Bug report #5598

CRS attribution algorithm for newly added layers broken in qgis-master (upcoming 1.8)

Added by Mathieu Pellerin - nIRV over 7 years ago. Updated over 7 years ago.

Status:Closed
Priority:Severe/Regression
Assignee:-
Category:Projection Support
Affected QGIS version:master Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:No Resolution:fixed
Crashes QGIS or corrupts data:No Copied to github as #:15172

Description

Something's seriously wrong with CRS attribution for newly added layers in qgis-master (i.e. soon to be 1.8). I have no idea what is happening, however the steps below will reproduce the issue:

1) Add moeareas.shp to a QGIS project
2) Open the Properties dialog and go to the General tab
3) Take note of the custom CRS applied to the layer

The above shapefile's coordinate reference system is EPSG:3148, defined as follow:
"+proj=utm +zone=48 +a=6377276.345 +b=6356075.41314024 +towgs84=198,881,317,0,0,0,0 +units=m +no_defs"

The custom CRS created by qgis 1.8 when loading the above shapefile:
"+proj=utm +zone=48 +a=6377276.345 +b=6356075.41314024 +units=m +no_defs"

While qgis 1.7.x will correctly attribute the newly loaded layer its proper CRS (EPSG:3148), qgis 1.8 creates the custom CRS with the exact same projection settings (minus the +towgs84). Something must be wrong in the CRS attribution algorithm. This issue leads to misalignment with on-the-fly re-projection projects in which wgs layers & utm layers are mixed together.

While I'm initially setting the priority as high, the issue should most probably considered a blocker as it's a pretty serious regression from 1.7.x. It's present on windows and linux

MoEAreas.zip (148 KB) Mathieu Pellerin - nIRV, 2012-05-14 07:26 PM

crstest.patch Magnifier (2.76 KB) Etienne Tourigny, 2012-05-29 01:59 PM


Related issues

Related to QGIS Application - Bug report #5066: (regression) QGIS 1.7.* misdetects EPSG::25884 datasets a... Closed 2012-02-22

History

#1 Updated by Mathieu Pellerin - nIRV over 7 years ago

  • Category set to Projection Support
  • Priority changed from High to Severe/Regression
  • Target version set to Version 1.8.0
  • Operating System set to all

#2 Updated by cgsbob - over 7 years ago

I have the same problem as you. I believe it is a problem with gdal. testepsg shows the towgs84 is being stripped off:

$ testepsg   moeareas.prj 
Validate Succeeds.
WKT[moeareas.prj] =
PROJCS["Indian_1960_UTM_Zone_48N",
    GEOGCS["GCS_Indian_1960",
        DATUM["D_Indian_1960",
            SPHEROID["Everest_Adjustment_1937",6377276.345,300.8017]],
        PRIMEM["Greenwich",0.0],
        UNIT["Degree",0.0174532925199433]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["False_Easting",500000.0],
    PARAMETER["False_Northing",0.0],
    PARAMETER["Central_Meridian",105.0],
    PARAMETER["Scale_Factor",0.9996],
    PARAMETER["Latitude_Of_Origin",0.0],
    UNIT["Meter",1.0]]

Simplified WKT[moeareas.prj] =
PROJCS["Indian_1960_UTM_Zone_48N",
    GEOGCS["GCS_Indian_1960",
        DATUM["D_Indian_1960",
            SPHEROID["Everest_Adjustment_1937",6377276.345,300.8017]],
        PRIMEM["Greenwich",0.0],
        UNIT["Degree",0.0174532925199433]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["False_Easting",500000.0],
    PARAMETER["False_Northing",0.0],
    PARAMETER["Central_Meridian",105.0],
    PARAMETER["Scale_Factor",0.9996],
    PARAMETER["Latitude_Of_Origin",0.0],
    UNIT["Meter",1.0]]

Old Style WKT[moeareas.prj] = PROJCS["Indian_1960_UTM_Zone_48N",GEOGCS["GCS_Indian_1960",DATUM["D_Indian_1960",SPHEROID["Everest_Adjustment_1937",6377276.345,300.8017]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Transverse_Mercator"],PARAMETER["False_Easting",500000.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",105.0],PARAMETER["Scale_Factor",0.9996],PARAMETER["Latitude_Of_Origin",0.0],UNIT["Meter",1.0]]
ESRI'ified WKT[moeareas.prj] = 
PROJCS["Indian_1960_UTM_Zone_48N",
    GEOGCS["GCS_Indian_1960",
        DATUM["D_Indian_1960",
            SPHEROID["Everest_Adjustment_1937",6377276.345,300.8017]],
        PRIMEM["Greenwich",0.0],
        UNIT["Degree",0.017453292519943295]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["False_Easting",500000.0],
    PARAMETER["False_Northing",0.0],
    PARAMETER["Central_Meridian",105.0],
    PARAMETER["Scale_Factor",0.9996],
    PARAMETER["Latitude_Of_Origin",0.0],
    UNIT["Meter",1.0]]
PROJ.4 rendering of [moeareas.prj] = +proj=utm +zone=48 +a=6377276.345 +b=6356075.41314024 +units=m +no_defs 

#3 Updated by Giovanni Manghi over 7 years ago

I can confirm the issue with the provided shapefile (tested on Windows/osgei4w) as actually I don't have a Linux machine with qgis and gdal 1.9

I have vectors in other CRSs with towg84 parameters and the issue is not surfacing.

The issue sounds a lot like this #5142 filed by me, where I arrived at the conclusion that there was something strange going on if QGIS is not compiled against gdal 1.9, but obviously this case is different as the tests were made using qgis compiled against gdal 1.9.

#4 Updated by Mathieu Pellerin - nIRV over 7 years ago

I don't think it's a gdal issue here: on a windows machine with both qgis (1.7.4) and qgis-dev (to-be 1.8) installed, qgis 1.7.4 assigns the right EPSG to the newly loaded layer, while qgis 1.8 fails. I believe both versions are compiled using the same gdal 1.9 library.

#5 Updated by Mathieu Pellerin - nIRV over 7 years ago

Also, regarding testepsg not showing the +towgs84 value, I believe this is not linked to the issue raised here. The abscence of +towgs84 in testepsg (or gdalsrsinfo) is a problem present in gdal (and discussing here: http://trac.osgeo.org/gdal/ticket/4345) but has not prevent previous version of qgis from assigning the right srs to loaded layers.

#6 Updated by Salvatore Larosa over 7 years ago

It seem depends on the PRJ file definition!
If you use the definition as below [1], QGIS acknowledges the correct reference system (3148)!

How you created the shp?
which software you used?

[1] - http://spatialreference.org/ref/epsg/3148/

PROJCS["Indian 1960 / UTM zone 48N",GEOGCS["Indian 1960",DATUM["D_Indian_1960",SPHEROID["Everest_1830_1937_Adjustment",6377276.345,300.8017]],PRIMEM["Greenwich",0],UNIT["Degree",0.017453292519943295]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",105],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["Meter",1]]

#7 Updated by Giovanni Manghi over 7 years ago

Salvatore Larosa wrote:

It seem depends on the PRJ file definition!
If you use the definition as below [1], QGIS acknowledges the correct reference system (3148)!

How you created the shp?
which software you used?

[1] - http://spatialreference.org/ref/epsg/3148/

[...]

but the bottom line is that qgis 1.7.4 reads correctly also the provided prj

#8 Updated by Jürgen Fischer over 7 years ago

Giovanni Manghi wrote:

but the bottom line is that qgis 1.7.4 reads correctly also the provided prj

In the same setup (GDAL and maybe PROJ.4)?

#9 Updated by Giovanni Manghi over 7 years ago

In the same setup (GDAL and maybe PROJ.4)?

yes, on osgeo4w

#10 Updated by Salvatore Larosa over 7 years ago

Also, I noticed that in PROJ4.7 EPSG 3148 is defined:

# Indian 1960 / UTM zone 48N
<3148> +proj=utm +zone=48 +a=6377276.345 +b=6356075.41314024 +units=m +no_defs  <>

instead in PROJ4.8 is:

# Indian 1960 / UTM zone 48N
<3148> +proj=utm +zone=48 +a=6377276.345 +b=6356075.41314024 +towgs84=198,881,317,0,0,0,0 +units=m +no_defs  <>

#11 Updated by Etienne Tourigny over 7 years ago

The problem is that OGR's MorphFromESRI() wkt fixing code (see gdal bug #4345) does not work with this file, because of a mismatch in SPHEROID name - "Everest 1830 (1937 Adjustment)" vs. "Everest_Adjustment_1937" despite identical spheroid parameters.

$ gdalsrsinfo moeareas.shp

PROJ.4 : '+proj=utm +zone=48 +a=6377276.345 +b=6356075.41314024 +units=m +no_defs '

OGC WKT :
PROJCS["Indian_1960_UTM_Zone_48N",
    GEOGCS["GCS_Indian_1960",
        DATUM["Indian_1960",
            SPHEROID["Everest_Adjustment_1937",6377276.345,300.8017]],
        PRIMEM["Greenwich",0.0],
        UNIT["Degree",0.0174532925199433]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["False_Easting",500000.0],
    PARAMETER["False_Northing",0.0],
    PARAMETER["Central_Meridian",105.0],
    PARAMETER["Scale_Factor",0.9996],
    PARAMETER["Latitude_Of_Origin",0.0],
    UNIT["Meter",1.0]]

[email protected]: /data/research/work/qgis/issues/5598 $ gdalsrsinfo EPSG:3148

PROJ.4 : '+proj=utm +zone=48 +a=6377276.345 +b=6356075.41314024 +towgs84=198,881,317,0,0,0,0 +units=m +no_defs '

OGC WKT :
PROJCS["Indian 1960 / UTM zone 48N",
    GEOGCS["Indian 1960",
        DATUM["Indian_1960",
            SPHEROID["Everest 1830 (1937 Adjustment)",6377276.345,300.8017,
                AUTHORITY["EPSG","7015"]],
            TOWGS84[198,881,317,0,0,0,0],
            AUTHORITY["EPSG","6131"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4131"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",105],
    PARAMETER["scale_factor",0.9996],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["Easting",EAST],
    AXIS["Northing",NORTH],
    AUTHORITY["EPSG","3148"]]

Created ticket in gdal : http://trac.osgeo.org/gdal/ticket/4673 , and commited changes to gdal which fix this issue. The fix is to compare SPHEROID parameters rather than node names. However, it is unlikely it will be in gdal 1.9.1 which is around the corner.

QGis 1.7 was not restrictive enough in comparing spatial reference definitions, which is why it works, despite the lack of TOWGS84 node in the .prj files.
It "works" because all other parameters are equal, but for some reason does not enforce strictly equal +towgs84 parameters (perhaps because they are missing in the definition).

This needs a little more investigation to find how it does recognize the srid with towgs84 parameters in 1.7 and not in 1.8.

#12 Updated by Etienne Tourigny over 7 years ago

Found a solution - the parameters passed to sql query must be trimmed.

This is a regression caused by 2a830339ed5fae9e6002b7d56442991fe1a18a8c "split proj4string on spaces followed by '+' (instead of spaces only).
see http://trac.osgeo.org/proj/ticket/132"

See my pull request: https://github.com/qgis/Quantum-GIS/pull/142

#13 Updated by Giuseppe Sucameli over 7 years ago

Happy to know that my commit doesn't work as I expected :)

But I really don't understand why trimming the string fixes the problem...

The string is splitted on space(s) followed by '+' sign, where spaces are in the param?

#14 Updated by Giuseppe Sucameli over 7 years ago

Giuseppe Sucameli wrote:

The string is splitted on space(s) followed by '+' sign, where spaces are in the param?

Found, if the last param ends with a space that space is kept and passed to the query.

#15 Updated by Etienne Tourigny over 7 years ago

in this case (and probably most cases, it's the +nodefs . Most (all?) proj strings end with a space:

'+proj=longlat +datum=WGS84 +no_defs '

#16 Updated by Giuseppe Sucameli over 7 years ago

  • Resolution set to fixed
  • Status changed from Open to Closed

The pull request (by Etienne) was merged in master branch.

#17 Updated by Giuseppe Sucameli over 7 years ago

Probably we should go deeper and search for the function that adds (or not removes) that trailing space, otherwise the problem may come back and break anything else.

I.e. trimming the user input, the OSRExportToProj4 output, and so on...

#18 Updated by Etienne Tourigny over 7 years ago

OSRExportToProj4 always inserts a trailing space, and we should consider that user input could have a trailing space, so you a right about that!

#19 Updated by Etienne Tourigny over 7 years ago

I have addressed your concern, and also added a test which should point out any new regressions.

https://github.com/qgis/Quantum-GIS/pull/144

#20 Updated by Mathieu Pellerin - nIRV over 7 years ago

Etienne, thanks for looking into this.

I've compiled the latest revision of qgis-master, which includes your commit, and I still can't get proper behavior with the shape file I attached to this issue. Did you check with the attached shape file whether your change fixed the issue on your computer?

#21 Updated by Etienne Tourigny over 7 years ago

Hi

this is probably because you already have that user-defined crs in your personal srs.db . This happened to me when testing - yes I did test your file.

Please delete $HOME/.qgis/qgis.db (or rename it) and try again. If not compile qgis with debug support and attach the log here.

#22 Updated by Etienne Tourigny over 7 years ago

Just saw this - you can delete the user-defined crs from your personal db with Settings-Custom CRS

#23 Updated by Mathieu Pellerin - nIRV over 7 years ago

I've done that, as well as removing the custom crs from the recently used crs. QGIS still ends up creating a custom CRS. I've noticed your pull request #144 isn't merged into qgis-master yet. Is it required to properly fix the issue?

#24 Updated by Etienne Tourigny over 7 years ago

What gdal version are you using? works here with gdal 1.9, even before changes in pull request 144.

#25 Updated by Mathieu Pellerin - nIRV over 7 years ago

Etienne, I've just tested a osgeo4w QGIS windows build (based on c64b715) and - yay - the attached shapefile is attributed the proper CRS.

Yesterday, the issue was still present on a QGIS linux build (based on 1b6b841545) which I had built myself. The gdal version I used was 1.9 (as confirmed in the QGIS about dialogue). However, I will try to verify this on a ubungis build today.

Are you using a linux or windows build?

#26 Updated by Etienne Tourigny over 7 years ago

I use and dev on linux, and sometimes test on windows. I will post another, permanent fix today(for stray buggy GEOGCS files) with more unit tests so stay tuned.

Again - it should work in linux now (as it does in the osgeo4w build), so re-check your build. I think there is a nightlybuild you can download and test.

https://issues.qgis.org/projects/quantum-gis/wiki/Download

#27 Updated by Etienne Tourigny over 7 years ago

I have commited more changes to my pull request. Once commited and built, I will send a mailing list email toask people to test, because there is a change that might have negative impacts in some cases.

I found a few more issues, notably with GEOGCS nodes only. Using gdal-1.8 they are not loaded completely, I don't see how to fix that as this is external to QGis (try them with testepsg...). So I set the tests as "Expected Failure" when gdal<1.9 for those. The solution for these problems is "upgrade to gdal 1.9".

The best solution I see is to use GDAL_FIX_ESRI_WKT=GEOGCS instead of TOWGS84 because gdal-1.9.1 has some bugs when using TOWGS84. These have been fixed, but will not make it in gdal-1.9.1. The downside is that GEOGCS changes more things than TOWGS84 and this might change the CRS. See http://trac.osgeo.org/gdal/ticket/4673 for a follow-up.

I also added the possibility to set GDAL_FIX_ESRI_WKT externally (it's a CPL option), so this could be used to resolve specific issues. For example, if GEOGCS has a negative impact, user can add GDAL_FIX_ESRI_WKT=TOWGS84 to environment.

Example GEOGCS definitions which do not work without this fix with gdal-1.8:

EPSG:4131
GEOGCS["GCS_Indian_1960",DATUM["D_Indian_1960",SPHEROID["Everest_Adjustment_1937",6377276.345,300.8017]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]]

EPSG:4618
GEOGCS["GCS_South_American_1969",DATUM["D_South_American_1969",SPHEROID["GRS_1967_Truncated",6378160.0,298.25]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]]

#28 Updated by Mathieu Pellerin - nIRV over 7 years ago

Etienne, I still see this issue on my linux machine. I've tried both a ubuntugis nightly build, as well as building qgis myself. In both linux builds, the issue remains.

Libraries used by the two builds are: gdal 1.9.0 (offered on ubuntugis ppa) and proj 4.7 (ubuntu package).

#29 Updated by Giovanni Manghi over 7 years ago

  • Status changed from Closed to Feedback

nirvn - wrote:

Etienne, I still see this issue on my linux machine. I've tried both a ubuntugis nightly build, as well as building qgis myself. In both linux builds, the issue remains.

Libraries used by the two builds are: gdal 1.9.0 (offered on ubuntugis ppa) and proj 4.7 (ubuntu package).

this needs to be reopened? plese leave feedback asap as this is/was a blocker.

#30 Updated by Etienne Tourigny over 7 years ago

as I said in the ML, please re-open this bug as I found and fixed other issues, and it seems that it didn't work for nirvn.

The fixes in my pull request should work for him : https://github.com/qgis/Quantum-GIS/pull/144

#31 Updated by Giovanni Manghi over 7 years ago

  • Resolution deleted (fixed)
  • Status changed from Feedback to Open
  • Operating System deleted (all)

#32 Updated by Etienne Tourigny over 7 years ago

Jurgen has just committed the changes I pushed. Nirv - if you can update your master and test it would be nice. If you have any problems please post a debug log here.

#33 Updated by Etienne Tourigny over 7 years ago

Autotests are failing in i386 linuxes because of a difference in the precision of the +b parameters (see the mailing list). Attached patch should work fine in all architectures, as it only looks for towgs84 parameter instead of matchin proj.4 string.

#34 Updated by Etienne Tourigny over 7 years ago

I've tested latest ubuntugis-nightly using ubuntu 11.10 AMD64 (gdal 1.9.0)

There might be an issue with i386 linux (because of a difference in the +b parameter of the PROJ.4 string).

Can someone please test the latest ubuntu-nightly with a 32 bit linux (with gdal-1.9)? This is the last blocker, so it would be nice to clear it!

Thanks

#35 Updated by Etienne Tourigny over 7 years ago

Also ok in the osgeo4w build (i386 VM)

#36 Updated by Mathieu Pellerin - nIRV over 7 years ago

OMG, it's now working, yay! Both on osgeo4w build and ubuntugis nightly ppa.

Kudos Etienne. Leaving you the honor of closing this bug. :)

#37 Updated by Etienne Tourigny over 7 years ago

That's great to know! I'd like to close it but redmine won't let me!

Can somebody please give me "Edit issues" permissions for the QGis or quantum-gis redmine project?

#38 Updated by Giovanni Manghi over 7 years ago

Etienne Tourigny wrote:

That's great to know! I'd like to close it but redmine won't let me!

Can somebody please give me "Edit issues" permissions for the QGis or quantum-gis redmine project?

try now (logout before)

#39 Updated by Etienne Tourigny over 7 years ago

  • % Done changed from 0 to 100
  • Status changed from Open to Closed
  • Resolution set to fixed

thanks

Also available in: Atom PDF