Bug report #8487

manual CRS being overwritten by QGIS

Added by Gavin Fleming over 6 years ago. Updated almost 6 years ago.

Status:Closed
Priority:Severe/Regression
Assignee:-
Category:Projection Support
Affected QGIS version:master Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:Yes Resolution:fixed/implemented
Crashes QGIS or corrupts data:No Copied to github as #:17252

Description

The new series of EPSG codes for South Africa (Hartebeesthoek/LO) with +axis=wsu might be correct for land survey data in South Africa, which is 'south-facing', but it is NOT correct for the way GIS is practised in South Africa, where conventionally-oriented coordinates are used.

QGIS is now forcefully overriding my manual CRS definitions so I cannot work in our local projections any more.

I define a CRS as such: +proj=tmerc +lat_0=0 +lon_0=21 +k=1 +x_0=0 +y_0=0 +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs, which QGIS 1.8 and even 1.9 till recently obeyed. Now QGIS replaces this with +proj=tmerc +lat_0=0 +lon_0=21 +k=1 +x_0=0 +y_0=0 +axis=wsu +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs (note the added +axis=wsu) which is completely unusable since no GIS data in South Africa has 'south-facing' coordinates. The map is flipped upside down and back to front and I have no way of correcting it.

QGIS is apparently detecting that my definition is similar to EPSG:2049 and imposing the EPSG:2049 definition on my own.

test_4326.csv Magnifier - triangle point up in longlat (48 Bytes) Gavin Fleming, 2013-09-02 08:06 AM

test_2050.csv Magnifier - triangle point up in "lo23/hartebeesthoek" (epsg:2050), i.e. south-facing (209 Bytes) Gavin Fleming, 2013-09-02 08:06 AM

test_SACRS_NO_23.csv Magnifier - triangle point up in "lo23/hartebeesthoek north facing" i.e our new SACRS_NO_23 (215 Bytes) Gavin Fleming, 2013-09-02 08:06 AM

test_ZA_BB.csv Magnifier - South Africa bounding box (190 Bytes) Gavin Fleming, 2013-09-02 08:16 AM

History

#1 Updated by Jürgen Fischer over 6 years ago

Is that a custom crs or did you modify srs.db?

#2 Updated by Giovanni Manghi over 6 years ago

  • Operating System deleted (ubuntu)
  • OS version deleted (13.04)

Jürgen Fischer wrote:

Is that a custom crs or did you modify srs.db?

it is a custom crs, I tested and qgis does indeed overwrite the crs string.

#3 Updated by Frank Sokolic about 6 years ago

I've been able to solve this bug on my system (Ubuntu 13.04, QGIS master): If I open /usr/share/qgis/reources/srs.db using spatialite and then edit the relevant rows in the tbl_srs table to remove +axis=wsu then Bug #8487 goes away. Is this a possible solution to this bug?

#4 Updated by Jürgen Fischer about 6 years ago

Giovanni Manghi wrote:

it is a custom crs, I tested and qgis does indeed overwrite the crs string.

What exactly happens? I suppose it's not assigning the user CRS to the layer, when loading the shape file and assigns the "standard" crs instead. But it doesn't modify (overwrite) the existing user CRS and manually assigning the user crs works as expected. Right?

#5 Updated by Leyan Ouyang about 6 years ago

It is due to the changes I made in the custom CRS handling. Previously, it was all manual, then I changed it to make use of QgsCoordinateReferenceSystem. When creating a custom CRS, a QgsCoordinateReferenceSystem is created using createFromProj4, then is saved. The second string is the result of toProj4 on this CRS. To clarify, the custom CRS is not saved as expected, the modified version is saved instead. Manually changing would not help, but modifying the database can allow to replace with the correct value.

However, createFromProj4 is trying too hard to match to an existing CRS: it will match with an existing CRS, even if this CRS has additional parameters. It is even documented here

I think the clean way would be to make smaller utility class to manage CRS (check validity, convert between representations), separated from QgsCoordinateReferenceSystem which tries very hard to fit to existing CRS. However, I was very close to the feature freeze when I submitted this, and couldn't make the changes. I was not very happy with this matching to existing CRS, as it can rearrange some parameters, but I did not think it would make some custom CRS unusable ...

#6 Updated by Giovanni Manghi about 6 years ago

Jürgen Fischer wrote:

What exactly happens?

you create the custom CRS with

+proj=tmerc +lat_0=0 +lon_0=21 +k=1 +x_0=0 +y_0=0 +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs

then click ok, and when you go edit the same CRS it turns it is

+proj=tmerc +lat_0=0 +lon_0=21 +k=1 +x_0=0 +y_0=0 +axis=wsu +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs

#7 Updated by Frank Sokolic about 6 years ago

Yes, on my system (Ubuntu 13.04, QGIS Master) if I create a custom CRS, e.g:

+proj=tmerc +lat_0=0 +lon_0=31 +k=1 +x_0=0 +y_0=0 +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs

and then click ok, and then edit the same CRS then +axis=wsu has been added:

+proj=tmerc +lat_0=0 +lon_0=31 +k=1 +x_0=0 +y_0=0 +axis=wsu +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs

#8 Updated by Frank Sokolic about 6 years ago

For me, the issue is that the proj4 definitions in SRS.db for these South African CRSs are not correct and shouldn't contain the +axis=wsu parameter as GIS users in South Africa (and Namibia and Lesotho) don't use south-facing coordinates. If the +axis=wsu parameter was removed from the relevant entries in SRS.db then southern African GIS users wouldn't have to use custom CRSs at all.

#9 Updated by Hien TRAN-QUANG about 6 years ago

From what Leyan said, it seems that the error occurs when the custom CRS has too few parameters, compared to the found ones in the srs.db.

Shouldn't it only match the exact same number of parameters (apart maybe from +datum?) and not try to find a corresponding one, as it is contradictory to the purpose of a custom crs ?

#10 Updated by Hien TRAN-QUANG about 6 years ago

I did a pull request [[https://github.com/qgis/Quantum-GIS/pull/838]]
Now createFromProj4 only checks for same parameters (except +datum)

#11 Updated by René-Luc ReLuc about 6 years ago

  • Pull Request or Patch supplied changed from No to Yes
  • Category set to Projection Support
  • Target version set to Version 2.0.0
  • Status info set to Review requested

#12 Updated by Gavin Fleming about 6 years ago

I found a simpler solution though it wasn't easily apparent. It you specify +axis=enu explicitly (the normal north-oriented axes) in the Custom CRS dialog when creating a new custom CRS then QGIS accepts the definition you provide. It does not show the +axis=enu switch in the resulting definition but does implement it and does not override it with +axis=wsu. This does seem to be a good approach, i.e. to force explicit definition of axis orientation, but does need to be documented somewhere. I am accustomed to not specifying axis orientation but am quite happy to specify it explicitly from now on. If that is the case then +axis=enu should perhaps remain in the CRS definition.

If I edit an existing custom CRS that I had tried to change to north facing when I created it, but where +axis=wsu was forcefully inserted by the bug, the change becomes effective and can be seen in Project Properties but does not show in the Custom CRS dialog.

e.g. to make a north-facing version of EPSG:22287 I edited the CRS I had created (follow the +axis flag in these definitions):
+proj=tmerc +lat_0=0 +lon_0=27 +k=1 +x_0=0 +y_0=0 +axis=wsu +a=6378249.145 +b=6356514.966398753 +towgs84=-136,-108,-292,0,0,0,0 +units=m +no_defs.

I changed this to:
+proj=tmerc +lat_0=0 +lon_0=27 +k=1 +x_0=0 +y_0=0 +axis=enu +a=6378249.145 +b=6356514.966398753 +towgs84=-136,-108,-292,0,0,0,0 +units=m +no_defs.

In Project properties it now appears as the following and most importantly, it behaves correctly:
+proj=tmerc +lat_0=0 +lon_0=27 +k=1 +x_0=0 +y_0=0 +a=6378249.145 +b=6356514.966398753 +towgs84=-136,-108,-292,0,0,0,0 +units=m +no_defs

But if I open Custom CRS again it still shows:
+proj=tmerc +lat_0=0 +lon_0=27 +k=1 +x_0=0 +y_0=0 +axis=wsu +a=6378249.145 +b=6356514.966398753 +towgs84=-136,-108,-292,0,0,0,0 +units=m +no_defs

#13 Updated by craigleat - about 6 years ago

Frank Sokolic wrote:

For me, the issue is that the proj4 definitions in SRS.db for these South African CRSs are not correct and shouldn't contain the +axis=wsu parameter as GIS users in South Africa (and Namibia and Lesotho) don't use south-facing coordinates. If the +axis=wsu parameter was removed from the relevant entries in SRS.db then southern African GIS users wouldn't have to use custom CRSs at all.

I believe the definitions are correct and it would be improper to alter a standard definition to suit a historical non-standard practice. GIS users have traditionally worked with negative values but this situation only arose because existing software was unable to implement south facing coordinates. Civil Engineers on the other hand have been correctly using TMSO for years via local software products. If you wish to use TMNO this should be viable by creating a custom CRS.

#14 Updated by Tim Sutton about 6 years ago

There are a complex of issues, here I try to outline a roadmap to address them:

Incorrect Proj4 definitions

/usr/share/proj/epsg lists Cape data with the with the easting/northing order switched. In other words in all of theses `+axis=wsu` needs to become `+axis=swu`. There is alread a ticket (filed in proj4 for this http://trac.osgeo.org/proj/ticket/18).

# Cape / Lo15
<22275> +proj=tmerc +lat_0=0 +lon_0=15 +k=1 +x_0=0 +y_0=0 +axis=wsu +a=6378249.145 +b=6356514.966398753 +towgs84=-136,-108,-292,0,0,0,0 +units=m +no_defs  <>
# Cape / Lo17
<22277> +proj=tmerc +lat_0=0 +lon_0=17 +k=1 +x_0=0 +y_0=0 +axis=wsu +a=6378249.145 +b=6356514.966398753 +towgs84=-136,-108,-292,0,0,0,0 +units=m +no_defs  <>
# Cape / Lo19
<22279> +proj=tmerc +lat_0=0 +lon_0=19 +k=1 +x_0=0 +y_0=0 +axis=wsu +a=6378249.145 +b=6356514.966398753 +towgs84=-136,-108,-292,0,0,0,0 +units=m +no_defs  <>
# Cape / Lo21
<22281> +proj=tmerc +lat_0=0 +lon_0=21 +k=1 +x_0=0 +y_0=0 +axis=wsu +a=6378249.145 +b=6356514.966398753 +towgs84=-136,-108,-292,0,0,0,0 +units=m +no_defs  <>
# Cape / Lo23
<22283> +proj=tmerc +lat_0=0 +lon_0=23 +k=1 +x_0=0 +y_0=0 +axis=wsu +a=6378249.145 +b=6356514.966398753 +towgs84=-136,-108,-292,0,0,0,0 +units=m +no_defs  <>
# Cape / Lo25
<22285> +proj=tmerc +lat_0=0 +lon_0=25 +k=1 +x_0=0 +y_0=0 +axis=wsu +a=6378249.145 +b=6356514.966398753 +towgs84=-136,-108,-292,0,0,0,0 +units=m +no_defs  <>
# Cape / Lo27
<22287> +proj=tmerc +lat_0=0 +lon_0=27 +k=1 +x_0=0 +y_0=0 +axis=wsu +a=6378249.145 +b=6356514.966398753 +towgs84=-136,-108,-292,0,0,0,0 +units=m +no_defs  <>
# Cape / Lo29
<22289> +proj=tmerc +lat_0=0 +lon_0=29 +k=1 +x_0=0 +y_0=0 +axis=wsu +a=6378249.145 +b=6356514.966398753 +towgs84=-136,-108,-292,0,0,0,0 +units=m +no_defs  <>
# Cape / Lo31
<22291> +proj=tmerc +lat_0=0 +lon_0=31 +k=1 +x_0=0 +y_0=0 +axis=wsu +a=6378249.145 +b=6356514.966398753 +towgs84=-136,-108,-292,0,0,0,0 +units=m +no_defs  <>
# Cape / Lo33
<22293> +proj=tmerc +lat_0=0 +lon_0=33 +k=1 +x_0=0 +y_0=0 +axis=wsu +a=6378249.145 +b=6356514.966398753 +towgs84=-136,-108,-292,0,0,0,0 +units=m +no_defs  <>

# Hartebeesthoek94 / Lo15
<2046> +proj=tmerc +lat_0=0 +lon_0=15 +k=1 +x_0=0 +y_0=0 +axis=wsu +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs  <>
# Hartebeesthoek94 / Lo17
<2047> +proj=tmerc +lat_0=0 +lon_0=17 +k=1 +x_0=0 +y_0=0 +axis=wsu +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs  <>
# Hartebeesthoek94 / Lo19
<2048> +proj=tmerc +lat_0=0 +lon_0=19 +k=1 +x_0=0 +y_0=0 +axis=wsu +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs  <>
# Hartebeesthoek94 / Lo21
<2049> +proj=tmerc +lat_0=0 +lon_0=21 +k=1 +x_0=0 +y_0=0 +axis=wsu +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs  <>
# Hartebeesthoek94 / Lo23
<2050> +proj=tmerc +lat_0=0 +lon_0=23 +k=1 +x_0=0 +y_0=0 +axis=wsu +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs  <>
# Hartebeesthoek94 / Lo25
<2051> +proj=tmerc +lat_0=0 +lon_0=25 +k=1 +x_0=0 +y_0=0 +axis=wsu +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs  <>
# Hartebeesthoek94 / Lo27
<2052> +proj=tmerc +lat_0=0 +lon_0=27 +k=1 +x_0=0 +y_0=0 +axis=wsu +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs  <>
# Hartebeesthoek94 / Lo29
<2053> +proj=tmerc +lat_0=0 +lon_0=29 +k=1 +x_0=0 +y_0=0 +axis=wsu +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs  <>
# Hartebeesthoek94 / Lo31
<2054> +proj=tmerc +lat_0=0 +lon_0=31 +k=1 +x_0=0 +y_0=0 +axis=wsu +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs  <>
# Hartebeesthoek94 / Lo33
<2055> +proj=tmerc +lat_0=0 +lon_0=33 +k=1 +x_0=0 +y_0=0 +axis=wsu +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs  <>

Until such time as an upstream fix happens, it would be good to fix these locally in the srs.db

There are no north facing CRS definitions for the Lo Hartebeeshoek and the Lo Cape CRSs

There is currently not a ticket with EPSG to add these so for the interim it is suggested that they are added to the QGIS srsr.db with auth_name 'ZANGI' (South Africa National Geospatial Information [department]).

In these cases all of the above definitions would be duplicated but with +axis=enu (which is the default but we should set it explictly). We suggest the following schema for the auth_id we would use for 'Lo projections this: SACRS_NO_<XX> and for Hartebeeshoek projections this: TM_CAPE_NO_<XX> where NO = North Orientated and <xx> is the zone.

Test dataset

We should have a test dataset which can be used to verify that the CRS is properly detected. We propose two test datasets:

  • A bounding box for South Africa in EPSG:4326
  • A simple feature in SACRS_NO_23 which when repropected on the fly to EPSG:4326 intesects with the country bbox above.

EPSG formalised definitions

In the longer term, we should have EPSG definitions defined for the above SACRS and TM_CAPE crs's and then remove the locally added definitions so that they are automatically added from EPSG.

#15 Updated by craigleat - about 6 years ago

Tim,

For your proposed schema did you mean the following, as you appear to have them switched around?
Lo Hartebeeshoek94 -> SACRS_NO_<XX>
Lo Cape -> TM_CAPE_NO_<XX>

#16 Updated by Gavin Fleming about 6 years ago

@craigleat: no, Tim is correct - we are proposing a new set of north-facing (or NO=north-oriented) definitions in addition to the existing set of south-facing (SO) definitions, such that each existing 'LO' code has a north-facing equivalent.

I attach here some test files to use. Each has the same triangle polygon in longlat (test_4326), SACRS_23 or "lo23/Hartebeesthoek"(test_2050) and SACRS_NO_23 or "lo23/Hartebeesthoek north-facing" (test_SACRS_NO_23).

These can be tested manually by loading with the delimited text loader and specifying the appropriate CRS and enabling on the fly CRS transformation. When all is working, all three polygons should coincide in the middle of South Africa.

When the bug that this ticket refers to is fixed and a new definition has been created, then test_SACRS_NO_23 should overlay 4326.

When the +axis flag in the existing definition has been changed from +axis=wsu to +axis=swu then test_SACRS_NO_23 should overlay 4326.

#17 Updated by Gavin Fleming about 6 years ago

#18 Updated by Gavin Fleming about 6 years ago

  • Priority changed from Severe/Regression to High

Let's hang five on Tim's "Incorrect project definitions" above. Technically they are correct according to the EPSG but the jury is out on which one of swu or wsu is the predominant axis order in practice in South Africa. Perhaps we should leave them as is and tell users they must conform with EPSG convention, until we can confirm otherwise. If we confirm in due course that swu is the local convention, then we should make the change.

Regarding the new north-facing SACRS ('ZANGI') CRS definitions, those are fine as Tim has described.

#19 Updated by Gavin Fleming about 6 years ago

  • Priority changed from High to Severe/Regression

I mistakenly downgraded this earlier - the original bug this Issue describes is still a blocker until resolved.

#20 Updated by Tim Sutton about 6 years ago

Further to this, I have will insert the following CRS definitions into the srs.db. Due to the last minute nature of this commit, any corrections to these will need to be made in future release of QGIS 2.x

<quote>
INSERT INTO "tbl_ellipsoid" VALUES;

INSERT INTO "tbl_srs" VALUES;
INSERT INTO "tbl_srs" VALUES;
INSERT INTO "tbl_srs" VALUES;
INSERT INTO "tbl_srs" VALUES;
INSERT INTO "tbl_srs" VALUES;
INSERT INTO "tbl_srs" VALUES;
INSERT INTO "tbl_srs" VALUES;
INSERT INTO "tbl_srs" VALUES;
INSERT INTO "tbl_srs" VALUES;
INSERT INTO "tbl_srs" VALUES;

INSERT INTO "tbl_srs" VALUES;
INSERT INTO "tbl_srs" VALUES;
INSERT INTO "tbl_srs" VALUES;
INSERT INTO "tbl_srs" VALUES;
INSERT INTO "tbl_srs" VALUES;
INSERT INTO "tbl_srs" VALUES;
INSERT INTO "tbl_srs" VALUES;
INSERT INTO "tbl_srs" VALUES;
INSERT INTO "tbl_srs" VALUES;
INSERT INTO "tbl_srs" VALUES;
</quote>

Notes on acronyms used:

  • ZANGI - South Africa National Geospatial Information [Dept]
  • CPNO<xx> (e.g. CPNO33) - Cape Datum North Oriented on centered longitude 33
  • HBKNO<xx> (e.g. HBKNO33) - Hartebeeshoek North Oriented on centered longitude 33
  • NO_<xx> (e.g. NO_15) - North Oriented on longitude 15

#21 Updated by Tim Sutton about 6 years ago

14f639d Adds the above CRS's

#22 Updated by Giovanni Manghi about 6 years ago

  • Target version changed from Version 2.0.0 to Future Release - High Priority

#23 Updated by Gavin Fleming about 6 years ago

bug appears to be fixed in 2.1.0 Master (though the new north-oriented CRS definitions Tim added don't appear yet)

#24 Updated by Giovanni Manghi almost 6 years ago

  • Status changed from Open to Feedback

Gavin Fleming wrote:

bug appears to be fixed in 2.1.0 Master (though the new north-oriented CRS definitions Tim added don't appear yet)

can we close this?

#25 Updated by Gavin Fleming almost 6 years ago

Yes you can close it, thanks all. The original issue as well as the related issues brought up in the discussion are all fixed in master.

#26 Updated by Giovanni Manghi almost 6 years ago

  • Status changed from Feedback to Closed
  • Resolution set to fixed/implemented

Also available in: Atom PDF