Bug report #3099

Projection Failure

Added by badams - over 13 years ago. Updated over 10 years ago.

Status:Closed
Priority:Low
Assignee:-
Category:Projection Support
Affected QGIS version:master Regression?:No
Operating System:All Easy fix?:No
Pull Request or Patch supplied:No Resolution:
Crashes QGIS or corrupts data:No Copied to github as #:13159

Description

QGIS does not operate with Australian Datums AGD66, AGD84 and GDA94. Although it will allow the selection of these datums, they are not applied to the data upon importing instead defaulting to WGS84. In the case of GDA94 V's AGD84 this results in an approx 200m offset error.

Similarly when exporting the data to either Shapefiles or Mapinfo TAB files, the projection is not sticking maintaining the default WGS84.

Roland Hill and Ken Norris(QGIS users) figured out the projection problem lies within the SRS.DB file and the Australia distortion grids (AGD66.gsb and AGD84.gsb), possibly indicating the problem lies with Proj4. I have attached Roland's edited files, he was able to make these work on the Linux platform but they didn't work for me on Windows 7. Not sure if he was able to get the projections to stick, I never did.

Attached CSV file contains data points in AGD66, AGD84, GDA94 and WSG84. This is the same grid in different datums. Note the AGD66 datum uses the "AGD66-Australia-GDA94" Local Datum Transformation. There are at least 10 other AGD66 - Local Datum Transformations however only AGD66-Australia-GDA94 covers all of Australia (or so I believe).

Brett.csv Magnifier - Data with different projections (29.7 KB) badams -, 2010-10-09 12:33 AM

project.zip (112 KB) Giovanni Manghi, 2010-10-10 01:39 AM

Screenshot.png (61.3 KB) Giovanni Manghi, 2010-10-10 01:39 AM

Screenshot-2.png (171 KB) Giovanni Manghi, 2010-10-10 01:40 AM

Associated revisions

Revision c1793662
Added by Jürgen Fischer over 13 years ago

[FEATURE] allow user specific PROJ.4 search paths and update srs.db to include grid reference (fixes #3099)

git-svn-id: http://svn.osgeo.org/qgis/trunk/qgis@14399 c8812cc2-4d05-0410-92ff-de0c093fc19c

Revision 6308416a
Added by Jürgen Fischer over 13 years ago

[FEATURE] allow user specific PROJ.4 search paths and update srs.db to include grid reference (fixes #3099)

git-svn-id: http://svn.osgeo.org/qgis/trunk@14399 c8812cc2-4d05-0410-92ff-de0c093fc19c

History

#1 Updated by Giovanni Manghi over 13 years ago

Did you created the necessary custom CRSs in order to make the ntv2 grid transformation?

the source SRS must include both the "+nadgrids" paramater as well the "+wktext"

the tagert SRS must include (if is a projected CRS) "+towgs84=0,0,0"

Did you copied the necessary .gsb file in the right place in your windows installation?

otherwise the grid transformation will not be correctly done, and will "fail" silently.

I tested the system on windows Seven with data and grids for Portugal, and it works. You also must be sure that the geographic area of your vectors in actually covered by the grid you are using.

#2 Updated by badams - over 13 years ago

Sorry, Remaining files are too big to attach. They can be downloaded at;

http://dl.dropbox.com/u/9546967/srs.db
http://dl.dropbox.com/u/9546967/agd66.gsb
http://dl.dropbox.com/u/9546967/agd84.gsb

Additional information "Geocentric Datum of Australia
Technical Manual"
http://www.icsm.gov.au/gda/gdatm/gdav2.3.pdf.

#3 Updated by Giovanni Manghi over 13 years ago

Can you attach data so I can make a test?

#4 Updated by Giovanni Manghi over 13 years ago

Additional information "Geocentric Datum of Australia
Technical Manual"
http://www.icsm.gov.au/gda/gdatm/gdav2.3.pdf.

too big, don't have time to study such document, just tell ma what are the source/target srs, the appropriate grid and attach some data.

#5 Updated by badams - over 13 years ago

Additional Projection information

Universal Transverse Mercator
AGD66 / AMG zone51
AGD84 / AMG zone51
GDA94 / MGA zone51
WGS84 / UTM zone51S
Sorry, I'll get the information in eventually. Brett

#6 Updated by Giovanni Manghi over 13 years ago

a ntv2 grid is appropriate for just one kind of transformation, from one specific datum to another specific one. I'll need to know them (along with vectors), if you want me to test here.

#7 Updated by badams - over 13 years ago

Sorry Lutra, I actually have no idea what your talking about. I'm way out of my depth here.

The test is simple. The CSV file contains the same grid, listed in 4 different datums, UTM: AGD66, AGD84, GDA94 and WGS84. The projections already exist in QGIS, you don't need to change anything. Forget about the srs.db and gsb files for the moment.

If it worked properly, all 4 datums would plot in the same place.

#8 Updated by Giovanni Manghi over 13 years ago

The projections already exist in QGIS, you don't need to change anything. Forget about the srs.db and gsb files for the moment.

If it worked properly, all 4 datums would plot in the same place.

The "re-projection" already exist in QGIS, right, but without transformation parameters such towgs84 or nadgrids, not all datum transformations will give good results. This is why in such situations you must use towgs84 or nadgrids parameters. Proj4 coordinate system definitions come without such parameters, so you need to add them manually.

Can you please specify, in terms of EPSG codes, what are the coordinate systems of the point grids in the csv file (beside obviously wgs84 that has the 4326 code)? thanks

#9 Updated by badams - over 13 years ago

Lutra, I think these are the codes your chasing

EPSG codes as listed in QGIS are as follows;

AGD66 / AMG zone51: 20251
AGD84 / AMG zone51: 20351
GDA94 / MGA zone51: 28351
WGS84 / UTM zone51S: 32651

I suspect ESPG: 4326 is a lat/long projection.

Brett

#10 Updated by Giovanni Manghi over 13 years ago

Ok, this is a start. Now I just miss to know what kind of transformation are supposed to do the grids (the gsb files) you linked. Each grid is good for one kind of datum transformation.

#11 Updated by badams - over 13 years ago

Sorry Lutra, I don't know how the transformation process works. Can't help you with that.

#12 Updated by Giovanni Manghi over 13 years ago

Supposing that the grids you linked are the right ones for the the geographic area of your interest (amg zone 51, but I don't think so, especially after reading this http://www.mail-archive.com/[email protected]/msg07255.html in particular http://www.mail-archive.com/[email protected]/msg07363.html and following answers), the transformation works like this

ogr2ogr -f "ESRI Shapefile" -s_srs "+proj=utm +zone=51 +south +ellps=aust_SA +units=m +no_defs +nadgrids=agd66.gsb +wktext" -t_srs "+proj=utm +zone=51 +south +ellps=GRS80 +towgs84=0,0,0 +units=m +no_defs" adg66_to_gda94.shp agd66_51.shp

where:

*) ogr2ogr is the program that underlies QGIS (and many other software) to handle vectors and make things like coordinate transformations

*) s_srs is the source coordinate system, here you need to add "+nadgrids=agd66.gsb" and "+wktext" and where "agd66.gsb" is the appropriate grid file you need to copy in the right folder (it depends on the OS and, under Windows, depends also in what kind of installation/program are you using)

*) t_srs is the target coordinate system, if is a projected one and you want the grid transformation to work you need to add always "+towgs84=0,0,0 "

Then you indicate the target shape and the source one. The source shape was obtained with the csv file you provided and saving the import result as shapefile with the appropriate CRS.

Obviously in the above example the operation is made by hand, with the CLI, very handy if you need to transform bunches of shapefiles, but it can be done obviously also with QGIS.

But let see before the results (see attachment).

after transforming the points grid from adg66 to gda94, using the ntv2 grid, the resulting grid (in gda94 crs) is just 1.3 meters away from the original grid in gda94, obtained by the coordinates in the csv file.

Usually grid transformations give better results, this is another reasons why I think that the grid you provided is not the right one for the geographic area of your sample data.

Back to QGIS:

the boring problem in QGIS is that in none of the GUIs that allow you to save a copy of a vector is available the field to let you choose the source SRS (I have already filed a enhancement ticket) and/or let the user add +nadgrids and +towgs84 parameters. So what you need to do:

create a custom CRS, to be used as source SRS

proj=utm +zone=51 +south +ellps=aust_SA +units=m +no_defs +nadgrids=agd66.gsb +wktext

save a copy of your source vector ("save as...") and give to it this custom coordinate system.

save again this "intermediate" vector and give to it the target coordinate system, in this case

"+proj=utm +zone=51 +south +ellps=GRS80 +towgs84=0,0,0 +units=m +no_defs" 

and don't forget to check if the Proj definition in the QGIS database included "+towgs84=0,0,0" , if not you'll need to create a custom CRS also for the target coordinate system.

It seems complex, but it is not. The method works, and the results are the same under Linux and Windows, by hand or with QGIS.

Now one question and one problem. Is the precision of the results the one you were looking for? If not you than probably need to find a more appropriate ntv2 grid.

The problem: while doing the transformation with QGIS under windows Seven (osgeo4w/trunk) I'm getting some warnings that I don't get under Linux. So I'll need to investigate more.

Open the attached project and tell if it makes sense to you.

#13 Updated by badams - over 13 years ago

Lutra,
I'll need some time to digest this but I'll have a go.

The proposed projections are correct. I've been playing with them for the last 15 years. The link below will take to the Geoscience Australia site where they are explained fully.

http://www.ga.gov.au/geodesy/datums/cosys.jsp

Thanks
Brett

#14 Updated by badams - over 13 years ago

Hi Lutra,
Sorry for the slow response. Needed to run some trials.

The 1.3m variation between your conversion and mine is acceptable. Ran some trials with a professional Mapinfo user and we able to get a similar variation out of Mapinfo depending on the transforms selected so I'm happy the parameters you have are correct.

As I don't have the understanding, or the time, to implement these changes myself, I'm going to put some $$$ up for a bug fix. I'll run it past Paolo Cavallini first but I need a "plug and play" solution. Ideally the results will become part of QGIS so that anyone working on Australian data will benefit.

Given you understand the problem, can I ask you to help me put the bug fix proposal together?

Thanks
Brett

#15 Updated by Giovanni Manghi over 13 years ago

Hi Brett, I'm resuming what would be needed for both the vector and raster dialogs. see here

https://trac.osgeo.org/qgis/ticket/2913#comment:3

#16 Updated by Giovanni Manghi over 13 years ago

see also #3118

#17 Updated by Jürgen Fischer over 13 years ago

6308416a (SVN r14400) add the possibility for user defined proj.4 paths and updates the srs.db to include references to the australian GSBs. The GSB themselves are not included. The user needs to either add them to the standard PROJ.4 directory or add a user defined path.

#18 Updated by badams - over 13 years ago

I loaded . The 4 vector projections now import in the correct location so thats a good start.

Projections are not sticking for a shape or mapinfo export. They seem to reload in the right place, just with the wrong projection attached.

#19 Updated by badams - over 13 years ago

How is this bug fix progressing? I haven't seen any updates for a long time.

#20 Updated by Giovanni Manghi over 13 years ago

no idea, meanwhile I compiled trunk and the indirect methods*** do still work.

***
copy the ntv2 grids in the proper system folder, create the proper custom CRS then use them for datum transformation (for vectors a middle passage is needed due to the lack of the "source srs" filed in the "save as..." dialogs).

#21 Updated by ramon . about 13 years ago

as at 1.7 a2020cbe (SVN r15531)

From a quick test with the dev version isn't (a2020cbe (SVN r15531)) and a set of single point shape files in AMGz51, MGAz51 and WGSz51s.
- the AMGz51 point won't display in MGAz51 or WGSz51s.
QGIS puts up a window saying "Could not draw ShapeFile because:" and not giving a reason.
The debug information says: "failed with error: failed to load datum shift file"

- the MGAz51 and WGSz51s points won't display in AMGz51 and fail with the same set of errors.

More debug available if anyone wants it.

(and this bug exists in my unbuntu install too).

#22 Updated by ramon . almost 13 years ago

If the following files:
ttp://www.icsm.gov.au/icsm/gda/gdatm/national66.zip
http://www.icsm.gov.au/icsm/gda/gdatm/national84.zip
are downloaded, and unzipped then renamed to agd66.gsb and agd84.gsb respectively and copied in the the projections folderr1 it appears to work.

If they could be included in the distribution I'd consider this fixed.

(any issues about these files not being recognised by other GIS programmes probably ought to be in a different bug)

r1 /usr/share/proj or C:/osge4w/share/proj or C:/Program Files/Quantum Version/share/proj and equivalent on a mac.

#23 Updated by Giovanni Manghi over 12 years ago

  • Target version changed from Version 1.7.0 to Version 1.7.4

#24 Updated by Bill Williamson about 12 years ago

  • Crashes QGIS or corrupts data set to No
  • Affected QGIS version set to master

A US$25 Bounty on this bug

#25 Updated by Giovanni Manghi about 12 years ago

  • Tracker changed from Bug report to 4
  • Assignee deleted (nobody -)
  • Start date set to 2012-01-09
  • Pull Request or Patch supplied set to No

#26 Updated by Alister Hood about 12 years ago

Also see the discussion at #3136, and #4404. The fix should really be upstream (gdal and proj), so I've been intending to start a discussion somewhere there about it. First I need to read a bit of the gdal mailing list to see whether it is best to start a discussion on it, or to file a ticket.

#27 Updated by Giovanni Manghi almost 12 years ago

  • Tracker changed from 4 to Bug report

#28 Updated by Paolo Cavallini almost 12 years ago

  • Target version changed from Version 1.7.4 to Version 1.8.0

#29 Updated by Paolo Cavallini over 11 years ago

  • Target version changed from Version 1.8.0 to Version 2.0.0

#30 Updated by Jürgen Fischer over 10 years ago

  • Status changed from Open to Closed

Also available in: Atom PDF