Bug report #9060

Area is wrong in QGIS master (again)

Added by Rhenriques Henriques about 6 years ago. Updated over 5 years ago.

Status:Closed
Priority:Severe/Regression
Assignee:-
Category:Vectors
Affected QGIS version:master Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:Yes Resolution:fixed/implemented
Crashes QGIS or corrupts data:No Copied to github as #:17707

Description

Qgis 2.1 DEV has a ressurgence of the "area calculation bug" with OTFR. For instance the NB e890e14 cannot calculate correctly the area. If we use "info" and check the geometry properties, the area is just right. We try to get it to a field, by using $area and results are awkward. The 0 ID geometry from the attached file should have 2520002 m2 and is giving 2523339 m2. This bug is not corrected. Use attached .shp to check.
Another thing is that, while opening an attribute table with several thousands of objects (about 400 000 points in my case), QGIS is extremely slow and some tables do not load beyond a few 1000 or 2000 objects. This is getting worse each NB. In QGIS 2.0.1 the table open is super fast.
Cheers

QUAD.zip (1.89 KB) Rhenriques Henriques, 2013-11-13 11:50 AM

QUAD.zip (148 KB) Rhenriques Henriques, 2013-11-15 10:39 AM

MeasureCalcTransformed.jpg (40.7 KB) Alvaro Huarte, 2013-12-27 04:39 AM


Related issues

Related to QGIS Application - Bug report #10392: Ellipsoid for measurement keeps getting reset Closed 2014-05-29

Associated revisions

Revision 7ea33909
Added by Martin Dobias over 5 years ago

Fix incorrectly reported area in identify tool (partly fixes #9060)

Occurred with projected layer CRS, non-projected map CRS, planimetric measuring.
It was assumed that units of the measurement were in map CRS.

History

#1 Updated by Giovanni Manghi about 6 years ago

  • Subject changed from Table issues to Area is wrong in QGIS master (again)
  • Category deleted (Browser)
  • Affected QGIS version changed from 2.0.1 to master

the slowness of the attribute table has been reported in another (blocking) issue.

#2 Updated by Matthias Kuhn about 6 years ago

I just checked and could not reproduce (yet).

OTF turned on:
With project CRS set to 4326 I get 2523339 in the identify dialog and the field calculator (opened from the attribute table)
With project CRS set to 20790 I get 2520002 in the identify dialog and the field calculator (opened from the attribute table)

Can you add a project file to the zip?

#3 Updated by Rhenriques Henriques about 6 years ago

See attached project and files. See the two PNG images included. One shows how the area is predicted in the calculation window (the value is right) and the other one shows how it appear in the table (wrong value - area_1). The column "area" as the expected value. This value comes from ArcGIS 10. Parallel to this there's another annoying issue - QGIS cannot remember the framed objects and zoom from the last session. We have to select a layer and do "zoom to layer extension" to frame the objects again.
Cheers

#4 Updated by Rhenriques Henriques almost 6 years ago

Problem finally solved in QGIS nightly build 55f8606!
Cheers

#5 Updated by Giovanni Manghi almost 6 years ago

  • Resolution set to fixed/implemented
  • Status changed from Open to Closed

#6 Updated by Rhenriques Henriques almost 6 years ago

Sorry Giovanni, but the bug surfaced again in nightly build 30900e9!! It has exactly the same symptoms it had before. Strange...
Cheers

#7 Updated by Giovanni Manghi almost 6 years ago

  • Resolution deleted (fixed/implemented)
  • Status changed from Closed to Feedback

Rhenriques Henriques wrote:

Sorry Giovanni, but the bug surfaced again in NB 30900e9!! It has exactly the same symptoms it had before. Strange...
Cheers

maybe the issue was never fixed and it surfaces only in specific conditions?

#8 Updated by Alvaro Huarte almost 6 years ago

Hi, I have found that the change in the calculated value of the area is due to the configuration of the project.

It has assigned an ellipsoid transformation for measure calculations.

I you change the 'ellipsoid' to 'none/planimetric', it give me ok. Or you can disable 'on the fly' CRS transformation in CRS panel.
Best Regards

#9 Updated by Rhenriques Henriques almost 6 years ago

Hi Alvaro

You are right! Maybe the none/planimetric option should be selected as "default" for every new project.
Cheers

#10 Updated by Alvaro Huarte almost 6 years ago

Hi Rhenriques, then this issue can be closed?
Best regards

#11 Updated by Rhenriques Henriques almost 6 years ago

Hi Alvaro

I suppose so. I've tested in several projects and values are always correctly calculated if 'none/planimetric' option is used, as expected.
Please ensure that this option is set as default in the next versions. No one wants miscalculated areas or any other wrong measurements in their tables!
Cheers and thank you.

#12 Updated by Giovanni Manghi almost 6 years ago

  • Status changed from Feedback to Open

Alvaro Huarte wrote:

Hi Rhenriques, then this issue can be closed?
Best regards

Hi Alvaro and Rui,
what are the (other) consequences of eventyally making 'none/planimetric' default?

Do older qgis releases computed the area right even with 'wgs84'? If yes then this must be fixed anyway, unless the old behavior/option was misleading.

Please raise this issue in the dev mailing list.

#13 Updated by Nyall Dawson almost 6 years ago

There is a side-effect of setting the ellipsoid to "None/planimetric":
- Set the otf projection to a non-projected crs, eg wgs84 (epsg:4326)
- In the general tab set the canvas units to "meters"
- Try to measure a distance on the map - the distance is shown in degrees, not metres

There's some relevant discussion in issue #7441.

#14 Updated by Rhenriques Henriques almost 6 years ago

Nyall is right! I've only tested in projects with projected units. The settings are not even stable. Sometimes things change (units; ellipsoid) each time we open the Project Settings dialog (NB).
This issue must be changed to blocker due to the major implications it represents. You can have the best GIS software but if it's unable to calculate correctly, in a stable basis, a distance or an area, it becomes useless in real world projects. Strange because it all seem much more stable in 1.8. Another strange thing is that the 'preview value' of the 'field calculator' is mostly right. How is this possible?
Cheers

#15 Updated by Alvaro Huarte almost 6 years ago

  • Pull Request or Patch supplied changed from No to Yes

Hi, I release a new pull request (https://github.com/qgis/QGIS/pull/1071) it attempts to correct this error.

Can you test it ?
Thank you very much

Alvaro

#16 Updated by Giovanni Manghi almost 6 years ago

  • Status changed from Open to Feedback

Rhenriques Henriques wrote:

Nyall is right! I've only tested in projects with projected units. The settings are not even stable. Sometimes things change (units; ellipsoid) each time we open the Project Settings dialog (NB).
This issue must be changed to blocker due to the major implications it represents. You can have the best GIS software but if it's unable to calculate correctly, in a stable basis, a distance or an area, it becomes useless in real world projects. Strange because it all seem much more stable in 1.8. Another strange thing is that the 'preview value' of the 'field calculator' is mostly right. How is this possible?
Cheers

but then the issue is now about the measure tool, and not the field calculator results? right?

As it is now it seems that the area/length are computed correctly with or without otfr, right?

Alvaro your patch fixes the measure tools problem and the inconsistency when setting project properties > canvas units?

If yes this ticket should be closed and a new one open, or at least change title, category and description.

#17 Updated by Alvaro Huarte almost 6 years ago

Hi, this patch fix the ellipsoid and units selection in the project properties dialog. The current settings are preserved when the dialog is showed and fix a bug to select the related ellipsoid of the current CRS (https://github.com/qgis/QGIS/pull/1071/files#diff-b335df9eb7a06c67dc9347624b7c3b53L1027).

I have not tested the measure tool, I guess it works fine for one so mature application ;-), but I tested the example of Rhenriques and It seems ok.

Best regards

#18 Updated by Giovanni Manghi almost 6 years ago

  • Category set to Vectors

Alvaro Huarte wrote:

Hi, this patch fix the ellipsoid and units selection in the project properties dialog. The current settings are preserved when the dialog is showed and fix a bug to select the related ellipsoid of the current CRS (https://github.com/qgis/QGIS/pull/1071/files#diff-b335df9eb7a06c67dc9347624b7c3b53L1027).

I have not tested the measure tool, I guess it works fine for one so mature application ;-), but I tested the example of Rhenriques and It seems ok.

Best regards

Rui, can you confirm then that the original issue is solved? Thanks

#19 Updated by Giovanni Manghi almost 6 years ago

  • Resolution set to fixed/implemented
  • Status changed from Feedback to Closed

This seems really fixed to me. Please reopen if necessary.

#20 Updated by Alvaro Huarte almost 6 years ago

Hi Giovanni, the pull request 'https://github.com/qgis/QGIS/pull/1071' has not yet been merged.
Do you think that it is not necessary to fix this issue ?

Best Regards
Alvaro

#21 Updated by Giovanni Manghi almost 6 years ago

Alvaro Huarte wrote:

Hi Giovanni, the pull request 'https://github.com/qgis/QGIS/pull/1071' has not yet been merged.
Do you think that it is not necessary to fix this issue ?

Best Regards
Alvaro

it seems to me that master is returning right values as it is now, but I would really appreciate if you can give a look. Also it would e important to have the feedback of the original reporter.

#22 Updated by Alvaro Huarte almost 6 years ago

Giovanni Manghi wrote:

Alvaro Huarte wrote:

Hi Giovanni, the pull request 'https://github.com/qgis/QGIS/pull/1071' has not yet been merged.
Do you think that it is not necessary to fix this issue ?

Best Regards
Alvaro

it seems to me that master is returning right values as it is now, but I would really appreciate if you can give a look. Also it would e important to have the feedback of the original reporter.

I think that 'https://github.com/ahuarte47/QGIS/commit/e0342dcf077c9be04b1de416abbc718b616d074d' fixes as least a bug searching the ellipsoid related to one CRS

#23 Updated by Rhenriques Henriques almost 6 years ago

Just tested in NB, code revision "d26831b" and is not right yet. If the original shape CRS is used in the project definition, the area is right. As soon as we change the project CRS to WGS84, the area calculation gets wrong. Same as before unfortunately.
Cheers

#24 Updated by Paolo Cavallini almost 6 years ago

  • Status changed from Closed to Feedback

#25 Updated by Giovanni Manghi over 5 years ago

  • Target version changed from Future Release - High Priority to Version 2.2
  • Resolution deleted (fixed/implemented)
  • Status changed from Feedback to Open

Rhenriques Henriques wrote:

Just tested in NB, code revision "d26831b" and is not right yet. If the original shape CRS is used in the project definition, the area is right. As soon as we change the project CRS to WGS84, the area calculation gets wrong. Same as before unfortunately.
Cheers

Ok, so for what I can see now qgis master computes wrong the area of a polygon if this has (for example) a projected CRS and OTFR is ON and set to a geographic CRS (or a different projected CRS), unless in the project properties the ellipsoid is set to "none/planimetric".

In the same conditions QGIS 1.8 computed the are always correctly (the ellipsoid was only configurable in the general options).

Correct?

#26 Updated by Rhenriques Henriques over 5 years ago

Correct!

#27 Updated by Bo Victor Thomsen over 5 years ago

IMHO, the correct solution would be that both the measurement tools and the field calculation functions should obey the same global set-up value regarding the ellipsoid choice. And the default ellipsoid should be the one used in the project default CRS. The use of planimetric measurement is a leftover from the days where you had to use a planimeter (http://en.wikipedia.org/wiki/Planimeter) to calculate areas on a paper map.

Would it be possible to the following choices ?:

"Use ellipsoid from default project CRS"
"Use specific ellipsoid" (together with a "ellipsoid chooser" dialogue)
"Use planimetric calculations"

Although the main point is to have the different tools to report the same area/length measurements

Regards
Bo Victor Thomsen
Aestas-GIS
Denmark

#28 Updated by Martin Dobias over 5 years ago

I am getting quite confused by the discussion as there have been various issues mentioned.

Can you please clearly describe (with the help of attached QUAD project) in what case things get wrong and what is the expected outcome?
- on-the-fly projections on/off
- project CRS
- project measurement ellipsoid
- tool used (measure tool / identify tool / field calculator / all)
- anything else that affects calculation?

The only issue I can see right now is with OTF projections on, project CRS: wgs84, ellipsoid: none.
The identify tool says the area is 31 227 946 258,332 km² which is completely off.

As a bonus, with these settings above, after opening the project properties dialog, the measurement ellipsoid says WGS 84 which is wrong (planimetric is being used).

Is this the problem or are there more cases to fix?

#29 Updated by Martin Dobias over 5 years ago

  • Status changed from Open to Feedback

#30 Updated by Giovanni Manghi over 5 years ago

Is this the problem or are there more cases to fix?

Hi Martin, I resume here what we discussed in private. Anyway it seems to me that if the issue of the completely wrong computed values is solved then the actual implementation make sense, and this introduce a different behavior since 1.8. In the future the approach used in the "add/export geometry columns" tool in the vector menu may be used.


Giovanni:

take as input a 10x10km square in a projected CRS

Hi Martin, I resume here what we discussed privately.

qgis 1.8:

the ellipsoid for measurements option was defined in the qgis general
options, wgs84 by default

now add your square, otfr off, compute area with field calculator -> 100sqkm

otfr on, project CRS same as layer CRS, compute area with field
calculator -> 100sqkm

otfr on, project CRS different from the layer CRS (geographic or
projected does not matter), compute area with field calculator ->
100sqkm

qgis master:

the ellipsoid for measurements option is defined in the project
options, wgs84 by default

now add your square, otfr off, compute area with field calculator -> 100sqkm

otfr on, project CRS same as layer CRS, compute area with field
calculator -> 100sqkm

otfr on, project CRS different from the layer CRS (geographic or
projected does not matter), compute area with field calculator ->
NOT 100sqkm... but a close value (as expected?). Problem is that
sometimes the result is a completely wrong value (not even close).

If the user changes the ellipsoid for measurements to "planar/none"
that the results are equals to the ones in qgis 1.8


Martin:

This is expected - or at least for me... instead of measuring the area
in layer's coordinate system, it will reproject it to lat/lon coords
on ellipsoid and calculate the area on ellipsoid - a relatively small
difference is understandable.

But maybe not what the users want?


Giovanni:

I agree, but this change was undocumented and probably unexpected by
many users, so
is leading to confusions.

the feedback I have is that most users expect to have the area/length
of layers in a projected CRS to be always
computed in the layers CRS.

But I also agree that the actual behaviour of qgis make sense (for the
more experienced user), so the above could
eventually be added as an option in general or project properties.

maybe the approach that can be used (as option in the project/general
options) is the one already available in the
"add/export geometry columns" tool in the vector menu.


Martin:

This looks like a good option for the future - should be relatively clear.

It seems that planimetric computation in layer's CRS is the usual
assumption. Then there are people with unprojected data (e.g. from
GPS) trying to measure something, for them it makes more sense to have
area computed on ellipsoid rather than area in squared degrees :-)
.... all the other cases are probably "advanced" and the user should
have a choice to decide how to do the measuring.

#31 Updated by Martin Dobias over 5 years ago

  • Status changed from Feedback to Closed

#32 Updated by Martin Dobias over 5 years ago

Current state of measuring:
- ellipsoid "none" - consistently returning 2.520 km^2 (with project's CRS being EPSG:20790, WGS84 or with OTFR off)
- ellipsoid "wgs84" - consistently returning 2.523 km^2 (with project's CRS being EPSG:20790 or WGS84)

Consistently means the same result for identify tool, $area and measure tool.

The only outlier is measure tool for project's CRS being WGS84 and ellipsoid "none" - the reported area is ~3.4 km2 due to measuring in degrees and then converting that value with a fixed degree-to-meters ratio, so that is acceptable (IMHO)

#33 Updated by Martin Dobias over 5 years ago

In case of any further issues please open a new bug as all the comments in this issue make it hard to follow.

#34 Updated by Giovanni Manghi over 5 years ago

Martin Dobias wrote:

In case of any further issues please open a new bug as all the comments in this issue make it hard to follow.

Hi Martin, does your fix apply also to the field calculator?

cheers!

#35 Updated by Martin Dobias over 5 years ago

No, it fixes only wrong value in identify tool. I am not aware of any issue in field calculator.

#36 Updated by Alvaro Huarte over 5 years ago

Hi Martin, there is an issue commented in #9060-14 that I think fixed in https://github.com/qgis/QGIS/pull/1071

Cheers!!!

#37 Updated by Martin Dobias over 5 years ago

  • Status changed from Closed to Reopened

#38 Updated by Giovanni Manghi over 5 years ago

Martin Dobias wrote:

No, it fixes only wrong value in identify tool. I am not aware of any issue in field calculator.

I'm afraid also the FC is affected.

#39 Updated by Rhenriques Henriques over 5 years ago

I'm paying attention to every single NB about this issue. Changes are not committed yet in the NB: f06457b. Changing flags in the project settings, about units and projection, are not honoured yet too. This happens under circumstances that are not easy to replicate sometimes, but mainly are related with turning on or off the OTFR. If we change to meters and planimetric, for instance, things change again the next time we open the project dialogue settings to default.
While using the 2.01 version, in projects with a high degree of responsibility, I use a shape with a perfect 1 sq km that I copy to every polygon shape, to ensure that I got the areas right in the field calculator. Very annoying...
Cheers

#40 Updated by Alvaro Huarte over 5 years ago

Hi Rhenriques, this pull (https://github.com/qgis/QGIS/pull/1071) (not merged) attempts to correct that behavior that are not respected previous settings.

Can you test it ?

#41 Updated by Martin Dobias over 5 years ago

I do not feel confident merging the proposed PR this close before the release as there are several non-trivial changes where I cannot foresee their impact. It needs more testing to see if various possible cases are covered.

#42 Updated by Alvaro Huarte over 5 years ago

Martin Dobias wrote:

I do not feel confident merging the proposed PR this close before the release as there are several non-trivial changes where I cannot foresee their impact. It needs more testing to see if various possible cases are covered.

ok, then there are two minor changes:

https://github.com/ahuarte47/QGIS/commit/4b352c342704f46e43ecee39e9c0e5bb4662e8d0
https://github.com/qgis/QGIS/pull/1054

#43 Updated by Martin Dobias over 5 years ago

Merged the two commits from Alvaro (35791aa and c784c0)

#44 Updated by Martin Dobias over 5 years ago

The real issue with the project properties and setting of on-the-fly reprojection flag, CRS, map units, ellipsoid is that it is poorly designed and the user has no clue which settings are forced by QGIS and which of them are recommended with the possibility to override them.

#45 Updated by Giovanni Manghi over 5 years ago

Martin Dobias wrote:

I do not feel confident merging the proposed PR this close before the release as there are several non-trivial changes where I cannot foresee their impact. It needs more testing to see if various possible cases are covered.

Hi Martin,

I just tested master with a code revision prior of your last commit.

This is what I see using the field calculator.

The input is a 1x1km grid in a projected CRS (tested 3763)

1) otfr off area = 1000000
2) otfr on, project CRS same as layer area = 1002464.5100 - 1002535.8531
3) otfr on, project CRS different from layer CRS (projected or geographic) area = 1002464.5100 - 1002535.8531

the Field Calculator preview always show 1000000

Maybe 3) makes sense, but 2) does not seems to.

#46 Updated by Rhenriques Henriques over 5 years ago

The behaviour that Giovanni described is exactly what I am obtaining too. As I told before, and as Giovanni wrote, the strange thing is that field calculator always show the correct value in the preview.

Cheers

#47 Updated by Martin Dobias over 5 years ago

Giovanni Manghi wrote:

The input is a 1x1km grid in a projected CRS (tested 3763)

1) otfr off area = 1000000
2) otfr on, project CRS same as layer area = 1002464.5100 - 1002535.8531
3) otfr on, project CRS different from layer CRS (projected or geographic) area = 1002464.5100 - 1002535.8531

the Field Calculator preview always show 1000000

Maybe 3) makes sense, but 2) does not seems to.

For 2) and 3), if you did not use "none / planimetric" ellipsoid for measuring, the result makes sense because the measurements are done on ellipsoid. Only with "none / planimetric" ellipsoid it will give you 1000000. I am not saying this logic is optimal, it is simply the behavior that someone has defined previously.

Btw. interesting that you can see preview in field calculator - for me the preview for "$area" is always empty.

#48 Updated by Giovanni Manghi over 5 years ago

For 2) and 3), if you did not use "none / planimetric" ellipsoid for measuring, the result makes sense because the measurements are done on ellipsoid. Only with "none / planimetric" ellipsoid it will give you 1000000. I am not saying this logic is optimal, it is simply the behavior that someone has defined previously.

as discussed previously this is of course right, and makes sense for power users, but if wgs84 is the defaut ellispoid (as it is) this can be confusing for most of the users, especially because it is change since 1.8.

I'm really not sure what would be the best option at this point.

Btw. interesting that you can see preview in field calculator - for me the preview for "$area" is always empty.

:)

#49 Updated by Martin Dobias over 5 years ago

Giovanni Manghi wrote:

For 2) and 3), if you did not use "none / planimetric" ellipsoid for measuring, the result makes sense because the measurements are done on ellipsoid. Only with "none / planimetric" ellipsoid it will give you 1000000. I am not saying this logic is optimal, it is simply the behavior that someone has defined previously.

as discussed previously this is of course right, and makes sense for power users, but if wgs84 is the defaut ellispoid (as it is) this can be confusing for most of the users, especially because it is change since 1.8.

I'm really not sure what would be the best option at this point.

Me neither - but it does not make sense to do any radical changes right now.

It is clear that for future versions we need to rethink how we will deal will measuring settings, map units and related things to clean the mess we have right now - a solution with sensible defaults and logic where one gets expected results all the time.

#50 Updated by Paolo Cavallini over 5 years ago

I confirm, the $area preview shows the result

#51 Updated by Martin Dobias over 5 years ago

Another fix (d92a0e) - make sure to show preview of "$area" + make the preview consistent with field calculator results

#52 Updated by Antonio Locandro over 5 years ago

Here are my thoughts on this

The measure tool dialog for me for selecting Ellipsoid is superfluous in the General Tab, units should be selected directly on the Measure tool that way I can go from m to km to NM to ft if I wanted.

- If project uses a Projected coordinate system then the measure tool should give me the measurements using that PCS by default in the units I have previously selected, only meters, ft and NM should be shown as options (BTW miles should be added as its used in the USA) no degrees options here as it doesn't make sense

- If the project is set to using a Geographic coordinate system then measurements should be used by selecting the ellipsoid from the definition, if OTF is off then only degrees should be shown as option if OTF is on only m,ft, NM should be shown. It makes absolutely no sense to have the QGIS project use Airy 1830 for measure calculations if WGS 84 is used for the project

In this scenario if I have a layer in WGS UTM

a) If project is set to use UTM then I will can measure in m, ft, etc
b) If project is in WGS then I can get geodetic distance in m, ft, etc

Field calculator should work similar

#53 Updated by Giovanni Manghi over 5 years ago

Antonio Locandro wrote:

Here are my thoughts on this

The measure tool dialog for me for selecting Ellipsoid is superfluous in the General Tab, units should be selected directly on the Measure tool that way I can go from m to km to NM to ft if I wanted.

- If project uses a Projected coordinate system then the measure tool should give me the measurements using that PCS by default in the units I have previously selected, only meters, ft and NM should be shown as options (BTW miles should be added as its used in the USA) no degrees options here as it doesn't make sense

- If the project is set to using a Geographic coordinate system then measurements should be used by selecting the ellipsoid from the definition, if OTF is off then only degrees should be shown as option if OTF is on only m,ft, NM should be shown. It makes absolutely no sense to have the QGIS project use Airy 1830 for measure calculations if WGS 84 is used for the project

In this scenario if I have a layer in WGS UTM

a) If project is set to use UTM then I will can measure in m, ft, etc
b) If project is in WGS then I can get geodetic distance in m, ft, etc

Field calculator should work similar

Hi Antonio, please send your notes also the dev mailing list, we need to figure out a way to solve this issues.

#54 Updated by Giovanni Manghi over 5 years ago

  • Resolution set to fixed/implemented
  • Status changed from Reopened to Closed

I'm closing this because as far as I can see now (in master) the area computations are right in both field calculator and identify tool ("derived") and when reprojection is ON the values are ok depending the ellipsoid set in project properties.

The real issue seems this #10392 and, as said many times in this ticket, the whole logic that need to be reviewed, see Antonio's comment #9060-52 for example

Also available in: Atom PDF