Bug report #8897

Results from field calculator wrong format

Added by Jonathan Moules over 10 years ago. Updated almost 6 years ago.

Affected QGIS version:2.0.1 Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:No Resolution:fixed/implemented
Crashes QGIS or corrupts data:No Copied to github as #:17575


- Open field calculator
- Expression 10/ toreal( 3)
- Set to create a new field type (integer), width 10.

Result? 3.33333333333333
This is obviously wrong. When I save the shapefile it will be truncated to "3", which is right. But QGIS should never return 3.333333333333 in the first place, because it is invalid for the data type of the output format. The fact the result changes when saved is anathema to all conventions of data handling.


#1 Updated by Nathan Woodrow over 10 years ago

This is more of a issue with the attribute table and attribute dialog itself. The expression engine is just doing what it is asked and returning the result. It seems the field is storing the full value even though it's an int although I don't know how.

#2 Updated by Alexander Bruy over 10 years ago

This is obviously correct. You divide integer to real and result surprisingly will be... real. Use toint() function in such cases or don't cast integer to float before division.

Also when you try save float value into integer-type field, this float will be converted (if possible) to int. So all, I stress, all works correct.

#3 Updated by Jonathan Moules over 10 years ago

Sorry Alexander, I disagree strongly.
Yes, mathematically the result (3.33333.) is obviously correct. However, because the output field is an integer, it cannot store anything beyond the "3". So QGIS should never say 3.33333333 in the attribute result if the output field can't handle that.

QGIS is doing the conversion of 3.33333 to 3 (float to int) at save time. I'm saying it should do it before it puts it into the attribute table in the first place.

Similarly, if the output field is set to only allow two decimal places, qgis should only show two decimal places when the result is returned.

To do otherwise will confuse the vast majority of users when their data "changes".

#4 Updated by Matthias Kuhn over 10 years ago

It's true, the expression engine produces a QVariant which can be of a variety of types.

And it seems these values are stored like they are returned to the local edit buffer until being committed. This affects much more than the representation in the attribute table.

Imagine a layer with an INT field a, calculated to 3.6
If you now create a new INT field b by multiplying a with 3 and you perform this calculation on uncommitted values the result will be 10.8 (later truncated to 10). If this calculation is done on committed values, the result will be 9.

It should be the edit buffer or vector layer taking care of field conversion on UPDATE / INSERT operations, so also values which originate from python scripts or other sources will be converted.

#5 Updated by Jürgen Fischer almost 10 years ago

  • Category set to Vectors

#6 Updated by Giovanni Manghi almost 7 years ago

  • Regression? set to No
  • Easy fix? set to No

#7 Updated by Giovanni Manghi about 6 years ago

  • Status changed from Open to Feedback

Please test with a recent QGIS release (2.18 or 3), if the issue/request is still valid change the affected version accordingly, if is fixed/implemented then close the ticket. Thanks!

#8 Updated by Alexander Bruy almost 6 years ago

  • Resolution set to fixed/implemented
  • Description updated (diff)
  • Status changed from Feedback to Closed

Also available in: Atom PDF