Bug report #8897
Results from field calculator wrong format
Status: | Closed | ||
---|---|---|---|
Priority: | Normal | ||
Assignee: | - | ||
Category: | Vectors | ||
Affected QGIS version: | 2.0.1 | Regression?: | No |
Operating System: | Easy fix?: | No | |
Pull Request or Patch supplied: | No | Resolution: | fixed/implemented |
Crashes QGIS or corrupts data: | No | Copied to github as #: | 17575 |
Description
- Open field calculator
- Expression 10/ toreal( 3)
- Set to create a new field type (integer), width 10.
Result? 3.33333333333333
This is obviously wrong. When I save the shapefile it will be truncated to "3", which is right. But QGIS should never return 3.333333333333 in the first place, because it is invalid for the data type of the output format. The fact the result changes when saved is anathema to all conventions of data handling.
History
#1 Updated by Nathan Woodrow about 11 years ago
This is more of a issue with the attribute table and attribute dialog itself. The expression engine is just doing what it is asked and returning the result. It seems the field is storing the full value even though it's an int although I don't know how.
#2 Updated by Alexander Bruy about 11 years ago
This is obviously correct. You divide integer to real and result surprisingly will be... real. Use toint() function in such cases or don't cast integer to float before division.
Also when you try save float value into integer-type field, this float will be converted (if possible) to int. So all, I stress, all works correct.
#3 Updated by Jonathan Moules about 11 years ago
Sorry Alexander, I disagree strongly.
Yes, mathematically the result (3.33333.) is obviously correct. However, because the output field is an integer, it cannot store anything beyond the "3". So QGIS should never say 3.33333333 in the attribute result if the output field can't handle that.
QGIS is doing the conversion of 3.33333 to 3 (float to int) at save time. I'm saying it should do it before it puts it into the attribute table in the first place.
Similarly, if the output field is set to only allow two decimal places, qgis should only show two decimal places when the result is returned.
To do otherwise will confuse the vast majority of users when their data "changes".
#4 Updated by Matthias Kuhn about 11 years ago
It's true, the expression engine produces a QVariant which can be of a variety of types.
And it seems these values are stored like they are returned to the local edit buffer until being committed. This affects much more than the representation in the attribute table.
E.g.
Imagine a layer with an INT field a
, calculated to 3.6
If you now create a new INT field b
by multiplying a
with 3 and you perform this calculation on uncommitted values the result will be 10.8 (later truncated to 10). If this calculation is done on committed values, the result will be 9.
It should be the edit buffer or vector layer taking care of field conversion on UPDATE / INSERT operations, so also values which originate from python scripts or other sources will be converted.
#5 Updated by Jürgen Fischer over 10 years ago
- Category set to Vectors
#6 Updated by Giovanni Manghi over 7 years ago
- Regression? set to No
- Easy fix? set to No
#7 Updated by Giovanni Manghi over 6 years ago
- Status changed from Open to Feedback
Please test with a recent QGIS release (2.18 or 3), if the issue/request is still valid change the affected version accordingly, if is fixed/implemented then close the ticket. Thanks!
#8 Updated by Alexander Bruy over 6 years ago
- Resolution set to fixed/implemented
- Description updated (diff)
- Status changed from Feedback to Closed