Bug report #11066
Error in Field Calculator Results, when using the 'division' operator
| Status: | Closed | ||
|---|---|---|---|
| Priority: | High | ||
| Assignee: | - | ||
| Category: | - | ||
| Affected QGIS version: | 2.4.0 | Regression?: | No |
| Operating System: | Easy fix?: | No | |
| Pull Request or Patch supplied: | No | Resolution: | wontfix |
| Crashes QGIS or corrupts data: | No | Copied to github as #: | 19400 |
Description
I created an attribute table for a point shapefile.
The attribute table had 3 columns, apart from the ‘id’ column.
Then, I tried to create a 4th column from two of the already existing columns, using field calculator.
The result of the division operator didn’t look good at all.
In the 4th column, it created a wrong result.
The results of dividing one column by the next, was really wrong. But when I tested the ‘addition’, ‘Subtraction’ and ‘Multiplication’ operator , they checked out fine, except ‘division’.
I observed that the results didn’t display correctly the decimal portion of the output.
I had set the precision level to 4.
For example, 3 / 7 gives 0.0000 (which is wrong) instead of 0. 4285714285714
In other words, it appears to throw away the decimal portions, instead of honour the precision level that was entered. And I think it is a fundamental flaw, which should be fixed.
History
#1
Updated by Matthias Kuhn over 10 years ago
- Resolution set to wontfix
- Status changed from Open to Closed
Hi Chinedu,
The calculator works with the format, that parts of the expression are. That means, when you are doing an operation based on two integers (whole numbers), the result will be the same format. That also means, that you just have to change one of the two numbers to a decimal number to get more precise results. In your case this means using:
3.0 / 7
or
3 / 7.0