Bug report #3395
Field Calculator Integer Conversion Error
|Affected QGIS version:||master||Regression?:||No|
|Operating System:||All||Easy fix?:||No|
|Pull Request or Patch supplied:||No||Resolution:||fixed/implemented|
|Crashes QGIS or corrupts data:||No||Copied to github as #:||13455|
I tried to use FC to extract an integer number using the following formula:
(SURV_BRG * 1000) - to int (to int(SURV_BRG * 100) * 10)
If the number is 62.233, the above formula will correctly return the number 3. However, when I encounter certain numbers, e.g. the number 65.1, the result is 9.99999999999272 instead of 0. I think this has to do with the to int function not returning a true integer and would appreciate if this problem could be solved.
#1 Updated by Anne Ghisla almost 12 years ago
I tried to reproduce the bug.
If I set the type of the new field as integer, the formula outputs 3 and 10 as expected.
If I set it as decimal, then I get the 9.99999.. even if the formula tries to output an integer.
Computing partial results in a new decimal field, with input value as 65.1, gives this:
SURV_BRG * 100 = 6510
to int(SURV_BRG ) = 65
to int(SURV_BRG * 100) = 6509
If affects Fedora, so changing platform to All.