Bug report #3395

Field Calculator Integer Conversion Error

Added by alexckp - over 13 years ago. Updated about 10 years ago.

Status:Closed
Priority:Low
Assignee:Jürgen Fischer
Category:Vectors
Affected QGIS version:master Regression?:No
Operating System:All Easy fix?:No
Pull Request or Patch supplied:No Resolution:fixed/implemented
Crashes QGIS or corrupts data:No Copied to github as #:13455

Description

I tried to use FC to extract an integer number using the following formula:

(SURV_BRG * 1000) - to int (to int(SURV_BRG * 100) * 10)

If the number is 62.233, the above formula will correctly return the number 3. However, when I encounter certain numbers, e.g. the number 65.1, the result is 9.99999999999272 instead of 0. I think this has to do with the to int function not returning a true integer and would appreciate if this problem could be solved.

History

#1 Updated by Anne Ghisla over 13 years ago

I tried to reproduce the bug.

If I set the type of the new field as integer, the formula outputs 3 and 10 as expected.

If I set it as decimal, then I get the 9.99999.. even if the formula tries to output an integer.

Computing partial results in a new decimal field, with input value as 65.1, gives this:

SURV_BRG * 100 = 6510

to int(SURV_BRG ) = 65

to int(SURV_BRG * 100) = 6509

If affects Fedora, so changing platform to All.

#2 Updated by Giovanni Manghi over 12 years ago

  • Target version changed from Version 1.7.0 to Version 1.7.4

#3 Updated by Paolo Cavallini over 12 years ago

  • Crashes QGIS or corrupts data set to No
  • Affected QGIS version set to master
  • Target version changed from Version 1.7.4 to Version 1.8.0

#4 Updated by Paolo Cavallini almost 12 years ago

  • Target version changed from Version 1.8.0 to Version 2.0.0

#5 Updated by Jürgen Fischer almost 11 years ago

  • Pull Request or Patch supplied set to No
  • Subject changed from Field Calculator Integer Conversion Errror to Field Calculator Integer Conversion Error

#6 Updated by Nathan Woodrow about 10 years ago

  • Resolution set to fixed/implemented
  • Status changed from Open to Closed

There is a new expression engine since 2.0 and this is no longer an issue.

Also available in: Atom PDF