Bug report #18062

floating point error in round function in Ubuntu

Added by Michel Stuyts almost 5 years ago. Updated almost 5 years ago.

Affected QGIS version:master Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:No Resolution:up/downstream
Crashes QGIS or corrupts data:No Copied to github as #:25958


The following procedure gives a floating point error in the results in Ubuntu Linux

  1. Open QGIS 2.99 in Ubuntu Linux 16.04 LTS
  2. Create a new shapefile for lines
  3. Start editing this shapefile and create a line
  4. Stop editing the shapefile and save the edits
  5. Go to the layer styling panel and select "Show labels for this layer"
  6. For "Label with" add round(123.456789,2) as value and click "Apply"
  7. The label now shows 123.459999999999
  8. Open attribute table
  9. Open field calculator
  10. Select Create new field, with output field type Real, output field length 10 and Precision 10 (that's the highest possible value). As a expression enter round(123.456789,2).
  11. The output preview says 123.459999999999
  12. The result in the table says 123.4600000000

On Windows 10, the result is always the correct 123.46.


#1 Updated by Kory Roberts almost 5 years ago

Looks similar to these already reported bugs:

#2 Updated by Nyall Dawson almost 5 years ago

  • Resolution set to up/downstream
  • Status changed from Open to Closed

This is an issue with the underlying Qt 5.6 library, and is not something we can fix in QGIS (believe us, we tried HARD). The ONLY solution is to use a build with a newer version of Qt (5.7 and above are fine) - that may mean updating your distro.

#3 Updated by Michel Stuyts almost 5 years ago

Since I just want to show it as a label and don't want to calculate with it, for now I'll just use the following custom function in the Function Editor:

@qgsfunction(args='auto', group='Custom')
def str_round(value1, value2, feature, parent):
    return str(round(value1,value2))

This will show the correct result as a string.

Also available in: Atom PDF