Bug report #18062

floating point error in round function in Ubuntu

Added by Michel Stuyts about 6 years ago. Updated about 6 years ago.

Status:Closed
Priority:Normal
Assignee:-
Category:Expressions
Affected QGIS version:master Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:No Resolution:up/downstream
Crashes QGIS or corrupts data:No Copied to github as #:25958

Description

The following procedure gives a floating point error in the results in Ubuntu Linux

  1. Open QGIS 2.99 in Ubuntu Linux 16.04 LTS
  2. Create a new shapefile for lines
  3. Start editing this shapefile and create a line
  4. Stop editing the shapefile and save the edits
  5. Go to the layer styling panel and select "Show labels for this layer"
  6. For "Label with" add round(123.456789,2) as value and click "Apply"
  7. The label now shows 123.459999999999
  8. Open attribute table
  9. Open field calculator
  10. Select Create new field, with output field type Real, output field length 10 and Precision 10 (that's the highest possible value). As a expression enter round(123.456789,2).
  11. The output preview says 123.459999999999
  12. The result in the table says 123.4600000000

On Windows 10, the result is always the correct 123.46.

History

#1 Updated by Kory Roberts about 6 years ago

Looks similar to these already reported bugs:
#17861
#18007

#2 Updated by Nyall Dawson about 6 years ago

  • Resolution set to up/downstream
  • Status changed from Open to Closed

This is an issue with the underlying Qt 5.6 library, and is not something we can fix in QGIS (believe us, we tried HARD). The ONLY solution is to use a build with a newer version of Qt (5.7 and above are fine) - that may mean updating your distro.

#3 Updated by Michel Stuyts about 6 years ago

Since I just want to show it as a label and don't want to calculate with it, for now I'll just use the following custom function in the Function Editor:

@qgsfunction(args='auto', group='Custom')
def str_round(value1, value2, feature, parent):
    return str(round(value1,value2))

This will show the correct result as a string.

Also available in: Atom PDF