Bug report #18062
floating point error in round function in Ubuntu
| Status: | Closed | ||
|---|---|---|---|
| Priority: | Normal | ||
| Assignee: | - | ||
| Category: | Expressions | ||
| Affected QGIS version: | master | Regression?: | No |
| Operating System: | Easy fix?: | No | |
| Pull Request or Patch supplied: | No | Resolution: | up/downstream |
| Crashes QGIS or corrupts data: | No | Copied to github as #: | 25958 |
Description
The following procedure gives a floating point error in the results in Ubuntu Linux
- Open QGIS 2.99 in Ubuntu Linux 16.04 LTS
- Create a new shapefile for lines
- Start editing this shapefile and create a line
- Stop editing the shapefile and save the edits
- Go to the layer styling panel and select "Show labels for this layer"
- For "Label with" add round(123.456789,2) as value and click "Apply"
- The label now shows 123.459999999999
- Open attribute table
- Open field calculator
- Select Create new field, with output field type Real, output field length 10 and Precision 10 (that's the highest possible value). As a expression enter round(123.456789,2).
- The output preview says 123.459999999999
- The result in the table says 123.4600000000
On Windows 10, the result is always the correct 123.46.
History
#1
Updated by Kory Roberts almost 7 years ago
#2
Updated by Nyall Dawson almost 7 years ago
- Resolution set to up/downstream
- Status changed from Open to Closed
This is an issue with the underlying Qt 5.6 library, and is not something we can fix in QGIS (believe us, we tried HARD). The ONLY solution is to use a build with a newer version of Qt (5.7 and above are fine) - that may mean updating your distro.
#3
Updated by Michel Stuyts almost 7 years ago
Since I just want to show it as a label and don't want to calculate with it, for now I'll just use the following custom function in the Function Editor:
@qgsfunction(args='auto', group='Custom')
def str_round(value1, value2, feature, parent):
return str(round(value1,value2))
This will show the correct result as a string.