Bug report #9081

Raster calculator result is not accurate

Added by Tim Sutton almost 6 years ago. Updated over 4 years ago.

Status:Closed
Priority:Normal
Assignee:Marco Hugentobler
Category:Raster Calculator
Affected QGIS version:2.0.1 Regression?:No
Operating System:OSX Easy fix?:No
Pull Request or Patch supplied:No Resolution:
Crashes QGIS or corrupts data:No Copied to github as #:17726

Description

When using the raster calculator with this simple example:

@layer1 / 10

the results do not come out as expected. We can verify this by using the identify tool and clicking on a pixel e.g.

Original Layer: 844.1530151367188
Calculated layer: 84.41529846191406

In other cases the difference can be more extreme.

It appears the difference may be due to resampling during the raster calculation process. It would be good so convey to the user the fact that resampling takes place during calculation, or to have an option to prevent resampling during recalculation.

Associated revisions

Revision f42f640d
Added by Nyall Dawson over 4 years ago

[rastercalc] Switch all internal calculations to doubles

...for more accurate calculations (fix #9081)

Also:
- Add test suite for raster calculator
- Fix errors with log/log10 calculation and inputs <= 0
- Fix loss of nodata values in certain circumstances

History

#1 Updated by Nyall Dawson over 4 years ago

  • Status changed from Open to Closed

Also available in: Atom PDF