Bug report #13336

Allow raster calculator with huge raster inputs

Added by Simona Z almost 3 years ago. Updated 9 days ago.

Status:Open
Priority:High
Assignee:-
Category:Raster Calculator
Affected QGIS version:3.3(master) Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:No Resolution:
Crashes QGIS or corrupts data:Yes

Description

Hello,
I am trying to do some raster operation with Qgis' Raster Calculator but I get a minidump error and Qgis crashes (like minidump written to C:\\Users\\PROPRI~1\\AppData\\Local\\Temp\\qgis-20150909-155310-3892-7148-d20c5b7.dmp) every time. I am working with two raster files I created, so I would like to know if it's a problem with my rasters or with QGis

Metadata of raster file (it's a binary file with 255 and 0 values):

Driver
GDAL provider
GTiff
GeoTIFF
Dataset Description
[raster pathname]
AREA_OR_POINT=Area
Band 1
STATISTICS_MAXIMUM=255
STATISTICS_MEAN=34.841265875025
STATISTICS_MINIMUM=0
STATISTICS_STDDEV=87.582012938486
Dimensions
X: 18821 Y: 15837 Bands: 1
Origin
1.71397e+06,5.13445e+06
Pixel Size
2.00004,-2.00001
No Data Value
-99999
Data Type
Float64 - Sixty four bit floating point
Pyramid overviews
Layer Spatial Reference System
+proj=tmerc +lat_0=0 +lon_0=9 +k=0.9996 +x_0=1500000 +y_0=0 +ellps=intl +towgs84=-104.1,-49.1,-9.9,0.971,-2.917,0.714,-11.68 +units=m +no_defs
Layer Extent (layer original source projection)
1713966.4223982943221927,5102780.3500192686915398 : 1751609.1579853519797325,5134454.5267527680844069
Band
Band 1
Band No
1
No Stats
No stats collected yet

qgis-20160120-163811-3596-4868-b403532.7z (2.46 MB) Nicolas Cadieux, 2016-01-20 01:50 PM

simulateRaster.7z (648 Bytes) Nicolas Cadieux, 2016-01-25 09:23 AM

backtrace.log Magnifier (2.74 KB) Johannes Kroeger, 2016-09-03 03:53 AM


Related issues

Related to QGIS Application - Bug report #17523: Raster calculator cannot handle huge raster files Closed 2017-11-22

Associated revisions

Revision a665b7d7
Added by Nyall Dawson over 2 years ago

Don't crash when using raster calculator with huge raster (refs #13336)

Now we abort and advise the user if insufficient memory is available
to perform the calculation.

At some future stage it would be nice to perform the calculations
in blocks to allow this scenario, but for now it's better not to
crash.

History

#1 Updated by Nyall Dawson almost 3 years ago

  • Status changed from Open to Feedback

Can you share your raster file and the expression you are using for the calculation?

#2 Updated by Nicolas Cadieux almost 3 years ago

Hi Nyall,

Same problem with 2.12.0 on Windows 7-64
Files are large .tiff(84GB each) but this has worked in the past with 2.6 or 2.8. In my case, both are in the same projection, and I am not changing the output projection. This is simple math. layer1 - layer 2 (creating a normalized LiDAR layer). Same result using virtual raster sets of the same data.

Hope this helps.
Nicolas

#3 Updated by Giovanni Manghi almost 3 years ago

  • Category set to Raster Calculator
  • Priority changed from Normal to High

Nicolas Cadieux wrote:

Hi Nyall,

Same problem with 2.12.0 on Windows 7-64
Files are large .tiff(84GB each) but this has worked in the past with 2.6 or 2.8. In my case, both are in the same projection, and I am not changing the output projection. This is simple math. layer1 - layer 2 (creating a normalized LiDAR layer). Same result using virtual raster sets of the same data.

so, to replicate the issue is needed to use such big tiffs or it happens with any size ones? Any shareable dataset?

#4 Updated by Nicolas Cadieux over 2 years ago

Hi,
I just saw the comments. Sorry for not responding. I though I would get an email. I could share a file but it's 80 GB. I bit big but I could send it by ftp server. Or I could wait to find a smaller file that crashes QGIS. I did a test with a 250MB file and it work ok. It crashes with the big stuff. Send me an email to nicolas.cadieux at archeotec.ca if you need files to resolve this issues.

Nicolas

#5 Updated by Giovanni Manghi over 2 years ago

  • Status changed from Feedback to Open
  • Subject changed from minidump error when using Raster calculator to QGIS crashes when using Raster calculator with huge raster inputs

Nicolas Cadieux wrote:

Send me an email to nicolas.cadieux at archeotec.ca if you need files to resolve this issues.

would be happy to try test this huge rasters: giovanni dot manghi at naturalgis dot pt

#6 Updated by Nyall Dawson over 2 years ago

Can you also send me a link to them? nyall dot dawson at gmail dot com

#7 Updated by Nyall Dawson over 2 years ago

Did you manage to send this data? I'd like to address this for 2.14

#8 Updated by Nicolas Cadieux over 2 years ago

Working on that file. For now, I can send you the dmp.

#9 Updated by Nicolas Cadieux over 2 years ago

trying again on the dmp. If it does not work, I will send it by email.

#10 Updated by Nicolas Cadieux over 2 years ago

Hi,

I have a solution. I have uploaded 2 files. A python 2.7 script that will create a simulated raster having the same general properties as my files. That will create a ASC file that has 145,857,241KB. (Will take time...) Then, use the gdal_translate file to save the ASC file as a tif. That file will have 84,013,848KB. With that, you can make a copy of this file (ex rename file1 and file2) and then do some simple raster math. It should crash as expected. (you will need to change the input and output files in these scripts (but as you are programs, you probably know this.)

Contact me if you have problems.
Nicolas

#11 Updated by Nyall Dawson over 2 years ago

  • Assignee deleted (Nyall Dawson)
  • Tracker changed from Bug report to Feature request
  • Subject changed from QGIS crashes when using Raster calculator with huge raster inputs to Allow raster calculator with huge raster inputs

#12 Updated by Nyall Dawson over 2 years ago

Crash is now avoided, so moving this to a feature request - at some future stage we should perform the calculation in "blocks" to avoid the memory allocation errors.

#13 Updated by Drew Ricketts over 2 years ago

Should this issue be resolved in QGIS 2.14? I'm trying to do a very simple raster calculation on a .tif using a mac, and QGIS crashes without giving a warning about memory.

#14 Updated by Nyall Dawson over 2 years ago

Drew - sounds like a totally different issue.

#15 Updated by Jürgen Fischer about 2 years ago

  • Tracker changed from Feature request to Bug report
  • Crashes QGIS or corrupts data changed from Yes to No

#16 Updated by Jürgen Fischer about 2 years ago

  • Tracker changed from Bug report to Feature request

#17 Updated by Giovanni Manghi about 2 years ago

  • Tracker changed from Feature request to Bug report
  • Target version set to Version 3.0
  • Affected QGIS version changed from 2.10.1 to 2.16.0
  • Crashes QGIS or corrupts data changed from No to Yes

It still crashes, see #15281 (tested on a clean 2.16 installation on a Linux box with 16GB or ram).

#18 Updated by Johannes Kroeger almost 2 years ago

I guess I ran into this issue as well with a simple subtraction between two ~40000x40000 rasters.
Attached is a backtrace from gdb.
QGIS eats about 7GB of RAM while trying to do this, there would be 12GB more available for it though so I am not sure if it fills up something.
QGIS version 2.16.1-Nødebo.

It works perfectly fine if I do the same calculation with the gdal_calc.py utility outside QGIS.

If needed I can share the rasters, they are about 700MB each (compressed).

#19 Updated by Giovanni Manghi over 1 year ago

  • Status changed from Open to Feedback

Please try the NEW QGIS raster calculator available in the Processing toolbox of QGIS master (QGIS3).

#20 Updated by Giovanni Manghi over 1 year ago

If needed I can share the rasters, they are about 700MB each (compressed).

I would like to give them a try, thanks!

#21 Updated by Giovanni Manghi over 1 year ago

  • Regression? set to No
  • Easy fix? set to No

#22 Updated by Giovanni Manghi about 1 year ago

  • Description updated (diff)
  • Status changed from Feedback to Open

#23 Updated by Alessandro Pasotti 9 months ago

  • Related to Bug report #17523: Raster calculator cannot handle huge raster files added

#24 Updated by Pawel Stankiewicz 14 days ago

Hi

Please change the affected version for Master and Regression for Yes because I've had to install 2.8 to make calculations for which there is not enough memory in 3.2.1 - 16 GB RAM, Windows 10, grid 50,553 x 32,083 = 1.6 Gpx, 6 GB not compressed tiff.
In the spring 3.0 started showing messages about not enough memory after Windows with running QGIS crashed, I think it was Windows fault - it crashes from time to time. Restoring a disk image restored also the calculator, reinstalling QGIS was not enough. Yesterday Windows with idle QGIS in the background had crashed, before I wanted to start using the calculator after a few months break.
I use Aoemi Backupper Standard for weekly disk images and I can explore files in disk images. I could restore last week image if it would be really necessary - it's time-consuming.

Maybe QGIS is too cautious to block calculations? I prefer to make the job done with a risk of crashes to no job done. It would be better to get an info about risk, to save changes, and than to risk.

Thanks for this great, usually, program.

#25 Updated by Giovanni Manghi 13 days ago

Pawel Stankiewicz wrote:

Hi

Please change the affected version for Master and Regression for Yes because I've had to install 2.8 to make calculations for which there is not enough memory in 3.2.1 - 16 GB RAM, Windows 10, grid 50,553 x 32,083 = 1.6 Gpx, 6 GB not compressed tiff.

interesting, any chance you can allow us download the dataset (and tell us the formula used)?

#26 Updated by Pawel Stankiewicz 13 days ago

Sure: https://drive.google.com/file/d/1WNW-dlJcr83td4nuZSw9hQYJraDaP2lI/view?usp=sharing
Even grid * 1 formula doesn't work, but after choosing the grid extent. After choosing another layer extent the message isn't displayed, the calculator calculates, the result file has ~the correct size in MB, but with no data.

#27 Updated by Giovanni Manghi 10 days ago

  • Affected QGIS version changed from 2.16.0 to 3.3(master)

Pawel Stankiewicz wrote:

Sure: https://drive.google.com/file/d/1WNW-dlJcr83td4nuZSw9hQYJraDaP2lI/view?usp=sharing
Even grid * 1 formula doesn't work, but after choosing the grid extent. After choosing another layer extent the message isn't displayed, the calculator calculates, the result file has ~the correct size in MB, but with no data.

Confirmed:
here on Linux on master it always works (no crashes) but the results (with your dataset) is always a raster with only 0.
Wrong result, leaving the ticket as "corrupts data".

#28 Updated by Pawel Stankiewicz 9 days ago

Giovanni, do you confirm there is a problem or my description of the problem, which is a differnet from what you got? Previously I didn't write clearly enough: calculator calculates after selecting another smaller layer extent from a different area. 2.8 also makes no data in such case.

Also available in: Atom PDF