Bug report #19594
"Memory Error" in Raster Merge
| Status: | Closed | ||
|---|---|---|---|
| Priority: | Normal | ||
| Assignee: | - | ||
| Category: | Processing/GDAL | ||
| Affected QGIS version: | 3.2.1 | Regression?: | No |
| Operating System: | Windows 10 Creators with 3000x2000 screen and 175% scaling | Easy fix?: | No |
| Pull Request or Patch supplied: | No | Resolution: | invalid |
| Crashes QGIS or corrupts data: | No | Copied to github as #: | 27421 |
Description
[To see this problem and others with more context, check #19584.]
I had three 400MB 1-meter DEM .tif files, and RiverGIS only lets you query one file name per project, so I wanted to consolidate them.
I tried Raster -> Miscellaneous -> Merge, but got a Memory Error every time. It did write the output .tif files, with the size expected, but they were solid black.
I cut the file size a bit, but Merge still got a memory error each time. And, for each new try, I found I had to close the Merge window and start over from the QGIS menu bar. If I just edited a setting and tried again, I got "Incorrect parameter value for INPUT". It would still show the files selected, and re-selecting the input files didn't help.
Win 10 Resource Monitor showed zero memory faults, and just a tiny blip from 41% to 51% memory used, for these "Memory Error" tries, so I don't think it was a hardware limitation.
I found a way to cut the file sizes more severely...
I checked the Merge command before clicking run this time:
gdal_merge.bat ot Float32 -of GTiff -o "C:/Users/loren/Giant Files/HEC-RAS Dam Break/QGIS Projects/FEMA808/DEM_2226_clip_man_merge.tif" --optfile C:/Users/loren/AppData/Local/Temp/processing_b1a1b0b73d594cc4a2747620a75d7b50\mergeInputFiles.txt
I was a bit suspicious of all the '/' dividers being valid in Windows... The file did exist:
C:\Users\loren\AppData\Local\Temp\processing_b1a1b0b73d594cc4a2747620a75d7b50\mergeInputFiles.txt
But it was blank before I clicked "go".
But it worked! With 225MB of input data instead of 955MB (or 1201MB input which had made an all-black 1565MB output file).
After the operation completed, the mergeInputFiles.txt file listed my input files.
Is this memory error a bug? Or an inherent limitation? Could there be a more informative error message, showing where the limit is and how much data could be handled? Maybe some documentation if there is a fixed limit?
In any case, it seems the "Incorrect parameter value for INPUT" error is a bug. Shouldn't we be able to edit a dialog and try again, without having to enter all the settings again?
Related issues
History
#1
Updated by Jürgen Fischer over 5 years ago
- Related to Bug report #19584: Raster Clip and Merge issues (QGIS 3.2.1) added
#2
Updated by Jürgen Fischer over 5 years ago
- Description updated (diff)
#3
Updated by Loren Amelang over 5 years ago
@Jürgen Fischer,
What is the proper procedure for including a link in a report? I've tried the "Link to a Wiki page" button, and just pasting in the link inline, and it seems like you have to fix them either way... Is there some way I could do it right the first time?
#4
Updated by Harrissou Santanna over 5 years ago
Hi,
When you are in edit/writing mode there's a help button at the right, on the toolbar that shows this
#5
Updated by Loren Amelang over 5 years ago
@Harrissou Santanna,
Even in English I find that incomprehensible. Am I supposed to write #19584 without pressing the (wiki) link button? Or inside the square brackets? Or do I need to put the "Demande/Issue" in front of it?
Oh - I scrolled farther and found "More Information".
Redmine links
Redmine allows hyperlinking between resources (issues, changesets, wiki pages...) from anywhere wiki formatting is used.
Link to an issue: #124 (displays #124, link is striked-through if the issue is closed)Guess that confirms #19584 without any other action or entry?
#6
Updated by Jürgen Fischer over 5 years ago
Loren Amelang wrote:
Guess that confirms #19584 without any other action or entry?
Yes. And links to comments with #ticket-commentno (ie. #19594-5). Links to commits with commit:sha (eg. 73f3164fe4d)
#7
Updated by Giovanni Manghi over 5 years ago
- Status changed from Open to Feedback
- Category changed from Processing/QGIS to Processing/GDAL
Please attach the data.
About the dialog: yes you can keep it open, look in the Processing general options.
#8
Updated by Loren Amelang over 5 years ago
The data are way beyond 5 MB! With 225 MB of input it still works. I don't know where the crossover is, but I know 1200 MB fails. If you really want to see them, I imagine any of the "1 meter DEM" tif files from https:\\viewer.nationalmap.gov\basic\?basemap=b1&category=ned,nedsrc&title=3DEP%20View#productGroupSearch would act the same. Mine are:
USGS_NED_one_meter_x46y436_CA_FEMA_R9_Russian_2017_IMG_2018
USGS_NED_one_meter_x47y435_CA_FEMA_R9_Russian_2017_IMG_2018
USGS_NED_one_meter_x47y436_CA_FEMA_R9_Russian_2017_IMG_2018
My project folder is 2.73 GB... Is there some tiny part of it that would help you?
The "keep open" option is checked by default, and the dialog stays open and editable - it just creates useless files if run a second time!
Except today all the exact same tests work! On the day I reported all of this, every time I tried to re-use a dialog it failed in some way, usually by creating bad files, all black, improperly located, or a processing error. I have no clue why.
#9
Updated by Giovanni Manghi over 5 years ago
Loren Amelang wrote:
The data are way beyond 5 MB! With 225 MB of input it still works. I don't know where the crossover is, but I know 1200 MB fails. If you really want to see them, I imagine any of the "1 meter DEM" tif files from https:\\viewer.nationalmap.gov\basic\?basemap=b1&category=ned,nedsrc&title=3DEP%20View#productGroupSearch would act the same. Mine are:
USGS_NED_one_meter_x46y436_CA_FEMA_R9_Russian_2017_IMG_2018
USGS_NED_one_meter_x47y435_CA_FEMA_R9_Russian_2017_IMG_2018
USGS_NED_one_meter_x47y436_CA_FEMA_R9_Russian_2017_IMG_2018My project folder is 2.73 GB... Is there some tiny part of it that would help you?
if is a memory error then is likely caused by your large datasets. Can you post a link from where we can download them?
Anyway... if is really a memory error this can be caused by: a limitation of your system (because of the large datasets/inputs) or (not likely) a GDAL bug. For this tools QGIS acts "just" a GUI for GDAL.
The "keep open" option is checked by default, and the dialog stays open and editable - it just creates useless files if run a second time!
Except today all the exact same tests work! On the day I reported all of this, every time I tried to re-use a dialog it failed in some way, usually by creating bad files, all black, improperly located, or a processing error. I have no clue why
not sure if I understand, it works for you now?
#10
Updated by Loren Amelang over 5 years ago
@Giovanni Manghi,
The huge DEM files are at:
https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/1m/IMG/USGS_NED_one_meter_x46y436_CA_FEMA_R9_Russian_2017_IMG_2018.zip
https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/1m/IMG/USGS_NED_one_meter_x47y435_CA_FEMA_R9_Russian_2017_IMG_2018.zip
https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/1m/IMG/USGS_NED_one_meter_x47y436_CA_FEMA_R9_Russian_2017_IMG_2018.zip
As I said, I watched my system memory usage and it only went up from 41% to 51% during the failed merges. I have 64-bit QGIS, but maybe there is some other limit?
I totally don't understand how all the re-used dialogs failed for all of that one day, and now they all seem to work. It is like why some days QGIS remembers its window size and position after it has been closed, and some days it starts as a tiny box in the middle of the screen - #19386. Hmmm... Could having been hibernated and re-awakened make a difference? Window size is in \AppData\Roaming\QGIS\QGIS3\profiles\default\QGIS, right? But how are connections to all these GDAL and other processes stored?
Still, within the same continuous session, the first run after opening a dialog would work, re-using that dialog would fail, closing it and going back to the QGIS menu would make it work again!
#11
Updated by Giovanni Manghi over 5 years ago
- Subject changed from "Memory Error" in Raster Merge, can't retry from same dialog? to "Memory Error" in Raster Merge
Loren Amelang wrote:
@Giovanni Manghi,
The huge DEM files are at:
https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/1m/IMG/USGS_NED_one_meter_x46y436_CA_FEMA_R9_Russian_2017_IMG_2018.zip
https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/1m/IMG/USGS_NED_one_meter_x47y435_CA_FEMA_R9_Russian_2017_IMG_2018.zip
https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/1m/IMG/USGS_NED_one_meter_x47y436_CA_FEMA_R9_Russian_2017_IMG_2018.zipAs I said, I watched my system memory usage and it only went up from 41% to 51% during the failed merges. I have 64-bit QGIS, but maybe there is some other limit?
see #19595-7
my bet is that is a limitation on your system (also Windows is known to have a terrible memory management when it starts to be very low). I have tested on a Windows VM with 8GB of RAM and had no issues.
I totally don't understand how all the re-used dialogs failed for all of that one day, and now they all seem to work. It is like why some days QGIS remembers its window size and position after it has been closed, and some days it starts as a tiny box in the middle of the screen - #19386. Hmmm... Could having been hibernated and re-awakened make a difference? Window size is in \AppData\Roaming\QGIS\QGIS3\profiles\default\QGIS, right? But how are connections to all these GDAL and other processes stored?
Still, within the same continuous session, the first run after opening a dialog would work, re-using that dialog would fail, closing it and going back to the QGIS menu would make it work again!
I have understand the issue you are seeing and I'm about to file a separate ticket.
#12
Updated by Giovanni Manghi over 5 years ago
Still, within the same continuous session, the first run after opening a dialog would work, re-using that dialog would fail, closing it and going back to the QGIS menu would make it work again!
I have understand the issue you are seeing and I'm about to file a separate ticket.
#13
Updated by Loren Amelang over 5 years ago
Glad to see the dialog re-use bug is repeatable for you!
As for "Memory Error", I just tried a few more times as reported in #19595. Only yesterday's two tries in #19596 have ever worked for the full size merge. And a third try with identical system conditions (but run from the OSGeo4W console) failed. I have 8GB of system memory, and it never shows using more than 58% of it.
Is there anything else I can do here?
#14
Updated by Giovanni Manghi over 5 years ago
- Resolution set to invalid
- Status changed from Feedback to Closed
Loren Amelang wrote:
Glad to see the dialog re-use bug is repeatable for you!
As for "Memory Error", I just tried a few more times as reported in #19595. Only yesterday's two tries in #19596 have ever worked for the full size merge. And a third try with identical system conditions (but run from the OSGeo4W console) failed. I have 8GB of system memory, and it never shows using more than 58% of it.
Is there anything else I can do here?
ok, the fact that in the same conditions (inputs, parameters) fails directly from the command line means that is absolutely not a QGIS issue. From the symptoms you describe to me is clearly a local issue (memory shortage, the fact that stops before using 100% may be not relevant, possibly the script does not uses 100$ of memory on purpose). You may want to inquiry the gdal mailing list, they may be useful to better troubleshoot your issue.