Bug report #19596: Can't re-use "Clip Raster by Extent" dialog; boundary offset? - QGIS Application - QGIS Issue Tracking

Bug report #19596

Can't re-use "Clip Raster by Extent" dialog; boundary offset?

Added by Loren Amelang over 6 years ago. Updated about 6 years ago.

Status:	Closed
Priority:	Normal
Assignee:	-
Category:	Processing/GDAL
Affected QGIS version:	3.2.1	Regression?:	No
Operating System:	Windows 10 Creators with 3000x2000 screen and 175% scaling	Easy fix?:	No
Pull Request or Patch supplied:	No	Resolution:	duplicate
Crashes QGIS or corrupts data:	No	Copied to github as #:	27423

Description

[To see this problem and others with more context, check #19584.]

I had three 400MB 1-meter DEM .tif files, and RiverGIS only lets you query one file name per project, so I wanted to consolidate them. I tried Raster -> Miscellaneous -> Merge, but got a Memory Error every time.

I thought I'd cut the file size, so I tried Raster -> Extraction -> Clip Raster by Extent (Select extent on canvas)
First file worked, second showed correct "zoom to" extent but no image; under the Layers entry it showed:
1.79769e+308
-1.79769e+308
instead of
238.166
822.751

As I've seen in other QGIS functions, it turns out you can't edit and re-use a dialog! You must close it and start over and re-enter everything each time. Wish I could have just changed two characters between each of my files!

After the manual clip, there were a couple of (narrow, V-shaped) empty cracks protruding in from the file boundaries along the line where the clipped segment should meet the adjoining file. They met perfectly on the screen before the clip. I found I has to draw my manual boundary at least 300' beyond the apparent edge of the DEM to get it all! Seems strange... Yes, all the files and the map image were in the same 2226 project CRS.

Incorrect input for merge - second try.JPG (133 KB) Loren Amelang, 2018-08-14 10:21 PM

Related issues

History

#1 Updated by Jürgen Fischer over 6 years ago

Related to Bug report #19584: Raster Clip and Merge issues (QGIS 3.2.1) added

#2 Updated by Jürgen Fischer over 6 years ago

Description updated (diff)

#3 Updated by Giovanni Manghi over 6 years ago

Category changed from Processing/QGIS to Processing/GDAL
Status changed from Open to Feedback

I had three 400MB 1-meter DEM .tif files, and RiverGIS only lets you query one file name per project, so I wanted to consolidate them. I tried Raster -> Miscellaneous -> Merge, but got a Memory Error every time.

does the same operation done directly by the command line (using the gdal_merge.py tool) works?

As I've seen in other QGIS functions, it turns out you can't edit and re-use a dialog! You must close it and start over and re-enter everything each time. Wish I could have just changed two characters between each of my files!

yes of course you can, among the Processing options you have one to keep the tools dialog open after a run

Please add project/data with exact steps you are following and results vs expected results. The description as is now is not very useful without the data.

#4 Updated by Loren Amelang over 6 years ago

    I had three 400MB 1-meter DEM .tif files, and RiverGIS only lets you query one file name per project, so I wanted to consolidate them. I tried Raster -> Miscellaneous -> Merge, but got a Memory Error every time.

does the same operation done directly by the command line (using the gdal_merge.py tool) works?

I'm way beyond my depth here! But trying to learn...

The Processing menu item makes this command:

cmd.exe /C gdal_merge.bat -ot Float32 -of GTiff -o C:/Users/loren/AppData/Local/Temp/processing_6d811e1bb3544572b1fa6f5c07aa1ad7/da78c07429e54d99b3ef7812227c7d73/OUTPUT.tif --optfile C:/Users/loren/AppData/Local/Temp/processing_6d811e1bb3544572b1fa6f5c07aa1ad7\mergeInputFiles.txt
Doesn't look like Python, looks like Windows command line... But:

PS C:\Program Files\QGIS 3.2\bin> ./gdal_merge.bat -ot Float32 -of GTiff -o C:/Users/loren/AppData/Local/Temp/processing_6d811e1bb3544572b1fa6f5c07aa1ad7/bd850c5cab81497d844340896a39ab74/OUTPUT.tif --optfile C:/Users/loren/AppData/Local/Temp/processing_6d811e1bb3544572b1fa6f5c07aa1ad7\mergeInputFiles.txt The system cannot find the path specified. ImportError: No module named site PS C:\Program Files\QGIS 3.2\bin>
Fixing the '/' dividers doesn't help:

PS C:\Program Files\QGIS 3.2\bin> ./gdal_merge.bat -ot Float32 -of GTiff -o C:\Users\loren\AppData\Local\Temp\processing_6d811e1bb3544572b1fa6f5c07aa1ad7\bd850c5cab81497d844340896a39ab74\OUTPUT.tif --optfile C:\Users\loren\AppData\Local\Temp\processing_6d811e1bb3544572b1fa6f5c07aa1ad7\mergeInputFiles.txt The system cannot find the path specified. ImportError: No module named site PS C:\Program Files\QGIS 3.2\bin>

Not sure what it is not finding:

I give up on getting this section and the ones below formatted as Code - it just doesn't work, whether I type the char or click the button!

PS C:\Program Files\QGIS 3.2\bin> ls gdal_merge.bat

Directory: C:\Program Files\QGIS 3.2\bin

Mode LastWriteTime Length Name
---- ------------- ------ ----
~~a---~~ 6/26/2018 4:15 PM 99 gdal_merge.bat

PS C:\Program Files\QGIS 3.2\bin> ls C:\Users\loren\AppData\Local\Temp\processing_6d811e1bb3544572b1fa6f5c07aa1ad7\bd850
c5cab81497d844340896a39ab74
PS C:\Program Files\QGIS 3.2\bin>

(Output file doesn't exist yet, but path works.)

@
PS C:\Program Files\QGIS 3.2\bin> ls C:\Users\loren\AppData\Local\Temp\processing_6d811e1bb3544572b1fa6f5c07aa1ad7\merge
InputFiles.txt

Directory: C:\Users\loren\AppData\Local\Temp\processing_6d811e1bb3544572b1fa6f5c07aa1ad7

Mode LastWriteTime Length Name
---- ------------- ------ ----
~~a---~~ 8/13/2018 10:46 AM 274 mergeInputFiles.txt

PS C:\Program Files\QGIS 3.2\bin>
@

Running it via the Processing menu shows this:
@
Processing algorithm…
Algorithm 'Merge' starting…
Input parameters: { 'DATA_TYPE' : 5, 'INPUT' : ['C:\Users\loren\Giant Files\HEC-RAS Dam Break\QGIS Projects\FEMA808\DEM_46_436_to2226.tif','C:\Users\loren\Giant Files\HEC-RAS Dam Break\QGIS Projects\FEMA808\DEM_47_435_to2226.tif','C:\Users\loren\Giant Files\HEC-RAS Dam Break\QGIS Projects\FEMA808\DEM_47_436_to2226.tif'], 'NODATA_INPUT' : None, 'NODATA_OUTPUT' : None, 'OPTIONS' : '', 'OUTPUT' : 'C:\Users\loren\AppData\Local\Temp\processing_6d811e1bb3544572b1fa6f5c07aa1ad7\da78c07429e54d99b3ef7812227c7d73\OUTPUT.tif', 'PCT' : False, 'SEPARATE' : False }

GDAL command:
cmd.exe \C gdal_merge.bat -ot Float32 -of GTiff -o C:\Users\loren\AppData\Local\Temp\processing_6d811e1bb3544572b1fa6f5c07aa1ad7\da78c07429e54d99b3ef7812227c7d73\OUTPUT.tif --optfile C:\Users\loren\AppData\Local\Temp\processing_6d811e1bb3544572b1fa6f5c07aa1ad7\mergeInputFiles.txt
GDAL command output:
0Traceback (most recent call last):

File "C:\OSGEO4~1\bin\gdal_merge.py", line 540, in <module>

sys.exit(main())

File "C:\OSGEO4~1\bin\gdal_merge.py", line 526, in main

fi.copy_into( t_fh, band, band, nodata )

File "C:\OSGEO4~1\bin\gdal_merge.py", line 270, in copy_into

nodata_arg )

File "C:\OSGEO4~1\bin\gdal_merge.py", line 77, in raster_copy

m_band )

File "C:\OSGEO4~1\bin\gdal_merge.py", line 128, in raster_copy_with_mask

data_dst = t_band.ReadAsArray( t_xoff, t_yoff, t_xsize, t_ysize )

File "C:\OSGEO4~1\apps\Python27\lib\site-packages\osgeo\gdal.py", line 2449, in ReadAsArray

callback_data = callback_data)

File "C:\OSGEO4~1\apps\Python27\lib\site-packages\osgeo\gdal_array.py", line 339, in BandReadAsArray

buf_obj = numpy.empty([buf_ysize,buf_xsize], dtype = typecode)

MemoryError

Execution completed in 9.28 seconds
Results: {'OUTPUT': <QgsProcessingOutputLayerDefinition {'sink':C:\Users\loren\AppData\Local\Temp\processing_6d811e1bb3544572b1fa6f5c07aa1ad7\da78c07429e54d99b3ef7812227c7d73\OUTPUT.tif, 'createOptions': {'fileEncoding': 'System'}}>}

Loading resulting layers
Algorithm 'Merge' finished
@

    As I've seen in other QGIS functions, it turns out you can't edit and re-use a dialog! You must close it and start over and re-enter everything each time. Wish I could have just changed two characters between each of my files!

yes of course you can, among the Processing options you have one to keep the tools dialog open after a run

That is checked, and the dialog stays open and editable - it just creates useless files if run a second time!

Except today the exact same tests work! On the day I reported all of this, every time I tried to re-use a dialog it failed in some way, usually by creating bad files, all black, improperly located, or a processing error. I have no clue why.

Please add project\data with exact steps you are following and results vs expected results. The description as is now is not very useful without the data.

Unfortunately the project and data are way beyond 5 MB. More like 1.5 GB! Not sure what I can do about that...

(Sorry this is such a mess. Some of the content apparently conflicts with the markup language...)

#5 Updated by Giovanni Manghi over 6 years ago

Loren Amelang wrote:

[...]
I'm way beyond my depth here! But trying to learn...

The Processing menu item makes this command:

@cmd.exe /C gdal_merge.bat -ot Float32 -of GTiff -o C:/Users/loren/AppData/Local/Temp/processing_6d811e1bb3544572b1fa6f5c07aa1ad7/da78c07429e54d99b3ef7812227c7d73/OUTPUT.tif --optfile C:/Users/loren/AppData/Local/Temp/processing_6d811e1bb3544572b1fa6f5c07aa1ad7\mergeInputFiles.txt

just open the OSGeo4W shell that you find in any QGIS installation (Windows start menu), then from there you can launch any of the GDAL/OGR utilities, i.e.:

gdal_merge ... ... ...

Unfortunately the project and data are way beyond 5 MB. More like 1.5 GB! Not sure what I can do about that...

any download link?

#6 Updated by Loren Amelang over 6 years ago

File Incorrect input for merge - second try.JPG added

@Giovanni Manghi,

A new toy to learn about! Copying the GUI command to the OSGeo4W shell before running it in the GUI fails because it hasn't written the input file names yet.
Running it in the GUI...

OMG! Today it worked! Made a 1.6 GB file with proper data! I've tried it at least a dozen time before and all of them failed.

Processing algorithm…
Algorithm 'Merge' starting…
Input parameters:
{ 'DATA_TYPE' : 5, 'INPUT' : ['C:/Users/loren/Giant Files/HEC-RAS Dam Break/QGIS Projects/FEMA808/DEM_46_436_to2226.tif','C:/Users/loren/Giant Files/HEC-RAS Dam Break/QGIS Projects/FEMA808/DEM_47_435_to2226.tif','C:/Users/loren/Giant Files/HEC-RAS Dam Break/QGIS Projects/FEMA808/DEM_47_436_to2226.tif'], 'NODATA_INPUT' : None, 'NODATA_OUTPUT' : None, 'OPTIONS' : '', 'OUTPUT' : 'C:/Users/loren/AppData/Local/Temp/processing_544af2b511d3484ab90a79f29ca1f08b/64f69b0b29424416b5b1187fc8f85bfe/OUTPUT.tif', 'PCT' : False, 'SEPARATE' : False }

GDAL command:
cmd.exe /C gdal_merge.bat -ot Float32 -of GTiff -o C:/Users/loren/AppData/Local/Temp/processing_544af2b511d3484ab90a79f29ca1f08b/64f69b0b29424416b5b1187fc8f85bfe/OUTPUT.tif --optfile C:/Users/loren/AppData/Local/Temp/processing_544af2b511d3484ab90a79f29ca1f08b\mergeInputFiles.txt
GDAL command output:
0...10...20...30...40...50...60...70...80...90...100 - done.

Execution completed in 39.03 seconds
Results:
{'OUTPUT': <QgsProcessingOutputLayerDefinition {'sink':C:/Users/loren/AppData/Local/Temp/processing_544af2b511d3484ab90a79f29ca1f08b/64f69b0b29424416b5b1187fc8f85bfe/OUTPUT.tif, 'createOptions': {'fileEncoding': 'System'}}>}

Loading resulting layers
Algorithm 'Merge' finished

So the only difference I can think of is that the OSGeo4W shell was open in the background this time...
Closed it, went back to the merge dialog:
Incorrect input for merge - second try.JPG

This time re-selecting the same three input files (making six) and then un-selecting the first three, made it run.
That never worked before (unless the precise order of selecting or un-selecting makes a difference...)
And it made another perfect 1.6GB layer!

So since I'm an incorrigible Aspie, I re-opened the OSGeo4W shell and pasted in the command:

C:\>gdal_merge.bat -ot Float32 -of GTiff -o C:/Users/loren/AppData/Local/Temp/processing_544af2b511d3484ab90a79f29ca1f08b/719c44590dae40cdb91a61288dbac23b/OUTPUT.tif --optfile C:/Users/loren/AppData/Local/Temp/processing_544af2b511d3484ab90a79f29ca1f08b\mergeInputFiles.txt
0Traceback (most recent call last):
  File "C:\OSGEO4~1\bin\gdal_merge.py", line 540, in <module>
    sys.exit(main())
  File "C:\OSGEO4~1\bin\gdal_merge.py", line 526, in main
    fi.copy_into( t_fh, band, band, nodata )
  File "C:\OSGEO4~1\bin\gdal_merge.py", line 270, in copy_into
    nodata_arg )
  File "C:\OSGEO4~1\bin\gdal_merge.py", line 77, in raster_copy
    m_band )
  File "C:\OSGEO4~1\bin\gdal_merge.py", line 131, in raster_copy_with_mask
    to_write = Numeric.choose( mask_test, (data_src, data_dst) )
  File "C:\OSGEO4~1\apps\Python27\lib\site-packages\numpy\core\fromnumeric.py", line 354, in choose
    return _wrapfunc(a, 'choose', choices, out=out, mode=mode)
  File "C:\OSGEO4~1\apps\Python27\lib\site-packages\numpy\core\fromnumeric.py", line 57, in _wrapfunc
    return getattr(obj, method)(*args, **kwds)
MemoryError

C:\>

I'm beginning to think I need to take this all up with an astrologer or shaman...
Can you think of anything that would inject such randomness into QGIS?
I guess my Firefox has its share of randomness, but I blame most of that on the crazy web pages it has to deal with.
I can do huge graphics manipulations in Photoshop and it is absolutely rock solid, so I don't think it is my hardware.

If you want to try your luck, the huge DEM files are at:
https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/1m/IMG/USGS_NED_one_meter_x46y436_CA_FEMA_R9_Russian_2017_IMG_2018.zip
https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/1m/IMG/USGS_NED_one_meter_x47y435_CA_FEMA_R9_Russian_2017_IMG_2018.zip
https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/1m/IMG/USGS_NED_one_meter_x47y436_CA_FEMA_R9_Russian_2017_IMG_2018.zip

Gah! I hate these systems that format regular text as wide as the widest image or "pre" section!
Hopefully I've edited it enough to make it readable...

#7 Updated by Giovanni Manghi over 6 years ago

If you want to try your luck, the huge DEM files are at:
https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/1m/IMG/USGS_NED_one_meter_x46y436_CA_FEMA_R9_Russian_2017_IMG_2018.zip
https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/1m/IMG/USGS_NED_one_meter_x47y435_CA_FEMA_R9_Russian_2017_IMG_2018.zip
https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/1m/IMG/USGS_NED_one_meter_x47y436_CA_FEMA_R9_Russian_2017_IMG_2018.zip

Gah! I hate these systems that format regular text as wide as the widest image or "pre" section!
Hopefully I've edited it enough to make it readable...

see #19595-7

I have no more to add. Here it works all as expected, both on Windows and Linux.

#8 Updated by Loren Amelang over 6 years ago

"I have no more to add."

I guess we're just going to blame my Surface Book? Today's tests are in #19595, all merges failed with the "Memory Error".

@
0Traceback (most recent call last):

File "C:\OSGEO4~1\bin\gdal_merge.py", line 540, in <module>

sys.exit(main())

File "C:\OSGEO4~1\bin\gdal_merge.py", line 526, in main

fi.copy_into( t_fh, band, band, nodata )
@

WHY IS IT NOTHING I CAN DO WILL MAKE THE "CODE" OPTION WORK HERE?

It apparently does not crash out of the copy loop, it goes on to the sys.exit call and traces back from there... Or is that yet another Python option I haven't seen yet?

Is there some way to set the Python this calls to a more detailed debug level?

#9 Updated by Giovanni Manghi about 6 years ago

Status changed from Feedback to Closed
Resolution set to duplicate

Closed in favor of #19628.

About the "offset" issue ("fter the manual clip, there were a couple of (narrow, V-shaped) empty cracks protruding in from the file boundaries along the line where the clipped segment should meet the adjoining file. They met perfectly on the screen before the clip. I found I has to draw my manual boundary at least 300' beyond the apparent edge of the DEM to get it all! Seems strange... Yes, all the files and the map image were in the same 2226 project CRS."), which I haven't understand well, it must be filed in a separate ticket.

#10 Updated by Giovanni Manghi about 6 years ago

I guess we're just going to blame my Surface Book? Today's tests are in #19595, all merges failed with the "Memory Error".

@
0Traceback (most recent call last):

File "C:\OSGEO4~1\bin\gdal_merge.py", line 540, in <module>

sys.exit(main())

File "C:\OSGEO4~1\bin\gdal_merge.py", line 526, in main

fi.copy_into( t_fh, band, band, nodata )
@

WHY IS IT NOTHING I CAN DO WILL MAKE THE "CODE" OPTION WORK HERE?

It apparently does not crash out of the copy loop, it goes on to the sys.exit call and traces back from there... Or is that yet another Python option I haven't seen yet?

Is there some way to set the Python this calls to a more detailed debug level?

this is not related to the description of this ticket.

Also available in: Atom PDF

QGIS » QGIS Application

Issues

Custom queries