Bug report #17298

Processing: gdal merge python error on QGIS 3

Added by Giovanni Manghi almost 3 years ago. Updated over 2 years ago.

Status:Closed
Priority:High
Assignee:Victor Olaya
Category:Processing/GDAL
Affected QGIS version:master Regression?:Yes
Operating System:Windows, MacOS Easy fix?:No
Pull Request or Patch supplied:No Resolution:fixed/implemented
Crashes QGIS or corrupts data:No Copied to github as #:25196

Description

Algorithm 'Merge' starting...
Input parameters:
{ 'INPUT' : ['C:/Users/qgis/Desktop/merge_null/OUTPUT1.tif','C:/Users/qgis/Desktop/merge_null/OUTPUT2.tif'], 'PCT' : False, 'SEPARATE' : False, 'OPTIONS' : '', 'DATA_TYPE' : 5, 'OUTPUT' : 'C:/Users/qgis/AppData/Local/Temp/processing_311e3738739c4ad08b8f21b50bdff7b5/2d82bd2adfdc43c4989ea7cff35be575/OUTPUT.tif' }

GDAL command:
cmd.exe /C gdal_merge.bat -ot Float32 -of GTiff -o C:/Users/qgis/AppData/Local/Temp/processing_311e3738739c4ad08b8f21b50bdff7b5/2d82bd2adfdc43c4989ea7cff35be575/OUTPUT.tif C:/Users/qgis/Desktop/merge_null/OUTPUT1.tif C:/Users/qgis/Desktop/merge_null/OUTPUT2.tif
GDAL command output:
File "C:\OSGEO4~1\apps\Python36\lib\site.py", line 177 
file=sys.stderr) 
^ 
SyntaxError: invalid syntax 
Execution completed in 0.13 seconds
Results:
{'OUTPUT': <QgsProcessingOutputLayerDefinition {'sink':C:/Users/qgis/AppData/Local/Temp/processing_311e3738739c4ad08b8f21b50bdff7b5/2d82bd2adfdc43c4989ea7cff35be575/OUTPUT.tif, 'createOptions': {'fileEncoding': 'System'}}>}

Loading resulting layers
Algorithm 'Merge' finished

Other GDAL tools seems not affected.


Related issues

Related to QGIS Application - Bug report #15749: Processing: gdal merge converts no data in 0 Closed 2016-10-24
Related to QGIS Application - Bug report #17846: Polygonize (raster to vector) fails Closed 2018-01-12
Duplicated by QGIS Application - Bug report #18413: Incorrect Python ver. with GDAL Merge tool in QGIS 3 Closed 2018-03-12

History

#1 Updated by Jürgen Fischer almost 3 years ago

#2 Updated by Nyall Dawson almost 3 years ago

  • Operating System set to Windows

Seems a windows-specific issue

#3 Updated by Nyall Dawson almost 3 years ago

  • Resolution set to up/downstream

I've looked into this - I don't believe this is a QGIS issue, but rather an osgeo4w packaging issue. The issue is that the gdal_merge.bat script runs gdal_merge.py under python2, not python3, which leads to the resulting error. (Indeed this issue applies to all the gdal_*.bat scripts). We'd require equivalent batch files which explicitly run these python utilities under python3.

#4 Updated by Jürgen Fischer almost 3 years ago

  • Description updated (diff)

#5 Updated by Jürgen Fischer almost 3 years ago

  • Status changed from Open to Feedback

python should point to python3 when run from QGIS3:

Algorithm 'Verschmelzen' starting...
Input parameters:
{ 'INPUT' : ['D:/TEMP/test/10438_230601_d_gr_01.tiff'], 'PCT' : False, 'SEPARATE' : False, 'OPTIONS' : '', 'DATA_TYPE' : 5, 'OUTPUT' : 'C:/Users/jef.NORBIT/AppData/Local/Temp/processing_21c16c31984843028ddf4bacebba901a/c3b0c4c14a65448da9569b9fab6b30b5/OUTPUT.tif' }

GDAL command:
cmd.exe /C gdal_merge.bat -ot Float32 -of GTiff -o C:/Users/jef.NORBIT/AppData/Local/Temp/processing_21c16c31984843028ddf4bacebba901a/c3b0c4c14a65448da9569b9fab6b30b5/OUTPUT.tif D:\TEMP\test\XXX.tiff
GDAL command output:

C:\WINDOWS\system32>python --version 
Python 3.6.0 

C:\WINDOWS\system32>path 
PATH=C:\OSGEO4~1\apps\Python36\lib\site-packages\numpy\core;C:\OSGEO4~1\apps\qgis-dev\bin;C:\OSGEO4~1\apps\grass\grass-7.2.2\lib;C:\OSGEO4~1\apps\grass\grass-7.2.2\bin;C:\OSGEO4~1\apps\Python36;C:\OSGEO4~1\apps\Python36\Scripts;C:\OSGEO4~1\apps\qt5\bin;C:\OSGEO4~1\apps\Python27\Scripts;C:\OSGEO4~1\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\system32\WBem;C:\OSGEO4~1\apps\Python36\lib\site-packages\pypiwin32_system32; 
Warning 1: TIFFFetchNormalTag:ASCII value for tag "Software" contains null byte in value; value incorrectly truncated during reading due to implementation limitations 
0Warning 1: TIFFFetchNormalTag:ASCII value for tag "Software" contains null byte in value; value incorrectly truncated during reading due to implementation limitations 
...10...20...30...40...50...60...70...80...90...100 - done. 
Execution completed in 0.64 seconds
Results:
{'OUTPUT': <QgsProcessingOutputLayerDefinition {'sink':C:/Users/jef.NORBIT/AppData/Local/Temp/processing_21c16c31984843028ddf4bacebba901a/c3b0c4c14a65448da9569b9fab6b30b5/OUTPUT.tif, 'createOptions': {'fileEncoding': 'System'}}>}

Lade Ergebnis Layer
Algorithm 'Verschmelzen' finished

with python --version and path added to gdal_merge.bat

#6 Updated by Giovanni Manghi almost 3 years ago

  • Resolution deleted (up/downstream)
  • Status changed from Feedback to Open

#7 Updated by Alexander Bruy over 2 years ago

#8 Updated by Salvatore Larosa over 2 years ago

  • Operating System changed from Windows to Windows, MacOS

I can confirm it even on macos.

#9 Updated by Giovanni Manghi over 2 years ago

Still true on the latest master.

#10 Updated by Luigi Pirelli over 2 years ago

not confirmed with qgis master installed yesterday on win10 on my laptop... twin check with Giovanni on his laptop confirm the issue. Checking what local condition generate the error

#11 Updated by Luigi Pirelli over 2 years ago

confirmed in this conditions:
Run qgis from start bar => e.g. run C:\<osgeo install>\bin\qgis-dev-bin-g7.4.0.exe

In my first test, that successfully end, I run qgis from cmd or osgeo shell running C:\<osgeo install>\bin\qgis-dev-g7.4.0.bat

e.g. the correct OSGeo4W env is loaded.

the problem seems a OSGe4W packaging problem

#12 Updated by Luigi Pirelli over 2 years ago

as suggested by @jürgen adding load of osge4w env just in any command wrapper fix the issue

#13 Updated by Luigi Pirelli over 2 years ago

what I'm not sure with the above fiex is that setting the osgeo4w env in each command wrapper does not allow to run all comands in a general custom osgeo4w env that can be useful to set for example to set a custom PATH or something else.

#14 Updated by Jürgen Fischer over 2 years ago

  • Resolution set to fixed/implemented
  • Status changed from Open to Closed

Fixed in OSGeo4W's python-core.

#15 Updated by Jürgen Fischer over 2 years ago

  • Duplicated by Bug report #18413: Incorrect Python ver. with GDAL Merge tool in QGIS 3 added

Also available in: Atom PDF