Bug report #15685

Processing gdalwarp issues

Added by Giovanni Manghi about 3 years ago. Updated almost 3 years ago.

Status:Closed
Priority:Severe/Regression
Assignee:Victor Olaya
Category:Processing/GDAL
Affected QGIS version:master Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:No Resolution:wontfix
Crashes QGIS or corrupts data:No Copied to github as #:23608

Description

Input: http://svn.simo-project.org/tools/trunk/txt2latlon/data/gt30w020n90.tif

1)
On QGIS 2.16 with gdal 1.11 (QGIS Ubuntu 16.04 repos), try reprojecting to 3857 gives:

2.16 gdal 1.11
2016-10-09T21:14:26 2 Uncaught error while executing algorithm
Traceback (most recent call last):
File "/usr/share/qgis/python/plugins/processing/core/GeoAlgorithm.py", line 203, in execute
self.processAlgorithm(progress)
File "/usr/share/qgis/python/plugins/processing/algs/gdal/GdalAlgorithm.py", line 52, in processAlgorithm
commands = self.getConsoleCommands()
File "/usr/share/qgis/python/plugins/processing/algs/gdal/warp.py", line 179, in getConsoleCommands
if rastext and len(rastext_crs) > 0:
TypeError: object of type 'NoneType' has no len()

2)
On master with gdal 1.11 (QGIS Ubuntu 16.04 repos) the "te" parameter has been added (automatically in fact), where with gdal 1.11 te_srs is not available. This seems to cause a lot of issues

GDAL command output:
ERROR 1: Attempt to create 0x0 dataset is illegal,sizes must be larger than zero.
Creating output file that is 0P x 0L.

in factI have not been able to reproject any raster.

3)
On 2.16 with gdal 2.1 (QGIS Ubuntu 16.04 repos + Ubuntugis)

with the above input

gdalwarp -ot Float32 -t_srs EPSG:3857 -r near -of GTiff -te -20.0 40.0
20.0 90.0 -te_srs EPSG:4326 -co COMPRESS=DEFLATE -co PREDICTOR=1 -co
ZLEVEL=6 -wo OPTIMIZE_SIZE=TRUE gt30w020n90.tif OUTPUT.tif
GDAL command output:
ERROR 1: tolerance condition error
ERROR 1: -te_srs ignored since coordinate transformation failed.

if a smaller extent rather than -te -20.0 40.0
20.0 90.0 is choosen than works ok. This does not seems to be a problem with many other rasters, maybe the issue if the extent for this large wgs84 raster.

The issue here is not this one for me, the real issue is understand why "te" has been made mandatory (and bu consequence "te_srs" is automatically added/computed by Processing).

Associated revisions

Revision dc2df62b
Added by Sandro Santilli about 3 years ago

Make Extent and Extent CRS GDAL parameters optional

Specify what the default is when extent CRS is not specified.
Allow using "auto" to have extent automatically set to min covering extent.

Fixes #15685

Revision 06976a2e
Added by Sandro Santilli about 3 years ago

Make Extent and Extent CRS GDAL parameters optional

Specify what the default is when extent CRS is not specified.
Allow using "auto" to have extent automatically set to min covering extent.

Fixes #15685

History

#1 Updated by Giovanni Manghi about 3 years ago

Not sure if this affects also other gdal tools.

#2 Updated by Sandro Santilli about 3 years ago

  • Status changed from Open to In Progress
  • Assignee changed from Victor Olaya to Sandro Santilli

I'm trying to reproduce, can you confirm I should be useing the "Warp (reproject)" processing tool ?
Opening the dialog of that tool, both with current 2.16 branch and master_2, I get an "Invalid value for parameter 'Destination SRS'", no matter setting it (by default it is unset). What are the exact steps you take to reproduce the subject issue ?

#3 Updated by Sandro Santilli about 3 years ago

Giovanni: maybe it would be more useful to split this report in 3 different tickets ?
It's not easy to understand what you think should be done here. For example: are you saying you think -te should not be added with GDAL is < 2.0 ?

#4 Updated by Giovanni Manghi about 3 years ago

Hi Sandro,

I'm sorry to me the description is clear and detailed

Sandro Santilli wrote:

I'm trying to reproduce, can you confirm I should be useing the "Warp (reproject)" processing tool ?

yes of course

Opening the dialog of that tool, both with current 2.16 branch and master_2, I get an "Invalid value for parameter 'Destination SRS'", no matter setting it

(by default it is unset). What are the exact steps you take to reproduce the subject issue ?

you must test with different qgis releseas compiled against gdal 1.* and 2.*

It is not complicated:

  • qgis 2.16, gdal 1.1, operation always fails with the abobe error
  • qgis master, gdal 1.1, the "te" parameter is used (in that gdal version there is not the "te_srs" one) and it causes the above error
  • using qgis 2.6/master with gdal 2.* it mostly work in any case, execpt for geographically large raster in geographic CRS:The "te" parameter is automagically added to the resulting command (and this anyway shuld not happen, should be optional) and the warping can fail. But in this specific cases you choose an extent for "te" that is slighly smaller than the input then it works. The latter is of course is a gdalwarp issue. The real issue in QGIS is that in this toll "te" must not be mandatory (and also te_srs when available).

I don't think we should split this ticket. It is all part of the very same tool/issue.

#5 Updated by Sandro Santilli about 3 years ago

And the issue is that "-te" is computed wrong ? Or "-te" should not be passed in at all ?

#6 Updated by Giovanni Manghi about 3 years ago

Sandro Santilli wrote:

And the issue is that "-te" is computed wrong ? Or "-te" should not be passed in at all ?

it is not wrong per se, is the real extent of the input, but this does not play well in the gdalwarp command line command. But if you try with other rasters, smaller areas, different CRSs, it works. There is a problem in gdalwarp with that specific type of input. If only "te" was optional as it should be... at least the tool would become usable also in such cases (or of course the trick) is to choose a "te" extent smaller (even slightly smaller) that the input one.

#7 Updated by Sandro Santilli about 3 years ago

I'm trying to figure what sets that raster extent, beause at the time getConsoleCommands is reached (in algs/gdal/warp.py) the extent is already set to the layer extent.

#8 Updated by Sandro Santilli about 3 years ago

I found the culprit of non-optionality being gui/ExtentSelectionPanel, which takes an empty string as meaning "use min covering extent". If we want to make the extent optional we'll have to find another "special" value to signify "min covering extent".
How about "min" or "min-covering" or "auto" ?

#9 Updated by Giovanni Manghi about 3 years ago

Sandro Santilli wrote:

I found the culprit of non-optionality being gui/ExtentSelectionPanel, which takes an empty string as meaning "use min covering extent". If we want to make the extent optional we'll have to find another "special" value to signify "min covering extent".
How about "min" or "min-covering" or "auto" ?

we already have non mandatory parameters in gdal/processing, we cannot just use the same as for them?

#10 Updated by Sandro Santilli about 3 years ago

The param is already non-mandatory, but the GUI currently intercept a blank value to signify "use min extent" rather than "do not specify an extent". Do you want to completely drop support for such "use min extent" behavior or do you want it to be done via a special value ?

#11 Updated by Sandro Santilli about 3 years ago

  • % Done changed from 0 to 20
  • Pull Request or Patch supplied changed from No to Yes

Giovanni can you test this change ?
https://github.com/qgis/QGIS/pull/3638

#12 Updated by Giovanni Manghi about 3 years ago

  • % Done changed from 20 to 0

I tested your patches on master_2 (gdal 2.*).

If the "raster extent..." is left blank is all ok, no automatically added "te" pameter with the full extent of input.

On the other end if the extent is added

2016-10-19T17:38:21 2 Uncaught error while executing algorithm
Traceback (most recent call last):
File "C:/OSGeo4W/apps/qgis-dev/./python/plugins\\processing\\core\\GeoAlgorithm.py", line 203, in execute
self.processAlgorithm(progress)
File "C:/OSGeo4W/apps/qgis-dev/./python/plugins\\processing\\algs\\gdal\\GdalAlgorithm.py", line 51, in processAlgorithm
commands = self.getConsoleCommands()
File "C:/OSGeo4W/apps/qgis-dev/./python/plugins\\processing\\algs\\gdal\\warp.py", line 203, in getConsoleCommands
return ['gdalwarp', GdalUtils.escapeAndJoin(arguments)]
File "C:/OSGeo4W/apps/qgis-dev/./python/plugins\\processing\\algs\\gdal\\GdalUtils.py", line 174, in escapeAndJoin
if s0 != '-' and ' ' in s:
IndexError: string index out of range

#13 Updated by Sandro Santilli about 3 years ago

  • Status changed from In Progress to Closed

#14 Updated by Giovanni Manghi about 3 years ago

  • Assignee deleted (Sandro Santilli)
  • Status changed from Closed to Reopened

The problems with gdal < 2.* has to be fixed and/or retested after this latest patches.

#15 Updated by Sandro Santilli about 3 years ago

  • Assignee set to Sandro Santilli
  • Status changed from Reopened to Closed

Please file a new ticket if you see problems with GDAL-1.x.

#16 Updated by Giovanni Manghi about 3 years ago

  • Status changed from Closed to Reopened

Sandro Santilli wrote:

Please file a new ticket if you see problems with GDAL-1.x.

why should I file a new ticket if this issue were reported here in the first place?

#17 Updated by Sandro Santilli about 3 years ago

Because this ticket reports too many bugs and is more useful to restrict the scope of tickets.
You can see from the ticket title how broad it is: "Processing gdalwrarp issues". "issues" ?
Please file a ticket for each issue.

#18 Updated by Giovanni Manghi about 3 years ago

Sandro Santilli wrote:

Because this ticket reports too many bugs and is more useful to restrict the scope of tickets.
You can see from the ticket title how broad it is: "Processing gdalwrarp issues". "issues" ?
Please file a ticket for each issue.

they are not different issues, but different symptoms of the same buggy tool. There are no different "warp" tools depending on the different underlying libraries versions, it is just one. The user do not need to know anything about the underlying libraries. And for example if under Ubuntu I install QGIS usig the official qgis repo that does not make use of the Ubuntugis one what I get is QGIS compiled against gdal 1.1.*

#19 Updated by Sandro Santilli about 3 years ago

  • Status changed from Reopened to Closed

#20 Updated by Victor Olaya about 3 years ago

Sandro

I had to revert your change. It breaks many places where extents are used, and also it modifies the syntax of extent params (where empty or none means using min extent). I suggest we try to find a better solution in 3.0, but not in 2. I am planning a change for extents in 3, where extents can have an asigned CRS as well, so it makes sense to add this changes as well.

Sorry for the revert

#21 Updated by Sandro Santilli about 3 years ago

  • Affected QGIS version changed from master_2 to master
  • Status changed from Closed to Reopened
  • Assignee changed from Sandro Santilli to Victor Olaya
  • Target version changed from Version 2.18 to Version 3.0

Ok, sorry for the light push (I was hoping the testsuite would have caught regressions)

For the record, master_2 revert is 8de56bf01a38fd75574a3a5bac1bcd52b0d88c55

Reopening and assigning to you.

Also, since master_2 won't be maintained and 2.14 is unaffected, I'm changing affected version to master and target version to 3.0, is that correct ?

#22 Updated by Sandro Santilli about 3 years ago

  • Pull Request or Patch supplied changed from Yes to No

#23 Updated by Victor Olaya about 3 years ago

yes, make sense to me

Thanks a lot!

#24 Updated by Alexander Bruy almost 3 years ago

  • Status changed from Reopened to Feedback

As QGIS 3 will require GDAL 2.x I think we can close this, as with GDAL 2.x it works fine

#25 Updated by Giovanni Manghi almost 3 years ago

  • Resolution set to wontfix
  • Status changed from Feedback to Closed

Also available in: Atom PDF