Bug report #19372

Named algorithm outputs in modeler fail when used as inputs

Added by Rudi von Staden over 2 years ago. Updated over 2 years ago.

Status:Closed
Priority:High
Assignee:-
Category:Processing/Modeller
Affected QGIS version:3.2 Regression?:Yes
Operating System:Windows 10 Easy fix?:No
Pull Request or Patch supplied:No Resolution:
Crashes QGIS or corrupts data:No Copied to github as #:27200

Description

Using the modeler in QGIS 3.2, if you give a name to the output of an algorithm, you can't use that output as input for a subsequent algorithm. Trying to do so results in a processing error (`Could not load source layer for INPUT: invalid value`). I have attached a test project with some data and a model to demonstrate the issue. Included is a folder with an equivalent project in QGIS 2.18 showing that the same model worked in previous versions.

The model runs two parallel processes which are identical (polygonize raster then extract by attribute) except the second names the output of the polygonize step before using it in the extract step. I am fairly sure that this is not an issue with the test algorithms used here, but more broadly with the mapping of input and output parameters in models. You can see in the model log below that the first 'extract by attribute' algorithm was able to find the input file (standard named `OUTPUT.shp`) but the second time the algorithm is executed it has an empty INPUT parameter. This is probably because the output of the previous step results in a file named `gdal_polygonize_2_DEM_vector.shp`, not `OUTPUT.shp` as usual.

I am not sure where the mapping of output and input names happens in the processing code (and whether it's executed in c++ or Python). If it's in Python I'd be happy to look into it if someone can tell me which class or file to start the investigation.

Processing algorithm…
Algorithm 'named_output_used_as_input' starting…
Input parameters: { 'dem' : 'C:/Users/rudi_000/Dropbox/GIS/shared/named_output_used_as_input_test/dem.tif', 'gdal:polygonize_2:DEM_vector' : 'C:/Users/rudi_000/AppData/Local/Temp/processing_37290c9f65694832953690d6bd83266d/bfa5cb1aca8a4c8b8a9b7da1b85da366/gdal_polygonize_2_DEM_vector.shp', 'native:extractbyattribute_1:Output 1' : 'memory:', 'native:extractbyattribute_2:Output 2 (named output)' : 'memory:' }
Prepare algorithm: gdal:polygonize_1
Running Polygonize 1 [1/4]
Input Parameters: { BAND: 1, EIGHT_CONNECTEDNESS: False, FIELD: 'DN', INPUT: 'C:/Users/rudi_000/Dropbox/GIS/shared/named_output_used_as_input_test/dem.tif', OUTPUT: 'C:/Users/rudi_000/AppData/Local/Temp/processing_37290c9f65694832953690d6bd83266d/e715a21a7ae64c79a5b864dc72c5a6b1/OUTPUT.shp' }
GDAL command:
cmd.exe /C gdal_polygonize.bat C:/Users/rudi_000/Dropbox/GIS/shared/named_output_used_as_input_test/dem.tif C:/Users/rudi_000/AppData/Local/Temp/processing_37290c9f65694832953690d6bd83266d/e715a21a7ae64c79a5b864dc72c5a6b1/OUTPUT.shp -b 1 -f "ESRI Shapefile" None DN
GDAL command output:
0...10...20...30...40...50...60...70...80...90...Creating output C:/Users/rudi_000/AppData/Local/Temp/processing_37290c9f65694832953690d6bd83266d/e715a21a7ae64c79a5b864dc72c5a6b1/OUTPUT.shp of format ESRI Shapefile.
100 - done.
OK. Execution took 29.186 s (1 outputs).
Prepare algorithm: native:extractbyattribute_1
Running Extract by attribute [2/4]
Input Parameters: { FIELD: 'DN', INPUT: 'C:/Users/rudi_000/AppData/Local/Temp/processing_37290c9f65694832953690d6bd83266d/e715a21a7ae64c79a5b864dc72c5a6b1/OUTPUT.shp', OPERATOR: 0, OUTPUT: 'memory:', VALUE: '1' }
OK. Execution took 0.116 s (1 outputs).
Prepare algorithm: gdal:polygonize_2
Running Polygonize 2 (named output) [3/4]
Input Parameters: { BAND: 1, EIGHT_CONNECTEDNESS: False, FIELD: 'DN', INPUT: 'C:/Users/rudi_000/Dropbox/GIS/shared/named_output_used_as_input_test/dem.tif', OUTPUT: 'C:/Users/rudi_000/AppData/Local/Temp/processing_37290c9f65694832953690d6bd83266d/bfa5cb1aca8a4c8b8a9b7da1b85da366/gdal_polygonize_2_DEM_vector.shp' }
GDAL command:
cmd.exe /C gdal_polygonize.bat C:/Users/rudi_000/Dropbox/GIS/shared/named_output_used_as_input_test/dem.tif C:/Users/rudi_000/AppData/Local/Temp/processing_37290c9f65694832953690d6bd83266d/bfa5cb1aca8a4c8b8a9b7da1b85da366/gdal_polygonize_2_DEM_vector.shp -b 1 -f "ESRI Shapefile" None DN
GDAL command output:
0...10...20...30...40...50...60...70...80...90...Creating output C:/Users/rudi_000/AppData/Local/Temp/processing_37290c9f65694832953690d6bd83266d/bfa5cb1aca8a4c8b8a9b7da1b85da366/gdal_polygonize_2_DEM_vector.shp of format ESRI Shapefile.
100 - done.
OK. Execution took 29.425 s (1 outputs).
Prepare algorithm: native:extractbyattribute_2
Running Extract by attribute [4/4]
Input Parameters: { FIELD: 'DN', INPUT: '', OPERATOR: 0, OUTPUT: 'memory:', VALUE: '1' }
Could not load source layer for INPUT: invalid value
Error encountered while running Extract by attribute
Error encountered while running Extract by attribute
Execution failed after 59.70 seconds
Loading resulting layers
Algorithm 'named_output_used_as_input' finished

named_output_used_as_input_test.zip - Sample project with model and test data (165 KB) Rudi von Staden, 2018-07-09 10:51 PM

Associated revisions

Revision 41825cab
Added by Nyall Dawson over 2 years ago

[processing] Also accept QgsProcessingOutputLayerDefinition for
layer input values

Since these may result from the output of an earlier child algorithm running
in a processing model.

Fixes #19372

Revision 257d64d1
Added by Nyall Dawson over 2 years ago

[processing] Also accept QgsProcessingOutputLayerDefinition for
layer input values

Since these may result from the output of an earlier child algorithm running
in a processing model.

Fixes #19372

(cherry-picked from 41825cab)

History

#1 Updated by Nyall Dawson over 2 years ago

  • Status changed from Open to In Progress

#2 Updated by Nyall Dawson over 2 years ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Closed

Also available in: Atom PDF