Bug report #20601

bug: Raster calculator produces empty results layer and no error message if input layer is one that has been renamed in QGIS layers panel

Added by Alister Hood 12 months ago. Updated 10 months ago.

Status:Reopened
Priority:Normal
Assignee:Alessandro Pasotti
Category:Processing/QGIS
Affected QGIS version:3.4.1 Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:No Resolution:
Crashes QGIS or corrupts data:No Copied to github as #:28421

Description

Right click on a layer in the QGIS "Layers" panel, and rename it.
Try to use it in the raster calculator (i.e. the one listed under QGIS raster analysis in processing). It will not work, but there will be no indication there has been an error (you will just get an empty result layer).
This is bad, because it appears to run successfully, but gives the wrong results.

Maybe this is related to the limitation in the implementation that prevents the raster calculator from listing and using more than one layer with the same name.

Associated revisions

Revision 18854dc7
Added by Alessandro Pasotti 12 months ago

Do not add duplicates in raster calc layers list

Related to #20601

Revision 1672d434
Added by Alessandro Pasotti 12 months ago

Raster calc: refresh layers list if a layer is renamed or added/removed

Fixes #20601 - bug: Raster calculator produces empty results layer and no error message if input layer is one that has been renamed in QGIS layers panel

Revision ecb6cde3
Added by Alessandro Pasotti 10 months ago

Pick up all layer entries references from the project context

This is the processing side of the duplicate layer
names bug, while the core part was already fixed,
the processing logic was extended to handle inputs
from models and full-path references.

Fixes #20601

History

#1 Updated by Alessandro Pasotti 12 months ago

That's probably the case, you can refer to layers/bands by [email protected] syntax, there is no way to tell one layer from another having the same name.
The solutions that comes up to my mind are:
- use the layer ID (which is very long, something like 'tmp_9a151baa_65a8_4abb_b425_0246bee6da0f')
- use the layer name and append "_x" like [email protected], [email protected] [email protected] etc.)

#2 Updated by Alister Hood 12 months ago

Hi Alessandro, yes, you are addressing this:

Maybe this is related to the limitation in the implementation that prevents the raster calculator from listing and using more than one layer with the same name.

The other obvious solution is to do it the same way as the gdal and saga raster calculators, i.e. assign rasters and bands to variables (A, B, C...). (It looks to me like using r.mapcalc from qgis has the same limitations as the qgis raster calculator - it says "Use original filenames, without extension)".

But I think at least in theory it should be possible to fix the main issue I was actually reporting even if that one isn't fixed.

#3 Updated by Alister Hood 12 months ago

I see the problem is more general - the raster calculator will not complain about any invalid raster name or band number. Type in a layer name that doesn't exist, maybe [email protected], or increase the band number for a raster that does exist, but doesn't have that many bands, and again you will just get an empty result layer.

IMHO this more general problem isn't as bad, although if you type in something that is wrong it would be nice to get an error message. But the specific example I gave is much worse i.e. you get the wrong result even though you don't type in anything wrong - QGIS lists a layer and you click on it, so you can quite rightly expect it to work!

#4 Updated by Alister Hood 12 months ago

  • Subject changed from bug: Raster calculator produces empty results layer if input layer has been renamed in QGIS layers panel to bug: Raster calculator produces empty results layer and no error message if input layer is one that has been renamed in QGIS layers panel

#5 Updated by Alessandro Pasotti 12 months ago

  • Assignee set to Alessandro Pasotti

#6 Updated by Alessandro Pasotti 12 months ago

Ok, here is the logic I'm working on in order to avoid the issue and prevent double loading in memory of layers:

Case 1: same data source but different layer name -> do not add to the list
Case 2: different data source but same layer name -> add _1, _2 etc to base name
Case 3: same data source and same layer name -> do not add to the list

For instance:

Source:      Layer name   Case  Reference
dem.tif      dem                [email protected]  (or [email protected], see below)
dem.tif      dem          3     -
dem.tif      dem_another: 1     -
slope.tif    slope              [email protected]
slope2.tif   slope        2     [email protected]

So, the idea is that we don't list multiple copies of the same data source because it would only mean to waste memory, in case of different data source with the same name we add _1, _2 etc. to the base name.

A tooltip in the dialog will show the real file path to the data source.

The only minor glitch here is that the order of layers in the code is not predictable, so we could end up with ([email protected] or [email protected]) depending on which layer comes first in the C++ container (we have no control over that order).

Case 1 is the only doubt I have, maybe more user friendly to list is and resolve data source identity in the code, but this is much more work, so for now I'll probably stick to this implementation.

#7 Updated by Alister Hood 12 months ago

I don't think case 1 is desirable.
Sometimes I have so many layers open (possibly several with the same name) that I lose track, and rather than figuring out which is the one I need to use it is quicker to add it again and rename it to something unique. If you implement case 1 then the unique name may not be the one that is listed.

I wouldn't worry too much about matching up identical layer sources to reduce memory usage - if memory usage is critical the user can take care not to use multiple layers with the same source in their formula!

#8 Updated by Alister Hood 12 months ago

For what it's worth, I'm not really clear on why the raster calculator has its own unique way of referring to layers that isn't used anywhere else. Is the advantage that you can just copy and paste from a list of calculations if you always use the same layer names?

#9 Updated by Anonymous 12 months ago

  • % Done changed from 0 to 100
  • Status changed from Open to Closed

#10 Updated by Alessandro Pasotti 12 months ago

Sorry I didn't get the notifications about your comments earlier (this is recurring a problem btw).

I think that case 1 (different layer names referring to the same data source) should result in a single reference in the calculator, the main reason to have different layers with the same data source is styling, and that doesn't apply to the calculator.

But I see your point and I added a tooltip to the list of references in the calculator, the tooltip shows the data source of the layer reference, so you can quickly identify which is what.

The synchronization is also implemented: the list of references is updated whenever something in the legend (the layers tree) changes.

Please test my patches (committed to master today) and let me know if that works well.

#11 Updated by Alessandro Pasotti 12 months ago

Alister Hood wrote:

For what it's worth, I'm not really clear on why the raster calculator has its own unique way of referring to layers that isn't used anywhere else. Is the advantage that you can just copy and paste from a list of calculations if you always use the same layer names?

I'm not clear either, but what's the alternative?

#12 Updated by Jürgen Fischer 12 months ago

Alessandro Pasotti wrote:

Sorry I didn't get the notifications about your comments earlier (this is recurring a problem btw).

with google mail. I added a txt record now. Please report if it helps - the SPF record apparently didn't.

#13 Updated by Alister Hood 11 months ago

  • Status changed from Closed to Reopened

the main reason to have different layers with the same data source is styling

Maybe, but I guess the main reason to have a layer name that doesn't match the file name is simply generating a "temporary" layer using a processing algorithm without specifying the output file.

Please test my patches (committed to master today) and let me know if that works well.

Sorry, I wasn't able to test quickly as my computer at work is "protected" by the moronic Trend Micro, so I can't install the osgeo4w dev builds. But I've now borrowed a laptop to test and it seems that the simple problem case has not been addressed.

To demonstrate, run a processing algorithm like v.torast without specifying an output file. Then try to use the resulting layer in the raster calculator and it will produce an empty layer.

#14 Updated by Giovanni Manghi 11 months ago

  • Status changed from Reopened to Feedback

To demonstrate, run a processing algorithm like v.torast without specifying an output file. Then try to use the resulting layer in the raster calculator and it will produce an empty layer.

confirmed but also unsure if is exactly the same issue as reported originally.

#15 Updated by Alister Hood 11 months ago

It is the same issue* - let's amend the description if you think it needs to be clarified.

Current description:

bug: Raster calculator produces empty results layer and no error message if input layer is one that has been renamed in QGIS layers panel

When I filed the ticket I tried to explain the problem as simply as I could. I didn't mean to imply that it affects only layers that (case 1) the user has manually renamed after they were loaded, and not layers that (case 2) were programmatically loaded with a name that doesn't match the filename.

^* Unless you are suggesting that case 1 works already, even though case 2 does not. Are you?

#16 Updated by Giovanni Manghi 11 months ago

  • Status changed from Feedback to Open

#17 Updated by Alessandro Pasotti 10 months ago

This is actually the processing side of the bug, processing uses the C++ core QgsRasterCalculator class (that I've fixed) but it adds some more logic for models and other path layer references, my previous fix was based on the C++ core side, I'll check processing wrappers too.

#18 Updated by Alessandro Pasotti 10 months ago

  • Status changed from Open to Closed

#19 Updated by Alister Hood 10 months ago

  • Status changed from Closed to Reopened

Hi Alessandro. Are some of your fixes only in the upcoming 3.5?

The core "bug" is still present in 3.4.4 as originally described. i.e. rename a layer in the layers panel and it will be listed in the raster calculator by the new name (this is good, as there are situations where the user would never have seen the actual filename unless they specifically looked in the layer properties), but trying to use it will not work, and there will be no indication there has been an error (you will just get an empty result layer).

To demonstrate, run a processing algorithm like v.torast without specifying an output file. Then try to use the resulting layer in the raster calculator and it will produce an empty layer.

For the record, you can't demonstrate with v.torast anymore - it seems it or processing itself has been changed to load the output with a layer name that matches the filename.

Also available in: Atom PDF