Bug report #16178

GDAL algorithms ignore feature filter

Added by Spencer Gardner about 7 years ago. Updated almost 6 years ago.

Status:Closed
Priority:Normal
Assignee:Victor Olaya
Category:Processing/GDAL
Affected QGIS version:2.18.3 Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:No Resolution:fixed/implemented
Crashes QGIS or corrupts data:No Copied to github as #:24088

Description

Note: does not affects only PostGIS layers

I ran a GDAL Processing algorithm today on a PostGIS layer and noticed that the algorithm ignores the feature filter on the input layer. This resulted in incorrect results since the output used data that I had filtered out of the input layer. It appears the problem is in the construction of the GDAL call, where the connection details are passed to GDAL but the -sql option is not used to pass the filter.

Steps to reproduce:
  1. Add PostGIS layer to map
  2. Set feature filter so that layer represents only a subset of the original table
  3. Run GDAL Rasterize (vector to raster) tool in Processing

Expected result:
New raster layer has nodata where features have been filtered out of the source layer.

Actual result:
New raster layer has data in places where features should have been filtered out.

rasterized.png (227 KB) Spencer Gardner, 2017-02-14 08:25 AM

filter.png - Feature filter (86.5 KB) Spencer Gardner, 2017-02-14 08:54 AM

dialog.png - Tool dialog (164 KB) Spencer Gardner, 2017-02-14 08:54 AM

rasterized_points.png - Results (103 KB) Spencer Gardner, 2017-02-14 08:54 AM

community_designations.zip (1.05 MB) Spencer Gardner, 2017-02-14 09:25 AM

History

#1 Updated by Giovanni Manghi about 7 years ago

  • Status changed from Open to Feedback

If we are speaking about a filter added in layer properties ("general" tab) then i cannot confirm, it works as expected for me on both Linux and Windows on 2.18.3 tested using a point postgis layer.

#2 Updated by Spencer Gardner about 7 years ago

Strange. I'm still reproducing this issue on 2.18.3 on Linux. I wonder if behavior is different for point data than for line or polygon.

I've attached a screengrab of the results to make the problem clearer. The blue "community_designations" layer is a PostGIS layer where I have used layer properties to filter features. As you can see, the results of the Rasterize tool produce values even in locations where features have been filtered out and I would have expected nodata results. I hope this helps.

#3 Updated by Giovanni Manghi about 7 years ago

Spencer Gardner wrote:

Strange. I'm still reproducing this issue on 2.18.3 on Linux. I wonder if behavior is different for point data than for line or polygon.

I've attached a screengrab of the results to make the problem clearer. The blue "community_designations" layer is a PostGIS layer where I have used layer properties to filter features. As you can see, the results of the Rasterize tool produce values even in locations where features have been filtered out and I would have expected nodata results. I hope this helps.

could you test with points please? thanks!

#4 Updated by Spencer Gardner about 7 years ago

I am reproducing this with point data as well. Attached are screengrabs where I set the feature filter, run the tool, and then the results, which show the output raster with values in locations where points have been filtered out. (The red points are the locations where I would expect data, everywhere else should be nodata.)

As I described before, I suspect the reason for this behavior is that the command line statement constructed in the tool dialog (see bottom of the screenshot) passes the table name and db connection details but doesn't include a SQL filter, so the GDAL routine runs on the unfiltered table.

#5 Updated by Giovanni Manghi about 7 years ago

As I described before, I suspect the reason for this behavior is that the command line statement constructed in the tool dialog (see bottom of the screenshot) passes the table name and db connection details but doesn't include a SQL filter, so the GDAL routine runs on the unfiltered table.

but this does not explain why here is all ok on two different OSes.

Are you using an old Processing version?
Do you have any "processing" folder inside "~/.qgis2/python/plugins"?

#6 Updated by Spencer Gardner about 7 years ago

Processing version is 2.12.99. No processing folder in ~/.qgis2/python/plugins. The VM this runs on was created last week and everything GIS-related was installed from the latest sources:

  • proj4 v 4.9.2
  • GDAL v 2.1.3
  • GEOS v 3.5.1

#7 Updated by Spencer Gardner about 7 years ago

For what it's worth, I'm also reproducing this issue in Windows on QGIS 2.14.6.

#8 Updated by Giovanni Manghi about 7 years ago

Spencer Gardner wrote:

For what it's worth, I'm also reproducing this issue in Windows on QGIS 2.14.6.

can you attach a sample of your data and the expression used for filtering? thanks!

#9 Updated by Spencer Gardner about 7 years ago

What's the best way to send you the data? pg_dump? Or can I export to GeoJSON and have you import on your DB?

#10 Updated by Giovanni Manghi about 7 years ago

Spencer Gardner wrote:

What's the best way to send you the data? pg_dump? Or can I export to GeoJSON and have you import on your DB?

or even a shapefile that I will import in a postgis db of mine. Whatever is handier for you. thanks!

#11 Updated by Spencer Gardner about 7 years ago

Polygon data attached. I'm filtering half of the features out with this query: "id % 2 = 0". I ran the GDAL Rasterize tool with attribute="abc_sort" and 50 x 50 tiles, although the default 100 x 100 should work too.

#12 Updated by Giovanni Manghi about 7 years ago

  • Subject changed from GDAL algorithms ignore feature filter on PostGIS layer to GDAL algorithms ignore feature filter
  • Status changed from Feedback to Open
  • Operating System deleted (Linux)

Yes there is something strange going on as I also can replicate now, but is not just a PostGIS layers thing.

Moreover I found that rasterizing using values from a column "id" on PostGIS layers does result in a only zeroes rasters (does not affect shapefiles).

#13 Updated by Spencer Gardner about 7 years ago

Unless I'm mistaken, the code is lacking the logic to identify a feature filter and pass it as a parameter to GDAL. The solution would be to have the tool append
-where
to the arguments array along with the feature filter logic.

#14 Updated by Giovanni Manghi almost 7 years ago

  • Regression? set to No
  • Easy fix? set to No

#15 Updated by Nyall Dawson almost 6 years ago

  • Status changed from Open to Closed
  • Description updated (diff)
  • Resolution set to fixed/implemented

Fixed in 3.0

Also available in: Atom PDF