Bug report #17131

Processing: Union and Intersection wrong results

Added by Giovanni Manghi over 6 years ago. Updated almost 6 years ago.

Status:Closed
Priority:High
Assignee:Victor Olaya
Category:Processing/QGIS
Affected QGIS version:2.18.12 Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:No Resolution:fixed/implemented
Crashes QGIS or corrupts data:Yes Copied to github as #:25030

Description

This ticket is a tentative to replace/resume other older tickets (#15962) about union/intersection issues in QGIS native tools.

Two sample projects with two sample datasets are attached, results vs expected results and results from other GIS packages are here below or added to the projects.

The sample datasets have been tested in QGIS master and on 2.18.12 with this https://github.com/qgis/QGIS/pull/5099 patched version of Processing that includes backports form master for union and intersection.

*) with the original version of the first dataset (taken from https://issues.qgis.org/attachments/download/10585/Sample1.zip):

the dataset has small geometry errors (auto-intersections).

**) Trying union/intersection on master it returns an error

Algorithm 'Union' starting...
Input parameters: {'INPUT': <QgsProcessingFeatureSourceDefinition {'source':AreaA20170911213135765, 'selectedFeaturesOnly': False}

, 'OUTPUT': <QgsProcessingOutputLayerDefinition {'sink':memory:, 'createOptions': {'fileEncoding': 'System'}}>, 'OVERLAY': <QgsProcessingFeatureSourceDefinition

{'source':AreaB20170911213135757, 'selectedFeaturesOnly': False}

}

Traceback (most recent call last): File "C:/OSGEO4~1/apps/qgis-dev/./python/plugins\processing\algs\qgis\Union.py", line 101, in processAlgorithm indexB = QgsSpatialIndex(sourceB.getFeatures(QgsFeatureRequest().setSubsetOfAttributes([]).setDestinationCrs(sourceA.sourceCrs())), feedback) Exception: unknown
Execution failed after 0.04 seconds

**) on 2.18.12 (with patched processing) the intersection on the unclean geometries return a clear message

GEOS geoprocessing error: One or more input features have invalid geometry.

while the union operation does not. The operations runs "normally", no warning/error is thrown (not even in QGIS log) and the a wrong result is added to the project (there are overlapped features should not exist, there are missing features and also there is no "union" of attributes for the resulting features representing the overlapped areas in the two inputs).

*) on a cleaned version of the first dataset:

**) Intersection on both 2.18.12 and master seems ok.

*) Union is *wrong (is slightly different in 2.18.12 and master, but still wrong in both cases): there are overlapped features that should not exist in the result (compare with SAGA, GRASS and ArcGIS results, all added to the test project) and also there is no "union" of attributes for the resulting features representing the overlapped areas in the two inputs (again compare with SAGA, GRASS and ArcGIS results)

*) second project/dataset:

meant to test the scenario as it is shown on ESRI docs and represented by the left part in the following image

where the circles and the rectangle are in separate layers.

Note: ESRI software allows doing an union operation with just one input. In this case if the circles and rectangle are in the same layer and an union operation is done in ArcGIS the expected result is the one on the right of the above image. Anyway this was not tested on QGIS (or SAGA or GRASS) because this case (1 input layer) is not allowed/implemented. In case the circles and the rectangle are in separate layers the expected results is not the one represented on the right part of the above image,

There are no differences in the results produced for the intersection and union operation in 2.18.12 and master, as in both cases they are wrong.

NOTE: curiously enough also SAGA results are wrong (same results as QGIS) leaving (for this scenario) only GRASS v.overlay as a viable option in QGIS.

**) intersection: it produces just 2 features instead of 4. The features created are the part of the circles that do overlap the rectangle, but then the overlapped parts of the two circles are not cut (so in the middle two more geometrically identical features should be created). As said the same issue affects SAGA (compare with GRASS and ArcGIS results).
Attribute table records for this two missing features should show the attributes of the rectangle and the attributes of each overlapping circle.

**) union: the problem is the same as for intersection. The overlapped areas of the circles should return two identical overlapped features in the output. Same as above regarding the attributes.

union_intersection_test_datasets.zip (103 KB) Giovanni Manghi, 2017-09-13 10:54 AM

dataset3.zip (682 KB) Giovanni Manghi, 2017-09-14 10:52 PM


Related issues

Duplicated by QGIS Application - Bug report #14846: QGIS Union: wrong results Closed 2016-05-20
Duplicated by QGIS Application - Bug report #17820: [Processing] Union algorithm does not pick attributes fro... Rejected 2018-01-08

History

#1 Updated by Giovanni Manghi over 6 years ago

Better leave the focus of this ticket for the problems regarding wrong results with clean geometries.

The errors/problems when running with unclean geometries can be addressed in #17132

#2 Updated by Giovanni Manghi over 6 years ago

Interesting enough:

the unpatched version of Processing in 2.18.12 (so without https://github.com/qgis/QGIS/pull/5099 ) produces the right result for the sample dataset1.
As said before, master does not, and this explains why https://github.com/qgis/QGIS/pull/5099 produces also wrong results.

Anyway also the unpatched processing in 2.18.12 produces the wrong result with another simple/sample datasets here attached now:
from a geometry point of view it seems correct (no overlapped features), and also part of the table of attributes seems correct. But you can notice that in the table of attributes there are number of records that are not associated to any geometry, and in fact they should not be there.

With this new dataset I have seem the patched version of Processing crash 2.18.12 and master produced the wrong result as it has overlapped geometries and no unioned attributes from input tables.

#3 Updated by Regis Haubourg over 6 years ago

For what it's worth, we are working on making Union more robust since we encountered many Null geometries in some outputs, both in QGIS and Postgis.

That ended up in modifying GEOS heuristics, with Sandro. That work partly solved some issues and requires upgrading packages to GEOS 3.7. We however still face issues, so we started a common plan with JTS and GEOS.

Waiting for that, I discovered some optimizations that we should include in QGIS Union algorithm.
I shared that here https://github.com/nyalldawson/QGIS/wiki/Processing-Algorithm-Bugs-and-Enhancements.

In two words, we must add snap2grid tolerance, dump all geometries to single parts and do a UnaryUnion on collected parts.

#4 Updated by Giovanni Manghi over 6 years ago

Regis Haubourg wrote:

...and Postgis.

wow, this is surprising.

That ended up in modifying GEOS heuristics, with Sandro. That work partly solved some issues and requires upgrading packages to GEOS 3.7. We however still face issues, so we started a common plan with JTS and GEOS.

Waiting for that, I discovered some optimizations that we should include in QGIS Union algorithm.
I shared that here https://github.com/nyalldawson/QGIS/wiki/Processing-Algorithm-Bugs-and-Enhancements.

In two words, we must add snap2grid tolerance, dump all geometries to single parts and do a UnaryUnion on collected parts.

many thanks for the infos Regis, and thanks for this effort!

#5 Updated by Jürgen Fischer over 6 years ago

#6 Updated by Alexander Bruy over 6 years ago

  • Duplicated by Bug report #17820: [Processing] Union algorithm does not pick attributes from the other layer added

#7 Updated by Martin Dobias almost 6 years ago

After merging https://github.com/qgis/QGIS/pull/6811, https://github.com/qgis/QGIS/pull/6859 and https://github.com/qgis/QGIS/pull/6961 this all should be resolved in QGIS 3.2 and algorithms are covered with tests for various situations.

Could you please test again and close if everything is resolved?

One note: Union(A,B) algorithm should now give results like SAGA does. This is slightly different from what GRASS and Arc produce because the overlapping features within one layer are not broken into multiple features. If there is need for such behavior, it is possible to use Union(C) algorithm where C=Union(A,B) to get results consistent with GRASS/Arc.

One more source of differences may be regarding handling of multi-part geometries: GRASS breaks multi-part polygons into multiple features with single-part polygons, SAGA has an option whether to split parts or not, don't know about Arc. QGIS algorithm does not split parts currently and it does not have an option to do so.

#8 Updated by Martin Dobias almost 6 years ago

  • Status changed from Open to Feedback

#9 Updated by Nyall Dawson almost 6 years ago

  • Resolution set to fixed/implemented
  • Status changed from Feedback to Closed

I think it's safe to close this, and new tickets can be opened with new test data if more issues are identified

Also available in: Atom PDF