Bug report #13167

ftools dissolve, intersection, clip broken

Added by Lene Fischer over 4 years ago. Updated almost 4 years ago.

Status:Closed
Priority:Severe/Regression
Assignee:-
Category:Data Provider/OGR
Affected QGIS version:2.10.0 Regression?:No
Operating System:Windows Easy fix?:No
Pull Request or Patch supplied:No Resolution:fixed/implemented
Crashes QGIS or corrupts data:No Copied to github as #:21230

Description

Hi,
I have installed QGIS 2.10.1 on Windows 7 (64 bit).

When I use the dialog from Vector>Geoprocessing tool>Dissolve the filename in the dialog disappear while the function are running. – Minor problem

But two other issues:
• The function runs extremely slowly
• But worst of all function Dissolve and add Unique ID field – All polygons that are unique are stored and all the rest deleted.

The problem is both in the vector menu and in the Processing toolbox.
Attached file with polygons used.

Regards
Lene Fischer

polygons.zip (1.37 MB) Lene Fischer, 2015-07-30 01:32 PM

Dissolve_process-End.png (9.94 KB) Michele Mor, 2015-10-01 09:18 AM


Related issues

Related to QGIS Application - Bug report #13467: Dissolve Vectors not working in QGIS 2.10 Closed 2015-09-30

History

#1 Updated by Lene Fischer over 4 years ago

Using other functions from Geoprocession tool: Intersection and Clip - The new layer come out empty.

#2 Updated by Lene Fischer over 4 years ago

Just tried the same functions in 2.8.2 - where they work fine.

#3 Updated by Giovanni Manghi over 4 years ago

  • Subject changed from Dissolve delete polygons to ftools dissolve, intersection, clip broken
  • Priority changed from High to Severe/Regression
  • Target version changed from Version 2.10 to Future Release - High Priority

#4 Updated by Giovanni Manghi over 4 years ago

  • Status changed from Open to Feedback

Hi Lene,

could you provide sample data? I cannot replicate here (Win 7 64 bit, qgis 2.10.1/osgeo4w).

About the slowness:

the QGIS geoprocessing tools are unfortunately historically slow and often unreliable (without forgetting the often they break from one release to another). Honestly it seems at least strange that among all the (new) cool features regularly added (or fixed) no one has ever felt the necessity to have them ported rewritten (maybe in C++) to make them robust, faster and returning right results.

I soon plan to add in Processing new geoprocessing tools (a few are already available, clip, dissolve) based on ogr2ogr and SQL, so at least reliable alternative will be avalable without needing to use GRASS or SAGA tools.

#5 Updated by Lene Fischer over 4 years ago

Hi Giovanni,
Exampledata added.

Bo Thomsen tried the last build 2.10.1 and 2.11 OSGEO 64 bit (Win 10) - with same result.

Regards
Lene

#6 Updated by Giovanni Manghi over 4 years ago

Hi Lene,

what is the field to be used for the dissolve? there are many in the table of attributes. Thanks.

Lene Fischer wrote:

Hi Giovanni,
Exampledata added.

Bo Thomsen tried the last build 2.10.1 and 2.11 OSGEO 64 bit (Win 10) - with same result.

Regards
Lene

#7 Updated by Lene Fischer over 4 years ago

Try "Regionnavn"

#8 Updated by Giovanni Manghi over 4 years ago

  • Category set to 44
  • Status changed from Feedback to Open

confirmed.

#9 Updated by Bernhard Ströbl about 4 years ago

  • Category changed from 44 to Data Provider/OGR
I checked with Dissolve in Processing in current master on Ubuntu;
result: in OGR-log there is an entry Feature geometry not imported (OGR error: ) Thus the geometry of the new feature cannot be set, resulting in an empty layer.
Investigating further I discovered your layer is wkb-Type -2147483645 (wkbPolygon25D) http://gdal.org/java/constant-values.html
Obviously QGIS (or OGR) cannot handle this type. If I create a new polygon shape file and copy your polygons over into the new layer, it is wkbType 3 (Polygon) and dissolves without any problems.
So the problem is not the algorithm but your data. However QGIS/Processing does not give any feedback stating the data is not suited.
Things to be done:
  1. improve the OGR-error log stating the error
  2. either insert a geometry check so Processing only processes proper wkbTypes and gives feedback otherwise (Victor?)
  3. or improve QGIS' geometry handling, if not already in the pipeline (Marco?)

#10 Updated by Michele Mor about 4 years ago

Hi.
I have reported a similar issue using dissolve and if I process my shape file using 2.8.3 everything works, if I process the same file using 2.10.1 it does not work.
Unfortunately I cannot upload the shape file.

#11 Updated by Bernhard Ströbl about 4 years ago

Hi,
can you load the layer, make it active and enter in the Python console
iface.activeLayer().wkbType()
What is the result?

#12 Updated by Michele Mor about 4 years ago

Hi Bernhard,

can you load the layer, make it active and enter in the Python console
iface.activeLayer().wkbType()

if I have done it correctly (never used the python console before), my result is 3.

#13 Updated by Bernhard Ströbl about 4 years ago

wkbType 3 is a straight polygon layer, different from Lene's and should work (according to what we know until now). If you cannot share the shape file publicly you might want to provide it to me for testing.
Could you describe what your result is or (better) if you already opened a ticket link to it?

#14 Updated by Michele Mor about 4 years ago

Hi Bernhard.
I have tried to reproduce the same error using a sample file but it works!
The file that I use is the Codepoint with Polygons, but they are copyrighted therefore I cannot send you a copy (see: https://www.ordnancesurvey.co.uk/business-and-government/products/code-point-with-polygons.html).

The whole file consist of 1.5 millions features and the shape file is almost 1.4GB.

If it's possible I could run the process again with some LOG or DEBUG options?

#15 Updated by Bernhard Ströbl about 4 years ago

This is a really large file, does it work with subsets of polygons (like 100000)
Are you using Processing 2.10.2? Maybe your computer runs out of memory, do you get an error message?

#16 Updated by Michele Mor about 4 years ago

Hi.
I am using 2.10.1.
I have checked yesterday and that was the version available on the website.

The same file, same computer does not run out of memory with 2.8.3, therefore I assume that the new version is at fault (but I could be wrong!).
I don't have any error message, the new shape file is smaller than it should be and if I open it nothing happens.
I can take a screenshot and upload screenshots.
I will try to find the time for testing 100,000 polygons.

#17 Updated by Bernhard Ströbl about 4 years ago

I meant the version of the Processing plugin, not QGIS, AFAIK you can update Processing independently of QGIS, although it is shipped with QGIS in the first place
All I konow is that when I install Processing 2.10.2 my recent changes to Dissolve are included. The former Dissolve took ages to run (when you used a dissolve field).

#18 Updated by Michele Mor about 4 years ago

Hi.
Sorry for the misunderstanding, but I was not aware of the Processing Plugin.
The version installed was 2.10.99, so I downgraded to 2.10.2.
I have tried again and had the same issue.

I have attached a screenshot.
As you can see, first of all as soon as I click "OK" the path to destination file becomes blank.
Then, at the end of the process, even if the option to "add result to canvas" is selected, no layer is added.
If I look in the folder, my new shape file exists but if compared to successfull process is smaller 27M vs 45M.
I have tried to open the DBF file using another software and I had an error.

I will try to test using 100000 polygons tomorrow.

#19 Updated by Bernhard Ströbl about 4 years ago

Please try with 100000 (or a selection) and report back.
I will be abroad for two weeks beginning this afternoon, so do not wonder when I do not answer.

#20 Updated by Michele Mor about 4 years ago

Hi.
I have tried a sample using 194997 polygons and they were successfully dissolved into 1342 polygons.
I will try with a bigger sample and let you know when it fails.

#21 Updated by Bernhard Ströbl about 4 years ago

ok, could be a memory problem. Are you using a dissolve field? If yes: is it faster than in QGIS 2.8?

#22 Updated by Michele Mor about 4 years ago

Hi.
I am using a dissolve field. Don't know if it is faster than QGIS 2.8, never timed them because it takes around 20/30 mins so I tend to start the process and check once in a while.
Is there a log that I can check and see how long it took?

#23 Updated by Bernhard Ströbl about 4 years ago

It should be faster. I tested with ~16000 polygons and the new algorithm takes a blink whereas it used to be minutes before. In processing history it is listed, when a process started. I do not know if there is more logging.

#24 Updated by Michele Mor about 4 years ago

Hi, I did a couple of tests: with 221739 polygons success, with 798145 polygons failure.
I can check the speed of the same process in QGIS 2.8 and QGIS 2.10, but frankly I'm not concerned about it.
Unless I can give you more details or help - I could install a test version of QGIS if needed - I'm not spending more time on it.
To me it's an issue cause by the number of polygons and hopefully you can replicate it easily.

#25 Updated by Bernhard Ströbl about 4 years ago

Currently my recent changes are reverted, so in the next version of QGIS Dissolve should behave as in 2.8

#26 Updated by Michele Mor about 4 years ago

Hi.
I have just downloaded and installed 2.12.0 Lyon.
The same layer that I can dissove with 2.8.3 fails in 2.12.0.
In the python panel I had the following error:
2015-10-27T16:06:05 1 Traceback (most recent call last):
File "C:\\PROGRA~1\\QGISLY~1\\apps\\qgis\\python\\plugins\\fTools\\tools\\doGeoprocessing.py", line 309, in run
geos, feature, match, error = self.dissolve(useField)
File "C:\\PROGRA~1\\QGISLY~1\\apps\\qgis\\python\\plugins\\fTools\\tools\\doGeoprocessing.py", line 749, in dissolve
feature.setGeometry(outFeats[k])
TypeError: QgsFeature.setGeometry(QgsGeometry): argument 1 has unexpected type 'NoneType'

The other part of the panel is:
2015-10-27T16:06:05 1 An error has occured while executing Python code:

TypeError: QgsFeature.setGeometry(QgsGeometry): argument 1 has unexpected type 'NoneType'
Traceback (most recent call last):
File "C:\\PROGRA~1\\QGISLY~1\\apps\\qgis\\python\\plugins\\fTools\\tools\\doGeoprocessing.py", line 309, in run
geos, feature, match, error = self.dissolve(useField)
File "C:\\PROGRA~1\\QGISLY~1\\apps\\qgis\\python\\plugins\\fTools\\tools\\doGeoprocessing.py", line 749, in dissolve
feature.setGeometry(outFeats[k])
TypeError: QgsFeature.setGeometry(QgsGeometry): argument 1 has unexpected type 'NoneType'
Python version: 2.7.5 (default, May 15 2013, 22:44:16) [MSC v.1500 64 bit (AMD64)]
QGIS version: 2.12.0-Lyon Lyon, cd9d645
Python Path:
C:/Users/administrator/.qgis2/python/plugins\\processingC:/PROGRA~1/QGISLY~1/apps/qgis/./pythonC:/Users/administrator/.qgis2/pythonC:/Users/administrator/.qgis2/python/pluginsC:/PROGRA~1/QGISLY~1/apps/qgis/./python/pluginsC:\\PROGRA~1\\QGISLY~1\\apps\\Python27\\lib\\site-packages\\matplotlib-1.3.1-py2.7-win-amd64.eggC:\\PROGRA~1\\QGISLY~1\\apps\\Python27\\lib\\site-packages\
ose-1.3.3-py2.7.eggC:\\PROGRA~1\\QGISLY~1\\apps\\Python27\\lib\\site-packages\\tornado-4.0.1-py2.7-win-amd64.eggC:\\PROGRA~1\\QGISLY~1\\apps\\Python27\\lib\\site-packages\\backports.ssl_match_hostname-3.4.0.2-py2.7.eggC:\\PROGRA~1\\QGISLY~1\\apps\\Python27\\lib\\site-packages\\certifi-14.05.14-py2.7.eggC:\\PROGRA~1\\QGISLY~1\\bin\\python27.zipC:\\PROGRA~1\\QGISLY~1\\apps\\Python27\\DLLsC:\\PROGRA~1\\QGISLY~1\\apps\\Python27\\libC:\\PROGRA~1\\QGISLY~1\\apps\\Python27\\lib\\plat-winC:\\PROGRA~1\\QGISLY~1\\apps\\Python27\\lib\\lib-tkC:\\PROGRA~1\\QGISLY~1\\binC:\\PROGRA~1\\QGISLY~1\\apps\\Python27C:\\PROGRA~1\\QGISLY~1\\apps\\Python27\\lib\\site-packagesC:\\PROGRA~1\\QGISLY~1\\apps\\Python27\\lib\\site-packages\\PILC:\\PROGRA~1\\QGISLY~1\\apps\\Python27\\lib\\site-packages\\jinja2-2.7.2-py2.7.eggC:\\PROGRA~1\\QGISLY~1\\apps\\Python27\\lib\\site-packages\\markupsafe-0.23-py2.7-win-amd64.eggC:\\PROGRA~1\\QGISLY~1\\apps\\Python27\\lib\\site-packages\\pytz-2012j-py2.7.eggC:\\PROGRA~1\\QGISLY~1\\apps\\Python27\\lib\\site-packages\\win32C:\\PROGRA~1\\QGISLY~1\\apps\\Python27\\lib\\site-packages\\win32\\libC:\\PROGRA~1\\QGISLY~1\\apps\\Python27\\lib\\site-packages\\PythonwinC:\\PROGRA~1\\QGISLY~1\\apps\\Python27\\lib\\site-packages\\Shapely-1.2.18-py2.7-win-amd64.eggC:\\PROGRA~1\\QGISLY~1\\apps\\Python27\\lib\\site-packages\\six-1.3.0-py2.7.eggC:\\PROGRA~1\\QGISLY~1\\apps\\Python27\\lib\\site-packages\\wx-2.8-msw-unicodeC:\\PROGRA~1\\QGISLY~1\\apps\\Python27\\lib\\site-packages\\xlrd-0.9.2-py2.7.eggC:\\PROGRA~1\\QGISLY~1\\apps\\Python27\\lib\\site-packages\\xlwt-0.7.5-py2.7.eggC:/Users/administrator/.qgis2//pythonC:\\PROGRA~1\\QGISLY~1\\apps\\qgis\\python\\plugins\\fTools\\tools

Obviously I don't know if that caused the process to fails or is a red herring and the process failed because of memory issues.

#27 Updated by Alexander Bruy almost 4 years ago

  • Status changed from Open to Closed
  • Resolution set to fixed/implemented

Vector geoprocessing tools now show error message if input layer has incompatible geometry type. As this was the problem of the original bugreport I close this. Please open new tickets for other issues if necessary.

Also available in: Atom PDF