Bug report #8434

join by spatial location locks up QGIS

Added by Jonathan Moules over 11 years ago. Updated over 7 years ago.

Status:Closed
Priority:High
Assignee:-
Category:Processing/QGIS
Affected QGIS version:master Regression?:
Operating System:Windows Easy fix?:
Pull Request or Patch supplied:No Resolution:fixed/implemented
Crashes QGIS or corrupts data:Yes Copied to github as #:17203

Description

See attached for data and screenshot of settings used for "join attributes by location" tool.

It gets to 15% and then no progress. I waited about 30 minutes but still nothing. ArcGIS managed it in about 2 minutes, so I figured it has crashed. Once it gets to 15% it starts using 100% of one of my CPU cores and reading from my HDD at 300MB/s (quite impressive given the datasets are so small).
Or maybe I just need to be more patient, but 30 mins on a high end workstation for a simple join indicates crash to me.

a13cdf8 - windows x64

Compass_logs_-_Copy.7z (2.3 MB) Jonathan Moules, 2013-08-08 07:47 AM

Desktop.zip (1.67 MB) Jonathan Moules, 2013-08-09 04:32 AM

Clipboard07.png (60.6 KB) Jonathan Moules, 2013-08-09 04:54 AM

Associated revisions

Revision c465c9f0
Added by Martin Dobias almost 11 years ago

Fix #8434 (join by spatial location locks up QGIS)

By caching the features of the provider in the inner loop, there is
a considerable speedup (compared to doing millions of queries)

History

#1 Updated by Giovanni Manghi over 11 years ago

  • Status changed from Open to Feedback

in the attached dataset there is only 1 layer. There is also no screenshot with settings used.

#2 Updated by Giovanni Manghi over 11 years ago

It is supposed to try the join with the shapefile added here? #8433

If yes then I tested on qgis master/osgeo4w and there is no freeze whatsoever. I tested both ways (target and join layer) and both options (keep all the records, only keep matching records). When the target layer is the SL layer then the operation takes a few minutes in a Windows VM, so very low end "hardware". The other way is pretty fast, just a few seconds.

#3 Updated by Giovanni Manghi over 11 years ago

  • Category set to 44

#4 Updated by Giovanni Manghi over 11 years ago

  • Crashes QGIS or corrupts data changed from Yes to No
  • Priority changed from Normal to Low

You are right, using the SL vector "processed" as target layer and "hex_1k" as join layer with the option "keep all the records" is very slow (but it does not freeze or crash, is "just" dead slow).

It happens that the "processed" vector as ~14k invalid geometries (check with the "check geometry validity" tool).

If you remove such records/geometries (they have areas = 0) and you run again the join, it works as expected.

#5 Updated by Jonathan Moules over 11 years ago

Apologies I attached the wrong file. :-(

Giovanni Manghi wrote:

You are right, using the SL vector "processed" as target layer and "hex_1k" as join layer with the option "keep all the records" is very slow (but it does not freeze or crash, is "just" dead slow).

It happens that the "processed" vector as ~14k invalid geometries (check with the "check geometry validity" tool).

Are you saying the SpatialLite has invalid geometries in? Because I've used spatailite_gui's "check validity" (which runs ST_IsValid()) and it comes back just fine. There are ~14,000 null geometries, but these don't appear to qualify as a "issue".

This issue was with two shapefiles. See the attached. It locks up on 15%. How long does it take you to run it?

#6 Updated by Jonathan Moules over 11 years ago

See attached screenshot, this is indeed a crash. After several minutes of waiting, a little black box appears in the bottom right of the screen (you can see it as white here because I've clicked it).
Clicking the little box brings up that crashed dialog.

#7 Updated by Giovanni Manghi over 11 years ago

  • Status changed from Feedback to Open
  • Crashes QGIS or corrupts data changed from No to Yes
  • Operating System set to Windows
  • Priority changed from Low to High

Jonathan Moules wrote:

See attached screenshot, this is indeed a crash. After several minutes of waiting, a little black box appears in the bottom right of the screen (you can see it as white here because I've clicked it).
Clicking the little box brings up that crashed dialog.

With the provided shapefile and settings it freezes, but only on Windows as on Linux runs without issues.

#8 Updated by Giovanni Manghi over 11 years ago

Jonathan Moules wrote:

There are ~14,000 null geometries, but these don't appear to qualify as a "issue".

they are invalid for the

vector -> geometry -> check geometry

tool, and fact is that if you remove them the operation runs ok. Don't know if such geometries are allowed or not, anyway seems something that can be improved.

#9 Updated by Martin Dobias almost 11 years ago

  • Status changed from Open to Closed

#10 Updated by Martin Dobias almost 11 years ago

  • Target version set to Version 2.2
  • Resolution set to fixed/implemented

#11 Updated by Giovanni Manghi over 7 years ago

The "ftools" category is being removed from the tracker, changing the category of this ticket to "Processing/QGIS" to not leave the category orphaned.

Also available in: Atom PDF