Feature request #3810

Join attributes by location crash

Added by bderstine - over 8 years ago. Updated over 2 years ago.

Status:Closed
Priority:Low
Assignee:cfarmer -
Category:Processing/QGIS
Pull Request or Patch supplied:No Resolution:invalid
Easy fix?: Copied to github as #:13868

Description

Target layer has 2.3 million points.
Join layer has a couple hundred polygons.

Process ran very slowly up to 27% completeness and then QGIS stopped responding and crashed.

History

#1 Updated by Giovanni Manghi over 8 years ago

please attach/link a sample dataset to try replicate the issue.

#2 Updated by cfarmer - over 8 years ago

A point of clarification:

Process ran very slowly up to 27% completeness and then QGIS stopped responding and crashed.

Does this mean that QGIS stopped responding and you closed/terminated the application, or did it indeed crash on it's own? If crash, is there any output that you can report?

Note: The Join by location tool is not particularly efficient (not optimized for large datasets), and for large datasets, you might be better off using postgis, or even a spatialite database, which are more appropriate for these types of operations. This is obviously not a 'solution' to this problem, but rather a viable workaround. I would also recommend using the 'Check geometry validity' tool to ensure that your polygons are all properly defined and not causing problems themselves.

Carson

#3 Updated by bderstine - over 8 years ago

lutra: I cannot attach the dataset as it is proprietary.

cfarmer: This means that QGIS stopped responding, the window turned slightly transparent, and I eventually shut it down. I may have been able to let it keep running, but it had already been an hour with no apparent signs of life (the progress bar ceased updating). No output to report, sadly.

I ran the same join (and a follow-up) in ArcGIS, just to test whether the files were ok, and it ran fine there (took a long time as well, but not as long and no crash).

#4 Updated by bderstine - over 8 years ago

lutra: I imagine a similar query would be to load the GNIS national file: http://geonames.usgs.gov/domestic/download_data.htm and join attributes by spatial location to some US Census boundary file such as the Census Incorporated Places.

#5 Updated by Giovanni Manghi over 8 years ago

Replying to [comment:5 bderstine]:

lutra: I imagine a similar query would be to load

Hi, can you please test if the datasets you say are enough to replicate the problem? I'm also not familiar with that datasets, so I guess it would be easier for you to try.

#6 Updated by bderstine - over 8 years ago

More detail: I tried again today with 1.8.0-10.
Attribute Summary set to "Take attributes of fist located feature"
Output table: Keep all records (including non-matching target records)

Process started at 8:35am, progress bar jumped to 15% immediately and then doesn't move, and I get the little "thinking" mouse pointer (likely due to all the processing that's happening behind the scenes, though this is not ideal UI design).

A spatial join in ArcGIS constantly displays a running count of # processed, in chunks, so you know that it's still actually doing something. Something like that would vastly improve the UI here.

At 9:14am, I get the pop-up window telling me that it "created output shapefile: ..." So I guess I was wrong. I guess I'll change this to "enhancement" and "major".

I feel strongly that either (a) the progress bar needs to be fixed so that it actually does update to show progress, or (b) some other indicator of progress should be added to inform the user that the process is still working.

#7 Updated by Giovanni Manghi over 8 years ago

I suggest then to close this ticket and open a more general one, like "improve handling of processes in ftools". My suggestion would be to manage the processes as in the "raster" tools menu (gdal tools). In this case launching a process does not "block" qgis as is run in the background, as a user can still work with the main program.

#8 Updated by Giovanni Manghi about 8 years ago

  • Status changed from Open to Closed
  • Resolution set to invalid
  • Pull Request or Patch supplied set to No

So closing this and opening a feature request one instead.

#9 Updated by Giovanni Manghi over 2 years ago

The "ftools" category is being removed from the tracker, changing the category of this ticket to "Processing/QGIS" to not leave the category orphaned.

Also available in: Atom PDF