Bug report #21694

Hang/Crash when attempting to drag&drop multiple layers into gpkg

Added by Brett Carlock 9 months ago. Updated 9 months ago.

Status:Open
Priority:High
Assignee:-
Category:Browser
Affected QGIS version:3.6.1 Regression?:No
Operating System:1809 17763.402 [x64 Pro] Easy fix?:No
Pull Request or Patch supplied:No Resolution:
Crashes QGIS or corrupts data:Yes Copied to github as #:29510

Description

The time it takes to import a number of layers into a geopackage seems to increase exponentially. For instance, there is no noticeable delay from time of drop to beginning of import at less than 10 layers. Somewhere between 10-20 layers there is a pause of 10s or more. Any more than 20 layers the pause becomes longer. Trying to drag/drop 105 layers results in an unresponsive QGIS GUI with 1 CPU thread pegged, no disk activity, and static (not growing) RAM usage for the QGIS process.

There should be the means to batch import data into a geopackage, especially if one is trying to migrate away from an ESRI environment where large file geodatabases are commonplace. Having to batch things a handful of layers at a time is tedious at best.


Related issues

Related to QGIS Application - Feature request #21695: Improve behavior of Geopackage Data Import Open 2019-03-28

History

#1 Updated by Giovanni Manghi 9 months ago

  • Priority changed from Normal to High

There should be the means to batch import data into a geopackage, especially if one is trying to migrate away from an ESRI environment where large file geodatabases are commonplace. Having to batch things a handful of layers at a time is tedious at best.

there will be soon an option, I hope (let's see if I'm able to implement it): https://issues.qgis.org/issues/21679

#2 Updated by Brett Carlock 9 months ago

Giovanni Manghi wrote:

There should be the means to batch import data into a geopackage, especially if one is trying to migrate away from an ESRI environment where large file geodatabases are commonplace. Having to batch things a handful of layers at a time is tedious at best.

there will be soon an option, I hope (let's see if I'm able to implement it): https://issues.qgis.org/issues/21679

That sounds amazing!

I'm not sure the behavior above is directly related though. It should be able to handle the import one by one as it does for # of layers less than about 20, but it is just stuck.

Currently 13min of CPU time at 100% for that core (8% overall CPU usage) and no change in RAM or disk activity. QGIS instance still Not responding...

#3 Updated by Giovanni Manghi 9 months ago

That sounds amazing!

I'm not sure the behavior above is directly related though. It should be able to handle the import one by one as it does for # of layers less than about 20, but it is just stuck.

Currently 13min of CPU time at 100% for that core (8% overall CPU usage) and no change in RAM or disk activity. QGIS instance still Not responding...

it will have the same performances as the GDAL/OGR command line tool, so you can already test it.

#4 Updated by Nyall Dawson 9 months ago

  • Status changed from Open to Feedback

There's already the "package layers" algorithm in the processing toolbox, which allows for many layers to be added to a geopackage. Does that work OK for you?

#5 Updated by Jürgen Fischer 9 months ago

#6 Updated by Brett Carlock 9 months ago

Giovanni Manghi wrote:

That sounds amazing!

I'm not sure the behavior above is directly related though. It should be able to handle the import one by one as it does for # of layers less than about 20, but it is just stuck.

Currently 13min of CPU time at 100% for that core (8% overall CPU usage) and no change in RAM or disk activity. QGIS instance still Not responding...

it will have the same performances as the GDAL/OGR command line tool, so you can already test it.

4hr25m of CPU time on this import and it still is hung. To clarify, this is to import 105 different layers totaling 289mb as shapefiles. There is definitely a bug/limitation in the import to geopackage function. There is no way this should take even a fraction of this speed.
i7-7800x
1TB SSD
16GB DDR4

#7 Updated by Giovanni Manghi 9 months ago

4hr25m of CPU time on this import and it still is hung

this is with the "package layers" tool?

#8 Updated by Brett Carlock 9 months ago

Giovanni Manghi wrote:

4hr25m of CPU time on this import and it still is hung

this is with the "package layers" tool?

Drag/drop from the Browser panel. Same method I opened the ticket about.

#9 Updated by Brett Carlock 9 months ago

Nyall Dawson wrote:

There's already the "package layers" algorithm in the processing toolbox, which allows for many layers to be added to a geopackage. Does that work OK for you?

Thanks for pointing that out. It isn't immediately obvious (I had never come across it, for instance) and can't be called from the Browser panel, but yes, 50s of CPU time to import all 105 layers into the geopackage. This works as intended, though please note that drag/drop from the Browser panel does not.

#10 Updated by Giovanni Manghi 9 months ago

  • Status changed from Feedback to Open

Also available in: Atom PDF