Bug report #13963

QGIS greadily allocates memory and crashes when editing moderately large shapefiles with the node tool

Added by Karl Kastner over 8 years ago. Updated about 7 years ago.

Status:Closed
Priority:High
Assignee:-
Category:Digitising
Affected QGIS version:2.8.4 Regression?:No
Operating System:Linux / Ubuntu Easy fix?:No
Pull Request or Patch supplied:No Resolution:fixed/implemented
Crashes QGIS or corrupts data:Yes Copied to github as #:21977

Description

When editing moderately large shapefiles (228428 points, 1.1MB << 1% total RAM). QGIS starts to allocate huge amounts of memory and either crashes the system when no ulimit is set or is killed by the kernel. This happens reproducibly by adding the dataset to a new project, going into edit mode, selecting the node tool, selecting more than 1/3 of the points at once and attempting to delete them. This is a downsampled dataset which can be generated and edited with other software on the same machine without hickups. QGIS version is 1:2.8.1-0+20trusty6.

dmesg-2.12.1.txt Magnifier (247 KB) Karl Kastner, 2015-12-09 01:00 AM

dmesg-2.8.4.txt Magnifier (248 KB) Karl Kastner, 2015-12-10 03:10 AM

kapuas-polygon-merged-100.tgz (1.07 MB) Karl Kastner, 2015-12-10 05:44 AM

Shapefile_test.zip (8.79 KB) Hynek Suchy, 2015-12-28 05:30 AM

Associated revisions

Revision e503c705
Added by Jürgen Fischer about 8 years ago

change QgsAbstractGeometryV2::coordinateSequence() to return a
implicitly shared copy of an internal cache instead of recreating the
coordinate sequence again and again.

Improves performance of the nodetool on large features a lot (refs #13963)

Also introduce Qgs(Coordinate|Ring|Point)SequenceV2 typedefs.

Revision 102ba2d3
Added by Nyall Dawson almost 8 years ago

Much faster selection of large number of nodes using node tool

Refs #13963

History

#1 Updated by Nyall Dawson over 8 years ago

  • Status changed from Open to Feedback

Please test with 2.8.4 or 2.12.1

#2 Updated by Karl Kastner over 8 years ago

Release 2.12.1 segfaults immediately at start up. This has to do with python plugins, because it starts up if the package python-qgis is removed. Release 2.8.4 is not in the repositories, at least neither in http://qgis.org/ubuntugis nor in http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu/.

#3 Updated by Karl Kastner over 8 years ago

Release 2.12.1 segfaults immediately at start up. This has to do with python plugins, because it starts up if the package python-qgis is removed. Release 2.8.4 is not in the repositories, at least neither in http://qgis.org/ubuntugis nor in http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu/.

#4 Updated by Jürgen Fischer over 8 years ago

Karl Kastner wrote:

Release 2.8.4 is not in the repositories, at least neither in http://qgis.org/ubuntugis nor in http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu/.

Because 2.8.4 is the LTR, which is in debian-ltr or ubuntugis-ltr (see download page)

#5 Updated by Karl Kastner over 8 years ago

Thanks for the advice to add the LTR repository. The bug also affects 2.8.4. It does not occur on every deletion though, the steps had to be repeated a couple of times until Qgis crashed. Qgis becomes also relatively slow when activating the node tool, as it automatically starts a background validation. Note that the dataset has some crossing lines due to the downsampling, but this should not make Qgis crash.

dpkg -l qgis
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                     Version                   Architecture              Description
+++-========================================-=========================-=========================-=====================================================================================
ii  qgis                                     1:2.8.4+20trusty-ubuntugi amd64                     Geographic Information System (GIS)

#6 Updated by Jürgen Fischer over 8 years ago

The kernel output is mostly useless. Please attach the shape file.

#8 Updated by Jürgen Fischer over 8 years ago

The archive is missing the dbf.

#9 Updated by Karl Kastner over 8 years ago

There is no dbf, as there are no attributes. This is a Matlab export. Matlab does not create a dbf if there are no attributes.

#10 Updated by Giovanni Manghi over 8 years ago

  • Priority changed from Low to High
  • Category changed from Actions to Digitising
  • Status changed from Feedback to Open

Karl Kastner wrote:

There is no dbf, as there are no attributes. This is a Matlab export. Matlab does not create a dbf if there are no attributes.

weird, but anyway I confirm the issue, at least on 2.8.4 on ubuntu 14.04.

#11 Updated by Giovanni Manghi over 8 years ago

  • Affected QGIS version changed from 2.8.1 to 2.8.4

#12 Updated by Hynek Suchy over 8 years ago

Hello!
I would like to confirm this bug.
I am on Windows Vista 32bits Ultimate, QGIS 2.12.1 Lyon, standalone installer.
I am working on a small shapefile with some 300 points.
While editing the shapefile, mainly deleting points or moving them, qgis starts to
slow down and in the end freezes.
I tried to open the shp file outside of current project and I easily recreated the bug.
1. select one point
2. delete it
repeat 11 times and qgis freezes (not responding), but after some time
refreshes and I can delete another point, after 14 repetitions qgis doesn't recover
and I have to kill it from Windows Task Manager.
Enclosing shapefile.

#13 Updated by Giovanni Manghi over 8 years ago

Jurgen, Nyall, shouldn't this be tagged as blocker?

#14 Updated by Gerhard Spieles over 8 years ago

Hi Hynek,
maybe a problem of installed plugins.
See #13869

#15 Updated by Giovanni Manghi over 8 years ago

Gerhard Spieles wrote:

Hi Hynek,
maybe a problem of installed plugins.
See #13869

usually in my testing boxes I have no 3rd party plugins, but I can test again.

#16 Updated by Hynek Suchy over 8 years ago

Sorry for making a false alarm!
It was CadTools, I deinstalled them (and Digitizing Tools to be sure) and now it works smoothly!
Thank you!

#17 Updated by Karl Kastner over 8 years ago

On my machine the bug can be reproduced when all plugins are disabled or by temporally moving the ~/.qgis2 folder before starting qgis.
The bug is also be reproduced with data sets having a valid geometry. (The polygon in the original sample shape file was self intersecting.)
The behaviour depends on the geometry for the same data points:
Polygon: QGIS becomes a memory bomb and crashes.
Line: QGIS takes very long (minutes) to complete the tasks, but does not fill the memory.
Point: QGIS deletes all selected points flawlessly within less than a second.

#18 Updated by Giovanni Manghi over 8 years ago

Karl Kastner wrote:

On my machine the bug can be reproduced when all plugins are disabled or by temporally moving the ~/.qgis2 folder before starting qgis.

yes, this is my observation too.

#19 Updated by Jürgen Fischer about 8 years ago

Improvements in PR #2817

#20 Updated by Giovanni Manghi about 8 years ago

  • Status changed from Open to Feedback

#21 Updated by Jürgen Fischer about 8 years ago

50f01a2 also reduces the memory consumption of the geometry classes, which should also help with this issue.

#22 Updated by Giovanni Manghi almost 8 years ago

  • Status changed from Feedback to Closed
  • Resolution set to fixed/implemented

seems fixed to me at least on 2.14.3, please reopen if necessary

#23 Updated by baditaflorin - almost 8 years ago

  • Status changed from Closed to Reopened

Tested in 2.14.3 on the Canada provinces and the results are horrible.

If i delete less then 10-20 nodes at a time it takes around 1-2 seconds.

If i select around 100-1000 nodes, it can take up to 10 minutes.

You can download the file from here https://www.dropbox.com/s/j5i1m1tt2cam1qy/canada_states.zip?dl=0

Here it`s a youtube video showing the selection https://youtu.be/tV7VOK4YAZE

#24 Updated by Giovanni Manghi about 7 years ago

  • Status changed from Reopened to Closed

baditaflorin - wrote:

Tested in 2.14.3 on the Canada provinces and the results are horrible.

If i delete less then 10-20 nodes at a time it takes around 1-2 seconds.

If i select around 100-1000 nodes, it can take up to 10 minutes.

You can download the file from here https://www.dropbox.com/s/j5i1m1tt2cam1qy/canada_states.zip?dl=0

Here it`s a youtube video showing the selection https://youtu.be/tV7VOK4YAZE

the original report/issue is solved, there is no more leak if you the node tool.
So in this cases please file a different ticket.
Anyway I tested you dataset on both 2.18.4 and master (qgis3) on both Windows and Linux and is not that slow removing nodes when you selected a lot of them. Is not lighting fast, but it takes a few seconds when you have selected a reasonable large amount. Please try with the latest releases and in case file a new ticket.

Also available in: Atom PDF