Bug report #21189

Indexing data takes a very long time

Added by Alexey T almost 2 years ago. Updated almost 2 years ago.

Status:Closed
Priority:High
Assignee:-
Category:Digitising
Affected QGIS version:3.4.4 Regression?:Yes
Operating System:win7 Easy fix?:No
Pull Request or Patch supplied:No Resolution:fixed/implemented
Crashes QGIS or corrupts data:Yes Copied to github as #:29007

Description

I have a large project with vector DEM loaded.
When I try to edit any feature QGis starts indexing data.
First, it takes a veeery long time to index.
Second, it takes more than 8 Gigs of RAM.

Looks like QGis indexes the whole project. I believe it is sufficient to index only the canvas.

History

#1 Updated by Giovanni Manghi almost 2 years ago

  • Status changed from Open to Feedback

can you attach a sample project with data?

#2 Updated by Alexey T almost 2 years ago

It is 8 Gb of DEM data in shapefiles. Snapping should be enabled. When you try to edit it, QGIS indexes it and consumes 19Gb of RAM.

#3 Updated by Giovanni Manghi almost 2 years ago

  • Category changed from Editing to Digitising

Alexey T wrote:

It is 8 Gb of DEM data in shapefiles. Snapping should be enabled. When you try to edit it, QGIS indexes it and consumes 19Gb of RAM.

a DEM is usually a raster map, in this case I guess you refer to a line vector map where lines represent the same terrain height, correct?
can we download this dataset somewhere?

#4 Updated by Alexey T almost 2 years ago

Yes, vector lines.
Here is the dataset. 4 Gb zipped.
https://yadi.sk/d/aR605PsV-vrp2A

#5 Updated by Giovanni Manghi almost 2 years ago

Alexey T wrote:

Yes, vector lines.
Here is the dataset. 4 Gb zipped.
https://yadi.sk/d/aR605PsV-vrp2A

the dataset is split in several "small" shapes. I don't have any issues while editing any of them. Did you merged those shapes in one unique vector?

#6 Updated by Alexey T almost 2 years ago

No, I didn't

#7 Updated by Alexey T almost 2 years ago

Snapping shold be enabled.

#8 Updated by Alexey T almost 2 years ago

Take a look at this video
https://yadi.sk/i/-b6yBfba3W967A

#9 Updated by Giovanni Manghi almost 2 years ago

Alexey T wrote:

Snapping shold be enabled.

I'm trying with one of those shapes (a 250MB one) with snapping enabled and I don't particular issues to edit on 3.4.4 (but it seems there is a regression on master).

What are the exact steps to follow to replicate the problem?

#10 Updated by Giovanni Manghi almost 2 years ago

Alexey T wrote:

Take a look at this video
https://yadi.sk/i/-b6yBfba3W967A

sorry seen this clink only after my last comment.

#11 Updated by Giovanni Manghi almost 2 years ago

  • Priority changed from Normal to High
  • Crashes QGIS or corrupts data changed from No to Yes
  • Status changed from Feedback to Open

Alexey T wrote:

Take a look at this video
https://yadi.sk/i/-b6yBfba3W967A

ok I replicated, it indeed eats up all memory and eventually will crashes.

Did it worked better on 2.18?

#12 Updated by Giovanni Manghi almost 2 years ago

  • Status changed from Open to Feedback

#13 Updated by Giovanni Manghi almost 2 years ago

  • Status changed from Feedback to Open
  • Regression? changed from No to Yes

Did it worked better on 2.18?

I answer myself: yes. No such issue on 2.18.

#14 Updated by Loïc BARTOLETTI almost 2 years ago

Can you try to enable the option "Settings->Options->Digitizing->Enable snapping on invisible features" and restart QGIS ?

#15 Updated by Giovanni Manghi almost 2 years ago

Loïc BARTOLETTI wrote:

Can you try to enable the option "Settings->Options->Digitizing->Enable snapping on invisible features" and restart QGIS ?

it does not make any difference.

#16 Updated by Giovanni Manghi almost 2 years ago

  • Resolution set to fixed/implemented
  • Status changed from Open to Closed

There have been a update of the linspatialindex library in osgeo4w, and now this problem is gone.

#17 Updated by Alexey T almost 2 years ago

In what version will it be fixed?

#18 Updated by Giovanni Manghi almost 2 years ago

Alexey T wrote:

In what version will it be fixed?

it is already fixed if you use the osgeo4w installer, if not yo will need to wait for the next build of the standalone installer (not sure if it was generated one in the last hours).

#19 Updated by Jürgen Fischer almost 2 years ago

Giovanni Manghi wrote:

it is already fixed if you use the osgeo4w installer, if not yo will need to wait for the next build of the standalone installer (not sure if it was generated one in the last hours).

The standalones are regularly only updated after releases.

Also available in: Atom PDF