Bug report #20008

Very Slow Table Operations while the Table is Open

Added by Kelley Jabr almost 2 years ago. Updated almost 2 years ago.

Status:Closed
Priority:Normal
Assignee:-
Category:Attribute table
Affected QGIS version:3.2.3 Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:No Resolution:fixed/implemented
Crashes QGIS or corrupts data:No Copied to github as #:27830

Description

I'm working with a geopackage on ~530k features, and just about everything I try to do in the table takes way longer than expected.
  • Adding a field took 5 minutes (table now has 22 total fields, so not massive)
  • Field Calculator on ~32k selected records took 8 minutes (literally all I was putting in the cells was: N)
  • Field calculator on ~480k records took so many hours that I had to leave it running over the weekend and when I came back, the Field Calculator had "closed unexpectedly"

(I'm running on a linux machine with an Intel Core i7 and 32GB of memory.)

I posed this question to the users list and the final solution was to close the attribute table and perform these operations. The improvement in speed was significant (what took hours before took just minutes when I had the attribute table closed). It'd be great to be able to do this with the table open, however!


Related issues

Related to QGIS Application - Bug report #19771: QGIS slow commit on large tables Closed 2018-09-05

History

#1 Updated by Luigi Pirelli almost 2 years ago

confirmed on windows too with master three days ago... not checked current master

#2 Updated by Giovanni Manghi almost 2 years ago

  • Operating System deleted (Linux)

This is an issue that affects QGIS since LONG ago. In fact I'm pretty sure that there should be already a ticket about it. For anyone working with large datasets this is almost a show stopper.

#3 Updated by Luigi Pirelli almost 2 years ago

not sure if it is duplicate of #19771

#4 Updated by Giovanni Manghi almost 2 years ago

#6 Updated by Antonio Viscomi almost 2 years ago

This slowness is also present in QGIS 2.18
and also occurs with fewer features in shapefiles (15000-16000 are enough),
slowness increases as the number of unsaved editing operations increases.

With the "attribute table" opened, even "merge elements" or "split feature" operations,
cause some problems (i.e. the counting of the records and the selection of the same does not seem synchronized with these editing operations)
and sometimes this cause freeze of QGIS

#7 Updated by Giovanni Manghi almost 2 years ago

  • Status changed from Open to Feedback

Antonio Viscomi wrote:

This slowness is also present in QGIS 2.18
and also occurs with fewer features in shapefiles (15000-16000 are enough),
slowness increases as the number of unsaved editing operations increases.

With the "attribute table" opened, even "merge elements" or "split feature" operations,
cause some problems (i.e. the counting of the records and the selection of the same does not seem synchronized with these editing operations)
and sometimes this cause freeze of QGIS

please try qgis master (it must contain https://github.com/qgis/QGIS/pull/8177) hopefully that patch helps.

#8 Updated by Antonio Viscomi almost 2 years ago

Giovanni Manghi wrote:

Antonio Viscomi wrote:

This slowness is also present in QGIS 2.18
and also occurs with fewer features in shapefiles (15000-16000 are enough),
slowness increases as the number of unsaved editing operations increases.

With the "attribute table" opened, even "merge elements" or "split feature" operations,
cause some problems (i.e. the counting of the records and the selection of the same does not seem synchronized with these editing operations)
and sometimes this cause freeze of QGIS

please try qgis master (it must contain https://github.com/qgis/QGIS/pull/8177) hopefully that patch helps.

I've just tested in Master 9cae77f262,
It seems to works fine

many thanks for the explanation

Regards

#9 Updated by Giovanni Manghi almost 2 years ago

  • Resolution set to fixed/implemented
  • Status changed from Feedback to Closed

Also available in: Atom PDF