Bug report #11631

field calculator update is very slow

Added by Gavin Fleming almost 6 years ago. Updated over 5 years ago.

Status:Closed
Priority:Normal
Assignee:-
Category:Field calculator
Affected QGIS version:2.6.0 Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:No Resolution:worksforme
Crashes QGIS or corrupts data:No Copied to github as #:19884

Description

I select a small number of records in a layer with a few hundred thousand records and use the field calculator to update them. What used to be (or should be) an almost instant operation takes many seconds. It is longer than I would take to update them manually in the table.

History

#1 Updated by Giovanni Manghi almost 6 years ago

  • Category set to Field calculator
  • Status changed from Open to Feedback

do you see any difference if you use the FC with the table of attributes open or closed?

#2 Updated by Gavin Fleming almost 6 years ago

540000 road LineString records, 8 core i7 with SSD, 8GB RAM

no difference updating 17 records whether table is open or not, both 30s.

Opening the table just adds another looong wait (how can this be minimised?).

#3 Updated by Giovanni Manghi almost 6 years ago

Gavin Fleming wrote:

540000 road LineString records, 8 core i7 with SSD, 8GB RAM

no difference updating 17 records whether table is open or not, both 30s.

Opening the table just adds another looong wait (how can this be minimised?).

postgis?

#4 Updated by Gavin Fleming almost 6 years ago

local PostgreSQL 9.3, PostGIS 2.1

#5 Updated by Giovanni Manghi almost 6 years ago

Gavin Fleming wrote:

540000 road LineString records, 8 core i7 with SSD, 8GB RAM

no difference updating 17 records whether table is open or not, both 30s.

Opening the table just adds another looong wait (how can this be minimised?).

I'm really not lucky as it doesn't seem I'm able to replicate any of the issues you file :)

In this case:

  • I downloaded the OSM UK roads shape, 3 million features and imported it into a local postgis installation (9.3, Linux, 4 cores, 8gb ram and SSD).
  • selected a few tens of records/features (20/30) and updated an attribute.
  • repeated the above after selecting ~200k features

results:

in both cases the update operation, after clicking the "ok" button in the field calculator, takes a few seconds (eventually what can be surprising is the fact that the time it takes does not seems to differ much between a few records selected and a lot of records selected).

In the past we had issues when updating (or creating) fields with the FC and at the same time having the table of attributes open. I remember also another similar issue that was related to the number of features that were visible the moment the FC was used. In both cases fixes/patches has been applied and the situation is much better now.

Opening a 3 million table of attributes table here take ~25 seconds from shapefile and ~40 from postgis.

Unchecking the "select at id" option when loading the postgis layer make a little faster the operation, ~30 seconds, but in this case there may be a bug as the operation is done 2 times (to notice is necessary to open a postgis table of at least 80/100k features). Jurgen, is this right?

To make faster opening large table is usually used the "show features visible on map" option, but this is known to be bugged for postgis layers.

#6 Updated by Giovanni Manghi over 5 years ago

  • Resolution set to worksforme
  • Status changed from Feedback to Closed

closing for lack of feedback.

Also available in: Atom PDF