Bug report #9315

Field Calculator, slow performance with 2.1.0-104

Added by joe larson almost 11 years ago. Updated over 7 years ago.

Status:Closed
Priority:Low
Assignee:Matthias Kuhn
Category:Vectors
Affected QGIS version:master Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:No Resolution:
Crashes QGIS or corrupts data:No Copied to github as #:17922

Description

using OSGeo4W64, 2.1.0-104 while trying to update a field in a large dataset (~300MB) with the Field Calculator ...very slow performance was experienced.

Another user in IRC, `lrssvt` also confirmed experiencing this with this dataset (CARTO1_Lin_6CNivelD.zip) available here #8725-23

Related issues

Related to QGIS Application - Bug report #9519: Rollback operations slow if lots of features changed Closed 2014-02-09
Related to QGIS Application - Bug report #9509: Discarding edits freezes qgis for a long time Closed 2014-02-07

Associated revisions

Revision ccde424d
Added by Jürgen Fischer almost 11 years ago

reduce debugging noise in attribute table (fixes #9315)

Revision 3ba33d73
Added by Matthias Kuhn almost 11 years ago

[attrtable] Only update table at the end of an edit command
Fix #9315

Revision d378f942
Added by Matthias Kuhn almost 11 years ago

[attrtable] No checks when the model is not based on a request
Huge speedup
Fix #9315

History

#1 Updated by joe larson almost 11 years ago

  • Operating System changed from Windows 7 to Windows
  • OS version set to 7

This only occurs when Attribute Table is open.

#2 Updated by Nathan Woodrow almost 11 years ago

  • Assignee set to Matthias Kuhn

#3 Updated by Jürgen Fischer almost 11 years ago

  • Status changed from Open to Closed

#4 Updated by Matthias Kuhn almost 11 years ago

I've noticed a similar latency recently, but in a release build (i.e. without debug output, so Jürgens commits should not make any differece there). Adding a new calculated column to ~1mio rows was incredibly slow. This was with a build from before the recent changes concerning reliable signalling of attribute changes. I hope to get time to looking into batch updating.

#5 Updated by dr - almost 11 years ago

  • Operating System deleted (Windows)
  • Status changed from Closed to Reopened
  • OS version deleted (7)

Also noticed that slow performance occurs only when Attribute Table is open (master, Ubuntu).

#6 Updated by Giovanni Manghi almost 11 years ago

  • Category set to Vectors
  • Target version set to Future Release - High Priority

dr - wrote:

Also noticed that slow performance occurs only when Attribute Table is open (master, Ubuntu).

confirmed very slow field calculator performance when table is open.

#7 Updated by Matthias Kuhn almost 11 years ago

Is it a regression?

#8 Updated by Giovanni Manghi almost 11 years ago

Matthias Kuhn wrote:

Is it a regression?

it seems faster in qgis 2.0.1

#9 Updated by Matthias Kuhn almost 11 years ago

I could (partially) revert 5c38775 which could probably help here. But this would in turn reintroduce bug #9268 (updates not pushed to the attribute table)
In the future I would like to have a solution which solves both. For now somebody has to decide, what is more important (or agree on e.g. a threshold feature count for which live-propagating will be turned on/off)

http://lists.osgeo.org/pipermail/qgis-developer/2014-January/030018.html

#10 Updated by Giovanni Manghi almost 11 years ago

Matthias Kuhn wrote:

I could (partially) revert 5c38775 which could probably help here. But this would in turn reintroduce bug #9268 (updates not pushed to the attribute table)
In the future I would like to have a solution which solves both. For now somebody has to decide, what is more important (or agree on e.g. a threshold feature count for which live-propagating will be turned on/off)

http://lists.osgeo.org/pipermail/qgis-developer/2014-January/030018.html

personally if I have to choose I would prefer speed (in table edits) rather than #9268 fixed.

#11 Updated by Matthias Kuhn almost 11 years ago

I have created a branch which could potentially fix both issues, but I did not have any time at all to test it.

If somebody is able to test this and give a feedback... It would be great to have these things solved, but didn't even dare to create a pull request so far.

https://github.com/matthias-kuhn/QGIS/tree/tblspeed

#12 Updated by Matthias Kuhn almost 11 years ago

  • Status changed from Reopened to Closed

#13 Updated by Matthias Kuhn almost 11 years ago

  • Status changed from Closed to Reopened

My propsed fix was based on wrong assumptions, the main reason for the bad performance was not the fix for #9268 but another change introduced for the relations, which led to tons of unnecessary requests. Fixing that brought back an acceptable speed and the proposed fix implemented this morning gave another remarkable speed boost.

Please help testing thiese fixes and report any related bugs or reopen if this issue should still not be solved for any reason.

#14 Updated by Matthias Kuhn almost 11 years ago

  • Status changed from Reopened to Closed

#15 Updated by Salvatore Larosa almost 11 years ago

Hi Matthias,
I just noticed that with large shapefiles (over ~70k features)
when turn off the edit mode and you choose "close without saving" QGIS freezes (not crashes).

#16 Updated by Giovanni Manghi almost 11 years ago

Salvatore Larosa wrote:

Hi Matthias,
I just noticed that with large shapefiles (over ~70k features)
when turn off the edit mode and you choose "close without saving" QGIS freezes (not crashes).

Hi Matthias and Salvatore,

are the fixes already committed in master or to test I have to use Matthias branch?

thanks!

#17 Updated by Salvatore Larosa almost 11 years ago

Giovanni Manghi wrote:

are the fixes already committed in master or to test I have to use Matthias branch?

Yes, they are in master branch now.

#18 Updated by Giovanni Manghi almost 11 years ago

Salvatore Larosa wrote:

Hi Matthias,
I just noticed that with large shapefiles (over ~70k features)
when turn off the edit mode and you choose "close without saving" QGIS freezes (not crashes).

very true, when discarding edits it take ages to exit the freeze state.

#19 Updated by Matthias Kuhn almost 11 years ago

I noticed the same. This is due to the undo operation requesting all the features in single requests AFAICT.

Please note that this also happens with the attribute table closed and should therefore be treated as a separate issue.

See #9519

#20 Updated by Jürgen Fischer almost 11 years ago

Matthias Kuhn wrote:

Please note that this also happens with the attribute table closed and should therefore be treated as a separate issue.

See #9519

Giovanni already filed #9509.

#21 Updated by Jürgen Fischer over 7 years ago

  • Description updated (diff)
  • Priority changed from Severe/Regression to Low

Also available in: Atom PDF