Bug report #12125

Very poor performance with complex projects (saving, adding/removing layers)

Added by Samuel Ingarfield over 4 years ago. Updated almost 4 years ago.

Status:Closed
Priority:Normal
Assignee:-
Category:Project Loading/Saving
Affected QGIS version:2.6.0 Regression?:No
Operating System:Windows Easy fix?:No
Pull Request or Patch supplied:No Resolution:duplicate
Crashes QGIS or corrupts data:No Copied to github as #:20319

Description

I think these two issues are related:

- Opening a project with many (>100) layers takes a very long time. One of my projects takes upwards of 30mins.
- Once open, adding or removing layers is fast (ie, instantaneous) until I save the project. From that point on, adding or removing layers is extremely slow. I had a quick look in the debugging output, it appears that after the save, the entire project is reloaded on each addition and removal of a layer. It goes without saying that this issue is very frustrating, and promotes workflows which involve minimal saving which is not good.

postgis_opening_log_17_layers.txt Magnifier - SQL query logs when opening qgis file with 17 layers (80.9 KB) Maxim Broos, 2015-02-09 03:23 PM


Related issues

Related to QGIS Application - Bug report #12170: Adding a print layout adds a lot of project loading time Closed 2015-02-10

Associated revisions

Revision 837c7ee8
Added by Nyall Dawson over 4 years ago

[composer] Don't redraw map if layers change and map is set to cached

Previously map was being redrawn every time a layer was added or
removed from the registry, which blocked the ui until the map
redraw is complete.

Now the map will only be redrawn as a result of layer changes
if it's set to Render mode.

Possible fix for 12234 and 12125. (refs #12234, #12125)

Revision d371fd63
Added by Nyall Dawson over 4 years ago

[composer] Don't redraw map if layers change and map is set to cached

Previously map was being redrawn every time a layer was added or
removed from the registry, which blocked the ui until the map
redraw is complete.

Now the map will only be redrawn as a result of layer changes
if it's set to Render mode.

Possible fix for 12234 and 12125. (refs #12234, #12125)

(cherry-picked from 837c7ee8d3ecacfce45ac0a299eb20c8a76f983b)

History

#1 Updated by Samuel Ingarfield over 4 years ago

Forgot to add, I see the same behaviour on both 2.6.1 and master.

#2 Updated by Giovanni Manghi over 4 years ago

  • Status changed from Open to Feedback

this seems a pretty important issue, but that I cannot easily replicate unless you can share a full project and data. I would suggest to raise the problem on the users and developers mailing lists, and get feedback from other power users. Thanks.

#3 Updated by Regis Haubourg over 4 years ago

Hi, we have faced such things here and found some issues that might be in cause:

  • layers are queried for extent and geometry type on project loading. Some formats can be long extracting those informations, if there is no statistic included. We have slowliness when using postgis view becauses stats are on table side only. This occurs only for very large tables (> 100K rows or very complex geometries) .
  • QGis project stores ghost layer definitions that are loaded but not used. See #10237. This occurs on large projects here. Solution by now is to clean up qgs file in a text editor (risky task, do some backups)
  • Databases can sometimes wait for connections to be released, and simply make QGIS wait. We had this here using postgis + PGpool. Default postgres behaviour is to reject connections if none is available, and that throws a error message in QGIS. PGpool does just wait, and QGIS does so.. sometimes for a very long time. Workaround is simple, set a higher max_connection value on your database or pooler tool.

Hope that helps you eliminate some upstream issues.

#4 Updated by Luca Lanteri over 4 years ago

I can confirm the issue but I can't replicate it every time. As said It seem to occurs after the first save and in any case it disappears after close and reopen the project.

#5 Updated by Samuel Ingarfield over 4 years ago

I've just tried to produce a minimal example, but unfortunately the issue doesn't seem to manifest itself with small project files. I'm not in a position to share a full project and data. (would be 10's of GB anyway so not very useful). I can say that this issue appears consistently across all my large projects, so is not just an artifact of some single dodgy project file.

Regis Haubourg wrote:

Hi, we have faced such things here and found some issues that might be in cause:
...

Thanks for the pointers Regis. Yeah, I've noticed that QGIS spends a long time querying extents and geometry types from my Postgis backend on loading - I'd say it probably accounts for half of my project load time. I have had some luck by moving from complex queries to materialised views, though I dislike this from an architectural point of view.

In a way I can live with very long project load times, so long as we can fix the very-slow-after-first-save issue.

Will cross-post on the dev list.

#6 Updated by Benoit de Cabissole over 4 years ago

  • OS version changed from 7 to 7 & 8.1

Same problem here with only shapefiles as vectors and geotiffs as rasters. No database connections. More than 80 layers having different CRS (4). On load most of the time is spent on loading plugins, the loading of layers is not fast but faster that plugins loading. Real problem is removing/adding layers (both raster and vector) once the project has been saved once: takes up to two minutes to complete the operation on a 16GB RAM, i7 computer.

#7 Updated by Martin Dobias over 4 years ago

Could you please ensure that the problem is not caused by one of the plugins? Either start QGIS with --noplugins argument on the command line - or manually disable all plugins in the plugin manager dialog.

I think in this bug report we should focus on the second part - poor performance after saving - as that is obviously wrong...

#8 Updated by Giovanni Manghi over 4 years ago

Martin Dobias wrote:

Could you please ensure that the problem is not caused by one of the plugins? Either start QGIS with --noplugins argument on the command line - or manually disable all plugins in the plugin manager dialog.

I think in this bug report we should focus on the second part - poor performance after saving - as that is obviously wrong...

I think I may have found one possible reason for the slow loading of projects.
I was writing that I has tested with a ~90 layers project

https://www.dropbox.com/s/tc7m15ngp5dbzed/test_slow_project.zip?dl=0

and seen no problems in loading speed, adding/removing layers, saving, etc.

Then I tried to add a composer layout... load time increased... a lot.

#9 Updated by Giovanni Manghi over 4 years ago

Giovanni Manghi wrote:

Martin Dobias wrote:

Could you please ensure that the problem is not caused by one of the plugins? Either start QGIS with --noplugins argument on the command line - or manually disable all plugins in the plugin manager dialog.

I think in this bug report we should focus on the second part - poor performance after saving - as that is obviously wrong...

I think I may have found one possible reason for the slow loading of projects.
I was writing that I has tested with a ~90 layers project

https://www.dropbox.com/s/tc7m15ngp5dbzed/test_slow_project.zip?dl=0

and seen no problems in loading speed, adding/removing layers, saving, etc.

Then I tried to add a composer layout... load time increased... a lot.

anyay confirmed also with much smaller projects: adding a print composer adds a lot of load time to projects (qgis master).

#10 Updated by Maxim Broos over 4 years ago

I can confirm this as well, adding a print composer adds loading time. But adding ten print composers doesn't add ten times the extra loading time.

When looking at opening a qgis file with 17 (!)EMPTY PostGis layers in it, I also find a lot of redundant queries:
"SELECT typname,typtype,typelem,typlen FROM pg_type WHERE oid=1043" gets executed 52 times.

In total i get over 500 queries to open a file with 17 layers. Is this by design? With these numbers, any network latency to the server becomes a big delay-issue.
(Not sure if this is related to the initial issue, i can open a new issue if that would be required)

#11 Updated by Giovanni Manghi over 4 years ago

Maxim Broos wrote:

I can confirm this as well, adding a print composer adds loading time. But adding ten print composers doesn't add ten times the extra loading time.

maybe the print composer issue is unrelated to the one that originated this ticket(?), but still seems a regression as I could not recall such problem in older releases.

#12 Updated by Giovanni Manghi over 4 years ago

Giovanni Manghi wrote:

Maxim Broos wrote:

I can confirm this as well, adding a print composer adds loading time. But adding ten print composers doesn't add ten times the extra loading time.

maybe the print composer issue is unrelated to the one that originated this ticket(?), but still seems a regression as I could not recall such problem in older releases.

in my case it affects only linux, not windows. I filed a separated ticket #12170

#13 Updated by Samuel Ingarfield over 4 years ago

Actually, it seems it may be related. As an experiment, I loaded one of my troublesome projects with plugins disabled (--noplugins; no difference) and then made a copy and deleted all the print composers. After saving, adding and deleting layers is just as fast as it was before the save.

So it seems it may be related to print composers after all. I'm on Win 7 (8GB RAM, i7), not linux.

#14 Updated by Benoit de Cabissole over 4 years ago

Also did some testing. Kept all plugins active, loaded a copy of one project having many composers, deleted all composers, saved. Deleted a layer = very fast, saved, deleted another layer = very fast, added a new layer = also very fast.
It really seems that having composers and saving a project will then slow down considerably the delete/add actions.
Behaviour is for 2.6.x & master; 2.2 does not have this problem, and if I remember correctly 2.4 exhibited the problem as well. All on Win8.1

#15 Updated by Giovanni Manghi over 4 years ago

Benoit de Cabissole wrote:

Also did some testing. Kept all plugins active, loaded a copy of one project having many composers, deleted all composers, saved. Deleted a layer = very fast, saved, deleted another layer = very fast, added a new layer = also very fast.
It really seems that having composers and saving a project will then slow down considerably the delete/add actions.
Behaviour is for 2.6.x & master; 2.2 does not have this problem, and if I remember correctly 2.4 exhibited the problem as well. All on Win8.1

With the latest master this issue seems to be gone. Please test.

#16 Updated by Samuel Ingarfield over 4 years ago

Hmm, the issue still exists with the latest master for me.

#17 Updated by Martin Dobias over 4 years ago

The only issue I can confirm right now is that whenever layers are added/removed, the composers' maps will be re-cached (even if the compositions are not opened). This can make QGIS freeze for some time for complex maps. (That is because the caching of map in composer is not done in background - compared to main map canvas).

Please note that it is not necessary to save the map first - this happens from the time the project is first opened (at least in master).

#18 Updated by Benoit de Cabissole over 4 years ago

QGIS Master 'ca86be7' (15/2/2015) still shows this behaviour.

If it can help: the composers are not specially complex: an A4 map (< 10 layers: 1 or 2 rasters, others vectors) with grid & coords + scale bar + simple legend + SVG image + png image. The map layers in the composers are generally different from the layers displayed in QGIS and the 'Lock layers for map item' checkbox is checked.

#19 Updated by Giovanni Manghi almost 4 years ago

  • Status changed from Feedback to Closed
  • Resolution set to duplicate

duplicate of #10828

Also available in: Atom PDF