Bug report #10828

project loading and adding/removing layers is slow if project has composers

Added by Per Gammerath over 5 years ago. Updated about 2 years ago.

Status:Closed
Priority:Normal
Assignee:-
Category:Project Loading/Saving
Affected QGIS version:2.18.4 Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:No Resolution:fixed/implemented
Crashes QGIS or corrupts data:No Copied to github as #:19203

Description

I don't know whether to report this as a bug or feature request, but QGIS is horribly slow at loading projects. What takes 19 seconds in Dufour, takes 32 seconds in Chugiak.

see also:
#12125

QGIS.zip (4.15 MB) Per Gammerath, 2015-06-23 01:26 AM

With_layouts.qgs (1.32 MB) Per Gammerath, 2015-06-23 01:39 AM


Related issues

Duplicated by QGIS Application - Bug report #15499: Qgis Desktop 2.14.5 and 2.16 Projects with Multiple Compo... Closed 2016-08-29

Associated revisions

Revision 18a1c8a6
Added by Martin Dobias over 5 years ago

Fix #10828 (slow project loading)

Revision 904443ff
Added by Martin Dobias over 5 years ago

Fix #10828 (slow project loading)

History

#1 Updated by Giovanni Manghi over 5 years ago

  • Category set to Project Loading/Saving
  • Status changed from Open to Feedback

it may be a bug, but anyway without attaching a project (and data) to replicate the issue it would be hard to troubleshoot it.

#2 Updated by Per Gammerath over 5 years ago

Here is my project.

#3 Updated by Nathan Woodrow over 5 years ago

Have you enabled multicore rendering?

Settings -> Options -> Rendering

#4 Updated by Per Gammerath over 5 years ago

I can't upload the project. The file is 12MB.

#5 Updated by Per Gammerath over 5 years ago

Yes, Multicore rendering is enabled. Rendering is very good once the project is loaded.

#6 Updated by Giovanni Manghi over 5 years ago

Per Gammerath wrote:

I can't upload the project. The file is 12MB.

dropbox (or similar)?

#8 Updated by Giovanni Manghi over 5 years ago

  • Status changed from Feedback to Open
  • Subject changed from Slow loading to Slow project loading in qgis 2.4/master when compared to 2.2
  • Affected QGIS version changed from 2.4.0 to master

I had to change the data source path in your project but after that I can confirm your observation:

  • in qgis 2.2 qgis loads the layers, at the same time builds the layers tree and shortly after in renders the canvas
  • in qgis 2.4 it seems to be sequential: qgis laods the layers, then it stays busy, then the TOC tree is rendered all together, and then the layers are rendered in the canvas.

I don't know if this depends on multi-threading, the legend refactory and/or other factors. If this has a potential fix then we should tag this as a regression.

#9 Updated by Per Gammerath over 5 years ago

I have been suspecting our project file of being messy and siltet, so I built a new project file and styled it to look like the original. The result was amazing, now the project loads in 16 seconds. This tells me, that there must be something wrong in the process of loading and saving project files. Our original project file was initially built in 2.0.1.
In the proces of building the new project file, QGIS crashed more than 20 times, just from organizing layers and loading styles. It's highly unstable.

#10 Updated by Per Gammerath over 5 years ago

Hmmm, just added all the composer layouts, and now everything is slow again. So maybe this is all about composer layouts.

#11 Updated by Giovanni Manghi over 5 years ago

In the proces of building the new project file, QGIS crashed more than 20 times, just from organizing layers and loading styles.

something that can be replicated every time?

#12 Updated by Giovanni Manghi over 5 years ago

  • Status changed from Open to Feedback

Per Gammerath wrote:

Hmmm, just added all the composer layouts, and now everything is slow again. So maybe this is all about composer layouts.

it is just enough to add one layout? does it need to contain a particular object to make slow the loading of the project?

#13 Updated by Per Gammerath over 5 years ago

Giovanni Manghi wrote:

In the proces of building the new project file, QGIS crashed more than 20 times, just from organizing layers and loading styles.

something that can be replicated every time?

It seems to be random. It crashed so many times that I had to save the project with each change to make sure I didn't have to start all over.

#14 Updated by Per Gammerath over 5 years ago

Giovanni Manghi wrote:

Per Gammerath wrote:

Hmmm, just added all the composer layouts, and now everything is slow again. So maybe this is all about composer layouts.

it is just enough to add one layout? does it need to contain a particular object to make slow the loading of the project?

I have 8 layouts, A4, A3, A2, A1, all with landscape and portrait orientation. I don't see how I can do without them.

#15 Updated by Per Gammerath over 5 years ago

If I'm right about the fact that Layouts slow down the loading of projects, would it be reasonable to let QGIS skip them and only load them when needed?

#16 Updated by Giovanni Manghi over 5 years ago

Per Gammerath wrote:

I have 8 layouts, A4, A3, A2, A1, all with landscape and portrait orientation. I don't see how I can do without them.

I didn't mean this.

As you said that you built the projects from scratch and that at some point you realized that it depends on layouts, then I would like to know if it was enough to add 1 layout (to get a slow to load project) and if the content of the layout made some difference.

#17 Updated by Per Gammerath over 5 years ago

For each layout I deleted, QGIS got faster. So I think it's about the number of layouts and not the content.
My layouts include a map, legend, small logo, date field and scale field.

#18 Updated by Nyall Dawson over 5 years ago

One thing that would help track this down would be if you could try removing different item types from the compositions and testing whether a single item type is causing the slow down. For instance, if you remove all the scale bars from all the compositions does this make any difference to the project load speed?

#19 Updated by Per Gammerath over 5 years ago

Nyall Dawson wrote:

One thing that would help track this down would be if you could try removing different item types from the compositions and testing whether a single item type is causing the slow down. For instance, if you remove all the scale bars from all the compositions does this make any difference to the project load speed?

I have tested some different scenariois, and load speeds are:
  • Project file with 8 layouts including a map, legend, scalebar, logo
    30 seconds
  • Project file with 8 layouts including a map, scalebar, logo
    24 seconds
  • Project file with 8 layouts including a map
    23 seconds
  • Project file with 2 layouts including a map, scalebar, legend, logo
    14 seconds

I think this indicates that the number of layouts and load speed is tied together.

#20 Updated by Nyall Dawson over 5 years ago

Are they the only elements in your layouts? ( map, scalebar, legend, logos ?). No other items at all, like arrows or shapes?

#21 Updated by Per Gammerath over 5 years ago

Nyall Dawson wrote:

Are they the only elements in your layouts? ( map, scalebar, legend, logos ?). No other items at all, like arrows or shapes?

No other objects.

#22 Updated by Martin Dobias over 5 years ago

  • Status changed from Feedback to Closed

#23 Updated by Paolo Cavallini over 5 years ago

  • Status changed from Closed to Feedback

Better backport it to 2.4 branch?

#24 Updated by Giovanni Manghi over 5 years ago

  • Resolution set to fixed/implemented

#25 Updated by Giovanni Manghi over 5 years ago

  • Status changed from Feedback to Closed

Paolo Cavallini wrote:

Better backport it to 2.4 branch?

#18a1c8a6360faf69a9b1bb9620b75baa7f1319cd#commitcomment-7044552

#26 Updated by Jürgen Fischer over 5 years ago

Giovanni Manghi wrote:

Paolo Cavallini wrote:

Better backport it to 2.4 branch?

#18a1c8a6360faf69a9b1bb9620b75baa7f1319cd#commitcomment-7044552

done in 904443ff0d49b3d6b6b883cb479ab2f0b5f5f6af

#27 Updated by Jürgen Fischer over 5 years ago

  • Tag set to backport

#28 Updated by Per Gammerath over 4 years ago

  • Status changed from Closed to Reopened

I've just downloaded and tested QGIS 2.8, and it strikes me that the issue with slow loading still exists, though the resolution is set to fixed. How come?

#29 Updated by Saber Razmjooei over 4 years ago

  • Status changed from Reopened to Feedback
  • Affected QGIS version changed from master to 2.8.2

Do you have some sample projects with data?

#30 Updated by Per Gammerath over 4 years ago

Try this these projects

#31 Updated by Saber Razmjooei over 4 years ago

It opens and saves pretty much instantly for wihtout layout.qgs and a couple of seconds with layout.qgs project on my QGIS 2.8.2.

#32 Updated by Per Gammerath over 4 years ago

On my computer, there is a 11 seconds difference. To emphasize the problem, try and the duplicate layouts, and you will see that it takes even longer.

#33 Updated by Per Gammerath over 4 years ago

Try this map

#34 Updated by Saber Razmjooei over 4 years ago

With 22 print layouts it loads in ~10 seconds for me now. With this no. of layout, I'd say it is acceptable!

#35 Updated by Per Gammerath over 4 years ago

You must have a really fast computer then. I'm running QGIS on a quad core 2.40 GHz, 4 GB ram. With 30 layouts, it takes about 30 seconds to load.
Nevertheless, I would like to know why QGIS has to process these layouts upon opening and not on demand?

#36 Updated by Saber Razmjooei over 4 years ago

  • Status changed from Feedback to Open

#37 Updated by Per Gammerath about 4 years ago

  • Target version set to Version 2.12

I see that this issue persists in QGIS Pisa. I still would like to know why the layouts can't be loaded on demand?

#38 Updated by Lene Fischer about 4 years ago

Times for saving project on local C: or network H:

1. Projekt with composers (1-2 composers) on C: 1-2 sec
2. Projekt with composers (1-2 composers)on H: 24 sec
3. Projekt without composers on c: < 1 sec
4. Projekt without composers on h: 9 sec

Test times are from save until mouse react again. Test in 2.12

#39 Updated by Giovanni Manghi about 4 years ago

Per Gammerath wrote:

I see that this issue persists in QGIS Pisa. I still would like to know why the layouts can't be loaded on demand?

not sure about that, anyway I retested all the projects available in this ticket and loading time for projects with composers is somwhow slower than the projects without, but it is always in the few seconds range, really nothing that would led me to think about an issue to solve. I retested on master on both linux and windows.

#40 Updated by Giovanni Manghi about 4 years ago

  • Status changed from Open to Feedback

#41 Updated by Giovanni Manghi about 4 years ago

  • Target version deleted (Version 2.12)

#42 Updated by Per Gammerath almost 4 years ago

  • Assignee set to Giovanni Manghi

It seems that this issue is more acute when projects are loaded from a network drive, as Lene Fischer stated in comment #39. Subjecting users to such slow speeds is quite unfortunate, especially when QGIS becomes unresponsive in Windows.

#43 Updated by Per Gammerath almost 4 years ago

I copied my project to a local drive and I realized that it didn't change the matter.

Project with 6 composers: 26 seconds
Project with 0 composers: 6 seconds

Please consider a solution for this annoyance.

#44 Updated by Giovanni Manghi almost 4 years ago

  • Resolution deleted (fixed/implemented)
  • Subject changed from Slow project loading in qgis 2.4/master when compared to 2.2 to project loading and adding/removing layers is slow if project has composers
  • Status changed from Feedback to Open
  • Assignee deleted (Giovanni Manghi)

see also #12125 and in particular #12125-17

#45 Updated by Kilian Hagemann over 3 years ago

OMG, can't believe I only found this bug now.

We deal with very large and complex (>150 layer) gis project files and they take forever to open. Now that I found this bug I was curious and did some benchmarks of our own:

  • test file: 193 layers and 106 print composers
  • test environment: QGIS 64bit on Windows 10, 8GB RAM, recent i3 dual core CPU (4 logical processors)
  • measured from end of missing layer dialogue (artificially inserted to remove file loading times which is disk/network dependent) to the time the canvas is rendered
  • with QGIS 2.8.8:
    • timing WITH print composers: 00:16:25 with 30-45% CPU usage
    • timing WITHOUT print composers: 00:00:03
  • with QGIS 2.14.3:
    • timing WITH print composers: 01:09:56 with 30-45% CPU usage
    • timing WITHOUT print composers: 00:00:04

So QGIS 2.8 takes about 9 seconds and QGIS 2.14 40 seconds (!) for each print composer saved inside the .qgs file. I hope this helps to stress the severity of this bug, so please could the QGIS devs look into the code that loads and parses the print composers and remove the bottlenecks? If the info provided by the OP and others isn't sufficient, I could help in reproducing the issue.

Until then I guess we will all remove print composers from our projects and manage them outside QGIS...

#47 Updated by Bjoern Gessler about 3 years ago

Got the same problem as well under Linux, as under Windows 7 and 10. QGIS VErsion 16.3 and 16.1.
I got a project with about 15 Print composers, some of them quite large (about 60 x 90 cm). The QGIS-project has risen to a size of about 16MB, loading the project takes about 20 Minutes on a i5-3,4 GHz computer with 8 GB of RAM.
Also saving the project sometimes takes up to 5 minutes.
Makes working sometimes really annoying...
Guess this might be a major issue witch then could be more or less simply fixed by not loading all map layouts into the memory at start up.

#48 Updated by Giovanni Manghi over 2 years ago

  • Status changed from Open to Closed
  • Resolution set to fixed/implemented
  • Affected QGIS version changed from master to 2.18.4

There have been several improvements in both 2.18.* and master (qgis3) and all the provided projects with/without layouts now open in a very reasonable amount of time, especially on master.

#49 Updated by Nyall Dawson over 2 years ago

With the changes merged in https://github.com/qgis/QGIS/pull/4279 all sample projects attached here load instantly.

#50 Updated by Ábel Kőszeghy about 2 years ago

My project files with many layouts still load painfully slow from network drives. I'm on version 2.18.14.

Also available in: Atom PDF