Bug report #17119

master using too much memory

Added by Patrick Dunford about 2 years ago. Updated 8 months ago.

Status:Feedback
Priority:Normal
Assignee:-
Category:Map Canvas
Affected QGIS version:master Regression?:No
Operating System:Debian 9.1 Easy fix?:No
Pull Request or Patch supplied:No Resolution:
Crashes QGIS or corrupts data:No Copied to github as #:25018

Description

I have a project with about a thousand layers including currently about 600 raster images totalling about 6 GB. These are not fixed layers, I just open some aerial photos for the area I am currently mapping, so I can have a variable number of rasters open at any one time.

If I open this in a January build e.g. 313ec55 it uses about 5 GB of memory in the gui editor.

If I open it in the latest nightly 1caaa2e it uses 19 GB of memory with 433 of the rasters displayed. The VM only has 12 GB physical ram so it starts paging like crazy.

If I turn off displaying the aerial images in the gui it drops memory usage massively.

This looks like the composer issue, loading all the layers in memory even if they aren't actually needed.

If there is an expectation of loading all the layers into memory to speed up refresh, we need a setting to be able to turn this behaviour on or off, as for projects with very large amounts of data, the disk paging needed to deal with the large memory allocation obviates any benefit from refreshing layer data from memory instead of disk.

History

#1 Updated by Giovanni Manghi about 2 years ago

  • Status changed from Open to Feedback

Hi,

how this differ from #17117 ? Thanks.

#2 Updated by Giovanni Manghi about 2 years ago

what is the "gui editor"? is this memory consumption issue about the print composer or also the QGIS canvas?

#3 Updated by Patrick Dunford about 2 years ago

I am not a developer, I don't know how you describe different parts of the software
so maybe "canvas"
and this is a new behaviour in recent builds
and I don't have money to run out and buy lots more RAM for my PC so it can run the new software version without paging everything to disk and slowing down massively.

#4 Updated by Giovanni Manghi about 2 years ago

  • Subject changed from 2.99 1caaa2e using too much memory to master using too much memory

Patrick Dunford wrote:

I am not a developer, I don't know how you describe different parts of the software
so maybe "canvas"
and this is a new behaviour in recent builds

that is not developer terminology, it is really a general user one.
When you launch the program it open what many of us refer as "qgis main window". In there the part that shows the map is the "canvas". Then you can open a separate window called the "print composer" that allows to create (stating the obvious here)... maps for printing.
In the original description you speak about a "gui editor" which I'm personally not sure what it is, then you also speak about the composer and as category yo choose "unknown", so it is not unusual that someone ask to clarify.

and I don't have money to run out and buy lots more RAM for my PC so it can run the new software version without paging everything to disk and slowing down massively.

I don't think that anyone has suggested you to do so. As asked yesterday my another maintainer of the bug tracker let's try to keep the attitude positive, if after a filing a ticket we are asking questions it is to allow understand better the issues, where the source of the problem can be as also possible solutions.

So... can you clarify a bit your original description? and again... how this ticket is different from #17117 ?

#5 Updated by Patrick Dunford about 2 years ago

The canvas behaviour is new in recent releases, whereas in older masters and in the LTS versions it's not happening, it doesn't gobble up all that memory for the raster layers if you have a lot of them. When I have filed the bug report for the composer problem, I used a January version of the master (313ec55) and that version doesn't exhibit the high memory usage, hence it was possible to extrapolate all those numbers showing how much memory the composer was using at different times.

I suspect but haven't yet tested that I will find with this version it won't be possible to determine the composer memory use because it will just re-use the existing memory buffers for the canvas instead of grabbing its own chunk of memory.

The composer, on the other hand, this behaviour appears to exist in multiple different versions, certainly in 2.18. I have 2.14 and 2.16 environments and haven't tried to see how the composer works in them.

#6 Updated by Giovanni Manghi about 2 years ago

Patrick Dunford wrote:

The canvas behaviour is new in recent releases, whereas in older masters and in the LTS versions it's not happening, it doesn't gobble up all that memory for the raster layers if you have a lot of them. When I have filed the bug report for the composer problem, I used a January version of the master (313ec55) and that version doesn't exhibit the high memory usage, hence it was possible to extrapolate all those numbers showing how much memory the composer was using at different times.

I suspect but haven't yet tested that I will find with this version it won't be possible to determine the composer memory use because it will just re-use the existing memory buffers for the canvas instead of grabbing its own chunk of memory.

The composer, on the other hand, this behaviour appears to exist in multiple different versions, certainly in 2.18. I have 2.14 and 2.16 environments and haven't tried to see how the composer works in them.

ok thanks. Sorry for insisting, how this ticket is different from #17117 ?

#7 Updated by Patrick Dunford about 2 years ago

Ticket 17117 refers specifically to the behaviour using composer
Ticket 17119 refers specifically to the behaviour using the canvas

They have different classifications in the bug reporting system.
And they are different software functions for different purposes.
Composer does different things from canvas.

#8 Updated by Giovanni Manghi about 2 years ago

  • Category changed from Unknown to Map Canvas
  • Status changed from Feedback to Open

Patrick Dunford wrote:

Ticket 17117 refers specifically to the behaviour using composer
Ticket 17119 refers specifically to the behaviour using the canvas

They have different classifications in the bug reporting system.
And they are different software functions for different purposes.
Composer does different things from canvas.

as the behavior is the same (or not?) I suspect (strictly from a user point of view) that the issues is the same. But I'll leave the real answer to a dev, of course.

See also #17117

#9 Updated by Patrick Dunford about 2 years ago

It depends on your perception of the user or usability experience of the different software functions. There are significant differences.

If I open the composer and use it like a canvas to browse (but not edit) the map it doesn't use all this extra memory.

It only starts gobbling the memory when I tell it I want it to render an image or PDF of the current view of the map I can see in its preview. At that point, the question is a very different one because loading all of the extra images, that are not actually needed to render the current view of the map, is a superfluous waste of time and resources. It might even slow down the generation of the image (or PDF). These resources then get discarded after the render and aren't used again until the next render.

The behaviour in the canvas (the main application) is quite different. Here, there is a legitimate reason for wanting to load all the images - fast refreshes of the screen as you move to different areas of the canvas. It's not in the same league as loading up resources that are never going to be used. I think it is possible to make a case that since the composer has already loaded the rasters needed to display the preview, it is hard to understand why it needs access to rasters that are outside this preview.

These distinctions are only relevant to the behaviour of the older masters and 2.18 LTS. In the latest masters it's possible the composer might be coded to assume all these resources are globally available to all parts of the application, since they appear to be loaded automatically with the project, judging by memory usage. At that level, it becomes more of a question of whether a possible solution to the canvas issue - per project settings to turn off raster caching - is further supported by a recoding of composer to eliminate the resource wastage.

You can make a case that the composer preview refresh is the same as canvas refresh and there is a case for caching images for use by composer - this is only the case in the latest masters, not older ones or 2.18, where the composer behaviour appears to be unrelated to preview refresh. I can see there is a point of the latest version in (presumably) attempting to leverage a beneficial speed increase in composer preview refresh off the back of canvas refresh. This has to be balanced against the possibility that the user might need to turn off this canvas rendering cache if it uses too many resources in their system. If they need to be able to reduce the memory usage of the canvas they won't be wanting to see the resource wastage in composer in less recent masters and current stable releases.

#10 Updated by Giovanni Manghi 8 months ago

  • Status changed from Open to Feedback

Please check if this issue is still valid on QGIS 3.4.5 or 3.6.

Also available in: Atom PDF