QGIS » QGIS Application

How about memory consumption by the time the crash occur?

It depends on whether you think the software should behave nicely and give some sort of error message or just crash.

OK so I would need to know how to monitor this
and these days with modern OS we expect virtual memory to handle this so I need to know if there is a particular configuration of the computer or in this case VM that is needed to ensure there is enough virtual memory available.
As you can edit the project in the main gui window with that number of layers open (over 1000 in this case), it is composer that is crashing.

sample project here, this is purely a sample, it is not real world because this project is far too complex, it would take me all day to archive the data and would be probably 10 GB of data. This sample works by loading the same group of rasters 5 times.

https://drive.google.com/drive/folders/0B_NF_W8CCnSgY1pCQlZuZTNmYk0?usp=sharing

I think the question is going to rise around whether composer actually needs the huge amount of memory it demands to produce an image (the issue here is the Save as Image export from Composer).
I have been monitoring the performance in top using a 12 GB virtual machine running Debian 9.1
The editor sits there using 5 GB of virtual memory with this project open. I have turned off a few layers that aren't needed in the composer.
I can use Save as Image from the editor and see no memory spike. The output image is 1326x816 pixels in a PNG.

However if I ask composer to do virtually the same thing it asks for 12 GB of virtual memory just for this task.
Whilst there may well be 12 GB of JPEG images in total in the project it does not need to load every single one of them to render the output.
The output is a PNG image 3425x2362 pixels i.e. 17 megapixels.

As we know the default DPI in composer for outputting to an image is 300 dpi. I imagine I could drop this down to 200 or 100 without losing too much quality for the purpose I am using these maps for.
However I think the question of it having an efficient usage of memory remains relevant.

Thanks for all the details. So the question stand: does the program crash when all the available memory (plus paging/swap) is used?

I don't see why crash is the only criteria, I don't see why I could not have legitimate concerns that the software is a resource hog, or have an option to configure this behaviour in the software.

Patrick Dunford wrote:

I don't see why crash is the only criteria, I don't see why I could not have legitimate concerns that the software is a resource hog, or have an option to configure this behaviour in the software.

I didn't say the contrary, I just asked a "simple" question: does it crashes when it eats up all the memory/swap or before?

Sorry, I don't know enough about linux virtual memory to test that.

Patrick Dunford wrote:

Sorry, I don't know enough about linux virtual memory to test that.

use "top", "htop" or "system monitor" application (name can change depending if you use Gnome/KDE/etc.). Then just have a look how memory consumption grows.

This issue does not occur in 2.14
Using the same project and rasters, in 2.14 the 280 MP image (20 000 x 14 000, or 1750 dpi) required about 1 GB of extra memory and took about 3 minutes to render. This was with all the 600 rasters loaded and 433 displayed. At that resolution the 2.18/master composer would have an astronomical memory demand. (I must test it to see how long it takes to crash)

Attempting a 1750 dpi image in master build 313ec55 resulted in a demand for 415 GB of memory (0.415 TB as displayed in top) and the system crashed very rapidly. This VM running Debian 9.1 has 12 GB of physical memory and 8 GB of swap.

Testing on the latest masters shows a very efficient composer rendering resource optimisation (whether through reusing the loaded layer buffers for the canvas or reverting to the 2.14 composer behaviour of only loading the actually required images) at the cost of the high resource use for buffering all of the layers in canvas.

Ideally the preferred solution for working with large projects of this nature (hopefully a per project setting) is going to be the 2.14 behaviour with only the minimum number of layers buffered in canvas and only the minimum number of layers actually needed loaded by the composer to render the output. Whilst it is possible to take down the DPI and reduce the memory footprint for an image render, there is no option to do this for PDFs which appear to render at some preconfigured number of DPI.