Bug report #16049

Opening a lot of JPG rasters all at once causes crash on Linux

Added by Patrick Dunford almost 3 years ago. Updated about 2 years ago.

Status:Closed
Priority:High
Assignee:-
Category:Rasters
Affected QGIS version:master Regression?:No
Operating System:Xubuntu Easy fix?:No
Pull Request or Patch supplied:No Resolution:invalid
Crashes QGIS or corrupts data:Yes Copied to github as #:23964

Description

I have a folder of 280 geo referenced JPEGs which is downloaded aerial photography.
If open all those images at once in 2.99 every time it will come up with 180 data conversion errors and then crash.

In 2.14.10 there are no issues and no errors. Although that version works across the network whereas 2.99 computer is loading the files from its own hard drive.

qgis_bug_16049_fail_layer_load_Xubuntu_1604_x64_015fdf2.png (130 KB) Patrick Dunford, 2017-01-10 02:52 AM

qgis_bug_16049_resource_use_Xubuntu_1604_x64_015fdf2.png (23.7 KB) Patrick Dunford, 2017-01-10 02:52 AM

qgis_bug_16049_Xubuntu_1604_x64_015fdf2.png (494 KB) Patrick Dunford, 2017-01-10 02:52 AM

OtagoSouthland21410.qgs (3.04 MB) Patrick Dunford, 2017-01-12 01:56 PM

CanterburyWestland21410.qgs.zip (213 KB) Patrick Dunford, 2017-01-12 02:02 PM

tmof.png (44 KB) Patrick Dunford, 2017-09-09 01:00 PM

History

#1 Updated by Giovanni Manghi almost 3 years ago

  • Category set to Rasters
  • Status changed from Open to Feedback
  • Priority changed from Normal to Severe/Regression
  • Target version set to Version 3.0

Is the data freely available? Does it happens also on 2.18 or only master?

#2 Updated by Patrick Dunford almost 3 years ago

I would have to put it onto Google drive because it is a 2 GB download.

#3 Updated by Patrick Dunford almost 3 years ago

Is there a 2.99 available for Windows, I can't see any build for it in OSGeo4W

That is necessary to dispel the strong suspicion that this is a capacity limitation inherent to the Linux platform edition of Qgis.

I can edit the project and add all the rasters no problem in 2.14 running on Windows 8.1 x64. But any version running on Linux (2.14, 2.18, 2.99 all x64) all run into the same problems.

  • If you try to add the rasters it crashes when it apparently reaches some sort of limit. You can try to add as few as 5 at a time and get the error message about the invalid format. If it doesn't crash, it will display only some of the existing rasters (my projects also have other existing raster layers)
  • If you add all the rasters in Windows, close the project then reopen it in Linux, it gets so far then displays the bad layers dialog. The layers it misses are not just rasters, they are shapefiles and other stuff as well. It just hit some limit to the number of layers it would load.

I have attached one screendump showing the system that was running 2.99 had plenty of free resources and Qgis was only using 1.5 GiB - this computer has 24 GB of RAM, there was heaps of free memory, the CPU was at a tiny load and none of the 48 GB swap file was in use. The system has /home mounted on a 2 TB volume with about 40% free.

#4 Updated by Giovanni Manghi almost 3 years ago

Patrick Dunford wrote:

I would have to put it onto Google drive because it is a 2 GB download.

it is doable? I would like to try replicate the issue.

#5 Updated by Patrick Dunford almost 3 years ago

OK I will try to nail down how easy it is to replicate i.e. if you just add the same layers in multiple different groups, otherwise it will take forever and use a lot of online space to provide enough data to replicate.

#6 Updated by Giovanni Manghi almost 3 years ago

Patrick Dunford wrote:

if you just add the same layers in multiple different groups, otherwise it will take forever and use a lot of online space to provide enough data to replicate.

I can provide you a shared folder on google drive with enough space. Is ok?

#7 Updated by Patrick Dunford almost 3 years ago

OK you can share it to my email address on my profile kahukowhai at gmail.com

All you need to do is create a new project then add the 280 images to it, it will just crash after it gets to about 250.

The project when I can work on it in Windows has more than 1000 layers in it and it has no trouble at all in Windows. I have attached the project file so you can count just how many there actually are.

#8 Updated by Patrick Dunford almost 3 years ago

Sorry, wrong project file. Here is the correct one.

#9 Updated by Patrick Dunford almost 3 years ago

Here is the project file zipped.

#11 Updated by Giovanni Manghi over 2 years ago

  • Priority changed from Severe/Regression to High
  • Subject changed from Opening a lot of rasters all at once causes data conversion errors and crash [2.99 110ffe2] to Opening a lot of JPG rasters all at once causes crash on Linux
  • Status changed from Feedback to Open

I can see it happen on Linux. On Windows seems ok. Not a regression anyway, older releases also affected.

#12 Updated by R. R. over 2 years ago

Loading many .jpg tiles makes QGIS (2.18) quite unusable here on Windows 7.

#13 Updated by Patrick Dunford over 2 years ago

If you mean it slows down a lot, what are the specs of your system.

My Windows system is running Windows 10, with brand new Skylake hardware and 8 GB of RAM.

#14 Updated by R. R. over 2 years ago

This is not a hardware issue. ECW files work fine. I'll do some tests next week.

#15 Updated by Patrick Dunford over 2 years ago

Well, the project I have had all the trouble with has 750 aerial photos totalling 6 GB. It is slow to update when I zoom or drag, but I half expect this to be the case.

#16 Updated by Giovanni Manghi over 2 years ago

  • Regression? set to No
  • Easy fix? set to No

#17 Updated by Even Rouault about 2 years ago

  • Description updated (diff)

This issue really sounds like the QGIS process reaches the limit of 1024 file descriptor simultaneously opened, which is the default on Linux

Patrick, could you open a terminal and type "ulimit -n 10000" and then launch "qgis" from that terminal ?

#18 Updated by Giovanni Manghi about 2 years ago

  • Status changed from Open to Feedback

#19 Updated by Patrick Dunford about 2 years ago

ulimit doesn't work. prlimit does.

#20 Updated by Giovanni Manghi about 2 years ago

Patrick Dunford wrote:

ulimit doesn't work. prlimit does.

we should document this (if not already) and close this ticket.

#21 Updated by Patrick Dunford about 2 years ago

Changed the limit permanently in /etc/security/limits.conf

#22 Updated by Patrick Dunford about 2 years ago

Reinhard Reiterer wrote:

This is not a hardware issue. ECW files work fine. I'll do some tests next week.

No. It's something to do with Windows. The performance is much faster on Linux now that I can get it to open hundreds of images at a time.

#23 Updated by Patrick Dunford about 2 years ago

Even Rouault wrote:

This issue really sounds like the QGIS process reaches the limit of 1024 file descriptor simultaneously opened, which is the default on Linux

Patrick, could you open a terminal and type "ulimit -n 10000" and then launch "qgis" from that terminal ?

Very kind of you as the people here so far have told me it is some sort of inherent architectural limitation with Linux. Which has always seemed unlikely to me as people are running massive server tasks on the platform.

Now all we need is being able to install different build numbers of the development edition on Linux instead of always being given the latest one with no choice.

#24 Updated by Giovanni Manghi about 2 years ago

Now all we need is being able to install different build numbers of the development edition on Linux instead of always being given the latest one with no choice.

what do you mean?

#25 Updated by Patrick Dunford about 2 years ago

I mean if you can have old releases of the Windows master edition archived on the web site, why not Linux?

At the moment, it looks like the only way I can get an older Linux edition is to build it from the source; a non trivial task.

Master releases convert the project file up from a stable edition, but you can't go back to stable from master because too much data gets lost going down. I don't see how you can adequately test except by using real world data or conditions.

When I run into a problem with a new master edition that has bugs, at least on Windows I can uninstall that version and install my choice of an older release.

#26 Updated by Giovanni Manghi about 2 years ago

  • Status changed from Feedback to Closed
  • Resolution set to invalid

see #17101

#27 Updated by Harrissou Santanna about 2 years ago

Patrick Dunford wrote:

At the moment, it looks like the only way I can get an older Linux edition is to build it from the source; a non trivial task.

Maybe some other solution here https://lists.osgeo.org/pipermail/qgis-user/2017-August/040618.html

#28 Updated by Patrick Dunford about 2 years ago

Still doesn't allow you to install a particular version, this is a serious issue with major usability bugs in the latest masters.

#29 Updated by Regis Haubourg about 2 years ago

Can you open a separate feature request for that master installer need ?

Considering Linux packages logic, the only possibility I am aware of are:

- compile yourself. This is not so hard in fact on Linux, maybe that's why nobody make another option be more advertised. The only drawback is that cold compile are long, from 20 minutes to 1h30 depending on the number of cores you have. Hot compile will only recompile changed files and take some minutes only. Once that learned, this let you the ability to easily test development branches which is very nice for early testing.

- Use docker images. see here https://hub.docker.com/r/kartoza/qgis-desktop/ for instance. We might need to keep some more history in master builds. This is not for free, this requires storage, CPU and some human work to automate the process and keep it working.

- Use virtual machines. Same as docker but each machine is a lot heavier than a docker image.

The two last solutions would require funding to make them happen.

#30 Updated by Patrick Dunford about 2 years ago

"- compile yourself. This is not so hard in fact on Linux, "
but there is no documentation of where to find a particular master build, the instructions appear to be for downloading the most recent release, not finding a particular release. The builds I pulled from browsing the Github site appear to be 2.18 builds, not 2.99

#31 Updated by Jürgen Fischer about 2 years ago

Patrick Dunford wrote:

"- compile yourself. This is not so hard in fact on Linux, "
but there is no documentation of where to find a particular master build, the instructions appear to be for downloading the most recent release, not finding a particular release. The builds I pulled from browsing the Github site appear to be 2.18 builds, not 2.99

Of course. 2.99 is master and will never be released. The release 3.0 will be branched of master and then released as such. And the last release before 3.0 is 2.18. The point releases are tagged (eg. final-2_18_12). So you can checkout tags or alternatively use a tar balls from qgis.org. The are no qgis builds on github. We don't keep old linux builds. For Windows you might find older builds - but due to disk space constraints we don't keep a full archive of those either.

#32 Updated by Patrick Dunford about 2 years ago

Even Rouault wrote:

This issue really sounds like the QGIS process reaches the limit of 1024 file descriptor simultaneously opened, which is the default on Linux

Patrick, could you open a terminal and type "ulimit -n 10000" and then launch "qgis" from that terminal ?

Although this fixes the drawing of maps in the main software, I am now getting composer crashes with a large number of rasters open. These are affecting both the master and 2.18 versions. So far tested a master 313ec55 on Xubuntu 17.04 and Debian 9.1 and a 2.18 on Xubuntu 17.10

Please advise would you like to reopen this bug or start a new one.

#33 Updated by Patrick Dunford about 2 years ago

Even Rouault wrote:

This issue really sounds like the QGIS process reaches the limit of 1024 file descriptor simultaneously opened, which is the default on Linux

Patrick, could you open a terminal and type "ulimit -n 10000" and then launch "qgis" from that terminal ?

#34 Updated by Giovanni Manghi about 2 years ago

Patrick Dunford wrote:

what the attached image means, that raising the limit for open files do not work?

#35 Updated by Patrick Dunford about 2 years ago

Sorry it was meant to show what happens with the default setting, this is the actual error message returned.

#36 Updated by Harrissou Santanna about 2 years ago

Patrick Dunford wrote:

"- compile yourself. This is not so hard in fact on Linux, "
but there is no documentation of where to find a particular master build, the instructions appear to be for downloading the most recent release, not finding a particular release.

FYI, https://htmlpreview.github.io/?https://github.com/qgis/QGIS/blob/master/doc/INSTALL.html in case you would like to build your own QGIS (link available from http://qgis.org/en/site/forusers/download.html, Sources tab)

Also available in: Atom PDF