Bug report #4013

Large Projects: add limitations to the manual

Added by Jaideep Gupte almost 9 years ago. Updated over 4 years ago.

Status:Closed
Priority:Normal
Assignee:-
Category:Documentation and Help
Affected QGIS version:master Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:No Resolution:
Crashes QGIS or corrupts data:No Copied to github as #:14000

Description

Hi, I'm on 1.7 (on snow leopard). I'm have trouble opening a project with 206 layers (mix of vector and raster). Only 84 layers will open, the remaining 122 will be flagged as bad layers. Nothing I do in the Handle Bad Layer window makes any difference, all layers are lost. When I individually try and add these layers to the project, they are not recognised (error: "not a valid or recognized data source"). However, each of these layers opens flawlessly in a new project, or a project with less than 84 layers already open.

Is there a maximum limit for layers? Is this related to a cache issue? Any help would be most appreciated. Thanks!

History

#1 Updated by William Kyngesburye almost 9 years ago

Same limit here, 84 layers. Using master build 6d788eb from June 4. And I'm creating a new project, not opening an old one.

#2 Updated by William Kyngesburye almost 9 years ago

From Royce Cline: the limit of the number of open files per process on OS X is 256. I now remember this problem a while back, but figured it was solved somehow. But then, I don't add a lot of shapefiles to a QGIS project, until recently, usually it's postgis connections.

Royce's solution has been to set ulimit in a Terminal and start QGIS from there, but this is not something for the average user, and is generally a bad way (at least the wrong way) to start any application on OS X.

ulimit only affects processes started from a shel (Terminal). OS X apps get their limits from sysctl/launchctl. You can check it with:

launchctl limit

which for me on 10.6 shows 256 soft limit and unlimited hard limit. Huh, if a soft limit only gives the app a warning but lets it continue opening files, maybe there is a problem in how Qt handles it.

The sysctl limits look different:

kern.maxfiles: 12288
kern.maxfilesperproc: 10240

#3 Updated by Paolo Cavallini over 8 years ago

  • Category set to Project Loading/Saving
  • Pull Request or Patch supplied set to No

#4 Updated by Jürgen Fischer over 8 years ago

  • Priority changed from 6 to Normal

#5 Updated by Giovanni Manghi over 8 years ago

  • Target version set to Version 1.7.4

#6 Updated by Paolo Cavallini almost 8 years ago

  • Affected QGIS version set to master
  • Target version changed from Version 1.7.4 to Version 1.8.0
  • Crashes QGIS or corrupts data set to No

#7 Updated by gcarrillo - almost 8 years ago

Same issue here. In Linux the limit seems to be 1024 by default. I wonder how other software address this problem since it is platform-specific.

#8 Updated by Giovanni Manghi almost 8 years ago

gcarrillo - wrote:

I wonder how other software address this problem since it is platform-specific.

As far as I have understand it is an operating system limitation/issue, not a QGIS one. Consider also that when adding a shapefile you are opening at least 3 files, not 1.

#9 Updated by gcarrillo - almost 8 years ago

Giovanni Manghi wrote:

gcarrillo - wrote:

I wonder how other software address this problem since it is platform-specific.

As far as I have understand it is an operating system limitation/issue, not a QGIS one. Consider also that when adding a shapefile you are opening at least 3 files, not 1.

Giovanni, if you read what I wrote, you'll see that I'm aware it's not a QGIS problem but a platform (OS) one. My question goes in the direction of finding the ways other GIS software tackle this, so that QGIS can also implement their solutions, if there are any.

#10 Updated by Giovanni Manghi almost 8 years ago

My question goes in the direction of finding the ways other GIS software tackle this, so that QGIS can also implement their solutions, if there are any.

I guess fist we have to know if other GIS packages overcome this limitation. They do? any open source?

#11 Updated by Paolo Cavallini over 7 years ago

  • Target version changed from Version 1.8.0 to Version 2.0.0

#12 Updated by Daniel Vaz over 6 years ago

I think that a good solution would be to keep the user informed about OS limitations.

In the GRASS Wiki they report OS limitations to users (see Number of open files limitation): http://grasswiki.osgeo.org/wiki/Large_raster_data_processing

So let's keep the information accessible by users and create/update a wiki page about Know Issues.

Thanks in advance.

#13 Updated by Daniel Vaz over 6 years ago

Closing related Bug #6404.

#14 Updated by Daniel Vaz over 6 years ago

I created a wikipage

Known Issues

I think there is no more reason to keep this issue open.

Please, if necessary reopen it.

#15 Updated by Paolo Cavallini over 6 years ago

  • Subject changed from Large Projects to Large Projects: add limitations to the manual
  • Category changed from Project Loading/Saving to Documentation and Help

Wiki pages are essentially out of reach for most users. Please add these notes to the manual, and to the Help button.

#16 Updated by Larry Shaffer over 6 years ago

The original issue (as reported for Mac) has been fixed in master branch with commit d8e5c2f8

#17 Updated by Jürgen Fischer almost 6 years ago

  • Target version changed from Version 2.0.0 to Future Release - Lower Priority

#18 Updated by William Kyngesburye almost 6 years ago

It partially works now. It works when I build a debug QGIS, but does not work when building a release (MinSizeRel) QGIS. Tested with new v2.4, also release build for 2.2 (didn't try debug for 2.2).

When I added a QgsLogger line to the setrlimit() if block, above the QgsDebugMsg line, to check if the rlimit was changed, it then worked in a release build. I wonder then, since QgsDebugMsg() is a conditional define that does nothing in a release build, then the whole if block gets optimized to nothing since the if block is essentially empty. When I added QgsLogger(), that's a real function, so the if block is then not empty, and it then works. Maybe the setrlimit() part of the if test needs to be put in a variable and the variable tested, so it doesn't get lost in optimization?

A side note: in searching for an example of how to get non-debug messages into the console log so I can check the rlimits in a release build, I noticed that non-debug messages are not getting to console any more. The usual QgsMessageLog::logMessage() that I see in gqisapp.cpp, that showed up in console logs in the past do not now. And I couldn't find any QGIS log file, it doesn't look like there is a default, but is only set from the environment in QGIS_LOG_FILE, which can't happen in a normal OS X app startup. I had to use QgsLogger::warning() to get anything to show in the console, and that's only when running from a Terminal, directly not with the open command.

#19 Updated by William Kyngesburye almost 6 years ago

It wasn't debug vs. release build that was the problem. It's starting from Terminal vs. starting as a normal application. I started the debug build from a terminal to see the debug output for the rlimit settings. Then I started my hacked release build to also see the console output. But when starting either debug or hacked release build as a normal app, the rlimits are not changed and I can't load more than 84 shapefiles.

So, this bug is not fixed yet.

#20 Updated by William Kyngesburye almost 6 years ago

And same with my old v2.2 - opens more than 84 shapefiles when started from Terminal.

#21 Updated by Larry Shaffer over 5 years ago

William Kyngesburye wrote:

... I noticed that non-debug messages are not getting to console any more. ... but is only set from the environment in QGIS_LOG_FILE, which can't happen in a normal OS X app startup. ...

I think that has been the case since Lion, but definitely for Mt. Lion and Mavericks. Maybe something to do with sandboxing changes.

I use the following in the .app's Info.plist for my local dev builds:

<key>LSEnvironment</key>
<dict>
  ...
  <key>QGIS_LOG_FILE</key>
  <string>/Users/<my_user>/Library/Logs/QGIS/qgis-dev.log</string>
  ...
</dict>

But this may not be reasonable for the stable version, because there is no shell-type expansion for any of those LSEnvironment values. I suppose you could set that to a log file location writable by all users, but would be best in ~/Library/Logs/QGIS/.

Retesting the file limit issue now.

#22 Updated by Larry Shaffer over 5 years ago

William Kyngesburye wrote:

It partially works now. It works when I build a debug QGIS, but does not work when building a release (MinSizeRel) QGIS. Tested with new v2.4, also release build for 2.2 (didn't try debug for 2.2).

I can not reproduce the limit issue here. Just downloaded/installed your QGIS 2.4, launched it by double-click and ran the following in PyQGIS console:

for i in range(0,200):
  vlayer = QgsVectorLayer("/path/to/shapefile.shp", "layer {0}".format(i), "ogr")
  QgsMapLayerRegistry.instance().addMapLayer(vlayer)

For which, Activity Monitor shows 600 files loaded. I also saved and closed QGIS. Upon relaunching QGIS, the project was successfully auto-loaded (I have the option set).

I have also tested all my various dev builds here, via double-click. The qgis.log I have set shows the following:

Mac RLIMIT_NOFILE Soft/Hard NEW: 10240 / 10240
Mac RLIMIT_NOFILE Soft/Hard ORIG: 256 / 9223372036854775807

Will have to wait until next week to get access to the 10.7 nightly build VM to test it.

Is there maybe something funky with your Mac's setup? Have you tested on a different Mac as well?

#23 Updated by Larry Shaffer over 5 years ago

William, I tested with 150 separate shapefiles, via your double-clicked 2.4 release install, to make sure repetitively loading the same shapefile wasn't masking the issue. Same results, everything loads fine, with Activity Monitor reporting 450 open shapefile components.

#24 Updated by William Kyngesburye over 5 years ago

Grr, something off with my Mac. Compiled on this Mac, OS X 10.8, with 10.7 SDK. Won't load > 84 shapefiles. Same problem in another login. Another Mac, OS X 10.7, it works. Yet another Mac 10.8, works.

#25 Updated by Médéric RIBREUX over 4 years ago

  • Status changed from Open to Feedback

Hello, bug triage...

I am able to open more than 250 Shapefile layers on QGIS 2.13 under Debian. Do you still have the same problem on OS X ?

If it is not the case, we could close this bug.

#26 Updated by William Kyngesburye over 4 years ago

I upgraded the system on the problem Mac recently. I can now open more than 84 shapefiles. I never figured what in the old system (and just that Mac) was overriding the limits.

#27 Updated by Médéric RIBREUX over 4 years ago

  • Status changed from Feedback to Closed

Okay, so I am closing this bug...

Also available in: Atom PDF