https://issues.qgis.org/https://issues.qgis.org/favicon.ico2011-06-27T08:18:38ZQGIS Issue TrackingQGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=250972011-06-27T08:18:38ZWilliam Kyngesburyekyngchaos@kyngchaos.com
<ul></ul><p>Same limit here, 84 layers. Using master build 6d788eb from June 4. And I'm creating a new project, not opening an old one.</p> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=251172011-06-30T15:17:42ZWilliam Kyngesburyekyngchaos@kyngchaos.com
<ul></ul><p>From Royce Cline: the limit of the number of open files per process on OS X is 256. I now remember this problem a while back, but figured it was solved somehow. But then, I don't add a lot of shapefiles to a QGIS project, until recently, usually it's postgis connections.</p>
<p>Royce's solution has been to set ulimit in a Terminal and start QGIS from there, but this is not something for the average user, and is generally a bad way (at least the wrong way) to start any application on OS X.</p>
<p>ulimit only affects processes started from a shel (Terminal). OS X apps get their limits from sysctl/launchctl. You can check it with:</p>
<pre>
launchctl limit
</pre>
<p>which for me on 10.6 shows 256 soft limit and unlimited hard limit. Huh, if a soft limit only gives the app a warning but lets it continue opening files, maybe there is a problem in how Qt handles it.</p>
<p>The sysctl limits look different:</p>
<pre>
kern.maxfiles: 12288
kern.maxfilesperproc: 10240
</pre> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=255562011-08-29T20:36:48ZPaolo Cavallinicavallini@faunalia.it
<ul><li><strong>Category</strong> set to <i>Project Loading/Saving</i></li><li><strong>Pull Request or Patch supplied</strong> set to <i>No</i></li></ul> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=257212011-09-24T00:30:15ZJürgen Fischerjef@norbit.de
<ul><li><strong>Priority</strong> changed from <i>6</i> to <i>Normal</i></li></ul> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=278562011-12-16T13:12:42ZGiovanni Manghigiovanni.manghi@gmail.com
<ul><li><strong>Target version</strong> set to <i>Version 1.7.4</i></li></ul> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=310132012-04-16T04:28:39ZPaolo Cavallinicavallini@faunalia.it
<ul><li><strong>Affected QGIS version</strong> set to <i>master</i></li><li><strong>Target version</strong> changed from <i>Version 1.7.4</i> to <i>Version 1.8.0</i></li><li><strong>Crashes QGIS or corrupts data</strong> set to <i>No</i></li></ul> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=329302012-06-21T05:48:41Zgcarrillo -
<ul></ul><p>Same issue here. In Linux the limit seems to be 1024 by default. I wonder how other software address this problem since it is platform-specific.</p> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=329312012-06-21T05:52:20ZGiovanni Manghigiovanni.manghi@gmail.com
<ul></ul><p>gcarrillo - wrote:</p>
<blockquote>
<p>I wonder how other software address this problem since it is platform-specific.</p>
</blockquote>
<p>As far as I have understand it is an operating system limitation/issue, not a QGIS one. Consider also that when adding a shapefile you are opening at least 3 files, not 1.</p> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=329462012-06-22T03:18:17Zgcarrillo -
<ul></ul><p>Giovanni Manghi wrote:</p>
<blockquote>
<p>gcarrillo - wrote:</p>
<blockquote>
<p>I wonder how other software address this problem since it is platform-specific.</p>
</blockquote>
<p>As far as I have understand it is an operating system limitation/issue, not a QGIS one. Consider also that when adding a shapefile you are opening at least 3 files, not 1.</p>
</blockquote>
<p>Giovanni, if you read what I wrote, you'll see that I'm aware it's not a QGIS problem but a platform (OS) one. My question goes in the direction of finding the ways other GIS software tackle this, so that QGIS can also implement their solutions, if there are any.</p> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=329642012-06-23T02:04:22ZGiovanni Manghigiovanni.manghi@gmail.com
<ul></ul><blockquote>
<p>My question goes in the direction of finding the ways other GIS software tackle this, so that QGIS can also implement their solutions, if there are any.</p>
</blockquote>
<p>I guess fist we have to know if other GIS packages overcome this limitation. They do? any open source?</p> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=346372012-09-04T10:01:30ZPaolo Cavallinicavallini@faunalia.it
<ul><li><strong>Target version</strong> changed from <i>Version 1.8.0</i> to <i>Version 2.0.0</i></li></ul> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=431012013-07-14T09:17:15ZDaniel Vazdanielvaz@gmail.com
<ul></ul><p>I think that a good solution would be to keep the user informed about OS limitations.</p>
<p>In the GRASS Wiki they report OS limitations to users (see Number of open files limitation): <a class="external" href="http://grasswiki.osgeo.org/wiki/Large_raster_data_processing">http://grasswiki.osgeo.org/wiki/Large_raster_data_processing</a></p>
<p>So let's keep the information accessible by users and create/update a wiki page about Know Issues.</p>
<p>Thanks in advance.</p> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=431032013-07-14T09:24:17ZDaniel Vazdanielvaz@gmail.com
<ul></ul><p>Closing related Bug <a class="issue tracker-1 status-5 priority-5 priority- closed" href="https://issues.qgis.org/issues/6404" title="Too many layers -> unknown exception (Closed)">#6404</a>.</p> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=433952013-07-20T06:08:28ZDaniel Vazdanielvaz@gmail.com
<ul></ul><p>I created a wikipage</p>
<p><a class="wiki-page" href="https://issues.qgis.org/projects/qgis/wiki/Known_Issues">Known Issues</a></p>
<p>I think there is no more reason to keep this issue open.</p>
<p>Please, if necessary reopen it.</p> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=434422013-07-21T21:41:10ZPaolo Cavallinicavallini@faunalia.it
<ul><li><strong>Subject</strong> changed from <i>Large Projects</i> to <i>Large Projects: add limitations to the manual</i></li><li><strong>Category</strong> changed from <i>Project Loading/Saving</i> to <i>Documentation and Help</i></li></ul><p>Wiki pages are essentially out of reach for most users. Please add these notes to the manual, and to the Help button.</p> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=461162013-11-03T09:59:37ZLarry Shaffer
<ul></ul><p>The original issue (as reported for Mac) has been fixed in master branch with commit <a class="changeset" href="https://issues.qgis.org/projects/qgis/repository/revisions/d8e5c2f8300cdbd31f71544a402f360cac12016d" title="Increase number of allowed open files on Mac - Increase from default 256 to 4096-to-8192 range, ...">d8e5c2f8</a></p> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=535682014-06-28T05:40:43ZJürgen Fischerjef@norbit.de
<ul><li><strong>Target version</strong> changed from <i>Version 2.0.0</i> to <i>Future Release - Lower Priority</i></li></ul> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=539012014-06-28T08:24:50ZWilliam Kyngesburyekyngchaos@kyngchaos.com
<ul></ul><p>It partially works now. It works when I build a debug QGIS, but does not work when building a release (MinSizeRel) QGIS. Tested with new v2.4, also release build for 2.2 (didn't try debug for 2.2).</p>
<p>When I added a QgsLogger line to the setrlimit() if block, above the QgsDebugMsg line, to check if the rlimit was changed, it then worked in a release build. I wonder then, since QgsDebugMsg() is a conditional define that does nothing in a release build, then the whole if block gets optimized to nothing since the if block is essentially empty. When I added QgsLogger(), that's a real function, so the if block is then not empty, and it then works. Maybe the setrlimit() part of the if test needs to be put in a variable and the variable tested, so it doesn't get lost in optimization?</p>
<p>A side note: in searching for an example of how to get non-debug messages into the console log so I can check the rlimits in a release build, I noticed that non-debug messages are not getting to console any more. The usual QgsMessageLog::logMessage() that I see in gqisapp.cpp, that showed up in console logs in the past do not now. And I couldn't find any QGIS log file, it doesn't look like there is a default, but is only set from the environment in QGIS_LOG_FILE, which can't happen in a normal OS X app startup. I had to use QgsLogger::warning() to get anything to show in the console, and that's only when running from a Terminal, directly not with the open command.</p> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=539052014-06-28T09:49:14ZWilliam Kyngesburyekyngchaos@kyngchaos.com
<ul></ul><p>It wasn't debug vs. release build that was the problem. It's starting from Terminal vs. starting as a normal application. I started the debug build from a terminal to see the debug output for the rlimit settings. Then I started my hacked release build to also see the console output. But when starting either debug or hacked release build as a normal app, the rlimits are not changed and I can't load more than 84 shapefiles.</p>
<p>So, this bug is not fixed yet.</p> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=539062014-06-28T09:52:36ZWilliam Kyngesburyekyngchaos@kyngchaos.com
<ul></ul><p>And same with my old v2.2 - opens more than 84 shapefiles when started from Terminal.</p> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=542132014-07-04T09:59:01ZLarry Shaffer
<ul></ul><p>William Kyngesburye wrote:</p>
<blockquote>
<p>... I noticed that non-debug messages are not getting to console any more. ... but is only set from the environment in QGIS_LOG_FILE, which can't happen in a normal OS X app startup. ...</p>
</blockquote>
<p>I think that has been the case since Lion, but definitely for Mt. Lion and Mavericks. Maybe something to do with sandboxing changes.</p>
<p>I use the following in the .app's <code>Info.plist</code> for my local dev builds:</p>
<pre>
<key>LSEnvironment</key>
<dict>
...
<key>QGIS_LOG_FILE</key>
<string>/Users/<my_user>/Library/Logs/QGIS/qgis-dev.log</string>
...
</dict>
</pre>
<p>But this may not be reasonable for the stable version, because there is no shell-type expansion for any of those <code>LSEnvironment</code> values. I suppose you could set that to a log file location writable by all users, but would be best in <code>~/Library/Logs/QGIS/</code>.</p>
<p>Retesting the file limit issue now.</p> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=542142014-07-04T10:20:57ZLarry Shaffer
<ul></ul><p>William Kyngesburye wrote:</p>
<blockquote>
<p>It partially works now. It works when I build a debug QGIS, but does not work when building a release (MinSizeRel) QGIS. Tested with new v2.4, also release build for 2.2 (didn't try debug for 2.2).</p>
</blockquote>
<p>I can not reproduce the limit issue here. Just downloaded/installed your QGIS 2.4, launched it by double-click and ran the following in PyQGIS console:</p>
<pre>
for i in range(0,200):
vlayer = QgsVectorLayer("/path/to/shapefile.shp", "layer {0}".format(i), "ogr")
QgsMapLayerRegistry.instance().addMapLayer(vlayer)
</pre>
<p>For which, Activity Monitor shows 600 files loaded. I also saved and closed QGIS. Upon relaunching QGIS, the project was successfully auto-loaded (I have the option set).</p>
<p>I have also tested all my various dev builds here, via double-click. The qgis.log I have set shows the following:</p>
<pre>
Mac RLIMIT_NOFILE Soft/Hard NEW: 10240 / 10240
Mac RLIMIT_NOFILE Soft/Hard ORIG: 256 / 9223372036854775807
</pre>
<p>Will have to wait until next week to get access to the 10.7 nightly build VM to test it.</p>
<p>Is there maybe something funky with your Mac's setup? Have you tested on a different Mac as well?</p> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=542162014-07-04T13:06:10ZLarry Shaffer
<ul></ul><p>William, I tested with 150 separate shapefiles, via your double-clicked 2.4 release install, to make sure repetitively loading the <em>same</em> shapefile wasn't masking the issue. Same results, everything loads fine, with Activity Monitor reporting 450 open shapefile components.</p> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=542222014-07-04T15:49:48ZWilliam Kyngesburyekyngchaos@kyngchaos.com
<ul></ul><p>Grr, something off with my Mac. Compiled on this Mac, OS X 10.8, with 10.7 SDK. Won't load > 84 shapefiles. Same problem in another login. Another Mac, OS X 10.7, it works. Yet another Mac 10.8, works.</p> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=660282015-12-03T05:25:45ZMédéric RIBREUX
<ul><li><strong>Status</strong> changed from <i>Open</i> to <i>Feedback</i></li></ul><p>Hello, bug triage...</p>
<p>I am able to open more than 250 Shapefile layers on QGIS 2.13 under Debian. Do you still have the same problem on OS X ?</p>
<p>If it is not the case, we could close this bug.</p> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=660342015-12-03T06:18:23ZWilliam Kyngesburyekyngchaos@kyngchaos.com
<ul></ul><p>I upgraded the system on the problem Mac recently. I can now open more than 84 shapefiles. I never figured what in the old system (and just that Mac) was overriding the limits.</p> QGIS Application - Bug report #4013: Large Projects: add limitations to the manualhttps://issues.qgis.org/issues/4013?journal_id=660542015-12-03T10:05:41ZMédéric RIBREUX
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Closed</i></li></ul><p>Okay, so I am closing this bug...</p>