Bug report #9526
File based layer edits and posts taking extremely long time
| Status: | Closed | ||
|---|---|---|---|
| Priority: | Severe/Regression | ||
| Assignee: | - | ||
| Category: | Vectors | ||
| Affected QGIS version: | master | Regression?: | No |
| Operating System: | Windows | Easy fix?: | No |
| Pull Request or Patch supplied: | No | Resolution: | worksforme |
| Crashes QGIS or corrupts data: | No | Copied to github as #: | 18116 |
Description
Since I updated to 2.1.0 dev (and also Windows 8.1) from 2.0 and Windows 8, I have noticed the layers I am working on are very slow to either save or go into edit mode or out of edit mode. These are file based layers and usually shp format.
Something that only took literally seconds to happen in 2.0 and earlier developmental editions I have run on my computer will now take at least 10 minutes to occur, while Windows says QGis is "not responding" and its Task Manager shows QGIS is using 51% of the CPU.
During this time QGIS is allocating memory because its total memory is climbing up, so we know something is happening. It goes up in small increments, maybe 4K - 10K. Sometimes more than this. But is extremely slow to happen. Almost like it had to count every single point in this layer (these layers have been points, paths and polygons, so they are all different types of layers) and allocate memory just for that point, then carry on with the next one etc. So far it has been going half an hour and its memory total has increased by 50 MB. I have no inclination to wait and see where it gets to or how long it takes. I'll come back in the morning and see what the final result was and add it as a comment to this. I have attached the layer in question in a Zip file. It is about 19KB. Why does so much memory have to be allocated to edit a shapefile of such a small size.
I don't see this happening with any other kind of file operations on Windows 8.1 so it must presumably be down to QGIS. I can get around the problem by creating a new shapefile and transferring everything into it from the old one. This works for a few days and then I get the same thing popping up again.
History
#1
Updated by Giovanni Manghi about 10 years ago
- Category set to Vectors
- Status changed from Open to Feedback
- Priority changed from Normal to Severe/Regression
- Target version set to Version 2.2
Can you confirm that on the same machine, the same files and operations are ok with qgis 2.0.1?
tagging as blocker awaiting for feedback.
#2
Updated by Patrick Dunford about 10 years ago
- Target version deleted (
Version 2.2)
OK. So this is what happened. It took over an hour to put the layer into edit mode. And QGIS increased its memory usage by over 100 MB, in order to edit a single layer that occupies around 19 KB of disk space.
Now I don't know why it needs so much memory just to edit a single layer, obviously the delay is down to the extremely slow rate of allocating this much memory for this task.
I needed to transfer a couple of features from this layer to another layer so after cutting them I put the second layer into edit mode. Then the same snail's pace performance is repeated to put the second, equally small layer, into edit mode. Still more many MBs of memory have to be allocated to put this layer into edit mode woth the resulting huge delay.
#3
Updated by Patrick Dunford about 10 years ago
- Target version set to Version 2.2
- File CWMSLFormerSiding.zip added
Reporting on the paste to the second layer, putting the second layer into edit mode required another 30 MB of memory. On the other hand, after pasting the paths into the second layer, taking it out of edit mode took mere seconds to achieve the posts to the layer. Likewise with the first layer, almost instantaneous save and reversion to read only. I have attached the second layer with this update.
After completing both saves and saving the project the memory used by Qgis had not reduced and it is now 144 MB more than before the two layers were edited. Must some big memory leaks going on there?
Will test with the 2.0 revision tonight. This appears to be the production 2.0.1 release -
QGIS version 2.0.1-Dufour QGIS code revision d94c044
Compiled against Qt 4.8.5 Running against Qt 4.8.5
Compiled against GDAL/OGR 1.10.0 Running against GDAL/OGR 1.10.1
Compiled against GEOS 3.4.2-CAPI-1.8.2 Running against GEOS 3.4.2-CAPI-1.8.2 r3921
PostgreSQL Client Version 9.2.4 SpatiaLite Version 4.1.1
QWT Version 5.2.3 PROJ.4 Version 480
QScintilla2 Version 2.7.2
Whereas the 2.1 release I have reports
QGIS version 2.1.0-Master QGIS code revision e3aef43
Compiled against Qt 4.8.5 Running against Qt 4.8.5
Compiled against GDAL/OGR 1.10.1 Running against GDAL/OGR 1.10.1
Compiled against GEOS 3.4.2-CAPI-1.8.2 Running against GEOS 3.4.2-CAPI-1.8.2 r3921
PostgreSQL Client Version 9.2.4 SpatiaLite Version 4.1.1
QWT Version 5.2.3 PROJ.4 Version 480
QScintilla2 Version 2.7.2 This copy of QGIS writes debugging output.
#4
Updated by Richard Duivenvoorde about 10 years ago
Hi Patrick,
can you confirm that this also is the case when you disable ALL your plugins?
There have been some issues with certain python plugins some time ago, see for example: #9229
You can start QGIS from command line with --noplugins option (or just move your .qgis2 folder for this test).
Hoping that you have one of the older versions of one of the plugins that had this issue, OR maybe you found a new plugin with the same issue.
#5
Updated by Patrick Dunford about 10 years ago
OK so using 2,01 with the same project and layer files, editing a layer is instantaneous. So are the saves.
I'll give it a few more days of work to see if I get the same problems developing again.
#6
Updated by Giovanni Manghi about 10 years ago
Patrick Dunford wrote:
OK so using 2,01 with the same project and layer files, editing a layer is instantaneous. So are the saves.
I'll give it a few more days of work to see if I get the same problems developing again.
I cannot confirm this issue on the latest master (with the provided data and other datasets). Maybe one of the fixes committed in the last days has also fixed this.
#7
Updated by Patrick Dunford about 10 years ago
Testing with the --noplugins option but haven't done enough on it yet to see if fixed. Haven't tried latest master (could be a plugin yet)
#8
Updated by Giovanni Manghi about 10 years ago
Patrick Dunford wrote:
Testing with the --noplugins option but haven't done enough on it yet to see if fixed. Haven't tried latest master (could be a plugin yet)
have you tested the latest master release?
#9
Updated by Patrick Dunford about 10 years ago
At the moment I'm running 141 noplugins. Might try it tonight with plugins enabled.
#10
Updated by Giovanni Manghi about 10 years ago
Patrick Dunford wrote:
At the moment I'm running 141 noplugins. Might try it tonight with plugins enabled.
the important is to try also the latest master release that you can install with the osgeo4w installer.
#11
Updated by Patrick Dunford about 10 years ago
141 is the latest master.
#12
Updated by Giovanni Manghi about 10 years ago
Patrick Dunford wrote:
141 is the latest master.
ok, then it is solved for you or not? cheers!
#13
Updated by Patrick Dunford about 10 years ago
- Status changed from Feedback to Closed
I'm happy with it for now. I've been using it for a few days and haven't seen a repeat of the problem.
#14
Updated by Giovanni Manghi about 10 years ago
- Resolution set to fixed/implemented
#15
Updated by Patrick Dunford about 10 years ago
- Status changed from Closed to Reopened
I think it's still happening, just not as extreme (yet). Taking 4 minutes to put a layer into edit mode and using 12 MB of memory (for a 32KB shapefile layer) could still be an issue.
#16
Updated by Patrick Dunford about 10 years ago
OK so I finished editing the two layers (25KB and 32KB, shapefile, file based layer) then hit Save All and waited for 15 minutes and qgis-dev.exe increased its memory usage by 40 MB.
If it's plugins, what is the difference between "installed" and "activated"? What does "--noplugins" do in terms of "installed" and "activated"?
#17
Updated by Giovanni Manghi about 10 years ago
- Resolution deleted (
fixed/implemented) - Status changed from Reopened to Feedback
Patrick Dunford wrote:
OK so I finished editing the two layers (25KB and 32KB, shapefile, file based layer) then hit Save All and waited for 15 minutes and qgis-dev.exe increased its memory usage by 40 MB.
can you please attach a sample project and data? A list of the 3rd party plugins you have installed would be useful too.
If it's plugins, what is the difference between "installed" and "activated"? What does "--noplugins" do in terms of "installed" and "activated"?
once installed a plugin can be active (loaded on qgis startup) or not. The --noplugins switch allow you to run qgis without loading any plugin on startup (even if you have them active in the plugin manager).
#18
Updated by Giovanni Manghi about 10 years ago
- Resolution set to worksforme
- Status changed from Feedback to Closed
Giovanni Manghi wrote:
Patrick Dunford wrote:
OK so I finished editing the two layers (25KB and 32KB, shapefile, file based layer) then hit Save All and waited for 15 minutes and qgis-dev.exe increased its memory usage by 40 MB.
can you please attach a sample project and data? A list of the 3rd party plugins you have installed would be useful too.
If it's plugins, what is the difference between "installed" and "activated"? What does "--noplugins" do in terms of "installed" and "activated"?
once installed a plugin can be active (loaded on qgis startup) or not. The --noplugins switch allow you to run qgis without loading any plugin on startup (even if you have them active in the plugin manager).
closing for lack of feedback, reopen if necessary.
#19
Updated by Patrick Dunford over 9 years ago
The problem plugin seems to be the Rectangles Ovals plugin. I just updated to 2.4.0 and have just reinstalled the version of the layer that was available from the plugins repository. Up until now I have had two shortcuts, one for starting Qgis without plugins and one with plugins, as this is the only plugin I use most of the time.
#20
Updated by Patrick Dunford over 8 years ago
- Status changed from Closed to Reopened
- Target version deleted (
Version 2.2)
This is still an occurrence in 2.10
Is there any progress on this issue?
#21
Updated by Jürgen Fischer over 8 years ago
- Status changed from Reopened to Feedback
Patrick Dunford wrote:
This is still an occurrence in 2.10
Is there any progress on this issue?
Check 2.12 or 2.8 (LTR). 2.10 is not maintained. But it this is actually a problem with the "Rectangle Ovals" plugin you should probably file a ticket there.
#22
Updated by Giovanni Manghi over 8 years ago
- Status changed from Feedback to Closed
Why reopening if is a 3rd party plugin issue? Reopen it if there is any evidence that is a QGIS core issue, otherwise report the issue to the plugin author. Thanks!