Bug report #18420

Excessive, erratic memory usage

Added by Johannes Kroeger about 2 years ago. Updated over 1 year ago.

Status:Closed
Priority:High
Assignee:Nyall Dawson
Category:Unknown
Affected QGIS version:3.0.0 Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:Yes Resolution:fixed/implemented
Crashes QGIS or corrupts data:Yes Copied to github as #:26309

Description

QGIS uses excessive amounts of memory, seemingly leaking some until it takes down my whole system. This seems to depend on adding and removing layers. In my case I noticed it with GeoJSON and Shapefile formats, I have no idea if it affects others).

Dataset used: https://transfer.sh/KghNi/b%C3%A4ume.7z (CC0 from https://opendata.bonn.de/dataset/baumstandorte)
There is a geojson and a shp inside.

QGIS takes about 320M of RAM with a clean (no project loaded, but some plugins) start for me.

1670M after loading the geojson after a clean start
2500M after adding the shp to that
3300M after removing the shp layer
4160M after adding the shp again
5000M after removing the shp layer again
(5000M after removing the geojson layer as well but that seems to get at least partially reclaimed as adding both the shp and the geojson again only made it grow to 5075M...)
 365M after loading the shp after a clean start
1720M after adding the geojson to that
1700M after removing the geojson layer
1760M after adding the geojson again

So this seemed fine (although 1.7G for a 25M GeoJSON is crazy).
 365M after loading the shp after a clean start
1720M after adding the geojson to that
2530M after removing the shp layer
3350M after adding the shp again

I did not bother going further, it would have taken more memory again.

So, loading and unloading the shp file while the geojson is loaded makes QGIS gradually eat all RAM and crash the system.

QGIS version 3.0.0-Girona
QGIS code branch Release 3.0
On Archlinux

Associated revisions

Revision f3b5838f
Added by Nyall Dawson about 2 years ago

[ogr] Fix ref/unref mismatch when loading OGR layers

Causes an extra connection reference which is never removed,
blocking ogr dataset closing.

Fixes #18420, probably others

Revision dff31a4a
Added by Nyall Dawson about 2 years ago

[ogr] Fix ref/unref mismatch when loading OGR layers

Causes an extra connection reference which is never removed,
blocking ogr dataset closing.

Fixes #18420, probably others

(cherry-picked from f3b5838)

History

#1 Updated by Nyall Dawson about 2 years ago

  • Status changed from Open to In Progress
  • Pull Request or Patch supplied changed from No to Yes

Can you test https://github.com/qgis/QGIS/pull/6593 and see if it helps?

#2 Updated by Johannes Kroeger about 2 years ago

I could not test on the same machine yet. On my other one (same 64 bit Archlinux but different GDAL) I could not replicate the high memory usage even in an older qgis (r44842.6ed078c889) so I am a bit confused. Maybe the GDAL version plays a role too? I will try to test it tomorrow.

#3 Updated by Nyall Dawson about 2 years ago

Gdal 2.3 will help a lot here (it's gdal which consumes all the memory on opening the geojson)

#4 Updated by Johannes Kroeger about 2 years ago

Yeah, that one had the latest GDAL from SVN I think.

On the machine where I noticed the problem:

QGIS 3.1.0-Master (193c554b4c) with the patch applied.
GDAL 2.2.3, released 2017/11/20

1660M added GeoJSON
2033M added shp
3000M removed shp
3800M added shp
4200M removed shp

It also does not free after unloading all layers or creating a new project.

#5 Updated by Nyall Dawson about 2 years ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Closed

#6 Updated by Johannes Kroeger about 2 years ago

  • Status changed from Closed to Reopened
  • Assignee set to Nyall Dawson

No luck :(
I built from Git r46587.10044fb1dd with the same GDAL 2.2.3, released 2017/11/20 and tried the files again:

1660 gj
2500 shp+
3300 shp-
3300 shp+
3300 shp-
4200 shp+
5000 shp-
5000 shp+
5000 shp-
5000 shp+
5000 shp-
5000 gj-
5000 new project
5000 shp+
5100 gj+
5100 shp-
5100 shp+
5100 open random other project

#7 Updated by Giovanni Manghi over 1 year ago

  • Status changed from Reopened to Feedback

Please try on QGIS 3.4.1, if the issue is still valid change the affected version, thanks.

#8 Updated by Johannes Kroeger over 1 year ago

  • Status changed from Feedback to Open

Tested in 3.2.3-Bonn (Compiled against GDAL/OGR 2.3.1, Running against GDAL/OGR 2.3.2).

I could not reproduce the excessive memory usage. So this issue can be closed.

#9 Updated by Giovanni Manghi over 1 year ago

  • Status changed from Open to Closed
  • Resolution set to fixed/implemented

You should test 3.4.1 as 3.2 is not supported anymore, so reopen if necessary.

Also available in: Atom PDF