Bug report #18740

is a WFS layer is large enough then identifying a feature in QGIS with the "auto open form" option active leads to memory leak/crashes

Added by Giovanni Manghi about 2 years ago. Updated about 2 years ago.

Status:Closed
Priority:High
Assignee:Even Rouault
Category:Web Services clients/WFS
Affected QGIS version:2.18.19 Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:No Resolution:fixed/implemented
Crashes QGIS or corrupts data:Yes Copied to github as #:26627

Description

This happens regardless the server used (I tested QGIS Server and Mapserver).

If you load one of those WFS layers the first identify works ok.
On the second identify it seems that loading the entire layer is triggered (by default the WFS client has the option "only request the features overlapping the view extent" enabled). The dialog that asks to "hide" or "abort" the operation shows, but the buttons are not responsive.
At this point QGIS starts eating memory (in my case 16gb od ram and almost 16gb of swap). A crash/freeze is usual, but if there are enough resources and a long time is waited then ultimately QGIS regains life and the identify dialog shows.

Note that this does not happen at all if the "auto open form" is not enabled, even with huge WFS layers.

You can test this (layer "ine_inspire_portugal")
http://www.naturalgis.pt/cgi-bin/opendata/qgis_mapserv.fcgi?service=WFS&request=GetCapabilities

or layer "portugal_addresses" in
http://www.naturalgis.pt/cgi-bin/opendata/mapserv?

this also affects QGIS 3.

Associated revisions

Revision c3c5c976
Added by Even Rouault about 2 years ago

Dummy commit to indicate that 0addae5b783e541953522eabe178426e820bf852 was about refs #18740

Revision 6cf1c502
Added by Even Rouault about 2 years ago

[WFS provider] Avoid request by feature id to cause a full layer download (fixes #18740)

Revision 2bbea0d4
Added by Even Rouault about 2 years ago

Merge pull request #7047 from rouault/fix_18740

[WFS provider] Avoid request by feature id to cause a full layer download (fixes #18740)

Revision 1121565d
Added by Even Rouault about 2 years ago

[WFS provider] Fix excessive memory consumption on big layers (refs #18740)

There was a pseudo memory leak in qgsgml.cpp
And the WFS downloader could also have to process big replies, causing a
lot of features to be instanciated at once.

Was seen on the 'portugal_addresses' layer of http://www.naturalgis.pt/cgi-bin/opendata/mapserv?

Backport of
0addae5b783e541953522eabe178426e820bf852
c3c5c9760a0b300548f9a669ff64f513988add7c
6cf1c502d1e057aa63ba7b48e2a532324a98194d

Revision b9be0a5b
Added by Even Rouault about 2 years ago

[WFS provider] Avoid request by feature id to cause a full layer download (fixes #18740)

Backport of 6cf1c502d1e057aa63ba7b48e2a532324a98194d

Revision e1708afc
Added by Even Rouault about 2 years ago

Merge pull request #7052 from rouault/fix_18740_qgis2_18

[WFS provider] 2.18 / Avoid request by feature id to cause a full layer download (fixes #18740)

History

#1 Updated by Jeremy Palmer about 2 years ago

  • Assignee set to Even Rouault

I also get this problem for large layers, but only for QGIS 3.0. When I load a layer with restrictToRequestBBOX='1' it loads into the map ok, but on the first feature info it causes the provider to download the whole layer. This didn't cause a cause for me, but did lock up the UI for about 20mins in downloading 5GB of data for a very large layer.

I was testing on QGIS 3.0.1 MacOSX.

Even are you able to help?

#2 Updated by Even Rouault about 2 years ago

I didn't do significant changes in the WFS provider in months, so if this is a regression, perhaps a "git bisect" exercise could help find the offending commit (probably in another component)

#3 Updated by Giovanni Manghi about 2 years ago

Even Rouault wrote:

I didn't do significant changes in the WFS provider in months, so if this is a regression, perhaps a "git bisect" exercise could help find the offending commit (probably in another component)

I'm not sure is a regression. I think is just that no one has possibly ever tested with a very large (millions of features) using the "auto open form" option enabled.

#4 Updated by Even Rouault about 2 years ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100

#5 Updated by Even Rouault about 2 years ago

  • Resolution set to fixed/implemented
  • Affected QGIS version changed from 3.1(master) to 2.18.19

Fixed in master 3.1dev and backported to 2.18 branch

Also available in: Atom PDF