Bug report #9348

Selection of thousands of points very slow in QGIS2

Added by Rhenriques Henriques over 6 years ago. Updated over 6 years ago.

Status:Closed
Priority:Severe/Regression
Assignee:-
Category:Vectors
Affected QGIS version:master Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:No Resolution:invalid
Crashes QGIS or corrupts data:No Copied to github as #:17950

Description

The selection of thousands of points from a 5 million point shapefile takes ages to process and eventually crashes QGIS 2 and QGIS 2 NB. I had to force quit after 2 hours. In QGIS 1.8 the same selection occurs in seconds. Glad I have a 1.8 copy still installed.
Used both Freehand and Select by Radius modes.
Cheers

QGIS 2 or NB under MacOSX Mavericks

RandomPoints.JPG (176 KB) Alvaro Huarte, 2014-02-01 01:34 AM

Screen_Shot_2014-02-01_at_2.40.33_PM.png (119 KB) Rhenriques Henriques, 2014-02-01 06:41 AM

History

#1 Updated by Giovanni Manghi over 6 years ago

  • Target version set to Future Release - High Priority
  • Category set to Vectors
  • Priority changed from High to Severe/Regression
  • Affected QGIS version changed from 2.0.1 to master

Then tagging as a regression.

#2 Updated by Rhenriques Henriques over 6 years ago

  • Target version deleted (Future Release - High Priority)

The same applies to the editing of values in the attribute table.

#3 Updated by Andreas Neumann over 6 years ago

Does the point shapefile have a spatial index. Can you share the data so we can test the issue?

#4 Updated by Rhenriques Henriques over 6 years ago

Hi Andreas

Thank you so much for the reply. The problem happens with or without spatial index. Tested both situations.
Unfortunately, the layer I'm using is restricted. But you can test the same problem by building a random point layer, with 2 or 3 columns of attributes, with more than 300000 points. I've made 2 videos for you to see the difference in versions 1.8 and 2.0. In the smaller set video, QGIS 1.8 selects points almost instantly while QGIS 2.0 takes more than 40 seconds. In the larger set selection, QGIS 1.8 takes 6 or 7 seconds to perform the selection while I had to force quit QGIS 2.0 after 8 minutes of processing without any selection. This layer has, sometimes, several dozens of points in the same positions.
Access the videos from here (you can fast forward the second one to near end because it takes 8 minutes to that point!!):

http://www.dct.uminho.pt/smallsetselection.mp4

http://www.dct.uminho.pt/largesetselection.mp4

This problem makes me use QGIS 1.8 often just for these kind of tasks. The same happens while selecting from attributes table with expressions or to change several thousands of attribute cells via calculator.

Cheers

#5 Updated by Alvaro Huarte over 6 years ago

I do not test this case use, but this patch

https://github.com/qgis/QGIS/pull/1114

can fix this issue

Alvaro

#6 Updated by Alvaro Huarte over 6 years ago

  • Status changed from Open to Feedback

#7 Updated by Rhenriques Henriques over 6 years ago

Hi Alvaro

Thank you so much for taking ongoing this annoying issue. I've made the modifications to the xxx.h files in the MacOSX latest NB, suggested by you from here "https://github.com/qgis/QGIS/pull/1116/files" but no luck. The problem remains the same. I'll wait for the next NB's, with the committed code, to check if the problem get solved because there's always a chance that I did something wrong.
Cheers

#8 Updated by Alvaro Huarte over 6 years ago

Rhenriques Henriques wrote:

Hi Alvaro

Thank you so much for taking ongoing this annoying issue. I've made the modifications to the xxx.h files in the MacOSX latest NB, suggested by you from here "https://github.com/qgis/QGIS/pull/1116/files" but no luck. The problem remains the same. I'll wait for the next NB's, with the committed code, to check if the problem get solved because there's always a chance that I did something wrong.
Cheers

Hi, can you attach the data to test ?
I can try solve it

Alvaro

#9 Updated by Rhenriques Henriques over 6 years ago

Hi Alvaro

The data that I'm using is restricted. However I've generated a Random 400000 Points layer for the Iberian Peninsula Area. You can test the issue with this layer. Even without any attribute, if you try to select, let's say, 1/6 of the points, with QGIS 1.8 it will get selected almost instantaneously. With QGIS 2.x it takes ages or crash.
While testing this issue, I've found another bug into the NB. Define a Shapefile, choose Vector -> Research Tools -> Random Points ; Choose 400000 point generation and flag "Add result to Canvas" - QGIS will crash.

Cheers

download from: https://www.dropbox.com/s/hputdz2b27sx7cl/TestPoints.zip

#10 Updated by Alvaro Huarte over 6 years ago

Hi Rhenriques, I get "normal" results, I have released a video (~18mb):

http://www.filedropper.com/testissue9348

could be some plugin? I have none installed
Alvaro

#11 Updated by Rhenriques Henriques over 6 years ago

Hi Alvaro

That example is before or after the code modification? Your selection is indeed quicker. Maybe it's an OS related issue or, as you say, a plugin. I need to test that further. Maybe any option flagged somewhere, I don't know. But, even so, as you can see, freehand selection lags a lot. Try the same in QGIS1.8 and you will see that it's a lot faster. Did you check the "Random Points" bug?
I'll make some plugin disabling and let you know the results.
Cheers

#12 Updated by Matthias Kuhn over 6 years ago

Try running with --noplugins to check if it is a plugin issue

#13 Updated by Giovanni Manghi over 6 years ago

Rhenriques Henriques wrote:

Hi Alvaro

That example is before or after the code modification? Your selection is indeed quicker. Maybe it's an OS related issue or, as you say, a plugin. I need to test that further. Maybe any option flagged somewhere, I don't know. But, even so, as you can see, freehand selection lags a lot. Try the same in QGIS1.8 and you will see that it's a lot faster. Did you check the "Random Points" bug?
I'll make some plugin disabling and let you know the results.
Cheers

Hi Rui,

I also just tested qgis master on Ubuntu and selecting even all your 400k points from both the table or the map it is very fast.

#14 Updated by Rhenriques Henriques over 6 years ago

Hi again Alvaro

Definitely a plug-in. I've just disabled all and selection is normal. I'm going to isolate the culprit. I hope that it's not an important one. QGIS is a lot better due to plug-ins.
Cheers

#15 Updated by Giovanni Manghi over 6 years ago

  • Status changed from Feedback to Closed
  • Resolution set to invalid

#16 Updated by Alvaro Huarte over 6 years ago

Rhenriques Henriques wrote:

Hi Alvaro

That example is before or after the code modification? Your selection is indeed quicker. Maybe it's an OS related issue or, as you say, a plugin. I need to test that further. Maybe any option flagged somewhere, I don't know. But, even so, as you can see, freehand selection lags a lot. Try the same in QGIS1.8 and you will see that it's a lot faster. Did you check the "Random Points" bug?
I'll make some plugin disabling and let you know the results.
Cheers

Hi, my apologies for not describing my results:
This example is after modification.
Freehand selection lags a lot, that makes sense, the clipping polygon has more points than a simple rectangle, It will study it if there is something wrong.
I dont check "Random Points", I will test soon.

Best Regards
Alvaro

#17 Updated by Rhenriques Henriques over 6 years ago

Hi again

I've isolated the culprit. This slowness is due to the experimental "Geometry copier" plug-in by Nikulin Evgeniy. Try to check if, in similar problems, people have this plug-in installed. Warn the author about this "painful" issue.
Cheers

#18 Updated by Alvaro Huarte over 6 years ago

I think that the problem is:
https://github.com/yellow-sky/GeometryCopier/blob/master/src/geometry_copier.py#L109

It loads all selected geometries in memory reading from disk one to one.

I think the solution is use...

QgsVectorLayer::selectedFeaturesIds() -> features = layer.selectedFeatures()

Alvaro

#19 Updated by Nathan Woodrow over 6 years ago

Or better yet just use `selectedFeaturesCount()`.

#21 Updated by Alvaro Huarte over 6 years ago

Nathan Woodrow wrote:

Or better yet just use `selectedFeaturesCount()`.

You are right, thanks Nathan

#22 Updated by Alvaro Huarte over 6 years ago

Hi, I have a question, it is possible use in QGIS/Qt something similar to IEnumerable/IEnumerator interfaces of c# ?

Thanks in advance

#23 Updated by Nathan Woodrow over 6 years ago

In C++ or Python.

What are you trying to do?

#24 Updated by Alvaro Huarte over 6 years ago

In c# we can write a method as...

IEnumerable<QgsFeature> QgsVectorLayer::selectedFeaturesEnum() {
QgsFeatureRequest req;
if ( geometryType() == QGis::NoGeometry )
req.setFlags( QgsFeatureRequest::NoGeometry );

foreach ( QgsFeatureId fid, mSelectedFeatureIds )
{
QgsFeature feature;
features.push_back( feature );
getFeatures( req.setFilterFid( fid ) ).nextFeature( feature );
yield return feature;
}
yield break;
}

It is a iterator in C++, but the data are sequentially read

#25 Updated by Nathan Woodrow over 6 years ago

Like we have in Python. Well layer.getFeatures returns a lazy iterator that only evals when you call next. A bit like a generator in Python.

I'm not sure if C++ has it's own concept of this or you have to write you own.

#26 Updated by Alvaro Huarte over 6 years ago

Thanks Nathan, It was a curious.
Best Regards

#27 Updated by Alvaro Huarte over 6 years ago

Rhenriques Henriques wrote:

Hi again

I've isolated the culprit. This slowness is due to the experimental "Geometry copier" plug-in by Nikulin Evgeniy. Try to check if, in similar problems, people have this plug-in installed. Warn the author about this "painful" issue.
Cheers

Hi Rhenriques, anyone has reported this "painful" issue ?
Alvaro

#28 Updated by Rhenriques Henriques over 6 years ago

Hi Alvaro

So far, in my case, things are behaving normally, after uninstalling the "Geometry copier" plug-in. I'll wait for a plug-in upgrade. I don't know if there are further reports of this issue. I guess that it's not that common that people need to use databases with hundreds of thousands of points or a few millions. Otherwise, with this plug-in installed, they would have notice for sure ;-)

Cheers

#29 Updated by Alvaro Huarte over 6 years ago

Hi, I added a new issue to GeometryCopier plugin for notify it to author:
https://github.com/yellow-sky/GeometryCopier/issues/1

Best regards
Alvaro

#30 Updated by Alvaro Huarte over 6 years ago

Hi Rhenriques, I tested successly the "Random Points" plugin (doRandPoints.py).

I created a new point shapefile with 40000 points using as filter a very complex polygon layer and "Add result to Canvas" checked.
I work fine here.

You could add a new issue with the data to cause the bug
Best Regards

Alvaro

#31 Updated by Rhenriques Henriques over 6 years ago

Hi Alvaro

Here it crashes immediately (NB vs ddec663. Maybe it's a Mavericks related issue or another Plug-In related issue. I'll try to check later for this.

Cheers

#32 Updated by Evgeniy Nikulin over 6 years ago

GeometryCopier fixed

#33 Updated by Alvaro Huarte over 6 years ago

Evgeniy Nikulin wrote:

GeometryCopier fixed

Thanks Evgeniy !

#34 Updated by Rhenriques Henriques over 6 years ago

Working perfectly now! Thanks Evgeniy.
Cheers

#35 Updated by Evgeniy Nikulin over 6 years ago

Thanks you!

Also available in: Atom PDF