Bug report #7822

Export to Spatialite is really slow, compared to commandline ogr2ogr

Added by Maxim Dubinin over 6 years ago. Updated almost 6 years ago.

Status:Closed
Priority:Severe/Regression
Assignee:-
Category:Vectors
Affected QGIS version:master Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:Yes Resolution:fixed/implemented
Crashes QGIS or corrupts data:No Copied to github as #:16707

Description

Testdata: http://gis-lab.info/forum/download/file.php?id=6540

Save as Spatialite takes more than an hour.

ogr2ogr -f SQLite testspatialite.sqlite geocoded.shp  -dsco SPATIALITE=YES

Produces result in 10 seconds.

Checked on 1fce8a3, GDAL 1.9.2

qgsvectorfilewriter.cpp.diff Magnifier - patch file for cpp (574 Bytes) Daniel Vaz, 2013-06-26 09:00 AM

qgsvectorfilewriter.h.diff Magnifier - patch file for header (143 Bytes) Daniel Vaz, 2013-06-26 09:00 AM

Associated revisions

Revision e491acb2
Added by Nathan Woodrow over 6 years ago

Merge pull request #680 from ddanielvaz/master

Fix #7822. Export to Spatialite is really slow, compared to commandline ogr2ogr

History

#1 Updated by Giovanni Manghi over 6 years ago

  • Status changed from Open to Feedback

confirmed. How was speed in 1.8?

#2 Updated by Maxim Dubinin over 6 years ago

it is equally slow in 1.8

#3 Updated by Larry Shaffer over 6 years ago

Confirmed here on Mac with latest master. I thought the app had crashed. It was not a large layer that was being saved: ~100 features.

#4 Updated by Daniel Vaz over 6 years ago

I read the OGR documentation and I suppose that it problem would be related with transaction system which isn't being used.

I create the patches for qgsvectorfilewriter class (see files attached). The patches were created from qgis-1.8.0.tar.bz2 downloaded from project site.

I ran QGis with Testdata and QGis takes minus than ten seconds to execute (Intel(R) Core(TM) i7-2620M CPU @ 2.70GHz - 8GB RAM)

#5 Updated by Daniel Vaz over 6 years ago

The issue #3002 reports the same problem described here.

#6 Updated by Giovanni Manghi over 6 years ago

  • Pull Request or Patch supplied changed from No to Yes

Daniel Vaz wrote:

I read the OGR documentation and I suppose that it problem would be related with transaction system which isn't being used.

I create the patches for qgsvectorfilewriter class (see files attached). The patches were created from qgis-1.8.0.tar.bz2 downloaded from project site.

I ran QGis with Testdata and QGis takes minus than ten seconds to execute (Intel(R) Core(TM) i7-2620M CPU @ 2.70GHz - 8GB RAM)

Hi!
would you mind make a pull request on github with your patch? otherwise the devs are likely not to see this patch, review it and eventually commit it. Thanks!

#7 Updated by Daniel Vaz over 6 years ago

Hi, I will do at home as soon as possible. Thanks for the answer and the directions. Best regards.

#8 Updated by Nathan Woodrow over 6 years ago

  • Status changed from Feedback to Closed

#9 Updated by Maxim Dubinin almost 6 years ago

  • Status changed from Closed to Reopened

The bug resurfaced and export in the latest nighly build is very slow. Reported both on Windows 7 and Ubuntu.

#10 Updated by Giovanni Manghi almost 6 years ago

  • Priority changed from Normal to Severe/Regression
  • Target version set to Future Release - High Priority

#11 Updated by Alexander Bruy almost 6 years ago

  • Status changed from Reopened to Feedback

Works like a charm in latest master (021441b361). Export of the attached file took about several seconds. Tested under Linux and Windows. I also tested larger file (~30 Mb point shapefile) and it also exported very fast.

#12 Updated by Alexander Bruy almost 6 years ago

Still cannot confirm. Export takes several seconds.

#13 Updated by Giovanni Manghi almost 6 years ago

  • Status changed from Feedback to Open

Alexander Bruy wrote:

Still cannot confirm. Export takes several seconds.

Hi Alex,

I can confirm the issue at some extent, the cause seems the same (already fixed by you) that slowed down a lot the process of placing off-line a postgis table with the off-line editing plugin. I tested on both Linux and Windows with qgis master.

It takes many minutes to export as SL the provided (~40000) point layer. The curios thing is that now it takes that long also from the CLI with ogr2ogr, so may be the issue is not QGIS after all. Maybe it is now necessary to use different parameters in og2ogr when converting?

#14 Updated by Giovanni Manghi almost 6 years ago

  • Status changed from Open to Closed
  • Resolution set to fixed/implemented

Please reopen if I'm wrong:

right now (on Ubuntu, GDAL/OGR 1.10.0) the operation is slow also from the command line.

For what I can see here

#8340

and here

http://trac.osgeo.org/gdal/ticket/5270

it seems that the upstream fix is available in gdal/ogr 1.10.2

Also available in: Atom PDF