Bug report #8101
"Save as" on a large PostGIS point layer doesn't work
|Affected QGIS version:||master||Regression?:||No|
|Operating System:||Easy fix?:||No|
|Pull Request or Patch supplied:||No||Resolution:||end of life|
|Crashes QGIS or corrupts data:||No||Copied to github as #:||16936|
On 1.8.0 and 1.9 192e130, I load a PostGIS layer with 2,889,598 points. These represent a few years' worth of a type of event along roadways in a state of the United States. I right-click on the layer, select Save As, select a path and filename, check Skip attribute creation (the layer has a lot of attributes; don't need them in the SHP), and press OK. I don't change the Format field from ERSI Shapefile.
QGIS just hangs for a few minutes. It eventually gives the attached error. It does produce a SHP (also attached in a ZIP), but the file sizes are far too small. The DBF only has 1253 data rows.
(Just to be clear, I can't share the data due to state law and confidentiality agreements.)
#3 Updated by Aren Cambre over 8 years ago
OK, on the export of the filtered subset of only about 6000 points, it takes about 6 minutes to complete. I've attached the output of the error dialog.
While this is happening, QGIS is using 100% of one of my CPUs.
Postgres has little activity, verified both through CPU usage and pgAdmin III's Server Status feature. All it does is receive occasional FETCH FORWARD 2000 FROM qgisf0 commands every minute or two, and that appears to produce 12 locks.
Something that could be distinctive is this PostGIS table has 148 columns (including the numeric index and geometry columns).
#8 Updated by Giovanni Manghi about 8 years ago
- Status changed from Feedback to Open
- Priority changed from Normal to Low
- Category changed from GUI to Vectors
- Subject changed from "Save as" on large Postgres point layer doesn't work to "Save as" on large a PostGIS point layer doesn't work
This are my findings:
- exporting large/huge postgis layers works ok: I tested on layers with up to 5 million features and in general it works ok (with "save as...")
- I confirm that with your dataset the operation (save as...) fails after a few minutes with a list of
Feature creation error (OGR error: Pointer 'hFeat' is NULL in 'OGR_L_SetFeature'.
)Stopping after 1001 errors
- exporting the same dataset with ogr2ogr is ok and gives the expected shapefile
- exporting with pgsql2shp seems to work fine, but beside not returning any error the number of features is not the same as the original postgis layer
#11 Updated by Giovanni Manghi about 8 years ago
- Subject changed from "Save as" on large a PostGIS point layer doesn't work to "Save as" on a large PostGIS point layer doesn't work
Matthias Kuhn wrote:
There's a hardcoded limit of 1000 allowed errors in QgsVectorFileWriter (qgsvectorfilewriter.cpp:860).
This could (and should) be made configurable. But this leaves open the question of which features fail to be written and for what reason.
as I said the operation in general works ok even for very large postgis vectors. With this particular dataset there is this problem.
#13 Updated by Bill Morris over 7 years ago
I've just run into the same error on a point file with 100 features and a single attribute column. Only 86 out of 100 exported. This bug is still out there.
Aren Cambre wrote:
Is there anything unique about this dataset? I haven't noticed any problems viewing it. It's just a bunch of points.
#15 Updated by Giovanni Manghi over 2 years ago
- Resolution set to end of life
- Status changed from Open to Closed
End of life notice: QGIS 2.18 LTR