QGIS » QGIS Application

belg4mit - wrote:

Here are the requested screenshots.

My hunch is that is has something to do with the 2GB file size of the DBF.

Now I see the images I realize that here (QGIS master and 2.18, Windows and Linux) the points that in your case are classified in red (because of the NULLs) are never loaded. The vector metadata shows 2+ million records, but opening the attributes table is shows only a limited amount. The header says that records were filtered (but no filter was applied). I'm wondering if here there are several different issues.

When using ogr2ogr to convert the sahepfile (to GPKG for example) it returns

ERROR 1: fread(1754) failed on DBF file.

not sure it means the DBF file is corrupted (the limit is 4gb, not 2, afaik). But also is similar to https://trac.osgeo.org/gdal/ticket/4203

I believe this issue should be raised first in the gdal/ogr mailing list and see what they say.

Reported to GDAL as https://github.com/OSGeo/gdal/issues/937

Giovanni Manghi wrote:

When using ogr2ogr to convert the sahepfile (to GPKG for example) it returns

ERROR 1: fread(1754) failed on DBF file.

not sure it means the DBF file is corrupted (the limit is 4gb, not 2, afaik). But also is similar to https://trac.osgeo.org/gdal/ticket/4203

I believe this issue should be raised first in the gdal/ogr mailing list and see what they say.

The fread()error in https://trac.osgeo.org/gdal/ticket/4203 was a different case and for making it happen you'd need to use an ancient version of shapelib <1.2.7 together with GDAL that is more than 4 years old (prior https://trac.osgeo.org/gdal/changeset/27577). That error was a systematic one. The shx part was always written to have a different content length value than the shp part. For users that meant that the last record of the shapefile was dropped in reading.

There are no technical limits in the size of DBF file. I wrote a test case into https://github.com/OSGeo/gdal/issues/937 that creates a shapefile with 18 GB sized .dbf part and GDAL and QGIS can read it. It is still not a valid shapefile because ESRI has specified that no part in shapefile is allowed to exceed 2 GB.

I guess that what happened is:
- QGIS tried to write a shapefile with so much attribute data that DBF part exceeded 2 GB
- QGIS applies the GDAL setting "2GB_LIMIT=YES" and at 2 GB limit the writing of DBF was stopped but writing of .shp was continued
- As a result the final .shp has more geometries than .dbf has rows

If there is a bug I would say that it is in the writing of shapefile. Now it seems that writer is using the option 2GB_LIMIT=YES (see https://www.gdal.org/drv_shapefile.html) but when it meets the 2 GB limit with .dbf it does not error. But because the writer adds data sequentially into the data files of shapefile the total rollback would require deletion of the .shp, .shx, .dfb, and .prj files from the file system.

@Jukka QGIS doesn't mess with setting 2GB_LIMIT=YES. There is all evidence from the dataset that is was produced by ESRI software as it contains sbn, sbx, mxd, shp.xml files etc. So I suspect that the corruption comes from non-GDAL & non-QGIS side.