Bug report #15407

Saving Shapefile edits corrupts file

Added by Jamie Portman almost 3 years ago. Updated over 2 years ago.

Status:Closed
Priority:Normal
Assignee:Even Rouault
Category:Data Provider/OGR
Affected QGIS version:2.14.5 Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:No Resolution:fixed/implemented
Crashes QGIS or corrupts data:Yes

Description

We have a number of shapefiles that have been created/edited in previous versions of QGIS where deleted features remained as phantom features (reportedly now fixed as per Issue 11007).

In the current LTR version (2.14.5) if these files are edited (feature deleted, split, merged or vertex added) then upon saving the edits, the entire file is corrupted - deleted features reappear, others disappear and attributes become completely reassigned to different features.
I have attached an example file that this occurs to.

I know this file suffers from the 'phantom' deleted features, because the feature count in the layers menu is higher than the feature count in the attribute table.
If a save-as of the file is done, the feature count of the new file matches that of the attribute table. This re-saved file can also have any edits done without resulting in corruption.

Not all files with these 'phantom' features behave this way - some can be edited and saving edits behaves exactly as expected, with the phantom features being cleaned out and all other data remaining as it should.

This file has had significant edits with large areas deleted, etc. so the resulting corruption is very obvious immediately.
Other files may be less obvious - how is a user to know when this will occur and ensure the file will not become corrupted after they save edits?

I had hoped that by making some edits that did not result in a change of object FID's (such as a simple attribute update) and saving those edits might result in a clean out ('packing') of the file, however this is not the case. It appears that perhaps the 'repack' function is only called upon saving edits such as deleting, merging or splitting features as well as adding a vertex to an existing feature. Even the addition of a new object does not seem to result in the cleanse of the layer :(

Centrelines.zip (283 KB) Jamie Portman, 2016-08-08 11:40 PM


Related issues

Related to QGIS Application - Bug report #11007: Deleted/edited features within SHAPEFILE are still recogn... Closed 2014-08-05

Associated revisions

Revision 3a906a18
Added by Even Rouault over 2 years ago

[OGR provider] Force REPACK at the first edit action.

In the case where we deal with a shapefile, it is possible that it has
pre-existing holes in the DBF (see #15407), so if using a GDAL version
recent enough (>=2.1.2) to have reliable packing, do a packing at the
first edit action.

Fixes #15407

Revision 6d5e7356
Added by Even Rouault over 2 years ago

[OGR provider] Force REPACK at the first edit action.

In the case where we deal with a shapefile, it is possible that it has
pre-existing holes in the DBF (see #15407), so if using a GDAL version
recent enough (>=2.1.2) to have reliable packing, do a packing at the
first edit action.

Fixes #15407

Revision aa5ec22c
Added by Even Rouault over 2 years ago

[OGR provider] Force REPACK at the first edit action.

In the case where we deal with a shapefile, it is possible that it has
pre-existing holes in the DBF (see #15407), so if using a GDAL version
recent enough (>=2.1.2) to have reliable packing, do a packing at the
first edit action.

Fixes #15407

Revision 872d86eb
Added by Even Rouault over 2 years ago

[OGR provider] Force REPACK at the first edit action.

In the case where we deal with a shapefile, it is possible that it has
pre-existing holes in the DBF (see #15407), so if using a GDAL version
recent enough (>=2.1.2) to have reliable packing, do a packing at the
first edit action.

Fixes #15407

Conflicts:
src/providers/ogr/qgsogrprovider.cpp

History

#1 Updated by Saber Razmjooei almost 3 years ago

I suggest to re-open #11007. It seems to be side-effects of the fix for #11007.

#2 Updated by Saber Razmjooei almost 3 years ago

  • Status changed from Open to Feedback

#3 Updated by Jamie Portman almost 3 years ago

Should I be doing that, or does an administrator need to?

I feel so bad... they were so thrilled to have finally resolved that issue after two years of feedback... :(

#4 Updated by Saber Razmjooei almost 3 years ago

The ticket has been re-opened.

I suggest to try editing the file in another GIS package and see if the corruption happens there too.

#5 Updated by Jamie Portman almost 3 years ago

MapInfo can't open a Shapefile that has 'phantom' features, the Universal Translator won't convert it and if you open in ArcGIS all the 'deleted' features reappear as it doesn't recognise the 'deleted' flag in the DBF!

#6 Updated by Danny Duong over 2 years ago

Does this issue occur in LTR 2.14.6?

Also, is it easily repeatable?
I don't seem to have the phantom deleted features anymore and haven't really noticed any discrepancies.

#7 Updated by Amy Taylor over 2 years ago

This is a problem we are seeing as well, and with shapefiles which have been created in QGIS as well as those older ones created using other software.

It seems to be that the split tool is the one which most commonly causes problems, where a shapefile will look ok until you hit the save button, then half of the split feature will disappear and attributes will become assigned to the wrong feature. Editing individual nodes is also flagging up problems, where if a node is deleted then sometimes the remaining adjacent nodes connect to a node on a totally separate and unconnected feature.

Unfortunately the problem isn't easily repeatable so we haven't been able to get an example shapefile which we can post on here.

#8 Updated by Even Rouault over 2 years ago

  • Status changed from Feedback to Closed

#9 Updated by Even Rouault over 2 years ago

  • Resolution set to fixed/implemented
  • Category set to Data Provider/OGR
  • Assignee set to Even Rouault
  • Target version set to Version 2.14

Fixes require GDAL 2.1.2 or above. Linked to #15570 and #15393

Also available in: Atom PDF