Feature request #21279

Processing/OGR: do not use shapefile as format for temp input files

Added by Tobias Wendorff almost 6 years ago. Updated over 5 years ago.

Status:Open
Priority:Normal
Assignee:-
Category:Processing/Core
Pull Request or Patch supplied:No Resolution:
Easy fix?:No Copied to github as #:29097

Description

The algorithms from GDAL's toolbox for vector processing still use shapefiles as intermediate format. This results in corrupted fieldnames, since those are limited in the DBF format. Also, field contents might get corrupted if they're longer than 255 chars.

ogr2ogr -f "ESRI Shapefile" C:/Users/tobwen/AppData/Local/Temp/processing_5fe343f547494d5e9fc31fe5cc38c868/680b663b94af46ab9516f2acd4141efd/OUTPUT.shp R:/demo.geojson

Please use GPKG for this.

History

#1 Updated by Jürgen Fischer almost 6 years ago

  • Tracker changed from Bug report to Feature request

#2 Updated by Tobias Wendorff almost 6 years ago

Sorry, Jürgen, I strongly disagree that this is a feature request only.

The use of shapefile is an intermediate format is creating unexpected results and is destroying the user's results. Furthermore, ogr2ogr actually is one of the most capable applications out there, which can handle GPKG files. Even if I don't like GPKG that much, it has removed limits, we had with prosperity shapefile over the last decades.

I believe patching this is damn easy, you only need to replace -F Esri Shapefile with -F GPKG and fix filenames from .shp to .gpkg.

Fixing this bug will save hours of processing, energy and might be even some kittens.

#3 Updated by Jürgen Fischer almost 6 years ago

Tobias Wendorff wrote:

I believe patching this is damn easy,

Patch? Pull request?

#4 Updated by Giovanni Manghi almost 6 years ago

  • Status changed from Open to Feedback

is this about specifically the OGR based "convert format" tool?

#5 Updated by Tobias Wendorff almost 6 years ago

Giovanni Manghi wrote:

is this about specifically the OGR based "convert format" tool?

I do not know whether this question was addressed to me or whether I can answer it at all. As far as I can tell, all the vector tools are affected, f.e. Vector geoprocessing -> Buffer vectors

ogr2ogr C:/Users/tobwen/AppData/Local/Temp/processing_08c1c2be50d4431a82d6d123c3eeec35/ae5cee8d9c484a1986ed5a3db42f1d50/OUTPUT.shp R:/demo.sqlite -dialect sqlite -sql "SELECT ST_Buffer(geometry, 10.0) AS geometry,* FROM 'demo'" -f "ESRI Shapefile"

#6 Updated by Giovanni Manghi almost 6 years ago

  • Status changed from Feedback to Open
  • Assignee set to Giovanni Manghi

#7 Updated by Giovanni Manghi almost 6 years ago

  • Subject changed from GDAL toolbox still use destructive SHP format to Processing/OGR: do not use shapefile as format for temp input files
  • Assignee deleted (Giovanni Manghi)

#8 Updated by Tobias Wendorff over 5 years ago

To prevent oblivion: this is still an issue in QGIS v3.7.0-Master, revision 8c21a9e848

Since latest GDAL (>= 2.5) supports proj6, the use of OGR might be interested to advanced users. Right now, you need to remove all (destroyed) columns, rejoin the original table and write it to a new file.

Also available in: Atom PDF