Feature request #19733

make GPKG the default output format for GRASS tools

Added by Markus Neteler almost 2 years ago. Updated over 1 year ago.

Status:Closed
Priority:Normal
Assignee:-
Category:Processing/GRASS
Pull Request or Patch supplied:Yes Resolution:fixed/implemented
Easy fix?:Yes Copied to github as #:27558

Description

new description
#19733-5

old description
Processing > GRASS 7 > v.rast.stats is not functional (reported also in several forums [1]). The reason is apparently a parse error since column names have multiple "_". When reducing the statistical measures to those without underscore (e.g."sum") it works. When having "percentile" is doesn't since the percentile value is transferred via underscore, e.g. "percentile=95" becomes "percentile_95" which is not parsed properly in QGIS processing.

By default all measure are active, i.e.
number,minimum,maximum,range,average,stddev,variance,coeff_var,sum,first_quartile,median,third_quartile,percentile

Issue is in both QGIS 2.18.x as well as 3.2.x.

Log:

Algorithmus v.rast.stats - Calculates univariate statistics from a raster map based on vector polygons and uploads statistics to new attribute columns. startet...
g.proj -c proj4="+proj=sterea +lat_0=52.15616055555555 +lon_0=5.38763888888889 +k=0.9999079 +x_0=155000 +y_0=463000 +ellps=bessel +towgs84=565.2369,50.0087,465.658,-0.406857,0.350733,-1.87035,4.0812 +units=m +no_defs"
v.in.ogr min_area=0.0001 snap=-1 input="/home/user/geodata_dav/mundialis_daten/projekte/who_geodata/data/Districts" layer=Districts output=tmp15356101025715 --overwrite -o
r.external input="/home/user/ES5_TemperatureCoolingModul/actualT_dailymean.tif" band=1 output=tmp15356101025716 --overwrite -o
g.region n=322441.565 s=312777.529 e=180172.152 w=172817.428 res=10.0064272109
v.rast.stats -c map="tmp15356101025715" raster="tmp15356101025716" column_prefix="s" method="third_quartile,percentile" percentile="2" --overwrite
v.out.ogr --overwrite -s -e input=tmp15356101025715 output="/tmp/processing5113522d9c174e7bb445f24ef38c5f5a/135e78c4ae1d430da568b502506b4ab0" format=ESRI_Shapefile output_layer=output
[...]
Preprocessing input data...
Processing input data (7 categories)...
0..3..6..9..12..15..18..21..24..27..30..33..36..39..42..45..48..51..54..57..60..63..66..69..72..75..78..81..84..87..90..93..96..99..100
Updating the database ...
ERROR 6: Failed to add field named 's_third_quartile'
ERROR 6: Failed to add field named 's_percentile_2'
Exporting 13 areas (may take some time)...
7..15..23..30..38..46..53..61..69..76..84..92..100

Assumed solution: fix the parsing of the column names (allow for multiple underscores).

[1] Reported without solution here:

History

#1 Updated by Giovanni Manghi almost 2 years ago

  • Affected QGIS version changed from 2.18.22 to 3.2.2
  • Status changed from Open to Feedback

Did it worked as expected in older qgis releases?

#2 Updated by Markus Neteler almost 2 years ago

Unfortunately no idea. But some of the related reports I found are also older than one year.

Potential example (4 years ago but no log provided):
https://gis.stackexchange.com/questions/75254/how-to-perform-grass-v-rast-stats-via-qgis/75283

#3 Updated by Giovanni Manghi almost 2 years ago

  • Status changed from Feedback to Open
  • Regression? changed from Yes to No

#4 Updated by Markus Neteler almost 2 years ago

I suspect that the underscore parsing error happens in

plugins/processing/algs/grass7/Grass7Algorithm.py

around line 553.

#5 Updated by Markus Neteler almost 2 years ago

  • Subject changed from v.rast.stats is broken due to parsing error of double underscore in column names to v.rast.stats support is broken due to column name length issue

Update from grass-dev [1, 2]: it appears to not be an underscore problem but a result of the column name length limit of 10 chars since SHP is used for data exchange in GRASS-Processing.

Proposed solution: switch to GeoPackage (in Grass7Algorithm.py)

[1] https://lists.osgeo.org/pipermail/grass-dev/2018-September/089579.html
[2] https://lists.osgeo.org/pipermail/grass-dev/2018-September/089580.html

#6 Updated by Giovanni Manghi almost 2 years ago

  • Subject changed from v.rast.stats support is broken due to column name length issue to make GPKG the default output format for GRASS tools
  • Tracker changed from Bug report to Feature request
  • Description updated (diff)

#7 Updated by Jürgen Fischer almost 2 years ago

  • Description updated (diff)

#8 Updated by Markus Neteler almost 2 years ago

Could you please indicate where the SHP format is hardcoded so that I can potentially provide a GPKG patch? I didn't find the respective line if code. Thanks

#9 Updated by Giovanni Manghi almost 2 years ago

Markus Neteler wrote:

Could you please indicate where the SHP format is hardcoded so that I can potentially provide a GPKG patch? I didn't find the respective line if code. Thanks

I think is the other way: QGIS uses the v.out.ogr default format (shapefile) when outputting to a temp file, so in this case I think that is GPKG that needs to be hardcoded.

#10 Updated by Markus Neteler almost 2 years ago

Giovanni Manghi wrote:

Markus Neteler wrote:

Could you please indicate where the SHP format is hardcoded so that I can potentially provide a GPKG patch? I didn't find the respective line if code. Thanks

I think is the other way: QGIS uses the v.out.ogr default format (shapefile) when outputting to a temp file, so in this case I think that is GPKG that needs to be hardcoded.

We switched 11 months to GPKG as default vector export format:
https://trac.osgeo.org/grass/log/grass/branches/releasebranch_7_4/vector/v.out.ogr
@71597 11 months neteler v.out.ogr: Set OGC GeoPackage? as default export format (see #3428); …

This was published with GRASS GIS 7.4.0,
https://trac.osgeo.org/grass/wiki/Release/7.4.0-News#Majormodulechanges

So, if a user has G7.4.x installed, then the problem might be gone.

However, I agree to your suggestion that GPKG needs to be hardcoded in Processing/GRASS to avoid that users with G7.2 installed face this issue. If I am not mistaken, that would be some additional lines in
plugins/processing/algs/grass7/Grass7Algorithm.py around l314?

#11 Updated by Giovanni Manghi almost 2 years ago

Markus Neteler wrote:

Giovanni Manghi wrote:

Markus Neteler wrote:

Could you please indicate where the SHP format is hardcoded so that I can potentially provide a GPKG patch? I didn't find the respective line if code. Thanks

I think is the other way: QGIS uses the v.out.ogr default format (shapefile) when outputting to a temp file, so in this case I think that is GPKG that needs to be hardcoded.

We switched 11 months to GPKG as default vector export format:
https://trac.osgeo.org/grass/log/grass/branches/releasebranch_7_4/vector/v.out.ogr
@71597 11 months neteler v.out.ogr: Set OGC GeoPackage? as default export format (see #3428); …

This was published with GRASS GIS 7.4.0,
https://trac.osgeo.org/grass/wiki/Release/7.4.0-News#Majormodulechanges

So, if a user has G7.4.x installed, then the problem might be gone.

I have 7.4.0 installed here on Kubuntu 18.04 bust still seems to output (temp files) to shp, so it seems it not that simple.

However, I agree to your suggestion that GPKG needs to be hardcoded in Processing/GRASS to avoid that users with G7.2 installed face this issue. If I am not mistaken, that would be some additional lines in
plugins/processing/algs/grass7/Grass7Algorithm.py around l314?

My hunch is that the format should be hardcoded here https://github.com/qgis/QGIS/blob/master/python/plugins/processing/algs/grass7/Grass7Algorithm.py#L879 but I gave it a try and didn't change anything (possibly because at the end shp is really hardcoded somewhere else).

#12 Updated by Markus Neteler almost 2 years ago

Giovanni Manghi wrote:

Markus Neteler wrote:

...

However, I agree to your suggestion that GPKG needs to be hardcoded in Processing/GRASS to avoid that users with G7.2 installed face this issue. If I am not mistaken, that would be some additional lines in
plugins/processing/algs/grass7/Grass7Algorithm.py around l314?

My hunch is that the format should be hardcoded here https://github.com/qgis/QGIS/blob/master/python/plugins/processing/algs/grass7/Grass7Algorithm.py#L879 but I gave it a try and didn't change anything (possibly because at the end shp is really hardcoded somewhere else).

Yes, probably 4 lines above in https://github.com/qgis/QGIS/blob/master/python/plugins/processing/algs/grass7/Grass7Algorithm.py#L876 :

outFormat = QgsVectorFileWriter.driverForExtension(os.path.splitext(fileName)[1]).replace(' ', '_')

This will predefine it (to whatever, this I didn't manage to find out).

#13 Updated by Giovanni Manghi almost 2 years ago

Yes, probably 4 lines above in https://github.com/qgis/QGIS/blob/master/python/plugins/processing/algs/grass7/Grass7Algorithm.py#L876 :

outFormat = QgsVectorFileWriter.driverForExtension(os.path.splitext(fileName)[1]).replace(' ', '_')

This will predefine it (to whatever, this I didn't manage to find out).

maybe worth commenting on github on those lines and pinging @nyalldawson and/or @volaya and/or @alexbruy

#14 Updated by Markus Neteler over 1 year ago

Giovanni Manghi wrote:

maybe worth commenting on github on those lines and pinging @nyalldawson and/or @volaya and/or @alexbruy

No answers from anyone :-(

#16 Updated by Markus Neteler over 1 year ago

  • Status changed from Feedback to Open
  • Pull Request or Patch supplied changed from No to Yes
  • Easy fix? changed from No to Yes

Thanks for the valuable hint!

I have prepared a PR against QGIS 3.4 accordingly: https://github.com/qgis/QGIS/pull/8431

Importantly, this should also reach QGIS 2.18 by a backport.

#17 Updated by Alexander Bruy over 1 year ago

  • Resolution set to fixed/implemented
  • Status changed from Open to Closed

Fixed in master.

#18 Updated by Markus Neteler over 1 year ago

Alexander Bruy wrote:

Fixed in master.

In which version will this update be shipped?

Also available in: Atom PDF