Bug report #11887

minimumValue() in 2.6.x for Windows different than Linux for NULL

Added by Jeff Cavner almost 6 years ago. Updated almost 6 years ago.

Status:Closed
Priority:Normal
Assignee:-
Category:Vectors
Affected QGIS version:2.6.0 Regression?:No
Operating System:Windows Easy fix?:No
Pull Request or Patch supplied:No Resolution:up/downstream
Crashes QGIS or corrupts data:No Copied to github as #:20101

Description

minimumValue() against the data provider or a vector layer object in Windows if a NULL value is present in the attributes for a field will return a 0 treating the NULL as the minimum value, instead of the minimum numeric value in that field (which is my case is greater than 0). In Linux for 2.6.x it works correctly and returns the minimum numeric value and does not treat NULLs as numeric.

Associated revisions

Revision 1f8dee0c
Added by Matthias Kuhn almost 6 years ago

Don't consider NULL values for minimumValue() and maximumValue()

Fix #11887

Revision 790916b2
Added by Matthias Kuhn almost 6 years ago

Don't quote column names for MIN and MAX for ogrsql

See https://trac.osgeo.org/gdal/ticket/5799#comment:9

Fixes #11887

History

#1 Updated by Matthias Kuhn almost 6 years ago

  • Status changed from Open to Feedback

Which data provider are you using?
Is the layer in edit mode and does it contain changes when the mistake happens?

#2 Updated by Jeff Cavner almost 6 years ago

The dataprovider from a vector layer returned from vectorLyr.dataProvider(). Also the same thing happens against just the vectorLyr, so vectorLyr.minimumValue(index) returns the same 0.0 for a null value. The layer is not in edit mode and does not contain changes.

#3 Updated by Matthias Kuhn almost 6 years ago

And which data provider as in postgis/spatialite/ogr ...?

#4 Updated by Jeff Cavner almost 6 years ago

ogr -- shapefile

#5 Updated by Matthias Kuhn almost 6 years ago

Are the OGR versions on your Windows and Linux machine the same?

#6 Updated by Jeff Cavner almost 6 years ago

windows ogr 1.11.1

linux ogr 1.9.2

#7 Updated by Matthias Kuhn almost 6 years ago

  • Status changed from Feedback to Open
  • Resolution set to up/downstream

Can you open an issue on their tracker?
There's a similar one open that is being fixed for GDAL/OGR 2.0
https://trac.osgeo.org/gdal/ticket/5333

#8 Updated by Jeff Cavner almost 6 years ago

Yes, just created a ticket over there. Thank you for all of the help.

#9 Updated by Jukka Rahkonen almost 6 years ago

By looking at this ESRI document http://webhelp.esri.com/arcgisdesktop/9.3/index.cfm?TopicName=Geoprocessing_considerations_for_shapefile_output NULL values in a numeric field are really stored as zeroes into the .dbf part of shapefile. And if NULL is stored as a zero into the .dbf file there is no way for GDAL to separate it from zero that represents a real zero. I may be wrong but this ticket and the corresponding GDAL ticket feel invalid for me. But I do not understand why Linux makes a difference for you. Could you add a shapefile with a few features and some NULLs for testing?

GDAL ticket 5333 is not related, it was about creating new functions for finding min/max/avg dates from a DATE field of a shapefile which actually a string "CCYYMMDD" in the .dbf file.

#10 Updated by Matthias Kuhn almost 6 years ago

I can reproduce the problem on Linux with GDAL 1.11

Taking the shapefile lakes from here:

https://github.com/opengisch/QGIS-Sampledata/tree/master/shapefiles

Minimum value for the xlabel column (values 28399 and a some NULLs):

iface.activeLayer().dataProvider().minimumValue( 3 )

0

Editing the attriubte table and changing a NULL value to 0 works. The change can be saved and reloaded properly, so GDAL seems to be able to distinguish 0 and NULL for shapefiles.

Upstream issue: https://trac.osgeo.org/gdal/ticket/5799

#11 Updated by Jukka Rahkonen almost 6 years ago

I downloaded the lakes shapefile and had a try with GDAL 2.0-dev on Windows 7 but this version does not show the issue with ogrinfo:

ogrinfo lakes.shp -sql "select min(xlabel) from lakes"
INFO: Open of `lakes.shp'
using driver `ESRI Shapefile' successful.

Layer name: lakes
Geometry: None
Feature Count: 1
Layer SRS WKT:
(unknown)
MIN_xlabel: Integer (8.0)
OGRFeature(lakes):0
MIN_xlabel (Integer) = 28399

#12 Updated by Matthias Kuhn almost 6 years ago

Hmmm, same result here with 1.11.1.

Debugging shows that ogr fails to interpret the sql qgis sends it (see below) and it's actually our fallback code that fails to properly handle NULL values.

QGIS debug output:

src/providers/ogr/qgsogrprovider.cpp: 2360: (minimumValue) [30361ms] Failed to execute SQL: SELECT MIN("xlabel") FROM "lakes" 

ogrinfo output:

ogrinfo lakes.shp -sql 'SELECT MIN("xlabel") FROM "lakes"'
INFO: Open of `lakes.shp'
      using driver `ESRI Shapefile' successful.
ERROR 1: Argument of column Summary Function 'MIN' should be a column.

Therefore the proper question is, should we send an unquoted column name to ogr or should that be changed on gdal side?

#13 Updated by Matthias Kuhn almost 6 years ago

  • Status changed from Open to Closed

Also available in: Atom PDF