Bug report #6829

Qgis won't draw raster tiles / hard crash. (mac, linux)

Added by Graeme Bell over 11 years ago. Updated over 11 years ago.

Status:Closed
Priority:Severe/Regression
Assignee:-
Category:-
Affected QGIS version:master Regression?:No
Operating System:Macos (ML) + Linux (Fedora) Easy fix?:No
Pull Request or Patch supplied:No Resolution:
Crashes QGIS or corrupts data:Yes Copied to github as #:15971

Description

(Blocker: can't do anything with generated rasters till this is fixed. Unsure if core qgis or postgis-raster bug).

General description: Bugs in raster rendering or extent calcuation.
Test case code provided to allow it to be quickly reproduced, and tested on 2 builds of Qgis on two OS (details at end).

Start by disabling 'on the fly reprojection' in qgis.

Bug 1 - no display of most of raster.

1. run this in psql


drop table bigtest, bigrast;
create table bigtest (row_number serial, x int, y int);

insert into bigtest (x, y)
select a,b
from generate_series(-80000,1120000-1,100000) a, generate_series(6440000,7950000-1,100000) b;

create table bigrast (rid serial primary key, rast raster);

insert into bigrast
select row_number, ST_MakeEmptyRaster(100,100,x,y,1000,-1000,0,0,25833) from bigtest;

update bigrast
set rast=st_addband(rast, '1BB'::text, 1,0);

SELECT AddRasterConstraints('bigrast'::name, 'rast'::name);
-------------
This sets up a test raster covering norway in UTM33 at low resolution.

Add the layer to QGis canvas using Database/DB manager.

Expected result:
Should be a black box from -80000, 8040000 top left to 1120000, 6440000 bottom right

Actual result:

In both versions of QGIS, generated appearance is a thin black strip from -80000, 6440000 top left to 1120000, 6340000.
e.g. one row of tiles, most of the raster is missing.

Layer extent given in properties/metadata as:

-80000, 4840000
112000, 6440000

This is wrong.

In postgis:

select ST_Raster2WorldCoordY(rast,1) from bigrast;

yields correct values.

Notes:

1. Where is the 4840000 coming from???

2. I believe qgis may be cropping the raster tiles with an upside down, incorrect extent somehow. Is it calculating extent for itself using pixel scale settings and top left position - rather than via postgis extents? Is it interpreting the 'y-pixel scale' wrong and forgetting to invert it?

I get the same results from the linux 1.8 and Macos 1.9dev versions.

Bug 2

Similar, but this time a positive y-scale pixel size value. This shouldn't work, but does.

----
drop table bigtest, bigrast;

create table bigtest (row_number serial, x int, y int);

insert into bigtest (x, y)
select a,b
from generate_series(-80000,1120000-1,100000) a, generate_series(6440000,7950000-1,100000) b;

create table bigrast (rid serial primary key, rast raster);

insert into bigrast
select row_number, ST_MakeEmptyRaster(100,100,x,y,1000,1000,0,0,25833) from bigtest;

update bigrast
set rast=st_addband(rast, '1BB'::text, 1,0);

SELECT AddRasterConstraints('bigrast'::name, 'rast'::name);
----

LINUX 1.8
Add to qgis with DB Manager -> QGis implodes - hard crash/core dump!

Debug crash output here
http://pastebin.com/U3Htd6p7

MACOS 1.9
Displays the correct raster output that was expected for bug1 test case!
(but shouldn't)

N.B. using a positive y-scale pixel size is the change. This was an attempt to stop the extents being made 'upside down' as in test1.

However, y-scale should be negative e.g. where it is auto-set by postgis raster:

http://postgis.refractions.net/documentation/manual-2.0/RT_ST_MakeEmptyRaster.html

"The last version use a single parameter to specify the pixel size (pixelsize). scalex is set to this argument and scaley is set to the negative value of this argument. skewx and skewy are set to 0."

also
http://en.wikipedia.org/wiki/World_file

"Line 4: E: y component of the pixel height (y-scale), almost always negative"

I believe that a negative y-pixel size is the correct way to represent things. I think there is at least one bug in how qgis interprets y-pixel scale and how it calculates the extents of the coverage.

Tested in Fedora Linux with this build:

QGIS version
1.8.0-Lisboa
QGIS code revision
exported
Compiled against Qt
4.8.2
Running against Qt
4.8.3
Compiled against GDAL/OGR
1.9.1
Running against GDAL/OGR
1.9.1
GEOS Version
3.3.2
PostgreSQL Client Version
9.1.4
SpatiaLite Version
2.4.0
QWT Version
5.2.2

And MacOS X Mountain Lion with this build:

QGIS version
1.9.0-Master
QGIS code revision
9dd4a92
Compiled against Qt
4.8.3
Running against Qt
4.8.3
Compiled against GDAL/OGR
1.9.1
Running against GDAL/OGR
1.9.2
GEOS Version
3.3.5
PostgreSQL Client Version
9.1.6
SpatiaLite Version
3.0.1
QWT Version
6.0.1
PROJ.4 Version
480
QScintilla2 Version
2.6.2

$ postgres --version
postgres (PostgreSQL) 9.2.1
postgis is version 2.0.1

History

#1 Updated by Graeme Bell over 11 years ago

1) Tested in windows: qgis 1.9 fcf108d, Mac 1.8.0 Windows 1.8.0
They all behave the same as Mac 1.9dev.
So the hard crash only seems to happen in Linux 1.8 version.

But the bad extents/mostly missing tiles exists in all versions/platforms.

2) Upgraded to postgis 2.0.2 (built on gdal 1.10dev) and re-tested. No change.

POSTGIS="2.0.2 r10789" GEOS="3.3.6-CAPI-1.7.6" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.10dev, released 2011/12/29" LIBXML="2.8.0" TOPOLOGY RASTER

#2 Updated by Giovanni Manghi over 11 years ago

  • Status changed from Open to Feedback

Graeme Bell wrote:

1) Tested in windows: qgis 1.9 fcf108d, Mac 1.8.0 Windows 1.8.0
They all behave the same as Mac 1.9dev.
So the hard crash only seems to happen in Linux 1.8 version.

But the bad extents/mostly missing tiles exists in all versions/platforms.

2) Upgraded to postgis 2.0.2 (built on gdal 1.10dev) and re-tested. No change.

POSTGIS="2.0.2 r10789" GEOS="3.3.6-CAPI-1.7.6" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.10dev, released 2011/12/29" LIBXML="2.8.0" TOPOLOGY RASTER

is this a regression in 1.8/master since 1.7.4?

#3 Updated by Graeme Bell over 11 years ago

I'm not sure if this very recent discussion on the Postgis-devel list is relevant here:

http://lists.osgeo.org/pipermail/postgis-devel/2012-November/022914.html

#4 Updated by Jorge Arevalo over 11 years ago

Hello,

I'm the GDAL PostGIS Raster driver developer. And I think this is a problem of the driver, actually. The QGIS PostGIS Raster plugin uses it underneath.

First thing I must say: don't use any stable version of GDAL. I committed a version with a lot of bugs fixed in August. It will be included in GDAL 1.10 (due to 2012/12/31). Meanwhile, use the daily snapshot (http://www.gdal.org/daily) or the last version from trunk (https://svn.osgeo.org/gdal/trunk/gdal). Sorry for the inconvenience.

Could you please update your GDAL version to, say, today's snapshot, and test again? The shortest way will be check the raster extent returned by gdalinfo (or the properties of the raster in QGIS)

#5 Updated by Graeme Bell over 11 years ago

Long story short: tried it, no change.

Graemes-MacBook-Pro:~ gbb$ gdalinfo --version
GDAL 1.10dev, released 2011/12/29

-> Looks good at the command line...

select postgis_full_version();

POSTGIS="2.0.2 r10789" GEOS="3.3.6-CAPI-1.7.6" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.10dev, released 2011/12/29" LIBXML="2.8.0" TOPOLOGY RASTER
(1 row)

gdalinfo.... (raster)

Driver: PostGISRaster/PostGIS Raster driver
Files: none associated
Size is 1200, 1600
Coordinate System is:
PROJCS["ETRS89 / UTM zone 33N",
GEOGCS["ETRS89",
DATUM["European_Terrestrial_Reference_System_1989",
SPHEROID["GRS 1980",6378137,298.257222101,
AUTHORITY["EPSG","7019"]],
TOWGS84[0,0,0,0,0,0,0],
AUTHORITY["EPSG","6258"]],
PRIMEM["Greenwich",0,
AUTHORITY["EPSG","8901"]],
UNIT["degree",0.0174532925199433,
AUTHORITY["EPSG","9122"]],
AUTHORITY["EPSG","4258"]],
UNIT["metre",1,
AUTHORITY["EPSG","9001"]],
PROJECTION["Transverse_Mercator"],
PARAMETER["latitude_of_origin",0],
PARAMETER["central_meridian",15],
PARAMETER["scale_factor",0.9996],
PARAMETER["false_easting",500000],
PARAMETER["false_northing",0],
AUTHORITY["EPSG","25833"],
AXIS["Easting",EAST],
AXIS["Northing",NORTH]]
Origin = (-80000.000000000000000,7940000.000000000000000)
Pixel Size = (1000.000000000000000,-1000.000000000000000)
Corner Coordinates:
Upper Left ( -80000.000, 7940000.000)
Lower Left ( -80000.000, 6340000.000)
Upper Right ( 1120000.000, 7940000.000)
Lower Right ( 1120000.000, 6340000.000)
Center ( 520000.000, 7140000.000)
Band 1 Block=100x100 Type=Byte, ColorInterp=Gray
NoData Value=0
Image Structure Metadata:
NBITS=1

BUT

Still does not display correctly in qgis, as before. Also, qgis was loading the raster even when I was using the gdal 1.9.2 without postgis raster support. So... might this be something internal to qgis? Or can I force it to use the gdal library to load the raster?

Interestingly, there is an option in qgis DB manager:

Extent:
(unknown) (find out)

(find out) does nothing

QGIS says it is running against '1.9.2' GDAL (I had uninstalled 1.9.2 completely before the rebuild of postgis).

I checked that 1.9.2. was deleted and had rebooted to clear out the shared library from memory.

#6 Updated by Graeme Bell over 11 years ago

  • Status changed from Feedback to Closed

Jorge, you are a superhero.

IT WORKS!

The guide at Kyngchaos is ... somewhat ... out of date, as are parts of the configure script used in gdal (--enable-macosx-framework, absent, presumed dead).

Here's what I did to get things working on the mac (was running gdal 1.9, postgis 2.0.1 and qgis 1.9).

--- BACKUP YOUR MAC, your DB, etc: carbon copy cloner or superduper is a good choice ---

rip out the stuff you installed with macports or homebrew
port uninstall postgis2 gdal (etc.)

mkdir -p /build/postgis2
cd /build/postgis2
(download, build, install postgis 2.0.2 here
e.g. curl http://download.osgeo.org/postgis/source/postgis-2.0.2.tar.gz
./configure; make; make install

to finish upgrading postgis2, run this in psql

CREATE EXTENSION postgis FROM unpackaged;
CREATE EXTENSION postgis_topology FROM unpackaged;)

next rebuild gdal2 from svn - do not follow the instructions at kyngchaos.

mkdir -p /build/gdal2
cd /build/gdal2
svn checkout https://svn.osgeo.org/gdal/trunk/gdal
./configure; make; make install

[no fancy configure options / file editing needed... it seems]

next, make sure the old frameworks are REALLY gone (especially if you previously installed the kyngchaos gdal framework packages)
Qgis links to these automatically. We want qgis to point to the new library.

mv /Library/Frameworks/GDAL.framework /Library/Frameworks/GDAL-old.framework
mkdir -p /Library/Frameworks/GDAL.framework/Versions/1.9/
ln -s /usr/local/lib/libgdal.dylib /Library/Frameworks/GDAL.framework/Versions/1.9/GDAL

Now run qgis, et voila: Bug STOMPED. Tiled rasters render as they should.

Please be warned: this may introduce new horrible bugs elsewhere. But for me, today is a good day.

#7 Updated by Graeme Bell over 11 years ago

n.b. if you have successfully persuaded qgis to run the new version of gdal, it will be reported as 1.10dev in the 'about' menu.

#8 Updated by Jorge Arevalo over 11 years ago

Good to hear that! I'll continue working on the driver :-D

Also available in: Atom PDF