Bug report #605

Building pyramids causes segfault

Added by Gary Sherman over 13 years ago. Updated about 11 years ago.

Status:Closed
Priority:Low
Assignee:Tim Sutton
Category:Rasters
Affected QGIS version: Regression?:No
Operating System:All Easy fix?:No
Pull Request or Patch supplied: Resolution:fixed
Crashes QGIS or corrupts data: Copied to github as #:10664

Description

Building pyramids segfaults. Image size was 8192x4096.

Backtrace:

(gdb) bt
#0  0xb62424c7 in memset () from /lib/tls/i686/cmov/libc.so.6
#3905  0x00002000 in ?? ()
#3906  0xb6be5c85 in _TIFFmemset () from /usr/local/lib/libgdal.so.1
#3907  0xb6bd4ab2 in LZWPreDecode () from /usr/local/lib/libgdal.so.1
#3908  0xb6be372c in TIFFStartTile () from /usr/local/lib/libgdal.so.1
#3909  0xb6be330d in TIFFFillTile () from /usr/local/lib/libgdal.so.1
#3910  0xb6be2bce in TIFFReadEncodedTile () from /usr/local/lib/libgdal.so.1
#3911  0xb6ab8bd5 in GTiffDataset::LoadBlockBuf () from /usr/local/lib/libgdal.so.1
#3912  0xb6aba436 in GTiffRasterBand::IReadBlock () from /usr/local/lib/libgdal.so.1
#3913  0xb6c10691 in GDALRasterBand::GetLockedBlockRef () from /usr/local/lib/libgdal.so.1
#10 0xb6c22235 in GDALRasterBand::IRasterIO () from /usr/local/lib/libgdal.so.1
#3914 0xb6c0f4fc in GDALRasterBand::RasterIO () from /usr/local/lib/libgdal.so.1
#3915 0xb6c1efc8 in GDALDownsampleChunk32R () from /usr/local/lib/libgdal.so.1
#3916 0xb6c20156 in GDALRegenerateOverviews () from /usr/local/lib/libgdal.so.1
#3917 0xb6abd0c1 in GTiffDataset::IBuildOverviews () from /usr/local/lib/libgdal.so.1
#3918 0xb6bff0b6 in GDALDataset::BuildOverviews () from /usr/local/lib/libgdal.so.1
#3919 0xb7ee7529 in [[QgsRasterLayer]]::buildPyramids (this=0x82abfa0, [email protected],
    [email protected]) at qgsrasterlayer.cpp:4216
#3920 0xb65ccaf1 in [[QgsRasterLayerProperties]]::on_buttonBuildPyramids_clicked (this=0x8319d80)
    at qgsrasterlayerproperties.cpp:478

History

#1 Updated by Tim Sutton over 13 years ago

Moved to milestone 0.8.2 since we wont be fixing any further issues before the 0.8.1 release

#2 Updated by Frank Warmerdam - almost 13 years ago

I believe an upgrade to GDAL 1.4.4 with internal libtiff, or libtiff 3.9 (from CVS) will resolve this problem. It is a libtiff bug, not a GDAL or QGIS bug.

#3 Updated by Maciej Sieczka - over 12 years ago

I have just checked in QGIS , built against GDAL 1.5.0+SVN 2008-03-04 (internal libtiff). No crashes building overviews in QGIS whatsoever. Checked with several Byte and Uint16 GeoTIFFs, all having colortables, compressed with lzw, deflate or packbits.

I still believe however that building external overviews in Erdas Imagine format would be more bulletproof and universal. The benefits of external overviews:

1. The input data are not modified.

2. Ergo, input data remain compatible with software that don't tolerate internal overviews and

3. there's no need to duplicate data for safety/usability reasons before building overviews.

4. Other software support external Erdas Imagine overviews that GDAL creates - eg. ArcGIS, ERDAS Imagine.

What do developers think about it?

For now, to create an external Erdas Imagin overwievs from CLI use eg.:

gdaladdo --config USE_RRD YES some.tif 2 4 8 16

#4 Updated by Tim Sutton about 12 years ago

  • Status changed from Open to In Progress

#5 Updated by Maciej Sieczka - about 12 years ago

Since my last #comment:4 regarding overwievs, I have learned that external overviews in GeoTIFF are also a nice option. They meet points 1-3 too, and although they are not so compatible as Erdas overviews, they are more compressable beacuse you can use DEFLATE for TIFF, which is more efficient than Erdas RLC compression method (the only available for Erdas, applied by default to Erdas overviews; and can't be disabled for overviews AFAIK).

Compare (the input is a 361 MB uncompressed GeoTIFF):

$ gdaladdo --config USE_RRD YES atif.tif 2 4 8 16
$ ls -sh atif.aux
125M atif.aux
$ chmod a-w atif.tif
$ gdaladdo --config COMPRESS_OVERVIEW DEFLATE atif.tif 2 4 8 16
$ ls -sh atif.tif.ovr
103M atif.tif.ovr

So, ideally, QGIS should let the user choose if he wants internal overviews (and if/what compresssion scheme) or external overviews - in Erdas or GeoTIFF format (the former for better compatibility, the latter for space saving - if chosen let the user choose DEFLATE/LZW/PACKBITS; hint that DEFLATE compresses best and better than Erdas compressions).

#6 Updated by jaadfoo - about 12 years ago

One question:

In the first example you show a ".aux" file of 125M. Shouldn't this be the ".rrd" file? From my understanding of aux files, this should contain a small amount of information, usually histogram / coordinate / projection info / etc. The rrd should contain the pyramids, and in my local tests, this is the case.

#7 Updated by Maciej Sieczka - about 12 years ago

Replying to [comment:8 dickeya]:

One question:

In the first example you show a ".aux" file of 125M. Shouldn't this be the ".rrd" file?

My *.aux file is a direct output of a command like 'gdaladdo --config USE_RRD YES some.tif 2 4 8 16' and the command does not produce anything else. If you get a file with different extension then I don't know why. I'm using GDAL 1.5.2 + SVN FWIW.

The rrd should contain the pyramids, and in my local tests, this is the case.

Do you mean gdaladdo output? Please copy&paste the command syntax, gdalinfo of input file, the list of output files, gdalinfo of what you believe is the overviews file and your GDAL version.

#8 Updated by jaadfoo - about 12 years ago

As it turns out, I had HFA_USE_RRD set to YES in my environment. After removal, gdaladdo only created an aux file.

Thanks.

#9 Updated by Tim Sutton about 12 years ago

  • Status changed from In Progress to Closed
  • Resolution set to fixed

With you can now specify internal or external overviews. Compressed overviews are not yet supported since http://www.gdal.org/classGDALDataset.html#3906aa6f88b3bbc840a5696236af11dde15 provides no obvious way to do this (I would need to go and read gdaladdo code and see how they did it there). If you really want this feature, please file a separate enhancement ticket since we are wondering way off the topic of the original ticket.

Best regards

Tim

#10 Updated by Maciej Sieczka - about 12 years ago

Replying to [comment:11 timlinux]:

With you can now specify internal or external overviews.

Cool. Smallish issue - there is a redundant test for file write access if the user chooses to build external overviews. It effectively disables external overviews creation for read-only rasters. As we are not going to alter the file, we don't care if it is writeable.

Compressed overviews (...) . If you really want this feature, please file a separate
enhancement ticket since we are wondering way off the topic of the original ticket.

Done in #1199.

#11 Updated by Tim Sutton about 12 years ago

Hi

Its strange, because that message is propogated from GDAL as shown in the following snippet:

1601 if (!res.isNull())
1602 {
1603 if (res "ERROR_WRITE_ACCESS")
1604 {
1605 QMessageBox::warning(this, tr("Write access denied"),
1606 tr("Write access denied. Adjust the file permissions and try again.\
\
") );
1607 }
1608 else if (res "ERROR_WRITE_FORMAT")
1609 {
1610 QMessageBox::warning(this, tr("Building pyramids failed."),
1611 tr("The file was not writeable. Some formats do not support pyramid overviews. Consult the GDAL documentation if in doubt.") );
1612 }
1613 else if (res == "FAILED_NOT_SUPPORTED")
1614 {
1615 QMessageBox::warning(this, tr("Building pyramids failed."),
1616 tr("Building pyramid overviews is not supported on this type of raster.") );
1617 }
1618 }

If gdal reports an error my feeling is that we should report it. Perhaps I will just add a message saying that the warning can be ignored if building external overviews and the dir is writable.

Regards

Tim

#12 Updated by Maciej Sieczka - about 12 years ago

Replying to [comment:13 timlinux]:

Hi

Its strange, because that message is propogated from GDAL as shown in the following snippet:

1601 if (!res.isNull())
1602 {
1603 if (res "ERROR_WRITE_ACCESS")
1604 {
1605 QMessageBox::warning(this, tr("Write access denied"),
1606 tr("Write access denied. Adjust the file permissions and try again.\

\
") );

1607 }
1608 else if (res "ERROR_WRITE_FORMAT")
1609 {
1610 QMessageBox::warning(this, tr("Building pyramids failed."),
1611 tr("The file was not writeable. Some formats do not support pyramid overviews. Consult the GDAL documentation if in doubt.") );
1612 }
1613 else if (res == "FAILED_NOT_SUPPORTED")
1614 {
1615 QMessageBox::warning(this, tr("Building pyramids failed."),
1616 tr("Building pyramid overviews is not supported on this type of raster.") );
1617 }
1618 }

If gdal reports an error my feeling is that we should report it.

Hmm, probably GDAL reports this error in error. It is logical to discard it IMO.

Perhaps I will just add a message saying that the warning can be ignored if building external
overviews and the dir is writable.

I think QGIS should not bother about theis error at all and just proceed. There is no point in reporting that the file is not writeable if we don't try to modify it.

Another small issue I noticed: even if QGIS fails to build the overviews for whatever reason, the icons for selected overviews change like if the overviews were actually created. Re-entering raster's properties shows they were really were not.

Regards

Tim

#13 Updated by Anonymous about 11 years ago

Milestone Version 1.0.0 deleted

Also available in: Atom PDF