Bug report #8356

VRT file opens instantly in 1.8, not opened after 10 mins on Master

Added by James Stott over 11 years ago. Updated over 11 years ago.

Status:Closed
Priority:Severe/Regression
Assignee:-
Category:-
Affected QGIS version:master Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:No Resolution:
Crashes QGIS or corrupts data:No Copied to github as #:17136

Description

I am trying to open a vrt file in Master. It is made up of 348 jpg files (each file ranging in size from 4 - 60mb).

The vrt file has overviews built on it (which is obviously a rather large file), but on QGIS 1.8 it opens almost instantaneously once I have confirmed the CRS to use. I can also pan around and the screen refreshes instantly.

Attached is the output of gdalinfo for the vrt.

I also have other vrt files that are doing the same thing. So its not just the one file.

Taking one of the individual jpgs on its own (a 47mb file), 1.8 opens it in 17 secs, and master in 21 secs - so not a massive difference for a single file (and certainly not the most scientific of tests on my part).

Marked as a blocker as to me it seems like a regression from 1.8. Please change if needed.

gdalinfo_ouput.txt Magnifier (10.6 KB) James Stott, 2013-07-24 04:02 AM

unsupported_data_source.jpg (47.8 KB) James Stott, 2013-07-25 06:12 AM

Associated revisions

Revision 299d152e
Added by Radim Blazek over 11 years ago

Workaround for GDAL ticket 5170, fixes #8356

Revision 83d15a51
Added by Radim Blazek over 11 years ago

Disable VSI cache before GDALRasterIO, workaround for GDAL ticket 5170, fixes #8356

Revision 6a1a4be7
Added by Radim Blazek over 11 years ago

Disable VSI cache before GDALGetOverviewCount, workaround for GDAL ticket 5170, hopefully fixes #8356

History

#1 Updated by Nathan Woodrow over 11 years ago

Do you have raster icons turned on?

#2 Updated by James Stott over 11 years ago

I did have that turned on yes. Is that a default setting as I don't remember turning it on? Should it be the default setting if it slows things down?

But I have unticked it and master still displays the same behavior as before (unopened after 10 mins). So this was not the cause of the problem.

Is there any logging info I can supply to help find why this is happening?

#3 Updated by Radim Blazek over 11 years ago

What renderer type and contrast enhancement are set when it opens?

Ho fast is it if you switch of rendering (check off Render in status bar) before you open the raster?

How fast is the second display refresh (whole extent)?

How fast is it to render a small part (possibly an area just from a single tile file but from VRT).

Which pyramids (overviews) are listed in raster properties Pyramids tab?

How fast is it to open the overviews file alone (Stavanger_2011.vrt.ovr)?

Are you able to deduce from the debug log what is happening all that time.

#4 Updated by James Stott over 11 years ago

What renderer type and contrast enhancement are set when it opens?

I haven't been able to get layers properties to open to find out.

Ho fast is it if you switch of rendering (check off Render in status bar) before you open the raster?

Opens quickly and appears in the layer tree, but properties takes too long to open. I gave up after 10 mins.

How fast is the second display refresh (whole extent)?

Haven't been able to achieve this far with the whole extent.

How fast is it to render a small part (possibly an area just from a single tile file but from VRT).

It opened in just over 1 minute (the tiles are 800m x 600m). Pretty much the same rendering time if I move the map.

Which pyramids (overviews) are listed in raster properties Pyramids tab?

Properties wont open after 10 mins of waiting.

How fast is it to open the overviews file alone (Stavanger_2011.vrt.ovr)?

This generates a unsupported data source dialog (see attached jpg). Maybe this is the problem?

Are you able to deduce from the debug log what is happening all that time.

QGIS just freezes if I have rendering turned on, or if open layer properties, so I cannot see what the log says. When I get it to open zoomed right in, QGIS does not post any error messages to the error log.

I have been able to get some of my other data to open. I also created a vrt and overviews for some new data I have received and it worked fine in master.

I am currently rebuilding the vrt and overviews on the Stavanger 2011 file and will report back tomorrow (it will not be finished before I finish work today).

#5 Updated by Radim Blazek over 11 years ago

James Stott wrote:

What renderer type and contrast enhancement are set when it opens?

I haven't been able to get layers properties to open to find out.

Probably it takes too long to generate thumbnail.

Ho fast is it if you switch of rendering (check off Render in status bar) before you open the raster?

Opens quickly and appears in the layer tree, but properties takes too long to open. I gave up after 10 mins.

That means that rendering is slow, not calculating stats etc.

How fast is it to open the overviews file alone (Stavanger_2011.vrt.ovr)?

This generates a unsupported data source dialog (see attached jpg). Maybe this is the problem?

It seems to be. It looks like the ovr file is in a format which is not supported by your QGIS build, strange however that it is reported correctly by GDAL, is it using the same GDAL lib?

#6 Updated by James Stott over 11 years ago

It seems to be. It looks like the ovr file is in a format which is not supported by your QGIS build, strange however that it is reported correctly by GDAL, is it using the same GDAL lib?

It was using the OSGeo4W shell that I ran gdalinfo (which reports GDAL 1.9.2). If I go into the about dialog in QGIS, QGIS 1.8 and QGIS Master both state GDAL 1.9.2.

If I run GDAL info from Raster > Miscellaneous > Information here are the results:

QGIS 1.8 - reports the vrt files has overviews when trying to open the vrt file. The ovr file reports these too.
QGIS 1.9 - does not report overviews when querying the vrt file. The ovr file generates the same error in the attached jpg unsupported_data_source.jpg

The errors from master are:

ERROR 1: TIFFFetchDirectory:Sanity check on directory count failed, this is probably not a valid IFD offset
ERROR 1: TIFFReadDirectory:Failed to read directory at offset 17891712530

So is QGIS master running a different version of GDAL than 1.9.2?

The file works in QGIS v1.8 (and it opens in ArcMap 10.1 SP1), so there does seem to be a problem with the overview file specific to master.

Very strange.

#7 Updated by Radim Blazek over 11 years ago

James Stott wrote:

It was using the OSGeo4W shell that I ran gdalinfo (which reports GDAL 1.9.2). If I go into the about dialog in QGIS, QGIS 1.8 and QGIS Master both state GDAL 1.9.2.

Than it should be the same GDAL. In theory, the same versions of GDAL may be compiled with different drivers but if both versions of QGIS are installed via OSGeo4W, they should be using the same GDAL lib.

The errors from master are:

ERROR 1: TIFFFetchDirectory:Sanity check on directory count failed, this is probably not a valid IFD offset
ERROR 1: TIFFReadDirectory:Failed to read directory at offset 17891712530

So is QGIS master running a different version of GDAL than 1.9.2?

One difference came to mind. To improve performance of Shapefiles over network I added to the start up batch some GDAL VSI variables. Please try to disable them, comment or delete in your qgis-dev.bat the rows:

set VSI_CACHE=TRUE
set VSI_CACHE_SIZE=1000000

#8 Updated by James Stott over 11 years ago

One difference came to mind. To improve performance of Shapefiles over network I added to the start up batch some GDAL VSI variables. Please try to disable them, comment or delete in your qgis-dev.bat the rows:

set VSI_CACHE=TRUE
set VSI_CACHE_SIZE=1000000

You have hit the nail right on the head.

It works with these two lines commented out/deleted.

Seeing as we rarely use shapefiles, I shall leave them commented out.

#10 Updated by Radim Blazek over 11 years ago

  • Status changed from Open to Closed

#11 Updated by Radim Blazek over 11 years ago

It was fixed in GDAL: http://trac.osgeo.org/gdal/ticket/5170 for future 1.9.3 and 1.10.1.

I have fixed that also in QGIS as Even recommended.

James, please test again with next nightly build and qgis-dev.bat not modified (i.e. with VSI cache globally enabled).

#12 Updated by James Stott over 11 years ago

  • Status changed from Closed to Reopened

I updated master this morning (29/07/2013) and this has not fixed the problem. Commenting out the lines still works.

#13 Updated by Paolo Cavallini over 11 years ago

Please check which version you are actually using - it is possible that the fix came in later.

#14 Updated by James Stott over 11 years ago

I am using f779592 (stated as 1.9.0-345 in the OSGeo4W installer). I do not know if this is before or after the fix.

#15 Updated by Salvatore Larosa over 11 years ago

  • Status changed from Reopened to Closed

Please try with 299d152 or next, your revision is before the fix!

#16 Updated by James Stott over 11 years ago

Salvatore Larosa wrote:

Please try with 299d152 or next, your revision is before the fix!

Thanks. Will check once the update becomes available.

#17 Updated by James Stott over 11 years ago

  • Status changed from Closed to Reopened

Apologies about yesterdays false alarm. I have learnt something new about the way the versioning works :)

Today I have tried with a13cdf8, which I believe has the fix implemented.

Slightly different behaviour.

I cannot load the vrt file - I gave up after 10 mins of waiting.

But I can open the overview file fine (it loads instantly once I have selected a CRS).

Commenting out the two lines in the qgis-dev.bat file still allows me to open the vrt.

#18 Updated by Radim Blazek over 11 years ago

It means that Even was right, the VSI cache has to be disabled also before each read data call, GDAL does not keep the overview file open.

Easy fix, just to do the same what was done for GDALOpen also for GDALRasterIO.

I cannot fix that however before the 13th of August.

#19 Updated by Radim Blazek over 11 years ago

  • Status changed from Reopened to Closed

#20 Updated by James Stott over 11 years ago

Thanks Radim.

I will test once qgis-dev updates (hasn't updated for a few days now).

#21 Updated by James Stott over 11 years ago

  • Status changed from Closed to Reopened

Tested with 4e69997 just now. This does not fix opening of VRT files. It loads fine if you open the .ovr file directly, but not the vrt file. I closed QGIS down after waiting for 10 minutes.

Commenting out the VSI lines in the bat file enables the vrt file to load.

#22 Updated by Radim Blazek over 11 years ago

  • Status changed from Reopened to Closed

#23 Updated by Radim Blazek over 11 years ago

I found that OVR files is opened neither in GDALOpen nor in GDALRasterIO but in GDALGetOverviewCount so I wrapped that function as well.

James, please test once more, thanks for patience.

#24 Updated by James Stott over 11 years ago

Thanks for continuing to look at the problem!

This has now fixed the problem. Vrt files with ovr files now open instantly with master.

Also available in: Atom PDF