Bug report #4421

Raster are very slow to load when calculating stats

Added by Paolo Cavallini almost 8 years ago. Updated over 7 years ago.

Status:Closed
Priority:High
Assignee:-
Category:Rasters
Affected QGIS version:master Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:No Resolution:worksforme
Crashes QGIS or corrupts data:No Copied to github as #:14353

Description

Rasterlite layers, e.g. http://download.gfoss.it/TrueMarble/TrueMarble-2km.sqlite are extremely slow to open (I remember a while ago they were surprisingly fast); I noticed it uses only one CPU out of four, BTW

History

#1 Updated by Giovanni Manghi almost 8 years ago

  • Status changed from Open to Feedback

Not confirmed here with qgis-trunk on Ubuntu 11.10, the linked sample data is still lightning fast to open. Please leave feedback.

#2 Updated by Paolo Cavallini almost 8 years ago

Verified: on one machine it is slow, on another is fast. It might depend on some setting, still to understand which

#3 Updated by Paolo Cavallini almost 8 years ago

  • Status changed from Feedback to Closed
  • Resolution set to worksforme

I remember it was working with OSM layers, but I may be wrong

#4 Updated by Paolo Cavallini almost 8 years ago

  • Resolution deleted (worksforme)
  • Status changed from Closed to Feedback

#5 Updated by Alexander Bruy almost 8 years ago

Maybe this is because in one PC contrast enhancement is set to something different than "No Stretch" (see #3867). Try to set contrast enhancement to "No Stretch" and try again.

#6 Updated by Paolo Cavallini almost 8 years ago

The raster has a colour table, so stretching does not apply

#7 Updated by Giovanni Manghi almost 8 years ago

  • Target version set to Version 1.7.4

#8 Updated by Giovanni Manghi over 7 years ago

  • Crashes QGIS or corrupts data set to No
  • Affected QGIS version set to master

I cannot add rasterlite layers anymore (datasource not supported), support is broken in master? please leave feedback.

#9 Updated by Paolo Cavallini over 7 years ago

I can open it with c25ae56, it's just very slow.

#10 Updated by Giovanni Manghi over 7 years ago

Paolo Cavallini wrote:

I can open it with c25ae56, it's just very slow.

On another machine it works (forget it).

The source of slowness seems to be if you some standard deviation value active in the raster properties.

#11 Updated by Giovanni Manghi over 7 years ago

  • Priority changed from Normal to 6

Giovanni Manghi wrote:

Paolo Cavallini wrote:

I can open it with c25ae56, it's just very slow.

On another machine it works (forget it).

The source of slowness seems to be if you some standard deviation value active in the raster properties.

Ok, I found the pattern.

In the raster properties, style tab, there are two properties that is possible to save (diskette icon) in order to make them persistent between QGIS sessions. Those two properties are "contrast enhancement" and "use standard deviation".

First Note) the two are saved to "no stretch" and the other to not use standard deviation ----> the raster (tested with a TIFF and a Rasterlite, both 3 bands rasters) is loaded very fast as the rasters statistics are not computed and the ".xml" file (the statistic raster file) is not created.

Second Note) the standard deviation parameter alone cannot be saved (when "contrast enhancement" is set to "no stretch"). Try choose it, set it to a certain value, save clicking on the diskette icon and load again a raster, you will see that it will show always unselected.

Third Note) if the "contrast enhancement" parameter is set and saved to something else than "no stretch" then raster statistics are computed on raster load taking a long time depending on raster size (the ".xml" will be also created). When doing this the "use standard deviation" parameter is set (automatically and wrongly) to "2" leading also the raster to load with a wrong colortable. In this case it is also not possible to override the "use standard deviation" configuration as it will alwaus revert to a checked state with a value of "2". Once the ".xml" file is created the raster will load fast, but with the already "use standard deviation" issue.

#12 Updated by Giovanni Manghi over 7 years ago

Ok, I found the pattern.

The title of this ticket should be changed to something more appropriate after the recent findings.

#13 Updated by Paolo Cavallini over 7 years ago

  • Subject changed from Some raster are very slow to load (color table?) to Raster are very slow to load when calculating stats

#14 Updated by Paolo Cavallini over 7 years ago

See also #4862

#15 Updated by Giovanni Manghi over 7 years ago

  • Priority changed from 6 to High

I have made a series of tests and took a series of notes that right now I'm not able to translate in english.

From what I see there seems to be a few facts:

a) calculating stats is very slow for the attached rasterlite example (and probably other RL datasets), not for rasters in general

b) calculatings stats can take a while depending on raster size, but after the .xml file is created loading is lightning fast

c) stats are always computed for GRASS rasters (when adding them to the canvas), this because for this format the stretching is always enabled and when is enabled then QGIS computes stats. Moreover stats for GRASS rasters are apparently not saved anywhere.

d) it is not clear why it should be necessary to compute stats/stretching to show rasters "as they are" -> stats/stretching should be an optional choice of the user in case he/she needs to have a more detailed colormap.

#16 Updated by Paolo Cavallini over 7 years ago

  • Target version changed from Version 1.7.4 to Version 1.8.0

#17 Updated by Giovanni Manghi over 7 years ago

  • Resolution set to worksforme
  • Status changed from Feedback to Closed

Probably the changes made by Alexander recently have provided benefit for this issue as also for #4862.

The overall behaviour when loading a raster is much more better now. Stats are not forced to be computed on load even when stretching is enabled, as no standard deviation is used by default (that caused rasters to load with a "funny" colormap).

Loading the raster attached in this ticket is fast (now).

Computing histograms is kind of slow (but acceptable), but this seems an overall issue for all rasters but not a "blocker" issue.

Also available in: Atom PDF