Bug report #8194

Heatmap plugin doesn't work for points along highway

Added by Aren Cambre over 6 years ago. Updated over 6 years ago.

Status:Closed
Priority:Normal
Assignee:Nyall Dawson
Category:C++ Plugins
Affected QGIS version:master Regression?:No
Operating System:Windows 7 x64 Easy fix?:No
Pull Request or Patch supplied:No Resolution:
Crashes QGIS or corrupts data:No Copied to github as #:17012

Description

I have a highway with a lot of points on it that represent crashes; this is actually a point layer with a filter.

These crashes occur more frequently around intersections, curves, entrances and exits, and more, and they are almost nonexistent in other areas. A heatmap will show clear, meaningful differences.

For reasons I cannot discern, the Heatmap plugin no longer works for these crashes. I set a radius of 500 meters, increased rows to 1500 (but may be 1499 due to #8193), and let columns be proportional.

I've attached a ZIP of the resulting TIF. Normally, when I make a heatmap, I see shades of gray, but this time, all I see is black. Furthermore, I can typically change to pseudocolor and just hit Classify for a spectral image. I can't in this case because there is nothing pre-filled in the Min field. Also, the Max field is bizarrely huge, signaling a problem. See weird min and max.png.

I've also attached screen shots of my QGIS view before and after making the heat map.

Increasing the Radius field to 1000m just gives a thicker black line and exact same problem with tying to go to pseudocolor.

Increasing the Radius field to 5000m and unchecking Advanced just gives an even fatter black line and still the exact same problem with tying to go to pseudocolor.

Yes, I am very sure I have the right Input point layer selected. The only other two point layers are not filtered, so a heatmap of them would show statewide crashes.

I-45_rural_areas_test.zip (21.4 KB) Aren Cambre, 2013-06-28 02:41 PM

before_heatmap.png (1.83 MB) Aren Cambre, 2013-06-28 02:41 PM

heatmap_is_just_black_stripe.png (1.82 MB) Aren Cambre, 2013-06-28 02:41 PM

weird_min_and_max.png (97.7 KB) Aren Cambre, 2013-06-28 02:41 PM

heat.tar.gz (770 KB) Giovanni Manghi, 2013-07-09 07:31 AM

Associated revisions

Revision 573efab5
Added by Nyall Dawson over 6 years ago

Fix calculation of heatmap raster size (fix #8194) - merged for further testing

Revision 7608faf6
Added by Nyall Dawson over 6 years ago

Skip missing geometries when calculating extent (fixes #8192, #8194)

Revision 6149b3a6
Added by Nyall Dawson over 6 years ago

Merge pull request #789 from nyalldawson/bug8192

Fix bounding box calculation for features with NULL geometry (fixes #8192, #8194)

History

#1 Updated by Aren Cambre over 6 years ago

I closed qgis, removed the .qgis2 directory, and started it back up. The heatmap started working again at default settings. So I gradually bumped the heatmap down and down until I got to 500 meters, and I was still getting meaningful information. Only when I checked Advanced and changed the Rows field to 1500 (I let Columns scale proportionally) did I start getting the bad heatmaps. Unchecking Advanced does not fix this.

BTW, I am on c8d3fa7.

#2 Updated by Nyall Dawson over 6 years ago

Can you share the source layer you are using for this?

#3 Updated by Nyall Dawson over 6 years ago

  • Assignee set to Nyall Dawson

#4 Updated by Aren Cambre over 6 years ago

It's a PostGIS layer, but QGIS won't let me export it due to #8101.

#5 Updated by Aren Cambre over 6 years ago

Would you take an SQL export of a PostGIS 2.0/Postgres 9.2 table?

#6 Updated by Giovanni Manghi over 6 years ago

Aren Cambre wrote:

Would you take an SQL export of a PostGIS 2.0/Postgres 9.2 table?

yes thanks, so we can test also the other issue.

#7 Updated by Aren Cambre over 6 years ago

Well, this is weird.

I've modified my dataset by combining 3 similar ones and reducing the number of columns. So instead of 148-170 columns (depending on which year's data you're looking at), I merged the 3 datasets into one table with 9 columns. I loaded that, applied the filter, and got the same problem. The columns were of type:
  • integer (ID column)
  • data
  • time without time zone
  • character varying(10)*
  • character varying(4)*
  • integer
  • integer*
  • boolean
  • geometry(Point, 3081)

\\*Was used to filter the dataset.

Then I further restricted the dataset by removing all columns except id and geom and only including rows that match the query, so the resulting, unfiltered dataset has 2 columns and 9874 rows. The heatmap works now, but the related issue (bug #8101) remains broken.

Why am I talking about this? There's a lot of the data that, for complex reasons, I cannot disclose. In the process of stripping the data down to both a manageable dataset and the portion thereof that I can disclose, I seem to have eliminated the problem. It's almost as if there's something about the presence of these other columns or many unfiltered rows in the original data that's tripping up the Heatmap plugin.

#8 Updated by Giovanni Manghi over 6 years ago

Aren Cambre wrote:

Well, this is weird.

I've modified my dataset by combining 3 similar ones and reducing the number of columns. So instead of 148-170 columns (depending on which year's data you're looking at), I merged the 3 datasets into one table with 9 columns. I loaded that, applied the filter, and got the same problem. The columns were of type:
  • integer (ID column)
  • data
  • time without time zone
  • character varying(10)*
  • character varying(4)*
  • integer
  • integer*
  • boolean
  • geometry(Point, 3081)

\\*Was used to filter the dataset.

Then I further restricted the dataset by removing all columns except id and geom and only including rows that match the query, so the resulting, unfiltered dataset has 2 columns and 9874 rows. The heatmap works now, but the related issue (bug #8101) remains broken.

Why am I talking about this? There's a lot of the data that, for complex reasons, I cannot disclose. In the process of stripping the data down to both a manageable dataset and the portion thereof that I can disclose, I seem to have eliminated the problem. It's almost as if there's something about the presence of these other columns or many unfiltered rows in the original data that's tripping up the Heatmap plugin.

Can't you apply and undisclosed (or random) shift to your points and fill the columns with random junk and then share the dataset?

#9 Updated by Aren Cambre over 6 years ago

Got it! Can you email me directly at ? I'd like discuss transferring the data outside this issue.

#10 Updated by Giovanni Manghi over 6 years ago

  • Status changed from Open to Feedback
  • File heat.tar.gz added

Aren Cambre wrote:

Got it! Can you email me directly at ? I'd like discuss transferring the data outside this issue.

I have tested you dataset, with the filter you sent me, using the heatmap tool back and forth, using advanced options or not, and always got a meaningful result, I think. I attach here a few samples. Please have a look and leave feedback.

#11 Updated by Aren Cambre over 6 years ago

Nyall was able to replicate the issue. (He told me over email instead of posting here.)

He has a proposed fix, although it looks like I'd have to compile, and it may be a few days before I can get around to that unless his fix can be pushed to master. I can refresh master using osgeo4w.

#12 Updated by Nyall Dawson over 6 years ago

Proposed fix is here:
https://github.com/nyalldawson/Quantum-GIS/commit/8d7a686a86961e1642176d91ad3a883b8b92d3b7

I'm going to double check it and run it through some more tests before committing.

#13 Updated by Aren Cambre over 6 years ago

Nyall, what happens when you press Zoom to Layer after applying my provided filter to that layer? If it doesn't zoom to fit that layer on the screen, then you may be another person to reproduce #8192. It would be interesting to see your findings.

#14 Updated by Nyall Dawson over 6 years ago

I've merged this fix to master for further testing -- would appreciate some feedback if this fixes the issue!

#15 Updated by Aren Cambre over 6 years ago

What daily release of master is this in? I'm on f4ac684 and don't see a change.

#16 Updated by Nyall Dawson over 6 years ago

It should hit the osgeo4w build today - f4ac684 doesn't have this fix applied.

#17 Updated by Aren Cambre over 6 years ago

Just updated to b1efd47. Same problem.

#18 Updated by Giovanni Manghi over 6 years ago

Aren Cambre wrote:

Just updated to b1efd47. Same problem.

I can't see Nyall commit

https://github.com/qgis/Quantum-GIS/commits/master

so I would not expect to see any difference, yet.

#19 Updated by Nyall Dawson over 6 years ago

No - it's in there (573efab5a87278d6795412cb3b4de65b5d259a3f) ... but it's not working for me either now. I'll have to look into it more.

#20 Updated by Nyall Dawson over 6 years ago

Think I've finally sorted this one out - pull request #789 fixes it.

It's caused by the filter returning records with null geometry - bug #8192 meant qgis was incorrectly calculating the bounding box for the resultant layer, thus breaking the heatmap.

If a project was saved with the filter, then loading the project caused QGIS to calculate the extent using the provider, resulting in a correct bounding box -- this led me to believe the earlier fix was working, when in reality it was just because I was working from a project with the filter applied.

#21 Updated by Nyall Dawson over 6 years ago

  • Status changed from Feedback to Closed

Also available in: Atom PDF