Bug report #16759: QGIS shows incomplete data when opening a .pbf/.osm file - QGIS Application - QGIS Issue Tracking

Bug report #16759

QGIS shows incomplete data when opening a .pbf/.osm file

Added by Kinakh Dmytro over 7 years ago. Updated about 6 years ago.

Status:	Closed
Priority:	Normal
Assignee:	-
Category:	Data Provider/OGR
Affected QGIS version:	master	Regression?:	No
Operating System:	Windows 10 x64	Easy fix?:	No
Pull Request or Patch supplied:	No	Resolution:
Crashes QGIS or corrupts data:	No	Copied to github as #:	24658

Description

Hi support team,

I'm trying to use QGIS to open Florida dataset (osm.pbf file) to get some data for my startup.
Link to source file - http://download.geofabrik.de/north-america.html
But I'm getting a lot of missed information as there are a lot of gaps - http://iddmitriko.s3.amazonaws.com/screens/3dm12.png (I've added Florida state shape to show how much is missed in terms of streets).

I can't use shp file directly from source I mentioned as there is a lot of information which I need and it's present in OSM file and not present in SHP.

Info: I'm using lates version of QGIS on my Windows machine - http://take.ms/MlZsF

Could you please adivese me what I can try to get that file open properly on QGIS so I'll have a chance to save it as shp.

florida_osm_lines.JPG (20.6 KB) Stephen Knox, 2018-09-27 10:31 AM

florida.png (427 KB) Stephen Knox, 2018-09-30 02:46 AM

Associated revisions

Revision f410b4a4
Added by Stephen Knox about 6 years ago

Fixes #16759 - ensures OSM layers use OGR random layer read/write

To guarantee all features are rendered with large files

History

#1 Updated by Giovanni Manghi over 7 years ago

Status changed from Open to Feedback
Subject changed from QGIS shows holes if I open osm.pbf file to QGIS shows incomplete data when opening a .pbf/.osm file
Category changed from Browser to Vectors
Priority changed from High to Normal
Crashes QGIS or corrupts data changed from Yes to No

I the issue also on Linux also with the .osm format, however translating to Spatialite or PostGIS is ok.

I smell a problem in the GDAL/OGR driver for obf/osm rather than QGIS.

#2 Updated by Giovanni Manghi over 6 years ago

Status changed from Feedback to Open

Giovanni Manghi wrote:

I the issue also on Linux also with the .osm format, however translating to Spatialite or PostGIS is ok.

I smell a problem in the GDAL/OGR driver for obf/osm rather than QGIS.

regardless is not nice, and still happens in the latest build of QGIS 3.

#3 Updated by Stephen Knox about 6 years ago

This is a bit of a punt, but could this be happening because GDAL switched the OSM driver to enable reading in random order (see https://trac.osgeo.org/gdal/wiki/rfc66_randomlayerreadwrite and https://www.gdal.org/drv_osm.html). I wonder if the QGIS code should be calling GDALDatasetGetNextFeature rather than OGR_L_GetNextFeature as it does at present?

#4 Updated by Nyall Dawson about 6 years ago

Status changed from Open to Feedback

Please try with latest 3.3 master -- a fix was recently committed relating to OSM data display.

#5 Updated by Stephen Knox about 6 years ago

File florida_osm_lines.JPG added

There doesn't seem to be any change. The attached file is with master 80723e89fd on Windows 10. I also tried on Linux with the same result.

Interestingly, I did also try with a smaller file (> 10x smaller - http://download.geofabrik.de/europe/great-britain/england/hertfordshire-latest.osm.pbf) which did not appear to give any gaps. I then did Bugaria (~90MB) which did have some gaps. I did turn on GDAL debugging for the Linux session and was getting a lot of errors like only got 3 of 8 nodes for way xxxx, which made me think it was a sequential ordering issue or memory. Perhaps the nodes table is not fully loaded into memory where there are a lot of nodes.

I think the idea above would mitigate that from what I understand about OSM file structure.

This does sort of beg the question is QGIS the right tool to use to load large OSM datasets (panning and zooming is not quick) but I think as a "shop window" for QGIS it should at least be correct, if not necessarily fast.