QGIS » QGIS Application

I tried reproducing with current master_2 branch (2.17) but I don't even get an "id" field with QuickOSM and I dont know where to find "OpenStreetMap" plugin (plugin manager doesn't show it to me) - can you help me reproducing it ?

ref: http://wiki.openstreetmap.org/wiki/QGIS_OSM_Plugin

Just to avoid further confusion in the future:

- the OSM plugin mentioned in comment 7 is looong gone (probably removed before 2.0, I don't remember)

- there is OSM support in menu Vector > OpenStreetMap, which does support 64bit identifiers.

If you have found any issue with 64bit identifiers anyway, using any recent QGIS version, please reopen AND attach test data.

Giovanni could you reproduce this ? With which version ?
The reporter mentions 2.18.3 but we're if I'm not wrong at 2.18.7 now.
Andre did you test any more recent version ? Can you try to reproduce on a Linux host ?

Still the same issue with the test data using the 3-part Vector -> Openstreetmap menu entry on QGIS 2.18.7, on Linux Mint 18 (non-Ubuntugis).

Only some building polygons are generated, while the OSM tile background has more (those with 64-bit node numbers). Same for the roads. I still get the view from testosm.png.

With Add Vector layer you get the complete data, and with the QuickOSM plugin reading the test file as well.

Same result with QGIS-dev cd248c6 from OSGEO4W64.

Andre I'm trying to reproduce the problem you are seeing but I'm missing full instructions.
What I'm doing:
1. Start qgis
2. From the menu: Vector -> OpenStreeMap -> Import Topology from XML...
3. Select the `testosm.osm` file you attached to this ticket in the "Input XML file (.osm)" field
4. Hit 'OK' (receiving "Import has been sucessful" message)
5. Close the OpenStreetMap dialogs
6. From the menu: Layer -> Add Layer -> Add SpatiaLite Layer...
7. Select the spatialite layer created by the previous import procedure
8. Click on Connect

At the final step (connecting to the newly created spatialite database) I get no table listed.
If I check the "Also list tables with no geometry" checkbox on the left then I do get some tables.
I add the "nodes" and "ways" tables to the canvas, then I select each and open attribute table, counting the number of entries.
For both tables (nodes and ways) there's an exact match between the records in the attribute table and the number of records in the .osm file (checked with `grep 'way id' testosm.osm | wc -l` and `grep 'node id' testosm.osm | wc -l` and obtaining respectively 32 and 127).

This is with QGIS from the 2.18 branch at 44d1454dd0d553466e3c62324f3ac73713844dbf

Are you seeing anything different ? If so please give exact step-by-step instructions as I did above.
I'm probably missing something because you reference a screenshot showing a map, while I see no map at all !

Sandro, you missed the third part of the import: Vector -> OpenStreetMap -> Export to Spatialite...

That`s why you only have the geometryless tables of nodes, ways and relations (the OSM raw data). You need Point, Line and Multipolygon tables.

For a reference, you can use Layer -> Add Vector Layer on the testosm.osm file. It delivers 24 multipolygons, while Vector -> OpenStreetMap -> Export to Spatialite only delivers 8 multipolygons. Same for the Lines.

The background map is from the QuickMapServices plugin, OSM Standard map.

Andre, could you give instructions in the form I gave them in #10790-16? I'll try to figure out but it's a lot of time wasted if you don't give step-by-step instructions to reproduce

Alright what "Export topology to Spatialite..." does is not really an "export" as it just creates more tables in an existing Spatialite database, which is used both as input and output. After this step using "Add Spatialite layer" also gives less polygons than found by "Add Vector Layer".

I also noticed that using "Save as" on the layer loaded via "Add Vector Layer" from the .osm file (the layer with all the polygons) to an "sqlite" export, doesn't loose any geometry, so that re-opening the newly created sqlite database (via "Add Vector Layer") gives all geometries. This latter "sqlite" database is not a spatialite one though, so this seems to be some OGR abstraction

`ways` identifiers go from 26553528 to 471592927.
`nodes` identifiers go from 291056122 to 4657544184.
2^32 is 4294967296 so the problem of integer overflow could be with nodes, not ways.

Adding debug lines I see 32 ways being analyzed and 23 of them being considered invalid by the QgsOSMDatabase::exportSpatiaLite method. One is skip for being non-area. This leaves us with the 8 polygons as reported.

For comparison, "Add Vector Layers" finds a total of 28 polygons from those 32 ways.

Now, this should be the list of ways that have nodes with identifiers above 2^32:
spatialite> select w.id from ways w where exists ( select * from ways_nodes where way_id = w.id and node_id > pow(2,32) );
26553528
74240953
80219147
202972863
471587860
471587866
471587870
471587876
471587883
471587884 *
471587887 *
471587892
471587895
471587904
471587906
471592919
471592920
471592921
471592922 *
471592923 *
471592924 *
471592925 *
471592926 *
471592927 *

But the 8 polygons which are loaded are all in the above set (marked with an asterisk)

The kind of invalidity spot by QgsOSMDatabase::exportSpatiaLite is that wayPoints( w.id() ) returns an empty polyline which in turn happens because sqlite3_column_type returns SQLITE_NULL. It isn't clear to me what that means, anyway skipping those points rather than aborting the operation makes 22 polygons appear instead of just 8. Still a bit shorter than the OSM direct extraction, but looks like a step forward.

ogr2ogr (which works in the background of Add vector layer) stores all fields as VARCHAR, while the QGIS importer uses INTEGER for the id field and TEXT for the rest. I suggest to follow the ogr2ogr way and store the id as TEXT too.

If you need an integer field for a primary key, you may create your own (e.g. qgis_id) from just auto-counting through all features. Noone will import the whole world with QGIS to exceed the 2^32 limit of that counter.

This will prevent to have some nodes dropped, and get invalid geometries dropped as a consequence.

possibly related to #8878

Giovanni Manghi wrote:

possibly related to #8878

as also #12727