Bug report #19299

"Attribute Table" rows open randomly ordered?

Added by Loren Amelang over 1 year ago. Updated 7 months ago.

Status:Open
Priority:High
Assignee:-
Category:Attribute table
Affected QGIS version:3.4.5 Regression?:Yes
Operating System: Easy fix?:No
Pull Request or Patch supplied:No Resolution:
Crashes QGIS or corrupts data:No Copied to github as #:27127

Description

I've pretty much concluded the "Attribute Table" rows are randomly ordered unless you intentionally sort them - even if the actual shapefile is required to be in a specific order! Can this be?

The default order is not a problem of sorting numbers stored as text in the wrong numerical order. Typically 20 or 30 rows from near the top are relocated near the bottom, and almost all rows are randomized over a distance of maybe 10, as in 257, 251, 263, 262, 259, 266... If there is a pattern I don't see it.

If this is not a bug, could it be a feature request? When I view a shapefile I have created in a specific order, I'd like to see it displayed in that order!

DB Mgr, Attrib local+fresh, FreeView, Excel.JPG - Screenshot, various views of shp with 674 deleted (243 KB) Loren Amelang, 2018-06-30 01:26 AM

1st id1, ID0, ATid3.JPG (135 KB) Loren Amelang, 2018-07-01 12:14 AM

test_19299.geojson (908 Bytes) Andrea Giudiceandrea, 2019-03-19 07:44 PM

test_19299.gml (2.12 KB) Andrea Giudiceandrea, 2019-03-19 07:44 PM

test_19299_shp.zip (1.64 KB) Andrea Giudiceandrea, 2019-03-19 07:44 PM

test_19299.gpkg (96 KB) Andrea Giudiceandrea, 2019-03-19 07:44 PM

History

#1 Updated by Loren Amelang over 1 year ago

Still trying to understand this. ESRI and Wikipedia agree "In each of the .shp, .shx, and .dbf files, the shapes in each file correspond to each other in sequence (i.e., the first record in the .shp file corresponds to the first record in the .shx and .dbf files, etc.)." ESRI offers [[https://support.esri.com/en/technical-article/000012218]] - "How To: Permanently change the order of records in a feature class, shapefile, or table", but that is as close as I can find to an assertion that ID order should not randomly change. I know that when I was writing software to capture NDVI data in agricultural fields, we and all the vendors who used our data expected the order to remain as written, and the whole system would have collapsed if some software randomized the order.

I just did a test. See attached screenshot - DB Mgr, Attrib local+fresh, FreeView, Excel.JPG

--> I deleted ID = 674 in QGIS. Every view except Attribute Table shows the IDs in order, higher IDs moved up one line. Some (not shown) files have contiguous IDs of river reaches re-ordered, but only Attribute Tables show all ID positions randomized.

Geomatica-FreeView starts at item 0 = ID 1.
DB Mgr starts at item 1 = ID 1.
Excel (of the dbf file) starts at item 2 = ID 1.

Attribute Tables start at item 1 = ID 602. Random ID = 600s up to item 42, then ID = 100s, ID = 1-20 at lines 170+, more or less on up to ID = 120s, ID = 400s begin at 298, last item is ID = 601. I see no pattern. File freshly read from disk matches file in project.

So every view I have except the QGIS Attribute Table shows my data still in order. I might believe QGIS DB Manager and Geomatica-FreeView could sort the IDs before displaying them, but I'm pretty sure Excel just stupidly opened the file and shows the actual order.

Certainly seems to me the Attribute Tables are wrongly randomizing the data they present!

#2 Updated by Loren Amelang over 1 year ago

One more bit of this...

I'm looking at a little 5 item line shapefile I created manually. When I first looked at the attribute table it showed my order as expected. Somewhere in the process of saving, re-opening, or being used as a data source for other layers, it acquired a weird random sort. I just tried the Identify Feature function (as shown in the "1st id1, ID0, ATid3.JPG" screenshot) and saw the results summarized below:

Creation order (and sort order of almost all wkt_geom points)
    id
        Feature ID
            Excel id
                Attribute Table id
1   1   0   1   3
2   2   1   2   4
3   3   2   3   1
4   4   3   4   2
5   5   4   5   5

I can't find any explanation of why there are both id and Feature ID, but at least they are in the same numerical order. The default sort order of the Attribute table and the data copied from it are incomprehensible:

wkt_geom    id
MultiLineString ((6183809.76566736679524183 2245431.97494861111044884, 6183256.77749805059283972 2245082.22174750547856092))    3
MultiLineString ((6184164.24526308197528124 2244916.79793617175891995, 6183615.98348837532103062 2244543.41276201792061329))    4
MultiLineString ((6182987.37300530634820461 2245616.30433838302269578, 6182510.00714974291622639 2245450.88052704930305481))    1
MultiLineString ((6183143.34402742143720388 2245436.70134322065860033, 6182892.84511311631649733 2244968.78827687632292509))    2
MultiLineString ((6184740.86540544591844082 2244274.00826927414163947, 6183833.3976404145359993 2244198.38595552183687687))    5

I guess this makes no difference in this file - IF you know the initial order is random and meaningless and you need to sort by id to see the actual file content. It sorts nicely. But some of my files have had their id positions re-ordered according to river reaches they cross... If I sort them simply, to get rid of the randomness, I lose the intentional groupings! I could probably create a complex sort involving other files that would duplicate my intentional order, but the point of viewing the table is to verify that order has already been achieved!

#3 Updated by Alessandro Pasotti 7 months ago

  • Category changed from Browser to Attribute table

#4 Updated by Andrea Giudiceandrea 7 months ago

  • Regression? changed from No to Yes
  • Operating System deleted (Windows 10 Creators with 3000x2000 screen and 175% scaling)
  • Affected QGIS version changed from 3.2 to 3.4.5

I can confirm the same strange annoying behaviour using the latest QGIS ltr 3.4.5-1 (89ee6f6e23) and QGIS master 3.7.0-6 (d890084f63) on Windows 7 64 bit testing vector data-sets in various formats (shapefile, gml, geojson, gpkg, spatialite).

It doesn't happen with QGIS 2.18.23.

See also http://osgeo-org.1560.x6.nabble.com/QGIS-Developer-random-order-after-open-attribute-table-td5396860.html

#5 Updated by Giovanni Manghi 7 months ago

  • Priority changed from Normal to High

#6 Updated by Andrea Giudiceandrea 7 months ago

A test vector layer with 5 points, exported in some file formats.

For example, the geojson file contains:

{
"type": "FeatureCollection",
"name": "test_19299",
"crs": { "type": "name", "properties": { "name": "urn:ogc:def:crs:EPSG::32633" } },
"features": [
{ "type": "Feature", "properties": { "id": 0 }, "geometry": { "type": "Point", "coordinates": [ 726412.054477424127981, 4446266.452965155243874 ] } },
{ "type": "Feature", "properties": { "id": 1 }, "geometry": { "type": "Point", "coordinates": [ 616447.858150468440726, 4481093.48317492660135 ] } },
{ "type": "Feature", "properties": { "id": 2 }, "geometry": { "type": "Point", "coordinates": [ 653174.098219732986763, 4415418.181722722016275 ] } },
{ "type": "Feature", "properties": { "id": 3 }, "geometry": { "type": "Point", "coordinates": [ 710272.649218724109232, 4368354.675654544495046 ] } },
{ "type": "Feature", "properties": { "id": 4 }, "geometry": { "type": "Point", "coordinates": [ 598740.939244971377775, 4362665.847607292234898 ] } }
]
}

but the Attribute table will display the features in a random order after the layer is added to a map.

Also available in: Atom PDF