Bug report #19904
Processing: v.net tools (GRASS) return empty outputs
| Status: | Closed | ||
|---|---|---|---|
| Priority: | High | ||
| Assignee: |  Alexander Bruy | ||
| Category: | Processing/GRASS | ||
| Affected QGIS version: | 3.4.4 | Regression?: | Yes |
| Operating System: | Easy fix?: | No | |
| Pull Request or Patch supplied: | Yes | Resolution: | fixed/implemented |
| Crashes QGIS or corrupts data: | No | Copied to github as #: | 27728 |
Description
The tools do run, they produce an output but this is always empty.
This set of tools works ok on 2.18.
log_v_net_iso_qgis_2.18.28.txt 
(13 KB)
log_v_net_iso_qgis_3.4.4.txt
(13.5 KB)
new_line_qgis344.jpg (383 KB)
Associated revisions
[processing] snap points to network by default (fix #19904)
Kudos to Pedro Venâncio for finding solution for this bug
[processing] snap points to network by default (fix #19904)
Kudos to Pedro Venâncio for finding solution for this bug
(cherry picked from commit 3abea3600024c4d329403e7462a47af4db503339)
History
#1
Updated by Nyall Dawson over 5 years ago
- Status changed from Open to Feedback
Hi Giovanni,
Are you able to attach some layers which I can use to test this? I've hit issues with this algorithm too, but couldn't work out the magic formula to get it to work in 2.18 either...
#2
Updated by kajo kaji over 5 years ago
Seems a more general issue with GRASS output generation (see another case in #19923)
#3
Updated by Giovanni Manghi over 5 years ago
Nyall Dawson wrote:
Hi Giovanni,
Are you able to attach some layers which I can use to test this? I've hit issues with this algorithm too, but couldn't work out the magic formula to get it to work in 2.18 either...
here (test data and screenshot courtesy of Pedro Venancio): https://drive.google.com/file/d/1zkK6BRU1OWElqadHAXI6uIrzPsrm0z5l/view?usp=sharing
#4
Updated by Jürgen Fischer over 5 years ago
- Status changed from Feedback to Open
- Priority changed from High to Normal
#5
Updated by Pedro Venâncio over 5 years ago
- File log_v_net_iso_qgis_3.4.4.txt added
- File log_v_net_iso_qgis_2.18.28.txt added
This issue still happen in QGIS 3.4.4.
I attach log files of v.net.distance in QGIS 2.18.28 and 3.4.4.
The differences are mainly in
- QGIS 2.18.28
g.proj -c proj4="+proj=tmerc +lat_0=39.66825833333333 +lon_0=-8.133108333333334 +k=1 +x_0=0 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs" v.in.ogr min_area=0.0001 snap=-1 input="D:\SIG\dados_exercicio_analise_redes\dados_exercicio" layer=rede_viaria_distrito_porto_break output=tmp1548434142196 --overwrite -o v.in.ogr min_area=0.0001 snap=-1 input="D:\SIG\dados_exercicio_analise_redes\dados_exercicio" layer=origem output=tmp1548434142197 --overwrite -o v.in.ogr min_area=0.0001 snap=-1 input="D:\SIG\dados_exercicio_analise_redes\dados_exercicio" layer=destino output=tmp1548434142198 --overwrite -o g.region n=200265.166245 s=149098.806787 e=21170.1340706 w=-54560.428457 res=100 v.net -s input=tmp1548434142196 points=tmp1548434142197 out=tmp1548434142199 op=connect threshold=50 arc_layer=1 node_layer=2 v.db.connect -o map=tmp1548434142199 table=tmp1548434142197 layer=2 v.net -s input=tmp1548434142199 points=tmp1548434142198 out=tmp15484341421910 op=connect threshold=50 arc_layer=1 node_layer=3 v.db.connect -o map=tmp15484341421910 table=tmp1548434142198 layer=3 v.net.distance input="tmp15484341421910" arc_type=line to_type=point arc_column="dist_duplo" arc_backward_column="dist_unico" from_layer="2" to_layer="3" output="output8df48fb12fc54499af89b4d27d5d4ab5" --overwrite v.out.ogr -s -e input=output8df48fb12fc54499af89b4d27d5d4ab5 type=line output="C:\Users\PEDRO~1.VEN\AppData\Local\Temp\processing87ba215f31944c14a0caf114e3623877\a8bea1cd64aa4c8d92d60aad792298ca" format=ESRI_Shapefile output_layer=output --overwrite
C:\OSGeo4W64\bin>v.net -s input=tmp1548434142196 points=tmp1548434142197 out=tmp1548434142199 op=connect threshold=50 arc_layer=1 node_layer=2 Copying features... 2..5..8..11..14..17..20..23..26..29..32..35..38..41..44..47..50..53..56..59..62..65..68..71..74..77..80..83..86..89..92..95..98..100 Building topology for vector map ... Registering primitives... 10000..20000..30000..40000..50000..60000..70000..80000..90000.. 94621 primitives registered 484198 vertices registered Number of nodes: 70208 Number of primitives: 94621 Number of points: 0 Number of lines: 94621 Number of boundaries: 0 Number of centroids: 0 Number of areas: - Number of isles: - Copying attributes... Building topology for vector map ... Registering primitives... 10000..20000..30000..40000..50000..60000..70000..80000..90000.. 94623 primitives registered 484201 vertices registered Building areas... 0..2..4..6..8..10..12..14..16..18..20..22..24..26..28..30..32..34..36..38..40..42..44..46..48..50..52..54..56..58..60..62..64..66..68..70..72..74..76..78..80..82..84..86..88..90..92..94..96..98..100 0 areas built 0 isles built Attaching islands... Attaching centroids... 1..3..5..7..9..11..13..15..17..19..21..23..25..27..29..31..33..35..37..39..41..43..45..47..49..51..53..55..57..59..61..63..65..67..69..71..73..75..77..79..81..83..85..87..89..91..93..95..97..99..100 Number of nodes: 70209 Number of primitives: 94623 Number of points: 1 Number of lines: 94622 Number of boundaries: 0 Number of centroids: 0 Number of areas: 0 Number of isles: 0 v.net complete. 1 lines (network arcs) written to output.
C:\OSGeo4W64\bin>v.net -s input=tmp1548434142199 points=tmp1548434142198 out=tmp15484341421910 op=connect threshold=50 arc_layer=1 node_layer=3 Copying features... 2..5..8..11..14..17..20..23..26..29..32..35..38..41..44..47..50..53..56..59..62..65..68..71..74..77..80..83..86..89..92..95..98..100 Building topology for vector map ... Registering primitives... 10000..20000..30000..40000..50000..60000..70000..80000..90000.. 94623 primitives registered 484201 vertices registered Number of nodes: 70209 Number of primitives: 94623 Number of points: 1 Number of lines: 94622 Number of boundaries: 0 Number of centroids: 0 Number of areas: - Number of isles: - Copying attributes... Building topology for vector map ... Registering primitives... 10000..20000..30000..40000..50000..60000..70000..80000..90000.. 94625 primitives registered 484204 vertices registered Building areas... 0..2..4..6..8..10..12..14..16..18..20..22..24..26..28..30..32..34..36..38..40..42..44..46..48..50..52..54..56..58..60..62..64..66..68..70..72..74..76..78..80..82..84..86..88..90..92..94..96..98..100 0 areas built 0 isles built Attaching islands... Attaching centroids... 1..3..5..7..9..11..13..15..17..19..21..23..25..27..29..31..33..35..37..39..41..43..45..47..49..51..53..55..57..59..61..63..65..67..69..71..73..75..77..79..81..83..85..87..89..91..93..95..97..99..100 Number of nodes: 70210 Number of primitives: 94625 Number of points: 2 Number of lines: 94623 Number of boundaries: 0 Number of centroids: 0 Number of areas: 0 Number of isles: 0 v.net complete. 1 lines (network arcs) written to output.
- QGIS 3.4.4
g.proj -c proj4="+proj=tmerc +lat_0=39.66825833333333 +lon_0=-8.133108333333334 +k=1 +x_0=0 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs" v.in.ogr min_area=0.0001 snap=-1.0 input="D:\SIG\dados_exercicio_analise_redes\dados_exercicio\rede_viaria_distrito_porto_break.shp" output="vector_5c4b3aaa0594f8" --overwrite -o v.in.ogr min_area=0.0001 snap=-1.0 input="D:\SIG\dados_exercicio_analise_redes\dados_exercicio\origem.shp" output="vector_5c4b3aaa0594f9" --overwrite -o v.in.ogr min_area=0.0001 snap=-1.0 input="D:\SIG\dados_exercicio_analise_redes\dados_exercicio\destino.shp" output="vector_5c4b3aaa0594f10" --overwrite -o g.region n=200265.16624542486 s=149098.80678702804 e=21170.13407058691 w=-54560.42845699884 v.net input=vector_5c4b3aaa0594f8 points=vector_5c4b3aaa0594f9 output=bufnet5c4b3aaa05d3511 operation=connect threshold=50.0 arc_layer=1 node_layer=2 v.net input=bufnet5c4b3aaa05d3511 points=vector_5c4b3aaa0594f10 output=net5c4b3aaa05d3512 operation=connect threshold=50.0 arc_layer=1 node_layer=3 v.db.connect -o map=net5c4b3aaa05d3512 table=vector_5c4b3aaa0594f9 layer=2 v.db.connect -o map=net5c4b3aaa05d3512 table=vector_5c4b3aaa0594f10 layer=3 v.net.distance input=net5c4b3aaa05d3512 arc_type="line" to_type="point" arc_column=dist_duplo arc_backward_column=dist_unico from_layer="2" to_layer="3" output=output9bb50432020a447183c3ae5e59c42c53 --overwrite v.out.ogr type="line" input="output9bb50432020a447183c3ae5e59c42c53" output="C:/Users/pedro.venancio/AppData/Local/Temp/processing_cbc40bf994154e01a653beaeee87a30e/a06799eb354e4ce0ab76811d555656bf/output.gpkg" format="GPKG" layer=1 --overwrite
C:\OSGeo4W64\bin>v.net input=vector_5c4b3aaa0594f8 points=vector_5c4b3aaa0594f9 output=bufnet5c4b3aaa05d3511 operation=connect threshold=50.0 arc_layer=1 node_layer=2 Copying features... 2..5..8..11..14..17..20..23..26..29..32..35..38..41..44..47..50..53..56..59..62..65..68..71..74..77..80..83..86..89..92..95..98..100 A construir topologia para mapa vectorial <bufnet5c4b3aaa05d3511@PERMANENT>... Registando primitivas... 10000..20000..30000..40000..50000..60000..70000..80000..90000.. 94621 primitives registered 484198 vertices registered Número de nós: 70208 Número de primitivos: 94621 Número de pontos: 0 Número de linhas: 94621 Número de fronteiras: 0 Número de centróides: 0 Número de áreas: - Número de ilhas: - Copying attributes... A construir topologia para mapa vectorial <bufnet5c4b3aaa05d3511@PERMANENT>... Registando primitivas... 10000..20000..30000..40000..50000..60000..70000..80000..90000.. 94624 primitives registered 484203 vertices registered A construir áreas... 0..2..4..6..8..10..12..14..16..18..20..22..24..26..28..30..32..34..36..38..40..42..44..46..48..50..52..54..56..58..60..62..64..66..68..70..72..74..76..78..80..82..84..86..88..90..92..94..96..98..100 0 areas built 0 isles built A anexar ilhas... A anexar centróides... 1..3..5..7..9..11..13..15..17..19..21..23..25..27..29..31..33..35..37..39..41..43..45..47..49..51..53..55..57..59..61..63..65..67..69..71..73..75..77..79..81..83..85..87..89..91..93..95..97..99..100 Número de nós: 70210 Número de primitivos: 94624 Número de pontos: 1 Número de linhas: 94623 Número de fronteiras: 0 Número de centróides: 0 Número de áreas: 0 Número de ilhas: 0 v.net completo. 2 lines (network arcs) written to output.
C:\OSGeo4W64\bin>v.net input=bufnet5c4b3aaa05d3511 points=vector_5c4b3aaa0594f10 output=net5c4b3aaa05d3512 operation=connect threshold=50.0 arc_layer=1 node_layer=3 Copying features... 2..5..8..11..14..17..20..23..26..29..32..35..38..41..44..47..50..53..56..59..62..65..68..71..74..77..80..83..86..89..92..95..98..100 A construir topologia para mapa vectorial <net5c4b3aaa05d3512@PERMANENT>... Registando primitivas... 10000..20000..30000..40000..50000..60000..70000..80000..90000.. 94624 primitives registered 484203 vertices registered Número de nós: 70210 Número de primitivos: 94624 Número de pontos: 1 Número de linhas: 94623 Número de fronteiras: 0 Número de centróides: 0 Número de áreas: - Número de ilhas: - Copying attributes... A construir topologia para mapa vectorial <net5c4b3aaa05d3512@PERMANENT>... Registando primitivas... 10000..20000..30000..40000..50000..60000..70000..80000..90000.. 94627 primitives registered 484208 vertices registered A construir áreas... 0..2..4..6..8..10..12..14..16..18..20..22..24..26..28..30..32..34..36..38..40..42..44..46..48..50..52..54..56..58..60..62..64..66..68..70..72..74..76..78..80..82..84..86..88..90..92..94..96..98..100 0 areas built 0 isles built A anexar ilhas... A anexar centróides... 1..3..5..7..9..11..13..15..17..19..21..23..25..27..29..31..33..35..37..39..41..43..45..47..49..51..53..55..57..59..61..63..65..67..69..71..73..75..77..79..81..83..85..87..89..91..93..95..97..99..100 Número de nós: 70212 Número de primitivos: 94627 Número de pontos: 2 Número de linhas: 94625 Número de fronteiras: 0 Número de centróides: 0 Número de áreas: 0 Número de ilhas: 0 v.net completo. 2 lines (network arcs) written to output.
C:\OSGeo4W64\bin>v.net.distance input=net5c4b3aaa05d3512 arc_type="line" to_type="point" arc_column=dist_duplo arc_backward_column=dist_unico from_layer="2" to_layer="3" output=output9bb50432020a447183c3ae5e59c42c53 --overwrite A construir grafo... Registando arcos... 1..3..5..7..9..11..13..15..17..19..21..23..25..27..29..31..33..35..37..39..41..43..45..47..49..51..53..55..57..59..61..63..65..67..69..71..73..75..77..79..81..83..85..87..89..91..93..95..97..99..ATENÇÃO: Registo da base de dados para a linha 94622 (cat = 94622, em frente/ambas as direções) não encontrada (em frente/ambas as direções da linha ignoradas) ATENÇÃO: Registo da base de dados para a linha 94622(cat = 94622, direção reversa) não encontrada (direção da linha ignorada) ATENÇÃO: Registo da base de dados para a linha 94626 (cat = 94623, em frente/ambas as direções) não encontrada (em frente/ambas as direções da linha ignoradas) ATENÇÃO: Registo da base de dados para a linha 94626(cat = 94623, direção reversa) não encontrada (direção da linha ignorada) 100 Achatando o grafo... Grafo construído Distances to 'to' features ... Tracing paths from 'from' features ... A construir topologia para mapa vectorial <output9bb50432020a447183c3ae5e59c42c53@PERMANENT>... Registando primitivas... 0 primitives registered 0 vertices registered A construir áreas... 100 0 areas built 0 isles built A anexar ilhas... A anexar centróides... Número de nós: 0 Número de primitivos: 0 Número de pontos: 0 Número de linhas: 0 Número de fronteiras: 0 Número de centróides: 0 Número de áreas: 0 Número de ilhas: 0 ATENÇÃO: 1 'from' feature was not reachable
A complete dataset for testing v.net.* tools can be found here: http://qgis.pt/workshops_qgis2016/dados_exercicio_analise_redes.zip
And the presentation of the exercises are here (unfortunately they are in portuguese): http://qgis.pt/workshops_qgis2016/Analise_de_Redes_com_QGIS_PedroVenancio.pdf
#6
Updated by Giovanni Manghi over 5 years ago
- Affected QGIS version changed from 3.2.3 to 3.4.4
- Priority changed from Normal to High
#7
Updated by Alexander Bruy about 5 years ago
- Status changed from Open to Feedback
Am I right that empty output produced only when executed via Processing and correct results produced when executing same commands in GRASS directly?
#8
Updated by Pedro Venâncio about 5 years ago
Hi Alex,
It works with same data in QGIS 2.18.28, using same GRASS version (now 7.6.0).
I was looking more carefully, and the problem only happens when using arc_column and arc_backward_column parameters.
This works in QGIS 3.4.4:
v.net.distance input=net5c51d46329c936 arc_type="line,boundary" to_type="point" from_layer="2" to_layer="3" output=outputd3ae2de5a7344e9a93974c6bf994e8cf --overwrite
And this does not work (gives empty output), using exactly the same input data:
v.net.distance input=net5c51d4e3326c711 arc_type="line,boundary" to_type="point" arc_column=tmpo_duplo arc_backward_column=tmpo_unico from_layer="2" to_layer="3" output=outputbda141567f7d41a988a0db1ac74c663d --overwrite
Running both commands directly in GRASS 7.6.0, with intermediary data generated by QGIS 3.4.4 (v.in.ogr, g.region, v.net, v.db.connect - data saved in tmp folder) the output is the same, that is, an empty file using arc_column and arc_backward_column parameters, and the message:
WARNING: Registo da base de dados para a linha 94622 (cat = 94622, em frente/ambas as direções) não encontrada (em frente/ambas as direções da linha ignoradas) WARNING: Registo da base de dados para a linha 94622(cat = 94622, direção reversa) não encontrada (direção da linha ignorada) WARNING: Registo da base de dados para a linha 94626 (cat = 94623, em frente/ambas as direções) não encontrada (em frente/ambas as direções da linha ignoradas) WARNING: Registo da base de dados para a linha 94626(cat = 94623, direção reversa) não encontrada (direção da linha ignorada)
So, given that all this works in QGIS 2.18.28 (with same data and same GRASS version), the problem must be somewhere in v.net / v.db.connect intermediary operations? The order is different, I don't know if this is relevant, but could be here:
- QGIS 2.18.28 (v.net + v.db.connect + v.net + v.db.connect)
v.net -s input=tmp1548434142196 points=tmp1548434142197 out=tmp1548434142199 op=connect threshold=50 arc_layer=1 node_layer=2 v.db.connect -o map=tmp1548434142199 table=tmp1548434142197 layer=2 v.net -s input=tmp1548434142199 points=tmp1548434142198 out=tmp15484341421910 op=connect threshold=50 arc_layer=1 node_layer=3 v.db.connect -o map=tmp15484341421910 table=tmp1548434142198 layer=3 v.net.distance input="tmp15484341421910" arc_type=line to_type=point arc_column="dist_duplo" arc_backward_column="dist_unico" from_layer="2" to_layer="3" output="output8df48fb12fc54499af89b4d27d5d4ab5" --overwrite
- QGIS 3.4.4 (v.net + v.net + v.db.connect + v.db.connect)
v.net input=vector_5c4b3aaa0594f8 points=vector_5c4b3aaa0594f9 output=bufnet5c4b3aaa05d3511 operation=connect threshold=50.0 arc_layer=1 node_layer=2 v.net input=bufnet5c4b3aaa05d3511 points=vector_5c4b3aaa0594f10 output=net5c4b3aaa05d3512 operation=connect threshold=50.0 arc_layer=1 node_layer=3 v.db.connect -o map=net5c4b3aaa05d3512 table=vector_5c4b3aaa0594f9 layer=2 v.db.connect -o map=net5c4b3aaa05d3512 table=vector_5c4b3aaa0594f10 layer=3 v.net.distance input=net5c4b3aaa05d3512 arc_type="line" to_type="point" arc_column=dist_duplo arc_backward_column=dist_unico from_layer="2" to_layer="3" output=output9bb50432020a447183c3ae5e59c42c53 --overwrite
#9
Updated by Giovanni Manghi about 5 years ago
- Status changed from Feedback to Open
#10
Updated by Pedro Venâncio about 5 years ago
- File new_line_qgis344.jpg added
Ok, it seems the order is not relevant for this issue, because the same happens with v.net.iso, and there we have only one v.net + v.db.connect.
I was looking to the temporary grass mapsets created both by QGIS 2.18.28 and 3.4.4, and I think I've found the source of the problem.
For instance, in v.net.iso from QGIS 3.4.4, v.net + v.db.connect creates a network layer that have a new line, connecting the point layer to the lines (network) layer. This new line has no data in the attribute table. Please, see the image attached.
In QGIS 2.18.28, this new line is not created.
Then, when running v.net.* algorithms with arc_column and arc_backward_column parameters, this new line has null data, when numeric data are expected. And the error is shown:
WARNING: Registo da base de dados para a linha 94623 (cat = 94622, em frente/ambas as direções) não encontrada (em frente/ambas as direções da linha ignoradas) WARNING: Registo da base de dados para a linha 94623(cat = 94622, direção reversa) não encontrada (direção da linha ignorada)
Temporary mapsets can be downloaded from here, if you need test:
https://cld.pt/dl/download/3dbf4723-66df-44ed-9424-0cf2d495812e/grassdata_21828.zip
https://cld.pt/dl/download/4f3fef72-bdac-424c-a5f5-891324db3833/grassdata_344.zip
So, why is this new line created in QGIS 3.4.4?
#11
Updated by Pedro Venâncio about 5 years ago
I found it!
v.net needs the -s flag to snap points to network for operation connect. By default, v.net add a new line from the point to the network.
In 2.18.28, the -s flag is present.
#12
Updated by Pedro Venâncio about 5 years ago
Simply changing
\python\plugins\processing\algs\grass7\ext\v_net.py line 61 to
command = u"v.net -s input={} points={} output={} operation=connect threshold={}".format(\python\plugins\processing\algs\grass7\ext\v_net_distance.py lines 51 and 56 to
command = u"v.net -s input={} points={} output={} operation=connect threshold={} arc_layer=1 node_layer=2".format(command = u"v.net -s input={} points={} output={} operation=connect threshold={} arc_layer=1 node_layer=3".format(do the work.
Other v.net.* algorithms needs to be checked also.
Can someone apply this changes to master and backport to 3.4.x?
Thanks!
#13
Updated by Alexander Bruy about 5 years ago
- Assignee set to Alexander Bruy
- Status changed from Open to In Progress
Pedro Venâncio wrote:
I found it!
Thanks for your work! I will take care about updating code and backporting
#14
Updated by Giovanni Manghi about 5 years ago
- Pull Request or Patch supplied changed from No to Yes
- Status changed from In Progress to Open
- Assignee deleted (
Alexander Bruy)
Can someone apply this changes to master and backport to 3.4.x?
for master https://github.com/qgis/QGIS/pull/9044
#15
Updated by Giovanni Manghi about 5 years ago
Giovanni Manghi wrote:
Can someone apply this changes to master and backport to 3.4.x?
for master https://github.com/qgis/QGIS/pull/9044
#16
Updated by Alexander Bruy about 5 years ago
- Status changed from Open to In Progress
- Assignee set to Alexander Bruy
#17
Updated by Alexander Bruy about 5 years ago
- % Done changed from 0 to 100
- Status changed from In Progress to Closed
Applied in changeset qgis|3abea3600024c4d329403e7462a47af4db503339.
#18
Updated by Alexander Bruy about 5 years ago
- Resolution set to fixed/implemented