Bug report #19904

Processing: v.net tools (GRASS) return empty outputs

Added by Giovanni Manghi about 6 years ago. Updated almost 6 years ago.

Status:Closed
Priority:High
Assignee:Alexander Bruy
Category:Processing/GRASS
Affected QGIS version:3.4.4 Regression?:Yes
Operating System: Easy fix?:No
Pull Request or Patch supplied:Yes Resolution:fixed/implemented
Crashes QGIS or corrupts data:No Copied to github as #:27728

Description

The tools do run, they produce an output but this is always empty.

This set of tools works ok on 2.18.

log_v_net_iso_qgis_2.18.28.txt Magnifier (13 KB) Pedro Venâncio, 2019-01-25 06:17 PM

log_v_net_iso_qgis_3.4.4.txt Magnifier (13.5 KB) Pedro Venâncio, 2019-01-25 06:17 PM

new_line_qgis344.jpg (383 KB) Pedro Venâncio, 2019-01-31 12:54 AM

Associated revisions

Revision 3abea360
Added by Alexander Bruy almost 6 years ago

[processing] snap points to network by default (fix #19904)

Kudos to Pedro Venâncio for finding solution for this bug

Revision ecd306e0
Added by Alexander Bruy almost 6 years ago

Merge pull request #9046 from alexbruy/grass-v-net

fix GRASS v.net modules (fix #19904) [processing]

Revision 0da67d27
Added by Alexander Bruy almost 6 years ago

[processing] snap points to network by default (fix #19904)

Kudos to Pedro Venâncio for finding solution for this bug

(cherry picked from commit 3abea3600024c4d329403e7462a47af4db503339)

History

#1 Updated by Nyall Dawson about 6 years ago

  • Status changed from Open to Feedback

Hi Giovanni,

Are you able to attach some layers which I can use to test this? I've hit issues with this algorithm too, but couldn't work out the magic formula to get it to work in 2.18 either...

#2 Updated by kajo kaji about 6 years ago

Seems a more general issue with GRASS output generation (see another case in #19923)

#3 Updated by Giovanni Manghi about 6 years ago

Nyall Dawson wrote:

Hi Giovanni,

Are you able to attach some layers which I can use to test this? I've hit issues with this algorithm too, but couldn't work out the magic formula to get it to work in 2.18 either...

here (test data and screenshot courtesy of Pedro Venancio): https://drive.google.com/file/d/1zkK6BRU1OWElqadHAXI6uIrzPsrm0z5l/view?usp=sharing

#4 Updated by Jürgen Fischer almost 6 years ago

  • Status changed from Feedback to Open
  • Priority changed from High to Normal

#5 Updated by Pedro Venâncio almost 6 years ago

This issue still happen in QGIS 3.4.4.

I attach log files of v.net.distance in QGIS 2.18.28 and 3.4.4.

The differences are mainly in

- QGIS 2.18.28

g.proj -c proj4="+proj=tmerc +lat_0=39.66825833333333 +lon_0=-8.133108333333334 +k=1 +x_0=0 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs" 
v.in.ogr min_area=0.0001 snap=-1 input="D:\SIG\dados_exercicio_analise_redes\dados_exercicio" layer=rede_viaria_distrito_porto_break output=tmp1548434142196 --overwrite -o
v.in.ogr min_area=0.0001 snap=-1 input="D:\SIG\dados_exercicio_analise_redes\dados_exercicio" layer=origem output=tmp1548434142197 --overwrite -o
v.in.ogr min_area=0.0001 snap=-1 input="D:\SIG\dados_exercicio_analise_redes\dados_exercicio" layer=destino output=tmp1548434142198 --overwrite -o
g.region n=200265.166245 s=149098.806787 e=21170.1340706 w=-54560.428457 res=100
v.net -s input=tmp1548434142196 points=tmp1548434142197 out=tmp1548434142199 op=connect threshold=50 arc_layer=1 node_layer=2
v.db.connect -o map=tmp1548434142199 table=tmp1548434142197 layer=2
v.net -s input=tmp1548434142199 points=tmp1548434142198 out=tmp15484341421910 op=connect threshold=50 arc_layer=1 node_layer=3
v.db.connect -o map=tmp15484341421910 table=tmp1548434142198 layer=3
v.net.distance input="tmp15484341421910" arc_type=line to_type=point arc_column="dist_duplo" arc_backward_column="dist_unico" from_layer="2" to_layer="3" output="output8df48fb12fc54499af89b4d27d5d4ab5" --overwrite
v.out.ogr -s -e input=output8df48fb12fc54499af89b4d27d5d4ab5 type=line output="C:\Users\PEDRO~1.VEN\AppData\Local\Temp\processing87ba215f31944c14a0caf114e3623877\a8bea1cd64aa4c8d92d60aad792298ca" format=ESRI_Shapefile output_layer=output --overwrite
C:\OSGeo4W64\bin>v.net -s input=tmp1548434142196 points=tmp1548434142197 out=tmp1548434142199 op=connect threshold=50 arc_layer=1 node_layer=2 
Copying features... 
2..5..8..11..14..17..20..23..26..29..32..35..38..41..44..47..50..53..56..59..62..65..68..71..74..77..80..83..86..89..92..95..98..100 
Building topology for vector map ... 
Registering primitives... 
10000..20000..30000..40000..50000..60000..70000..80000..90000.. 
94621 primitives registered 
484198 vertices registered 
Number of nodes: 70208 
Number of primitives: 94621 
Number of points: 0 
Number of lines: 94621 
Number of boundaries: 0 
Number of centroids: 0 
Number of areas: - 
Number of isles: - 
Copying attributes... 
Building topology for vector map ... 
Registering primitives... 
10000..20000..30000..40000..50000..60000..70000..80000..90000.. 
94623 primitives registered 
484201 vertices registered 
Building areas... 
0..2..4..6..8..10..12..14..16..18..20..22..24..26..28..30..32..34..36..38..40..42..44..46..48..50..52..54..56..58..60..62..64..66..68..70..72..74..76..78..80..82..84..86..88..90..92..94..96..98..100 
0 areas built 
0 isles built 
Attaching islands... 
Attaching centroids... 
1..3..5..7..9..11..13..15..17..19..21..23..25..27..29..31..33..35..37..39..41..43..45..47..49..51..53..55..57..59..61..63..65..67..69..71..73..75..77..79..81..83..85..87..89..91..93..95..97..99..100 
Number of nodes: 70209 
Number of primitives: 94623 
Number of points: 1 
Number of lines: 94622 
Number of boundaries: 0 
Number of centroids: 0 
Number of areas: 0 
Number of isles: 0 
v.net complete. 1 lines (network arcs) written to output. 
C:\OSGeo4W64\bin>v.net -s input=tmp1548434142199 points=tmp1548434142198 out=tmp15484341421910 op=connect threshold=50 arc_layer=1 node_layer=3 
Copying features... 
2..5..8..11..14..17..20..23..26..29..32..35..38..41..44..47..50..53..56..59..62..65..68..71..74..77..80..83..86..89..92..95..98..100 
Building topology for vector map ... 
Registering primitives... 
10000..20000..30000..40000..50000..60000..70000..80000..90000.. 
94623 primitives registered 
484201 vertices registered 
Number of nodes: 70209 
Number of primitives: 94623 
Number of points: 1 
Number of lines: 94622 
Number of boundaries: 0 
Number of centroids: 0 
Number of areas: - 
Number of isles: - 
Copying attributes... 
Building topology for vector map ... 
Registering primitives... 
10000..20000..30000..40000..50000..60000..70000..80000..90000.. 
94625 primitives registered 
484204 vertices registered 
Building areas... 
0..2..4..6..8..10..12..14..16..18..20..22..24..26..28..30..32..34..36..38..40..42..44..46..48..50..52..54..56..58..60..62..64..66..68..70..72..74..76..78..80..82..84..86..88..90..92..94..96..98..100 
0 areas built 
0 isles built 
Attaching islands... 
Attaching centroids... 
1..3..5..7..9..11..13..15..17..19..21..23..25..27..29..31..33..35..37..39..41..43..45..47..49..51..53..55..57..59..61..63..65..67..69..71..73..75..77..79..81..83..85..87..89..91..93..95..97..99..100 
Number of nodes: 70210 
Number of primitives: 94625 
Number of points: 2 
Number of lines: 94623 
Number of boundaries: 0 
Number of centroids: 0 
Number of areas: 0 
Number of isles: 0 
v.net complete. 1 lines (network arcs) written to output.

- QGIS 3.4.4

g.proj -c proj4="+proj=tmerc +lat_0=39.66825833333333 +lon_0=-8.133108333333334 +k=1 +x_0=0 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs" 
v.in.ogr min_area=0.0001 snap=-1.0 input="D:\SIG\dados_exercicio_analise_redes\dados_exercicio\rede_viaria_distrito_porto_break.shp" output="vector_5c4b3aaa0594f8" --overwrite -o
v.in.ogr min_area=0.0001 snap=-1.0 input="D:\SIG\dados_exercicio_analise_redes\dados_exercicio\origem.shp" output="vector_5c4b3aaa0594f9" --overwrite -o
v.in.ogr min_area=0.0001 snap=-1.0 input="D:\SIG\dados_exercicio_analise_redes\dados_exercicio\destino.shp" output="vector_5c4b3aaa0594f10" --overwrite -o
g.region n=200265.16624542486 s=149098.80678702804 e=21170.13407058691 w=-54560.42845699884
v.net input=vector_5c4b3aaa0594f8 points=vector_5c4b3aaa0594f9 output=bufnet5c4b3aaa05d3511 operation=connect threshold=50.0 arc_layer=1 node_layer=2
v.net input=bufnet5c4b3aaa05d3511 points=vector_5c4b3aaa0594f10 output=net5c4b3aaa05d3512 operation=connect threshold=50.0 arc_layer=1 node_layer=3
v.db.connect -o map=net5c4b3aaa05d3512 table=vector_5c4b3aaa0594f9 layer=2
v.db.connect -o map=net5c4b3aaa05d3512 table=vector_5c4b3aaa0594f10 layer=3
v.net.distance input=net5c4b3aaa05d3512 arc_type="line" to_type="point" arc_column=dist_duplo arc_backward_column=dist_unico from_layer="2" to_layer="3" output=output9bb50432020a447183c3ae5e59c42c53 --overwrite
v.out.ogr type="line" input="output9bb50432020a447183c3ae5e59c42c53" output="C:/Users/pedro.venancio/AppData/Local/Temp/processing_cbc40bf994154e01a653beaeee87a30e/a06799eb354e4ce0ab76811d555656bf/output.gpkg" format="GPKG" layer=1 --overwrite
C:\OSGeo4W64\bin>v.net input=vector_5c4b3aaa0594f8 points=vector_5c4b3aaa0594f9 output=bufnet5c4b3aaa05d3511 operation=connect threshold=50.0 arc_layer=1 node_layer=2
Copying features...
2..5..8..11..14..17..20..23..26..29..32..35..38..41..44..47..50..53..56..59..62..65..68..71..74..77..80..83..86..89..92..95..98..100
A construir topologia para mapa vectorial <bufnet5c4b3aaa05d3511@PERMANENT>...
Registando primitivas...
10000..20000..30000..40000..50000..60000..70000..80000..90000..
94621 primitives registered
484198 vertices registered
Número de nós: 70208
Número de primitivos: 94621
Número de pontos: 0
Número de linhas: 94621
Número de fronteiras: 0
Número de centróides: 0
Número de áreas: -
Número de ilhas: -
Copying attributes...
A construir topologia para mapa vectorial <bufnet5c4b3aaa05d3511@PERMANENT>...
Registando primitivas...
10000..20000..30000..40000..50000..60000..70000..80000..90000..
94624 primitives registered
484203 vertices registered
A construir áreas...
0..2..4..6..8..10..12..14..16..18..20..22..24..26..28..30..32..34..36..38..40..42..44..46..48..50..52..54..56..58..60..62..64..66..68..70..72..74..76..78..80..82..84..86..88..90..92..94..96..98..100
0 areas built
0 isles built
A anexar ilhas...
A anexar centróides...
1..3..5..7..9..11..13..15..17..19..21..23..25..27..29..31..33..35..37..39..41..43..45..47..49..51..53..55..57..59..61..63..65..67..69..71..73..75..77..79..81..83..85..87..89..91..93..95..97..99..100
Número de nós: 70210
Número de primitivos: 94624
Número de pontos: 1
Número de linhas: 94623
Número de fronteiras: 0
Número de centróides: 0
Número de áreas: 0
Número de ilhas: 0
v.net completo. 2 lines (network arcs) written to output.
C:\OSGeo4W64\bin>v.net input=bufnet5c4b3aaa05d3511 points=vector_5c4b3aaa0594f10 output=net5c4b3aaa05d3512 operation=connect threshold=50.0 arc_layer=1 node_layer=3
Copying features...
2..5..8..11..14..17..20..23..26..29..32..35..38..41..44..47..50..53..56..59..62..65..68..71..74..77..80..83..86..89..92..95..98..100
A construir topologia para mapa vectorial <net5c4b3aaa05d3512@PERMANENT>...
Registando primitivas...
10000..20000..30000..40000..50000..60000..70000..80000..90000..
94624 primitives registered
484203 vertices registered
Número de nós: 70210
Número de primitivos: 94624
Número de pontos: 1
Número de linhas: 94623
Número de fronteiras: 0
Número de centróides: 0
Número de áreas: -
Número de ilhas: -
Copying attributes...
A construir topologia para mapa vectorial <net5c4b3aaa05d3512@PERMANENT>...
Registando primitivas...
10000..20000..30000..40000..50000..60000..70000..80000..90000..
94627 primitives registered
484208 vertices registered
A construir áreas...
0..2..4..6..8..10..12..14..16..18..20..22..24..26..28..30..32..34..36..38..40..42..44..46..48..50..52..54..56..58..60..62..64..66..68..70..72..74..76..78..80..82..84..86..88..90..92..94..96..98..100
0 areas built
0 isles built
A anexar ilhas...
A anexar centróides...
1..3..5..7..9..11..13..15..17..19..21..23..25..27..29..31..33..35..37..39..41..43..45..47..49..51..53..55..57..59..61..63..65..67..69..71..73..75..77..79..81..83..85..87..89..91..93..95..97..99..100
Número de nós: 70212
Número de primitivos: 94627
Número de pontos: 2
Número de linhas: 94625
Número de fronteiras: 0
Número de centróides: 0
Número de áreas: 0
Número de ilhas: 0
v.net completo. 2 lines (network arcs) written to output.
C:\OSGeo4W64\bin>v.net.distance input=net5c4b3aaa05d3512 arc_type="line" to_type="point" arc_column=dist_duplo arc_backward_column=dist_unico from_layer="2" to_layer="3" output=output9bb50432020a447183c3ae5e59c42c53 --overwrite
A construir grafo...
Registando arcos...
1..3..5..7..9..11..13..15..17..19..21..23..25..27..29..31..33..35..37..39..41..43..45..47..49..51..53..55..57..59..61..63..65..67..69..71..73..75..77..79..81..83..85..87..89..91..93..95..97..99..ATENÇÃO: Registo da base de dados para a linha 94622 (cat = 94622, em frente/ambas as direções) não encontrada (em frente/ambas as direções da linha ignoradas)
ATENÇÃO: Registo da base de dados para a linha 94622(cat = 94622, direção reversa) não encontrada (direção da linha ignorada)
ATENÇÃO: Registo da base de dados para a linha 94626 (cat = 94623, em frente/ambas as direções) não encontrada (em frente/ambas as direções da linha ignoradas)
ATENÇÃO: Registo da base de dados para a linha 94626(cat = 94623, direção reversa) não encontrada (direção da linha ignorada)
100
Achatando o grafo...
Grafo construído
Distances to 'to' features ...
Tracing paths from 'from' features ...
A construir topologia para mapa vectorial <output9bb50432020a447183c3ae5e59c42c53@PERMANENT>...
Registando primitivas...
0 primitives registered
0 vertices registered
A construir áreas...
100
0 areas built
0 isles built
A anexar ilhas...
A anexar centróides...
Número de nós: 0
Número de primitivos: 0
Número de pontos: 0
Número de linhas: 0
Número de fronteiras: 0
Número de centróides: 0
Número de áreas: 0
Número de ilhas: 0
ATENÇÃO: 1 'from' feature was not reachable

A complete dataset for testing v.net.* tools can be found here: http://qgis.pt/workshops_qgis2016/dados_exercicio_analise_redes.zip
And the presentation of the exercises are here (unfortunately they are in portuguese): http://qgis.pt/workshops_qgis2016/Analise_de_Redes_com_QGIS_PedroVenancio.pdf

#6 Updated by Giovanni Manghi almost 6 years ago

  • Affected QGIS version changed from 3.2.3 to 3.4.4
  • Priority changed from Normal to High

#7 Updated by Alexander Bruy almost 6 years ago

  • Status changed from Open to Feedback

Am I right that empty output produced only when executed via Processing and correct results produced when executing same commands in GRASS directly?

#8 Updated by Pedro Venâncio almost 6 years ago

Hi Alex,

It works with same data in QGIS 2.18.28, using same GRASS version (now 7.6.0).

I was looking more carefully, and the problem only happens when using arc_column and arc_backward_column parameters.

This works in QGIS 3.4.4:

v.net.distance input=net5c51d46329c936 arc_type="line,boundary" to_type="point" from_layer="2" to_layer="3" output=outputd3ae2de5a7344e9a93974c6bf994e8cf --overwrite

And this does not work (gives empty output), using exactly the same input data:

v.net.distance input=net5c51d4e3326c711 arc_type="line,boundary" to_type="point" arc_column=tmpo_duplo arc_backward_column=tmpo_unico from_layer="2" to_layer="3" output=outputbda141567f7d41a988a0db1ac74c663d --overwrite

Running both commands directly in GRASS 7.6.0, with intermediary data generated by QGIS 3.4.4 (v.in.ogr, g.region, v.net, v.db.connect - data saved in tmp folder) the output is the same, that is, an empty file using arc_column and arc_backward_column parameters, and the message:

WARNING: Registo da base de dados para a linha 94622 (cat = 94622, em frente/ambas as direções) não encontrada (em frente/ambas as direções da linha ignoradas)
WARNING: Registo da base de dados para a linha 94622(cat = 94622, direção reversa) não encontrada (direção da linha ignorada)
WARNING: Registo da base de dados para a linha 94626 (cat = 94623, em frente/ambas as direções) não encontrada (em frente/ambas as direções da linha ignoradas)
WARNING: Registo da base de dados para a linha 94626(cat = 94623, direção reversa) não encontrada (direção da linha ignorada)

So, given that all this works in QGIS 2.18.28 (with same data and same GRASS version), the problem must be somewhere in v.net / v.db.connect intermediary operations? The order is different, I don't know if this is relevant, but could be here:

- QGIS 2.18.28 (v.net + v.db.connect + v.net + v.db.connect)

v.net -s input=tmp1548434142196 points=tmp1548434142197 out=tmp1548434142199 op=connect threshold=50 arc_layer=1 node_layer=2
v.db.connect -o map=tmp1548434142199 table=tmp1548434142197 layer=2
v.net -s input=tmp1548434142199 points=tmp1548434142198 out=tmp15484341421910 op=connect threshold=50 arc_layer=1 node_layer=3
v.db.connect -o map=tmp15484341421910 table=tmp1548434142198 layer=3
v.net.distance input="tmp15484341421910" arc_type=line to_type=point arc_column="dist_duplo" arc_backward_column="dist_unico" from_layer="2" to_layer="3" output="output8df48fb12fc54499af89b4d27d5d4ab5" --overwrite

- QGIS 3.4.4 (v.net + v.net + v.db.connect + v.db.connect)

v.net input=vector_5c4b3aaa0594f8 points=vector_5c4b3aaa0594f9 output=bufnet5c4b3aaa05d3511 operation=connect threshold=50.0 arc_layer=1 node_layer=2
v.net input=bufnet5c4b3aaa05d3511 points=vector_5c4b3aaa0594f10 output=net5c4b3aaa05d3512 operation=connect threshold=50.0 arc_layer=1 node_layer=3
v.db.connect -o map=net5c4b3aaa05d3512 table=vector_5c4b3aaa0594f9 layer=2
v.db.connect -o map=net5c4b3aaa05d3512 table=vector_5c4b3aaa0594f10 layer=3
v.net.distance input=net5c4b3aaa05d3512 arc_type="line" to_type="point" arc_column=dist_duplo arc_backward_column=dist_unico from_layer="2" to_layer="3" output=output9bb50432020a447183c3ae5e59c42c53 --overwrite

#9 Updated by Giovanni Manghi almost 6 years ago

  • Status changed from Feedback to Open

#10 Updated by Pedro Venâncio almost 6 years ago

Ok, it seems the order is not relevant for this issue, because the same happens with v.net.iso, and there we have only one v.net + v.db.connect.

I was looking to the temporary grass mapsets created both by QGIS 2.18.28 and 3.4.4, and I think I've found the source of the problem.

For instance, in v.net.iso from QGIS 3.4.4, v.net + v.db.connect creates a network layer that have a new line, connecting the point layer to the lines (network) layer. This new line has no data in the attribute table. Please, see the image attached.

In QGIS 2.18.28, this new line is not created.

Then, when running v.net.* algorithms with arc_column and arc_backward_column parameters, this new line has null data, when numeric data are expected. And the error is shown:

WARNING: Registo da base de dados para a linha 94623 (cat = 94622, em frente/ambas as direções) não encontrada (em frente/ambas as direções da linha ignoradas)
WARNING: Registo da base de dados para a linha 94623(cat = 94622, direção reversa) não encontrada (direção da linha ignorada)

Temporary mapsets can be downloaded from here, if you need test:

https://cld.pt/dl/download/3dbf4723-66df-44ed-9424-0cf2d495812e/grassdata_21828.zip

https://cld.pt/dl/download/4f3fef72-bdac-424c-a5f5-891324db3833/grassdata_344.zip

So, why is this new line created in QGIS 3.4.4?

#11 Updated by Pedro Venâncio almost 6 years ago

I found it!

v.net needs the -s flag to snap points to network for operation connect. By default, v.net add a new line from the point to the network.

In 2.18.28, the -s flag is present.

#12 Updated by Pedro Venâncio almost 6 years ago

Simply changing

\python\plugins\processing\algs\grass7\ext\v_net.py line 61 to

command = u"v.net -s input={} points={} output={} operation=connect threshold={}".format(

\python\plugins\processing\algs\grass7\ext\v_net_distance.py lines 51 and 56 to

command = u"v.net -s input={} points={} output={} operation=connect threshold={} arc_layer=1 node_layer=2".format(
command = u"v.net -s input={} points={} output={} operation=connect threshold={} arc_layer=1 node_layer=3".format(

do the work.

Other v.net.* algorithms needs to be checked also.

Can someone apply this changes to master and backport to 3.4.x?

Thanks!

#13 Updated by Alexander Bruy almost 6 years ago

  • Assignee set to Alexander Bruy
  • Status changed from Open to In Progress

Pedro Venâncio wrote:

I found it!

Thanks for your work! I will take care about updating code and backporting

#14 Updated by Giovanni Manghi almost 6 years ago

  • Pull Request or Patch supplied changed from No to Yes
  • Status changed from In Progress to Open
  • Assignee deleted (Alexander Bruy)

Can someone apply this changes to master and backport to 3.4.x?

for master https://github.com/qgis/QGIS/pull/9044

#15 Updated by Giovanni Manghi almost 6 years ago

Giovanni Manghi wrote:

Can someone apply this changes to master and backport to 3.4.x?

for master https://github.com/qgis/QGIS/pull/9044

for 3.4 https://github.com/qgis/QGIS/pull/9045

#16 Updated by Alexander Bruy almost 6 years ago

  • Status changed from Open to In Progress
  • Assignee set to Alexander Bruy

#17 Updated by Alexander Bruy almost 6 years ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Closed

#18 Updated by Alexander Bruy almost 6 years ago

  • Resolution set to fixed/implemented

Also available in: Atom PDF