Bug report #14241

GRASS plugin tries to call t.list.exe instead of t.list.bat when listing temporal input on windows

Added by Stefan Blumentrath over 4 years ago. Updated over 4 years ago.

Status:Closed
Priority:Normal
Assignee:Radim Blazek
Category:GRASS
Affected QGIS version:2.12.2 Regression?:No
Operating System:Windows Easy fix?:No
Pull Request or Patch supplied:No Resolution:
Crashes QGIS or corrupts data:No Copied to github as #:22238

Description

On Windows, when a temporal module is started, which takes space time datasets as input, the GRASS plugin calls t.list.exe, instead of t.list.py.

Error message is:
"Cannot start module command: C:/OSGeo4W/apps/grass/GRASS-~1.3RC/scripts/t.list.exe type=strds"

I guess the problem is src/providers/grass/qgsgrass.cpp, line 1545.

QGIS-GRASS.log Magnifier - Debug log of temporal modules in the QGIS-GRASS7-plugin (272 KB) Stefan Blumentrath, 2016-02-11 12:51 AM

QGIS-GRASS-2.log Magnifier (959 KB) Stefan Blumentrath, 2016-02-11 01:14 AM

Associated revisions

Revision dfdb420f
Added by Radim Blazek over 4 years ago

[GRASS] fixed module search on windows, fixes #14241

Revision 5dd86aab
Added by Radim Blazek over 4 years ago

[GRASS] removed absolute path from t.list, fixes #14241

History

#1 Updated by Radim Blazek over 4 years ago

It should be fixed in 8e48e97d but it was not tested on Windows. Please test with the next build.

#2 Updated by Stefan Blumentrath over 4 years ago

Hi, and thanks for the quick fix. I tired to test. But on Windows the GRASS plugin is broken for me at the moment due to package incompatibilities...
When I installed QGIS Master I accidentally also updated GRASS to 7.0.3. The QGIS GRASS plugin is (still?) build against GRASS 7.0.3RC2, while OSGeo only ships GRASS 7.0.3 and GRASS 7.0.2. Since also my setup.ini for OSGeo4W got overwritten, there is currently no (convenient) way back to RC2. But maybe the nightly with the relevant changes get available first tomorrow anyway?

Just saw, that the setup.ini is from 06.02.2016. So I assume qgis-dev has not been built and packaged with your fix yet. I shall try again as soon as it gets available...

#3 Updated by Radim Blazek over 4 years ago

I have also moved QgsGrassModule::findExec() to QgsGrass::findModule() and used it in QgsGrass::runModule() in 58beed2 as Jef suggested: #8e48e97d4a233b523c92e4b9ca84677ab0916e0c#commitcomment-15954822

#4 Updated by Giovanni Manghi over 4 years ago

  • Status changed from Open to Feedback

#5 Updated by Stefan Blumentrath over 4 years ago

I can confirm that this particular issue is fixed!

However, the time series data input widget does not find any STRDS; although t.list works and returns all three STRDS in my current mapset.
Should I open a new ticket for that or can we continue here?

In addition, at least in t.sample input and sample STDS have a text input widget, maybe there is no equivalent for G_OPT_STDS_INPUTS and G_OPT_STDS_INPUT in QGIS yet?

Finally, I could not access the tgis/sqlite.db I had on a network drive (CIFS). But that is probably a GRASS issue (I shall investigate)...

#6 Updated by Radim Blazek over 4 years ago

Maybe it is not really fixed and it just does not show an error because I also added missing try-catch.

However, the time series data input widget does not find any STRDS; although t.list works and returns all three STRDS in my current mapset.

Is it Windows only problem? Which module? t.rast.list works for me on Linux. Can you open debug logs and filter t.list, you should get something like:

src/providers/grass/qgsgrass.cpp: 2002: (findModule) [0ms] not found /t.rast.list
src/providers/grass/qgsgrass.cpp: 2002: (findModule) [0ms] not found /home/user/grass/grass-7.1.svn/bin/t.rast.list
src/providers/grass/qgsgrass.cpp: 1997: (findModule) [0ms] found /home/user/grass/grass-7.1.svn/scripts/t.rast.list

In addition, at least in t.sample input and sample STDS have a text input widget, maybe there is no equivalent for G_OPT_STDS_INPUTS and G_OPT_STDS_INPUT in QGIS yet?

Yes, stds is not supported only strds, stvds and str3ds. I'll fix that.

#7 Updated by Stefan Blumentrath over 4 years ago

Is it Windows only problem? Which module? t.rast.list works for me on Linux.

Possibly. I have not tested on Ubuntu yet, but it is independend from the network storage issue...

Can you open debug logs and filter t.list, you should get something like:

src/providers/grass/qgsgrass.cpp: 2002: (findModule) [0ms] not found /t.rast.list
src/providers/grass/qgsgrass.cpp: 2002: (findModule) [0ms] not found /home/user/grass/grass-7.1.svn/bin/t.rast.list
src/providers/grass/qgsgrass.cpp: 1997: (findModule) [0ms] found /home/user/grass/grass-7.1.svn/scripts/t.rast.list

I am installing qgis-dev-pdb now... Where do I find the debug logs (on Windows)?

Yes, stds is not supported only strds, stvds and str3ds. I'll fix that.

Great, thanks!

BTW, would you mind backporting or are we too close to 2.14, so that does not make much sense?

#8 Updated by Radim Blazek over 4 years ago

You don't need qgis-dev-pdb to see debug logs. Just start dbgview.exe (https://technet.microsoft.com/en-us/sysinternals/debugview.aspx) and logs are printed there, you can filter etc.

Backport does not make sense to me, especially because t.* modules are rarely used.

#9 Updated by Stefan Blumentrath over 4 years ago

Backport does not make sense to me, especially because t.* modules are rarely used.

I fully understand.

Could not find anything suspicious in the debug logs (log file attached)...

#10 Updated by Radim Blazek over 4 years ago

The t.list call from QgsGrassModuleInputModel is missing in the log, it is called once during the session (or when data changed). Start first dbgview then qgis and when you start t.rast.list it should log findModule() with t.list.

#11 Updated by Stefan Blumentrath over 4 years ago

So I did. Attached a full debug log including QGIS start...

#12 Updated by Radim Blazek over 4 years ago

qgsgrass.cpp(2002) : (QgsGrass::findModule) [0ms] not found /C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/t.list.bat

because

QString full = path + "/" + module + ext;

adds "/" even when path is empty (trying module with full path).

I'll fix that, thanks for debug.

#13 Updated by Radim Blazek over 4 years ago

  • Status changed from Feedback to Closed

#14 Updated by Radim Blazek over 4 years ago

Stds input is fixed in c8161aa.

#15 Updated by Stefan Blumentrath over 4 years ago

  • Status changed from Closed to Reopened

I can confirm that the STDS part is fixed. However, list of Space Time Datasets (STDRS, STDVS...) in the input widgets remains empty on windows, although such datasets exist in the mapset:

[6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(1536) : (QgsGrass::grassObjects) [0ms] mapsetPath = C:/data/grassdata/Portugal/ninsbl_new type = strds
[6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2769) : (QgsGrass::defaultGisbase) [0ms] gisbase from envar = C:/OSGeo4W/apps/grass/GRASS-~1.3
[6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2086) : (QgsGrass::runModule) [0ms] gisdbase = C:/data/grassdata location = Portugal timeOut = -1
[6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2027) : (QgsGrass::startModule) [0ms] gisdbase = C:/data/grassdata location = Portugal
[6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(1980) : (QgsGrass::findModule) [0ms] called.
[6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2016) : (QgsGrass::findModule) [0ms] not found C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/t.list.bat
[6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2016) : (QgsGrass::findModule) [0ms] not found C:/OSGeo4W/apps/grass/GRASS-~1.3/bin/C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/t.list.bat
[6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2016) : (QgsGrass::findModule) [0ms] not found C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/t.list.bat
[6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2016) : (QgsGrass::findModule) [0ms] not found C:/OSGeo4W/apps/qgis-dev/./grass/scripts/C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/t.list.bat
[6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2016) : (QgsGrass::findModule) [0ms] not found C:/OSGeo4W/apps/qgis-dev/./grass/modules/C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/t.list.bat
[6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2011) : (QgsGrass::findModule) [0ms] found C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/t.list.py
[6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2062) : (QgsGrass::startModule) [1ms] C:/Users/STEFAN~1.BLU/AppData/Local/Temp/qt_temp.az6876
[6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2073) : (QgsGrass::startModule) [0ms] C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/t.list.py type=strds
[6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(1591) : (QgsGrass::grassObjects) [1ms] Cannot run C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/t.list: Cannot start module
[6876] command: C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/t.list type=strds

The correct module call for t-list on Windows is something like this:

C:/OSGeo4W/apps/grass/GRASS-~1.3/bin/t.list.bat type=strds separator=pipe

#16 Updated by Radim Blazek over 4 years ago

  • Status changed from Reopened to Closed

#17 Updated by Radim Blazek over 4 years ago

There was absolute path left for t.list. Hopefully it is fixed now, sorry.

#18 Updated by Stefan Blumentrath over 4 years ago

Thanks for your swift reply and fix! I shall test the next build and report back in case of issues. But lets assume that that was it. Looking forward to using the temporal modules in QGIS!

#19 Updated by Stefan Blumentrath over 4 years ago

Thanks for your swift reply and fix! I shall test the next build and report back in case of issues. But lets assume that that was it. Looking forward to using the temporal modules in QGIS!

#20 Updated by Radim Blazek over 4 years ago

Don't forget that t.list is terribly slow (about 500ms!) and it has to be run for each mapset in search path for each type (strds, stvds, str3ds, stds).

It would be probably better to access the sqlite db directly from the GRASS provider in QGIS.

#21 Updated by Jürgen Fischer over 4 years ago

  • Subject changed from GRASS plugin tries to call t.list.exe instead of t.list.py when listing temporal input to GRASS plugin tries to call t.list.exe instead of t.list.bat when listing temporal input on windows

#22 Updated by Stefan Blumentrath over 4 years ago

Radim Blazek wrote:

Don't forget that t.list is terribly slow (about 500ms!) and it has to be run for each mapset in search path for each type (strds, stvds, str3ds, stds).

It would be probably better to access the sqlite db directly from the GRASS provider in QGIS.

If listing the datasets can be done faster that t.list does, would nt it be better to apply performance improvements upstreams?

#23 Updated by Radim Blazek over 4 years ago

Stefan Blumentrath wrote:

If listing the datasets can be done faster that t.list does, would nt it be better to apply performance improvements upstreams?

That was originally my idea and I tried to look into the module but that does not seem to be that simple. See my discussion with Soeren http://osgeo-org.1560.x6.nabble.com/t-list-slow-td5230931.html
If the author does not see a solution, I could hardly find it. OTOH, it is quite simple to query

select title from strds_metadata;

Unfortunately the temporal modules are quite non standard.

#24 Updated by Stefan Blumentrath over 4 years ago

  • Status changed from Closed to Reopened

Unfortunately, this does not seem to work on Windows, as the list of STDS remains empty.

Here is some debugview output:
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(1601) : (QgsGrass::grassObjects) [0ms] time (ms) = 16
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(1536) : (QgsGrass::grassObjects) [0ms] mapsetPath = R:/grassdata/ETRS_33N/U_stefan.blumentrath type = strds
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2086) : (QgsGrass::runModule) [78ms] gisdbase = R:/grassdata location = ETRS_33N timeOut = -1
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2027) : (QgsGrass::startModule) [0ms] gisdbase = R:/grassdata location = ETRS_33N
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(1980) : (QgsGrass::findModule) [0ms] called.
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2016) : (QgsGrass::findModule) [0ms] not found t.list.bat
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2011) : (QgsGrass::findModule) [0ms] found C:/OSGeo4W/apps/grass/GRASS-~1.3/bin/t.list.bat
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2062) : (QgsGrass::startModule) [0ms] C:/Users/STEFAN~1.BLU/AppData/Local/Temp/qt_temp.MS5012
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2073) : (QgsGrass::startModule) [0ms] C:/OSGeo4W/apps/grass/GRASS-~1.3/bin/t.list.bat type=strds
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2096) : (QgsGrass::runModule) [93ms] process->exitCode() = 1
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(1591) : (QgsGrass::grassObjects) [0ms] Cannot run t.list: Cannot run module
[5012] command: t.list type=strds
[5012] stdout:
[5012] stderr: Traceback (most recent call last):
[5012] File "C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/t.list.py", line 80, in <module>
[5012] import grass.script as gscript
[5012] ImportError: No module named grass.script
[5012]
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(1600) : (QgsGrass::grassObjects) [0ms] list =
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(1601) : (QgsGrass::grassObjects) [0ms] time (ms) = 171
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(1536) : (QgsGrass::grassObjects) [0ms] mapsetPath = R:/grassdata/ETRS_33N/U_stefan.blumentrath type = stvds
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2086) : (QgsGrass::runModule) [0ms] gisdbase = R:/grassdata location = ETRS_33N timeOut = -1
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2027) : (QgsGrass::startModule) [0ms] gisdbase = R:/grassdata location = ETRS_33N
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(1980) : (QgsGrass::findModule) [0ms] called.
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2016) : (QgsGrass::findModule) [0ms] not found t.list.bat
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2011) : (QgsGrass::findModule) [0ms] found C:/OSGeo4W/apps/grass/GRASS-~1.3/bin/t.list.bat
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2062) : (QgsGrass::startModule) [0ms] C:/Users/STEFAN~1.BLU/AppData/Local/Temp/qt_temp.jS5012
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2073) : (QgsGrass::startModule) [0ms] C:/OSGeo4W/apps/grass/GRASS-~1.3/bin/t.list.bat type=stvds
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2096) : (QgsGrass::runModule) [31ms] process->exitCode() = 1
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(1591) : (QgsGrass::grassObjects) [0ms] Cannot run t.list: Cannot run module
[5012] command: t.list type=stvds
[5012] stdout:
[5012] stderr: Traceback (most recent call last):
[5012] File "C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/t.list.py", line 80, in <module>
[5012] import grass.script as gscript
[5012] ImportError: No module named grass.script
[5012]

BTW: I noticed that all several thousand raster maps got listed in the debug output, like:

[5012] c:\\src\\qgis\\src\\plugins\\grass\\qgsgrassmoduleinput.cpp(243) : (QgsGrassModuleInputModel::refreshMapset) [0ms] add map : precipitation_2013_02_21

before the error occurred. Naturally, that took quite a while. Maybe that adds to the perormance issues of t.list?

Just as a note... There is a related GRASS issue when temporal modules are used in multi-user or mixed environments (Linux / Windows):
https://trac.osgeo.org/grass/ticket/2943, which is why I suspected the issue (empty STDS drop down menu) to be a GRASS one...

I can send the full debug log if necessary...

#25 Updated by Stefan Blumentrath over 4 years ago

Forgot to mention: When I call t.list manually it lists all STRDS without an issue...

#26 Updated by Radim Blazek over 4 years ago

  • Status changed from Reopened to Closed

It was problem on all platforms, path to GRASS python files was missing in PYTHONPATH and path to g.parser (used by t.list.py) was missing in PATH. t.list is the only python module run in this context.

Fixed in 1d3cbd9 and backported to 2.14 in 8958204.

#27 Updated by Radim Blazek over 4 years ago

Stefan Blumentrath wrote:

BTW: I noticed that all several thousand raster maps got listed in the debug output, like:

[5012] c:\\src\\qgis\\src\\plugins\\grass\\qgsgrassmoduleinput.cpp(243) : (QgsGrassModuleInputModel::refreshMapset) [0ms] add map : precipitation_2013_02_21

before the error occurred. Naturally, that took quite a while. Maybe that adds to the perormance issues of t.list?

I don't think so. The list of standard raster layers is created by listing directory content which is fast. The problem of t.list is that takes too long to start regardless how many temporal sets / maps there are in GRASS database.

#28 Updated by Stefan Blumentrath over 4 years ago

I just noticed it while watching the Debugview logging... For time series data there can be > 40k data sets in on mapset...

In general opening a module seems quite slow (not only temporal, but also raster or vector modules), but that may be because the grassdb is on the network (1Gbps connection)...?

BTW, thanks for the quick fix and backport!

Also available in: Atom PDF