Bug report #14241
GRASS plugin tries to call t.list.exe instead of t.list.bat when listing temporal input on windows
Status: | Closed | ||
---|---|---|---|
Priority: | Normal | ||
Assignee: | |||
Category: | GRASS | ||
Affected QGIS version: | 2.12.2 | Regression?: | No |
Operating System: | Windows | Easy fix?: | No |
Pull Request or Patch supplied: | No | Resolution: | |
Crashes QGIS or corrupts data: | No | Copied to github as #: | 22238 |
Description
On Windows, when a temporal module is started, which takes space time datasets as input, the GRASS plugin calls t.list.exe, instead of t.list.py.
Error message is:
"Cannot start module command: C:/OSGeo4W/apps/grass/GRASS-~1.3RC/scripts/t.list.exe type=strds"
I guess the problem is src/providers/grass/qgsgrass.cpp, line 1545.
Associated revisions
[GRASS] fixed module search on windows, fixes #14241
[GRASS] removed absolute path from t.list, fixes #14241
History
#1
Updated by Radim Blazek over 8 years ago
It should be fixed in 8e48e97d but it was not tested on Windows. Please test with the next build.
#2
Updated by Stefan Blumentrath over 8 years ago
Hi, and thanks for the quick fix. I tired to test. But on Windows the GRASS plugin is broken for me at the moment due to package incompatibilities...
When I installed QGIS Master I accidentally also updated GRASS to 7.0.3. The QGIS GRASS plugin is (still?) build against GRASS 7.0.3RC2, while OSGeo only ships GRASS 7.0.3 and GRASS 7.0.2. Since also my setup.ini for OSGeo4W got overwritten, there is currently no (convenient) way back to RC2. But maybe the nightly with the relevant changes get available first tomorrow anyway?
Just saw, that the setup.ini is from 06.02.2016. So I assume qgis-dev has not been built and packaged with your fix yet. I shall try again as soon as it gets available...
#3
Updated by Radim Blazek over 8 years ago
I have also moved QgsGrassModule::findExec() to QgsGrass::findModule() and used it in QgsGrass::runModule() in 58beed2 as Jef suggested: #8e48e97d4a233b523c92e4b9ca84677ab0916e0c#commitcomment-15954822
#4
Updated by Giovanni Manghi over 8 years ago
- Status changed from Open to Feedback
#5
Updated by Stefan Blumentrath over 8 years ago
I can confirm that this particular issue is fixed!
However, the time series data input widget does not find any STRDS; although t.list works and returns all three STRDS in my current mapset.
Should I open a new ticket for that or can we continue here?
In addition, at least in t.sample input and sample STDS have a text input widget, maybe there is no equivalent for G_OPT_STDS_INPUTS and G_OPT_STDS_INPUT in QGIS yet?
Finally, I could not access the tgis/sqlite.db I had on a network drive (CIFS). But that is probably a GRASS issue (I shall investigate)...
#6
Updated by Radim Blazek over 8 years ago
Maybe it is not really fixed and it just does not show an error because I also added missing try-catch.
However, the time series data input widget does not find any STRDS; although t.list works and returns all three STRDS in my current mapset.
Is it Windows only problem? Which module? t.rast.list works for me on Linux. Can you open debug logs and filter t.list, you should get something like:
src/providers/grass/qgsgrass.cpp: 2002: (findModule) [0ms] not found /t.rast.list
src/providers/grass/qgsgrass.cpp: 2002: (findModule) [0ms] not found /home/user/grass/grass-7.1.svn/bin/t.rast.list
src/providers/grass/qgsgrass.cpp: 1997: (findModule) [0ms] found /home/user/grass/grass-7.1.svn/scripts/t.rast.list
In addition, at least in t.sample input and sample STDS have a text input widget, maybe there is no equivalent for G_OPT_STDS_INPUTS and G_OPT_STDS_INPUT in QGIS yet?
Yes, stds is not supported only strds, stvds and str3ds. I'll fix that.
#7
Updated by Stefan Blumentrath over 8 years ago
Is it Windows only problem? Which module? t.rast.list works for me on Linux.
Possibly. I have not tested on Ubuntu yet, but it is independend from the network storage issue...
Can you open debug logs and filter t.list, you should get something like:
src/providers/grass/qgsgrass.cpp: 2002: (findModule) [0ms] not found /t.rast.list
src/providers/grass/qgsgrass.cpp: 2002: (findModule) [0ms] not found /home/user/grass/grass-7.1.svn/bin/t.rast.list
src/providers/grass/qgsgrass.cpp: 1997: (findModule) [0ms] found /home/user/grass/grass-7.1.svn/scripts/t.rast.list
I am installing qgis-dev-pdb now... Where do I find the debug logs (on Windows)?
Yes, stds is not supported only strds, stvds and str3ds. I'll fix that.
Great, thanks!
BTW, would you mind backporting or are we too close to 2.14, so that does not make much sense?
#8
Updated by Radim Blazek over 8 years ago
You don't need qgis-dev-pdb to see debug logs. Just start dbgview.exe (https://technet.microsoft.com/en-us/sysinternals/debugview.aspx) and logs are printed there, you can filter etc.
Backport does not make sense to me, especially because t.* modules are rarely used.
#9
Updated by Stefan Blumentrath over 8 years ago
- File QGIS-GRASS.log
added
Backport does not make sense to me, especially because t.* modules are rarely used.
I fully understand.
Could not find anything suspicious in the debug logs (log file attached)...
#10
Updated by Radim Blazek over 8 years ago
The t.list call from QgsGrassModuleInputModel is missing in the log, it is called once during the session (or when data changed). Start first dbgview then qgis and when you start t.rast.list it should log findModule() with t.list.
#11
Updated by Stefan Blumentrath over 8 years ago
- File QGIS-GRASS-2.log
added
So I did. Attached a full debug log including QGIS start...
#12
Updated by Radim Blazek over 8 years ago
qgsgrass.cpp(2002) : (QgsGrass::findModule) [0ms] not found /C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/t.list.bat
because
QString full = path + "/" + module + ext;
adds "/" even when path is empty (trying module with full path).
I'll fix that, thanks for debug.
#13
Updated by Radim Blazek over 8 years ago
- Status changed from Feedback to Closed
Fixed in changeset dfdb420f23e33e3ed3df064d25b35582244de594.
#14
Updated by Radim Blazek over 8 years ago
Stds input is fixed in c8161aa.
#15
Updated by Stefan Blumentrath over 8 years ago
- Status changed from Closed to Reopened
I can confirm that the STDS part is fixed. However, list of Space Time Datasets (STDRS, STDVS...) in the input widgets remains empty on windows, although such datasets exist in the mapset:
[6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(1536) : (QgsGrass::grassObjects) [0ms] mapsetPath = C:/data/grassdata/Portugal/ninsbl_new type = strds [6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2769) : (QgsGrass::defaultGisbase) [0ms] gisbase from envar = C:/OSGeo4W/apps/grass/GRASS-~1.3 [6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2086) : (QgsGrass::runModule) [0ms] gisdbase = C:/data/grassdata location = Portugal timeOut = -1 [6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2027) : (QgsGrass::startModule) [0ms] gisdbase = C:/data/grassdata location = Portugal [6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(1980) : (QgsGrass::findModule) [0ms] called. [6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2016) : (QgsGrass::findModule) [0ms] not found C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/t.list.bat [6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2016) : (QgsGrass::findModule) [0ms] not found C:/OSGeo4W/apps/grass/GRASS-~1.3/bin/C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/t.list.bat [6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2016) : (QgsGrass::findModule) [0ms] not found C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/t.list.bat [6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2016) : (QgsGrass::findModule) [0ms] not found C:/OSGeo4W/apps/qgis-dev/./grass/scripts/C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/t.list.bat [6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2016) : (QgsGrass::findModule) [0ms] not found C:/OSGeo4W/apps/qgis-dev/./grass/modules/C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/t.list.bat [6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2011) : (QgsGrass::findModule) [0ms] found C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/t.list.py [6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2062) : (QgsGrass::startModule) [1ms] C:/Users/STEFAN~1.BLU/AppData/Local/Temp/qt_temp.az6876 [6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2073) : (QgsGrass::startModule) [0ms] C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/t.list.py type=strds [6876] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(1591) : (QgsGrass::grassObjects) [1ms] Cannot run C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/t.list: Cannot start module [6876] command: C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/t.list type=strds
The correct module call for t-list on Windows is something like this:
C:/OSGeo4W/apps/grass/GRASS-~1.3/bin/t.list.bat type=strds separator=pipe
#16
Updated by Radim Blazek over 8 years ago
- Status changed from Reopened to Closed
Fixed in changeset 5dd86aabfa3fbcd25b07953eec1842d9936f626f.
#17
Updated by Radim Blazek over 8 years ago
There was absolute path left for t.list. Hopefully it is fixed now, sorry.
#18
Updated by Stefan Blumentrath over 8 years ago
Thanks for your swift reply and fix! I shall test the next build and report back in case of issues. But lets assume that that was it. Looking forward to using the temporal modules in QGIS!
#19
Updated by Stefan Blumentrath over 8 years ago
Thanks for your swift reply and fix! I shall test the next build and report back in case of issues. But lets assume that that was it. Looking forward to using the temporal modules in QGIS!
#20
Updated by Radim Blazek over 8 years ago
Don't forget that t.list is terribly slow (about 500ms!) and it has to be run for each mapset in search path for each type (strds, stvds, str3ds, stds).
It would be probably better to access the sqlite db directly from the GRASS provider in QGIS.
#21
Updated by Jürgen Fischer over 8 years ago
- Subject changed from GRASS plugin tries to call t.list.exe instead of t.list.py when listing temporal input to GRASS plugin tries to call t.list.exe instead of t.list.bat when listing temporal input on windows
#22
Updated by Stefan Blumentrath over 8 years ago
Radim Blazek wrote:
Don't forget that t.list is terribly slow (about 500ms!) and it has to be run for each mapset in search path for each type (strds, stvds, str3ds, stds).
It would be probably better to access the sqlite db directly from the GRASS provider in QGIS.
If listing the datasets can be done faster that t.list does, would nt it be better to apply performance improvements upstreams?
#23
Updated by Radim Blazek over 8 years ago
Stefan Blumentrath wrote:
If listing the datasets can be done faster that t.list does, would nt it be better to apply performance improvements upstreams?
That was originally my idea and I tried to look into the module but that does not seem to be that simple. See my discussion with Soeren http://osgeo-org.1560.x6.nabble.com/t-list-slow-td5230931.html
If the author does not see a solution, I could hardly find it. OTOH, it is quite simple to query
select title from strds_metadata;
Unfortunately the temporal modules are quite non standard.
#24
Updated by Stefan Blumentrath over 8 years ago
- Status changed from Closed to Reopened
Unfortunately, this does not seem to work on Windows, as the list of STDS remains empty.
Here is some debugview output:[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(1601) : (QgsGrass::grassObjects) [0ms] time (ms) = 16
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(1536) : (QgsGrass::grassObjects) [0ms] mapsetPath = R:/grassdata/ETRS_33N/U_stefan.blumentrath type = strds
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2086) : (QgsGrass::runModule) [78ms] gisdbase = R:/grassdata location = ETRS_33N timeOut = -1
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2027) : (QgsGrass::startModule) [0ms] gisdbase = R:/grassdata location = ETRS_33N
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(1980) : (QgsGrass::findModule) [0ms] called.
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2016) : (QgsGrass::findModule) [0ms] not found t.list.bat
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2011) : (QgsGrass::findModule) [0ms] found C:/OSGeo4W/apps/grass/GRASS-~1.3/bin/t.list.bat
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2062) : (QgsGrass::startModule) [0ms] C:/Users/STEFAN~1.BLU/AppData/Local/Temp/qt_temp.MS5012
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2073) : (QgsGrass::startModule) [0ms] C:/OSGeo4W/apps/grass/GRASS-~1.3/bin/t.list.bat type=strds
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2096) : (QgsGrass::runModule) [93ms] process->exitCode() = 1
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(1591) : (QgsGrass::grassObjects) [0ms] Cannot run t.list: Cannot run module
[5012] command: t.list type=strds
[5012] stdout:
[5012] stderr: Traceback (most recent call last):
[5012] File "C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/t.list.py", line 80, in <module>
[5012] import grass.script as gscript
[5012] ImportError: No module named grass.script
[5012]
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(1600) : (QgsGrass::grassObjects) [0ms] list =
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(1601) : (QgsGrass::grassObjects) [0ms] time (ms) = 171
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(1536) : (QgsGrass::grassObjects) [0ms] mapsetPath = R:/grassdata/ETRS_33N/U_stefan.blumentrath type = stvds
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2086) : (QgsGrass::runModule) [0ms] gisdbase = R:/grassdata location = ETRS_33N timeOut = -1
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2027) : (QgsGrass::startModule) [0ms] gisdbase = R:/grassdata location = ETRS_33N
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(1980) : (QgsGrass::findModule) [0ms] called.
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2016) : (QgsGrass::findModule) [0ms] not found t.list.bat
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2011) : (QgsGrass::findModule) [0ms] found C:/OSGeo4W/apps/grass/GRASS-~1.3/bin/t.list.bat
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2062) : (QgsGrass::startModule) [0ms] C:/Users/STEFAN~1.BLU/AppData/Local/Temp/qt_temp.jS5012
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2073) : (QgsGrass::startModule) [0ms] C:/OSGeo4W/apps/grass/GRASS-~1.3/bin/t.list.bat type=stvds
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(2096) : (QgsGrass::runModule) [31ms] process->exitCode() = 1
[5012] c:\\src\\qgis\\src\\providers\\grass\\qgsgrass.cpp(1591) : (QgsGrass::grassObjects) [0ms] Cannot run t.list: Cannot run module
[5012] command: t.list type=stvds
[5012] stdout:
[5012] stderr: Traceback (most recent call last):
[5012] File "C:/OSGeo4W/apps/grass/GRASS-~1.3/scripts/t.list.py", line 80, in <module>
[5012] import grass.script as gscript
[5012] ImportError: No module named grass.script
[5012]
BTW: I noticed that all several thousand raster maps got listed in the debug output, like:
[5012] c:\\src\\qgis\\src\\plugins\\grass\\qgsgrassmoduleinput.cpp(243) : (QgsGrassModuleInputModel::refreshMapset) [0ms] add map : precipitation_2013_02_21
before the error occurred. Naturally, that took quite a while. Maybe that adds to the perormance issues of t.list?
Just as a note... There is a related GRASS issue when temporal modules are used in multi-user or mixed environments (Linux / Windows):
https://trac.osgeo.org/grass/ticket/2943, which is why I suspected the issue (empty STDS drop down menu) to be a GRASS one...
I can send the full debug log if necessary...
#25
Updated by Stefan Blumentrath over 8 years ago
Forgot to mention: When I call t.list manually it lists all STRDS without an issue...
#26
Updated by Radim Blazek over 8 years ago
- Status changed from Reopened to Closed
#27
Updated by Radim Blazek over 8 years ago
Stefan Blumentrath wrote:
BTW: I noticed that all several thousand raster maps got listed in the debug output, like:
[5012] c:\\src\\qgis\\src\\plugins\\grass\\qgsgrassmoduleinput.cpp(243) : (QgsGrassModuleInputModel::refreshMapset) [0ms] add map : precipitation_2013_02_21
before the error occurred. Naturally, that took quite a while. Maybe that adds to the perormance issues of t.list?
I don't think so. The list of standard raster layers is created by listing directory content which is fast. The problem of t.list is that takes too long to start regardless how many temporal sets / maps there are in GRASS database.
#28
Updated by Stefan Blumentrath over 8 years ago
I just noticed it while watching the Debugview logging... For time series data there can be > 40k data sets in on mapset...
In general opening a module seems quite slow (not only temporal, but also raster or vector modules), but that may be because the grassdb is on the network (1Gbps connection)...?
BTW, thanks for the quick fix and backport!