Bug report #19731

Python error at qgis startup after changing profile folder

Added by Andreas Mueller over 1 year ago. Updated 7 months ago.

Status:Feedback
Priority:High
Assignee:Luigi Pirelli
Category:Build/Install
Affected QGIS version:3.2.2 Regression?:Yes
Operating System:Windows 7 Easy fix?:No
Pull Request or Patch supplied:Yes Resolution:
Crashes QGIS or corrupts data:No Copied to github as #:27556

Description

I put my osgeo4w/qgis install onto a usb stick and use the nice new profile feature. Qgis is called with a fix profile via command line (--profiles-path) which points to a directory on the same usb sick. Unfortunately, there are absolute pathes stored in the QGIS3.ini file inside the profile, so if the drive letter changes, these pathes are invalid. This is the error message i get.

FileNotFoundError: [WinError 3] Das System kann den angegebenen Pfad nicht finden: 'C:\\tmp\\gp180828\\Home\\.qis3\\profiles\\default\\processing\\scripts' 
Traceback (most recent call last):
  File "C:/tmp/gp180823/osgeo4w/apps/qgis/./python/plugins\processing\script\ScriptAlgorithmProvider.py", line 106, in loadAlgorithms
    items = [f for f in os.listdir(folder) if os.path.isfile(os.path.join(folder, f))]
FileNotFoundError: [WinError 3] Das System kann den angegebenen Pfad nicht finden: 'C:\\tmp\\gp180828\\Home\\.qis3\\profiles\\default\\processing\\scripts'

Python-Version: 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 07:18:10) [MSC v.1900 32 bit (Intel)] 
QGIS-Version: 3.2.2-Bonn Bonn, 26842169e9 

Python-Pfad:
C:/tmp/gp180823/osgeo4w/apps/qgis/./python
C:\tmp\gp180823\Home\.qis3\profiles\default/python
C:\tmp\gp180823\Home\.qis3\profiles\default/python/plugins
C:/tmp/gp180823/osgeo4w/apps/qgis/./python/plugins
C:\tmp\gp180823\osgeo4w\bin\python36.zip
C:\tmp\gp180823\osgeo4w\apps\Python36\DLLs
C:\tmp\gp180823\osgeo4w\apps\Python36\lib
C:\tmp\gp180823\osgeo4w\bin
C:\tmp\gp180823\osgeo4w\apps\Python36
C:\tmp\gp180823\osgeo4w\apps\Python36\lib\site-packages
C:\tmp\gp180823\Home\.qis3\profiles\default/python

As you can see, the error raises inside the list comprehension in ScriptAlgorithmProvider.py.

My suggestion is, to check if the pathes exist else fall back to default values, e.g. the pathes inside the active profile.
Because the values for these pathes are stored in ini files inside the profile folder, may be qgis could use relative pathes and expand them with the profile path?
There is also a global ini file (qgis_global_settings.ini) in apps/qgis/resources which may can provide default values?

I think the problem is not only related to my "custom portable qgis", but with cases where people have profiles or qgis installs on network drives and more.

History

#1 Updated by Giovanni Manghi over 1 year ago

  • Category changed from Customisation Framework to Build/Install

Maybe the category should be "Processing", not sure.

#2 Updated by Giovanni Manghi over 1 year ago

  • Status changed from Open to Feedback

Did it worked as expected on 2.18?

#3 Updated by Andreas Mueller over 1 year ago

  • Assignee set to Giovanni Manghi

Giovanni Manghi wrote:

Did it worked as expected on 2.18?

Yes, the only absolute pathes i can find in QGIS2.ini are lastDirPath and alike. If they do not exists, it doesn't matter.

#4 Updated by Giovanni Manghi over 1 year ago

  • Priority changed from Normal to High
  • Regression? changed from No to Yes
  • Status changed from Feedback to Open
  • Assignee deleted (Giovanni Manghi)

#5 Updated by Luigi Pirelli 10 months ago

  • Assignee set to Luigi Pirelli

investigating

#6 Updated by Luigi Pirelli 10 months ago

  • Pull Request or Patch supplied changed from No to Yes

#7 Updated by Luigi Pirelli 10 months ago

please Andreas Mueller give a look of the related unit test if it cover your requirements

https://github.com/qgis/QGIS/blob/a2bf5ccefd5363bb06d607d2dd5925fd82816cad/python/plugins/processing/tests/ScriptUtilsTest.py#L53

or better build with the path and check if it owrks in your use cases.

#8 Updated by Andreas Mueller 10 months ago

Thank you for your efforts Luigi,

i looked into the test, but i don't totally understand the code, I'm sorry. Also I never build qgis from source and it is not an option at the moment.

But i tried to use the new scripts ScriptAlgorithmProvider.py and ScriptUtils.py with my installed qqis (3.4.3) which gives an error, so i think there are other changes in the code since 3.4.3. Error is:

2019-02-14T10:57:07     WARNING    Traceback (most recent call last):
File "H:/gp181030/osgeo4w/apps/qgis/./python\qgis\utils.py", line 309, in loadPlugin
import(packageName)
File "H:/gp181030/osgeo4w/apps/qgis/./python\qgis\utils.py", line 672, in import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "H:/gp181030/osgeo4w/apps/qgis/./python/plugins\processing\
_init__.py", line 29, in
from processing.tools.general import * # NOQA
File "H:/gp181030/osgeo4w/apps/qgis/./python\qgis\utils.py", line 672, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "H:/gp181030/osgeo4w/apps/qgis/./python/plugins\processing\tools\general.py", line 39, in
from processing.core.Processing import Processing
File "H:/gp181030/osgeo4w/apps/qgis/./python\qgis\utils.py", line 672, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "H:/gp181030/osgeo4w/apps/qgis/./python/plugins\processing\core\Processing.py", line 55, in
from processing.script import ScriptUtils
File "H:/gp181030/osgeo4w/apps/qgis/./python\qgis\utils.py", line 672, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "H:/gp181030/osgeo4w/apps/qgis/./python/plugins\processing\script\ScriptUtils.py", line 28, in
from qgis.processing import alg as algfactory
File "H:/gp181030/osgeo4w/apps/qgis/./python\qgis\utils.py", line 672, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
ModuleNotFoundError: No module named 'qgis.processing'

What i have done to overcome the originally issue is running a script to change entries in the ini-Files QGIS3.ini in Profiles/default/QGIS and qgis_global_settings.ini in apps/qgis/resources:

import os
from PyQt5.QtCore import QSettings
inipath_profile = os.path.join(os.environ['Home'],'.qis3/profiles/default/QGIS/QGIS3.ini')
if os.path.exists(inipath_profile):
settings_profile = QSettings(inipath_profile, QSettings.IniFormat);
else:
settings_profile = None
inipath_global = os.path.join(os.environ['OSGEO4W_ROOT'],'apps/qgis/resources/qgis_global_settings.ini')
settings_global = QSettings(inipath_global, QSettings.IniFormat);
envdict = {}
try:
home =os.environ['HOME'] # G:\\gp180823\\Home
envdict["Processing/Configuration/MODELS_FOLDER"] = os.path.realpath(home + "\.qis3\profiles\default\processing\models")
envdict["Processing/Configuration/OUTPUTS_FOLDER"] = os.path.realpath(home + "\.qis3\profiles\default\processing\outputs")
envdict["Processing/Configuration/SCRIPTS_FOLDERS"] = os.path.realpath(home + "\.qis3\profiles\default\processing\scripts")
envdict["qgis/projectTemplateDir"] = os.path.realpath(home + "\.qis3\profiles\default/project_templates")
except Exception as e:
exit("Environment variable 'Home' not set!")
try:
prefix = os.environ['GISBASE'] # G:\gp180823\osgeo4w\apps\grass\grass-7.4.1
envdict["Processing/Configuration/GRASS7_FOLDER"] = os.path.realpath(prefix)
envdict["Processing/Configuration/GRASS_HELP_PATH"] = os.path.realpath(prefix + "\docs\html")
except Exception as e:
print ("Environment variable 'GISBASE' not set!")
for k in envdict.keys():
settings_global.setValue(k, envdict[k])
print('globals', k, envdict[k])
if settings_profile is not None and settings_profile.contains(k):
print('key %s exists in profile'%k)
if not os.path.exists(settings_profile.value(k)):
print('path %s does not exists, writing default to profile'%settings_profile.value(k))
print('profiles', k, envdict[k])
settings_profile.setValue(k, envdict[k])
else:
print('path %s exists, left it in profiles!'%settings_profile.value(k))
else:
print('key %s does not exists in profile, left empty', k)
settings_global.sync()
if settings_profile is not None:
settings_profile.sync()

I know that's not a proper solution, but may be it shows better, what/where the issue is...

#9 Updated by Luigi Pirelli 10 months ago

Andreas Mueller wrote:

Thank you for your efforts Luigi,

i looked into the test, but i don't totally understand the code, I'm sorry. Also I never build qgis from source and it is not an option at the moment.

no problem

But i tried to use the new scripts ScriptAlgorithmProvider.py and ScriptUtils.py with my installed qqis (3.4.3) which gives an error, so i think there are other changes in the code since 3.4.3. Error is:

confirmed... but the Patch is for master not for 3.4.x where ScriptUtils has a different code base... you should use qgis-dev in osgeo4w

What i have done to overcome the originally issue is running a script to change entries in the ini-Files QGIS3.ini in Profiles/default/QGIS and qgis_global_settings.ini in apps/qgis/resources:

I know that's not a proper solution, but may be it shows better, what/where the issue is...

my patch works only for Script path but the function I added can be used to reset any config path to the acrual profile path

#10 Updated by Luigi Pirelli 10 months ago

BTW no idea why you using .qgis3... scripts re usually saved in:
C:\Users\<yout user>\AppData\Roaming\QGIS\QGIS3\profiles\<yout profile>\processing\scripts

what kind of installer are you using? I'm using the advanced OSGeo4W installer (not the standalone application) that shoudl istall stuffs in the same paths

#11 Updated by Andreas Mueller 10 months ago

I use apt by Matt Wilkie (https://github.com/maphew/apt), which i changed to use modified postinstall batch-files for a installation on removable drives. But the reason why i use .qgis3 for profiles is that i wanted to store the profiles along with qgis on an usb-stick and I start qgis with --profiles-path. I used the according option under qgis2 too, then it pointed to .qgis2.
Since i have more "portable" software on usb sticks, i encountered, that it is possible to redirect even AppData through an environment variable, so this could also be an option. Some Qt-Applications I tested unfortunately don't use the environment variables APPDATA and LOCALAPPDATA and i didn't test it for qgis.

#12 Updated by Luigi Pirelli 10 months ago

I'll check if my patch works in your custom configuration

#13 Updated by Luigi Pirelli 7 months ago

partially fixed by https://github.com/qgis/QGIS/pull/9172 but IMHO I do not think it cover all. Please check @andreas

#14 Updated by Giovanni Manghi 7 months ago

  • Status changed from Open to Feedback

Also available in: Atom PDF