Bug report #15293

Encoding problem with PyQGIS on Windows

Added by Patrice Pineault over 8 years ago. Updated over 7 years ago.

Status:Closed
Priority:Normal
Assignee:-
Category:Python plugins
Affected QGIS version:2.16.0 Regression?:No
Operating System:Windows Easy fix?:No
Pull Request or Patch supplied:No Resolution:fixed/implemented
Crashes QGIS or corrupts data:No Copied to github as #:23227

Description

The bug is about character encoding when using geoprocessing algorithm from the toolbox, using PyQGIS.

It affects 2.16 but also previous versions (Windows).

The bug is not reproducable in 2.4 (Linux Debian)

However, the behaviour is a slightly different in 2.16, so I am creating the issue for 2.16.

Here are the steps for reproducing the bug:

  • Installing QGIS 2.16 from osgeo4w.
  • Installing my Python plugin (the plugin is in C:\\OSGeo4W64\\apps\\qgis\\python\\plugins\\myplugin)
  • Run the plugin. There, the output has correct encoding (same as input, UTF-8).
  • Exit QGIS
  • Re-Open QGIS
    - ERROR:
    An error occurred during execution of following code:
import pyplugin_installer

UnicodeEncodeError: 'ascii' codec can't encode character u'\\xe9' in position 12: ordinal not in range(128)

Python version:
2.7.5 (default, May 15 2013, 22:44:16) [MSC v.1500 64 bit (AMD64)]

QGIS version:
2.16.0-Nødebo 'Nødebo', d0b3e39

Python path:
['C:/OSGEO4~1/apps/qgis/./python/plugins\\\\processing', 'C:/OSGEO4~1/apps/qgis/./python', u'C:/Users/Gobtron/.qgis2/python', u'C:/Users/Gobtron/.qgis2/python/plugins', 'C:/OSGEO4~1/apps/qgis/./python/plugins', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\matplotlib-1.3.1-py2.7-win-amd64.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\
ose-1.3.3-py2.7.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\tornado-4.0.1-py2.7-win-amd64.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\backports.ssl_match_hostname-3.4.0.2-py2.7.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\certifi-14.05.14-py2.7.egg', 'c:\\\\osgeo4~1\\\\apps\\\\python27\\\\lib\\\\site-packages\\\\python_dateutil-2.1-py2.7.egg', 'C:\\\\Python27\\\\ArcGIS10.2', 'C:\\\\OSGEO4~1\\\\bin\\\\python27.zip', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\DLLs', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\plat-win', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\lib-tk', 'C:\\\\OSGEO4~1\\\\bin', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\PIL', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\jinja2-2.7.2-py2.7.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\markupsafe-0.23-py2.7-win-amd64.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\pytz-2012j-py2.7.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\win32', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\win32\\\\lib', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\Pythonwin', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\Shapely-1.2.18-py2.7-win-amd64.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\wx-2.8-msw-unicode', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\xlrd-0.9.2-py2.7.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\xlwt-0.7.5-py2.7.egg', u'C:/Users/Gobtron/.qgis2//python']
  • Another error:
    An error occurred during execution of following code:
pyplugin_installer.initPluginInstaller()

Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'pyplugin_installer' is not defined

Python version:
2.7.5 (default, May 15 2013, 22:44:16) [MSC v.1500 64 bit (AMD64)]

QGIS version:
2.16.0-Nødebo 'Nødebo', d0b3e39

Python path:
['C:/OSGEO4~1/apps/qgis/./python/plugins\\\\processing', 'C:/OSGEO4~1/apps/qgis/./python', u'C:/Users/Gobtron/.qgis2/python', u'C:/Users/Gobtron/.qgis2/python/plugins', 'C:/OSGEO4~1/apps/qgis/./python/plugins', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\matplotlib-1.3.1-py2.7-win-amd64.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\
ose-1.3.3-py2.7.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\tornado-4.0.1-py2.7-win-amd64.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\backports.ssl_match_hostname-3.4.0.2-py2.7.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\certifi-14.05.14-py2.7.egg', 'c:\\\\osgeo4~1\\\\apps\\\\python27\\\\lib\\\\site-packages\\\\python_dateutil-2.1-py2.7.egg', 'C:\\\\Python27\\\\ArcGIS10.2', 'C:\\\\OSGEO4~1\\\\bin\\\\python27.zip', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\DLLs', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\plat-win', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\lib-tk', 'C:\\\\OSGEO4~1\\\\bin', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\PIL', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\jinja2-2.7.2-py2.7.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\markupsafe-0.23-py2.7-win-amd64.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\pytz-2012j-py2.7.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\win32', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\win32\\\\lib', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\Pythonwin', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\Shapely-1.2.18-py2.7-win-amd64.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\wx-2.8-msw-unicode', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\xlrd-0.9.2-py2.7.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\xlwt-0.7.5-py2.7.egg', u'C:/Users/Gobtron/.qgis2//python']
  • Then QGIS opens, but my plugin is absent
  • If I open 'Manage and Install Plugins...', I get this error:
    An error occurred during execution of following code:
pyplugin_installer.instance().showPluginManagerWhenReady()

Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'pyplugin_installer' is not defined

Python version:
2.7.5 (default, May 15 2013, 22:44:16) [MSC v.1500 64 bit (AMD64)]

QGIS version:
2.16.0-Nødebo 'Nødebo', d0b3e39

Python path:
['C:/OSGEO4~1/apps/qgis/./python/plugins\\\\processing', 'C:/OSGEO4~1/apps/qgis/./python', u'C:/Users/Gobtron/.qgis2/python', u'C:/Users/Gobtron/.qgis2/python/plugins', 'C:/OSGEO4~1/apps/qgis/./python/plugins', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\matplotlib-1.3.1-py2.7-win-amd64.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\
ose-1.3.3-py2.7.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\tornado-4.0.1-py2.7-win-amd64.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\backports.ssl_match_hostname-3.4.0.2-py2.7.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\certifi-14.05.14-py2.7.egg', 'c:\\\\osgeo4~1\\\\apps\\\\python27\\\\lib\\\\site-packages\\\\python_dateutil-2.1-py2.7.egg', 'C:\\\\Python27\\\\ArcGIS10.2', 'C:\\\\OSGEO4~1\\\\bin\\\\python27.zip', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\DLLs', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\plat-win', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\lib-tk', 'C:\\\\OSGEO4~1\\\\bin', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\PIL', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\jinja2-2.7.2-py2.7.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\markupsafe-0.23-py2.7-win-amd64.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\pytz-2012j-py2.7.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\win32', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\win32\\\\lib', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\Pythonwin', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\Shapely-1.2.18-py2.7-win-amd64.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\wx-2.8-msw-unicode', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\xlrd-0.9.2-py2.7.egg', 'C:\\\\OSGEO4~1\\\\apps\\\\Python27\\\\lib\\\\site-packages\\\\xlwt-0.7.5-py2.7.egg', u'C:/Users/Gobtron/.qgis2//python']
  • Exit QGIS
  • Re-Open QGIS
  • Open 'Manage and Install Plugin...'
  • My plugin is uninstalled
  • Re-install my plugin
  • Run the plugin
  • Encoding is broken and different from source (UTF-8).

History

#1 Updated by Regis Haubourg over 8 years ago

Hi, we have the same problem in a plugin.
I tried all good practices with character encoding but it seems - on windows at least - that we have something strange with that new version name:
We should largely advertise plugin authors since it breaks many plugins (see here for instance https://github.com/etiennesky/profiletool/issues/15 )

Can someone help find what is wrong here, this is what I tried from python console supposing it's Utf-8:

v = u'2.16.0-Nødebo'
type(v)
<type 'unicode'>
>>>v
u'2.16.0-N\\xf8debo'
>>>v = v.encode('utf-8')
Traceback (most recent call last):
  File "<input>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 8: ordinal not in range(128)
>>>v = v.encode('ISO-8859-2')
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\\OSGEO4~1\\apps\\Python27\\lib\\encodings\\iso8859_2.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 8: ordinal not in range(128)

and supposing it's not utf-8:

>>>v = '2.16.0-Nødebo'
>>>v
'2.16.0-N\\xc3\\xb8debo'
>>>type(v)
<type 'str'>
>>>v = v.encode('utf-8')
Traceback (most recent call last):
  File "<input>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 8: ordinal not in range(128)

>>>v = v.encode('ISO-8859-2')
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\\OSGEO4~1\\apps\\Python27\\lib\\encodings\\iso8859_2.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 8: ordinal not in range(128)

#2 Updated by Alain FERRATON over 8 years ago

QGIS 2.16 has a problem when there are accented characters in file metadata.txt of a plugin.

We usually encode our files metadata.txt in UTF-8 and it was working fine until 2.14.

But QGIS 2.16 seems to open that file with the ASCII encoding.

Hence the errors "UnicodeEncodeError: 'ascii' codec can not encode character ..." if accents. If we eliminate the accented characters in metadata.txt, the bug no longer occurs.

Replace open ("medata.txt", "r") with codecs.open ("medata.txt", 'r','utf8') in the code of plugin manager ?

#3 Updated by Johannes Kroeger over 8 years ago

I get the same bug on Archlinux.

First message:

An error occurred during execution of following code:
import pyplugin_installer

Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/share/qgis/python/qgis/utils.py", line 313, in loadPlugin
    showException(sys.exc_info()[0], sys.exc_info()[1], sys.exc_info()[2], msg, messagebar=True)
  File "/usr/share/qgis/python/qgis/utils.py", line 107, in showException
    open_stack_dialog(type, value, tb, msg)
  File "/usr/share/qgis/python/qgis/utils.py", line 185, in open_stack_dialog
    pypath=u"".join(u" 
{}
".format(path) for path in sys.path))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 8: ordinal not in range(128)

Python version:
2.7.12 (default, Jun 28 2016, 08:31:05) 
[GCC 6.1.1 20160602]

QGIS version:
2.16.0-Nødebo 'Nødebo', exported

Python path:
['/usr/share/qgis/python', u'/home/me/.qgis2/python', u'/home/me/.qgis2/python/plugins', '/usr/share/qgis/python/plugins', '/usr/lib/python27.zip', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/home/me/.local/lib/python2.7/site-packages', '/usr/lib/python2.7/site-packages', '/usr/lib/python2.7/site-packages/gst-0.10', '/usr/lib/python2.7/site-packages/gtk-2.0', '/usr/lib/python2.7/site-packages/wx-3.0-gtk2', u'/home/me/.qgis2//python']

I replaced my username with "me", my username is just standard lowercase ascii.

After I hit "close" and when I try to access the plugins menu I get similar errors like mentioned by OP above so I won't repeat them.

What really confuses me is that if I replace the supposed offending line
pypath=u"".join(u"<li>{}</li>".format(path) for path in sys.path))
with
pypath="")
I still get

Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/share/qgis/python/qgis/utils.py", line 314, in loadPlugin
    showException(sys.exc_info()[0], sys.exc_info()[1], sys.exc_info()[2], msg, messagebar=True)
  File "/usr/share/qgis/python/qgis/utils.py", line 107, in showException
    open_stack_dialog(type, value, tb, msg)
  File "/usr/share/qgis/python/qgis/utils.py", line 185, in open_stack_dialog
    pypath="")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 8: ordinal not in range(128)

If I comment out the whole txt= block, I get similar errors now about the processing plugin but then I can actually open the plugins menu.

If I put both QGis.QGIS_VERSION ("2.16.0-Nødebo") and QGis.QGIS_RELEASE_NAME ("Nødebo") in quotes (so the string is used, not the variable value), QGIS starts without these errors (but others which I am ignoring for now) and I can open the plugins manager.

#4 Updated by Johannes Kroeger over 8 years ago

Decoding both QGis.QGIS_VERSION and QGis.QGIS_RELEASE_NAME as UTF-8 resolves the error but I assume the unicode handling needs fixing higher up.

    txt = txt.format(msg="msg",
                     main_error=main_error,
                     error=error,
                     version_label=version_label,
                     num=sys.version,
                     qgis_label=qgis_label,
                     qversion=QGis.QGIS_VERSION.decode("utf-8"),
                     qgisrelease=QGis.QGIS_RELEASE_NAME.decode("utf-8"),
                     devversion=QGis.QGIS_DEV_VERSION,
                     pypath_label="pypath_label",
                     pypath=u"".join(u"<li>{}</li>".format(path) for path in sys.path))

#5 Updated by Giovanni Manghi over 7 years ago

  • Priority changed from High to Normal
  • Crashes QGIS or corrupts data changed from Yes to No

#6 Updated by Alexander Bruy over 7 years ago

  • Status changed from Open to Feedback
  • OS version deleted (Windows 8, Windows 10)

As 2.18 already out and will be next LTR I think we can close this issue.

#7 Updated by Giovanni Manghi over 7 years ago

  • Regression? set to No
  • Easy fix? set to No

#8 Updated by Giovanni Manghi over 7 years ago

  • Resolution set to fixed/implemented
  • Description updated (diff)
  • Status changed from Feedback to Closed

Also available in: Atom PDF