Bug report #17864

Graphical modeler doesn`t preserve utf 8 encoding

Added by Christian Metzger almost 3 years ago. Updated about 2 years ago.

Status:Closed
Priority:Normal
Assignee:Victor Olaya
Category:Processing/Modeller
Affected QGIS version:2.18.15 Regression?:No
Operating System:windows 7 64 bit Easy fix?:Yes
Pull Request or Patch supplied:No Resolution:fixed/implemented
Crashes QGIS or corrupts data:No Copied to github as #:25759

Description

qgis processing algorithms which I use in a model don`t preserve utf 8 encoding.

test.model (3.01 KB) Christian Metzger, 2018-01-16 06:46 AM

testshp.7z (1.81 KB) Christian Metzger, 2018-01-16 06:49 AM

Setting_environmentvariables.png (6.45 KB) Christian Metzger, 2018-02-14 10:26 AM

History

#1 Updated by Mathieu Pellerin - nIRV over 2 years ago

  • Status changed from Open to Feedback

Christian, can you be a bit more specific here? I'm a regular user of utf8-encoded datasets (including shapefiles), and QGIS algorithms within models are fine on my machine.

FYI, using GRASS algorithm will most likely result in loss of utf8-encoded VARCHAR/TEXT fields, but that's not related to QGIS or its modeler. You can preserve utf8-encoding of your shapefile by adding the following system variable:

SHAPE_ENCODING=UTF-8

You can add that through QGIS's preference window (System -> Environment [variables]). Cheers

#2 Updated by Christian Metzger over 2 years ago

Hi Mathieu,

thank you for the answer I tried the hint with the environment variables but it still doesn`t work (I attached a screenshot with my settings of environment variable)
Do I have to reboot my computer to set the environment variables or only restart QGIS?

My utf 8 encoding is still corrupted when running a model or executing a GRASS algorithm.
I am using QGIS 2.18.16 now. I installed from OsGeo4W.
I attached a test model (very simple, save selected features) and a test shapefile to my issue.
Could you may run this model with the attached shapefile and check if the utf 8 encoding (Laotian script) has been preserved?

#3 Updated by Nyall Dawson about 2 years ago

Please try with 3.3 nightlies and report if this is still an issue - the whole modeler backend has been rewritten and this is likely no longer an issue.

#4 Updated by Christian Metzger about 2 years ago

Thank you Nyall for the hint.
I found a solution that also works for QGIS 2.18. The problem was that utf 8 encoding wasn`t enabled for the processing environment. To enable it in windows environment I needed to go to the registry HKEY_CURRENT_USER\Software\QGIS\QGIS2\Processing and enable utf 8 encoding

#5 Updated by Nyall Dawson about 2 years ago

  • Resolution set to fixed/implemented
  • Status changed from Feedback to Closed

Also available in: Atom PDF