I understand, that Ignore Shapefile Encoding has been set to true for new created vector layers:

75dc85b

But why has it been generally set to true since QGIS 2.0 ?

ddb5117

This default true option leads to corrupted encoding if doing the following:

1. Creating new vector layer, adding objects (with special characters in the attribute) and saving as Shapefile (UTF-8 is used as default, .cpg is automatically created)
2. Later: Opening created shapefile using drag-and-drop (Encoding is then ignored upon loading since Ignore Shapefile Encoding is true by default)
3. Selecting objects from vector layer and saving it to new shapefile (only selected objects), again using UTF-8 (as default encoding)

Surprise: special chars in file are now corrupted!

Same happens when saving file to kml (always using UTF-8!).

In short: IMHO the default option is not compatible to drag-and-drop usage.

Please explain and/or add comments to documentation - aspecially in the shapefile part as mentioned above, I know there is a really short description in options part but this doesn't really help if someone does not expect such a behaviour. It still seems to be a bug to me (according to what a user expects), although I am pretty sure this option was not set to true just by accident. But I do not understand why it was. In QGIS 1.9 this option was originally set to false which makes more sense to me.

Well, after more investigating into the issue, I found the following discussion going back to QGIS 1.8, why this was changed #6500

What would be the downside of setting QGIS to ignore GDAL's shapefile encoding by default?

Probably nothing. I also think it is preferable to disable encoding conversion of Shapefile layer in the present situation, for example, the handling of LDID/87 and OGR interface about encoding. All the non-ASCII characters using users will need the option, so it should be checked by default.

My question: Is this still true for current QGIS versions? As I mentioned above, there are downsides...

All the non-ASCII characters using users will need the option, so it should be checked by default.

All users using non-ASCII characters currently need to ignore GDAL shapefile encoding feature, and all users using ASCII characters will not be affected by turning this option on by default.

That's a win win solution :)

Again: IMHO it's not any more a win win situation since it interferes with drag and drop usage of QGIS 2.x.

Anyone who was in this discussion 3 years ago, could you please check if this default option still makes sense? I could not see any problems when disabling this option (Ignore Shapefile Encoding), but if option is active by default if interferes with drag and drop usage.

Hi,

In Japan, we usually get shapefiles that encoding is CP932 (the code page in Japanese windows). Bad thing is that they sometimes have LDID/87 in the LDID field of dbf file. OGR shapefile driver handles LDID/87 as ISO-8859-1 so character corruption occurs.
Edited: 2015-09-05

When the "Ignore Shapefile Encoding" option is not checked and a shapefile to be loaded has non-zero LDID or a cpg file, the encoding selection on the open vector layer dialog is not applied to the layer. In this case, the user selection is ignored. Experienced users might be able to avoid the corruption by creating a cpg file or checking the option, but I am afraid of beginners' (or general users') confusion. So I think the option should be checked by default.

IMHO, a more flexible way to deal with this issue:
- When a shapefile is opened by drag & drop, QGIS tries to read the LDID / cpg file in its side and apply the encoding to the layer.
- The Ignore Shapefile Encoding option is always checked (maybe will be removed), so the layer encoding should be able to change on the properties dialog.

I am not sure if I get it right, but if I understand it correctly the automatic shapefile encoding detection is blocking manual selection of encoding in dialog (if not ignored). Therefore, if detection fails like e.g. for Japanese CP932 Encoding (or other non ASCII compatible encodings), user has to select encoding manually, but can not do so if "Ignore Shapefile Encoding" is not true.
What happens if a Japanese CP932 Shapefile is being opened by drag & drop? Will it fail? Does the "Ignore" option has any effect on it?

IMHO, same flexible way as mentioned by Minoru Akagi, but from different view:
- Moving the "Ignore Shapefile Encoding" option from general properties to the dialog (can be always checked unless deselected by user) so it will only have effect in the dialog (but not on drag & drop)
- Drag & Drop will not be affected by "Ignore Shapefile Encoding" at all (if shapefile is ISO-8859-1 encoding it should be detected properly, in any other case it is better to at least try to detect encoding instead of always using default ISO-8859-1 which may be wrong)

Adrian Klink wrote:

What happens if a Japanese CP932 Shapefile is being opened by drag & drop? Will it fail? Does the "Ignore" option has any effect on it?

What the layer encoding is set to and whether characters in the attribute table are right or not when a CP932 shapefile is opened by drag & drop:

• Ignore Shapefile Encoding option: ON

  OS＼LDID	LDID/0	LDID/19
  Win	System (CP932) *	System (CP932) *
  Ubuntu	System (UTF-8) **	System (UTF-8) **

• Ignore Shapefile Encoding option: OFF

  OS＼LDID	LDID/0	LDID/19
  Win	System (CP932) *	UTF-8 (grayed-out) *
  Ubuntu	System (UTF-8) **	UTF-8 (grayed-out) *

*   Fine
**  Characters are wrong, but user can correct the encoding setting on the layer properties dialog.

LDID/19 means CP932.

Edited: 2015-09-05

@Adrian Klink, I'm sorry. I had a misunderstanding. I do not get CP932 shapefiles with LDID/87. I get shapefiles with LDID/0 or LDID/19. I've edited my above comments.

And there is a nice plugin to fix the encoding declaration on GUI: Shapefile Encoding Fixer

@Minoru Akagi: Thank you very much for investigating into Japanese CP932 Shapefile opening by Drag&Drop with and w/o Ignore Shapefile Encoding option. And thank you for the link to the Shapefile Encoding Fixer Plugin. It is very usefull.

So I think we do agree that Option "Ignore Shapefile Encoding" should be handled differently for Drag&Drop (disabled) and opening via Dialog (enabled), unlike it is currently implemented. The remaining question is how this can be done the best way without any side effects.

I just ran into the same issue...

Opening Shapefiles using Drag and Drop should definitely use .cpg-File-Settings.

So as mentioned above
we should move
"the "Ignore Shapefile Encoding" option from general properties to the dialog (can be always checked unless deselected by user) so it will only have effect in the dialog (but not on drag & drop)"

So
"Drag & Drop will not be affected by "Ignore Shapefile Encoding" at all (if shapefile is ISO-8859-1 encoding it should be detected properly, in any other case it is better to at least try to detect encoding instead of always using default ISO-8859-1 which may be wrong)

Thanks Peter

This is really unexpected behaviour. Ignoring the .cpg file seems weird to me. Users can override the encoding the open dialog so if they must, the option exists.

The option is not clear, what exactly does it mean if one disables "on-the-fly conversion to UTF-8"? Our files might be declared as UTF-8 in the .cpg, what happens if we untick the option?

Please at least notify the user in a message about the ignoring. If magic must happen, maybe use something to try to detect the encoding?

.cpg files are there for a reason, it is not a good idea to assume that the system encoding is a sane, modern one (Hi Windows!). This just gave me flashbacks to ArcGIS...

+1, same problem here on Windows...

I agree to it. The default "IgnoreShapefileEncoding" causes way too much confusion! It should be off by default and can be enabled if necessary.

@Minoru Akagi: Can you please make a Pull request to revert this commit ddb5117 for further qgis versions?

Commit ddb5117:

src/providers/ogr/qgsogrprovider.cpp

CPLSetConfigOption( "SHAPE_ENCODING", settings.value( "/qgis/ignoreShapeEncoding", true ).toBool() ? "" : 0 );

cbxIgnoreShapeEncoding->setChecked( settings.value( "/qgis/ignoreShapeEncoding", true ).toBool() );

Hi,
This problem is also on Windows 7 64x and QGIS 3.2.2 Drag & Drop don't use the CPG file. I need set it manually in source panel

It's in relation with ignoreShapeEncoding

Because default value is true. So there is no autodetection in drag&drop

But if you set ignoreShapeEncoding=false in QGIS.ini the encoding is UTF-8 and in layer properties, encoding is not editable (grey combobox is shaded)

I edit this directety in QGIS.ini textfile because there is an error on parameters dialogbox #19741

Giovanni Manghi wrote:

that error is caused by a 3rd party plugin, not qgis itself. If you remove/disable the plugin it works as expected?

I have desactive all 3rd party plugin

I do a test to activate an desactivate the ignore encoding checkbox. There is no crash but there is no modification to...

Encoding is gray all time

My test¶

1. Save a layer in CP1252 > It is added in list of layers

1. Check encoding in properties. > combobox is gray and value is UTF8

1. Save project and change Ignore Shapefile Encoding have no effect > there is an other problem here

1. I set Ignore Shapefile Encoding to True in properties panel

1. Drag&Drop my file in map

1. Check properties > That is in encoding System so that is not ignored? The value are revesed???

1. I set Ignore Shapefile Encoding to False

1. Drag&Drop my file in map

1. Check properties > That is in encoding utf-8 so that is not ignored? The value are really revesed!

That is not ignore shapefile encoding but use shapefile encoding

My test¶

1. Save a layer in CP1252 > It is added in list of layers

1. Check encoding in properties. > combobox is gray and value is UTF8

1. Save project and change Ignore Shapefile Encoding have no effect > there is an other problem here

1. I set Ignore Shapefile Encoding to True in properties panel

1. Drag&Drop my file in map

1. Check properties > That is in encoding System so that is not ignored? The value are revesed???

1. I set Ignore Shapefile Encoding to False

1. Drag&Drop my file in map

1. Check properties > That is in encoding utf-8 so that is not ignored? The value are really revesed!

That is not ignore shapefile encoding but use shapefile encoding

what qgis version?

Version de QGIS
3.2.2-Bonn
Révision du code
26842169e9
Compilé avec Qt
5.9.2
Utilisant Qt
5.9.2
Compilé avec GDAL/OGR
2.2.4
Utilisé avec GDAL/OGR
2.2.4
Compilé avec GEOS
3.6.1-CAPI-1.10.1
Utilisé avec GEOS
3.6.1-CAPI-1.10.1 r0
Version du client PostgreSQL
9.2.4
Version de SpatiaLite
4.3.0
Version de QWT
6.1.3
Version de QScintilla2
2.10.1
Version de PROJ.4 :
493

Related #21313