Bug report #20053

decimal separator in csv files

Added by Soren Lindqvist about 6 years ago. Updated over 5 years ago.

Status:Closed
Priority:Normal
Assignee:Alessandro Pasotti
Category:Unknown
Affected QGIS version:3.5(master) Regression?:No
Operating System:Mac OSX 10.14.3 Easy fix?:No
Pull Request or Patch supplied:Yes Resolution:no timely feedback
Crashes QGIS or corrupts data:No Copied to github as #:27875

Description

When opening cvs files with data that contains decimals separator with "." it change after opening in QGIS to "," the problem then is that when saving the csv file to GeoJson it will become a "string" instead of numbers. This habit has come with version 3, it does not do this in 2.18 (never had).

Associated revisions

Revision 859b39a6
Added by Alessandro Pasotti about 6 years ago

Fixes #20053 decimal separator in csv files

Actually it had nothing to do with CSV being the
source, but it was the json exporter passing
the values through all field formatters except for
the fallback.

This resulted in all fields using a 'Range' formatter
(which is the default for all numeric types) passing
through the formatter and being returned as strings
in the json. Worse, if the locale was not a "dot"
locale and decimal separator was on, the resulting
string could not be easily converted into its original
numeric type.

Now, instead of checking for the fallback formatter
only, there is a white list of formatters that
can be applied when we want a json.

This is a temporary solution because the "right" way
to do it would be either a flag in the formatter to
tell if it can be applied when converting to json
and/or other "data" formats (csv etc.) or a different
new method similar to representValue.

Revision b80c95c7
Added by Alessandro Pasotti about 6 years ago

Merge pull request #8150 from elpaso/bugfix-20053-json-exporter-values

Fixes #20053 decimal separator in csv files

History

#1 Updated by Alessandro Pasotti about 6 years ago

  • Assignee set to Alessandro Pasotti

Just to be sure: if you change your locale number formatting options to "English" or "C" in QGIS options general tab, does that fixes the problem (override current locale must be checked) ?

#2 Updated by Giovanni Manghi about 6 years ago

  • Regression? changed from No to Yes
  • Status changed from Open to Feedback
  • Priority changed from Normal to High

was the csv added with the "add delimited text layer" tool?

#3 Updated by Alessandro Pasotti about 6 years ago

  • Pull Request or Patch supplied changed from No to Yes
  • Status changed from Feedback to In Progress
  • Priority changed from High to Normal
  • Regression? changed from Yes to No

#4 Updated by Alessandro Pasotti about 6 years ago

Giovanni, this had actually nothing to do with CSV, it was a bug in the json formatter.

#5 Updated by Soren Lindqvist about 6 years ago

Alessandro Pasotti wrote:

Just to be sure: if you change your locale number formatting options to "English" or "C" in QGIS options general tab, does that fixes the problem (override current locale must be checked) ?

Yes, I did right now change the locale formatting options to "C", the import does not change the "." to "," and exporting to GeoJson is now correct. It seems like the locale settings override "." and also export it as such and turning it to a string instead of number.

Great if you have find the solutions.
Thank you

#6 Updated by Anonymous about 6 years ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Closed

#7 Updated by Alessandro Pasotti about 6 years ago

  • Resolution set to fixed/implemented

Thanks for checking, actually I think that even with C locale the output was not entirely correct: you did not get the ',' decimal separator and you did not get the thousand group separator (like "'" in italian locale), but you still did get the numer as a string (surrounded by quotes).

#8 Updated by Soren Lindqvist almost 6 years ago

  • Status changed from Closed to Reopened

Problem back again....

#9 Updated by Giovanni Manghi almost 6 years ago

  • Status changed from Reopened to Feedback
  • Resolution deleted (fixed/implemented)

Soren Lindqvist wrote:

Problem back again....

on 3.4.4, correct?

#10 Updated by Giovanni Manghi almost 6 years ago

Soren Lindqvist wrote:

Problem back again....

same OS (and version)?

#11 Updated by Soren Lindqvist almost 6 years ago

Yes
went back and installed v 2.18.28, export to geojson was correct!
Problem is now back in v 3.4.4 and in the master 3.5

#12 Updated by Soren Lindqvist almost 6 years ago

Mac OSX 10.14.3

#13 Updated by Giovanni Manghi almost 6 years ago

  • Operating System changed from OSX version 10.11.6 to Mac OSX 10.14.3
  • Status changed from Feedback to Open
  • Affected QGIS version changed from 3.2.3 to 3.5(master)

#14 Updated by Alessandro Pasotti almost 6 years ago

  • Status changed from Open to Feedback

Can you please provide some more detail and a step-by-step procedure and sample data to reproduce?

The original issue has been identified and fixed and it is protected by a test which is not failing, hence it highly likely that you have found a completely new issue.

#15 Updated by Soren Lindqvist almost 6 years ago

I will, at the moment on the road traveling...

#16 Updated by Giovanni Manghi over 5 years ago

Soren Lindqvist wrote:

I will, at the moment on the road traveling...

please leave further feedback.

#17 Updated by Alessandro Pasotti over 5 years ago

  • Status changed from Feedback to Closed
  • Resolution set to no timely feedback

Also available in: Atom PDF