Bug report #17221

imports delimited text: the decimal separator (,) is not recognized

Added by salvatore fiandaca about 7 years ago. Updated about 7 years ago.

Status:Closed
Priority:High
Assignee:-
Category:Data Provider/Delimited Text
Affected QGIS version:master Regression?:Yes
Operating System:win 10 x64 and Ubuntu 16.04 - 64 bit Easy fix?:No
Pull Request or Patch supplied:No Resolution:invalid
Crashes QGIS or corrupts data:No Copied to github as #:25120

Description

importing a csv with decimal separator (,) is read only as a table, while with (.) is imported as a layer point.

test on win 10 x64 and Ubuntu 16.04 - 64 bit
Local settings: ITA both system and QGIS

the problem does not exist in QGIS 2.18

I attach two CSV files for the test

Thanks for the excellent work

test_not_ok.csv Magnifier - decimal separator (,) (308 Bytes) salvatore fiandaca, 2017-09-29 03:55 PM

test_ok.csv Magnifier - decimal separator (.) (308 Bytes) salvatore fiandaca, 2017-09-29 03:55 PM

csvnotok.png - working in master (188 KB) Richard Duivenvoorde, 2017-10-01 11:19 AM

N1.jpg - field options were hidden (365 KB) salvatore fiandaca, 2017-10-01 12:02 PM

History

#1 Updated by Richard Duivenvoorde about 7 years ago

For what I see, the code [0] is not using the locale of the machine, but is rather looking at the 'option' "Decimal separator is comma".
Here with me that is the third checkbox in the row 'Field options'

If I check that one (and use for all the other options the right choices...) I can load your 'test_not_ok.csv' just fine.
BUT I must say I use the english locale here. So not sure if the result is different if you use a european locale.

Did you check using the "Decimal separator is comma" option?

Mmm, looking at [1] it looks like it triest to hardcoded replace comma's with points, which in case of an european locale results in non valid numbers I think?
Can somebody confirm?

[0] https://github.com/qgis/QGIS/blob/release-2_18/src/providers/delimitedtext/qgsdelimitedtextprovider.cpp#L521
[1] https://github.com/qgis/QGIS/blob/release-2_18/src/providers/delimitedtext/qgsdelimitedtextprovider.cpp#L873

#2 Updated by Richard Duivenvoorde about 7 years ago

Ok, did some more testing on a Windows laptop with nl_NL locale, and both your files work there too (as long as you check the 'comma' checkbox.

Ah, talking about master here: this is a little more 'hidden' in the 'fields' options.

I also tried to start a discussion on dev list: https://lists.osgeo.org/pipermail/qgis-developer/2017-September/049892.html

But my main conclusion is that:
- the csv provider 'just works', but the user itself has to let QGIS know that the strings in the text file are comma-decimal-separated
- it would be cool if QGIS checks the locale of the machine and based on that maybe auto-checks this 'comma' checkbox
- but fact is that even european people (like me) often just use the point as a decimal separator..

Closing this issue, but please reopen with some discussion if you think above conclusion is wrong.

#3 Updated by salvatore fiandaca about 7 years ago

Richard Duivenvoorde wrote:

Ok, did some more testing on a Windows laptop with nl_NL locale, and both your files work there too (as long as you check the 'comma' checkbox.

Ah, talking about master here: this is a little more 'hidden' in the 'fields' options.

I also tried to start a discussion on dev list: https://lists.osgeo.org/pipermail/qgis-developer/2017-September/049892.html

But my main conclusion is that:
- the csv provider 'just works', but the user itself has to let QGIS know that the strings in the text file are comma-decimal-separated
- it would be cool if QGIS checks the locale of the machine and based on that maybe auto-checks this 'comma' checkbox
- but fact is that even european people (like me) often just use the point as a decimal separator..

Closing this issue, but please reopen with some discussion if you think above conclusion is wrong.

as described in the description, the problem does not exist in QGIS 2.18

#4 Updated by Giovanni Manghi about 7 years ago

  • Status changed from Open to Feedback

as described in the description, the problem does not exist in QGIS 2.18

As Richard said using the option "decimal separator is comma" makes qgis import it normally as point layer. Are you using that?

#5 Updated by Richard Duivenvoorde about 7 years ago

See screenshot, with me both csv files are being correctly opened both in master and in 2.18 both on Linux and Windows and in en_US locale and in nl_NL locale. That is IF you enable the 'comma option'

#6 Updated by salvatore fiandaca about 7 years ago

Richard Duivenvoorde wrote:

See screenshot, with me both csv files are being correctly opened both in master and in 2.18 both on Linux and Windows and in en_US locale and in nl_NL locale. That is IF you enable the 'comma option'

Thanks, WORK !!!
field options were hidden !!!

#7 Updated by salvatore fiandaca about 7 years ago

Giovanni Manghi wrote:

as described in the description, the problem does not exist in QGIS 2.18

As Richard said using the option "decimal separator is comma" makes qgis import it normally as point layer. Are you using that?

The field option is hidden, now it works !!!
thank you

#8 Updated by Giovanni Manghi about 7 years ago

  • Resolution set to invalid
  • Status changed from Feedback to Closed

Also available in: Atom PDF