Bug report #16833

Unicode character in query in db_manager makes appear a never ending hourglass

Added by Martin HOFFMANN over 7 years ago. Updated over 7 years ago.

Status:Closed
Priority:High
Assignee:Sandro Santilli
Category:DB Manager
Affected QGIS version:2.18.10 Regression?:Yes
Operating System: Easy fix?:No
Pull Request or Patch supplied:Yes Resolution:
Crashes QGIS or corrupts data:No Copied to github as #:24732

Description

Experiencing big regression which put QGIS in an unstable state (endless rolling cursor) under Windows 10.

The minimal step to reproduce is to put this query in DB Manager connected to a Postgresql instance

SELECT 'é'::text --being an unicode accentuated character

This however work fine under 2.18.3 that is still installed on my colleague workstation.
Might also be related to this bug report #16756

Associated revisions

Revision 9efd666e
Added by Sandro Santilli over 7 years ago

Test that PostGIS query can be passed as both unicode and string literal

See https://issues.qgis.org/issues/16833

Revision 3120068b
Added by Sandro Santilli over 7 years ago

Do not assume input SQL is ASCII

Fixes #16833

History

#1 Updated by Giovanni Manghi over 7 years ago

It works as expected up until 2.18.9

#2 Updated by Martin HOFFMANN over 7 years ago

Guess it was time I learn how to downgrade using OSGeo4W...
I confirm that downgrading to 2.18.9 solve my issue.

I strongly discourage Windows10 users that need unicode support to upgrade until this is solved.

#3 Updated by Jürgen Fischer over 7 years ago

  • Subject changed from Unicode caracter crash db_manager to Unicode character crashes db_manager

#4 Updated by Giovanni Manghi over 7 years ago

  • Subject changed from Unicode character crashes db_manager to Unicode caracter in query in db_manager makes appear a never ending hourglass
  • Crashes QGIS or corrupts data changed from Yes to No

It is also true that neither DB Manager of QGIS crash or are blocked/froze. Just the very annoying hourglass.

#5 Updated by Martin HOFFMANN over 7 years ago

Sorry for the confusion, in fact the hourglass don't prevent manipulating the application and permit for instance to save the current document.
What seems to happen is that the query is stuck and never get_back to the preview window or a layer.

Apparently the error is only triggered where the input SQL query contain Unicode character, Unicode character retunned by the database are handled just fine.

Here is a test table to create in your DB using external tool to check that only input is concerned.

SELECT * INTO test_unicode FROM (VALUES ('ascii','e'),('unicode','é')) foo(test,value)

So if you try to access to that table through database manager query

SELECT * FROM test_unicode                      --This will work
SELECT * FROM test_unicode WHERE value='e'      --This as well
SELECT * FROM test_unicode WHERE value='é'      --This will never return and start and endless hourglass

I tested against Debian Jessie and I get the same error, here is the python log for the event

2017-07-11T13:50:05    1    Traceback (most recent call last):
              File "/usr/share/qgis/python/plugins/db_manager/dlg_sql_window.py", line 169, in loadAsLayerToggled
                self.fillColumnCombos()
              File "/usr/share/qgis/python/plugins/db_manager/dlg_sql_window.py", line 312, in fillColumnCombos
                c = connector._execute(None, sql)
              File "/usr/share/qgis/python/plugins/db_manager/db_plugins/connector.py", line 81, in _execute
                cursor.execute(str(sql))
            UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 23: ordinal not in range(128)

This seems to confirm that this might actually be the same bug than #16756
Sorry for duplicating I'm more a user than a developer and maybe I don't communicate "the good way".

#6 Updated by Borys Jurgiel over 7 years ago

  • Operating System deleted (Windows 10 (with OSGeo4W))
  • Assignee set to Sandro Santilli

Assigning to Sandro, as 14ab5eb0bda is the reason of this regression (line 81 of connector.py)

#7 Updated by Jürgen Fischer over 7 years ago

  • Subject changed from Unicode caracter in query in db_manager makes appear a never ending hourglass to Unicode character in query in db_manager makes appear a never ending hourglass

#8 Updated by Sandro Santilli over 7 years ago

I think I made those changes because I could not build otherwise, but trying again now I can build and confirm the revert fixes the issue. PR is here: https://github.com/qgis/QGIS/pull/4882 -- it would be good to have a testcase for this but I don't know when I'll have time for that.

#9 Updated by Sandro Santilli over 7 years ago

  • Status changed from Open to In Progress

#10 Updated by Sandro Santilli over 7 years ago

  • Pull Request or Patch supplied changed from No to Yes

#11 Updated by Sandro Santilli over 7 years ago

A second, simpler, pull request is here:
https://github.com/qgis/QGIS/pull/4888

This second one removes some assumptions about the input being ASCII (or encoded utf8).
Can the original reporter run some tests with that ?
A test is included and I run a manual test, but maybe you have more paths to test..

#12 Updated by Sandro Santilli over 7 years ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

#13 Updated by Sandro Santilli over 7 years ago

Fix is now merged

Also available in: Atom PDF