Bug report #5597

External URL access via PyQt crashes QGIS on Mac OS X

Added by Larry Shaffer over 7 years ago. Updated over 7 years ago.

Status:Closed
Priority:High
Assignee:-
Category:Python plugins
Affected QGIS version:master Regression?:No
Operating System:Mac OS X Easy fix?:No
Pull Request or Patch supplied:No Resolution:fixed
Crashes QGIS or corrupts data:Yes Copied to github as #:15171

Description

After typing the following commands at the Python Console, QGIS crashes (after entering last command):

from PyQt4.QtWebKit import QWebView
from PyQt4.QtCore import QUrl
webview = QWebView()
webview.load(QUrl('http://qgis.org/api'))

If the QUrl is local (i.e. file:// scheme) then there is no crash. Attached is the debug of the startup of QGIS (no debug info on crash, excepting notification of segfault). Between end of debug info and segfault is where Python Console commands were entered.

When testing this with a help browser, the base HTML of the page appears to render, then QGIS crashes. Possibly when loading additional resources. Pages with no images, only HTML (like above) cause the crash, as well.

This did not happen with 1.7.4-4 (Kyngchaos.com release), though he reports the same crash (on entering the above) with his 1.8 builds, too.

QGIS version: 1.8.0-Lisboa
QGIS code revision: a1255fc
Compiled against Qt: 4.8.1
Running against Qt: 4.8.1
Compiled against GDAL/OGR: 1.9.0
Running against GDAL/OGR: 1.9.0
GEOS Version: 3.3.2dev
PostgreSQL Client Version: 9.1.1
SpatiaLite Version: 2.4.0
QWT Version: 5.2.2
This copy of QGIS writes debugging output.

This bug may be related to #5338

This is fairly serious, as it may affect any plugin that has an HTML file that loads remotely into a QWebView, like help, etc.

Larry

qgis_webview-crash-debug-log.txt Magnifier - debug log output (85.6 KB) Larry Shaffer, 2012-05-14 06:22 PM

QGIS_2012-05-14-185752_Larry-Ss-iMac.crash - Mac textual crash report (59.6 KB) Larry Shaffer, 2012-05-14 06:22 PM

qgscustomization_system.log.txt Magnifier (30.3 KB) Larry Shaffer, 2012-05-22 03:07 PM

QGIS_2012-05-22-155544_Larry-Ss-iMac.crash.txt Magnifier (59.4 KB) Larry Shaffer, 2012-05-22 03:11 PM

qgis_prenotify-fix.png - OpenLayers plugin loaded. (272 KB) Larry Shaffer, 2012-05-23 12:59 AM

QGIS_2012-05-23-094658_Tull.crash (69.1 KB) John Tull, 2012-05-23 09:51 AM

QGIS_2012-05-23-101745_Alita.crash (71.7 KB) John Tull, 2012-05-23 10:22 AM

qgis_prenotify_fix-860f66d-.png (319 KB) Larry Shaffer, 2012-05-23 10:42 AM


Related issues

Related to QGIS Application - Bug report #5338: QGIS master crashes adding a WFS layer on OsX Snow Leopard Closed 2012-04-10

Associated revisions

Revision d77069d2
Added by Radim Blazek over 7 years ago

check thread in customization, it should fix #5597

History

#1 Updated by Giovanni Manghi over 7 years ago

  • Status changed from Open to Feedback

If this is a regression since 1.7.4 please tag this as blocker. I cannot test myself as I have not access to a Mac.

#2 Updated by Larry Shaffer over 7 years ago

Giovanni Manghi wrote:

If this is a regression since 1.7.4 please tag this as blocker.

How do I change the status to blocker? I can find no method to do so. Maybe I do not have those privileges.

#3 Updated by Giovanni Manghi over 7 years ago

  • Priority changed from High to Severe/Regression

Larry Shaffer wrote:

Giovanni Manghi wrote:

If this is a regression since 1.7.4 please tag this as blocker.

How do I change the status to blocker? I can find no method to do so. Maybe I do not have those privileges.

so it is confirmed is a regression?

#4 Updated by Larry Shaffer over 7 years ago

I have tested it on latest 1.8 under Mac OS X 10.7.3, too. Also crashes.

Here is where William Kyngesburye mentioned it also crashed his build:
http://osgeo-org.1560.n6.nabble.com/Config-fails-on-find-of-spatialindex-library-tp4939392p4941036.html

I would be more comfortable having some other Mac users confirming this, with latest 1.8 RC, before setting the issue to blocker.

I'm no C++ coder, but, if I were to guess, I think it might be an incompatibility of Qt 4.8.1 on Mac, possibly with some networking routines, like src/core/qgsnetworkaccessmanager.cpp (though I don't use a proxy). 1.7.4 on Mac used Qt 4.8.0.

Note: similar PyQt QWebView/QUrl functions (like a HTML help browser I am working on for QGIS) work fine with Qt 4.8.1 when run outside of the QGIS.app 1.8 RC bundle, which is created from the same Qt install.

#5 Updated by Marco Hugentobler over 7 years ago

  • Priority changed from Severe/Regression to High

That looks like a packaging or Qt compilation problem on OS X. Therefore I think it should not be classified as a blocker.

#6 Updated by Larry Shaffer over 7 years ago

Hi Marco,

I agree that it should just be classified as high until more testing can show if QGIS src to be the cause. I'll test the compilation under 10.6.8/Qt 4.7.4 to see if it reappears.

It does not occur with 1.8 RC under Ubuntu 11.10/Qt 4.7.4 (compiled and tested today).

#7 Updated by Larry Shaffer over 7 years ago

Ok, I tested under 10.6.8/Qt 4.7.4 with no issues (external URLs loaded fine in QWebViews).

To remove PyQt from the equation (when compiling against Qt 4.8.1), I added/linked a QWebView into one of the C++ plugin's forms and connected one it buttons to load http://www.qgis.org into the web view when clicked. Results: same as with PyQt, crashes QGIS immediately. So, it seems to have nothing to do with the Python bindings or bundling.

Will now look into any recent changes in Qt bundling/linking that might be an issue for 4.8.1, but not 4.7.4.

#8 Updated by Martin Dobias over 7 years ago

Does that code produce a crash also outside of QGIS environment?

#9 Updated by Larry Shaffer over 7 years ago

Hi Martin,

Does that code produce a crash also outside of QGIS environment?

Not with PyQt. My help browser works fine while testing in Eric4 IDE. I'm too much of a C++ neophyte to test in a separate program outside QGIS. It also works fine when tested with Qt 4.8.0 (QGIS Mac ver. 1.7.4-4). It seems to be some kind of incompatibility or missing component with bundling specific to Qt 4.8.1.

There have been some changes in QNetwork with 4.8.1, mostly mobile stuff, so I tried bundling/relinking the .dyld 'bearer' Qt plugins, but that didn't make a difference. That's the only missing network stuff not bundled/relinked that I could see.

http://qt-project.org/doc/qt-4.8/bearer-management.html

Also, Qt-bundled WebKit version was increased with 4.8.1:

http://qt.nokia.com/products/changes/changes-4.8.1/

This appears to happen whenever an external URL resource is requested, regardless of C++ or Python binding. Here with OpenLayers plugin:
#5027

Not sure how many plugins/tools use external URL resources, but I'd guess it would be enough to hamper many Mac user's experience with QGIS, especially if their projects already include plugin layers like OpenLayers.

I'm starting to be at a loss for what to test next. Maybe install Qt debug libraries, but I have no knowledge of their use.

#10 Updated by Larry Shaffer over 7 years ago

Done some more research, and think I have maybe found where to look for the problem/fix. A core C++ dev will need to verify my guesstimation on whether QGIS source code is causing the issue. I think it may be, which means it may be a blocking issue for any QGIS compiled against Qt >= 4.8.1.

I now think this issue is the same problem causing the crashes of WMS and WFS connections as well (#5338 for WFS issue ). I tested with the default WMS servers added, also a crash.

QgsNetworkAccessManager incompatible with QNetworkAccessManager (4.8.1)?

So here's my guess... According to this info on threads, there have been changes to QNetworkAccessManager.

http://qt-project.org/wiki/Threads_Events_QObjects#605fc5d1bdd8d9f08192df140937a7c7

As of Qt 4.8, QNetworkAccessManager now handles HTTP requests in a separate thread by default...

The QgsNetworkAccessManager class looks like it re-implements QNetworkAccessManager to intercept QGIS network connections, allowing the insertion of proxy settings from app-level options (Network section of QGIS preferences, yes?). Maybe QgsNetworkAccessManager needs updated to jive with changes in Qt 4.8.1.

Does QgsNetworkAccessManager intercept any external URL request from within QGIS, including those from a QWebView in a plugin, or only those that have QgsNetworkAccessManager instantiated? If it does intercept all outgoing connections, it would explain a lot.

QgsNetworkAccessManager is utilized in the WMS and WFS providers, both of which are crashing the app on connection, but not always immediately.

If anyone on a platform other than Mac, that has QGIS 1.8 compiled against Qt 4.8.1, can test this issue, it would help. No sense looking for a possible incompatibility in QgsNetworkAccessManager unless necessary.

#11 Updated by John Tull over 7 years ago

Let's hope you have found the source of the phantom crashes in trunk on OSX. Perhaps this is also related to the crashing open layers plugin, #5027.

#12 Updated by Larry Shaffer over 7 years ago

Well, I just installed Ubuntu Pangolin 12.04 and loaded up QGIS 1.8, which is compiled against Qt 4.8.1 and it does not have the issue. In fact it works very well.

It appears to be Mac-only, when bundled in QGIS, and tied to QNetwork somehow. I don't see how QgsNetworkAccessManager could intercept plugins, just merely offer a different proxy setup. So, it apparently is falling prey to the same underlying connection issue. The OpenLayers plugin has its own proxy setup, and my test code (in initial issue report) has no proxy setup at all.

I browsed the commits on Qt 4.8.2 to see if there is a possible fix in the pipeline, regarding QNetworkAccessManager on Mac, and only came up with this:

http://qt.gitorious.org/qt/qt/commit/ba166b618e2360d2fa7e927adf239d737a443f5a

Since I have seen affected QWebViews actually load most of a page, and start rendering, just before they crash, it looks like the 'premature' quit() of the HTTP connection thread may be causing the crash (though unrelated to SSL, as in the above bug). Kind of a chore to compile Qt for Mac only to incorporate the 3 lines of 5-second waits, but it may do the trick.

I will load up a copy of Lion under Parallels and give it a try.

#13 Updated by John Tull over 7 years ago

I can tell you that the crash still occurs with 4.8.2 installed on my machine, so it does not appear that the commit has fixed this. Unless you were suggesting something different from the linked commit you pointed out.

#14 Updated by Jürgen Fischer over 7 years ago

John Tull wrote:

I can tell you that the crash still occurs with 4.8.2 installed on my machine, so it does not appear that the commit has fixed this. Unless you were suggesting something different from the linked commit you pointed out.

Might https://gist.github.com/2771088 help?

#15 Updated by Larry Shaffer over 7 years ago

Jürgen Fischer wrote:

Might https://gist.github.com/2771088 help?

Why, yes, it does! Wow, I was all over the place trying to find/fix that.

Can you explain your edit, please?

Here is the OpenLayers plugin working:
http://dl.dropbox.com/u/4058089/qgis/qt4.8.1-fix.png

And my plugin's help browser working as well:

http://dl.dropbox.com/u/4058089/qgis/qt4.8.1-fix-helpbrowser.png

Thank you very much, Jürgen!

Thanks for your help as well, John.

#16 Updated by Jürgen Fischer over 7 years ago

Larry Shaffer wrote:

Jürgen Fischer wrote:

Might https://gist.github.com/2771088 help?

Why, yes, it does! Wow, I was all over the place trying to find/fix that.

Can you explain your edit, please?

The customization code tries to filter out events and apparently in some situations that causes a crash. We need to dig deeper to see what.

So we're not done - just starting to debug. Please revert the last patch and try this one: https://gist.github.com/2771300
That might catch an exception instead of crashing and give some more info.

#17 Updated by Larry Shaffer over 7 years ago

Jürgen Fischer wrote:

So we're not done - just starting to debug.

I was afraid you were going to say that.

Please revert the last patch and try this one: https://gist.github.com/2771300
That might catch an exception instead of crashing and give some more info.

Sorry for the delay. I had to stash my changes and do a reset on my local git repository, then I could ensure I had the latest source from github.

No luck on catching an exception, just crashes again, with no more info in the log than original bug report.

Here is what that section of qgsapplication.cpp looks like after applying your diff:

bool QgsApplication::notify( QObject * receiver, QEvent * event )
{
  bool done = false;

  try
  {
    emit preNotify( receiver, event, &done );

    if ( done )
      return true;

    // Send event to receiver and catch unhandled exceptions
    done = true;
    done = QApplication::notify( receiver, event );
  }
  catch ( QgsException & e )
  {
    QMessageBox::critical( activeWindow(), tr( "Exception" ), e.what() );
  }
  catch ( std::exception & e )
  {
    QMessageBox::critical( activeWindow(), tr( "Exception" ), e.what() );
  }
  catch ( ... )
  {
    QMessageBox::critical( activeWindow(), tr( "Exception" ), tr( "unknown exception.\
receiver: %1\
class: %2\
event: %3" )
                           .arg( receiver->objectName() )
                           .arg( receiver->metaObject()->className() )
                           .arg( event->type() )
                         );
  }

  return done;
}

I built QGIS with RelWithDebInfo, but still don't get anything useful output to log for the crash.

#18 Updated by Jürgen Fischer over 7 years ago

Larry Shaffer wrote:

I built QGIS with RelWithDebInfo, but still don't get anything useful output to log for the crash.

unfortunate - please try the updated gist. It'll probably produce loads of output. What's the last line?

#19 Updated by Larry Shaffer over 7 years ago

Jürgen Fischer wrote:

unfortunate - please try the updated gist. It'll probably produce loads of output. What's the last line?

Attached file contains all logged events from around the last Python console entry (using code in original report for testing).

#20 Updated by Larry Shaffer over 7 years ago

Mac crash reporter report to go with last update.

#21 Updated by Jürgen Fischer over 7 years ago

Larry Shaffer wrote:

Mac crash reporter report to go with last update.

please try the updated gist.

#22 Updated by Larry Shaffer over 7 years ago

Jürgen Fischer wrote:

please try the updated gist.

Yes, that works.

Do you think this stems from the HTTP connections running in separate thread? If so, I wonder why it didn't show up with Qt 4.8.0.

Hopefully it doesn't cause problems with anything using other threads. I'm aware of some Python plugins that use processing threads. I'm guessing those threads just can't be run through QgsCustomization. Probably not an issue, if they are non-gui?

I compiled this on my personal workstation (Mac Lion 10.7.3). Will try again in the morning at work with Snow Leopard 10.6.8.

WMS and WFS connections and layer loads appear to function as well. I did have a crash using WMS (DM Solutions server), but I think it was unrelated to this issue.

#23 Updated by Jürgen Fischer over 7 years ago

Larry Shaffer wrote:

Do you think this stems from the HTTP connections running in separate thread? If so, I wonder why it didn't

show up with Qt 4.8.0.

Maybe threads are now used more widely in Qt - dunno. Anyway, the customization stuff is all about the UI (Radim correct me, if I'm wrong). Therefore restricting the signal to stuff in the UI thread shouldn't do harm.

#24 Updated by Larry Shaffer over 7 years ago

Posting results (and a link to your gist) to:

#5338
#5027

It may fix those issues as well. Also, be good to have more testing before committing.

#25 Updated by Jürgen Fischer over 7 years ago

Larry Shaffer wrote:

It may fix those issues as well. Also, be good to have more testing before committing.

Checking if the customization stuff (never used it so far) still works would be nice too.

#26 Updated by Larry Shaffer over 7 years ago

Jürgen Fischer wrote:

Checking if the customization stuff (never used it so far) still works would be nice too.

After setting a custom configuration, clicking Apply does not work (live applies may not for Mac). Also, I get crashes on relaunch about 50% of the time.

However, I opened up a saved nightly archive QGIS.app from April and customizations appear to work/crash the same. After one crash, the app launches and the customizations are in effect.

The crashes are probably Mac Lion trying to (annoyingly) auto-reopen the previous session's windows, which are now different. It appears to be functional, barring the launch crashes, on 10.7. May work fine on 10.6.

So, your fix doesn't appear to cause any change, but that's the first time I have set any customizations.

#27 Updated by Radim Blazek over 7 years ago

Jürgen Fischer wrote:

Anyway, the customization stuff is all about the UI (Radim correct me, if I'm wrong). Therefore restricting the signal to stuff in the UI thread shouldn't do harm.

Right. Should I add the

thread() == receiver->thread()
check to QgsCustomization::preNotify to keep it closer to where the problem appears?

We should probably add an enable/disable checkbox in customization dialog to disable it (by default) completely.

Sorry for problems.

#28 Updated by Jürgen Fischer over 7 years ago

Radim Blazek wrote:

Right. Should I add the [...] check to QgsCustomization::preNotify to keep it closer to where the problem appears?

I'm not sure that the slot is executed in the same thread. Signals can also be used for communication between threads. QCoreApplication::instance()->thread() == receiver->thread() is probably better there.

We should probably add an enable/disable checkbox in customization dialog to disable it (by default) completely.

Sounds good.

#29 Updated by Radim Blazek over 7 years ago

  • Status changed from Feedback to Closed

#30 Updated by Radim Blazek over 7 years ago

I have added the thread check in d77069d2 , please test if it helps. For other 'normal' customization crashes please open another issue.

#31 Updated by Radim Blazek over 7 years ago

  • Status changed from Closed to Reopened

#32 Updated by John Tull over 7 years ago

Attached is my crash log with Radim's update applied. This crash was produced running the python code that Larry provided earlier in the thread. The OpenLayers plugin also produces a crash.

#33 Updated by Larry Shaffer over 7 years ago

Radim Blazek wrote:

I have added the thread check in d77069d2 , please test if it helps. ...

Hi, Radim. Unfortunately, moving the check to the slot does not fix. External HTTP access still crashes on Mac 10.7.3. Initial issue report, OpenLayers plugin, adding WMS or WFS layer all still cause crashes.

Maybe move check back to signal emit code, but possibly leave a comment in the slot about the check?

We should probably add an enable/disable checkbox in customization dialog to disable it (by default) completely.

I'm not understanding this. Do you see use cases where a user may want to turn off the check? Or, is this a "don't know now, but better leave it optional" type of setting?

... For other 'normal' customization crashes please open another issue.

I will test on Mac 10.6.8, to verify, before creating an issue.

#34 Updated by Larry Shaffer over 7 years ago

Radim, instead of a checkbox, do you think that after applying this stopgap fix for 1.8 release maybe the QgsCustomization signal/slot setup could be adapted to switch to the proper receiver thread? This might allow the receiver to accept customizations even if in a different thread. Could be a lot of work for limited need.

I don't have experience with coding threads, though I think having the customization code working for all use cases (as originally designed) would be a good thing.

#35 Updated by Jürgen Fischer over 7 years ago

Larry Shaffer wrote:

Maybe move check back to signal emit code, but possibly leave a comment in the slot about the check?

Or like in b600ff7

I'm not understanding this. Do you see use cases where a user may want to turn off the check? Or, is this a "don't know now, but better leave it optional" type of setting?

That was just about disabling the customization in general by default - having no active customization is the default case, so the signal processing is useless most of the time anyway.

#36 Updated by Larry Shaffer over 7 years ago

Jürgen Fischer wrote:

Or like in b600ff7

That fix does not work. Same crashes in PyQt code, OpenLayers plugin, adding WMS or WFS layer. WMS and WFS did connect, but maybe the server I was testing was cached somehow. Choosing a different server caused an immediate crash again on connect.

I'm not understanding this. Do you see use cases where a user may want to turn off the check? Or, is this a "don't know now, but better leave it optional" type of setting?

That was just about disabling the customization in general by default - having no active customization is the default case, so the signal processing is useless most of the time anyway.

Ok, now I understand.

#37 Updated by John Tull over 7 years ago

Here is the crash report from the last update to the code.

#38 Updated by John Tull over 7 years ago

It looks like this latest patch has fixed it, at least from my limited testing. Larry can test further and close it once he's put it through the paces also.

#39 Updated by Larry Shaffer over 7 years ago

Jürgen, your take III (860f66d) works as before.

Attached screen snap shows OpenLayers, WMS and WFS layers, and QWebView all working.

#40 Updated by Jürgen Fischer over 7 years ago

  • Status changed from Reopened to Resolved
  • Resolution set to fixed

#41 Updated by Jürgen Fischer over 7 years ago

  • Status changed from Resolved to Closed

#42 Updated by Radim Blazek over 7 years ago

Larry Shaffer wrote:

Radim, instead of a checkbox, do you think that after applying this stopgap fix for 1.8 release maybe the QgsCustomization signal/slot setup could be adapted to switch to the proper receiver thread? This might allow the receiver to accept customizations even if in a different thread. Could be a lot of work for limited need.

If customization crashes even with the last fix, then there must be some other problem, not related to threads. It is crashing in Qt core functions, I have no idea where is the problem. It is difficult to fix it blindly trying patches. It must be fixed by somebody on Mac.

#43 Updated by Larry Shaffer over 7 years ago

Confirmed 860f66d works for Mac OS X 10.6.8 as well.

Also available in: Atom PDF