Bug report #20337

Crash loading layer from DB2 spatial table

Added by David Adler about 1 year ago. Updated about 1 year ago.

Status:Open
Priority:High
Assignee:David Adler
Category:Data Provider
Affected QGIS version:3.3(master) Regression?:Yes
Operating System: Easy fix?:No
Pull Request or Patch supplied:No Resolution:
Crashes QGIS or corrupts data:Yes Copied to github as #:28158

Description

User Feedback

Attempting to load a layer from a DB2 spatial table when double-clicking on a DB2 table name.
This information is from running without debugging.
When running the debugger, it appears to be crashing in the ODBC code called by the QSqlQuery::prepare method.

It worked fine in QGIS 2.18 but fails most but not all the time in QGIS 3.3 and when built from QGIS Master.

I'm wondering if the problem could have something to do with threading in the DB2 provider code.
When the code crashes, it appears that the qgsdb2featureiterator and qgsdb2provider code is running in different threads for the same request to access a layer.
This is a snip of the trace when it fails:
E:\qgisdev\src\core\qgsmaplayer.cpp(178) : (QgsMapLayer::name) [2ms] returning name 'CUSTOMERS'
E:\qgisdev\src\core\qgscoordinatetransform_p.cpp(210) : (QgsCoordinateTransformPrivate::initialize) [3ms] Source/Dest CRS not equal, shortcircuit is not set.
E:\qgisdev\src\core\qgscoordinatetransform_p.cpp(210) : (QgsCoordinateTransformPrivate::initialize) [2ms] Source/Dest CRS not equal, shortcircuit is not set.
E:\qgisdev\src\core\qgscoordinatetransform_p.cpp(210) : (QgsCoordinateTransformPrivate::initialize) [3ms] Source/Dest CRS not equal, shortcircuit is not set.
E:\qgisdev\src\core\qgsmaprendererparalleljob.cpp(64) : (QgsMapRendererParallelJob::start) [2ms] QThreadPool max thread count is 8
E:\qgisdev\src\core\qgsmaprendererparalleljob.cpp(253) : (QgsMapRendererParallelJob::renderLayerStatic) [4ms] [thread:0x2eb55698840] job 2eb57ceecc0 start (layer CUSTOMERS_e0a5be6a_6d31_4690_a52d_d10756f7ce67)
E:\qgisdev\src\providers\db2\qgsdb2featureiterator.cpp(235) : (QgsDb2FeatureIterator::BuildStatement) [4ms] [thread:0x2eb55698840] compileExpressions: true
E:\qgisdev\src\providers\db2\qgsdb2featureiterator.cpp(288) : (QgsDb2FeatureIterator::BuildStatement) [3ms] [thread:0x2eb55698840] SELECT OBJECTID,DB2GSE.ST_ASBINARY(GEOM) AS GEOM FROM OSUSER.CUSTOMERS WHERE DB2GSE.ENVELOPESINTERSECT = 1
E:\qgisdev\src\providers\db2\qgsdb2provider.cpp(164) : (QgsDb2Provider::getDatabase) [3ms] [thread:0x2eb55698840] expanded connInfo: dbname='OSTEST' service='OSTEST64'
E:\qgisdev\src\providers\db2\qgsdb2provider.cpp(174) : (QgsDb2Provider::getDatabase) [4ms] [thread:0x2eb55698840] driver: ''; host: ''; databaseName: 'OSTEST'
E:\qgisdev\src\providers\db2\qgsdb2provider.cpp(188) : (QgsDb2Provider::getDatabase) [3ms] [thread:0x2eb55698840] connectionName: OSTEST64
E:\qgisdev\src\providers\db2\qgsdb2provider.cpp(197) : (QgsDb2Provider::getDatabase) [3ms] [thread:0x2eb55698840] found existing connection, use the existing one
E:\qgisdev\src\core\qgscredentials.cpp(68) : (QgsCredentials::put) [1095ms] [thread:0x2eb5590cf90] inserting realm:OSTEST username:osuser password:osuserpw
E:\qgisdev\src\providers\db2\qgsdb2featureiterator.cpp(418) : (QgsDb2FeatureIterator::rewind) [0ms] [thread:0x2eb5590cf90] Execute mStatement: SELECT OBJECTID,DB2GSE.ST_ASBINARY(GEOM) AS GEOM FROM OSUSER.CUSTOMERS WHERE DB2GSE.ENVELOPESINTERSECT = 1
E:\qgisdev\src\core\qgscredentials.cpp(48) : (QgsCredentials::get) [8ms] [thread:0x2eb55698840] retrieved realm:OSTEST username:osuser password:osuserpw
E:\qgisdev\src\providers\db2\qgsdb2provider.cpp(242) : (QgsDb2Provider::getDatabase) [7ms] [thread:0x2eb55698840] ODBC connection string: OSTEST64
E:\qgisdev\src\app\qgscrashhandler.cpp(34) : (QgsCrashHandler::handle) [336ms] [thread:0x2eb5590cf90] CRASH!!!
E:\qgisdev\src\app\qgscrashhandler.cpp(57) : (QgsCrashHandler::handle) [9ms] [thread:0x2eb5590cf90] C:/Users/davea2/AppData/Local/Temp/qgis-crash-info-12560
E:\qgisdev\src\app\qgscrashhandler.cpp(102) : (QgsCrashHandler::handle) [14ms] [thread:0x2eb5590cf90] E:/qgisdev/ms-windows/osgeo4w/build-qgis-test-x86_64/output/bin/RelWithDebInfo/qgiscrashhandler.exe

This is a snip of the trace when it succeeds:
E:\qgisdev\src\core\qgsmaplayer.cpp(178) : (QgsMapLayer::name) [3ms] returning name 'CUSTOMERS'
E:\qgisdev\src\core\qgscoordinatetransform_p.cpp(210) : (QgsCoordinateTransformPrivate::initialize) [2ms] Source/Dest CRS not equal, shortcircuit is not set.
E:\qgisdev\src\core\qgscoordinatetransform_p.cpp(210) : (QgsCoordinateTransformPrivate::initialize) [2ms] Source/Dest CRS not equal, shortcircuit is not set.
E:\qgisdev\src\core\qgscoordinatetransform_p.cpp(210) : (QgsCoordinateTransformPrivate::initialize) [3ms] Source/Dest CRS not equal, shortcircuit is not set.
E:\qgisdev\src\core\qgsmaprendererparalleljob.cpp(64) : (QgsMapRendererParallelJob::start) [2ms] QThreadPool max thread count is 8
E:\qgisdev\src\core\qgsmaprendererparalleljob.cpp(253) : (QgsMapRendererParallelJob::renderLayerE:\qgisdev\src\core\qgsmaprendererparalleljob.cpp(253) : (QgsMapRendererParallelJob::renderLayerStatic) [0ms] [thread:0x24fda6eb760] job 24fdd489b70 start (layer SpatialTable_1daffcad_555d_4c84_8b91_ada8a3d72729)
E:\qgisdev\src\core\qgsmaplayer.cpp(178) : (QgsMapLayer::name) [3ms] returning name 'SpatialTable'
E:\qgisdev\src\providers\mssql\qgsmssqlfeatureiterator.cpp(293) : (QgsMssqlFeatureIterator::BuildStatement) [1ms] [thread:0x24fda6eb760] SELECT [id],[GeomCol1] FROM [dbo].[SpatialTable] WHERE [GeomCol1].STIsValid() = 1 AND [GeomCol1].Filter([geometry]::STGeomFromText('POLYGON)',4326)) = 1
E:\qgisdev\src\providers\db2\qgsdb2featureiterator.cpp(235) : (QgsDb2FeatureIterator::BuildStatement) [0ms] [thread:0x24fdb506a80] compileExpressions: true
E:\qgisdev\src\providers\db2\qgsdb2featureiterator.cpp(288) : (QgsDb2FeatureIterator::BuildStatement) [2ms] [thread:0x24fdb506a80] SELECT OBJECTID,DB2GSE.ST_ASBINARY(GEOM) AS GEOM FROM OSUSER.CUSTOMERS WHERE DB2GSE.ENVELOPESINTERSECT = 1
E:\qgisdev\src\providers\db2\qgsdb2provider.cpp(164) : (QgsDb2Provider::getDatabase) [3ms] [thread:0x24fdb506a80] expanded connInfo: dbname='OSTEST' service='OSTEST64'
E:\qgisdev\src\providers\db2\qgsdb2provider.cpp(174) : (QgsDb2Provider::getDatabase) [3ms] [thread:0x24fdb506a80] driver: ''; host: ''; databaseName: 'OSTEST'
E:\qgisdev\src\core\qgsmaprendererparalleljob.cpp(273) : (QgsMapRendererParallelJob::renderLayerStatic) [1ms] [thread:0x24fda6eb760] job 24fdd489b70 end [21 ms] (layer SpatialTable_1daffcad_555d_4c84_8b91_ada8a3d72729)
E:\qgisdev\src\providers\db2\qgsdb2provider.cpp(188) : (QgsDb2Provider::getDatabase) [2ms] [thread:0x24fdb506a80] connectionName: OSTEST64
E:\qgisdev\src\providers\db2\qgsdb2provider.cpp(197) : (QgsDb2Provider::getDatabase) [3ms] [thread:0x24fdb506a80] found existing connection, use the existing one
E:\qgisdev\src\core\qgscredentials.cpp(48) : (QgsCredentials::get) [1322ms] [thread:0x24fdb506a80] retrieved realm:OSTEST username:osuser password:osuserpw
E:\qgisdev\src\providers\db2\qgsdb2provider.cpp(242) : (QgsDb2Provider::getDatabase) [4ms] [thread:0x24fdb506a80] ODBC connection string: OSTEST64
E:\qgisdev\src\core\qgscredentials.cpp(68) : (QgsCredentials::put) [1874ms] [thread:0x24fdb506a80] inserting realm:OSTEST username:osuser password:osuserpw
E:\qgisdev\src\providers\db2\qgsdb2featureiterator.cpp(419) : (QgsDb2FeatureIterator::rewind) [13ms] [thread:0x24fdb506a80] Execute mStatement: SELECT OBJECTID,DB2GSE.ST_ASBINARY(GEOM) AS GEOM FROM OSUSER.CUSTOMERS WHERE DB2GSE.ENVELOPESINTERSECT = 1
E:\qgisdev\src\providers\db2\qgsdb2featureiterator.cpp(426) : (QgsDb2FeatureIterator::rewind) [342ms] [thread:0x24fdb506a80] leaving rewind
E:\qgisdev\src\providers\db2\qgsdb2featureiterator.cpp(427) : (QgsDb2FeatureIterator::rewind) [8ms] [thread:0x24fdb506a80]
E:\qgisdev\src\providers\db2\qgsdb2featureiterator.cpp(310) : (QgsDb2FeatureIterator::fetchFeature) [8ms] [thread:0x24fdb506a80] Entering fetchFeature
E:\qgi

Report Details

Crash ID: 597ecc7423c2116f73ac10c08f473806a26f2353

Stack Trace

RtlEnterCriticalSection :
ODBCSetTryWaitValue :
SQLExecute :
QgsDb2FeatureIterator::rewind qgsdb2featureiterator.cpp:427
QgsDb2FeatureIterator::QgsDb2FeatureIterator qgsdb2featureiterator.cpp:68
QgsDb2FeatureSource::getFeatures qgsdb2featureiterator.cpp:485
QgsVectorLayerFeatureIterator::QgsVectorLayerFeatureIterator qgsvectorlayerfeatureiterator.cpp:264
QgsVectorLayerFeatureSource::getFeatures qgsvectorlayerfeatureiterator.cpp:98
QgsVectorLayerRenderer::render qgsvectorlayerrenderer.cpp:237
QgsMapRendererParallelJob::renderLayerStatic qgsmaprendererparalleljob.cpp:256
QtConcurrent::FunctionWrapper1<void,LayerRenderJob & __ptr64>::operator() qtconcurrentfunctionwrappers.h:84
QtConcurrent::MapKernel<QList<LayerRenderJob>::iterator,QtConcurrent::FunctionWrapper1<void,LayerRenderJob & __ptr64> >::runIteration qtconcurrentmapkernel.h:70
QtConcurrent::MapKernel<QList<LayerRenderJob>::iterator,QtConcurrent::FunctionWrapper1<void,LayerRenderJob & __ptr64> >::runIterations qtconcurrentmapkernel.h:79
QtConcurrent::IterateKernel<QList<LayerRenderJob>::iterator,void>::forThreadFunction qtconcurrentiteratekernel.h:256
QtConcurrent::IterateKernel<QList<LayerRenderJob>::iterator,void>::threadFunction qtconcurrentiteratekernel.h:218
QtConcurrent::ThreadEngineBase::run :
QThreadPoolPrivate::reset :
QThread::start :
BaseThreadInitThunk :
RtlUserThreadStart :

QGIS Info
QGIS Version: 3.3.0-Master
QGIS code revision: dfb28fed44
Compiled against Qt: 5.9.2
Running against Qt: 5.9.2
Compiled against GDAL: 2.2.4
Running against GDAL: 2.2.4

System Info
CPU Type: x86_64
Kernel Type: winnt
Kernel Version: 10.0.17134

History

#1 Updated by David Adler about 1 year ago

  • Assignee set to David Adler

I think this has to do with threading between getting database connection and executing query - researching.

#2 Updated by Nyall Dawson about 1 year ago

There's been many bug fixes added to the mssql provider since it was forked for the DB2 provider. I'd suggest starting by looking at the git history of the mssql provider and porting all the relevant fixes.

#3 Updated by Giovanni Manghi about 1 year ago

  • Status changed from Open to Feedback

Please try on QGIS 3.4.1, if the issue is still valid change the affected version, thanks.

#4 Updated by David Adler about 1 year ago

  • Status changed from Feedback to Open

Nyall - thank you for the feedback.
Migrating over the mssql changes to DB2 seems to have fixed the problem which had to do with reusing the connection with different threads.

#5 Updated by Nyall Dawson about 1 year ago

Ah - that thorny issue! It's a direct result of change in behavior with Qt 5.11, but fixing it should also make the provider more stable with earlier Qt versions.

Do you have a PR inbound with your fixes?

I'd also encourage you to try running the db2 provider tests again if you've got an up to date build. There's been a large number of additional checks added to that test since the db2 provider was added, but unfortunately I'm not aware of any core developers with access to a db2 instance in order to run these newer checks on a real db2 database. The provider test is likely to expose some more issues which may need addressing to further improve the provider's stability.

Also available in: Atom PDF