Skip to content

Commit 96eb75a

Browse files
troopa81troopa81
committedAug 23, 2021
[ExternalStorage] add WebDAV external storage implementation
·
ltr-3_40final-3_22_0
1 parent 0ce7f90 commit 96eb75a

23 files changed

+619
-22
lines changed
 

‎.docker/docker-compose-testing.yml

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,12 +11,20 @@ services:
1111
httpbin:
1212
image: kennethreitz/httpbin:latest
1313

14+
webdav:
15+
image: nginx
16+
volumes:
17+
- ${GH_WORKSPACE}/.docker/webdav/nginx.conf:/etc/nginx/conf.d/default.conf
18+
- ${GH_WORKSPACE}/.docker/webdav/passwords.list:/etc/nginx/.passwords.list
19+
- /tmp/webdav_tests:/tmp/webdav_tests_root/webdav_tests
20+
1421
qgis-deps:
1522
tty: true
1623
image: qgis3-build-deps-binary-image
1724
volumes:
1825
- ${GH_WORKSPACE}:/root/QGIS
19-
# links:
26+
links:
27+
- webdav
2028
# - mssql
2129
links:
2230
- httpbin

‎.docker/docker-variables.env

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,3 +20,6 @@ QGIS_CONTINUOUS_INTEGRATION_RUN=true
2020
PUSH_TO_CDASH=false
2121

2222
XDG_RUNTIME_DIR=/tmp
23+
24+
QGIS_WEBDAV_HOST=webdav
25+
QGIS_WEBDAV_PORT=80

‎.docker/webdav/nginx.conf

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
server {
2+
listen 80;
3+
listen [::]:80;
4+
server_name localhost;
5+
6+
location /webdav_tests {
7+
8+
auth_basic realm_name;
9+
auth_basic_user_file /etc/nginx/.passwords.list;
10+
11+
dav_methods PUT DELETE MKCOL COPY MOVE;
12+
#dav_ext_methods PROPFIND OPTIONS;
13+
dav_access user:rw group:rw all:r;
14+
15+
autoindex on;
16+
17+
client_max_body_size 0;
18+
create_full_put_path on;
19+
root /tmp/webdav_tests_root;
20+
}
21+
}

‎.docker/webdav/passwords.list

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
qgis:$apr1$cxID/nB1$3tG4J0FkYvEHyWAB.yqjo.

‎.github/workflows/run-tests.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -386,4 +386,5 @@ jobs:
386386
[[ ${{ matrix.test-batch }} == "ORACLE" ]] && sudo rm -rf /usr/share/dotnet/sdk
387387
echo "TEST_BATCH=$TEST_BATCH"
388388
echo "DOCKERFILE=$DOCKERFILE"
389+
mkdir -p /tmp/webdav_tests && chmod 777 /tmp/webdav_tests
389390
docker-compose -f .docker/$DOCKERFILE run qgis-deps /root/QGIS/.docker/docker-qgis-test.sh $TEST_BATCH

‎python/core/auto_generated/network/qgsblockingnetworkrequest.sip.in

Lines changed: 27 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,7 @@ can be retrieved by calling :py:func:`~QgsBlockingNetworkRequest.errorMessage`.
6969
.. seealso:: :py:func:`post`
7070
%End
7171

72-
ErrorCode post( QNetworkRequest &request, const QByteArray &data, bool forceRefresh = false, QgsFeedback *feedback = 0 );
72+
ErrorCode post( QNetworkRequest &request, QIODevice *data, bool forceRefresh = false, QgsFeedback *feedback = 0 );
7373
%Docstring
7474
Performs a "post" operation on the specified ``request``, using the given ``data``.
7575

@@ -89,6 +89,15 @@ If an error was encountered then a specific ErrorCode will be returned, and a de
8989
can be retrieved by calling :py:func:`~QgsBlockingNetworkRequest.errorMessage`.
9090

9191
.. seealso:: :py:func:`get`
92+
93+
.. versionadded:: 3.22
94+
%End
95+
96+
ErrorCode post( QNetworkRequest &request, const QByteArray &data, bool forceRefresh = false, QgsFeedback *feedback = 0 );
97+
%Docstring
98+
This is an overloaded function.
99+
100+
Performs a "post" operation on the specified ``request``, using the given ``data``.
92101
%End
93102

94103
ErrorCode head( QNetworkRequest &request, bool forceRefresh = false, QgsFeedback *feedback = 0 );
@@ -113,7 +122,7 @@ can be retrieved by calling :py:func:`~QgsBlockingNetworkRequest.errorMessage`.
113122
.. versionadded:: 3.18
114123
%End
115124

116-
ErrorCode put( QNetworkRequest &request, const QByteArray &data, QgsFeedback *feedback = 0 );
125+
ErrorCode put( QNetworkRequest &request, QIODevice *data, QgsFeedback *feedback = 0 );
117126
%Docstring
118127
Performs a "put" operation on the specified ``request``, using the given ``data``.
119128

@@ -129,6 +138,15 @@ by calling :py:func:`~QgsBlockingNetworkRequest.reply`.
129138
If an error was encountered then a specific ErrorCode will be returned, and a detailed error message
130139
can be retrieved by calling :py:func:`~QgsBlockingNetworkRequest.errorMessage`.
131140

141+
.. versionadded:: 3.22
142+
%End
143+
144+
ErrorCode put( QNetworkRequest &request, const QByteArray &data, QgsFeedback *feedback = 0 );
145+
%Docstring
146+
This is an overloaded function.
147+
148+
Performs a "put" operation on the specified ``request``, using the given ``data``.
149+
132150
.. versionadded:: 3.18
133151
%End
134152

@@ -192,6 +210,13 @@ Emitted when when data arrives during a request.
192210
void downloadFinished();
193211
%Docstring
194212
Emitted once a request has finished downloading.
213+
%End
214+
215+
void uploadProgress( qint64, qint64 );
216+
%Docstring
217+
Emitted when when data are sent during a request.
218+
219+
.. versionadded:: 3.22
195220
%End
196221

197222
};

‎python/core/auto_generated/network/qgsnetworkcontentfetcher.sip.in

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,14 @@ Emitted when content has loaded
9595
Emitted when data is received.
9696

9797
.. versionadded:: 3.2
98+
%End
99+
100+
void errorOccurred( QNetworkReply::NetworkError code, const QString &errorMsg );
101+
%Docstring
102+
Emitted when an error with ``code`` error occured while processing the request
103+
``errorMsg`` is a textual description of the error
104+
105+
.. versionadded:: 3.22
98106
%End
99107

100108
};

‎python/core/auto_generated/network/qgsnetworkcontentfetcherregistry.sip.in

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,8 @@ FetchedContent holds useful information about a network content being fetched
3232
Failed
3333
};
3434

35-
explicit QgsFetchedContent( const QString &url, QTemporaryFile *file = 0, ContentStatus status = NotStarted );
35+
explicit QgsFetchedContent( const QString &url, QTemporaryFile *file = 0, ContentStatus status = NotStarted,
36+
const QString &authConfig = QString() );
3637
%Docstring
3738
Constructs a FetchedContent with pointer to the downloaded file and status of the download
3839
%End
@@ -54,6 +55,11 @@ Returns the status of the download
5455
QNetworkReply::NetworkError error() const;
5556
%Docstring
5657
Returns the potential error of the download
58+
%End
59+
60+
QString authConfig() const;
61+
%Docstring
62+
Returns the authentication configuration id use for this fetched content
5763
%End
5864

5965
public slots:
@@ -74,6 +80,14 @@ Cancel the download operation.
7480
void fetched();
7581
%Docstring
7682
Emitted when the file is fetched and accessible
83+
%End
84+
85+
void errorOccurred( QNetworkReply::NetworkError code, const QString &errorMsg );
86+
%Docstring
87+
Emitted when an error with ``code`` error occured while processing the request
88+
``errorMsg`` is a textual description of the error
89+
90+
.. versionadded:: 3.22
7791
%End
7892

7993
};
@@ -103,12 +117,13 @@ Create the registry for temporary downloaded files
103117

104118
~QgsNetworkContentFetcherRegistry();
105119

106-
const QgsFetchedContent *fetch( const QString &url, Qgis::ActionStart fetchingMode = Qgis::ActionStart::Deferred );
120+
QgsFetchedContent *fetch( const QString &url, Qgis::ActionStart fetchingMode = Qgis::ActionStart::Deferred, const QString &authConfig = QString() );
107121
%Docstring
108122
Initialize a download for the given URL
109123

110124
:param url: the URL to be fetched
111125
:param fetchingMode: defines if the download will start immediately or shall be manually triggered
126+
:param authConfig: authentication configuration id to be used while fetching
112127

113128
.. note::
114129

‎python/core/auto_generated/network/qgsnetworkcontentfetchertask.sip.in

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,14 @@ of whether the fetch was successful or not.
9191
Users of QgsNetworkContentFetcherTask should connect to this signal,
9292
and from the associated slot they can then safely access the network :py:func:`~QgsNetworkContentFetcherTask.reply`
9393
without danger of the task being first removed by the :py:class:`QgsTaskManager`.
94+
%End
95+
96+
void errorOccurred( QNetworkReply::NetworkError code, const QString &errorMsg );
97+
%Docstring
98+
Emitted when an error with ``code`` error occured while processing the request
99+
``errorMsg`` is a textual description of the error
100+
101+
.. versionadded:: 3.22
94102
%End
95103

96104
};

‎src/core/CMakeLists.txt

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,7 @@ set(QGIS_CORE_SRCS
133133
externalstorage/qgsexternalstorage.cpp
134134
externalstorage/qgsexternalstorageregistry.cpp
135135
externalstorage/qgssimplecopyexternalstorage.cpp
136+
externalstorage/qgswebdavexternalstorage.cpp
136137

137138
layertree/qgscolorramplegendnode.cpp
138139
layertree/qgscolorramplegendnodesettings.cpp
@@ -1750,8 +1751,8 @@ set(QGIS_CORE_PRIVATE_HDRS
17501751
qgsspatialindexkdbush_p.h
17511752

17521753
editform/qgseditformconfig_p.h
1753-
17541754
externalstorage/qgssimplecopyexternalstorage_p.h
1755+
externalstorage/qgswebdavexternalstorage_p.h
17551756

17561757
proj/qgscoordinatereferencesystem_p.h
17571758
proj/qgscoordinatetransformcontext_p.h

‎src/core/externalstorage/qgsexternalstorageregistry.cpp

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,10 +17,12 @@
1717

1818
#include "qgsexternalstorage.h"
1919
#include "qgssimplecopyexternalstorage_p.h"
20+
#include "qgswebdavexternalstorage_p.h"
2021

2122
QgsExternalStorageRegistry::QgsExternalStorageRegistry()
2223
{
2324
registerExternalStorage( new QgsSimpleCopyExternalStorage() );
25+
registerExternalStorage( new QgsWebDAVExternalStorage() );
2426
}
2527

2628
QgsExternalStorageRegistry::~QgsExternalStorageRegistry()

‎src/core/externalstorage/qgswebdavexternalstorage.cpp

Lines changed: 200 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,200 @@
1+
/***************************************************************************
2+
qgswebdavexternalstorage.cpp
3+
--------------------------------------
4+
Date : March 2021
5+
Copyright : (C) 2021 by Julien Cabieces
6+
Email : julien dot cabieces at oslandia dot com
7+
***************************************************************************
8+
* *
9+
* This program is free software; you can redistribute it and/or modify *
10+
* it under the terms of the GNU General Public License as published by *
11+
* the Free Software Foundation; either version 2 of the License, or *
12+
* (at your option) any later version. *
13+
* *
14+
***************************************************************************/
15+
16+
#include "qgswebdavexternalstorage_p.h"
17+
18+
#include "qgsnetworkcontentfetcherregistry.h"
19+
#include "qgsblockingnetworkrequest.h"
20+
#include "qgsnetworkaccessmanager.h"
21+
#include "qgsapplication.h"
22+
23+
#include <QFile>
24+
#include <QPointer>
25+
#include <QFileInfo>
26+
27+
QgsWebDAVExternalStorageStoreTask::QgsWebDAVExternalStorageStoreTask( const QUrl &url, const QString &filePath, const QString &authCfg )
28+
: QgsTask( tr( "Storing %1" ).arg( QFileInfo( filePath ).baseName() ) )
29+
, mUrl( url )
30+
, mFilePath( filePath )
31+
, mAuthCfg( authCfg )
32+
, mFeedback( new QgsFeedback( this ) )
33+
{
34+
}
35+
36+
bool QgsWebDAVExternalStorageStoreTask::run()
37+
{
38+
QgsBlockingNetworkRequest request;
39+
request.setAuthCfg( mAuthCfg );
40+
41+
QNetworkRequest req( mUrl );
42+
QgsSetRequestInitiatorClass( req, QStringLiteral( "QgsWebDAVExternalStorageStoreTask" ) );
43+
44+
QFile *f = new QFile( mFilePath );
45+
f->open( QIODevice::ReadOnly );
46+
47+
connect( &request, &QgsBlockingNetworkRequest::uploadProgress, this, [ = ]( qint64 bytesReceived, qint64 bytesTotal )
48+
{
49+
if ( !isCanceled() && bytesTotal > 0 )
50+
{
51+
const int progress = ( bytesReceived * 100 ) / bytesTotal;
52+
setProgress( progress );
53+
}
54+
} );
55+
56+
QgsBlockingNetworkRequest::ErrorCode err = request.put( req, f, mFeedback );
57+
58+
if ( err != QgsBlockingNetworkRequest::NoError )
59+
{
60+
mErrorString = request.errorMessage();
61+
}
62+
63+
return !isCanceled() && err == QgsBlockingNetworkRequest::NoError;
64+
}
65+
66+
void QgsWebDAVExternalStorageStoreTask::cancel()
67+
{
68+
mFeedback->cancel();
69+
QgsTask::cancel();
70+
}
71+
72+
QString QgsWebDAVExternalStorageStoreTask::errorString() const
73+
{
74+
return mErrorString;
75+
}
76+
77+
QgsWebDAVExternalStorageStoredContent::QgsWebDAVExternalStorageStoredContent( const QString &filePath, const QString &url, const QString &authcfg )
78+
{
79+
QString storageUrl = url;
80+
if ( storageUrl.endsWith( "/" ) )
81+
storageUrl.append( QFileInfo( filePath ).fileName() );
82+
83+
mUploadTask = new QgsWebDAVExternalStorageStoreTask( storageUrl, filePath, authcfg );
84+
85+
connect( mUploadTask, &QgsTask::taskCompleted, this, [ = ]
86+
{
87+
mUrl = storageUrl;
88+
mStatus = Qgis::ContentStatus::Finished;
89+
emit stored();
90+
} );
91+
92+
connect( mUploadTask, &QgsTask::taskTerminated, this, [ = ]
93+
{
94+
reportError( mUploadTask->errorString() );
95+
} );
96+
97+
connect( mUploadTask, &QgsTask::progressChanged, this, [ = ]( double progress )
98+
{
99+
emit progressChanged( progress );
100+
} );
101+
}
102+
103+
void QgsWebDAVExternalStorageStoredContent::store()
104+
{
105+
mStatus = Qgis::ContentStatus::Running;
106+
QgsApplication::instance()->taskManager()->addTask( mUploadTask );
107+
}
108+
109+
110+
void QgsWebDAVExternalStorageStoredContent::cancel()
111+
{
112+
if ( !mUploadTask )
113+
return;
114+
115+
disconnect( mUploadTask, &QgsTask::taskTerminated, this, nullptr );
116+
connect( mUploadTask, &QgsTask::taskTerminated, this, [ = ]
117+
{
118+
mStatus = Qgis::ContentStatus::Canceled;
119+
emit canceled();
120+
} );
121+
122+
mUploadTask->cancel();
123+
}
124+
125+
QString QgsWebDAVExternalStorageStoredContent::url() const
126+
{
127+
return mUrl;
128+
}
129+
130+
131+
QgsWebDAVExternalStorageFetchedContent::QgsWebDAVExternalStorageFetchedContent( QgsFetchedContent *fetchedContent )
132+
: mFetchedContent( fetchedContent )
133+
{
134+
connect( mFetchedContent, &QgsFetchedContent::fetched, this, &QgsWebDAVExternalStorageFetchedContent::onFetched );
135+
connect( mFetchedContent, &QgsFetchedContent::errorOccurred, this, [ = ]( QNetworkReply::NetworkError code, const QString & errorMsg )
136+
{
137+
Q_UNUSED( code );
138+
reportError( errorMsg );
139+
} );
140+
}
141+
142+
void QgsWebDAVExternalStorageFetchedContent::fetch()
143+
{
144+
if ( !mFetchedContent )
145+
return;
146+
147+
mStatus = Qgis::ContentStatus::Running;
148+
mFetchedContent->download();
149+
150+
// could be already fetched/cached
151+
if ( mFetchedContent->status() == QgsFetchedContent::Finished )
152+
{
153+
mStatus = Qgis::ContentStatus::Finished;
154+
emit fetched();
155+
}
156+
}
157+
158+
QString QgsWebDAVExternalStorageFetchedContent::filePath() const
159+
{
160+
return mFetchedContent ? mFetchedContent->filePath() : QString();
161+
}
162+
163+
void QgsWebDAVExternalStorageFetchedContent::onFetched()
164+
{
165+
if ( !mFetchedContent )
166+
return;
167+
168+
if ( mFetchedContent->status() == QgsFetchedContent::Finished )
169+
{
170+
mStatus = Qgis::ContentStatus::Finished;
171+
emit fetched();
172+
}
173+
}
174+
175+
void QgsWebDAVExternalStorageFetchedContent::cancel()
176+
{
177+
mFetchedContent->cancel();
178+
}
179+
180+
QString QgsWebDAVExternalStorage::type() const
181+
{
182+
return QStringLiteral( "WebDAV" );
183+
};
184+
185+
QString QgsWebDAVExternalStorage::displayName() const
186+
{
187+
return QObject::tr( "WebDAV Storage" );
188+
};
189+
190+
QgsExternalStorageStoredContent *QgsWebDAVExternalStorage::doStore( const QString &filePath, const QString &url, const QString &authcfg ) const
191+
{
192+
return new QgsWebDAVExternalStorageStoredContent( filePath, url, authcfg );
193+
};
194+
195+
QgsExternalStorageFetchedContent *QgsWebDAVExternalStorage::doFetch( const QString &url, const QString &authConfig ) const
196+
{
197+
QgsFetchedContent *fetchedContent = QgsApplication::instance()->networkContentFetcherRegistry()->fetch( url, Qgis::ActionStart::Deferred, authConfig );
198+
199+
return new QgsWebDAVExternalStorageFetchedContent( fetchedContent );
200+
}

‎src/core/externalstorage/qgswebdavexternalstorage_p.h

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
/***************************************************************************
2+
qgswebdavexternalstorage.h
3+
--------------------------------------
4+
Date : March 2021
5+
Copyright : (C) 2021 by Julien Cabieces
6+
Email : julien dot cabieces at oslandia dot com
7+
***************************************************************************
8+
* *
9+
* This program is free software; you can redistribute it and/or modify *
10+
* it under the terms of the GNU General Public License as published by *
11+
* the Free Software Foundation; either version 2 of the License, or *
12+
* (at your option) any later version. *
13+
* *
14+
***************************************************************************/
15+
16+
#ifndef QGSWEBDAVEXTERNALSTORAGE_H
17+
#define QGSWEBDAVEXTERNALSTORAGE_H
18+
19+
#include "qgis_core.h"
20+
#include "qgis_sip.h"
21+
#include "qgstaskmanager.h"
22+
23+
#include "externalstorage/qgsexternalstorage.h"
24+
25+
#include <QPointer>
26+
27+
class QgsWebDAVExternalStorageStoreTask;
28+
class QgsFetchedContent;
29+
30+
///@cond PRIVATE
31+
#define SIP_NO_FILE
32+
33+
/**
34+
* \ingroup core
35+
* \brief External storage implementation using the protocol WebDAV.
36+
*
37+
* \since QGIS 3.22
38+
*/
39+
class CORE_EXPORT QgsWebDAVExternalStorage : public QgsExternalStorage
40+
{
41+
public:
42+
43+
QString type() const override;
44+
45+
QString displayName() const override;
46+
47+
protected:
48+
49+
QgsExternalStorageStoredContent *doStore( const QString &filePath, const QString &url, const QString &authcfg = QString() ) const override;
50+
51+
QgsExternalStorageFetchedContent *doFetch( const QString &url, const QString &authConfig = QString() ) const override;
52+
};
53+
54+
/**
55+
* \ingroup core
56+
* \brief Class for WebDAV stored content
57+
*
58+
* \since QGIS 3.22
59+
*/
60+
class QgsWebDAVExternalStorageStoredContent : public QgsExternalStorageStoredContent
61+
{
62+
Q_OBJECT
63+
64+
public:
65+
66+
QgsWebDAVExternalStorageStoredContent( const QString &filePath, const QString &url, const QString &authcfg = QString() );
67+
68+
void cancel() override;
69+
70+
QString url() const override;
71+
72+
void store() override;
73+
74+
private:
75+
76+
QPointer<QgsWebDAVExternalStorageStoreTask> mUploadTask;
77+
QString mUrl;
78+
};
79+
80+
/**
81+
* \ingroup core
82+
* \brief Class for WebDAV fetched content
83+
*
84+
* \since QGIS 3.22
85+
*/
86+
class QgsWebDAVExternalStorageFetchedContent : public QgsExternalStorageFetchedContent
87+
{
88+
Q_OBJECT
89+
90+
public:
91+
92+
QgsWebDAVExternalStorageFetchedContent( QgsFetchedContent *fetchedContent );
93+
94+
QString filePath() const override;
95+
96+
void cancel() override;
97+
98+
void fetch() override;
99+
100+
private slots:
101+
102+
void onFetched();
103+
104+
private:
105+
106+
QPointer<QgsFetchedContent> mFetchedContent;
107+
};
108+
109+
110+
/**
111+
* \ingroup core
112+
* \brief Task to store a file to a given WebDAV url
113+
*
114+
* \since QGIS 3.22
115+
*/
116+
class QgsWebDAVExternalStorageStoreTask : public QgsTask
117+
{
118+
Q_OBJECT
119+
120+
public:
121+
122+
QgsWebDAVExternalStorageStoreTask( const QUrl &url, const QString &filePath, const QString &authCfg );
123+
124+
bool run() override;
125+
126+
void cancel() override;
127+
128+
QString errorString() const;
129+
130+
private:
131+
132+
const QUrl mUrl;
133+
const QString mFilePath;
134+
const QString mAuthCfg;
135+
QgsFeedback *mFeedback = nullptr;
136+
QString mErrorString;
137+
};
138+
139+
140+
141+
#endif // QGSWEBDAVEXTERNALSTORAGE_H

‎src/core/network/qgsblockingnetworkrequest.cpp

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
#include <QWaitCondition>
2828
#include <QNetworkCacheMetaData>
2929
#include <QAuthenticator>
30+
#include <QBuffer>
3031

3132
QgsBlockingNetworkRequest::QgsBlockingNetworkRequest()
3233
{
@@ -60,6 +61,14 @@ QgsBlockingNetworkRequest::ErrorCode QgsBlockingNetworkRequest::get( QNetworkReq
6061
}
6162

6263
QgsBlockingNetworkRequest::ErrorCode QgsBlockingNetworkRequest::post( QNetworkRequest &request, const QByteArray &data, bool forceRefresh, QgsFeedback *feedback )
64+
{
65+
QByteArray ldata( data );
66+
QBuffer buffer( &ldata );
67+
buffer.open( QIODevice::ReadOnly );
68+
return post( request, &buffer, forceRefresh, feedback );
69+
}
70+
71+
QgsBlockingNetworkRequest::ErrorCode QgsBlockingNetworkRequest::post( QNetworkRequest &request, QIODevice *data, bool forceRefresh, QgsFeedback *feedback )
6372
{
6473
mPayloadData = data;
6574
return doRequest( Post, request, forceRefresh, feedback );
@@ -71,6 +80,14 @@ QgsBlockingNetworkRequest::ErrorCode QgsBlockingNetworkRequest::head( QNetworkRe
7180
}
7281

7382
QgsBlockingNetworkRequest::ErrorCode QgsBlockingNetworkRequest::put( QNetworkRequest &request, const QByteArray &data, QgsFeedback *feedback )
83+
{
84+
QByteArray ldata( data );
85+
QBuffer buffer( &ldata );
86+
buffer.open( QIODevice::ReadOnly );
87+
return put( request, &buffer, feedback );
88+
}
89+
90+
QgsBlockingNetworkRequest::ErrorCode QgsBlockingNetworkRequest::put( QNetworkRequest &request, QIODevice *data, QgsFeedback *feedback )
7491
{
7592
mPayloadData = data;
7693
return doRequest( Put, request, true, feedback );
@@ -177,6 +194,7 @@ QgsBlockingNetworkRequest::ErrorCode QgsBlockingNetworkRequest::doRequest( QgsBl
177194
// * or the owner thread of mReply is currently not doing anything because it's blocked in future.waitForFinished() (if it is the main thread)
178195
connect( mReply, &QNetworkReply::finished, this, &QgsBlockingNetworkRequest::replyFinished, Qt::DirectConnection );
179196
connect( mReply, &QNetworkReply::downloadProgress, this, &QgsBlockingNetworkRequest::replyProgress, Qt::DirectConnection );
197+
connect( mReply, &QNetworkReply::uploadProgress, this, &QgsBlockingNetworkRequest::replyProgress, Qt::DirectConnection );
180198

181199
auto resumeMainThread = [&waitConditionMutex, &authRequestBufferNotEmpty ]()
182200
{
@@ -288,7 +306,10 @@ void QgsBlockingNetworkRequest::replyProgress( qint64 bytesReceived, qint64 byte
288306
}
289307
}
290308

291-
emit downloadProgress( bytesReceived, bytesTotal );
309+
if ( mMethod == Put || mMethod == Post )
310+
emit uploadProgress( bytesReceived, bytesTotal );
311+
else
312+
emit downloadProgress( bytesReceived, bytesTotal );
292313
}
293314

294315
void QgsBlockingNetworkRequest::replyFinished()
@@ -351,6 +372,7 @@ void QgsBlockingNetworkRequest::replyFinished()
351372

352373
connect( mReply, &QNetworkReply::finished, this, &QgsBlockingNetworkRequest::replyFinished, Qt::DirectConnection );
353374
connect( mReply, &QNetworkReply::downloadProgress, this, &QgsBlockingNetworkRequest::replyProgress, Qt::DirectConnection );
375+
connect( mReply, &QNetworkReply::uploadProgress, this, &QgsBlockingNetworkRequest::replyProgress, Qt::DirectConnection );
354376
return;
355377
}
356378
}

‎src/core/network/qgsblockingnetworkrequest.h

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,14 @@ class CORE_EXPORT QgsBlockingNetworkRequest : public QObject
103103
* can be retrieved by calling errorMessage().
104104
*
105105
* \see get()
106+
* \since 3.22
107+
*/
108+
ErrorCode post( QNetworkRequest &request, QIODevice *data, bool forceRefresh = false, QgsFeedback *feedback = nullptr );
109+
110+
/**
111+
* This is an overloaded function.
112+
*
113+
* Performs a "post" operation on the specified \a request, using the given \a data.
106114
*/
107115
ErrorCode post( QNetworkRequest &request, const QByteArray &data, bool forceRefresh = false, QgsFeedback *feedback = nullptr );
108116

@@ -143,6 +151,14 @@ class CORE_EXPORT QgsBlockingNetworkRequest : public QObject
143151
* If an error was encountered then a specific ErrorCode will be returned, and a detailed error message
144152
* can be retrieved by calling errorMessage().
145153
*
154+
* \since 3.22
155+
*/
156+
ErrorCode put( QNetworkRequest &request, QIODevice *data, QgsFeedback *feedback = nullptr );
157+
158+
/**
159+
* This is an overloaded function.
160+
*
161+
* Performs a "put" operation on the specified \a request, using the given \a data.
146162
* \since 3.18
147163
*/
148164
ErrorCode put( QNetworkRequest &request, const QByteArray &data, QgsFeedback *feedback = nullptr );
@@ -207,6 +223,12 @@ class CORE_EXPORT QgsBlockingNetworkRequest : public QObject
207223
*/
208224
void downloadFinished();
209225

226+
/**
227+
* Emitted when when data are sent during a request.
228+
* \since QGIS 3.22
229+
*/
230+
void uploadProgress( qint64, qint64 );
231+
210232
private slots:
211233
void replyProgress( qint64, qint64 );
212234
void replyFinished();
@@ -227,7 +249,9 @@ class CORE_EXPORT QgsBlockingNetworkRequest : public QObject
227249
QNetworkReply *mReply = nullptr;
228250

229251
Method mMethod = Get;
230-
QByteArray mPayloadData;
252+
253+
//! payload data used in PUT/POST request
254+
QIODevice *mPayloadData;
231255

232256
//! Authentication configuration ID
233257
QString mAuthCfg;

‎src/core/network/qgsnetworkcontentfetcher.cpp

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,17 @@ void QgsNetworkContentFetcher::fetchContent( const QNetworkRequest &r, const QSt
7171
mReply->setParent( nullptr ); // we don't want thread locale QgsNetworkAccessManagers to delete the reply - we want ownership of it to belong to this object
7272
connect( mReply, &QNetworkReply::finished, this, [ = ] { contentLoaded(); } );
7373
connect( mReply, &QNetworkReply::downloadProgress, this, &QgsNetworkContentFetcher::downloadProgress );
74+
75+
auto onError = [ = ]( QNetworkReply::NetworkError code )
76+
{
77+
emit errorOccurred( code, mReply->errorString() );
78+
};
79+
80+
#if QT_VERSION < QT_VERSION_CHECK(5, 15, 0)
81+
connect( mReply, qOverload<QNetworkReply::NetworkError>( &QNetworkReply::error ), this, onError );
82+
#else
83+
connect( mReply, &QNetworkReply::errorOccurred, this, onError );
84+
#endif
7485
}
7586

7687
QNetworkReply *QgsNetworkContentFetcher::reply()
@@ -190,7 +201,3 @@ void QgsNetworkContentFetcher::contentLoaded( bool ok )
190201
mReply->deleteLater();
191202
fetchContent( redirect.toUrl(), mAuthCfg );
192203
}
193-
194-
195-
196-

‎src/core/network/qgsnetworkcontentfetcher.h

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,13 @@ class CORE_EXPORT QgsNetworkContentFetcher : public QObject
105105
*/
106106
void downloadProgress( qint64 bytesReceived, qint64 bytesTotal );
107107

108+
/**
109+
* Emitted when an error with \a code error occured while processing the request
110+
* \a errorMsg is a textual description of the error
111+
* \since QGIS 3.22
112+
*/
113+
void errorOccurred( QNetworkReply::NetworkError code, const QString &errorMsg );
114+
108115
private:
109116

110117
QString mAuthCfg;

‎src/core/network/qgsnetworkcontentfetcherregistry.cpp

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,8 @@
2020

2121
#include "qgsapplication.h"
2222
#include <QUrl>
23+
#include <QFileInfo>
24+
#include <QDir>
2325

2426
QgsNetworkContentFetcherRegistry::~QgsNetworkContentFetcherRegistry()
2527
{
@@ -31,15 +33,15 @@ QgsNetworkContentFetcherRegistry::~QgsNetworkContentFetcherRegistry()
3133
mFileRegistry.clear();
3234
}
3335

34-
const QgsFetchedContent *QgsNetworkContentFetcherRegistry::fetch( const QString &url, const Qgis::ActionStart fetchingMode )
36+
QgsFetchedContent *QgsNetworkContentFetcherRegistry::fetch( const QString &url, const Qgis::ActionStart fetchingMode, const QString &authConfig )
3537
{
3638

3739
if ( mFileRegistry.contains( url ) )
3840
{
3941
return mFileRegistry.value( url );
4042
}
4143

42-
QgsFetchedContent *content = new QgsFetchedContent( url, nullptr, QgsFetchedContent::NotStarted );
44+
QgsFetchedContent *content = new QgsFetchedContent( url, nullptr, QgsFetchedContent::NotStarted, authConfig );
4345

4446
mFileRegistry.insert( url, content );
4547

@@ -126,9 +128,11 @@ void QgsFetchedContent::download( bool redownload )
126128
status() == QgsFetchedContent::NotStarted ||
127129
status() == QgsFetchedContent::Failed )
128130
{
129-
mFetchingTask = new QgsNetworkContentFetcherTask( mUrl );
131+
mFetchingTask = new QgsNetworkContentFetcherTask( mUrl, mAuthConfig );
130132
// use taskCompleted which is main thread rather than fetched signal in worker thread
131133
connect( mFetchingTask, &QgsNetworkContentFetcherTask::taskCompleted, this, &QgsFetchedContent::taskCompleted );
134+
connect( mFetchingTask, &QgsNetworkContentFetcherTask::taskTerminated, this, &QgsFetchedContent::taskCompleted );
135+
connect( mFetchingTask, &QgsNetworkContentFetcherTask::errorOccurred, this, &QgsFetchedContent::errorOccurred );
132136
QgsApplication::instance()->taskManager()->addTask( mFetchingTask );
133137
mStatus = QgsFetchedContent::Downloading;
134138
}
@@ -163,7 +167,12 @@ void QgsFetchedContent::taskCompleted()
163167
QNetworkReply *reply = mFetchingTask->reply();
164168
if ( reply->error() == QNetworkReply::NoError )
165169
{
166-
QTemporaryFile *tf = new QTemporaryFile( QStringLiteral( "XXXXXX" ) );
170+
// keep extension, it can be usefull when guessing file content
171+
// (when loading this file in a Qt WebView for instance)
172+
const QString extension = QFileInfo( reply->request().url().fileName() ).completeSuffix();
173+
174+
QTemporaryFile *tf = new QTemporaryFile( extension.isEmpty() ? QString( "XXXXXX" ) :
175+
QString( "%1/XXXXXX.%2" ).arg( QDir::tempPath(), extension ) );
167176
mFile = tf;
168177
tf->open();
169178
mFile->write( reply->readAll() );

‎src/core/network/qgsnetworkcontentfetcherregistry.h

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,10 +51,12 @@ class CORE_EXPORT QgsFetchedContent : public QObject
5151
};
5252

5353
//! Constructs a FetchedContent with pointer to the downloaded file and status of the download
54-
explicit QgsFetchedContent( const QString &url, QTemporaryFile *file = nullptr, ContentStatus status = NotStarted )
54+
explicit QgsFetchedContent( const QString &url, QTemporaryFile *file = nullptr, ContentStatus status = NotStarted,
55+
const QString &authConfig = QString() )
5556
: mUrl( url )
5657
, mFile( file )
5758
, mStatus( status )
59+
, mAuthConfig( authConfig )
5860
{}
5961

6062
~QgsFetchedContent() override
@@ -79,6 +81,11 @@ class CORE_EXPORT QgsFetchedContent : public QObject
7981
//! Returns the potential error of the download
8082
QNetworkReply::NetworkError error() const {return mError;}
8183

84+
/**
85+
* Returns the authentication configuration id use for this fetched content
86+
*/
87+
QString authConfig() const {return mAuthConfig;}
88+
8289
public slots:
8390

8491
/**
@@ -96,6 +103,13 @@ class CORE_EXPORT QgsFetchedContent : public QObject
96103
//! Emitted when the file is fetched and accessible
97104
void fetched();
98105

106+
/**
107+
* Emitted when an error with \a code error occured while processing the request
108+
* \a errorMsg is a textual description of the error
109+
* \since QGIS 3.22
110+
*/
111+
void errorOccurred( QNetworkReply::NetworkError code, const QString &errorMsg );
112+
99113
private slots:
100114
void taskCompleted();
101115

@@ -106,6 +120,8 @@ class CORE_EXPORT QgsFetchedContent : public QObject
106120
QgsNetworkContentFetcherTask *mFetchingTask = nullptr;
107121
ContentStatus mStatus = NotStarted;
108122
QNetworkReply::NetworkError mError = QNetworkReply::NoError;
123+
QString mAuthConfig;
124+
QString mErrorString;
109125
};
110126

111127
/**
@@ -134,9 +150,10 @@ class CORE_EXPORT QgsNetworkContentFetcherRegistry : public QObject
134150
* \brief Initialize a download for the given URL
135151
* \param url the URL to be fetched
136152
* \param fetchingMode defines if the download will start immediately or shall be manually triggered
153+
* \param authConfig authentication configuration id to be used while fetching
137154
* \note If the download starts immediately, it will not redownload any already fetched or currently fetching file.
138155
*/
139-
const QgsFetchedContent *fetch( const QString &url, Qgis::ActionStart fetchingMode = Qgis::ActionStart::Deferred );
156+
QgsFetchedContent *fetch( const QString &url, Qgis::ActionStart fetchingMode = Qgis::ActionStart::Deferred, const QString &authConfig = QString() );
140157

141158
#ifndef SIP_RUN
142159

‎src/core/network/qgsnetworkcontentfetchertask.cpp

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,8 +41,13 @@ bool QgsNetworkContentFetcherTask::run()
4141
{
4242
mFetcher = new QgsNetworkContentFetcher();
4343
QEventLoop loop;
44+
45+
// We need to set the event loop (and not 'this') as receiver for all signal to ensure execution
46+
// in the same thread and in the same order of emission. Indeed 'this' and 'loop' lives in
47+
// different thread because they have been created in different thread.
48+
4449
connect( mFetcher, &QgsNetworkContentFetcher::finished, &loop, &QEventLoop::quit );
45-
connect( mFetcher, &QgsNetworkContentFetcher::downloadProgress, this, [ = ]( qint64 bytesReceived, qint64 bytesTotal )
50+
connect( mFetcher, &QgsNetworkContentFetcher::downloadProgress, &loop, [ = ]( qint64 bytesReceived, qint64 bytesTotal )
4651
{
4752
if ( !isCanceled() && bytesTotal > 0 )
4853
{
@@ -53,12 +58,22 @@ bool QgsNetworkContentFetcherTask::run()
5358
setProgress( progress );
5459
}
5560
} );
61+
62+
63+
bool hasErrorOccurred = false;
64+
connect( mFetcher, &QgsNetworkContentFetcher::errorOccurred, &loop, [ &hasErrorOccurred, this ]( QNetworkReply::NetworkError code, const QString & errorMsg )
65+
{
66+
hasErrorOccurred = true;
67+
emit errorOccurred( code, errorMsg );
68+
} );
69+
5670
mFetcher->fetchContent( mRequest, mAuthcfg );
5771
loop.exec();
5872
if ( !isCanceled() )
5973
setProgress( 100 );
6074
emit fetched();
61-
return true;
75+
76+
return !isCanceled() && !hasErrorOccurred;
6277
}
6378

6479
void QgsNetworkContentFetcherTask::cancel()

‎src/core/network/qgsnetworkcontentfetchertask.h

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,9 +23,9 @@
2323
#include "qgstaskmanager.h"
2424
#include "qgis_core.h"
2525
#include <QNetworkRequest>
26+
#include <QNetworkReply>
2627

2728
class QgsNetworkContentFetcher;
28-
class QNetworkReply;
2929

3030
/**
3131
* \class QgsNetworkContentFetcherTask
@@ -103,12 +103,20 @@ class CORE_EXPORT QgsNetworkContentFetcherTask : public QgsTask
103103
*/
104104
void fetched();
105105

106+
/**
107+
* Emitted when an error with \a code error occured while processing the request
108+
* \a errorMsg is a textual description of the error
109+
* \since QGIS 3.22
110+
*/
111+
void errorOccurred( QNetworkReply::NetworkError code, const QString &errorMsg );
112+
106113
private:
107114

108115
QNetworkRequest mRequest;
109116
QString mAuthcfg;
110117
QgsNetworkContentFetcher *mFetcher = nullptr;
111-
118+
QString mMode;
119+
QIODevice *mContent = nullptr;
112120
};
113121

114122
#endif //QGSNETWORKCONTENTFETCHERTASK_H

‎tests/src/python/CMakeLists.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,7 @@ ADD_PYTHON_TEST(PyQgsExpressionBuilderWidget test_qgsexpressionbuilderwidget.py)
8080
ADD_PYTHON_TEST(PyQgsExpressionLineEdit test_qgsexpressionlineedit.py)
8181
ADD_PYTHON_TEST(PyQgsExtentGroupBox test_qgsextentgroupbox.py)
8282
ADD_PYTHON_TEST(PyQgsExtentWidget test_qgsextentwidget.py)
83+
ADD_PYTHON_TEST(PyQgsExternalStorageWebDAV test_qgsexternalstorage_webdav.py)
8384
ADD_PYTHON_TEST(PyQgsFeature test_qgsfeature.py)
8485
ADD_PYTHON_TEST(PyQgsFeatureSink test_qgsfeaturesink.py)
8586
ADD_PYTHON_TEST(PyQgsFeatureSource test_qgsfeaturesource.py)

‎tests/src/python/test_qgsexternalstorage_webdav.py

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
# -*- coding: utf-8 -*-
2+
"""QGIS Unit tests for WebDAV external storage
3+
4+
External storage backend must implement a test based on TestPyQgsExternalStorageBase
5+
6+
.. note:: This program is free software; you can redistribute it and/or modify
7+
it under the terms of the GNU General Public License as published by
8+
the Free Software Foundation; either version 2 of the License, or
9+
(at your option) any later version.
10+
"""
11+
12+
__author__ = 'Julien Cabieces'
13+
__date__ = '31/03/2021'
14+
__copyright__ = 'Copyright 2021, The QGIS Project'
15+
16+
from shutil import rmtree
17+
import os
18+
import tempfile
19+
import time
20+
21+
from utilities import unitTestDataPath, waitServer
22+
from test_qgsexternalstorage_base import TestPyQgsExternalStorageBase
23+
24+
from qgis.PyQt.QtCore import QCoreApplication, QEventLoop, QUrl
25+
26+
from qgis.core import (
27+
QgsApplication,
28+
QgsAuthMethodConfig,
29+
QgsExternalStorageFetchedContent)
30+
31+
from qgis.testing import (
32+
start_app,
33+
unittest,
34+
)
35+
36+
37+
class TestPyQgsExternalStorageWebDAV(TestPyQgsExternalStorageBase, unittest.TestCase):
38+
39+
storageType = "WebDAV"
40+
badUrl = "http://nothinghere/"
41+
42+
@classmethod
43+
def setUpClass(cls):
44+
"""Run before all tests:"""
45+
46+
super().setUpClass()
47+
48+
cls.url = "http://{}:{}/webdav_tests".format(
49+
os.environ.get('QGIS_WEBDAV_HOST', 'localhost'), os.environ.get('QGIS_WEBDAV_PORT', '80'))
50+
51+
52+
if __name__ == '__main__':
53+
unittest.main()

0 commit comments

Comments
 (0)
Please sign in to comment.