[feature] virtual raster data provider (#44195)

* try to commit some changes in the branch * addedd 3 file, cmake, vrp .h and .cpp * updating VRP, no good results * try to solve some issue to the virtual raster provider * metadata class added, it should be developed * registred in qgsproviderregistry the new VRP * on line to test debug in the new VRP * chenge of 1_06 * minor change 2 06 * updating VRP, it can add some useless color to map canvas * add baseclass test fro new provider * Delete testqgsvirtualrasterprovider.cpp * move VRP to src/providers * Delete src/core/providers/virtualrasterprovider directory * little change to cmake * added as dynamic provider * move the VRP to src/providers and let it work * move the VRP to src/providers and let it work * including qgis analysis * updating block function with some calc tools * delete block function, updating readblock function * modified block method, used setValue --> to update with bits * modified block method * modified block method, adding takedata to the resultmatrix * modified test for the new provider * Add some checks in the block method, for raster entries * tries for the calculator tool * work on the hardcoded part and test * work on the hardcoded test * minor change in order to make the calc functionaltiy work * minor change to the loop in which the computation occurs * try to understand how qgsrastermatrixworks * work on debugging new class * work on debugging new class * updating test and block method * updating test and and block, now it works running in test mode * updating test and and block, now it works running in test mode with bits * updating test and and block, now it works running in test mode with bits * update test * update test and provider class * add decodeuri and encodeui in metadata, at the moment they do nothing * modify qgsrasterdataprovider by adding decoding and encoding uri methods * workin on encode and decode funct * work on encode/decode after Martin advices * work on encode/decode after Martin advices, and update unit test * work on encode/decode and update Cmakelist of provider test * work on encode/decode and update Cmakelist and provider test * work on encode/decode and update test + constructor * work on encode/decode and constructor * decode is now non trivial, he same as constructor * working on encoding * working on encoding/decoding * change a little the code/decode mehods in order to return and get a struct as output/input * change of decode method * change the encode method, I should polish the code and optimize the encode for raster layers * add flag ptr bool to decode mehod * add flag ptr and modified the constructor of vrprovider * work on the constructor and deconstructor, the flag does not work properly * work on the constructor anf decoder * work on the constructor and decoder * work on the constructor and add some lines to popoluta mRasterEntries * delete hardcoded data * work on constructor and copy constructor 2 * add the parsing of the formula in the constructor, I should make a private var with calcnode * add the parsing of the formula in the constructor, I should make a private var with calcnode2 * add some checks in the constructor2 * modify the copy constructor, add some code to adjust the qgsrasterdataprovider.sip * modify checks in the constructor and in the decode method, in order to avoid nested if stat * delete the logic in the decode method and add in the constructor * delete the logic in the decode method and add in the constructor, add some redundant check in the constructor * delete the logic in the decode method and add in the constructor, add some redundant check in the constructor 2 * test is now more general * test is now more general2 * delete of a comment * banned keywords check * little error * add copyright * add copyright2 * add assignment operator= delete * add documentation in qgsrasterprovider * documentation * modify qgsstringutils * little modifications * add the checkbox to the ui * come back to old ui for raster calc * adjust some qt::endl in the test of vrprovider * start to work on the ui dialog * layot of the code * add some logic to the dialog of the raster calc * add some test to qgisapp, need to modify the encoder and maybe put it in the qgrastercalcdialog * modify some logic in rastercalcdialog and in the app, I should modify and move the encode method * modify the comment * add method to calcnode, modify the constructor of the provider and the open of the dialog * add documentation * some modifications according to the first review * sadd some changes according to Martin's review * change qgsrastercalcnode methods that returns raster names, small change to ui * change the ui * change the ui - using QString instead of QStringLiteral * change the ui - using QString instead of QStringLiteral * change name of subdirectory containing the virtual raster provider * change name of subdirectory containing the virtual raster provider2 * change name of subdirectory containing the virtual raster provider3 * minor changes to makefiles * minor change to dialog, to hide and show the output format and path * provider key changed to the shorter virtualraster * add some code to QgsRasterLayer::encodedSource and decodedSource * delete some comments * updating VRP, no good results * made some change after the the second review * delete the comments * change some part of the code (delete of comments and cleaning the code) * correct some layout errors * change the mail contatct in my prvider * change details on the method in qgsrastercalcnode * change some method of the test class of my provider * delete some leakage in the virtual raster provider * modify some parts according to the review + change two lines of code for encoding/decoding methods * modify a line of qgisapp.cpp * small change according to the comments on th PR * add a test method for second generation virtual raster and modify checks in qgisapp * correction of the test method Co-authored-by: franc <Franc-Brs>
qgis · Aug 17, 2021 · 179cb1d · 179cb1d
1 parent 9a4f0f5
commit 179cb1d
Show file tree

Hide file tree

Showing 17 changed files with 1,237 additions and 39 deletions.
diff --git a/python/analysis/auto_generated/raster/qgsrastercalcnode.sip.in b/python/analysis/auto_generated/raster/qgsrastercalcnode.sip.in
@@ -98,6 +98,25 @@ Returns a list of nodes of a specific ``type``
 
     static QgsRasterCalcNode *parseRasterCalcString( const QString &str, QString &parserErrorMsg ) /Factory/;
 
+    QStringList referencedLayerNames( );
+%Docstring
+Returns a list of raster layer names that are referenced in the formula without the quotation marks.
+It uses QgsRasterCalcNode.cleanRasterReferences
+
+.. note::
+
+   since QGIS 3.22
+%End
+
+    QStringList cleanRasterReferences();
+%Docstring
+Returns a list of raster layer references that are addressed in the formula, without quotation marks.
+
+.. note::
+
+   since QGIS 3.22
+%End
+
   private:
     QgsRasterCalcNode( const QgsRasterCalcNode &rh );
 };

diff --git a/python/core/auto_generated/raster/qgsrasterdataprovider.sip.in b/python/core/auto_generated/raster/qgsrasterdataprovider.sip.in
@@ -474,6 +474,50 @@ Returns a list of pyramid resampling method name and label pairs
 for given provider
 %End
 
+    struct VirtualRasterInputLayers
+    {
+      QString name;
+      QString uri;
+      QString provider;
+    };
+
+    struct VirtualRasterParameters
+    {
+      QgsCoordinateReferenceSystem crs;
+      QgsRectangle extent;
+      int width;
+      int height;
+      QString formula;
+      QList <QgsRasterDataProvider::VirtualRasterInputLayers> rInputLayers;
+
+    };
+
+    static QgsRasterDataProvider::VirtualRasterParameters decodeVirtualRasterProviderUri( const QString &uri, bool *ok = 0 );
+%Docstring
+Decodes the URI returning a struct with all the parameters for :py:class:`QgsVirtualRasterProvider` class
+
+.. note::
+
+   used by Virtual Raster Provider only
+
+.. note::
+
+   since QGIS 3.22
+%End
+
+    static QString encodeVirtualRasterProviderUri( const VirtualRasterParameters &parts );
+%Docstring
+Encodes the URI starting from the struct .
+
+.. note::
+
+   used by Virtual Raster Provider only
+
+.. note::
+
+   since QGIS 3.22
+%End
+
     virtual QString validateCreationOptions( const QStringList &createOptions, const QString &format );
 %Docstring
 Validates creation options for a specific dataset and destination format.

diff --git a/src/analysis/raster/qgsrastercalcnode.cpp b/src/analysis/raster/qgsrastercalcnode.cpp
@@ -382,3 +382,38 @@ QgsRasterCalcNode *QgsRasterCalcNode::parseRasterCalcString( const QString &str,
   return localParseRasterCalcString( str, parserErrorMsg );
 }
 
+QStringList QgsRasterCalcNode::referencedLayerNames()
+{
+  QStringList referencedRasters;
+
+  QStringList rasterRef = this->cleanRasterReferences();
+  for ( const auto &i : rasterRef )
+  {
+    if ( referencedRasters.contains( i.mid( 0, i.lastIndexOf( "@" ) ) ) ) continue;
+    referencedRasters << i.mid( 0, i.lastIndexOf( "@" ) );
+  }
+
+  return referencedRasters;
+}
+
+QStringList QgsRasterCalcNode::cleanRasterReferences()
+{
+  QStringList rasterReferences;
+  const QList<const QgsRasterCalcNode *> rasterRefNodes =  this->findNodes( QgsRasterCalcNode::Type::tRasterRef );
+
+  for ( const QgsRasterCalcNode *r : rasterRefNodes )
+  {
+
+    QString layerRef( r->toString() );
+    if ( layerRef.at( 0 ) == QStringLiteral( "\"" ) && layerRef.at( layerRef.size() - 1 ) == QStringLiteral( "\"" ) )
+    {
+      layerRef.remove( 0, 1 );
+      layerRef.chop( 1 );
+
+    }
+    layerRef.remove( QChar( '\\' ), Qt::CaseInsensitive );
+    rasterReferences << layerRef;
+  }
+
+  return rasterReferences;
+}
diff --git a/src/analysis/raster/qgsrastercalcnode.h b/src/analysis/raster/qgsrastercalcnode.h
@@ -125,6 +125,19 @@ class ANALYSIS_EXPORT QgsRasterCalcNode
 
     static QgsRasterCalcNode *parseRasterCalcString( const QString &str, QString &parserErrorMsg ) SIP_FACTORY;
 
+    /**
+     * Returns a list of raster layer names that are referenced in the formula without the quotation marks.
+     * It uses QgsRasterCalcNode::cleanRasterReferences
+     * \note since QGIS 3.22
+     */
+    QStringList referencedLayerNames( );
+
+    /**
+     * Returns a list of raster layer references that are addressed in the formula, without quotation marks.
+     * \note since QGIS 3.22
+     */
+    QStringList cleanRasterReferences();
+
   private:
 #ifdef SIP_RUN
     QgsRasterCalcNode( const QgsRasterCalcNode &rh );

diff --git a/src/app/qgisapp.cpp b/src/app/qgisapp.cpp
@@ -6562,7 +6562,47 @@ void QgisApp::newGpxLayer()
 void QgisApp::showRasterCalculator()
 {
   QgsRasterCalcDialog d( qobject_cast<QgsRasterLayer *>( activeLayer() ), this );
-  if ( d.exec() == QDialog::Accepted )
+  if ( d.exec() != QDialog::Accepted )
+  {
+    return;
+  }
+  if ( d.useVirtualProvider() )
+  {
+    QgsRasterDataProvider::VirtualRasterParameters virtualCalcParams;
+    virtualCalcParams.crs = d.outputCrs();
+    virtualCalcParams.extent = d.outputRectangle();
+    virtualCalcParams.width = d.numberOfColumns();
+    virtualCalcParams.height = d.numberOfRows();
+    virtualCalcParams.formula = d.formulaString();
+
+    QString errorString;
+    std::unique_ptr< QgsRasterCalcNode > calcNodeApp( QgsRasterCalcNode::parseRasterCalcString( d.formulaString(), errorString ) );
+    if ( !calcNodeApp )
+    {
+      return;
+    }
+    QStringList rLayerDictionaryRef = calcNodeApp->cleanRasterReferences();
+    QSet<QPair<QString, QString>> uniqueRasterUriTmp;
+
+    for ( const auto &r : QgsRasterCalculatorEntry::rasterEntries() )
+    {
+      if ( ( ! rLayerDictionaryRef.contains( r.ref ) ) ||
+           uniqueRasterUriTmp.contains( QPair( r.raster->source(), r.ref.mid( 0, r.ref.lastIndexOf( "@" ) ) ) ) ) continue;
+      uniqueRasterUriTmp.insert( QPair( r.raster->source(), r.ref.mid( 0, r.ref.lastIndexOf( "@" ) ) ) );
+
+      QgsRasterDataProvider::VirtualRasterInputLayers projectRLayer;
+      projectRLayer.name = r.ref.mid( 0, r.ref.lastIndexOf( "@" ) );
+      projectRLayer.provider = r.raster->dataProvider()->name();
+      projectRLayer.uri = r.raster->source();
+
+      virtualCalcParams.rInputLayers.append( projectRLayer );
+    }
+
+    addRasterLayer( QgsRasterDataProvider::encodeVirtualRasterProviderUri( virtualCalcParams ),
+                    d.virtualLayerName().isEmpty() ? d.formulaString() : d.virtualLayerName(),
+                    QStringLiteral( "virtualraster" ) );
+  }
+  else
   {
     //invoke analysis library
     QgsRasterCalculator rc( d.formulaString(),
@@ -6589,7 +6629,7 @@ void QgisApp::showRasterCalculator()
       case QgsRasterCalculator::Success:
         if ( d.addLayerToProject() )
         {
-          addRasterLayer( d.outputFile(), QFileInfo( d.outputFile() ).completeBaseName(), QString() );
+          addRasterLayer( d.outputFile(), QFileInfo( d.outputFile() ).completeBaseName(), QStringLiteral( "gdal" ) );
         }
         visibleMessageBar()->pushMessage( tr( "Raster calculator" ),
                                           tr( "Calculation complete." ),

diff --git a/src/app/qgsrastercalcdialog.cpp b/src/app/qgsrastercalcdialog.cpp
@@ -95,6 +95,14 @@ QgsRasterCalcDialog::QgsRasterCalcDialog( QgsRasterLayer *rasterLayer, QWidget *
   mOutputLayer->setDialogTitle( tr( "Enter result file" ) );
   mOutputLayer->setDefaultRoot( settings.value( QStringLiteral( "/RasterCalculator/lastOutputDir" ), QDir::homePath() ).toString() );
   connect( mOutputLayer, &QgsFileWidget::fileChanged, this, [ = ]() { setAcceptButtonState(); } );
+
+  connect( mUseVirtualProviderCheckBox, &QCheckBox::clicked, this, &QgsRasterCalcDialog::setOutputToVirtual );
+
+  if ( ! useVirtualProvider() )
+  {
+    setOutputToVirtual();
+  }
+
 }
 
 QString QgsRasterCalcDialog::formulaString() const
@@ -149,6 +157,16 @@ bool QgsRasterCalcDialog::addLayerToProject() const
   return mAddResultToProjectCheckBox->isChecked();
 }
 
+bool QgsRasterCalcDialog::useVirtualProvider() const
+{
+  return mUseVirtualProviderCheckBox->isChecked();
+}
+
+QString QgsRasterCalcDialog::virtualLayerName() const
+{
+  return mVirtualLayerName->text();
+}
+
 QVector<QgsRasterCalculatorEntry> QgsRasterCalcDialog::rasterEntries() const
 {
   QVector<QgsRasterCalculatorEntry> entries;
@@ -305,7 +323,7 @@ void QgsRasterCalcDialog::mExpressionTextEdit_textChanged()
   if ( expressionValid() )
   {
     mExpressionValidLabel->setText( tr( "Expression valid" ) );
-    if ( filePathValid() )
+    if ( filePathValid() || useVirtualProvider() )
     {
       mButtonBox->button( QDialogButtonBox::Ok )->setEnabled( true );
       return;
@@ -320,7 +338,7 @@ void QgsRasterCalcDialog::mExpressionTextEdit_textChanged()
 
 void QgsRasterCalcDialog::setAcceptButtonState()
 {
-  if ( expressionValid() && filePathValid() )
+  if ( ( expressionValid() && filePathValid() ) || ( expressionValid() && useVirtualProvider() ) )
   {
     mButtonBox->button( QDialogButtonBox::Ok )->setEnabled( true );
   }
@@ -330,6 +348,33 @@ void QgsRasterCalcDialog::setAcceptButtonState()
   }
 }
 
+void QgsRasterCalcDialog::setOutputToVirtual()
+{
+  if ( useVirtualProvider() )
+  {
+    mOutputFormatComboBox->hide();
+    mOutputLayer->hide();
+    mOutputLayerLabel->hide();
+    mOutputFormatLabel->hide();
+    mAddResultToProjectCheckBox->isChecked();
+    mAddResultToProjectCheckBox->setEnabled( false );
+    mVirtualLayerLabel->show();
+    mVirtualLayerName->show();
+    setAcceptButtonState();
+  }
+  else
+  {
+    mOutputFormatComboBox->show();
+    mOutputLayer->show();
+    mOutputLayerLabel->show();
+    mOutputFormatLabel->show();
+    mAddResultToProjectCheckBox->setEnabled( true );
+    mVirtualLayerLabel->hide();
+    mVirtualLayerName->hide();
+    setAcceptButtonState();
+  }
+}
+
 bool QgsRasterCalcDialog::expressionValid() const
 {
   QString errorString;
@@ -355,6 +400,8 @@ bool QgsRasterCalcDialog::filePathValid() const
 void QgsRasterCalcDialog::mRasterBandsListWidget_itemDoubleClicked( QListWidgetItem *item )
 {
   mExpressionTextEdit->insertPlainText( quoteBandEntry( item->text() ) );
+  //to enable the "ok" button if someone checks the virtual provider checkbox before adding a valid expression,
+  if ( expressionValid() && useVirtualProvider() ) setAcceptButtonState();
 }
 
 void QgsRasterCalcDialog::mPlusPushButton_clicked()

diff --git a/src/app/qgsrastercalcdialog.h b/src/app/qgsrastercalcdialog.h
@@ -42,6 +42,10 @@ class APP_EXPORT QgsRasterCalcDialog: public QDialog, private Ui::QgsRasterCalcD
     QString outputFormat() const;
     QgsCoordinateReferenceSystem outputCrs() const;
     bool addLayerToProject() const;
+    //! True if Virtual is checked
+    bool useVirtualProvider() const;
+    //! Return the name written in the qlineedit
+    QString virtualLayerName() const;
 
     //! Bounding box for output raster
     QgsRectangle outputRectangle() const;
@@ -64,6 +68,8 @@ class APP_EXPORT QgsRasterCalcDialog: public QDialog, private Ui::QgsRasterCalcD
     void mExpressionTextEdit_textChanged();
     //! Enables OK button if calculator expression is valid and output file path exists
     void setAcceptButtonState();
+    //! Disables some options that are not required if using Virtual Provider
+    void setOutputToVirtual();
     void showHelp();
 
     //calculator buttons