Navigation Menu

Skip to content

Commit

Permalink
[FEATURE][processing] one-to-many join support to the join attribute …
Browse files Browse the repository at this point in the history 
…table alg. (#6499)
  • Loading branch information
@nirvn
nirvn committed Mar 2, 2018
1 parent 2238b42 commit 6f47b25
Show file tree
Hide file tree
Showing 6 changed files with 207 additions and 10 deletions.
4 changes: 2 additions & 2 deletions python/plugins/processing/algs/qgis/SpatialJoin.py
View file
Expand Up @@ -87,8 +87,8 @@ def initAlgorithm(self, config=None):
'crosses': 'crosses'}

self.methods = [
self.tr('Create separate feature for each located feature'),
self.tr('Take attributes of the first located feature only')
self.tr('Create separate feature for each located feature (one-to-one)'),
self.tr('Take attributes of the first located feature only (one-to-many)')
]

self.addParameter(QgsProcessingParameterFeatureSource(self.INPUT,
Expand Down
41 changes: 41 additions & 0 deletions python/plugins/processing/tests/testdata/expected/join_attribute_table_all_match.gfs
View file
@@ -0,0 +1,41 @@
<GMLFeatureClassList>
<GMLFeatureClass>
<Name>join_attribute_table</Name>
<ElementPath>join_attribute_table</ElementPath>
<GeometryType>1</GeometryType>
<SRSName>EPSG:4326</SRSName>
<DatasetSpecificInfo>
<FeatureCount>10</FeatureCount>
<ExtentXMin>0.00000</ExtentXMin>
<ExtentXMax>8.00000</ExtentXMax>
<ExtentYMin>-5.00000</ExtentYMin>
<ExtentYMax>3.00000</ExtentYMax>
</DatasetSpecificInfo>
<PropertyDefn>
<Name>id</Name>
<ElementPath>id</ElementPath>
<Type>Integer</Type>
</PropertyDefn>
<PropertyDefn>
<Name>id2</Name>
<ElementPath>id2</ElementPath>
<Type>Integer</Type>
</PropertyDefn>
<PropertyDefn>
<Name>id_2</Name>
<ElementPath>id_2</ElementPath>
<Type>Integer</Type>
</PropertyDefn>
<PropertyDefn>
<Name>NUM_A</Name>
<ElementPath>NUM_A</ElementPath>
<Type>Real</Type>
</PropertyDefn>
<PropertyDefn>
<Name>ST_A</Name>
<ElementPath>ST_A</ElementPath>
<Type>String</Type>
<Width>8</Width>
</PropertyDefn>
</GMLFeatureClass>
</GMLFeatureClassList>
111 changes: 111 additions & 0 deletions python/plugins/processing/tests/testdata/expected/join_attribute_table_all_match.gml
View file
@@ -0,0 +1,111 @@
<?xml version="1.0" encoding="utf-8" ?>
<ogr:FeatureCollection
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation=""
xmlns:ogr="http://ogr.maptools.org/"
xmlns:gml="http://www.opengis.net/gml">
<gml:boundedBy>
<gml:Box>
<gml:coord><gml:X>0</gml:X><gml:Y>-5</gml:Y></gml:coord>
<gml:coord><gml:X>8</gml:X><gml:Y>3</gml:Y></gml:coord>
</gml:Box>
</gml:boundedBy>

<gml:featureMember>
<ogr:join_attribute_table fid="points.0">
<ogr:geometryProperty><gml:Point srsName="EPSG:4326"><gml:coordinates>1,1</gml:coordinates></gml:Point></ogr:geometryProperty>
<ogr:id>1</ogr:id>
<ogr:id2>2</ogr:id2>
<ogr:id_2>1</ogr:id_2>
<ogr:NUM_A>1.100000</ogr:NUM_A>
<ogr:ST_A>string a</ogr:ST_A>
</ogr:join_attribute_table>
</gml:featureMember>
<gml:featureMember>
<ogr:join_attribute_table fid="points.1">
<ogr:geometryProperty><gml:Point srsName="EPSG:4326"><gml:coordinates>3,3</gml:coordinates></gml:Point></ogr:geometryProperty>
<ogr:id>2</ogr:id>
<ogr:id2>1</ogr:id2>
<ogr:id_2>2</ogr:id_2>
<ogr:NUM_A>2.400000</ogr:NUM_A>
<ogr:ST_A>string c</ogr:ST_A>
</ogr:join_attribute_table>
</gml:featureMember>
<gml:featureMember>
<ogr:join_attribute_table fid="points.1">
<ogr:geometryProperty><gml:Point srsName="EPSG:4326"><gml:coordinates>3,3</gml:coordinates></gml:Point></ogr:geometryProperty>
<ogr:id>2</ogr:id>
<ogr:id2>1</ogr:id2>
<ogr:id_2>2</ogr:id_2>
<ogr:NUM_A>2.200000</ogr:NUM_A>
<ogr:ST_A>string a</ogr:ST_A>
</ogr:join_attribute_table>
</gml:featureMember>
<gml:featureMember>
<ogr:join_attribute_table fid="points.2">
<ogr:geometryProperty><gml:Point srsName="EPSG:4326"><gml:coordinates>2,2</gml:coordinates></gml:Point></ogr:geometryProperty>
<ogr:id>3</ogr:id>
<ogr:id2>0</ogr:id2>
<ogr:id_2>3</ogr:id_2>
<ogr:NUM_A>3.300000</ogr:NUM_A>
<ogr:ST_A>string a</ogr:ST_A>
</ogr:join_attribute_table>
</gml:featureMember>
<gml:featureMember>
<ogr:join_attribute_table fid="points.3">
<ogr:geometryProperty><gml:Point srsName="EPSG:4326"><gml:coordinates>5,2</gml:coordinates></gml:Point></ogr:geometryProperty>
<ogr:id>4</ogr:id>
<ogr:id2>2</ogr:id2>
<ogr:id_2>4</ogr:id_2>
<ogr:NUM_A>4.400000</ogr:NUM_A>
<ogr:ST_A>string b</ogr:ST_A>
</ogr:join_attribute_table>
</gml:featureMember>
<gml:featureMember>
<ogr:join_attribute_table fid="points.4">
<ogr:geometryProperty><gml:Point srsName="EPSG:4326"><gml:coordinates>4,1</gml:coordinates></gml:Point></ogr:geometryProperty>
<ogr:id>5</ogr:id>
<ogr:id2>1</ogr:id2>
<ogr:id_2>5</ogr:id_2>
<ogr:NUM_A>5.500000</ogr:NUM_A>
<ogr:ST_A>string b</ogr:ST_A>
</ogr:join_attribute_table>
</gml:featureMember>
<gml:featureMember>
<ogr:join_attribute_table fid="points.5">
<ogr:geometryProperty><gml:Point srsName="EPSG:4326"><gml:coordinates>0,-5</gml:coordinates></gml:Point></ogr:geometryProperty>
<ogr:id>6</ogr:id>
<ogr:id2>0</ogr:id2>
<ogr:id_2>6</ogr:id_2>
<ogr:NUM_A>6.600000</ogr:NUM_A>
<ogr:ST_A>string b</ogr:ST_A>
</ogr:join_attribute_table>
</gml:featureMember>
<gml:featureMember>
<ogr:join_attribute_table fid="points.6">
<ogr:geometryProperty><gml:Point srsName="EPSG:4326"><gml:coordinates>8,-1</gml:coordinates></gml:Point></ogr:geometryProperty>
<ogr:id>7</ogr:id>
<ogr:id2>0</ogr:id2>
<ogr:id_2>7</ogr:id_2>
<ogr:NUM_A>7.700000</ogr:NUM_A>
<ogr:ST_A>string b</ogr:ST_A>
</ogr:join_attribute_table>
</gml:featureMember>
<gml:featureMember>
<ogr:join_attribute_table fid="points.7">
<ogr:geometryProperty><gml:Point srsName="EPSG:4326"><gml:coordinates>7,-1</gml:coordinates></gml:Point></ogr:geometryProperty>
<ogr:id>8</ogr:id>
<ogr:id2>0</ogr:id2>
<ogr:id_2>8</ogr:id_2>
<ogr:NUM_A>8.800000</ogr:NUM_A>
<ogr:ST_A>string b</ogr:ST_A>
</ogr:join_attribute_table>
</gml:featureMember>
<gml:featureMember>
<ogr:join_attribute_table fid="points.8">
<ogr:geometryProperty><gml:Point srsName="EPSG:4326"><gml:coordinates>0,-1</gml:coordinates></gml:Point></ogr:geometryProperty>
<ogr:id>9</ogr:id>
<ogr:id2>0</ogr:id2>
</ogr:join_attribute_table>
</gml:featureMember>
</ogr:FeatureCollection>
21 changes: 20 additions & 1 deletion python/plugins/processing/tests/testdata/qgis_algorithm_tests.yaml
View file
Expand Up @@ -2772,8 +2772,9 @@ tests:
type: vector

- algorithm: native:joinattributestable
name: join the attribute table by common field
name: join the attribute table by common field (one-to-one)
params:
METHOD: 0
INPUT:
name: points.gml
type: vector
Expand All @@ -2787,9 +2788,27 @@ tests:
name: expected/join_attribute_table.gml
type: vector

- algorithm: native:joinattributestable
name: join the attribute table by common field (one-to-many)
params:
METHOD: 1
INPUT:
name: points.gml
type: vector
INPUT_2:
name: table.dbf
type: table
FIELD: id
FIELD_2: ID
results:
OUTPUT:
name: expected/join_attribute_table_all_match.gml
type: vector

- algorithm: native:joinattributestable
name: Join attributes table with subset of fields
params:
METHOD: 0
FIELD: id
FIELDS_TO_COPY:
- NUM_A
Expand Down
Binary file modified python/plugins/processing/tests/testdata/table.dbf
View file
Binary file not shown.
40 changes: 33 additions & 7 deletions src/analysis/processing/qgsalgorithmjoinbyattribute.cpp
View file
Expand Up @@ -46,6 +46,10 @@ QString QgsJoinByAttributeAlgorithm::groupId() const

void QgsJoinByAttributeAlgorithm::initAlgorithm( const QVariantMap & )
{
QStringList methods;
methods << QObject::tr( "Take attributes of the first matching feature only (one-to-one)" )
<< QObject::tr( "Create separate feature for each matching feature (one-to-many)" );

addParameter( new QgsProcessingParameterFeatureSource( QStringLiteral( "INPUT" ),
QObject::tr( "Input layer" ), QList< int>() << QgsProcessing::TypeVector ) );
addParameter( new QgsProcessingParameterField( QStringLiteral( "FIELD" ),
Expand All @@ -61,6 +65,11 @@ void QgsJoinByAttributeAlgorithm::initAlgorithm( const QVariantMap & )
QVariant(), QStringLiteral( "INPUT_2" ), QgsProcessingParameterField::Any,
true, true ) );

addParameter( new QgsProcessingParameterEnum(
QStringLiteral( "METHOD" ),
QObject::tr( "Join type" ),
methods, false, 0 ) );

addParameter( new QgsProcessingParameterFeatureSink( QStringLiteral( "OUTPUT" ), QObject::tr( "Joined layer" ) ) );
}

Expand All @@ -69,7 +78,8 @@ QString QgsJoinByAttributeAlgorithm::shortHelpString() const
return QObject::tr( "This algorithm takes an input vector layer and creates a new vector layer that is an extended version of the "
"input one, with additional attributes in its attribute table.\n\n"
"The additional attributes and their values are taken from a second vector layer. An attribute is selected "
"in each of them to define the join criteria." );
"in each of them to define the join criteria.\n\n"
"The algorithm will output one feature per matching row(s) from the second vector layer." );
}

QgsJoinByAttributeAlgorithm *QgsJoinByAttributeAlgorithm::createInstance() const
Expand All @@ -79,6 +89,8 @@ QgsJoinByAttributeAlgorithm *QgsJoinByAttributeAlgorithm::createInstance() const

QVariantMap QgsJoinByAttributeAlgorithm::processAlgorithm( const QVariantMap &parameters, QgsProcessingContext &context, QgsProcessingFeedback *feedback )
{
int joinMethod = parameterAsEnum( parameters, QStringLiteral( "METHOD" ), context );

std::unique_ptr< QgsFeatureSource > input( parameterAsSource( parameters, QStringLiteral( "INPUT" ), context ) );
std::unique_ptr< QgsFeatureSource > input2( parameterAsSource( parameters, QStringLiteral( "INPUT_2" ), context ) );
if ( !input || !input2 )
Expand Down Expand Up @@ -129,7 +141,7 @@ QVariantMap QgsJoinByAttributeAlgorithm::processAlgorithm( const QVariantMap &pa


// cache attributes of input2
QHash< QVariant, QgsAttributes > input2AttributeCache;
QMultiHash< QVariant, QgsAttributes > input2AttributeCache;
QgsFeatureIterator features = input2->getFeatures( QgsFeatureRequest().setFlags( QgsFeatureRequest::NoGeometry ).setSubsetOfAttributes( fields2Fetch ) );
double step = input2->featureCount() > 0 ? 50.0 / input2->featureCount() : 1;
int i = 0;
Expand All @@ -144,7 +156,7 @@ QVariantMap QgsJoinByAttributeAlgorithm::processAlgorithm( const QVariantMap &pa

feedback->setProgress( i * step );

if ( input2AttributeCache.contains( feat.attribute( joinField2Index ) ) )
if ( joinMethod == 0 && input2AttributeCache.contains( feat.attribute( joinField2Index ) ) )
continue;

// only keep selected attributes
Expand Down Expand Up @@ -173,10 +185,24 @@ QVariantMap QgsJoinByAttributeAlgorithm::processAlgorithm( const QVariantMap &pa

feedback->setProgress( 50 + i * step );

QgsAttributes attrs = feat.attributes();
attrs.append( input2AttributeCache.value( feat.attribute( joinField1Index ) ) );
feat.setAttributes( attrs );
sink->addFeature( feat, QgsFeatureSink::FastInsert );
if ( input2AttributeCache.count( feat.attribute( joinField1Index ) ) > 0 )
{
QgsAttributes attrs = feat.attributes();

QList< QgsAttributes > attributes = input2AttributeCache.values( feat.attribute( joinField1Index ) );
QList< QgsAttributes >::iterator attrsIt = attributes.begin();
for ( ; attrsIt != attributes.end(); ++attrsIt )
{
QgsAttributes newAttrs = attrs;
newAttrs.append( *attrsIt );
feat.setAttributes( newAttrs );
sink->addFeature( feat, QgsFeatureSink::FastInsert );
}
}
else
{
sink->addFeature( feat, QgsFeatureSink::FastInsert );
}
}

QVariantMap outputs;
Expand Down

4 comments on commit 6f47b25

@anitagraser
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice feature! Seems like the definitions of one-to-one and one-to-many are inconsistent though:

SpatialJoin.py

self.tr('Create separate feature for each located feature (one-to-one)'),
self.tr('Take attributes of the first located feature only (one-to-many)')

qgsalgorithmjoinbyattribute.cpp

QObject::tr( "Take attributes of the first matching feature only (one-to-one)" )
QObject::tr( "Create separate feature for each matching feature (one-to-many)" );

@nirvn
Copy link
Contributor Author

@nirvn nirvn commented on 6f47b25 Mar 2, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@anitagraser , you mean the matching vs. located? I think it needs to be different here, since we're speaking of spatial join vs attribute join. Or am I missing your point? Glad I'm not the only one liking this one 😄 give it a try alongside the aggregate algorithm as part of a model, that made my day.

@anitagraser
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant that in SpatialJoin, "one-to-one" = "create feature for each located/matching" while in JoinByAttribute "one-to-one" is "first matching feature only".

@nirvn
Copy link
Contributor Author

@nirvn nirvn commented on 6f47b25 Mar 2, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@anitagraser , oh, doh, I see it now.

Please sign in to comment.