Bug report #13279

Processing & SAGA

Added by Paolo Cavallini almost 5 years ago. Updated almost 4 years ago.

Status:Closed
Priority:High
Assignee:Victor Olaya
Category:Processing/SAGA
Affected QGIS version:master Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:No Resolution:
Crashes QGIS or corrupts data:No Copied to github as #:21339

Description

SAGA 2.2.0 is currently on Debian (testing and sid). As a result, a number of commend do not work, e.g.:

2015-08-28T14:22:11    0    SAGA execution commands
            io_gdal -TRANSFORM -INTERPOL 0 -GRIDS "/tmp/processing/c97eff6cecfa4d57ab70373762af8c64/dtm.sgrd" -FILES "/home/paolo/Documents/Didattica/Corsi_Formazione_Faunalia/QGIS/QGIS_data/dtm.tif" 
            ta_morphometry "Slope, Aspect, Curvature" -ELEVATION "/tmp/processing/c97eff6cecfa4d57ab70373762af8c64/dtm.sgrd" -METHOD 6 -UNIT_SLOPE 1 -UNIT_ASPECT 1 -SLOPE "/tmp/processing/603f7f0a7b254a94830e2b22f2f3ee17/SLOPE.sdat" -ASPECT "/tmp/processing/90fd8d1c501748d1bb8b5a097ca8063b/ASPECT.sdat" -C_GENE "/tmp/processing/b9631b34c537485f8cfda97691c8510c/CGENE.sdat" -C_PLAN "/tmp/processing/254e029f993f4a6aa22607ab48d613da/CPLAN.sdat" -C_PROF "/tmp/processing/74d209817d3f421991325581eaa4f530/CPROF.sdat" -C_TANG "/tmp/processing/b9843ee4dbeb4df4a79338363be582b0/CTANG.sdat" -C_LONG "/tmp/processing/38ff7d18c92743d3af99c53ec29a2b61/CLONG.sdat" -C_CROS "/tmp/processing/b1efc31b658742a19640e67f7b47d84a/CCROS.sdat" -C_MINI "/tmp/processing/3b0079a261934cd9a30d1e4f764a67ec/CMINI.sdat" -C_MAXI "/tmp/processing/347e1e2bd02744419fa4a8f7e8c16d02/CMAXI.sdat" -C_TOTA "/tmp/processing/683095d37fb34739b1957c09781e5fdc/CTOTA.sdat" -C_ROTO "/tmp/processing/dd85022b267b43cdac433ae9f11be8fe/CROTO.sdat" 
2015-08-28T14:22:12    0    Risultato della console di esecuzione di SAGA
            Error: select a tool
            ____________________________

            #####   ##   #####    ##
            ###     ###  ##       ###
            ###   # ## ##  #### # ##
            ### ##### ##    # #####
            ##### #   ##  ##### #   ##
            ____________________________

            SAGA Version: 2.2.0

            tools:
            0    - GDAL: Import Raster
            1    - GDAL: Export Raster
            2    - GDAL: Export Raster to GeoTIFF
            3    - OGR: Import Vector Data
            4    - OGR: Export Vector Data
            5    - OGR: Export Shapes to KML
            6    - GDAL: Import NetCDF

            type -h or --help for further information

            Error: Grid file could not be opened.
            Error: input file [/tmp/processing/c97eff6cecfa4d57ab70373762af8c64/dtm.sgrd]
            Error: Elevation

saga_diff_2_2_0_vs_2_2_2.txt Magnifier (242 KB) Paolo Cavallini, 2015-10-19 02:14 AM

algdiffs.txt Magnifier (19.6 KB) Victor Olaya, 2015-10-19 11:13 PM


Related issues

Related to QGIS Application - Bug report #14300: SAGA 2.2.3 "Create Graticule" needs a parameter changed Closed 2016-02-15
Duplicated by QGIS Application - Bug report #14735: SagaAlgorithm since Saga 2.2.4: RESAMPLING instead of INT... Rejected 2016-04-27

Associated revisions

Revision a9dab06a
Added by Victor Olaya almost 5 years ago

[processing]fixed SAGA when using a version newer than the latest supported one

fixes #13279

History

#1 Updated by Giovanni Manghi almost 5 years ago

  • Status changed from Open to Feedback

On Windows the above 2.2.0 module works fine. My test under Linux is inconclusive because SAGA 2.2.0 io_gdal seg faults on Ubuntu.

Do you have any other evidence that already existing modules have parameters names changed?

#2 Updated by Paolo Cavallini almost 5 years ago

Tried a few more tools, I always get:

Error: select a tool

Chengelog is here:

http://sourceforge.net/p/saga-gis/wiki/Changelog%202.2.0/

#3 Updated by Paolo Cavallini almost 5 years ago

Here one can find the diff in interfaces:

SAGA 2.2.0

https://sourceforge.net/p/saga-gis/wiki/Changelog%202.2.0/attachment/saga_cmd_interface_v2_2_0.txt

SAGA 2.1.4

https://sourceforge.net/p/saga-gis/wiki/Changelog%202.1.4/attachment/saga_cmd_interface_v2_1_4.txt

So one can simply do a diff between both files to see the changes. This also works with files created for older SAGA versions, e.g. in case you have skipped SAGA 2.1.4.

Thanks Volker for pointing it out.

#4 Updated by Giovanni Manghi almost 5 years ago

So one can simply do a diff between both files to see the changes.

there are a few changes that will need to be addressed, but the above module (ta_morphometry "Slope, Aspect, Curvature") is identical in 2.1.4 and 2.2.0. As you get the same error for many other modules and as it generally seems to work (tested just on Windows for now, but haven't found yet a 2.2 module that does not work with the 2.1.4 descriptions) it seems anyway that you may have a local issue.

#5 Updated by Paolo Cavallini almost 5 years ago

Agreed, thanks for pointing this out.
My system is all from standard packages, a local issue is unlikely, or anyway unexpected.
Does anyone have the same?

#6 Updated by Paolo Cavallini almost 5 years ago

The options:

`io_gdal -TRANSFORM `

should be replaced by:

`io_gdal 0 -TRANSFORM 1`

so the import does not fail.
Thanks Giovanni for spotting this.

#7 Updated by Paolo Cavallini almost 5 years ago

  • Tag set to easy

#8 Updated by Paolo Cavallini almost 5 years ago

Basic problem fixed with 72665e0. More comprehensive solution needed.

#9 Updated by Victor Olaya almost 5 years ago

If this patch fixes the problem and 2.2.0 works fine, i think this can be closed, considering that we have 2.2.0 now as a supported version in the code

#10 Updated by Paolo Cavallini almost 5 years ago

Works well here, I leave it open for further feedback.

#11 Updated by Jeff G almost 5 years ago

I applied this patch on my 2.8.1 QGIS installation on Linux Mint, using SAGA 2.2.0. It solved the problem for me. (Thank you Paolo!)

#12 Updated by Paolo Cavallini almost 5 years ago

Thank Victor for this.
Unfortunately now (saga 2.2.1) I can see only one (!) saga alg. From the log:

...
2015-09-30T21:19:25 2 Could not open SAGA algorithm:
/usr/share/qgis/python/plugins/processing/algs/saga/description/2.1.4/FragmentationClassesfromDensityandConnectivity.txt
'2.2.1'
2015-09-30T21:19:25 2 Could not open SAGA algorithm:
/usr/share/qgis/python/plugins/processing/algs/saga/description/2.1.4/FlatDetection.txt
'2.2.1'

#13 Updated by Victor Olaya almost 5 years ago

  • Status changed from Feedback to Closed

#14 Updated by Paolo Cavallini almost 5 years ago

  • Status changed from Closed to Reopened
  • Affected QGIS version changed from 2.10.1 to master

2.2.2 is out. Have to check whether there is some API changes.
Cannot find clear info here:
http://sourceforge.net/p/saga-gis/wiki/SAGA%20Release%20Information/

#15 Updated by Paolo Cavallini almost 5 years ago

Diff in command syntax between 2.2.0 and 2.2.2 attached here.

#16 Updated by Paolo Cavallini almost 5 years ago

Steps to reproduce also for other versions:

wget http://sourceforge.net/p/saga-gis/wiki/Changelog%202.2.2/attachment/saga_cmd_interface_v2_2_2.txt

wget http://sourceforge.net/p/saga-gis/wiki/Changelog%202.2.0/attachment/saga_cmd_interface_v2_2_0.txt

diff saga_cmd_interface_v2_2_2.txt saga_cmd_interface_v2_2_0.txt > saga_diff_2_2_0_vs_2_2_2.txt

#17 Updated by Paolo Cavallini almost 5 years ago

  • Subject changed from Please upgrade Processing to SAGA 2.2.0 to Please upgrade Processing to SAGA 2.2.x

#18 Updated by Victor Olaya almost 5 years ago

Attached you can find a processed version of the diff file, showing only those algorithms with differences

I added a new folder with 2.2.2 descriptions in the QGIS repo, so if anyone wants to start applying those changes, now you can. I suggest leaving this issue open until all changes have been applied

Here's is the scritp that i used, just for the record. I added some manual editing in the end to separate new algs from modified ones

with open("diff.txt") as f:
    lines = f.readlines()
lines = [line for line in lines if "Usage" in line]

for old, new in zip(lines,lines[1:])[::2]:
    old = old.replace("[","").replace("]","")[1:]
    new = new.replace("[","").replace("]","")[1:]
    if old != new:
        print "---\
%s%s" %(old,new)    

Thanks to the SAGA crew for making this so hard to mantain!

#19 Updated by Paolo Cavallini over 4 years ago

  • Status changed from Reopened to Feedback

Seems fixed now, right?

#20 Updated by Alexander Bruy over 4 years ago

No, thanks to the SAGA devs. In SAGA 2.2.3 there are some commands with changed parameters, see #14300

#21 Updated by Giovanni Manghi over 4 years ago

I still find modules description for 2.2.0 (and by consequence on 2.2.2 and 2.2.3) that are wrong (just made a pull request for "resampling") because they were not changed when the files were copied from 2.1.*

Is there a way we can check what is the actual status in master?

#22 Updated by Paolo Cavallini over 4 years ago

Thanks a lot Giovanni for this work!
In the meantime, saga 2.2.4 is out, so probably there will be more to fix.

#23 Updated by Giovanni Manghi over 4 years ago

Paolo Cavallini wrote:

In the meantime, saga 2.2.4 is out, so probably there will be more to fix.

this is getting ridiculous, we should really stick to one SAGA version and eventually update only for LTR releases.

Meanwhile I would really appreciate if someone is able to easily check what

  • modules we still miss to add in 2.2.0 and 2.2.2 (and now 2.2.4). Yesterday I realized we were missing 5 tools from the shapes_polygons group. I already added the new ones in 2.2.3.
  • modules in 2.2.* that we still need to update (since we copied the description files from 2.1.*, and is not unusual still to find broken tools, see the above comment about "resampling").

#24 Updated by Paolo Cavallini over 4 years ago

Agreed, it's really a PITA. Unfortunately, the upgrade cycle for Linux distros at least is outside our control.

#25 Updated by Giovanni Manghi over 4 years ago

Paolo Cavallini wrote:

Agreed, it's really a PITA. Unfortunately, the upgrade cycle for Linux distros at least is outside our control.

Honestly as a Linux user I don't care about this specific problem. For example Linux users have virtually disappeared from trainings, and I would prefer much more to be able to rely 100% on a specific version of SAGA on Windows (that is what the vast majority of users use). Linux users can always be instructed on how to get/compile a specific version of SAGA.

#26 Updated by Paolo Cavallini over 4 years ago

This would be discriminatory, and obviously against our policy.
One possible solution would be to embed a specific saga version, to be merged from upstream when appropriate.
The whole issue should be discussed in the ML or as a QEP however.

#27 Updated by Giovanni Manghi over 4 years ago

Paolo Cavallini wrote:

This would be discriminatory, and obviously against our policy.

it would be a practical decision, not a discriminatory one.
Because otherwise among ~240 (and growing) tools we will have always a good chunk broken, and shipping at each release broken tools is admittedly very bad for the users and qgis image.

If the project want to task me to keep the descriptions files up to date at each saga release I'm available.

One possible solution would be to embed a specific saga version, to be merged from upstream when appropriate.

this would work for me.

#28 Updated by Alexander Bruy over 4 years ago

If SAGA devs don't care about their users its their problem, not ours. Maybe users will move to another tools, e.g. GRASS and OTB which are much more stable and SAGA will die in neareast future :-).

I think that we should not add support for new SAGA versions without real need. Our priority should be average users, not early adopters. Maybe I'm wrong, but new SAGA version appears in repositories, espesially stable repositories, with delay, not in the same day as it was released. So there is no harm in providing support only for version from stable repositories. We already do this with lot of dependencies, e.g. minimum supported Qt version is 4.8.0 not latest 4.8.6 and so on. Why not use same approach with SAGA? Declare support for version, available for example in Debian Stable (or another repo, we use as minimal supported version), and make sure it works. All other versions may be supported but without any warranty.

Also Victor and me discussed SAGA support recently. Maybe we can implement something like "real-time" description generation for SAGA algs using its Python API. But this will work only for some algs, because many single SAGA algs should be splitted into two or more Processing algs and also there are some differences in parameters handling (e.g. extent passed to SAGA differently). So such automated generator is not a "silver bullet", a lot of manual work will be needed. So not much difference with current situation.

#29 Updated by Giovanni Manghi over 4 years ago

I think that we should not add support for new SAGA versions without real need. Our priority should be average users, not early adopters. Maybe I'm wrong, but new SAGA version appears in repositories, especially stable repositories, with delay, not in the same day as it was released. So there is no harm in providing support only for version from stable repositories.

agree

#30 Updated by Giovanni Manghi over 4 years ago

Can we close this ticket?

#31 Updated by Paolo Cavallini over 4 years ago

IMHO making SAGA usable for all is still an open issue, although the solution may not be a simple one.
Please keep it open until we have a proper solution.

#32 Updated by Giovanni Manghi over 4 years ago

Paolo Cavallini wrote:

IMHO making SAGA usable for all is still an open issue, although the solution may not be a simple one.
Please keep it open until we have a proper solution.

what this has to do with the description/subject of this ticket?

#33 Updated by Giovanni Manghi about 4 years ago

  • Priority changed from Severe/Regression to High
  • Subject changed from Please upgrade Processing to SAGA 2.2.x to Processing & SAGA
  • Status changed from Feedback to Open
  • Tag deleted (easy)

This ticket is not a matter anymore of updating SAGA to 2.2.2 (done) or whatever else SAGA version. Now is about what to do with SAGA in QGIS: recently a lot of new SAGA releases have been published (6 releases since october) and they not only contain new algorithms, that is fine, but also changes in parameter names of existing tools (like recently a fundamental parameter in io_gdal) that is not so cool (broken scripts and models the first side effect).

#34 Updated by Alexander Bruy about 4 years ago

According to https://sourceforge.net/p/saga-gis/mailman/message/35146184/ SAGA 2.3 will support reading data via GDAL.

#35 Updated by Alexander Bruy almost 4 years ago

  • Status changed from Open to Closed

Closed as not relevant.

Also available in: Atom PDF