https://issues.qgis.org/https://issues.qgis.org/favicon.ico2012-12-30T08:46:48ZQGIS Issue TrackingQGIS Application - Bug report #6483: Qgis crashes when trying to classify huge vector (postgis db) with Natural Breaks(Jenks) https://issues.qgis.org/issues/6483?journal_id=379862012-12-30T08:46:48ZGiovanni Manghigiovanni.manghi@gmail.com
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li></ul> QGIS Application - Bug report #6483: Qgis crashes when trying to classify huge vector (postgis db) with Natural Breaks(Jenks) https://issues.qgis.org/issues/6483?journal_id=426802013-07-08T00:04:52ZVincent Moravincent.mora@oslandia.com
<ul></ul><p>I'd like a dataset to reproduce the bug.</p> QGIS Application - Bug report #6483: Qgis crashes when trying to classify huge vector (postgis db) with Natural Breaks(Jenks) https://issues.qgis.org/issues/6483?journal_id=426952013-07-08T07:51:56ZGiovanni Manghigiovanni.manghi@gmail.com
<ul><li><strong>Status</strong> changed from <i>Open</i> to <i>Feedback</i></li></ul><p>Just tested a 50k polygon layer, double precision column, values between 0 and 1, natural breaks, no crash.</p>
<p>Please test the latest master and if it happens again then attach sample data. Thanks!</p> QGIS Application - Bug report #6483: Qgis crashes when trying to classify huge vector (postgis db) with Natural Breaks(Jenks) https://issues.qgis.org/issues/6483?journal_id=428532013-07-11T03:01:34ZVincent Moravincent.mora@oslandia.com
<ul><li><strong>Assignee</strong> set to <i>Vincent Mora</i></li></ul> QGIS Application - Bug report #6483: Qgis crashes when trying to classify huge vector (postgis db) with Natural Breaks(Jenks) https://issues.qgis.org/issues/6483?journal_id=428552013-07-11T03:58:40ZVincent Moravincent.mora@oslandia.com
<ul><li><strong>File</strong> <a href="/attachments/download/5972/test_qgsissue6483.py">test_qgsissue6483.py</a><a href="/attachments/5972/test_qgsissue6483.py"><img alt="Magnifier" src="/images/magnifier.png" /></a> added</li></ul><p>Created a SL DB with provided script.</p>
<p>In qgis (today's master 56210eb0b5c), <br />- Add SpatiaLite Layer<br />- Properties->Style->Graduated and Mode->Natural Breaks (Jenks) OK</p>
<p>No problem.</p> QGIS Application - Bug report #6483: Qgis crashes when trying to classify huge vector (postgis db) with Natural Breaks(Jenks) https://issues.qgis.org/issues/6483?journal_id=428632013-07-11T05:57:42ZGiovanni Manghigiovanni.manghi@gmail.com
<ul></ul><p>Vincent Mora wrote:</p>
<blockquote>
<p>Created a SL DB with provided script.</p>
<p>In qgis (today's master 56210eb0b5c), <br />- Add SpatiaLite Layer<br />- Properties->Style->Graduated and Mode->Natural Breaks (Jenks) OK</p>
<p>No problem.</p>
</blockquote>
<p>you have to test with a very large vector. I tested with a 3 million record one, and it freezes with natural breaks, while it is ok with equal interval (don't know about the other modes). Not sure what is the number of records that start to be trouble.</p> QGIS Application - Bug report #6483: Qgis crashes when trying to classify huge vector (postgis db) with Natural Breaks(Jenks) https://issues.qgis.org/issues/6483?journal_id=428702013-07-11T06:34:24ZVincent Moravincent.mora@oslandia.com
<ul><li><strong>Pull Request or Patch supplied</strong> changed from <i>No</i> to <i>Yes</i></li></ul><p><a class="external" href="https://github.com/qgis/Quantum-GIS/pull/717">https://github.com/qgis/Quantum-GIS/pull/717</a></p> QGIS Application - Bug report #6483: Qgis crashes when trying to classify huge vector (postgis db) with Natural Breaks(Jenks) https://issues.qgis.org/issues/6483?journal_id=428722013-07-11T06:36:47ZGiovanni Manghigiovanni.manghi@gmail.com
<ul></ul><p>Vincent Mora wrote:</p>
<blockquote>
<p><a class="external" href="https://github.com/qgis/Quantum-GIS/pull/717">https://github.com/qgis/Quantum-GIS/pull/717</a></p>
</blockquote>
<p>this patch does not seems related to this ticket :)</p> QGIS Application - Bug report #6483: Qgis crashes when trying to classify huge vector (postgis db) with Natural Breaks(Jenks) https://issues.qgis.org/issues/6483?journal_id=428742013-07-11T06:40:39ZVincent Moravincent.mora@oslandia.com
<ul><li><strong>Pull Request or Patch supplied</strong> changed from <i>Yes</i> to <i>No</i></li></ul><p>Sorry for that, it was for <a class="issue tracker-1 status-5 priority-10 priority- closed" href="https://issues.qgis.org/issues/7244" title="Split multipolygon geometry fails (Closed)">#7244</a>. Working on this one right now.</p> QGIS Application - Bug report #6483: Qgis crashes when trying to classify huge vector (postgis db) with Natural Breaks(Jenks) https://issues.qgis.org/issues/6483?journal_id=428772013-07-11T07:31:31ZVincent Moravincent.mora@oslandia.com
<ul></ul><p>Giovanni: <strong>what do you mean by "freezes"</strong> ? Is it still computing or crashes ?</p>
<p>I'm trying with a few millions polygons (qgis compiled in debug), it hangs (computing) but no crash yet (15min in the process).</p>
<p>Apparently this classification (Jenks) is an iterative optimization process... The <strong>complexity is k x n^2</strong>, where k is the nb of classes (the default is 5) and <strong>n is the number of records</strong> (<a class="external" href="http://wiki.objectvision.nl/index.php/Fisher%27s_Natural_Breaks_Classification">http://wiki.objectvision.nl/index.php/Fisher%27s_Natural_Breaks_Classification</a>).</p>
<p>If it crashed it's a problem, but classifying 2M records with this method can surely take a long time. I'll give qgis a chance to compute a result before tomorrow morning, but I'm not really holding my breath.</p> QGIS Application - Bug report #6483: Qgis crashes when trying to classify huge vector (postgis db) with Natural Breaks(Jenks) https://issues.qgis.org/issues/6483?journal_id=428802013-07-11T08:35:50ZGiovanni Manghigiovanni.manghi@gmail.com
<ul></ul><p>Vincent Mora wrote:</p>
<blockquote>
<p>Giovanni: <strong>what do you mean by "freezes"</strong> ? Is it still computing or crashes ?</p>
</blockquote>
<p>freezes means that qgis become not responsive for a long time, and killing the program is the only exit</p>
<blockquote>
<p>I'm trying with a few millions polygons (qgis compiled in debug), it hangs (computing) but no crash yet (15min in the process).<br />Apparently this classification (Jenks) is an iterative optimization process... The <strong>complexity is k x n^2</strong>, where k is the nb of classes (the default is 5) and <strong>n is the number of records</strong> (<a class="external" href="http://wiki.objectvision.nl/index.php/Fisher%27s_Natural_Breaks_Classification">http://wiki.objectvision.nl/index.php/Fisher%27s_Natural_Breaks_Classification</a>).</p>
<p>If it crashed it's a problem, but classifying 2M records with this method can surely take a long time. I'll give qgis a chance to compute a result before tomorrow morning, but I'm not really holding my breath.</p>
</blockquote>
<p>My 2 cents:</p>
<p>we need to understand what is the (more or less) number of records that makes qgis freeze.</p>
<p>These days datasets of >100k records are pretty usual, so I would say that at least up to 200/300k records the classification should work. For more records than the problematic limit a warning should be issued before computing the classes, so it would be a user decision to wait a lot of time, or not.</p> QGIS Application - Bug report #6483: Qgis crashes when trying to classify huge vector (postgis db) with Natural Breaks(Jenks) https://issues.qgis.org/issues/6483?journal_id=428902013-07-11T11:24:31ZVincent Moravincent.mora@oslandia.com
<ul></ul><p>It took a couple hours (more or less, I wasn't waiting in from of the machine) but my 2M records where eventually sorted out, <strong>no crash</strong>. But it surely wasn't responsive for this time.</p>
<p>Giovanni, I agree with you: an acceptable solution would be to pop-up a warning window with a time estimate which should be a function of the nb of records (<em>a priori</em> n²).</p>
<p>I can try a few datasets with increasing nb of records tomorrow to try and model this processing time in order to validate the n² hypotheses.</p>
<p>Of course processing time will depend of your machines config, and maybe of your record length (not sure). So if we want an accurate enough estimate we may need to pre-process a subset of the data before popping-up the warning (and use the model to extrapolate for the complete set).</p>
<p>Another solution (simpler I guess) is to use the size of the dataset and modify the Natural Breaks entry of the drop-down-list accordingly:<br />- long processing for more than 10k records, <br />- very long for more than 100k, <br />- do that before lunch break for more than 1M, <br />- do that before going home for more than 10M,<br />- don't think about it, consider yourself lucky you got this far for 100M and above.</p>
<p>Of course, since I only tried in debug, I may be completely off with the dataset sizes, but I guess you got the idea :)</p> QGIS Application - Bug report #6483: Qgis crashes when trying to classify huge vector (postgis db) with Natural Breaks(Jenks) https://issues.qgis.org/issues/6483?journal_id=429012013-07-11T23:08:31ZVincent Moravincent.mora@oslandia.com
<ul><li><strong>File</strong> <a href="/attachments/download/5984/timing_jenks.pdf">timing_jenks.pdf</a> added</li></ul><p>So it's n² complexity all right ! (see the attached plot).</p>
<p>I'm going to try and implement the proposed warning.</p> QGIS Application - Bug report #6483: Qgis crashes when trying to classify huge vector (postgis db) with Natural Breaks(Jenks) https://issues.qgis.org/issues/6483?journal_id=429032013-07-12T00:06:11ZVincent Moravincent.mora@oslandia.com
<ul><li><strong>Pull Request or Patch supplied</strong> changed from <i>No</i> to <i>Yes</i></li></ul><p><a class="external" href="https://github.com/qgis/Quantum-GIS/pull/720">https://github.com/qgis/Quantum-GIS/pull/720</a></p> QGIS Application - Bug report #6483: Qgis crashes when trying to classify huge vector (postgis db) with Natural Breaks(Jenks) https://issues.qgis.org/issues/6483?journal_id=429052013-07-12T00:52:30ZVincent Moravincent.mora@oslandia.com
<ul><li><strong>Crashes QGIS or corrupts data</strong> changed from <i>Yes</i> to <i>No</i></li></ul><p>BTW, since a crash was observed with a <strong>postgis DB</strong>, I just tryied with 80k and 640k datasets, for the latter it sure takes a few minutes, but it classifies ok, <strong>no crash</strong>.</p> QGIS Application - Bug report #6483: Qgis crashes when trying to classify huge vector (postgis db) with Natural Breaks(Jenks) https://issues.qgis.org/issues/6483?journal_id=432212013-07-16T01:18:19ZHugo Mercierhugo.mercier@oslandia.com
<ul><li><strong>Resolution</strong> set to <i>fixed</i></li><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li></ul> QGIS Application - Bug report #6483: Qgis crashes when trying to classify huge vector (postgis db) with Natural Breaks(Jenks) https://issues.qgis.org/issues/6483?journal_id=432782013-07-17T02:10:48ZGiovanni Manghigiovanni.manghi@gmail.com
<ul><li><strong>Resolution</strong> deleted (<del><i>fixed</i></del>)</li><li><strong>Status</strong> changed from <i>Resolved</i> to <i>Open</i></li></ul><p>Vincent Mora wrote:</p>
<blockquote>
<p>BTW, since a crash was observed with a <strong>postgis DB</strong>, I just tryied with 80k and 640k datasets, for the latter it sure takes a few minutes, but it classifies ok, <strong>no crash</strong>.</p>
</blockquote>
<p>The patch that solves this has not yet been committed</p>
<p><a class="external" href="https://github.com/qgis/Quantum-GIS/pull/720">https://github.com/qgis/Quantum-GIS/pull/720</a></p>
<p>re-close if I'm wrong :)</p> QGIS Application - Bug report #6483: Qgis crashes when trying to classify huge vector (postgis db) with Natural Breaks(Jenks) https://issues.qgis.org/issues/6483?journal_id=432902013-07-17T02:39:20ZHugo Mercierhugo.mercier@oslandia.com
<ul></ul><p>The patch has not yet been committed, you're right.<br />I thought "resolved" represented a state where a patch had been supplied (a pull request here), but waiting for commiters to accept it or discuss it.</p>
<p>The strict workflow is not very clear for me. Can we agree on something like <a class="external" href="https://issues.qgis.org/wiki/quantum-gis/Tickets_workflow">https://issues.qgis.org/wiki/quantum-gis/Tickets_workflow</a> ?</p> QGIS Application - Bug report #6483: Qgis crashes when trying to classify huge vector (postgis db) with Natural Breaks(Jenks) https://issues.qgis.org/issues/6483?journal_id=432942013-07-17T05:48:21ZGiovanni Manghigiovanni.manghi@gmail.com
<ul><li><strong>Status</strong> changed from <i>Open</i> to <i>Resolved</i></li><li><strong>Resolution</strong> set to <i>fixed</i></li></ul><p>Hugo Mercier wrote:</p>
<blockquote>
<p>The patch has not yet been committed, you're right.<br />I thought "resolved" represented a state where a patch had been supplied (a pull request here), but waiting for commiters to accept it or discuss it.</p>
<p>The strict workflow is not very clear for me. Can we agree on something like <a class="external" href="https://issues.qgis.org/wiki/quantum-gis/Tickets_workflow">https://issues.qgis.org/wiki/quantum-gis/Tickets_workflow</a> ?</p>
</blockquote>
<p>Hi Hugo, that's fine for me.</p>
<p>We can eventually discuss how to simplify the workflow at the next dev meeting.</p> QGIS Application - Bug report #6483: Qgis crashes when trying to classify huge vector (postgis db) with Natural Breaks(Jenks) https://issues.qgis.org/issues/6483?journal_id=447072013-09-14T22:18:17ZTim Suttontim@linfiniti.com
<ul><li><strong>Status</strong> changed from <i>Resolved</i> to <i>Closed</i></li></ul><p>Fixed in changeset <a class="changeset" href="https://issues.qgis.org/projects/qgis/repository/revisions/7e171ef7b0eec2101c60671e7625cfea52688a7e" title="Merge pull request #720 from Oslandia/issue6483 Fix #6483 - Thanks Vincent!">7e171ef7b0eec2101c60671e7625cfea52688a7e</a>.</p> QGIS Application - Bug report #6483: Qgis crashes when trying to classify huge vector (postgis db) with Natural Breaks(Jenks) https://issues.qgis.org/issues/6483?journal_id=447102013-09-15T00:10:37ZVincent Moravincent.mora@oslandia.com
<ul></ul><p>Bug fixed thanks to fundings from Agence de l'Eau Adour-Garonne.</p>