## Bug report #5153

### Unexpected rounding in field calculator (documentation needed)

Status: | Closed | ||
---|---|---|---|

Priority: | Normal | ||

Assignee: | Nathan Woodrow | ||

Category: | Vectors | ||

Affected QGIS version: | master | Regression?: | No |

Operating System: | Easy fix?: | No | |

Pull Request or Patch supplied: | No | Resolution: | |

Crashes QGIS or corrupts data: | No | Copied to github as #: | 14905 |

**Description**

(possibly related to #3395)

If I do this I get the expected result: `4/2`

If I do this I get the expected result: `0.5`

If I do this the result is 0, regardless of the precision of the output field!: `2/4`

If I do this the results are shown with many decimal places, regardless of the precision of the output field!^: `$x`

^ However, *when the layer is saved* the results are rounded to the expected precision. N.B. this is the same as #3606

**Related issues**

### Associated revisions

[expressions] Make int/int return double results.

Also add a new "//" operator to perform integer division.

(fix #5153)

### History

#### #1 Updated by Alister Hood over 12 years ago

**Category**set to*Vectors*

#### #2 Updated by Martin Dobias over 12 years ago

If both numbers in division are integers, then integer division is used. If any of those numbers is a floating point number (e.g. 2.0/4 or 2/4.0) then real division is used. The same semantics is can be found in PostgreSQL and Sqlite3 (and in python2.x)

For the record, mysql (and Python3.x) always use real division.

#### #3 Updated by Alister Hood over 12 years ago

Do you think it should stay like that?

Things like this should at least be made really obvious to the user...

#### #4 Updated by Martin Dobias over 12 years ago

I am fine with the current behavior.

What do you suggest to make it more obvious to the user?

#### #5 Updated by Alister Hood over 12 years ago

Say the help for this (not currently implemented in Nathan's expression builder) is "a/b : a divided by b", change it to this:

"a/b : a divided by b.

If both numbers are integers, integer division is used, e.g. 1/2=0, but 1/2.0=0.5"

And because people might not look at the help for an operator as simple as /, also create a section in the user manual which lists issues like this where the expression parser behaves differently from what people would expect (unless they have used some other particular software). Even this probably isn't enough.

#### #6 Updated by Alister Hood over 12 years ago

I don't have QGIS here to test, but I'm wondering:

- what happens if you divide two fields with precision 0 by each other? Do you need to do field_1*1.0/field_2 if you don't want integer division?

- if the precision of a field is greater than 0, but the field value is a round number, does the expression parser treat it as a real number or an integer?

- what is the point in integer division? Is it a lot faster? Wouldn't it be a lot more user friendly to always do real division, and if someone wants to round the answer then they can build that into the expression?

#### #7 Updated by Alister Hood over 12 years ago

Alister Hood wrote:

I don't have QGIS here to test, but I'm wondering:

- what happens if you divide two fields with precision 0 by each other? Do you need to do field_1*1.0/field_2 if you don't want integer division?

Yes, you do.

- if the precision of a field is greater than 0, but the field value is a round number, does the expression parser treat it as a real number or an integer?

As a real number, thankfully.

#### #8 Updated by Alister Hood over 12 years ago

If I do this the results are shown with many decimal places, regardless of the precision of the output field!^:

$x^ However, when the layer is saved the results are rounded to the correct precision. N.B. this is the same as #3606

Except it isn't rounded to the correct precision when saved if it is a memory layer.

#### #9 Updated by Martin Dobias over 12 years ago

Alister Hood wrote:

I don't have QGIS here to test, but I'm wondering:

- what happens if you divide two fields with precision 0 by each other? Do you need to do field_1*1.0/field_2 if you don't want integer division?

Or use type conversions: toreal(field_1)/field_2

- if the precision of a field is greater than 0, but the field value is a round number, does the expression parser treat it as a real number or an integer?

Yes. Look into vector layer properties dialog to find out about types of each field.

- what is the point in integer division? Is it a lot faster? Wouldn't it be a lot more user friendly to always do real division, and if someone wants to round the answer then they can build that into the expression?

Nothing related to performance. The point is to have a choice - someone wants real division, someone wants integer division.

#### #10 Updated by Giovanni Manghi about 12 years ago

**Priority**changed from*High*to*Normal*

#### #11 Updated by Paolo Cavallini almost 12 years ago

**Target version**set to*Version 2.0.0*

#### #12 Updated by Alister Hood almost 12 years ago

**Subject**changed from*Problems with incorrect rounding in field calculator*to*Unexpected rounding in field calculator (documentation needed*

#### #13 Updated by Alister Hood almost 12 years ago

**Subject**changed from*Unexpected rounding in field calculator (documentation needed*to*Unexpected rounding in field calculator (documentation needed)*

If I do this the results are shown with many decimal places, regardless of the precision of the output field!^:

$x

#### #14 Updated by Stuart Smith over 11 years ago

Hello, my first QGIS post. May I add my thoughts? I was confused by this situation as well. Spent several hours trying to debug it until I came across this report.

I created a real field with precision = 2. Using the field calculator, I populated this new field by dividing two integer fields (e.g. integer_a / integer_b). I expected real numbers, but the result was either a zero or 1. I assumed that the zero/1 result was a bug/error because the output field was formatted as real.

It sure seems to me that if the output field type is real, then real division ought to be used regardless of the input type. If a user creates a real field, they're expecting "real" results. The integer_a * 1.0 / integer_b * 1.0 workaround is cumbersome and unintuitive.

Thanks!

#### #15 Updated by Nathan Woodrow almost 11 years ago

**Assignee**set to*Nathan Woodrow***Target version**changed from*Version 2.0.0*to*Future Release - High Priority*

#### #16 Updated by Giovanni Manghi over 9 years ago

Nothing related to performance. The point is to have a choice - someone wants real division, someone wants integer division.

I really think that the vast majority if users do expect to have always the real division.

And I just checked also other gis sw, and the real division is always used.

#### #17 Updated by Giovanni Manghi over 9 years ago

my suggestion would be to change the behavior instead of documenting it, is cumbersome for any common user.

#### #18 Updated by Nyall Dawson over 9 years ago

**Status**changed from*Open*to*Closed*

Fixed in changeset 232aaaca66114e69b9ac2b201c65a26e7379f475.