Bug report #21316

Processing: precision loss on decimal number

Added by Tobias Wendorff about 5 years ago. Updated about 5 years ago.

Status:Closed
Priority:Normal
Assignee:Nyall Dawson
Category:Processing/Core
Affected QGIS version:3.5(master) Regression?:No
Operating System:Microsoft Windows 7, 64-bit Easy fix?:No
Pull Request or Patch supplied:No Resolution:
Crashes QGIS or corrupts data:No Copied to github as #:29134

Description

Bug report

1. create a new Temporary Scratch Layer
2. add some fields
name: one
type: decimal number (real)
provider type: double
- length: 32
- precision: 30

name: two
type: decimal number (numeric)
provider type: numeric
- length: 20
- precision: 18

name: three
type: decimal number (decimal)
provider type: decimal
- length: 20
- precision: 18

name: four
type: decimal number (real)
provider type: real
- length: internally set to -1
- precision: internally set to -1

name: five
type: decimal number (double)
provider type: double precision
- length: internally set to -1
- precision: internally set to -1

3. add a new feature
4. add the value "0.01234567890123456789" into each field (length 22, precision 20)
5. save & check results
- one : 0.012345678901234568430878013601
- two : 0.012345678901234567
- three: 0.012345678901234567
- four : 0.012345678901234568
- five : 0.012345678901234568

6. When running "Refactor Fields", "Extract by Attribute" or others in Processing, the result is:
- one : 0.012345678901234568430878013601
- two : 0.012345678901234568
- three: 0.012345678901234568
- four : 0.01235 ["source field" shows, *precision gets smashed to 5*]
- five : 0.01235 ["source field" shows, *precision gets smashed to 5*]

Expected results

The output field should have the same precision as the input field.

Associated revisions

Revision f3cb3488
Added by Nyall Dawson about 5 years ago

Fix loss of precision when converting fields to uri in memory provider

Fixes #21316

Revision e098b9a3
Added by Nyall Dawson about 5 years ago

Fix loss of precision when converting fields to uri in memory provider

Fixes #21316

(cherry picked from commit f3cb3488ec5b1edca298833c6a982fe77b056e59)

History

#1 Updated by Nyall Dawson about 5 years ago

  • Assignee set to Nyall Dawson
  • Status changed from Open to In Progress

#2 Updated by Nyall Dawson about 5 years ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Closed

Also available in: Atom PDF