...
This section describes techniques to normalize numeric values in your datasets. Ideally, your source systems are configured to capture and deliver data using a consistent set of units in a standardized structure and format. In practice, data from multiple systems can illuminate differences in the level of precision used in numeric data or differences in text entries that reference the same thing. Within
| D s product | ||
|---|---|---|
|
Numeric precision
In
| D s product |
|---|
| -636074.22 |
| -2465086.34 |
Suppose you apply the following transformation:
D trans RawWrangle true p03Value MySum Type step WrangleText derive type: single value: (-636074.22 + -2465086.34) as: 'MySum' p01Name Formula type p01Value Single row formula p02Name Formula p02Value (-636074.22 + -2465086.34) p03Name New column name SearchTerm New formula
The expected output in the MySum column: -3101160.56
The actual output for in the MySum column: -3101160.5599999996
| Info |
|---|
NOTE: For 64-bit floating point mathematical operations, deviations like the above are intrinsic to the Decimal data type and how the platform performs computations. |
Depending on your precision requirements, you can manage precision across your columns using a transformation like the following, which rounds off MySum to three digits:
| D trans | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
For more information on floating point computations, see https://en.wikipedia.org/wiki/Numeric_precision_in_Microsoft_Excel.
Standardize decimal precision
...
