Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DEV and version next

...

This section describes techniques to normalize numeric values in your datasets. Ideally, your source systems are configured to capture and deliver data using a consistent set of units in a standardized structure and format. In practice, data from multiple systems can illuminate differences in the level of precision used in numeric data or differences in text entries that reference the same thing. Within

D s product
rtrue
, you can use the following techniques to address some of the issues you might encounter in the standardization of units and values for numeric types.

Numeric precision

In 

D s product
, mathematical computations are performed using 64-bit floating point operations to 15 decimals of precision. However, due to rounding off, truncation, and other technical factors, small discrepancies in outputs can be expected. Example:

-636074.22
-2465086.34

Suppose you apply the following transformation:

D trans
RawWrangletrue
p03ValueMySum
Typestep
WrangleTextderive type: single value: (-636074.22 + -2465086.34) as: 'MySum'
p01NameFormula type
p01ValueSingle row formula
p02NameFormula
p02Value(-636074.22 + -2465086.34)
p03NameNew column name
SearchTermNew formula

The expected output in the MySum column: -3101160.56

The actual output for in the MySum column: -3101160.5599999996

Info

NOTE: For 64-bit floating point mathematical operations, deviations like the above are intrinsic to the Decimal data type and how the platform performs computations.

Depending on your precision requirements, you can manage precision across your columns using a transformation like the following, which rounds off MySum to three digits:

D trans
RawWrangletrue
Typestep
WrangleTextset col: WM_Week value: round($col, 3)
p01NameColumns
p01ValueMySum
p02NameFormula
p02ValueROUND($col,3)
SearchTermEdit column with formula

For more information on floating point computations, see https://en.wikipedia.org/wiki/Numeric_precision_in_Microsoft_Excel.

Standardize decimal precision

...