Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DEV and version next

D toc

Excerpt

As part of your data cleansing steps, you might need to remove duplicate rows of data from your dataset.

Validate Duplicate Data

In some cases, it might be acceptable to have duplicated data. For example, additional records using the same primary key might be included in a dataset as amendments or detail records.

...

D trans
Typestep
p01NameFormula type
p01ValueSingle row formula
p02NameFormula
p02ValueLOWER(Description)
SearchTermNew formula
For more information, see Normalize Numeric Values.

Deduplicate Rows Based on a Primary Key

...

In the generated column, values that are true indicate duplicate data. If all values are true, then you can remove one of the columns.

D s also
inCQLtrue
label(label = "deduplicate")