Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DEV and version r093

D toc

Excerpt

You might encounter problems with how data has been structured or formatted that you must fix prior to providing the content to your target system. You can use the methods in this section to locate problems with the content or data typing of your data. 

Locate mismatched values

When

D s product
rtrue
 evaluates a dataset sample, it interprets the values in a column against its expectations for the values. Based on the column's specified data type and internal pattern matching, values are categorized as valid, mismatched, or missing. These value categories are represented in a slender bar at the top of each column. 

...

Tip

Tip: Before you start performing transformations on your data based on mismatched values, you should verify the data type for these columns to ensure that they are correct. The type against which values are checked is displayed to the upper left of the data quality bar. Below, the data type is ZIP for U.S. Zip code data. For more information, see Supported Data Types.

D caption
typefigure
Mismatched values in red

...

Info

NOTE: Remember that you are working on a sample of your data. If the sample indicator at the top left corner of the Transformer page does not indicate Full Data, then some values in your full dataset may not be represented in the sample displayed in the gridFor small datasets, the Initial Data sample includes all rows of the dataset and is unsampled.

  • From the Transformer page, click the mismatched values in a column's data quality bar to see their count, highlight them in the rows of the data grid, and trigger a set of suggestions for your review.
  • To refine the data grid view, click the Show Only Affected Rows checkbox in the status bar at the bottom of the screen. Only the rows that are affected by the previewed transform are displayed.

    Tip

    Tip: This step highlights specific values that are mismatched. You can take note of individual values.

  • To locate a specific value, click the Filters icon on the right side of the screen. In the Rows tab, enter the specific value to locate. Rows containing this value are highlighted. Back in the data grid, you can select one of these highlighted values to be prompted for suggestions.   

...

Tip

Tip: You can also use the IFMISMATCHED function to test for mismatched values. Unlike the above construction, however, IFMISMATCHED does not support an else clause when the value does match the listed data type. For more information, see IFMISMATCHED Function.

Bad data typing

Tip

Tip: Particularly for dates, data is often easiest to manage as String data type.

D s product
has a number of functions that you can deploy to manage strings. After the data has been properly formatted, you can change it to the proper data type. If you change data type immediately, you may have some challenges in reformatting and augmenting it. Do this step last.

...

Tip

Tip: If possible, you should review and refer to an available schema of your dataset, as generated from the source system. If the data has also been mis-typed in the source system, you should fix it there as well, so any future exports from that system show the correct type.

D s also
inCQLtrue
label((label = "data_type") OR (label = "mismatched") OR (label = "transformation_ui"))