Standardize Non-Numeric Values

This section describes techniques to standardize text values in your datasets. You can use the following techniques to address some common issues you might encounter in the standardization of text and other non-numeric values.

Standardize Using Patterns

From the Column Details panel, you can review and select patterns in the column's data. These selections can be used as the basis for converting all applicable values to the selected format.

In the Patterns tab, click a group of patterns and then review the Convert suggestion to define how the pattern matches can be converted to a single standardized format.

• Click Edit to modify the task.
• Click Add to add the task as a step to your recipe.

Example - Phone number patterns

For columns containing phone number data, you can use the Patterns tab to standardize formatting options. Consider the following values, which are valid phone numbers. Next to each value is a pattern representing the value:

\({{digit}{3})\) ({digit}{3})\-({digit}{4})

({digit}{3})\-({digit}{3})\-({digit}{4})

({digit}{3}).({digit}{3}).({digit}{3})

({digit}{3}) ({digit}{3})\-({digit}{4})

1\+{digit}{3}\-{digit}{3}\-{digit}{4}

In the Patterns tab, you can select the patterns to which you would like the other patterns in the same pattern group to be converted. Below, the selected target pattern becomes the pattern to which other patterns in the column values are converted:

Generic conversions

Below are types of conversions that are supported and not supported.

Supported:

Not supported:

Datetime patterns

For columns of Datetime type, the available Convert mappings are based upon the supported date formats in the platform. Standardization of Datetime patterns is a specific implementation. 

Notes on Datetime patterns:

Two-digit years (YY) do not yield four-digit year (YYYY) suggestions due to ambiguity. For example, it is unclear if 50 should map to 1950 or 2050.

For performance reasons, a maximum of two semantic standardizations can be applied at once. Examples:

For more information on supported formats, see Datetime Data Type.

For more information on converting Datetime values to a different format, see DATEFORMAT Function.

Standardize Using Functions

Convert to strings to modify

For manipulation of individual values, it is often easiest to work with the String data type, which is the most flexible. Depending on your approach, you may choose to convert some of your columns into String type:

settype col: col1,col2, col3 type:'String'

For more information, Valid Data Type Strings.

Trim whitespace from text

You can trim out whitespace from an individual column via transform. The TRIM function applied to string values removes the leading and trailing whitespace:

set col: myCol value: TRIM(myCol)

set col: * value: TRIM($col)

• The asterisk (*) is a wildcard, which represents all possible value. In this case, it matches with all column names in the dataset.
  • You may need to move columns or use range values to apply this transform to only non-numeric column types.
• The $col entry denotes a reference to the current column. So for any column to which this transform is applied, the source values are pulled from the column itself and then trimmed.
• See Set Transform.

In some cases, you may wish to remove all spaces, including those in between words or digits, in your strings:

set col:* value:REMOVEWHITESPACE($col)

Standardize case

You can use the following steps to set all text values in a column to be the same case.

Lower case:

set col:myStrings value:LOWER(myStrings)

set col:myStrings value:UPPER(myStrings)

set col:myStrings value:PROPER(myStrings)

Fixed length strings

You can limit the maximum size of a column or set of columns to a fixed string length. For example:

set col:col1,col2 value:IF(LENGTH($col)>32,LEFT($col,32),$col)

Reset types

After modifying non-text values as strings, remember to conver them back to their original types.