View Source

In the Column Details panel, you can review additional details about a column of your dataset. Select Column Details from any column menu or the Action menu in the column browser.

Tip: Use the Column Details panel to explore values in an individual column, when the context of the value is not important for your current exploration. For example, you can identify outlier values for the column or compare the number of unique values to number of rows to determine whether the column could be a key value.

Overview tab

Release 8.7.1 > Column Details Panel > ColumnDetailsPanel-OverviewTab.png

Column Details panel - Overview tab

Column statistics:

You can use this view to review basic counts and percentages of the values in the currently selected column. In addition to basic computations on valid, mismatched, and missing values, you can see breakdowns for the most frequent values and outlier values.

Depending on the data type of the column, additional statistics provide information on data quality and variation. For more information, see Column Statistics Reference.

Actions:

To change the data type, click the type indicator next to the column title in the Column Details panel.
To perform commands on the column, select from the drop-down next to the column title. For more information, see Column Menus.
Use the data quality bar to select categories of values: valid, mismatched, or missing. The context panel is updated based on your selection with recommended recipe steps. See Selection Details Panel.

Patterns tab

In the Patterns tab, you can review patterns identified by the platform in the selected column's data and then create steps based on patterns that you select. Pattern profiling automatically finds and groups clusters of the column's values based on similarities in format and structure, such as differently formatted phone numbers, addresses, log entries, and name fields. For example, if some of your dataset's address values include apartment numbers, you can create a Split transformation based on a pattern that includes the apartment numbers.

NOTE: In this tab, the count of values and the all patterns category do not include missing values.

NOTE: Wide columns, such as Arrays, Objects, or freeform text, might take a while to profile.

Each non-blank value in the column is represented by one of the displayed patterns. Patterns are specified as a combination of literal values and . For more information on these patterns, see Text Matching.
Patterns might be more generalized than the constraints of the column's data type.
Token values are without braces.

Release 8.7.1 > Column Details Panel > ColumnDetailsPanel-PatternsTab.png

Column Details panel - Patterns tab

All non-blank values are captured in the all patterns category, which you can expand to display the patterns that capture subsets of all values. Patterns are displayed in a tree structure, with each lower level describing a subset of the parent pattern.

Tip: Hover over a pattern or sub-pattern to see the affected values in the example data beneath it.

Tip: When you select a pattern group, you may be presented with suggestions for standardizing the values in the column to a single format. In some cases, you might want to remove unnecessary data first. For example, standardization of phone numbers is easier if any +1 country codes are removed from the beginning of values.

Tip: Pattern suggestions are created based on the first few thousand rows of data in your sample. For best results, you should generate a random sample with a representative set of patterns in the first rows in the column.

Below the top level, patterns are displayed in order of decreasing frequency in the column, allowing you to choose the level of granularity for which you wish to address data issues in the column. For each pattern, you can review the counts of values matching the pattern.

In the above example, all values that have been identified as matching the url are contained in the first category.

Select a pattern to trigger a set of suggestion cards to apply to the represented data.
- When you select values from a pattern's histogram, all suggestions match the pattern. You cannot select the values that do not match the pattern from the histogram.
- For more information, see Explore Suggestions.
Select a token within a pattern or a highlighted block of text among the example values to trigger suggestion cards that apply the token within the pattern.
You can modify the selected suggestion in the Transform Builder. See Transform Builder.
- When you apply the transformation to your recipe, the Patterns tab is updated automatically.
  Tip: When you see a pattern that you wish to reuse, select the pattern and one of its suggestion cards and then modify the step.

Expand the caret next to any pattern to explore its sub-patterns, which identify subsets of values within the broader pattern.

NOTE: The Other pattern is a special category that contains values and counts not recognized by the currently selected pattern or sub-pattern. For example, when you select url pattern, the Other pattern captures the non-URL values. When you explore a sub-pattern of URLs, the Other category captures the values not recognized within the sub-pattern.

For more information on pattern standardization, see Standardize Using Patterns.

For more information on standardizing numeric values, see Normalize Numeric Values.

Pattern reuse

After patterns have been selected, they can be reused through the Transform Builder.