provides multiple mechanisms to transform and standardize data to meet usage needs, including profile visualizations and type-based quality bars to identify potential anomalies and quality problems. Data quality checks can be applied during data import, transformation, or export in the form of visual profiling.

Broadly speaking, data quality identifies the degree to which data is usable and responsive to your use case. When you assess data quality, you are designing tests to assess its suitability for generic usage and for your specific uses. 

Data Quality Characteristics

Data quality covers the following characteristics:

Schema Validation

Type inference

When data is imported, the  attempts to infer the data types in the source and to type columns in the dataset accordingly. Type inference uses the first 20-25 rows of the initial sample to assess the appropriate data type to apply to the column. For more information, see Type Conversions.

Some imported data, such as relational tables, may include schema information to identify the data type of each column. In some cases you can disable type inferencing on imported data:

Assign targets

To assist in your transformation efforts, you can assign a target schema for each recipe. This target schema is super-imposed on the columns of your data. Using visual tools to review differences and select changes, you can rapidly convert the structure of your dataset in development to meet the expected target schema. For more information, see Overview of RapidTarget

Identify Anomalies

In the Transformer page, you can use the available visual tools to review the data quality characteristics of the columns in your data. These data visualizations and type-based quality bars can assist in identifying potential anomalies and quality problems.

Data quality bar

At the top of each column, you can see a data quality bar, which uses the following color coding to validate the column values against the selected column type.

Color barDescription
greenValues that are valid for the current data type of the column
redValues that are mismatched for the column data type
blackMissing or null values

Tip: Click any of the color bars to receive suggestions for transformations to add to your recipe. See Overview of Predictive Transformation.


Tip: You can change a column's data type in the column header. See Column Menus.

For more information, see Data Quality Bars.

Column histogram

In the column header, you can review the count and distribution of values in the column. A column's histogram can be useful for identifying anomalies or for selecting specific sets of values in the column for further exploration.  

Tip: Click and drag over any set of values to receive suggestions for transformations to add to your recipe. See Overview of Predictive Transformation.

See Column Histograms.

Column details

Through the Column Details panel, you can explore the quality and distribution of the values in the column. The contents of the panel vary depending on the data type. For example, if the column is typed for Datetime values, then the Column Details panel includes information on the distribution of values across the days of the week and days of the month. 

For all data types, you can review useful statistics on statistical quartiles, the uniqueness of values, mismatches, and outliers. 

Tip: The Column Details panel is very useful for acquiring statistical information on column values in a visual format. Click any data quality bar to be prompted for suggestions of transformation steps. See Overview of Predictive Transformation.

For more information, see Column Details Panel.

Standardization

You can use the Standardization tool to standardized clustered sets of column values to values that are common and consistent throughout your enterprise's data. For more information, see Overview of Standardization

Data Quality Functions

The following functions are available for assessing data quality.

Count functions

The following functions measure counts of values within a column, optionally counted by group. 

ItemDescription
COUNT Function

COUNTA Function

COUNTDISTINCT Function

UNIQUE Function

Aggregation functions

ItemDescription
AVERAGE Function
SUM Function

MIN Function

MAX Function

MODE Function

MINDATE Function

MAXDATE Function

MODEDATE Function

Statistical functions - single column

Variations in these functions:

General statistics

ItemDescription
VAR Function

STDEV Function

MEDIAN Function

QUARTILE Function

PERCENTILE Function


ItemDescription
APPROXIMATEMEDIAN Function

APPROXIMATEQUARTILE Function

APPROXIMATEPERCENTILE Function

Statistical functions - multi-column

ItemDescription
COVAR Function

CORREL Function

Data Quality in Job Details

When you run a job and generate results, you can review the the quality of the data of the generated output. 

Visual profiling

In parallel with executing the job, you can generate a visual profile of the generated results. This visual profile provides graphical representations of the valid and mismatched values against each column's data type, as well as indications about missing values in the output. 

Tip: Visual profiles can be downloaded in PDF or JSON format for offline analysis.

Visual profiling is selected as part of the job definition process. See Run Job Page.

For more information, see Overview of Visual Profiling.

For more information, see Job Details Page.