Skip to main content

Schema Changes Dialog

When schema validation is enabled, the schemas of the job's datasets are checked against stored versions of the schema from the previous job run and reported in the Job Details page. Through the Schema Changes dialog, you can explore differences (findings) between the stored and read schemas for each dataset.

SchemaChangesDialog.png

Figure: Schema Changes Dialog

Columns:

  • Column: Name of the column in the dataset.

  • Finding: Description on the change between the column in the stored schema and the column in the schema read during this job execution. Additional details on these finds is below.

Actions:

Use the tabs at the top of the screen to filter the list of findings.

Tip

In all cases, these issues should be explored.

Note

Renamed columns appear as a deletion of the original column and an addition of the renamed column.

Finding Type

Description

Risk

Added column

The indicated number of columns have been added to the dataset.

Risk:Moderate

If these columns are added at the end of the dataset, they may not cause breakages. However, they may be omitted from some dataset reshaping transformation steps.

If these columns are inserted in the middle of the dataset, they have caused some columns to be moved. See below.

Missing column

The indicated number of columns are missing in the new dataset.

Risk: High

Depending on the use of the missing columns, these could cause errors in your job execution. Additionally, downstream consumers of your data may be counting on these columns.

Moved column

The indicated number of columns have been moved to new locations in the dataset.

Risk: Moderate

Named references to the columns should still function. However, if you reference column ranges to specify a selection of columns, then these steps may no longer include the moved columns.

  • Use the controls at the bottom to move to additional pages of findings.

  • To open the flow from which the job and these findings were detected, click Open flow.

  • To close the dialog, click Close.

Limits:

Maximum number of findings per type:

  • Columns added: 125

  • Columns missing: 125

  • Columns moved: 50

  • Total number of findings: 300