Schema Changes Dialog
When schema validation is enabled, the schema of the job's datasets is checked against the stored schema versions from the previous job run and reported on the Job Details page. Through the Schema Changes dialog, you can explore differences (findings) between the stored and read schema for each dataset.
The Schema changes dialog consists of these columns:
Column: Name of the column in the dataset.
Finding: Description of the change between the column in the stored schema and the column in the schema read during this job execution. Additional details on these findings are available below. Use the tabs at the top of the dialog to filter the list of findings.
Use the pagination controls at the bottom to move to additional pages of findings. To close the dialog, select Close.
Important
In all cases, these issues should be explored. Renamed columns appear as a deletion of the original column and an addition of the renamed column.
Schema Changes Findings Definitions
Finding Type | Description | Risk |
|---|---|---|
Added Column | The indicated number of columns has been added to the dataset. | Risk: Moderate If these columns are added at the end of the dataset, they might not cause breakages. However, they might be omitted from some dataset reshaping transformation steps. If these columns are inserted in the middle of the dataset, some columns are moved. See below. |
Missing Column | The indicated number of columns is missing in the new dataset. | Risk: High Depending on the use of the missing columns, these could cause errors in your job execution. Additionally, downstream consumers of your data might be counting on these columns. |
Moved Column | The indicated number of columns has been moved to new locations in the dataset. | Risk: Moderate Named references to the columns should still function. However, if you reference column ranges to select columns, these steps might no longer include the moved columns. |
Schema Changes Findings Limits
Maximum number of findings per type:
Columns added: 125
Columns missing: 125
Columns moved: 50
Total number of findings: 300