Schema Changes Dialog
When schema validation is enabled, the schemas of the job's datasets are checked against stored versions of the schema from the previous job run and reported in the Job Details page. Through the Schema Changes dialog, you can explore differences (findings) between the stored and read schemas for each dataset.
Figure: Schema Changes Dialog
Columns:
Column: Name of the column in the dataset.
Finding: Description on the change between the column in the stored schema and the column in the schema read during this job execution. Additional details on these finds is below.
Actions:
Use the tabs at the top of the screen to filter the list of findings.
Tip
In all cases, these issues should be explored.
Note
Renamed columns appear as a deletion of the original column and an addition of the renamed column.
Finding Type | Description | Risk |
|---|---|---|
Added column | The indicated number of columns have been added to the dataset. | Risk:Moderate If these columns are added at the end of the dataset, they may not cause breakages. However, they may be omitted from some dataset reshaping transformation steps. If these columns are inserted in the middle of the dataset, they have caused some columns to be moved. See below. |
Missing column | The indicated number of columns are missing in the new dataset. | Risk: High Depending on the use of the missing columns, these could cause errors in your job execution. Additionally, downstream consumers of your data may be counting on these columns. |
Moved column | The indicated number of columns have been moved to new locations in the dataset. | Risk: Moderate Named references to the columns should still function. However, if you reference column ranges to specify a selection of columns, then these steps may no longer include the moved columns. |
Use the controls at the bottom to move to additional pages of findings.
To open the flow from which the job and these findings were detected, click Open flow.
To close the dialog, click Close.
Limits:
Maximum number of findings per type:
Columns added: 125
Columns missing: 125
Columns moved: 50
Total number of findings: 300