When schema validation is enabled, the schemas of the job's datasets are checked against stored versions of the schema from the previous job run and reported in the Job Details page. Through the Schema Changes dialog, you can explore differences (findings) between the stored and read schemas for each dataset.
For more information, see Overview of Schema Management.
- Column: Name of the column in the dataset.
- Finding: Description on the change between the column in the stored schema and the column in the schema read during this job execution. Additional details on these finds is below.
Use the tabs at the top of the screen to filter the list of findings.
Tip: In all cases, these issues should be explored.
NOTE: Renamed columns appear as a deletion of the original column and an addition of the renamed column.
|Added column||The indicated number of columns have been added to the dataset.|
If these columns are added at the end of the dataset, they may not cause breakages. However, they may be omitted from some dataset reshaping transformation steps.
If these columns are inserted in the middle of the dataset, they have caused some columns to be moved. See below.
|Missing column||The indicated number of columns are missing in the new dataset.|
Depending on the use of the missing columns, these could be directly referenced in the steps of your recipe. Additionally, downstream consumers of your data may be counting on these columns.
|Moved column||The indicated number of columns have been moved to new locations in the dataset.|
Named references to the columns should still function. However, if your recipe steps use column ranges to specify a selection of columns, then these steps may no longer include the moved columns.
- Use the controls at the bottom to move to additional pages of findings.
- To open the flow from which the job and these findings were detected, click Open flow.
- To close the dialog, click Close.
Maximum number of findings per type:
- Columns added: 125
- Columns missing: 125
- Columns moved: 50
- Total number of findings: 300
This page has no comments.