When working with datasets with parameters, it may be useful to do the following on a periodic basis. following if you expect the underlying datasets to be less than 100% consistent with each other.
- Recreate the dataset with parameters, except deselect the Detect Structure option during the import step.
In the Transformer page, collect a Random Sample using a Full Scan. This step attempts to gather data from multiple individual files, which may illuminate problems across the data.
Tip: If you suspect that there is a problem with a specific file or rows of data (e.g. from a specific date), you can create a static dataset from the file in question.