NOTE: Avoid creating datasets with parameters where individual files or tables have differing schemas. Either import these sources separately and then correct in the application before performing a union on the datasets, or make corrections in the source application to standardize the schemas.
When working with datasets with parameters, it may be useful to do the following if you expect the underlying datasets to be less than 100% consistent with each other.
For example, suppose you have two environments: Dev and Prod. You can create an environment parameter called
env.sourceBucketName to store the name of the bucket from which all data in the workspace or project is imported.
|Environment Name||Source Bucket Name||Environment Parameter Value|
For more information, see Overview of Parameterization.
In the screen above, you can see an example of pattern-based parameterization. In this case, you are trying to parameterize the two digits after the value:
Include nested folders
NOTE: You cannot create multiple wildcard parameters when the Include nested folders option is selected. When this option is selected for the dataset, only one wildcard parameter is supported.
When you create a wildcard or pattern-based parameter, you have the option to scan any nested folders for matching sources.