...
You can create a dataset from one or more files stored in ADLS.
Wildcards:
You can parameterize your input paths to import source files as part of the same imported dataset. For more information, see Overview of Parameterization.
Folder selection:
When you select a folder in ADLS to create your dataset, you select all files in the folder to be included. Notes:
- This option selects all files in all sub-folders. If your sub-folders contain separate datasets, you should be more specific in your folder selection.
- All files used in a single dataset must be of the same format and have the same structure. For example, you cannot mix and match CSV and JSON files if you are reading from a single directory.
- When a folder is selected from ADLS, the following file types are ignored:
*_SUCCESS
and*_FAILED
files, which may be present if the folder has been populated by HDI.- If you have stored files in ADLS that begin with an underscore (
_
), these files cannot be read during batch transformation and are ignored. Please rename these files through ADLS so that they do not begin with an underscore.
...