For more information on parameterization of datasets and other types of parameters, see Overview of Parameterization.
From File System
When browsing for data on your default storage layer, you can choose to parameterize elements of the path. Through the Import Data page, you can select elements of the path, apply one of the supported parameter types and then create the dataset with parameters.
NOTE: When you import a file, the data is not stored in the Dataprep by Trifacta Cloud . What you create is an imported dataset that is simply a reference to the source of the data. The Dataprep by Trifacta Cloud never stores or modifies source data.
When you create a dataset with parameters in the Dataprep by Trifacta Cloud , you can replace segments of the input path with parameters. Suppose you have the following files that you'd like to capture through a parameterized dataset:
A parameterized reference to all of these files would look something like:
Through the application, you can specify the parameters to match all values for:
##- You can use a wildcard or (better) a pattern to replace these values.
YYYY-MM-DD- A formatted Datetime parameter can replace these values.
For more information, see Parameterize Files for Import.
Parameterize bucket names
You can create environment parameters for your bucket names.
From Relational Sources
You can create datasets from a relational source by applying parameters to the custom SQL that pulls the data from the source. During import of database tables through relational connections, you can apply parameters to the SQL query that you use to define the imported dataset. In some scenarios, you may need to define the table to import using a variable parameter or to parameterize the time value associated with a table name. Using parameters, you can define the tables, columns, and conditions of the query that you use to bring in data from a relational database.
For more information, see Parameterize Tables for Import.
After you have created your dataset with parameters, you can edit the parameter as needed.
- In the left nav bar, select Library.
- In the Library page, locate the dataset. From its context menu, select either of the following:
- Files: Select Edit parameters. In the Edit Dataset with Parameters, click the parameter to modify its definition.
- Tables: Click Edit Custom SQL. In the Custom SQL window, you can modify the SQL statement, including any parameters in it. For more information, see Create Dataset with SQL.
Apply Parameter Overrides
After you have created a parameterized dataset, you can apply overrides to the default value. These override values can be applied in the following order of precedence.
- Job: When you choose to execute a job, you can set a new value for the parameter, which is applied for the specified job only.
Flow: If your imported dataset containing a parameter is added to a flow, you can define an override value for the dataset's parameter through Flow View. Whenever a job is executed on the imported dataset within the flow, the override value is applied to the dataset.
NOTE: If a job-level override is applied on top of a flow-level override, the job override value is applied to the job.
- Default: The default value for the parameter is used if no override is applied.
Apply parameter overrides for your flow
- Open the flow.
- In Flow View, select the icon for your dataset with parameters.
- From the context menu, select Parameter.
- In the Manager Parameters dialog, click the Overrides tab.
- Edit the required values, click Save.
For more information, see Manage Parameters Dialog.
Apply parameter overrides for your job
You can apply parameter overrides to your job.
- In Flow View, select the output that you wish to generate.
- In the right context panel, click Run Job.
- In the Run Job page, you can specify job-level overrides at the bottom of the screen.
For more information, see Run Job Page.
In the Edit Dataset with Parameters screen, select the parameter that you wish to remove.
NOTE: Before you remove parameter, you may want to take note of the default value, which may need to be applied to the path or query after you remove the parameter.
- In the popup, click Delete.
- Save your changes.
- The parameter is removed from the imported dataset definition.
This page has no comments.