In , a parameter is a user-defined variable that can be applied to flows, outputs, datasets, or the entire environment. This section outlines the different types and how to define them.
Each of the following types of parameters has a different scope in usage.
A project owner or workspace administrator can define a String parameter that is available to all users of the environment.
For example, if your environment uses a consistent bucket for source data, you can define the bucket name as an environment parameter and use it accordingly. If you move your flows between projects or workspaces and your source bucket changes between them, you can change the environment parameter for the bucket name in the new project or workspace to reconnect your datasets.
Environment parameters must be defined by an administrator.
When creating an imported dataset, you can parameterize parts of the source path to the file(s) or table(s).
For example, if you want to create a dataset from all of the files in a single directory, you can define a pattern parameter to match all of the filenames. Whenever new files land in the directory that match the pattern, they are automatically added to the dataset and processed whenever you run a job.
Dataset parameters can be patterns (as above), wildcards, variables, or Datetime values.
Flow parameters are defined at the flow level and can be referenced in your recipes.
For example, you may wish to pass into your recipe the name of the client to which a job applies. You can define this value as a flow parameter and then reference the parameter in your steps.
Flow parameters can be variables.
You can define parameters to specify output file or table destinations.
For example, you may wish to append the filename of your output with a timestamp for when the file is written.
Output parameters can be timestamps or variables.
In some cases, parameters can be overridden with new values at runtime. A parameter override is a new value set for a variable parameter when specifying a job in the Run Job page. Using parameter overrides, you can perform custom executions of your jobs to test specific outputs based on changing conditions. Parameter overrides apply only to the current job.
The following table shows the different types of parameters and the types of data that you can define for them.
An environment parameter is a project- or workspace-level variable that is accessible to all users. These parameters are useful for defining values that apply to the entire environment.
NOTE: You must be a project owner or workspace administrator to define or modify environment parameters.
Environment parameters can be used in imported datasets and output paths.
To create an environment parameter, please complete the following steps.
Name: Enter a unique name within the workspace.
NOTE: To distinguish them from other parameters, the prefix
Tip: If you plan to use this environment parameter with different values in other projects or workspaces, you should use a consistent naming convention. References to an environment parameter from one project or workspace can be imported into another and mapped to environment parameters that exist in the new environment.
You can export and import sets of these parameters through the Environment Parameters page
You can create a dataset with parameters to capture multiple datasource files or tables into a single dataset. When you create a dataset, you can specify parts of the file path or SQL query as parameters, which allows you to capture a broader set of data into a single imported dataset.
Tip: You can parameterize paths to folders, which allows you to select files located in parallel directories.
In this manner, you can manage consistently named and located data through a single imported dataset object.
When you create an imported dataset, you can parameterize parts of the path to capture multiple files or folders.
Tip: Depending on your environment, you may also be able to parameterize the names of the buckets in your paths.
Add Pattern Parameter: Create a pattern-based or wildcard parameter.
NOTE: Patterns can be based on a modified form of regex or .
When you create a dataset using custom SQL, you can choose to parameterize values in the SQL statements that you use to query for data.
At the flow level, you can define parameters for use in your recipes and apply overrides to parameters used within your flow.
The available parameters for the flow and any overrides are displayed.
Tip: Overrides values for use in the flow can be applied to any parameter that is accessible to this flow, including environment, flow, and output parameters.
Value: Enter a default value for the parameter.
Tip: Default values can be overridden when you are running a job.
When you add a flow task to your plan, you can review and manage any parameters pertaining to the flow or its outputs.
Tip: Changes made to parameters through the flow task only apply as overrides to execution of the flow within the plan. Parameter values defined within the flow are not affected.
When the flow task is executed within the plan, the new parameter value is applied.
When defining an output, you can parameterize parts of the output path using timestamp or variable parameters.