In , a parameter is a user-defined variable that can be applied to flows, outputs, datasets, or the entire environment. This section outlines the different types and how to define them.

Parameter Overview

Each of the following types of parameters has a different scope in usage. 

Parameter types

Parameter typeDescription

Environment

A project owner or workspace administrator can define a String parameter that is available to all users of the environment.

For example, if your environment uses a consistent bucket for source data, you can define the bucket name as an environment parameter and use it accordingly. If you move your flows between projects or workspaces and your source bucket changes between them, you can change the environment parameter for the bucket name in the new project or workspace to reconnect your datasets.

Environment parameters must be defined by an administrator.

Dataset

When creating an imported dataset, you can parameterize parts of the source path to the file(s) or table(s).

For example, if you want to create a dataset from all of the files in a single directory, you can define a pattern parameter to match all of the filenames. Whenever new files land in the directory that match the pattern, they are automatically added to the dataset and processed whenever you run a job.

Dataset parameters can be patterns (as above), wildcards, variables, or Datetime values.

Flow

Flow parameters are defined at the flow level and can be referenced in your recipes.

For example, you may wish to pass into your recipe the name of the client to which a job applies. You can define this value as a flow parameter and then reference the parameter in your steps.

Flow parameters can be variables.

Output

You can define parameters to specify output file or table destinations.

For example, you may wish to append the filename of your output with a timestamp for when the file is written.

Output parameters can be timestamps or variables.

Parameter overrides

In some cases, parameters can be overridden with new values at runtime. A parameter override is a new value set for a variable parameter when specifying a job in the Run Job page. Using parameter overrides, you can perform custom executions of your jobs to test specific outputs based on changing conditions. Parameter overrides apply only to the current job. 

The following table shows the different types of parameters and the types of data that you can define for them.

Parameter typeDatetime/Timestamp?Variable?Pattern?Overrides?
EnvironmentNYNY
DatasetYYYN
FlowNYNY
OutputYYNY

Environment Parameters

An environment parameter is a project- or workspace-level variable that is accessible to all users. These parameters are useful for defining values that apply to the entire environment.

NOTE: You must be a project owner or workspace administrator to define or modify environment parameters.

Environment parameters can be used in imported datasets and output paths.

Steps:

To create an environment parameter, please complete the following steps.

  1. Login to the  as an administrator.
  2. From the menu, select User menu > Admin console.
  3. In the Admin console, select Environment parameters.
  4. In the Environment Parameters page, click Create.
  5. For the environment parameter:

    1. Name: Enter a unique name within the workspace. 

      NOTE: To distinguish them from other parameters, the prefix env. is added to all environment parameter names.

      Tip: If you plan to use this environment parameter with different values in other projects or workspaces, you should use a consistent naming convention. References to an environment parameter from one project or workspace can be imported into another and mapped to environment parameters that exist in the new environment.

    2. Value: Specify a default value for the environment parameter. Default values are String values.
  6. To create additional parameters, click Add another.
  7. To save your environment parameters, click Save.

Export and Import

You can export and import sets of these parameters through the Environment Parameters page

Steps:

  1. To export the current set of environment parameters, click More (...) menu > Export in the Environment Parameters page. 
  2. The ZIP file is downloaded to your local desktop.
  3. In the target project or workspace, click More (...) menu > Import in the Environment Parameters page.
  4. In the Import Environment Parameters dialog, you can map the set of incoming environment parameters to existing parameters in the environment. If the environment parameter names match, then the new parameter overwrites the existing one in the target environment. 
    1. To delete parameters from import, click the Trash icon next to the parameter name.
  5. To save your changes and complete the import, click Save.
  6. The environment parameters are imported.

Dataset Parameters

You can create a dataset with parameters to capture multiple datasource files or tables into a single dataset. When you create a dataset, you can specify parts of the file path or SQL query as parameters, which allows you to capture a broader set of data into a single imported dataset.

Tip: You can parameterize paths to folders, which allows you to select files located in parallel directories.

In this manner, you can manage consistently named and located data through a single imported dataset object.

Dataset with parameters for files

When you create an imported dataset, you can parameterize parts of the path to capture multiple files or folders. 

Tip: Depending on your environment, you may also be able to parameterize the names of the buckets in your paths.

Steps:

  1. In the , click the Library icon in the left nav bar.
  2. In the Library page, click Import Data.
  3. In the Import Data page, select the connection that contains your files. 
  4. Navigate through the connection to locate one of the files or folders that you wish to parameterize for input. 
  5. To create a dataset with parameters:
    1. To parameterize at the folder level, click the Create Dataset with Parameters button.
    2. To parameterize based on an individual file, click the Parameterize button on the right side of the file's listing. 
  6. In the Create Dataset with Parameters dialog, click and drag to highlight the part of the path that you wish to turn into a parameter.
  7. Click one of the three icons that appears over your selection:
    1. Add Datetime Parameter: Parameterize a Datetime value.
    2. Add Variable: Create a variable that you can specify and later override to replace the highlighted value.

    3. Add Pattern Parameter: Create a pattern-based or wildcard parameter.

      NOTE: Patterns can be based on a modified form of regex or .

       

  8. Specify the parameter and click Save.
  9. Click Update matches. The list of matching files for your parameter is displayed.
  10. To save the parameter, click Create.
  11. In the Import Data page, review the settings or make other selections as needed before creating the dataset.

Dataset with parameters for tables

When you create a dataset using custom SQL, you can choose to parameterize values in the SQL statements that you use to query for data. 

Flow Parameters

At the flow level, you can define parameters for use in your recipes and apply overrides to parameters used within your flow. 

Steps:

  1. Open the flow in Flow View.
  2. From the Flow View context menu, select Parameters.

  3. The available parameters for the flow and any overrides are displayed. 

    Tip: Overrides values for use in the flow can be applied to any parameter that is accessible to this flow, including environment, flow, and output parameters.

  4. To add a new flow parameter, Add parameter.
    1. Name: Enter a descriptive name for the parameter.

    2. Value: Enter a default value for the parameter.

      Tip: Default values can be overridden when you are running a job.

        

  5. To save your configured flow parameter, click Save.

Flow parameters in plans

When you add a flow task to your plan, you can review and manage any parameters pertaining to the flow or its outputs.

Tip: Changes made to parameters through the flow task only apply as overrides to execution of the flow within the plan. Parameter values defined within the flow are not affected.

 Steps:

  1. In Plan View, create or select the flow task to generate the output. 
  2. After it is created, select the flow task. In the right panel, click the Parameters tab.
  3. To apply overrides, click Manage.
  4. In the Manage Parameters dialog, hover over the parameter to edit. Then, click the Edit icon.
  5. Modify the value, and click Save.

When the flow task is executed within the plan, the new parameter value is applied.

Output Parameters

When defining an output, you can parameterize parts of the output path using timestamp or variable parameters. 

Steps:

  1. In Flow View, select or create the output object. 
  2. In the context panel on the right, click the Output Destinations tab. 
  3. For Manual or Scheduled destinations, click Edit.
  4. In the Destinations window, add or edit the destination. 
  5. When you have the file or table selected or specified, click Parameterize destination in the upper-right corner.
  6. In the Defined parameterized destination window, highlight the part of the output path that you wish to parameter.
  7. Choose the parameter type:
    1. Add Timestamp Parameter: Select the timestamp and timezone to insert in the output path.
    2. Add Variable: Define the variable name and value to insert.
  8. To save your parameter, click Save.
  9. To save your parameterized destination, click Submit.