Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DEV and version r095

D toc

Excerpt

At the flow level, you can define flow parameters to reference in your recipes.

...

flow parameter

...

 is a (Variable) value or a set of enumerated values (Selector type)

Info

NOTE: Flow parameters apply to recipe steps only.

...

  1. Open the flow where you wish to apply the flow parameter. 
  2. From the Flow View context menu, select Parameters.
  3. In the Manage Parameters dialog, click the Parameters tab. 
  4. Click Add parameter.
  5. Enter a Name for your parameter.

    Info

    NOTE: Name values are case-sensitive. After saving a flow parameter, its name cannot be changed.

  6. (optional) Specify a user-friendly Description value.
  7. For Type, select the type of parameter. See below.

Variable type parameter

Variable type parameters are single values that can be one of the following formats:

FormatExample
String
Code Block
Here is my String value.

D s item
itempatterns

Any string of letters:

Code Block
`{letter}+`
regular expression

Any string of letters:

Code Block
/[a-zA-z]+/

Additional steps:

  1. Enter a default value for this parameter.

    Info

    NOTE: Input Values are evaluated as String type.

  2. Click Save.

The parameter is available for use in any recipe in your flow. See "Use Parameter."

Selector type parameter

Selector type parameters are lists of one or more permitted values.

Additional steps:

  1. Under Choices, enter a value.

    Tip

    Tip: The first value that you add is the default value.

  2. Click Add.
  3. Repeat the above steps until all permitted values have been added.
  4. Click Save.

The parameter is available for use in any recipe in your flow. See "Use Parameter."

...

Info

NOTE: You can apply override values for any parameter of any type that is referenced in the flow: dataset parameters, flow parameters, and object parameters.

Info

NOTE: For selector type flow parameters, the override value must be a value from the defined list of values.

  • Upstream parameter values: Parameter values can be inherited from upstream recipes and datasets. 

    Info

    NOTE: Override values applied in a downstream flow are applied to the upstream flow when its objects are invoked for purposes of generating data for use in the downstream flow.

  •  Downstream parameter values: Downstream flows receive parameter values, default or overridden, from upstream flows. These values can be overridden at the flow level.

...

This override value is applied to all references to the parameter in the flow. 

Tip

Tip:

...

Through Flow View, overrides can also be applied to the recipe parameters that are included when flow tasks are executed as part of a plan

...

.

Override Evaluation

Override values can be applied in multiple locations. Parameter values are evaluated in the following order of precedence (highest to lowest):

...

Info

NOTE: When aparameter value is displayed in a column, the column type in the data grid may be correctly inferring the type to your desired data type. However, the underlying type is still String type. To convert the underlying type, you must use one of the PARSE functions on your String values.

Example - String parameter

In this example, data is segmented by time zone. You must create a parameter to capture the following U.S. time zones, which must be specified explicitly:

Code Block
'Hawaii'
'Alaska'
'Pacific'
'Mountain'
'Central'
'Eastern'

In your flow, you create the following flow parameter:

...

Tip

Tip: It's a good habit to specify named variables in an identifiable way. By adding the param prefix, you identify references to it as a parameter. If you change the name to param-recipeTimeZone or similar to distinguish it as a flow parameter, then overrides specified at the flow level do not apply to any other parameter types that are performing the same function in the data.

...

Since this value must be specified explicitly, you set this value as thee default value. If this value appears in the generated output, then the flow parameter was not specified when the job was run.

Info

NOTE: Before you begin working with this parameter in your dataset, you should consider setting an override for it to a valid value.


In the following transformation, the parameter value is inserted into a new column, paramTZ in your dataset:

D trans
RawWrangletrue
p03Value'paramTZ'
Typestep
WrangleTextderive type:single value:${paramTimeZone} as:'paramTZ'
p01NameFormula type
p01ValueSingle row formula
p02NameFormula
p02Value${paramTimeZone}
p03NameNew column name
SearchTermNew formula

You can also use the parameter as an input to a function. In the following example, the paramTimeZone parameter is merged with the values in the Store_Nbr to compute primary key storeId field:

Info

NOTE: You cannot use the Merge transformation column for the following transformation, since it requires named columns as inputs.

D trans
RawWrangletrue
p03Value'storeId'
Typestep
WrangleTextderive type: single value: merge([$paramTimeZone,Store_Nbr], '-') as: 'storeId'
p01NameFormula type
p01ValueSingle row formula
p02NameFormula
p02Valuemerge([$paramTimeZone,Store_Nbr], '-')
p03NameNew column name
SearchTermNew formula

Example - parameter with multiple values

Suppose you wish to create a flow parameter that contains multiple values. Typically, you must track these values through an array, such as the following containing a set of colors:

Code Block
["red","white","blue","black"]

Flow parameters that are literals are String values only. As a workaround, you can define the above as a 

D s lang
itempattern

...

Code Block
`red|white|blue|black`

...

Note how the value is specified using backticks (`), which are used to indicate a

D s lang
itempattern
.

The vertical bars are delimiters to separate the values, when they are processed within the application.

Within your recipe, you can test for the presence of a parameter value. In the following transformation, a value of true is set in the new column isBlue if the value of $myColors is blue:

D trans
RawWrangletrue
p03Value'isBlue'
Typestep
WrangleTextderive type:single value:MATCHES([blue], $myColors, true) as:'isBlue'
p01NameFormula type
p01ValueSingle row formula
p02NameFormula
p02ValueMATCHES([blue], $myColors, true)
p03NameNew column name
SearchTermNew formula

Example Variable type example - Integer parameter

Instead of segmenting the data by named time zone values, suppose your data is segmented by regions, which are numeric in number. Your flow parameter definition could look like the following:

...

In the column histogram for the paramRegionId column, you can verify that the value 0 is present. Set an override outside at the flow level to insert a different value in the column. 

For more information, see PARSEINT Function.

...

Variable type example - Date parameter

Suppose you need to be able to pass a date into the execution of a recipe. If no date is passed in, then the current time is used. The variable is declared as follows:

...

D trans
RawWrangletrue
p03ValueparamDate
Typestep
WrangleTextderive type:single value:PARSEDATE(${paramDate},['yyyy-mm-dd HH:MM:SS']) as:'paramDate'
p01NameFormula type
p01ValueSingle row formula
p02NameFormula
p02ValuePARSEDATE(${paramDate},['yyyy-mm-dd HH:MM:SS'])
p03NameNew column name
SearchTermNew formula
For more information, see PARSEDATE Function.

If the inserted value is empty or null, you can insert the current timestamp:

...

If there is a valid value, then it is written back to the column.See NOW Function.

You can use the following to extract the time value from the parsed date param:

...

Since this value is not the parameter value specifically, the column name was listed simply as Time.

Variable type example - parameter with range of values

Suppose you need to create a parameter that can contain any two-digit value from 0 to 99. To do so, you create a flow parameter of variable type, using a 

D s lang
itempattern
 as the value:

Code Block
`{digit}{digit}`

The above matches two-digit values only. It does not match single-digit values. To expand the pattern to also match single digit values, you can use this one:

Code Block
`({digit}|{digit}{digit})`
  • Note how the value is specified using backticks (`), which are used to indicate a
    D s lang
    itempattern
    .
  • The vertical bars are delimiters to separate the values, when they are processed within the application.

This parameter now accepts the following values:

Code Block
0-9
00-99

Selector type example - parameter with multiple values

Suppose you wish to create a flow parameter that can contain one of multiple values for color. Typically, you must track these values through an array, such as the following containing a set of colors:

Code Block
["red","white","blue","black"]

Using a selector type of flow parameter, you can specify these values as choices in your permitted values list for the $myColors  flow parameter:

Code Block
red
white
blue
black

In this scenario, red  is the default value.

Within your recipe, you can test for the presence of a parameter value. In the following transformation, a value of true is set in the new column isBlue if the value of $myColors is blue:

D trans
RawWrangletrue
p03Value'isBlue'
Typestep
WrangleTextderive type:single value:IF($myColors == "blue",true,false)
p01NameFormula type
p01ValueSingle row formula
p02NameFormula
p02ValueF($myColors == "blue",true,false)
p03NameNew column name
SearchTermNew formula

Tip

Tip: When you reference a selector type flow parameter in a transformation step, you are presented the set of accepted values from which you apply the preferred one.

Apply Parameter Override via API

...

For more information, see API Workflow Task - Run Job.

D s also
inCQLtrue
label(label = "parameter")