Page tree

Trifacta Dataprep



Contents:


   


Feature Availability: This feature is available in the following editions:

  • Dataprep by Trifacta® Enterprise Edition
  • Dataprep by Trifacta Professional Edition
  • Dataprep by Trifacta Premium

Contents:


In the body and header of HTTP tasks in your plans, you can reference the following elements of metadata from the plan run for additional contextual information.

General Syntax

All plan metadata references follow the following basic syntax:

{{$plan.path.to.reference}}
  • All references can be entered with $ in the Trifacta application. These references are turned into {{$ in the code definition. The double-curly braces forms the environment for metadata replacement. 

    Tip: In the Trifacta application, you can start by typing $.

  • Nodes in the tree are separated with a . period.

Reference values that contain whitespace must be listed in the following manner:

{{$plan.path['path with white space in it'].rest.of.path}}

Notes:

  • In the Trifacta application, you can use double-quotes when specifying a whitespace value. However, these double-quotes get escaped in the actual request. It is safer and more consistent to use single quotes.

Whitespace values typically appear when referencing the display name values for underlying objects, like recipes executed as part of a flow task.

$plan References

These references apply to the plan definition or current plan run.

Text to enter:

$plan.


ReferenceDescription
nameName of the plan that is run.
duration

Length of time that the plan ran or has run so far

Tip: To return a more readable form of this duration value, use the following reference:

{{$plan.duration|humanizeDuration}}
startTimeTimestamp for when the plan run began
runIdInternal identifier for this run of the plan
userInternal identifier of the user who launched this run.
taskCountCount of tasks in the plan run.

$http References

These references apply to HTTP tasks in the plan run.

Enter the following, after which you can see the two-letter codes for the HTTP tasks that have already executed in the current plan run:

$http_ax.


ReferenceDescription
nameName of the HTTP task
statusCurrent status of the task execution
durationLength of time that the task ran or has run so far
startTimeTimestamp for when the task began. A null value if the task has not begun.
endTimeTimestamp for when the task ended. A null value if it has not ended yet.
statusCodeStatus code (if any) returned from the receiving endpoint
responseResponse information. See below.

Response references

These references apply to the response returned as part of the task execution.

Enter the following, after which you can see the two-letter codes for the HTTP tasks that have already executed in the current plan run:

$http_ax.response.


ReferenceDescription
bodyBody of the response
jsonJSON-formatted version of the response
headersHeaders returned with the response

$slack References

You can reference metadata from Slack tasks in the current plan run using the following reference types:

$slack_ax.

Supported metadata is identical to the metadata for HTTP tasks. See the previous section for details.

$delete References

You can reference metadata from Delete tasks in the current plan run using the following reference types:

$delete_92.

Supported metadata is identical to the metadata for HTTP tasks. See the previous section for details.

ReferenceDescription
nameName of the HTTP task
statusCurrent status of the task execution
pathFull path to the file(s) or folder that was deleted.
durationLength of time that the task ran or has run so far
startTimeTimestamp for when the task began. A null value if the task has not begun.
endTimeTimestamp for when the task ended. A null value if it has not ended yet.
deletedFilesCountNumber of files that were successfully deleted.

$flow References

These references apply to flow tasks in the plan run.

Enter the following, after which you can see the two-letter codes for the HTTP tasks that have already executed in the current plan run:

$flow_ax.


ReferenceDescription
nameName of the flow task
statusCurrent status of the task execution
durationLength of time that the task ran or has run so far
startTimeTimestamp for when the task began. A null value if the task has not begun.
endTimeTimestamp for when the task ended. A null value if it has not ended yet.
jobIdsInternal identifiers for the jobs that were run as part of this flow task
flowNameName of the flow underlying this flow task
outputMetadata from the flow task's output. See below.

Output references

These references apply to the outputs that are generated in the flow tasks of the plan run.

Enter the following for flow task 7p with output My Output Name:

$flow_7p['My Output Name'].


ReferenceDescription
nameName of the flow
statusCurrent status of the flow
durationLength of time that the flow execution ran or has run so far
startTimeTimestamp for when the flow execution began. A null value if the run has not begun.
endTimeTimestamp for when the flow execution ended. A null value if it has not ended yet.
lastUpdateTimestamp for when the flow was last modified
jobIdInternal identifier for the job that was run or is running for the flow
userInternal identifier for the user who executed the job
jobType

The type of job that was executed. Values:

  • manual - executed through the Trifacta application
  • scheduled - executed according to a defined schedule
  • api - executed via API
fileSizeIf the output generates a file or files, this value captures the size in KB of the output.
environmentRunning environment where the job was executed
columnCountCount of columns generated in the output
rowCountCount of rows generated in the output
dataTypeCount

Count of Trifacta data types detected in the output

validValuesCountCount of valid values in the output
mismatchedValuesCountCount of mismatched values in the output
emptyValuesCountCount of missing or empty values in the output
columns

Column information from the selected output for the flow. See below.

Tip: If you have defined any data quality rules on the column, they are listed, too. For more information, see Data Quality Rules Reference.

sourcesSource filename and table information from the imported datasets. See below.
publishing actionsData on the publishing actions defined for the output. See below.

Output column references

Tip: The metrics calculated for output columns may also appear in profiles of your output data.

These references are available for output columns in the following syntax:

$flow_7p['My Output Name'].output.
ReferenceDescription
nameColumn name
typeData type of column
validValuesCountCount of valid values in the column
mismatchedValuesCountCount of mismatched values in the column
emptyValuesCountCount of empty values in the column
topValuesList of top values in the column
minimumValueLowest value in the column
lowerQuartileValue25th percentile value in the column
medianValue50th percentile value in the column
upperQuartileValue75th percentile value in the column
maximumValueMaximum value in the column

Sources references

These references apply to the datasource files or tables that were used to generate the output.

ReferenceDescription
nameName of the datasource file or table

Publishing actions references

These references apply to the publishing actions defined for the output.

ReferenceDescription
nameName of the publishing action
actionDefinition of the publishing action
locationLocation where the publishing action is published
typeType of publishing action

Metadata Functions

The following functions can be applied to select metadata references to further filter the raw values. 

humanizeDuration

In raw form, the duration metadata references return values that look like the following:

PT7.523S

You can apply the HumanizeDuration function to render the above into a more readable format:

{{ $plan.duration | humanizeDuration }}

The output of the above applied to the first value is the following:

7.523 seconds

uuid

You can generate a universally unique identifier, which can be delivered as part of a messaging payload:

{{ uuid() }}

Parameter References

Flow parameters

To reference flow parameters as part of a flow task, use the following pattern:

$plan.params.<myParamName>

Environment parameters

You can reference environment parameters by name in your plan task metadata. For example:

env.MyEnvironmentParameter1

Additional References

Plan metadata reference information leverages the Nunjucks templating language, which provides additional capabilities such as loops, conditions, filters, and helper functions.

NOTE: These additional capabilities are available through the language, but their implementation in the Trifacta application has not been certified. For Nunjucks capabilities not listed on this page, you should experiment with them in a development environment first.

For more information, see https://mozilla.github.io/nunjucks/templating.html.


This page has no comments.