JavaScript UDFs

Wrangle provides a wide range of functions to transform your data across all supported data types. However, you may need to be able to define functions to meet your specific use cases.

Examples:

• Your enterprise has a specific method for calculating asset deprecation, which must be applied consistently.
• You use specialized statistical calculations for managing risk. 
• Your industry has commonly used metrics that are not available in the Dataprep by Trifacta application.

To support more specific use cases, you can create functions in JavaScript and make them available for use within the recipes that you create in the Dataprep by Trifacta application.

Features:

• Easy to create and modify directly through the Dataprep by Trifacta application.
• JavaScript UDFs are available through the Search panel and can be referenced in formulas like any other  Wrangle function.
• Executable for both Trifacta Photon and Dataflow jobs
• Internationalization is supported through built-in Java libraries. For more information, see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl.

Prerequisites

Required access levels

Access level for JavaScript UDF feature and UDF code is determined by the User defined functions privilege. 

For more information, see  Overview of Authorization . 

For more information, see Privileges and Roles Reference.

Enable

This feature is enabled by default. See Dataprep Project Settings Page.

Limitations

The following limitations apply to JavaScript UDFs:

• JavaScript UDFs execute function calls for each row of data. 
  • UDFs cannot guarantee maintaining global variable state across an entire sorted dataset that is being processed.
  • Aggregate functions across multiple rows are not supported.
• Each UDF can add a new column or edit an existing column's values. Other types of data transformations are not supported.
• Functions cannot be implemented to make external calls over the network. External API calls are not supported.
  • JavaScript libraries cannot be imported for use in creating your UDFs.
    
• UDFs cannot be used in macros.
• UDFs cannot be defined to replace functions in  Wrangle. 

• You cannot reference  Wrangle functions from within your UDF code.

  Tip: Wrangle references can be used as inputs to a UDF or in a Formula of which your UDF is a part. Details are below.

• Export and import of JavaScript UDFs is not supported. 

  Tip: JavaScript UDFs can be imported into another project or workspace when they are referenced in a flow that you are importing. See "Importing through Flows" below.

• If assets are transferred to a new user via API, UDFs are not included in the transfer.

Supported JavaScript

JavaScript implementation

• Supports only Javascript ES6 version. For more information, see https://www.w3schools.com/js/js_es6.asp.
• JavaScript V8 libraries are used internally. For more information on licensing, see https://github.com/v8/v8/blob/master/LICENSE.
• Memory and stack depth limits may impact the number of nested loops or recursive calls that can be executed within a single function.
• HTTP request objects cannot be referenced from within your JavaScript.
• JavaScript definitions are not stored in encrypted format.
• The export keyword can be used for the following definitions only:

  • The UDF signature constant:
    
    

    export const signature

    
    

  • The main function:
    
    

    export function trifactaUdf()

    NOTE: Use of the export keyword in other areas of your code cause failures if the UDF is executed as part of a BigQuery pushdown transformation. Other uses of export are not supported.

    
    

Supported JavaScript objects

Fundamental objects:

• Object

• Function

• Boolean

• Symbol

Error handling:

• Error
• AggregateError
• EvalError
• InternalError
• RangeError
• ReferenceError
• SyntaxError
• TypeError
• URIError

Data types:

• Number
• BigInt
• Math
• Date
• String
• RegExp
• Array

Array types:

• Int8Array
• Uint8Array
• Uint8ClampedArray
• Int16Array
• Uint16Array
• Int32Array
• Uint32Array
• Float32Array
• Float64Array
• BigInt64Array
• BigUint64Array

Other supported objects and collections:

• Map

• Set
• WeakMap
• WeakSet
• JSON
• ArrayBuffer
• DataView
  
• Promise
• Generator
• GeneratorFunction

Supported JavaScript value properties

• undefined

• NaN

• Infinity
• isFinite
• isNaN

Supported JavaScript functions

Parsing functions:

• parseFloat

• parseInt

URI encoding functions:

• encodeURI
• encodeURIComponent
• decodeURI
• decodeURIComponent

Internationalization functions:

• Intl
• Intl.Collator
• Intl.DateTimeFormat
• Intl.ListFormat
• Intl.NumberFormat
• Intl.PluralRules
• Intl.RelativeTimeFormat
• Intl.Locale

Type Conversions

The following tables indicate how Dataprep by Trifacta data types are converted to JavaScript data types during input and also for output back to Dataprep by Trifacta data types.

For more information on the string values for Dataprep by Trifacta data types, see Valid Data Type Strings.

Input type conversions

Output data types

On output, JavaScript data types are exported to their corresponding Dataprep by Trifacta data types, with the following specific mappings: 

Security Measures

To minimize the risk of code execution on the platform, the following security measures have been applied:

• Some language constructs like eval are not supported.

• During design time, web workers are used.

• During runtime (job execution): 

  • v8 Isolates are deployed for execution. For more information, see  https://v8.dev/docs/untrusted-code-mitigations.

  • Jobs are executed in sandboxed compute resources for the following running environments:
    • Trifacta Photon: remote Kubernetes job
    • Dataflow: customer VPC
    • Users that are not executing the job are isolated from job data and Dataprep by Trifacta metadata through UDFs.
• JavaScript UDFs have execution time limits placed on them to prevent runaway processes. 
• JavaScript UDFs have other security related restrictions to prevent malicious code injection.
• Sensitive APIs of the Dataprep by Trifacta Cloud are not exposed in the global object.

Enable

For more information on enabling this feature in your project, please contact  Google Support.

UDF Requirements

The following sections define the requirements for creating your UDF.

Metadata

The following metadata objects must be defined as part of your JavaScript UDF:

Signature

The signature defines the inputs of the function. A JavaScript UDF signature has the following generic structure. 

export const signature = {
  "inputs": [   
   {
      "alias": "Input1",
      "description": "Input1's description",
      "displayType": "<DataType>",
      "examples": [
        "COL1"
      ]
    },
    {
      "alias": "Input2",
      "description": "Input2's description",
      "displayType": "<DataType>",
      "examples": [
        "COL2"
      ]
    }
  ],
  "output": {
    "type" : "String"
  }
};

Notes:

• The signature and function definition must be marked for export in the JavaScript module. The following line must be present at the start of the signature:

  export const signature = {

  NOTE: Use of the export keyword outside of the UDF signature constant or main function declaration cause failures if the UDF is executed as part of a BigQuery pushdown transformation. Other uses of export are not supported.

• Individual inputs are specified as elements of an array. Each array element can contain the following attribute and values:

  Attribute	Description
  alias	(Required) Unique name of the input. This value can be referenced within the function.
  description	User-friendly description of the input. Appears in the popup that is displayed when the function has been selected.
  displayType	Text value representing the data type of the input. This value is for display purposes only. Appears in the function popup.
  examples	Text value or values to indicate the type of input. Appears in the function popup.

• The number of inputs in the signature array must equal the number of inputs in the main function.
• The output of the function is specified according to the Dataprep by Trifacta data type. For more information on the values to insert for type, see Valid Data Type Strings.

Function definition

The definition of your function has the following basic structure:

export function trifactaUdf(<alias1>,<alias2>) {
   <your_JavaScript>
   }
   return <values_returned>;
}

Notes:

• The signature and function definition must be marked for export in the JavaScript module. The following line must be present at the start of the signature:

  export function trifactaUdf(<any_inputs>) {
  

  • In the above, the inputs to the function must match in name the values for each alias in the signature. 

• The values returned from the function are specified as part of the return entry.

Type inference

JavaScript is a dynamically typed language. As a result, actual data types are assigned to inputs based on type inferencing performed by the Dataprep by Trifacta application during design time.

Flow parameters are always passed into the function for execution as String values and must explicitly cast to other data types within the function.

Other requirements

Name scoping

Internally, all UDFs are scoped in name using the convention:

UDF.<yourFunctionName>

This naming convention prevents name collision with existing  Wrangle functions.

Syntax

Please use Javascript ES6 syntax. For more information, see https://www.w3schools.com/js/js_es6.asp.

Commenting

You can insert comments in your UDF code. The following is an example:

// This is a comment.

Examples

Example - Net Present Value

This example computes net present value.

export const signature = {
  "inputs": [   
   {
      "alias": "R",
      "description": "Cash flow at time t",
      "displayType": "Integer",
      "examples": [
        "COL1"
      ]
    },
   {
      "alias": "i",
      "description": "Discount rate as a decimal value",
      "displayType": "Float",
      "examples": [
        "COL2"
      ]
    },
    {
      "alias": "t",
      "description": "Number of time units",
      "displayType": "Float",
      "examples": [
        "COL3"
      ]
    }
  ],
  "output": {
    "type" : "Float"
  }
};


export function trifactaUdf(R, i, t) {
   let discount = Math.pow(1 + i, t);  
   return (R / discount);
}

Other examples

For additional examples, please see https://community.trifacta.com/s/article/JavaScript-UDFs-Examples.

Deploy Your UDF

User Defined Functions page

When you have created your UDF in JavaScript, you can upload it to your project in the User Defined Functions page. Select Library > User defined functions. 

See User Defined Functions Page.

Importing through flows

JavaScripts UDFs cannot be separately exported and imported from the Dataprep by Trifacta application. However, when you export a flow definition that contains references to a JavaScript UDF, the function definition is also exported.

When this flow is imported into a new project or workspace:

• If no UDF exists with same name in the target system, then a new UDF is created.
• If a UDF already exists with same name, then you can choose one of the following: 
  • Use the existing UDF in the new project or workspace,

  • Overwrite the existing UDF with the UDF in the flow import.

    NOTE: You must have Editor or better privileges on the existing UDF to overwrite it during import.

Job execution

Javascript UDFs are executed as part of jobs on Dataflow and Trifacta Photon. 

Error handling

• When errors are encountered using JavaScript UDFs:
  • Transformer Page: errors in the UDF code are reported on the page. The code must be fixed to eliminate these errors for the sampled rows.
  • Job execution: null values are written as the output
• All errors encountered in executing a UDF using JavaScript v8 are collected and submitted to the Dataprep by Trifacta job logs.

Storage

• Signature information is stored in a dedicated table in the Dataprep by Trifacta database.
• UDF definitions are stored separately as artifacts using the Artifact Storage Service.

Use

Transform Builder

In the Transform Builder:

• You can search for your UDF by name or by entering UDF.
• In a Formula field, you can reference the function by name or by entering UDF.  

UDFs and  Wrangle

You cannot reference   Wrangleobjects from within your UDFs. 

• When specifying your UDFs in the Transform Builder, you can pass in parameters as inputs.
• You can combine UDFs with  Wrangle functions as part of your Formula.