Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DEV and version r093

D toc

...

D s support

...

D beta

Info

NOTE: This feature is available for

D s product
productdp
editions only.

...

Prerequisites

Warning

Creation of UDFs requires JavaScript development experience.

...

Access level for JavaScript UDF feature and UDF code is determined by the User defined functions privilege. 

Tip

Tip: The Default role is automatically assigned the Author level of access for the User defined functions privilege. This privilege level is sufficient for creating and editing JavaScript UDFs.

Tip

Tip: To create UDFs, you must have the Author level for your User defined functions privilege in one of your account roles. The default role provides Author level access, by default.

For more information, see  Overview of Authorization 

For more information, see Privileges and Roles Reference.

Enable

This feature is enabled by default. See Dataprep Project Settings Page.

Limitations

Info

NOTE:  If execution of the UDF fails on one or more rows, the values in the affected rows are empty, and the job is permitted to continue execution.

The following limitations apply to JavaScript UDFs:

  • JavaScript UDFs execute function calls for each row of data. 
    • UDFs cannot guarantee maintaining global variable state across an entire sorted dataset that is being processed.
    • Aggregate functions across multiple rows are not supported.
  • Each UDF can add a new column or edit an existing column's values. Other types of data transformations are not supported.
  • Functions cannot be implemented to make external calls over the network. External API calls are not supported.
    • JavaScript libraries cannot be imported for use in creating your UDFs.
  • UDFs cannot be used during pushdown job execution to BigQuery.
  • UDFs cannot be used in macros.
  • UDFs cannot be defined to replace functions in 
    D s lang
  • You cannot reference 

    D s lang
     functions from within your UDF code.

    Tip

    Tip:

    D s lang
    references can be used as inputs to a UDF or in a Formula of which your UDF is a part. Details are below.

  • Export and import of JavaScript UDFs is not supported. 

    Tip

    Tip: JavaScript UDFs can be imported into another project or workspace when they are referenced in a flow that you are importing. See "Importing through Flows" below.

Supported JavaScript

JavaScript implementation

  • If assets are transferred to a new user via API, UDFs are not included in the transfer.

Supported JavaScript

JavaScript implementation

  • Supports only Javascript ES6 version. For more information, see https://www.w3schools.com/js/js_es6.asp.
  • JavaScript V8 libraries are used internally. For more information on licensing, see https://github.com/v8/v8/blob/master/LICENSE.
  • Memory and stack depth limits may impact the number of nested loops or recursive calls that can be executed within a single function.
  • HTTP request objects cannot be referenced from within your JavaScript.
  • JavaScript definitions are not stored in encrypted format.The export keyword can be used for the following definitions only:The UDF signature constant:
    Code Block
    export const signature
    The main function:
    Code Block
    export function trifactaUdf()
    Info

    NOTE: Use of the export keyword in other areas of your code cause failures if the UDF is executed as part of a BigQuery pushdown transformation. Other uses of export are not supported.

Supported JavaScript objects

Fundamental objects:

  • Object

  • Function

  • Boolean
  • Symbol

Error handling:

  • Error
  • AggregateError
  • EvalError
  • InternalError
  • RangeError
  • ReferenceError
  • SyntaxError
  • TypeError
  • URIError

Data types:

  • Number
  • BigInt

Supported JavaScript objects

Fundamental objects:

  • Object

  • Function

  • Boolean
  • Symbol

Error handling:

  • Error
  • AggregateError
  • EvalError
  • InternalError
  • RangeError
  • ReferenceError
  • SyntaxError
  • TypeError
  • URIError

Data types:

  • Number
  • BigInt
  • Math
  • Date
  • String
  • RegExp
  • Array

...

Other supported objects and collections:

  • Map

  • Set
  • WeakMap
  • WeakSet
  • JSON
  • ArrayBuffer
  • DataView
  • Promise
  • Generator
  • GeneratorFunction

Supported JavaScript value properties

  • undefined

  • NaN

  • Infinity
  • isFinite
  • isNaN

Supported JavaScript functions

Parsing functions:

  • parseFloat
  • parseInt

URI encoding functions:

...

The following tables indicate how how

D s item
itemdata types
 are converted to JavaScript data types during input and also for output back to to
D s item
itemdata types
.

For more information on the string values for 

D s item
itemdata types
, see Valid Data Type Strings.

Input type conversions

D s item
itemData Type

JavaScript UDF Data TypeNotes
StringJS String
IntegerJS NumbersJavaScript has no data type for integers. All numeric values are interpreted as JS Numbers.
DecimalJS Numbers
Boolean1/0
Social Security NumberJS String
Phone NumberJS String
Email AddressJS String
Credit CardJS String
GenderJS String
ObjectJS Objects
ArrayJS Arrays
IP AddressJS String
URLJS String
HTTP CodeJS String
Zip CodeJS String
StateJS String
DatetimeJS String

Output data types

On output, JavaScript data types are exported to their corresponding corresponding

D s item
itemdata types
, with the following specific mappings: 

Tip

Tip: The fallback data type is String. Exported String values may require conversion to other

D s item
itemdata types
.

JavaScript UDF Data Type

D s item
itemData Type

Notes
JS StringString
JS Numbers

Integer

Decimal

JavaScript returns values as JS Numbers.

D s webapp
interprets if Integer or Decimal.

1/0Boolean
JS ObjectsObject
JS ArraysArray

Security Measures

To minimize the risk of code execution on the platform, the following security measures have been applied:

  • Some language constructs like eval are not supported.
  • During design time, web workers are used.

  • During runtime (job execution): 
    • v8 Isolates are deployed for execution. For more information, see   https://v8.dev/docs/untrusted-code-mitigations.

    • Jobs are executed in sandboxed compute resources for the following running environments:
      • D s photon
        : remote Kubernetes job
      • D s dataflow
        : customer VPC
      • Users that are not executing the job are isolated from job data and 
        D s item
        itemmetadata
         through UDFs.
  • JavaScript UDFs have execution time limits placed on them to prevent runaway processes. 
  • JavaScript UDFs have other security related restrictions to prevent malicious code injection.
  • Sensitive APIs of the the
    D s platform
     are not exposed in the global object.

Enable

For more information on enabling this feature in your project, please contact 

D s support
.

UDF Requirements

The following sections define the requirements for creating your UDF.

...

The following metadata objects must be defined as part of your JavaScript UDF:

ObjectDescription
Name

Name of your JavaScript function.

  • Only alphanumeric characters and underscores (_) are supported. No special characters.
  • Function names must be unique within the project or workspace.
Tip

Tip: All UDF function names are prepended with UDF., which makes them easier to discover in the Transform Builder. Do not prepend this value or similar here.

DescriptionUser-friendly description of the function. This description appears in the Transform Builder and should be kept to a single line of text.
SignatureThe function's signature defines the inputs that it accepts. See below for more information.
Function definitionThis section defines the custom JavaScript code that is executed when the function is invoked.

Signature

Warning

Changing the signature of a JavaScript UDF that is already in use may break recipe steps and may cause jobs to fail. For example, if you add required inputs to the function, recipe steps become broken. When you make changes to the function's signature, you should review the recipes where the function is in use.

...

  • The signature and function definition must be marked for export in the JavaScript module. The following line must be present at the start of the signature:

    Code Block
    export const signature = {
    Info

    NOTE: Use of the export keyword outside of the UDF signature constant or main function declaration cause failures if the UDF is executed as part of a BigQuery pushdown transformation. Other uses of export are not supported.

  • Individual inputs are specified as elements of an array. Each array element can contain the following attribute and values:

    Individual inputs are specified as elements of an array. Each array element can contain the following attribute and values:

    AttributeDescription
    alias(Required) Unique name of the input. This value can be referenced within the function.
    descriptionUser-friendly description of the input. Appears in the popup that is displayed when the function has been selected.
    displayType

    Text value representing the data type of the input. This value is for display purposes only. Appears in the function popup.

    examplesText value or values to indicate the type of input. Appears in the function popup.
  • The number of inputs in the signature array must equal the number of inputs in the main function.
  • The output of the function is specified according to the 
    D s item
    itemdata type
    . For more information on the values to insert for type, see Valid Data Type Strings.

Function definition

The definition of your function has the following basic structure:

...

  • The signature and function definition must be marked for export in the JavaScript module. The following line must be present at the start of the signature:

    Code Block
    export function trifactaUdf(<any_inputs>) {
    
    • In the above, the inputs to the function must match in name the values for each alias in in the signature. 

  • The values returned from the function are specified as part of the return entry entry.

Type inference

JavaScript is a dynamically typed language. As a result, actual data types are assigned to inputs based on type inferencing performed by the 

D s webapp
 during design time.

Flow parameters are always passed into the function for execution as String execution as String values and must explicitly cast to other data types within the function.

...

This example computes net present value.

export const signature = { "inputs": [ { "alias": "R", "description": "Cash flow at time t", "displayType": "Integer", "examples": [ "COL1" ] }, { "alias": "i", "description": "Discount rate as a decimal value", "displayType": "Float", "examples": [ "COL2" ] }, { "alias": "t", "description": "Number of time units", "displayType": "Float", "examples": [
Code Block
languagejs
export const signature = {
  "inputs": [   
   {
      "alias": "R",
      "description": "Cash flow at time t",
      "displayType": "Integer",
      "examples": [
        "COL1"
      ]
    },
   {
      "alias": "i",
      "description": "Discount rate as a decimal value",
      "displayType": "Float",
      "examples": [
        "COL2"
      ]
    },
    {
      "alias": "t",
      "description": "Number of time units",
      "displayType": "Float",
      "examples": [
        "COL3"
      ]
    }
  ],
  "output": {
    "type" : "Float"
  }
};


export function trifactaUdf(R, i, t) {
   let discount = Math.pow(1 + i, t);  
   return (R / discount);
}

Example - Levenshtein distance

The following code example implements the Levenshtein distance algorithm for calculating the similarity of two strings. For more information, see https://en.wikipedia.org/wiki/Levenshtein_distance.

Code Block
languagejs
export const signature = {
  "inputs": [   
   {
      "alias": "str1",
      "description": "This is String 1 to compare.",
      "displayType": "String",
      "examples": [
        COL1
      ]
    },
    {
      "alias": "str2",
      "description": "This is String 2 to compare.",
      "displayType": "String",
      "examples": [
        COL2
      ]
    }
  ],
  "output": {
    "type" : "Float"
  }
};

 
export function trifactaUdf(str1, str2) {
   const track = Array(str2.length + 1).fill(null).map(() =>
   Array(str1.length + 1).fill(null));
   for (let i = 0; i <= str1.length; i += 1) {
      track[0][i] = i;
   }
   for (let j = 0; j <= str2.length; j += 1) {
      track[j][0] = j;
   }
   for (let j = 1; j <= str2.length; j += 1) {
      for (let i = 1; i <= str1.length; i += 1) {
         const indicator = str1[i - 1] === str2[j - 1] ? 0 : 1;
         track[j][i] = Math.min(
            track[j][i - 1] + 1, // deletion
        "COL3"
    track[j - 1][i]
 + 1, // }insertion
   ],
  "output": {
    "type" : "Float"
  }
]


export function trifactaUdf(R, i, t) { track[j - 1][i - 1] + indicator, // substitution
   let discount = Math.pow(1 + i, t);
      }
   return}
  (R / discount);
}

Other examples

...

return track[str2.length][str1.length];
}

Notes:

  • The signature indicates two input strings.  

Deploy Your UDF

User Defined Functions page

When you have created your UDF in JavaScript, you can upload it to your project in the User Defined Functions page. Select Select Library > User defined  functions. 

See User Defined Functions Page.

...

When this flow is imported into a new project or workspace:

  • If no UDF exists with same name in the target system, then a new UDF is created.
  • If a UDF already exists with same name, then you can choose one of the following: 
    • Use the existing UDF in the new project or workspace,
    • Overwrite the existing UDF with the UDF in the flow import,

      Info

      NOTE: You must have Editor or better privileges on the existing UDF to overwrite it during import.

...

  • When specifying your UDFs in the Transform Builder, you can pass in parameters as inputs.
  • You can combine UDFs with 
    D s lang
     functions as part of your Formula.

D s also
inCQLtrue
label((label = "udf") OR (label = "javascript"))