Page tree


Support | BlogContact Us | 844.332.2821

Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DEV and version r0411


The following changes have been applied to 

D s lang
 in this release.

Release 4.1

Standardization page and transform have been removed

For a number of releases, the Standardization page and its related transform have been available via feature flag, due to a number of issues.


NOTE: This feature was not typically enabled in

D s product

This feature has been removed from the product altogether and will be replaced in the future by a more robust standardization and normalization capability. 

Syntax Changes

New Transforms

Transform NameDocumentationNotes
commentComment Transform

In previous releases, you could insert comments of the following format:

Code Block
// This is a comment.

Beginning in Release 4.1, comments are supported by formal transform.

New Functions

Function NameDocumentation

Release 4.0.1

Map data type is now Object data type

The Map data type has been renamed to the Object data type. There are no changes to the behavior.


NOTE: This change is not reflected in the Release 4.0.1 PDF documentation.


Splitrows transform now permits specifying of quote escaping character

For text-based formats that format fields between quotes, you can specify the character that is used to signify the escaping of the quote character in the data. The quoteEscapeChar parameter identifies the character in the data the precedes quotes that are supposed to be part of the data, instead of the marker for a field. See Splitrows Transform.

Release 4.0

Script steps displayed in natural language


  • set and settype transforms now support multiple input columns.
  • When working with multiple columns, set transform now accepts a placeholder variable in the formula. 
  • See Set Transform.
  • See Settype Transform.

Syntax Changes

Terminology Changes

rollingaverage function accepts two windowing parameters.


  • The function name has changed to ROLLINGAVERAGE.
  • The behavior of the first parameter has changed. It captures rows before the current one, instead of rows after the current one.
    • The default values are -1 and 0, which capture all values from the current row back to the first row of the dataset.
  • During the upgrade process, transform steps using this function are automatically migrated to the new method of specification. 
  • For more information, see ROLLINGAVERAGE Function TBD.
Function Changes

The following name changes have been applied to existing functions to use more familiar names.


Release 3.2.1

Syntax Changes

Terminology Changes


NOTE: Beginning in Release 3.2.1, values that are considered empty are now referred to as missing.


This change was made to align the behaviors of the Photon running environment with the JavaScript running environment.

Release 3.2

Syntax Changes

Transform Changes

  • multisplit transform has been deprecated. All multisplit capabilities are now supported by the split transform. See Split Transform.
  • pivot transform now supports multiple columns. See Pivot Transform.
  • unnest transform now requires the keys parameter, which was optional in previous releases. See Unnest Transform.
    • To unnest arrays without specifying keys, use the flatten transform. See Flatten Transform.
  • arraylength and arraystomap now accept functions that return arrays as inputs to the function.
  • domain and subdomain functions have been updated to reflect standard interpretations of domain and sub-domain values for URLs:

    ReleaseExample URLDomain FunctionSubdomain Function
    Release 3.1.2 and earlier
    Code Block
    Code Block
    Code Block
    Release 3.2 and later
    Code Block
    Code Block
    Code Block
  • The following parameter values are no longer supported as special capture groups in the with parameter for the replace transform. These references in the with parameter do not work in the Photon and Hadoop running environments and are unlikely to work at scale in any other running environment:

    Code Block