Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DEV and version next

...

A 

D s lang
 recipe is a sequence of transforms applied transformation steps applied to your dataset in order to produce your results.

Transform and transformation:

  • transform is  is a single action applied to a dataset. For most transformsA transform is part of the underlying 
    D s lang
    . Transforms are not directly accessible to users. 
  • transformation is a user-facing action that you can apply to your dataset through the Transformer page. A transformation is typically a use-specific or more sophisticated manifestation of a transform. 

    D s transforms

    Tip

    Tip: Except for the reference documentation for individual transforms, the language documentation references transformations that you can apply through the Transformer page.

For most of these actions, you can pass one or more parameters to define the context (columns, rows, or conditions).

Function:

  • Some parameters accept one or more functions. A function is a computational action performed on one or more columns of data in your dataset.
  • Recipes are built in the Transformer Page. See Transformer Page.

When you select suggestions in the Transformer Page, your selection is converted into a

D s lang
 command and added a transformation that you can add to your recipe.

Tip

Tip: Where possible, you should make selections in the data grid to build transform transformation steps. These selections prompt a series of cards to be displayed. You can select different cards to specify a basic transform transformation for your selected data, choose a variant of that transformtransformation, and then modify the underlying

D s lang
recipe as necessary. For more information, see Overview of Predictive Transformation.

For more information on the suggestion cards, see Selection Details Panel.

Some complex transformstransformations, such as joins and unions, must be created through dedicated screens. See Transformer Page.

...

  • D s lang
     is a proprietary language designed for data transformation. Every supported transformation is designed to make changes to a dataset. It cannot be used to read from or write to a datastore. 
    • Users interact with 
      D s lang
       exclusively through the the
      D s webapp
      . There is no direct interaction with the language.
  • SQL (Structured Query Language) is designed for querying, transforming, and writing for relational datasources. It cannot be applied to file-based datasets.
    • SQL cannot be used to transform data in 
      D s product
      .

...

Transform ElementDescription
transformA

In

D s lang
, a transform (or verb) is a single keyword that identifies the type of change you are applying to your dataset.

A transform is always the first keyword in a recipe step. Details are below.

D s transforms

The other elements in each step are contextual parameters for the transform. Some transforms do not require parameters.

parameter1:, parameter2:

Additional parameters may be optional or required for any transform. 

Info

NOTE: A parameter is always followed by a colon. A parameter may appear only one time in a transform step.

...

  1. As you build 
    D s lang
     steps in the Transform Builder, your syntax is validated for you. You cannot add steps containing invalid syntax. 
    1. Error messages are reported back to the application, so you can make immediate modifications to correct the issue.
    2. Type-ahead support can provide guidance to the supported transforms, functions, and column references.
    3. For more information, see Transform Builder.
  2. When you have entered a valid transform step, the results are previewed for you in the data grid.
    1. This preview is generated by applying the transform transformation to the sample in the data grid. 

      Info

      NOTE: The generated output applies only to the values displayed in the data grid. The function is applied across the entire dataset only during job execution.

    2. If the previewed transform transformation is invalid, the data grid is grayed out.
    3. For more information, see Transform Preview.
  3. When you add the transform transformation to your recipe: 
    1. It is applied to the sample in the application, and the data grid is updated to the current state.
    2. Column histograms are updated with new values and counts.
    3. Column data types may be re-inferred for affected columns.
  4. Making changes:
    1. You can edit any transform transformation step in your recipe whenever needed.
      1. When you edit a transform transformation step in your recipe, the context of the data grid is changed to display the state of your data up to the point of previewing the step you're editing. 
      2. All subsequent steps are still part of the recipe, but they are not applied to the sample yet.
      3. You can insert recipe steps between existing steps.
    2. When you delete a recipe step, the state remains at the point where the step was removed.
      1. You can insert a new step if needed.
      2. When you complete your edit, select the final step of the recipe, which displays the results of all of your transform transformation steps in the data grid. Your changes may cause some recipe steps to become invalid.
    3. See Recipe Panel.

Transforms

...

Transformation

A transformation is an action applied to rows or columns of your data. Transforms are the essential set of changes that you can apply to your dataset.for which you can browse or search through the Transform Builder in the Transformer page. When specified and added to the recipe, these sometimes complex actions are rendered in the recipe as steps using the underlying transforms of the language. 

Tip

Tip: Through transformations,

D s product
guides you through creation of more sophisticated steps that would be difficult to create in raw
d-s-

...

lang
.

For more information on the list of available transformations, see TransformsTransformation Reference.

For more information on creating transformation steps in the Transformer page, see Transform Builder.  

Function Categories

A function is an action that is applied to a set of values as part of a transform step. Functions can apply to the values in a transform for specific data types, such as strings, or to types of transforms, such as aggregate and window function categories. A function cannot be applied to data without a transform.

...

Operator CategoryDescription
Logical Operatorsand, or, and not operators
Numeric OperatorsAdd, subtract, multiply, and divide
Comparison OperatorsCompare two values with greater than, equals, not equals, and less than operators
Ternary OperatorsUse ternary operators to create if/then/else logic in your transforms.

Transforms

 transform, or verb, is an action applied to rows or columns of your data. Transforms are the essential set of changes that you can apply to your dataset.

D s transforms

Transforms are described in the Language Appendices. For more information, see Transforms.

Documentation

Documentation for 

D s lang
 is also available through 
D s product
. Select Help menu > Documentation.

...