Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DEV and version next

D toc

Excerpt

Explore the objects that you create and their relationships. Flows, imported datasets, and recipes are created to transform your sampled data. After you build your output objects, you can run a job to transform the entire dataset based on your recipe and deliver the results according to your output definitions.

Flow Structure and Objects

...

When you select a recipe's reference object, you can add it to another flow. This object is then added as a reference dataset in the target flow. A reference dataset is a read-only version of the output data generated from the execution of a recipe's steps.

Working with recipes

Recipes are edited in the Transformer page, which provides multiple methods for quickly selecting and building recipe steps.Macros: As needed, you can create reusable sequences of steps that can be parameterized for use in other recipes. For more information, see Overview of Macros

Run Jobs: When you are satisfied with the recipe that you have created in the Transformer page, you can execute a job. A job may be composed of one or more of the following job types:

  • Transform job: Executes the set of recipe steps that you have defined against your sample(s), generating the transformed set of results across the entire dataset.
  • Profile job: Optionally, you can choose to generate a visual profile of the results of your transform job. This visual profile can provide important feedback on data quality and can be a key for further refinement of your recipe.
  • When a job completes, you can review the resulting data and identify data that still needs fixing. See Job Details Page.
  • For more information on the process, see Running Job Basics.

Flow Example

The following diagram illustrates the flexibility of object relationships within a flow. 

...

D caption
typefigure
Flow Example
TypeDatasetsDescription
Standard job executionRecipe 1/Job 1

Results of the job are used to create a new imported dataset (I-Dataset 2).

See
Create dataset from generated resultsRecipe 2/Job 2

Recipe 2 is created off of I-Dataset 2 and then modified. A job has been specified for it, but the results of the job are unused.

 

Chaining datasetsRecipe 3/Job 3

Recipe 3 is chained off of Recipe 2. The results of running jobs off of Recipe 2 include all of the upstream changes as specified in I-Dataset 1/Recipe1 and I-Dataset 2/Recipe 2.

Reference datasetRecipe 4/Job 4I-Dataset 4 is created as a reference off of Recipe 3. It can have its own recipe, job, destinations, and results.

Flows are created in the Flows page. See Flows Page.

Working with recipes

Recipes are edited in the Transformer page, which provides multiple methods for quickly selecting and building recipe steps.
Macros: As needed, you can create reusable sequences of steps that can be parameterized for use in other recipes. For more information, see Overview of Macros

Run Jobs: When you are satisfied with the recipe that you have created in the Transformer page, you can execute a job. A job may be composed of one or more of the following job types:

  • Transform job: Executes the set of recipe steps that you have defined against your sample(s), generating the transformed set of results across the entire dataset.
  • Profile job: Optionally, you can choose to generate a visual profile of the results of your transform job. This visual profile can provide important feedback on data quality and can be a key for further refinement of your recipe.
  • When a job completes, you can review the resulting data and identify data that still needs fixing. See Job Details Page.
  • For more information on the process, see Running Job Basics.