Page tree

 

Support | BlogContact Us | 844.332.2821

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DEV and version r0411

...

Through the Flow View page, you can access and manage all objects in your flow. For each imported dataset, recipe, or wrangled dataset in your flow, you can perform a variety of actions to effectively manage flow development and job execution through a single page.

Image RemovedImage Added

D caption
Flow View page

The imported datasets in the flow or wrangled datasets added to the flow are listed on the left side of the screen. Associated with each imported dataset can be one or more recipes, which are used to transform the source data into wrangled datasets.

  • These To begin working with an imported recipe, select it and click Add new recipe. A new, empty recipe is associated with the database. To open in the Transformer page, click the recipe icon and select Edit Recipe. See Transformer Page.
  • When created, these objects are connected together by lines flowing between them, which show the relationships between the objects of the flow.
  • For any dataset, any objects on which it depends are displayed to the left of the object on one of the flowing lines leading from the dataset. In the above example, the POS-01 dataset is dependent on all of the objects in the flow, while the REF_CAL dataset is only dependent on its recipe and the REF_CAL.txt imported dataset. 

    Tip

    Tip: When you run a job from a wrangled dataset, all of the recipes steps for the preceding datasets are executed as part of the job, and only the results of the terminal wrangled dataset are generated.

     

    • In the above example, the POS-01 dataset is dependent on all of the objects in the flow, while the REF_CAL dataset is only dependent on its recipe and the REF_CAL.txt imported dataset. 

    • The other datasets have been integrated with the POS-01 dataset and have not yet had a recipe or wrangled dataset created for them. 

  • For more information on these objects, see Object Overview.

Select an object from your flow to open an object-specific panel on the right side of the screen.

Actions:

...

Tip

Tip: You can right-click any object in Flow View to see the list of available actions that appear when you select it and choose from the right panel.

Tip

Tip: Double-click any recipe to open it and its associated wrangled dataset. See Transformer Page.


Actions:

Add Datasets: Add new datasets to the flow.

...

Details are below.

...

Make a copy

...

Create a copy of the flow for another user.

Info

NOTE: The copied flow is

...

independent of the source flow, but the original source datasets are connected.

Edit name and description

...

(Available to flow owner only) Change the name and description of the flow.

...

 

Delete(Available to flow owner only) Delete the flow.

Warning

Deleting a flow removes all wrangled datasets that are contained in the flow. If copies of these datasets exist in other flows, they are not touched. Imported datasets are not deleted by this action.

Add Datasets to Flow

From the Flow View page, you can add imported or wrangled datasets to your flow. These datasets are added as independent objects in the flow and can be joined, unioned, or referenced by other datasets in the flow.

D caption
Add datasets to current flow
  1. Search for or select the dataset to add.
    1. Use the page view controls to browse for other datasets, or select the appropriate tab to filter the list to Wrangled or Imported datasets.
    2. To import new datasets from external sources, click Import Datasets. See Import Dataset Page.
  2. When you have made your selections, click Add.
  3. The dataset is added as a new object in flow view.

    Info

    NOTE: For imported datasets that do not have a published schema, such as (CSV, TXT, LOG, or JSON files), a recipe, including steps for inferring structure, and a wrangled dataset are automatically created as part of the process.

View for Imported Datasets

When you select an imported dataset, you can preview the data contained in it, swap replace the source object, and more from the right-side panel.

D caption
Imported Dataset view

Key Fields:

FieldDescription
Data Preview

In the Data Preview window, you can see a small section of the data that is contained in the imported dataset. This window can be useful for verifying that you are looking at the proper data.

Tip

Tip: You Click the preview to open a larger dialog, where you can select and copy data from this preview window.

TypeIndicates where the data is sourced or the type of file.
File SizeSize in KB.of the file. Units may vary.
LocationPath to the location of the file.Used InCount of flows where the imported dataset is used, including the current one.

 

Actions:

ActionDescription
SwapReplace

Swap out Replace the current source for a new source for the imported dataset. You can swap out any dataset in the flow as long as it has another recipe that appears after it in the flow.

Info

NOTE: This action removes the imported dataset and all connections (edges) coming out of it. The replacement must be reconnected with any downstream objects.

Info

NOTE: If the swapped-in source does not have the same schema as the original source, recipe steps in the current flow and any flow that uses the imported dataset may be broken.

For more information, see Change Dataset BrowserDialog.

Add new RecipeCreate a new recipe and wrangled dataset from the imported dataset. This recipe and dataset combination is independent of the original one.
Edit name and description...Change the name and description of the imported dataset.
Remove structure...If the imported dataset had structuring steps applied to it during import, you can choose to remove these steps. The dataset is converted to a raw dataset. No structuring steps are applied.
Remove

Remove the imported dataset from the flow.

Info

NOTE: Any recipe and wrangled dataset using the imported dataset are also removed. In the Remove Dataset dialog, click Details to review the imported dataset.

More detailsSee Dataset Details Page.

...

For each recipe, you can review or edit its steps or create new recipes altogether.

D caption
Recipe view

Key Fields:

FieldDescription
Steps PreviewPreview the first steps in the recipe.
StepsTotal count of the steps in the recipe.

...

ActionDescription
Edit RecipeOpen the recipe and begin editing. See Transformer Page.Make a Copy
Change input

Change the input dataset associated with the recipe.

Info

NOTE: This action substitutes only the primary input from a recipe, which does not include any datasets that are integrated from joins, unions, lookups, or other multi-dataset options.

Make a copy

Create a copy of the recipe and a new wrangled dataset. You can create the copy with the same inputs or without inputs at all.

The copied recipe is owned by the user who copied it.

Move...Move the recipe to a different flow, or create a new flow to contain it.
Delete

Delete the recipe.

Warning

This step cannot be undone.

See data

View

...

for Wrangled Datasets

...

View for Wrangled Datasets

...

Image Added

D caption
Wrangled Dataset view

Key Fields:

flows is
FieldDescription
Data Preview

In the Data Preview window, you can see a small section of the data that is contained in the wrangled dataset. This window can be useful for verifying that you are looking at the proper data.

Tip

Tip: You Click the preview to open a larger dialog, where you can select and copy data from this preview window. 

SizeCount of columns and data types in the wrangled dataset.Used In
Run
Count of
jobs where the wrangled dataset
has been used.
RanCount of jobs launched for the wrangled dataset.
Last RanTimestamp for when the job last ran.

 

Click the link to review details. See Dataset Details Page.

Actions:

ActionDescription
Edit RecipeEdit the recipe of the wrangled dataset. See Transformer Page.
Run Job

Launch a job for the wrangled dataset, its recipes, and all preceding datasets.

Info

NOTE: This

feature

option is not available in

D s product
productpe
.

Add new RecipeCreate a new recipe and wrangled dataset from the wrangled dataset. This recipe and dataset combination is independent of the original one.
Edit name and description...(Available to flow owner only) Change the name and description for the wrangled dataset.
See recipeView the steps of the recipe associated with the wrangled dataset. See View for Recipes above.
More detailsReview details on the flows where the dataset is used.

Publishing Settings tab

...

D caption
Publishing Settings tab

Destinations:

When you click the Run Job button, these destinations are populated by default in the job specification.

...

View for Referenced Datasets

...

To add a referenced dataset, click Add Datasets from the main Flow View page and select one from a different flow. See Add Datasets to Flow above.

D caption
Referenced Dataset view

Key Fields:

FieldDescription
SizeNumber of columns and data types in the referenced dataset.
Source FlowFlow that contains the dataset. Click the link to open the Flow View page for that dataset.

...