Through the Flow View page, you can access and manage all objects in your flow. For each imported dataset, recipe, or other object in your flow, you can perform a variety of actions to effectively manage flow development and job execution through a single page in the .

Flow View page

The imported datasets in the flow or reference datasets added to the flow are listed on the left side of the screen. Associated with each dataset can be one or more recipes, which are used to transform the source data.

NOTE: Objects marked with a red dot indicate a problem with the object's configuration. Please select the object to begin investigating the error. Error information may be displayed in the right panel.

Datasets:

Recipes:

A recipe is a set of steps to transform source data into the results you desire.

For more information on these objects, see Object Overview.

Select an object from your flow to open an object-specific panel on the right side of the screen.

Tip: You can right-click any object in Flow View to see the list of available actions that appear when you select it and choose from the right panel.

Tip: Double-click any recipe to edit it. See Transformer Page.

Actions:

Rename: Select the name of the object to rename it within the platform. This rename does not apply to the source of the object, if it exists elsewhere. 

Add Datasets: Click to add new datasets to the flow. Details are below.

From the flow's context menu: Make a copy: Create a copy of the flow.


NOTE: The copied flow is independent of the source flow, but the original source datasets are connected.

Move to: Move the flow to a new or existing folder. See Manage Flows with Folders.

Edit Flow name and description:  (Available to flow owner only) Change the name and description of the flow.

Delete Flow:  (Available to flow owner only) Delete the flow.

Deleting a flow removes all recipes that are contained in the flow. If copies of these objects exist in other flows, they are not touched. Imported datasets are not deleted by this action.

Add Datasets to Flow

From the Flow View page, you can add imported or reference datasets to your flow. These datasets are added as independent objects in the flow and can be joined, unioned, or referenced by other datasets in the flow.

Add datasets to current flow

View for Imported Datasets

When you select an imported dataset, you can preview the data contained in it, replace the source object, and more from the right-side panel.

Imported Dataset view

Key Fields:

Actions:




View for Recipes

For each recipe, you can review or edit its steps or create new recipes altogether. You can also create references to the recipe, modify outputs, and create new recipes off of the recipe.

When you select a recipe:

Recipe view

Actions:

Recipe tab

Preview the first steps in the recipe.

Key Fields:

Data tab

Preview the data as reflected by the recipe.

NOTE: To render this data preview, some of the data must be loaded, and all steps in the recipe must be executed to generate the preview. Some delays may be expected.

Key Fields:

Target tab

When a target has been assigned for this recipe, you can review its schema information in the Target tab. This tab appears only after a target has been assigned to the recipe.

To remove the current target, select Remove Target from the context menu.

Columns:

View for Outputs

Associated with each recipe is one or more outputs, which are publishing destinations. Through outputs, you can execute and track jobs for the related recipe.

Destinations tab

The Destinations tab contains all configured destinations associated with the recipe. 



Destinations tab

Key Fields:  

For more information, see  Run Job Page.Actions:

Jobs tab

Jobs tab

Each entry in the Jobs tab identifies a job that has been queued for execution for the selected output. You can track the progress, success, or failure of execution. When a job has finished execution you can review the results. Click the link to the job. For more information, see Job Details Page.

Actions:

For a job, you can do the following:

View Results: Click to view the results of your completed job. For more information, see Job Details Page.

Cancel job: Select to cancel a job that is currently being executed.

Delete job: Delete the job from the platform.

Deleting a job cannot be undone.


View for References

When you select a recipe, you can choose to create a reference dataset off of that recipe. A reference dataset is a dataset that is a reference to the output generated from a recipe contained in another flow. Whenever the upstream recipe and its output data are changed, the results are automatically inherited through the reference to the reference dataset. 

NOTE: You cannot select or use a reference dataset until a reference has been created in the source flow from the recipe to use.

To create a reference dataset from a recipe, click the Paper Clip icon. The following options appear in the right panel.

Reference view

Key Fields:

Used In: Indicates the number of flows where the reference appears. If this number is greater than one, click More details to review the flows. See Dataset Details Page.

Actions:

Add to Flow: Click to add the reference dataset to a new or existing flow.

Edit name and description: (Available to flow owner only) Change the name and description for the object.

 

Delete Reference Dataset: Remove the reference dataset from the flow.

Deleting a reference dataset in the source flow causes all references to it to be broken in the flows where it is referenced. These broken references should be fixed by swapping in new sources.

View for Unstructured Datasets

An unstructured dataset is an imported dataset that does not contain any initial parsing steps. All parsing steps must be added through recipes that are applied to the dataset. 

Tip: You can remove initial parsing during import or through the context menu for an imported dataset. See Initial Parsing Steps.

Unstructured Dataset view

Key Fields:

Data Preview: In the Data Preview window, you can see a small section of the data that is contained in the imported dataset. This window can be useful for verifying that you are looking at the proper data.

Tip: Click the preview to open a larger dialog, where you can select and copy data.

Type: Indicates where the data is sourced or the type of file.

File Size: Size of the file. Units may vary.

Location: Path to the location of the imported dataset.

Actions:

View for Reference Datasets

reference dataset is a reference to a recipe's outputs that has been added to a flow other than the one where the recipe is located.

NOTE: A reference dataset is a read-only object in the flow where it is referenced. You cannot select or use a reference dataset until a reference has been created in the source flow from the recipe to use. See View for Recipes above.

To add a reference dataset, you can:

From the source flow, select the reference object for a recipe. In the context panel, click Add to Flow....

Click Add Datasets from the main Flow View page and select one from a different flow.


View for referenced dataset in a new flow

NOTE: Reference datasets marked with a red dot no longer have a source dataset for them in the other flow. These upstream dependencies should be fixed. See Fix Dependency Issues.

When you select a reference dataset in flow view, the following are available in the right-hand panel.

Key Fields:

Source Flow: Flow that contains the dataset. Click the link to open the Flow View page for that dataset.

Actions:

Add new Recipe: Add a new recipe for the object. If a recipe already exists for it, this new recipe is created as a branch in the flow.

Remove from Flow: Remove the reference dataset from the flow. The source dataset in the other flow is untouched.

Go to original reference: Open in Flow View the flow containing the original dataset for this reference.