Plans and Tasks

Overview

In many scenarios, data pipelines have dependencies in them. Data sourced from one dataset must be cleaned and delivered for use in another data pipeline. When a pipeline is built, it needs to be scheduled for periodic execution. Ideally, when the execution completes, downstream stakeholders must be informed that the pipeline has succeeded or failed in execution.

In  Designer Cloud, plans provide the mechanism for building these robust data pipelines within the application, in which sequences of flows can be executed to deliver more diverse datasets. As a result of the success or failure of these flow executions, the Designer Cloud application can deliver messages to receiving applications, such as Slack.

Example

In this example, you want to create a plan that executes the following tasks:

1. Execute the flow task that performs the initial cleaning of your data. The results data is saved to a known location.
2. Execute the flow task that transforms the cleaned data for downstream uses. 
3. Based on the outcome of step 2:
   1. On success: Send a success message to a Slack channel.
   2. On failure: Send a failure message to a Slack channel.

In Plan View, this plan might look like the following:

After creating the above plan, you can perform test runs. 

If the tests are successful, you can schedule the plan for periodic execution.

Create a Plan

You build plans in Plan View. See Build a Plan.

Schedule a Plan

You can schedule executions of your plan. See Schedule a Plan Run.