...

In the following sections, you can review short summaries of specific features and explore more detailed information on them.

Single-Flow Operations

These features can be applied to individual flows to simplify job execution.

Parameterization

Parameterization enables you to specify parameters that capture variability in your data source paths or names. For example, you can parameterize the names of folders in your filepaths to capture files within multiple folders. Or, you can parameterize your inputs to capture datasets named within a specific time range. Nested folders of data can be parameterized, too.

...

Parameterization is available for the following:

File systems

For more information, see Overview of Parameterization.

Scheduling

The scheduling feature enables you to schedule the execution of individual flows on a specified frequency. Frequencies can be specified through the 

...

For more information, see Overview of Scheduling.

Job

...

Monitoring

After a job has been launched, detailed monitoring permits you to track the progress of your job during all phases of execution. Status, job stats, inputs, outputs and a flow snapshot are available through the

Email

...

Notifications

After a job has completed, you can send email notifications to stakeholders based on the success or failure of the job.These notifications are defined within Flow View. See Email Notifications Page.

Webhooks

Webhook notifications let you define outgoing HTTP messages to any REST API. The message form and body can be customized to include job execution metadata. For more information, see  Create Flow Webhook Task.

Orchestration

Orchestration is a set of functionality that supports the scheduled execution of jobs across multiple flows. These jobs could be external processes, other flows, or even HTTP requests.

Terms

...

Plans

A plan is a sequence of tasks

...

executed

...

...

Task types

The following types of tasks are available.

...

Limitations

• You cannot specify parameter overrides to be applied to plans specifically.
  • Plans inherit parameter values from the objects referenced in the plan's tasks.

  • If overrides are applied to flow parameters, those overrides are passed to the plan at the time of flow execution.

    Tip
    Tip: Prior to plan execution, you can specify parameter overrides at the flow level. These values are passed through to the plan for execution. For more information, see Manage Parameters Dialog.

Basic task

You create a plan and schedule it using the following basic task.

1. Create the plan. A plan is the container for definition of the tasks, triggers, and other objects. See Plans Page.
2. In Plan View, you specify the objects that are part of your plan. See Plan View Page.
   1. Schedule: The schedule defines the set of triggers that queue the plan for execution.
      1. Trigger: A trigger defines the schedule and frequency at which the plan is executed. A plan can have multiple triggers (e.g. monthly versus weekly executions).
   2. Task(s): Next, you specify the tasks that are executed in order.

      1. Flow task: A flow task includes the specification of the flow to run and the outputs from the flow to generate.

         Info
         NOTE: You can select the outputs from the recipe that you wish to generate. You do not need to generate all outputs.

         Info
         NOTE: When a flow task is executed, the execution plan works back from the selected outputs to execute all of the recipes required to generate the output, including the upstream dependencies of those recipes.

         See Plan View for Flow Tasks.

      2. HTTP task: An HTTP task is a request issued when it is triggered from the application to a target endpoint. This request supports a variety of API methods. See Plan View for HTTP Tasks.
      3. Slack task: A Slack task is a message between

         D s product

         and a specified Slack channel that is triggered within the plan. See Plan View for Slack Tasks.
      4. Delete task: A Delete task deletes specific files or folders from backend storage. See Plan View for Delete Tasks.
      5. Continue building tasks in a sequence.
3. As needed, you can apply override values to any flow parameters that are included in the tasks of your recipe. These overrides are applied during a plan run. For more information, see Manage Parameters Dialog.
4. To test:
   
   1. Click Run now.
   2. To track progress, click the Runs link.
   3. In the Run Details page, you can track the progress.
   4. The first task is executed and completes, before the second task is started.
   5. Individual tasks are executed as separate jobs, which you can track through the Job History page. See Job History Page.
   6. When the plan has completed, you can verify the results through the Job details page. See Job Details Page.
5. If you are satisfied with the plan definition and your test run, the plan will execute according to the scheduled trigger.

Plan scheduling

Through the Plan View page, you can configure the scheduled executions of the plan. Plan schedules are defined using triggers.

• These schedules are independent of schedules for individual flows.
• You cannot create schedules for individual tasks.

Plan execution

When a plan is triggered for execution, a snapshot of the plan is taken. This snapshot is used to execute the plan. Tasks are executed in the sequence listed in Plan View.

Important notes:

...

At the flow level, you can define webhooks and email notifications that are triggered based on the successful generation of outputs. When you execute a plan containing an output with one of these messages, the message is triggered and delivered to stakeholders.

See "Webhooks" and "Email notifications" above.

Enable

Enable the following setting:

Plans feature

Plan sharing, import, and export must also be enabled.For more information, see Workspace Settings Page.

Logging

For more information on debugging plans, see Diagnose Failed Plan Runsin the