On January 27, 2021, Google is changing the required permissions for attaching IAM roles to service accounts. If you are using IAM roles for your Google service accounts, please see Changes to User Management.
After you have finished developing the recipes in your flow, you can define scheduled executions of the recipe or recipes within the flow to deliver outputs to known locations. Using the Automator, you can automate execution of jobs on source data, which can be replenished with fresh data asynchronously.
NOTE: Before you begin, you should verify that your data management pipeline into and out of the platform has been appropriately defined. This pipeline includes how data is written to the output location. For more information, see Overview of Automator.
When you schedule a job, you create two objects in Flow View:
NOTE: You must create both of these objects to schedule a job execution.
|Schedule||A schedule applies to the entire flow. It contains one or more intervals at which the recipes of the flow are executed. Recipes are executed if their outputs include a scheduled destination.|
|Scheduled destination||A scheduled destination is an output location, format, and other settings that is populated with the results of executing the related recipes.|
Tip: You can create schedules for datasets with parameters. Any overrides specified through Flow View are automatically applied at runtime for the scheduled job. See Flow View Page.
Notes on scheduled Cloud Dataflow jobs:
NOTE: If your scheduled flow uses an imported output object where Cloud Dataflow execution overrides have been specified, then the overrides are applied to the scheduled job. Property values that do not appear in the imported output object are taken from the Project Settings page. See Dataflow Execution Settings.
Add a Schedule
- Open the flow in Flow View.
- From the context menu, select Schedule.
- In the Add Schedule dialog, select your scheduling options:
- Timezone: Select the timezone to use to determine when to execute the specified schedule.
Frequency: Select the time and frequency of execution: Hourly, Daily, Weekly, Monthly, or cron.
NOTE: Scheduling supports a modified version of cron scheduling syntax. For more information, see cron Schedule Syntax Reference.
To add another scheduled time, click Add.
- To save your schedule, click Save.
- A Calendar icon appears in Flow View to indicate that the flow has a schedule associated with it.
For more information, see Add Schedule Dialog.
Schedule a Destination
- To specify a destination for your schedule, click the recipe you wish to execute at the scheduled time.
- If you have not done so already, click the Output icon next to the recipe to create an output for it.
- In the right panel, locate the Scheduled destinations header. Click Add.
Specify an output location, format, and updating method.
For more information, see Run Job Page.
- To edit the scheduled times, click the Calendar icon. Then, click Edit. Make changes as needed and save.
- To edit a scheduled destination, select the output in Flow View. In the right panel, click Edit next to the appropriate scheduled destination.
- To disable a schedule you control, click the Calendar icon in Flow View. Then, move the slider to disable it.
NOTE: If you receive an email indicating
We have disabled one of your schedules when you did not choose to disable it, you need to login to
Cloud Dataprep by TRIFACTA INC., which automatically refreshes your access token. Then, you can re-enable your schedule through Flow View.
Delete a schedule
In Flow View, click the Calendar icon. Then, click Delete.
Delete a destination
Tip: If you have deleted the schedule for the flow, you do not need to delete the scheduled destination. It cannot run without a schedule.
- In Flow View, select the output.
- In the right panel, select Delete Output from the context menu.
This page has no comments.