...
- Long or complex recipes
- Consider breaking recipes into smaller steps. You can change recipes together.
- Number of columns in your data
- The entire width of a dataset must be represented in the sample.
Delete unnecessary columns early in your recipe.
Tip Tip: If your data is sourced in relational systems, you can apply optimizations to your imported datasets to pre-filter out columns in your dataset before they are ingested into the system. See Flow Optimization Settings Dialog.
You can also use custom SQL statements to collect only the columns that are needed from source tables. See Create Dataset with SQL.
- Complexity of transformations
- Transformations that blend datasets (join and union) or that perform complex transformations on your dataset (aggregate, window, pivot, etc.) can be expensive to process.
- If your recipe contains too many of them, it can negatively impact job processing. Consider breaking these across multiple recipes instead.
...
If there are issues with job execution that cannot be resolved by reviewing the job log, workspace administrators can download a support bundle, which contains additional log information from the platform. For more information, see Support Bundle Contents.
Job Heartbeat and Timeouts
You can configure the how the
| D s webapp |
|---|
| D s config |
|---|
General
These settings configure heartbeat checks for all jobs initiated from the
| D s webapp |
|---|
| Setting | Description |
|---|---|
| feature.enableJobHeartbeat | When enabled, any job that was running when a restart is executed is explicitly marked as Failed. Default is |
| batchserver.workers.jobHeartbeat.jobHeartbeatDelayedThreshold | This setting in minutes defines the maximum permitted time that a job is allowed to run without returning a valid heartbeat check. Default is |
| jobHeartbeat.jobWaitingMaxThreshold | This setting in minutes defines the maximum permitted time that a job is allowed to run. The default is |
| batchserver.workers.jobHeartbeat.frequencyInMs | Interval between heartbeat checks. Default is |
For
| D s photon |
|---|
| D s photon |
|---|
| D s platform |
|---|
| Setting | Description | |
|---|---|---|
| batchserver.workers.photon.timeoutEnabled | When enabled,
| |
| batchserver.workers.photon.timeoutMinutes | This setting in minutes defines the maximum time that a
|
Running Environments
| D children | ||
|---|---|---|
|
...
