Overview of Job Monitoring
The Designer Cloud Powered by Trifacta platform supports detailed monitoring of a job throughout each phase of its execution.
Limitations:
Applies only to ingest and publishing jobs
Applies to JDBC datasets
Monitoring Phases
These phases apply to ingest and publishing jobs. Information on them is surfaced in the application.
Connect
In the Connect phases, the Designer Cloud Powered by Trifacta platform uses the specified connection for the flow to connect to the source of the job.
Note
Errors in this phase typically involve issues in the connection definition or in the network configuration or availability.
Request
After the platform has been able to connect to the datastore, the Request phase entails the submission of the request to the datastore for the assets. For example, for JDBC-based datasets, this phase covers the SQL query of the database through the response that the query was successfully executed.
Note
Errors in this phase typically reflect errors in the SQL query, which can include renaming or moving of assets in the datastore.
Note
If assets are retrieved via custom SQL query, you may need to review the query and validate it through the Trifacta Application. For more information, see Create Dataset with SQL.
Transfer
This phase covers the transfer of assets from the datastore to the platform.
Note
Errors in this phase typically indicate issued with permissions.
Prepare
Note
This phase applies to publishing jobs only.
Depending on the destination, the Prepare phase includes the creation of temporary tables, generation of manifest files, and the fetching of extra connections for parallel data transfer.
Process
After the data has been transferred to the platform, this phase covers the processing of cleanup after data transfer, including the dropping of temporary tables or copying data within the instance.
Enable
The base feature is enabled by default.
"feature.enableJobMonitoring": true,
Configure
Optionally, you can enable the following capabilities in the Trifacta Application. You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.
Enable phases in Data sources tab
To display separate columns in the Data sources tab of the Job Details page for each phase on an ingest job, set the following parameter to true:
"jobMonitoring.enablePhasesInDatasourcesTable": true,
Enable phases in Outputs tab
To display separate columns in the Outputs tab of the Job Details page for each phase for a publish job, set the following parameter to true:
"jobMonitoring.enablePhasesInOutputsTable": true,
Save your changes and restart the platform.
Monitoring Jobs in the Application
When the base feature is enabled, you can monitor jobs in the following locations.
Flow View
Track phases in the Jobs panel in Flow View. Hover the mouse over the link to the job.
See Flow View Page.
Import
Note
This feature may require enablement in your deployment. See Configure JDBC Ingestion.
Import Data:
For long-loading datasets, you can track the progress of the import through the Import Data page as you specify the import. See Import Data Page.
Library:
After specifying the import, if the data is continuing to be ingested, you can track progress through the Library page. See Library Page.
Dataset Details Page:
In the Dataset Details page, you can monitor the ingest progress. Hover over the Status link.
 |
Figure: Dataset Details Page - Job Monitoring
Job Details Page
Track phases of progress by hovering over the job in progress in the Job Details page.
Review new and better detail in the Job Details page. Click View Details for the job listing.
For more information, see Job Details Page.
If job monitoring phases have been enabled for the Datasources tab, the tab looks like the following:
Figure: Job monitoring in the Datasources tab
View details:
If an ingest job succeeds or fails, you can click View details in the status column for additional information on each phase of the ingest job:
Figure: View details on monitoring ingest jobs
If job monitoring phases have been enabled for the Output Destinations tab, the tab looks like the following:
Figure: Job monitoring in the Datasources tab
View details:
If a publishing job succeeds or fails, you can click View details in the status column for additional information on each phase of the publishing job:
Figure: View details on monitoring ingest jobs