API Workflow - Run Deployment

This section describes how to run a deployment using the APIs available in  Designer Cloud Enterprise Edition.

• A deployment is a packaging mechanism for versioning your production-level flows. 
• Deployments are created and managed through a separate interface. 
• For more information, see Overview of Deployment Manager. 

A note about API URLs:

In the listed examples, URLs are referenced in the following manner:

<protocol>://<platform_base_url>/

In your product, these map references map to the following:

<http or https>://<hostname>:<port_number>/

For more information, see API Reference.

Pre-requisites

Before you begin, you should verify the following:

1. Get authentication credentials. As part of each request, you must pass in authentication credentials to the Designer Cloud Powered by Trifacta platform. 

   Tip: The recommended method is to use an access token, which can be generated from the Designer Cloud application. For more information, see Access Tokens Page.

   For more information, see API Authentication.

2. Develop your flow. Before you deploy a flow through the Deployment Manager, you should build and test your flow in a development environment. See API Workflow - Develop a Flow.
3. Deploy your flow. After you have developed a flow, you can deploy it. See API Workflow - Deploy a Flow.

4. Acquire deployment identifier. In Deployment Manager, acquire the numeric value for the deployment. See Deployment Manager Page.

Step - Run Deployment

Through the APIs, you can run a deployment. When a deployment is executed:

• All of the recipes that are included in the active release of the deployment are executed. 
• All default values for outputs are applied.

In the following example, the deployment identifier is 2.

{
    "data": [
        {
            "reason": "JobStarted",
            "sessionId": "dd6a90e0-c353-11e7-ad4e-7f2dd2ae4621",
            "id": 33
        }
    ]
}

If the 201 response code is returned, then the deployment job has been queued for execution. 

For more information, see API Workflow - Deploy a Flow.

Step - Monitoring Your Deployment Job

You can monitor the status of your deploy job through the following endpoint using the id value that was returned in the previous step:

<protocol>://<platform_base_url>/v4/jobGroups/<id>/status

For more information, see API Workflow - Run Job.

Step - Run Deployment with Overrides

When you run a deployment, you cannot apply overrides to the request. However, you can complete the following steps to apply overrides when you execute the jobs within the active release. In this workflow, you run jobs on the specific recipes of the active deployment, applying overrides as needed.

Suppose you are running the jobs for deployment id 2, and you want to apply some overrides.

Acquire the active outputs for the deployment

The first step is to acquire all of the active outputs for the deployment.

{
    "data": [
        {
            "outputObjectId": 6,
            "flowNodeId": 27,
            "recipeName": "USDA_Farmers_Market_2014"
        }
    ]
}

in the above response:

• The flowNodeId value corresponds to the recipe (wrangledDataset) identifier that you wish to modify.
• The outputObjectId value corresponds to the output object that is produced by default from the recipe.

Acquire output object information

The next step is to review the output object to determine what needs to be overridden. Since you are overriding a file-based publication, you can query directly for the writeSettings objects associated with the output object:

{
    "data": [
        {
            "delim": ",",
            "id": 17,
            "path": "hdfs://hadoop:50070/example/joe@example.com/USDA_Farmers_Market_2014.csv",
            "action": "create",
            "format": "csv",
            "compression": "none",
            "header": true,
            "asSingleFile": true,
            "prefix": null,
            "suffix": "_increment",
            "hasQuotes": true,
            "createdAt": "2019-11-05T18:26:31.972Z",
            "updatedAt": "2019-11-05T18:30:56.756Z",
            "creator": {
                "id": 2
            },
            "updater": {
                "id": 2
            },
            "outputObject": {
                "id": 6
            }
        },
        {
            "delim": ",",
            "id": 16,
            "path": "hdfs://hadoop:50070/example/joe@example.com/USDA_Farmers_Market_2014.json",
            "action": "create",
            "format": "json",
            "compression": "none",
            "header": false,
            "asSingleFile": false,
            "prefix": null,
            "suffix": "_increment",
            "hasQuotes": false,
            "createdAt": "2019-11-05T18:26:44.983Z",
            "updatedAt": "2019-11-05T18:30:56.743Z",
            "creator": {
                "id": 2
            },
            "updater": {
                "id": 2
            },
            "outputObject": {
                "id": 6
            }
        }
    ]
}

Run deployment with overrides

Now that you have access to the outputs generated by the deployment, you decide to override the following for each file:

• Filename:
  • Remove the year information at the end of the filename
  • Store in a separate folder called final
• Wipe the table and reload it each time (action=overwrite)
• Disable profiling

From the activeOutputs endpoint, you retrieved the flowNodeId (27).

{
  "wrangledDataset": {
    "id": 27
  },
  "overrides": {
    "profiler": false,
    "writesettings": [
      {
        "path": "hdfs://hadoop:50070/example/joe@example.com/final/USDA_Farmers_Market.csv",
        "format": "csv",
        "action": "overwrite"
      },
      {
        "path": "hdfs://hadoop:50070/example/joe@example.com/final/USDA_Farmers_Market.json",
        "format": "json",
        "action": "overwrite"
      }
    ]
  },
  "ranfrom": null
}

{
    "sessionId": "b29467c3-fc59-499e-aed6-d797746a86eb",
    "reason": "JobStarted",
    "jobGraph": {
        "vertices": [
            10,
            11,
            12
        ],
        "edges": [
            {
                "source": 10,
                "target": 11
            },
            {
                "source": 10,
                "target": 12
            }
        ]
    },
    "id": 4,
    "jobs": {
        "data": [
            {
                "id": 10
            },
            {
                "id": 11
            },
            {
                "id": 12
            }
        ]
    }
}

You can use the job identifier (4) to monitor the job status.