Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • deployment is a packaging mechanism for versioning your production-level flows. 
  • Deployments are created and managed through a separate interface. 
  • For more information, see Overview of Deployment Manager

D s api baseurl

Run Job Endpoints

Depending on the type of job that you are running, you must use one of the following endpoints:

Run job

Run a job to generate the outputs from a single recipe in a flow.

Endpoint/v4/jobGroups/:id
MethodPOST
Reference documentation


D s api refdoclink
operation/runJobGroup


Run flow

Run all outputs specified in a flow. Optionally, you can run all scheduled outputs. 

Endpoint/v4/flows/:id/run
MethodPOST
Reference documentation


D s api refdoclink
operation/runFlow


Run deployment

Run the primary flow in the active release of the specified deployment.

Tip

Tip: This method is covered on this page.

Deployments are available only through the Deployment Manager. For more information, see Overview of Deployment Manager.

Endpoint/v4/deployments/:id/run
MethodPOST
Reference documentation


D s api refdoclink
operation/runDeployment


Pre-requisites

Before you begin, you should verify the following:

...

In the following example, the deployment identifier is 2.

Endpoint<protocol>://<platform_base_url>/v4/deployments/2/run
AuthenticationRequired
MethodPOST
Request Body

None.

Response Code201 - Created
Response Body


Code Block
{
    "data": [
        {
            "reason": "JobStarted",
            "sessionId": "dd6a90e0-c353-11e7-ad4e-7f2dd2ae4621",
            "id": 33
        }
    ]
}


If the 201 response code is returned, then the deployment job has been queued for execution. 

...

The first step is to acquire all of the active outputs for the deployment.

Endpoint<protocol>://<platform_base_url>/v4/deployments/2/activeoutputs
AuthenticationRequired
MethodGET
Request Body

None.

Response Code200 - OK
Response Body


Code Block
{
    "data": [
        {
            "outputObjectId": 6,
            "flowNodeId": 27,
            "recipeName": "USDA_Farmers_Market_2014"
        }
    ]
}


in the above response:

  • The flowNodeId value corresponds to the recipe (wrangledDataset) identifier that you wish to modify.
  • The outputObjectId value corresponds to the output object that is produced by default from the recipe.

...

The next step is to review the output object to determine what needs to be overridden. Since you are overriding a file-based publication, you can query directly for the writeSettings objects associated with the output object:

Endpoint<protocol>://<platform_base_url>/v4/outputObjects/6/writeSettings
AuthenticationRequired
MethodGET
Request Body

None.

Response Code200 - OK
Response Body


Code Block
{
    "data": [
        {
            "delim": ",",
            "id": 17,
            "path": "hdfs://hadoop:50070/example/joe@example.com/USDA_Farmers_Market_2014.csv",
            "action": "create",
            "format": "csv",
            "compression": "none",
            "header": true,
            "asSingleFile": true,
            "prefix": null,
            "suffix": "_increment",
            "hasQuotes": true,
            "createdAt": "2019-11-05T18:26:31.972Z",
            "updatedAt": "2019-11-05T18:30:56.756Z",
            "creator": {
                "id": 2
            },
            "updater": {
                "id": 2
            },
            "outputObject": {
                "id": 6
            }
        },
        {
            "delim": ",",
            "id": 16,
            "path": "hdfs://hadoop:50070/example/joe@example.com/USDA_Farmers_Market_2014.json",
            "action": "create",
            "format": "json",
            "compression": "none",
            "header": false,
            "asSingleFile": false,
            "prefix": null,
            "suffix": "_increment",
            "hasQuotes": false,
            "createdAt": "2019-11-05T18:26:44.983Z",
            "updatedAt": "2019-11-05T18:30:56.743Z",
            "creator": {
                "id": 2
            },
            "updater": {
                "id": 2
            },
            "outputObject": {
                "id": 6
            }
        }
    ]
}


Run deployment with overrides

...

From the activeOutputs endpoint, you retrieved the flowNodeId (27).

Endpoint<protocol>://<platform_base_url>/v4/jobGroups/
AuthenticationRequired
MethodPOST
Request Body


Code Block
{
  "wrangledDataset": {
    "id": 27
  },
  "overrides": {
    "profiler": false,
    "writesettings": [
      {
        "path": "hdfs://hadoop:50070/example/joe@example.com/final/USDA_Farmers_Market.csv",
        "format": "csv",
        "action": "overwrite"
      },
      {
        "path": "hdfs://hadoop:50070/example/joe@example.com/final/USDA_Farmers_Market.json",
        "format": "json",
        "action": "overwrite"
      }
    ]
  },
  "ranfrom": null
}


Response Code201 - Created
Response Body


Code Block
{
    "sessionId": "b29467c3-fc59-499e-aed6-d797746a86eb",
    "reason": "JobStarted",
    "jobGraph": {
        "vertices": [
            10,
            11,
            12
        ],
        "edges": [
            {
                "source": 10,
                "target": 11
            },
            {
                "source": 10,
                "target": 12
            }
        ]
    },
    "id": 4,
    "jobs": {
        "data": [
            {
                "id": 10
            },
            {
                "id": 11
            },
            {
                "id": 12
            }
        ]
    }
}



Tip

Checkpoint: Your job with overrides has been queued for execution.

...

All other default values are applied. In the following example, the deployment identifier is 2.

Endpoint<protocol>://<platform_base_url>/v4/deployments/2/run
AuthenticationRequired
MethodPOST
Request Body


Code Block
{
  "sparkOptions": [
    {
      "key": "spark.executor.cores",
      "value": "2"
    },
    {
      "key": "spark.executor.memory",
      "value": "4GB"
    }
  ]
}


Response Code201 - Created
Response Body


Code Block
{
    "data": [
        {
            "reason": "JobStarted",
            "sessionId": "dd6a90e0-c353-11e7-ad4e-7f2dd2ae4621",
            "id": 33
        }
    ]
}


If the 201 response code is returned, then the deployment job has been queued for execution. 

...