Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DEV and version r0682

D toc

D s api refdoclink
operation/runJobGroup

This section describes how to run a job using the APIs available in

D s product
rtrue
.

...

Endpoint<protocol>://<platform_base_url>/v4/jobGroups/<id>/status
AuthenticationRequired
MethodGET
Request BodyNone.
Response Code200 - Ok
Response Body
Code Block
{
    "id": 961247,
    "name": null,
    "description": null,
    "ranfrom": "ui",
    "ranfor": "recipe",
    "status": "Complete",
    "profilingEnabled": true,
    "runParameterReferenceDate": "2019-08-20T17:46:27.000Z",
    "createdAt": "2019-08-20T17:46:28.000Z",
    "updatedAt": "2019-08-20T17:53:17.000Z",
    "workspace": {
        "id": 22
    },
    "creator": {
        "id": 38
    },
    "updater": {
        "id": 38
    },
    "snapshot": {
        "id": 774476
    },
    "wrangledDataset": {
        "id": 28629
    },
    "flowRun": null
}

...

  1. Acquire the internal identifier for the recipe for which you wish to execute a job. In the previous example, this identifier was 28629.
  2. Construct a request using the following:

    Endpoint<protocol>://<platform_base_url>/v4/jobGroups
    AuthenticationRequired
    MethodPOST

    Request Body:

    Code Block
    {
      "wrangledDataset": {
        "id": 28629
      },
      "overrides": {
        "profiler": true,
        "execution": "spark",
        "writesettings": [
          {
            "path": "<new_path_to_output>",
            "format": "csv",
            "header": true,
            "asSingleFile": true
          }
        ]
      },
      "ranfrom": null
    }
    
  3.  In the above example, the job has been launched with the following overrides:
    1. Job will be executed on the Spark cluster. Other supported values depend on your deployment:

      Value for overrides.executionDescription
      photon

      Running environment on 

      D s node

      sparkSpark on integrated cluster, with the following exceptions.
      databricksSpark

      Spark on Azure Databricks

      emrSpark

      Spark on AWS EMR

      For more information, see Running Environment Options.

    2. Job will be executed with profiling enabled.
    3. Output is written to a new file path.
    4. Output format is CSV to the designated path. 
    5. Output has a header and is generated as a single file.
  4. A response code of 201 - Created is returned. The response body should look like the following:

    Code Block
    {
    
        "sessionId": "79276c31-c58c-4e79-ae5e-fed1a25ebca1",
        "reason": "JobStarted",
        "jobGraph": {
            "vertices": [
                21,
                22
            ],
            "edges": [
                {
                    "source": 21,
                    "target": 22
                }
            ]
        },
        "id": 962221,
        "jobs": {
            "data": [
                {
                    "id": 21
                },
                {
                    "id": 22
                }
            ]
        }
    }
  5. Retain the id value, which is the job identifier, for monitoring.

...

  • Path to database to which to write (must have write access)
  • Connection to write to the target.

    Tip

    Tip: This identifier is for the connection used to write to the target system. This connection must already exist. For more information on how to retrieve the identifier for a connection, see API Connections Get List v4.

  • Name of output table
  • Target table type

    Tip

    Tip: You can acquire the target type from the vendor value in the connection response. For more information, see API Connections Get List v4.

  • action:

    Key valueDescription
    createCreate a new table with each publication.
    createAndLoadAppend your data to the table.
    truncateAndLoadTruncate the table and load it with your data.

    dropAndLoad

    Drop the table and write the new table in its place.
  • Identifier of connection to use to write data.

...

Step - Run Job with Overrides - Webhooks Webhooks

When you execute a job, you can pass in a set of parameters as overrides to generate a webhook message to a third-party application, based on the success or failure of the job.

For more information on webhooks, see see Create Flow Webhook Task.

  1. Acquire the internal identifier for the recipe for which you wish to execute a job. In the previous example, this identifier was 28629.
  2. Construct a request using the following:

    Endpoint<protocol>://<platform_base_url>/v4/jobGroups
    AuthenticationRequired
    MethodPOST

    Request Body:

    Code Block
    {
      "wrangledDataset": {
        "id": 28629
      },
      "overrides": {
        "webhooks": [{
          "name": "webhook override",
          "url": "http://example.com",
          "method": "post",
          "triggerEvent": "onJobFailure",
          "body": {
            "text": "override" 
           },
          "headers": {
            "testHeader": "val1" 
           },
          "sslVerification": true,
          "secretKey": "123",
      }]
     }
    }
  3.  In the above example, the job has been launched with the following overrides:

    Override settingDescription
    nameName of the webhook.
    urlURL to which to send the webhook message.
    methodThe HTTP method to use. Supported values: POST, PUT, PATCH, GET, or DELETE. Body is ignored for GET and DELETE methods.
    triggerEvent

    Supported values: onJobFailure - send webhook message if job fails onJobSuccess - send webhook message if job completes successfully onJobDone - send webhook message when job fails or finishes successfully

    body

    (optional) The value of the text field is the message that is sent.

    Info

    NOTE: Some special token values are supported. See Create Flow Webhook Task.

    header(optional) Key-value pairs of headers to include in the HTTP request.
    sslVerification(optional) Set to true if SSL verification should be completed. If not specified, the value is true.
    secretKey(optional) If enabled, this value should be set to the secret key to use.
  4. A response code of 201 - Created is returned. The response body should look like the following:

    Code Block
    {
    
    
        "sessionId": "79276c31-c58c-4e79-ae5e-fed1a25ebca1",
        "reason": "JobStarted",
        "jobGraph": {
            "vertices": [
                21,
                22
            ],
            "edges": [
                {
                    "source": 21,
                    "target": 22
                }
            ]
        },
        "id": 962222,
        "jobs": {
            "data": [
                {
                    "id": 21
                },
                {
                    "id": 22
                }
            ]
        }
    }
  5. Retain the id value, which is the job identifier, for monitoring.

...