Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DEV and version r0712

D toc

Review the changes to the publicly available REST APIs for the 

D s platform
rtrue
 for the current release and past releases. 

Changes for Release 7.1

Introducing in-app API reference documentation

Beginning in Release 7.1, API reference documentation is available directly from the 

D s webapp
. From the menu bar, select Help menu > API Documentation.

Key features:

  • More endpoints and attributes are now available!
  • Reference content comes directly from the codebase.
  • Searchable and browsable

More content and features will be added to this new capability of the next few releases. 

Info

NOTE: The API reference doc that was published with the product documentation is no longer available. Please use the in-app API reference documentation.


Info

NOTE: The API reference documentation may require enablement in your environment. For more information, see API Reference.


Workflow documentation is still available in the product documentation. For more information on these workflows, see API Reference.

API changes for authorization

Release 7.1 introduces authorization, which provides finer-grained access controls to user-defined objects in the platform. For more information, see Changes to Authorization.

As a result of authorization, the following changes have been applied to the listed API endpoints.

Connection permissions - POST

Endpoint/v4/connections/:id/permissions/
MethodPOST

Change:

When setting permissions, the request body now must include the policyTag parameter:

Info

NOTE: This is required.


Code Block
"policyTag": "connection_viewer",
ParameterDescription
policyTag

Defines the policy level to assign to the change in connection permissions. Accepted values:

  • connection_viewer = can use the connection

For more information, see the API reference documentation:

D s api refdoclink
operation/createConnectionPermission


Connection permissions - GET

Endpoint/v4/connections/:id/permissions/
MethodGET

Change:

Response no longer includes connectionPermissions parameter.

Info

NOTE: Above is removed.

Response does include the following new parameters:

Code Block
"policyTag": "connection_viewer",
"workspaceAdmin": false,
"isCreatedBy": true,
ParameterDescription
policyTag

See previous.

workspaceAdminIf true, then the permitted user is a workspace admin.
isCreatedByIf true, then the permitted user created the connection.

For details on these parameters, please see the API Reference documentation:

D s api refdoclink
operation/getConnectionPermissions

Connections - GET LIST

Endpoint/v4/connections
MethodGET

Change:

Response now includes an authorizationPermission field in the associatedPeople block for each connection:

Code Block
{
    "data": [
        {
            "id": 14,
            ....
            "associatedPeople": {
                "data": [
                    {
                        "id": 1,
                        ...
                        "authorizationPermission": {
                            "policyName": null,
                            "policyTag": "workspace_admin",
                            "resourceOperationList": [
                                {
                                    "operations": [
                                        "execute",
                                        "delete",
                                        "update",
                                        "read",
                                        "create",
                                        "share"
                                    ],
                                    "policyTag": "plan_author",
                                    "resourceType": "plan"
                                },
                                {
                                    "operations": [
                                        "execute",
                                        "delete",
                                        "update",
                                        "read",
                                        "create",
                                        "share"
                                    ],
                                    "policyTag": "flow_author",
                                    "resourceType": "flow"
                                },

For details on these parameters, please see the API Reference documentation:

D s api refdoclink
operation/listConnections


Connections - GET

Endpoint/v4/connections/:id
MethodGET

Change:

Same change as previous.

For details on these parameters, please see the API Reference documentation:

D s api refdoclink
operation/getConnection

People - GET LIST

Endpoint/v4/people
MethodGET

Change:

Response now includes the following attributes:

  • maximalPrivileges
  • authorizationRoles
Info

NOTE: Above attributes appear only if the authenticating user is an administrator or the specific user.


Code Block
{
    "data": [
        {
            "id": 29,
            ....
           "maximalPrivileges": [
                {
                    "operations": [
                        "delete",
                        "update",
                        "read",
                        "create",
                        "share"
                    ],
                    "policyTag": "connection_author",
                    "resourceType": "connection"
                },
                {
                    "operations": [
                        "execute",
                        "delete",
                        "update",
                        "read",
                        "create",
                        "share"
                    ],
                    "policyTag": "plan_author",
                    "resourceType": "plan"
                },
                {
                    "operations": [
                        "execute",
                        "delete",
                        "update",
                        "read",
                        "create",
                        "share"
                    ],
                    "policyTag": "flow_author",
                    "resourceType": "flow"
                }
            ],
            "authorizationRoles": [
                {
                    "policyId": 13,
                    "name": "default",
                    "tag": null,
                    "workspaceId": 1,
                    "createdAt": "2020-04-22T07:50:04.502+0000",
                    "updatedAt": "2020-04-22T07:50:04.502+0000",
                    "resourceOperations": [
                        {
                            "operations": [
                                "execute",
                                "delete",
                                "update",
                                "read",
                                "create",
                                "share"
                            ],
                            "policyTag": "plan_author",
                            "resourceType": "plan"
                        },
                        {
                            "operations": [
                                "execute",
                                "delete",
                                "update",
                                "read",
                                "create",
                                "share"
                            ],
                            "policyTag": "flow_author",
                            "resourceType": "flow"
                        },
                        {
                            "operations": [
                                "delete",
                                "update",
                                "read",
                                "create",
                                "share"
                            ],
                            "policyTag": "connection_author",
                            "resourceType": "connection"
                        }
                    ]
                }
            ],

For details on these parameters, please see the API Reference documentation:

D s api refdoclink
operation/listPeople

People - GET

Endpoint/v4/people/:id
MethodGET

Change:

Same change as previous.

For details on these parameters, please see the API Reference documentation:

D s api refdoclink
operation/getPerson

Create Salesforce connection changes

Beginning in Release 7.1, the structure of a Salesforce connection object has been changed. In prior releases, a Salesforce connection could not be shared without also sharing credentials. These changes allow a Salesforce connection to be shared without requiring the sharing of credentials. 

Prior to Release 7.1, when creating a Salesforce connection using the APIs, the /v4/connections POST method required the insertion of a Salesforce security token as part of the params section of the request:

Code Block
{
  "vendor": "salesforce",
  "vendorName": "salesforce",
  "type": "jdbc",
  "credentialType": "basic",
  "ssl": true,
  "name": "Salesforce",
  "description": "example_salesforce_connection",
  "disableTypeInference": true,
  "isGlobal": true,
  "credentialsShared": true,
  "host": "salesforce.example.com",
  "params": {
    "servername": "myServer",
    "securityToken": "string"
  },
  "credentials": [
    {
      "username": "string",
      "password": "string"
    }
  ]
}

Beginning in Release 7.1, the following changes have been applied to this request structure:

  • Change credentialType to securityToken

  • Move securityToken key from params to credentials area of the API request

  • A host and port number value are not required.
Code Block
{
  "vendor": "salesforce",
  "vendorName": "salesforce",
  "type": "jdbc",
  "credentialType": "securityToken",
  "ssl": true,
  "name": "Salesforce",
  "description": "example_salesforce_connection",
  "disableTypeInference": true,
  "isGlobal": true,
  "credentialsShared": true,
  "host": "salesforce.example.com",
  "params": {
    "servername": "myServer"
  },
  "credentials": [
    {
      "username": "string",
      "password": "string",
      "securityToken": "string"
    }
  ]
}

For more information:

Apply overrides to file-based data sources when you run a job

You can now apply overrides to your file-based data sources when you run a job using the APIs. See API Workflow - Run Job.

Changes for Release 7.0

v3 endpoints are no longer available

The v3 version of the API endpoints are no longer available in the platform.

You must use the v4 versions. See API Reference.

Path to file-based imported datasets can be specified by URI

In Release 6.8 and earlier, you could create an imported dataset from a file-based source using a request like the following:

Code Block
{
  "path": "/tri-hdfs/uploads/1/4aee9852-cf92-47a8-8c6a-9ff2adeb3b4a/POS-r02.txt",
  "type": "hdfs",
  "bucket": null,
  "name": "POS-r02b.txt",
  "description": "POS-r02 - copy"
}

Beginning in this release, you can specify the source using a single uri value:

Code Block
{
  "uri": "hdfs:///tri-hdfs/uploads/1/4aee9852-cf92-47a8-8c6a-9ff2adeb3b4a/POS-r02.txt",
  "name": "POS-r02b.txt",
  "description": "POS-r02 - copy"
}
Info

NOTE: The prior format using path and type is still supported.

For more information, see 

D s api refdoclink
operation/createImportedDataset


awsRoles GET method removed

Prior to Release 7.0, the platform supported the ability to retrieve awsRole objects based on awsRoleId value.

In Release 7.0 and later, this endpoint has been disabled in the product. 

Tip

Workaround: You can still acquire awsRole identifiers and objects via API. For more information, see

D s api refdoclink
operation/listAwsRoles

Connections endpoints simplified

Prior to Release 7.0, the request to create a connection looked liked the following:

Code Block
{
    "connectParams": {
        "vendor": "sqlserver",
        "vendorName": "sqlserver",
        "host": "sqlserver.example.com",
        "port": "1433"
    },
    "host": "sqlserver.example.com",
    "port": 1433,
    "vendor": "sqlserver",
    "params": {
        "connectStrOpts": ""
    },
    "ssl": false,
    "vendorName": "sqlserver",
    "name": "sqlserver_test2",
    "description": "",
    "type": "jdbc",
    "isGlobal": false,
    "credentialType": "basic",
    "credentialsShared": true,
    "disableTypeInference": false,
    "credentials": [
        {
            "username": "<username>",
            "password": "<password>"
        }
    ]
}

In the above, the connectParams information is duplicated elsewhere in the request. 

Beginning in Release 7.0, the connectParams elements of the Connections endpoint have been removed. 

The above request now looks like the following:

Code Block
{
    "host": "sqlserver.example.com",
    "port": 1433,
    "vendor": "sqlserver",
    "params": {
        "connectStrOpts": ""
    },
    "ssl": false,
    "vendorName": "sqlserver",
    "name": "sqlserver_test2",
    "description": "",
    "type": "jdbc",
    "isGlobal": false,
    "credentialType": "basic",
    "credentialsShared": true,
    "disableTypeInference": false,
    "credentials": [
        {
            "username": "<username>",
            "password": "<password>"
        }
    ]
}

Required changes:

  • The params and name attributes are now required when creating a new connection. 
  • If you have downstream scripts that utilize the connectParams objects, these must be updated to reference the corresponding attributes after upgrade to Release 7.0 or later.

For more information, see 

D s api refdoclink
operation/getConnection


Publishing for a jobGroup has been simplified

After you have run a job, you can publish the results from the job using the following endpoint:

Code Block
/v4/jobGroups/<id>/publish

Prior to Release 7.0, the request to submit to the endpoint looked like the following:

Code Block
{
  "connection": {
    "id": 1
  },
  "path": ["default"],
  "table": "test_table",
  "action": "create",
  "inputFormat": "pqt",
  "flowNodeId": 27
}

However, the flowNodeId value is not necessary, as it can be inferred from the id value that is part of the endpoint URI. 

Beginning in Release 7.0, the request no longer requires the flowNodeId value:

Code Block
{
  "connection": {
    "id": 1
  },
  "path": ["default"],
  "table": "test_table",
  "action": "create",
  "inputFormat": "pqt"
}
Info

NOTE: No changes are required to existing scripts. Values submitted for the flowNodeId or the outputObjectId are ignored.

For more information, see 

D s api refdoclink
operation/publishJobGroup


WASB URI format has changed

In Release 6.8 and earlier, the supported format for WASB URIs was the following:

Code Block
wasbs://blobhost.blob.core.windows.net/container/path/to/file.csv

This format was not an Azure standard. Beginning in Release 7.0, the following standards-based format is supported:

Code Block
wasbs://container@blobhost.blob.core.windows.net/path/to/file.csv
Info

NOTE: Any references in your API tooling to the previous WASB UI format must be updated to the new format.

Saving Databricks Personal Access Token has different response

In Release 6.8 and earlier, when saving a Databricks Personal Access Token, the response looked like the following:

Response Code204 - No Content
Response BodyEmpty.

Beginning in Release 7.0, the response looks like the following:

Response Code200 - Ok
Response Body
Code Block
{
  "message": "Databricks Personal Access Token has been stored for user <user_email>"
}

For more information, see

D s api refdoclink
operation/saveDatabricksAccessToken


Documentation for awsRoles object corrected

In Release 6.8 and earlier, the following issues appeared in the documentation for the awsRoles object. These issues have been corrected in this release:

Inadequate permissions:

The POST and PUT methods for the awsRoles endpoint require either of the following user permissions:

  • Workspace admin
  • D s item
    itemadmin

Optional workspaceid/personid attributes:

For the POST and PUT methods for the awsRoles endpoint, you can optionally assign the role to a workspace or person.

See

D s api refdoclink
operation/createAwsRole

See

D s api refdoclink
operation/updateAwsRole

Changes for Release 6.8.1

Improvements to AssetTransfer API documentation

Before disabling or deleting a user's account, an admin may wish to transfer the user's assets to the ownership of a different user. For more information on how to transfer all assets via API, see 

D s api refdoclink
operation/transferAssets


Changes for Release 6.8

Overrides to relational sources and targets

Through the APIs, you can now apply overrides to relational sources and targets during job execution or deployment import. 

jobGroups

When you are running a job, you can override the default publication settings for the job using overrides in the request. For more information, see API Workflow - Run Job.

Deployments

When you import a flow package into a deployment, you may need to remap the source and output of the flow to use production versions of your data. This capability has been present in the product for file-based sources and targets. Now, it's available for relational sources and targets. For more information, see Define Import Mapping Rules

New flow object definition

Release 6.8 introduces a new version of the flow object definition. This new version will support cross-product and cross-version import and export of flows in the future. For more information see Changes to the Object Model.

Info

NOTE: The endpoints to use to manage flow packages remain unchanged. Similarly, the methods used to define import mapping rules remains unchanged. The API responses that contain flow definitions has changed. See below.

Export and import macros

Beginning in Release 6.8, you can export and import macro definitions via API.

  • Export: 

    D s api refdoclink
    operation/getMacroPackage
  • Import: 

    D s api refdoclink
    operation/importMacroPackage

Changes for Release 6.4

Request format for assigning connection permissions endpoint has changed

For this endpoint:

Code Block
/v4/connections/<cid>/permissions/

where:

  • <cid> is in the internal connection identifier.

The request payload format has changed.

Before Release 6.4, the request format was as follows:

Code Block
[
  {
    "personId": 3,
    "role": "readOnly"
  }
]

Beginning in Release 6.4, the request format is as follows:

Code Block
{
  "data": [
    {
      "person": {
        "id": 3
      },
      "role": "readOnly"
    }
  ]
}
Info

NOTE: The old request format is no longer supported.

For more information, see

D s api refdoclink
operation/createConnectionPermission

v4 version of password reset request endpoint

To assist in migration from the command-line interface to using the APIs, a v4 version of an API endpoint has been made available to allow for administrators to generate password reset codes.

Changes to awsConfig object

Info

NOTE: No action is required.

In Release 6.0, the awsConfig object was introduced to enable the assignment of AWS configurations to individual users (per-user auth) via API. This version of the awsConfig object supported a mapping of a single IAM role to an awsConfig object.

Beginning in Release 6.4, per-user authentication now supports mapping of multiple possible IAM roles to an individual user's configuration. To enable this one-to-many mapping, the awsRoles object was introduced.

  • An awsRoles object creates a one-to-one mapping between an IAM role and an awsConfig object.
  • An awsConfig object can have multiple awsRoles assigned to it.

Changes to awsConfig object:

  • The role field in the object has been replaced by activeRoleId, which maps to the active role for the configuration object.
  • For each role reference in the awsConfig objects, a corresponding awsRole object has been created and mapped to it.

Beginning in Release 6.4, you can create, edit, and delete awsRoles objects, which can be used to map an AWS IAM role ARN to a specified AWSConfig object. You can map multiple awsRoles to a single awsConfig.

For more information, see API Workflow - Manage AWS Configurations.

Changes for Release 6.3

Assign AWSConfigs to a user at create time

Beginning in Release 6.3, you can assign an AWSConfig object to a user when you create the object. This shortcut reduces the number of REST calls that you need to make. 

Info

NOTE: For security reasons, AWSConfig objects must be assigned to users at the time of creation. Admin users can assign to other users. Non-admin users are automatically assigned the AWSConfig objects that they create.

Prior to Release 6.3, AWSConfig objects were assigned through the following endpoint. Example:

Code Block
/v4/people/2/awsConfigs/6
Info

NOTE: This endpoint has been removed from the platform. Please update any scripts that reference the above endpoint to manage AWS configuration assignments through the new method described in the following link.

See API Workflow - Manage AWS Configurations.

Changes for Release 6.0

Error in Release 6.0.x API docs

In Release 6.0 - Release 6.0.2, the online and PDF versions of the documentation referenced the following endpoint: API JobGroups Get Status v4. According to the docs, this endpoint was triggered in this manner:

MethodGET
Endpoint
Code Block
/v4/jobGroups/<id>/status

This endpoint exists in v3 of the API endpoints. It does not exist in v4.

Instead, you should monitor the status field for the base GET endpoint for jobGroups. For more information, see

D s api refdoclink
operation/getJobGroup

Planned End of Life of v3 API endpoints

In Release 6.0, the v3 API endpoints are supported.

Warning

In the next release of 

D s product
productee
 after Release 6.0, the v3 API endpoints will be removed from the product (End of Life).

You must migrate to using the v4 API endpoints before upgrading to the next release after Release 6.0. 

Changes for Release 5.9

Introducing Access Tokens

Each request to the API endpoints of the 

D s platform
 requires submission of authentication information. In Release 5.1 and earlier:

  • A request could include clear-text username/password combinations. This method is not secure.
  • A request could include a browser cookie. This method does not work for well for use cases outside of the browser (e.g. scripts).

Beginning in Release 5.9, API users can manage authentication using access tokens. These tokens obscure any personally identifiable information and represent a standards-based method of secure authentication.

Info

NOTE: All previous methods of API authentication are supported in this release. Access tokens is the preferred method of authentication.

The basic process works in the following manner:

  1. API user requests generation of a new token.
    1. This initial request must contain a valid username and password.
    2. Request includes expiration.
    3. Token value is returned in the response.
  2. The token value inserted into the Authorization header of each request to the platform.
  3. User monitors current time and expiration time of the token. At any time, the user can request a new token to be generated using the same endpoint used in the initial request. 

Access tokens can be generated via API or the 

D s webapp
.

Info

NOTE: This feature must be enabled in your instance of the platform. See Enable API Access Tokens.

  • API: For more information, see 

    D s api refdoclink
    operation/createApiAccessToken
  • D s webapp
    :
     For more information, see Access Tokens Page.

For more information on API authentication methods, see 

D s api refdoclink
section/Authentication

Changes for Release 5.1

None.

Changes for Release 5.0

Introducing v4 APIs

D beta

Release 5.0 signals the introduction of version 4 of the REST APIs. 

Info

NOTE: At this time, a very limited number of v4 REST APIs are publicly available. Where possible, you should continue to use the v3 endpoints. For more information, see API Reference.

v4 conventions

The following conventions apply to v4 and later versions of the APIs:

  • Parameter lists are consistently enveloped in the following manner:

    Code Block
    { "data": [
        {
          ...
        }
      ]
    }
  • Field names are in camelCase and are consistent with the resource name in the URL or with the embed URL parameter.
  • From early API versions, foreign keys have been replaced with identifiers like the following:

    v3 and earlierv4 and later
    Code Block
    "createdBy": 1,
    Code Block
    "creator": {
      "id": 1
    },
    Code Block
    "updatedBy": 2,
    Code Block
    "updater": {
      "id": 2
    },
  • Publication endpoint references database differently. This change is to make the publishing endpoint for relational targets more flexible in the future. 

    v3 and earlierv4 and later
    Code Block
    "database": "dbName",
    Code Block
    "path": ["dbName"],

Changes for Release 4.2

Create Hive and Redshift connections via API

You can create connections of these types via API:

  • Only one global Hive connection is still supported.
  • You can create multiple Redshift connections.

WrangledDataset endpoints are still valid 

In Release 4.1.1 and earlier, the WrangledDataset endpoints enabled creation, modification, and deletion of a wrangled dataset object, which also created the associated recipe object.  

In Release 4.2, wrangled datasets have been removed from the application interface. However, the WrangledDataset API endpoints remain, although they now apply to the recipe directly.

The following endpoints and methods are still available:

Info

NOTE: In a future release, these endpoints may be migrated to recipe-based endpoints. API users should review this page for each release.

For more information, see Changes to the Object Model.