Limitations

• Import-only connection type
• A limited set of request methods is supported. See Method entry below.
• Using a passphrase when generating an SSH key is not supported.
• JSON response from API endpoint is required. API endpoints that return XML responses are not supported.
• OAuth 2.0 authentication is not supported.
• After the initial connection to an endpoint is made, the schema is cached. The schema is not updated again until the connection is edited.
• By default, the number of endpoints that you can specify to use an individual connection is 10. This limit can be modified.
  

• 
  
  • For more information, see Dataprep Project Settings Page.

Prerequisites

• You should identify the tables and (optional) data models for them that you wish to access. 

• You should acquire the credentials to access your target endpoints for one of the supported authentication methods. 

  • If you are using a key or token to access the endpoints, you should generate this token before you begin. 

  • See below. 

Configure

To create this connection, in the Connections page , select the Applications tab. Click the  card. See Connections Page.

Modify the following properties as needed:

https://exampleserver.sharepoint.com/sites/SharePointTest

Authentication types

The following types of authentication are supported for REST API connections. For each type, additional properties may require configuration. 

Basic auth

A username/password combination is submitted as part of any request for authentication.

HTTP Header Based Auth

Authentication is submitted using a key/value pair submitted in the HTTP request header.

HTTP Query Based Auth

Authentication is submitted using a query parameter key/value pair submitted as part of the URL.

Configure endpoints

Each endpoint and method combination must be configured. To add an endpoint, click Add endpoint. 

The properties are described below.

<endpoint_url>?<key1>=<value1>&<key2>=<value2>

Pagination

For the selected endpoint, you can specify the type of pagination in use by the target application. Specifying the pagination allows the  to retrieve larger sets of records when pagination is in use. For more information on these pagination methods, see http://cdn.cdata.com/help/DWE/ado/pg_customschemaselect.htm.

None:

(default) No pagination is applied by the endpoint.

Next page URL: 

When this method is selected, the URL of the next page of results is returned as part of the response body. 

• Page URL path defines the XPath in the response to the attribute containing the URL of the next page. 

Paging token:

A paging token may be returned as part of the response. To acquire the next page of results, this token must be submitted in the subsequent request as the value associated with the paging parameter.

• Page token path is the XPath to the token that must be submitted with the next request. 
• Page token param is the parameter in the request into which the page token must be submitted.
• More pages param (optional) is used when the page token path must be submitted as a query parameter. This value defines the query parameter for which it is submitted. 

Record offset: 

Under this pagination method, subsequent pages of results can be queried based on defining the number of results (records) to offset with the query.

• Page offset param defines the query parameter where you can specify the page offset to query.

• Page size param defines the parameter in the request where you define the size in records of each request (page) of records.

• Page size defines the number of records to request in a page.

Page number:

Similar to record offset, this method queries results based on specified page numbers.

• Page number param defines the query parameter where you can specify the page number to query.

• Page size param defines the parameter in the request where you define the size in records of each page of records.

• Page size defines the number of records to request in a page.

Connect string options

Too many requests error

During execution, you may receive an error similar from the driver to the following:

PlatformErrors: Retries errors based on the exception message. E.g. Other=PlatformErrors="Too Many Requests"
MaximumRequestRetries: The number of times the driver will attempt to retry the request (Default 4)

In this case, the default wait time for retrying a request (2 seconds) is not enough time, and the requests are piling up. You can address this issue by inserting the following connect string options:

Other='RetryWaitTime=15000'

The above option sets the wait time before retrying to 15000ms (15 seconds). You can experiment with this value as needed.

For more information on available connect string options, see https://cdn.cdata.com/help/DWE/ado/Connection.htm.

Create via API

This connection can also be created using the API.

• Type: jdbc_rest
• Vendor: jdbc_rest

Example - single GET method

The following example request creates a REST API connection with the following characteristics:

• Query parameters are used for authentication
• A single endpoint is enabled:
  • Method: GET
  • Target: table1
  • dataModel: Document  

{
  "vendor": "jdbc_rest",
  "vendorName": "jdbc_rest",
  "name": "REST API test",
  "description": "",
  "type": "jdbc",
  "params": {
    "base_URI": "some base URI",
    "connectStrOpts": ""
  },
  "credentialType": "httpQueryBasedAuth",
  "credentials": [
    {
      "queryKey": "user",
      "queryValue": "token"
    }
  ],
  "endpoints": [
    {
      "tableName": "table1",
      "httpMethod": "get",
      "endpoint": "endpoint1",
      "requestBody": "",
      "xPath": "",
      "dataModel": "document",
      "headers": {},
      "queryParams": {}
    }
  ]
}

Example - rate limiting

The following example creates a REST API connection to polygon.io with the following characteristics:

• Query-based authentication using key/value pair
• Connect String Options:

  Other='RetryWaitTime=15000';

  • Wait for retry: 15000 milliseconds
• Single endpoint to GET stock ticker information:
  • DataModel: Document
  • XPath: $./results
  • Rate limiting on the endpoint (maximum queries per minute): 1000
  • queryParams.date is a parameter that is passed in for this specific connection type.
  • Pagination: nextPageURL method

{
  "vendor": "jdbc_rest",
  "vendorName": "jdbc_rest",
  "name": "REST API",
  "description": "",
  "type": "jdbc",
  "params": {
    "base_URI": "https://api.polygon.io/",
    "connectStrOpts": "Other='RetryWaitTime=15000';"
  },
  "credentialType": "httpQueryBasedAuth",
  "credentials": [
    {
      "queryKey": "apiKey",
      "queryValue": "someKey"
    }
  ],
  "endpoints": [
    {
      "tableName": "tickers",
      "httpMethod": "get",
      "endpoint": "/v3/reference/tickers",
      "requestBody": "",
      "xPath": "$./results",
      "dataModel": "document",
      "headers": {},
      "queryParams": {
        "limit": "1000",
        "date": "2021-11-04T00:00:00Z"
      },
      "pagination": {
        "pageurlpath": "$./next_url",
        "paginationType": "nextPageURL"
      }
    }
  ]
}

For more information, see

#operation/createConnection

Use

You can import datasets from  through the Import Data page. See Import Data Page.

Using REST API Connections

This section describes how you interact through  with your REST data.

Uses

 can use  connections for the following tasks:

1. Import datasets

Before you begin

• Read Access: You must have credentials to create access the specific endpoints required to retrieve your data. 

• Write Access: Not supported

Secure access

SSL is available over HTTPS for  connections.

Reading data

You can create a  from the following data models: 

1. documents
2. flattened documents
3. relational tables

These source objects are represented as tables in the import browser and as grid data in the Transformer page.

For more information, see Database Browser.

Writing data

Not supported.

Reference

• Read: Supported
• Write: Not supported