...

AWS Databricks is a unified data analytics platform that has been optimized for use on the  AWS infrastructure.

• For more information, see https://databricks.com/aws.
  
• For documentation on AWS Databricks, see https://databricks.com/documentation. 

Additional Databricks features supported by the platform:

• Credential passthrough (AWS Databricks only): https://docs.databricks.com/security/credential-passthrough/iam-passthrough.html
• Table access control: https://docs.databricks.com/security/access-control/table-acls/object-privileges.html

Prerequisites

• The 

  D s platform

   must be installed in a customer-managed AWS environment.
• The base storage layer must be set to S3. For more information, see Set Base Storage Layer.

...

Supported versions of Databricks

• Azure AWS Databricks 8.3

• AWS Databricks 7.3 (Recommended)

• Azure AWS Databricks 5.5 LTS

Job Limits

...

1. D s config
   method ws

2. Locate the following parameter, which enables 

   D s photon

    for smaller job execution. Set it to Enabled:
   

   Code Block
   Photon execution

3. You do not need to save to enable the above configuration change.

4. D s config

5. Locate the following parameters. Set them to the values listed below, which enables AWS Databricks (small to extra-large jobs) running environments:

   Code Block
   "webapp.runInDatabricks": true, "webapp.runWithSparkSubmit": false, "webapp.runInDataflow": false,

6. Do not save your changes until you have completed the following configuration section.

Configure

Configure cluster mode 

When a user submits a job, the 

For more information on job clusters, see https://docs.databricks.com/clusters/configure.html .

The job clusters automatically terminate after the job is completed. A new cluster is automatically created when the user next requests access to AWS Databricks access. 

Configure use of cluster policies

Optionally, you can configure the 

Prerequisites

Notes:

• When this feature is enabled, each user may select the appropriate Databricks cluster policy to use for jobs. If none is selected by a user, jobs are launched without a cluster policy for the user using the Databricks properties set in platform configuration. 

  Info
  NOTE: Except for Spark version and cluster policy identifier in job-level overrides, other Databricks cluster configuration in the  D s platform  is ignored when this feature is in use. Other job-level overrides are also ignored.

• If a cluster policy is modified and existing clusters are using it, then subsequent job executions using that policy attempt to use the same cluster. This can cause issues in performance and even job failures.
  

  Tip
  Tip: Avoid editing cluster policies that are in use, as these changed policies may be applied to clusters generated under the old policies. Instead, you should create a new policy and assign it for use.

• If the cluster policy references a Databricks instance pool that does not exist, the job fails. 

Steps:

1. D s config
   method ws

2. Locate the following parameter and set it to Enabled:

   Code Block
   Databricks Cluster Policies

3. Save your changes and restart the platform.

Job overrides:

A user's cluster policy can overridden when a job is executed via API. Set the request attribute for the clusterPolicyId.

For more information, see API Workflow - Run Job.

Policy template for AWS - without instance pools:

The following example cluster policy can provide a basis for creating your own AWS cluster policies when instance pools are not in use:

{
  "autoscale.max_workers": {
    "type": "fixed",
    "value": 3,
    "hidden": true
  },
  "autoscale.min_workers": {
    "type": "fixed",
    "value": 1,
    "hidden": true
  },
  "autotermination_minutes": {
    "type": "fixed",
    "value": 10,
    "hidden": true
  },
  "aws_attributes.availability": {
    "type": "fixed",
    "value": "SPOT_WITH_FALLBACK",
    "hidden": false
  },
  "aws_attributes.ebs_volume_count": {
    "type": "fixed",
    "value": 0,
    "hidden": false
  },
  "aws_attributes.ebs_volume_size": {
    "type": "fixed",
    "value": 0,
    "hidden": false
  },
  "aws_attributes.first_on_demand": {
    "type": "fixed",
    "value": 1,
    "hidden": false
  },
  "aws_attributes.spot_bid_price_percent": {
    "type": "fixed",
    "value": 100,
    "hidden": false
  },
  "aws_attributes.instance_profile_arn": {
    "type": "fixed",
    "value": "arn:aws:iam::9999999999999:instance-profile/SOME_Role_ARN",
    "hidden": false
  },
  "driver_node_type_id": {
    "type": "fixed",
    "value": "i3.xlarge",
    "hidden": true
  },
  "enable_local_disk_encryption": {
    "type": "fixed",
    "value": false
  },
  "node_type_id": {
    "type": "fixed",
    "value": "i3.xlarge",
    "hidden": true
  }
}

Policy template for AWS - without instance pools:

The following example cluster policy can provide a basis for creating your own AWS cluster policies when instance pools are in use:

{
  "autoscale.max_workers": {
    "type": "fixed",
    "value": 3,
    "hidden": true
  },
  "autoscale.min_workers": {
    "type": "fixed",
    "value": 1,
    "hidden": true
  },
  "aws_attributes.instance_profile_arn": {
    "type": "fixed",
    "value": "arn:aws:iam::9999999999:instance-profile/SOME_POLICY",
    "hidden": false
  },
  "enable_local_disk_encryption": {
    "type": "fixed",
    "value": false
  },
  "instance_pool_id": {
    "type": "fixed",
    "value": "SOME_POOL",
    "hidden": true
  },
  "driver_instance_pool_id": {
    "type": "fixed",
    "value": "SOME_POOL",
    "hidden": true
  },
  "autotermination_minutes": {
    "type": "fixed",
    "value": 10,
    "hidden": true
  },
}

Configure Instance Profiles in AWS Databricks

To configure the instance profile for AWS Databricks, you must provide an IAM instance profile ARN in databricks.awsAttributes. instanceProfileArn parameter.

For more information, see  Configure for AWS Authentication .

Configure instance pooling

Instance pooling reduces cluster node spin-up time by maintaining a set of idle and ready instances. The 

...

• All cluster nodes used by the 

  D s platform

   are taken from the pool. If the pool has an insufficient number of nodes, cluster creation fails.
  
• Each user must have access to the pool and must have at least the ATTACH_TO permission.
• Each user must have a personal access token from the same Azure AWS Databricks workspace. See Configure personal access token below.

...

1. Acquire your pool identifier or pool name from Azure AWS Databricks.
   

   Info
   NOTE: You can use either the Databricks pool identifier or pool name.If both poolId and poolName are specified, poolId is used first. If that fails to find a matching identifier, then the poolName value is checked.

   Tip
   Tip: If you specify a poolName value only, then you can run your Databricks jobs against the available clusters across multiple D s item item workspaces . This mechanism allows for better resource allocation and broader execution options.

2. D s config

3. Set either of the following parameters: 

   1. Set the following parameter to the Azure AWS Databricks pool identifier:

      Code Block
      "databricks.poolId": "<my_pool_id>",

   2. Or, you can set the following parameter to the Azure AWS Databricks pool name:

      Code Block
      "databricks.poolName": "<my_pool_name>",

4. Save your changes and restart the platform.

...

...

Configure AWS Databricks workspace overrides

A single AWS Databricks account can have access to multiple Databricks workspaces. You can create more than one workspace by using Account API if you are account is on the E2 version of the platform or on a selected custom plan that allows multiple workspaces per account.

For more information, see https://docs.databricks.com/administration-guide/account-api/new-workspace.html

Each workspace has a unique deployment name associated with it that defines the workspace URL. For example:  https://<deployment-name>. cloud.databricks.com .  

...

I ndividual users can specify the name of the cluster to which they are permissioned to access Databricks Tables. This cluster can also be shared among users. For more information, see Databricks Settings Page. 

Configure maximum retries for REST API 

There is a limit of 30 requests per second per workspace on the Databricks REST APIs. If this limit is reached, then a HTTP status code 429 error is returned, indicating that rate limiting is being applied by the server.   By default, the

If you want to change the number of retries, change the value for the  databricks.maxAPICallRetries  flag.  

...