...

AWS Databricks is a unified data analytics platform that has been optimized for use on the  AWS infrastructure.

• For more information, see https://databricks.com/aws.
  
• For documentation on AWS Databricks, see https://databricks.com/documentation. 

Additional Databricks features supported by the platform:

• Credential passthrough (AWS Databricks only): https://docs.databricks.com/security/credential-passthrough/iam-passthrough.html
• Table access control: https://docs.databricks.com/security/access-control/table-acls/object-privileges.html

Prerequisites

• The 

  D s platform

   must be installed in a customer-managed AWS environment.
• The base storage layer must be set to S3. For more information, see Set Base Storage Layer.
• AWS Secrets Manager is required for AWS Databricks use. For more information, see Configure for AWS Secrets Manager.

...

1. D s config
   method ws

2. Locate the following parameter, which enables 

   D s photon

    for smaller job execution. Set it to Enabled:
   
   

   Code Block
   Photon execution

3. You do not need to save to enable the above configuration change.

4. D s config

5. Locate the following parameters. Set them to the values listed below, which enables AWS Databricks (small to extra-large jobs) running environments:

   Code Block
   "webapp.runInDatabricks": true, "webapp.runWithSparkSubmit": false, "webapp.runInDataflow": false,

6. Do not save your changes until you have completed the following configuration section.

Configure

Configure cluster mode 

When a user submits a job, the 

For more information on job clusters, see https://docs.databricks.com/clusters/configure.html .

The job clusters automatically terminate after the job is completed. A new cluster is automatically created when the user next requests access to AWS Databricks access. 

...

Optionally, you can configure the 

...

{
  "autoscale.max_workers": {
    "type": "fixed",
    "value": 3,
    "hidden": true
  },
  "autoscale.min_workers": {
    "type": "fixed",
    "value": 1,
    "hidden": true
  },
  "aws_attributes.instance_profile_arn": {
    "type": "fixed",
    "value": "arn:aws:iam::9999999999:instance-profile/SOME_POLICY",
    "hidden": false
  },
  "enable_local_disk_encryption": {
    "type": "fixed",
    "value": false
  },
  "instance_pool_id": {
    "type": "fixed",
    "value": "SOME_POOL",
    "hidden": true
  },
  "driver_instance_pool_id": {
    "type": "fixed",
    "value": "SOME_POOL",
    "hidden": true
  },
  "autotermination_minutes": {
    "type": "fixed",
    "value": 10,
    "hidden": true
  },
}

Configure Instance Profiles in AWS Databricks

To configure the instance profile for AWS Databricks, you must provide an IAM instance profile ARN in databricks.awsAttributes. instanceProfileArn parameter.

For more information, see  Configure for AWS Authentication .

Configure instance pooling

...

Instance pooling for worker nodes

Pre-requisitesPrerequisites:

• All cluster nodes used by the 

  D s platform

   are taken from the pool. If the pool has an insufficient number of nodes, cluster creation fails.
  
• Each user must have access to the pool and must have at least the ATTACH_TO permission.
• Each user must have a personal access token from the same AWS Databricks workspace. See Configure personal access token below.

...

Following is the list of parameters that can be reviewed or modified based on your requirements:

Optional Parameters

Configure Databricks Job Management

...

Optionally, you can provide to the 

Pre-requisitesPrerequisites:

...

Configure AWS Databricks workspace overrides

A single AWS Databricks account can have access to multiple Databricks workspaces. You can create more than one workspace by using Account API if you are account is on the E2 version of the platform or on a selected custom plan that allows multiple workspaces per account.

For more information, see https://docs.databricks.com/administration-guide/account-api/new-workspace.html

Each workspace has a unique deployment name associated with it that defines the workspace URL. For example:  https://<deployment-name>. cloud.databricks.com .  

...

I ndividual users can specify the name of the cluster to which they are permissioned to access Databricks Tables. This cluster can also be shared among users. For more information, see Databricks Settings Page. 

Configure maximum retries for REST API 

There is a limit of 30 requests per second per workspace on the Databricks REST APIs. If this limit is reached, then a HTTP status code 429 error is returned, indicating that rate limiting is being applied by the server.   By default, the

If you want to change the number of retries, change the value for the  databricks.maxAPICallRetries  flag.   

...