AWS Databricks is a unified data analytics platform that has been optimized for use on the AWS infrastructure.

• For more information, see https://databricks.com/aws.
  
• For documentation on AWS Databricks, see https://databricks.com/documentation. 

Additional Databricks features supported by the platform:

• Credential passthrough (AWS Databricks only): https://docs.databricks.com/security/credential-passthrough/iam-passthrough.html
• Table access control: https://docs.databricks.com/security/access-control/table-acls/object-privileges.html

Prerequisites

• The  must be installed in a customer-managed AWS environment.
• The base storage layer must be set to S3. For more information, see Set Base Storage Layer.
• AWS Secrets Manager is required for AWS Databricks use. For more information, see Configure for AWS Secrets Manager.

Limitations

• Import datasets created from nested folders is not supported for running jobs from AWS Databricks.

• If the job is submitted using the User cluster mode and no cluster is available, the following are the launch times for a new cluster with and without instance pools:
  • Without instance pools: Up to 5 minutes to launch
  • With instance pools : Up to 30 seconds to launch
• If the job is canceled during cluster startup:
  • The cluster startup continues. After the cluster is running, the job is terminated, and the cluster remains. 
  • As a result, there is a delay in reporting the job cancellation in the Job Details page. The job should be reported as canceled not failed.

• AWS Databricks integration works with Spark 2.4.x only.

  NOTE: The version of Spark for AWS Databricks must be applied to the platform configuration through the databricks.sparkVersion property. Details are provided later.

  
  

Supported versions of Databricks

• AWS Databricks 8.3

• AWS Databricks 7.3 (Recommended)

• AWS Databricks 5.5 LTS

Job Limits

By default, the number of jobs permitted on an AWS workspace is set to 1000.

• The number of jobs that can be created per workspace in an hour is limited to 1000.
• The number of jobs a workspace can create in an hour is limited to 5000 when using the run-submit API. This limit also affects jobs created by the REST API and notebook workflows. For more information, see "Configure Databricks job management" below.
• The number of actively concurrent job runs in a workspace is limited to 150.

These limits apply to any jobs that use workspace data on the cluster.

Managing Limits

To enable retrieval and auditing of job information after a job has been completed, the  does not delete jobs from the cluster. As a result, jobs can accumulate over time to exceed the number of jobs permitted on the cluster. If you reach these limits, you may receive a Quota for number of jobs has been reached limit. For more information, see https://docs.databricks.com/user-guide/jobs.html.

Optionally, you can allow the  to manage your jobs to avoid these limitations. For more information, see "Configure Databricks job management" below.

Enable

To enable AWS Databricks, perform the following configuration changes:

Steps:

1. 

2. Locate the following parameter, which enables  for smaller job execution. Set it to Enabled:
   
   

   Photon execution

   
   

3. You do not need to save to enable the above configuration change.
4. 

5. Locate the following parameters. Set them to the values listed below, which enables AWS Databricks (small to extra-large jobs) running environments:

   "webapp.runInDatabricks": true, "webapp.runWithSparkSubmit": false, "webapp.runInDataflow": false,

   
   

6. Do not save your changes until you have completed the following configuration section.

Configure

Configure cluster mode 

When a user submits a job, the  provides all the cluster specifications in the Databricks API and it creates cluster only for per-user or per-job, that means once the job is complete, the cluster is terminated.  Cluster creation may take less than 30 seconds if instance pools are used. If the instance pools are not used, it may take 10-15 minutes.

For more information on job clusters, see https://docs.databricks.com/clusters/configure.html.

The job clusters automatically terminate after the job is completed. A new cluster is automatically created when the user next requests access to AWS Databricks access. 

Configure Instance Profiles in AWS Databricks

  EC2 instances can be configured with permissions to access AWS resources like S3 by attaching an IAM instance profile. Similarly, instance profiles can be attached to EC2 instances for use with AWS Databricks clusters.

To configure the instance profile for AWS Databricks, you must provide an IAM instance profile ARN in databricks.awsAttributes.instanceProfileArn parameter.

For more information, see Configure for AWS Authentication.

Configure instance pooling

Instance pooling reduces cluster node spin-up time by maintaining a set of idle and ready instances. The  can be configured to leverage instance pooling on the AWS Databricks cluster for both worker and driver nodes.

For more information, see https://docs.azuredatabricks.net/clusters/instance-pools/index.html.

Instance pooling for worker nodes

Pre-requisites:

• All cluster nodes used by the  are taken from the pool. If the pool has an insufficient number of nodes, cluster creation fails.
  
• Each user must have access to the pool and must have at least the ATTACH_TO permission.
• Each user must have a personal access token from the same AWS Databricks workspace. See Configure personal access token below.

To enable:

1. Acquire your pool identifier or pool name from AWS Databricks.
   
   

   NOTE: You can use either the Databricks pool identifier or pool name. If both poolId and poolName are specified, poolId is used first. If that fails to find a matching identifier, then the poolName value is checked.

   
   

   Tip: If you specify a poolName value only, then you can run your Databricks jobs against the available clusters across multiple . This mechanism allows for better resource allocation and broader execution options.

   
   

2. 

3. Set either of the following parameters: 

   1. Set the following parameter to the AWS Databricks pool identifier:

      "databricks.poolId": "<my_pool_id>",

      
      

   2. Or, you can set the following parameter to the AWS Databricks pool name:

      "databricks.poolName": "<my_pool_name>",

      
      

4. Save your changes and restart the platform.

Instance pooling for driver nodes

The  can be configured to use Databricks instance pooling for driver pools.

To enable:

1. Acquire your driver pool identifier or driver pool name from Databricks.
   
   

   NOTE: You can use either the Databricks driver pool identifier or driver pool name. If both driverPoolId and driverPoolName are specified, driverPoolId is used first. If that fails to find a matching identifier, then the driverPoolName value is checked.

   
   

   Tip: If you specify a driverPoolName value only, then you can run your Databricks jobs against the available clusters across multiple . This mechanism allows for better resource allocation and broader execution options.

   
   

2. 

3. Set either of the following parameters: 

   1. Set the following parameter to the Databricks driver pool identifier:

      "databricks.driverPoolId": "<my_pool_id>",

      
      

   2. Or, you can set the following parameter to the Databricks driver pool name:

      "databricks.driverPoolName": "<my_pool_name>",

      
      

4. Save your changes and restart the platform.

Configure Platform

Review and modify the following configuration settings, as required:

 Following is the list of parameters that have to be set to integrate the AWS Databricks with :

Required Parameters

Following is the list of parameters that can be reviewed or modified based on your requirements:

Optional Parameters

Configure Databricks Job Management

AWS Databricks enforces a hard limit of 1000 created jobs per workspace, and by default cluster jobs are not deleted. To support jobs more than 1000 jobs per cluster, you can enable job management for AWS Databricks.

Steps:

1. 

2. Locate the following property and set it to one of the values listed below:
   
   

   Databricks Job Management

   
   

   Property Value	Description
   Never Delete	(default) Job definitions are never deleted from the AWS Databricks cluster.
   Always Delete	The AWS Databricks job definition is deleted during the clean-up phase, which occurs after a job completes.
   Delete Successful Only	When a job completes successfully, the AWS Databricks job definition is deleted during the clean-up phase. Failed or canceled jobs are not deleted, which allows you to debug as needed.
   Skip Job Creation	For jobs that are to be executed only one time, the can be configured to use a different mechanism for submitting the job. When this option is enabled, the submits jobs using the run-submit API, instead of the run-now API. The run-submit API does not create an AWS Databricks job definition. Therefore the submitted job does not count toward the enforced job limit.
   Default	Inherits the default system-wide setting.

   
   

3. When this feature is enabled, the platform falls back to use the runs/submit API as a fallback when the job limit for the Databricks workspace has been reached:

   Databricks Job Runs Submit Fallback

   
   

4. Save your changes and restart the platform.

Enable shared cluster for Databricks Tables

Optionally, you can provide to the  the name of a shared Databricks cluster to be used to access Databricks Tables. 

Pre-requisites:

• Shared clusters are not provisioned per-user and cannot be used to run Spark jobs.
• If you have enabled table access control on your high-concurrency cluster, you must configure access to data objects for users. For more information, see https://docs.databricks.com/security/access-control/table-acls/object-privileges.html.
• If only a limited number of users are using the shared cluster, you must configure the attach permission on the shared cluster in the Databricks workspace. 

Configure cluster

Depending on the credential provider type, the following properties must be specified in the Spark configuration for the shared cluster.

Default credential provider:

"fs.s3a.access.key"
"fs.s3a.secret.key"
"spark.hadoop.fs.s3a.access.key"
"spark.hadoop.fs.s3a.secret.key"

For more information, see https://docs.databricks.com/data/data-sources/aws/amazon-s3.html.

Instance credential provider:

"fs.s3a.credentialsType"
"fs.s3a.stsAssumeRole.arn"
"fs.s3a.canned.acl"
"fs.s3a.acl.default"
"spark.hadoop.fs.s3a.credentialsType"
"spark.hadoop.fs.s3a.stsAssumeRole.arn"
"spark.hadoop.fs.s3a.canned.acl"
"spark.hadoop.fs.s3a.acl.default"

For more information, see https://docs.databricks.com/administration-guide/cloud-configurations/aws/instance-profiles.html.

Temporary credential provider:

"fs.s3a.canned.acl"
"fs.s3a.acl.default"
"spark.hadoop.fs.s3a.canned.acl"
"spark.hadoop.fs.s3a.acl.default"

For more information, see  https://docs.databricks.com/administration-guide/cloud-configurations/aws/assume-role.html.

Enable

Steps:

1. 

2. Locate the following parameter, and add the name of the Databricks cluster to use to browse Databricks Table:
   
   

   "feature.databricks.connection.clusterName": "<your_cluster_name>",

   
   

3. Save your changes and restart the cluster.

When a cluster name is provided:

• If the cluster is available, all users of the  attempt to connect to Databricks Tables through the listed cluster.
• If the cluster has been terminated, a message indicates that the cluster must be restarted. After it has been restarted, you can try again.
• If the cluster name is invalid, the  fails any read/write operations to Databricks Tables.

If a cluster name is not provided:

• In USER mode, the default behavior is to create one interactive cluster for each user to browse Databricks Tables.
• In JOB mode, the interactive cluster is created only when a user needs to browse Databricks Tables.

Configure user-level override for Databricks Tables access cluster

Individual users can specify the name of the cluster that accesses Databricks Tables through the Databricks settings. See "Specify Databricks Tables cluster name" below.

Configure for Secrets Manager

The AWS Secrets Manager is a secure vault for storing access credentials to AWS resources. 

For more information, see Configure for AWS Secrets Manager.

Configure for Users

Configure AWS Databricks workspace overrides

A single AWS Databricks account can have access to multiple Databricks workspaces. You can create more than one workspace by using Account API if you are account is on the E2 version of the platform or on a selected custom plan that allows multiple workspaces per account.

For more information, see https://docs.databricks.com/administration-guide/account-api/new-workspace.html

Each workspace has a unique deployment name associated with it that defines the workspace URL. For example: https://<deployment-name>.cloud.databricks.com.  

Configure Databricks job throttling

By default, Databricks workspaces apply limits on the number of jobs that can be submitted before the cluster begins to fail jobs. These limits are the following:

• Maximum number of concurrent jobs per cluster

• Max number of concurrent jobs per workspace

• Max number of concurrent clusters per workspace

Depending on how your clusters are configured, these limits can vary. For example, if the maximum number of concurrent jobs per cluster is set to 20, then the 21st concurrent job submitted to the cluster fails. 

To prevent unnecessary job failure, the  submits the throttling of jobs to Databricks. When job throttling is enabled and the 21 concurrent job is submitted, the  holds that job internally the first of any of the following events happens:

• An active job on the cluster completes, and space is available for submitting a new job. The job is then submitted.
• The user chooses to cancel the job.
• One of the timeout limits described below is reached.

Steps:

Please complete the following steps to enable job throttling.

1. 
2. In the , select User menu > Admin console > Admin settings.

3. Locate the following settings and set their values accordingly:

   Setting	Description
   databricks.userClusterThrottling.enabled	When set to true, job throttling per Databricks cluster is enabled. Please specify the following settings.
   databricks.userClusterthrottling.maxTokensAllottedPerUserCluster	Set this value to the maximum number of concurrent jobs that can run on one user cluster. Default value is 20.
   databricks.userClusterthrottling.tokenExpiryInMinutes	The time in minutes after which tokens reserved by a job are revoked, irrespective of the job status. If a job is in progress and this limit is reached, then the Databricks token is expired, and the token is revoked under the assumption that it is stale. Default value is 120 (2 hours). Tip: Set this value to 0 to prevent token expiration. However, this setting is not recommended, as jobs can remain in the queue indefinitely.
   jobMonitoring.queuedJobTimeoutMinutes	The maximum time in minutes in which a job is permitted to remain in the queue for a slot on Databricks cluster. If this limit is reached, the job is marked as failed.
   batch-job-runner.cleanup.enabled	When set to true, the Batch Job Runner service is permitted to clean up throttling tokens and job-level personal access tokens. Tip: Unless you have reason to do otherwise, you should leave this setting to true.

   
   

4. Save your changes and restart the platform.

Configure personal access token

Each user must insert a Databricks Personal Access Token to access Databricks resources. For more information, see Databricks Settings Page.

Specify Databricks Tables cluster name

Individual users can specify the name of the cluster to which they are permissioned to access Databricks Tables. This cluster can also be shared among users. For more information, see Databricks Settings Page. 

Configure maximum retries for REST API 

There is a limit of 30 requests per second per workspace on the Databricks REST APIs. If this limit is reached, then a HTTP status code 429 error is returned, indicating that rate limiting is being applied by the server.  By default, the  re-attempts to submit a request 5 times and then fails the job if the request is not accepted. 

If you want to change the number of retries, change the value for the databricks.maxAPICallRetries  flag.  

Use

Run Job From Application

When the above configuration has been completed, you can select the running environment through the application.

See Run Job Page.

Run Job via API

You can use API calls to execute jobs.

Make sure that the request body contains the following:

    "execution": "databricksSpark",

For more information, see 

operation/runJobGroup

Troubleshooting

Spark job on AWS Databricks fails with "Invalid spark version" error

When running a job using Spark on AWS Databricks, the job may fail with the above invalid version error. In this case, the Databricks version of Spark has been deprecated.

Solution:

Since an AWS Databricks cluster is created for each user, the solution is to identify the cluster version to use, configure the platform to use it, and then restart the platform.

1. .
2. Acquire the value for databricks.sparkVersion.

3. In AWS Databricks, compare your value to the list of supported AWS Databricks version. If your version is unsupported, identify a new version to use. 

   NOTE: Ensure to note the version of Spark supported for the version of AWS Databricks that you have chosen.

   
   

4. In the  configuration, Set databricks.sparkVersion to the new version to use.

   NOTE: The value for spark.version does not apply to Databricks.

   
   

5. Restart the .
6. The platform is restarted. A new AWS Databricks cluster is created for each user using the specified values, when the user runs a job.

Spark job fails with "spark scheduler cannot be cast" error

When you run a job on Databricks, the job may fail with the following error:

java.lang.ClassCastException: org.apache.spark.scheduler.ResultTask cannot be cast to org.apache.spark.scheduler.Task
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:616)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

The job.log  file may contain something similar to the following:

2022-07-19T15:41:24.832Z - [sid=0cf0cff5-2729-4742-a7b9-4607ca287a98] - [rid=83eb9826-fc3b-4359-8e8f-7fbf77300878] - [Async-Task-9] INFO com.trifacta.databricks.spark.JobHelper - Got error org.apache.spark.SparkException: Stage 0 failed. Error: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, ip-10-243-149-238.eu-west-1.compute.internal, executor driver): java.lang.ClassCastException: org.apache.spark.scheduler.ResultTask cannot be cast to org.apache.spark.scheduler.Task
 at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:616)
...

This error is due to a class mismatch between the  and Databricks.

Solution:

The solution is to disable the precedence of using the Spark JARs provided from the  over the Databricks Spark JARs. Please perform the following steps:

1. 

2. Locate the spark.props  section and add the following configuration elements:

   "spark": { ... "props": { "spark.driver.userClassPathFirst": false, "spark.executor.userClassPathFirst": false, ... } },

   
   

3. Save your changes and restart the platform.