This section describes how to configure the  to integrate with Databricks hosted in Azure. 

• Azure Databricks is an Apache Spark implementation that has been optimized for use on the Azure platform. For more information, see https://databricks.com/product/azure.

Additional Databricks features supported by the platform:

• Table access control: https://docs.databricks.com/security/access-control/table-acls/object-privileges.html

Pre-requisites

• The  must be deployed in Microsoft Azure.  

  NOTE: If you are using Azure Databricks as a datasource, please verify that openJDKv1.8.0_242 or earlier is installed on the . Java 8 is required. If necessary, downgrade the Java version and restart the platform. There is a known issue with TLS v1.3.

  
  

Simba driver

Beginning in Release 7.1, the integration with Azure Databricks switched from using a Hive-based driver to a Simba driver for the integration with Spark. 

No installation is required to use this new driver.

For more information on the Simba driver, see https://databricks.com/spark/odbc-driver-download.

Limitations

• The  must be installed on Microsoft Azure.

• Import from nested folders is not supported for running jobs from Azure Databricks. 

• When a job is started and no cluster is available, a cluster is initiated, which can take up to four minutes. If the job is canceled during cluster startup:
  • The job is terminated, and the cluster remains. 
  • The job is reported in the application as Failed, instead of Canceled.

• Azure Databricks integration works with Spark 2.4.x only.

  NOTE: The version of Spark for Azure Databricks must be applied to the platform configuration through the databricks.sparkVersion property. Details are provided later.

  
  

• Azure Databricks integration does not work with Hive.

Supported versions of Azure Databricks

Supported versions:

• Azure Databricks 7.3 (Recommended)

• Azure Databricks 6.x
• Azure Databricks 5.5 LTS 

For more information on supported versions, see https://docs.databricks.com/release-notes/runtime/releases.html#supported-databricks-runtime-releases-and-support-schedule.

Job limits

By default, the number of jobs permitted on an Azure Databricks cluster is set to 1000.

• The number of jobs that can be created per workspace in an hour is limited to 1000.
  • If Databricks Job Management is enabled in the platform, then this limit is raised to 5000 by using the run-submit API. For more information, see "Configure Databricks job management" below.
  • For more information, see https://docs.databricks.com/dev-tools/api/latest/jobs.html#run-now.  
• These limits apply to any jobs run for workspace data on the cluster.
• The number of actively concurrent job runs in a workspace is limited to 150. 

Managing limits:

To enable retrieval and auditing of job information after a job has been completed, the does not delete jobs from the cluster. As a result, jobs can accumulate over time to exceeded the number of jobs permitted on the cluster. If you reach these limits, you may receive a Quota for number of jobs has been reached limit. For more information, see https://docs.databricks.com/user-guide/jobs.html.

Optionally, you can allow the  to manage your jobs to avoid these limitations. For more information, see "Configure Databricks job management" below.

Create Cluster

When a user first requests access to Azure Databricks, a new Azure Databricks cluster is created for the user. Access can include a request to run a job or to browse Databricks Tables. Cluster creation may take a few minutes.

A new cluster is also created when a user launches a job after:

• The Azure Databricks configuration properties or Spark properties are changed in platform configuration.
• A JAR file is updated on the 
  

A user's cluster automatically terminates after a configurable time period. A new cluster is automatically created when the user next requests access to Azure Databricks access. See "Configure Platform" below.

Enable

To enable Azure Databricks, please perform the following configuration changes. 

Steps:

1. 

2. Locate the following parameter, which enables  for smaller job execution. Set it to Enabled:
   
   

   Photon execution

   
   

3. You do not need to save to enable the above configuration change.
4. 

5. Locate the following parameters. Set them to the values listed below, which enables Azure Databricks (small to extra-large jobs) running environments:

   "webapp.runInDatabricks": true, "webapp.runWithSparkSubmit": false, "webapp.runinEMR": false, "webapp.runInDataflow": false,

   
   

6. Locate the following parameter and set it to azure:

   "metadata.cloud": "azure",

   
   

   NOTE: Do not set this parameter to any value other than azure while using Azure Databricks.

   
   

7. Do not save your changes until you have completed the following configuration section.

Configure

Configure cluster mode 

When a user submits a job, the  provides all the cluster specifications in the Databricks API and it creates cluster only for per-user or per-job. that means once the job is complete, the cluster is terminated.  Cluster creation may take less than 30 seconds if instance pools are used. If the instance pools are not used, it may take 10-15 minutes.

For more information on job clusters, see https://docs.databricks.com/clusters/configure.html.

The job clusters automatically terminate after the job is completed. A new cluster is automatically created when the user next requests access to Azure Databricks access. 

Configure Platform

Please review and modify the following configuration settings.

Configure instance pooling

Instance pooling reduces cluster node spin-up time by maintaining a set of idle and ready instances. The  can be configured to leverage instance pooling on the Azure Databricks cluster for both worker and driver nodes.

For more information, see https://docs.azuredatabricks.net/clusters/instance-pools/index.html.

Instance pooling for worker nodes

Pre-requisites:

• All cluster nodes used by the are taken from the pool. If the pool has an insufficient number of nodes, cluster creation fails.
  
• Each user must have access to the pool and must have at least the ATTACH_TO permission.
• Each user must have a personal access token from the same Azure Databricks workspace. See Configure personal access token below.

To enable:

1. Acquire your pool identifier or pool name from Azure Databricks.
   
   

   NOTE: You can use either the Databricks pool identifier or pool name. If both poolId and poolName are specified, poolId is used first. If that fails to find a matching identifier, then the poolName value is checked.

   
   

   Tip: If you specify a poolName value only, then you can run your Databricks jobs against the available clusters across multiple . This mechanism allows for better resource allocation and broader execution options.

   
   

2. 

3. Set either of the following parameters: 

   1. Set the following parameter to the Azure Databricks pool identifier:

      "databricks.poolId": "<my_pool_id>",

      
      

   2. Or, you can set the following parameter to the Azure Databricks pool name:

      "databricks.poolName": "<my_pool_name>",

      
      

4. Save your changes and restart the platform.

Instance pooling for driver nodes

The  can be configured to use Databricks instance pooling for driver pools.

To enable:

1. Acquire your driver pool identifier or driver pool name from Databricks.
   
   

   NOTE: You can use either the Databricks driver pool identifier or driver pool name. If both driverPoolId and driverPoolName are specified, driverPoolId is used first. If that fails to find a matching identifier, then the driverPoolName value is checked.

   
   

   Tip: If you specify a driverPoolName value only, then you can run your Databricks jobs against the available clusters across multiple . This mechanism allows for better resource allocation and broader execution options.

   
   

2. 

3. Set either of the following parameters: 

   1. Set the following parameter to the Databricks driver pool identifier:

      "databricks.driverPoolId": "<my_pool_id>",

      
      

   2. Or, you can set the following parameter to the Databricks driver pool name:

      "databricks.driverPoolName": "<my_pool_name>",

      
      

4. Save your changes and restart the platform.

Configure Databricks job management

Azure Databricks enforces a hard limit of 1000 created jobs per workspace, and by default cluster jobs are not deleted. To support jobs more than 1000 jobs per cluster, you can enable job management for Azure Databricks.

Steps:

1. 

2. Locate the following property and set it to one of the values listed below:
   
   

   Databricks Job Management

   
   

   Property Value	Description
   Never Delete	(default) Job definitions are never deleted from the Azure Databricks cluster.
   Always Delete	The Azure Databricks job definition is deleted during the clean-up phase, which occurs after a job completes.
   Delete Successful Only	When a job completes successfully, the Azure Databricks job definition is deleted during the clean-up phase. Failed or canceled jobs are not deleted, which allows you to debug as needed.
   Skip Job Creation	For jobs that are to be executed only one time, the can be configured to use a different mechanism for submitting the job. When this option is enabled, the submits jobs using the run-submit API, instead of the run-now API. The run-submit API does not create an Azure Databricks job definition. Therefore the submitted job does not count toward the enforced job limit.
   Default	Inherits the default system-wide setting.

   
   

3. When this feature is enabled, the platform falls back to use the runs/submit API as a fallback when the job limit for the Databricks workspace has been reached:

   Databricks Job Runs Submit Fallback

   
   

4. Save your changes and restart the platform.

Configure Databricks job throttling

By default, Databricks workspaces apply limits on the number of jobs that can be submitted before the cluster begins to fail jobs. These limits are the following:

• Maximum number of concurrent jobs per cluster

• Max number of concurrent jobs per workspace

• Max number of concurrent clusters per workspace

Depending on how your clusters are configured, these limits can vary. For example, if the maximum number of concurrent jobs per cluster is set to 20, then the 21st concurrent job submitted to the cluster fails. 

To prevent unnecessary job failure, the  submits the throttling of jobs to Databricks. When job throttling is enabled and the 21 concurrent job is submitted, the  holds that job internally the first of any of the following events happens:

• An active job on the cluster completes, and space is available for submitting a new job. The job is then submitted.
• The user chooses to cancel the job.
• One of the timeout limits described below is reached.

Steps:

Please complete the following steps to enable job throttling.

1. 
2. In the , select User menu > Admin console > Admin settings.

3. Locate the following settings and set their values accordingly:

   Setting	Description
   databricks.userClusterThrottling.enabled	When set to true, job throttling per Databricks cluster is enabled. Please specify the following settings.
   databricks.userClusterthrottling.maxTokensAllottedPerUserCluster	Set this value to the maximum number of concurrent jobs that can run on one user cluster. Default value is 20.
   databricks.userClusterthrottling.tokenExpiryInMinutes	The time in minutes after which tokens reserved by a job are revoked, irrespective of the job status. If a job is in progress and this limit is reached, then the Databricks token is expired, and the token is revoked under the assumption that it is stale. Default value is 120 (2 hours). Tip: Set this value to 0 to prevent token expiration. However, this setting is not recommended, as jobs can remain in the queue indefinitely.
   jobMonitoring.queuedJobTimeoutMinutes	The maximum time in minutes in which a job is permitted to remain in the queue for a slot on Databricks cluster. If this limit is reached, the job is marked as failed.
   batch-job-runner.cleanup.enabled	When set to true, the Batch Job Runner service is permitted to clean up throttling tokens and job-level personal access tokens. Tip: Unless you have reason to do otherwise, you should leave this setting to true.

   
   

4. Save your changes and restart the platform.

Configure personal access token

Each user must insert a Databricks Personal Access Token to access Databricks resources. For more information, see Databricks Settings Page.

Combine transform and profiling for Spark jobs

When profiling is enabled for a Spark job, the transform and profiling tasks are combined by default. As needed, you can separate these two tasks. Publishing behaviors vary depending on the approach. For more information, see Configure for Spark.

Additional Configuration

Enable SSO for Azure Databricks

To enable SSO authentication with Azure Databricks, you enable SSO integration with Azure AD. For more information, see Configure SSO for Azure AD.

Enable Azure Managed Identity access

For enhanced security, you can configure the to use an Azure Managed Identity. When this feature is enabled, the platform queries the Key Vault for the secret holding the applicationId and secret to the service principal that provides access to the Azure services.

To enable, the following parameters for the  must be specified.

Save your changes.

Enable shared cluster for Databricks Tables

Optionally, you can provide to the  the name of a shared Databricks cluster to be used to access Databricks Tables. 

Pre-requisites:

• Shared clusters are not provisioned per-user and cannot be used to run Spark jobs.
• If you have enabled table access control on your high-concurrency cluster, you must configure access to data objects for users. For more information, see https://docs.databricks.com/security/access-control/table-acls/object-privileges.html.
• If only a limited number of users are using the shared cluster, you must configure the attach permission on the shared cluster in the Databricks workspace. 

Configure cluster

Depending on the credential provider type, the following properties must be specified in the Spark configuration for the shared cluster.

ADLS Gen1 Credentials:

"dfs.adls.oauth2.client.id"
"dfs.adls.oauth2.credential"
"dfs.adls.oauth2.refresh.url"
"dfs.adls.oauth2.access.token.provider.type"
"dfs.adls.oauth2.access.token.provider"
"fs.azure.account.oauth2.client.endpoint"
"spark.hadoop.dfs.adls.oauth2.client.id"
"spark.hadoop.dfs.adls.oauth2.credential"
"spark.hadoop.dfs.adls.oauth2.refresh.url"
"spark.hadoop.fs.azure.account.oauth2.client.endpoint"
"spark.hadoop.dfs.adls.oauth2.access.token.provider.type"
"spark.hadoop.dfs.adls.oauth2.access.token.provider"

For more information, see https://docs.databricks.com/data/data-sources/azure/azure-datalake.html.

ADLS Gen2 Credentials:

"fs.azure.account.auth.type"
"fs.azure.account.oauth.provider.type"
"fs.azure.account.oauth2.client.id"
"fs.azure.account.oauth2.client.secret"
"fs.azure.account.oauth2.client.endpoint"
"spark.hadoop.fs.azure.account.auth.type"
"spark.hadoop.fs.azure.account.oauth.provider.type"
"spark.hadoop.fs.azure.account.oauth2.client.id"
"spark.hadoop.fs.azure.account.oauth2.client.secret"
"spark.hadoop.fs.azure.account.oauth2.client.endpoint"

For more information, see https://docs.databricks.com/data/data-sources/azure/azure-datalake-gen2.html.

WASB Credentials:

fs.azure.sas.<container-name>.<storage-account-name>.blob.core.windows.net
spark.hadoop.fs.azure.sas.<container-name>.<storage-account-name>.blob.core.windows.net
azure.wasb.stores

For more information, see https://docs.databricks.com/data/data-sources/azure/azure-storage.html.

Enable

Steps:

1. 

2. Locate the following parameter, and add the name of the Databricks cluster to use to browse Databricks Table:
   
   

   "feature.databricks.connection.clusterName": "<your_cluster_name>",

   
   

3. Save your changes and restart the cluster.

When a cluster name is provided:

• If the cluster is available, all users of the  attempt to connect to Databricks Tables through the listed cluster.
• If the cluster has been terminated, a message indicates that the cluster must be restarted. After it has been restarted, you can try again.
• If the cluster name is invalid, the  fails any read/write operations to Databricks Tables.

If a cluster name is not provided:

• In USER mode, the default behavior is to create one interactive cluster for each user to browse Databricks Tables.
• In JOB mode, the interactive cluster is created only when a user needs to browse Databricks Tables.

Configure user-level override for Databricks Tables access cluster

Individual users can specify the name of the cluster that accesses Databricks Tables through the Databricks settings. See "Specify Databricks Tables cluster name" below.

Specify Databricks Tables cluster name

Individual users can specify the name of the cluster to which they are permissioned to access Databricks Tables. This cluster can also be shared among users. For more information, see Databricks Settings Page. 

Pass additional Spark properties

As needed, you can pass additional properties to the Spark running environment through the spark.props configuration area.

Steps:

1. 
2. Search for the following property: spark.props.

3. Insert new Spark properties. For example, you can specify the spark.props.spark.executor.memory property, which changes the memory allocated to the Spark executor on each node by using the following in the spark.props area:
   
   

   "spark": { ... "props": { "spark.executor.memory": "6GB" } ... }

   
   

4. Save your changes and restart the platform.

For more information on modifying these settings, see Configure for Spark.

Configure Maximum Retries for REST API 

There is a limit of 30 requests per second per workspace on the Databricks REST APIs. If this limit is reached, then a HTTP status code 429 error is returned, indicating that rate limiting is being applied by the server.  By default, the  re-attempts to submit a request 5 times and then fails the job if the request is not accepted. 

If you want to change the number of retries, change the value for the databricks.maxAPICallRetries  flag.  

Use

Run job from application

When the above configuration has been completed, you can select the running environment through the application.

See Run Job Page.

Run job via API

You can use API calls to execute jobs.

Please make sure that the request body contains the following:

    "execution": "databricksSpark",

For more information, see 

operation/runJobGroup

Troubleshooting

Spark job on Azure Databricks fails with "Invalid spark version" error

When running a job using Spark on Azure Databricks, the job may fail with the above invalid version error. In this case, the Databricks version of Spark has been deprecated.

Solution:

Since an Azure Databricks cluster is created for each user, the solution is to identify the cluster version to use, configure the platform to use it, and then restart the platform.

1. 
2. Acquire the value for databricks.sparkVersion.

3. In Azure Databricks, compare your value to the list of supported Azure Databricks version. If your version is unsupported, identify a new version to use. 

   NOTE: Please make note of the version of Spark supported for the version of Azure Databricks that you have chosen.

   
   

4. In the  configuration, set databricks.sparkVersion to the new version to use.

   NOTE: The value for spark.version does not apply to Databricks.

   
   

5. Restart the .
6. The platform is restarted. A new Azure Databricks cluster is created for each user using the specified values, when the user runs a job.

Spark job fails with "spark scheduler cannot be cast" error

When you run a job on Databricks, the job may fail with the following error:

java.lang.ClassCastException: org.apache.spark.scheduler.ResultTask cannot be cast to org.apache.spark.scheduler.Task
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:616)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

The job.log  file may contain something similar to the following:

2022-07-19T15:41:24.832Z - [sid=0cf0cff5-2729-4742-a7b9-4607ca287a98] - [rid=83eb9826-fc3b-4359-8e8f-7fbf77300878] - [Async-Task-9] INFO com.trifacta.databricks.spark.JobHelper - Got error org.apache.spark.SparkException: Stage 0 failed. Error: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, ip-10-243-149-238.eu-west-1.compute.internal, executor driver): java.lang.ClassCastException: org.apache.spark.scheduler.ResultTask cannot be cast to org.apache.spark.scheduler.Task
 at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:616)
...

This error is due to a class mismatch between the  and Databricks.

Solution:

The solution is to disable the precedence of using the Spark JARs provided from the  over the Databricks Spark JARs. Please perform the following steps:

1. 

2. Locate the spark.props  section and add the following configuration elements:

   "spark": { ... "props": { "spark.driver.userClassPathFirst": false, "spark.executor.userClassPathFirst": false, ... } },

   
   

3. Save your changes and restart the platform.