This section describes how to configure the to integrate with Databricks hosted in Azure.
NOTE: You cannot integrate with existing Azure Databricks clusters.
NOTE: If you are using Azure AD to integrate with an Azure Databricks cluster, the Azure AD secret value stored in
Nested folders are not supported when running jobs from Azure Databricks.
NOTE: Avoid including spaces in the paths to your ADLS sources. Spaces in the path value can cause errors during execution on Databricks.
Azure Databricks integration works with Spark 2.4.x only.
NOTE: The version of Spark for Azure Databricks must be applied to the platform configuration through the
By default, the number of jobs permitted on an Azure Databricks cluster is set to
NOTE: To enable retrieval and auditing of job information after a job has been completed, the does not delete jobs from the cluster. As a result, jobs can accumulate over time to exceeded the number of jobs permitted on the cluster. You should periodically delete jobs on your Azure Databricks cluster to prevent reaching these limits and receiving a
For more information, see https://docs.databricks.com/user-guide/jobs.html.
NOTE: Integration with pre-existing Azure Databricks clusters is not supported.
When a user first requests access to Azure Databricks, a new Azure Databricks cluster is created for the user. Access can include a request to run a job or to browse Databricks Tables. Cluster creation may take a few minutes.
A new cluster is also created when a user launches a job after:
A user's cluster automatically terminates after a configurable time period. A new cluster is automatically created when the user next requests access to Azure Databricks access. See "Configure Platform" below.
To enable Azure Databricks, please perform the following configuration changes.
Locate the following parameters. Set them to the values listed below, which enable the (smaller jobs) and Azure Databricks (small to extra-large jobs) running environments:
"webapp.runInTrifactaServer": true, "webapp.runInDatabricks": true, "webapp.runInHadoop": false, "webapp.runinEMR": false, "webapp.runInDataflow": false,
Please review and modify the following configuration settings.
NOTE: When you have finished modifying these settings, save them and restart the platform to apply.
|feature.parameterization.maxNumberOfFilesForExecution.databricksSpark||Maximum number of parameterized source files that are permitted to be executed as part of an Azure Databricks job.|
|feature.parameterization.matchLimitOnSampling.databricksSpark||Maximum number of parameterized source files that are permitted for matching in a single dataset with parameters.|
|databricks.workerNodeType||Type of node to use for the Azure Databricks Workers/Executors. There are 1 or more Worker nodes per cluster.|
For more information, see the sizing guide for Azure Databricks.
|databricks.sparkVersion||Azure Databricks cluster version which also includes the Spark Version.|
Please do not change unless you are using a non-default version of Spark. For more information, see Configure for Spark.
|databricks.serviceUrl||URL to the Azure Databricks Service where Spark jobs will be run (Example: https://westus2.azuredatabricks.net)|
|databricks.minWorkers||Initial number of Worker nodes in the cluster, and also the minimum number of Worker nodes that the cluster can scale down to during auto-scale-down|
Increasing this value can increase compute costs.
|databricks.maxWorkers||Maximum number of Worker nodes the cluster can create during auto scaling|
Minimum value: Not less than
Increasing this value can increase compute costs.
|databricks.logsDestination||DBFS location that cluster logs will be sent to every 5 minutes||Leave this value as |
|databricks.enableAutotermination||Set to true to enable auto-termination of a user cluster after N minutes of idle time, where N is the value of the autoterminationMinutes property.||Unless otherwise required, leave this value as |
|databricks.driverNodeType||Type of node to use for the Azure Databricks Driver. There is only 1 Driver node per cluster.|
For more information, see the sizing guide for Databricks.
|databricks.clusterStatePollerDelayInSeconds||Number of seconds to wait between polls for Azure Databricks cluster status when a cluster is starting up|
|databricks.clusterStartupWaitTimeInMinutes||Maximum time in minutes to wait for a Cluster to get to Running state before aborting and failing an Azure Databricks job|
Maximum time in minutes to wait for a Cluster to complete syncing its logs to DBFS before giving up on pulling the cluster logs to the .
|Set this to |
|databricks.clusterLogSyncPollerDelayInSeconds||Number of seconds to wait between polls for a Databricks cluster to sync its logs to DBFS after job completion|
|databricks.autoterminationMinutes||Idle time in minutes before a user cluster will auto-terminate.||Do not set this value to less than the cluster startup wait time value.|
After the above configuration has been performed, each user must insert their personal access token into their User Settings page. This configuration enables the user to authenticate using the Azure Databricks REST APIs, which enables the execution of jobs.
NOTE: Each user must apply a personal access token to their User Profile. Users that do not provide a personal authentication token cannot run jobs on Azure Databricks, including transformation, sampling, and profiling jobs.
In the Databricks Personal Access Token field, paste your token.
Databricks user configuration
Azure Databricks personal access tokens are saved in the Azure key vault.
To enable SSO authentication with Azure Databricks, you enable SSO integration with Azure AD. For more information, see Configure SSO for Azure AD.
For enhanced security, you can configure the to use an Azure Managed Identity. When this feature is enabled, the platform queries the Key Vault for the secret holding the applicationId and secret to the service principal that provides access to the Azure services.
NOTE: This feature is supported for Azure Databricks only.
NOTE: Your Azure Key Vault must already be configured, and the applicationId and secret must be available in the Key Vault. See Configure for Azure.
To enable, the following parameters for the must be specified.
|azure.managedIdentities.enabled||Set to |
|azure.managedIdentities.keyVaultApplicationidSecretName||Specify the name of the Azure Key Vault secret that holds the service principal Application Id.|
|azure.managedIdentities.keyVaultApplicationSecretSecretName||Specify the name of the Key Vault secret that holds the service principal secret.|
Save your changes.
When the above configuration has been completed, you can select the running environment through the application. See Run Job Page.
You can run jobs on Azure Databricks via CLI. When executing, the
job_type parameter must be set to
databricksSpark. See CLI for Jobs.
You can use API calls to execute jobs.
Please make sure that the request body contains the following:
For more information, see API JobGroups Create v4.
When running a job using Spark on Azure Databricks, the job may fail with the above invalid version error. In this case, the Databricks version of Spark has been deprecated.
Since an Azure Databricks cluster is created for each user, the solution is to identify the cluster version to use, configure the platform to use it, and then restart the platform.
In Azure Databricks, compare your value to the list of supported Azure Databricks version. If your version is unsupported, identify a new version to use.
NOTE: Please make note of the version of Spark supported for the version of Azure Databricks that you have chosen.
databricks.sparkVersionto the new version to use.
spark.versionto the appropriate version of Spark to use.