This section provides high-level information on how to configure the to integrate with Databricks hosted on AWS.
AWS Databricks is a unified data analytics platform that has been optimized for use on the AWS infrastructure.
Additional Databricks features supported by the platform:
Import datasets created from nested folders is not supported for running jobs from AWS Databricks.
AWS Databricks integration works with Spark 2.4.x only.
NOTE: The version of Spark for AWS Databricks must be applied to the platform configuration through the
Azure Databricks 7.3 (Recommended)
Azure Databricks 5.5 LTS
By default, the number of jobs permitted on an AWS workspace is set to
5000when using the run-submit API. This limit also affects jobs created by the REST API and notebook workflows. For more information, see "Configure Databricks job management" below.
These limits apply to any jobs that use workspace data on the cluster.
To enable retrieval and auditing of job information after a job has been completed, the does not delete jobs from the cluster. As a result, jobs can accumulate over time to exceed the number of jobs permitted on the cluster. If you reach these limits, you may receive a
Quota for number of jobs has been reached limit. For more information, see https://docs.databricks.com/user-guide/jobs.html.
Optionally, you can allow the to manage your jobs to avoid these limitations. For more information, see "Configure Databricks job management" below.
To enable AWS Databricks, perform the following configuration changes:
Locate the following parameter, which enables for smaller job execution. Set it to
Locate the following parameters. Set them to the values listed below, which enables AWS Databricks (small to extra-large jobs) running environments:
"webapp.runInDatabricks": true, "webapp.runWithSparkSubmit": false, "webapp.runInDataflow": false,
When a user submits a job, the provides all the cluster specifications in the Databricks API and it creates cluster only for per-user or per-job, that means once the job is complete, the cluster is terminated. Cluster creation may take less than 30 seconds if instance pools are used. If the instance pools are not used, it may take 10-15 minutes.
For more information on job clusters, see https://docs.databricks.com/clusters/configure.html.
The job clusters automatically terminate after the job is completed. A new cluster is automatically created when the user next requests access to AWS Databricks access.
When a user submits a job, creates a new cluster and persists the cluster ID in metadata for the user if the cluster does not exist or invalid. If the user already has an existing interactive valid cluster, then the existing cluster is reused when submitting the job.
Reset to JOB mode to run jobs in AWS Databricks.
When a user submits a job, provides all the cluster specifications in the Databricks API. Databricks creates a cluster only for this job and terminates it as soon as the job completes.Default cluster mode to run jobs in AWS Databricks.
EC2 instances can be configured with permissions to access AWS resources like S3 by attaching an IAM instance profile. Similarly, instance profiles can be attached to EC2 instances for use with AWS Databricks clusters.
NOTE: You must register the instance profiles in the Databricks workspace, or your Databricks clusters reject the instance profile ARNs and display an error. For more information, see https://docs.databricks.com/administration-guide/cloud-configurations/aws/instance-profiles.html#step-5-add-the-instance-profile-to-databricks.
To configure the instance profile for AWS Databricks, you must provide an IAM instance profile ARN in
NOTE: For AWS Databricks, you can configure the instance profile value in
|aws.credentialProvider||AWS Databricks permissions|
or Databricks jobs gets all permissions directly from the instance profile.
or Databricks jobs use temporary credentials that are issued based on system or user IAM roles.
NOTE: If the
For more information, see Configure for AWS Authentication.
Instance pooling reduces cluster node spin-up time by maintaining a set of idle and ready instances. The can be configured to leverage instance pooling on the Azure Databricks cluster for both worker and driver nodes.
NOTE: When instance pooling is enabled, the following parameters are not used:
For more information, see https://docs.azuredatabricks.net/clusters/instance-pools/index.html.
Acquire your pool identifier or pool name from Azure Databricks.
NOTE: You can use either the Databricks pool identifier or pool name. If both poolId and poolName are specified, poolId is used first. If that fails to find a matching identifier, then the poolName value is checked.
Tip: If you specify a poolName value only, then you can run your Databricks jobs against the available clusters across multiple . This mechanism allows for better resource allocation and broader execution options.
Set either of the following parameters:
Set the following parameter to the Azure Databricks pool identifier:
Or, you can set the following parameter to the Azure Databricks pool name:
The can be configured to use Databricks instance pooling for driver pools.
Acquire your driver pool identifier or driver pool name from Databricks.
NOTE: You can use either the Databricks driver pool identifier or driver pool name. If both driverPoolId and driverPoolName are specified, driverPoolId is used first. If that fails to find a matching identifier, then the driverPoolName value is checked.
Tip: If you specify a driverPoolName value only, then you can run your Databricks jobs against the available clusters across multiple . This mechanism allows for better resource allocation and broader execution options.
Set either of the following parameters:
Set the following parameter to the Databricks driver pool identifier:
Or, you can set the following parameter to the Databricks driver pool name:
Review and modify the following configuration settings, as required:
NOTE: Restart the platform after you modify the configuration settings for the system to take affect.
Following is the list of parameters that have to be set to integrate the AWS Databricks with :
|URL to the AWS Databricks Service where Spark jobs will be run||-|
Must be set to
Following is the list of parameters that can be reviewed or modified based on your requirements:
Number of initial cluster nodes to be placed on on-demand instances. The remainder is placed on availability instances
Availability type used for all subsequent nodes past the firstOnDemandInstances.
Identifier for the availability zone/datacenter in which the cluster resides. The provided availability zone must be in the same region as the Databricks deployment.
The max price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. When spot instances are requested for this cluster, only spot instances whose max price percentage matches this field will be considered.
The type of EBS volumes that will be launched with this cluster.
EC2 instance profile ARN for the cluster nodes. This is only used when AWS credential provider is set to temporary/instance. The instance profile must have previously been added to the Databricks environment by an account administrator.
|For more information, see Configure for AWS Authentication.|
Determines the cluster mode for running a Databricks job.
|feature.parameterization.matchLimitOnSampling.databricksSpark||Maximum number of parameterized source files that are permitted for matching in a single dataset with parameters.||Default: 0|
|databricks.workerNodeType||Type of node to use for the AWS Databricks Workers/Executors. There are 1 or more Worker nodes per cluster.|
|databricks.sparkVersion||AWS Databricks runtime version which also references the appropriate version of Spark.|
Depending on your version of AWS Databricks, please set this property according to the following:
Please do not use other values.
|databricks.minWorkers||Initial number of Worker nodes in the cluster, and also the minimum number of Worker nodes that the cluster can scale down to during auto-scale-down.|
Increasing this value can increase compute costs.
|databricks.maxWorkers||Maximum number of Worker nodes the cluster can create during auto scaling.|
Minimum value: Not less than
Increasing this value can increase compute costs.
If you have enabled instance pooling in AWS Databricks, you can specify the pool identifier here.
|databricks.poolName||If you have enabled instance pooling in AWS Databricks, you can specify the pool name here.|
Type of node to use for the AWS Databricks Driver. There is only one Driver node per cluster.
For more information, see the sizing guide for Databricks.
|databricks.driverPoolId||If you have enabled instance pooling in AWS Databricks, you can specify the driver node pool identifier here. For more information, see Configure instance pooling below.|
|databricks.driverPoolName||If you have enabled instance pooling in AWS Databricks, you can specify the driver node pool name here. For more information, see Configure instance pooling below.|
|databricks.logsDestination||DBFS location that cluster logs will be sent to every 5 minutes||Leave this value as |
|databricks.enableAutotermination||Set to true to enable auto-termination of a user cluster after N minutes of idle time, where N is the value of the autoterminationMinutes property.||Unless otherwise required, leave this value as |
|databricks.clusterStatePollerDelayInSeconds||Number of seconds to wait between polls for AWS Databricks cluster status when a cluster is starting up|
|databricks.clusterStartupWaitTimeInMinutes||Maximum time in minutes to wait for a Cluster to get to Running state before aborting and failing an AWS Databricks job.||Default: 60|
Maximum time in minutes to wait for a Cluster to complete syncing its logs to DBFS before giving up on pulling the cluster logs to the .
|Set this to |
|databricks.clusterLogSyncPollerDelayInSeconds||Number of seconds to wait between polls for a Databricks cluster to sync its logs to DBFS after job completion.||Default: 20|
|databricks.autoterminationMinutes||Idle time in minutes before a user cluster will auto-terminate.||Do not set this value to less than the cluster startup wait time value.|
|databricks.maxAPICallRetries||Maximum number of retries to perform in case of 429 error code response||Default: 5. For more information, see Configure Maximum Retries for REST API section below.|
Enables encryption of data like shuffle data that is temporarily stored on cluster's local disk.
|databricks.patCacheTTLInMinutes||Lifespan in minutes for the Databricks personal access token in-memory cache||Default: 10|
AWS Databricks enforces a hard limit of 1000 created jobs per workspace, and by default cluster jobs are not deleted. To support jobs more than 1000 jobs per cluster, you can enable job management for AWS Databricks.
NOTE: This feature covers the deletion of the job definition on the cluster, which counts toward the enforced limits. The never deletes the outputs of a job or the job definition stored in the platform. When cluster job definitions are removed, the jobs remain listed in the Jobs page, and job metadata is still available. There is no record of the job on the AWS Databricks cluster. Jobs continue to run, but users on the cluster may not be aware of them.
Tip: Regardless of your job management option, when you hit the limit for the number of job definitions that can be created on the Databricks workspace, the platform by default falls back to using the runs/submit API, if the Databricks Job Runs Submit Fallback setting has been enabled.
Locate the following property and set it to one of the values listed below:
Databricks Job Management
|Never Delete||(default) Job definitions are never deleted from the AWS Databricks cluster.|
|Always Delete||The AWS Databricks job definition is deleted during the clean-up phase, which occurs after a job completes.|
|Delete Successful Only||When a job completes successfully, the AWS Databricks job definition is deleted during the clean-up phase. Failed or canceled jobs are not deleted, which allows you to debug as needed.|
|Skip Job Creation|
For jobs that are to be executed only one time, the can be configured to use a different mechanism for submitting the job. When this option is enabled, the submits jobs using the run-submit API, instead of the run-now API. The run-submit API does not create an AWS Databricks job definition. Therefore the submitted job does not count toward the enforced job limit.
|Default||Inherits the default system-wide setting.|
When this feature is enabled, the platform falls back to use the runs/submit API as a fallback when the job limit for the Databricks workspace has been reached:
Databricks Job Runs Submit Fallback
Optionally, you can provide to the the name of a shared Databricks cluster to be used to access Databricks Tables.
NOTE: Any shared cluster must be maintained by the customer.
Depending on the credential provider type, the following properties must be specified in the Spark configuration for the shared cluster.
Default credential provider:
"fs.s3a.access.key" "fs.s3a.secret.key" "spark.hadoop.fs.s3a.access.key" "spark.hadoop.fs.s3a.secret.key"
For more information, see https://docs.databricks.com/data/data-sources/aws/amazon-s3.html.
Instance credential provider:
"fs.s3a.credentialsType" "fs.s3a.stsAssumeRole.arn" "fs.s3a.canned.acl" "fs.s3a.acl.default" "spark.hadoop.fs.s3a.credentialsType" "spark.hadoop.fs.s3a.stsAssumeRole.arn" "spark.hadoop.fs.s3a.canned.acl" "spark.hadoop.fs.s3a.acl.default"
For more information, see https://docs.databricks.com/administration-guide/cloud-configurations/aws/instance-profiles.html.
Temporary credential provider:
"fs.s3a.canned.acl" "fs.s3a.acl.default" "spark.hadoop.fs.s3a.canned.acl" "spark.hadoop.fs.s3a.acl.default"
For more information, see https://docs.databricks.com/administration-guide/cloud-configurations/aws/assume-role.html.
Locate the following parameter, and add the name of the Databricks cluster to use to browse Databricks Table:
When a cluster name is provided:
NOTE: While using a single cluster shared across users to access Databricks Tables, each user must have a valid Databricks Personal Access Token to the shared cluster.
If a cluster name is not provided:
Individual users can specify the name of the cluster that accesses Databricks Tables through the Databricks settings. See "Specify Databricks Tables cluster name" below.
The AWS Secrets Manager is a secure vault for storing access credentials to AWS resources. It is mandatory to use with AWS Databricks. For more information, see Configure for AWS Secrets Manager.
A single AWS Databricks account can have access to multiple Databricks workspaces. You can create more than one workspace by using Account API if you are account is on the E2 version of the platform or on a selected custom plan that allows multiple workspaces per account.
For more information, see https://docs.databricks.com/administration-guide/account-api/new-workspace.html
Each workspace has a unique deployment name associated with it that defines the workspace URL. For example:
For more information, see Databricks Settings Page.
For more information, see Configure Platform section above.
By default, Databricks workspaces apply limits on the number of jobs that can be submitted before the cluster begins to fail jobs. These limits are the following:
Maximum number of concurrent jobs per cluster
Max number of concurrent jobs per workspace
Max number of concurrent clusters per workspace
Depending on how your clusters are configured, these limits can vary. For example, if the maximum number of concurrent jobs per cluster is set to
20, then the 21st concurrent job submitted to the cluster fails.
To prevent unnecessary job failure, the submits the throttling of jobs to Databricks. When job throttling is enabled and the 21 concurrent job is submitted, the holds that job internally the first of any of the following events happens:
NOTE: The supports throttling of jobs based on the maximum number of concurrent jobs per cluster. Throttling against the other limits listed above is not supported at this time.
Please complete the following steps to enable job throttling.
Locate the following settings and set their values accordingly:
When set to
Set this value to the maximum number of concurrent jobs that can run on one user cluster. Default value is
The time in minutes after which tokens reserved by a job are revoked, irrespective of the job status. If a job is in progress and this limit is reached, then the Databricks token is expired, and the token is revoked under the assumption that it is stale. Default value is
|jobMonitoring.queuedJobTimeoutMinutes||The maximum time in minutes in which a job is permitted to remain in the queue for a slot on Databricks cluster. If this limit is reached, the job is marked as failed.|
When set to
Each user must insert a Databricks Personal Access Token to access Databricks resources. For more information, see Databricks Settings Page.
Individual users can specify the name of the cluster to which they are permissioned to access Databricks Tables. This cluster can also be shared among users. For more information, see Databricks Settings Page.
There is a limit of 30 requests per second per workspace on the Databricks REST APIs. If this limit is reached, then a HTTP status code 429 error is returned, indicating that rate limiting is being applied by the server. By default, the re-attempts to submit a request
5 times and then fails the job if the request is not accepted.
If you want to change the number of retries, change the value for the
(default) When a request is submitted through the AWS Databricks REST APIs, up to
When an API call fails, the request fails. As the number of concurrent jobs increases, more jobs may fail.
|5+||Increasing this setting above the default value may result in more requests eventually getting processed. However, increasing the value may consume additional system resources in a high concurrency environment and jobs might take longer to run due to exponentially increasing waiting time.|
When the above configuration has been completed, you can select the running environment through the application.
NOTE: When a Databricks job fails, the failure is reported immediately in the . In the background, the job logs are collected from Databricks and may not be immediately available.
See Run Job Page.
You can use API calls to execute jobs.
Make sure that the request body contains the following:
For more information, see
When running a job using Spark on AWS Databricks, the job may fail with the above invalid version error. In this case, the Databricks version of Spark has been deprecated.
Since an AWS Databricks cluster is created for each user, the solution is to identify the cluster version to use, configure the platform to use it, and then restart the platform.
In AWS Databricks, compare your value to the list of supported AWS Databricks version. If your version is unsupported, identify a new version to use.
NOTE: Ensure to note the version of Spark supported for the version of AWS Databricks that you have chosen.
In the configuration, Set
databricks.sparkVersion to the new version to use.
NOTE: The value for