Page tree

Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DEV and version r0871



This section provides high-level information on how to configure the

D s platform
to integrate with Databricks hosted on AWS.

AWS Databricks is a unified data analytics platform that has been optimized for use on the 
AWS infrastructure.

Additional Databricks features supported by the platform:


  • The 
    D s platform
     must be installed in a customer-managed AWS environment.
  • The base storage layer must be set to S3. For more information, see Set Base Storage Layer.
  • AWS Secrets Manager is required for AWS Databricks use. For more information, see Configure for AWS Secrets Manager.


  1. D s config
  2. Locate the following parameter, which enables 

    D s photon
     for smaller job execution. Set it to Enabled:

    Code Block
    Photon execution
  3. You do not need to save to enable the above configuration change.
  4. D s config
  5. Locate the following parameters. Set them to the values listed below, which enables AWS Databricks (small to extra-large jobs) running environments:

    Code Block
    "webapp.runInDatabricks": true,
    "webapp.runWithSparkSubmit": false,
    "webapp.runInDataflow": false,
  6. Do not save your changes until you have completed the following configuration section.


Configure cluster mode 

When a user submits a job, the 

D s product
 provides all the cluster specifications in the Databricks API and it creates cluster only for per-user or per-job, that means once the job is complete, the cluster is terminated.  Cluster creation may take less than 30 seconds if instance pools are used. If the instance pools are not used, it may take 10-15 minutes.

For more information on job clusters, see .

The job clusters automatically terminate after the job is completed. A new cluster is automatically created when the user next requests access to AWS Databricks access. 

Cluster ModeDescription

When a user submits a job,

D s product
 creates a new cluster and persists the cluster ID in 
D s product
 metadata for the user if the cluster does not exist or invalid. If the user already has an existing interactive valid cluster, then the existing cluster is reused when submitting the job.

Reset to JOB mode to run jobs in AWS Databricks.


When a user submits a job,

D s product
provides all the cluster specifications in the Databricks API. Databricks creates a cluster only for this job and terminates it as soon as the job completes.

Default cluster mode to run jobs in AWS Databricks.


Optionally, you can configure the 

D s platform
 to use the Databricks cluster policies that have been specified by your Databricks administrator for creating and using clusters. These policies are effectively templates for creation and use of Databricks clusters and govern aspects of clusters such as the type and count of nodes the resources that can be accessed via the cluster, and other settings. For more information on Databricks cluster policies, see


Code Block
  "autoscale.max_workers": {
    "type": "fixed",
    "value": 3,
    "hidden": true
  "autoscale.min_workers": {
    "type": "fixed",
    "value": 1,
    "hidden": true
  "aws_attributes.instance_profile_arn": {
    "type": "fixed",
    "value": "arn:aws:iam::9999999999:instance-profile/SOME_POLICY",
    "hidden": false
  "enable_local_disk_encryption": {
    "type": "fixed",
    "value": false
  "instance_pool_id": {
    "type": "fixed",
    "value": "SOME_POOL",
    "hidden": true
  "driver_instance_pool_id": {
    "type": "fixed",
    "value": "SOME_POOL",
    "hidden": true
  "autotermination_minutes": {
    "type": "fixed",
    "value": 10,
    "hidden": true

Configure Instance Profiles in AWS Databricks

D s platform
  EC2 instances can be configured with permissions to access AWS resources like S3 by attaching an IAM instance profile. Similarly, instance profiles can be attached to EC2 instances for use with AWS Databricks clusters.


NOTE: You must register the instance profiles in the Databricks workspace, or your Databricks clusters reject the instance profile ARNs and display an error. For more information, see

To configure the instance profile for AWS Databricks, you must provide an IAM instance profile ARN in databricks.awsAttributes. instanceProfileArn parameter.


NOTE: For AWS Databricks, you can configure the instance profile value in  databricks.awsAttributes.instanceProfileArn , only when the  aws.credentialProvider is set to instance or temporary.

aws.credentialProviderAWS Databricks permissions

D s platform
 or Databricks jobs gets all permissions directly from the instance profile.


D s platform
  or Databricks jobs use temporary credentials that are issued based on system or user IAM roles.


NOTE: The instance profile must have policies that allow  

D s platform
 or Databricks to assume those roles.



NOTE: If the aws.credentialProvider is set to temporary or instance while using AWS Databricks:

  • databricks.awsAttributes.instanceProfileArn must be set to a valid value f or Databricks jobs to run successfully.
  • aws.ec2InstanceRoleForAssumeRole flag is ignored for Databricks jobs.

For more information, see  Configure for AWS Authentication .

Configure instance pooling


Instance pooling for worker nodes


  • All cluster nodes used by the 
    D s platform
     are taken from the pool. If the pool has an insufficient number of nodes, cluster creation fails.
  • Each user must have access to the pool and must have at least the ATTACH_TO permission.
  • Each user must have a personal access token from the same AWS Databricks workspace. See Configure personal access token below.


Following is the list of parameters that can be reviewed or modified based on your requirements:

Optional Parameters



Number of initial cluster nodes to be placed on on-demand instances. The remainder is placed on availability instances

Default: 1


Availability type used for all subsequent nodes past the firstOnDemandInstances.



Identifier for the availability zone/datacenter in which the cluster resides. The provided availability zone must be in the same region as the Databricks deployment.


The max price for AWS spot instances, as a percentage of the corresponding instance type's on-demand price. When spot instances are requested for this cluster, only spot instances whose max price percentage matches this field will be considered.

Default: 100


The type of EBS volumes that will be launched with this cluster.

Default: None


EC2 instance profile ARN for the cluster nodes. This is only used when AWS credential provider is set to temporary/instance. The instance profile must have previously been added to the Databricks environment by an account administrator.

For more information, see Configure for AWS Authentication.

Determines the cluster mode for running a Databricks job.

Default: JOB
feature.parameterization.matchLimitOnSampling.databricksSparkMaximum number of parameterized source files that are permitted for matching in a single dataset with parameters.Default: 0
databricks.workerNodeTypeType of node to use for the AWS Databricks Workers/Executors. There are 1 or more Worker nodes per cluster.

Default: i3.xlarge


databricks.sparkVersionAWS Databricks runtime version which also references the appropriate version of Spark.

Depending on your version of AWS Databricks, please set this property according to the following:

  • AWS Databricks 8.x: 8.3.x-scala2.12

  • AWS Databricks 7.x: 7.3.x-scala2.12

  • AWS Databricks 5.5 LTR: 5.5.x-scala2.11

Please do not use other values.

databricks.minWorkersInitial number of Worker nodes in the cluster, and also the minimum number of Worker nodes that the cluster can scale down to during auto-scale-down.

Minimum value: 1

Increasing this value can increase compute costs.

databricks.maxWorkersMaximum number of Worker nodes the cluster can create during auto scaling.

Minimum value: Not less than databricks.minWorkers.

Increasing this value can increase compute costs.


If you have enabled instance pooling in AWS Databricks, you can specify the pool identifier here.


NOTE: If both poolId and poolName are specified, poolId is used first. If that fails to find a matching identifier, then the poolName value is checked.

databricks.poolNameIf you have enabled instance pooling in AWS Databricks, you can specify the pool name here.

See previous.


Tip: If you specify a poolName value only, then you can use the instance pools with the same poolName available across multiple Databricks workspaces when you create a new cluster.


Type of node to use for the AWS Databricks Driver. There is only one Driver node per cluster.

Default: i3.xlarge

For more information, see the sizing guide for Databricks.


NOTE: This property is unused when instance pooling is enabled. For more information, see Configure instance pooling below.

databricks. driverPoolIdIf you have enabled instance pooling in AWS Databricks, you can specify the driver node pool identifier here. For more information, see Configure instance pooling below.

NOTE: If both driverPoolId and driverPoolName are specified, driverPoolId is used first. If that fails to find a matching identifier, then the driverPoolName value is checked.

databricks.driverPoolNameIf you have enabled instance pooling in AWS Databricks, you can specify the driver node pool name here. For more information, see Configure instance pooling below.

See previous.


Tip: If you specify a driverPoolName value only, then you can use the instance pools with the same driverPoolName available across multiple Databricks workspaces when you create a new cluster.

databricks.logsDestinationDBFS location that cluster logs will be sent to every 5 minutesLeave this value as /trifacta/logs.
databricks.enableAutoterminationSet to true to enable auto-termination of a user cluster after N minutes of idle time, where N is the value of the autoterminationMinutes property.Unless otherwise required, leave this value as true.
databricks.clusterStatePollerDelayInSecondsNumber of seconds to wait between polls for AWS Databricks cluster status when a cluster is starting up
databricks.clusterStartupWaitTimeInMinutesMaximum time in minutes to wait for a Cluster to get to Running state before aborting and failing an AWS Databricks job. Default: 60

Maximum time in minutes to wait for a Cluster to complete syncing its logs to DBFS before giving up on pulling the cluster logs to the

D s node

Set this to 0 to disable cluster log pulls.
databricks.clusterLogSyncPollerDelayInSecondsNumber of seconds to wait between polls for a Databricks cluster to sync its logs to DBFS after job completion. Default: 20
databricks.autoterminationMinutesIdle time in minutes before a user cluster will auto-terminate.Do not set this value to less than the cluster startup wait time value.
databricks.maxAPICallRetriesMaximum number of retries to perform in case of 429 error code responseDefault: 5. For more information, see Configure Maximum Retries for REST API section below.

Enables encryption of data like shuffle data that is temporarily stored on cluster's local disk.

databricks.patCacheTTLInMinutesLifespan in minutes for the Databricks personal access token in-memory cacheDefault: 10

When true, the platform bypasses shipping its installed Spark libraries to the cluster with each job's execution.


NOTE: This setting is ignored. The vendor Spark libraries are always used for AWS Databricks.

Configure Databricks Job Management


Optionally, you can provide to the 

D s platform
 the name of a shared Databricks cluster to be used to access Databricks Tables. 



NOTE: Any shared cluster must be maintained by the customer.


Configure AWS Databricks workspace overrides

A single AWS Databricks account can have access to multiple Databricks workspaces. You can create more than one workspace by using Account API if you are account is on the E2 version of the platform or on a selected custom plan that allows multiple workspaces per account.

For more information, see

Each workspace has a unique deployment name associated with it that defines the workspace URL. For example:  https://<deployment-name>. .  



  • The existing property databricks.serviceUrl is used to configure the URL to the Databricks Service to run Spark jobs.
  • The databricks.serviceUrl defines the default Databricks workspace for all user in the
    D s product
  • Individual user can override this setting in the User Preferences in the Databricks Personal Access Token page.

For more information, see Databricks Settings Page.

For more information, see Configure Platform section above.


I ndividual users can specify the name of the cluster to which they are permissioned to access Databricks Tables. This cluster can also be shared among users. For more information, see Databricks Settings Page

Configure maximum retries for REST API 

There is a limit of 30 requests per second per workspace on the Databricks REST APIs. If this limit is reached, then a HTTP status code 429 error is returned, indicating that rate limiting is being applied by the server.   By default, the

D s platform
 re-attempts to submit a request  5 times and then fails the job if the request is not accepted. 

If you want to change the number of retries, change the value for the  databricks.maxAPICallRetries  flag.   


(default) When a request is submitted through the AWS Databricks REST APIs, up to 5 retries can be performed in the case of failures.

  • The waiting period increases exponentially for every retry. For example, for the 1st retry, the wait time is 10 seconds, 20 seconds for the next retry, 40 seconds for the third retry and so on.
  • You can set the values accordingly based on number of minutes /seconds you want to try.

When an API call fails, the request fails. As the number of concurrent jobs increases, more jobs may fail.


NOTE: This setting is not recommended.

5+Increasing this setting above the default value may result in more requests eventually getting processed. However, increasing the value may consume additional system resources in a high concurrency environment and jobs might take longer to run due to exponentially increasing waiting time.