Page tree

Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.


Supported versions of Databricks

  • Azure AWS Databricks 7.3 (Recommended)

  • Azure AWS Databricks 5.5 LTS

Job Limits


Instance pooling reduces cluster node spin-up time by maintaining a set of idle and ready instances. The 

D s platform
 can be configured to leverage instance pooling on the Azure AWS Databricks cluster for both worker and driver nodes.


  • All cluster nodes used by the 
    D s platform
     are taken from the pool. If the pool has an insufficient number of nodes, cluster creation fails.
  • Each user must have access to the pool and must have at least the ATTACH_TO permission.
  • Each user must have a personal access token from the same Azure AWS Databricks workspace. See Configure personal access token below.


  1. Acquire your pool identifier or pool name from Azure AWS Databricks.


    NOTE: You can use either the Databricks pool identifier or pool name.If both poolId and poolName are specified, poolId is used first. If that fails to find a matching identifier, then the poolName value is checked.


    Tip: If you specify a poolName value only, then you can run your Databricks jobs against the available clusters across multiple

    D s item
    . This mechanism allows for better resource allocation and broader execution options.

  2. D s config
  3. Set either of the following parameters: 

    1. Set the following parameter to the Azure AWS Databricks pool identifier:

      Code Block
      "databricks.poolId": "<my_pool_id>",

    2. Or, you can set the following parameter to the Azure AWS Databricks pool name:

      Code Block
      "databricks.poolName": "<my_pool_name>",

  4. Save your changes and restart the platform.