Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Instance pooling reduces cluster node spin-up time by maintaining a set of idle and ready instances. The 

D s platform
 can be configured to leverage instance pooling on the Azure AWS Databricks cluster for both worker and driver nodes.

...

  • All cluster nodes used by the 
    D s platform
     are taken from the pool. If the pool has an insufficient number of nodes, cluster creation fails.
  • Each user must have access to the pool and must have at least the ATTACH_TO permission.
  • Each user must have a personal access token from the same Azure AWS Databricks workspace. See Configure personal access token below.

...

  1. Acquire your pool identifier or pool name from Azure AWS Databricks.

    Info

    NOTE: You can use either the Databricks pool identifier or pool name.If both poolId and poolName are specified, poolId is used first. If that fails to find a matching identifier, then the poolName value is checked.


    Tip

    Tip: If you specify a poolName value only, then you can run your Databricks jobs against the available clusters across multiple

    D s item
    itemworkspaces
    . This mechanism allows for better resource allocation and broader execution options.


  2. D s config
  3. Set either of the following parameters: 

    1. Set the following parameter to the Azure AWS Databricks pool identifier:

      Code Block
      "databricks.poolId": "<my_pool_id>",


    2. Or, you can set the following parameter to the Azure AWS Databricks pool name:

      Code Block
      "databricks.poolName": "<my_pool_name>",


  4. Save your changes and restart the platform.

...