NOTE: If you are using Azure AD to integrate with an Azure Databricks cluster, the Azure AD secret value stored in
Nested folders are not supported when running jobs from Azure Databricks.
NOTE: Avoid including spaces in the paths to your ADLS sources. Spaces in the path value can cause errors during execution on Databricks.
- When a job is started and no cluster is available, a cluster is initiated, which can take up to four minutes. If the job is canceled during cluster startup:
- The job is terminated, and the cluster remains.
- The job is reported in the application as Failed, instead of Canceled.
Azure Databricks integration works with Spark 2.4.0 only.
NOTE: The version of Spark for Azure Databricks must be applied to the platform configuration through the
databricks.sparkVersionproperty. Details are provided later.
- Azure Databricks integration does not work with Hive.
- Use of partitioned tables in Azure Databricks is not supported.