must be installed on Microsoft Azure.
D s platform
Nested folders are not supported when running jobs from Azure Databricks.
NOTE: Avoid including spaces in the paths to your ADLS sources. Spaces in the path value can cause errors during execution on Databricks.
- When a job is started and no cluster is available, a cluster is initiated, which can take up to four minutes. If the job is canceled during cluster startup:
- The job is terminated, and the cluster remains.
- The job is reported in the application as Failed, instead of Canceled.
Azure Databricks integration works with Spark 2.4.x only.
NOTE: The version of Spark for Azure Databricks must be applied to the platform configuration through the
databricks.sparkVersionproperty. Details are provided later.
- Azure Databricks integration does not work with Hive.
- Use of partitioned tables in Azure Databricks is not supported.