NOTE: Spark 3.0.1 is supported only on specific deployments and versions of the following environments:
Azure Databricks 8.x
Azure Databricks 8.3 (Recommended)
AWS Databricks 7.x
AWS Databricks 7.3 (Recommended)
Spark 2.4.6 Recommended
- Spark 2.3.x
NOTE: Spark History Server is not supported. It should be used only for short-term debugging tasks, as it requires considerable resources.
Before you begin, please verify the following:
For additional pre-requisites prerequisites for a kerberized environment, see Configure for Kerberos Integration.
- Additional configuration is required for secure impersonation. See Configure for Secure Impersonation.
The per-container memory allocation in Spark (
spark.driver.memory) must not exceed the YARN thresholds. See Spark tuning properties above.
Problem - Job fails with ask timed out error