...
Spark 3.0.1
Info NOTE: Spark 3.0.1 is supported only on specific deployments and versions of the following environments:
Azure Databricks 8.x
Azure Databricks 8.3 (Recommended)
AWS Databricks 7.x
AWS Databricks 7.3 (Recommended)
EMR 6.2
Spark 2.4.6 Recommended
- Spark 2.3.x
...
Prerequisites
| Info |
|---|
NOTE: Spark History Server is not supported. It should be used only for short-term debugging tasks, as it requires considerable resources. |
...
Before you begin, please verify the following:
For additional pre-requisites prerequisites for a kerberized environment, see Configure for Kerberos Integration.
- Additional configuration is required for secure impersonation. See Configure for Secure Impersonation.
...
The per-container memory allocation in Spark (spark.executor.memory and spark.driver.memory) must not exceed the YARN thresholds. See Spark tuning properties above.
Problem - Job fails with ask timed out error
...
