Page tree

Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DEV and version r0871


  • Spark 3.0.1


    NOTE: Spark 3.0.1 is supported only on specific deployments and versions of the following environments:

    Azure Databricks 8.x

    Azure Databricks 8.3 (Recommended)

    AWS Databricks 7.x

    AWS Databricks 7.3 (Recommended)

    EMR 6.2

  • Spark 2.4.6 Recommended

  • Spark 2.3.x




NOTE: Spark History Server is not supported. It should be used only for short-term debugging tasks, as it requires considerable resources.


Before you begin, please verify the following:

  1. For additional pre-requisites prerequisites for a kerberized environment, see Configure for Kerberos Integration.

  2. Additional configuration is required for secure impersonation. See Configure for Secure Impersonation.


The per-container memory allocation in Spark (spark.executor.memory and spark.driver.memory) must not exceed the YARN thresholds. See Spark tuning properties above.

Problem - Job fails with ask timed out error