...
| Warning |
|---|
Before you modify these parameters, you should review with your cluster administrator what are appropriate settings for each parameter. In some cases, you can set these values to cause failures on the cluster. No validation is performed for inputted values. |
Spark parameter | Description | |
|---|---|---|
| Spark Driver Memory | Amount of physical RAM in GB on each Spark node that is made available for the Spark drivers. By raising this number:
| |
| Spark Executor Memory | Amount of physical RAM in GB on each Spark node that is made available for the Spark executors. By raising this number:
| |
| Spark Executor Cores | Number of cores on each Spark executor that is made available to Spark. By raising this number:
| |
| Transformer Dataframe Checkpoint Threshold | When checkpointing is enabled, the Spark DAG is checkpointed when the approximate number of expressions in this parameter has been added to the DAG. Checkpointing assists in managing the volume of work that is processed through Spark at one time; by checkpointing after a set of steps,
By raising this number:
|
For more details on setting these parameters, see Tune Cluster Performance.
...
