Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Published by Scroll Versions from space DEV and version next

...

Spark parameter

Description

Transformer Dataframe Checkpoint Threshold

When checkpointing is enabled, the Spark DAG is checkpointed when the approximate number of expressions in this parameter has been added to the DAG. Checkpointing assists in managing the volume of work that is processed through Spark at one time; by checkpointing after a set of steps, the

D s platform
 can reduce the chances of execution errors for your jobs.

By raising this number:

  • You increase the upper limit of steps between checkpoints.
  • You may reduce processing time.
  • It may result in a higher number of job failures.

Default value:200

Enable whole-stage code generation for SparkWhen enabled, whole-stage code generation optimizes Spark SQL queries for execution performance on the cluster.
Maximum number of fields that whole-stage code generation supports

This defines the number of fields (columns) that are permitted in a whole-stage code generation query. If the number of fields in the query exceeds this value, then the

D s platform
disables whole-stage code generation to prevent performance issues and memory exceptions.

Info

NOTE: Avoid modifying this value unless you have a clear understanding of the implications.

Default value: 100