Table 4 Selected Spark configuration parameters

Spark.executor.memory

Amount of memory to use per executor process, in GB

1

12

Spark.executor.cores

The number of cores to use on each executor

#

2–12

Spark.driver.memory

Amount of memory to use for the driver process, in GB

1

4

Spark.driver.cores

The Number of cores to use for the driver process.

1

3

Spark.shuffle.file.buffer

Size of the in-memory buffer for each shuffle file output stream, in KB

32

48

Spark.reducer.maxSizeInFlight

Maximum size of map outputs to fetch simultaneously from each reduce task, in MB

48

96

Spark.default.parallelism

The default number of partitions in RDDs returned by transformations like join, reduceByKey, and parallelize when not set by the user

#

8–100

Spark.python.worker.memory

Amount of memory to use per python worker process during aggregation, in MB

512

512–1024

Spark.python.worker.reuse

Reuse Python worker or not

True

True

Spark.rdd.compress

Whether to compress serialized RDD partitions

False

True/False

Spark.serializer

Class to use for serializing objects that will be sent over the network or need to be cached in serialized form

Java

Java

Spark.memory.fraction

Fraction of heap space used for execution and storage

0.6

0.1–0.4

Spark.memory.storageFraction

Amount of storage memory immune to eviction expressed as a fraction of the size of the region

0.5

0.1–0.4

Spark.task.maxFailures

Number of failures of any particular task before giving up on the job

4

5

Spark.speculation

If set to "true", performs speculative execution of tasks

False

True/False

Spark.rpc.message.maxSize

Maximum message size to allow in "control plane" communication, in MB

128

256

Spark.io.compression.codec

Compress map output files

snappy

lz4/lzf/snappy

Spark.io.compression.snappy.blockSize

Block size in Snappy compression, in KB

32

32–128