Table 4 Spark configuration parameters
Configuration parameters category | Spark | Tuned values |
|---|---|---|
Resource utilization | num-executors | 50 |
| Â | executor-cores | 4 |
| Â | executor-memory | 8Â GB |
Input split | spark.hadoop.MapReduce.input .fileinputformat.split.minsize | 128Â MB (default), 256MB, 512MB, 1024MB |
Shuffle | spark.shuffle.file.buffer | 16Â k, 32Â k (default), 48Â k, 64Â k |
spark.reducer.maxSizeInFlight | 32Â M, 48Â M (default), 64Â M, 96Â M | |
spark.hadoop.dfs.replication | 1 | |
spark.default.parallelism | 80, 100, 200, 300 |