Configuring spark-defaults.conf

This topic describes how to prevent Spark errors when making OS system calls from BDD Shell.

You can make call() or os.system() calls from BDD Shell. For example, you can call the DP CLI, as described in Creating a new BDD data set from BDD Shell.

However, if the spark-defaults.conf is misconfigured, making such a call may result in an error similar to this example:

org.apache.spark.SparkException: Job aborted due to stage failure: 
Exception while getting task result: com.esotericsoftware.kryo.KryoException: 
java.lang.UnsupportedOperationException

The error occurs because the Spark spark-defaults.conf file contains the following property:

spark.serializer=org.apache.spark.serializer.KryoSerializer

Remove this property and re-try the operation.