6.2.1 Spark Streaming DataServer Properties

Provides the streaming properties that are specific to Spark Technology that are added to the Spark Execution Unit.

Table 6-2 Spark Streaming DataServer Properties

Key Value
spark.checkpointingBaseDir This property defines the base directory and under this base directory every Spark EU will create a sub-directory with the name EU.

Example: hdfs://cluster-ns1/user/oracle/spark/checkpoints
spark.checkpointingInterval Displays the time in seconds
spark.restartFromCheckpoint

  • If set to true, the Spark Streaming application will restart from an existing checkpoint.

  • If set to false, the Spark Streaming application will ignore any existing checkpoints.

  • If there is no checkpoint, it will start normally.
spark.batchDuration Displays the time in seconds and Spark batches up Stream input data.
spark.rememberDuration Displays the time in seconds and the sets the Spark Streaming context to remember RDDs.
spark.checkpointing Enables Spark checkpointing.
spark.streaming.timeout Displays the time in seconds and the Spark waits before stopping a Streaming applications.

Default is 60.
odi-execution-mode

  • SYNCHRONOUS: Spark application is submitted and monitored through OdiOSCommand.

  • ASYNCHRONOUS: Spark application is submitted asynchronously through OdiOSCommand and then monitored through Spark REST APIs.
spark.ui.enabled Enables the Spark Live REST API.

Note:

Set to true for asynchronous execution.
spark.eventLog.enabled Enables Spark event logs. This allows the logs to be accessible by the Spark History Server.

Note:

Set to true for asynchronous execution.
principal Kerberized User name.
keytab Kerberized Password.
odi.spark.enableUnsupportedSparkModes This check is introduced, as only yarn-client and yarn-cluster are supported.