Provides the streaming properties that are specific to Spark Technology that are added to the Spark Execution Unit.
Table 6-2 Spark Streaming DataServer Properties
Key | Value |
---|---|
spark.checkpointingBaseDir | This property defines the base directory and under this base directory every Spark EU will create a sub-directory with the name EU.
Example: hdfs://cluster-ns1/user/oracle/spark/checkpoints |
spark.checkpointingInterval | Displays the time in seconds |
spark.restartFromCheckpoint |
|
spark.batchDuration | Displays the time in seconds and Spark batches up Stream input data. |
spark.rememberDuration | Displays the time in seconds and the sets the Spark Streaming context to remember RDDs. |
spark.checkpointing | Enables Spark checkpointing. |
spark.streaming.timeout | Displays the time in seconds and the Spark waits before stopping a Streaming applications.
Default is 60. |
odi-execution-mode |
|
spark.ui.enabled | Enables the Spark Live REST API.
Note: Set to true for asynchronous execution. |
spark.eventLog.enabled | Enables Spark event logs. This allows the logs to be accessible by the Spark History Server.
Note: Set to true for asynchronous execution. |
principal | Kerberized User name. |
keytab | Kerberized Password. |
odi.spark.enableUnsupportedSparkModes | This check is introduced, as only yarn-client and yarn-cluster are supported. |