1. Stream Analytics Documentation
  2. Configure
  3. Configure Runtime Environment
  4. Optional Configurations
  5. Changing Spark Log Rollover based on Time

3.1.2.7 Changing Spark Log Rollover based on Time

To change Spark log rollover based on time:

  1. Stop Spark service:
    1. sudo systemctl stop spark-slave.service
    2. sudo systemctl stop spark-master.service
  2. Edit spark-env.sh
    1. Navigate to SPARK_HOME/conf, and comment or delete SPARK_WORKER_OPTS variable and its value.
  3. Edit spark-defaults.conf by adding the below lines:
    • spark.executor.logs.rolling.maxRetainedFiles 7
    • spark.executor.logs.rolling.strategy time
    • spark.executor.logs.rolling.time.interval minutely

    Note:

    You can change spark.executor.logs.rolling.time to daily, hourly, minutely. This is to enable log rollover based on time.
  4. Start Spark service:
    1. sudo systemctl start spark-master.service
    2. sudo systemctl start spark-slave.service

    You will see the application and driver data ( files and logs) under /u02/spark when you publish the pipeline again.