3.1.2.6 Changing Spark Work Directory
To change Spark work directory:
- Stop Spark service:
sudo systemctl stop spark-slave.service
sudo systemctl stop spark-master.service
- Create work directory under u02:
- Navigate to
/u02
sudo mkdir spark
. Herespark
is the work folder name, you can create folder with the name of your choice.chmod 777 spark
, to change permission.
- Navigate to
- Edit
spark-env.sh
- Navigate to
SPARK_HOME/conf
, and editspark-env.sh
- Add
SPARK_WORKER_DIR=/u02/spark
at the end of the filespark-env.sh
to point to newly created folder under/u02
- Navigate to
- Start Spark service:
sudo systemctl start spark-master.service
sudo systemctl start spark-slave.service
You will see the application and driver data ( files and logs) under
/u02/spark
when you publish the pipeline again.