Tune an NVIDIA GPU Cluster for Apache Spark
You can tune your NVIDIA GPU clusters to optimize their performance by using recommendations from the GPU provider and by installing optional libraries.
Tuning GPU clusters can help optimize the performance of those clusters when called on by jobs in your Oracle AI Data Platform Workbench.
For NVIDIA GPU-based clusters, you can follow NVIDIA's Tuning Guide for recommendations and steps you can take to optimize performance.
You also have the option of installing NVIDIA cuDF and cuML plug-ins for Apache Spark libraries to assist with optimization:
- cuDF plugin for Apache Spark library is an accelerator for Apache Spark and provides a set of plugins that leverage GPUs to accelerate processing.
- cuML plug-in for Apache Spark library enables GPU-accelerated, distributed machine learning on Apache Spark and provides several PySpark ML compatible algorithms powered by the cuML library.
The cuDF plugin library is commonly used first for feature engineering and data cleaning, and then cross validation is performed at scale using the cuML plugin library. You can use these libraries for use cases like fraud detection (time series), web clickstream, and A/B experimentation.
Table 9-2 Recommended Spark Configurations
| Setting | Value | Note |
|---|---|---|
| spark.task.resource.gpu.amount | 0.0625 | 1 / spark.executor.cores |
| spark.rapids.sql.concurrentGpuTasks | 3 | GPU memory / 8GB, maximum of 4 |
| spark.rapids.shuffle.multiThreaded.writer.threads | 32 | CPU cores / GPU count per worker |
| spark.rapids.shuffle.multiThreaded.reader.threads | 32 | CPU cores / GPU count per worker |
| spark.shuffle.manager | com.nvidia.spark.rapids.spark350.RapidsShuffleManager | - |
| spark.rapids.shuffle.mode | MULTITHREADED | - |
| spark.plugins | com.nvidia.spark.SQLPlugin | - |
| spark.executor.resource.gpu.amount | 1 | - |
| spark.sql.files.maxPartitionBytes | 2 GB | Optional, recommended for large datasets |
| spark.rapids.sql.batchSizeBytes | 2 GB | Optional, recommended for large datasets |
| spark.rapids.memory.host.spillStorageSize | 32 G | Optional, recommended for large datasets |
| spark.rapids.memory.pinnedPool.size | 8 G | Optional, recommended for large datasets |
| spark.sql.adaptive.coalescePartitions.minPartitionSize | 32 MB | Optional, recommended for large datasets |
| spark.sql.adaptive.advisoryPartitionSizeInBytes | 160 MB | Optional, recommended for large datasets |
| spark.rapids.filecache.enabled | True | Optional, recommended if workloads will be reusing datasets |