Performance Tuning
Tuning Runtime Configuration
The table below lists the parameters for fine tuning the runtime configuration of the connected system agent, and suggests specific values for small, medium, and large scale implementations.
Details of how to configure these parameters can be found in Agent Parameters.
| Parameter | Description | Small Scale | Medium Scale | Large Scale |
|---|---|---|---|---|
|
idoConfig.sparkMaxResultSizeInGB |
Limit of total size of serialized results of all partitions for each action (e.g. collect) in bytes. Should be at least 1M, or 0 for unlimited. Jobs will be aborted if the total size is above this limit. Having a high limit may cause out-of-memory errors in driver (depends on spark.driver.memory and memory overhead of objects in JVM). Setting a proper limit can protect the driver from out-of-memory errors. |
2 |
5 |
7 |
|
idoConfig.sparkExecutorMemoryInGB |
Amount of additional memory to be allocated per executor process, in MiB unless otherwise specified. This is memory that accounts for things like VM overheads, interned strings, other native overheads, etc. |
2 |
5 |
7 |
|
idoConfig.numberOfPartition |
Number of partitions. |
3 |
5 |
7 |