Heap Memory
Recommended heap to run cycle is usually between 4gb to 8gb size. A heap size beyond 8gb can have higher memory fragmentation and in order to deal with this memory fragmentation for large heap, garbage collectors have to work extra for compactness of the heap space and efficient management of the old and young generation spaces of heap. Lower memory footprints are usually easier to manage from a GC standpoint.
As far as Coherence is designed it efficiently manages the cache among the cluster nodes. It distributes the cache entries and copies such that no single heap is overloaded. No need to have large heap sizes per JVM unless application demands and running out of heap space. It is advisable to add another Cycle Agent (JVM) with 8gb heap rather than making heap from 8gb to 16gb.
Make sure JVMs initial and maximum head size is always set to equal. This will allow heap to be set to max at the time of startup for the app server and will not grow heap when needed. Ensure at-least 30% of heap is always free.