This appendix includes the following sections:
Coherence recommends keeping heap sizes at 1-8GB per JVM. However, larger heap sizes, up to 20GB, are suitable for some applications where the simplified management of fewer, larger JVMs outweighs the performance benefits of many smaller JVMs. Using multiple cache servers allows a single computer to achieve higher capacities. With Oracle's JVMs, heap sizes beyond 8GB are reasonable, though GC tuning is still advisable to minimize long GC pauses. See Introduction in Java Platform, Standard Edition HotSpot Virtual Machine Garbage Collection Tuning Guide. It is also advisable to run with fixed sized heaps as this generally lowers GC times. See JVM Tuning.
When available Coherence uses the highly concurrent AtomicLong class, which allows concurrent atomic updates to long values without requiring synchronization.
It is suggested to run in server mode to ensure that the stable and highly concurrent version can be used. To run the JVM in server mode include the -server option on the Java command line.
JVMs that experience an OutOfMemoryError can be left in an indeterministic state which can have adverse effects on a cluster. We recommend configuring JVMs to exit upon encountering an OutOfMemoryError instead of allowing the JVM to attempt recovery. Here is the parameter to configure this setting on Sun JVMs:
UNIX:
-XX:OnOutOfMemoryError="kill -9 %p"
Windows:
-XX:OnOutOfMemoryError="taskkill /F /PID %p"
Additionally, it is recommended to configure the JVM to generate a heap dump if an OutOfMemoryError is thrown to assist the investigation into the root cause for the error. Use the following flag to enable this feature on the Sun JVM:
-XX:+HeapDumpOnOutOfMemoryError
JVMs that experience an OutOfMemoryError can be left in an indeterministic state which can have adverse effects on a cluster. We recommend configuring JVMs to exit upon encountering an OutOfMemoryError instead of allowing the JVM to attempt recovery. Here is the parameter to configure this setting on IBM JVMs:
UNIX:
-Xdump:tool:events=throw,filter=java/lang/OutOfMemoryError,exec="kill -9 %pid"
Windows:
-Xdump:tool:events=throw,filter=java/lang/OutOfMemoryError,exec="taskkill /F /PID %pid"
IBM does not recommend fixed size heaps for JVMs. In many cases, it is recommended to use the default for -Xms (in other words, omit this setting and only set -Xmx). See this link for more details:
http://www.ibm.com/developerworks/java/jdk/diagnosis/
It is recommended to configure the JVM to generate a heap dump if an OutOfMemoryError is thrown to assist the investigation into the root cause for the error. IBM JVMs generate a heap dump on OutOfMemoryError by default; no further configuration is required.
Linux has several high resolution timesources to choose from, the fastest TSC (Time Stamp Counter) unfortunately is not always reliable. Linux chooses TSC by default, and during startup checks for inconsistencies, if found it switches to a slower safe timesource. The slower time sources can be 10 to 30 times more expensive to query then the TSC timesource, and may have a measurable impact on Coherence performance. For more details on TSC, see
https://lwn.net/Articles/209101/
Coherence and the underlying JVM are not aware of the timesource which the operating system is using. It is suggested that you check your system logs (/var/log/dmesg) to verify that the following is not present.
kernel: Losing too many ticks! kernel: TSC cannot be used as a timesource. kernel: Possible reasons for this are: kernel: You're running with Speedstep, kernel: You don't have DMA enabled for your hard disk (see hdparm), kernel: Incorrect TSC synchronization on an SMP system (see dmesg). kernel: Falling back to a sane timesource now.
As the log messages suggest, this can be caused by a variable rate CPU (SpeedStep), having DMA disabled, or incorrect TSC synchronization on multi CPU computers. If present, it is suggested that you work with your system administrator to identify the cause and allow the TSC timesource to be used.
When running on Solaris 10, there are known issues relate to packet corruption and multicast disconnections. These most often manifest as either EOFExceptions, "Large gap" warnings while reading packet data, or frequent packet timeouts. It is highly recommend that the patches for both issues below be applied when using Coherence on Solaris 10 systems.
Possible Data Integrity Issues on Solaris 10 Systems Using the e1000g Driver for the Intel Gigabit Network Interface Card (NIC)
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=ALERT&id=1000972.1
IGMP(1) Packets do not Contain IP Router Alert Option When Sent From Solaris 10 Systems With Patch 118822-21 (SPARC) or 118844-21 (x86/x64) or Later Installed
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=ALERT&id=1000940.1
If running on Solaris 10, review the above Solaris 10 (x86 and SPARC) issues which relate to packet corruption and multicast disconnections. These most often manifest as either EOFExceptions, "Large gap" warnings while reading packet data, or frequent packet timeouts. It is highly recommend that the patches for both issues be applied when using Coherence on Solaris 10 systems.
Solaris M series systems include an on-board NIC (bge) and PCI connected NIC (nxge). The on-board gigabit ethernet ports are used for low-bandwidth administrative networking connectivity for the domain and are not intended for high-performance cluster interconnect workloads with high traffic demands. Coherence cluster members must always use the dedicated PCIe NICs for high bandwidth cluster interconnects.
Solaris 11 supports two types of NIC link aggregation: trunk aggregation and Datalink multipathing (DLMP) aggregations. Trunk aggregation requires the use of a network switch, which must support the Link Aggregation Control Protocol (LCAP). DLMP aggregation requires the use of at least one network switch. However, when using DLMP aggregations, make sure any switches are not configured to use trunk aggregation with LCAP. If you change from a trunk aggregation to a DLMP aggregation, you must remove the switch configuration that was previously created for the trunk aggregation. Failure to do so, can result in packet loss and under utilization of the network bandwidth.
The default Windows configuration is not optimized for background processes, heavy network loads, and network interruptions. This may be addressed by running the optimize.reg script included in the Coherence installation's bin directory. See Operating System Tuning.
If running a firewall on a computer you may have difficulties in forming a cluster consisting of multiple computers. This can be resolved by either:
Disabling the firewall, though this is generally not recommended.
Granting full network access to the Java executable which runs Coherence.
Opening up individual address and ports for Coherence. See Configuring Firewalls for Cluster Members in Developing Applications with Oracle Coherence.
On Microsoft Windows, if the Network Interface Card (NIC) is unplugged from the network, the operating system invalidates the associated IP address. The effect of this is that any socket which is bound to that IP address enters an error state. This results in the Coherence nodes exiting the cluster and residing in an error state until the NIC is reattached to the network. In cases where it is desirable to allow multiple collocated JVMs to remain clustered during a physical outage Windows must be configured to not invalidate the IP address.
To adjust this parameter:
While the name of the keyword includes DHCP, the setting effects both static and dynamic IP addresses. See Microsoft Windows TCP/IP Implementation Details for additional information:
http://technet.microsoft.com/en-us/library/bb726981.aspx#EDAA
OS X defaults to running multicast over IPv6 rather than IPv4. If you run in a mixed IPv6/IPv4 environment, configure your JVMs to explicitly use IPv4. This can be done by setting the java.net.preferIPv4Stack system property to true on the Java command line.
Generally, Coherence prefers 2MB or higher buffers, but for OS X this may result in unexpectedly high kernel CPU time, which in turn reduces throughput. For OS X, the suggested buffers size is 768KB, though your own tuning may find a better size. See Configuring the Size of the Packet Buffers in Developing Applications with Oracle Coherence.
When deploying Coherence into environments where the default character set is EBCDIC rather than ASCII, ensure that Coherence configuration files which are loaded from JAR files or off of the classpath are in ASCII format. Configuration files loaded directly from the file system should be stored in the systems native format of EBCDIC.
Under some circumstances, Coherence cluster nodes that run within the same logical partition (LPAR) on z/OS on IBM zSeries cannot communicate with each other. (This problem does not occur on the zSeries when running on Linux.)
The root cause is that z/OS may bind the MulticastSocket that Coherence uses to an automatically-assigned port, but Coherence requires the use of a specific port in order for cluster discovery to operate correctly. (Coherence does explicitly initialize the java.net.MulitcastSocket to use the necessary port, but that information appears to be ignored on z/OS when there is an instance of Coherence running within that same LPAR.)
The solution is to run only one instance of Coherence within a z/OS LPAR; if multiple instances are required, each instance of Coherence should be run in a separate z/OS LPAR. Alternatively, well known addresses may be used. See Using Well Known Addresses in Developing Applications with Oracle Coherence.
AIX 5.2 and above default to running multicast over IPv6 rather than IPv4. If you run in a mixed IPv6/IPv4 environment, configure your JVMs to explicitly use IPv4. This can be done by setting the java.net.preferIPv4Stack system property to true on the Java command line. See the IBM 32-bit SDK for AIX User Guide for details.
Oracle Coherence follows the support policies of Oracle Fusion Middleware. See Supported Virtualization and Partitioning Technologies for Oracle Fusion Middleware.
Virtualization adds another layer to your network topology and it must be properly configured to support multicast networking. See Configuring Multicast Communication in Developing Applications with Oracle Coherence.
It is less likely that a process running in a virtualized operating system can fully use gigabit Ethernet. This is not specific to Coherence and is visible on most network intensive virtualized applications.
Additional configuration is required to ensure that cache entry backups reside on physically separate hardware. See Specifying a Cluster Member's Identity in Developing Applications with Oracle Coherence.
Some Cisco switches may run out of buffer space and exhibit frequent multi-second communication pauses under heavy packet load some. These communication pauses can be identified by a series of Coherence log messages referencing communication delays with multiple nodes which cannot be attributed to local or remote GCs.
Experienced a 4172 ms communication delay (probable remote GC) with Member(Id=7, Timestamp=2008-09-15 12:15:47.511, Address=xxx.xxx.x.xx:8089, MachineId=13838); 320 packets rescheduled, PauseRate=0.31, Threshold=512
The Cisco 6500 series support configuring the amount of buffer space available to each Ethernet port or ASIC. In high load applications it may be necessary to increase the default buffer space. This can be accomplished by executing:
fabric buffer-reserve high
See Cisco's documentation for additional details on this setting.
Cisco's default switch configuration does not support proper routing of multicast packets between switches due to the use of IGMP snooping. See the Cisco's documentation regarding the issue and solutions.
Some Cisco switches have shown difficulty in maintaining multicast group membership resulting in existing multicast group members being silently removed from the multicast group. This cause a partial communication disconnect for the associated Coherence node(s) and they are forced to leave and rejoin the cluster. This type of outage can most often be identified by the following Coherence log messages indicating that a partial communication problem has been detected.
A potential network configuration problem has been detected. A packet has failed to be delivered (or acknowledged) after 60 seconds, although other packets were acknowledged by the same cluster member (Member(Id=3, Timestamp=Sat Sept 13 12:02:54 EST 2008, Address=192.168.1.101, Port=8088, MachineId=48991)) to this member (Member(Id=1, Timestamp=Sat Sept 13 11:51:11 EST 2008, Address=192.168.1.101, Port=8088, MachineId=49002)) as recently as 5 seconds ago.
To confirm the issue, use the same multicast address and port as the running cluster. If the issue affects a multicast test node, its logs show that it suddenly stopped receiving multicast test messages. See Performing a Multicast Connectivity Test.
The following test logs show the issue:
Example A-1 Log for a Multicast Outage
Test Node 192.168.1.100: Sun Sept 14 16:44:22 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? Sun Sept 14 16:44:23 GMT 2008: Received test packet 76 from ip=/192.168.1.101, group=/224.3.2.0:32367, ttl=4. Sun Sept 14 16:44:23 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? Sun Sept 14 16:44:23 GMT 2008: Sent packet 85. Sun Sept 14 16:44:23 GMT 2008: Received test packet 85 from self. Sun Sept 14 16:44:24 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? Sun Sept 14 16:44:25 GMT 2008: Received test packet 77 from ip=/192.168.1.101, group=/224.3.2.0:32367, ttl=4. Sun Sept 14 16:44:25 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? Sun Sept 14 16:44:25 GMT 2008: Sent packet 86. Sun Sept 14 16:44:25 GMT 2008: Received test packet 86 from self. Sun Sept 14 16:44:26 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? Sun Sept 14 16:44:27 GMT 2008: Received test packet 78 from ip=/192.168.1.101, group=/224.3.2.0:32367, ttl=4. Sun Sept 14 16:44:27 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? Sun Sept 14 16:44:27 GMT 2008: Sent packet 87. Sun Sept 14 16:44:27 GMT 2008: Received test packet 87 from self. Sun Sept 14 16:44:28 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? Sun Sept 14 16:44:29 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? Sun Sept 14 16:44:29 GMT 2008: Sent packet 88. Sun Sept 14 16:44:29 GMT 2008: Received test packet 88 from self. Sun Sept 14 16:44:30 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? Sun Sept 14 16:44:31 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? Sun Sept 14 16:44:31 GMT 2008: Sent packet 89. Sun Sept 14 16:44:31 GMT 2008: Received test packet 89 from self. Sun Sept 14 16:44:32 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? Sun Sept 14 16:44:33 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? Test Node 192.168.1.101: Sun Sept 14 16:44:22 GMT 2008: Sent packet 76. Sun Sept 14 16:44:22 GMT 2008: Received test packet 76 from self. Sun Sept 14 16:44:22 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:22 GMT 2008: Received test packet 85 from ip=/192.168.1.100, group=/224.3.2.0:32367, ttl=4. Sun Sept 14 16:44:23 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:24 GMT 2008: Sent packet 77. Sun Sept 14 16:44:24 GMT 2008: Received test packet 77 from self. Sun Sept 14 16:44:24 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:24 GMT 2008: Received test packet 86 from ip=/192.168.1.100, group=/224.3.2.0:32367, ttl=4. Sun Sept 14 16:44:25 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:26 GMT 2008: Sent packet 78.Sun Sept 14 16:44:26 GMT 2008: Received test packet 78 from self. Sun Sept 14 16:44:26 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:26 GMT 2008: Received test packet 87 from ip=/192.168.1.100, group=/224.3.2.0:32367, ttl=4. Sun Sept 14 16:44:27 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:28 GMT 2008: Sent packet 79. Sun Sept 14 16:44:28 GMT 2008: Received test packet 79 from self. Sun Sept 14 16:44:28 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:28 GMT 2008: Received test packet 88 from ip=/192.168.1.100, group=/224.3.2.0:32367, ttl=4. Sun Sept 14 16:44:29 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:30 GMT 2008: Sent packet 80. Sun Sept 14 16:44:30 GMT 2008: Received test packet 80 from self. Sun Sept 14 16:44:30 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:30 GMT 2008: Received test packet 89 from ip=/192.168.1.100, group=/224.3.2.0:32367, ttl=4. Sun Sept 14 16:44:31 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:32 GMT 2008: Sent packet 81.Sun Sept 14 16:44:32 GMT 2008: Received test packet 81 from self. Sun Sept 14 16:44:32 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:32 GMT 2008: Received test packet 90 from ip=/192.168.1.100, group=/224.3.2.0:32367, ttl=4. Sun Sept 14 16:44:33 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:34 GMT 2008: Sent packet 82.
Note that at 16:44:27 the first test node stops receiving multicast packets from other computers. The operating system continues to properly forward multicast traffic from other processes on the same computer, but the test packets (79 and higher) from the second test node are not received. Also note that both the test packets and the cluster's multicast traffic generated by the first node do continue to be delivered to the second node. This indicates that the first node was silently removed from the multicast group.
If you encounter this multicast issue it is suggested that you contact Cisco technical support, or you may consider changing your configuration to unicast-only by using the well known addresses. See Using Well Known Addresses in Developing Applications with Oracle Coherence.
The Cisco 6500 series router may become overloaded if too many packets with a time-to-live (TTL) value of 1 are received. In addition, a low TTL setting may overload single group members. Set the Coherence multicast TTL setting to at least the size of the multicast domain (127 or 255) and make sure that clusters do not use overlapping groups. See Specifying the Multicast Time-to-Live in Developing Applications with Oracle Coherence.
Foundry switches have shown to exhibit difficulty in handing multicast traffic. When deploying on with Foundry switches, ensure that all computers that are part of the Coherence cluster can communicate over multicast. See Performing a Multicast Connectivity Test.
If you encounter issues with multicast you may consider changing your configuration to unicast-only by using the well-known-addresses feature. See Using Well Known Addresses in Developing Applications with Oracle Coherence.
A typical deployment on a BladeCenter may include blades with two NICs where one is used for administration purposes and the other for cluster traffic. By default, the MAC addresses assigned to the blades of a BladeCenter are uniform enough that the first NIC generally has an even MAC address and the second has an odd MAC address. If the BladeCenter's uplink to a central switch also has an even number of channels, then layer 2 (MAC based) load balancing may prevent one set of NICs from making full use of the available uplink bandwidth as they are all bound to either even or odd channels. This issue arises due to the assumption in the switch that MAC addresses are essentially random, which in BladeCenter's is untrue. Remedies to this situation include:
Use layer 3 (IP based) load balancing (if the IP addresses do not follow the same even/odd pattern).
This setting must be applied across all switches carrying cluster traffic.
Randomize the MAC address assignments by swapping them between the first and second NIC on alternating computers.
Linux enables you to change a NIC's MAC address using the ifconfig command.
For other operating systems custom tools may be available to perform the same task.