The following sections in this appendix provide information on deploying Coherence to various platforms.
When deploying Coherence on AIX please be aware of the following:
There is an issue with IBM's 1.4.2, and 1.5 JVMs for AIX which may prevent them from allocating socket buffers larger then 64K (Oracle 2MB). This issue has been addressed in IBM's 1.4.2 SR7 SDK and 1.5 SR3 SDK. See "Operating System Tuning".
AIX 5.2 and above default to running multicast over IPv6 rather then IPv4. If you run in a mixed IPv6/IPv4 environment you will need to configure your JVMs to explicitly use IPv4. This can be done by setting the java.net.preferIPv4Stack system property to true on the Java command line. See the IBM 32-bit SDK for AIX User Guide for details.
On AIX it is suggested that each Coherence cluster use a unique multicast address and port, as some versions of AIX will not take both into account when delivering packets. See "multicast-listener" for details on configuring the address.
When deploying Coherence on JRockit JVMs please be aware of the following:
When running JRockit on Linux, BEA recommends using 2.6 kernels, and ensuring that the NPTL is enabled. Please see BEA's documentation regarding this issue.
When available Coherence will make use of the highly concurrent AtomicLong class, which allows concurrent atomic updates to long values without requiring synchronization. BEA 1.4 JVMs do not fully support AtomicLong and thus if Coherence detects that it is being run on a BEA 1.4 JVM it will default to a safe but slower synchronized implementation, and will output the following log message.
sun.misc.AtomicLong is not supported on this JVM; using a synchronized counter.
Upgrading to JRockit 1.5 will allow the use of the highly concurrent implementation.
When deploying Coherence with Cisco switches please be aware of the following:
Under heavy UDP packet load some Cisco switches may run out of buffer space and exhibit frequent multi-second communication pauses. These communication pauses can be identified by a series of Coherence log messages referencing communication delays with multiple nodes which cannot be attributed to local or remote GCs.
Experienced a 4172 ms communication delay (probable remote GC) with Member(Id=7, Timestamp=2008-09-15 12:15:47.511, Address=xxx.xxx.x.xx:8089, MachineId=13838); 320 packets rescheduled, PauseRate=0.31, Threshold=512
The Cisco 6500 series support configuration the amount of buffer space available to each Ethernet port or ASIC. In high load applications it may be necessary to increase the default buffer space. This can be accomplished by executing:
fabric buffer-reserve high
See Cisco's documentation for additional details on this setting.
Cisco's default switch configuration does not support proper routing of multicast packets between switches due to the use of IGMP snooping. See the Cisco's documentation regarding the issue and solutions.
Some Cisco switches have shown difficulty in maintaining multicast group membership resulting in existing multicast group members being silently removed from the multicast group. This will cause a partial communication disconnect for the associated Coherence node(s) and they will be forced to leave and rejoin the cluster. This type of outage can most often be identified by the following Coherence log messages indicating that a partial communication problem has been detected.
A potential network configuration problem has been detected. A packet has failed to be delivered (or acknowledged) after 60 seconds, although other packets were acknowledged by the same cluster member (Member(Id=3, Timestamp=Sat Sept 13 12:02:54 EST 2008, Address=xxx.xxx.x.xxx, Port=8088, MachineId=48991)) to this member (Member(Id=1, Timestamp=Sat Sept 13 11:51:11 EST 2008, Address=xxx.xxx.x.xxx, Port=8088, MachineId=49002)) as recently as 5 seconds ago.
To confirm the issue you may run the using the same multicast address and port as the running cluster. If the issue affects a multicast test node its logs will show that at some point it will suddenly stop receiving multicast test messages. See Chapter 16, "Performing a Multicast Connectivity Test".
The following test logs show the issue:
Example M-1 Log for a Multicast Outage
Test Node 192.168.1.100: Sun Sept 14 16:44:22 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? Sun Sept 14 16:44:23 GMT 2008: Received test packet 76 from ip=/192.168.1.101, group=/224.3.2.0:32367, ttl=4. Sun Sept 14 16:44:23 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? Sun Sept 14 16:44:23 GMT 2008: Sent packet 85. Sun Sept 14 16:44:23 GMT 2008: Received test packet 85 from self. Sun Sept 14 16:44:24 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? Sun Sept 14 16:44:25 GMT 2008: Received test packet 77 from ip=/192.168.1.101, group=/224.3.2.0:32367, ttl=4. Sun Sept 14 16:44:25 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? Sun Sept 14 16:44:25 GMT 2008: Sent packet 86. Sun Sept 14 16:44:25 GMT 2008: Received test packet 86 from self. Sun Sept 14 16:44:26 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? Sun Sept 14 16:44:27 GMT 2008: Received test packet 78 from ip=/192.168.1.101, group=/224.3.2.0:32367, ttl=4. Sun Sept 14 16:44:27 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? Sun Sept 14 16:44:27 GMT 2008: Sent packet 87. Sun Sept 14 16:44:27 GMT 2008: Received test packet 87 from self. Sun Sept 14 16:44:28 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? Sun Sept 14 16:44:29 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? Sun Sept 14 16:44:29 GMT 2008: Sent packet 88. Sun Sept 14 16:44:29 GMT 2008: Received test packet 88 from self. Sun Sept 14 16:44:30 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? Sun Sept 14 16:44:31 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? Sun Sept 14 16:44:31 GMT 2008: Sent packet 89. Sun Sept 14 16:44:31 GMT 2008: Received test packet 89 from self. Sun Sept 14 16:44:32 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? Sun Sept 14 16:44:33 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? Test Node 192.168.1.101: Sun Sept 14 16:44:22 GMT 2008: Sent packet 76.Sun Sept 14 16:44:22 GMT 2008: Received test packet 76 from self. Sun Sept 14 16:44:22 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:22 GMT 2008: Received test packet 85 from ip=/192.168.1.100, group=/224.3.2.0:32367, ttl=4. Sun Sept 14 16:44:23 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:24 GMT 2008: Sent packet 77.Sun Sept 14 16:44:24 GMT 2008: Received test packet 77 from self. Sun Sept 14 16:44:24 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:24 GMT 2008: Received test packet 86 from ip=/192.168.1.100, group=/224.3.2.0:32367, ttl=4. Sun Sept 14 16:44:25 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:26 GMT 2008: Sent packet 78.Sun Sept 14 16:44:26 GMT 2008: Received test packet 78 from self. Sun Sept 14 16:44:26 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:26 GMT 2008: Received test packet 87 from ip=/192.168.1.100, group=/224.3.2.0:32367, ttl=4. Sun Sept 14 16:44:27 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:28 GMT 2008: Sent packet 79.Sun Sept 14 16:44:28 GMT 2008: Received test packet 79 from self. Sun Sept 14 16:44:28 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:28 GMT 2008: Received test packet 88 from ip=/192.168.1.100, group=/224.3.2.0:32367, ttl=4. Sun Sept 14 16:44:29 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:30 GMT 2008: Sent packet 80.Sun Sept 14 16:44:30 GMT 2008: Received test packet 80 from self. Sun Sept 14 16:44:30 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:30 GMT 2008: Received test packet 89 from ip=/192.168.1.100, group=/224.3.2.0:32367, ttl=4. Sun Sept 14 16:44:31 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:32 GMT 2008: Sent packet 81.Sun Sept 14 16:44:32 GMT 2008: Received test packet 81 from self. Sun Sept 14 16:44:32 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:32 GMT 2008: Received test packet 90 from ip=/192.168.1.100, group=/224.3.2.0:32367, ttl=4. Sun Sept 14 16:44:33 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:34 GMT 2008: Sent packet 82.
Note that at 16:44:27 the first test node stops receiving multicast packets from other machines. The operating system continues to properly forward multicast traffic from other processes on the same machine, but the test packets (79 and higher) from the second test node are not received. Also note that both the test packets and the cluster's multicast traffic generated by the first node do continue to be delivered to the second node. This indicates that the first node was silently removed from the multicast group.
If you encounter this multicast issue it is suggested that you contact Cisco technical support, or you may consider changing your configuration to unicast-only by using the Coherence well-known-addresses feature. See "well-known-addresses".
When deploying Coherence with Foundry switches please be aware of the following:
Foundry switches have shown to exhibit difficulty in handing multicast traffic. When deploying on with Foundry switches it is recommend that you use the to ensure that all machines which will be part of the Coherence cluster can communicate over multicast. See Chapter 16, "Performing a Multicast Connectivity Test".
If you encounter issues with multicast you may consider changing your configuration to unicast-only by using the well-known-addresses feature. See "well-known-addresses".
When deploying Coherence on IBM BladeCenters please be aware of the following:
A typical deployment on a BladeCenter may include blades with two NICs where one is used for administration purposes and the other for cluster traffic. By default the MAC addresses assigned to the blades of a BladeCenter are uniform enough that the first NIC will generally have an even MAC address and the second will have an odd MAC address. If the BladeCenter's uplink to a central switch also has an even number of channels then layer 2 (MAC based) load balancing may prevent one set of NICs from making full use of the available uplink bandwidth as they will all be bound to either even or odd channels. This issue arises due to the assumption in the switch that MAC addresses are essentially random, which in BladeCenter's is untrue. Remedies to this situation include:
Use layer 3 (IP based) load balancing, assuming that the IP addresses do not follow the same even/odd pattern.
This setting would need to be applied across all switches carrying cluster traffic.
Randomize the MAC address assignments by swapping them between the first and second NIC on alternating machines.
Linux enables you to change a NIC's MAC address using the ifconfig command.
For other operating systems custom tools may be available to perform the same task.
When deploying Coherence on IBM JVMs please be aware of the following:
There is an issue with IBM's 1.4.2, and 1.5 JVMs which may prevent them from allocating socket buffers larger then 64K (Note that buffers of 2MB are recommended for Coherence). This issue has been addressed in IBM's 1.4.2 SR7 SDK and 1.5 SR3 SDK. For performance reasons it is suggested that the patch be applied.
When deploying Coherence on Linux please be aware of the following:
Early versions of the NPTL are prone to deadlock, especially when combined with 2.4 Linux Kernels. The kernel version and NPTL version can be obtained by executing the following commands:
uname -a getconf GNU_LIBPTHREAD_VERSION
If running on a 2.4 kernel, it is recommended that you avoid using any version of the NPTL, and revert to using LinuxThreads. This can be done by setting the LD_ASSUME_KERNEL environment variable before launching Java.
export LD_ASSUME_KERNEL=2.4.19 getconf GNU_LIBPTHREAD_VERSION
If running on a 2.6 kernel, it is recommended that you use a 1.0 or higher version of NPTL. If upgrading the NPTL version is not possible then it is then recommended that you switch to LinuxThreads.
NPTL related issues are known to occur with Red Hat Linux 9 and Red Hat Enterprise Linux 3, and are also likely to effect any 2.4 based Linux distribution with a backported version of the NPTL. See http://java.sun.com/developer/technicalArticles/JavaTechandLinux/RedHat for more details on this issue.
Linux has several high resolution timesources to choose from, the fastest TSC (Time Stamp Counter) unfortunately is not always reliable. Linux chooses TSC by default, and during boot checks for inconsistencies, if found it switches to a slower safe timesource. The slower time sources can be 10 to 30 times more expensive to query then the TSC timesource, and may have a measurable impact on Coherence performance. Note that Coherence and the underlying JVM are not aware of the timesource which the operating system is using. It is suggested that you check your system logs (/var/log/dmesg) to verify that the following is not present.
kernel: Losing too many ticks! kernel: TSC cannot be used as a timesource. kernel: Possible reasons for this are: kernel: You're running with Speedstep, kernel: You don't have DMA enabled for your hard disk (see hdparm), kernel: Incorrect TSC synchronization on an SMP system (see dmesg). kernel: Falling back to a sane timesource now.
As the log messages suggest, this can be caused by a variable rate CPU (SpeedStep), having DMA disabled, or incorrect TSC synchronization on multi CPU machines. If present it is suggested that you work with your system administrator to identify the cause and allow the TSC timesource to be used.
When deploying Coherence on OS X please be aware of the following:
OS X defaults to running multicast over IPv6 rather then IPv4. If you run in a mixed IPv6/IPv4 environment you will need to configure your JVMs to explicitly use IPv4. This can be done by setting the java.net.preferIPv4Stack system property to true on the Java command line.
On OS X it is suggested that each Coherence cluster use a unique multicast address and port, as some versions of OS X will not take both into account when delivering packets. See the multicast-listener for details on configuring the address.
Generally Coherence prefers 2MB or higher buffers, but in the case of OS X this may result in unexpectedly high kernel CPU time, which in turn reduces throughput. For OS X the suggested buffers size is 768KB, though your own tuning may find a better sweet spot. See "packet-buffer" for details on specifying the amount of buffer space Coherence will request.
When deploying Coherence on Solaris please be aware of the following:
Note:
If running on Solaris 10, please review Sun issues 102712 and 102741 which relate to packet corruption and multicast disconnections. These will most often manifest as either EOFExceptions, "Large gap" warnings while reading packet data, or frequent packet timeouts. It is highly recommend that the patches for both issues be applied when using Coherence on Solaris 10 systems.Possible Data Integrity Issues on Solaris 10 Systems Using the e1000g Driver for the Intel Gigabit Network Interface Card (NIC)
IGMP(1) Packets do not Contain IP Router Alert Option When Sent From Solaris 10 Systems With Patch 118822-21 (SPARC) or 118844-21 (x86/x64) or Later Installed
If running on Solaris 10, please review Sun issues 102712 and 102741 which relate to packet corruption and multicast disconnections. These will most often manifest as either EOFExceptions, "Large gap" warnings while reading packet data, or frequent packet timeouts. It is highly recommend that the patches for both issues be applied when using Coherence on Solaris 10 systems.
When deploying Coherence on Sun JVMs please be aware of the following:
With 1.4 JVMs Coherence recommends keeping heap sizes below 1GB in size per JVM. Multiple cache servers can be used allow a single machine to achieve higher capacities. With Sun's 1.5 JVMs, heap sizes beyond 1GB are reasonable, though GC tuning is still advisable to minimize long GC pauses. See Sun's GC Tuning Guide for tuning details. It is also advisable to run with fixed sized heaps as this generally lowers GC times.
When available Coherence will make use of the highly concurrent AtomicLong class, which allows concurrent atomic updates to long values without requiring synchronization. Sun 1.4 client JVMs include an implementation which is not stable on some multiprocessor systems. If Coherence detects that it is being run on a Sun 1.4 client JVM it will default to a safe but slower synchronized implementation, and will output the following log message.
sun.misc.AtomicLong is not supported on this JVM; using a synchronized counter.
It is suggested that you run your 1.4 JVMs in server mode to ensure that the stable and highly concurrent version can be used. To run the JVM in server mode include the -server option on the Java command line.
Coherence is supported within virtual machine environments, and there should see no functional differences between running it there or in a non-virtualized operating system.
Using virtualization adds another layer to your network topology, and like all other layers it must be properly configured to support multicast networking. See "multicast-listener".
It is less likely that a process running in a virtualized OS will be able to fully use gigabit Ethernet. This is not specific to Coherence, and will be visible most network intensive virtualized applications.
See the following VMWare article covering their network performance as compared to non-virtualized operating systems.
From a Coherence fault tolerance perspective there is more configuration which needs to occur to ensure that cache entry backups reside on physically separate hardware. Manual machine identity must be configured so that Coherence can ensure that backups are not inadvertently stored on the same physical machine as the primary. This can be configured by using the machine-id element within the operational configuration file. See the configuration for "unicast-listener" for details.
When deploying Coherence on Windows please be aware of the following:
Out of the box Windows is not optimized for background processes and heavy network loads. This may be addressed by running the optimize.reg script included in the Coherence installation's bin directory. See "Operating System Tuning" for details on the optimizations which will be performed.
If running a firewall on a machine you may have difficulties in forming a cluster consisting of multiple computers. This can be resolved by either:
Disable the firewall, though this is generally not recommended.
Grant full network access to the Java executable which will run Coherence.
Open up individual address and ports for Coherence.
By default Coherence will use TCP and UDP ports starting at 8088, subsequent nodes on the same machine will use increasing port numbers. Coherence may also communicate over multicast, the default address and port will differ with based on the release. See "unicast-listener" and "multicast-listener" for details on address and port configuration.
When deploying Coherence on z/OS please be aware of the following:
When deploying Coherence into environments where the default character set is EBCDIC rather than ASCII, please make sure that Coherence the configuration files which are loaded from JAR files or off of the classpath are in ASCII format. Configuration files loaded directly from the file system should be stored in the systems native format of EBCDIC.
Under some circumstances, Coherence cluster nodes that run within the same logical partition (LPAR) on z/OS on IBM zSeries cannot communicate with each other. (This problem does not occur on the zSeries when running on Linux.)
The root cause is that z/OS may bind the MulticastSocket that Coherence uses to an automatically-assigned port, but Coherence requires the use of a specific port in order for cluster discovery to operate correctly. (Coherence does explicitly initialize the java.net.MulitcastSocket to use the necessary port, but that information appears to be ignored on z/OS when there already is an instance of Coherence running within that same LPAR.)
The solution is to run only one instance of Coherence within a z/OS LPAR; if multiple instances are required, each instance of Coherence should be run in a separate z/OS LPAR. Alternatively well known addresses may be used. See "well-known-addresses" for more information.