2 Documentation Errata

This chapter describes changes, enhancements, and corrections made to the Oracle Coherence documentation library for the 3.5.3 release. The library can be found at the following URL:

https://download.oracle.com/docs/cd/E14526_01/index.htm

2.1 Changes to the Setting Single Server Mode

There has been a change in the text of the Setting Single Server Mode chapter of the Developer's Guide for Oracle Coherence.

Table 2-1 Change to Setting Single Server Mode

Old Text New Text

The unicast IP address is defined by the <address> sub-element of the unicast-listener element. This sub-element specifies the IP address that a Socket will listen or publish on. Setting this sub-element to an IP address that is never used will prevent Coherence from joining the network.

The unicast IP address is defined by the <address> sub-element of the unicast-listener element. This sub-element specifies the IP address that a Socket will listen or publish on and must be routed to loopback as well. Setting this sub-element to an IP address that is never used will prevent Coherence from joining the network.

Note: The localhost setting may not work on systems that define localhost as the loopback address; in that case, specify the machine name or the specific IP address.


2.2 Changes to Specifying a Custom Eviction Policy

There has been an addition to the description of Specifying a Custom Eviction Policy in the Developer's Guide for Oracle Coherence:

If using Coherence 3.5, you may run into an IllegalStateException using this example. As a workaround, modify the eviction policy to implement LocalCache.EvitionPolicy. For example:

public class SimpleEvictionPolicy
extends AbstractEvictionPolicy
implements LocalCache.EvictionPolicy

2.3 Changes to init-params and class-name Elements

There has been a change in the descriptions of the init-params and class-name elements in Appendix A, Operational Configuration Elements, in the Developer's Guide for Oracle Coherence. These elements can now be used as sub-elements of the default-serializer element.

2.4 Changes to Well Known Addresses

There is an error in Appendix A, Coherence Features by Edition in Getting Started with Oracle Coherence. The description states that Multicast-free operation (also known as "Well Known Addresses", or "WKA") is supported by the Standard Edition. It is not.

There is also an error in the description of the well-known-addresses element in the Developer's Guide for Oracle Coherence. The description states: "Use of the Well Known Addresses (WKA) feature is not supported by Caching Edition..." Instead of Caching Edition, this should read Standard Edition.

2.5 Changes to packet-buffer Description

The following section has been added to the description of the packet-buffer element in Appendix A, Operational Configuration Elements, in the Developer's Guide for Oracle Coherence:

Latency Versus Throughput

When setting this for transmit (that is, within packet-publisher), higher values may allow for increased throughput, while lower values may allow for decreased latency. If you make any changes to this value, it is recommended that you evaluate how it effects performance in all of these dimensions.

2.6 Changes to local-scheme Element

Changes have been made to the descriptions of the low-units and high-units attributes of the local-scheme element in Appendix B, Cache Configuration Elements in the Developer's Guide for Oracle Coherence.

Table 2-2 Changes to high-units Description

Old text New Text

high-units—(Optional) Used to limit the size of the cache. Contains the maximum number of units that can be placed in the cache before pruning occurs. An entry is the unit of measurement, unless it is overridden by an alternate unit-calculator. Once this limit is exceeded, the cache will begin the pruning process, evicting entries according to the eviction policy until the low-units size is reached.

high-units—(Optional) Used to limit the size of the cache. Contains the maximum number of units that can be placed in the cache before pruning occurs. An entry is the unit of measurement, unless it is overridden by an alternate unit-calculator. Once this limit is exceeded, the cache will begin the pruning process, evicting entries according to the eviction policy.


Table 2-3 Changes to low-units Description

Old Text New Text

low-units—(Optional) Contains the number of units that the cache will be pruned down to when pruning takes place. An entry is the unit of measurement, unless it is overridden by an alternate unit-calculator. When pruning occurs entries will continue to be evicted according to the eviction policy until this size.

low-units—(Optional) Contains the lowest number of units that a cache will be pruned down to when pruning takes place. A pruning will not necessarily result in a cache containing this number of units, however a pruning will never result in a cache containing less than this number of units. An entry is the unit of measurement, unless it is overridden by an alternate unit-calculator. When pruning occurs entries will continue to be evicted according to the eviction policy until this size.


2.7 Changes to tcp-acceptor and tcp-initiator Elements

In the Developer's Guide for Oracle Coherence, the reusable subelement of tcp-acceptor element has been deprecated. Use the new reuse-address element instead. The reuse-address element specifies whether a TCP/IP socket can be bound to an address if a previous connection is in a timeout state. When a TCP/IP connection is closed the connection may remain in a timeout state for a period after the connection is closed (typically known as the TIME_WAIT state or 2MSL wait state). For applications using a well-known socket address or port, it may not be possible to bind a socket to a required address if there is a connection in the timeout state involving the socket address or port. If no value for reuse-address is specified, the default value is true.

The reuse-address element has also been added as a sub-element of tcp-initiator. The definition of reuse-address element is the same, but in this case the default value is false.

2.8 Changes to JVM Tuning

The following text resolves inconsistencies and consolidates information on JVM tuning that appears in the Best Practices appendix in Getting Started with Oracle Coherence and the Performance Tuning chapter in the Developer's Guide for Oracle Coherence.

2.8.1 Basic Sizing Recommendation

The recommendations in this section are sufficient for general use cases and require minimal setup effort. The primary issue to consider when sizing your JVMs is a balance of available RAM versus garbage collection (GC) pause times.

2.8.1.1 Cache Servers

The standard, safe recommendation for Coherence cache servers is to run a fixed size heap of up to 1GB. Additionally, it is recommended to utilize an incremental garbage collector to minimize GC pause durations. Lastly, run all Coherence JVMs in server mode, by specifying the -server on the JVM command line. This allows for several performance optimizations for long running applications.

For example:

java -server -Xms1g -Xmx1g -Xincgc -Xloggc: -cp coherence.jar com.tangosol.net.DefaultCacheServer

This sizing allows for good performance without the need for more elaborate JVM tuning. For more information on garbage collection, see "GC Monitoring and Tuning".

Note:

It is possible to run cache servers with larger heap sizes; though it becomes more important to monitor and tune the JVMs to minimize the GC pauses. It may also be necessary to alter the storage ratios such that the amount of scratch space is increased to facilitate faster GC compactions. Additionally it is recommended that you make use of an up to date JVM version such as HotSpot 1.6 as it includes significant improvements for managing large heaps. See "Heap Size Considerations".

2.8.1.2 TCMP Clients

Coherence TCMP clients should be configured similarly to cache servers as long GCs could cause them to be misidentified as being dead.

2.8.1.3 Extends Clients

Coherence Extend clients are not technically speaking cluster members and, as such, the effect of long GCs is less detrimental. For extend clients it is recommended that you follow the existing guidelines as set forth by the application in which you are embedding coherence.

2.8.2 Heap Size Considerations

This section will help you decide:

  • How many CPUs you will need for your system

  • How much memory you will need for each system

  • How many JVMs to run per system

  • How much heap to configure with each JVM

Since all applications are different, this section should be read as guidelines. You will need to answer the following questions to choose the configuration that is right for you:

  • How much data will be stored in Coherence caches?

  • What are the application requirements in terms of latency and throughput?

  • How CPU or Network intensive is the application?

Sizing is an imprecise science. There is no substitute for frequent performance and stress testing.

The following topics are included in this section:

2.8.2.1 General Guidelines

Running with a fixed sized heap will save your JVM from having to grow the heap on demand and will result in improved performance. To specify a fixed size heap use the -Xms and -Xmx JVM options, setting them to the same value. For example:

java -server -Xms4G -Xmx4G ...

A JVM process consumes more system memory then the specified heap size. The heap size settings specify the amount of heap which the JVM makes available to the application, but the JVM itself will also consume additional memory. The amount consumed differs depending on the OS, and JVM settings. For instance, a HotSpot JVM running on Linux configured with a 1GB JVM will consume roughly 1.2GB of RAM. It is important to externally measure the JVMs memory utilization to ensure that RAM is not over committed. Tools such as top, vmstat, and Task Manager are useful in identifying how much RAM is actually being utilized.

2.8.2.1.1 Storage Ratios

The basic recommendation for how much data can be stored within a cache server of a given size is to use up to 1/3rd of the heap for primary cache storage. This leaves another 1/3rd for backup storage, and the final 1/3rd for scratch space. Scratch space is then used for things such as holding classes, temporary objects, network transfer buffers, and GC compaction. You may instruct Coherence to limit primary storage on a per-cache basis by configuring the <high-units> element and specifying a BINARY value for the <unit-calculator> element. These settings are automatically applied to backup storage as well.

Ideally both the primary and backup storage will also fit within the JVMs tenured space (for HotSpot based JVMs). See HotSpot's Tuning Garbage Collection guide for details on sizing the collectors generations:

http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html

2.8.2.1.2 Cache Topologies and Heap Size

For large datasets, Partitioned or Near caches are recommended. As the scalability of the Partitioned cache is linear for both reading and writing, varying the number of Coherence JVMs will not significantly affect cache performance. Using a Replicated cache will put significant pressure on GC.

2.8.2.1.3 Deciding How Many JVMs to Run Per System

The number of JVMs (nodes) to run per system depends on the system's number of processors/cores and amount of memory. As a starting point, we recommend starting with a plan to run one JVM for every two cores. This recommendation balances the following factors:

  • Multiple JVMs per server allow Coherence to make more efficient use of network resources. Coherence's packet-publisher and packet-receiver have a fixed number of threads per JVM; as you add cores, you'll want to add JVMs to scale across these cores.

  • Too many JVMs will increase contention and context switching on processors.

  • Too few JVMs may not be able to handle available memory and may not fully utilize the NIC.

  • Especially for larger heap sizes, JVMs must have available processing capacity to avoid long GC pauses.

Depending on your application, you can add JVMs up toward one per core. The recommended number of JVMs and amount of configured heap may also vary based on the number of processors/cores per socket and on the machine architecture.

2.8.2.1.4 Sizing Your Heap

When considering heap size, it is important to find the right balance. The lower bound is determined by per-JVM overhead (and also, manageability of a potentially large number of JVMs). For example, if there is a fixed overhead of 100MB for infrastructure software (for example, JMX agents, connection pools, internal JVM structures), then the use of JVMs with 256MB heap sizes will result in close to 40% overhead for non-cache data.The upper bound on JVM heap size is governed by memory management overhead, specifically the maximum duration of GC pauses and the percentage of CPU allocated to GC (and other memory management tasks).

GC can affect the following:

  • The latency of operations against Coherence. Larger heaps will cause longer and less predictable latency than smaller heaps.

  • The stability of the cluster. With very large heaps, lengthy long garbage collection pauses can trick TCMP into believing a cluster member is dead since the JVM is unresponsive during GC pauses. Although TCMP takes GC pauses into account when deciding on member health, at some point it may decide the member is dead.

We offer the following guidelines:

  • With older JVMs such as Sun 1.4 and 1.5, we recommend not allocating more than 1GB and 2GB heap respectively.

  • For newer JVMs (Sun 1.6 or JRockit, for example), we recommend allocating up to a 4GB heap. We also recommend using Sun's Concurrent Mark and Sweep GC or JRockit's Deterministic GC.

The length of a GC pause scales worse than linearly to the size of the heap. That is, if you double the size of the heap, pause times due to GC will, in general, more than double. GC pauses are also impacted by application usage:

  • Pause times increase as the amount of live data in the heap increases. We recommend not exceeding 70% live data in your heap. This includes primary data, backup data, indexes, and application data.

  • High object allocation rates will increase pause times. Even "simple" Coherence applications can cause high object allocation rates since every network packet generates many objects.

  • CPU-intensive computation will increase contention and may also contribute to higher pause times.

Depending on your latency requirements, you can increase allocated heap space beyond the above recommendations, but be sure to stress test your system.

2.8.2.2 Moving the Cache Out of the Application Heap

Using dedicated Coherence cache server instances for Partitioned cache storage will minimize the heap size of application JVMs as the data is no longer stored locally. As most Partitioned cache access is remote (with only 1/N of data being held locally), using dedicated cache servers does not generally impose much additional overhead. Near cache technology may still be used, and it will generally have a minimal impact on heap size (as it is caching an even smaller subset of the Partitioned cache). Many applications are able to dramatically reduce heap sizes, resulting in better responsiveness.

Local partition storage may be enabled (for cache servers) or disabled (for application server clients) with the tangosol.coherence.distributed.localstorage Java property (for example, -Dtangosol.coherence.distributed.localstorage=false).

It may also be disabled by modifying the <local-storage> setting in the tangosol-coherence.xml (or tangosol-coherence-override.xml) file as follows:

Example 2-1 Disabling Partition Storage

<!--
Example using tangosol-coherence-override.xml
-->
<coherence>
  <cluster-config>
    <services>
      <!--
      id value must match what's in tangosol-coherence.xml for DistributedCache
      service
      -->
      <service id="3">
        <init-params>
          <init-param id="4">
            <param-name>local-storage</param-name> 
            <param-value system-property="tangosol.coherence.distributed.localstorage">false</param-value>   
          </init-param>
        </init-params>
      </service>
    </services>
  </cluster-config>
</coherence>

At least one storage-enabled JVM must be started before any storage-disabled clients access the cache.

2.8.3 GC Monitoring and Tuning

Lengthy GC pause times can negatively impact the Coherence cluster and are, for the most part, indistinguishable from node death. During these pauses, a Java application is unable to send or receive packets and in the case of receiving the operating system buffered packets, the packets may be discarded and need to be retransmitted. For these reasons, it is very important that cluster nodes are sized and/or tuned to ensure that their GC times remain minimal. As a good rule of thumb, a node should spend less than 10% of its time paused in GC, normal GC times should be under 100ms, and maximum GC times should be around 1 second.

GC activity can be monitored in a number of ways; some standard mechanisms include:

  • JVM switch -verbose:gc

  • JVM switch -Xloggc: (similar to verbose GC but includes timestamps)

  • Over JMX using tools such as JConsole

Log messages will be generated when one cluster node detects that another cluster node has been unresponsive for a period, generally indicating that a target cluster node was in a GC cycle.

Example 2-2 Message Indicating Target Cluster Node is in Garbage Collection Mode

Experienced a 4172 ms communication delay (probable remote GC) with Member(Id=7, Timestamp=2006-10-20 12:15:47.511, Address=192.168.0.10:8089, MachineId=13838); 320 packets rescheduled, PauseRate=0.31, Threshold=512

PauseRate indicates the percentage of time for which the node has been considered unresponsive since the statistics were last reset. Nodes reported as unresponsive for more then a few percent of their lifetime may be worth investigating for GC tuning.

2.9 Changes to Features By Edition

A sub-row for C++ has been added to the API Language row of the Coherence Client Editions table in Appendix A, Coherence Features by Edition in Getting Started with Oracle Coherence. The sub-row indicates that C++ API are available for the Data Client and the Real Time Client configured as an Extend/TCP Client. There is no available C++ API for the Real Time Client configured as a Compute Client.

2.10 Changes to Local Cache Example

A correction was made to the example in the Configuring the Local Cache section of Getting Started with Oracle Coherence. In the example,. the configuration of the <flush-delay> element has been removed.

2.11 Changes to Supported Environments for Coherence for C++

There has been a change to the table in the Supported Environments section of the Requirements, Installation, and Deployment chapter in the Client Guide for Oracle Coherence.

Note 6 below the table should now read as follows:

6. Specifically, GCC 3.4.6-8 and above, as well as GCC 4.x versions are supported.

2.12 Changes to the Logger Configuration Example for .NET Clients

The example in the Logger section of the Configuration and Usage for .NET Clients chapter of Client Guide for Oracle Coherence has been corrected. The example incorrectly displayed log4net as the value for the destination element. This value has been corrected to common-logger:

<coherence>
  <logging-config>
    <destination>common-logger</destination> 
    <severity-level>5</severity-level>
    <message-format>(thread={thread}): {text}</message-format>
    <character-limit>8192</character-limit>
  </logging-config>
</coherence>

2.13 Changes to .NET Installation

The following changes have been made to the Requirements, Installation and Deployment for Coherence for .NET chapter in the Client Guide for Oracle Coherence.

  • An additional package requiement has been added for running .NET:

    MSHelp 2.x Runtime, which is included in Visual Studio 200x, as well as the applications at this URL:

    http://www.helpware.net/mshelp2/h20.htm#MS_H2_Runtime

  • Note that after Step 2, "Run the installer by double clicking on the installer file":

    If the installer indicates that it is rolling back the installation, then you must run the installer in elevated execution mode. For example, executing the MSI file from a command line shell that was started as Administrator should enable the installation process to complete.

2.14 Changes to Tomcat Server Type Aliases for Coherence*Web

In previous releases, the User's Guide for Oracle Coherence*Web listed the Server Type Aliases for the Tomcat 5.5.x and Tomcat 6.0.x application servers as Generic. This has changed in the current release. Use Tomcat/5.5.x as the Server Type Alias for Tomcat 5.5.x and Tomcat/6.0.x as the alias for Tomcat 6.0.x.

2.15 Deprecation of the Coherence*Web APIs

As of this release, the Coherence*Web API should be considered to be deprecated, and their use should be discouraged. They will be removed in a future release.

2.16 Disable Page Tag Pooling when Deploying on IBM WebSphere

When deploying on IBM WebSphere thread and page tag pooling must be disabled. For more information on disabling page tag pooling on IBM Websphere, see:

http://www.ibm.com/developerworks/websphere/techjournal/0403_johnson/0403_johnson.html