2 Using JMX to Manage Oracle Coherence

JMX management is enabled and configured using cluster operational settings. Once configured, you can access Oracle Coherence MBeans using tools such as VisualVM and JConsole. Management information is also accessible using REST.

This chapter includes the following sections:

• Configuring JMX Management
  Coherence includes many configuration options, such as enabling management, stopping cluster members from being managed, filtering MBeans, and configuring management refresh properties.
• Accessing Oracle Coherence MBeans
  Coherence supports many tools for interacting with Coherence MBeans, such as VisualVM, JConsole, the HTML Adapter, and an MBean connector.
• Accessing Management Information Using REST
  You can use HTTP to access Coherence MBeans that are exposed as REST resources. REST enables a broad range of clients to access management information.
• Using the Coherence-JVisualVM Plug-In
  The Coherence-JVisualVM plug-in provides management and monitoring of a single Coherence cluster using the VisualVM management utility.

Configuring JMX Management

Coherence includes many configuration options, such as enabling management, stopping cluster members from being managed, filtering MBeans, and configuring management refresh properties. JMX management is configured within the <management-config> element in a tangosol-coherence-override.xml file or by setting management system properties at startup. See management-config in Developing Applications with Oracle Coherence.

This section includes the following topics:

• Using Dynamic Management Mode
  
• Explicitly Enabling Remote JMX Management on a Cluster Member
  
• Enabling Local JMX Management on a Cluster Member
  
• Enabling JMX Management When Using the Startup Scripts
  
• Stopping a Cluster Member from Being Managed Remotely
  
• Disabling JMX Management
  
• Filtering MBeans
  
• Configuring Management Refresh
  
• Using an Existing MBean Server
  

Using Dynamic Management Mode

Coherence is configured by default to start in dynamic management mode. Dynamic management mode is a high availability feature that automatically selects the senior cluster member as the JMX cluster member. The JMX cluster member hosts an MBean server that is responsible for aggregating management information from all cluster members. If the JMX cluster member is not operational, then the next most senior cluster member is automatically selected as the JMX cluster member.

You can find the JMX service URL by querying the NameService service on any cluster member using the com.tangosol.discovery.NSLookup class. The class has a command line interface or you can call the lookupJMXServiceURL method. For example:

java -cp %COHERENCE_HOME%\lib\coherence.jar com.tangosol.discovery.NSLookup -name management/JMXServiceURL

Cluster JRCluster:      service:jmx:rmi://127.0.0.1:62427/stub/rO0ABXNyAC5qYXZheC5tYW5hZ2VtZW50LnJlbW90ZS5ybWkuUk1...

All cluster members can potentially be selected as the JMX cluster member. To specify that a cluster member should be excluded from being selected as the JMX member, set the <managed-nodes> element to none. For example:

<?xml version='1.0'?>

<coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
   xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config
   coherence-operational-config.xsd">
   <management-config>
      <managed-nodes system-property="coherence.management">none
      </managed-nodes>
   </management-config>
</coherence>

The default value if no value is specified is dynamic and indicates that a cluster member can become the JMX cluster member.

The coherence.management system property also excludes a cluster member from being selected as the JMX cluster member. For example:

-Dcoherence.management=none

Explicitly Enabling Remote JMX Management on a Cluster Member

One or more cluster members can be explicitly configured to host an MBean server that is responsible for the managed objects of all cluster members. Accessing the MBean servers on these cluster members shows management information for all cluster members. The use of dedicated JMX cluster members is a common practice because it avoids loading JMX software into every single cluster member while still providing fault tolerance if a single JMX member fails.

Note:

By default, Coherence is configured to use dynamic management mode and a JMX cluster member is automatically selected. Dynamic management is a recommended best practice. See Using Dynamic Management Mode. These instructions allow you to explicitly configure JMX cluster members. However, cluster members that are not configured as the JMX cluster member must be explicitly disabled from being dynamically selected as the JMX cluster member. See Disabling JMX Management.

In smaller clusters, a common practice is to have dedicated JMX JVMs on two existing cluster members to ensure fault tolerance. In very large clusters, it is often practical to have two computers that are dedicate solely for JMX; however, this is not always necessary.

To enable remote JMX management on a cluster member, set the <managed-nodes> element to all or remote-only. For example:

<?xml version='1.0'?>

<coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
   xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config
   coherence-operational-config.xsd">
   <management-config>
      <managed-nodes system-property="coherence.management">all
      </managed-nodes>
   </management-config>
</coherence>

The remote-only setting starts an MBean server that manages only remote MBeans. The all setting starts an MBean server that manages remote MBeans and local (within the same JVM) MBeans.

The coherence.management system property also enables remote JMX management. For example:

-Dcoherence.management=all

Enabling Local JMX Management on a Cluster Member

Local JMX management constrains an MBean server to manage only the MBeans that are local (within the same JVM) to the cluster member. Accessing the MBean server on the cluster member shows only local management information. However, the member's MBeans can still be managed by a cluster member that has been enabled for remote JMX management. Local JMX management is typically used for extend clients or transient cluster clients.

To enable local JMX management on a cluster member, set the <managed-nodes> element to local-only. For example:

<?xml version='1.0'?>

<coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
   xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config
   coherence-operational-config.xsd">
   <management-config>
      <managed-nodes system-property="coherence.management">local-only
      </managed-nodes>
   </management-config>
</coherence>

The coherence.management system property also enables local JMX management. For example:

-Dcoherence.management=local-only

Enabling JMX Management When Using the Startup Scripts

As a convenience, the COHERENCE_HOME/bin/cache-server and COHERENCE_HOME/bin/coherence startup scripts include a -jmx argument that enables JMX management on a cluster member. The argument is not required when using dynamic management, which is the default management configuration.

cache-server -jmx

The argument automatically sets the management system properties, which can be changed as required within the script. The default settings are the following:

-Dcoherence.management=all
-Dcoherence.management.remote=true

Stopping a Cluster Member from Being Managed Remotely

By default, all cluster members allow their MBeans to be managed by a remote MBean server. To restrict remote management of a member's MBeans, set the <allow-remote-management> element to false. For example:

<?xml version='1.0'?>

<coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
   xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config
   coherence-operational-config.xsd">
   <management-config>
      <allow-remote-management
         system-property="coherence.management.remote">false
      </allow-remote-management>
   </management-config>
</coherence>

The coherence.management.remote system property also disables remote management. For example:

-Dcoherence.management.remote=false

Disabling JMX Management

To disable JMX management on a cluster member, set the <managed-nodes> element to none. For example:

<?xml version='1.0'?>

<coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
   xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config
   coherence-operational-config.xsd">
   <management-config>
      <managed-nodes system-property="coherence.management">none</managed-nodes>
   </management-config>
</coherence>

Disabling JMX management on a member does not stop the member from being remotely managed. You must also set the <allow-remote-management> to false. The following example disables JMX management and stops the member from being remotely managed:

<?xml version='1.0'?>

<coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
   xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config
   coherence-operational-config.xsd">
   <management-config>
      <managed-nodes system-property="coherence.management">none</managed-nodes>
      <allow-remote-management
         system-property="coherence.management.remote">false
      </allow-remote-management>
   </management-config>
</coherence>

Filtering MBeans

The Oracle Coherence management framework provides the ability to filter MBeans before they are registered in the MBean server. An out-of-the-box MBean filter is provided, and custom filters can be created as required. The included MBean filter (com.tagosol.net.management.ObjectNameExcludeFilter) excludes MBeans from being registered based on their JMX object name using standard regex patterns. For example, the pattern .*type=Service,name=Management,.* excludes MBeans with type=Service and name=Management. As configured out of the box, the filter excludes some platform MBeans from being registered in the management framework. MBean filters are defined using the <mbean-filter> element.

The following example shows the out-of-the-box configuration:

...
<mbean-filter>
   <class-name>com.tangosol.net.management.ObjectNameExcludeFilter</class-name>
   <init-params>
      <init-param>
         <param-type>string</param-type>
         <param-value system-property="coherence.management.exclude">
              .*type=Service,name=Management,.*
              .*type=Platform,Domain=java.lang,subType=ClassLoading,.*
              .*type=Platform,Domain=java.lang,subType=Compilation,.*
              .*type=Platform,Domain=java.lang,subType=MemoryManager,.*
              .*type=Platform,Domain=java.lang,subType=Threading,.*
         </param-value>
      </init-param>
   </init-params>
</mbean-filter>
...

To enable the management service or platform MBeans, remove the corresponding object names from the list of names in the <param-value> element. To exclude an MBean from being registered, add the MBean object name to the list.

The coherence.management.exclude system property also filters MBeans. For example:

-Dcoherence.management.exclude=.*type=Service,name=Management,.*

Configuring Management Refresh

The <refresh-expiry>, <refresh-policy>, and <refresh-timeout> elements control the latency of management information. The following example configures each of these settings, which are described in detail after the example:

<?xml version='1.0'?>

<coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
   xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config
   coherence-operational-config.xsd">
   <management-config>
      <refresh-policy
         system-property="coherence.management.refresh.policy">
         refresh-ahead</refresh-policy>
      <refresh-expiry
         system-property="coherence.management.refresh.expiry">1s
      </refresh-expiry>
      <refresh-timeout
         system-property="coherence.management.refresh.timeout">300ms
      </refresh-timeout>
   </management-config>
</coherence>

This section includes the following topics:

• Setting the Management Refresh Expiry
  
• Setting the Management Refresh Policy
  
• Setting the Management Refresh Timeout
  

Setting the Management Refresh Expiry

The <refresh-expiry> element specifies the minimum time interval between the remote retrieval of management information from remote members. The value of this element must be in the following format:

(\d)+((.)(\d)+)?[MS|ms|S|s|M|m|H|h|D|d]?

The first non-digits (from left to right) indicate the unit of time duration:

• MS or ms (milliseconds)

• S or s (seconds)

• M or m (minutes)

• H or h (hours)

• D or d (days)

A unit of milliseconds is assumed if the value does not contain a unit. The default value is 1s.

The coherence.management.refresh.expiry system property also sets the expiry. For example:

-Dcoherence.management.refresh.expiry=2s

Setting the Management Refresh Policy

The <refresh-policy> element specifies how to refresh remote management information. Each policy uses a different refresh algorithm that improves latency for a given MBean usage pattern. Table 2-1 describes each policy.

Table 2-1 Refresh Policies

Setting	Description
refresh-ahead (default)	Refreshes MBeans before they are requested based on prior usage patterns after the expiry delay has passed. This setting can reduce latency of the management information with a minor increase in network consumption. This setting is best when MBeans are accessed in a repetitive or programmatic pattern.
refresh-behind	Refreshes each MBean after the data is accessed. This method ensures optimal response time. However, the information returned is offset by the last refresh time.
refresh-expired	Refreshes each MBean from the remote member when the MBean is accessed and the expiry delay has passed from the last refresh. This setting is best used when MBeans are accessed in a random pattern.

The coherence.management.refresh.policy system property also sets the policy. For example:

-Dcoherence.management.refresh.policy=refresh-expired

Setting the Management Refresh Timeout

The <refresh-timeout> element specifies the duration which the management member waits for a response from a remote member when refreshing MBean information. This value must be less than the <refresh-expiry> value. The value of this element must be in the following format:

(\d)+((.)(\d)+)?[MS|ms|S|s|M|m|H|h|D|d]?

The first non-digits (from left to right) indicate the unit of time duration:

• MS or ms (milliseconds)

• S or s (seconds)

• M or m (minutes)

• H or h (hours)

• D or d (days)

A unit of milliseconds is assumed if the value does not contain a unit. The default value is 250ms.

The coherence.management.refresh.timeout system property also sets the timeout. For example:

-Dcoherence.management.refresh.timeout=300ms

Using an Existing MBean Server

The Oracle Coherence management framework allows the use of an existing MBean server to expose Oracle Coherence MBeans. The MBean server must be located in the same JVM process as the cluster member, and the cluster member must have JMX management enabled.

To use an existing MBean server, enter the MBean server's domain name using the <default-domain-name> element. MBean registration fails if an MBean server with the provided domain name is not found. The following example specifies an existing MBean server with the MyDomainName domain name:

<?xml version='1.0'?>

<coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
   xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config
   coherence-operational-config.xsd">
   <management-config>
      <managed-nodes system-property="coherence.management">all
      </managed-nodes>
      <default-domain-name>MyDomainName</default-domain-name>
   </management-config>
</coherence>

Implement the com.tangosol.net.management.MBeanServerFinder interface to customize how the Oracle Coherence management framework locates an MBean server. Add the fully qualified name of the implementation class within the <server-factory> element using the <class-name> element and include any initialization parameters using the <init-params> element. Use the <default-domain-name> element to specify an existing MBean server domain name to use when instantiating the class.

The following example specifies the MyMBeanServerFinder implementation class and passes the MyDomainName domain name to the implementation class.

<?xml version='1.0'?>

<coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
   xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config
   coherence-operational-config.xsd">
   <management-config>
      <managed-nodes system-property="coherence.management">all
      </managed-nodes>
      <default-domain-name>MyDomainName</default-domain-name>
      <server-factory>
         <class-name
            system-property="coherence.management.serverfactory">
               package.MyMBeanServerFinder</class-name>
         <init-params>
            <init-param>
               <param-type>int</param-type>
               <param-value>0</param-value>
            </init-param>
         </init-params>
     </server-factory>
   </management-config>
</coherence>

The coherence.management.serverfactory system property also specifies the implementation class. For example:

-Dcoherence.management.serverfactory=package.MyMBeanServerFinder

Accessing Oracle Coherence MBeans

Coherence supports many tools for interacting with Coherence MBeans, such as VisualVM, JConsole, the HTML Adapter, and an MBean connector. JMX management must be enabled on a cluster member before you can access Oracle Coherence MBeans. See Configuring JMX Management.

This section includes the following topics:

• Accessing MBeans Locally Using VisualVM
  
• Accessing MBeans Locally Using the JConsole Utility
  
• Allowing Remote Access to Oracle Coherence MBeans
  
• Accessing MBeans of a Running Coherence Cluster Using the JConsole Utility
  
• Accessing MBeans Using the HTML Adapter Application
  
• Accessing Coherence MBeans by Using WLST
  When you run Coherence within WebLogic Server in a managed Coherence Servers environment, WebLogic Server domain runtime MBean server collects JMX information from the management proxy. You can access this information by using WLST.
• Setting Up the Oracle Coherence MBean Connector
  

Accessing MBeans Locally Using VisualVM

VisualVM is a management utility that provides the ability to view and interact with MBeans. MBean functionality is provided as a plug-in to VisualVM. The plug-in must be installed from the VisualVM Plugins Center. From the tool, use the Tools -> Plugins option to install the VisualVM-MBeans plug-in.

To locally access Coherence MBeans using the VisualVM utility:

1. Start a cache server using the COHERENCE_HOME\bin\cache-server script. For example:

   COHERENCE_HOME\bin\cache-server

2. Start VISUALVM_HOME\bin\visualvm. The VisualVM window displays.
3. From the Applications tree, click to expand Local and double-click the cluster member's process. The process information appears in a process tab on the right side of the window.
4. From the selected process tab, click the MBeans tab and expand the Coherence node to access the MBeans.
   
   Description of the illustration jvisualvm.png

Accessing MBeans Locally Using the JConsole Utility

JConsole is a management utility that is included with the JDK (JDK_HOME\bin\jconsole) and provides the ability to view and interact with MBeans.

To locally access MBeans using the JConsole utility:

1. Start a cache server using the COHERENCE_HOME\bin\cache-server script. For example:

   COHERENCE_HOME\bin\cache-server
   

2. Start JDK_HOME\bin\jconsole. The Java Monitoring & Management Console window appears and the JConsole: New Connection dialog box is displayed.
3. From the JConsole: New Connection dialog box, select the Local Process: option and then select the cluster member's process. For example:
   
   Description of the illustration new_connection.gif
4. Click Connect.
5. From the Java Monitoring & Management Console window, select the MBeans tab and expand the Coherence node to access the MBeans.
   
   Description of the illustration example-jmx-jconsole.gif

Allowing Remote Access to Oracle Coherence MBeans

Remote JMX management allows JMX clients that are located on remote computers to access Oracle Coherence MBeans. System properties configure remote access and are often set when starting a JVM that hosts an MBean server: that is, the system properties are set when starting cluster members that host an MBean server for the cluster. See Monitoring and Management Using JMX Technology in Java SE Monitoring and Management Guide. The following instructions provide only the minimum configuration that is required.

To allow remote access to Oracle Coherence MBeans without security enabled, set the following system properties when you start a cluster member:

-Dcom.sun.management.jmxremote.port=port
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

Note:

Disabling SSL for remote JMX connections is not recommended and is insecure. It is recommended that you enable SSL for remote JMX connections. For recommendations on securing remote JMX connections, refer to your JDK documentation.

Set the com.sun.management.jmxremote.port value to any available port on the cluster member. Specify the host and port in the JMX client (for example, JConsole) when you connect to the remote MBean server.

In some cases (such as virtualized environments or when using network address translation), the host name of the computer is incorrectly identified or the default value (127.0.0.1) is returned in the RMI stubs that are sent to the JMX client. As a result, the JMX client fails to connect to the MBean server. To ensure that the correct IP address is included in the RMI stubs, explicitly set the IP address of the host by using the java.rmi.server.hostname system property. For example:

-Djava.rmi.server.hostname=IP_Address

See java.rmi Properties in Java Platform Standard Edition 8 Documentation.

Accessing MBeans of a Running Coherence Cluster Using the JConsole Utility

To remotely access MBeans using the jconsole utility, use the Unix or Windows shell script jmxserviceurl.[sh | cmd] from the Coherence distribution coherence/bin by using the following command:

% $JAVA_HOME/bin/jconsole `jmxserviceurl.sh <clusterName> <clusterPort> <hostName>`

In the above command:

• clusterName is the Coherence cluster name and is required.
• clusterPort is optional when no <hostName> parameter is used. It defaults to 7574.
• hostName is the machine name or IP address of a Coherence cluster member. It defaults to localhost.

For example:

% $JAVA_HOME/bin/jconsole `./jmxserviceurl.sh mycluster 7574 100.111.142.210`

Note:

The example assumes that there is a Coherence cluster member for cluster mycluster at the default clusterport 7574 running on machine IP address 100.111.142.210. This machine does not have to be the Coherence management senior, it just has to be running the NameService for mycluster on the specified clusterport.

When the jxmserviceurl shell script is run by itself, it has the following output that is a JMX RMI URL that jconsole uses to connect to the MBean server.

$ ./jmxserviceurl.sh mycluster 7574 100.111.142.210
service:jmx:rmi://100.111.142.210:42788/stub/rO0ABXNyAC5qYXZheC5tYW5hZ2VtZW50LnJlbW90ZS5ybWkuUk1JU2VydmVySW1wbF9TdHViAAAAAAAAAAICAAB4cgAaamF2YS5ybWkuc2VydmVyLlJlbW90ZVN0dWLp

The JConsole application starts and auto connects to the Coherence MBean server.

1. Click Insecure Connection.
2. From the Java Monitoring & Management Console window, select the MBeans tab and expand the Coherence node to access the Coherence MBeans.

Accessing MBeans Using the HTML Adapter Application

The HTML Adapter Web Application allows remote access to Oracle Coherence MBeans on cache clients and uses the HTML adapter (HtmlAdaptorServer) that is shipped as part of the JMX reference implementation. The adapter requires both the jmxri.jar and jmxtools.jar libraries to be in the classpath. The JMX reference implementation is available at Java Platform Technology Downloads.

To access MBeans using the HTML adapter:

1. Edit the COHERENCE_HOME\bin\coherence script to include the jmxri.jar and jmxtools.jar in the classpath. For example, on Windows:

   -cp "jmxri-1.2.1.jar;jmxtools-1.2.1.jar;%coherence_home%\lib\coherence.jar"
   

2. Start a cache factory instance using the script. For example:

   COHERENCE_HOME\bin\coherence
   

3. After the cache factory instance starts, enter jmx 8082 at the command prompt. This starts an HTTP adapter at port 8082 on the cluster member.
4. Using a Web browser, access the adapter by using the host name and port 8082 for the address as shown in the following screen:
   
   Description of the illustration example-jmx-httpadapter.gif

Accessing Coherence MBeans by Using WLST

When you run Coherence within WebLogic Server in a managed Coherence Servers environment, WebLogic Server domain runtime MBean server collects JMX information from the management proxy. You can access this information by using WLST.

For details, see Accessing Coherence MBeans by Using WLST in Administering Clusters for Oracle WebLogic Server.

Setting Up the Oracle Coherence MBean Connector

Oracle Coherence ships with a program to start a cluster member as a dedicated MBean server host. This program provides remote access to Oracle Coherence MBeans by using the JMX Remote API using RMI or the HTTP server provided by the JMX reference implementation. Configure the RMI and HTTP ports setting, if required, to allow access through a firewall. The server is started using the following command:

java -Dcoherence.management=all 
     -Dcom.sun.management.jmxremote.ssl=false 
     -Dcom.sun.management.jmxremote.authenticate=false 
     -cp coherence.jar;jmxri.jar;jmxtools.jar 
     com.tangosol.net.management.MBeanConnector [-http -rmi]

Note:

Disabling SSL for remote JMX connections is not recommended and is insecure. It is recommended that you enable SSL for remote JMX connections. For recommendations on securing remote JMX connections, refer to your JDK documentation.

To allow access by using JMX RMI, include the -rmi argument. To allow access by using HTTP and a Web browser, include the -http argument. Both arguments may be included; however, at least one must be present for the member to start.

Table 2-2 describes optional properties for JMX RMI configuration.

Table 2-2 Optional Properties That Can Be Used for JMX RMI Configuration

Property	Description
tangosol.coherence.management.remote.host	The host to which the JMX server binds. The default value is localhost. On Linux, the value may have to be changed to the host name or IP address.
tangosol.coherence.management.remote.registryport	The port used for the JMX RMI registry. The default value is 9000.
tangosol.coherence.management.remote.connectionport	The port used for the JMX RMI connection. The default value is 3000.

Table 2-3 describes optional properties available for HTTP configuration.

Table 2-3 Optional Properties That Can Be Used for Http Configuration

Property	Description
tangosol.coherence.management.remote.httpport	The port used for the HTTP connection. The default value is 8888.

To connect to the MBean server using JConsole with the default settings, use the following command:

jconsole service:jmx:rmi://localhost:3000/jndi/rmi://localhost:9000/server

To connect to the MBean server using HTTP with the default settings, use the following URL:

http://localhost:8888

Note:

See the JMX Agent documentation to set up secure access using authentication and Secure Socket Layer (SSL):

Monitoring and Management Using JMX Technology in Java SE Monitoring and Management Guide.

Accessing Management Information Using REST

You can use HTTP to access Coherence MBeans that are exposed as REST resources. REST enables a broad range of clients to access management information.

To secure HTTP Management over the REST server, see Securing Oracle Coherence HTTP Management Over REST Server.

This section includes the following topics:

• Enabling the HTTP Management Server
  
• Connecting to Management Resources
  
• Changing the HTTP Management Server Address
  

Enabling the HTTP Management Server

REST management is enabled through an HTTP management server that runs as a proxy on the Coherence cluster. The HTTP management server requires the COHERENCE_HOME\lib\coherence-management.jar library as well as its third-party dependencies. The server starts when <http-managed-nodes> element in management-config is set to either all or inherit and the required libraries are found on the classpath. If the libraries are not found, then Coherence log messages indicate that the management libraries are missing and that management over HTTP is not available. The Coherence distribution does not include the third-party dependencies.

It is a best practice to manage dependencies using Maven. It is assumed that the latest Oracle Coherence artifacts are installed in your local maven repository using the Oracle Maven synchronization plug-in as described in Populating the Maven Repository Manager.

To generate a classpath containing third party libraries, run the following Maven command with the provided pom.xml below. Add the generated classpath to start cache server.

mvn dependency:build-classpath

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <artifactId>management</artifactId>
  <groupId>management</groupId>
  <version>12.2.1-4-0</version>
  <name>Coherence Management Over REST dependencies</name>
  <packaging>pom</packaging>

  <dependencies>
    <dependency>
      <groupId>com.oracle.coherence</groupId>
      <artifactId>coherence-management</artifactId>
      <version>${project.version}</version>
    </dependency>
  </dependencies>
  
  <profiles>
    <profile>
      <id>jdk11</id>
      <properties>
        <com.sun.xml.bind.version>2.3.0</com.sun.xml.bind.version>
        <javax.activation.version>1.1.1</javax.activation.version>
      </properties>
      <dependencies>
	<dependency>
          <groupId>com.sun.xml.bind</groupId>
          <artifactId>jaxb-core</artifactId>
          <version>${com.sun.xml.bind.version}</version>
        </dependency>
        <dependency>
          <groupId>com.sun.xml.bind</groupId>
          <artifactId>jaxb-impl</artifactId>   
          <version>${com.sun.xml.bind.version}</version>
        </dependency>
        <dependency>
          <groupId>javax.activation</groupId>
          <artifactId>activation</artifactId>
          <version>${javax.activation.version}</version>
        </dependency>
        <dependency>
          <groupId>javax.xml.bind</groupId>
          <artifactId>jaxb-api</artifactId>
          <version>${com.sun.xml.bind.version}</version>
        </dependency>
      </dependencies>
    </profile>
  </profiles>

</project>

All the required libraries are automatically downloaded. To see the complete list of libraries, run the following Maven command:

mvn dependency:list

Connecting to Management Resources

The HTTP management server is not enabled by default. It can be configured to use dynamic management mode to inherit and select port 30000 by default. See Changing the HTTP Management Server Address, and <http-managed-nodes> in management-config. If the senior cluster member is not operational, then the management server automatically restarts on the next most senior cluster member. The Coherence logs indicate the host and port of the server or you can query the NameService service on any cluster member using the com.tangosol.discovery.NSLookup class. The class has a command line interface or you can call the lookupHTTPManagementURL method. For example:

java -cp %COHERENCE_HOME%\lib\coherence.jar com.tangosol.discovery.NSLookup -name management/HTTPManagementURL

Cluster MyCluster:      [http://127.0.0.1:63660/management/coherence/cluster]

The following example uses a cURL request to get cluster information:

curl -i -X GET http://127.0.0.1:63660/management/coherence/cluster

Response:

HTTP / 1.1 200 OK
Content - Type: application / json
content - length: 1330
connection: keep - alive

{
	"links": [{
		"rel": "parent",
		"href": "http://127.0.0.1:63660/management/coherence"
	}, {
		"rel": "self",
		"href": "http://127.0.0.1:63660/management/coherence/cluster"
	}, {
		"rel": "canonical",
		"href": "http://127.0.0.1:63660/management/coherence/cluster"
	}, {
		"rel": "services",
		"href": "http://127.0.0.1:63660/management/coherence/cluster/services"
	}, {
		"rel": "caches",
		"href": "http://127.0.0.1:63660/management/coherence/cluster/caches"
	}, {
		"rel": "members",
		"href": "http://127.0.0.1:63660/management/coherence/cluster/members"
	}, {
		"rel": "management",
		"href": "http://127.0.0.1:63660/management/coherence/cluster/management"
	}, {
		"rel": "journal",
		"href": "http://127.0.0.1:63660/management/coherence/cluster/journal"
	}, {
		"rel": "hotcache",
		"href": "http://127.0.0.1:63660/management/coherence/cluster/hotcache"
	}, {
		"rel": "webApplications",
		"href": "http://127.0.0.1:63660/management/coherence/cluster/webApplications"
	}],
	"refreshTime": "2018-12-12T18:04:34.225-05:00",
	"licenseMode": "Development",
	"clusterSize": 1,
	"localMemberId": 1,
	"version": "19.1.0.0.0",
	"running": true,
	"clusterName": "MyCluster",
	"membersDeparted": [],
	"memberIds": [1],
	"membersDepartureCount": 0,
	"members": ["Member(Id=1, Timestamp=2018-12-12 18:01:38.132, Address=127.0.0.1:63621, MachineId=46461, Location=site:Burlington,rack:100A,process:12124,member:COH-001, Role=Cache Server)"],
	"oldestMemberId": 1,
	"type": "Cluster"
}

For a detailed reference of the REST API, see REST API for Managing Oracle Coherence.

Changing the HTTP Management Server Address

You can explicitly define the address and port of the HTTP management server. For most use cases, changing the address is not required and it is a best practice to configure dynamic management mode to enable HTTP management. The default coherence.management.http.port of 30000 need not be changed for virtual containers. For scenarios where coherence HTTP management server running in a non-virtual container is experiencing conflicts with other coherence cache servers/applications on default port of 30000, setting the coherence.management.http.port to 0 (specifying to use an ephemeral port) avoids potential conflicts. See Connecting to Management Resources.

Note:

If multiple coherence cache servers with HTTP management server configured with <http-managed-nodes> element set to all and are started on same machine, the coherence.management.http.port must be unique for each server. Otherwise, the subsequent ones with same coherence.management.http.port value will get an address which is already in use resulting in an error in starting up the HTTP management server. The coherence cache server will start up, but the HTTP management server will only be available on one coherence cache server on that machine.

Override the default HTTP management configuration using the <http-managed-nodes> element and set the value to all:

<?xml version='1.0'?>

<coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
   xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config
   coherence-operational-config.xsd">
   <management-config>
      <http-managed-nodes system-property="coherence.management.http">all
      </http-managed-nodes>
   </management-config>
</coherence>

The coherence.management.http system property also configures management. For example:

-Dcoherence.management.http=all

Specify the address and port of the HTTP management server using the following system properties:

-Dcoherence.management.http.address
-Dcoherence.management.http.port

Using the Coherence-JVisualVM Plug-In

The Coherence-JVisualVM plug-in provides management and monitoring of a single Coherence cluster using the VisualVM management utility.

The plug-in aggregates Coherence MBean data and shows a concise operational view of a single Coherence cluster. Some management information is presented over time, which allows real-time analysis and troubleshooting.

Note:

• The Coherence-JVisualVM plug-in is a design-time tool that is intended to monitor a single Coherence clusters during development and testing. Enterprise-level management products, such as Oracle Enterprise Manager, should be used for monitoring, managing, and alerting in production environments.
• Oracle strongly recommends you to use the open source Coherence VisualVM Plug-in instead of the plug-in that is shipped with the Coherence installer. For more information about installing the plug-in, see coherence-visualvm.

This section includes the following topics:

• Install the Coherence-JVisualVM Plug-In
  
• Monitor a Coherence Cluster Using the Coherence-JVisualVM Plug-in
  
• Overview of Coherence-JVisualVM Plug-in Tabs
  
• Configure the Coherence-JVisualVM Plug-In to Use REST
  
• Monitoring Managed Coherence Servers with the Coherence-JVisualVM Plug-in
  
• Changing the Coherence-JVisualVM Plug-in Refresh Interval
  

Install the Coherence-JVisualVM Plug-In

The Coherence-JVisualVM plug-in file (coherence-jvisualvm.nbm) is included in the COHERENCE_HOME\plugins\jvisualvm directory. Use the Java VisualVM Plugins tool to install the plug-in. The Coherence-JVisualVM plug-in is supported on JDK 7 update 79 or higher and JDK 8 update 40 and higher.

To install the Coherence-JVisualVM plug-in:

1. Run the VISUALVM_HOME\bin\visualvm file. The VisualVM window displays.
2. From the Tools menu, select Plugins. The Plugins screen displays.
3. Select the Downloaded Tab and click Add Plugins.... Use the Add Plugins dialog box to find the COHERENCE_HOME\plugins\jvisualvm\coherence-jvisualvm.nbm file.
4. Click Open. The Coherence-JVisualVM plug-in is imported into the tool and displays on the list of downloaded plugins.
5. From the list of plugins, select Coherence-JVisualVM and click Install. The Plugin Installer displays. Follow the online instructions to install the plugin.
6. Select the Installed tab to verify that the plug-in is installed and activated.
7. Click Close to exit the Plugins screen.

Monitor a Coherence Cluster Using the Coherence-JVisualVM Plug-in

A Coherence cluster is monitored in the VisualVM tool by connecting to a Coherence cluster member JVM process and using the Oracle Coherence tab. Coherence JMX management must be enabled on a cluster member before you can access management data. See Configuring JMX Management.

To monitor a Coherence cluster using the Coherence-JVisualVM plug-in:

1. Start a Coherence cluster.
2. From the VisualVM Applications tree, click to expand Local and double-click a Coherence cluster member process that hosts an MBean server (JMX management-enabled member). The process information appears in a process tab on the right side of the VisualVM window.
3. From the selected process tab, click Oracle Coherence to monitor the Coherence cluster using the Coherence-JVisualVM plug-in as shown below.

   Note:

   Enlarge the VisualVM window to at least 1280x800 in order to view all data. In addition, you can use the zoom system property to enable zooming of all graphs. Zooming allows closer inspection of a timeframe on a graph after VisualVM has been running for a long period of time. To enable zooming, specify the following system property when starting VisualVM:

   -J-Dcoherence.jvisualvm.zoom.enabled=true

   
   Description of the illustration coherence-jvisualvm.png

Overview of Coherence-JVisualVM Plug-in Tabs

Table 2-4 provides a high-level description of the tabs that are available with the Coherence JVisualVM Plug-in.

Table 2-4 Coherence-JVisualVM Plug-In Tab Descriptions

Tab	Description
Cluster Overview	Displays high-level information about the cluster including the cluster name, version, member count, and high availability status. Summary graphs are aggregated from the other tabs and show the total cluster memory available and used, load averages for cluster machines, and packet publisher and receiver success rates. Use the Cluster Overview tab to get a quick snapshot of how the cluster is operating and to identify potential performance and high availability issues.
Machines	Displays a list of the physical servers that are in the cluster and includes a memory profile for each server. A summary graph shows the average load on these servers. Use the Machine tab to monitor the load on each server and to identify any servers that are in danger of using all of their available memory.
Members	Displays a list of all cluster members, their address and role in the cluster, and includes both publisher and receiver success rates, heap memory usage, and send queue sizes. A summary graph shows the total cluster memory available and used. Right-click a member row and select Show details to view a complete list of statistics or select Report node state to get a full thread dump for the node. Use the Members tab to monitor cluster membership and to identify and diagnose potential memory issues for both a member and the cluster as a whole.
Services	Displays information about cluster services including service partition counts and statusHA values. Select a service to see detailed thread information for each node of the service as well as graphs of that information after the next data refresh. Right-click a service to view service partition statistics. Raw data for service thread performance is included to the left of the graph. Right-click a node row and select Show details to view a complete list of statistics. Use the Services tab to monitor services in the cluster and to identify potential performance issues with service threads.
Caches	Displays information about cluster caches including size and memory usage. To get the correct information to be displayed for memory usage, the backing map scheme for a cache must be configured to use the Binary unit calculator. The default unit calculator is Fixed if no unit calculator is defined. Select a cache to see detailed information on the next data refresh. If a near cache is selected, then additional statistics are available for the front cache. Right-click a node row and select Show details to view a complete list of either cache size and access statistics or cache storage statistics depending on which tab is active, respectively. Use the Caches tab to get a detailed view of cache statistics and to identify potential performance issues with cache queries.
Federation	Displays information about federated services in the context of the current participant cluster of the federated service. Each participant cluster to which the current participant cluster is connected is listed. Select a cluster participant to view performance statistics related to the members of the current cluster participant. Both outbound and inbound statistics are provided depending on whether the current participant is configured to send, receive, or send and receive data. Select a specific cluster member to see its bandwidth utilization and its replication statistics. Use the Federation tab to monitor federated participants and to identify potential performance issues among federated participants. The Federation tab supports performing federation operations. Right-click a cluster participant to view a list of operations that can be selected.
Persistence	Displays information about cache persistence including available and used storage, latency introduced by persistence operations, and the number of cache snapshots. Performance statistics are only applicable for active persistence mode. Use the Persistence tab to get a detailed view of active persistence and to identify potential issues with performance and space consumption. The Persistence tab supports performing persistence operations. Right-click a service to view a list of operations that can be selected.
Elastic Data	Displays information about RAM and flash journal memory usage. Select each progress bar to view details for RAM and flash journal, respectively. Journal statistics include internal usage of elastic data that is required by the federated caching feature. Use the Elastic Data tab to monitor elastic data memory usage and to identify when memory may potentially be exhausted
Proxy Servers	Displays information about proxy servers in the cluster including the number of connections across each proxy server and total messages that were sent and received. A summary graph shows the total number of proxy server connections. Use the Proxy Servers tab to monitor Coherence*Extend clients that are connecting to a proxy and to identify potential performance issues on the proxy. Note: This tab is only available if a proxy server is configured in the cluster. Also, you may need to reconnect to a Coherence JVM process if a proxy server was started after other Coherence JVM processes.
HTTP Servers	Displays information for HTTP acceptors that are configured on a proxy server. Information includes the number of connections across each server, total connections and graphs of response codes, and errors and requests over time for a selected service. Use the HTTP Servers tab to monitor resource utilization and identify potential performance issues. Note: This tab is only available if an HTTP acceptor is configured on a proxy server. Also, you may need to reconnect to a Coherence JVM process if a proxy server was started after other Coherence JVM processes.
Coherence*Web	Displays information about HTTP session storage in the cluster including the number of applications deployed, the number of HTTP sessions being stored, as well as other information regarding session reaping. Use the Coherence*Web tab to monitor HTTP session storage and to identify potential performance issues with session storage. Note: This tab is only available if a Coherence cluster is being used for HTTP session storage.
JCache	Displays configuration information and performance statistic information for JCache caches. Use the JCache tab to troubleshoot configuration and performance issues. Note: This tab is only available if a Coherence cluster is being used to store JCache caches.

Configure the Coherence-JVisualVM Plug-In to Use REST

The Coherence-JVisualVM Plug-In uses JMX by default to gather management information. You can also configure the plug-in to use REST. The management information is exactly the same whether you use JMX or REST. REST is typically used when running Coherence within a virtual container. Enable the HTTP management server prior to configuring the plug-in. See Enabling the HTTP Management Server.

To configure the Coherence-JVisualVM Plug-In to use REST:

1. Start the VisualVM application.
2. From the Applications Tab, select Coherence Clusters.
   The Add Coherence Cluster dialog box displays.
3. In the Name field, enter the cluster name that you want to manage.
4. In the Management REST URL field, enter http://host:port/management/coherence/cluster. Replace host and port with the HTTP management server host and port.
   A new cluster entry is created.
5. Click on the cluster entry to view management information.

Note:

The cluster entry is persisted during VisualVM restarts. If the port of the HTTP management server changes, you must create a new cluster entry and enter the new management REST URL.

Monitoring Managed Coherence Servers with the Coherence-JVisualVM Plug-in

The Coherence-JVisualVM plug-in can be used to monitor Coherence clusters that are managed in a WebLogic Server domain. One managed Coherence server is automatically selected as the management proxy. You can configure which managed Coherence servers can be selected as the management proxy. See Removing a Coherence Management Proxy in Administering Clusters for Oracle WebLogic Server. The WebLogic Server domain runtime MBean server collects JMX information from the management proxy.

Note:

• When connecting to a WebLogic Server multitenant domain, the Coherence-JVisualVM plugin only displays when connected as the WebLogic domain administrator. You cannot connect to a multitenant domain as the partition administrator. As the WebLogic Server domain administration, you can access all domain partitions. Ensure that you choose the correct partition when performing any service, cache, or persistence operations.
• If you want to access Coherence MBeans in a WebLogic server domain from a standalone cluster member (a member that is not part of the WebLogic server domain), then you must set the coherence.management.extendedmbeanname system property to true. The property ensures that managed object names include the cluster member name as part of the global MBean name.

To connect to the domain runtime MBean server using VisualVM:

1. Source your WebLogic Server environment by executing the wlserver\server\bin\setWLSEnv script.
2. Start VisualVM and include the following WebLogic Server libraries in the classpath. In addition, include the following options to ensure that MBean checking is disabled.

   jvisualvm --cp WLS_HOME\server\lib\wljmxclient.jar;
     WLS_HOME\server\lib\weblogic.jar -J-Djmx.remote.protocol.provider.pkgs=
     weblogic.management.remote
     -J-Dcoherence.jvisualvm.disable.mbean.check=true
   

3. From the VisualVM Applications tree, right-click Local and select Add JMX Connection. The Add JMX Connection dialog box displays.
4. Enter the following connection URL and replace hostname and port with your WebLogic Server host name and port:

   service:jmx:iiop://hostname:port/jndi/weblogic.management.mbeanservers.
   domainruntime
   

5. Click Use security credentials and enter the WebLogic Server username and password.
6. Click OK.
7. Double-click the new connection.
8. From the selected process tab, click the Oracle Coherence tab to monitor the Coherence cluster.

Changing the Coherence-JVisualVM Plug-in Refresh Interval

The Coherence-JVisualVM plug-in is set to refresh data from the Coherence MBean server by default every 30 seconds. Data on the details section of most tabs (Services tab, Caches tab, Federation tab, and so on) is immediately cleared when a different cache is selected and is refreshed on the next refresh cycle.

To change the refresh setting, use the following system property when starting VisualVM:

visualvm -J-Dcoherence.jvisualvm.refreshtime=10

If refreshing management data is adversely affecting performance (especially on large clusters), then consider leaving the default value for the plug-in refresh interval and changing the default Coherence management refresh expiry time of 1 second to a higher value. The value can be specified within an operational override file or by using a system property. See Configuring Management Refresh.

In addition, the Coherence-JVisualVM plug-in can be configured to include query times in the console logs. The logs are helpful when troubleshooting performance issues.

To enable query times in the logs, use the following system property when starting VisualVM:

-J-Dcoherence.jvisualvm.log.query.times=true 

To view VisualVM logs in the console, select Help | About and then from the About VisualVM screen, click Logfile.