This chapter includes the following sections:
JDK_HOME
\bin\jvisualvm
).<management-config>
element in a tangosol-coherence-override.xml
file or by setting management system properties at startup. See management-config in Developing Applications with Oracle Coherence.This section includes the following topics:
Coherence is configured to start in dynamic management mode. Dynamic management mode automatically selects the senior cluster member as the JMX cluster member. The JMX cluster member hosts an MBean server that is responsible for aggregating management information from all cluster members. If the JMX cluster member is not operational, then the next most senior cluster member is automatically selected as the JMX cluster member.
All cluster members can potentially be selected as the JMX cluster member. To specify that a cluster member should be excluded from being selected as the JMX member, set the <managed-nodes>
element to none
. For example:
<?xml version='1.0'?> <coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config" xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config coherence-operational-config.xsd"> <management-config> <managed-nodes system-property="coherence.management">none </managed-nodes> </management-config> </coherence>
The default value if no value is specified is dynamic
and indicates that a cluster member can become the JMX cluster member.
The coherence.management
system property also excludes a cluster member from being selected as the JMX cluster member. For example:
-Dcoherence.management=none
One or more cluster members can be explicitly configured to host an MBean server that is responsible for the managed objects of all cluster members. Accessing the MBean servers on these cluster members shows management information for all cluster members. The use of dedicated JMX cluster members is a common practice because it avoids loading JMX software into every single cluster member while still providing fault tolerance if a single JMX member fails.
Note:
By default, Coherence is configured to use dynamic management mode and a JMX cluster member is automatically selected. See Using Dynamic Management Mode.
In smaller clusters, a common practice is to have dedicated JMX JVMs on two existing cluster members to ensure fault tolerance. In very large clusters, it is often practical to have two computers that are dedicate solely for JMX; however, this is not always necessary.
To enable remote JMX management on a cluster member, set the <managed-nodes>
element to all
or remote-only
. For example:
<?xml version='1.0'?> <coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config" xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config coherence-operational-config.xsd"> <management-config> <managed-nodes system-property="coherence.management">all </managed-nodes> </management-config> </coherence>
The remote-only
setting starts an MBean server that manages only remote MBeans. The all
setting starts an MBean server that manages remote MBeans and local (within the same JVM) MBeans.
The coherence.management
system property also enables remote JMX management. For example:
-Dcoherence.management=all
Note:
Cluster members that are not configured as the JMX cluster member must explicitly be disabled from being dynamically selected as the JMX cluster member. See Disabling JMX Management.
Local JMX management constrains an MBean server to manage only the MBeans that are local (within the same JVM) to the cluster member. Accessing the MBean server on the cluster member shows only local management information. However, the member's MBeans can still be managed by a cluster member that has been enabled for remote JMX management. Local JMX management is typically used for extend clients or transient cluster clients.
To enable local JMX management on a cluster member, set the <managed-nodes>
element to local-only
. For example:
<?xml version='1.0'?> <coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config" xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config coherence-operational-config.xsd"> <management-config> <managed-nodes system-property="coherence.management">local-only </managed-nodes> </management-config> </coherence>
The coherence.management
system property also enables local JMX management. For example:
-Dcoherence.management=local-only
As a convenience, the COHERENCE_HOME
/bin/cache-server
and COHERENCE_HOME
/bin/
coherence
startup scripts include a -jmx
argument that enables JMX management on a cluster member. For example:
cache-server -jmx
The argument automatically sets the management system properties, which can be changed as required within the script. The default settings are the following:
-Dcoherence.management=all -Dcoherence.management.remote=true
By default, all cluster members allow their MBeans to be managed by a remote MBean server. To restrict remote management of a member's MBeans, set the <allow-remote-management>
element to false
. For example:
<?xml version='1.0'?> <coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config" xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config coherence-operational-config.xsd"> <management-config> <allow-remote-management system-property="coherence.management.remote">false </allow-remote-management> </management-config> </coherence>
The coherence.management.remote
system property also disables remote management. For example:
-Dcoherence.management.remote=false
To disable JMX management on a cluster member, set the <managed-nodes>
element to none
. For example:
<?xml version='1.0'?> <coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config" xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config coherence-operational-config.xsd"> <management-config> <managed-nodes system-property="coherence.management">none</managed-nodes> </management-config> </coherence>
Disabling JMX management on a member does not stop the member from being remotely managed. You must also set the <allow-remote-management>
to false
. The following example disables JMX management and stops the member from being remotely managed:
<?xml version='1.0'?> <coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config" xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config coherence-operational-config.xsd"> <management-config> <managed-nodes system-property="coherence.management">none</managed-nodes> <allow-remote-management system-property="coherence.management.remote">false </allow-remote-management> </management-config> </coherence>
The Oracle Coherence management framework provides the ability to filter MBeans before they are registered in the MBean server. An out-of-the-box MBean filter is provided, and custom filters can be created as required. The included MBean filter (com.tagosol.net.management.ObjectNameExcludeFilter
) excludes MBeans from being registered based on their JMX object name using standard regex patterns. For example, the pattern .*type=Service,name=Management,.*
excludes MBeans with type=Service
and name=Management
. As configured out of the box, the filter excludes some platform MBeans from being registered in the management framework. MBean filters are defined using the <mbean-filter>
element.
The following example shows the out-of-the-box configuration:
... <mbean-filter> <class-name>com.tangosol.net.management.ObjectNameExcludeFilter</class-name> <init-params> <init-param> <param-type>string</param-type> <param-value system-property="coherence.management.exclude"> .*type=Service,name=Management,.* .*type=Platform,Domain=java.lang,subType=ClassLoading,.* .*type=Platform,Domain=java.lang,subType=Compilation,.* .*type=Platform,Domain=java.lang,subType=MemoryManager,.* .*type=Platform,Domain=java.lang,subType=Threading,.* </param-value> </init-param> </init-params> </mbean-filter> ...
To enable the management service or platform MBeans, remove the corresponding object names from the list of names in the <param-value>
element. To exclude an MBean from being registered, add the MBean object name to the list.
The coherence.management.exclude
system property also filters MBeans. For example:
-Dcoherence.management.exclude=.*type=Service,name=Management,.*
The <refresh-expiry>
, <refresh-policy>
, and <refresh-timeout>
elements control the latency of management information. The following example configures each of these settings, which are described in detail after the example:
<?xml version='1.0'?> <coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config" xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config coherence-operational-config.xsd"> <management-config> <refresh-policy system-property="coherence.management.refresh.policy"> refresh-ahead</refresh-policy> <refresh-expiry system-property="coherence.management.refresh.expiry">1s </refresh-expiry> <refresh-timeout system-property="coherence.management.refresh.timeout">300ms </refresh-timeout> </management-config> </coherence>
This section includes the following topics:
The <refresh-expiry>
element specifies the minimum time interval between the remote retrieval of management information from remote members. The value of this element must be in the following format:
(\d)+((.)(\d)+)?[MS|ms|S|s|M|m|H|h|D|d]?
The first non-digits (from left to right) indicate the unit of time duration:
MS or ms (milliseconds)
S or s (seconds)
M or m (minutes)
H or h (hours)
D or d (days)
A unit of milliseconds is assumed if the value does not contain a unit. The default value is 1s
.
The coherence.management.refresh.expiry
system property also sets the expiry. For example:
-Dcoherence.management.refresh.expiry=2s
The <refresh-policy>
element specifies how to refresh remote management information. Each policy uses a different refresh algorithm that improves latency for a given MBean usage pattern. Table 2-1 describes each policy.
Table 2-1 Refresh Policies
Setting | Description |
---|---|
|
Refreshes MBeans before they are requested based on prior usage patterns after the expiry delay has passed. This setting can reduce latency of the management information with a minor increase in network consumption. This setting is best when MBeans are accessed in a repetitive or programmatic pattern. |
|
Refreshes each MBean after the data is accessed. This method ensures optimal response time. However, the information returned is offset by the last refresh time. |
|
Refreshes each MBean from the remote member when the MBean is accessed and the expiry delay has passed from the last refresh. This setting is best used when MBeans are accessed in a random pattern. |
The coherence.management.refresh.policy
system property also sets the policy. For example:
-Dcoherence.management.refresh.policy=refresh-expired
The <refresh-timeout>
element specifies the duration which the management member waits for a response from a remote member when refreshing MBean information. This value must be less than the <refresh-expiry>
value. The value of this element must be in the following format:
(\d)+((.)(\d)+)?[MS|ms|S|s|M|m|H|h|D|d]?
The first non-digits (from left to right) indicate the unit of time duration:
MS or ms (milliseconds)
S or s (seconds)
M or m (minutes)
H or h (hours)
D or d (days)
A unit of milliseconds is assumed if the value does not contain a unit. The default value is 250ms
.
The coherence.management.refresh.timeout
system property also sets the timeout. For example:
-Dcoherence.management.refresh.timeout=300ms
The Oracle Coherence management framework allows the use of an existing MBean server to expose Oracle Coherence MBeans. The MBean server must be located in the same JVM process as the cluster member, and the cluster member must have JMX management enabled.
To use an existing MBean server, enter the MBean server's domain name using the <default-domain-name>
element. MBean registration fails if an MBean server with the provided domain name is not found. The following example specifies an existing MBean server with the MyDomainName
domain name:
<?xml version='1.0'?> <coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config" xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config coherence-operational-config.xsd"> <management-config> <managed-nodes system-property="coherence.management">all </managed-nodes> <default-domain-name>MyDomainName</default-domain-name> </management-config> </coherence>
Implement the com.tangosol.net.management.MBeanServerFinder
interface to customize how the Oracle Coherence management framework locates an MBean server. Add the fully qualified name of the implementation class within the <server-factory>
element using the <class-name>
element and include any initialization parameters using the <init-params>
element. Use the <default-domain-name>
element to specify an existing MBean server domain name to use when instantiating the class.
The following example specifies the MyMBeanServerFinder
implementation class and passes the MyDomainName
domain name to the implementation class.
<?xml version='1.0'?> <coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config" xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config coherence-operational-config.xsd"> <management-config> <managed-nodes system-property="coherence.management">all </managed-nodes> <default-domain-name>MyDomainName</default-domain-name> <server-factory> <class-name system-property="coherence.management.serverfactory"> package.MyMBeanServerFinder</class-name> <init-params> <init-param> <param-type>int</param-type> <param-value>0</param-value> </init-param> </init-params> </server-factory> </management-config> </coherence>
The coherence.management.serverfactory
system property also specifies the implementation class. For example:
-Dcoherence.management.serverfactory=package.MyMBeanServerFinder
-jmx
argument of the Oracle Coherence startup script to enable JMX management.This section includes the following topics:
Java VisualVM is a management utility that is included with the JDK (JDK_HOME
\bin\jvisualvm
) and provides the ability to view and interact with MBeans. MBean functionality is provided as a plug-in to Java VisualVM. The plug-in must be installed from the Java VisualVM Plugins Center. From the tool, use the Tools -> Plugins option to install the VisualVM-MBeans plug-in.
To locally access Coherence MBeans using the Java VisualVM utility:
JConsole is a management utility that is included with the JDK (JDK_HOME
\bin\jconsole
) and provides the ability to view and interact with MBeans.
To locally access MBeans using the JConsole utility:
Remote JMX management allows JMX clients that are located on remote computers to access Oracle Coherence MBeans. System properties configure remote access and are often set when starting a JVM that hosts an MBean server: that is, the system properties are set when starting cluster members that host an MBean server for the cluster. See Monitoring and Management Using JMX Technology in Java SE Monitoring and Management Guide. The following instructions provide only the minimum configuration that is required.
To allow remote access to Oracle Coherence MBeans without security enabled, set the following system properties when you start a cluster member. For ease of use, set the system properties in the Oracle Coherence startup scripts so that they are automatically set when you use the -jmx
argument.
-Dcom.sun.management.jmxremote.port=port
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
Set the com.sun.management.jmxremote.port
value to any available port on the cluster member. Specify the host and port in the JMX client (for example, JConsole) when you connect to the remote MBean server.
In some cases (such as virtualized environments or when using network address translation), the host name of the computer is incorrectly identified or the default value (127.0.0.1) is returned in the RMI stubs that are sent to the JMX client. As a result, the JMX client fails to connect to the MBean server. To ensure that the correct IP address is included in the RMI stubs, explicitly set the IP address of the host by using the java.rmi.server.hostname
system property. For example:
-Djava.rmi.server.hostname=IP_Address
See java.rmi
Properties in Java Platform Standard Edition 8 Documentation.
The HTML Adapter Web Application allows remote access to Oracle Coherence MBeans on cache clients and uses the HTML adapter (HtmlAdaptorServer
) that is shipped as part of the JMX reference implementation. The adapter requires both the jmxri.jar
and jmxtools.jar
libraries to be in the classpath. The JMX reference implementation is available at Java Platform Technology Downloads.
To access MBeans using the HTML adapter:
Oracle Coherence ships with a program to start a cluster member as a dedicated MBean server host. This program provides remote access to Oracle Coherence MBeans by using the JMX Remote API using RMI or the HTTP server provided by the JMX reference implementation. Configure the RMI and HTTP ports setting, if required, to allow access through a firewall. The server is started using the following command:
java -Dcoherence.management=all -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -cp coherence.jar;jmxri.jar;jmxtools.jar com.tangosol.net.management.MBeanConnector [-http -rmi]
To allow access by using JMX RMI, include the -rmi
argument. To allow access by using HTTP and a Web browser, include the -http
argument. Both arguments may be included; however, at least one must be present for the member to start.
Table 2-2 describes optional properties for JMX RMI configuration.
Table 2-2 Optional Properties That Can Be Used for JMX RMI Configuration
Property | Description |
---|---|
|
The host to which the JMX server binds. The default value is |
|
The port used for the JMX RMI registry. The default value is |
|
The port used for the JMX RMI connection. The default value is |
Table 2-3 describes optional properties available for HTTP configuration.
Table 2-3 Optional Properties That Can Be Used for Http Configuration
Property | Description |
---|---|
|
The port used for the HTTP connection. The default value is |
To connect to the MBean server using JConsole with the default settings, use the following command:
jconsole service:jmx:rmi://localhost:3000/jndi/rmi://localhost:9000/server
To connect to the MBean server using HTTP with the default settings, use the following URL:
http://localhost:8888
Note:
See the JMX Agent documentation to set up secure access using authentication and Secure Socket Layer (SSL):
Monitoring and Management Using JMX Technology in Java SE Monitoring and Management Guide.
JDK_HOME
\bin\jvisualvm
).Note:
The Coherence-JVisualVM plug-in is a design-time tool that is intended to monitor a single Coherence clusters during development and testing. Enterprise-level management products, such as Oracle Enterprise Manager, should be used for monitoring, managing, and alerting in production environments.
This section includes the following topics:
The Coherence-JVisualVM plug-in file (coherence-jvisualvm.nbm
) is included in the COHERENCE_HOME
\plugins\jvisualvm
directory. Use the Java VisualVM Plugins tool to install the plug-in. The Coherence-JVisualVM plug-in is supported on JDK 7 update 79 or higher and JDK 8 update 40 and higher.
To install the Coherence-JVisualVM plug-in:
JDK_HOME
\bin\jvisualvm
file. The Java VisualVM window displays.COHERENCE_HOME
\plugins\jvisualvm\coherence-jvisualvm.nbm
file.A Coherence cluster is monitored in the Java VisualVM tool by connecting to a Coherence cluster member JVM process and using the Oracle Coherence tab. Coherence JMX management must be enabled on a cluster member before you can access management data. See Configuring JMX Management.
To monitor a Coherence cluster using the Coherence-JVisualVM plug-in:
Table 2-4 provides a high-level description of the tabs that are available with the Coherence JVisualVM Plug-in.
Table 2-4 Coherence-JVisualVM Plug-In Tab Descriptions
Tab | Description |
---|---|
Cluster Overview |
Displays high-level information about the cluster including the cluster name, version, member count, and high availability status. Summary graphs are aggregated from the other tabs and show the total cluster memory available and used, load averages for cluster machines, and packet publisher and receiver success rates. Use the Cluster Overview tab to get a quick snapshot of how the cluster is operating and to identify potential performance and high availability issues. |
Machines |
Displays a list of the physical servers that are in the cluster and includes a memory profile for each server. A summary graph shows the average load on these servers. Use the Machine tab to monitor the load on each server and to identify any servers that are in danger of using all of their available memory. |
Members |
Displays a list of all cluster members, their address and role in the cluster, and includes both publisher and receiver success rates, heap memory usage, and send queue sizes. A summary graph shows the total cluster memory available and used. Right-click a member row and select Show details to view a complete list of statistics or select Report node state to get a full thread dump for the node. Use the Members tab to monitor cluster membership and to identify and diagnose potential memory issues for both a member and the cluster as a whole. |
Services |
Displays information about cluster services including service partition counts and statusHA values. Select a service to see detailed thread information for each node of the service as well as graphs of that information after the next data refresh. Right-click a service to view service partition statistics. Raw data for service thread performance is included to the left of the graph. Right-click a node row and select Show details to view a complete list of statistics. Use the Services tab to monitor services in the cluster and to identify potential performance issues with service threads. |
Caches |
Displays information about cluster caches including size and memory usage. To get the correct information to be displayed for memory usage, the backing map scheme for a cache must be configured to use the binary unit-calculator, which is the default unit calculator if no unit-calculator is defined. Select a cache to see detailed information on the next data refresh. If a near cache is selected, then additional statistics are available for the front cache. Right-click a node row and select Show details to view a complete list of either cache size and access statistics or cache storage statistics depending on which tab is active, respectively. Use the Caches tab to get a detailed view of cache statistics and to identify potential performance issues with cache queries. |
Federation |
Displays information about federated services in the context of the current participant cluster of the federated service. Each participant cluster to which the current participant cluster is connected is listed. Select a cluster participant to view performance statistics related to the members of the current cluster participant. Both outbound and inbound statistics are provided depending on whether the current participant is configured to send, receive, or send and receive data. Select a specific cluster member to see its bandwidth utilization and its replication statistics. Use the Federation tab to monitor federated participants and to identify potential performance issues among federated participants. The Federation tab supports performing federation operations. Right-click a cluster participant to view a list of operations that can be selected. |
Persistence |
Displays information about cache persistence including available and used storage, latency introduced by persistence operations, and the number of cache snapshots. Performance statistics are only applicable for active persistence mode. Use the Persistence tab to get a detailed view of active persistence and to identify potential issues with performance and space consumption. The Persistence tab supports performing persistence operations. Right-click a service to view a list of operations that can be selected. |
Elastic Data |
Displays information about RAM and flash journal memory usage. Select each progress bar to view details for RAM and flash journal, respectively. Journal statistics include internal usage of elastic data that is required by the federated caching feature. Use the Elastic Data tab to monitor elastic data memory usage and to identify when memory may potentially be exhausted |
Proxy Servers |
Displays information about proxy servers in the cluster including the number of connections across each proxy server and total messages that were sent and received. A summary graph shows the total number of proxy server connections. Use the Proxy Servers tab to monitor Coherence*Extend clients that are connecting to a proxy and to identify potential performance issues on the proxy. Note: This tab is only available if a proxy server is configured in the cluster. Also, you may need to reconnect to a Coherence JVM process if a proxy server was started after other Coherence JVM processes. |
HTTP Servers |
Displays information for HTTP acceptors that are configured on a proxy server. Information includes the number of connections across each server, total connections and graphs of response codes, and errors and requests over time for a selected service. Use the HTTP Servers tab to monitor resource utilization and identify potential performance issues. Note: This tab is only available if an HTTP acceptor is configured on a proxy server. Also, you may need to reconnect to a Coherence JVM process if a proxy server was started after other Coherence JVM processes. |
Coherence*Web |
Displays information about HTTP session storage in the cluster including the number of applications deployed, the number of HTTP sessions being stored, as well as other information regarding session reaping. Use the Coherence*Web tab to monitor HTTP session storage and to identify potential performance issues with session storage. Note: This tab is only available if a Coherence cluster is being used for HTTP session storage. |
JCache |
Displays configuration information and performance statistic information for JCache caches. Use the JCache tab to troubleshoot configuration and performance issues. Note: This tab is only available if a Coherence cluster is being used to store JCache caches. |
The Coherence-JVisualVM plug-in can be used to monitor Coherence clusters that are managed in a WebLogic Server domain. One managed Coherence server is automatically selected as the management proxy. You can configure which managed Coherence servers can be selected as the management proxy. See Removing a Coherence Management Proxy in Administering Clusters for Oracle WebLogic Server. The WebLogic Server domain runtime MBean server collects JMX information from the management proxy.
Note:
When connecting to a WebLogic Server multitenant domain, the Coherence-JVisualVM plugin only displays when connected as the WebLogic domain administrator. You cannot connect to a multitenant domain as the partition administrator. As the WebLogic Server domain administration, you can access all domain partitions. Ensure that you choose the correct partition when performing any service, cache, or persistence operations.
To connect to the domain runtime MBean server using Java VisualVM:
The Coherence-JVisualVM plug-in is set to refresh data from the Coherence MBean server by default every 30 seconds. Data on the details section of most tabs (Services tab, Caches tab, Federation tab, and so on) is immediately cleared when a different cache is selected and is refreshed on the next refresh cycle.
To change the refresh setting, use the following system property when starting Java VisualVM:
jvisualvm -J-Dcoherence.jvisualvm.refreshtime=10
If refreshing management data is adversely affecting performance (especially on large clusters), then consider leaving the default value for the plug-in refresh interval and changing the default Coherence management refresh expiry time of 1 second to a higher value. The value can be specified within an operational override file or by using a system property. See Configuring Management Refresh.
In addition, the Coherence-JVisualVM plug-in can be configured to include query times in the console logs. The logs are helpful when troubleshooting performance issues.
To enable query times in the logs, use the following system property when starting Java VisualVM:
-J-Dcoherence.jvisualvm.log.query.times=true
To view Java VisualVM logs in the console, select Help | About and then from the About Java VisualVM screen, click Logfile.