This chapter describes the procedure to discover and monitor a Coherence cluster using Oracle Enterprise Manager Cloud Control 13c. The following sections are covered in this chapter:
Oracle Coherence is an in-memory data-grid and distributed caching solution. It is composed of many individual nodes or java processes which work together to provide highly reliable and high speed virtual caching.
Enterprise Manager provides deep visibility into performance of all the artifacts such as caches, nodes, and services. Nodes and caches can be proactively monitored by the Incident Management feature. You can create a monitoring template by pre-populating the monitoring template with metrics for a Coherence target. You can export and import monitoring templates to share monitoring settings between different Enterprise Manager deployments.
Metric Extensions are the next generation of User-Defined Metrics, which enable you to extend Enterprise Manager to monitor conditions specific to the enterprise's environment by creating new metrics for any target type. By including metric extensions in export or imported monitoring templates, multiple metric extensions can be easily shared at the same time between Enterprise Manager deployments.
You can correlate cluster nodes with the underlying hosts to determine CPU and memory utilization on those hosts in order to make better decisions for scaling your clusters. You can see the association between the caches, nodes, hosts, and Oracle WebLogic targets.
Highly customizable performance views for monitoring performance charts and trends are available. You can overlay metrics for multiple nodes or caches in the same or different cluster for detail analysis to provide detailed visibility at the desired level. The drill down views allows you to determine the root cause of performance problems or simply identify performance trends in the Coherence Cluster.
Enterprise Manager provides a centralized cache data management feature that allows you to perform various cache operations such as add/remove index, view cache data, view query explain plan, and so on.
Enterprise Manager monitors the changing configuration of the nodes over a period of time. The Topology Viewer provides a high level topology of the entire cluster and shows the relation between caches, nodes and hosts.
All of the Coherence Management features are integrated with JVM Diagnostics and provide real-time visibility into the node JVMs. You can drill down to a Coherence node's JVM from within the context of a cache and a cluster to identify the method or thread that is causing a delay. The JVM Diagnostics feature is part of the WLS Management Pack EE and Management Pack for NonOracle Middleware.
Enterprise Manager provides a complete provisioning solution. You can maintain an Oracle Coherence setup image or gold image in the Software Library and deploy it throughout the infrastructure to create completely new clusters or add nodes an existing cluster. You can use the same deployment procedure to updates nodes as well.
Target Navigation: The target navigation menu is hidden by default. Click the Target Navigation icon on the left hand side of the page to display the Target Navigation tree. The navigation tree makes it easier to navigate to any node or cache from any page. Nodes are grouped based on the hosts on which they are running, and caches are grouped based on services. See Navigation Treefor details.
Heat Map: The Cluster Home page now has a Heat Map tab which provides a graphical representation of all the targets in the cluster.
Log Viewer: You can now scan the log files and view the log file data.
Remove Down Members: You can now delete any members of the cluster that have a Down status.
Managed Coherence Clusters: The discovery and monitoring of managed Coherence clusters has now been integrated with Oracle Fusion Middleware/Weblogic Domain discovery and monitoring.
Custom Topology Viewer: The custom topology viewer provides a customized view of all the targets in a Coherence cluster.
Federated Caching: You can synchronize cache instances across clusters by replicating updates from a cache in a source cluster to a cache in a remote cluster.
Discovery of Dynamic Management Node: The coherence 12.2.1.x and above can be configured to start in dynamic management mode. Dynamic management mode automatically selects the senior cluster member as the management node (JMX cluster member). If the central management node is not operational, then the next most senior coherence node is automatically selected as the management node.
Federated Operations: A new Federated Caching Operations region is added to Service Home Page. This allows the user to submit operation request. Based on operation selection, federation operation will be executed with reference to that remote participant.
This section covers the configuration procedure for a standalone Coherence cluster.
For details on configuring a managed Coherence cluster, refer to the WebLogic documentation.
Oracle Coherence standalone deployments can be monitored using Enterprise Manager by configuring the Coherence nodes with a set of Coherence and JMX system properties (start arguments). In addition, one of the nodes will have to be configured as a central JMX management node. This JMX management node must expose all Coherence MBeans and attributes. See Creating and Starting a JMX Management Node for details. In addition to configuring the JMX management node, the Management Agent must also be installed and configured on the same host as JMX management node. This is required to discover and monitor the Coherence cluster in Enterprise Manager.
Figure 23-1 shows the configuration for monitoring standalone Coherence clusters using Enterprise Manager.
Figure 23-1 Coherence Cluster Configuration (Standalone Coherence Cluster)
As shown in the figure, Coherence Management (JMX) node's MBean server will expose MBeans for entire Coherence cluster. Enterprise Manager will connect to this management node to discover and monitor Coherence cluster.
The Management Agent uses the JMX management node (centralized MBean server) to discover and monitor the entire Coherence cluster, including the nodes and caches. As a best practice, it is recommended that the Management Agent be present on the same host as the JMX management node that is used to discover and monitor the Coherence cluster. The Management Agent must be setup on all the machines on which the Coherence nodes are running to monitor and provision the cluster. For more information on using JMX to manage Oracle Coherence, see Using JMX to Manage Coherence in the Oracle Coherence Management guide. To configure the JMX management node, you must:
Specify Additional System Properties
Include Additional Class Path
Use the Enterprise Manager Custom Start Class
Oracle recommends that the management node is configured as a storage disabled node to ensure minimal performance impact on any Coherence caches.
The following start arguments must be added to one of the Coherence nodes to configure it as the JMX central management node.
-Dtangosol.coherence.management.extendedmbeanname=true (allows any restarted node to be automatically detected by Enterprise Manager. This parameter is available in Coherence 18.104.22.168 and later versions)
If set to true, the status of the node is automatically refreshed when a node is restarted.
If this property is not set, you must use the Refresh Cluster option to update the status of a node when it is restarted.
If you start a node after setting this property to true, all nodes in the cluster must be started after the
extendedmbeanname property is set to true.
-Dtangosol.coherence.management=all (enables monitoring for all nodes)
-Dcom.sun.management.jmxremote.port=<port number> (required for remote connection for coherence 12.2.1.x or older versions)
-Dtangosol.coherence.distributed.localstorage=false (disables caching and ensures that the node is a dedicated monitoring node)
-Dtangosol.coherence.member=<member name> (required for target name)
-Doracle.coherence.machine=<fully qualified hostname> (must match the name of the host discovered in Enterprise Manager)
If you are using JMX credentials, you must set the following additional start arguments.
If no JMX credentials are used, you must set these arguments to false.
You must include the path to both Enterprise Manager custom jar files,
Coherence cluster with version 12.2.1 and above, must use the
coherenceEMIntg.jar file available in the
The location of the .jar files may change based on the plugin version.
In addition to configuring the system properties and the class path when starting Coherence management node, it is also required that you use the Enterprise Manager
EMIntegrationServer class as the start class. This class allows you to register the custom MBeans required for the Cache Data Management feature of Management Pack for Oracle Coherence.
# !/bin/sh CP=$CP:<EM CC_Agent_Home>/plugins/oracle.sysman.emas.agent.plugin_22.214.171.124.0/archives/coherence/coherenceEMIntg.jar: <EM CC_Agent_Home>/plugins/oracle.sysman.emas.agent.plugin_126.96.36.199.0/archives/coherence/bulkoperationsmbean.jar COH_OPTS="$COH_OPTS -cp $CP" $JAVA_HOME/bin/java $COH_OPTS -Dtangosol.coherence.management.extendedmbeanname=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dtangosol.coherence.management=all -Dtangosol.coherence.member=<unique member name> -Doracle.coherence.machine=<hostname_as_discovered_in_EM> -Dcom.sun.management.jmxremote.port=<OpenTCP_Port> -Doracle.coherence.home=$COHERENCE_HOME -Dtangosol.coherence.distributed.localstorage=false -Dtangosol.coherence.management.refresh.expiry=1m -server -Xms2048m -Xmx2048m oracle.sysman.integration.coherence.EMIntegrationServer
In addition to configuring the Coherence JMX management node, you must configure all other Coherence cluster nodes with additional Coherence specific system properties (start arguments) used by Enterprise Manager.
The following system properties must be added to all other Coherence nodes.
-Dtangosol.coherence.management.extendedmbeanname=true -Dtangosol.coherence.management.remote=true –Dtangosol.coherence.member=<unique member name> -Doracle.coherence.home=<coherence home> -Doracle.coherence.machine=<machine name> should be the same as the name of the host discovered in Enterprise Manager.
If you are using JMX credentials, you must set the following additional start arguments.
If no JMX credentials are used, you must set these arguments to false.
#!/bin/sh COH_OPTS="$COH_OPTS -cp $CP" $JAVA_HOME/bin/java $COH_OPTS -Dtangosol.coherence.management.extendedmbeanname=true -Dtangosol.coherence.management.remote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Doracle.coherence.home=<coherence home> -Dtangosol.coherence.member=<unique member name> -Doracle.coherence.machine=<hostname> -Dcom.tangosol.net.DefaultCacheServer
To test the Coherence cluster configuration for use in Enterprise Manager, you must verify that the central management (JMX) node has information regarding the managed objects of all other Coherence cluster nodes, caches, services, and so on. Additionally, you must verify that the central management node is accessible remotely, either through
<hostname>:<port> OR the JMX Service URL. If JMX credentials are used, they should also be specified.
JConsole is a Java tool available through JDK. You can use this to verify remote access to the MBean objects of entire Coherence cluster nodes, caches, services, and so on.
Figure 23-2 JConsole
To verify remote access, open JConsole and select "New Connection". In New Connection page, select Remote Process and provide connection details where
<hostname> is the name of the machine where central management node is running,
<port> is what you have specified in the
-Dcom.sun.management.jmxremote.port parameter while starting the management node. If successful, you will see the MBean object tree.
Figure 23-3 MBean Object Tree
If you see MBeans for all Coherence nodes in the System MBean Browser or JConsole, you can now discover and monitor the Coherence cluster and its associated elements in Enterprise Manager.
This section covers the following:
Enterprise Manager monitors the entire Coherence cluster and its artifacts. The key targets that can be monitored are Oracle Coherence Cluster, Oracle Coherence Node, and Oracle Coherence Cache. The Oracle Coherence Cluster target provides a high level view of the health of the entire cluster. The Oracle Coherence Node and Oracle Coherence Cache are child targets of the Oracle Coherence Cluster. In addition to monitoring the above target types, additional Coherence components such as Services, Connections, and Applications can also monitored.
To provision new Coherence nodes, start, and stop nodes, the Management Agent must be installed on all the hosts on which the nodes are running. For more details on provisioning Coherence nodes, see the Enterprise Manager Lifecycle Management Guide.
Created a Coherence cluster with one JMX management node and one or more other nodes.
Started the JMX management node with the necessary parameters as defined in Creating and Starting a JMX Management Node.
Started the other nodes with the necessary parameters as defined in Configuring All Other Nodes.
To discover an already running Coherence cluster, follow these steps:
Alternatively, you can add a Coherence target from the Setup menu. From the Setup menu, select Add Target, then select Add Targets Manually. In the Add Targets Manually page, select the Add Non-Host Targets Using Guided Process option. Follow the steps in the wizard to add the Coherence target.
If you have selected option Standalone Coherence cluster configured with dedicated management node, you can select either of the following options to provide MBean Server details:
Figure 23-4 Add Coherence Target
Host, Port, and Service: Enter the following details:
Management Node Host: Select the host on which the Management Node is running.
JMX Remote Port: The port used for the JMX RMI connection. If you are using the MBean connector for Coherence MBeans, specify the
It is recommended that you use the
Service Name: The service name used for the connection. The default is
JMX Service URL: Service URL that will be used for the connection. If you enter the URL, the values specified in the Machine Name, Port, Communication Protocol, and ServiceName fields will be ignored. For example,
service:jmx:rmi://localhost:3000/jndi/rmi://localhost:9000/server. For more details on the URL format, refer to
You may need to specify the Service URL only in complex cases like when the RMI registry and the MBean Server ports are different. It is recommended that you use the Machine Name and Port option for the MBean server connection.
Figure 23-5 Add Coherence Target
Cluster Port: In dynamic management mode any coherence node configured with dynamic managed mode can be a Management Node. Hence, you must enter coherence Cluster Port to discover Coherence target instead of JMX port.
Host Name: You must enter the host name where nodes with dynamic management mode are running. You can see a list of host names on which coherence nodes with dynamic management mode are running. If the current management node is down then the next management node is discovered using these host names.
To automatically discover a new node or target in Enterprise Manager, you must refresh the cluster as described in Refreshing a Cluster.
You can manually synchronize the cluster targets with the running Coherence cluster. Click Refresh Cluster from the Oracle Coherence Cluster menu. A message indicating that new Coherence nodes and caches that have been discovered will be added as Enterprise Manager targets is displayed. Nodes are updated if there are any changes to their attributes. Click Continue to refresh the cluster. This ensures that the latest changes are applied.
Figure 23-6 Refresh Cluster
Click Close. The list of nodes and caches that can be added are displayed. Click Add Targets to add the targets to the cluster.
Decommissioned nodes and caches will not be removed during the Refresh process. You must remove them manually.
While discovering a Coherence cluster, all nodes must be started with the proper guidelines as described in Configuring a Coherence Cluster.
If a node is improperly configured or has been started without the necessary guidelines, it will be categorized as a mis-configured node and will not be a part of the newly discovered cluster. During discovery, if any improperly configured nodes are present in the cluster, you will see the following screen:
Figure 23-7 Mis-Configured Nodes
This indicates that there are some improperly configured nodes in the cluster. Click Close. The following page is displayed.
Figure 23-8 Mis-configured Nodes II
The list of improperly configured nodes along with the reasons for their failure is listed in this page. You can either choose to cancel the discovery process and fix these nodes or continue with the discovery with the properly configured nodes.
If you wish to continue with the discovery process, follow the steps listed in Discovering Coherence Targets.
If you click Cancel, the discovery process is aborted and the cluster is not refreshed. If mis-configured nodes are found during the Refresh process, they must be fixed before you can run the Refresh operation again. For more information, see Refreshing a Cluster and then discover the cluster.
You can discover a managed Coherence cluster while discovering an Oracle Fusion Middleware / WebLogic Domain by following these steps:
Figure 23-9 Add Oracle Fusion Middleware / WebLogic Domain: Find Targets
Administration Server Host: Enter the host name on which the Administration Server is installed.
Port: Enter the WebLogic Administration Server port.
Username and Password: Enter the user name and password for the WebLogic Administration Server.
Agent: Enter The host name for a Management Agent that will be used to discover the Fusion Middleware targets.
Figure 23-10 Targets and Agents Assignments
For more details on Oracle Fusion Middleware / WebLogic Domain discovery, see the Oracle Fusion Middleware documentation.
Note:If the management node of a 12.2.1 managed cluster is restarted, you must manually refresh the WebLogic Domain before you can continue monitoring the cluster.
For managed Coherence clusters, you must enable the Oracle Cloud Management Pack for Oracle Fusion Middleware.
You must enable the Management Pack for Oracle Coherence if you want to use any custom features. If the management pack is not enabled, you can access only the Home pages and base platform features. To enable the Management Pack, do the following:
Apart from enabling the Management Pack, you must grant
VIEW privileges to all users on the Management Agent that is monitoring the Coherence targets. This ensures that all targets being monitored by the Management Agent are visible to the user.