15 Provisioning Coherence Nodes and Clusters

This chapter explains how you can provision Coherence nodes or clusters across multiple targets in a farm. In particular, this chapter covers the following:

Getting Started

This section helps you get started with this chapter by providing an overview of the steps involved in provisioning Coherence nodes and clusters. The Coherence Deployment Procedure allows you to do the following:

  • Add one or more nodes to a new cluster and add this cluster as an Enterprise Manager monitored target.

  • Add a management node to an existing cluster and add this cluster as an Enterprise Manager monitored target.

  • Add one or more nodes to a cluster that is already being monitored by Enterprise Manager.

Consider this section to be a documentation map to understand the sequence of actions you must perform to successfully deploy a Coherence node. Click the reference links provided against the steps to reach the relevant sections that provide more information.

Table 15-1 Getting Started with Deploying a Coherence Node

Step Description Reference Links

Step 1

Understanding the Deployment Procedure

Understand the Deployment Procedure that is offered by Enterprise Manager Grid Control for deploying a Coherence node. Know how the Deployment Procedure functions, what use cases it covers, and so on.

To learn about the Deployment Procedure, see Deployment Procedure.

Step 2

Knowing About the Supported Releases

Know what releases of Oracle Coherence are supported.

To learn about the releases supported by the Deployment Procedure, see Supported Releases.

Step 3

Meeting the Prerequisites

Before you run any Deployment Procedure, you must meet the prerequisites, such as adding the Coherence node as a target, and setting up of Oracle Software Library.

To learn about the prerequisites for deploying a Coherence node, see Prerequisites.

Step 4

Running the Deployment Procedure

Run the Deployment Procedure to successfully deploy a Coherence node.

To deploy a Coherence node, follow the steps explained in Procedure.

Deployment Procedure

Enterprise Manager Grid Control offers the following Deployment Procedure:

  • Coherence Node Provisioning

Coherence Architecture

Before you can monitor a coherence node, you must identify the management node in the cluster and add its details in Enterprise Manager. The following diagram shows the Coherence architecture:

Figure 15-1 Architecture Diagram: Monitoring the Coherence Cluster

Coherence Architecture Diagram

The management node hosts the Coherence MBeanServer. All Coherence MBeans, Enterprise Manager MBeans for Coherence, and customer specific MBeans are registered in this MBeanServer. Since the number of MBeans may be large, we recommend that you use the BulkOperationsMBean to upload data from all the MBeans quickly and efficiently.

When the Enterprise Manager Agent is installed, the $ORACLE_HOME/sysman/jlib/coherenceEMIntg.jar and the $ORACLE_HOME/modules/bulkoperationsmbean_11.1.1.jar files are also installed. The bulkoperationsmbean_11.1.1.jar contains all required classes needed for the BulkOperationsMBean. The coherenceEMIntg.jar contains the oracle.sysman.integration.coherence.EMIntegrationServer class that starts a new Coherence management node with the MBeanServer, registers all the MBeans in this MBeanServer along with the BulkOperationsMBean. We recommend that you add this new management node using the BulkOperationsMBean. You can monitor a cluster without the BulkOperationsMBean but data collection may be slow and inefficient.

Note:

You can deploy a node on remote hosts only if these remote hosts are being monitored by Enterprise Manager. If a host is not monitored by Enterprise Manager, you cannot create new nodes on that host.

Supported Releases

This section lists the releases of Oracle Coherence supported by this Deployment Procedure:

  • Oracle Coherence 3.3, 3.4, 3.5, and 3.6

Deploying a Coherence Node

This section describes how you can deploy a Coherence node using the Coherence Node Provisioning procedure.

This section covers the following:

Prerequisites

Before running the Deployment Procedure, meet the following prerequisites:

  • The host on which a Coherence node needs to be deployed should be a monitored target in Oracle Enterprise Manager.

  • A software library with the necessary components that include Coherence jars, configuration files, and application specific jars must be created. We recommend that you add all software components with the product name Coherence.

  • A zip file with the Coherence software, default configuration files and start scripts must be created. This zip file must be added as a software component to the Enterprise Manager software library.

  • To facilitate the creation of the management node, you must add coherenceEMIntg.jar and bulkoperationsmbean_11.1.1.jar as software components to the library.

Procedure

To deploy a Coherence cluster, follow these steps:

  1. In Grid Control, click Deployments.

  2. On the Deployments page, click Deployment Procedures.

  3. All the available deployment procedures are listed on the Deployment Procedure Manager page. Click the Coherence Node Provisioning deployment procedure on the list.

  4. The View Procedure page is displayed. Click Schedule Deployment.

  5. The Source Selection page which is the first page of the Coherence Node Provisioning wizard is displayed. On this page, do the following:

    Figure 15-2 Source Selection Page

    Source Selection Page

    1. Click Add to add a software component. The Select Source popup is displayed. By default all software components with the product name Coherence are displayed. Select required components and click Select.

    2. All selected components appear in Sources table in the Source Selection page. Specify the destination directory information for all selected components. The destination directory must be an absolute or relative path from COHERENCE_HOME (contains all the Coherence related files).

      Note:

      You can skip this step if all the required software components are available on the target machine.

    3. Click Next to continue with the next step in the wizard.

  6. The Target Selection page is displayed. On this page, you can add one or more nodes that need to be deployed. You can add nodes to:

    1. EM Monitored Coherence Cluster Target: Click the Search icon and select a target from the list.

    2. New EM Coherence Cluster Target: This option allows you to add a new Coherence target. Enter the following details:

      Figure 15-3 Target Selection Page

      Target Selection Page

      Table 15-2 Target Selection Page

      Field Name Description

      Cluster Name

      Enter the name of the cluster. This is a mandatory field.

      Cluster Port / Address

      If you do not specify the values for the Cluster Port and Cluster Address, the values in the tangosol_coherence.xml configuration file are used.

      Coherence Version

      The default version used is 3.3 but you can specify 3.4 or later.

    3. Click Add to add nodes to the cluster. The Add Coherence Node page is displayed. Enter the following details:

      Figure 15-4 Add Coherence Node Page

      Add Coherence Node Page

      Table 15-3 Add Coherence Node Page

      Field Name Description

      Host Details

      Host Name

      Select the host on which the node is to be added.You may have more than one node on the host depending on the machine configuration and the node configuration.

      Note: The Host Name you select here is used to set two start up parameters. Use the tangosol_coherence_machine environment variable to set the tangosol.coherence.machine parameter and oracle_coherence_machine to set the oracle.coherence.machine parameter.

      Number of Nodes

      Specify the number of nodes that need to be added. By default, this field has the value of 1 but you can add as many nodes as required. If the value is more than 1, then all the nodes will have the following properties:

      • Each node will use the same COHERENCE_HOME and start script.

      • The Node Name value will be added as the prefix and a number will be appended to each node. For example <node_name>_1, <node_name>_2 and so on. Each name should be a unique one in the cluster.

      • The JMX Remote Port value will be increased by 1 for each additional node. For example, if the value of the JMX Remote Port for the first node is 8088, the value for the second node will be 8089 and so on.

      Credentials

      Use Preferred Credentials

      Select this option, if you already entered the credentials on the Preferred Credentials page under Preferences.

      Override Preferred Credentials

      Select this option to explicitly specify the user name and password and override the preferred credentials.

      Node Details

      Node Name

      Enter a unique name for the node

      Site Name

      This is the location of the Coherence node. This is geographical physical site name which identifies the racks and machines on which the node is running.

      Rack Name

      The name of the rack in the site on which the machine is located.

      Role Name

      The role could be storage/data, application/process, proxy or management node.

      Note: The Node Name, Site Name, Rack Name, and Role Name cannot exceed 32 characters.

      Do not Copy Software Components

      If you are creating more than one node with same software components on a host, you need not copy these components again. By selecting this option, you can skip copying of software components.

      Management Node Details

      Management Node with MBeanServer

      You can define multiple management nodes in the cluster but only one management node can be marked as the Primary Management Node. We recommend that you add at least two management nodes preferably running on different hosts / machines to support fail over.

      JMX Remote Port

      The port number of the EMIntegration Mbean server.

      JMX User Name

      The user name for the JMX server if authentication is enabled.

      JMX Password

      The password for the JMX server if authentication is enabled.

      Note: To enable the JMX authentication, you need to set com.sun.management.jmxremote.authenticate=true. The JMX User name and JMX Password need to be set in the $JDK_HOME/jre/lib/management/jmxremote.password and $JDK_HOME/jre/lib/management/jmxremote.access files.

      Primary Management Node used for Monitoring

      Select this checkbox to mark the management node you are adding as the Primary Management Node used for Monitoring. If several nodes are being added to a cluster, only one management node can be marked as the primary one. If the primary management node fails, you can configure any of the other management nodes for monitoring. If no other management node is available, you can add a new primary management node to an existing cluster and this node can be used to monitoring.

      Note: To configure the management node, navigate to the Monitoring Configuration by clicking this link under the Related Links in the Cluster Home page.

      Use Bulk Operations MBean

      This checkbox is selected by default. When this option is selected and bulkoperationsmbean_11.1.1.jar and the coherenceEMIntg.jar files have been added to classpath, a new management node with BulkOperationsMBean will be started.

      Environment Details

      		  

      Coherence Home

      Enter the absolute path to the folder under which the Coherence software components reside. The path specified here will be used as the Destination Directory specified on the Coherence Node Provisioning: Source Selection page. This value could be different for each node or the same for one or more nodes.

      Start Script

      This script is used to bring up the Coherence node. This script is operating system specific and sets the proper environment required for the node by specifying the relevant system parameters. See sample script for an example.

      The following table lists the environment variables and their values that can be used to set the Coherence System Parameters in the start script. The deployment procedure sets these environment variables with the inputs you specify in the interview wizard. You can execute a separate script to set different values for these environment variables in the start script. You can use these environment variables to set start parameters for the Coherence node or override them by specifying appropriate values.

      Table 15-4 Environment Variables

      UI Parameter Environment Variable Set Coherence System Parameter

      COHERENCE_HOME

      coherence_home

      oracle.coherence.home

      JAVA_HOME

      JAVA_HOME

      JAVA_HOME

      APP_HOME

      APP_HOME

      APP_HOME

      ORACLE_HOME

      ORACLE_HOME

      ORACLE_HOME

      CLUSTER_NAME

      tangosol_coherence_cluster

      tangosol.coherence.cluster

      CLUSTER_ADDRESS

      tangosol_coherence_clusteraddress

      tangosol.coherence.clusteraddress

      CLUSTER_PORT

      tangosol_coherence_clusterport

      tangosol.coherence.clusterport

      NODE_NAME (32 chars)

      tangosol_coherence_member

      tangosol.coherence.member

      SITE_NAME (32 chars)

      tangosol_coherence_site

      tangosol.coherence.site

      RACK_NAME (32 chars)

      tangosol_coherence_rack

      tangosol.coherence.rack

      MACHINE_NAME

      tangosol_coherence_machine

      tangosol.coherence.machine

      ROLE_NAME (32 chars)

      tangosol_coherence_role

      tangosol.coherence.role

      JMX_REMOTE_PORT

      jmx_remote_port

      com.sun.management.jmxremote.port

      PROCESS_NAME

      tangosol_coherence_process

      tangosol.coherence.process

      BULK_MBEAN

      bulk_mbean

      		  

      JMX_ENABLE_AUTH

      jmx_enable_auth

      com.sun.management.jmxremote.authenticate

      START_SCRIPT

      start_script

      oracle.coherence.startscript

    4. After adding the node, click Continue to return to the Coherence Node Provisioning: Target Selection page.

    5. Click Next to go to the next step in the wizard.

  7. The Schedule page is displayed. On this page, you can specify the schedule for deploying the node. You can choose to deploy the node immediately or at a later date.

    Figure 15-5 Schedule Page

    Schedule Page

    Note:

    If you set the Grace Period as Indefinite, Enterprise Manager will keep trying to deploy the node for an indefinite period. If you specify a date / time in this field, the deployment process will be aborted after this period.

  8. Click Next to go to the next step in the wizard.

  9. The Review page is displayed. On this page, review the details you have provided for deploying the node. Click Finish to deploy the node.

    Note:

    Depending on the schedule you have defined, the deployment will take place either immediately or at a later date as specified.

Sample Script

A sample script to start a coherence management node is shown below. When this script is executed from Deployment Procedure, it sets some environment variables. If you want to override those variables you may run a local script as part of this script and override those settings. If the JMX Remote Port is defined, then this would create a Coherence management node with a MBeanServer. A BulkOperationsMBean also will be registered.

Sample Script (Unix): You can use the default-start-script.pl script:

#!/usr/local/bin/perl
# 
# Sample script to demonstrate starting of following Coherence nodes.
# When this script is passed in as a start script in Coherence Node Provisioning
# Deployment Procedure, while executing start node step, deployment procedure sets
# all user entered options as environment variables. Based on the values of these
# environment variables, you can start different types of Coherence nodes.
#
# - Management Node with Oracle Bulk Operation MBean is started when
#  "bulk_mbean" and "jmx_remote_port" variables are set. For this option,
#   oracle.sysman.integration.coherence.EMIntegrationServer Java class is executed
    that starts a MBeanServer in this node and registers Oracle Bulk Operation
    MBean. You need coherenceEMIntg.jar and bulkoperationsmbean_11.1.1.jar in the
    classpath. Oracle recommends the usage of Bulk Operation MBean as it improves
    performance of metric collection.
# 
# - Management Node is started when "jmx_remote_port" is set, but "bulk_mbean" is
    NOT set.
#
# - Managed node when "jmx_remote_port" is not set.
#
#
# Following variables are set from the deployment procedure. Use these values to
# define required system parameters to override Coherence default settings.
#
 
my $coherence_home=$ENV{'COHERENCE_HOME'};
my $start_script=$ENV{'START_SCRIPT'};
my $java_home=$ENV{'JAVA_HOME'};
my $app_home=$ENV{'APP_HOME'};
my $oracle_home=$ENV{'ORACLE_HOME'};
 
my $member=$ENV{'tangosol_coherence_member'};
my $site=$ENV{'tangosol_coherence_site'};
my $rack=$ENV{'tangosol_coherence_rack'};
my $machine=$ENV{'tangosol_coherence_machine'};

# tangosol.coherence.machine has a limitation of 32 chars. As a workaround, use
# oracle.coherence.machine to set machine name. This parameter is used to identify
# hosts for cluster management features
my $oracle_coherence_machine=$ENV{'oracle_coherence_machine'};
my $role=$ENV{'tangosol_coherence_role'};
 
my $jmxport=$ENV{'jmx_remote_port'};
my $cluster=$ENV{'tangosol_coherence_cluster'};
my $clusteraddr=$ENV{'tangosol_coherence_clusteraddress'};
my $clusterport=$ENV{'tangosol_coherence_clusterport'};
my $bulkmbean=$ENV{'bulk_mbean'};
my $jmx_auth=$ENV{'jmx_enable_auth'};
 
my $SYS_OPT="";
my $JVM_OPT="";
 
my $psep="";
my $dsep="";
if ( !&IsWindows() ) {
        $psep=":";
        $dsep="/";
}
else
{
        $psep=";";
        $dsep="\\";
}
 
my $cohlogfile=$coherence_home.$dsep.$member.".cohout";
 
my $logfile=$coherence_home.$dsep.$member.".log";
print "$logfile\n";
open(LOGFL,"> $logfile");
 
print LOGFL "Starting Node : $member\n";
print LOGFL "Coherence Home : $coherence_home \n";
print LOGFL "Start Script : $start_script \n";
print LOGFL "Java Home : $java_home \n";
print LOGFL "Oracle Home : $oracle_home \n";
print LOGFL "APP Home : $app_home \n";
 
print LOGFL "Site Name : $site \n";
print LOGFL "Rack Name : $rack \n";
print LOGFL "Machine Name : $machine \n";
print LOGFL "Oracle Coherence Machine Name : $oracle_coherence_machine \n";
print LOGFL "Role Name : $role \n";

print LOGFL "Cluster Name : $cluster \n";
print LOGFL "Cluster Addr : $clusteraddr \n";
print LOGFL "Cluster Port : $clusterport \n";
print LOGFL "JMX Port : $jmxport \n";
print LOGFL "Bulk MBean : $bulkmbean \n";
print LOGFL "JMX Auth Enabled : $jmx_auth \n";
 
#
# you may run a local script as part of this script and override those
# settings.
# Override JAVA_HOME variable by setting it locally
#
#. ./set-env.sh
#echo "After setting JAVA_HOME locally, JAVA_HOME: $JAVA_HOME"
 
# Options for Java Virtual Machine.
$JVM_OPT="-server -Xms512m -Xmx512m -Xincgc -verbose:gc";
 
#
# Set system parameters to Coherence node
$SYS_OPT="-Djava.net.preferIPv4Stack=true";
 
# This param allows the mbeans on this node to be registered to mbean servers running on management nodes
$SYS_OPT="$SYS_OPT -Dtangosol.coherence.management.remote=true";
 
$SYS_OPT="$SYS_OPT -Dcom.sun.management.jmxremote.ssl=false";
$SYS_OPT="$SYS_OPT -Dtangosol.coherence.cluster=$cluster";
$SYS_OPT="$SYS_OPT -Dtangosol.coherence.member=$member";
$SYS_OPT="$SYS_OPT -Dtangosol.coherence.site=$site";
$SYS_OPT="$SYS_OPT -Dtangosol.coherence.rack=$rack";
$SYS_OPT="$SYS_OPT -Dtangosol.coherence.machine=$machine";
# set this if machine name > 32 chars, tangosol.coherence.machine has a limitation of 32 chars
$SYS_OPT="$SYS_OPT -Doracle.coherence.machine=$oracle_coherence_machine";
$SYS_OPT="$SYS_OPT -Dtangosol.coherence.role=$role";

# Set coh home and start script so they will be part of input args$SYS_OPT="$SYS_OPT -Doracle.coherence.home=$coherence_home";$SYS_OPT="$SYS_OPT -Doracle.coherence.startscript=$start_script";

# set jmxremote.authenticate=true if $jmx_enable_auth is present.
# username/password needs to be set in $JDK_HOME/jre/lib/management/jmxremote.password and
# $JDK_HOME/jre/lib/management/jmxremote.access files. Uncomment the following block after adding these files.
#
#if ("$jmx_auth" ne "") {
#      $SYS_OPT="$SYS_OPT -Dcom.sun.management.jmxremote.authenticate=$jmx_auth";
#} else {
#       $SYS_OPT="$SYS_OPT -Dcom.sun.management.jmxremote.authenticate=false";
#}
 
# Default is true, so make sure to set it to false if not using authentication
        $SYS_OPT="$SYS_OPT -Dcom.sun.management.jmxremote.authenticate=false";
 
 
# set jmxremote.port only for management nodes, if user passes in.
# It enables monitoring from remote systems through this port.
if ($jmxport ne  "") {
        $SYS_OPT="$SYS_OPT -Dcom.sun.management.jmxremote.port=$jmxport";
}
 
 
#
# Define clusteraddress and clusterport if we have valid values.
#
if($clusteraddr ne "") {
        $SYS_OPT="$SYS_OPT -Dtangosol.coherence.clusteraddress=$clusteraddr";
}
 
if("$clusterport" ne "") {
        $SYS_OPT="$SYS_OPT -Dtangosol.coherence.clusterport=$clusterport";
}
 
my $startup_class="";
my $cmd="";
 
# Note that Coherence lib is under $COHERENCE_HOME/coherence. Add any application specific jars to this classpath, if needed.

my $CLASSPATH=$coherence_home.$dsep."coherence".$dsep."lib".$dsep."coherence.jar".$psep.$coherence_home.$dsep."coherence".$dsep."lib".$dsep."reporter.jar";

print LOGFL "CLASSPATH: $CLASSPATH\n";

if ($bulkmbean ne "" && $jmxport ne "") {
        # Management node with Bulk Operation MBean.
        # add Oracle supplied jars for Bulk Operation MBean

$CLASSPATH=$CLASSPATH.$psep.$coherence_home.$dsep."coherenceEMIntg.jar".$psep.$coherence_home.$dsep."bulkoperationsmbean_11.1.1.jar";
# Start MBeanServer
     $SYS_OPT="$SYS_OPT -Dtangosol.coherence.management=all";
      print LOGFL "Starting a management node with Bulk Operation MBean \n";
     $startup_class="oracle.sysman.integration.coherence.EMIntegrationServer";
     $cmd=$java_home.$dsep."bin".$dsep."java -cp $CLASSPATH $JVM_OPT $SYS_OPT $startup_class > $cohlogfile 2<&1";
} elsif ($jmxport ne "") {
        # Management Node with out Bulk Operation MBean
        # Start MBeanServer
        $SYS_OPT="$SYS_OPT -Dtangosol.coherence.management=all";
        print LOGFL "Starting a management node ...\n";
        $startup_class="com.tangosol.net.DefaultCacheServer";
$cmd=$java_home.$dsep."bin".$dsep."java -cp $CLASSPATH $JVM_OPT $SYS_OPT $startup_class > $cohlogfile 2<&1";
} else {  
        # A simple managed node. Do not start MBeanServer.
        $SYS_OPT="$SYS_OPT -Dtangosol.coherence.management=none";
        print LOGFL "Starting a simple managed node ...\n";
        $startup_class="com.tangosol.net.DefaultCacheServer";
$cmd=$java_home.$dsep."bin".$dsep."java -cp $CLASSPATH $JVM_OPT $SYS_OPT $startup_class > $cohlogfile 2<&1";
}
                   
 
if ( !&IsWindows() ) {
    if (fork() == 0) {
        print LOGFL "Executing start script from child process... $cmd \n";
        exec("$cmd") or die "Could not execute $cmd\n";
    }
} else {
        print LOGFL "Command used to start node = $cmd\n";
        exec($cmd);
}
close(LOGFL);
exit 0;
 
sub lsWindows {
    $osname = $^O;
    if (   $osname eq "Windows_NT"
        || $osname eq "MSWin32"
        || $osname eq "MSWin64" )
    {
        return 1;
    }
    else {
        return 0;
    }
}

Sample Script (Windows): You need to run the start-node.vbs script which in turn launches the default-start-script.pl script.

start-node.vbs

Set WshShell = WScript.CreateObject("WScript.Shell")Set objEnv = WshShell.Environment("Process")Dim run_cmd
run_cmd=objEnv("ORACLE_HOME") & "\perl\bin\perl.exe " & objEnv("ORACLE_HOME") & "\sysman\admin\scripts\coherence\default-start-script.pl"obj = WshShell.Run(run_cmd, 0)
set WshShell = Nothing

Troubleshooting

We recommend that you have at least two management nodes running on different machines in a Coherence cluster. If a monitoring failure occurs, the second management node can be used. Some of the common failure scenarios are listed below:

  • Error Condition: Loss of Management Node

    Solution: If the primary management node fails, you need to change the monitoring configuration of the cluster to point to another management node in the cluster. If no other management node is present in the cluster, you can use the Coherence Node Provisioning deployment procedure, select an existing cluster and add a new management node. This process will update the monitoring configuration for the cluster.

  • Error Condition: Loss of Agent Monitoring the Cluster

    Solution: If the Agent is not available, you need to use another Enterprise Manager Agent to point to the management node of Coherence cluster.

  • Error Condition: Loss of Host with EM Agent and Management Node

    Solution: If the Host is not available, you need to switch to another management node that is running on a different machine.