5 Multiserver Domains with Oracle Coherence

You can create, configure, and administer multiserver domains that are based on Oracle Coherence. With Oracle Coherence, the domain receives replicated and distributed (partitioned) data management services on top of a reliable and highly scalable peer-to-peer clustering protocol.

This chapter includes the following sections:

If you plan to deploy an Oracle Stream Analytics high availability application and require scalability, you might need to create an Oracle Stream Analytics high availability notification group. For more information, see High Availability Broadcast Adapter in Schema Reference for Oracle Stream Analytics.

5.1 Create a Multiserver Domain

To create a multiserver domain, start by using the Configuration Wizard to create one domain with one server. This is a standalone-server domain.

To convert the standalone-server domain to a multiserver domain, you can either:

  • Copy and rename the server you just created and then edit the config.xml file for each server so that they all have the same multicast name, multicast port, and domain name.

  • Use the Configuration Wizard to generate additional stand-alone servers and then edit the config.xml file for each server so that they all have the same multicast name, multicast port, and domain name.

The procedures in this chapter use the first approach.

5.2 Create a Multiserver Domain with Default Groups

You can create a multiserver domain that uses only the two predefined deployment groups, singleton and domain. These groups are described in Singleton Server Deployment Group. In a domain that uses default groups, all servers must be completely homogeneous.

If a domain must support servers that are not completely homogeneous, configure this by creating custom server groups. See Create a Multiserver Domain with Custom Groups.

Create a Multiserver Domain with Default Server Groups

  1. Use the Configuration Wizard to create a standalone-server domain.
  2. Change to the directory where you put the new standalone-server domain when you created it.

    By default, the location is

    Oracle/Middleware/my_oep/user_projects/domains/<domainname>/

  3. Copy and rename the server directory until you have the number of servers that you need under the domain.
  4. Update the config.xml file for each member server by adding a cluster child element of the root config element. The following example shows a server that is part of a domain called myDomain.
    <config>
      <domain>
         <name>myDomain</name>
      </domain>
      <cluster>
         <server-name>myServer1</server-name>
         <multicast-address>239.255.0.1</multicast-address>
         <enabled>true</enabled>
      </cluster>
    ...
    </config>
    

    You can use the following child elements of cluster:

    • server-name: The server-name child element of cluster specifies a unique name for the server. Oracle Stream Analytics Visualizer Using Visualizer for Oracle Stream Analytics uses the value of this element when it displays the server in its console. The default is the Oracle Coherence member name if that is set or WLEvServer-MEMBERID.

    • server-host-name: Specifies the host address/IP used for point-to-point HTTP multiserver communication. Default value is localhost. This element is mandatory if one or more Oracle Stream Analytics servers in your multiserver domain are on different hosts and you plan to manage the multiserver domain using Oracle Stream Analytics Visualizer. It is also mandatory if a server is deployed on a host machine that has multiple IP addresses configured (whether in a multiserver or standalone-server environment).

    • multicast-address: The multicast-address element is required unless all servers of the multiserver domain are hosted on the same computer; in that case you can omit the multicast-address element and Oracle Stream Analytics automatically assigns a multicast address to the multiserver domain based on the computer's IP address. If the servers are hosted on different computers, then you must provide an appropriate domain-local address. Oracle recommends that you use an address of the form 239.255.X.X. This is the basis for the auto-assigned multicast address. All the Oracle Stream Analytics servers using this multicast-address must be on the same subnet. With Oracle Coherence, you can specify a unicast address here and Oracle Coherence will use WKA (Well Known Addressing).

    • enabled: By default the clustering of the servers in a multiserver domain is disabled for Oracle Coherence. The element <enabled>true</enabled> is required if you want to enable Oracle Coherence.

    For each server of the multiserver domain, the multicast-address elements must contain the same value. The server-name element must be different for each server in the multiserver domain. The following example shows the config.xml file of a second server, called myServer2, in the myDomain multiserver domain.

    <config>
      <domain>
         <name>myDomain</name>
      </domain>
      <cluster>
         <server-name>MyServer2</server-name>
         <multicast-address>239.255.0.1</multicast-address>
         <enabled>true</enabled>
      </cluster> 
    ...
    </config>
    

    See Secure the Messages Sent Between Servers for a description of additional multiserver-related configuration elements and the required order of child elements.

  5. Optionally, override the default Oracle Coherence clustering configuration on all of the servers in the multiserver domain, if necessary.
  6. Optionally, secure the messages that are shared among all of the servers in the multiserver domain by configuring encryption and digital signatures.
  7. Consider enabling Oracle Stream Analytics Visualizer on a subset of machines within the multiserver domain so that in the event of a failure, you can use Visualizer to troubleshoot from a machine that is still operating.

    Note:

    Enabling Oracle Stream Analytics Visualizer on a given Oracle Stream Analytics Server can impact the performance of the server depending on the Oracle Stream Analytics Visualizer workload.

  8. Start all of the servers in your multiserver domain.

5.3 Create a Multiserver Domain with Custom Groups

Use this procedure if you plan to deploy applications that are not homogeneous and require a custom configuration.

If all of the servers in your domain are completely homogeneous, you do not need to create custom server groups. Instead, use the predefined default server groups: the singleton group and the domain group. See Create a Multiserver Domain with Default Groups.

In this procedure, assume you have created three servers: myServer1, myServer2, and myServer3. You want myServer1 to be a member of the selector server group and myServer2 and myServer3 to be members of the strategy server group.

Create a Multiserver Domain with Custom Server Groups

  1. Use the Configuration Wizard to create a standalone-server domain.
  2. Copy and rename the server directory until you have the number of servers that you want under the domain.Update the config.xml file for each member server by adding a cluster child element of the root config element. Within the cluster child element, add a group element to specify the server group.

    Note:

    When you add cluster element child elements, observe the correct element order as Order of Cluster Element Child Elements describes.

    The groups element can include more than one server group name in a case where the server is a member of more than one server group. Separate multiple server group names with commas.

    The groups element is optional. If a server configuration does not have a groups element, then the server is a member of the default server groups (domain and singleton). For more information about the domain and singleton groups, see Server Groups.

    The following examples show the relevant snippets from the config.xml file for myServer1, myServer2, and myServer3. The groups entry for myServer1 puts it in the selector group. The groups entry for myServer2 and myServer3, put them in the strategy group.

    myServer1:

    <config>
      <domain>
         <name>myDomain</name>
      </domain>
      <cluster>
         <server-name>myServer1</server-name>
         <multicast-address>239.255.0.1</multicast-address>
         <enabled>true</enabled>
         <groups>selector</groups>
      </cluster> 
    ...
    </config>
    

    myServer2:

    <config>
      <domain>
         <name>myDomain</name>
      </domain>
      <cluster>
         <server-name>myServer2</server-name>
         <multicast-address>239.255.0.1</multicast-address>
         <enabled>true</enabled>
         <groups>strategy</groups>
      </cluster> 
    ...
    </config>
    

    myServer3:

    <config>
      <domain>
         <name>myDomain</name>
      </domain>
      <cluster>
         <server-name>myServer3</server-name>
         <multicast-address>239.255.0.1</multicast-address>
         <enabled>true</enabled>
         <groups>strategy</groups>
      </cluster> 
    ...
    </config>
    
  3. Optionally, override the default Oracle Coherence clustering configuration on all of the servers in the multiserver domain, if necessary.
  4. Optionally, secure the messages that are shared among all of the servers in the multiserver domain by configuring encryption and digital signatures.
  5. Consider enabling Oracle Stream Analytics Visualizer on a subset of machines within the multiserver domain so that in the event of a failure, you can use Visualizer to troubleshoot from a machine that is still operating.

    Note:

    Enabling Oracle Stream Analytics Visualizer on a given Oracle Stream Analytics Server can impact the performance of the server depending on the Oracle Stream Analytics Visualizer workload.

  6. Start all of the servers in your multiserver domain.

5.4 Configure the Oracle Coherence Cluster

For Oracle Coherence cluster configuration, Oracle Stream Analytics uses the Oracle Coherence tangosol-coherence-override.xml configuration file.

The tangosol-coherence-override.xml file is a global per-server configuration file. It is referred to as operational configuration in the Oracle Coherence documentation. Put this file with the config.xml server configuration file, which is in the Oracle Stream Analytics server config directory.

When you declare that a caching system uses the Oracle Coherence provider, make sure that all of the caches of the caching system map to an Oracle Coherence configuration and not to an Oracle Stream Analytics local configuration. If you do not do this and you have one or more caches mapping to an Oracle Stream Analytics local configuration, Oracle Stream Analytics throws an exception.

Servers in a multiserver domain must be configured with the same multicast address and port number and the same domain name. For example, Oracle Stream Analytics throws an error if you configure servers with the same multicast address and port number, but with different domain names.

The tangosol-coherence-override.xml file supports the following elements: <cluster-config>, <management-config>, and <logging-config>. You cannot override the cluster name because Oracle Stream Analytics always sets the cluster name to the domain name. Choose a unique name for each Oracle Stream Analytics domain to prevent accidental cluster discovery between different domains.

The following sample shows a simple configuration that specifies the time-to-live setting that determines the maximum number of hops a packet can traverse. A hop is measured as a traversal from one network segment to another via a router.

<?xml version='1.0'?>
<coherence xml-override="/tangosol-coherence-override.xml">
  <cluster-config>
    <multicast-listener>
       <time-to-live>3</time-to-live>
    </multicast-listener>
    ...
</coherence>

For detailed information about the tangosol-coherence-override.xml file, see Setting Up a Cluster in Developing Applications with Oracle Coherence.

5.5 Update a Multiserver Domain

You update servers in a multiserver domain the same way that you update the one server in a standalone-server domain: You either launch the Configuration Wizard or edit the XML properties file to use silent mode.

The only difference between updating one server and multiple servers is that in a multiserver domain, you perform the update on each server individually. Using an XML properties file in silent mode might be your best option in a multiserver domain.

When you use the Configuration Wizard to update a server, you can update only the listen port and the JDBC data source configuration. In silent mode, you can add a server and update anything for which there is a data value.

To update the existing configuration of an existing server you provide values for the following data values in the XML properties file:

  • Set CONFIGURATION_OPTION to updateDomain.

  • Set the DOMAIN_NAME and DOMAIN_LOCATION options to the name and location of the existing domain. In our example, the values are myDomain and C:\Oracle\Middleware\my_oep\user_projects\domains, respectively.

  • Set the SERVER_NAME option to the name of the new server you want to add to the existing domain. In our example, this would be myServer2.

  • If this server is running on the same computer as the other servers in the multiserver domain, then be sure that the new server configuration options, such as NETIO_PORT are different than the options for any existing server in the domain. The database options can be the same if you want the new server to connect to the same database as the existing servers.

    If the server is on a different machine than the other servers in the multiserver domain, then the ports do not have to be different.

The following example is an XML properties file that updates an existing server in a multiserver domain.

<?xml version="1.0" encoding="UTF-8"?>
<bea-installer xmlns="http://www.bea.com/plateng/wlevs/config/silent">
  <input-fields>
    <data-value name="CONFIGURATION_OPTION" value="updateDomain" />
    <data-value name="USERNAME" value="wlevs" />
    <data-value name="PASSWORD" value="wlevs" />
    <data-value name="SERVER_NAME" value="myServer2" />
    <data-value name="DOMAIN_NAME" value="myDomain" />
    <data-value name="DOMAIN_LOCATION" value="C:\Oracle\Middleware\my_oep\user_projects\domains" />
    <data-value name="NETIO_PORT" value="9102" />
    <data-value name="KEYSTORE_PASSWORD" value="my_keystore_password" /> 
    <data-value name="PRIVATEKEY_PASSWORD" value="my_privatekey_password" /> 
    <data-value name="DB_URL" value="jdbc:bea:oracle://localhost:1521:XE" />
    <data-value name="DB_USERNAME" value="db_user" />
    <data-value name="DB_PASSWORD" value="db_password" />
  </input-fields>
</bea-installer>

Note:

After you create the servlet and if the two servlets are on the same machine, modify the SSL port to recognize the new servlet.

5.6 Secure the Messages Sent Between Servers

The servers in a multiserver domain update their state by exchanging multiserver-related messages. These messages should be checked for integrity.

You can use a private key to ensure integrity. The private key must be shared by all of the servers within the domain.

When you use the Oracle Coherence clustering implementation, you can secure the messages sent between servers in a multiserver domain.

Secure Messages Sent Between Servers

  1. Stop all servers in your multiserver domain, if they are currently running.
  2. Edit the config.xml file of each server in the multiserver domain by adding the security child element to the cluster element:
    <config>
      <domain>
         <name>myDomain</name>
      </domain>
      <cluster>
         <server-name>myServer1</server-name>
         <multicast-address>239.255.0.1</multicast-address>
         <identity>1</identity>
         <enabled>coherence</enabled>
         <security>encrypt</security>
      </cluster> 
    ...
    </config>
    

    By default the config.xml file is in Oracle/Middleware/my_oep/user_projects/domains/<domainname>/<servername>/config.

    You must specify one of the following values for the security child element:

    • none: No security is configured for the multiserver domain. This is the default value.

    • encrypt: Encrypt multiserver messages.

    Observe the correct order of child elements in the cluster element. See Order of Cluster Element Child Elements.

  3. Edit the Oracle security-config.xml file for each server in the multiserver domain by adding the encryption-service child element to the config root element, as shown.

    By default the security-config.xml files are in Oracle/Middleware/my_oep/user_projects/domains/<domainname>/<servername>/config/

    <config>
      <encryption-service>
         <signature-enabled>true</signature-enabled>
      </encryption-service>
      <css-realm>
       ...
    </config>
    
  4. Ensure that the <domainname/<servername>/.aesinternal.dat file for each server in the multiserver domain is exactly the same by copying the file from one server to the other servers.

    This file is created by the Configuration Wizard when you first created the server. Oracle Stream Analytics uses this file for encrypting messages.

  5. Start one of the servers in your domain.

    See Start and Stop a Server in a Multiserver Domain.

    Because of the encryption-service element that you added to the security-config.xml file, Oracle Stream Analytics creates the *.msasig.dat file in the main server directory. Oracle Stream Analytics uses this file for digitally signing messages.

  6. Stop the server you just started.
  7. Copy the *.msasig.dat file to the other servers.
  8. Perform the following steps on each server in the cluster:
    • Change to the server directory for your domain.

    • Create a keystore coherence-identity.jks containing the boot user using the JDK keytool utility and the following command line (broken here for readability; in practice the full command should be on one line):

      keytool -genkey -v -keystore config/coherence-identity.jks 
      -storepass PASSWORD -alias BOOT-USER -keypass BOOT-USER-PASSWORD 
      -dname CN=BOOT-USER
      

      Where:

      • PASSWORD is the password used to secure the keystore.

      • BOOT-USER is the user name you used to log into the Oracle Stream Analytics server host.

      • BOOT-USER-PASSWORD is the password you used when you logged into the Oracle Stream Analytics server host.

    • Create a permissions.xml file.

      By default the permissions.xml file goes in Oracle/Middleware/my_oep/user_projects/domains/<domainname>/<servername>/config

    • Edit the permissions.xml file to add the following permission for the boot user:

      <permissions>
          <grant>
              <principal>
                  <class>javax.security.auth.x500.X500Principal</class>
                  <name>CN=BOOT-USER</name>
              </principal>
      
              <permission>
                  <target>*</target>
                  <action>all</action>
              </permission>
          </grant>
      </permissions>
      

      Where BOOT-USER is the user name you used to log into the Oracle Stream Analytics server host.

    • Save and close the permissions.xml file.

    • Create a login.config file.

      By default the login.config file goes in Oracle/Middleware/my_oep/ user_projects/domains/<domainname>/<servername>/config/

    • Edit the login.config file to add the following:

      Coherence {
          com.tangosol.security.KeystoreLogin required
              keyStorePath=".${/}config${/}coherence-identity.jks";
      }; 
      
    • Save and close the login.config file.

    • Update the server startup script for your platform, startwlevs.cmd (Windows) or startwlevs.sh (UNIX), by adding the following property to the java command that actually starts the server:

      -Djava.security.auth.login.config=./login.config
      

      For example (put the full command should be on one line):

      "%JAVA_HOME%/bin/java" %DGC% %DEBUG% 
      -Djava.security.auth.login.config=./login.config 
      -Dwlevs.home="%USER_INSTALL_DIR%" -Dbea.hoe="%BEA_HOME%" 
      -jar "%USER_INSTALL_DIR%/bin/wlevs.jar" %1 %2 %3 %4 %5 %6
      
  9. If you plan to use Oracle Stream Analytics Visualizer with the servers in this domain, see Configure SSL in a Multiserver Domain for Visualizer.
  10. Start all servers in your multiserver domain.

5.7 Use Multiserver Domain APIs to Manage Group Membership

In an active-active system, you deploy actively executing applications homogeneously across several servers. There are cases when these homogeneously-deployed applications need to elect a coordinator application to lead.

In this case, events that result from the coordinator application are kept and passed to the next component in the EPN. The results of secondary servers are dropped. If the coordinator fails, then one of the secondary servers must be elected as the new coordinator.

To enable this behavior in an application, the adapter or event bean, usually in the role of an event sink, must implement the following interface:

com.bea.wlevs.ede.api.cluster.GroupMembershipListener

This interface enables the event sink to listen for multiserver domain group membership changes. At runtime, Oracle Stream Analytics calls the onMembershipChange callback method whenever membership changes occur.

The callback method signature follows:

onMembershipChange(Server localIdentity, Configuration groupConfiguration); 

In the implementation of the onMembershipChange callback method, the event sink uses the Server object (localIdentity) to verify if it is the leader. This can be done be comparing localIdentity with the result of Configuration.getCoordinator() run on the second parameter, groupConfiguration. This parameter also allows a server to know what the current members of the group are by executing Configuration.getMembers().

Note:

There is a new API for notification groups. For more information, see Java API Reference for Oracle Stream Analytics.

To only keep events when it is a coordinator, the event sink must get a new Server identity every time membership in the group changes. Group membership changes occur when, for example, another server within the group fails and is no longer the coordinator.

The following interface is for listening to membership changes to the domain as a whole, rather than changes to the server group.

com.bea.wlevs.ede.api.cluster.DomainMembershipListener

In a hot-hot configuration, there is a non-zero delay in failure notification. If you use the notification APIs to implement clustering, you will lose and not process events that occur in the window of time between the server failure and the notification being delivered to the new master server.

See Java API Reference for Oracle Stream Analytics.

5.8 Start and Stop a Server in a Multiserver Domain

To start the servers in a multiserver domain, start each server separately by running its start script. This is the same way you start a server in a standalone server domain.

See Start and Stop a Server in a Standalone-Server Domain for details.

If you have not configured custom groups for the multiserver domain, then all servers are members of the predefined domain group, which contains all of the servers in the multiserver domain, and a singleton group, one for each member server. This means, for example, if there are three servers in the multiserver domain, then there are three singleton groups.

If, however, you have configured custom groups for the multiserver domain, then the servers are members of the groups for which they have been configured and the predefined groups.

Note:

On Windows, do not stop the Oracle Stream Analytics server by clicking the Close button in the command prompt in which you started it. Always stop the Oracle Stream Analytics server using the stopwlevs.cmd script or Ctrl-C.