3 Administrating Oracle CEP Multi-Server Domains

This section contains information on the following subjects:

Section 3.1, "Overview of Oracle CEP Multi-Server Domain Administration"
Section 3.2, "Creating an Oracle CEP Multi-Server Domain Using Oracle CEP Native Clustering"
Section 3.3, "Creating an Oracle CEP Multi-Server Domain Using Oracle Coherence"
Section 3.4, "Updating an Oracle CEP Multi-Server Domain"
Section 3.5, "Securing the Messages Sent Between Servers in a Multi-Server Domain"
Section 3.6, "Using the Multi-Server Domain APIs to Manage Group Membership Changes"
Section 3.7, "Starting and Stopping an Oracle CEP Server in a Multi-Server Domain"
Section 3.8, "Deploying an Application to an Oracle CEP Multi-Server Domain"
Section 3.9, "Next Steps"

3.1 Overview of Oracle CEP Multi-Server Domain Administration

An Oracle CEP multi-server domain (or cluster) is a domain to which you can add one or more servers that become logically connected for the purposes of management, and physically connected using a shared User Datagram Protocol (UDP) multicast address and port. All servers in an Oracle CEP multi-server domain are aware of all other servers in the domain and any one server can be used as an access point for making changes to the deployments in the domain.

Management of the multi-server infrastructure is done at the domain level. Thus server failure, start, or restart is detected by every member of the multi-server domain. Each member of the multi-server domain has a consistent, agreed notion of domain membership enforced by the multi-server infrastructure.

To properly configure servers in a multi-server domain, you must configure them with the same multicast address and port and the same domain name. It is an error to configure servers using the same multicast address and port and port but different domain names.

Applications deployed to the default group in a multi-server domain are deployed homogenously to all servers of that domain, therefore all servers must have the appropriate configuration resources required by the application.

Oracle CEP supports the following clustering systems:

Oracle CEP native clustering: provides a native clustering implementation based on TOTEM.

For more information, see Section 3.2, "Creating an Oracle CEP Multi-Server Domain Using Oracle CEP Native Clustering".
Oracle Coherence: provides replicated and distributed (partitioned) data management services on top of a reliable, highly scalable peer-to-peer clustering protocol. Oracle Coherence has no single points of failure; it automatically and transparently fails over and redistributes its clustered data management services when a server becomes inoperative or is disconnected from the network. When a new server is added, or when a failed server is restarted, it automatically joins the cluster and Oracle Coherence fails back services to it, transparently redistributing the cluster load.

Note:
Before you can use Oracle CEP with Oracle Coherence, you must obtain a valid Oracle Coherence license such as a license for Coherence Enterprise Edition, Coherence Grid Edition, or Oracle WebLogic Application Grid. For more information on Oracle Coherence, see http://www.oracle.com/technology/products/coherence/index.html.

For more information, see Section 3.3, "Creating an Oracle CEP Multi-Server Domain Using Oracle Coherence".

This section describes:

Section 3.1.1, "Groups"
Section 3.1.2, "Multi-Server Notifications and Messaging"
Section 3.1.3, "Multi-Server Domain Directory Structure"
Section 3.1.4, "Order of cluster Element Child Elements"

For more information, see:

3.1.1 Groups

In order to support the deployment to, and management of, the multi-server domain at a finer grained-level than the domain, Oracle CEP introduces the concept of groups. A group is a set of one or more servers with a unique name within the domain. In an Oracle CEP domain, an arbitrary number of groups may exist with a configurable group membership. A server may be a member of more than one group, although typically this information is transparent to the user.

When you deploy an application to a multi-server domain, you deploy it to a particular group. Applications deployed to any group must have a unique name across the domain.

Applications deployed to a group in a multi-server domain are deployed homogenously to all servers of that group, therefore all servers must have the appropriate configuration resources required by the application.

An application that is deployed to a server can be uninstalled from another server under the same domain.

The following pre-defined groups always exist:

Singleton server group—Consists of only the local server. This means that the membership of this group depends on the server from which it is accessed. This group can be used to pin deployments to a single server.
Domain (or default or all) group—Contains all live members of the domain. Its membership is automatically managed and cannot be changed by the user.

The domain name is determined by the Oracle CEP server config.xml file domain element. For example, the domain is named mydomain if your config.xml file is like this:
```
<domain>
    <name>mydomain</name>
</domain>
```
The default name is WLEventServerDomain.

Alternatively, you can create a custom group (see Section 3.1.1.1, "Custom Groups").

3.1.1.1 Custom Groups

There are cases where the application logic cannot simply be replicated across a homogenous set of servers in a multi-server domain. Examples of these types of applications are those that must determine the best price provided by different pricing engines, or applications that send an alert when a position crosses a threshold. In these cases, the application is not idempotent; it must calculate only once or send a single event. In other cases, the application has a singleton nature, such as a monitoring application, the HTTP pub-sub server, and so on.

As a more complex example, consider a domain that has two applications: the strategies application uses several strategies for calculating different prices for some derivative and then feeds its results to a selector application. The selector application then selects the best price amongst the different options provided by the strategies' application results. The strategies application can be replicated to achieve fault-tolerance. However, the selector application must be able to keep state so as to determine the best price; for this reason, the selector application cannot be replicated in a hot/hot fashion.

If a domain must support servers that are not completely homogeneous, you configure this by creating custom groups.

Applications deployed to a custom group in a multi-server domain are deployed homogenously to all servers of that group, therefore all servers must have the appropriate configuration resources required by the application.

For more information, see Section 3.3.2, "How to Create an Oracle CEP Multi-Server Domain With Custom Groups Using Oracle Coherence".

3.1.2 Multi-Server Notifications and Messaging

In order to provide high availability (HA)-like capabilities to adapter and event bean implementations, Oracle CEP provides a number of notification and messaging APIs at both the group- and domain-level. Using these APIs, you can configure a server to receive notification when its group or domain membership changes, either because an administrator deliberately changed it or due to a server failure. Similarly you can use these APIs to send messages to both individual groups and to the domain.

You can configure Oracle CEP servers in a multi-server domain to communicate securely. For more information, see Section 3.5, "Securing the Messages Sent Between Servers in a Multi-Server Domain".

3.1.3 Multi-Server Domain Directory Structure

Servers in an Oracle CEP domain store their files in a single directory. By convention, the directories of the servers in a multi-server domain are sub-directories of the domain directory. Additionally, the name of the servers and domain correspond to the name of the server directories and domain directory, respectively. This is by convention only, and not required, although Oracle recommends you set up your domains this way for simplicity and consistency. If the servers of the multi-server domain are located on different computers, you can replicate the directory structure on both computers, also for simplicity and consistency.

Figure 3-1 shows a multi-server domain directory with three servers.

Figure 3-1 Multi-Server Domain Directory Structure

Description of "Figure 3-1 Multi-Server Domain Directory Structure"

The myServer1 configuration file snippet shows how the domain directory and domain object are configured with the same name, as well as the server directory and server name. The domain directory is located in the ORACLE_CEP_HOME/user_projects/domains directory, which is the default location for Oracle CEP domains.

3.1.4 Order of cluster Element Child Elements

The order of cluster element child elements in the config.xml file is important; if you include elements in the incorrect order you may encounter an error. The following list describes the order in which you should list the child elements:

server-name
server-host-name: Specifies the host address/IP used for point-to-point HTTP multi-server communication. Default value is localhost.

If all the Oracle CEP servers in your multi-server domain are on the same host, this element is optional.

If one or more Oracle CEP servers in your multi-server domain are on different hosts, this element is mandatory if you plan to manage the multi-server domain using the Oracle CEP Visualizer.
multicast-address
multicast-port: Optional. Specifies the port used for multicast traffic. Default value is 9001.
identity: Mandatory only for Oracle CEP native clustering; not used for Oracle Coherence.
enabled
security
groups
operation-timeout—Optional. Specifies, in milliseconds, the timeout for point-to-point HTTP multi-server requests. Default value is 30000.

For the complete XSD Schema of the config.xml file, including a description of the cluster element, see "Server Configuration XSD Schema: wlevs_server_config.xsd" in the Oracle CEP IDE Developer's Guide for Eclipse.

3.2 Creating an Oracle CEP Multi-Server Domain Using Oracle CEP Native Clustering

This section describes how to create and configure a multi-server domain from two or more Oracle CEP servers using Oracle CEP native clustering.

3.2.1 How to Create an Oracle CEP Multi-Server Domain With Default Groups Using Oracle CEP Native Clustering

This procedure describes how to create a multi-server domain that uses only the two predefined groups: the singleton group and domain group. In a domain that uses default groups, all servers must be completely homogenous.

If a domain must support servers that are not completely homogeneous, you configure this by creating custom groups. See Section 3.2.2, "How to Create an Oracle CEP Multi-Server Domain With Custom Groups Using Oracle CEP Native Clustering".

For more information about default and custom groups, see Section 3.1.1, "Groups".

Note:

In this section it is assumed that you have already created a domain that contains a single server and that you want to add additional servers to the domain to make it a multi-server domain. See Chapter 2, "Administrating Oracle CEP Standalone-Server Domains" for details on creating a domain.

To create an Oracle CEP multi-server domain with default groups using Oracle CEP Native Clustering:

Create a domain that contains a single, initial server.

See Chapter 2, "Administrating Oracle CEP Standalone-Server Domains".
Add one or more servers to the domain using the Configuration Wizard.

Note:
Even though the Configuration WIzard does not support adding new servers to a multi-server domain, one can use the Configuration Wizard to generate a new stand-alone server, and then manually update its configuration to join a multi-server domain.

See Section 3.4, "Updating an Oracle CEP Multi-Server Domain."
Configure all the servers in the multi-server domain by manually editing their config.xml files and adding a cluster element with specific information.

To configure the servers in a multi-server domain using default groups, update the config.xml file for each member server by adding a cluster child element of the root config element as Example 3-1 shows.

Include the following child elements of cluster:
- server-name: The server-name child element of cluster specifies a unique name for the server. Oracle CEP Visualizer uses the value of this element when it displays the server in its console. The default value if the element is not set is Server-identity where identity is the value of the identity element.
- server-host-name: Specifies the host address/IP used for point-to-point HTTP multi-server communication. Default value is localhost.
  
  If all the Oracle CEP servers in your multi-server domain are on the same host, this element is optional.
  
  If one or more Oracle CEP servers in your multi-server domain on on different hosts, this element is mandatory if you plan to manage the multi-server domain using the Oracle CEP Visualizer.
- multicast-address: The multicast-address element is required unless all servers of the multi-server domain are hosted on the same computer; in that case you can omit the multicast-address element and Oracle CEP automatically assigns a multicast address to the multi-server domain based on the computer's IP address.
  
  If, however, the servers are hosted on different computers, then you must provide an appropriate domain-local address. Oracle recommends you use an address of the form 239.255.X.X, which is what the auto-assigned multicast address is based on.
- identity: The identity element identifies the server's identity and must be an integer between 1 and INT_MAX. Oracle CEP numerically compares the server identities during multi-server operations; the server with the lowest identity becomes the domain coordinator. Be sure that each server in the multi-server domain has a different identity; if servers have the same identity, the results of multi-server operations are unpredictable.
- enabled: By default the clustering of the servers in a multi-server domain is enabled for Oracle Coherence, so to enable Oracle CEP native clustering use <enabled>evs4j</enabled>.
Note:
When adding cluster element child elements, observe the correct element order as Section 3.1.4, "Order of cluster Element Child Elements" describes.
Example 3-1 myServer1 config.xml File
```
<config>
  <domain>
        <name>myDomain</name>
  </domain>
  <cluster>
      <server-name>myServer1</server-name>
      <multicast-address>239.255.0.1</multicast-address>
      <identity>1</identity> 
      <enabled>evs4j</enabled>
  </cluster> 
...
</config>
```
In Example 3-1, the server is part of a domain called myDomain.

For each server of the multi-server domain, the multicast-address elements must contain the same value. The identity and server-name elements, however, must be different for each server in the multi-server domain.

Example 3-2 shows the config.xml file of a second server, called myServer2, in the myDomain multi-server domain; note that this server's identity is 2.
Example 3-2 myServer2 config.xml File
```
<config>
  <domain>
        <name>myDomain</name>
  </domain>
  <cluster>
      <server-name>myServer2</server-name>
      <multicast-address>239.255.0.1</multicast-address>
      <identity>2</identity>
      <enabled>evs4j</enabled>
  </cluster> 
...
</config>
```
See Section 3.1.4, "Order of cluster Element Child Elements" for a description of additional multi-server-related configuration elements and the required order of child elements.
Optionally, secure the messages that are shared between the servers in a domain by configuring encryption and digital signatures.

See Section 3.5.1, "How to Secure the Messages Sent Between Servers in a Multi-Server Domain Using Oracle CEP Native Clustering".
To avoid a single point of failure, consider enabling Oracle CEP Visualizer on a small subset of n machines in the domain.

See "How to Start Oracle CEP Visualizer in a Multi-Server Domain" in the Oracle CEP Visualizer User's Guide.

Note:
Enabling Oracle CEP Visualizer on a given Oracle CEP Server may impact the performance of the server depending on the Oracle CEP Visualizer workload.
Start all servers in your multi-server domain.

See Section 3.7, "Starting and Stopping an Oracle CEP Server in a Multi-Server Domain".

3.2.2 How to Create an Oracle CEP Multi-Server Domain With Custom Groups Using Oracle CEP Native Clustering

This procedure describes how to create a multi-server domain that uses custom groups to accommodate servers which are not completely homogenous.

If all the servers in your domain are completely homogeneous, you do not need to create custom groups. Instead, you can use the predefined groups: the singleton group and domain group. See Section 3.2.1, "How to Create an Oracle CEP Multi-Server Domain With Default Groups Using Oracle CEP Native Clustering".

For more information about default and custom groups, see Section 3.1.1, "Groups".

Note:

In this procedure, assume you have created three servers: myServer1, myServer2, and myServer3. You want myServer1 to be a member of the selector group and myServer2 and myServer3 to be members of the strategy group.

To create an Oracle CEP multi-server domain with custom groups using Oracle CEP Native Clustering:

Create a domain that contains a single, initial server.

See Chapter 2, "Administrating Oracle CEP Standalone-Server Domains".
Add one or more servers to the domain using the Configuration Wizard.

Note:
Even though the Configuration WIzard does not support adding new servers to a multi-server domain, one can use the Configuration Wizard to generate a new stand-alone server, and then manually update its configuration to join a multi-server domain.

See Section 3.4, "Updating an Oracle CEP Multi-Server Domain."
Configure all the servers in the multi-server domain by manually editing their config.xml files and adding a cluster element with specific information.

To configure the servers in a multi-server domain using custom groups, update the config.xml file for each member server by adding (if one does not already exist) a groups child element of cluster and specifying the name of the group as the value of the groups element.

Note:
When adding cluster element child elements, observe the correct element order as Section 3.1.4, "Order of cluster Element Child Elements" describes.

The groups element can include more than one group name in the case that the server is a member of more than one group; separate multiple group names using commas.

The groups element is optional; if a server configuration does not include one, then the server is a member of the default groups (domain and singleton). For more information about the domain and singleton groups, see Section 3.1.1, "Groups".

Example 3-3, Example 3-4, and Example 3-5 show the relevant snippets of the config.xml file for each server.
Example 3-3 Server Configuration File config.xml for myServer1
```
<config>
  <domain>
        <name>myDomain</name>
  </domain>
  <cluster>
      <server-name>myServer1</server-name>
      <multicast-address>239.255.0.1</multicast-address>
      <identity>1</identity> 
      <enabled>evs4j</enabled>
      <groups>selector</groups>
  </cluster> 
...
</config>
```
Example 3-4 Server Configuration File config.xml for myServer2
```
<config>
  <domain>
        <name>myDomain</name>
  </domain>
  <cluster>
      <server-name>myServer2</server-name>
      <multicast-address>239.255.0.1</multicast-address>
      <identity>2</identity> 
      <enabled>evs4j</enabled>
      <groups>strategy</groups>
  </cluster> 
...
</config>
```
Example 3-5 Server Configuration File config.xml for myServer3
```
<config>
  <domain>
        <name>myDomain</name>
  </domain>
  <cluster>
      <server-name>myServer3</server-name>
      <multicast-address>239.255.0.1</multicast-address>
      <identity>3</identity> 
      <enabled>evs4j</enabled>
      <groups>strategy</groups>
  </cluster> 
...
</config>
```
Optionally, secure the messages that are shared between the servers in a domain by configuring encryption and digital signatures.

See Section 3.5.1, "How to Secure the Messages Sent Between Servers in a Multi-Server Domain Using Oracle CEP Native Clustering".
To avoid a single point of failure, consider enabling Oracle CEP Visualizer on a small subset of n machines in the domain.

See "How to Start Oracle CEP Visualizer in a Multi-Server Domain" in the Oracle CEP Visualizer User's Guide.

Note:
Enabling Oracle CEP Visualizer on a given Oracle CEP Server may impact the performance of the server depending on the Oracle CEP Visualizer workload.
Start all servers in your multi-server domain.

See Section 3.7, "Starting and Stopping an Oracle CEP Server in a Multi-Server Domain".

3.3 Creating an Oracle CEP Multi-Server Domain Using Oracle Coherence

This section describes how to create and configure a multi-server domain from two or more Oracle CEP servers using Oracle Coherence.

Note:

Before you can use Oracle CEP with Oracle Coherence, you must obtain a valid Oracle Coherence license such as a license for Coherence Enterprise Edition, Coherence Grid Edition, or Oracle WebLogic Application Grid. For more information on Oracle Coherence, see http://www.oracle.com/technology/products/coherence/index.html.

3.3.1 How to Create an Oracle CEP Multi-Server Domain With Default Groups Using Oracle Coherence

If a domain must support servers that are not completely homogeneous, you configure this by creating custom groups. See Section 3.3.2, "How to Create an Oracle CEP Multi-Server Domain With Custom Groups Using Oracle Coherence".

For more information about default and custom groups, see Section 3.1.1, "Groups".

Note:

To create an Oracle CEP multi-server domain with default groups using Coherence:

Create a domain that contains a single, initial server.

See Chapter 2, "Administrating Oracle CEP Standalone-Server Domains".
Add one or more servers to the domain using the Configuration Wizard.

Note:
Even though the Configuration WIzard does not support adding new servers to a multi-server domain, one can use the Configuration Wizard to generate a new stand-alone server, and then manually update its configuration to join a multi-server domain.

See Section 3.4, "Updating an Oracle CEP Multi-Server Domain."
Configure all the servers in the multi-server domain by manually editing their config.xml files and adding a cluster element with specific information.

To configure the servers in a multi-server domain using default groups, update the config.xml file for each member server by adding a cluster child element of the root config element as Example 3-6 shows.

Include the following child elements of cluster:
- server-name: The server-name child element of cluster specifies a unique name for the server. Oracle CEP Visualizer uses the value of this element when it displays the server in its console.
- server-host-name: Specifies the host address/IP used for point-to-point HTTP multi-server communication. Default value is localhost.
  
  If all the Oracle CEP servers in your multi-server domain are on the same host, this element is optional.
  
  If one or more Oracle CEP servers in your multi-server domain on on different hosts, this element is mandatory if you plan to manage the multi-server domain using the Oracle CEP Visualizer.
- multicast-address: The multicast-address element is required unless all servers of the multi-server domain are hosted on the same computer; in that case you can omit the multicast-address element and Oracle CEP automatically assigns a multicast address to the multi-server domain based on the computer's IP address.
  
  If, however, the servers are hosted on different computers, then you must provide an appropriate domain-local address. Oracle recommends you use an address of the form 239.255.X.X, which is what the auto-assigned multicast address is based on.
  
  Using Oracle Coherence, you can specify a unicast address here and Oracle Coherence will use WKA (Well Known Addressing).
- enabled: By default the clustering of the servers in a multi-server domain is enabled for Oracle Coherence, so the element <enabled>true</enabled> is optional.
  
  Note:
  Before you can legally use Oracle CEP with Oracle Coherence, you must obtain a valid Coherence license such as a license for Coherence Enterprise Edition, Coherence Grid Edition, or Oracle WebLogic Application Grid. For more information on Oracle Coherence, see http://www.oracle.com/technology/products/coherence/index.html.
Example 3-6 myServer1 config.xml File
```
<config>
  <domain>
        <name>myDomain</name>
  </domain>
  <cluster>
      <server-name>myServer1</server-name>
      <multicast-address>239.255.0.1</multicast-address>
      <enabled>true</enabled>
  </cluster> 
...
</config>
```
In Example 3-6, the server is part of a domain called myDomain.

For each server of the multi-server domain, the multicast-address elements must contain the same value. The server-name element, however, must be different for each server in the multi-server domain.

Example 3-7 shows the config.xml file of a second server, called myServer2, in the myDomain multi-server domain.
Example 3-7 myServer2 config.xml File
```
<config>
  <domain>
        <name>myDomain</name>
  </domain>
  <cluster>
      <server-name>myServer2</server-name>
      <multicast-address>239.255.0.1</multicast-address>
      <enabled>true</enabled>
  </cluster> 
...
</config>
```
See Section 3.1.4, "Order of cluster Element Child Elements" for a description of additional multi-server-related configuration elements and the required order of child elements.
Optionally, override the default Oracle Coherence clustering configuration, if necessary.

See Section 3.3.3, "Configuring the Oracle Coherence Cluster".
Optionally, secure the messages that are shared between the servers in a domain by configuring encryption and digital signatures.

See Section 3.5.2, "How to Secure the Messages Sent Between Servers in a Multi-Server Domain Using Oracle Coherence".
To avoid a single point of failure, consider enabling Oracle CEP Visualizer on a small subset of n machines in the domain.

See "How to Start Oracle CEP Visualizer in a Multi-Server Domain" in the Oracle CEP Visualizer User's Guide.

Note:
Enabling Oracle CEP Visualizer on a given Oracle CEP Server may impact the performance of the server depending on the Oracle CEP Visualizer workload.
Start all servers in your multi-server domain.

See Section 3.7, "Starting and Stopping an Oracle CEP Server in a Multi-Server Domain".

3.3.2 How to Create an Oracle CEP Multi-Server Domain With Custom Groups Using Oracle Coherence

This procedure describes how to create a multi-server domain using Oracle Coherence that uses custom groups to accommodate servers which are not completely homogenous.

Note:

If all the servers in your domain are completely homogeneous, you do not need to create custom groups. Instead, you can use the predefined groups: the singleton group and domain group. See Section 3.3.1, "How to Create an Oracle CEP Multi-Server Domain With Default Groups Using Oracle Coherence".

For more information about default and custom groups, see Section 3.1.1, "Groups".

Note:

To create an Oracle CEP multi-server domain with custom groups using Oracle Coherence:

Create a domain that contains a single, initial server.

See Chapter 2, "Administrating Oracle CEP Standalone-Server Domains".
Add one or more servers to the domain using the Configuration Wizard.

Note:
Even though the Configuration WIzard does not support adding new servers to a multi-server domain, one can use the Configuration Wizard to generate a new stand-alone server, and then manually update its configuration to join a multi-server domain.

See Section 3.4, "Updating an Oracle CEP Multi-Server Domain."
Configure all the servers in the multi-server domain by manually editing their config.xml files and adding a cluster element with specific information.

To configure the servers in a multi-server domain using custom groups, update the config.xml file for each member server by adding (if one does not already exist) a groups child element of cluster and specifying the name of the group as the value of the groups element.

Note:
When adding cluster element child elements, observe the correct element order as Section 3.1.4, "Order of cluster Element Child Elements" describes.

The groups element can include more than one group name in the case that the server is a member of more than one group; separate multiple group names using commas.

The groups element is optional; if a server configuration does not include one, then the server is a member of the default groups (domain and singleton). For more information about the domain and singleton groups, see Section 3.1.1, "Groups".

Example 3-8, Example 3-9, and Example 3-10 show the relevant snippets of the config.xml file for each server.
Example 3-8 Server Configuration File config.xml for myServer1
```
<config>
  <domain>
        <name>myDomain</name>
  </domain>
  <cluster>
      <server-name>myServer1</server-name>
      <multicast-address>239.255.0.1</multicast-address>
      <enabled>true</enabled>
      <groups>selector</groups>
  </cluster> 
...
</config>
```
Example 3-9 Server Configuration File config.xml for myServer2
```
<config>
  <domain>
        <name>myDomain</name>
  </domain>
  <cluster>
      <server-name>myServer2</server-name>
      <multicast-address>239.255.0.1</multicast-address>
      <enabled>true</enabled>
      <groups>strategy</groups>
  </cluster> 
...
</config>
```
Example 3-10 Server Configuration File config.xml for myServer3
```
<config>
  <domain>
        <name>myDomain</name>
  </domain>
  <cluster>
      <server-name>myServer3</server-name>
      <multicast-address>239.255.0.1</multicast-address>
      <enabled>true</enabled>
      <groups>strategy</groups>
  </cluster> 
...
</config>
```
Optionally, override the default Oracle Coherence clustering configuration, if necessary.

See Section 3.3.3, "Configuring the Oracle Coherence Cluster".
Optionally, secure the messages that are shared between the servers in a domain by configuring encryption and digital signatures.

See Section 3.5.2, "How to Secure the Messages Sent Between Servers in a Multi-Server Domain Using Oracle Coherence".
To avoid a single point of failure, consider enabling Oracle CEP Visualizer on a small subset of n machines in the domain.

See "How to Start Oracle CEP Visualizer in a Multi-Server Domain" in the Oracle CEP Visualizer User's Guide.

Note:
Enabling Oracle CEP Visualizer on a given Oracle CEP Server may impact the performance of the server depending on the Oracle CEP Visualizer workload.
Start all servers in your multi-server domain.

See Section 3.7, "Starting and Stopping an Oracle CEP Server in a Multi-Server Domain".

3.3.3 Configuring the Oracle Coherence Cluster

Oracle CEP leverages the native configuration provided by Oracle Coherence. You do this by packaging the following Oracle Coherence configuration file, with the indicated names, in the application bundle that uses the Oracle Coherence cache:

tangosol-coherence-override.xml—Oracle Coherence cluster configuration. See Section 3.3.3.1, "The tangosol-coherence-override.xml File" for information about this file as well as an example.

When assembling your application, consider the following:

tangosol-coherence-override.xml is a global per-server file (referred to as "operational configuration" in the Oracle Coherence documentation); put this file in the Oracle CEP server config directory.

When you declare that a caching system uses the Oracle Coherence provider, be sure that all of the caches of this caching system also map to an Oracle Coherence configuration and not an Oracle CEP local configuration, or Oracle CEP throws an exception.

3.3.3.1 The tangosol-coherence-override.xml File

The tangosol-coherence-override.xml file configures Oracle Coherence clustering.

The following sample shows a simple configuration. See the explanation after the sample for information about the sections in bold.

<?xml version='1.0'?>
<coherence xml-override="/tangosol-coherence-override.xml">
  <cluster-config>
    <member-identity>
      <cluster-name>com.bea.wlevs.example.provider</cluster-name>
    </member-identity>
...
</coherence>

This configuration file is fairly standard. The main thing to note is that you should specify a cluster-name element to prevent Oracle Coherence from attempting to join existing Oracle Coherence clusters when Oracle CEP starts up; this can cause problems and sometimes even prevent Oracle CEP from starting.

Refer to your Oracle Coherence documentation (see http://www.oracle.com/technology/products/coherence/index.html) for detailed information about the tangosol-coherence-override.xml file.

3.4 Updating an Oracle CEP Multi-Server Domain

Use the Configuration Wizard to add a new server to an existing standalone server domain so as to later convert it into a multi-server domain. The procedure is similar to creating a new domain, so be sure you read Section 2.2, "Creating an Oracle CEP Standalone-Server Domain" before continuing with this section.

3.4.1 How to Update an Oracle CEP Multi-Server Domain Using the Configuration Wizard in Graphical Mode

The following procedure shows how to invoke and use the Configuration Wizard in graphical mode by executing the relevant command script for both Windows or Unix.

Note:

On Windows, you can also invoke the Configuration Wizard using the Start menu:

Start > All Programs > Oracle Complex Event Processing 10gR3 > Tools > Configuration Wizard

For clarity, it is assumed that:

You have already created a new domain and its domain directory is C:\oracle_cep\user_projects\domains\mydomain.
The domain includes a single server called defaultserver and the server files are located in the C:\oracle_cep\user_projects\domains\myDomain\myServer1 directory.
You want to create a new server in the existing mydomain domain called myServer2.

To update an Oracle CEP multi-server domain using the Configuration Wizard in graphical mode:

Open a command window and set your environment as described in "Setting Your Development Environment" in the Oracle CEP Getting Started.
Change to the ORACLE_CEP_HOME/ocep_11.1/common/bin directory, where ORACLE_CEP_HOME refers to the main Oracle CEP installation directory, such as /oracle_cep:
```
prompt> cd /oracle_cep/ocep_11.1/common/bin
```
Invoke the config.cmd (Windows) or config.sh (UNIX) command to invoke the wizard:
```
prompt> config.sh
```
After the Configuration Wizard has finished loading, you will see a standard Oracle Welcome window.

Note:
The Oracle CEP Configuration Wizard is self-explanatory; however, if you want more information about using the tool, continue reading this procedure.
Click Next.
In the Choose Create or Update Domain window, choose Create a New Oracle CEP Domain.
Click Next.
Enter the name and password of the administrator user for the new server you are adding to the domain.'
Click Next.
Enter basic configuration information about the new server in the domain. If the new server is located on the same computer as any other servers in the domain, be sure the following information is different from that of the other servers to prevent conflicts when starting all servers. In particular:
- Enter the name of the new server. This name will also be used as the name of the directory that contains the new server's files. Following our example, this value is myServer1.
- The listen port for Oracle CEP itself.
Click Next.
Enter and confirm the password for the Oracle CEP identity keystore. By default, the password for the certificate private key will be the same as the identity keystore; if you want it to be different, uncheck Use Keystore Password and enter the private key password.
Click Next.
In the Configuration Options window, choose:
1. Yes if you want to change the default JDBC data source configuration,
2. No to accept the defaults.
Note:
When you deploy an application to a group in the domain, Oracle CEP replicates the application to each server that is a member of the group. This means that if your application uses a datasource, and you have configured the datasource differently for each server in the domain, then the storage and retrieval of data to and from this data source will differ depending on the server on which the application is running.
Click Next.
If you chose to change the default JDBC data source configuration, enter the information in the Configure Database Properties window.

In the top section, enter the name of the datasource. Then select the database type (Oracle or Microsoft SQL Server) and corresponding drivers; you can also browse to new drivers using the Browse/Append button.

In the lower section, enter the details about the database to which this data source connects, such as its name, the name of the computer that hosts the database server, the port, and the name and password of the user that connects to the database. The JDBC connection URL is automatically generated for you based on this information.
Click Next.
In the Configure Server window, enter the name of the existing domain and the full pathname of its location. Following our example, you would enter myDomain for the domain name and C:\oracle_cep\user_projects\domains for the domain location.
Click Create.
If the creation of the new server succeeded, you will see a message similar to the following in the Creating Domain window:
```
Domain created successfully!
Domain location: C:\oracle_cep\user_projects\domains\myDomain
```
Click Done.

3.5 Securing the Messages Sent Between Servers in a Multi-Server Domain

The servers in a multi-server domain update their state by exchanging multi-server-related messages. It is important that these messages be at least checked for integrity. A private key can be used to achieve integrity. This key must be shared by all servers in the domain.

This section describes:

Section 3.5.1, "How to Secure the Messages Sent Between Servers in a Multi-Server Domain Using Oracle CEP Native Clustering"
Section 3.5.2, "How to Secure the Messages Sent Between Servers in a Multi-Server Domain Using Oracle Coherence"

3.5.1 How to Secure the Messages Sent Between Servers in a Multi-Server Domain Using Oracle CEP Native Clustering

You can secure the messages sent between servers in a multi-server domain using Oracle CEP native clustering.

To secure the messages sent between servers in an Oracle CEP multi-server domain using Oracle CEP Native Clustering:

Stop all servers in your multi-server domain, if they are currently running.

See Section 3.7, "Starting and Stopping an Oracle CEP Server in a Multi-Server Domain".
Edit the config.xml file of each server in the multi-server domain by adding the security child element to the cluster element, as Example 3-11 shows.
Example 3-11 The cluster Element security Child Element
```
<config>
  <domain>
        <name>myDomain</name>
  </domain>
  <cluster>
      <server-name>myServer1</server-name>
      <multicast-address>239.255.0.1</multicast-address>
      <identity>1</identity> 
      <enabled>coherence</enabled>
      <security>encrypt</security>
  </cluster> 
...
</config>
```
The config.xml file is located in the DOMAIN_DIR/servername/config directory of each server, where DOMAIN_DIR refers to the domain directory and servername refers to the name of your server, such as d:\oracle_cep_home\user_projects\domains\mydomain\myserver1\config.

You must specify one of the following values for the security child element:
- none—Default value. Specifies that no security is configured for the multi-server domain.
- encrypt—Specifies that multi-server messages should be encrypted.
Observe the correct order of child elements in the cluster element. See Section 3.1.4, "Order of cluster Element Child Elements".
Edit the DOMAIN_DIR/servername/config/security-config.xml file of each server in the multi-server domain by adding the encryption-service child element of the config root element, as Example 3-12 shows.
Example 3-12 The security-config.xml File encryption-service Element
```
<config>
  <encryption-service>
     <signature-enabled>true</signature-enabled>
  </encryption-service>
  <css-realm>
   ...
</config>
```
Ensure that the DOMAIN_DIR/servername/.msainternal.dat file for each server in the multi-server domain is exactly the same by copying the file from one server to the other servers.

This file is automatically created by the Configuration Wizard when you first created the server; Oracle CEP uses this file for encrypting messages.

For example, assume all the servers in your domain are located in the d:\oracle_cep\user_projects\domains\mydomain directory, and that the domain has three servers: server1, server2, and server3. To ensure they all have the same .msainternal.dat file, copy the one from server1 to the other servers:
```
prompt> cd d:\oracle_cep\user_projects\domains\mydomain\server1
prompt> cp .msainternal.dat ..\server2
prompt> cp .msainternal.dat ..\server3
```
Start one of the servers in your domain.

See Section 3.7, "Starting and Stopping an Oracle CEP Server in a Multi-Server Domain".

Because of the encryption-service element that you added to the security-config.xml file in step 3, Oracle CEP automatically creates the .msasig.dat file in the main server directory. Oracle CEP uses this file for digitally signing messages.
Stop the server you just started.

See Section 3.7, "Starting and Stopping an Oracle CEP Server in a Multi-Server Domain".

Copy the .msasig.dat file you created in step 5 to the other servers.

For example:

prompt> cd d:\oracle_cep\user_projects\domains\mydomain\server1
prompt> cp .msasig.dat ..\server2
prompt> cp .msasig.dat ..\server3

If you plan to use Oracle CEP Visualizer with the servers in this domain, see Section 4.5.3, "How to Configure SSL in a Multi-Server Domain for Oracle CEP Visualizer".
Start all servers in your multi-server domain.

See Section 3.7, "Starting and Stopping an Oracle CEP Server in a Multi-Server Domain".

3.5.2 How to Secure the Messages Sent Between Servers in a Multi-Server Domain Using Oracle Coherence

You can secure the messages sent between servers in a multi-server domain using the Oracle Coherence clustering implementation.

To secure the messages sent between servers in a multi-server domain using Oracle Coherence:

Stop all servers in your multi-server domain, if they are currently running.

See Section 3.7, "Starting and Stopping an Oracle CEP Server in a Multi-Server Domain".
Edit the config.xml file of each server in the multi-server domain by adding the security child element to the cluster element, as Example 3-11 shows.
Example 3-13 The cluster Element security Child Element
```
<config>
  <domain>
        <name>myDomain</name>
  </domain>
  <cluster>
      <server-name>myServer1</server-name>
      <multicast-address>239.255.0.1</multicast-address>
      <identity>1</identity> 
      <enabled>coherence</enabled>
      <security>encrypt</security>
  </cluster> 
...
</config>
```
The config.xml file is located in the DOMAIN_DIR/servername/config directory of each server, where DOMAIN_DIR refers to the domain directory and servername refers to the name of your server, such as d:\oracle_cep_home\user_projects\domains\mydomain\myserver1\config.

You must specify one of the following values for the security child element:
- none—Default value. Specifies that no security is configured for the multi-server domain.
- encrypt—Specifies that multi-server messages should be encrypted.
Observe the correct order of child elements in the cluster element. See Section 3.1.4, "Order of cluster Element Child Elements".
Edit the DOMAIN_DIR/servername/config/security-config.xml file of each server in the multi-server domain by adding the encryption-service child element of the config root element, as Example 3-12 shows.
Example 3-14 The security-config.xml File encryption-service Element
```
<config>
  <encryption-service>
     <signature-enabled>true</signature-enabled>
  </encryption-service>
  <css-realm>
   ...
</config>
```
Ensure that the DOMAIN_DIR/servername/.msainternal.dat file for each server in the multi-server domain is exactly the same by copying the file from one server to the other servers.

This file is automatically created by the Configuration Wizard when you first created the server; Oracle CEP uses this file for encrypting messages.

For example, assume all the servers in your domain are located in the d:\oracle_cep\user_projects\domains\mydomain directory, and that the domain has three servers: server1, server2, and server3. To ensure they all have the same .msainternal.dat file, copy the one from server1 to the other servers:
```
prompt> cd d:\oracle_cep\user_projects\domains\mydomain\server1
prompt> cp .msainternal.dat ..\server2
prompt> cp .msainternal.dat ..\server3
```
Start one of the servers in your domain.

See Section 3.7, "Starting and Stopping an Oracle CEP Server in a Multi-Server Domain".

Because of the encryption-service element that you added to the security-config.xml file in step 3, Oracle CEP automatically creates the .msasig.dat file in the main server directory. Oracle CEP uses this file for digitally signing messages.
Stop the server you just started.

See Section 3.7, "Starting and Stopping an Oracle CEP Server in a Multi-Server Domain".

Copy the .msasig.dat file you created in step 5 to the other servers.

For example:

prompt> cd d:\oracle_cep\user_projects\domains\mydomain\server1
prompt> cp .msasig.dat ..\server2
prompt> cp .msasig.dat ..\server3

Perform the following steps on each server in the cluster:
- Open a command window and set your environment as described in "Setting Your Development Environment" in the Oracle CEP Getting Started.
- Change to the DOMAIN_DIR/servername directory, where DOMAIN_DIR refers to the domain directory and servername refers to the name of your server, such as d:\oracle_cep_home\user_projects\domains\mydomain\myserver1.
```
prompt> cd d:\oracle_cep_home\user_projects\domains\mydomain\myserver1
```
- Create a keystore coherence-identity.jks containing the boot user using the JDK keytool utility and the following command line:
```
prompt> keytool -genkey -v -keystore config/coherence-identity.jks -storepass PASSWORD -alias BOOT-USER -keypass BOOT-USER-PASSWORD -dname CN=BOOT-USER
```
  Where:
  - PASSWORD is the password used to secure the keystore.
  - BOOT-USER is the user name you used to log into the Oracle CEP server host.
  - BOOT-USER-PASSWORD is the password you used when you logged into the Oracle CEP server host.
- Create a permissions.xml file.
- Edit the permissions.xml file to add the following permissiosn for the boot user:
```
<permissions>
    <grant>
        <principal>
            <class>javax.security.auth.x500.X500Principal</class>
            <name>CN=BOOT-USER</name>
        </principal>

        <permission>
            <target>*</target>
            <action>all</action>
        </permission>
    </grant>
</permissions>
```
  Where BOOT-USER is the user name you used to log into the Oracle CEP server host.
- Save and close the permissions.xml file.
- Create a login.config file.
- Edit the login.config file to add the following:
```
Coherence {
    com.tangosol.security.KeystoreLogin required
        keyStorePath=".${/}config${/}coherence-identity.jks";
}; 
```
- Save and close the login.config file.
- Update the server startup script for your platform, startwlevs.cmd (Windows) or startwlevs.sh (UNIX), by adding the following property to the java command that actually starts the server:
```
-Djava.security.auth.login.config=./login.config
```
  For example:
```
"%JAVA_HOME%\bin\java" %DGC% %DEBUG% -Djava.security.auth.login.config=./login.config -Dwlevs.home="%USER_INSTALL_DIR%" -Dbea.hoe="%BEA_HOME%" -jar "%USER_INSTALL_DIR%\bin\wlevs.jar" %1 %2 %3 %4 %5 %6
```
If you plan to use Oracle CEP Visualizer with the servers in this domain, see Section 4.5.3, "How to Configure SSL in a Multi-Server Domain for Oracle CEP Visualizer".
Start all servers in your multi-server domain.

See Section 3.7, "Starting and Stopping an Oracle CEP Server in a Multi-Server Domain".

3.6 Using the Multi-Server Domain APIs to Manage Group Membership Changes

In an active-active system, applications are deployed homogeneously across several servers and are actively executing.

There are cases, however, when these homogeneously-deployed applications need to elect a primary one as the coordinator or leader. In this case, events that result from the coordinator application are kept and passed on to the next component in the EPN; the results of secondary servers are dropped. However, if the coordinator fails, then one of the secondary servers must be elected as the new coordinator.

To enable this in an application, the adapter or event bean, generally in the role of an event sink, must implement the com.bea.wlevs.ede.api.cluster.GroupMembershipListener interface which allows the event sinks to listen for multi-server domain group membership changes. At runtime, Oracle CEP automatically invokes the onMembershipChange callback method whenever membership changes occur.

The signature of the callback method is as follows:

onMembershipChange(Server localIdentity, Configuration groupConfiguration);

In the implementation of the onMembershipChange callback method, the event sink uses the Server object (localIdentity) to verify if it is the leader. This can be done be comparing localIdentity with the result of Configuration.getCoordinator() run on the second parameter, groupConfiguration. This parameter also allows a server to know what the current members of the group are by executing Configuration.getMembers().

In order to only keep events if it is a coordinator, the event sink must get a new Server identity every time membership in the group changes. Group membership changes occur if, for example, another server within the group fails and is no longer the coordinator.

A similar interface com.bea.wlevs.ede.api.cluster.DomainMembershipListener exists for listening to membership changes to the domain as a whole, rather than just changes to the group.

Note that in a hot-hot configuration, there is a non-zero delay in failure notification. If you are using the notification APIs to implement clustering, you will lose and not process events that occur in the window between the server failure and the notification being delivered to the new master server.

For more information, see Oracle CEP Java API Reference.

3.7 Starting and Stopping an Oracle CEP Server in a Multi-Server Domain

To start the servers in a multi-server domain, start each server separately by running its start script. This is the same way you start a server in a standalone server domain. See Section 2.4, "Starting and Stopping an Oracle CEP Server in a Standalone-Server Domain" for details.

If you have not configured custom groups for the multi-server domain, then all servers are members of just the pre-defined domain group, which contains all the servers in the multi-server domain, and a singleton group, one for each member server. This means, for example, if there are three servers in the multi-server domain then there are three singleton groups.

If, however, you have configured custom groups for the multi-server domain, then the servers are members of the groups for which they have been configured, as well as the pre-defined groups.

3.8 Deploying an Application to an Oracle CEP Multi-Server Domain

When you deploy an application to a multi-server domain, you typically specify a target group, and Oracle CEP then deploys the application to the set of running servers in that group. Oracle CEP dynamically maintains group membership based on running servers. This means that if new servers in the group are started, Oracle CEP automatically propagates the appropriate set of deployments to the new server.

Take, for example, the simple multi-server domain configured in the section Section 3.3, "Creating an Oracle CEP Multi-Server Domain Using Oracle Coherence." Assume that only myServer1 had been started, and then an application is deployed to the domain group, which includes myServer1 and myServer2. At that point, because only myServer1 of the multi-server domain has been started, the application will be deployed only to myServer1. When myServer2 is subsequently started, Oracle CEP automatically replicates and propagates the deployment of the application to myServer2 without the user having the explicitly deploy it.

Deployment propagation occurs based on application version. When you deploy a new version of an application, the new version is propagated to all servers accordingly.

If different configuration is required on different servers for an application then currently it is best to achieve this by using system properties.

This section describes how to perform the following tasks:

Section 3.8.1, "Deploying to an Oracle CEP Server Using the Oracle CEP Visualizer"
Section 3.8.2, "Deploying to an Oracle CEP Server Singleton Group Using the Deployer Utility"
Section 3.8.3, "Deploying to an Oracle CEP Server Domain Group Using the Deployer Utility"
Section 3.8.4, "Deploying to an Oracle CEP Server Custom Group Using the Deployer Utility"
Section 3.8.5, "Troubleshooting Multi-Server Domain Deployment"

For more information, see:

3.8.1 Deploying to an Oracle CEP Server Using the Oracle CEP Visualizer

The simplest way to deploy an Oracle CEP application to a multi-server domain is to use the Oracle CEP Visualizer.

For more information, see "Deploying an Application" in the Oracle CEP Visualizer User's Guide.

3.8.2 Deploying to an Oracle CEP Server Singleton Group Using the Deployer Utility

If you do not specify a group when you deploy an application, Oracle CEP deploys the application to the singleton server group that includes only the specific server to which you deploy the application. This is the standard case in single-server domains, but is also applicable to multi-server domains.

Note:

When you upgrade a 2.0 domain to execute in a multi-server domain, any deployed applications are deployed to the singleton server group.

The following example shows how to deploy to a singleton group; note that the command does not specify a -group option:

prompt> java -jar wlevsdeploy.jar -url http://ariel:9002/wlevsdeployer -install myapp_1.0.jar

In the example, the myapp_1.0.jar application will be deployed to the singleton server group that contains a single server: the one running on host ariel and listening to port 9002. If the domain is multi-server and other servers are members of the domain group, the application will not be deployed to these servers.

For more information about groups, see Section 3.1.1, "Groups".

3.8.3 Deploying to an Oracle CEP Server Domain Group Using the Deployer Utility

The domain group is a live group that always exists and contains all servers in a domain. In another words, all servers are always a member of the domain group. However, you must still explicitly deploy applications to the domain group. The main reason for this is for simplicity and consistency in usage.

When you explicitly deploy an application to the domain group, Oracle CEP guarantees that all servers of this homogenous environment have this deployment.

To deploy to the domain group, use the -group all option. The following example shows how to deploy to a domain group:

prompt> java -jar wlevsdeploy.jar -url http://ariel:9002/wlevsdeployer -install myapp_1.0.jar -group all

In the example, the myapp_1.0.jar application will be deployed to all servers of the domain group on host ariel listening to port 9002.

For more information about groups, see Section 3.1.1, "Groups".

3.8.4 Deploying to an Oracle CEP Server Custom Group Using the Deployer Utility

To deploy to a custom group, use the -group groupname option of the deploy command.

In the following examples, assume the multi-server domain has been configured as described in Section 3.3.2, "How to Create an Oracle CEP Multi-Server Domain With Custom Groups Using Oracle Coherence."

The following example shows how to deploy an application called strategies_1.0.jar to the strategygroup:

prompt> java -jar wlevsdeploy.jar -url http://ariel:9002/wlevsdeployer -install strategies_1.0.jar -group strategygroup

Based on the multi-server domain configuration, the preceding command deploys the application to myServer2 and myServer3, the members of the group strategygroup.

The following example shows how to deploy an application called selector_1.0.jar to the selectorgroup:

prompt> java -jar wlevsdeploy.jar -url http://ariel:9002/wlevsdeployer -install selector_1.0.jar -group selectorgroup

Based on the multi-server domain configuration, the preceding command deploys the application only to myServer1, which is the sole member of group selectorgroup.

Note that both commands are executed to the same server (the one on host ariel listening to port 9002). However, you can specify any of the servers in the domain in the deploy command, even if the server is not part of the group to which you want to deploy the application.

For more information about groups, see Section 3.1.1, "Groups".

3.8.5 Troubleshooting Multi-Server Domain Deployment

This section describes common problems that you may encounter when deploying applications to an Oracle CEP multi-server domain, including:

Section 3.8.5.1, "Oracle CEP Server Stops Application After Deployment"

3.8.5.1 Oracle CEP Server Stops Application After Deployment

Problem: After you deploy an application to an Oracle CEP multi-server domain, Oracle CEP stops the application after about 30 seconds.

Solution: Be sure you do not have more than one VPN software package installed on the same computer hosting your multi-server domain.

3.9 Next Steps

After creating your own Oracle CEP multi-server domain, consider the administration tasks that Section 1.5, "Understanding Oracle CEP Server Administration Tasks" describes.

For example, you can:

Optionally configure the server.

See Section 1.3, "Understanding Oracle CEP Server Configuration."
Create an Oracle CEP application.

See Oracle CEP IDE Developer's Guide for Eclipse for a description of the programming model, details about the various components that make up an application, how they all fit together, and typical steps to create a new application.
Deploy your new, or existing, Oracle CEP application to the domain.

For more information, see:
- Section 3.8, "Deploying an Application to an Oracle CEP Multi-Server Domain"
- "Assembling and Deploying Oracle CEP Applications" in the Oracle CEP IDE Developer's Guide for Eclipse
Manage your applications, servers, and domains:
- Using the Oracle CEP Visualizer.
  
  See Oracle CEP Visualizer User's Guide.
- Using the wlevs.Admin command line tool.
  
  See Appendix A, "wlevs.Admin Command-Line Reference".
- Using JMX and MBeans.
  
  See Section 6.3, "Managing With JMX".