Configuring and Using Oracle CEP Clustered Domains

The following sections introduce use cases and terminology related to the management and availability of a set of Oracle Complex Event Processing (or Oracle CEP for short) instances.

Oracle CEP Clustered Domain

An Oracle CEP clustered domain is a set of servers logically connected for the purposes of management, and physically connected using a shared User Datagram Protocol (UDP) multicast address. All servers in an Oracle CEP clustered domain are aware of all other servers in the domain and any one server can be used as an access point for making changes to the deployments in the domain.

Management of the clustering infrastructure is done at the domain level. Thus server failure, start, or restart is detected by every member of the clustered domain. Each member of the clustered domain has a consistent, agreed notion of domain membership enforced by the clustering infrastructure.

For servers to be considered a part of the same clustered domain they must share both the same multicast address and the same domain name.

Groups

In order to support the deployment to and management of the servers clustered domains at a finer grained-level than the domain, Oracle CEP introduces the concept of groups. A group is a set of one or more servers with a unique name within the domain. In an Oracle CEP domain, an arbitrary number of groups may exist with a configurable group membership. A server may be a member of more than one group, although typically this information is transparent to the user. The following pre-defined groups always exist:

Singleton server group—Consists of only the local server. This means that the membership of this group depends on the server from which it is accessed. This group can be used to pin deployments to a single server.
Domain (or default) group—Contains all live members of the domain. Its membership is automatically managed and cannot be changed by the user.

When you deploy an application, you deploy it to a particular group. Applications deployed to the domain or custom groups must have a unique name across the domain.

Cluster Notifications and Messaging

In order to provide high availibility (HA)-like capabilities to adapter implementations, Oracle CEP provides a number of notification and messaging APIs at both the group- and domain-level. Using these APIs, you can configure a server to receive notification when its group or domain membership changes, either because an administrator deliberately changed it or due to a server failure. Similarly you can use these APIs to send messages to both individual groups and to the domain.

Creating and Configuring a Simple Clustered Domain

This section describes how to create and configure a simple clustered domain from two or more Oracle CEP servers. The cluster is simple because it is not configured with any custom groups, other than the two predefined ones (singleton group and domain group.) See Configuring Custom Groups for a Clustered Domain for a description of how to configure custom groups within the clustered domain.

To create a simple clustered domain, update the config.xml file for each member server by adding a <cluster> child element of the root <config> element. Include the <name>, <multicast-address>, and <identity> child elements of <cluster>, as shown in the following example:

  <cluster>
      <name>myCluster</name>
      <multicast-address>239.255.0.1</multicast-address>
      <identity>1</identity> 
      <enabled>true</enabled>
  </cluster>

In the example, the server called myServer1 is part of the domain myDomain and also part of the clustered domain myCluster. By default, clustering is disabled, so you must explicitly enable it with the <enabled> element.

For each server of the clustered domain, the <name> and <multicast-address> elements must contain the same value. The <identity> element, however, must be different for each server in the clustered domain.

The <multicast-address> element is required unless all servers of the clustered domain are hosted on the same computer; in that case you can omit the <multicast-address> element and Oracle CEP automatically assigns a multicast address to the clustered domain based on the computer’s IP address. If, however, the servers are hosted on different computers, then you must provide an appropriate domain-local address. Oracle recommends you use an address of the form 239.255.X.X, which is what the auto-assigned multicast address is based on.

The <identity> element identifies the server’s identity and must be an integer between 1 and INT_MAX. Oracle CEP numerically compares the server identities during cluster operations; the server with the lowest identity becomes the domain coordinator. Be sure that each server in the clustered domain has a different identity; if servers have the same identity, the results of cluster operations are unpredictable.

An example of configuring a second server, called myServer2, in the myCluster clustered domain is shown below; note that its identity is 2:

  <cluster>
      <name>myCluster</name>
      <multicast-address>239.255.0.1</multicast-address>
      <identity>2</identity> 
      <enabled>true</enabled>
  </cluster>

Configuring Custom Groups for a Clustered Domain

There are cases where the application logic cannot simply be replicated across a homogenous set of clustered servers. Examples of these types of applications are those that must determine the best price provided by different pricing engines, or applications that send an alert when a position crosses a threshold. In these cases, the application is not idempotent; it must calculate only once or send a single event. In other cases, the application has a singleton nature, such as a monitoring application, the HTTP pub-sub server, and so on.

As a more complex example, consider a domain that has two applications: the strategies application uses several strategies for calculating different prices for some derivative and then feeds its results to a selector application. The selector application then selects the best price amongst the different options provided by the strategies’ application results. The strategies application can be replicated to achieve fault-tolerance. However, the selector application must be able to keep state so as to determine the best price; for this reason, the selector application cannot be replicated in a hot/hot fashion.

For all these reasons, a domain must support servers that are not completely homogeneous; you configure this by creating custom groups.

To configure a group, update the config.xml file of each member of the group, adding (if one does not already exist) a <groups> child element of <cluster> and specifying the name of the group as the value of the <groups> element. The <groups> element can include more than one group name in the case that the server is a member of more than one group; separate multiple group names using commas. The <groups> element is optional; if a server configuration does not include one, then the server is a member of the default groups (domain and singleton).

For example, assume you have created three servers (myServer1, myServer2, and myServer3) and you want server1 to be a member of the selector group and server2 and server3 to be members of the strategy group. The relevant snippets of the config.xml file for each server are shown below:

<config>
  <domain>
        <name>myDomain</name>
       <server>
           <name>myServer1</name>
       </server>
  </domain>
  <cluster>
      <name>myCluster</name>
      <multicast-address>239.255.0.1</multicast-address>
      <identity>1</identity> 
      <groups>selector</groups>  
      <enabled>true</enabled>
  </cluster>

<config>
  <domain>
        <name>myDomain</name>
       <server>
           <name>myServer2</name>
       </server>
  </domain>
  <cluster>
      <name>myCluster</name>
      <multicast-address>239.255.0.1</multicast-address>
      <identity>2</identity> 
      <groups>strategy</groups>  
      <enabled>true</enabled>
  </cluster>

<config>
  <domain>
        <name>myDomain</name>
       <server>
           <name>myServer3</name>
       </server>
  </domain>
  <cluster>
      <name>myCluster</name>
      <multicast-address>239.255.0.1</multicast-address>
      <identity>3</identity> 
      <groups>strategy</groups>  
      <enabled>true</enabled>
  </cluster>

Starting the Servers in a Clustered Domain

To start the servers in a clustered domain, start each server separately by running its start script. This is the same way you start a server even if it’s not part of a clustered domain. See Stopping and Starting the Server for details.

If you have not configured custom groups for the clustered domain, then all servers are members of just the pre-defined domain group, which contains all the servers in the clustered domain, and a singleton group, one for each member server. This means, for example, if there are three servers in the clustered domain then there are three singleton groups.

If, however, you have configured custom groups for the clustered domain, then the servers are members of the groups for which they have been configured, as well as the pre-defined groups.

Deploying Applications to a Clustered Domain

When you deploy an application to a clustered domain, you typically specify a target group, and Oracle CEP then deploys the application to the set of running servers in that group. Oracle CEP dynamically maintains group membership based on running servers. This means that if new servers in the group are started, Oracle CEP automatically propagates the appropriate set of deplyments to the new server.

Take, for example, the simple cluster configured in the section Creating and Configuring a Simple Clustered Domain. Assume that only myServer1 had been started, and then an application is deployed to the domain group, which includes myServer1 and myServer2. At that point, because only myServer1 of the clustered domain has been started, the application will be deployed only to myServer1. When myServer2 is subsequently started, Oracle CEP automatically propagates the deployment of application to myServer2 without the user having the explicitly deploy it.

Deployment propagation only occurs when a server is missing a required deployments. If a server already has a local deployment, then this deployment is used. This means that Oracle CEP does not automatically propagate changes to a deployment on one server of the clustered domain to the other servers if they already have that deployment. This means that it is possible to have slightly different versions of the same deployment on different servers if the deployment is configured manually through copying application jar files.

If different configuration is required on different servers for an application then currently it is best to achieve this by using system properties.

Deploying to the Singleton Server Group

If you do not specify a group when you deploy an application, Oracle CEP deploys the application to the singleton server group that includes only the specific server to which you deploy the application. This is the standard case in single-server domains, but is also applicable to clustered domains.

The following example shows how to deploy to a singleton group; note that the command does not specify a -group option:

In the example, the myapp_1.0.jar application will be deployed to the singleton server group that contains a single server: the one running on host ariel and listening to port 9002. If the domain is clustered and other servers are members of the domain group, the application will not be deployed to these servers.

Deploying to the Domain Group

The domain group is a live group that always exists and contains all servers in a domain. In another words, all servers are always a member of the domain group. However, you must still explicitly deploy applications to the domain group. The main reason for this is for simplicity and consistency in usage.

When you explicitly deploy an application to the domain group, Oracle CEP guarantees that all servers of this homogenous environment have this deployment.

To deploy to the domain group, use the -group all option. The following example shows how to deploy to a domain group:

In the example, the myapp_1.0.jar application will be deployed to all servers of the domain group on host ariel listening to port 9002.

Deploying to a Custom Group

To deploy to a custom group, use the -group groupname option of the deploy command.

The following example shows how to deploy an application called strategies_1.0.jar to the strategygroup:

Based on the cluster configuration, the preceding command deploys the application to myServer2 and myServer3, the members of the group strategygroup.

The following example shows how to deploy an application called selector_1.0.jar to the selectorgroup:

Based on the cluster configuration, the preceding command deploys the application only to myServer1, which is the sole member of group selectorgroup.

Note that both commands are executed to the same server (the one on host ariel listening to port 9002). However, you can specify any of the servers in the domain in the deploy command, even if the server is not part of the group to which you want to deploy the application.

Using the Cluster APIs to Manage Group Membership Changes

In an active-active system, applications are deployed homogeneously across several servers and are actively executing.

There are cases, however, when these homogeneously-deployed applications need to elect a primary one as the coordinator or leader. In this case, events that result from the coordinator application are kept and passed on to the next component in the EPN; the results of secondary servers are dropped. However, if the coordinator fails, then one of the secondary servers must be elected as the new coordinator.

To enable this in an application, the adapter or event bean, generally in the role of an event sink, must implement the


com.bea.wlevs.ede.api.cluster.GroupMembershipListener

interface which allows the event sinks to listen for cluster group membership changes. At runtime, Oracle CEP automatically invokes the onMembershipChange callback method whenever membership changes occur.

    onMembershipChange(Server localIdentity, Configuration groupConfiguration);

In the implementation of the onMembershipChange callback method, the event sink uses the Server object (localIdentity) to verify if it is the leader. This can be done be comparing localIdentity with the result of Configuration.getCoordinator() run on the second paramter, groupConfiguration. This parameter also allows a server to know what the current members of the group are by executing Configuration.getMembers().

In order to only keep events if it is a coordinator, the event sink must get a new Server identity every time membership in the group changes. Group membership changes occur if, for example, another server within the group fails and is no longer the coordinator.

Securing Cluster Messages

The servers in a clustered domain update their state by exchanging cluster-related messages. It is important that these messages be at least checked for integrity. A private key can be used to achieve integrity. This key must be shared by all servers in the domain.

To configure security for a clustered domain, add the <security> child element to the <cluster> element for each server in the clustered domain, as shown:

<config>
  <domain>
        <name>myDomain</name>
       <server>
           <name>myServer1</name>
       </server>
  </domain>
  <cluster>
      <name>myCluster</name>
      <multicast-address>239.255.0.1</multicast-address>
      <identity>1</identity> 
      <security>sign</security>
      <enabled>true</enabled>
  </cluster>

none—Default value. Specifies that no security is configured for the clustered domain.
sign—Specifies that cluster messages should be digitally signed.
encrypt—Specifies that cluster messages should be encrypted.

After you have generated the key for signing or encrypting, whichever is appropriate, put the key in the file called.msainternal.dat, located in the servername directory of the main domain directory; this file is generated when the server first starts up.

Additional Child Elements of the <cluster> Element

This section briefly describes additional child elements. For the complete XSD Schema of the config.xml file, including a description of the <cluster> element, see Server Configuration XSD Schema.

You can add the following optional child elements to the <cluster> element of the config.xml file to further configure your clustered domain:

<server-name>—Specifies a name for the server. Visualizer uses the value of this element when it displays the server in its console. Default value if element is not set is Server-<identity>.
<server-host-name>—Specifies the host address/IP used for point-to-point HTTP cluster communication. Default value is localhost.
<multicast-port>—Specifies the port used for multicast traffic. Default value is 9100.
<operation-timeout>—Specifies, in milliseconds, the timeout for point-to-point HTTP clustering requests. Default value is 30000.

Administration and Configuration Guide