17 High Availability Applications

High availability is critical to Oracle Stream Explorer applications because they continuously monitor streaming data. Oracle Stream Explorer provides application design patterns and high availability adapters, to enable you to increase the backup and failover processing capabilities of you applications.

This chapter includes the following sections:

17.1 Oracle Coherence

Oracle Stream Explorer high availability options depend on Oracle Coherence. You cannot implement Oracle Stream Explorer high availability options without Oracle Coherence. When considering performance tuning, be sure to evaluate your Oracle Coherence configuration in addition to your Oracle Stream Explorer application.

For more information, see:

High Availability Performance Tuning
Administering Oracle Stream Explorer
Oracle Coherence Developer's Guide at https://download.oracle.com/docs/cd/E15357_01/coh.360/e15723/toc.htm.

17.2 Architecture

Oracle Stream Explorer supports an active-active high availability architecture. This approach has the advantages of high performance, simplicity, and short failover time to mitigate the likelihood and impact of data and service faults.

Deploy high availability applications to a group of two or more Oracle Stream Explorer servers running in a multiserver domain. Oracle Stream Explorer chooses one server in the group to be the active primary server. The other servers become active secondary servers.

The primary and secondary servers are configured to receive the same input events and process them in parallel but only the primary server outputs events to the Oracle Stream Explorer application client. Depending on the quality of service you choose, the secondary servers buffer their output events using in-memory queues and the primary server keeps the secondary servers up to date with which events the primary has already output.

Figure 17-1 shows a typical configuration with one active server and two primary servers.

Figure 17-1 Oracle Stream Explorer High Availability: Primary and Secondary Servers

Description of "Figure 17-1 Oracle Stream Explorer High Availability: Primary and Secondary Servers"

17.3 Life Cycle and Failover

Figure 17-2 shows a state diagram for the Oracle Stream Explorer high availability life cycle. In this diagram, the state names (SECONDARY, BECOMING_PRIMARY, and PRIMARY) correspond to the Oracle Stream Explorer high availability adapter RuntimeMBean method getState return values. These states are specific to Oracle Stream Explorer. See High Availability Input Adapter for more information about the high availability adapter.

Figure 17-2 Oracle Stream Explorer High Availability Life Cycle State Diagram

Description of "Figure 17-2 Oracle Stream Explorer High Availability Life Cycle State Diagram"

You cannot specify the initial primary server. Initially, the first server in the multiserver domain to start up becomes the primary so by starting servers in a particular order, you can influence primary selection. There is no way to force a particular, running server to become the primary. If a primary fails and comes back up, then it does not automatically become the primary again unless the current primary fails and causes a failover.

17.3.1 Secondary Failure

In general, when a secondary server fails, there is no effect on Oracle Stream Explorer application operation as Figure 17-3 shows. Regardless of the quality of service you choose, there are no missed or duplicate events.

Figure 17-3 Secondary Failure

Description of "Figure 17-3 Secondary Failure"

17.3.2 Primary Failure and Failover

When a primary server fails, as Figure 17-4 shows, Oracle Stream Explorer performs a failover that can cause missed or duplicate events, depending on the quality of service you choose.

Figure 17-4 Primary Failure and Failover

Description of "Figure 17-4 Primary Failure and Failover"

During failover, Oracle Stream Explorer selects a new primary and the new primary transitions from the SECONDARY state to the BECOMING_PRIMARY state. Depending on the quality of service you choose, the new primary does not transition to PRIMARY state until a configurable readiness threshold is met. For details, see the specific quality of service option in Choose a Quality of Service Option.

17.3.3 Rejoining the High Availability MultiServer Domain

When a new Oracle Stream Explorer server is added to an Oracle Stream Explorer high availability multiserver domain or an existing failed server restarts, the server does not fully join the Oracle Stream Explorer high availability deployment and notification groups until all applications deployed to it have fully joined. The type of application determines when it can fully join.

If the application must generate exactly the same sequence of output events as existing secondaries (a Type 1 application), then it must be able to rebuild its internal state by processing input streams for some finite period of time (the warm-up-window-length period). This warm-up-window-length time determines the minimum time it takes for the application to fully join the Oracle Stream Explorer high availability deployment and notification groups.

If the application does not need to generate exactly the same sequence of output events as existing secondaries (a Type 2 application), then it does not require a warm-up-window-length time and fully joins the Oracle Stream Explorer high availability deployment and notification groups when it deploys.

For more information, see Choose an Adequate warm-up-window-length Time.

17.4 Deployment Group and Notification Group

All of the servers in the multiserver domain belong to the same deployment group. the deployment group is the group to which you deploy an application. For the purposes of Oracle Stream Explorer high availability, you must deploy the same application to all of the servers in this group.

By default, all of the servers in the multiserver domain also belong to the same notification group. The servers listen to the notification group for membership notifications that indicate when a server fails (and exits the group) or resumes operation (and rejoins the group), and for synchronization notifications from the primary.

If you need to scale your Oracle Stream Explorer high availability application, use the ActiveActiveGroupBean to define a notification group that allows two or more servers to function as a primary server unit while retaining the convenience of a single deployment group that spans all servers (primaries and secondaries).

You must use Oracle Coherence-based clustering to create the multiserver domain deployment group. You can use either default groups or custom groups.

For more information, see:

17.5 High Availability Adapters

To implement Oracle Stream Explorer high availability, add the optional high availability input adapter and the required high availability output adapters to the EPN.

High Availability Input Adapter

The optional high availability input adapter in the primary server communicates with the corresponding high availability input adapters in each secondary server to normalize event time stamps. Oracle Stream Explorer high availability provides one type of high availability input adapter.

See High Availability Input Adapter.

High Availability Output Adapters

To have high availability functionality in your application, put a high availability output adapter before each output adapter in your EPN. The high availability output adapter in the primary server outputs events to the output streams that connect the Oracle Stream Explorer application to its downstream client.

The high availability output adapter in the primary also communicates with the corresponding high availability output adapters in each secondary, and depending on the high availability quality of service you choose, can instruct the secondary output adapters to trim their in-memory queues of output events.

For information about the high availability output adapters, see Buffering Output Adapter, Broadcast Output Adapter, and Correlating Output Adapter. Which output adapter you choose is determined by the high availability quality of service you choose. See Choose a Quality of Service Option.

Figure 17-5 shows a simplified EPN with all possible high availability adapters in place. The figure shows no channels and one processor.

Figure 17-5 High Availability Adapters in the EPN

Description of "Figure 17-5 High Availability Adapters in the EPN"

17.5.1 High Availability Input Adapter

Each event is associated with a point in time at which the event occurred. There are two different approaches to generating event timestamps: application timestamps and system timestamps (see Channels for information about application and system time stamps). Application time means that a time value is assigned to each event externally by the application before the event enters a CQL processor. System time means that a time value is associated with an event automatically by Oracle Stream Explorer when it arrives at a CQL processor.

Application time is generally the best approach for applications that need to be highly available. The application time is associated with an event before the event is sent to Oracle Stream Explorer, so it is consistent across active primary and secondary instances. System time can cause application instances to generate different results because the time value associated with an event can be different on each instance due to system clocks not being synchronized.

Using system time is not a problem for applications whose CQL queries do not use time-based windows. Applications that use only event-based windows depend only on the arrival order of events rather than the arrival time, so system time can be used in this case.

For applications that use time-based windows and do not have externally generated (application) time stamps, the optional Oracle Stream Explorer high availability input adapter can be used to provide a consistent time value across all servers. The input adapter instance on the primary server assigns a time (in nanoseconds) to events as they arrive at the adapter and forwards the time values assigned for each event to all secondary servers.

Because a time value is assigned to each event before the event reaches any downstream channels in the EPN, downstream channels should be configured to use application time so that they do not assign a new time value to events as they arrive at the channel.

Input events must have a key that uniquely identifies each event in order to use this adapter.

You can configure the Oracle Stream Explorer high availability input adapter to send heartbeat events.

The Oracle Stream Explorer high availability input adapter is applicable to all high availability quality of service options. However, because the high availability input adapter increases performance overhead, it is not appropriate for some high availability quality of service options (such as Simple Failover and Simple Failover with Buffering). For these options, you should instead consider using application time with some incoming event property.

For more information, see:

17.5.2 Buffering Output Adapter

The Oracle Stream Explorer high availability buffering output adapter implements a buffered queue trimming strategy. The buffer is a sliding window of output events from the stream. The size of the window is measured in milliseconds.

The Oracle Stream Explorer high availability buffering output adapter applies to simple failover, and simple failover with buffering high availability quality of service options.

For more information, see:

17.5.3 Broadcast Output Adapter

The Oracle Stream Explorer high availability broadcast output adapter implements a distributed queue trimming strategy. The active primary instance broadcasts messages to the active secondary instances in the notification group telling them when to trim their local representation of the queue.

The Oracle Stream Explorer high availability broadcast output adapter applies to the light-weight queue trimming high availability quality of service option.

For more information, see:

17.5.4 Correlating Output Adapter

The Oracle Stream Explorer high availability correlating output adapter correlates two event streams, usually from JMS. This adapter correlates an inbound buffer of events with a second source of the same event stream, outputting the buffer if correlation fails after a configurable time interval. Correlated events are trimmed from the queue. Correlated events are assumed to be in-order.

The Oracle Stream Explorer high availability correlating output adapter applies to precise recovery with JMS high availability quality of service option.

For more information, see:

17.6 High Availability and Scalability

If you need to scale your Oracle Stream Explorer high availability application, use the notification groups Spring bean, ActiveActiveGroupBean to increase scalability in JMS applications. The ActiveActiveGroupBean defines a notification group that allows two or more servers to function as a high availability unit while retaining the convenience of a single deployment group that spans all servers (primaries and secondaries).

Figure 17-6 shows three Oracle Stream Explorer application scenarios progressing from the simplest configuration, to high availability, and then to both high availability and scalability.

Figure 17-6 High Availability and Scalability

Description of "Figure 17-6 High Availability and Scalability"

Most applications begin in a single-server domain without high availability. In this case, the simplest scenario, which is an Oracle Stream Explorer application running on one Oracle Stream Explorer server processes an input event stream and produces output events.

High availability scenario: The application is configured to use Oracle Stream Explorer high availability options, is deployed to the deployment group of a multiserver domain composed of two server, and only the primary server outputs events.

High availability and scalability scenario: The high availability application is configured to use the ActiveActiveGroupBean to define notification groups. Each notification group contains two or more servers that function as a single, high availability unit. In this scenario, only the primary server in each notification group outputs events. Should the primary server in a notification group go down, an Oracle Stream Explorer high availability fail over occurs and a secondary server in that notification group is declared the new primary and resumes outputting events according to the Oracle Stream Explorer high availability quality of service you configure.

For more information, see Configure Partitioning with High Availability.

17.7 Choose a Quality of Service Option

You can choose any of the quality of service options that Table 17-1 lists. Choose the quality of service option that suits your application's tolerance for missed and duplicate events and expected event throughput. Note that primary and secondary server hardware requirements increase as the quality of service becomes more precise.

Table 17-1 Oracle Stream Explorer High Availability Quality of Service

High Availability Option	Missed Events?	Duplicate Events?	Performance Overhead
Simple Failover	Yes (many)	Yes (few)	Negligible
Simple Failover with Buffering	Yes (few)	Yes (many)	Low
Light-Weight Queue Trimming	No	Yes (few)	Low-Medium
Precise Recovery with JMS	No	No	High

17.7.1 Simple Failover

The simple failover high availability quality of service is characterized by the lowest performance overhead (fastest recovery time) and the least data integrity (both missed events and duplicate events are possible during failover).

The primary server outputs events and secondary servers discard their output events because they do not buffer output events. If the current active primary fails, a new active primary is chosen and begins sending output events once it is notified.

During failover, many events can be missed or duplicated by the new primary depending on whether it is running ahead of or behind the old primary, respectively.

During the failover window, events can be missed. For example, if you process 100 events per second and failover takes 10 seconds, then you miss 1000 events

The new primary server enters the PRIMARY state immediately. There is no configurable readiness threshold that must be met before the new primary server transitions out of the BECOMING_PRIMARY state. When an Oracle Event Processing server rejoins the multiserver domain, it is immediately available as a secondary.

To implement this high availability quality of service, you configure your EPN with a high availability buffering output adapter (with a sliding window of size zero) before each output adapter. To reduce performance overhead, rather than use a high availability input adapter, use application time with some incoming event property.

For more information, see Configure a Simple Failover.

17.7.2 Simple Failover with Buffering

The simple failover with buffering high availability quality of service is characterized by a low performance overhead (faster recovery time) and increased data integrity (no missed events but many duplicate events are possible during failover).

The primary server outputs events and the secondary servers buffer their output events. If the current active primary fails, a new active primary is chosen and begins sending output events once it is notified.

During the failover window, events might be missed. For example, if you are processing 100 events per second and failover takes 10 seconds, then you miss 1000 events. If the secondary buffers are large, a significant number of duplicates can be output. On the other hand, a larger buffer reduces the chances of missed messages.

When an Oracle Stream Explorer server rejoins the multiserver domain, if your application is an Oracle Stream Explorer high availability Type 1 application (the application must generate exactly the same sequence of output events as existing secondaries), then it must wait the warm-up-window-length time you configure for the Oracle Stream Explorer high availability output adapter before it is available as a secondary.

To implement this high availability quality of service, you configure your EPN with a high availability buffering output adapter with a sliding window of size greater than zero before each output adapter. To reduce performance overhead, rather than use a high availability input adapter, use application time with some incoming event property.

For more information, see:

17.7.3 Light-Weight Queue Trimming

This high availability quality of service is characterized by a low performance overhead (faster recovery time) and increased data integrity (no missed events but a few duplicate events are possible during failover).

The active primary communicates to the secondaries the events that it has actually processed. This enables the secondaries to trim their buffer of output events so that it contains only those events that have not been sent by the primary at a particular point in time. Because events are only trimmed after they have been sent by the current primary, this allows the secondary to avoid missing any output events when there is a failover.

The frequency with which the active primary sends queue trimming messages to active secondaries is configurable:

Every n events (n>0)

This limits the number of duplicate output events to at most n events at failover.
Every n milliseconds (n>0)

The queue trimming adapter requires a way to identify events consistently among the active primary and secondaries. The recommended approach is to use application time to identify events, but any key value that uniquely identifies events works.

The advantage of queue trimming is that output events are never lost. There is a slight performance overhead at the active primary, however, for sending the trimming messages that need to be communicated. This overhead increases as the frequency of queue trimming messages increases.

During failover, the new primary enters the BECOMING_PRIMARY state and does not transition into the PRIMARY state until its event queue (that it was accumulating as a secondary) has been flushed. During this transition, new input events are buffered and some duplicate events can be output.

When an Oracle Stream Explorer server rejoins the multiserver domain, if your application is an Oracle Stream Explorer high availability Type 1 application (an application that must generate exactly the same sequence of output events as existing secondaries), then it must wait the warm-up-window-length time you configure for the Oracle Stream Explorer high availability output adapter before it is available as a secondary.

To implement this high availability quality of service, you configure your EPN with a high availability input adapter after each input adapter and a high availability broadcast output adapter before each output adapter.

For more information, see Configure Light-Weight Queue Trimming.

17.7.4 Precise Recovery with JMS

The precise recovery with JMS high availability quality of service is characterized by a high performance overhead (slower recovery time) and maximum data integrity (no missed events and no duplicate events during failover). This high availability quality of service is compatible with only JMS input and output adapters.

In the precise recovery with JMS high availability quality of service, the focus is not on transactional guarantees along the event path for a single-server, but on guaranteeing a single output from a set of servers. To achieve guarantee, secondary servers listen over JMS to the event stream being published by the primary. As Figure 17-7 shows, this incoming event stream is a source of reliable queue-trimming messages that the secondaries use to trim their output queues. If JMS is configured for reliable delivery, we can be sure that the stream of events seen by the secondary is precisely the stream of events output by the primary and so failover allows the new primary server to output precisely those events not delivered by the old primary server.

Figure 17-7 Precise Recovery with JMS

Description of "Figure 17-7 Precise Recovery with JMS"

During failover, the new primary server enters the BECOMING_PRIMARY state and does not transition into the PRIMARY state in its event queue (that was accumulating as a secondary) has been flushed. During this transition, new input events are buffered and no duplicate events are output.

When an Oracle Stream Explorer server rejoins the multiserver domain, if your application is an Oracle Stream Explorer high availability Type 1 application (the application must generate exactly the same sequence of output events as existing secondaries), it must wait the warm-up-window-length time you configure for the Oracle Stream Explorer high availability output adapter before it is available as a secondary server.

To implement the precise recovery with JMS high availability quality of service, you configure your EPN with a high availability input adapter after each input adapter and a high availability correlating output adapter before each output adapter.

To increase scalability, you can also use the cluster groups bean with high availability quality of service.

For more information, see:

17.8 Design Applications for High Availability

Although you can implement Oracle Stream Explorer high availability declaratively, to fully benefit from the high availability quality of service you choose, you must design your Oracle Stream Explorer application to take advantage of the high availability options that Oracle Stream Explorer provides.

When you design your application for high availability, consider the primary use case, design patterns, and Oracle CQL query restrictions discussed here.

17.8.1 Primary High Availability Use Case

You can adapt high availability options to various Oracle Stream Explorer application designs but in general, Oracle Stream Explorer high availability is designed for the following use case:

An application receives input events from one or more external systems.
The external systems are publish-subscribe style systems that allow multiple instances of the application to connect simultaneously and receive the same stream of messages.
The application does not update any external systems in a way that would cause conflicts when multiple copies of the application run concurrently.
The application sends output events to an external downstream system. Multiple instances of the application can connect to the downstream system simultaneously, although only one instance of the application can send messages at any one time.

Within these constraints, the following different cases are of interest:

The application is allowed to skip sending some output events to the downstream system when there is a failure. Duplicates are also allowed. Use the "Simple Failover" high availability quality of service option.
The application is allowed to send duplicate events to the downstream system, but must not skip any events when there is a failure. Use the Simple Failover with Buffering and Light-Weight Queue Trimming high availability quality of service options.
The application must send exactly the same stream of messages/events to the downstream system when there is a failure, modulo a brief pause during which events may not be sent when there is a failure. Use the Precise Recovery with JMS high availability quality of service option.

17.8.2 High Availability Design Patterns

When designing your Oracle Stream Explorer application for use with Oracle Stream Explorer high availability options, observe the following design patterns:

17.8.2.1 Select the Minimum High Availability Your Application can Tolerate

Be sure that the extra cost of precise recovery (per-stage throughput decrease) is actually necessary for your application.

17.8.2.2 Use High Availability Components at All Ingress and Egress Points

Use a high availability input adapter after each regular input adapter and use a high availability output adapter before each regular output adapter.

17.8.2.3 Preserve What You Need

Oracle Stream Explorer systems receive a large number of raw input events that are queried to generate a smaller number of relevant events. In general it makes sense to preserve the relevant event because there are fewer of them, and they are more valuable.

17.8.2.4 Limit Oracle Stream Explorer Application State

Oracle Stream Explorer systems enable you to query windows of events. It can be tempting to build systems with very large windows, but this increases the state that needs to be rebuilt when failure occurs. In general it is better to think of long-term state as something kept in stable storage, such as a distributed cache or a database to leverage the high availability facilities of these technologies.

17.8.2.5 Choose an Adequate warm-up-window-length Time

When a new Oracle Stream Explorer server is added to a high availability multiserver domain or when an existing failed server restarts, the server does not fully join the Oracle Stream Explorer high availability deployment and notification groups until all applications deployed to it have fully joined. The type of application determines when it fully joins.

Oracle Stream Explorer high availability applications can be described as Type 1 or Type 2 applications as Table 17-2 shows.

Table 17-2 Oracle Stream Explorer High Availability Application Types

Application Type	Must generate exactly the same sequence of output events?	Must be able to rebuild internal state by processing input streams within a finite period of time?	Must wait this period of time before it has fully joined?
Type 1	Yes	Yes	Yes
Type 2	No	No	No

For more information, see Rejoining the High Availability MultiServer Domain.

17.8.2.5.1 Type 1 Applications

A Type 1 application requires the new secondary server to generate exactly the same sequence of output events as existing secondary servers once it fully joins the high availability deployment and notification groups.

A Type 1 application must be able to rebuild its internal state by processing its input streams for a finite period of time (warm-up-window-length time), after which it generates the same stream of output events as other secondary servers running the application.

Configure the warm-up-window-length time on a high availability output adapter. Specify the length of the warm-up-window-length time length in seconds or minutes. For example, if the application contains Oracle CQL queries with range-based windows of 5, 7, and 15 minutes, then the minimum warm-up-window-length time is 15 minutes (the maximum range-based window size). Oracle recommends that the maximum window length be padded with a few minutes to ensure that the necessary state is available. So, in the previous example 17 minutes or 20 minutes would be a good length for the warm-up-window-length time.

The server uses system time during the warm-up-window-length time period, so the server time is not directly correlated with the application time associated with events being processed.

Type 1 applications must only be interested in events that occurred during a finite amount of time. All range-based Oracle CQL windows must be shorter than the warm-up-window-length time and tuple-based windows must be qualified by time. For example, the application should only query the last 10 events if they occurred within the last five minutes. Applications that do not have this property cannot be Type 1 applications and cannot use the warm-up-window-length period.

For example, an application that uses an tuple-based partitioned window that has no time qualification cannot use the warm-up-window-length period because an arbitrary amount of time is required to rebuild the state of the window.

If a Type 1 application uses the high availability broadcast output adapter, it can trim events with a unique application-specific key, or a monotonic key like application time. Trimming events with application time is encouraged because it is more robust and less susceptible to bugs in the application that can cause an output event to not be generated.

For more information, see:

17.8.2.5.2 Type 2 Applications

A Type 2 application does not require the new secondary server to generate the same sequence of output events as existing secondary servers once it fully joins the high availability deployment and notification groups. It simply requires that the new cluster member generate valid output events with respect to the point in time at which it begins processing input events.

A Type 2 application does not require a warm-up-window-length period.

Most applications are Type 2 applications. It is common for an application to be brought up at an arbitrary point in time (on the primary Oracle Stream Explorer server), begin processing events from input streams at that point, and generate valid output events. The input stream is not paused while the application starts and input events are constantly being generated and arriving. You can assume that in many cases a secondary stage that does the same thing, but at a slightly different time, also produces output events that are valid from the point of view of the application, although not necessarily identical to those events produced by the primary server because of slight timing differences.

For example, a financial application that only runs while the market is open might operate as a Type 2 application as follows: All servers can be brought up before the market opens and begin processing incoming events at the same point in the market data stream. Multiple secondary servers can be run to protect against failure, and if the number of secondary servers is sufficient while the market is open, then do not restart secondary servers that fail or add additional secondary servers because no secondary server needs to recover its state.

17.8.2.6 Ensure Applications are Idempotent

You scan run two copies of an application on different servers and the copies do not conflict in a shared cache or database. If you use an external relation (such as a cache or table), then you must ensure that when a server rejoins the cluster, your application accesses the same cache or table as before. The application must join to the same external relation again. The data source defined on the server must not have been changed to ensure that you are pulling data from same data source.

17.8.2.7 Source Event Identity Externally

Many high availability solutions require that events be correlated between different servers. To correlate events, the events must be universally identifiable. The best way to make event universally identifiable is use external information, preferably a time stamp, to seed the event. For more information, see Prefer Application Time .

17.8.2.8 Understand the Importance of Event Ordering

Primary and secondary servers must generate the same output events and in exactly the same order when you choose high availability quality of service options that use queue trimming and equality-based event identify (non-monotonic event identifiers that do not increase continually). Generating output events in different orders can cause missed output events or unnecessary duplicate output events when a failure occurs.

Consider the output event streams shown in Figure 17-8. The primary server has output events a, b, and c. After outputting event c, the primary sends the secondary a queue trimming message.

Figure 17-8 Event Order

Description of "Figure 17-8 Event Order"

The secondary server trims all events in its queue generated prior to event c including event c itself. In this case, the set of events trimmed will be {a, b, e, d, c}, which is wrong because the primary server has not yet output events d and e. If a failover occurs after processing the trimming message for event c, events are lost.

17.8.2.8.1 Prefer Deterministic Behavior

For an application to generate events in the same order when run on multiple instances, it must be deterministic. The application must not rely on things like:

Random number generators that can return different results on different machines.
Methods like System.getTimeMillis or System.nanoTime that can return different results on different machines because the system clocks are not synchronized.

17.8.2.8.2 Avoid Multithreading

Because thread scheduling algorithms are very timing dependent, multithreading can be a source of nondeterministic behavior in applications. Different threads can be scheduled at different times on different machines.

For example, avoid creating an EPN in which multiple threads send events to a high availability adapter in parallel. If such a channel is an event source for a high availability adapter, it would cause events to be sent to the adapter in parallel by different threads and could make the event order nondeterministic. Also, do not send events to the mediator (JMS server) with multiple threads, which acts as an event source.

For more information on channel configuration to avoid, see:

Customizing Oracle Stream Explorer
max-threads in Schema Reference for Oracle Stream Explorer.

17.8.2.8.3 Prefer Monotonic Event Identifiers

Event identifiers can be monotonic or non-monotonic. A monotonic identifier is one that increases continually (such as a time value). A non-monotonic identifier does not increase continually and may contain duplicates.

In general, design your application with monotonic event identifiers. With a monotonic event identifier, the Oracle Stream Explorer high availability adapter can handle an application that may produce events out of order.

17.8.2.9 Write Oracle CQL Queries with High Availability in Mind

Not all Oracle CQL query usage is supported when using Oracle Stream Explorer high availability. You might need to redefine your Oracle CQL queries to address these restrictions. For more information, see Oracle CQL Query Restrictions.

17.8.2.10 Avoid Coupling Servers

The best high availability performance for Oracle Stream Explorer systems comes when servers can run without requiring coordination. Generally this can be done when there is no shared state, and the downstream system can tolerate duplicates. Increasing levels of high availability are targeted at increasing the fidelity of event stream that the downstream system sees, but the increase in fidelity has a performance penalty.

17.8.2.11 Plan for Server Recovery

When a secondary server rejoins the multiserver domain, the server needs time to rebuild the application state to match the current primary and active secondaries. See Choose an Adequate warm-up-window-length Time.

The time it takes for a secondary server to become available as an active secondary server after it rejoins the multiserver domain is a factor in the number of active secondary servers you require.

If a secondary is declared to be the new primary server before it is ready, the secondary server throw an exception.

17.8.3 Oracle CQL Query Restrictions

In a high availability application, Oracle CQL queries have the following restrictions. For more information about Oracle CQL and the topics covered in this section, see Oracle CQL Language Reference for Oracle Stream Explorer.

17.8.3.1 Range-Based Windows

In a Type 1 application where the application must generate exactly the same sequence of output events as existing secondaries, all range-based Oracle CQL windows must be shorter than the warm-up-window-length time. See Choose an Adequate warm-up-window-length Time.

Channels must use application time when Oracle CQL queries contain range-based Windows. See Prefer Application Time .

17.8.3.2 Tuple-Based Windows

In a Type 1 application where the application must generate exactly the same sequence of output events as existing secondaries, all tuple-based windows must be qualified by time. See Choose an Adequate warm-up-window-length Time.

17.8.3.3 Partitioned Windows

Avoid partitioned windows because there are situations in which a partition cannot be rebuilt. If you do use partitioned windows, configure a warm-up-window-length time long enough to give the Oracle Stream Explorer server time to rebuild the partition. See Choose an Adequate warm-up-window-length Time.

17.8.3.4 Sliding Windows

Oracle CQL queries should not use sliding windows when new stages that join the multiserver domain are expected to generate exactly the same output events as existing stages. See Rejoining the High Availability MultiServer Domain.

17.8.3.5 DURATION Clause and Non-Event Detection

You must use application time when Oracle CQL queries contain a DURATION clause for non-event detection. See Prefer Application Time .

17.8.3.6 Prefer Application Time

In Oracle Stream Explorer each event is associated with a point in time at which the event occurred. Oracle CQL recognizes two types of time:

Application time: A time value assigned to each event outside of Oracle CQL by the application before the event enters the Oracle CQL processor.
System time: A time value associated with an event when it arrives at the Oracle CQL processor, essentially by calling System.nanoTime().

Application time is generally the best approach for applications that need to be highly available. The application time is associated with an event before the event is sent downstream, so it is consistent across active primary and secondary servers.

System time can cause application instances to generate different results because the time value associated with an event can be different on each server due to system clocks not being synchronized. You can use system time for applications when Oracle CQL queries do not use time-based windows. Applications that use only event-based windows depend only on the arrival order of events rather than the arrival time, so you can use system time in this case.

If you must use system time with Oracle CQL queries that do use time-based windows, then you must use a Oracle Stream Explorer high availability input adapter that intercepts incoming events and assigns a consistent time that spans primary and secondary servers.

17.9 Configure High Availability Quality of Service

You configure Oracle Stream Explorer high availability quality of service in the assembly and component configuration files. After you make Oracle Stream Explorer high availability configuration changes, you must redeploy your Oracle Stream Explorer application.

This section includes the following procedures:

17.9.1 Configure a Simple Failover

You configure simple failover with the Oracle Stream Explorer high availability buffering output adapter with a sliding window size of zero (0).

This procedure starts with the example EPN that Figure 17-9 shows and adds the required components to configure it for a simple failover.

Figure 17-9 Simple Failover EPN

Description of "Figure 17-9 Simple Failover EPN"

Configure a simple failover:

Create a multiserver domain using Oracle Coherence.
For more information, see Administering Oracle Stream Explorer.
Create an Oracle Stream Explorer application.
Edit the MANIFEST.MF file to add the following Import-Package entries:
- com.bea.wlevs.ede.api.cluster
- com.oracle.cep.cluster.hagroups
- com.oracle.cep.cluster.ha.adapter
- com.oracle.cep.cluster.ha.api

Configure your Oracle Stream Explorer application EPN assembly file to add an Oracle Stream Explorer high availability buffering output adapter as the following assembly file entries show.

Add a wlevs:adapter element with provider set to ha-buffering after channel helloworldOutputChannel.
Update the wlevs:listener element in channel helloworldOutputChannel to reference the ha-buffering adapter by its id.
Add a wlevs:listener element to the ha-buffering adapter that references the HelloWorldBean class.

<wlevs:event-type-repository>
  <wlevs:event-type type-name="HelloWorldEvent">            <wlevs:class>com.bea.wlevs.event.example.helloworld.HelloWorldEvent
    </wlevs:class>
  </wlevs:event-type>
</wlevs:event-type-repository>

<wlevs:adapter id="helloworldAdapter" 
  class="com.bea.wlevs.adapter.example.helloworld.HelloWorldAdapter" >
<wlevs:instance-property name="message" value="HelloWorld - 
  the current time is:"/>
</wlevs:adapter>

<wlevs:channel id="helloworldInputChannel" event-type="HelloWorldEvent" >
  <wlevs:listener ref="helloworldProcessor"/>
</wlevs:channel>

<wlevs:processor id="helloworldProcessor" />

<wlevs:channel id="helloworldOutputChannel" event-type="HelloWorldEvent"
  advertise="true">
  <wlevs:listener ref="myHaSlidingWindowAdapter"/>
  <wlevs:source ref="helloworldProcessor"/>
 </wlevs:channel>

 <wlevs:adapter id="myHaSlidingWindowAdapter" provider="ha-buffering" >
   <wlevs:listener>
     <bean class="com.bea.wlevs.example.helloworld.HelloWorldBean"/>
   </wlevs:listener>
 </wlevs:adapter>

Optionally, configure the channel downstream from the input adapter (helloworldInputChannel) to configure an application time stamp based on an appropriate event property as assembly file entries show.
For simple failover, you can use system time stamps because events are not correlated between servers. However, it is possible that slightly different results might be output from the buffer if application time stamps are not used.

In this example, event property arrivalTime is used.

The wlevs:expression should be set to this event property.
```
<wlevs:channel id="helloworldInputChannel" event-type="HelloWorldEvent" >
   <wlevs:listener ref="helloworldProcessor"/>
   <wlevs:source ref="myHaInputAdapter"/>
   <wlevs:application-timestamped>
      <wlevs:expression>arrivalTime</wlevs:expression>
   </wlevs:application-timestamped>
 </wlevs:channel>
```

Configure the Oracle Stream Explorer high availability buffering output adapter.

Set the instance property windowLength to zero (0) as shown.

<wlevs:adapter id="myHaSlidingWindowAdapter" provider="ha-buffering" >
   <wlevs:listener>
     <bean class="com.bea.wlevs.example.helloworld.HelloWorldBean"/>
   </wlevs:listener>
   <wlevs:instance-property name="windowLength" value="0"/>
</wlevs:adapter>

Optionally, configure the component configuration file to include the Oracle Stream Explorer high availability buffering output adapter as shown.

<processor>
  <name>helloworldProcessor</name>
  <rules>
    <query id="helloworldRule">
      <![CDATA[ select * from helloworldInputChannel [Now] >
    </query>
  </rules>
</processor>

<ha:ha-buffering-adapter >
  <name>myHaSlidingWindowAdapter</name>
  <window-length>0</window-length>
</ha:ha-buffering-adapter >

Deploy your application to the deployment group you created in step 1.
Oracle Stream Explorer automatically selects one of the Oracle Stream Explorer servers as the primary.

17.9.2 Configure Simple Failover With Buffering

You configure simple failover using the Oracle Stream Explorer buffering output adapter with a sliding window size greater than zero (0).

This procedure starts with the example EPN that Figure 17-10 shows and adds the required components to configure it for simple failover with buffering.

Figure 17-10 Simple Failover With Buffering EPN

Description of "Figure 17-10 Simple Failover With Buffering EPN"

Configure simple failover with buffering:

Create a multiserver domain using Oracle Coherence.
For more information, see Administering Oracle Stream Explorer.
Create an Oracle Stream Explorer application.
Edit the MANIFEST.MF file to add the following Import-Package entries:
- com.bea.wlevs.ede.api.cluster
- com.oracle.cep.cluster.hagroups
- com.oracle.cep.cluster.ha.adapter
- com.oracle.cep.cluster.ha.api

Configure your Oracle Stream Explorer application EPN assembly file to add an Oracle Stream Explorer high availability buffering output adapter as the following assembly file entries show.

Add a wlevs:adapter element with provider set to ha-buffering after channel helloworldOutputChannel.
Update the wlevs:listener element in channel helloworldOutputChannel to reference the ha-buffering adapter by its id.
Add a wlevs:listener element to the ha-buffering adapter that references the HelloWorldBean class.

<wlevs:event-type-repository>
  <wlevs:event-type type-name="HelloWorldEvent">
     <wlevs:class>com.bea.wlevs.event.example.helloworld.HelloWorldEvent</wlevs:class>
  </wlevs:event-type>
</wlevs:event-type-repository>

<wlevs:adapter id="helloworldAdapter" 
  class="com.bea.wlevs.adapter.example.helloworld.HelloWorldAdapter" >
  <wlevs:instance-property name="message" value="HelloWorld - the current time is:"/>
</wlevs:adapter>

<wlevs:channel id="helloworldInputChannel" event-type="HelloWorldEvent" >
  <wlevs:listener ref="helloworldProcessor"/>
</wlevs:channel>

<wlevs:processor id="helloworldProcessor" />

<wlevs:channel id="helloworldOutputChannel" event-type="HelloWorldEvent" advertise="true">
   <wlevs:listener ref="myHaSlidingWindowAdapter"/>
   <wlevs:source ref="helloworldProcessor"/>
</wlevs:channel>

<wlevs:adapter id="myHaSlidingWindowAdapter" provider="ha-buffering" >
  <wlevs:listener>
     <bean class="com.bea.wlevs.example.helloworld.HelloWorldBean"/>
  </wlevs:listener>
</wlevs:adapter>

Optionally, configure the channel downstream from the input adapter (helloworldInputChannel) to configure an application time stamp based on an appropriate event property as shown.
For simple failover with buffering, you can use system time stamps because events are not correlated between servers. However, it is possible that slightly different results might be output from the buffer if application time stamps are not used.

In this example, event property arrivalTime is used.

The wlevs:expression should be set to this event property.
```
<wlevs:channel id="helloworldInputChannel" event-type="HelloWorldEvent" >
  <wlevs:listener ref="helloworldProcessor"/>
  <wlevs:source ref="myHaInputAdapter"/>
  <wlevs:application-timestamped>
    <wlevs:expression>arrivalTime</wlevs:expression>
  </wlevs:application-timestamped>
</wlevs:channel>
```

Configure the Oracle Stream Explorer high availability buffering output adapter.

Set the instance property windowLength to a value greater than zero (0) as shown.

<wlevs:adapter id="myHaSlidingWindowAdapter" provider="ha-buffering" >
  <wlevs:listener>
    <bean class="com.bea.wlevs.example.helloworld.HelloWorldBean"/>
  </wlevs:listener>
     <wlevs:instance-property name="windowLength" value="15000"/>
</wlevs:adapter>

Optionally, configure the component configuration file to include the Oracle Stream Explorer high availability buffering output adapter as shown.

<processor>
  <name>helloworldProcessor</name>
  <rules>
    <query id="helloworldRule">
      <![CDATA[ select * from helloworldInputChannel [Now] >
    </query>
  </rules>
</processor>

<ha:ha-buffering-adapter >
  <name>myHaSlidingWindowAdapter</name>
  <window-length>15000</window-length>
</ha:ha-buffering-adapter >

If your application is an Oracle Stream Explorer high availability Type 1 application where the application must generate exactly the same sequence of output events as existing secondaries) configure the warm-up-window-length for the buffering output adapter.
For more information, see Choose an Adequate warm-up-window-length Time
Deploy your application to the deployment group you created in step 1.
Oracle Stream Explorer automatically selects one of the Oracle Stream Explorer servers as the primary.

17.9.3 Configure Light-Weight Queue Trimming

You configure light-weight queue trimming using the Oracle Stream Explorer high availability input adapter and the broadcast output adapter.

This procedure starts with the example EPN that Figure 17-11 shows and adds the required components to configure it for light-weight queue trimming.

Figure 17-11 Light-Weight Queue Trimming EPN

Description of "Figure 17-11 Light-Weight Queue Trimming EPN"

Configure light-weight queue trimming:

Create a multiserver domain using Oracle Coherence.
For more information, see Administering Oracle Stream Explorer.
Create an Oracle Stream Explorer application.
Edit the MANIFEST.MF file to add the following Import-Package entries:
- com.bea.wlevs.ede.api.cluster
- com.oracle.cep.cluster.hagroups
- com.oracle.cep.cluster.ha.adapter
- com.oracle.cep.cluster.ha.api

Configure your Oracle Stream Explorer application EPN assembly file to add an Oracle Stream Explorer high availability input adapter as the following example shows.

Add a wlevs:adapter element with provider set to ha-inbound after the regular input adapter helloworldAdapter.
Add a wlevs:listener element to the regular input adapter helloworldAdapter that references the ha-inbound adapter by its id.
Add a wlevs:source element to the helloworldInputChannel that references the ha-inbound adapter by its id.

    <wlevs:event-type-repository>
        <wlevs:event-type type-name="HelloWorldEvent">
            <wlevs:class>com.bea.wlevs.event.example.helloworld.HelloWorldEvent</wlevs:class>
  </wlevs:event-type>
</wlevs:event-type-repository>

<wlevs:adapter id="helloworldAdapter" 
  class="com.bea.wlevs.adapter.example.helloworld.HelloWorldAdapter" >
<wlevs:instance-property name="message" value="HelloWorld - 
  the current time is:"/>
   <wlevs:listener ref="myHaInputAdapter"/>
</wlevs:adapter>

<wlevs:adapter id="myHaInputAdapter" provider="ha-inbound" >
</wlevs:adapter>

<wlevs:channel id="helloworldInputChannel" event-type="HelloWorldEvent" >
  <wlevs:listener ref="helloworldProcessor"/>
  <wlevs:source ref="myHaInputAdapter"/>
</wlevs:channel>

<wlevs:processor id="helloworldProcessor" />

<wlevs:channel id="helloworldOutputChannel" event-type="HelloWorldEvent"
  advertise="true">
  <wlevs:listener>
    <bean class="com.bea.wlevs.example.helloworld.HelloWorldBean"/>
  </wlevs:listener>
    <wlevs:source ref="helloworldProcessor"/>
  </wlevs:channel>

Configure your Oracle Stream Explorer application EPN assembly file to add an Oracle Stream Explorer high availability broadcast output adapter as shown.

Add a wlevs:adapter element with provider set to ha-broadcast after channel helloworldOutputChannel.
Update the wlevs:listener element in channel helloworldOutputChannel to reference the ha-broadcast adapter by its id.
Add a wlevs:listener element to the ha-broadcast adapter that references the HelloWorldBean class.

<wlevs:event-type-repository>
  <wlevs:event-type type-name="HelloWorldEvent">
    <wlevs:class>com.bea.wlevs.event.example.helloworld.HelloWorldEvent
    </wlevs:class>
  </wlevs:event-type>
</wlevs:event-type-repository>

<wlevs:adapter id="helloworldAdapter" 
   class="com.bea.wlevs.adapter.example.helloworld.HelloWorldAdapter" >
 <wlevs:instance-property name="message" value="HelloWorld - 
    the current time is:"/>
  <wlevs:listener ref="myHaInputAdapter"/>
</wlevs:adapter>

<wlevs:adapter id="myHaInputAdapter" provider="ha-inbound" >
</wlevs:adapter>

<wlevs:channel id="helloworldInputChannel" event-type="HelloWorldEvent" >
  <wlevs:listener ref="helloworldProcessor"/>
  <wlevs:source ref="myHaInputAdapter"/>
</wlevs:channel>

<wlevs:processor id="helloworldProcessor" />

<wlevs:channel id="helloworldOutputChannel" event-type="HelloWorldEvent" 
  advertise="true">
  <wlevs:listener ref="myHaBroadcastAdapter"/>
  <wlevs:source ref="helloworldProcessor"/>
</wlevs:channel>

<wlevs:adapter id="myHaBroadcastAdapter" provider="ha-broadcast" >
  <wlevs:listener>
    <bean class="com.bea.wlevs.example.helloworld.HelloWorldBean"/>
  </wlevs:listener>
</wlevs:adapter>

Configure the Oracle Stream Explorer high availability input adapter.

Consider using one of the following example configurations.

<wlevs:adapter id="myHaInputAdapter" provider="ha-inbound" >
  <wlevs:instance-property name="timeProperty" value="arrivalTime"/>
</wlevs:adapter>

This example shows a high availability input adapter configuration using all defaults. The mandatory key is based on all event properties and the event property that the high availability input adapter assigns a time value to is an event property named arrivalTime. Because the events are tuple-based events, you must specify the event type (MyEventType) using the eventType property.

<wlevs:adapter id="myHaInputAdapter" provider="ha-inbound" >
   <wlevs:instance-property name="timeProperty" value="arrivalTime"/>
   <wlevs:instance-property name="eventType" value="MyEventType"/>
</wlevs:adapter>

This example shows a high availability input adapter configuration where the mandatory key is based on one event property (named id) and the event property that the high availability input adapter assigns a time value to is an event property named arrivalTime.

<wlevs:adapter id="myHaInputAdapter" provider="ha-inbound" >
  <wlevs:instance-property name="keyProperties" value="id"/>
  <wlevs:instance-property name="timeProperty" value="arrivalTime"/>
</wlevs:adapter>

This example shows a high availability input adapter configuration where the mandatory key is based on more than one event property (properties orderID and accountID) and the event property that the high availability input adapter assigns a time value to is an event property named arrivalTime.

<wlevs:adapter id="myHaInputAdapter" provider="ha-inbound" >
  <wlevs:instance-property name="timeProperty" value="arrivalTime"/>
  <wlevs:instance-property name="keyClass" value="com.acme.MyCompoundKeyClass"/>
</wlevs:adapter>

A compound key Java class (com.acme.MyCompoundKeyClass) is mandatory and its implementation is shown. The hashCode and equals methods are required. When you specify a keyClass, the keyProperties instance property is ignored: Oracle Stream Explorer assumes that the compound key is based on all the getter methods in the keyClass.

package com.acme;

public class MyCompoundKeyClass {
    private int orderID;
    private int accountID;

    public MyCompoundKeyClass() {}

    public int getOrderID() {
        return orderID;
    }
    public setOrderID(int orderID) {
        this.orderID = orderID;
    }
    public int getAccountID() {
        return accountID;
    }
    public setOrderID(int accountID) {
        this.accountID = accountID;
    }

    public int hashCode() {
    int hash = 1;
    hash = hash * 31 + orderID.hashCode();
    hash = hash * 31 + (accountID == null ? 0 : accountID.hashCode());
    return hash;
    }

    public boolean equals(Object obj) {
        if (obj == this) return true;
        if (obj == null) return false;
        if (!(obj instanceof MyCompoundKeyClass)) return false;
        MyCompoundKeyClass k = (MyCompoundKeyClass) obj;
        return k.accountID == accountID && k.orderID == orderID;
    }
}

Configure the channel downstream from the high availability input adapter (helloworldInputChannel) to configure an application time stamp based on the high availability input adapter timeProperty setting as the following example shows.

The wlevs:expression should be set to the timeProperty value.

<wlevs:adapter id="myHaInputAdapter" provider="ha-inbound" >
   <wlevs:instance-property name="keyProperties" value="id"/>
   <wlevs:instance-property name="eventType" value="HelloWorldEvent"/>
   <wlevs:instance-property name="timeProperty" value="arrivalTime"/>
</wlevs:adapter>

<wlevs:channel id="helloworldInputChannel" event-type="HelloWorldEvent" >
  <wlevs:listener ref="helloworldProcessor"/>
  <wlevs:source ref="myHaInputAdapter"/>
  <wlevs:application-timestamped>
    <wlevs:expression>arrivalTime</wlevs:expression>
  </wlevs:application-timestamped>
</wlevs:channel>

Configure the Oracle Stream Explorer high availability broadcast output adapter.

Consider using one of the following example configurations:

This example shows a broadcast output adapter configuration using all defaults. The mandatory key is based on all event properties, key values are non-monotonic (do not increase continually) and total order (unique).

<wlevs:adapter id="myHaSlidingWindowAdapter" provider="ha-broadcast" >
   <wlevs:listener>
      <bean class="com.bea.wlevs.example.helloworld.HelloWorldBean"/>
   </wlevs:listener>
</wlevs:adapter>

This example shows a broadcast output adapter configuration where the mandatory key is based on one event property (named timeProperty), key values are monotonic (they do increase continually) and not total order (not unique).

<wlevs:adapter id="myHaSlidingWindowAdapter" provider="ha-broadcast" >
   <wlevs:listener>
     <bean class="com.bea.wlevs.example.helloworld.HelloWorldBean"/>
   </wlevs:listener>
     <wlevs:instance-property name="keyProperties" value="timeProperty"/>
     <wlevs:instance-property name="monotonic" value="true"/>
     <wlevs:instance-property name="totalOrder" value="false"/>
  </wlevs:adapter>

This example shows a broadcast output adapter configuration where the mandatory key is based on more than one event property (properties timeProperty and accountID), key values are monotonic (they do increase continually) and total order (unique).

<wlevs:adapter id="myHaSlidingWindowAdapter" provider="ha-broadcast" >
 <wlevs:listener>
    <bean class="com.bea.wlevs.example.helloworld.HelloWorldBean"/>
  </wlevs:listener>
  <wlevs:instance-property name="keyClass" value="com.acme.MyCompoundKeyClass"/>
  <wlevs:instance-property name="monotonic" value="true"/>
  <wlevs:instance-property name="totalOrder" value="true"/>
</wlevs:adapter>

A compound key Java class (com.acme.MyCompoundKeyClass) is mandatory and its implementation is shown in the following example. The hashCode and equals methods are required. When you specify a keyClass, the keyProperties instance property is ignored: Oracle Stream Explorer assumes that the compound key is based on all the getter methods in the keyClass.

package com.acme;

public class MyCompoundKeyClass {
    private int timeProperty;
    private int accountID;

    public MyCompoundKeyClass() {}

    public int getTimeProperty() {
        return orderID;
    }
    public setTimeProperty(int timeProperty) {
        this.timeProperty = timeProperty;
    }
    public int getAccountID() {
        return accountID;
    }
    public setOrderID(int accountID) {
        this.accountID = accountID;
    }

    public int hashCode() {
    int hash = 1;
    hash = hash * 31 + timeProperty.hashCode();
    hash = hash * 31 + (accountID == null ? 0 : accountID.hashCode());
    return hash;
    }

    public boolean equals(Object obj) {
        if (obj == this) return true;
        if (obj == null) return false;
        if (!(obj instanceof MyCompoundKeyClass)) return false;
        MyCompoundKeyClass k = (MyCompoundKeyClass) obj;
        return k.accountID == accountID && k.orderID == orderID;
    }
}

Optionally, configure the component configuration file to include the Oracle Stream Explorer high availability input adapter and buffering output adapter as shown.

<processor>
  <name>helloworldProcessor</name>
  <rules>
    <query id="helloworldRule">
      <![CDATA[ select * from helloworldInputChannel [Now] >
    </query>
  </rules>
</processor>

<ha:ha-inbound-adapter>
 <name>myHaInputAdapter</name>
</ha:ha-inbound-adapter>
 
<ha:ha-broadcast-adapter>
  <name>myHaBroadcastAdapter</name>
  <trimming-interval units="events">10</trimming-interval>
</ha:ha-broadcast-adapter>

If your application is an Oracle Stream Explorer high availability Type 1 application where the application must generate exactly the same sequence of output events as existing secondaries, configure the warm-up-window-length for the broadcast output adapter.
Oracle Stream Explorer automatically selects one of the Oracle Stream Explorer servers as the primary.

17.9.4 Configure Precise Recovery With JMS

You configure precise recovery with JMS using the Oracle Stream Explorer high availability input adapter and correlating output adapter.

This procedure describes how to create the example EPN that Figure 17-12 shows. For more information about this Oracle Stream Explorer high availability quality of service, see Precise Recovery with JMS.

Note:

The JMS destination used by JMS adapters for precise recovery must be topics, rather than queues.

Figure 17-12 Precise Recovery With JMS EPN

Description of "Figure 17-12 Precise Recovery With JMS EPN"

Configure precise recovery with JMS:

Create a multiserver domain using Oracle Coherence.
For more information, see Administering Oracle Stream Explorer.
Create an Oracle Stream Explorer application.
Edit the MANIFEST.MF file to add the following Import-Package entries:
- com.bea.wlevs.ede.api.cluster
- com.oracle.cep.cluster.hagroups
- com.oracle.cep.cluster.ha.adapter
- com.oracle.cep.cluster.ha.api

Configure your Oracle Stream Explorer application EPN assembly file to add an Oracle Stream Explorer high availability input adapter as shown:

Add a wlevs:adapter element with provider set to ha-inbound after the regular input adapter JMSInboundAdapter.
Add a wlevs:listener element to the regular input adapter JMSInboundAdapter that references the ha-inbound adapter by its id.
Add a wlevs:source element to the channel channel1 that references the ha-inbound adapter by its id.

    <wlevs:event-type-repository>
        <wlevs:event-type type-name="StockTick">
            <wlevs:properties>
                <wlevs:property name="lastPrice" type="double" />
                <wlevs:property name="symbol" type="char" />
            </wlevs:properties>
        </wlevs:event-type>
    </wlevs:event-type-repository>

    <wlevs:adapter id="JMSInboundAdapter" provider="jms-inbound">
        <wlevs:listener ref="myHaInputAdapter"/>
    </wlevs:adapter>

    <wlevs:adapter id="myHaInputAdapter" provider="ha-inbound" >
    </wlevs:adapter>

    <wlevs:channel id="channel1" event-type="StockTick">
        <wlevs:listener ref="processor1" />
        <wlevs:source ref="myHaInputAdapter"/>
    </wlevs:channel>

Configure your Oracle Stream Explorer application EPN assembly file to add an Oracle Stream Explorer high availability correlating output adapter as shown.

Add a wlevs:adapter element with provider set to ha-correlating after channel channel2.
Update the wlevs:listener element in channel channel2 to reference the ha-correlating adapter by its id.
Add a wlevs:listener element to the ha-correlating adapter that references the regular output adapter JMSOutboundAdapter.

<wlevs:event-type-repository>
  <wlevs:event-type type-name="StockTick">
    <wlevs:properties>
      <wlevs:property name="lastPrice" type="double" />
      <wlevs:property name="symbol" type="char" />
    </wlevs:properties>
  </wlevs:event-type>
</wlevs:event-type-repository>

<wlevs:adapter id="JMSInboundAdapter" provider="jms-inbound">
  <wlevs:listener ref="myHaInputAdapter"/>
</wlevs:adapter>

<wlevs:adapter id="myHaInputAdapter" provider="ha-inbound" >
</wlevs:adapter>

<wlevs:channel id="channel1" event-type="StockTick">
  <wlevs:listener ref="processor1" />
  <wlevs:source ref="myHaInputAdapter"/>
</wlevs:channel>

<wlevs:processor id="processor1">
  <wlevs:listener ref="channel2" />
</wlevs:processor>

<wlevs:channel id="channel2" event-type="StockTick">
  <wlevs:listener ref="myHaCorrelatingAdapter" />
</wlevs:channel>

<wlevs:adapter id="myHaCorrelatingAdapter" provider="ha-correlating" >
  <wlevs:listener ref="JMSOutboundAdapter"/>
</wlevs:adapter>

<wlevs:adapter id="JMSOutboundAdapter" provider="jms-outbound">
</wlevs:adapter>

Configure the Oracle Stream Explorer high availability input adapter.

Consider using one of the following example configurations:

<wlevs:adapter id="myHaInputAdapter" provider="ha-inbound" >
  <wlevs:instance-property name="timeProperty" value="arrivalTime"/>
</wlevs:adapter>

<wlevs:adapter id="myHaInputAdapter" provider="ha-inbound" >
  <wlevs:instance-property name="timeProperty" value="arrivalTime"/>
  <wlevs:instance-property name="eventType" value="MyEventType"/>
</wlevs:adapter>

This example shows a high availability input adapter configuration where the mandatory key is based on one event property (named sequenceNo) and the event property that the high availability input adapter assigns a time value to is an event property named inboundTime.

<wlevs:adapter id="myHaInputAdapter" provider="ha-inbound" >
  <wlevs:instance-property name="keyProperties" value="sequenceNo"/>
  <wlevs:instance-property name="timeProperty" value="inboundTime"/>
</wlevs:adapter>

<wlevs:adapter id="myHaInputAdapter" provider="ha-inbound" >
  <wlevs:instance-property name="timeProperty" value="arrivalTime"/>
  <wlevs:instance-property name="keyClass" value="com.acme.MyCompoundKeyClass"/>
</wlevs:adapter>

package com.acme;

public class MyCompoundKeyClass {
    private int orderID;
    private int accountID;

    public MyCompoundKeyClass() {}

    public int getOrderID() {
        return orderID;
    }
    public setOrderID(int orderID) {
        this.orderID = orderID;
    }
    public int getAccountID() {
        return accountID;
    }
    public setOrderID(int accountID) {
        this.accountID = accountID;
    }

    public int hashCode() {
    int hash = 1;
    hash = hash * 31 + orderID.hashCode();
    hash = hash * 31 + (accountID == null ? 0 : accountID.hashCode());
    return hash;
    }

    public boolean equals(Object obj) {
        if (obj == this) return true;
        if (obj == null) return false;
        if (!(obj instanceof MyCompoundKeyClass)) return false;
        MyCompoundKeyClass k = (MyCompoundKeyClass) obj;
        return k.accountID == accountID && k.orderID == orderID;
    }
}

Configure the channel downstream from the high availability input adapter (channel1) to configure an application time stamp based on the high availability input adapter timeProperty setting as the following example shows.

The wlevs:expression should be set to the timeProperty value.

<wlevs:adapter id="myHaInputAdapter" provider="ha-inbound" >
  <wlevs:instance-property name="eventType" value="HelloWorldEvent"/>
  <wlevs:instance-property name="keyProperties" value="sequenceNo"/>
  <wlevs:instance-property name="timeProperty" value="inboundTime"/>
</wlevs:adapter>

<wlevs:channel id="channel1" event-type="StockTick">
  <wlevs:listener ref="processor1" />
  <wlevs:source ref="myHaInputAdapter"/>
  <wlevs:application-timestamped>
    <wlevs:expression>inboundTime</wlevs:expression>
  </wlevs:application-timestamped>
</wlevs:channel>

Configure the Oracle Stream Explorer high availability correlating output adapter failOverDelay.

The following example shows a correlating output adapter configuration where the failOverDelay is 2000 milliseconds.

<wlevs:adapter id="myHaCorrelatingAdapter" provider="ha-correlating" >
  <wlevs:listener ref="JMSOutboundAdapter"/>
  <wlevs:instance-property name="failOverDelay" value="2000"/>
</wlevs:adapter>

Create a second regular JMS input adapter.

The following example shows a JMS adapter named JMSInboundAdapter2.

    <wlevs:adapter id="JMSInboundAdapter2" provider="jms-inbound">
    </wlevs:adapter>

The following JMS input adapter must be configured identically to the first JMS input adapter (in this example, JMSInboundAdapter). The following example shows the component configuration file for both the JMS input adapters. Note that both have exactly the same configuration, including the same provider.

    <jms-adapter>
        <name>JMSInboundAdapter</name>
        <jndi-provider-url>t3://localhost:7001</jndi-provider-url>
        <destination-jndi-name>./Topic1</destination-jndi-name>
        <user>weblogic</user>
        <password>weblogic</password>
        <work-manager>JettyWorkManager</work-manager>
        <concurrent-consumers>1</concurrent-consumers>
    </jms-adapter>

    <jms-adapter>
        <name>JMSInboundAdapter2</name>
        <jndi-provider-url>t3://localhost:7001</jndi-provider-url>
        <destination-jndi-name>./Topic2</destination-jndi-name>
        <user>weblogic</user>
        <password>weblogic</password>
        <work-manager>JettyWorkManager</work-manager>
        <concurrent-consumers>1</concurrent-consumers>
    </jms-adapter>
    ...
</wlevs:config>

Create a channel to function as the correlated source.

You must configure this channel with the second regular JMS input adapter as its source.

The following example shows a correlated source named clusterCorrelatingOutstream whose source is JMSInboundAdapter2.

    <wlevs:adapter id="JMSInboundAdapter2" provider="jms-inbound">
    </wlevs:adapter>

    <wlevs:channel id="clusterCorrelatingOutstream" event-type="StockTick" advertise="true">
        <wlevs:source ref="JMSInboundAdapter2"/>
    </wlevs:channel>

Configure the Oracle Stream Explorer high availability correlating output adapter with the correlatedSource.

The following example shows a correlating output adapter configuration where the correlatedSource is clusterCorrelatingOutstream.

<wlevs:adapter id="myHaCorrelatingAdapter" provider="ha-correlating" >
  <wlevs:listener ref="JMSOutboundAdapter"/>
  <wlevs:instance-property name="failOverDelay" value="2000"/>
  <wlevs:instance-property name="correlatedSource"       ref="clusterCorrelatingOutstream"/>
</wlevs:adapter>

If your application is an Oracle Stream Explorer high availability Type 1 application where the application must generate exactly the same sequence of output events as existing secondaries, configure the warm-up-window-length for the correlating output adapter.

Configure the component configuration file to enable session-transacted for both inbound JMS adapters and the outbound JMS adapter as the following example shows:

    <jms-adapter>
        <name>JMSInboundAdapter</name>
        <jndi-provider-url>t3://localhost:7001</jndi-provider-url>
        <destination-jndi-name>./Topic1</destination-jndi-name>
        <user>weblogic</user>
        <password>weblogic</password>
        <work-manager>JettyWorkManager</work-manager>
        <concurrent-consumers>1</concurrent-consumers>
        <session-transacted>true</session-transacted>
    </jms-adapter>

    <jms-adapter>
        <name>JMSInboundAdapter2</name>
        <jndi-provider-url>t3://localhost:7001</jndi-provider-url>
        <destination-jndi-name>./Topic2</destination-jndi-name>
        <user>weblogic</user>
        <password>weblogic</password>
        <work-manager>JettyWorkManager</work-manager>
        <concurrent-consumers>1</concurrent-consumers>
        <session-transacted>true</session-transacted>
    </jms-adapter>
    ...
    <jms-adapter>
        <name>JMSOutboundAdapter</name>
        <event-type>JMSEvent</event-type>
        <jndi-provider-url>t3://localhost:7001</jndi-provider-url>
        <destination-jndi-name>./Topic2</destination-jndi-name>
        <delivery-mode>nonpersistent</delivery-mode>
        <session-transacted>true</session-transacted>
    </jms-adapter>
    ...
</wlevs:config>

Optionally, configure the component configuration file to include the Oracle Stream Explorer high availability input adapter and correlating output adapter as shown.

    <ha:ha-inbound-adapter>
        <name>myHaInputAdapter</name>
    </ha:ha-inbound-adapter>
    ...
    <ha:ha-correlating-adapter>
        <name>myHaBroadcastAdapter</name>
        <fail-over-delay>2000</fail-over-delay>
    </ha:ha-correlating-adapter>

Optionally, add an ActiveActiveGroupBean to your EPN to improve scalability.
For more information, see Partition an Incoming JMS Event Stream.
Oracle Stream Explorer automatically selects one of the Oracle Stream Explorer servers as the primary.

17.10 Configure High Availability Adapters

You configure Oracle Stream Explorer high availability adapters in the EPN assembly file and component configuration files, similar to how you configure other components in the EPN, such as channels or processors.

After making any Oracle Stream Explorer high availability configuration changes, you must redeploy your Oracle Stream Explorer application. See Deploy an OSGi Bundle.

This section includes the following procedures:

17.10.1 Configure the High Availability Input Adapter

The Oracle Stream Explorer high availability broadcast input adapter is implemented by the BroadcastInputAdapter interface.

Assembly File

The root element to declare an Oracle Stream Explorer high availability input adapter is wlevs:adapter with the provider element set to ha-inbound. You also specify a wlevs:listener element for the Oracle Stream Explorer high availability input adapter in the input adapter.

<wlevs:adapter id="jmsAdapter" provider="jms-inbound"
    <wlevs:listener ref="myHaInputAdapter"/>	
</wlevs:adapter>

<wlevs:adapter id="myHaInputAdapter" provider="ha-inbound">
    <wlevs:instance-property name="keyProperties" value="id"/>
    <wlevs:instance-property name="timeProperty" value="arrivalTime"/>
    <wlevs:instance-property name="eventType" value="MyEventType"/>
</wlevs:adapter>

<wlevs:channel id="inputChannel" event-type="MyEventType ">
    <wlevs:source ref="myHaInputAdapter"/>
    <wlevs:application-timestamped>
        <wlevs:expression>arrivalTime</wlevs:expression>
    </wlevs:application-timestamped>
</wlevs:channel>

Table 17-3 describes the additional child element.

Table 17-3 Child Elements of wlevs:adapter for the High Availability Input Adapter

Child Element	Description
`wlevs:instance-property`	Specify one or more `instance-property` element `name` and `value` attributes.

Table 17-4 lists the supported instance properties with their name and value attributes.

Table 17-4 High Availability Input Adapter Instance Properties

Name	Value
`timeProperty`	Specify the name of the event property to which the high availability input adapter assigns a time value. This is the same property that you use in the `wlevs:application-timestamped` element of the downstream EPN component to which the high availability input adapter is connected.
`keyProperties`	Specify a space delimited list of one or more event properties that the Oracle Stream Explorer high availability input adapter uses to identify event instances. If you specify more than one event property, you must specify a keyClass. Default: all event properties.
`keyClass`	Specify the fully qualified Java class name of to use for the compound key. By default, all JavaBean properties in the `keyClass` are assumed to be `keyProperties`, unless the `keyProperties` setting is used.
`eventType`	Specify the type name of the events that the Oracle Stream Explorer high availability input adapter receives from the actual input adapter. This is the same event type that you use in the downstream EPN component to which the high availability input adapter is connected. For tuple events, this property is mandatory. For all other Java class-based event types, this property is optional.

Configuration File

The root element for configuring an Oracle Stream Explorer high availability input adapter is ha-inbound-adapter. The name child element must match the id attribute of the corresponding wlevs:adapter element in the assembly file as shown.

<ha:ha-inbound-adapter>
    <name>myHaInputAdapter</name>
    <heartbeat units="millis">1000</heartbeat>
    <batch-size>10</batch-size>
</ha:ha-inbound-adapter>

Table 17-5 describes the additional child elements.

Table 17-5 Child Elements

Child Element Description

heartbeat

Specify the length of time that the high availability input adapter can be idle before it generates a heartbeat event to advance time.

Valid integer values for the units attribute:

nanos: wait the specified number of nanoseconds.
millis: wait the specified number of milliseconds.
secs: wait the specified number of seconds.

Default: Heartbeats are not sent.

batch-size

Specify the number of events in each timing message that the primary broadcasts to its secondaries. A value of n means that n {key, time} pairs are sent in each message. You can use this property for performance tuning.

Default: 1 (disable batching).

17.10.2 Configure the Buffering Output Adapter

The Oracle Stream Explorer high availability buffering output adapter is implemented by the SlidingWindowQueueTrimmingAdapter interface.

Assembly File

The root element for declaring an Oracle Stream Explorer high availability buffering output adapter is wlevs:adapter with provider element set to ha-buffering as the following example shows.

<wlevs:adapter id="mySlidingWindowingAdapter" provider ="ha-buffering">
  <wlevs:listener>
     <bean class="com.bea.wlevs.example.cluster.ClusterAdapterBean"/>
  </wlevs:listener>
  <wlevs:instance-property name="windowLength" value="15000"/>
</wlevs:adapter>

Table 17-6 describes the additional child elements.

Table 17-6 Child Elements

Child Element	Description
`wlevs:listener`	Specify the regular output adapter downstream from this Oracle Stream Explorer high availability buffering output adapter.
`wlevs:instance-property`	Specify one or more `instance-property` element `name` and `value` attributes as Table 17-7 describes.

Table 17-7 lists the instance properties.

Table 17-7 Instance Properties

Name Value

windowLength

Specify the sliding window size as an integer number of milliseconds.

Default: 15000.

Configuration File

The root element for configuring an Oracle Stream Explorer high availability buffering output adapter is ha-buffering-adapter. The name child element for a particular adapter must match the id attribute of the corresponding wlevs:adapter element in the assembly file as shown.

<ha:ha-buffering-adapter >
    <name>mySlidingWindowingAdapter</name>
    <window-length>15000</window-length>
    <warm-up-window-length units="minutes">6</warm-up-window-length>
</ha:ha-buffering-adapter >

Table 17-8 describes the additional child elements of ha-buffering-adapter you can configure for an Oracle Stream Explorer high availability buffering output adapter.

Table 17-8 Child Elements

Child Element Description

window-length

Specify the sliding window size as an integer number of milliseconds.

Default: 15000.

warm-up-window-length

Specify the length of time that the high availability input adapter can be idle before it generates a heartbeat event to advance tim

Valid integer values for the units attribute:

seconds: wait the specified number of seconds.
minutes: wait the specified number of minutes.

Default: units is seconds.

17.10.3 Configure the Broadcast Output Adapter

The Oracle Stream Explorer high availability broadcast output adapter is implemented by the GroupBroadcastQueueTrimmingAdapter class.

Assembly File

The root element to declare an Oracle Stream Explorer high availability broadcast output adapter is wlevs:adapter with the provider element set to ha-broadcast as shown.

<wlevs:adapter id="myBroadcastAdapter" provider="ha-broadcast">
    <wlevs:listener ref="actualAdapter"/>
    <wlevs:instance-property name="keyProperties" value="time"/>
    <wlevs:instance-property name="monotonic" value="true"/>
</wlevs:adapter>

Table 17-9 describes the additional child elements.

Table 17-9 Child Elements

Child Element	Description
`wlevs:listener`	Specify the regular output adapter downstream from this Oracle Stream Explorer high availability broadcast output adapter.
`wlevs:instance-property`	Specify one or more `instance-property` element `name` and `value` attributes as Table 17-10 describes.

Table 17-10 lists the instance properties.

Table 17-10 Instance Properties

Name	Value
`keyProperties`	Specify a space delimited list of one or more event properties that the Oracle Stream Explorer high availability broadcast output adapter uses to identify event instances. If you specify more than one event property, you must specify a keyClass. Default: all event properties.
`keyClass`	Specify the fully qualified class name of a Java class used as a compound key. By default, all JavaBean properties in the `keyClass` are assumed to be `keyProperties`, unless the `keyProperties` setting is used. A compound key may be monotonic and may be totalOrder.
`monotonic`	Specify whether the key value is constantly increasing (like a time value). Valid values are: `true`: the key is constantly increasing. `false`: the key is not constantly increasing. Default: `false`.
`totalOrder`	Specify whether event keys are unique. Applicable only when instance property monotonic is set to `true`. Valid values are: `true`: event keys are unique. `false`: event keys are not unique. Default: `true`.

Configuration File

The root element for configuring an Oracle Stream Explorer high availability broadcast output adapter is ha-broadcast-adapter. The name child element for a particular adapter must match the id attribute of the corresponding wlevs:adapter element in the EPN assembly file that declares this adapter as shown.

<ha:ha-broadcast-adapter>
    <name>myBroadcastAdapter</name>
    <trimming-interval units="events">10</trimming-interval>
    <warm-up-window-length units="minutes">6</warm-up-window-length>
</ha:ha-broadcast-adapter>

Table 17-11 describes the additional child elements.

Table 17-11 Child Elements

Child Element Description

trimming-interval

Specify the interval at which trimming messages are broadcast as an integer number of units. You can use this property for performance tuning (see High Availability Performance Tuning).

Valid values for attribute units:

events: broadcast trimming messages after the specified number of events are processed.
millis: broadcast trimming messages after the specified number milliseconds.

Default: units is events.

warm-up-window-length

Specify the length of time it takes the application to rebuild state after a previously failed secondary restarts or a new secondary is added.

Valid integer values for the units attribute:

seconds: wait the specified number of seconds.
minutes: wait the specified number of minutes.

Default: units is seconds.

For more information, see Choose an Adequate warm-up-window-length Time.

17.10.4 Configure the Correlating Output Adapter

The Oracle Stream Explorer high availability correlating output adapter is implemented by the CorrelatedQueueTrimmingAdapter interface.

Assembly File

The root element to declare an Oracle Stream Explorer high availability correlating output adapter is wlevs:adapter with the provider element set to ha-correlating as shown.

<wlevs:adapter id="myCorrelatingAdapter" provider="ha-correlating">
    <wlevs:listener>
        <bean class="com.bea.wlevs.example.cluster.ClusterAdapterBean"/>
    </wlevs:listener>
    <wlevs:instance-property name="correlatedSource" ref="clusterCorrOutstream"/>
    <wlevs:instance-property name="failOverDelay" value="2000"/>
</wlevs:adapter>

Table 17-12 describes the additional child elements.

Table 17-12 Child Elements

Child Element	Description
`wlevs:listener`	Specify the regular output adapter downstream from this Oracle Stream Explorer high availability buffering output adapter.
`wlevs:instance-property`	Specify one or more `instance-property` element `name` and `value` attributes as Table 17-13 describes.

Table 17-13 lists the instance properties.

Table 17-13 Instance Properties

Name Value

correlatedSource

Specify the event source to correlate against. Events seen from this source are purged from the trimming queue. Events still in the queue at failover are replayed.

failOverDelay

Specify the delay timeout in milliseconds that is used to decide how soon after failover correlation should restart.

Default: 0 ms.

Configuration File

The root element for configuring an Oracle Stream Explorer high availability correlating output adapter is ha-correlating-adapter. The name child element for a particular adapter must match the id attribute of the corresponding wlevs:adapter element in the EPN assembly file that declares this adapter as shown.

<ha:ha-correlating-adapter>
    <name>myCorrelatingAdapter</name>
    <window-length>15000</window-length>
    <warm-up-window-length units="minutes">6</warm-up-window-length>
</ha:ha-correlating-adapter>

Table 17-14 describes the child elements.

Table 17-14 Child Elements

Child Element	Description
`fail-over-delay`	Specify the delay timeout in milliseconds that is used to decide how soon after failover correlation should restart. Default: 0 ms.
`warm-up-window-length`	Specify the length of time it takes the application to rebuild state after a previously failed secondary restarts or a new secondary is added as an integer number of `units`. Valid values for attribute `units`: `seconds`: wait the specified number of seconds. `minutes`: wait the specified number of minutes. Default: `units` is `seconds`. For more information, see Choose an Adequate warm-up-window-length Time.
window-length	The length of the saved buffer of events in milliseconds.
trimming-interval	The interval at which events should be trimmed from a secondary buffer. Units can be `events` or `millis`.
heartbeat	The value (n) for the heartbeat time out on this adapter. A heartbeat is generated when `n` time units go by without any event being generated on this adapter. The default time unit is nanoseconds.
batch-size	The batch size in terms of events for sending event time stamps to the secondary. By default, batching is disabled.