1 Overview of Oracle CEP

This section contains information on the following subjects:

1.1 Introduction to Oracle Complex Event Processing

Oracle CEP (formally known as the WebLogic Event Server) is a Java server for the development and deployment of high-performance event driven applications. It is a lightweight Java application container based on Equinox OSGi, with shared services, including the Oracle CEP Service Engine, which provides a rich, declarative environment based on Oracle Continuous Query Language (Oracle CQL) - a query language based on SQL with added constructs that support streaming data - to improve the efficiency and effectiveness of managing business operations. Oracle CEP supports ultra-high throughput and microsecond latency using JRockit Real Time and provides Oracle CEP Visualizer and Oracle CEP IDE for Eclipse developer tooling for a complete real time end-to-end Java Event-Driven Architecture (EDA) development platform.

Oracle CEP has the capability of deploying user Java code (POJOs) which contain the business logic. Running the business logic within Oracle CEP provides a highly tuned framework for time and event driven applications.

1.2 Conceptual Overview of Oracle CEP

Figure 1-1 provides a high level view of an event-driven system.

Figure 1-1 Example Event -Driven System

Description of Figure 1-1 follows
Description of "Figure 1-1 Example Event -Driven System"

An event-driven system is generally comprised of several event sources, the real-time event-driven applications, and event sinks. Oracle CEP server and the Oracle CEP applications you deploy to it comprises the event-driven applications. The event sources generate streams of ordinary event data. The Oracle CEP applications listen to the event streams, process these events, and generate notable events. Event sinks receive the notable events.

Event sources, event-driven applications, and event sinks are de-coupled from each other; one can add or remove any of these components without causing changes to the other components. This is a key attribute of event-driven architectures.

Event-driven applications are rule-driven. In Oracle CEP, rules are expressed as queries using the Oracle Continuous Query Language (Oracle CQL). These queries are persisted to a data store and are used for processing the inbound stream of events and generating the outbound stream of events. Queries typically perform filtering and aggregation functions to discover and extract notable events from the inbound event streams. As a result, the number of outbound events is generally much lower than that of the inbound events.

Oracle CEP is a middleware for the development of event-driven applications. An Oracle CEP application is essentially an event-driven application.

Next, consider the application itself, which is hosted by the Oracle CEP server, a light-weight container as shown in Figure 1-2.

Figure 1-2 Oracle CEP Application

Description of Figure 1-2 follows
Description of "Figure 1-2 Oracle CEP Application"

An Oracle CEP application is typically composed of the following main component types:

  • Adapters interface directly to the inbound and outbound stream and relation sources and sinks. Adapters understand the inbound and outbound protocol, and are responsible for converting the event data into a normalized form that can be queried by a processor. Adapters forward the normalized event data into channels or outbound stream and relation sinks.

  • Channels are event processing endpoints. Among other things, streams are responsible for queuing event data until the event processing agent can act upon it.

  • Processors (or event processing agents) consume normalized event data from a channel, process it using queries, and may generate new events to an output channel.

  • Beans register to listen to the output channel, and are triggered by the insertion of a new event into the output channel. This user code is generally a plain-old-Java-object (POJO). The user application makes use of a set of external services, such as JMS, Web services, and file writers, to forward the generated events to external event sinks.

  • Event Beans register to listen to the output channel, and is triggered by the insertion of a new event into the output channel. This user code uses the Oracle CEP event bean API so that the bean can be managed by Oracle CEP.

1.3 Event Processing Networks

Adapters, channels, processors, and business logic POJOs can be connected arbitrarily to each other, forming event processing networks (EPN). Examples of topologies of EPNs are:

  • Adapter - Channel - Business Logic POJO

    Scenario: no processing is needed; only adaptation from proprietary protocol to some normalized model.

  • Adapter - Channel - Processor - Channel - Business Logic POJO

    Scenario: straight through processing to user code.

  • Adapter - Channel - Processor - Channel - Business Logic POJO - Channel - Processor - Channel - Business Logic POJO

    Scenario: two layers of event processing; the first processor creates causality between events and the second processor aggregates events into complex (notable) events.

EPNs have two important attributes:

  • Event processing networks can be used to create a hierarchy of processing agents, and thus achieve very complex processing of events. Each layer of the EPN aggregates events of its layer into complex events that become simple events in the layer above it.

  • Event processing networks improve integrability, that is, the quality of having separately developed components work correctly together. For example, one can add user code and reference to external services at several places in the network.

1.4 Use Cases

The use cases for Oracle CEP span a variety of businesses and applications. Just a few of these diverse use cases include:

  • Financial: Algorithmic Trading

    Automate stock trading based on market movement. Sample query: if, within any 20 second window, StockB rises by more than 2% and StockA does not, then automatically buy StockA.

    For an example, see Section 3.8, "Signal Generation Example".

  • Transportation: Security and Fraud Detection

    Discover fraudulent activity by detecting patterns among events. Sample query: if a single ID card is used twice in less than 5 seconds to gain access to a city's subway system, alert security for piggybacking.

  • Energy and Telecommunications: Alarm Correlation

    Reduce false positive alarms. Sample query: When 15 alarms are received within any 5 second window, but less than 5 similar alarms detected within 30 seconds, then do nothing.

  • Health Care: Patient Monitoring

    Monitor the vital signs of a patient and perform some task if a particular event happens. Sample query: When a change in medication is followed by a rise in blood pressure within 20% of maximum allowable for this patient within any 10 second window, alert nearest nurse.

1.5 Summary of Oracle CEP Features

The following list summarizes the main features of Oracle CEP:

  • New in Release 11gR1 (11.1.1): Oracle Continuous Query Language (Oracle CQL) is a query language based on SQL with added constructs that support streaming data. Using Oracle CQL, you can express queries on data streams to perform complex event processing (CEP) using Oracle CEP. Oracle CQL is scalable and comprehensive. It provides a wide range of operators (including extensive window operators), functions (including built-in, Colt, and java.lang.Math functions), and statements. Oracle CQL supersedes EPL.

  • New in Release 11gR1 (11.1.1): Support for Coherence clustering and distributed cache.

  • New in Release 11gR1 (11.1.1): Oracle CEP Visualizer query constructor, query plan generator, and support for Coherence clustering and distributed cache.

  • Event Caching—Applications can optionally publish or consume events to and from a cache to increase the availability of the events and increase the performance of their applications.

  • Event Record and Playback—The event repository feature of Oracle CEP allows you to record events flowing through an EPN and store them so you can later play back the events.

  • Built-in HTTP Publish-Subscribe Adapters—The three built-in HTTP publish-subscribe adapters allow an application to easily to publish (locally and remotely) and subscribe to an HTTP publish-subscribe server channel.

  • Built-in JMS Adapters—The two JMS adapters (inbound and outbound) allow you to send and receive messages to and from a JMS queue, respectively, from your application without writing any Java code

  • Oracle CEP Visualizer—A Web 2.0 application that consumes data Oracle CEP, displays it in a useful and intuitive way to system administrators and operators, and, for specified tasks, accepts data that is then passed back to Oracle CEP so as to change it configuration

  • Multi-server domains (sometimes referred to as clustering)—Oracle CEP now allows multiple servers to be logically connected together for the purposes of management, and physically connected using a shared User Datagram Protocol (UDP) multicast address and port.

  • An application server that supports deployment of Plain Old Java applications (POJOs), or Spring applications, for handling large volumes of streaming data with low latency requirements.

  • Oracle CEP applications are developed and deployed as event driven applications, that is, a set of custom Spring tags is used to define the event processing network in the EPN assembly file, which extends the standard Spring context file, of your application.

  • The application server contains a set of real time services that include a complex event processor (CEP), adapters, and streams. The server is highly tuned for high message throughput and low latency and deterministic behavior.

  • The complex event processor is a high performance, continuous query engine for processing high volumes of streaming data. It has full support for filtering, correlation, and aggregation of streaming data from one or more streams.

  • The Event Processing Language (EPL), an SQL-like language that allows event data from streams to be declaratively filtered, correlated, aggregated, and merged, with the ability to insert results into other streams for further downstream processing. You define the EPL rules either in an XML file that configures the complex event processor or programmatically using APIs. Oracle CQL supersedes EPL.

  • An Adapter SDK that provides all the tools you need to create adapters that listen to incoming data feeds.

  • A set of product samples that show both a simple Hello World scenario to get you started and more complex foreign exchange and algorithmic trading scenarios to showcase additional features of Oracle CEP.

  • A load generator utility that simulates a data feed, useful for testing your application without needing to connect to a live data feed.

  • A monitoring service that includes pre-built instrumentation for measuring throughput and latency at the component level.

  • A static and dynamic configuration framework. Static configuration is performed using XML files; dynamic configuration is performed by accessing configuration and runtime MBeans using JMX and with the command-line utility wlevs.Admin.

  • Oracle CEP is built on the Oracle microServices Architecture (mSA) which uses an OSGi-based framework to manage services provided by modules or feature sets. Oracle mSA provides the following services:

  • Jetty, an HTTP container for running servlets.

  • javax.sql.DataSource implementation and thin JDBC drivers for accessing a relational database.

  • Logging and debugging.

  • Authentication and authorization security.

1.6 Supported Platforms

For detailed information on the platforms that Oracle CEP supports, see: http://www.oracle.com/technology/software/products/ias/files/oracle%20fusion%20middleware%2011gr1%20%2811%201%201%201%200%29%20certification%20matrix.xls.

You can find the installation program appropriate for your platform here: http://www.oracle.com/technology/software/products/middleware/htdocs/111110_fmw.html. For more information, see Section 2.2, "Installation Overview".

1.7 Oracle CEP IDE for Eclipse

Oracle CEP IDE for Eclipse is targeted specifically to programmers that want to develop Oracle CEP applications as Figure 1-3 shows.

Figure 1-3 Oracle CEP IDE for Eclipse

Description of Figure 1-3 follows
Description of "Figure 1-3 Oracle CEP IDE for Eclipse"

The Oracle CEP IDE for Eclipse is a set of plugins for the Eclipse IDE designed to help develop, deploy, and debug Oracle CEP applications.

The key features of Oracle CEP IDE for Eclipse are:

  • Project creation wizards and templates to quickly get started building event driven applications.

  • Advanced editors for source files including Java and XML files common to Oracle CEP applications.

  • Integrated server management to seamlessly start, stop, and deploy to Oracle CEP server instances all from within the IDE.

  • Integrated debugging.

  • Event Processing Network (EPN) visual design views for orienting and navigating in event processing applications.

  • Integrated support for the Oracle CEP Visualizer so you can use the Oracle CEP Visualizer from within the IDE (see Section 1.8, "Oracle CEP Visualizer").

For details, see:

1.8 Oracle CEP Visualizer

Oracle provides an advanced run-time administration console called the Oracle CEP Visualizer as Figure 1-4 shows.

Figure 1-4 Oracle CEP Visualizer

Description of Figure 1-4 follows
Description of "Figure 1-4 Oracle CEP Visualizer"

Using Oracle CEP Visualizer, you can manage, tune, and monitor Oracle CEP server domains and the Oracle CEP applications you deploy to them all from a browser. Oracle CEP Visualizer provides a variety of sophisticated run-time administration tools, including support for Oracle CQL and EPL rule maintenance and creation.

For details, see Oracle CEP Visualizer User's Guide

1.9 New Features in Oracle CEP Release 11gR1 (11.1.1)

This version of Oracle CEP includes the following new features:

1.9.1 Multi-Server Domains and Clustering

You can now configure multiple servers in an Oracle CEP domain, and cluster them together to achieve high availability, using Oracle Coherence.

See "Administrating Oracle CEP Standalone-Server Domains" in the Oracle CEP Administrator's Guide.

1.9.2 Caching

Oracle CEP applications can optionally publish or consume events to and from a cache to increase the availability of the events and increase the performance of their applications. A cache is a temporary storage area for events, created exclusively to improve the overall performance of your application; it is not necessary for the application to function correctly.

In this release, you can configure caching using the Oracle CEP local cache or Oracle Coherence.

See "Configuring Oracle CEP Caching" in the Oracle CEP IDE Developer's Guide for Eclipse.

1.9.3 Record and Playback of Events

The event repository feature of Oracle CEP allows you to record events flowing through an event processing network (EPN) and store them so you can later play back the events. The event repository is configured per stage, such as a processor or channel.

In this release, you can also schedule event record and playback.

See:

1.9.4 Visualizer Administration Console

Oracle CEP Visualizer is a Web 2.0 application that consumes data from Oracle CEP, displays it in a useful and intuitive way to system administrators and operators, and, for specified tasks, accepts data that is then passed back to Oracle CEP so as to change it configuration.

See "Overview of Oracle CEP Visualizer" in the Oracle CEP Visualizer User's Guide.

1.9.5 HTTP Publish-Subscribe Adapter

An HTTP Publish-Subscribe Server is a mechanism whereby Web clients, such as browser-based clients, subscribe to channels, receive messages as they become available, and publish messages to these channels, all using asynchronous messages over HTTP. Every instance of Oracle CEP includes a pub-sub server that programmers can use to implement HTTP publish-subscribe functionality in their applications.

See "Configuring HTTP Publish-Subscribe Server Adapters" in the Oracle CEP IDE Developer's Guide for Eclipse.

1.9.6 JMS Adapter

Oracle CEP provides both inbound and outbound JMS adapters that you can use in your event applications to send and receive messages to and from a JMS queue, respectively, without writing any Java code.

See "Configuring JMS Adapters" in the Oracle CEP IDE Developer's Guide for Eclipse.

1.9.7 Configuration Wizard for Creating Domains

The Configuration Wizard is a new tool for creating Oracle CEP domains.

See:

1.9.8 Oracle Continuous Query Language (Oracle CQL)

Oracle Continuous Query Language (Oracle CQL) is a query language based on SQL with added constructs that support streaming data. Using Oracle CQL, you can express queries on data streams to perform complex event processing (CEP) using Oracle CEP.

Oracle CQL replaces Event Processing Language (EPL) in Oracle CEP Release 11gR1 (11.1.1). Oracle CEP supports EPL for backwards compatibility. In this release, all Oracle CEP examples are expressed in Oracle CQL.

See Oracle CEP CQL Language Reference.

Some Oracle CQL features include:

  • Stream and relation operators—to perform the operations with streams and relations:

    • Relation-to-relation operators—to produce relations.

      See "Relation-to-Relation Operators" in the Oracle CEP CQL Language Reference.

    • Stream-to-relation operators—to produce a relation from a stream using a variety of window operators, including:

      • Range: time-based

        S[Range T], or, optionally,

        S[Range T1 Slide T2]

      • Range: time-based unbounded

        S[Range Unbounded]

      • Range: time-based now

        S[Now]

      • Range: constant value

        S[Range C on ID]

      • Tuple-based:

        S[Rows N], or, optionally,

        S[Rows N1 Slide N2]

      • Partitioned:

        S[Partition By A1 ... Ak Rows N] or, optionally,

        S[Partition By A1 ... Ak Rows N Range T], or

        S[Partition By A1 ... Ak Rows N Range T1 Slide T2]

      See "Stream-to-Relation Operators" in the Oracle CEP CQL Language Reference.

    • Relation-to-stream operators—to produce a stream from a relation, including:

      • IStream: insert stream.

        IStream(R) contains all (r,T) where r is in R at time T but r is not in R at time T-1.

      • DSteam: delete stream.

        DStream(R) contains all (r,T) where r is in R at time T-1 but r is not in R at time T.

      • RStream: relation stream.

        RStream(R) contains all (r,T) where r is in R at time T.

      See "Relation-to-Stream Operators" in the Oracle CEP CQL Language Reference.

    • Stream-to-stream operators—to produce a stream from one or more other streams

      See "Stream-to-Stream Operators" in the Oracle CEP CQL Language Reference.

  • Queries, views, and joins:

    • An Oracle CQL query is an operation that you express in Oracle CQL syntax that you execute on the Oracle CEP Service Engine to retrieve data from one or more streams or views. A top-level SELECT statement that you create using the [REGISTER|CREATE] QUERY statement is called a query.

      In particular, you can use the Oracle CQL MATCH_RECOGNIZE condition and its various clauses to succinctly express complex conditions among stream elements to perform advanced comparisons optimized for data streams. You can use the MATCH_RECOGNIZE clause wherever Oracle CQL supports a relation variable.

      Example 1-1 MATCH_RECOGNIZE Query

      <query id="detectPerish"><![CDATA[ 
    select
        its.itemId
    from 
        ItemTempStream 
    MATCH_RECOGNIZE (
        PARTITION BY 
            itemId
        MEASURES 
            A.itemId as itemId
        PATTERN (A B* C)
        DEFINE
            A  AS  (A.temp >= 25),
            B  AS  ((B.temp >= 25) and 
                    (B.element_time - A.element_time < INTERVAL "0 00:00:05:00" DAY TO SECOND)),
            C  AS  (C.element_time - A.element_time >= INTERVAL "0 00:00:05:00" DAY TO SECOND)
    ) as its
]]></query>

    • An Oracle CQL view represents an alternative selection on a stream or relation. In Oracle CQL, you use a view instead of a subquery. A top-level VIEW statement that you create using the [REGISTER|CREATE] VIEW statement is called a view.

    • A join is a query that combines rows from two or more streams, views, or relations.

    See "Queries, Views, and Joins" in the Oracle CEP CQL Language Reference.

  • Oracle CQL statements—a wide variety of statements for creating and using queries, views, functions, and windows.

    See "Oracle CQL Statements" in the Oracle CEP CQL Language Reference.

  • Functions—a wide variety of both built-in functions and base classes to extend to create user-defined functions, including:

    • single-row

    • aggregate

    • statistical and advanced arithmetic operations based on the Colt open source libraries for high performance scientific and technical computing

    • statistical and advanced arithmetic operations based on the java.lang.Math class

    See "Functions" in the Oracle CEP CQL Language Reference.

  • Time—control over timestamps and advanced scheduling, including:

    • application or system timestamped relations

    • application or system timestamped streams

    • derived timestamped streams

    See "Time" in the Oracle CEP CQL Language Reference.

1.9.9 Event Processing Language (EPL)

The event processing language (EPL) has the following new features:

Note:

Oracle CQL replaces Event Processing Language (EPL) in Oracle CEP Release 11gR1 (11.1.1). Oracle CEP supports EPL for backwards compatibility. For more information, see Section 1.9.8, "Oracle Continuous Query Language (Oracle CQL)."

  • Parameterized Queries—Parameterized queries allow you to put placeholders inside of an EPL query in the form of a question mark. At runtime you bind these placeholders with values and they are then compiled into regular statements. The process is much like the PreparedStatement in JDBC.

    See "Parameterized Queries" in the Oracle CEP EPL Language Reference.

  • Subqueries—EPL supports both simple subqueries as well as correlated subqueries. In a simple subquery, the inner query is not correlated to the outer query. Correlated subqueries allow SELECT clauses to be embedded within another SELECT or WHERE clause. Both IN and EXISTS keywords are available when subqueries are used in the WHERE clause. Previously they were available only in the FROM clause and without the ability to correlate the inner and outer query.

    See "Simple and Correlated Subqueries" in the Oracle CEP EPL Language Reference.

  • Dynamic Event Properties—Dynamic (unchecked) properties are event properties that need not be known at statement compilation time. Such properties are resolved during runtime. The idea behind dynamic properties is that for a given underlying event representation, the properties are not always known in advance. An underlying event may have additional properties that are not known at statement compilation time, but these properties might still be required in an EPL query. The concept is especially useful for events that represent rich, object-oriented domain models or when generic container events are used.

    In conjunction with dynamic event properties, the following functions have also been added to this release: INSTANCEOF, CAST, and EXISTS.

    See "Dynamic Event Properties" in the Oracle CEP EPL Language Reference.

1.9.10 Security

Oracle CEP provides a variety of mechanisms to protect server resources such as data and event streams, configuration, username and password data, security policy information, remote credentials, and network traffic.

Oracle CEP supports various security providers for authentication, authorization, role mapping, and credential mapping. As initially installed, Oracle CEP is configured to use the file-based providers for both authentication and authorization. You can also configure the system to use an LDAP or DBMS provider.

In this release, you can configure custom users and groups and use them to control access to your Oracle CEP applications.

In this release, Oracle CEP supports Federal Information Processing Standards (FIPS) using the com.rsa.jsafe.provider.JsafeJCE security provider. Using this provider, you can configure Oracle CEP to use a FIPS-certified pseudo-random number generator for SSL.

See "Configuring Security for Oracle CEP" in the Oracle CEP Administrator's Guide.

1.9.11 Event Beans and Spring Beans

There is a new component type called event bean. Event beans are very similar to basic Spring beans, but they can take advantage of the full Oracle CEP framework, such as monitoring and record/playback. Event beans are functionally the same as adapters, but should be used as intermediate nodes in the EPN when adapters are used as beginning or ending nodes.

You can now also use standard Spring beans as nodes in the EPN, although the beans cannot take advantage of Oracle CEP framework.

Spring tags for event beans now include attributes for setting lifecycle callback methods via the EPN assembly file

For event beans and Spring beans that want to run in a thread, their Java class should now implement com.bea.wlevs.ede.api.RunnableBean.

1.9.12 Suspend and Resume Applications

Applications can now be suspended and resumed

1.9.13 Domain Directory Structure Changes

To support clustering and a multi-server environment, the directory structure of the out-of-the-box and sample domains has changed.

Previously, all domain files were located directly under the domain directory, such as ORACLE_CEP_HOME/ocep_11.1/samples/domains/helloworld_domain, where ORACLE_CEP_HOME refers to the top-level installation directory of Oracle CEP.

In this release, the domain directory contains one or more sub-directories that correspond to a single server; this directory contains all the configuration files for that server. For example, the HelloWorld domain contains a single server whose configuration files are located in the following directory: ORACLE_CEP_HOME/ocep_11.1/samples/domains/helloworld_domain/defaultserver.

The new Configuration Wizard for creating domains follows this updated directory structure.

See:

1.10 Next Steps