This chapter provides an overview of Oracle Event Processing. It describes key concepts, features, and use cases, including event processing networks, developing with the Eclipse IDE, and managing applications with Oracle Event Processing Visualizer.
This chapter includes the following sections:
Oracle Event Processing is middleware for creating applications that receive potentially large amounts of streaming data representing events, then respond in real time based on the event data. Oracle Event Processing applications can receive event data from sources near and remote, including from sensors (such as for environment conditions), social media, financial feeds, and moving sources such as vehicles or mobile devices.
Some of the logic in Oracle Event Processing applications is typically implemented as Oracle Continuous Query Language (Oracle CQL) queries that examine event data in real time as it is received, filtering the data or watching for patterns. Other logic implemented in Java code can respond to discoveries made from the data, such as by initiating other processes or sending related data to other components (including reporting tools for display in a user interface).
Through connections with external components and resources -- including relational databases, "big data" sources such as Hadoop and NoSQL, Java Message Service (JMS) systems, and caches -- Oracle Event Processing applications can combine or correlate event data with other kinds of data. In this way, these applications integrate ongoing streaming data with existing, relatively static systems.
Very Large Amounts of Data from a Growing List of Sources
Oracle Event Processing provides a way to take advantage of the world's ever-growing amount of event data. Both on public networks such as the Internet and private, corporate networks, the increasing number of event sources and amount of event data provides opportunities that event-driven applications can take advantage of.
Sources of events include:
Applications on mobile devices such as smartphones can provide location data as events that include geographical coordinates.
Social media such as Twitter can offer events related to tweets that mention specific companies or products.
Devices such as the sensors in a data center can send regular, frequent events that contain information about environmental conditions in the rooms containing servers.
The events in each of these examples represent a potentially huge amount of streaming data. To take advantage of the data, an application should be able to filter out irrelevancies and identify patterns of interest that can be used to a company's advantage or to defuse immediate business threats. In many cases, making the most of events means being able to execute application logic in response to particular patterns -- and to do so in real time.
In this context, with so much data moving so quickly, an asynchronous processing model event is needed to ensure that processing scales well. Oracle Event Processing is designed to support very low latency and high throughput in an application context that assumes a large amount of data and a high requirement for immediate response.
The use cases described in this section illustrate specific uses for Oracle Event Processing applications.
Financial: Responsive Customer Relationship
Acting on an initiative to improve relationships with customers, a retail bank designs an effort to provide coupons tailored to each customer's purchase pattern and geography.
The bank collects automated teller machine (ATM) data, including about the geographical region for the customer's most common ATM activity. The bank also captures credit card transaction activity. Using this data, the bank can push purchase incentives (such as coupons) to the customer in real time based on where they are and what they tend to buy.
An Oracle Event Processing application receives event data in the form of ATM and credit card activity. Oracle CQL queries filter incoming events for patterns that isolate the customer's geography (via GPS coordinates) and likely purchase interests nearby. This transient data is matched against the bank's stored customer profile data. If a good match is found, a purchase incentive is sent to the customer in real time, such as through their mobile device.
Telecommunications: Real-Time Billing
Due to significant growth in mobile data usage, a telecommunications company with a large mobile customer base wants to shift billing for data usage from a flat-rate system to a real-time per-use system.
The company tracks IP addresses allocated to mobile devices and correlates this with stored user account data. Additional data is collect from deep-packet inspection (DPI) devices (for finer detail about data plan usage) and IP servers, then inserted into a Hadoop-based big data warehouse.
An Oracle Event Processing application receives usage information as event data in real time. Through Oracle CQL queries, and by correlating transient usage data with stored customer account data, the application determines billing requirements.
Energy: Improving Efficiency Through Analysis of Big Data
A company offering data management devices and services needs to improve its data center coordination and energy management in order to reduce total cost of ownership. The company needs finer-grained, more detailed sensor and data center reporting.
The company receives energy usage sensor data from disparate resources. Data from each sensor must be analyzed for its local relevance, then must be aggregated with data from other sensors to identify patterns that can be used to improve efficiency.
Separate Oracle Event Processing applications provide a two-tiered approach. One application, deployed in each of thousands of data centers, receives sensor data as event data. Through Oracle CQL queries against events representing the sensor data, this application analyzes local usage, filtering for fault and problem events and sending alerts when needed. The other application, deployed in multiple central management sites, receives event data from lower-tier the applications. This application aggregates and correlates data from across the system to identify consistency issues and produce data to be used in reports on patterns.
Understanding the Oracle Event Processing framework includes looking at it from the perspective of its key concepts, development artifacts and process, and constituent technologies.
The following sections describe concepts that are important to understanding how applications work
The concepts described in this section represent ideas that are unique to the event-driven model on which Oracle Event Processing applications are based. If you are new to this model, getting to know these concepts should help provide a basis for understanding more specific and technical aspects of Oracle Event Processing.
Event data is data received from (or sent to) a component that is external to the EPN. Event data is typically in a form that is understandable to the external component with which Oracle Event Processing is communicating. For example, the data coming from a sensing device will likely be in a form native to that device.
Event data is made up of tuples, or ordered lists of elements. For example, the following list of values could be a tuple representing a trade of 50 shares of Oracle stock at 30.86 a share:
{orcl,30.86,50}
You can think of event data tuples as similar to rows in a database. The data has a schema -- here, {symbol as string,price as double,volume as integer} -- that defines a consistent shape for the data.
An event type provides structure to represent event data inside an EPN -- it normalizes event data received by the application. The recommended way to create an event type is to create a JavaBean in which properties correspond to elements in event data tuples.
When event data arrives at an EPN, it is bound to an instance of the event type that was developed to represent it. The event type instance -- usually referred to simply as an "event" -- provides developers with a predictable, normalized way to access the event data from code in EPN stages.
Connected stages in an EPN provide a way to execute different kinds of code against events passing through the EPN. Kinds of stages can include an adapter, a processor, and a bean.
For example, an adapter could receive event data and bind it to an event type instance, then pass the event along to a processor. The processor's Continuous Query Language (CQL) code can query the events (as SQL code queries database rows), looking for particular patterns in the data as it flows through the EPN. Events that meet the pattern criteria could be passed along to a bean (written in Java), where the data could be used in a calculation with data retrieved from an external source. A further bean could use the calculation result to execute a process using an external component.
Any component designed to receive or send events in an EPN (such as EPN stages) must have been implemented specifically to do so. Components that are able to receive events are known as event sinks, while components that send events are known as event sources. A single component is usually both.
The set of stage components included in Oracle Event Processing, such as adapters and the components on which CQL processors are based, already support the required functionality. Developers can add event sink and source support to beans, new adapters, and other code they write by implementing interfaces from the OEP API.
The concepts of streams and relations are unique to Oracle Event Processing applications. As a result, they can seem more esoteric than other application ideas. However, they're important for describing changes in the condition of events as a result of queries.
A stream of events is in sequential order by time -- one after the other. Events are by definition time-based, so a stream is that sense the natural condition of events. It is how event data arrives at an Oracle Event Processing application.
A relation is, in a manner of speaking, a bucket of related events. A relation is the result produced by many CQL queries. So you can think of a relation as being similar to the results of a database query. Events in a relation are, well, related by some criteria, such as those used in the query. They aren't in sequential order.
An event processing network (EPN) is the core of each Oracle Event Processing application. The application processes the events it receives as each event moves from one part of the application to another.
To process events with Oracle Event Processing, you build an application whose core is an EPN. The EPN is made up of stages that each serve a distinct role in processing events, from receiving event data to querying the data to executing logic based on what is discovered about the events. The application receives raw event data, binds the data to event types, then routes the events from stage to stage for processing.
Event processing networks improve the ability to integrate through separately developed components working correctly together. For example, one can add user code and reference to external services at several places in the network.
Figure 1-1 Oracle Event Processing Application
A developer creates an EPN by assembling stages in a sequence so that events can be filtered, analyzed, and responded in real time. Each stage in the sequence provides its own logic for filtering and responding to event data.
Oracle Event Processing includes support for the following kinds of stages:
Adapters provide a way for the EPN to exchange data with components that are external to the network. Adapters understand the inbound and outbound protocols for the component they're interacting with. They are also responsible for converting the event data into the normalized form represented by an event type. Oracle Event Processing includes adapters for exchanging data as CSV files (for testing and debugging purposes), as well as with the Java Message Service (JMS) and an included HTTP Publish-Subscribe server. Developers can also create their own adapters using the Oracle Event Processing API.
Processors contain query code written in Continuous Query Language (CQL) and consume normalized event data from a channel that precedes them in the network. As part of their work, processors may also generate new events to an output channel. CQL extends the database query language, SQL. CQL includes functionality specifically designed for querying streaming data, and is particularly useful for fine-grained queries of events. Within a processor, a CQL developer can write multiple query blocks in which one block executes against the results of another.
Bean stages provide a place for logic written in Java. Java code in a bean implements Oracle Event Processing API interfaces through which the code can receive events (an event sink), send events (an event source), or both. A developer can implement a bean following either the standard Spring bean model or the OEP event bean model, which supports OEP-specific functionality. Support for the Spring bean model is particularly useful in cases where existing Spring bean code can be leveraged in the EPN.
Channels connect some kinds of stages, and also feature configuration options that can be used to tune and enhance performance. For example, channels can queue event data until a processor agent can act upon it.
The following very brief EPN topology examples suggest how stages can be combined to achieve certain goals:
Adapter -> Channel -> Bean
Scenario: Events do not need to be processed by an Oracle CQL query. Events received are only adapted from a proprietary protocol to some normalized model.
Adapter -> Channel -> Processor -> Channel -> Bean
Scenario: Events must be processed by a single Oracle CQL stage, then sent through to logic coded in Java. For example, the processor might filter to only those events that are relevant to action taken by logic in the bean.
Adapter -> Channel -> Processor -> Channel -> Bean -> Channel -> Processor -> Channel -> Bean
Scenario: Events pass through two layers of event processing, each with its own Oracle CQL query code. The first processor discovers causal relationships between events, while the second processor aggregates events to pass along those that are notable for logic implemented in the bean.
Developing event-driven applications with Oracle Event Processing is in most ways like developing applications on more common models, such as models for JEE applications. However, specific aspects of event-driven applications, in particular the time-oriented quality of working with streaming data, drive a need for specific tools and processes.
The following sections introduce the application development lifecycle as it pertains specifically to Oracle Event Processing applications.
The development stages described in this section assume that you have installed and configured the Oracle Event Processing server. This step installs the server, but also supporting functionality such as clustering, security, network I/O, Java Management Extensions (JMX), JDBC, HTTP Publish-Subscribe server, and logging.
Once you install and configure the server, the following stages include both development and administrative work.
Design and develop applications using an integrated development environment (IDE).
Deploy applications using an included command-line or browser-based tool.
Configure applications either by directly editing configuration artifacts or through an included browser-based tool.
Test applications with event data, including by injecting and tracing data against specific stages in an event processing network.
Monitor aspects of the application, including channels, cluster nodes, and caches.
The following sections provides more detailed information on each of these lifecycle phases.
With application goals in hand, developers define the schema for events that the application will handle. Event schemas used within the application are based on event data received from external components (such as a monitoring device). The event properties in a schema is often a subset of the data received since only certain elements might be needed. Application designers also define for external components (if any) that the application will interact with, such as a cache containing identifying data needed in combination with event data to interact with an external component.
In developing the application, a developer builds out an event processing network by selecting and configuring stages in the network. The developer might write logic code for certain stages, such as CQL code in processors and Java code in beans.
For example, a CQL developer might add query code to a processor stage designed to identify events that are part of a particular pattern (such as a dramatic temperature increase within 10 minutes' time). Then, within a JavaBean added as an event bean stage, a developer could include logic in Java to take action based on the content of events found in the pattern (for example, by sending an alert using an event property value that indicates where the temperature increase is occurring).
All of this is most easily done using an integrated development environment (IDE) that includes design tools specifically created for building Oracle Event Processing applications. Current releases of Oracle Event Processing include an Eclipse plugin with these tools. With an IDE, a developer can iteratively code, deploy, and debug an application until it is ready for testing and production.
Oracle Event Processing applications support deployment to standalone and multi-server domains. You can use one of two methods to deploy an application for testing and production: the Deployer command-line utility and Oracle Event Processing Visualizer. (During development, it's common to use an IDE to deploy an application in progress.)
The Deployer is a command-line utility that uses HTTP to connect to the OEP server and deploy the specified application. Deployer can also be used for other operations, such as suspending, resuming, and uninstalling.
Oracle Event Processing Visualizer is a browser-based tool for managing OEP servers and applications. The tool simplifies deployment and runtime configuration for both servers and the applications that run on them.
Oracle Event Processing provides multiple layers of configuration for tuning. In addition to full configuration options for the server itself, Oracle Event Processing provides configuration for event processing network stages and other application characteristics. In particular, Oracle CQL queries are configurable at runtime.
Many of these configuration options may be used to tune the server and applications for performance and high availability.
In addition to providing server and application management, Oracle Event Processing Visualizer provides a means to test an application by tracing and injecting events. Event tracing displays messages for events as they pass through the EPN. With event injection, you can inject an event into a particular EPN stage in order to discover or confirm how that stage handles an event with particular properties.
Through the Oracle Event Processing Visualizer and programmatic resources, you can monitor aspects of an application in production. For example, Oracle Event Processing Visualizer provides the ability to monitor event throughput and latency. It also provides the ability to monitor Coherence cluster nodes and caching.
The Oracle Event Processing platform is made up of standard technologies that provide functionality for developing application logic, as well as deploying and configuring applications.
Java
Much of the Oracle Event Processing server's functionality is written in the Java programming language. Java is the language you use to write some components that make up an OEP application. These include event beans and Spring beans.
Spring
Spring is a collection of technologies designed to ease and augment enterprise application development with Java. Oracle Event Processing makes significant use of the Spring configuration model, which developers use to connect and configure parts of an application. OEP applications also support adding logic as Spring beans, Java components that support Spring framework features.
You can find out more about Spring at the project's web site: http://www.springsource.org/get-started
OSGi
OSGi (formerly the Open Services Gateway initiative) is the module system through which Oracle Event Processing application components are assembled in a deployment. By following OSGi component guidelines, OEP application components support a modular assembly model that eases management.
You can find out more about OSGi at wikipedia: http://en.wikipedia.org/wiki/OSGi
XML
As with many other enterprise applications, Oracle Event Processing application configuration files are written in XML. These files include the EPN assembly file, which defines relationships between EPN stages, along with other design-time configuration. A separate configuration XML file contains configuration that can be modified after the application is deployed, including Oracle CQL queries.
SQL
Structured Query Language (SQL) is the basis for Oracle Continuous Query Language (Oracle CQL), which extends SQL with functionality designed to address the needs of applications that use streaming data.
Hadoop
Oracle CQL developers can access "big data" Hadoop data sources from query code. Hadoop is an open source technology that provides access to large data sets that are distributed across clusters. One strength of the Hadoop software is that it provides access to large quantities of data not stored in a relational database.
If you need more information about Hadoop, you might start with the Hadoop project web site at http://hadoop.apache.org/
.
NoSQL
Oracle CQL developers can access "big data" NoSQL data sources from query code. The Oracle NoSQL Database is a distributed key-value database. In it, data is stored as key-value pairs, which are written to particular storage node(s). Storage nodes are replicated to ensure high availability, rapid failover in the event of a node failure and optimal load balancing of queries.
If you need more information about Oracle NoSQL, be sure to see its Oracle Technology Network page at http://www.oracle.com/technetwork/products/nosqldb/
.
Oracle Event Processing includes tools designed to ease development and management of the Oracle Event Processing server and applications.
Oracle Event Processing IDE for Eclipse
Oracle Event Processing includes a plugin for use with the Eclipse integrated development environment (IDE). The Oracle Event Processing IDE for Eclipse includes features specifically designed to ease development of Oracle Event Processing applications. Figure 1-2 shows the IDE open to show the editor for designing an event processing network (EPN).
Figure 1-2 Oracle Event Processing IDE for Eclipse
The Oracle Event Processing IDE for Eclipse is a set of plugins for the Eclipse IDE designed to help develop, deploy, and debug Oracle Event Processing applications.
The key features of Oracle Event Processing IDE for Eclipse are:
Project creation wizards and templates to quickly get started building event driven applications.
Advanced editors for source files including Java and XML files common to Oracle Event Processing applications.
Integrated server management to seamlessly start, stop, and deploy to Oracle Event Processing server instances all from within the IDE.
Integrated debugging.
Event processing network (EPN) visual design views for orienting and navigating in event processing applications.
Integrated support for the Oracle Event Processing Visualizer so you can use the Oracle Event Processing Visualizer from within the IDE (see Oracle Event Processing Visualizer).
For more information, see:
Section 2.7, "Installing the Oracle Event Processing IDE for Eclipse"
Oracle Fusion Middleware Developer's Guide for Oracle Event Processing for Eclipse
Oracle Event Processing Visualizer
Oracle Event Processing provides an administration console called the Oracle Event Processing Visualizer. This tool provides access to server and application configuration options that are available at runtime, including the ability to edit Oracle CQL queries and to test an application by injecting events.
Figure 1-3 shows Oracle Event Processing Visualizer displaying the Oracle CQL design user interface.
Figure 1-3 Oracle Event Processing Visualizer
Using Oracle Event Processing Visualizer, you can manage, tune, and monitor Oracle Event Processing server domains and the Oracle Event Processing applications you deploy to them all from a browser. Oracle Event Processing Visualizer provides a variety of sophisticated run-time administration tools, including support for Oracle CQL and EPL rule maintenance and creation.
Oracle Event Processing Visualizer is pre-installed in every Oracle Event Processing server.
For more information, see Oracle Fusion Middleware Visualizer User's Guide for Oracle Event Processing
Oracle Event Processing provides the following documentation.
Oracle Fusion Middleware Getting Started Guide for Oracle Event Processing
Use this guide to get an introduction to Oracle Event Processing. The guide describes where it is useful and what its framework is made up of.
Oracle Fusion Middleware Administrator's Guide for Oracle Event Processing
Use this guide to deploy and manage Oracle Event Processing applications. It describes how to manage Oracle Event Processing in single- and multi-server domains. It also describes how to configure component parts of an installation, including security, Jetty, JMX, and JDBC.
Oracle Fusion Middleware Developer's Guide for Oracle Event Processing for Eclipse
This guide describes how to develop Oracle Event Processing applications. It introduces key concepts at a more technical level. It also describes the parts of a typical application, along with how to assemble them.
Oracle Fusion Middleware Visualizer User's Guide for Oracle Event Processing
Use this guide when using the browser-based Oracle Event Processing Visualizer to manage and configure the Oracle Event Processing server and applications deployed on it.
Oracle Fusion Middleware CQL Language Reference for Oracle Event Processing
Use this reference for descriptions and syntax of the Oracle CQL language. Oracle CQL extends Structured Query Language (SQL) to address the particular needs of querying streaming data.
Oracle Fusion Middleware EPL Language Reference for Oracle Event Processing
This reference describes the deprecated Oracle Event Processing Language (EPL). All new development should use Oracle Continuous Query Language instead. Support and documentation for this language will be removed in a future release.
For detailed information on the platforms that Oracle Event Processing supports, see: http://www.oracle.com/technetwork/middleware/ias/downloads/fusion-certification-100350.html
.
You can find the installation program appropriate for your platform here:
http://www.oracle.com/technetwork/middleware/complex-event-processing/downloads/index.html
. For more information, see Section 2.1, "Installation Overview".
Install Oracle Event Processing 11g Release 1 (11.1.1).
Understand how the sample applications have been programmed by viewing the source and configuration files and then building them from their respective source directories.
For more about the samples, see "Oracle Event Processing Samples" in the Oracle Fusion Middleware Developer's Guide for Oracle Event Processing for Eclipse.
Create your own Oracle Event Processing domain.
See:
"Creating an Oracle Event Processing Standalone-Server Domain" in the Oracle Fusion Middleware Administrator's Guide for Oracle Event Processing
"Creating an Oracle Event Processing Multi-Server Domain Using Oracle Coherence" in the Oracle Fusion Middleware Administrator's Guide for Oracle Event Processing
"Creating an Oracle Event Processing Multi-Server Domain Using Oracle Event Processing Native Clustering" in the Oracle Fusion Middleware Administrator's Guide for Oracle Event Processing
Create a new Oracle Event Processing application and deploy it to your new domain.
See "Overview of Creating Oracle Event Processing Applications" in the Oracle Fusion Middleware Developer's Guide for Oracle Event Processing for Eclipse.
Pose questions and find solutions using the Oracle Event Processing forum: http://forums.oracle.com/forums/forum.jspa?forumID=820