Modern App Development - Messaging

Messaging solutions connect your app's components, enabling them to exchange data reliably, scale transparently, and achieve a high level of availability. They enable you to decouple processing from data producers, to efficiently buffer unprocessed messages, and to provide message durability, processing scalability, and application resiliency.

Message-oriented applications span a range of architectures – data transfer between components may be part of a well-defined distributed and converged processing pipeline, or components may publish messages to multiple independent downstream systems that evolve independently. There are three common modern messaging models, each with shared and distinct application requirements:

Event Streaming

The following diagram shows an example event streaming application that uses the OCI Streaming Service or Oracle Transactional Event Queues (TEQ) as the underlying messaging solution, Oracle Functions for real-time processing, and the Oracle Autonomous Database as the backing store.

Description of maf-messaging-streaming.png follows
Description of the illustration maf-messaging-streaming.png

maf-messaging-streaming-oracle.zip

Event Streaming applications ingest high-volume and high-velocity streams of data that must be processed in real-time. The raw data must usually be processed to extract insights that may either be used by other application components, for monitoring, or may be stored for later analysis.

Publish-Subscribe

The following diagram for an example of how to use the OCI Streaming Service or Oracle TEQ to implement the Publish-subscribe messaging pattern.

Description of maf-messaging-publishers-subscribers.png follows
Description of the illustration maf-messaging-publishers-subscribers.png

maf-messaging-publishers-subscribers-oracle.zip

Publish-subscribe is a communication pattern where data producers publish data to specific topics which can then be consumed by any number of downstream consumers by subscribing to those topics. The coupling between producers and consumers is quite loose, so that consumers may evolve independently without impacting the upstream producers.

Message Queueing

The following diagram illustrates how to use Oracle TEQ to implement message queuing.

Description of maf-messaging-queuing.png follows
Description of the illustration maf-messaging-queuing.png

maf-messaging-queuing-oracle.zip

Message Queuing enables distributed stateful processing where upstream and downstream components are tightly bound and together implement an application workflow. The messaging solution must support semantics such as at-least-once delivery to ensure that messages do not get lost before they are consumed.

Design Principles

Use the following design principles to build your messaging applications or platform.
  • Build apps as a suite of services that communicate using REST APIs

    Use industry-standard APIs such as Kafka and JMS to provide application interoperability and seamlessly build hybrid and multicloud messaging applications. Oracle Cloud Infrastructure (OCI) Streaming provides a Kafka compatibility API, and Oracle Transaction Event Queues (TEQ) support both Kafka and JMS APIs. Both Kafka and JMS are widely supported by third party products. For example, you can use the Confluent Kafka JMS Connector to transfer messages between Oracle TEQ and Kafka. Use Kafka Connect deployed on Oracle Container Engine for Kubernetes cluster to connect with third-party products.

  • Use managed services to eliminate complexity in app development and operations

    Use managed services with built-in infrastructure maintenance and security patching such as OCI Streaming and the Transactional Event Queues (TEQ) and Advanced Queuing (AQ) features of the Oracle Autonomous Database (ADB). These services are highly available with automatic replication across availability domains and support scaling automation in response to changing loads.

  • Keep app tier stateless

    Messaging-related state such as the position in a message queue should never be stored in the app or on the local file system, since doing so may result in loss of data if an app instance fails. Apps may cache message contents for processing, but the messaging solution should remain the single source of truth for all messaging data. Related metadata such as the position in a message queue should be stored in a database or in object storage to avoid lost messages and ensure idempotent operation. This aids with failure recovery while also making it easier to scale a service up or down without loss of correctness.

  • Instrument end-to-end monitoring and tracing

    Maintain an authoritative understanding of your application's health, performance, and operational state. Use the Oracle Cloud Observability and Management portfolio of services to gain visibility and actionable insights across all layers of the application stack, from data producers and consumers to the messaging pipelines themselves. Monitor queue lengths and processing duration to catch errors and bottlenecks and to detect problems with services subscribing to topics.

  • Eliminate single points of failure through automated data replication and failure recovery

    Back up messaging data to persistent storage to fulfill regulatory and compliance needs. Use cross-region backup for disaster recovery. Incorporate idempotency into your messaging applications, and write unrecoverable errors to a separate stream, a dead letter queue, or persistent storage without blocking the primary execution pipeline.

  • Implement automated defense-in-depth approach to secure your app and its data

    Implement Oracle Cloud Infrastructure Identity and Access Management (IAM) policies to allow only authorized users to create, send, or receive data from the streams. Apply a principle of minimum reachability to the endpoint by securing access to messaging endpoints using private endpoints and service gateway, which limits access from the internet. Use the out-of-the-box capability of the OCI Streaming service and TEQ to encrypt data at rest and in transit to achieve data confidentiality. However, if you need increased ownership of key rotation, use the Oracle Cloud Infrastructure Vault service to securely manage your master keys.

Architecture

The following diagram shows how you can use OCI Streaming to implement the event streaming
Description of maf-messaging.png follows
Description of the illustration maf-messaging.png

maf-messaging-oracle.zip

Use OCI Streaming service to implement the event streaming and pub/sub messaging patterns. Oracle Functions can be used to process event data before it is consumed by downstream application components. Use the Oracle Autonomous Database to persist processed event data.

Implement network isolation by using dedicated subnets for your application, for the database, and for the messaging services.

Use Oracle Cloud Infrastructure Object Storage for long-term message retention. Use a serverless service like Service Connector to seamlessly move data from OCI Streaming to Object Storage, and enable Object Storage's cross-region backup to achieve multiregion backup. Implement a cross-region disaster recovery strategy using Kafka MirrorMaker 2.0 deployed on a fault-tolerant Oracle Container Engine for Kubernetes (OKE) environment to asynchronously replicate data between streams. This setup enables a Recovery Time Objective (RTO) and Recovery Point Objective (RPO) of minutes. Use remote VCN peering to ensure minimal latency during the data transfer.

Incorporate idempotency into applications by storing the offsets of the processed messages in external storage like Object Storage. Detect and discard duplicates by querying the external store. Categorize errors that are easily recoverable and allow for a replay of messages.

This architecture uses the following service and technologies:

  • Streaming

    Oracle Cloud Infrastructure Streaming provides a fully managed, scalable, and durable storage solution for ingesting continuous, high-volume streams of data that you can consume and process in real time. You can use Streaming for ingesting high-volume data, such as application logs, operational telemetry, web click-stream data; or for other use cases where data is produced and processed continually and sequentially in a publish-subscribe messaging model.

  • Functions

    Oracle Functions is a fully managed, multitenant, highly scalable, on-demand, Functions-as-a-Service (FaaS) platform. It is powered by the Fn Project open source engine. Functions enable you to deploy your code, and either call it directly or trigger it in response to events. Oracle Functions uses Docker containers hosted in Oracle Cloud Infrastructure Registry.

  • Service connectors

    Oracle Cloud Infrastructure Service Connector Hub is a cloud message bus platform that orchestrates data movement between services in OCI. You can use it to move data between services in Oracle Cloud Infrastructure. Data is moved using service connectors. A service connector specifies the source service that contains the data to be moved, the tasks to perform on the data, and the target service to which the data must be delivered when the specified tasks are completed.

    You can use Oracle Cloud Infrastructure Service Connector Hub to quickly build a logging aggregation framework for SIEM systems. An optional task might be a function task to process data from the source or a log filter task to filter log data from the source.

Database-Centric Alternative Architecture

This architecture uses Transactional Event Queues (TEQ) to implement messaging patterns in modern applications. TEQ is a built-in feature of the Oracle Autonomous Database.

Description of maf-messaging-alternate.png follows
Description of the illustration maf-messaging-alternate.png

maf-messaging-alternate-oracle.zip

This architecture provides simplicity by eliminating the need to utilize external streaming or queueing services and provides transactional messaging capabilities that simplify common microservices patterns.

TEQ combines data and message processing in a scalable infrastructure that simplifies life cycle management, security, and disaster recovery while providing high performance. TEQ supports common messaging patterns including streaming, queuing, and pub/sub. You can implement transactions across messaging and database operations using TEQ, and easily implement messaging patterns like transactional outbox and exactly-once messaging with little or no additional code. Oracle TEQ provides exactly-once messaging for applications running in the database .This means you do not need to maintain message ids to check for the duplicate messages or build idempotent consumers at the application level. TEQ supports both small messages that are typical in event processing as well as larger payloads associated with business workflows, and can also function as a secure event mesh.

Eliminating external streaming or queueing services simplifies state management. Events and messages are stored in the same database used by the application. This allows you to easily achieve and maintain consistency among your events, messages, and application changes. If there is a failure that requires a point-in-time recovery or a disaster recovery, everything (events, messages, and application data) is automatically recovered to a consistent state.

TEQ benefits from the high availability of the Autonomous Database. Messaging data in TEQ is automatically backed up and protected by ADB’s cross region replication using Autonomous Data Guard. You can deploy Oracle TEQ in a highly available manner using Oracle Real Application Clusters and Oracle Active Data Guard, both of which are built-in features of the Oracle Autonomous Database. Oracle Real Application Clusters provide availability within a region while Oracle Active Data Guard provides cross-region disaster recovery protection.

Use this architecture if your application does any of the following:

  • Needs to implement message queues and requires semantics like transactional outbox
  • Doesn't need independent scaling of the database and the messaging substrate

This architecture uses the following services and technologies:

  • Transactional Even Queues (TEQ) and Advanced Queuing (AQ)

    Transactional Event Queues (TEQ) and Advanced Queuing (AQ) are robust and feature-rich message queuing systems integrated with Oracle Database. Transactional Event Queues (TEQ) are a high performance partitioned in-memory implementation with multiple event streams per queue. Advanced Queuing (AQ) is suitable for simpler workflow use cases. These features leverage the Oracle Database to persist messages and provide high throughput and scalability.

  • Functions

    Oracle Functions is a fully managed, multitenant, highly scalable, on-demand, Functions-as-a-Service (FaaS) platform. It is powered by the Fn Project open source engine. Functions enable you to deploy your code, and either call it directly or trigger it in response to events. Oracle Functions uses Docker containers hosted in Oracle Cloud Infrastructure Registry.

Non-Recommended Architectures

  • Monolithic Message and Service Bus Applications

    Messaging solutions such as RabbitMQ may support integration with other components through open standards and APIs. However, these solutions require significant SME administrative effort and might not offer distributed redundancy and high availability without self-managed complex topologies.

  • Kafka Clusters in a Self-Managed Cloud or On-Premises Environment

    This solution, although offering scalability and high availability, demands significant specialized developer knowledge and extensive SME operational administration overhead. Give careful consideration before selecting this option due to the lead time to production and the risk of high Total Cost of Ownership (TCO).

Considerations

When implementing the Messaging design pattern, consider these implementation options.

Choose the right messaging platform based on your application requirements

Foundation platforms and services might appear similar and share common features. However, each platform has unique features and strengths that might be better aligned with your application requirements. For example:

  • Use OCI Streaming or Transactional Event Queues if your applications require a real-time, high-throughput messaging platform that offers message replay and pub-sub abilities.
  • Use Transactional Event Queues if you need a scalable and reliable point-to-point buffer to asynchronously move data.
  • Use Service Connector Hub (SCH) to enable integration with infrastructure resources.
  • Use Transactional Event Queues (TEQ) in the database when designing new microservices with messaging integrated with the Oracle Autonomous Database.

Public Case Studies

Oracle Streaming Based Architecture

Tango Eye converts surveillance video into actionable insights for the retail industry.

  • Oracle Cloud Infrastructure (OCI) Streaming serves as a low-maintenance publish-subscribe messaging system for various microservices. 
  • Oracle Functions executes serverless jobs without any engineering oversight.
  • Lifecycle policies for Oracle Cloud Infrastructure Object Storage automatically archive and purge data, reducing costs without diminishing the value of Tango Eye's AI-based analytics.

Oracle TEQ Based Architecture

FedEx uses Oracle E-Business Suite and the business event manager built with Oracle TEQ, for accounts receivables of their 15.5 million packages delivered every day.

  • FedEx has moved E-Business Suite to Oracle Cloud Infrastructure Object Storage using Exadata Cloud Service. E-Business Suite workflows and business event system are entirely built on Oracle Advanced Queuing (AQ) messaging.
  • Oracle AQ is simple to use and suitable for workflow orchestration. For high throughput events, Oracle TEQ is the high performance drop-in replacement with multiple event streams per queue, interoperable with apache Kafka.

Change Log

This log lists significant changes:

Acknowledgments

  • Authors: Harshad Kasture, Randall Barnes
  • Contributors: Hassan Ajan, James Emerson, Joshua Stanley, Parvez Syed Mohamed, Sajan Parihar, Wei Hu