Modern App Development - Messaging
Message-oriented applications span a range of architectures – data transfer between components may be part of a well-defined distributed and converged processing pipeline, or components may publish messages to multiple independent downstream systems that evolve independently. There are three common modern messaging models, each with shared and distinct application requirements:
Event Streaming
Event Streaming applications ingest high-volume and high-velocity streams of data that must be processed in real-time. The raw data must be processed to extract insights that may either be used by other application components, for monitoring, or may be stored for later analysis.
Description of the illustration maf-messaging-streaming.png
maf-messaging-streaming-oracle.zip
This example event streaming application uses the Oracle Cloud Infrastructure Streaming or Oracle Transactional Event Queues (TEQ) as the underlying messaging solution, Oracle Cloud Infrastructure Functions for real-time processing, and the Oracle Autonomous Database as the backing store.
Publish-Subscribe
Publish-subscribe or pub/sub is a communication pattern where data producers publish data to specific topics which can then be consumed by any number of downstream consumers by subscribing to those topics. The coupling between producers and consumers is quite loose, so that consumers may evolve independently without impacting the upstream producers.
Description of the illustration maf-messaging-publishers-subscribers.png
maf-messaging-publishers-subscribers-oracle.zip
This example shows how to use Oracle Cloud Infrastructure Streaming or Oracle TEQ to implement a pub/sub messaging pattern.
Message Queueing
Message queuing enables distributed stateful processing where upstream and downstream components are tightly bound and together implement an application workflow. The messaging solution must support semantics such as at-least-once delivery to ensure that messages do not get lost before they are consumed.
Description of the illustration maf-messaging-queuing.png
maf-messaging-queuing-oracle.zip
This example illustrates how to use Oracle Cloud Infrastructure Queue or Oracle TEQ to implement message queuing.
Design Principles
- Build apps as a suite of services that communicate using REST APIs
Use industry-standard APIs such as Kafka and JMS to provide application interoperability and seamlessly build hybrid and multicloud messaging applications. Oracle Cloud Infrastructure (OCI) Streaming provides a Kafka compatibility API, and Oracle Transaction Event Queues (TEQ) support both Kafka and JMS APIs. Both Kafka and JMS are widely supported by third party products. For example, you can use the Confluent Kafka JMS Connector to transfer messages between Oracle TEQ and Kafka. You can also use Kafka Connect deployed on Oracle Container Engine for Kubernetes cluster to connect with third-party products. Oracle Cloud Infrastructure Queue can be called using RESTful API definition (with an OpenAPI specification) or by using the industry standard STOMP protocol.
- Use managed services to eliminate complexity in app development and operations
Use fully-managed services with built-in infrastructure maintenance and security patching such as OCI Streaming, OCI Queue, and the Oracle TEQ and Advanced Queuing (AQ) features of the Oracle Autonomous Database (ADB). These services are highly available with automatic replication across availability domains and support scaling automation in response to changing loads.
- Keep app tier stateless
Messaging-related state such as the position in a message queue should never be stored in the app or on the local file system, since doing so may result in loss of data if an app instance fails. Apps may cache message contents for processing, but the messaging solution should remain the single source of truth for all messaging data. Related metadata such as the position in a message queue should be stored in a database or in object storage to avoid lost messages and ensure idempotent operation. This aids with failure recovery while also making it easier to scale a service up or down without loss of correctness.
- Instrument end-to-end monitoring and tracing
Maintain an authoritative understanding of your application's health, performance, and operational state. Use the Oracle Cloud Observability and Management Platform portfolio of services to gain visibility and actionable insights across all layers of the application stack, from data producers and consumers to the messaging pipelines themselves. Monitor queue lengths and processing duration to catch errors and bottlenecks and to detect problems with services subscribing to topics.
- Eliminate single points of failure through automated data replication and failure recovery
Back up messaging data to persistent storage to fulfill regulatory and compliance needs. Use cross-region backup for disaster recovery. Incorporate idempotency into your messaging applications, for example by using message visibility timeouts in OCI Queue. Write unrecoverable errors to a separate stream, a dead letter queue, or persistent storage without blocking the primary execution pipeline.
- Implement automated defense-in-depth approach to secure your app and its data
Implement Oracle Cloud Infrastructure Identity and Access Management (IAM) policies to allow only authorized users to create, send, or receive data from OCI streams and queues. Apply the principle of minimum reachability by securing access to messaging endpoints using private endpoints and service gateway, which limits access from the internet. Use the out-of-the-box capability of OCI Streaming, OCI Queue, and TEQ to encrypt data at rest and in transit to achieve data confidentiality. However, if you need increased ownership of key rotation, use the Oracle Cloud Infrastructure Vault service to securely manage your master keys.
Architecture
Description of the illustration maf-messaging.png
Use OCI Streaming to implement the event streaming and pub/sub messaging patterns, and Oracle Cloud Infrastructure Queue to implement message queuing. Use the Oracle Autonomous Database to persist processed event data. OCI Functions can be used to process event data before it is consumed by downstream application components or persisted to the database.
Implement network isolation by using dedicated subnets for your application and for the database, and for the messaging services. Secure access to your streams by using private endpoints. Use Oracle Cloud Infrastructure Identity and Access Management (IAM) policies to limit access to queues
Use Oracle Cloud Infrastructure Object Storage for long-term message retention. Use a serverless service like Service Connector to seamlessly move data from OCI Streaming to Object Storage, and enable Object Storage's cross-region backup to achieve multiregion backup. Implement a cross-region disaster recovery strategy using Kafka MirrorMaker 2.0 deployed on a fault-tolerant Oracle Container Engine for Kubernetes (OKE) environment to asynchronously replicate data between streams. This setup enables a Recovery Time Objective (RTO) and Recovery Point Objective (RPO) of minutes. Use remote VCN peering to ensure minimal latency during the data transfer.
Use dead letter queues to isolate problematic messages. Dead letter queues are automatically created when you create a queue. Dead letter queues help you avoid having failing messages block the primary execution pipeline. Messages in the dead letter queue can then be analyzed to determine why they failed.
Incorporate idempotency into applications by storing the offsets of the processed messages in external storage like Object Storage. Detect and discard duplicates by querying the external store. Categorize errors that are easily recoverable and allow for a replay of messages.
This architecture uses the following service and technologies:
- Streaming
Oracle Cloud Infrastructure Streaming provides a fully managed, scalable, and durable storage solution for ingesting continuous, high-volume streams of data that you can consume and process in real time. You can use Streaming for ingesting high-volume data, such as application logs, operational telemetry, web click-stream data; or for other use cases where data is produced and processed continually and sequentially in a publish-subscribe messaging model.
- Queue
Oracle Cloud Infrastructure Queue provides a scalable system to process messages while handling complex management tasks such as guaranteed at-least-once processing, tracking, and client isolation. This centralized service also manages message ordering and processing state, which allows stateless client processes to offload cursor tracking.
- Functions
Oracle Cloud Infrastructure Functions is a fully managed, multitenant, highly scalable, on-demand, Functions-as-a-Service (FaaS) platform. It is powered by the Fn Project open source engine. Functions enable you to deploy your code, and either call it directly or trigger it in response to events. Oracle Functions uses Docker containers hosted in Oracle Cloud Infrastructure Registry.
- Service connectors
Oracle Cloud Infrastructure Service Connector Hub is a cloud message bus platform that orchestrates data movement between services in OCI. You can use service connectors to move data from a source service to a target service. Service connectors also enable you to optionally specify a task (such as a function) to perform on the data before it is delivered to the target service.
You can use Oracle Cloud Infrastructure Service Connector Hub to quickly build a logging aggregation framework for security information and event management (SIEM) systems.
Database-Centric Alternative Architecture
This architecture uses Transactional Event Queues (TEQ) to implement messaging patterns in modern applications. TEQ is a built-in feature of the Oracle Autonomous Database.
Description of the illustration maf-messaging-alternate.png
maf-messaging-alternate-oracle.zip
This architecture provides simplicity by eliminating the need to utilize external streaming or queueing services and provides transactional messaging capabilities that simplify common microservices patterns.
TEQ combines data and message processing in a scalable infrastructure that simplifies life cycle management, security, and disaster recovery while providing high performance. TEQ supports common messaging patterns including streaming, queuing, and pub/sub. You can implement transactions across messaging and database operations using TEQ, and easily implement messaging patterns like transactional outbox and exactly-once messaging with little or no additional code. Oracle TEQ provides exactly-once messaging for applications running in the database .This means you do not need to maintain message ids to check for the duplicate messages or build idempotent consumers at the application level. TEQ supports both small messages that are typical in event processing as well as larger payloads associated with business workflows, and can also function as a secure event mesh.
Eliminating external streaming or queueing services simplifies state management. Events and messages are stored in the same database used by the application. This allows you to easily achieve and maintain consistency among your events, messages, and application changes. If there is a failure that requires a point-in-time recovery or a disaster recovery, everything (events, messages, and application data) is automatically recovered to a consistent state.
TEQ benefits from the high availability of the Autonomous Database. Messaging data in TEQ is automatically backed up and protected by ADB’s cross region replication using Autonomous Data Guard. You can deploy Oracle TEQ in a highly available manner using Oracle Real Application Clusters and Oracle Active Data Guard, both of which are built-in features of the Oracle Autonomous Database. Oracle Real Application Clusters provide availability within a region while Oracle Active Data Guard provides cross-region disaster recovery protection.
Use this architecture if your application does any of the following:
- Needs to implement message queues and requires semantics like transactional outbox
- Doesn't need independent scaling of the database and the messaging substrate
This architecture uses the following services and technologies:
- Transactional Event Queues (TEQ) and Advanced Queuing (AQ)
Transactional Event Queues (TEQ) and Advanced Queuing (AQ) are robust and feature-rich message queuing systems integrated with Oracle Database. Transactional Event Queues (TEQ) are a high performance partitioned in-memory implementation with multiple event streams per queue. Advanced Queuing (AQ) is suitable for simpler workflow use cases. These features leverage the Oracle Database to persist messages and provide high throughput and scalability.
- Functions
Oracle Cloud Infrastructure Functions is a fully managed, multitenant, highly scalable, on-demand, Functions-as-a-Service (FaaS) platform. It is powered by the Fn Project open source engine. Functions enable you to deploy your code, and either call it directly or trigger it in response to events. Oracle Functions uses Docker containers hosted in Oracle Cloud Infrastructure Registry.
Non-Recommended Architectures
-
Monolithic Message and Service Bus Applications
Messaging solutions such as RabbitMQ may support integration with other components through open standards and APIs. However, these solutions require significant SME administrative effort and might not offer distributed redundancy and high availability without self-managed complex topologies.
-
Kafka Clusters in a Self-Managed Cloud or On-Premises Environment
This solution, although offering scalability and high availability, demands significant specialized developer knowledge and extensive SME operational administration overhead. Give careful consideration before selecting this option due to the lead time to production and the risk of high Total Cost of Ownership (TCO).
Considerations
When implementing the Messaging design pattern, consider these implementation options.
Choose the right messaging platform based on your application requirements
Foundation platforms and services might appear similar and share common features. However, each platform has unique features and strengths that might be better aligned with your application requirements. For example:
- Use OCI Streaming or Transactional Event Queues if your applications require a real-time, high-throughput messaging platform that offers message replay and pub-sub abilities.
- Use Transactional Event Queues if you need a scalable and reliable point-to-point buffer to asynchronously move data.
- Use Service Connector Hub (SCH) to enable integration with infrastructure resources.
- Use Transactional Event Queues (TEQ) in the database when designing new microservices with messaging integrated with the Oracle Autonomous Database.
Public Case Studies
Oracle Streaming Based Architecture
Tango Eye converts surveillance video into actionable insights for the retail industry.
- Oracle Cloud Infrastructure (OCI) Streaming serves as a low-maintenance publish-subscribe messaging system for various microservices.
- Oracle Cloud Infrastructure Functions executes serverless jobs without any engineering oversight.
- Lifecycle policies for Oracle Cloud Infrastructure Object Storage automatically archive and purge data, reducing costs without diminishing the value of Tango Eye's AI-based analytics.
Oracle TEQ Based Architecture
FedEx uses Oracle E-Business Suite and the business event manager built with Oracle TEQ, for accounts receivables of their 15.5 million packages delivered every day.
- FedEx has moved E-Business Suite to Oracle Cloud Infrastructure Object Storage using Exadata Cloud Service. E-Business Suite workflows and business event system are entirely built on Oracle Advanced Queuing (AQ) messaging.
- Oracle AQ is simple to use and suitable for workflow orchestration. For high throughput events, Oracle TEQ is the high performance drop-in replacement with multiple event streams per queue, interoperable with apache Kafka.
Deploy
- Go to GitHub.
- Clone or download the repository to your local computer.
- Follow the instructions in the
READMEdocument.
Change Log
This log lists significant changes:
| Sep 21, 2023 |
|
| April 27, 2023 |
|
| July 12, 2022 |
|