Modern App Development - Messaging

Build messaging solutions that are highly available, reliable, and flexible. Use Oracle's platforms, services, and design recommendations that follow best practices to enable you to deliver messaging solutions based on your business needs. These recommendations help minimize development integration, deployment overhead, and long-term management burden.

Messaging solutions connect application components, including your existing on-premises systems, to cloud solutions. They enable data transfer either as part of a well-defined distributed processing pipeline, or publish messages to multiple independent downstream systems that evolve independently.

These solutions should also transparently accommodate unplanned spikes in message load by buffering data and dynamically adjusting process resources. In the past, it was challenging for enterprises to deploy and manage solutions that meet these objectives without undue complexity and expenses. However, this implementation can be straightforward in the cloud.

You can this pattern to decouple processing from data producers, to efficiently buffer unprocessed messages, and to provide message durability, processing scalability, and application resiliency.

Design Principles

Use the following design principles to build your messaging applications or platform.
  • Build apps as services that communicate through APIs

    Leverage Oracle Cloud Infrastructure (OCI) Messaging's Kafka compatibility to build applications using Kafka APIs. The use of Kafka APIs provides application interoperability and lets you seamlessly build hybrid and multi-cloud messaging applications. Implement interoperability with Oracle Cloud Infrastructure (OCI) and third-party products. Use the OCI Service Connector Hub to build a serverless integration with OCI services such as Object Storage, Notifications, Logging, Logging Analytics, Functions, and Messaging. Use Kafka Connect deployed on Oracle Container Engine for Kubernetes cluster to connect with third-party products.

  • Use fully-managed services to eliminate complexity across application development, runtimes, and data management

    Run applications on fully-managed services with built-in infrastructure maintenance and security patching. You can leverage scaling automation in response to the changing loads.

  • Use converged databases with fully-featured support across all data

    Use databases that natively support different types of data—JSON, relational, graph, spatial, and so on. Use database functionality to simplify application logic. For example, use SQL for queries, joins, and analysis. Use transactions to guarantee consistency and isolation, and built-in machine learning algorithms and analytics to avoid unnecessary data transfers. Use the database’s security features and access control to protect sensitive data, and use replication to improve the availability, scalability, and resiliency of your app.

  • Instrument end-to-end monitoring and tracing

    Maintain an authoritative understanding of your application’s health, performance, and operational state. Use the Oracle Cloud Observability and Management portfolio of services to gain visibility and actionable insights across all layers of the application stack, from data producers and consumers to the messaging pipelines themselves. Monitor queue lengths and processing duration to catch errors and bottlenecks and to detect problems with services subscribing to topics.

  • Eliminate single points of failure through horizontal scaling, and automate failure recovery

    To fulfill regulatory and compliance needs, back up messaging data in Object Storage for long-term retention. Use a serverless service like Service Connector to seamlessly move data from Messaging to Object Storage, and enable Object Storage's cross-region backup to achieve multi-region backup. Implement a cross-region disaster recovery strategy using Kafka MirrorMaker 2.0 deployed on a fault-tolerant Oracle Container Engine for Kubernetes (OKE) environment to asynchronously replicate data between streams. This setup enables a Recovery Time Objective (RTO) and Recovery Point Objective (RPO) of minutes. Use remote VCN peering to ensure minimal latency during the data transfer.

    Incorporate idempotency into applications by storing the offsets of the processed messages in external storage like Object Storage, or Oracle Autonomous Database (ADB). Detect and discard duplicates by querying the external store. Categorize errors that are easily recoverable and allow for a replay of messages. Write unrecoverable errors to a separate stream, a dead letter queue, or Object Storage without blocking the primary execution pipeline.

  • Implement a defense-in-depth approach to secure the app lifecycle

    Implement Identity and Access Control (IAM) policies to allow only authorized users to create, send, or receive data from the streams. Apply a principle of minimum reachability to the endpoint by securing access to messaging endpoints using private endpoints and service gateway, which limits access from the Internet. Use Messaging's out-of-the-box capability to encrypt data at rest and in transit to achieve data confidentiality. However, if you need increased ownership of key rotation, use the OCI Vault service to securely manage your master keys.

Architecture

Messaging enables applications to exchange data reliably and asynchronously, scale transparently, and achieve a high level of overall availability. Individual components may evolve independently while minimizing the risk of making system-wide changes. Message producers and consumers evolve independently and may have little knowledge of, or dependencies upon each other, except for the message data format and service bus network end-point.

You can implement the design principles by using a deployment based on this opinionated architecture.

Description of maf-messaging.png follows
Description of the illustration maf-messaging.png

There are three common modern messaging models, each with shared and distinct application requirements.

  • Data Exchange: Uses Oracle's Streaming service for asynchronous message transfer. Send messages to consumers by triggering Functions in response to events.
  • Publish-Subscribe Model: Uses Oracle's Streaming service for asynchronous message transfer to consumers who have subscribed to receive messages.
  • Stateful Distributed Processing and Queue Management: Enables management of a message queue and processes the messages sequentially depending on the order specified in the message.

This architecture includes the following technology recommendations:

  • Virtual cloud network (VCN) and subnet

    A VCN is a customizable, software-defined network that you set up in an Oracle Cloud Infrastructure region. Like traditional data center networks, VCNs give you complete control over your network environment. A VCN can have multiple non-overlapping CIDR blocks that you can change after you create the VCN. You can segment a VCN into subnets, which can be scoped to a region or to an availability domain. Each subnet consists of a contiguous range of addresses that don't overlap with the other subnets in the VCN. You can change the size of a subnet after creation. A subnet can be public or private.

  • Streaming

    Oracle Cloud Infrastructure Streaming provides a fully managed, scalable, and durable storage solution for ingesting continuous, high-volume streams of data that you can consume and process in real time. You can use Streaming for ingesting high-volume data, such as application logs, operational telemetry, web click-stream data; or for other use cases where data is produced and processed continually and sequentially in a publish-subscribe messaging model.

  • Functions

    Oracle Functions is a fully managed, multitenant, highly scalable, on-demand, Functions-as-a-Service (FaaS) platform. It is powered by the Fn Project open source engine. Functions enable you to deploy your code, and either call it directly or trigger it in response to events. Oracle Functions uses Docker containers hosted in Oracle Cloud Infrastructure Registry.

  • Service connectors

    Oracle Cloud Infrastructure Service Connector Hub is a cloud message bus platform that orchestrates data movement between services in OCI. You can use it to move data between services in Oracle Cloud Infrastructure. Data is moved using service connectors. A service connector specifies the source service that contains the data to be moved, the tasks to perform on the data, and the target service to which the data must be delivered when the specified tasks are completed.

    You can use Oracle Cloud Infrastructure Service Connector Hub to quickly build a logging aggregation framework for SIEM systems. An optional task might be a function task to process data from the source or a log filter task to filter log data from the source.

  • Queue service

    Oracle Queue Service provides a scalable system to process messages while handling complex management tasks such as guaranteed at-least-once processing, tracking, and client isolation. This centralized service also manages message ordering and processing state, which allows stateless client processes to offload cursor tracking.

Alternative Architecture

This architecture provides alternative designs to accommodate existing investments, operational familiarity, or other constraints. Use this architecture as an alternative recommendation for building your Messaging application using Oracle's Modern App Development design principles.

Description of maf-messaging-alternate.png follows
Description of the illustration maf-messaging-alternate.png

Application architects can consider the following alternative architecture:

  • Use Database Queues for Asynchronous Messaging

    An alternative architecture for reliable messaging is an Event Mesh in the database, using Oracle Transactional Event Queues (TEQ) along with other transactional features of the data platform. Oracle TEQ provides a combination of JMS message queues and Kafka-like pub-sub capabilities, used in streaming data and managing application workflows. Transactional Event Queues in the database simplify an app architecture by providing transactions across messaging and database operations. They also simplify configuration, high availability, and life cycle management by eliminating the need for a separate messaging infrastructure.

  • Use Monolithic Message and Service Bus Applications

    Applications such as RabbitMQ offer a mature solution and may support integration with other components through open standards and APIs. These solutions are typically easy to deploy and offer core features out of the box. However, these solutions require significant SME administrative effort and might not offer distributed redundancy and high availability without self-managed complex topologies.

  • Use Kafka Clusters in a Self-Managed Cloud or On-Premises Environment

    This solution, although offering scalability and high availability, demands significant specialized developer knowledge and extensive SME operational administration overhead. Give careful consideration before selecting this option due to the lead time to production and the risk of high Total Cost of Ownership (TCO).

Oracle doesn't recommend these antipatterns for building modern messaging architectures.

  • Non-persistent storage buffers that result in unplanned spikes in message load by buffering data, and may queue up and buffer unprocessed messaging.
  • Rigid schema definitions that can reduce flexibility and backward compatibility, and will not support incremental or deferred updates.

Considerations

When implementing the Messaging design pattern, consider these implementation options.

  • Choose the right messaging platform based on your application requirements

    Foundation platforms and services might appear similar and share common features. However, each platform has unique features and strengths that might be better aligned with your application requirements. For example, use OCI Streaming if your applications require a real-time, high-throughput messaging platform that offers message replay and pub-sub abilities. Use OCI Queuing if you need a scalable and reliable point-to-point buffer to asynchronously move data. Select Service Connector Hub (SCH) to enable integration with infrastructure resources.

  • Instrument an effective schema management strategy

    Use schema definitions to maintain coordination between app components and to implement a tight API contract between services. Create schema versions with backward compatibility, where practical, to facilitate incremental and deferred updates to distributed legacy components.

Public Case Study

Tango Eye converts surveillance video into actionable insights for the retail industry.

  • Oracle Cloud Infrastructure (OCI) Streaming serves as a low-maintenance publish-subscribe messaging system for various microservices. 
  • Oracle Functions executes serverless jobs without any engineering oversight.
  • Lifecycle policies for Oracle Cloud Infrastructure Object Storage automatically archive and purge data, reducing costs without diminishing the value of Tango Eye’s AI-based analytics.