Identify the Most Appropriate Messaging Service
Learn about the considerations for selecting a messaging service and then use the decision matrix table to review the different options and limitations of the OCI services based on your business needs.
Considerations for Selecting a Messaging Service
Consider the following evaluation factors to make your decision.
Choreography or Orchestration
Choreography and Orchestration are two architectural approaches to how different decoupled components are organized to work together. Choreography adopts the principle that each component performs independently but listens and watches for cues for when to perform their next action. A message broker transmits the cues to the next component. Components can be very autonomous and easy to scale. The downside is that each component requires a greater understanding of it's role in the overall solution and adds greater complexity. A message broker is a critical element and it's simplicity reduces the workload and enables greater throughput.
Orchestration refers to the idea of components assembled like an orchestra. Each component only needs to know their task and then wait for the conductor or orchestrator to prompt them to perform the action. Thus the conductor must understand and have the intelligence to direct all the components. The conductor role is key and its resilience and ability to scale, directly influences the scalability and resilience of the overall solution.
Some orchestration products include the ability to choreograph (such as inclusion of a broker) or create services that use components to achieve the effect of brokered communication.
Pay per Message or Events Vs. Bulk Purchase
Consider the different payment models based on the cost impact to your solution. You can choose to pay per message if you will potentially have unused message volume from a bulk purchase. Consider the fact that paying per message or event may cost more at volume because the minimum footprint cost for a service must be spread over fewer uses.
Single or Multiple Message Consumers
The difference between a Queue and a Topic is that a message can only be consumed once with a queue. A topic can have many different consumers. Queues and topics can have many instances of a consumer type. Multiple instances of the same consumer type connected to a single topic or queue is often described as a Race Condition where the next available consumer takes the next message available.
Message Retention After Reading
Some message brokers will delete a message once read (or after all consumers have read the message) to manage resources. Apache Kafka tracks how far a consumer has gone through the messages but not delete them once read. This provides the ability to replay messages if there is a problem. Hence Kafka acts as a data store as well as a message broker and supports creating data analytics on the message data store.
Guaranteed Order
Guaranteeing that messages are delivered in the order in which they are received, rather than the order in which they are stored, introduces complexity and impacts performance. Latency issues can be introduced when working through a broker as it may wait before allowing consumers to get a message to ensure that there aren't any delayed messages that must be inserted into the right order before they're consumed.
For example, message ordering is essential for events representing an order being canceled. The message for the order creation has the potential to cause application issues if the messages don't follow the order.
There are application-level patterns and strategies that can help mitigate ordering issues and offset broker constraints.
Asynchronous Operation
The benefit of using a broker mechanism is that the message provider doesn't need to know about the consumer and doesn't slow the provider down. The broker knows who the consumer is, whether they're available or not, and manages transmission if there are any issues with the consumer (such as performance). The use of the broker enables asynchronous communication and the provider and consumer don't need to be in sync.
Maximum Message Rate, Message Retention Period, and Maximum Retention Volume
These factors are all influencers on the cost of providing a serverless solution and will often require hard limits to be set. Software doesn't always provide uniform performance as it scales expectations. You can set limits to control expectations and provide predictable behaviors. You must understand these limits to determine whether they can impact the solution you're building.
For example, a share trading system will receive messages when share prices change. Each message will be small, but message volume will be high due to the high frequency of updates. A broker with a low throughput but large message size can't support this requirement.
Industry-Standards Input Support and Industry-Standards Output Support
Using industry-standard formats and protocols to send and receive messages increases the ability for the messaging mechanism to be reconfigured and used with different deployments and implementation providers. You can leverage common libraries to address the low-level mechanics of communication and also reduce vendor tie-in.
Third-Party Adapter Support
To help develop message providers and consumers, having support through third-party adapters can be advantageous as it increases development options.
Individual Deployments per Compartment
You can deploy individual messaging channels or flows to specific compartments and allow the services in a solution to potentially have fine-grained-collective security controls through a common control surface. This approach assumes that all elements can be controlled at the compartment level. The solutions that can't accommodate compartment-level controls per channel can offer alternatives. Alternative approaches may require additional configuration considerations.
Use OCI IAM Authentication
Oracle Cloud Infrastructure Identity and Access Management (OCI IAM) provides a robust enterprise-class level of security in cases where security can be federated to other providers. Working through a single IAM provider (with or without federation) makes security control of data flow access more manageable.
Data is Encrypted In Flight and At Rest
If the data passing through the service is sensitive, it must be protected both during transmission and while being held by the broker until delivery. Sensitive messages must be encrypted by the broker's encryption mechanism to guarantee no message loss.
Message Transformation
Depending on the architectural principles adopted, the broker may need to have the ability to translate message formatting. The message provider or consumers don't need to understand either common data semantics or know how the provider or consumers want the data formatted. This consideration may also extend to changes in protocols being used. For example, translating from REST to STOMP.
Audit Trail
To help remediate issues with distributed applications, an audit trail enables tracking where a message was transmitted to and received from. Having an audit trail can be useful when you must comply with legislative requirements. Therefore, it may be necessary to learn how the service can support this requirement.
Review the Decision Matrix
Use the following decision matrix to learn about the available choices of OCI services and the considerations or evaluation factors for choosing a particular service.
| Service/Factor | Queue | Streaming | Notifications | Oracle Integration (Gen2, Gen3) | Autonomous Database (TEQ and AQ) |
|---|---|---|---|---|---|
| Choreography versus Orchestration |
Choreography |
Choreography | Choreography | Choreography and Orchestration (Depends on integration configuration) | Choreography and Orchestration (Depends on integration configuration) |
| Pay per message or event versus bulk purchase | Per API call + multiplier for messages > 64 KB |
Per GB data transfer + Storage |
Depends upon the communication channel. Priced by quantity sent. | Price in blocks of 5000 integration calls per hour | No, Database pricing |
| Single or multiple consumers for a message |
Single consumer. (Race conditions allowed, but only one consumer per message). |
Single and Multiple | Single and Multiple | Single and Multiple (Depends on integration definition) | Single and Multiple |
| Message retention after reading | No | Yes | No | Yes (Depends on integration definition) | Yes |
| Guaranteed order | Best effort | Yes, within a partition | No | Yes (Depends on integration definition) | Partial (Rules about priorities can be set, which can dictate consumption order) |
| Asynchronous operation | Yes | Yes | Yes | Yes (Depends on integration definition) | Yes |
| Industry standards supported for inputs | REST API, STOMP | Kafka Connect | Cloud Events from OCI Events, Alarms | Multiple different adapters |
JMS and SQL Provide additional exposure with custom ORDS implementations by providing REST APIs |
| Third-Party Adapter support |
Open source STOMP implementations |
Any Kafka-compliant tool, library, or SDK | No | Wide range of industry adapters. You can generate from REST interface or a standard JMS client |
Use of standard JMS standard library SQL is widely supported |
| Industry standards supported for outputs | STOMP, REST API | Kafka Connect | Email, Slack, PagerDuty, HTTP/S | Multiple different adapters | JMS and SQL |
| Maximum message rate | 10 MBPS |
1 MB per second per partition Up to 250 |
60 per minute for HTTP/S 10 per minute for Email 60 per minute for notification topics |
None specified | Depends on Database sizing |
| Message retention period | 7 days (or shorter if configured) | 7 days | Until exhausted delivery retries | None defined | Configurable |
| Maximum Retention Volume | 2 GB per queue (20 GB per tenancy) | Retention is dictated by the maximum write rate of 1 MB per second per partition for 7 days. | The function of message rate and delivery retry duration. | Not applicable | Depends on Database configuration |
| Supported by OCI SDK or CLI to send messages | Yes | Yes | Yes | No | No |
| Individual deployments per compartment | Yes | Yes | Yes |
No. Oracle Integration instances separated at compartment level, not individual integrations |
No. Database instances align to compartments, not individual queues |
| Uses IAM Authentication (RBAC and so on) | Yes | Yes | Yes | Yes | Yes |
| Data is encrypted in flight and at rest | Yes | Yes | Yes | Yes | Depends on Database configuration |
| Message transformation | No | No |
Limited. Friendly formatting for email and so on. If the endpoint is using Oracle Functions, then Functions can transform the message. |
Yes | Yes (Stored SQL procedures) |
| Audit trail | Yes (using Logging) | Yes (using Logging) | Yes (using Logging) | Yes (Includes Oracle Integration with Logging) | Yes |
You can interpret the matrix using the following information:
- Payment model: Refers to per message or bulk purchase of capacity.
- Race condition: When a queue has multiple consumers, the consumers are described as being in a 'race condition' because the next messages are processed by the consumer that is first to be ready for another message.
- REST-based endpoints: Some of the services offer REST-based endpoints. If the REST payload is modeled using an Open API specification, it may not provide a payload-specific SDK. In such cases you can use API Gateway services to generate SDKs. See the OCI Documentation on Generating SDKs for an API Resource for more information.
In addition to the OCI services, the matrix refers to these technologies:
- STOMP: Simple (or Streaming) Text Orientated Messaging Protocol. It provides an interoperable wire format so that STOMP clients can communicate with any STOMP message broker to provide easy and widespread messaging interoperability among many languages, platforms, and brokers.
- STOMP Implementations: Refers to the known STOMP-compliant message servers.
- Kafka Connect: This tool enables reliably streaming and scaling data between Apache Kafka and other systems. Enables you to quickly define connectors that move large data collections into and out of Kafka.
You can read additional information about the technologies in the Explore section.