2 Core Concepts
The RIB is designed as an asynchronous publication and subscription messaging integration architecture. This allows the decoupling of applications and their systems. For example, a publishing application need not know about the subscribing applications, other than the requirement that at least one durable subscriber must exist. It decouples the systems operationally. Once a subscriber is registered, the RIB persists all published messages until all subscribers have seen them.
The publishing adapter does not know, or care, how many subscribers are waiting for the message, what types of adapters the subscribers are, what the subscribers' current states are (running or stopped), or where the subscribers are located. Delivering the message to all subscribing adapters is the responsibility of the RIB with the help of the underlying JMS server.
Physically, the message must reside somewhere so that it is available until all subscribers have processed it. The RIB uses the JMS specification for its messaging infrastructure. The JMS accepts the message from the publisher and saves it to stable storage, a JMS topic, until it is ready to be picked up by a subscriber. In all cases, message information must be kept on the JMS until all subscribers have read and processed it.
The RIB interfaces are organized by message family. Each message family contains information specific to a related set of operations on a business entity or related business entities. The publisher is responsible for publishing messages in response to actions performed on these business entities in the same sequence as they occur.
Each message family has specific message payloads based on agreed upon business elements between the Oracle Retail applications.
Key Functional Requirements
The design and architecture of the RIB infrastructure is based on two key requirements driven by the Oracle Retail application business model.
Guaranteed Once-and-Only-Once Successful Delivery
The RIB must preserve and persist all business events (messages) until all applications (subscribers) have looked at the message and have successfully consumed it or decided they do not care about that event (message). In other words, RIB must deliver to every subscriber all messages except those filtered as per a subscribing application's requirements.
A business event (message) must be redelivered to the consumer application if the business event (message) was not consumed successfully. The redelivery process is bound by the same rules of sequencing as normal (non-redelivered) business event (message).
Preservation of Publication Sequence
The business event (message) must be delivered to all the subscribing applications in the order (FIFO) the business event (messages) was published by the publishing application.
To enable this, the publishing application defines a business object ID whose existence informs RIB that this and all subsequent messages with the same business object ID have to be processed in order. Business event (message) ordering (FIFO) is assured only for messages with the same business object ID within the same message family.
Message Family and Message Types
The RIB messaging adapters and payloads are designed around the concept of a message family.
Each RIB message belongs to a specific message family. Each message family contains information specific to a related set of operations on a business entity or related business entities. The publisher is responsible for publishing messages in response to actions performed on these entities in the same sequence as they occur.
One example of a message family is the Order message family used to contain information about purchase order events.
A message family may contain multiple message types. Each message type encapsulates the information specific to a business entity within one or more business events. For example, the Order message family is published for events such as Create PO Header, Create PO Detail, Update PO Header, or Delete PO Detail.
A single business event, such as updating a purchase order, may involve multiple business entities, such as a line item within the purchase order.
Because a single business event may involve multiple business entities, the application may publish messages for this event from multiple message families for a single business transaction. More than one message type within a message family may also be created.
There are two broadly defined types of functional interfaces in the RIB (message families): foundation data and transactional data.
Foundation Messages
After populating application tables with initial company seed data, item foundation information is needed. Foundation messages are defined as those with payload that carry basic product data.
This table is an example from the Oracle Retail Integration Bus Integration Guide.
Functional Area | Publishing Applications | Subscribing Applications |
---|---|---|
Items |
RMS |
RWMS, SIM |
Item Locations |
RMS |
SIM |
Locations |
RIB |
RWMS |
Stores |
RMS |
RWMS, SIM |
Vendor |
RMS |
RWMS, SIM |
Warehouses |
RMS |
RWMS, SIM |
Transactional Messages
After populating application tables with initial seed data and after all required item foundation data messages have been subscribed to, all applications are prepared to publish and subscribe transaction data messages. Transactional messages communicate business events involving two or more organizations within a retail supply chain, for instance, among Oracle Retail Merchandising System (RMS), Oracle Retail Store Inventory Management (SIM), and Oracle Retail Warehouse Management System (RWMS), external suppliers and financial systems.
This table is an example from the Oracle Retail Integration Bus Integration Guide.
Functional Area | Publishing Applications | Subscribing Applications |
---|---|---|
Allocations |
RMS |
RWMS, SIM |
Appointments |
RWMS |
RMS, SIM |
ASN Outbound |
RWMS, SIM, RMS, RFM |
RMS, SIM, RWMS, |
ASN Inbound |
RWMS, External, RMS RFM |
RMS, SIM, RWMS |
Inventory Adjustments |
RWMS, SIM |
RMS |
Inventory Request |
SIM |
RMS |
Receipts |
RWMS, SIM |
RMS |
Purchase Order |
RMS, SIM |
RWMS, SIM |
Stock Order Status |
RWMS, SIM |
RMS, SIM |
Transfers |
RMS |
RWMS, SIM |
RIB Message Envelope and Payloads
Whenever a publishing application adapter publishes a message, it wraps the message in an envelope known as the RIB message envelope. The envelope is a standard message delivery format where the message information, the data payload, is contained within the overall delivery information. The envelope itself provides information that the RIB uses, such as RIB hospital information and routing information.
Note:
Payloads do not support time zone formats.
Message Life Cycle
The publishing application is responsible for creating the initial message contents. The RIB publishing adapter publishes it to the JMS Server and makes it available to any JMS subscribers. The RIB knows what subscribers are to receive the message due to the RIB configuration—this configuration associates a set of subscribers to each publisher and message family combination.
For PL/SQL Applications, database tables associated with the publishing application typically stage message information. One or more RIB publishing adapters poll the application via a stored procedure call. For Java EE Applications, the application calls a RIB Enterprise Java Bean (EJB) with the payload information to be published. Similarly, SOAP Applications calls with the payload information in the request to be published.
The message resides on a Java Message Service (JMS) immediately after publication. The JMS topic provides stable storage for the message in case a system crash occurs before all message subscribers receive and process it.
A fundamental RIB system requirement is that a message must be delivered to and processed successfully exactly once by each subscriber. Furthermore, all work performed by the subscriber and the RIB must be atomically committed or rolled back, even if the JMS server is on a remote host. The standard way to perform this is by using an XA compliant interface and two-phase commit protocol.
After initial publication, a message may undergo a series of transformation, filtering, or routing operations. A RIB component that implements these operations is known as a Transformation and Address Filter/Router (TAFR) component. TAFR is the acronym for Transform, Address, Filter, and Route. A TAFR is completely internal to the RIB and does not reside in either the publishing or subscribing application. The RIB performs these intermediate transformation and routing operations on some messages before making them available to the subscribing application.
A single TAFR may only transform a given message, only filter the message, only route it, or combine any of the three operations.
-
Transform - A message may be transformed from one message type into another, for example, WH (warehouse) from RMS to Location for RWMS.
-
Filter - A message may be filtered. Filtering can occur based on message type or based on content.
-
Route - A TAFR may route a message. For example, whenever a stock order message is published for a warehouse with an instance of RWMS, the TAFR routes it to the particular RWMS instance from where the stock will be fulfilled and not to warehouses that do not stock the order's items.
TAFR operations are specific to the set of subscribers to a specific message family. Multiple TAFRs may process a single message for a specific subscriber and different specific TAFRs may be present for different subscribers. Different sets of TAFRs are necessary for different message families. If all subscribers to a message can process all messages within a message family without any TAFR operations, then no TAFR components are needed.
Message processing continues until a subscribing adapter successfully processes the message or determines that no subscriber needs this message.
When a subscriber gets a message to be processed, the adapter checks to see if the RIB Hospital contains any messages associated with the same entity as the current message. If so, then the adapter places the current message in the hospital as well. This is to ensure messages are always processed in the proper sequence. If proper sequencing is not maintained, the subscribing application's data can be corrupted.
If an error occurs during message processing, the subscribing adapter notes this internally and rolls back all database work associated with the message. When the message is re-processed (because it has yet to be processed successfully), the adapter now recognizes this message is problematic and checks it into the hospital. If adding the message to error hospital fails, the subscribing adapter writes the message to the file system. This becomes a poison message (.xml).
After a message is checked into the RIB Hospital, a retry adapter extracts the message from the hospital and re-publishes it to the JMS topic for reprocessing. The message remains in the hospital during all re-tries until the subscribing adapter successfully processes it. Subscribing retry adapter also processes the poison message. It extracts the message from the poison-message file and adds it to the error hospital to be retried. The poison message file will be renamed to processed message (.processed). If the retry adapter fails to process the poison message, the file is moved to human-workflow file (.humanworkflow).
The unprocessed poison messages should be corrected with a human intervention. They are made available in object storage bucket at a regular interval. These messages should be downloaded from object-store, corrected and uploaded back to object store. RIB will process these uploaded messages through subscriber retry adapter.
Messaging Components
The RIB is a messaging system made-up of components that are packaged and shipped as an integration solution between the Oracle Retail applications. The application boundary between RIB and Oracle Retail applications can be confusing at times, so this section defines the RIB components and their responsibility and ownership. A diagram illustrating the RIB integration message flow follows:
RIB Subsystem Components
This section describes the components of the RIB subsystem.
Adapters
A RIB adapter is a component that coordinates business event (message) generation and processing with the respective Oracle Retail application interface. Each adapter in the RIB is created to handle a specific functional interface. RIB adapters are developed using Enterprise Java Beans (EJB) components architecture, subscribing adapters use Message Driven Beans (MDBs) and publishing adapters use Stateless Session Beans (SLSBs).
RIB provides four types of adapters that Oracle Retail applications can exploit to integrate with one another. These adapter types are: publisher, subscriber, TAFR, and hospital retry. They have been built using different technologies based on their particular needs.
Subscriber and TAFR adapters use Message Driven Bean (MDB) technology to register with JMS topics and receive messages for further processing.
Publisher and hospital retry adapters make use of the Java SE (Standard Edition) timer facility to schedule repetitive events that trigger calls to Stateless Session Beans (SLSBs) to query application tables for messages to publish to the JMS server.
As stated in the introduction, a fifth type of adapter exists for publishing messages in a pushing fashion. The Oracle Retail applications invoke this adapter at will for publishing messages.
These adapters have not been considered part of the scope of this technical document in regard to providing a mechanism for starting and stopping them.
Due to the variety of technologies used by the adapters, the goal of this technical design has been to isolate users from these differences and provide them with a common management interface that can be used to control the state of the adapters. During the last few years, the Java Management Extensions (JMX) specification has become a well known standard that defines the management layer for enterprise Java applications. JMX defines standard methodologies for declaring enterprise application components as manageable resources that can be exposed in a consistent way such that any JMX compliant management application can access and provide means for control.
JMS Domains, Destinations, Subscriptions
JMS defines two types of messaging domains: point-to-point and publish/subscribe. RIB uses publish/subscribe types of messaging domains for all its communication. Publish/subscribe is a one-to-many type of message distribution model where one source application en-queues the message and many destination applications can de-queue the same message and process independently of the other peer applications. In publish/subscribe the destinations are known as topics, the en-queue application is known as publisher, and the de-queue is known as subscriber. Unlike point-to-point, in publish/subscribe the publisher and subscriber are totally ignorant of each other and do not and should not know about each other's existence. The JMS Topics retain the messages only as long as it takes to distribute them to current active (running) subscribers. There is also a timing dependency between publishers and subscribers.
A client that subscribes to a topic can consume only messages published after the client has created a subscription, and the subscriber must continue to be active in order for it to consume messages. The JMS specification relaxes this timing dependency to some extent by allowing clients to create durable subscriptions. By creating durable subscriptions the JMS server will continue to hold the messages for all registered subscribers for that topic until the subscriber consumes the message or deletes the subscription.
There are two types of subscribers, non-durable and durable subscribers. The RIB uses only durable subscribers which allow the Oracle Retail edge applications to be in up or down state independently but still not lose any messages and catch up when the application comes back up. Every subscribing RIB adapter registers its durable subscriber with a subscription name that contains its rib-<app> application name and the adapter name in it.
RIB defines logical grouping of retail specific business objects (BO) and business functions in a concept called message family. For every message family there is a corresponding JMS topic. These JMS topics are used as communication pipelines between the source and destination Oracle Retail applications for exchanging the business objects.
The list of JMS topics used by RIB components is detailed in the Reports section of the Oracle Retail Integration Bus Integration Guide.
JMS Message Selector
A key aspect of the JMS usage that the RIB relies on is the attachment of message properties to published messages and the use of selectors by message subscribers. Message properties are used to convey information about the message outside of the actual message data to establish a logical channel for messages.
JMS message selectors are used by the RIB to filter the messages that each subscriber picks up. In other words, using the message properties, selectors act as a filter to weed out messages a subscriber should not process.
The message property set and used by the RIB messages is called threadValue. The thread value is associated with a logical channel of a message stream. All messages for a specific family with a specific business object ID always contain the same threadValue property. This, combined with the standard first in, first out (FIFO) message ordering on the topic, is integral to message sequencing. Messages with different threadValue properties are not guaranteed to be processed in the same relative order as publishing.
Messages published without JMS Message Property present will not be picked up by the standard subscribing RIB adapters.
Pseudo code for message selector:
( ( (appName is not null) AND (appName == $APP_NAME) ) AND ( (retryLocation is not null) AND (retryLocation LIKE $ADP_CLASS_DEF) ) ) OR ( ( (appName is null) OR (appName != $APP_NAME) ) AND ( (retryLocation is null) OR (retryLocation LIKE $ADP_CLASS_DEF) ) ) AND (threadValue == $ADP_INSTANCE_NUMBER)
Additional RIB JMS Message Properties
Every message published by the rib-<app> applications includes a number of JMS user defined header properties. In the current release, these properties are only set, not used by any RIB components. In the future, these properties will be used for intelligent performance enhancement and optimization and for traceability and auditability of RIB messages.
The message properties are as follows:
-
Property Name: appName
Type: java.lang.String
Required Property: false
Example: appName=rib-rms
Description: The appName property contains the rib-<app> application name that published this particular message.
-
Property Name: adapterInstance
Type: java.lang.String
Required Property: false
Example: adapterInstance=Item_pub_1
Description: The adapterInstance property contains the rib-<app> adapter instance name that published this particular message.
-
Property Name: family
Type: java.lang.String
Required Property: false
Example: family=Item
Description: The family property contains the name of the RIB family name to which the message belongs.
-
Property Name: needMessageOrderPreservation
Type: boolean
Required Property: false
Example: needMessageOrderPreservation=true
Description: This property will have a value of true if any ribMessage node within the RibMessages xml has a message that has businessObjectId set. This property will allow us to take advantage of the fact that now we know which messages need message order preserving at JMS header level (without opening the message). In the future, we will be able to take advantage of that information, make our processing parallel, and get better throughput without losing message sequencing.
-
Property Name: topic
Type: java.lang.String
Required Property: false
Example: topic=etItem
Description: This topic property contains the RIB topic name that this particular message is published to or subscribed from.
-
Property Name: ribKernelVersion
Type: java.lang.String
Required Property: false
Example: ribKernelVersion=22.0
Description: The system determines the rib kernel jar version number at runtime and includes its value in this JMS property.
-
Property Name: ribFuncArtifactVersion
Type: java.lang.String
Required Property: false
Example: ribFuncArtifactVersion=22.0
Description: This is a place holder for future enhancement. The idea is the system will somehow determine the runtime payload version and include that information in the message for better compatibility management. This property will be enhanced in a future release.
-
Property Name: ribMessageCount
Type: int
Required Property: false
Example: ribMessageCount=12
Description: This property contains the number of ribMessage nodes there are in a RibMessages xml message. This value gives us some indication of message aggregation in play. It might be used in the future to better optimize message flow paths based on the size/number of the messages.
-
Property Name: uuid
Type: java.lang.String
Required Property: false
Example: uuid=116cfabd-8949-4f93-bb61-aaa88e168f30
Description: This property contains a universally unique identifier for every message. This unique identifier will provide better traceability of a message within the JMS system. This property complements the ribMessageID xml element that is there to trace messages within the RIB logs.
Simple Message Flow
The typical lifecycle of a message through the RIB is as follows:
-
The publishing adapter creates the message. The event that triggers the message creation may be a polling operation in case of PL/SQL applications or a synchronous invoke in case of Java EE applications or a request in case of SOAP application. The message is published to a predetermined JMS topic.
-
The message is now available for all registered subscribers to the JMS topic for pick up. Subscription is based on the message family.
-
Once a subscriber gets the message, it is free to process that message according to its own rules. In the case of a transformer adapter, the adapter can open the message, modify its contents, and then publish the modified message to a new topic. The source topic and destination topic that a TAFR uses must always be distinct/different topics. There may be new subscribers to the modified message, and the scenario is repeated for each of these subscribers.
-
When each subscriber has finished (commit) processing a message, the JMS server updates the state of the message to reflect that it has been processed by this subscriber.
-
The JMS Server deletes the messages on the topic after delivering it to all the registered subscribers.
Two types of applications require this data and subscribe to it. One type of subscribing application requires a certain transformation be applied to the data, but the other type of subscriber can process the message without any transformations.
The RIB Hospital
The RIB Hospital is a collective term for a set of Java Classes and database tables whose purpose is to provide a mechanism to handle system and business related errors while meeting the fundamental RIB requirements:
-
Guaranteed once-and-only-once successful delivery.
-
Preservation of publication sequence (even in case of failures).
When a message is processed, the adapter checks to see if the RIB Hospital contains any messages associated with the same businessObjectId as the current message. If so, then the adapter places the current message in the hospital as well. This is to ensure messages are always processed in the proper sequence. If proper sequencing is not maintained, then the subscribing application's data can get corrupted.
If an error occurs during message processing, the subscribing adapter notes this internally and rolls back all work associated with the message. When the message is re-processed (since it is yet to be processed successfully), the adapter now recognizes this message is problematic and checks it into the hospital.
For Publication, there are some RMS publishers that return an 'H' status to denote a problem creating a new message for a specific business object. This status may be due to database locks being held by on-line users of an Oracle Forms application or it could also be due to some data incompatibility found in the GETNXT() procedure. Whenever a publisher recognizes that a message for a business object cannot be published due to one of these conditions, the message must go into the RIB Hospital.
After a message is checked into the RIB Hospital, a retry adapter extracts the message from the hospital and tries to re-publish it to the integration bus.
RIB Hospital Dependency Check
The RIB Hospital dependency check logic assumes that each message family has a single unique businessObjectId for all business object entities its messages are associated with. This businessObjectId must be the same for the same business entity across all message types within the message family. If any message for a specific business entity is placed into the RIB Hospital, then the RIB Hospital dependency check logic automatically inserts any subsequent messages for the same business object. This is to preserve the message sequencing and guaranteed exactly once successful message processing. Otherwise, multiple update messages for a business object may be processed in an incorrect order and create incompatibilities between applications.
If the businessObjectid is not set, then there is no dependency check. Not all message families set the businessObjectId or it is not set on all message types. See the Oracle Retail application documentation (for example, "Message Publication and Subscription Designs" in the Oracle Retail Merchandising System Operations Guide Volume 2).
RIB Hospital Insert
In an event of failure during message subscription, the error is flagged within the RIB Hospital software, resulting in rollback of the work done in the retail application, the adapter returns failure so that the database transaction is rolled back as well, and the message is kept on the integration bus topic. This is because subscribing adapters are executed within the context of a distributed transaction, using the XA two-phase commit protocol. This transaction is controlled by the Java EE Application Server. Immediately after the roll back, JMS re-delivers the message back to the subscribing adapter and this time the RIB Hospital software detects the previously flagged message and inserts the message in to the RIB Hospital tables and message is removed from the JMS topic.
When the initial failure occurs while processing the message, the error is flagged within the RIB Hospital software, the adapter returns failure so that the database transaction is rolled back, and the message is kept on the integration bus topic.
Note:
The XA interface is a standard protocol between a transaction manager and a database or resource manager. Note that both the JMS topic connection and the database connection must support the XA protocol. For more information regarding the XA standard, see the URL http://www.opengroup.org
.
RIB Hospital Tables
The RIB Hospital tables are:
-
RIB_MESSAGE - contains the message payload, all single-field envelope information, and a concatenated string made from <id> tags. It also contains a unique hospital ID identifying this record within the hospital.
-
RIB_MESSAGE_FAILURE - contains all failure information for each time the message was processed.
-
RIB_MESSAGE_ROUTING_INFO - contains all of the routing element information found in the message envelope.
-
RIB_MESSAGE_HOSPITAL_REF - contains all of the hospital reference information found in the message envelope.
A database sequence, RIB_MESSAGE_SEQ, is used to maintain a unique message number associated with each message placed into the RIB Hospital.
![Database Table Relationships Database Table Relationships](img/chapter_3.3.png)
These tables will have been created during the database portion of the Oracle Retail application installation (for example, RWMS, SIM, RPM, AIP, RFM, OMS, or RMS).
The RIB Hospital tables are internal system tables that maintain the RIB runtime state of the system. The entries in these tables must not be manipulated by non RIB tools when the RIB is running.
RIB Hospital Retry
After a message is inserted into the RIB Hospital, the hospital retry adapter is used to re-post the message to the JMS in order to retry its processing. The assumption is that the error is a transitory one; records locked or there is an external dependency that has not been met. The number of times a message is retried is configurable.
The hospital retry is responsible for maintaining state information for hospital records or what has happened to the record or message information. Each time the message is reprocessed, a record is kept of the event along with the results. The design is to provide a means to halt processing for messages that cause errors while allowing continued processing for the good messages.
One element of this information is whether the message has been queued to the JMS topic for re-try processing. So manually deleting messages from the hospital database using SQL directly may produce severe processing problems. Also, deleting messages directly from the JMS provider may result in a message that is never retried again, as the logic in the retry assumes the message is queued within the JMS.
There are three kinds of hospital retry adapters:
-
Sub Retry Adapter
-
JMS Retry Adapter
-
Pub Retry Adapter
All subscriber side retrying of messages are handled by the Sub Retry Adapter. The Sub Retry Adapter looks at all messages with reason code SUB, then filters and identifies the messages that are ready to be reprocessed, keeping message ordering in mind.
Oracle Retail applications are unaware that the integrations of the business data is happening through a JMS server. RIB abstracts the fact it is using a JMS server from the retail applications. When the JMS server is down or RIB has some problem publishing to the JMS server, RIB will not rollback the transaction as long as it is a recoverable problem. In such situation all messages are inserted to the RIB Hospital with a reason code of JMS and publications continues on. The JMS Retry Adapter retries all messages with reason code of JMS at a later time.
All messages with reason code of PUB are retried by the Pub Retry Adapter. RMS is the only retail application that needs the Pub Retry Adapter.
Hospital Attempt (Retry) Count
When the message first comes through the subscriber, if there is no businessObjectid, then there is no dependency check performed. If the message cannot be processed, it is then inserted into the hospital with an attempt_count = 1.
A message that comes through the subscriber, that has a businessObjectid, a dependency check is performed. If there is no dependency and the message cannot be processed, it is then inserted into the hospital with an attempt_count = 1.
A message that comes through the subscriber that does match the ID and family of another message in the hospital is known to be dependent, so it goes to the hospital immediately, with an attempt_count = 0.
Exception to this rib-tafr app, in case of rib-tafr attempt_count is 1, even if the message is inserted into the hospital as a dependent message because tafr adapters work with two topics and message would already be subscribed once by the tafr, therefore it always has attempt_count=1.
JMS Delivery Count
JMSXDeliveryCount is a message property set by AQ JMS. This property is checked to see if the message is being redelivered by the JMS. If the count MAX_REDELIVERY_THRESHOLD (set to 2) is reached, the RIB subscribers assume that the message is being re-delivered; the message will be determined as a poison message. The message is written to the file system (at the same location where application log files are written), and the adapter is shut down in such scenarios. An administrator must decide how this message will be handled.