This section contains common issues that need to be thought about and addressed by a retailer as they progress towards a production environment involving the RIB. It is not a comprehensive list, nor does it seek to answer the questions, since they are very dependent on the retailer implementation. The intent of this section is to provide a starting point for a site-specific RIB Operations planning effort.
The RIB has built in alerts and notification through JMX. An external system can subscribe to all of the built-ins.
Because the RIB is a subsystem that runs with no console, it is important to monitor the various log file that are created. Not only for the content (looking for exceptions), but also their size and growth.
RDMT includes several tools to assist in scanning and can provide examples on how to customize them to conform to particular site.
The RIB use log4j for all of its logging control. It manages the logs size via its control file.
Note: See Apache Software Foundationhttp://logging.apache.org/log4j/docs/documentation.html for details. |
In various phases of deployment and in triaging a problem it is often desirable or necessary to archive the logs so the logs are smaller and scanning by tools or people is easier. RDMT includes tools to assist and can provide examples on how to customize them to conform to particular site.
The Hospital tables, wherever they are, need to be monitored for size and growth. They have a huge effect on the performance of the entire RIB. As it gets larger, several interfaces dramatically slow down.
RDMT includes tools to assist and can provide examples on how to customize them to conform to particular site.
The MFQ and Upload table size and growth need to be monitored. They can indicate a poorly performing (hung) adapter or forecast a slow interface because the Hospital tables are filling. In the case of some of the slower interfaces there will be slow down of dependency records being processed.
RDMT includes tools to assist and can provide examples on how to customize them to conform to particular site.
If the situation exists where a retailer is deploying instances of RWMS in different geographic locations connect by a WAN then there are several RIB deployment architectural alternatives that need to be considered and decided.
The RIB component must be started and stopped in particular order, and there are recommendations on when and how to do this and tools to assist in building out operational processes to suite a retailers site requirements.
It is always recommended that the order of startup be SUB, TAFR, PUB and the shutdown is in the reverse order. The RIB supplies tools to control the adapter start and stop process in the proper sequence in the rib-app-builder tool called rib-adapter-controller.
The RIB application environment often presents a new dimension to a retailer's infrastructure, and there are training and support issues that do not fit the existing organization and current staff skill sets.
The RIB contains code and configurations that are critical to the Enterprise. This version of the RIB is designed to be centrally managed and contains tools for tracking inventory and versions and configuration changes. A backup strategy also needs to be developed specific to the site.
The RIB has an inventory tracking mechanism that is maintained by the tools in the RIB App Builder. These tools also manage the application of defects and tracking the defects applied in the inventory.
The RIB is usually considered a HA requirement, so an architecture and operations plan to handle this needs to be developed.
Note: See RIB Install Guide: The RIB and Oracle Database Cluster (RAC)Oracle® Application Server High Availability Guide 10g Release 3 (10.1.3.3)Oracle® Database Administrator's Guide 10g Release 2 (10.2) |
In addition to the HA requirements there is the issue of message retention, auditing and recovery. It is a common issue that an end-point application experiences an issue such as a crash, and requires recovery or rebuild. Syncing the data that the other applications have been publishing and subscribing to during the down time presents a major challenge.
It is important for a site to develop a plan and approach for this. In a large volume site, the JMS topics can build to huge numbers very quickly and over-run a system or the ability of the recovered system to catch-up in a time frame the business finds acceptable.
Note: See the section, "RIB Audit Logs", in this manual.See Chapter 8, "Diagnostic and Monitoring Tools" in this manual. |
Regardless of the organization structure or where the staff reports to, there are two distinct sets of roles and capabilities needed; the RIB system administrator role and RIB application administrator role. The number of the persons filling those roles is dependent on the size of the deployment, breadth of the products being integrated, levels of customization and schedule compression.
Integration support is a team effort, with one or two strong RIB Admin people who can help work through difficult failure modes using the RIB logs to help isolate the issue and determine type. There also needs to be Oracle Retail Application knowledgeable persons (RMS, RWMS, SIM, etc.) that also have a good level of RIB understanding. As a team they triage the issue and then work them. By the Integration Test phase of an implementation, the types of RIB failure issues become more related to complicated data sets for business case tests. Gross level functionality issues are generally solved by then.
Production requirements are similar, but need to reflect the realities of pager rotation, 24x7 issues, as well as how many applications are deployed and over what geography.
UNIX (strong) - shell scripts, Unix tools
Oracle Database and Stored Procedures
Oracle Application Server (strong)
JavaEE (strong) - ability to read and understand exceptions and log files.
Message-Oriented-Middleware (MOM) or communication technologies.
Installation or OAS and patches
Configuration of Oracle Application Server
Installation and configuration of the RIB
Support and configuration of Adapters and well as patches.
Operational issues such as backup/restore, failure analysis using RIBLOGS and App Server logs as well as tools and various UNIX scripts and programs, and aid in the determination of error causes resulting in RIB Hospital entries.
Under normal operations, messages go into the hospital, get retried and are automatically deleted from the hospital. But if there is a steady increase in hospitalized messages, the reasons should be immediately determined and worked.
Triage of messages placed in the RIB Hospital is a time consuming task. This is a tough task when only Oracle Retail applications are involved; adding other outside applications, as many retailers do, further complicates this process. Problems can be introduced at the application level, in the extract, or the transformation process.
Having the integration team take a first look at the messages is another common practice at Oracle Retail customer sites. This team's success at resolving and correcting data issues is dependant on their access to business analysts who understand the desired function.
The RIB Hospital tables need to be monitored for size and growth. The number of entries in the RIB Hospital has a large impact on the performance of the entire RIB. Each adapter checks the RIB Hospital for previous related failures for each message (to see if the message should be held until any previous errors have been resolved). As the RIB Hospital gets larger interfaces can dramatically slow down.
The RIB Hospital is a crucial component in the operation and performance of the RIB. Processes and procedures to handle it are very important, and should be decided on and practiced early. It is suggested that discussions and planning be started as soon as possible in the implementation phase to work through the possible scenarios and develop tools and procedures to handle them.
There are tools in RDMT that can be leveraged to not only build monitoring scripts but to aid in the initial triage of issues.
RIHA is the recommended tool for maintenance of the Hospital. It understands the Hospital table structure and how to appropriately correct, submit and, as needed, delete messages. The use of tools such as SQLDeveloper or TOAD is discouraged. Although they allow similar activities, they do not provide the safe guards of RIHA in maintaining the integrity of the tables and the JMS.
RIB Hospital tables are packaged with applications and therefore reside in the base schema of the applications. To reduce maintenance, upgrade and support concerns, users may choose to extract Hospital tables from application schemas.
Using the RIB Application Builder tool, error Hospital tables can be removed from the application space and placed under the control of the RIB kernel, where data sources meant for Hospital-related database operations are differentiated from application calls (such as GetNext and Consume). The datasource, hosp-managed-datasource, supports the separation of the Hospital schema from the application schema.
To facilitate the externalization of the RIB Hospital tables from the application schema, two placeholders (one for PL/SQL applications and one for JavaEE applications) exist in the rib-deployment-env-info.xml file, as described in Chapter 3 of this guide.