| Oracle® Retail Integration Bus Operations Guide Release 16.0 E80332-01 |
|
 Previous |
 Next |
This chapter address common issues faced while operating the RIB.
This section contains common issues that need to be thought about and addressed by a retailer as they progress towards a production environment involving RIB. It is not a comprehensive list, nor does it seek to answer the questions, since they are very dependent on the retailer implementation. The intent of this section is to provide a starting point for a site-specific RIB operations planning effort.
RIB has built in alerts and notification through JMX. An external system can subscribe to all of the built-ins.
The RIB code has been instrumented to send alerts to notify interesting internal events like the following:
Adapter status changes
Dynamically changing runtime configuration
Fatal error condition that needs user intervention
Inconsistent persistence store because of user error
The following are the alerting mechanism supported in RIB:
JMX – JSR-174, JSR-160
Email - JavaMail
Any standard JMX client. (E.g. JConsole) can subscibe to RIB JMX notifications and receive notification/alerts. Follow the steps described in chapter 4 ”RIB and JMX” to configure jconsole and view notification/alerts in jconsole.
The JMX Client can also be coded to monitor jmx notifications and act upon the notifications (sample code see below).
public class Client { /** * Inner class that will handle the notifications. */ public static class ClientListener implements NotificationListener { public void handleNotification(Notification notification,Object handback) { // logic to act upon notifications goes here echo("\nReceived notification:"); ... echo("\tSource: " + notification.getSource()); echo("\tMessage: " + notification.getMessage()); }}public static void main(String[] args) throws Exception { ... echo("Connecting to the remote server");String protocol = "t3"; Integer portInteger = Integer.valueOf(portString); int port = portInteger.intValue(); String jndiroot = "/jndi/"; String mserver = "weblogic.management.mbeanservers.runtime";JMXServiceURL serviceURL = new JMXServiceURL(protocol, hostname, port, jndiroot + mserver); ... jmxc = JMXConnectorFactory.connect(serviceURL, h); ClientListener listener = new ClientListener(); // Get an MBeanServerConnection // mbsc = jmxc.getMBeanServerConnection(); // Construct the ObjectName for adding listener to ObjectName mbeanName = new ObjectName( "rib-rms:appName=rib-rms,name=ribLogManager"); // Add notification listener on ribLogManager mbean // echo("\nAdd notification listener..."); mbsc.addNotificationListener(mbeanName, listener, null, null); echo("\nWaiting for notification..."); echo("\nClose the connection to the server"); jmxc.close();}}
For email alerts, email IDs need to be configured at install time for RIB kernel to send out emails. The RIB deployment config file (rib-deployment-env-info.xml) contains notifications tag for each rib-<app> as below. Set all the values correctly to configure email alert for that app. RIB kernel sends out poison messages to email ids configured at install.Example:
<notifications> <email> <email-server-host>mail.example.com</email-server-host> <email-server-port>25</email-server-port> <from-address>rib-email@example.com</from-address> <to-address-list>rib@example.com</to-address-list> </email> <jmx/></notifications>
|
Note: RDMT also sends email alerts. For more information on configuring email alerts in RDMT, see the RIB Support Tool Guide. |
Because RIB is a subsystem that runs with no console, it is important to monitor the various log file that are created. Not only for the content (looking for exceptions), but also their size and growth.
RDMT includes several tools to assist in scanning and can provide examples on how to customize them to conform to a particular site.
RIB uses log4j2 for all of its logging control. It manages the logs size via its control file.
|
Note: See Apache Software Foundation>http://logging.apache.org/log4j2/2.x/manual/index.html for details. |
In various phases of deployment and in triaging a problem it is often desirable or necessary to archive the logs so that the logs are smaller and scanning them by tools or by people is easier. RDMT includes tools to assist and can provide examples on how to customize them to conform to a particular site.
The Hospital tables, wherever they are, need to be monitored for size and growth. They have a huge effect on the performance of the entire RIB. As it gets larger, several interfaces dramatically slow down.
RDMT includes tools to assist and can provide examples on how to customize them to conform to a particular site.
The MFQ and Upload table size and growth needs to be monitored. They can indicate a poorly performing (hung) adapter or forecast a slow interface because the Hospital tables are filling. In the case of some of the slower interfaces there will be slow down of dependency records being processed.
RDMT includes tools to assist and can provide examples on how to customize them to conform to a particular site.
If the situation exists where a retailer is deploying instances of RWMS in different geographic locations connect by a WAN then there are several RIB deployment architectural alternatives that need to be considered and decided.
The RIB components must be started and stopped in particular order, and there are recommendations on when and how to do this and tools to assist in building out operational processes to suite a retailer's site requirements.
It is always recommended that the order of startup be SUB, TAFR, PUB and the shutdown be in the reverse order. RIB supplies tools to control the adapter start and stop process in the proper sequence in the rib-app-builder tool called rib-adapter-controller.
The RIB application environment often presents a new dimension to a retailer's infrastructure, and there are training and support issues that do not fit the existing organization and current staff skill sets.
RIB contains code and configurations that are critical to the Enterprise. This version of RIB is designed to be centrally managed and contains tools for tracking inventory and versions and configuration changes. A backup strategy also needs to be developed specific to the site.
RIB has an inventory tracking mechanism that is maintained by the tools in the RIB Application Builder. These tools also manage the application of defect fixes and tracking the defect fixes applied in the inventory.
RIB is usually considered a HA requirement, so an architecture and operations plan to handle this needs to be developed.
|
Note: See the Oracle Retail Integration Bus Installation Guide: The RIB and Oracle Database Cluster (RAC)Oracle® WebLogic Application Server High Availability Guide Oracle® Database Administrator's Guide 12c Release 1 |
In addition to the HA requirements, there is the issue of message retention, auditing and recovery. It is common for an end-point application to experience an issue such as a crash that requires recovery or a rebuild. Syncing the data that the other applications have been publishing and subscribing to during the down time presents a major challenge.
It is important for a site to develop a plan and approach for this. In a large volume site, the JMS topics can build to huge numbers very quickly and over-run a system or the ability of the recovered system to catch-up in a time frame the business finds acceptable.
Regardless of the organization structure or where the staff reports to, there are two distinct sets of roles and capabilities needed: the RIB system administrator role and RIB application administrator role. The number of persons filling those roles is dependent on the size of the deployment, breadth of the products being integrated, levels of customization and schedule compression.
Integration support is a team effort, with one or two strong RIB administrators who can work through difficult failure modes using the RIB logs to help isolate the issue and determine type. Users with knowledge of Oracle Retail application (such as RMS, RWMS, and SIM) must also have a good level of RIB understanding. As a team, they triage issues and then work on them. By the Integration Test phase of an implementation, the types of RIB failure issues become more related to complicated data sets for business case tests. Gross level functionality issues are generally solved by then.
Production requirements are similar, but need to reflect the realities of pager rotation, 24x7 issues, as well as how many applications are deployed and over what geography.
This section describes the RIB System Administrator role and responsibilities.
UNIX (strong) - shell scripts and Unix tools
Oracle Database and Stored Procedures
Oracle WebLogic Application Server (strong)
Java EE (strong)—ability to read and understand exceptions and log files.
Message Oriented Middleware (MOM) or communication technologies.
Installation of WLS and patches
Configuration of Oracle WebLogic Application Server
Installation and configuration of RIB
Support and configuration of Adapters and patches
Operational issues such as backup/restore, failure analysis using RIBLOGS and Application Server logs as well as tools and various UNIX scripts and programs, and aid in the determination of error causes resulting in RIB Hospital entries.
Under normal operations, messages go into the hospital, get retried and are automatically deleted from the hospital. But if there is a steady increase in hospitalized messages, the reasons should be immediately determined and addressed.
Triage of messages placed in the RIB Hospital is a time consuming task. This is a difficult task when only Oracle Retail applications are involved; adding other outside applications, as many retailers do, further complicates this process. Problems can be introduced at the application level, in the extract, or the transformation process.
Having the integration team take a first look at the messages is another common practice at Oracle Retail customer sites. This team's success at resolving and correcting data issues is dependent on their access to business analysts who understand the desired function.
The RIB Hospital tables need to be monitored for size and growth. The number of entries in the RIB Hospital has a large impact on the performance of the entire RIB. Each adapter checks the RIB Hospital for previous related failures for each message (to see if the message should be held until any previous errors have been resolved). As the RIB Hospital gets larger, interfaces can dramatically slow down.
The RIB Hospital is a crucial component in the operation and performance of RIB. Processes and procedures to handle it are very important, and should be decided on and practiced early. It is suggested that discussions and planning be started as soon as possible in the implementation phase to work through the possible scenarios and develop tools and procedures to handle them.
There are tools in RDMT that can be leveraged to not only build monitoring scripts but to aid in the initial triage of issues.
Oracle Retail Integration Bus Hospital Administration (RIHA) is the recommended tool for maintenance of the Hospital. It understands the Hospital table structure and how to appropriately correct, submit and, as needed, delete messages. The use of tools such as SQLDeveloper or TOAD is discouraged. Although they allow similar activities, they do not provide the safe guards that RIHA has to maintain the integrity of the tables and the JMS.
RIB Hospital tables are packaged with applications and therefore reside in the base schema of the applications. To reduce maintenance, upgrade and support concerns, users may choose to extract Hospital tables from application schemas.
Using the RIB Application Builder tool, error Hospital tables can be removed from the application space and placed under the control of the RIB kernel, where data sources meant for Hospital-related database operations are differentiated from application calls (such as GetNext and Consume). The data source, hosp-managed-datasource, supports the separation of the Hospital schema from the application schema.
To facilitate the externalization of the RIB Hospital tables from the application schema, two placeholders (one for PL/SQL applications and one for JavaEE applications) exist in the rib-deployment-env-info.xml file, as described in Chapter 3, "Backend System Administration and Logging ."
