| Oracle® Retail Integration Bus Cloud Service Operations Guide Release 19.1.000 F31988-02 |
|
 Previous |
 Next |
| Oracle® Retail Integration Bus Cloud Service Operations Guide Release 19.1.000 F31988-02 |
|
Previous |
Next |
This chapter address common issues faced while operating the RIB.
This section contains common issues that need to be thought about and addressed by a retailer as they progress towards a production environment involving RIB. It is not a comprehensive list, nor does it seek to answer the questions, since they are very dependent on the retailer implementation. The intent of this section is to provide a starting point for a site-specific RIB operations planning effort.
RIB has built in alerts and notification through JMX. An external system can subscribe to all of the built-ins.
The RIB code has been instrumented to send alerts to notify interesting internal events like the following:
Adapter status changes
Dynamically changing runtime configuration
Fatal error condition that needs user intervention
Inconsistent persistence store because of user error
The following are the alerting mechanism supported in RIB:
JMX – JSR-174, JSR-160
Email - JavaMail
Any standard JMX client (for example JConsole) can subscibe to RIB JMX notifications and receive notification/alerts. Follow the steps described in chapter 4 ”RIB and JMX” to configure jconsole and view notification/alerts in jconsole.
The JMX Client can also be coded to monitor jmx notifications and act upon the notifications (sample code see below).
public class Client {
/**
* Inner class that will handle the notifications.
*/
public static class ClientListener implements NotificationListener {
public void handleNotification(Notification notification,Object handback) {
// logic to act upon notifications goes here
echo("\nReceived notification:");
...
echo("\tSource: " + notification.getSource());
echo("\tMessage: " + notification.getMessage());
}
}
public static void main(String[] args) throws Exception {
...
echo("Connecting to the remote server");
String protocol = "t3";
Integer portInteger = Integer.valueOf(portString);
int port = portInteger.intValue();
String jndiroot = "/jndi/";
String mserver = "weblogic.management.mbeanservers.runtime";
JMXServiceURL serviceURL = new JMXServiceURL(protocol, hostname, port,
jndiroot + mserver);
...
jmxc = JMXConnectorFactory.connect(serviceURL, h);
ClientListener listener = new ClientListener();
// Get an MBeanServerConnection
//
mbsc = jmxc.getMBeanServerConnection();
// Construct the ObjectName for adding listener to
ObjectName mbeanName = new ObjectName(
"rib-rms:appName=rib-rms,name=ribLogManager");
// Add notification listener on ribLogManager mbean
//
echo("\nAdd notification listener...");
mbsc.addNotificationListener(mbeanName, listener, null, null);
echo("\nWaiting for notification...");
echo("\nClose the connection to the server");
jmxc.close();
}
}
For email alerts, email IDs need to be configured at install time for RIB kernel to send out emails. The RIB deployment config file (rib-deployment-env-info.xml) contains notifications tag for each rib-<app> as below. Set all the values correctly to configure email alert for that app. RIB kernel sends out poison messages to email ids configured at install.Example:
<notifications>
<email>
<email-server-host>mail.example.com</email-server-host>
<email-server-port>25</email-server-port>
<from-address>rib-email@example.com</from-address>
<to-address-list>rib@example.com</to-address-list>
</email>
<jmx/>
</notifications>
|
Note: RDMT also sends email alerts. For more information on configuring email alerts in RDMT, see the RIB Support Tool Guide. |
Because RIB is a subsystem that runs with no console, it is important to monitor the various log file that are created. Not only for the content (looking for exceptions), but also their size and growth.
RDMT includes several tools to assist in scanning and can provide examples on how to customize them to conform to a particular site.
RIB uses log4j2 for all of its logging control. It manages the logs size via its control file.
|
Note: See Apache Software Foundation>http://logging.apache.org/log4j2/2.x/manual/index.html for details. |
In various phases of deployment and in triaging a problem it is often desirable or necessary to archive the logs so that the logs are smaller and scanning them by tools or by people is easier. RDMT includes tools to assist and can provide examples on how to customize them to conform to a particular site.
The Hospital tables, wherever they are, need to be monitored for size and growth. They have a huge effect on the performance of the entire RIB. As it gets larger, several interfaces dramatically slow down.
RDMT includes tools to assist and can provide examples on how to customize them to conform to a particular site.
The MFQ and Upload table size and growth needs to be monitored. They can indicate a poorly performing (hung) adapter or forecast a slow interface because the Hospital tables are filling. In the case of some of the slower interfaces there will be slow down of dependency records being processed.
RDMT includes tools to assist and can provide examples on how to customize them to conform to a particular site.
If the situation exists where a retailer is deploying instances of RWMS in different geographic locations connect by a WAN then there are several RIB deployment architectural alternatives that need to be considered and decided.
The RIB components must be started and stopped in particular order, and there are recommendations on when and how to do this and tools to assist in building out operational processes to suite a retailer's site requirements.
It is always recommended that the order of startup be SUB, TAFR, PUB and the shutdown be in the reverse order. RIB supplies tools to control the adapter start and stop process in the proper sequence in the rib-app-builder tool called rib-adapter-controller.
The RIB application environment often presents a new dimension to a retailer's infrastructure, and there are training and support issues that do not fit the existing organization and current staff skill sets.
RIB contains code and configurations that are critical to the Enterprise. This version of RIB is designed to be centrally managed and contains tools for tracking inventory and versions and configuration changes. A backup strategy also needs to be developed specific to the site.
RIB has an inventory tracking mechanism that is maintained by the tools in the RIB Application Builder. These tools also manage the application of defect fixes and tracking the defect fixes applied in the inventory.
RIB is usually considered a high availability (HA) requirement, so an architecture and operations plan to handle this need is developed.
To fulfill the HA requirement for RIB, a useful feature was added to RIB that allows using single rib-home for multiple environments. There is a new script added in rib-home for setting up multiple environment from same rib-home. This script must be run before rib-app compilation. There will be different deployment file for each environment. For example, there are two WebLogic domains available: say node1 in RICS VM1 and node2 in RICS VM2. To deploy RIB on these two servers, rename the specific deployment file for that with an extension sample below.
rib-deployment-xml.node1
rib-deployment-xml.node2
Then, before compilation, run the script to specify which environment you want to deploy to.
For example: select-deployment-options.sh -deployment-file-extension {active|passive|node1|node2} -set-adapter-status {adapter-up|adapter-down} -for-rib-app {all|rib-rms|rib-sim}
Usage:
deployment-file-extension is a required parameter and a file name needs to be passed for value. For example, if node1 is passed as a value for this parameter, rib-deployment-env-info.xml.node1 will be used for deployment configuration.
set-adapter-status is optional. Default is up. If this option is not provided, there won't be any change in the initial state of the adapters. If adapter-up is provided for this parameter, the initial state of all the adapters will be changed from stopped to running and vice versa.
for-rib-app is an optional parameter. Default is all.
|
Note: See the Oracle Retail Integration Bus Installation Guide: RIB High Availability Installation InstructionsSee the Oracle Retail Integration Bus Installation Guide: The RIB and Oracle Database Cluster (RAC) Oracle® WebLogic Application Server High Availability Guide Oracle® Database Administrator's Guide 12c Release 1 |
In addition to the HA requirements, there is the issue of message retention, auditing and recovery. It is common for an end-point application to experience an issue such as a crash that requires recovery or a rebuild. Syncing the data that the other applications have been publishing and subscribing to during the down time presents a major challenge.
It is important for a site to develop a plan and approach for this. In a large volume site, the JMS topics can build to huge numbers very quickly and over-run a system or the ability of the recovered system to catch-up in a time frame the business finds acceptable.
Regardless of the organization structure or where the staff reports to, there are two distinct sets of roles and capabilities needed: the RIB system administrator role and RIB application administrator role. The number of persons filling those roles is dependent on the size of the deployment, breadth of the products being integrated, levels of customization and schedule compression.
Integration support is a team effort, with one or two strong RIB administrators who can work through difficult failure modes using the RIB logs to help isolate the issue and determine type. Users with knowledge of Oracle Retail application (such as RMS, RWMS, and SIM) must also have a good level of RIB understanding. As a team, they triage issues and then work on them. By the Integration Test phase of an implementation, the types of RIB failure issues become more related to complicated data sets for business case tests. Gross level functionality issues are generally solved by then.
Production requirements are similar, but need to reflect the realities of pager rotation, 24x7 issues, as well as how many applications are deployed and over what geography.
This section describes the RIB System Administrator role and responsibilities.
UNIX (strong) - shell scripts and Unix tools
Oracle Database and Stored Procedures
Oracle WebLogic Application Server (strong)
Java EE (strong)—ability to read and understand exceptions and log files.
Message Oriented Middleware (MOM) or communication technologies.
Installation of WLS and patches
Configuration of Oracle WebLogic Application Server
Installation and configuration of RIB
Support and configuration of Adapters and patches
Operational issues such as backup/restore, failure analysis using RIBLOGS and Application Server logs as well as tools and various UNIX scripts and programs, and aid in the determination of error causes resulting in RIB Hospital entries.
Under normal operations, messages go into the hospital, get retried and are automatically deleted from the hospital. But if there is a steady increase in hospitalized messages, the reasons should be immediately determined and addressed.
Triage of messages placed in the RIB Hospital is a time consuming task. This is a difficult task when only Oracle Retail applications are involved; adding other outside applications, as many retailers do, further complicates this process. Problems can be introduced at the application level, in the extract, or the transformation process.
Having the integration team take a first look at the messages is another common practice at Oracle Retail customer sites. This team's success at resolving and correcting data issues is dependent on their access to business analysts who understand the desired function.
The RIB Hospital tables need to be monitored for size and growth. The number of entries in the RIB Hospital has a large impact on the performance of the entire RIB. Each adapter checks the RIB Hospital for previous related failures for each message (to see if the message should be held until any previous errors have been resolved). As the RIB Hospital gets larger, interfaces can dramatically slow down.
The RIB Hospital is a crucial component in the operation and performance of RIB. Processes and procedures to handle it are very important, and should be decided on and practiced early. It is suggested that discussions and planning be started as soon as possible in the implementation phase to work through the possible scenarios and develop tools and procedures to handle them.
There are tools in RDMT that can be leveraged to not only build monitoring scripts but to aid in the initial triage of issues.
Oracle Retail Integration Bus Hospital Administration (RIHA) is the recommended tool for maintenance of the Hospital. It understands the Hospital table structure and how to appropriately correct, submit and, as needed, delete messages. The use of tools such as SQLDeveloper or TOAD is discouraged. Although they allow similar activities, they do not provide the safe guards that RIHA has to maintain the integrity of the tables and the JMS.
RIB Hospital tables are packaged with applications and therefore reside in the base schema of the applications. To reduce maintenance, upgrade and support concerns, users may choose to extract Hospital tables from application schemas.
Using the RIB Application Builder tool, error Hospital tables can be removed from the application space and placed under the control of the RIB kernel, where data sources meant for Hospital-related database operations are differentiated from application calls (such as GetNext and Consume). The data source, hosp-managed-datasource, supports the separation of the Hospital schema from the application schema.
To facilitate the externalization of the RIB Hospital tables from the application schema, two placeholders (one for PL/SQL applications and one for JavaEE applications) exist in the rib-deployment-env-info.xml file, as described in Chapter 3, "Backend System Administration and Logging."
The RIB runtime framework provides a set of APIs that help in preserving runtime configuration. This feature makes upgrading user-friendly by preserving user-edited configurations that can be used in subsequent installs. Updated configurations will be persisted across restart and re-deployment of rib-applications.
The new value updated through the UI is first saved into the database. Then the in-memory object is updated with the same value.
When someone restarts the app, the same process happen automatically. Values are first read from database, then the in-memory model is updated with the value from the database.
During the upgrade process, values in the database will not be purged. Then user-saved runtime values will also be used in the next install.
Database Table for Runtime Configuration
'RIB_RUNTIME_CONFIG_OVERRIDE' is the table name that is automatically created during the deployment of rib-apps. The runtime table shares the schema with the error hospital which is rib-schema.
RIB comprises of following configuration files, any updates to below files gets saved in runtime table.
rib-system.propertires and rib-<app> (kernel properties and application properties)
soap-service-consumer-config.xml (only for soap-app)
adapters-in-scope (only for rib-ext)
log4j.xml
The RIB administration GUI provisions updating the above configuration files:
RIB Properties - The System Options tab in Manage Configuration lists all the RIB properties available for update at runtime. Any updates to properties here is stored in database runtime table. See chapter "Backend System Administration and Logging" for a detailed description of all the RIB properties and what users should and should no update.
Injector Service - The Injector Service tab in Manage Configurations exposes the injector service end-point URL for primary-secondary, rest-app and soap-app. Any updates to the end-point URL and policy name will persist in the runtime table.
Adapter Selection - The Adapter Selection tab under Manage Configurations allows users to pick a small set of adapters at runtime for rib-ext. The selected adapters list is written to the runtime table.
Log Level Change -To update the log level for any RIB application logger, you can login to the RIB Admin GUI, go to Log Manager then select the needed log level against each logger, then click Apply. The selected log level for each logger is persisted into the RUNTIME_OVERRIDE_CONFIG table in the RIB error hospital schema. These values are persisted across restart and re-deployment of rib-applications.
Until the previous version, the log levels were persisted to log4j.xml.But from this version onwards, the log levels for the loggers will not be persisted to the file log4j2.xml, and instead they will persist into database to facilitate upgrades.
The log levels for many adapter logs are initially loaded from the lof4j2.xml file when the managed server starts. Further, if any override log-level value exists for the logger in the RUNTIME_OVERRIDE_CONFIG table, the log level is updated to this value on application startup. The RIB LogManager UI will show these log levels.
RIB uses a database store instead of wallet files for storing runtime credentials. There are no changes in the installer to support this. Compilation and deployment steps remains same as before. During the rib-<app> deployment, the deployer creates runtime credentials in the database store instead of copying the wallet files from the managed server home. These runtime credentials can be viewed from WebLogic Enterprise Manager console:
Log into the WebLogic Enterprise Manager.
Click Domain -> Security - > Credentials.
Open appTag to view or edit credentials.
