Troubleshooting
When an Oracle Utilities Network Management System application or Service is experiencing problems, some helpful information would include an isis dump of the applications process, the log file associated with that application, and the output of the processes list.
Generating an Isis Dump File
To generate an isis dump file, complete these steps:
1. On each node of concern:
ps -aef > $(hostname)_ps.out
2. Identify the process ID of the problem application(s).
grep -i [application] $(hostname)_ps.out
3. Use the following command to generate an isis dump file for a specific process:
kill -USR2 [pid]
The process will not be affected and will continue to operate, but upon receiving the USR2 signal, it will generate an isis dump ([pid].log) in the directory from which that tool was launched.
Note: Any subsequent USR2 messages will result in the process appending a new isis dump to the [pid].log file. Only the user running the applications can perform this action.
Generating an Isis Dump File for All Applications
An alternative is to issue the cmd snapshot command, which will create an isis dump for all applications connected to the isis message bus. The applications will continue to run, but every single application running will create an isis dump file. This will clutter the file system, but it is sometimes the best way to gather all the information you need to investigate a problem.
Use the Oracle Utilities Network Management Systems nms-snapshot to execute an isis cmd snapshot command and properly gather the relevant logs. Run nms-snapshot with no options or the ‑h option for information on what options are available.
Most commonly, nms-snapshot is run with the -a option, which will run cmd snapshot and get stack dumps from DBService and dbrdwr processes twice (waiting 20 seconds between snapshots/dumps), and gather all logs for the last day. It will create a zip file in $NMS_LOG_DIR that can be sent to Customer Support for investigation. With the full set of logs, Customer Support can track interactive messaging for problem investigation and resolution.
Reporting a Problem to Customer Support
In general, when reporting a problem to Customer Support, the following information can speed the problem identification and resolution process:
• An explanation of what the observed symptoms were and where they occurred.
• An explanation of how to repeat the problem, if possible.
• An explanation of expected behavior.
• A specific time frame when the problem occurred.
• Example data demonstrating the problem (e.g., event numbers, crew names, etc.).
• Service logs and environment log files of the affected Services/Applications.
• Isis dumps of the affected application and services at the time the problem was observed. A complete isis dump of all processes may be requested if the problem is repeatable, along with a process list output file.
• The core file trace, if a core file exists for the process.
• Any other activity that occurred prior to, or concurrent with, the issue that may stand out as a possible contributor.