1. X7-2 Deployment and User’s Guide
  2. Troubleshooting Oracle Database Appliance

17 Troubleshooting Oracle Database Appliance

Understand tools you can use to validate changes and troubleshoot Oracle Database Appliance problems.

Viewing Component Information on the Appliance

View details of all the components installed on the appliance, and the RPM drift information.

Viewing the Bill of Materials in the Web Console

Use the Appliance tab in the Web Console to view information about your deployment and the installed components. The Advanced Information tab displays information about the following components:

  • Grid Infrastructure Version, and the home directory

  • Database Version, Home location, and Edition

  • Location and details about the databases configured

  • All patches applied to the appliance

  • Firmware Controller and Disks

  • ILOM information

  • BIOS version

  • List of RPMs

In the List of RPMs section, click Show and then click RPM Drift to view the differences between the RPMs installed on the appliance, and the RPMs shipped in the latest Oracle Database Appliance Patch Bundle Update release.

Click Download to save the components report. You can use this report to help diagnose any deployment issues.

Viewing the Bill of Materials from the Command Line

The bill of materials is also available through the command line for bare metal and virtualized platforms deployments. The information about the installed components is collected according to a set schedule, and stored in the location /opt/oracle/dcs/Inventory/ for bare metal deployments and in the /opt/oracle/oak/Inventory/ directory for virtualized platforms. The file is stored in the format oda_bom_TimeStamp.json. Use the command describe-system to view the bill of materials on the command line. See the Oracle Database Command-Line Interface chapter for command options and usage notes.

Example 17-1 Example Command to View the Bill of Materials from the Command Line for Bare Metal Deployments

# odacli describe-system -b
ODA Components Information 
------------------------------
Component Name                Comonent Details                                             
---------------               ----------------------------------------------------------------------------------------------- 
NODE                          Name : rwsoda6m003 
                              Domain Name :  
                              Time Stamp : July 29, 2018 7:00:12 PM UTC 

  
RPMS                          Installed RPMS : acl-2.2.49-7.el6_9.1.x86_64,
                                               aide-0.14-11.el6.x86_64,
                                               alsa-lib-1.1.0-4.el6.x86_64,
                                               at-3.1.10-49.el6.x86_64,
                                               atk-1.30.0-1.el6.x86_64,
                                               attr-2.4.44-7.el6.x86_64,
                                               audit-2.4.5-6.el6.x86_64,
                                               audit-libs-2.4.5-6.el6.x86_64,
                                               audit-libs-python-2.4.5-6.el6.x86_64,
                                               augeas-libs-1.0.0-10.el6.x86_64,
                                               authconfig-6.1.12-23.el6.x86_64,
                                               avahi-libs-0.6.25-17.el6.x86_64,
                                               b43-openfwwf-5.2-10.el6.noarch,
                                               basesystem-10.0-4.0.1.el6.noarch,
                                               bash-4.1.2-48.el6.x86_64,
                                               bc-1.06.95-1.el6.x86_64,
                                               bind-libs-9.8.2-0.62.rc1.el6_9.5.x86_64,
                                               bind-utils-9.8.2-0.62.rc1.el6_9.5.x86_64,
                                               binutils-2.20.51.0.2-5.47.el6_9.1.x86_64,
                                               biosdevname-0.7.2-1.el6.x86_64,
                                               bridge-utils-1.2-10.el6.x86_64,
                                               busybox-1.15.1-21.el6_6.x86_64,
                                               bzip2-1.0.5-7.el6_0.x86_64,
                                               bzip2-libs-1.0.5-7.el6_0.x86_64,
                                               ca-certificates-2017.2.14-65.0.1.el6_9.noarch,
                                               cairo-1.8.8-6.el6_6.x86_64,
                                               celt051-0.5.1.3-0.el6.x86_64,
                                               checkpolicy-2.0.22-1.el6.x86_64,
                                               chkconfig-1.3.49.5-1.el6.x86_64,
                                               cloog-ppl-0.15.7-1.2.el6.x86_64,
                                               compat-libcap1-1.10-1.x86_64,
                                               compat-libstdc++-33-3.2.3-69.el6.x86_64,
                                               compat-readline5-5.2-17.1.el6.x86_64,
                                               compat-sap-c++-4.8.2-16.el6.x86_64,
                                               ConsoleKit-0.4.1-6.el6.x86_64,
                                               ConsoleKit-libs-0.4.1-6.el6.x86_64,
                                               ConsoleKit-x11-0.4.1-6.el6.x86_64,
                                               coreutils-8.4-46.0.1.el6.x86_64,
                                               coreutils-libs-8.4-46.0.1.el6.x86_64,
                                               cpio-2.10-13.el6.x86_64,
                                               cpp-4.4.7-18.el6.x86_64,
                                               cpupowerutils-1.3-2.el6.x86_64,
                                               cpuspeed-1.5-22.0.1.el6.x86_64,
                                               cracklib-2.8.16-4.el6.x86_64,
                                               cracklib-dicts-2.8.16-4.el6.x86_64,
                                               crash-7.1.4-1.0.1.el6_7.x86_64,
                                               crda-3.13_2015.10.22-3.el6.x86_64,
                                               createrepo-0.9.9-27.el6_9.noarch,
                                               cronie-1.4.4-16.el6_8.2.x86_64,
                                               cronie-anacron-1.4.4-16.el6_8.2.x86_64,
                                               crontabs-1.10-33.el6.noarch,
                                               cryptsetup-luks-1.2.0-11.el6.x86_64,
                                               cryptsetup-luks-libs-1.2.0-11.el6.x86_64,
                                               cups-libs-1.4.2-78.el6_9.x86_64,

....
....
....

Example 17-2 Example Command to View the Bill of Materials from the Command Line for Virtualized Platforms

# oakcli describe-system -b

Example 17-3 Example Command to View the Bill of Materials Report from the Stored Location

# /opt/oracle/dcs/Inventory/
total 264
-rw-r--r-- 1 root root 83550 Apr 26 05:41 oda_bom_2018-04-26_05-41-36.json

Parent topic: Troubleshooting Oracle Database Appliance

Errors When Logging into the Web Console

If you have problems logging into the Web Console, then it may be due to your browser or credentials.

Note:

Oracle Database Appliance uses self-signed certificates. Your browser determines how you log into the Web Console. Depending on the browser and browser version, you may receive a warning or error that the certificate is invalid or not trusted because it is self-signed, or that the connection is not private. Ensure that you accept the self-signed certificate for the agent and Web Console.

Follow these steps to log into the Web Console:

  1. Open a browser window.
  2. Go to the following URL: https://ODA-host-ip-address:7093/mgmt/index.html
  3. Get the security certificate (or certificate), confirm the security exception, and add an exception.
  4. Log in with your Oracle Database Appliance credentials.
    If you have not already set the oda-admin password, then a message is displayed, advising you to change the default password to comply with your system security requirements.
  5. If you have not added an exception for the agent security certificate, then a message about accepting agent certificate is displayed.
  6. Using a different tab in your browser, go to the following URL: https://ODA-host-ip-address:7070/login
  7. Get the security certificate (or certificate), confirm the security exception, and add an exception.
  8. Refresh the Web Console URL : https://ODA-host-ip-address:7093/mgmt/index.html

Related Topics

Parent topic: Troubleshooting Oracle Database Appliance

Errors when re-imaging Oracle Database Appliance

Understand how to troubleshoot errors that occur when re-imaging Oracle Database Appliance.

If re-imaging Oracle Database Appliance fails, with old header issues such as errors in storage discovery, or in running GI root scripts, or disk group RECO creation, then use the force mode with cleanup.pl. 

# perl cleanup.pl -f

To ensure that re-imaging is successful, remove the old headers from the storage disks by running the secure erase tool. Verify that the OAK/ASM headers are removed.

# cleanup.pl -erasedata
# cleanup.pl -checkHeader

Retry the re-imaging operation.

Related Topics

Parent topic: Troubleshooting Oracle Database Appliance

Using the Oracle ORAchk Health Check Tool

Use the ORAchk Health Check Tool to audit configuration settings and check system health.

Oracle ORAchk Health Check Tool performs proactive heath checks for the Oracle software stack and scans for known problems.

Oracle ORAchk Health Check Tool audits important configuration settings for Oracle RAC two-node deployments in the following categories:

  • Operating system kernel parameters and packages

  • Oracle Database

  • Database parameters, and other database configuration settings

  • Oracle Grid Infrastructure, which includes Oracle Clusterware and Oracle Automatic Storage Management

ORAchk is aware of the entire system. It checks the configuration to indicate if best practices are being followed.

See Also:

For more information about ORAchk, see My Oracle Support note 1268927.2, "ORAchk Health Checks for the Oracle Stack" at https://support.oracle.com/rs?type=doc&id=1268927.2

Note:

Before running ORAchk, check for the latest version of ORAchk, and download and install it.

Running ORAchk on Oracle Database Appliance 18.3 Baremetal Systems for New Installation

When you install Oracle Database Appliance 18.3, the ORAchk RPMs are installed in the directory /opt/oracle.SupportTools/orachk/. You can verify that ORAchk is installed by running the following command: 

[root@oak bin]# rpm -q orachk 
orachk-18.3.0_20180808-2.x86_64

Run orachk as follows:

[root@oak bin]# orachk   
This computer is for [S]ingle instance database or part of a [C]luster to run 
RAC database [S|C] [C]: S 
orachk did not find the inventory location on oak from environment. Does oak 
have Oracle software installed [y/n][n]? n 
... 
Detailed report (html) - 
/opt/oracle.SupportTools/orachk/orachk_oak_091918_182425/orachk_oak_091918_182 
425.html 

UPLOAD [if required] - 
/opt/oracle.SupportTools/orachk/orachk_oak_091918_182425.zip

Running orachk on Oracle Database Appliance Baremetal Systems for Releases Earlier than 18.3

  1. Open the command-line interface as root.
  2. Navigate to the ORAchk tool in the /suptools directory.
    /u01/app/12.2.0.1/grid/suptools/orachk
  3. Run the utility.
    ./orachk
    When all checks are finished, a detailed report is available. The output displays the location of the report in an HTML format and the location of a zip file if you want to upload the report.
  4. Review the Oracle Database Appliance Assessment Report and system health and troubleshoot any issues that are identified.
    The report includes a summary and filters that enable you to focus on specific areas. For example, you can choose the filter to show failed checks only, show checks with a Fail, Warning, Info, or Pass status, or any combination.

Parent topic: Troubleshooting Oracle Database Appliance

About Trace File Analyzer Collector

Trace File Analyzer (TFA) Collector simplifies diagnostic data collection on Oracle Grid Infrastructure and Oracle Real Application Clusters systems.

TFA behaves in a similar manner to the ion utility packaged with Oracle Clusterware. Both tools collect and package diagnostic data. However, TFA is much more powerful than ion, because TFA centralizes and automates the collection of diagnostic information.

TFA provides the following key benefits and options:

  • Encapsulation of diagnostic data collection for all Oracle Grid Infrastructure and Oracle RAC components on all cluster nodes into a single command, which you run from a single node

  • Option to "trim" diagnostic files during data collection to reduce data upload size

  • Options to isolate diagnostic data collection to a given time period, and to a particular product component, such as Oracle ASM, RDBMS, or Oracle Clusterware

  • Centralization of collected diagnostic output to a single node in Oracle Database Appliance, if desired

  • On-Demand Scans of all log and trace files for conditions indicating a problem

  • Real-Time Scan Alert Logs for conditions indicating a problem (for example, Database Alert Logs, Oracle ASM Alert Logs, and Oracle Clusterware Alert Logs)

See Also:

Refer to My Oracle Support note 1513912.1 "TFA Collector - Tool for Enhanced Diagnostic Gathering" for more information. https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=1513912.1

Parent topic: Troubleshooting Oracle Database Appliance

Running Trace File Analyzer (TFA) Collector Commands

Understand the installed location of tfactl and the options for the command.

About Using tfactl to Collect Diagnostic Information

Trace File Analyzer (TFA) Collector is installed in the directory /opt/oracle/tfa/tfa_home, The command line utility for TFA, tfactl can be invoked from the directory /opt/oracle/tfa/tfa_home/bin/tfactl.

Use the following command to run tfactl: 

/opt/oracle/tfa/tfa_home/bin/tfactl diagcollect -ips|-oda|-odalite|-dcs|-odabackup|
-odapatching|-odadataguard|-odaprovisioning|-odaconfig|-odasystem|-odastorage|-database|
-asm|-crsclient|-dbclient|-dbwlm|-tns|-rhp|-procinfo|-afd|-crs|-cha|-wls|
-emagent|-oms|-ocm|-emplugins|-em|-acfs|-install|-cfgtools|-os|-ashhtml|-ashtext|
-awrhtml|-awrtext -mask -sanitize

Table 17-1 Command Options for tfactl Tool

Option Description
-h

(Optional) Describes all the options for this command.
-ips

(Optional) Use this option to view the diagnostic logs for the specified component.
-oda

(Optional) Use this option to view the logs for the entire Appliance.
-odalite

(Optional) Use this option to view the diagnostic logs for the odalite component.
-dcs

(Optional) Use this option to view the DCS log files.
-odabackup

(Optional) Use this option to view the diagnostic logs for the backup components for the Appliance.
-odapatching

(Optional) Use this option to view the diagnostic logs for patching components of the Appliance.
-odadataguard

(Optional) Use this option to view the diagnostic logs for Oracle Data Guard component of the Appliance.
-odaprovisioning

(Optional) Use this option to view provisioning logs for the Appliance.
-odaconfig

(Optional) Use this option to view configuration-related diagnostic logs.
-odasystem

(Optional) Use this option to view the diagnostic logs for the specified component.
-odastorage

(Optional) Use this option to view the diagnostic logs for the Appliance storage.
-database

(Optional) Use this option to view database-related log files.
-asm

(Optional) Use this option to view the diagnostic logs for the Appliance.
-crsclient

(Optional) Use this option to view the diagnostic logs for the Appliance.
-dbclient

(Optional) Use this option to view the diagnostic logs for the Appliance.
-dbwlm

(Optional) Use this option to view the diagnostic logs for the specified component.
-tns

(Optional) Use this option to view the diagnostic logs for TNS.
-rhp

(Optional) Use this option to view the diagnostic logs for Rapid Home Provisioning.
-afd

(Optional) Use this option to view the diagnostic logs for Oracle ASM Filter Driver.
-crs

(Optional) Use this option to view the diagnostic logs for Oracle Clusterware.
-cha

(Optional) Use this option to view the diagnostic logs for the Cluster Health Monitor.
-wls

(Optional) Use this option to view the diagnostic logs for Oracle WebLogic Server.
-emagent

(Optional) Use this option to view the diagnostic logs for the Oracle Enterprise Manager agent.
-oms

(Optional) Use this option to view the diagnostic logs for the Oracle Enterprise Manager Management Service.
-ocm

(Optional) Use this option to view the diagnostic logs for the specified component.
-emplugins

(Optional) Use this option to view the diagnostic logs for Oracle Enterprise Manager plug-ins.
-em

(Optional) Use this option to view the diagnostic logs for Oracle Enterprise Manager deployment.
-acfs

(Optional) Use this option to view the diagnostic logs for Oracle ACFS storage.
-install

(Optional) Use this option to view the diagnostic logs for installation.
-cfgtools

(Optional) Use this option to view the diagnostic logs for the configuration tools.
-os

(Optional) Use this option to view the diagnostic logs for the operating system.
-ashhtml

(Optional) Use this option to view the diagnostic logs for the specified component.
-ashtext

(Optional) Use this option to view the diagnostic logs for the Appliance.
-awrhtml

(Optional) Use this option to view the diagnostic logs for the Appliance.
-awrtext

(Optional) Use this option to view the diagnostic logs for the specified component.

-mask

(Optional) Use this option to choose to mask sensitive data in the log collection.

-sanitize

(Optional) Use this option to choose to sanitize (redact) sensitive data in the log collection.

Usage Notes

You can use Trace File Collector (the tfactl command) to collect all log files for the Oracle Database Appliance components.

The following types of sensitive information can be redacted using the -mask or the -sanitize option:

  • Host names

  • IP addresses

  • Database names

  • Tablespace names

  • Service names

  • Ports

  • Operating System user names

For example, when the -mask option is used, all instances of a sensitive name such as a database name called "payrolldb" are replaced with "*********" in the TFA collection.

For example, when the -sanitize option is used, all instances of a sensitive name such as a database name called "payrolldb" are replaced with another string, such as "oCjlN7F8P", in the TFA collection.

Parent topic: Troubleshooting Oracle Database Appliance

Running the Disk Diagnostic Tool

Use the Disk Diagnostic Tool to help identify the cause of disk problems.

The tool produces a list of 14 disk checks for each node. To display details, where n represents the disk resource name, enter the following command:

# odaadmcli stordiag n
For example, to display detailed information for NVMe pd_00:
# odaadmcli stordiag pd_00

Parent topic: Troubleshooting Oracle Database Appliance

Running the Oracle Database Appliance Hardware Monitoring Tool

The Oracle Database Appliance Hardware Monitoring Tool displays the status of different hardware components in Oracle Database Appliance server.

The tool is implemented with the Trace File Analyzer collector. Use the tool both on bare-metal and on virtualized systems. The Oracle Database Appliance Hardware Monitoring Tool reports information only for the node on which you run the command. The information it displays in the output depend on the component that you select to review.

Bare Metal Platform

You can see the list of monitored components by running the command odaadmcli show -h

To see information about specific components, use the command syntax odaadmcli show component, where component is the hardware component that you want to query. For example, the command odaadmcli show power shows information specifically about the Oracle Database Appliance power supply: 

# odaadmcli show power

NAME            HEALTH  HEALTH_DETAILS   PART_NO.  	SERIAL_NO.
Power_Supply_0  OK            -          7079395     476856Z+1514CE056G

(Continued)
LOCATION    INPUT_POWER   OUTPUT_POWER   INLET_TEMP         EXHAUST_TEMP
PS0         Present       112 watts      28.000 degree C    34.938 degree C

Virtualized Platform

You can see the list of monitored components by running the command oakcli show -h

To see information about specific components, use the command syntax oakcli show component, where component is the hardware component that you want to query. For example, the command oakcli show power shows information specifically about the Oracle Database Appliance power supply: 

# oakcli show power

NAME            HEALTH HEALTH DETAILS PART_NO. SERIAL_NO.          
Power Supply_0  OK      -             7047410   476856F+1242CE0020
Power Supply_1  OK     -              7047410   476856F+1242CE004J

(Continued)

LOCATION  INPUT       POWER OUTPUT POWER INLET TEMP   EXHAUST TEMP
PS0       Present     88 watts     31.250 degree C    34.188 degree C
PS1       Present     66 watts     31.250 degree C    34.188 degree C

Note:

Oracle Database Appliance Server Hardware Monitoring Tool is enabled during initial startup of ODA_BASE on Oracle Database Appliance Virtualized Platform. When it starts, the tool collects base statistics for about 5 minutes. During this time, the tool displays the message "Gathering Statistics…" message.

Parent topic: Troubleshooting Oracle Database Appliance

Preparing Log Files for Oracle Support Services

If necessary, use the command odaadmcli manage diagcollect to collect diagnostic files to send to Oracle Support Services.

Use the Bill Of Materials report saved in the /opt/oracle/dcs/Inventory/ directory, to enable Oracle Support to help troubleshoot errors, if necessary.

If you have a system fault that requires help from Oracle Support Services, then you may need to provide log records to help Oracle support diagnose your issue.

You can use Trace File Collector (the tfactl command) to collect all log files for the Oracle Database Appliance components.

You can also collect log file information by running the command odaadmcli manage diagcollect. This command consolidates information from log files stored on Oracle Database Appliance into a single log file for use by Oracle Support Services. The location of the file is specified in the command output.

The following is an example of collecting log file information.

# odaadmcli manage diagcollect --dataMask --dataSanitize
Current TFA version is:183000
Current TFA supports dataMask or dataSanitize options.
DataMask is as true
DataSanitize is as true
Only collect dcslogs is as false
TFACTL command is: /opt/oracle/tfa/tfa_home/bin/tfactl
Data mask is set.
Running command: /opt/oracle/tfa/tfa_home/bin/tfactl diagcollect -mask -last 12h
Please wait for several minutes for the collection to complete.
Command return status code is 0
Logs are being collected to:
/opt/oracle/tfa/tfa_home/repository/collection_Mon_Sep_10_21_56_53_GMT_2018_node_all/
rwsoda6m006.tfa_Mon_Sep_10_21_56_53_GMT_2018.zip

Parent topic: Troubleshooting Oracle Database Appliance