17 Troubleshooting Oracle Database Appliance
Understand tools you can use to validate changes and troubleshoot Oracle Database Appliance problems.
- Viewing Component Information on the Appliance
View details of all the components installed on the appliance, and the RPM drift information. - Errors When Logging into the Web Console
If you have problems logging into the Web Console, then it may be due to your browser or credentials. - Errors when re-imaging Oracle Database Appliance
Understand how to troubleshoot errors that occur when re-imaging Oracle Database Appliance. - Using the Oracle ORAchk Health Check Tool
Use Oracle ORAchk Health Check Tool to audit configuration settings and check system health. - About Oracle Trace File Analyzer Collector
Oracle Trace File Analyzer (TFA) Collector simplifies diagnostic data collection on Oracle Grid Infrastructure and Oracle Real Application Clusters systems. - Running Oracle Trace File Analyzer (TFA) Collector Commands
Understand the installed location oftfactland the options for the command. - Running the Disk Diagnostic Tool
Use the Disk Diagnostic Tool to help identify the cause of disk problems. - Running the Oracle Database Appliance Hardware Monitoring Tool
The Oracle Database Appliance Hardware Monitoring Tool displays the status of different hardware components in Oracle Database Appliance server. - Configuring a Trusted SSL Certificate for Oracle Database Appliance
The Web Console and DCS Controller use SSL-based HTTPS protocol for secure communication. Understand the implications of this added security and the options to configure SSL certificates. - Disabling the Web Console
You can also disable the Web Console. Disabling the Web Console means you can only manage your appliance through the command-line interface. - Preparing Log Files for Oracle Support Services
If necessary, use the commandodaadmcli manage diagcollectto collect diagnostic files to send to Oracle Support Services.
Viewing Component Information on the Appliance
View details of all the components installed on the appliance, and the RPM drift information.
Viewing the Bill of Materials in the Web Console
Use the Appliance tab in the Web Console to view information about your deployment and the installed components. The Advanced Information tab displays information about the following components:
-
Grid Infrastructure Version, and the home directory
-
Database Version, Home location, and Edition
-
Location and details about the databases configured
-
All patches applied to the appliance
-
Firmware Controller and Disks
-
ILOM information
-
BIOS version
-
List of RPMs
In the List of RPMs section, click Show and then click RPM Drift to view the differences between the RPMs installed on the appliance, and the RPMs shipped in the latest Oracle Database Appliance Patch Bundle Update release.
Click Download to save the components report. You can use this report to help diagnose any deployment issues.
Viewing the Bill of Materials from the Command Line
The bill of materials is also available through the command line for bare metal and virtualized platforms deployments. The information about the installed components is collected according to a set schedule, and stored in the location /opt/oracle/dcs/Inventory/ for bare metal deployments and in the /opt/oracle/oak/Inventory/ directory for virtualized platforms. The file is stored in the format oda_bom_TimeStamp.json. Use the command describe-system to view the bill of materials on the command line. See the Oracle Database Command-Line Interface chapter for command options and usage notes.
Example 17-1 Example Command to View the Bill of Materials from the Command Line for Bare Metal Deployments
# odacli describe-system -b
ODA Components Information
------------------------------
Component Name Comonent Details
--------------- -----------------------------------------------------------------------------------------------
NODE Name : rwsoda6m003
Domain Name :
Time Stamp : July 29, 2018 7:00:12 PM UTC
RPMS Installed RPMS : acl-2.2.49-7.el6_9.1.x86_64,
aide-0.14-11.el6.x86_64,
alsa-lib-1.1.0-4.el6.x86_64,
at-3.1.10-49.el6.x86_64,
atk-1.30.0-1.el6.x86_64,
attr-2.4.44-7.el6.x86_64,
audit-2.4.5-6.el6.x86_64,
audit-libs-2.4.5-6.el6.x86_64,
audit-libs-python-2.4.5-6.el6.x86_64,
augeas-libs-1.0.0-10.el6.x86_64,
authconfig-6.1.12-23.el6.x86_64,
avahi-libs-0.6.25-17.el6.x86_64,
b43-openfwwf-5.2-10.el6.noarch,
basesystem-10.0-4.0.1.el6.noarch,
bash-4.1.2-48.el6.x86_64,
bc-1.06.95-1.el6.x86_64,
bind-libs-9.8.2-0.62.rc1.el6_9.5.x86_64,
bind-utils-9.8.2-0.62.rc1.el6_9.5.x86_64,
binutils-2.20.51.0.2-5.47.el6_9.1.x86_64,
biosdevname-0.7.2-1.el6.x86_64,
bridge-utils-1.2-10.el6.x86_64,
busybox-1.15.1-21.el6_6.x86_64,
bzip2-1.0.5-7.el6_0.x86_64,
bzip2-libs-1.0.5-7.el6_0.x86_64,
ca-certificates-2017.2.14-65.0.1.el6_9.noarch,
cairo-1.8.8-6.el6_6.x86_64,
celt051-0.5.1.3-0.el6.x86_64,
checkpolicy-2.0.22-1.el6.x86_64,
chkconfig-1.3.49.5-1.el6.x86_64,
cloog-ppl-0.15.7-1.2.el6.x86_64,
compat-libcap1-1.10-1.x86_64,
compat-libstdc++-33-3.2.3-69.el6.x86_64,
compat-readline5-5.2-17.1.el6.x86_64,
compat-sap-c++-4.8.2-16.el6.x86_64,
ConsoleKit-0.4.1-6.el6.x86_64,
ConsoleKit-libs-0.4.1-6.el6.x86_64,
ConsoleKit-x11-0.4.1-6.el6.x86_64,
coreutils-8.4-46.0.1.el6.x86_64,
coreutils-libs-8.4-46.0.1.el6.x86_64,
cpio-2.10-13.el6.x86_64,
cpp-4.4.7-18.el6.x86_64,
cpupowerutils-1.3-2.el6.x86_64,
cpuspeed-1.5-22.0.1.el6.x86_64,
cracklib-2.8.16-4.el6.x86_64,
cracklib-dicts-2.8.16-4.el6.x86_64,
crash-7.1.4-1.0.1.el6_7.x86_64,
crda-3.13_2015.10.22-3.el6.x86_64,
createrepo-0.9.9-27.el6_9.noarch,
cronie-1.4.4-16.el6_8.2.x86_64,
cronie-anacron-1.4.4-16.el6_8.2.x86_64,
crontabs-1.10-33.el6.noarch,
cryptsetup-luks-1.2.0-11.el6.x86_64,
cryptsetup-luks-libs-1.2.0-11.el6.x86_64,
cups-libs-1.4.2-78.el6_9.x86_64,
....
....
....
Example 17-2 Example Command to View the Bill of Materials from the Command Line for Virtualized Platforms
# oakcli describe-system -bExample 17-3 Example Command to View the Bill of Materials Report from the Stored Location
# ls -la /opt/oracle/dcs/Inventory/
total 264
-rw-r--r-- 1 root root 83550 Apr 26 05:41 oda_bom_2018-04-26_05-41-36.jsonParent topic: Troubleshooting Oracle Database Appliance
Errors When Logging into the Web Console
If you have problems logging into the Web Console, then it may be due to your browser or credentials.
Note:
Oracle Database Appliance uses self-signed certificates. Your browser determines how you log into the Web Console. Depending on the browser and browser version, you may receive a warning or error that the certificate is invalid or not trusted because it is self-signed, or that the connection is not private. Ensure that you accept the self-signed certificate for the agent and Web Console.Follow these steps to log into the Web Console:
Errors when re-imaging Oracle Database Appliance
Understand how to troubleshoot errors that occur when re-imaging Oracle Database Appliance.
If re-imaging Oracle Database Appliance fails, with old header issues such as errors in storage discovery, or in running GI root scripts, or disk group RECO creation, then use the force mode with cleanup.pl.
# perl cleanup.pl -fTo ensure that re-imaging is successful, remove the old headers from the storage disks by running the secure erase tool. Verify that the OAK/ASM headers are removed.
# cleanup.pl -erasedata
# cleanup.pl -checkHeaderRetry the re-imaging operation.
Related Topics
Parent topic: Troubleshooting Oracle Database Appliance
Using the Oracle ORAchk Health Check Tool
Use Oracle ORAchk Health Check Tool to audit configuration settings and check system health.
Oracle ORAchk Health Check Tool performs proactive heath checks for the Oracle software stack and scans for known problems.
Oracle ORAchk Health Check Tool audits important configuration settings for Oracle RAC two-node deployments in the following categories:
-
Operating system kernel parameters and packages
-
Oracle Database
-
Database parameters, and other database configuration settings
-
Oracle Grid Infrastructure, which includes Oracle Clusterware and Oracle Automatic Storage Management
ORAchk is aware of the entire system. It checks the configuration to indicate if best practices are being followed.
See Also:
For more information about ORAchk, see My Oracle Support note 1268927.2, "ORAchk Health Checks for the Oracle Stack" at https://support.oracle.com/rs?type=doc&id=1268927.2Note:
Before running ORAchk, check for the latest version of ORAchk, and download and install it.Running ORAchk on Oracle Database Appliance 18.7 Baremetal Systems for New Installation
When you provision or upgrade to Oracle Database Appliance 18.7, the ORAchk RPMs are installed in the directory
/opt/oracle.SupportTools/orachk/. You can verify that
ORAchk is installed by running the following command:
[root@oak bin]# rpm -q orachk
orachk-18.3.0_20180808-2.x86_64 orachk that you downloaded from My Oracle Support into the
/opt/oracle.SupportTools/orachk/ directory, and run the
following command:orachk -upgradeThe command upgrades your orachk utility to the latest version.
To run orachk, use the following command:
[root@oak bin]# orachk
This computer is for [S]ingle instance database or part of a [C]luster to run
RAC database [S|C] [C]: S
orachk did not find the inventory location on oak from environment. Does oak
have Oracle software installed [y/n][n]? n
...
Detailed report (html) -
/opt/oracle.SupportTools/orachk/orachk_oak_091918_182425/orachk_oak_091918_182
425.html
UPLOAD [if required] -
/opt/oracle.SupportTools/orachk/orachk_oak_091918_182425.zipRunning ORAchk on Oracle Database Appliance 18.7 Virtualized Platform
When you provision or upgrade to Oracle Database Appliance 18.7, the ORAchk RPMs are installed in the directory
/opt/oracle.SupportTools/orachk/.
orachk, use the following
command:[root@oak bin]# oakcli orachkRunning ORAchk on Oracle Database Appliance Baremetal Systems for Releases Earlier than 18.7
Parent topic: Troubleshooting Oracle Database Appliance
About Oracle Trace File Analyzer Collector
Oracle Trace File Analyzer (TFA) Collector simplifies diagnostic data collection on Oracle Grid Infrastructure and Oracle Real Application Clusters systems.
TFA behaves in a similar manner to the ion utility packaged with Oracle Clusterware. Both tools collect and package diagnostic data. However, TFA is much more powerful than ion, because TFA centralizes and automates the collection of diagnostic information.
TFA provides the following key benefits and options:
-
Encapsulation of diagnostic data collection for all Oracle Grid Infrastructure and Oracle RAC components on all cluster nodes into a single command, which you run from a single node
-
Option to "trim" diagnostic files during data collection to reduce data upload size
-
Options to isolate diagnostic data collection to a given time period, and to a particular product component, such as Oracle ASM, RDBMS, or Oracle Clusterware
-
Centralization of collected diagnostic output to a single node in Oracle Database Appliance, if desired
-
On-Demand Scans of all log and trace files for conditions indicating a problem
-
Real-Time Scan Alert Logs for conditions indicating a problem (for example, Database Alert Logs, Oracle ASM Alert Logs, and Oracle Clusterware Alert Logs)
See Also:
Refer to My Oracle Support note 1513912.1 "TFA Collector - Tool for Enhanced Diagnostic Gathering" for more information. https://support.oracle.com/rs?type=doc&id=1513912.1
Parent topic: Troubleshooting Oracle Database Appliance
Running Oracle Trace File Analyzer (TFA) Collector Commands
Understand the installed location of tfactl and the options
for the command.
About Using tfactl to Collect Diagnostic Information
When you provision or upgrade to Oracle Database Appliance 18.7, Oracle Trace File Analyzer (TFA) Collector is
installed in the directory
/opt/oracle/tfa/tfa_home, You can
invoke the command line utility for TFA, tfactl
from the directory
/opt/oracle/tfa/tfa_home/bin/tfactl, or
simply type tfactl.
You can use the following command options to run tfactl:
/opt/oracle/tfa/tfa_home/bin/tfactl diagcollect -ips|-oda|-odalite|-dcs|-odabackup|
-odapatching|-odadataguard|-odaprovisioning|-odaconfig|-odasystem|-odastorage|-database|
-asm|-crsclient|-dbclient|-dbwlm|-tns|-rhp|-procinfo|-afd|-crs|-cha|-wls|
-emagent|-oms|-ocm|-emplugins|-em|-acfs|-install|-cfgtools|-os|-ashhtml|-ashtext|
-awrhtml|-awrtext -mask -sanitizeTable 17-1 Command Options for tfactl Tool
| Option | Description |
|---|---|
-h |
(Optional) Describes all the options for this command. |
-ips |
(Optional) Use this option to view the diagnostic logs for the specified component. |
-oda |
(Optional) Use this option to view the logs for the entire Appliance. |
-odalite |
(Optional) Use this option to view the diagnostic logs for the odalite component. |
-dcs |
(Optional) Use this option to view the DCS log files. |
-odabackup |
(Optional) Use this option to view the diagnostic logs for the backup components for the Appliance. |
-odapatching |
(Optional) Use this option to view the diagnostic logs for patching components of the Appliance. |
-odadataguard |
(Optional) Use this option to view the diagnostic logs for Oracle Data Guard component of the Appliance. |
-odaprovisioning |
(Optional) Use this option to view provisioning logs for the Appliance. |
-odaconfig |
(Optional) Use this option to view configuration-related diagnostic logs. |
-odasystem |
(Optional) Use this option to view system information. |
-odastorage |
(Optional) Use this option to view the diagnostic logs for the Appliance storage. |
-database |
(Optional) Use this option to view database-related log files. |
-asm |
(Optional) Use this option to view the diagnostic logs for the Appliance. |
-crsclient |
(Optional) Use this option to view the diagnostic logs for the Appliance. |
-dbclient |
(Optional) Use this option to view the diagnostic logs for the Appliance. |
-dbwlm |
(Optional) Use this option to view the diagnostic logs for the specified component. |
-tns |
(Optional) Use this option to view the diagnostic logs for TNS. |
-rhp |
(Optional) Use this option to view the diagnostic logs for Rapid Home Provisioning. |
-afd |
(Optional) Use this option to view the diagnostic logs for Oracle ASM Filter Driver. |
-crs |
(Optional) Use this option to view the diagnostic logs for Oracle Clusterware. |
-cha |
(Optional) Use this option to view the diagnostic logs for the Cluster Health Monitor. |
-wls |
(Optional) Use this option to view the diagnostic logs for Oracle WebLogic Server. |
-emagent |
(Optional) Use this option to view the diagnostic logs for the Oracle Enterprise Manager agent. |
-oms |
(Optional) Use this option to view the diagnostic logs for the Oracle Enterprise Manager Management Service. |
-ocm |
(Optional) Use this option to view the diagnostic logs for the specified component. |
-emplugins |
(Optional) Use this option to view the diagnostic logs for Oracle Enterprise Manager plug-ins. |
-em |
(Optional) Use this option to view the diagnostic logs for Oracle Enterprise Manager deployment. |
-acfs |
(Optional) Use this option to view the diagnostic logs for Oracle ACFS storage. |
-install |
(Optional) Use this option to view the diagnostic logs for installation. |
-cfgtools |
(Optional) Use this option to view the diagnostic logs for the configuration tools. |
-os |
(Optional) Use this option to view the diagnostic logs for the operating system. |
-ashhtml |
(Optional) Use this option to view the diagnostic logs for the specified component. |
-ashtext |
(Optional) Use this option to view the diagnostic logs for the Appliance. |
-awrhtml |
(Optional) Use this option to view the diagnostic logs for the Appliance. |
-awrtext |
(Optional) Use this option to view the diagnostic logs for the specified component. |
|
|
(Optional) Use this option to choose to mask sensitive data in the log collection. |
|
|
(Optional) Use this option to choose to sanitize (redact) sensitive data in the log collection. |
Usage Notes
You can use Trace File Collector (the tfactl command) to collect all log files for the Oracle Database Appliance components.
The following types of sensitive information can be redacted using the -mask or the -sanitize option:
-
Host names
-
IP addresses
-
Database names
-
Tablespace names
-
Service names
-
Ports
-
Operating System user names
For example, when the -mask option is used, all instances of a sensitive name such as a database name called "payrolldb" are replaced with "*********" in the TFA collection.
For example, when the -sanitize option is used, all instances of a sensitive name such as a database name called "payrolldb" are replaced with another string, such as "oCjlN7F8P", in the TFA collection.
Parent topic: Troubleshooting Oracle Database Appliance
Running the Disk Diagnostic Tool
Use the Disk Diagnostic Tool to help identify the cause of disk problems.
The tool produces a list of 14 disk checks for each node. To display details, where n represents the disk resource name, enter the following command:
# odaadmcli stordiag n# odaadmcli stordiag pd_00Parent topic: Troubleshooting Oracle Database Appliance
Running the Oracle Database Appliance Hardware Monitoring Tool
The Oracle Database Appliance Hardware Monitoring Tool displays the status of different hardware components in Oracle Database Appliance server.
The tool is implemented with the Trace File Analyzer collector. Use the tool both on bare-metal and on virtualized systems. The Oracle Database Appliance Hardware Monitoring Tool reports information only for the node on which you run the command. The information it displays in the output depend on the component that you select to review.
Bare Metal Platform
You can see the list of monitored components by running the command odaadmcli show -h
To see information about specific components, use the command syntax odaadmcli show component, where component is the hardware component that you want to query. For example, the command odaadmcli show power shows information specifically about the Oracle Database Appliance power supply:
# odaadmcli show power
NAME HEALTH HEALTH_DETAILS PART_NO. SERIAL_NO.
Power_Supply_0 OK - 7079395 476856Z+1514CE056G
(Continued)
LOCATION INPUT_POWER OUTPUT_POWER INLET_TEMP EXHAUST_TEMP
PS0 Present 112 watts 28.000 degree C 34.938 degree CVirtualized Platform
You can see the list of monitored components by running the command oakcli show -h
To see information about specific components, use the command syntax oakcli show component, where component is the hardware component that you want to query. For example, the command oakcli show power shows information specifically about the Oracle Database Appliance power supply:
# oakcli show power
NAME HEALTH HEALTH DETAILS PART_NO. SERIAL_NO.
Power Supply_0 OK - 7047410 476856F+1242CE0020
Power Supply_1 OK - 7047410 476856F+1242CE004J
(Continued)
LOCATION INPUT POWER OUTPUT POWER INLET TEMP EXHAUST TEMP
PS0 Present 88 watts 31.250 degree C 34.188 degree C
PS1 Present 66 watts 31.250 degree C 34.188 degree C
Note:
Oracle Database Appliance Server Hardware Monitoring Tool is enabled during initial startup of ODA_BASE on Oracle Database Appliance Virtualized Platform. When it starts, the tool collects base statistics for about 5 minutes. During this time, the tool displays the message "Gathering Statistics…" message.
Parent topic: Troubleshooting Oracle Database Appliance
Configuring a Trusted SSL Certificate for Oracle Database Appliance
The Web Console and DCS Controller use SSL-based HTTPS protocol for secure communication. Understand the implications of this added security and the options to configure SSL certificates.
The Web Console provides an added layer of security using certificates and encryption, when an administrator interacts with the appliance. Encryption of data ensures that:
- Data is sent to the intended recipient, and not to any malicious third-party.
- When data is exchanged between the server and the browser, data interception cannot occur nor can the data be edited.
When you connect to the Web Console through HTTPS, the DCS Controller presents your browser with a certificate to verify the identity of appliance. If the web browser finds that the certificate is not from a trusted Certificate Authority (CA), then the browser assumes it has encountered an untrusted source, and generates a security alert message. The security alert dialog boxes display because Web Console security is enabled through HTTPS and SSL, but you have not secured your Web tier properly with a trusted matching certificate from a Certificate Authority. It is possible to purchase commercial certificates from a Certificate Authority or create your own and register them with a Certificate Authority. However, the server and browser certificates must use the same public certificate key and trusted certificate to avoid the error message produced by the browser.
There are three options to configure your certificates:
- Create your own key and Java keystore, ensure it is signed by a Certificate Authority (CA) and then import it for use.
- Package an existing Privacy Enhanced Mail (PEM) format key and certificates in a new Java keystore.
- Convert an existing PKCS or PFX keystore to a Java keystore
and configure it for the Web Console.
Note:
For Oracle Database Appliance High-Availability hardware models, run the configuration steps on both nodes.
The following topics explain how to configure these options:
- Option 1: Creating a Key and Java Keystore and Importing a Trusted Certificate
Usekeytool, a key and certificate management utility, to create a keystore and a signing request. - Option 2: Packaging an Existing PEM-format Key and Certificates in a New Java Keystore
Use the OpenSSL tool to package Privacy Enhanced Mail (PEM) files in a PKCS keystore. - Option 3: Converting an Existing PKCS or PFX Keystore to a Java Keystore
If you have an existing PKCS or PFX keystore for your server's domain, convert it to a Java keystore. - Configuring the DCS Server to Use Custom Keystore
After packaging or converting your keystore into Java keystore, configure the DCS server to use your keystore. - Configuring the DCS Agent for Custom Certificate
After you import the certificate into the keystore, configure the DCS agent to use the same certificate.
Parent topic: Troubleshooting Oracle Database Appliance
Option 1: Creating a Key and Java Keystore and Importing a Trusted Certificate
Use keytool, a key and certificate management utility, to
create a keystore and a signing request.
Option 2: Packaging an Existing PEM-format Key and Certificates in a New Java Keystore
Use the OpenSSL tool to package Privacy Enhanced Mail (PEM) files in a PKCS keystore.
Option 3: Converting an Existing PKCS or PFX Keystore to a Java Keystore
If you have an existing PKCS or PFX keystore for your server's domain, convert it to a Java keystore.
Configuring the DCS Server to Use Custom Keystore
After packaging or converting your keystore into Java keystore, configure the DCS server to use your keystore.
Disabling the Web Console
You can also disable the Web Console. Disabling the Web Console means you can only manage your appliance through the command-line interface.
Parent topic: Troubleshooting Oracle Database Appliance
Preparing Log Files for Oracle Support Services
If necessary, use the command odaadmcli manage diagcollect to collect diagnostic files to send to Oracle Support Services.
Use the Bill Of Materials report saved in the /opt/oracle/dcs/Inventory/ directory, to enable Oracle Support to help troubleshoot errors, if necessary.
If you have a system fault that requires help from Oracle Support Services, then you may need to provide log records to help Oracle support diagnose your issue.
You can use Trace File Collector (the tfactl command) to collect all log files for the Oracle Database Appliance components.
You can also collect log file information by running the command odaadmcli manage diagcollect. This command consolidates information from log files stored on Oracle Database Appliance into a single log file for use by Oracle Support Services. The location of the file is specified in the command output.
Example 17-4 Collecting log file information for a time period, masking sensitive data
# odaadmcli manage diagcollect --dataMask --fromTime 2019-08-12 --toTime 2019-08-25
DataMask is set as true
FromTime is set as: 2019-08-12
ToTime is set as: 2019-08-25
TFACTL command is: /opt/oracle/tfa/tfa_home/bin/tfactl
Data mask is set.
Collect data from 2019-08-12
Collect data to 2019-08-25Parent topic: Troubleshooting Oracle Database Appliance