18 Troubleshooting Oracle Database Appliance
Understand tools you can use to validate changes and troubleshoot Oracle Database Appliance problems.
- Viewing Component Information on the Appliance
View details of all the components installed on the appliance, and the RPM drift information. - Errors When Logging into the Browser User Interface
If you have problems logging into the Browser User Interface, then it may be due to your browser or credentials. - Errors when re-imaging Oracle Database Appliance
Understand how to troubleshoot errors that occur when re-imaging Oracle Database Appliance. - Backup and Recovery for Oracle Database Appliance Upgrades on Virtualized Platform
Before patching your deployment to Oracle Database Appliance release 19.8 on virtualized platform, understand how you can take a backup before the upgrade and then restore from the backup, if necessary. - Analyzing the Pre-Checks Report for Operating System Upgrades on Virtualized Platform
Understand the patching pre-checks report and the components checked during the operating system upgrade when patching Oracle Database Appliance Virtualized Platform. - Analyzing the Preupgrade Checks Report for Operating System Upgrades on Virtualized Platform
Understand the patching preupgrade checks report before performing the operating system upgrade when patching Oracle Database Appliance Virtualized Platform. - Analyzing the Post Upgrade Checks Report for Operating System Upgrades on Virtualized Platform
Understand the patching post upgrade checks report after performing the operating system upgrade when patching Oracle Database Appliance Virtualized Platform. - Using Oracle Autonomous Health Framework for Running Diagnostics
Oracle Autonomous Health Framework collects and analyzes diagnostic data collected, and proactively identifies issues before they affect the health of your system. - Running the Disk Diagnostic Tool
Use the Disk Diagnostic Tool to help identify the cause of disk problems. - Running the Oracle Database Appliance Hardware Monitoring Tool
The Oracle Database Appliance Hardware Monitoring Tool displays the status of different hardware components in Oracle Database Appliance server. - Configuring a Trusted SSL Certificate for Oracle Database Appliance
The Browser User Interface and DCS Controller use SSL-based HTTPS protocol for secure communication. Understand the implications of this added security and the options to configure SSL certificates. - Disabling the Browser User Interface
You can also disable the Browser User Interface. Disabling the Browser User Interface means you can only manage your appliance through the command-line interface. - Preparing Log Files for Oracle Support Services
If you have a system fault that requires help from Oracle Support Services, then you may need to provide log records to help Oracle support diagnose your issue.
Viewing Component Information on the Appliance
View details of all the components installed on the appliance, and the RPM drift information.
Viewing the Bill of Materials in the Browser User Interface
Use the Appliance tab in the Browser User Interface to view information about your deployment and the installed components. The Advanced Information tab displays information about the following components:
-
Grid Infrastructure Version, and the home directory
-
Database Version, Home location, and Edition
-
Location and details about the databases configured
-
All patches applied to the appliance
-
Firmware Controller and Disks
-
ILOM information
-
BIOS version
-
List of RPMs
In the List of RPMs section, click Show and then click RPM Drift to view the differences between the RPMs installed on the appliance, and the RPMs shipped in the latest Oracle Database Appliance Patch Bundle Update release.
Click Download to save the components report. You can use this report to help diagnose any deployment issues.
Viewing the Bill of Materials from the Command Line
The bill of materials is also available through the command line for bare metal and virtualized platforms deployments. The information about the installed components is collected according to a set schedule, and stored in the location /opt/oracle/dcs/Inventory/
for bare metal deployments and in the /opt/oracle/oak/Inventory/
directory for virtualized platforms. The file is stored in the format oda_bom_TimeStamp.json
. Use the command describe-system
to view the bill of materials on the command line. See the Oracle Database Command-Line Interface chapter for command options and usage notes.
Example 18-1 Example Command to View the Bill of Materials from the Command Line for Bare Metal Deployments
# odacli describe-system -b
ODA Components Information
------------------------------
Component Name Component Details
--------------- -----------------------------------------------------------------------------------------------
NODE Name : oda1
Domain Name : testdomain.com
Time Stamp : April 21, 2020 6:21:15 AM UTC
RPMS Installed RPMS : abrt-2.1.11-55.0.1.el7.x86_64,
abrt-addon-ccpp-2.1.11-55.0.1.el7.x86_64,
abrt-addon-kerneloops-2.1.11-55.0.1.el7.x86_64,
abrt-addon-pstoreoops-2.1.11-55.0.1.el7.x86_64,
abrt-addon-python-2.1.11-55.0.1.el7.x86_64,
abrt-addon-vmcore-2.1.11-55.0.1.el7.x86_64,
abrt-addon-xorg-2.1.11-55.0.1.el7.x86_64,
abrt-cli-2.1.11-55.0.1.el7.x86_64,
abrt-console-notification-2.1.11-55.0.1.el7.x86_64,
abrt-dbus-2.1.11-55.0.1.el7.x86_64,
abrt-libs-2.1.11-55.0.1.el7.x86_64,
abrt-python-2.1.11-55.0.1.el7.x86_64,
abrt-tui-2.1.11-55.0.1.el7.x86_64,
acl-2.2.51-14.el7.x86_64,
adwaita-cursor-theme-3.28.0-1.el7.noarch,
adwaita-icon-theme-3.28.0-1.el7.noarch,
aic94xx-firmware-30-6.el7.noarch,
aide-0.15.1-13.0.1.el7.x86_64,
alsa-firmware-1.0.28-2.el7.noarch,
alsa-lib-1.1.8-1.el7.x86_64,
alsa-tools-firmware-1.1.0-1.el7.x86_64,
at-3.1.13-24.el7.x86_64,
at-spi2-atk-2.26.2-1.el7.x86_64,
at-spi2-core-2.28.0-1.el7.x86_64,
atk-2.28.1-1.el7.x86_64,
attr-2.4.46-13.el7.x86_64,
audit-2.8.5-4.el7.x86_64,
audit-libs-2.8.5-4.el7.x86_64,
audit-libs-python-2.8.5-4.el7.x86_64,
augeas-libs-1.4.0-9.el7.x86_64,
authconfig-6.2.8-30.el7.x86_64,
autogen-libopts-5.18-5.el7.x86_64,
avahi-libs-0.6.31-19.el7.x86_64,
basesystem-10.0-7.0.1.el7.noarch,
bash-4.2.46-33.el7.x86_64,
bash-completion-2.1-6.el7.noarch,
bc-1.06.95-13.el7.x86_64,
bind-export-libs-9.11.4-9.P2.el7.x86_64,
bind-libs-9.11.4-9.P2.el7.x86_64,
bind-libs-lite-9.11.4-9.P2.el7.x86_64,
bind-license-9.11.4-9.P2.el7.noarch,
bind-utils-9.11.4-9.P2.el7.x86_64,
binutils-2.27-41.base.0.7.el7_7.2.x86_64,
biosdevname-0.7.3-2.el7.x86_64,
blktrace-1.0.5-9.el7.x86_64,
bnxtnvm-1.40.10-1.x86_64,
boost-date-time-1.53.0-27.el7.x86_64,
boost-filesystem-1.53.0-27.el7.x86_64,
boost-iostreams-1.53.0-27.el7.x86_64,
....
....
....
Example 18-2 Example Command to View the Bill of Materials from the Command Line for Virtualized Platforms
# oakcli describe-system -b
Example 18-3 Example Command to View the Bill of Materials Report from the Stored Location
# ls -la /opt/oracle/dcs/Inventory/
total 264
-rw-r--r-- 1 root root 83550 Apr 26 05:41 oda_bom_2018-04-26_05-41-36.json
Parent topic: Troubleshooting Oracle Database Appliance
Errors When Logging into the Browser User Interface
If you have problems logging into the Browser User Interface, then it may be due to your browser or credentials.
Note:
Oracle Database Appliance uses self-signed certificates. Your browser determines how you log into the Browser User Interface. Depending on the browser and browser version, you may receive a warning or error that the certificate is invalid or not trusted because it is self-signed, or that the connection is not private. Ensure that you accept the self-signed certificate for the agent and Browser User Interface.Follow these steps to log into the Browser User Interface:
Note:
If you have any issues logging into the Oracle Database Appliance Browser User Interface on browsers such as macOS Catalina and Google Chrome, then you may need to use any workaround as described on the official site for the product.Errors when re-imaging Oracle Database Appliance
Understand how to troubleshoot errors that occur when re-imaging Oracle Database Appliance.
If re-imaging Oracle Database Appliance fails, with old header issues such as errors in storage discovery, or in running GI root scripts, or disk group RECO creation, then use the force mode with cleanup.pl
.
# cleanup.pl -f
To ensure that re-imaging is successful, remove the old headers from the storage disks by running the secure erase tool. Verify that the OAK/ASM headers are removed.
# cleanup.pl -erasedata
# cleanup.pl -checkHeader
Retry the re-imaging operation.
Related Topics
Parent topic: Troubleshooting Oracle Database Appliance
Backup and Recovery for Oracle Database Appliance Upgrades on Virtualized Platform
Before patching your deployment to Oracle Database Appliance release 19.8 on virtualized platform, understand how you can take a backup before the upgrade and then restore from the backup, if necessary.
Backing Up the ODA_BASE Domain
- Login into
domain0
asroot
user. - Stop the ODA_BASE
domain.
$ oakcli stop oda_base
- Run the
rsync
command to take a backup of the ODA_BASE domain.$ /usr/bin/rsync -test --delete --progress --exclude 'exclude files' Source location: ODA_BASE location Target location: External NFS storage to move the backup
For example:$ /usr/bin/rsync -test --delete --progress --exclude *zip --exclude *gz ' /OVS/Repositories/odabaseRepo /external
The above command backs up the
You can also compress the/OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1
directory to the external NFS directory, for example,/external
./OVS/Repositories/odabaseRepo
file to reduce the size, and then perform thersync
operation on thebackup.tar.gz
file.For example:$ /bin/tar -zcvf backup.tar.gz /OVS/Repositories/odabaseRepo --exclude=*.gz --exclude=*.zip
- Start the ODA_BASE
domain.
$ oakcli start oda_base
- Repeat steps 1-4 on the ODA_BASE domain on the other node.
Restoring the ODA_BASE Domain
Follow these steps to restore the ODA_BASE domain from a backup.
- Login into
domain0
asroot
user. - Stop the ODA_BASE domain.
$ oakcli stop oda_base
- Create directory, if it does not already
exist.
$ mkdir -p /OVS/Repositories
- Run the
rsync
command to copy the ODA_BASE domain from the backup.$ /usr/bin/rsync -vaz --delete --progress --exclude 'exclude files' Backup location restore location
For example:If the backup is a$ /usr/bin/rsync -vaz --delete --progress --exclude *zip --exclude *gz /external/OVS/Repositories/
backup.tar.gz
file, then transfer the file using the same method, and then uncompress thebackup.tar.gz
file.For example:The above command restores the backup of$ tar -zxvf backup.tar.gz -C /OVS/Repositories/odabaseRepo
/OVS/Repositories/odabaseRepo/VirtualMachines/oakDom1
directory to/OVS/Repositories/
from/external
directory. - Start the ODA_BASE domain.
$ oakcli start oda_base
- Validate the environment using the
oakcli validate
command and ensure that ODA_BASE is functioning normally.$ oakcli validate –d
Parent topic: Troubleshooting Oracle Database Appliance
Analyzing the Pre-Checks Report for Operating System Upgrades on Virtualized Platform
Understand the patching pre-checks report and the components checked during the operating system upgrade when patching Oracle Database Appliance Virtualized Platform.
When you patch your Oracle Database Appliance Virtualized Platform deployment to release 19.8, your operating system is upgraded to Oracle Linux 7. The patching pre-checks report contains a section on the operating system upgrade checks run, similar to the following:
# oakcli update -patch 19.8.0.0.0 --pre-patchreport --local
INFO: Running OS prepatch checks ...
INFO: Validate supported versions
SUCCESS: Validated minimum supported versions.
INFO: Validate patching tag
SUCCESS: Validated patching tag: 19.8.0.0.0
INFO: Is patch location available
SUCCESS: Patch location is available.
INFO: Space checks for OS upgrade
SUCCESS: Validated space checks.
INFO: Install OS upgrade software
INFO: This action migth take a while if packages need to be uncompressed
SUCCESS: Extracted OS upgrade patches into /root/oda-upgrade.
Do not remove this directory until OS upgrade completes.
INFO: Verify OS upgrade by running preupgrade checks
INFO: This operation will take long, please wait
SUCCESS: Results stored in: '/root/preupgrade-results/preupg_results-200626121721.tar.gz' .
Read complete report file '/root/preupgrade/result.html' before attempting OS upgrade.
INFO: Validate custom rpms installed
SUCCESS: No additional RPMs found installed
SUCCESS: All OS prepatch tasks ran correctly
INFO: Running Storage prepatch checks ...
INFO: Validate patching tag
SUCCESS: Validated patching tag: 19.8.0.0.0
INFO: Patch location validation
SUCCESS: Verified patch location.
INFO: Patch tag validation
SUCCESS: Verified patching tag.
INFO: Verify ASM disks status
SUCCESS: ASM disk are online.
SUCCESS: All Storage prepatch tasks ran correctly
SUCCESS: Pre patch report ended successfully
Following is a description of the pre-checks for the operating system upgrade. Note
that you can override these pre-check failures with the –-force
flag,
but it is is recommended that you fix these failures before proceeding with the
operating system upgrade.
Operating System Checks
Validate supported versions: Validates that the current OAK version can be upgraded to the patch version.
Validated minimum supported versions.
Unable to get RPM version
Reason: RPM command failed or formatting is wrong.
Current system is not up-to-date
Could not get minimum compatible version
Current version current_oak_version is incompatible
Validate patching tag: Validates that the -patch parameter has the correct format and that the metadata file exists.
Validated patching tag: 19.8.0.0.0
Supplied version patch is invalid.
Reason: Either the -patch format is invalid, or the metadata file does not exist.
Is patch location available: Make sure that the patching OS tools are available. The directory that contains oda-upgrade.tar and the OS upgrade resources must exist.
Patch location is available.
Unavailable location: /opt/oracle/oak/pkgrepos/oda-os-upgrade/
Unavailable location: patch location
Validated
space checks: Checks the next space requirements. On ODA_BASE, checks for 7
GB of free space on /root
, 150 MB of free space on
/boot
. On dom0, checks for 3 GB of free space on
/root
and 150 MB of free space on /boot
.
Validated space checks.
Current node OS is up to date
Minimum 7 GB free space required under "/root"
/root
.
Minimum 150 MB free space required under "/boot"
/boot
.
Dom 0 space checks failed
/root
directory, or
less than 150 MB free space in the /boot
directory on dom0.
Install OS upgrade software: Prepares upgrade tools by
extracting oda-upgrade.tar
to /root
.
Extracted OS upgrade patches into /root/oda-upgrade.
Minimum 5 Gb free space required under "/root" to extract the oda-upgrade.tar
Component Metadata file or related is not accessible
oda-upgrade
metadata file is not available.
Could not extract file: oda-upgrade.tar
Tar file extracted but oda-upgrade.sh does not exist
oda-upgrade.sh
script is missing.
Verify operating system upgrade by running preupgrade checks:
This precheck runs the RHEL upgrade tool. Check the
/root/preupgrade/result.html
file and fix any critical failures
before performing the operating system upgrade. Results are stored in
/root/preupgrade-results/preupg_results-200626121721.tar.gz
.
Read the complete report file /root/preupgrade/result.html
before
attempting the operating system upgrade.
Current node OS is up to date
Not enough space on /root. At least 1 GB is required.
Reason: Not enough space to run the oda-upgrade
script.
Upgrade script is not available
oda-upgrade
script not found.
Could not store the name logFilename into our internal flat file
Error executing upgradeos.exp: cmdResult
User action is required, results are not successful.
Validate
custom RPMs installed: ODACLI handles upgrade of only those RPMs that are
part of the Oracle Database Appliance operating system image, as listed in the
versionlock file, or those included as part of the Oracle Database Appliance
software, such as DCS and HMP RPMs, or those required to perform the operating
system upgrade itself. As part of prechecks, any other Oracle supplied RPMs are
listed in /root/oda-upgrade/rpms-added-from-Oracle
and other
third-party RPMs in /root/oda-upgrade/rpms-added-from-Thirdparty
files respectively. For custom RPMs, you must separately upgrade these RPMs on each
node after the operating system is successfully upgraded to Oracle Linux 7. These
files are preserved in the same location after the operating system upgrade
completes.
No additional RPMs found installed
Skip check for custom RPMs, as OS is OL7
Found additional RPMs installed warning: user is responsible for updating custom RPM files
The custom rpm checks may not have run
Reason: The evidence that the rpm checks were performed could not be found.
Storage Checks
Validate patching tag: Validates that the -patch parameter has the correct format and that the metadata file exists.
Validated patching tag: 19.8.0.0.0
Supplied version patch is invalid.
Reason: Either the -patch format is invalid, or the metadata file does not exist.
patch location validation: Verifies that the metadata file exists.
Verified patch location.
System patch metadata does not exist.
patch tag validation: Builds disk, expander, and controller patching tags from the metadata file.
Verified patching tag.
Firmware patches for storage components are not present.
Verify ASM disks status: If CRS is running, check that disks are online. If CRS is not running, this check is skipped.
ASM disks are online.
CRS is not running, skipping ASM check.
One or more ASM disk are not online. Rolling storage update cannot proceed.
Failed to get ASM disk status.
Parent topic: Troubleshooting Oracle Database Appliance
Analyzing the Preupgrade Checks Report for Operating System Upgrades on Virtualized Platform
Understand the patching preupgrade checks report before performing the operating system upgrade when patching Oracle Database Appliance Virtualized Platform.
The patching preupgrade checks report on Virtualized Platform checks that the virtual machines and shared repositories are stopped before the upgrade.
Update DOM0
If the DOM0 agent is updated, then the following message is displayed:
SUCCESS: dom0 agent updated
Action Required: If there is an error in the dom0 update, then review the logs to find out the reason, and re-run the pre-osupgrade step.
Stopping Vms and Repos on both nodes
If virtual machines and shared repositories are stopped on both nodes, then the following message is displayed:
SUCCESS: Repos and vms stopped
Action Required: If there is an error in stopping virtual machines and shared
repositories, use oakcli
commands to stop them manually, and then
re-run the pre-osupgrade step.
Stopping CRS in local node
If Oracle Clusterware is stopped successfully on the local node, then the following message is displayed:
SUCCESS: CRS stopped in local node
Action Required: If a message Could not stop CRS
is
displayed, then check the log files and stop Oracle Clusterware manually, and re-run
the pre-osupgrade step.
Check loop devices
If the operation to check loop devices on dom0 completed successfully, then the following message is displayed:
SUCCESS: Loop devices check passed
Action Required: If any error is encountered in this step, then
run the command loseup -a
on dom0 and verify there are extra loop
devices besides the ODA_BASE.
Sample Preupgrade Checks Report
Sample output from a system before the upgrade.
# oakcli update -patch 19.8.0.0.0 --pre-osupgrade --local
INFO: Trying to ping dom0 agent
INFO: dom0 agent is live
INFO: dom0 agent is not updated.
INFO: Updating dom0 agent, this will take a while...
SUCCESS: dom0 agent updated
INFO: Stopping Vms and Repos on both nodes. This may take a while...
SUCCESS: Repos and vms stopped
INFO: Disabling start repo
Start repo operation is now DISABLED on node 0
Start repo operation is now DISABLED on node 1
SUCCESS: Start repo disabled
INFO: Stopping CRS in local node
INFO: 2020-06-26 12:22:53: Clusterware is running on local node
INFO: 2020-06-26 12:22:53: Attempting to stop clusterware and its resources locally
SUCCESS: 2020-06-26 12:24:01: Successfully stopped the clusterware on local node
SUCCESS: CRS stopped in local node
SUCCESS: Loop devices check passed
SUCCESS: Pre-OSupgrade finished successfully
Parent topic: Troubleshooting Oracle Database Appliance
Analyzing the Post Upgrade Checks Report for Operating System Upgrades on Virtualized Platform
Understand the patching post upgrade checks report after performing the operating system upgrade when patching Oracle Database Appliance Virtualized Platform.
The patching post upgrade checks report on Virtualized Platform validates operating system upgrade, OAK upgrade, Oracle Grid Infrastructure upgrade, RPMs, and enabling of NFS service after upgrade.
Operating System Upgrade (OS upgrade check)
If the operating system is upgraded, then the following message is displayed:
SUCCESS: OS has been upgraded to 4.14.35-1902.303.4.1.el7uek successfully.
OAK Upgrade
If OAK is upgraded successfully, then the following message is displayed:
SUCCESS: ODA software has been upgraded to 19.8.0.0.0 successfully.
Oracle Grid Infrastructure (GI Upgrade Check)
If Oracle Grid Infrastructure version is current, then the following message is displayed:
SUCCESS: GI software is running at correct version: 18.8.0.0.191015(30112122,30581079).
RPM (Extra RPM Check)
If extra RPMS were discovered during the upgrade, then the following message is displayed:
WARNING: There are additional RPMs installed on the system
WARNING: Please upgrade these RPMs.
Action Required: Check the directory
/root/oda-upgrade/rpms-added-from-ThirdParty
and
/root/oda-upgrade/rpms-added-from-Oracle
. Download and install
the equivalent Oracle Linux 7 RPMs for all the RPMs listed in these directories.
Enable NFS Service
If NFS service is enabled successfully after upgrade, then the following message is displayed:
SUCCESS: NFS service has been started successfully
Action Required: If the error message ERROR: Cannot start
NFS service
is shown in the ouput, then check the NFS status using the
command systemctl status nfs -l
.
Sample Post Upgrade Checks Report
Sample output from a system after all upgrading all components.
# oakcli update -patch 19.8.0.0.0 --post-osupgrade --local
INFO: Validating OS version
SUCCESS: OS has been upgraded to 4.14.35-1902.303.4.1.el7uek successfully.
INFO: Validating OAK version
SUCCESS: ODA software has been upgraded to 19.8.0.0.0 successfully.
INFO: Validating GI version
SUCCESS: GI software is running at correct version: 18.8.0.0.191015(30112122,30581079).
INFO: Starting CRS
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
INFO: Starting NFS service
SUCCESS: NFS service has been started successfully.
INFO: Removing ODA OS patching software
SUCCESS: ODA OS patching software has been removed
SUCCESS: Patching post-osupgrade command finished successfully
Parent topic: Troubleshooting Oracle Database Appliance
Using Oracle Autonomous Health Framework for Running Diagnostics
Oracle Autonomous Health Framework collects and analyzes diagnostic data collected, and proactively identifies issues before they affect the health of your system.
- About Installing Oracle Autonomous Health Framework
Oracle Autonomous Health Framework is installed automatically when you provision or patch to Oracle Database Appliance release 19.8. - Using the Oracle ORAchk Health Check Tool
Run Oracle ORAchk to audit configuration settings and check system health. - Generating and Viewing Oracle ORAchk Health Check Tool Reports in the Browser User Interface
Generate Oracle ORAchk Health Check Tool reports using the Browser User Interface. - Running Oracle Trace File Analyzer (TFA) Collector Commands
Understand the installed location oftfactl
and the options for the command. - Sanitizing Sensitive Information in Diagnostic Collections
Oracle Autonomous Health Framework uses Adaptive Classification and Redaction (ACR) to sanitize sensitive data. - Sanitizing Sensitive Information in Oracle Trace File Analyzer Collections
You can redact (sanitize or mask) Oracle Trace File Analyzer diagnostic collections. - Sanitizing Sensitive Information in Oracle ORAchk Output
You can sanitize Oracle ORAchk output.
Parent topic: Troubleshooting Oracle Database Appliance
About Installing Oracle Autonomous Health Framework
Oracle Autonomous Health Framework is installed automatically when you provision or patch to Oracle Database Appliance release 19.8.
When you provision or patch your appliance to Oracle Database Appliance
release 19.8, Oracle Autonomous Health Framework is
installed in the path /opt/oracle/dcs/oracle.ahf
.
[root@oak ~]# rpm -q oracle-ahf
oracle-ahf-193000-########.x86_64
Note:
When you provision or patch to Oracle Database Appliance release 19.8, Oracle Autonomous Health Framework automatically provides Oracle ORAchk Health Check Tool and Oracle Trace File Analyzer Collector.- Operating system kernel parameters and packages
- Oracle Database Database parameters, and other database configuration settings
- Oracle Grid Infrastructure, which includes Oracle Clusterware and Oracle Automatic Storage Management
- Encapsulation of diagnostic data collection for all Oracle Grid Infrastructure and Oracle RAC components on all cluster nodes into a single command, which you run from a single node
- Option to "trim" diagnostic files during data collection to reduce data upload size
- Options to isolate diagnostic data collection to a given time period, and to a particular product component, such as Oracle ASM, Oracle Database, or Oracle Clusterware
- Centralization of collected diagnostic output to a single node in Oracle Database Appliance, if desired
- On-Demand Scans of all log and trace files for conditions indicating a problem
- Real-Time Scan Alert Logs for conditions indicating a problem (for example, Database Alert Logs, Oracle ASM Alert Logs, and Oracle Clusterware Alert Logs)
Using the Oracle ORAchk Health Check Tool
Run Oracle ORAchk to audit configuration settings and check system health.
Note:
Before running ORAchk, check for the latest version of Oracle Autonomous Health Framework, and download and install it. See My Oracle Support Note 2550798.1 for more information about downloading and installing the latest verion of Oracle Autonomous Health Framework.Running ORAchk on Oracle Database Appliance 19.8 Baremetal Systems for New Installation
When you provision or upgrade to Oracle Database Appliance 19.8, ORAchk is installed using Oracle Autonomous Framework in the
directory /opt/oracle/dcs/oracle.ahf
.
[root@oak bin]# orachk
When all checks are finished, a detailed report is available. The output displays the location of the report in an HTML format and the location of a zip file if you want to upload the report. For example, you can choose the filter to show failed checks only, show checks with a Fail, Warning, Info, or Pass status, or any combination.
Review the Oracle Database Appliance Assessment Report and system health and troubleshoot any issues that are identified. The report includes a summary and filters that enable you to focus on specific areas.
Running ORAchk on Oracle Database Appliance 19.8 Virtualized Platform
When you provision or upgrade to Oracle Database Appliance 19.8, ORAchk is installed using Oracle Autonomous
Framework in the directory /opt/oracle.ahf
.
orachk
, use the following
command:[root@oak bin]# oakcli orachk
Generating and Viewing Oracle ORAchk Health Check Tool Reports in the Browser User Interface
Generate Oracle ORAchk Health Check Tool reports using the Browser User Interface.
Running Oracle Trace File Analyzer (TFA) Collector Commands
Understand the installed location of tfactl
and the options
for the command.
About Using tfactl to Collect Diagnostic Information
When you provision or upgrade to Oracle Database Appliance 19.8, Oracle Trace File Analyzer (TFA) Collector is
installed in the directory
/opt/oracle.ahf/bin/tfactl
. You can
invoke the command line utility for TFA, tfactl
from the directory /opt/oracle.ahf/bin/tfactl
,
or simply type tfactl
.
You can use the following command options to run tfactl
:
/opt/oracle.ahf/bin/tfactl diagcollect -ips|-oda|-odalite|-dcs|-odabackup|
-odapatching|-odadataguard|-odaprovisioning|-odaconfig|-odasystem|-odastorage|-database|
-asm|-crsclient|-dbclient|-dbwlm|-tns|-rhp|-procinfo|-afd|-crs|-cha|-wls|
-emagent|-oms|-ocm|-emplugins|-em|-acfs|-install|-cfgtools|-os|-ashhtml|-ashtext|
-awrhtml|-awrtext -mask -sanitize
Table 18-1 Command Options for tfactl Tool
Option | Description |
---|---|
-h |
(Optional) Describes all the options for this command. |
-ips |
(Optional) Use this option to view the diagnostic logs for the specified component. |
-oda |
(Optional) Use this option to view the logs for the entire Appliance. |
-odalite |
(Optional) Use this option to view the diagnostic logs for the odalite component. |
-dcs |
(Optional) Use this option to view the DCS log files. |
-odabackup |
(Optional) Use this option to view the diagnostic logs for the backup components for the Appliance. |
-odapatching |
(Optional) Use this option to view the diagnostic logs for patching components of the Appliance. |
-odadataguard |
(Optional) Use this option to view the diagnostic logs for Oracle Data Guard component of the Appliance. |
-odaprovisioning |
(Optional) Use this option to view provisioning logs for the Appliance. |
-odaconfig |
(Optional) Use this option to view configuration-related diagnostic logs. |
-odasystem |
(Optional) Use this option to view system information. |
-odastorage |
(Optional) Use this option to view the diagnostic logs for the Appliance storage. |
-database |
(Optional) Use this option to view database-related log files. |
-asm |
(Optional) Use this option to view the diagnostic logs for the Appliance. |
-crsclient |
(Optional) Use this option to view the diagnostic logs for the Appliance. |
-dbclient |
(Optional) Use this option to view the diagnostic logs for the Appliance. |
-dbwlm |
(Optional) Use this option to view the diagnostic logs for the specified component. |
-tns |
(Optional) Use this option to view the diagnostic logs for TNS. |
-rhp |
(Optional) Use this option to view the diagnostic logs for Rapid Home Provisioning. |
-afd |
(Optional) Use this option to view the diagnostic logs for Oracle ASM Filter Driver. |
-crs |
(Optional) Use this option to view the diagnostic logs for Oracle Clusterware. |
-cha |
(Optional) Use this option to view the diagnostic logs for the Cluster Health Monitor. |
-wls |
(Optional) Use this option to view the diagnostic logs for Oracle WebLogic Server. |
-emagent |
(Optional) Use this option to view the diagnostic logs for the Oracle Enterprise Manager agent. |
-oms |
(Optional) Use this option to view the diagnostic logs for the Oracle Enterprise Manager Management Service. |
-ocm |
(Optional) Use this option to view the diagnostic logs for the specified component. |
-emplugins |
(Optional) Use this option to view the diagnostic logs for Oracle Enterprise Manager plug-ins. |
-em |
(Optional) Use this option to view the diagnostic logs for Oracle Enterprise Manager deployment. |
-acfs |
(Optional) Use this option to view the diagnostic logs for Oracle ACFS storage. |
-install |
(Optional) Use this option to view the diagnostic logs for installation. |
-cfgtools |
(Optional) Use this option to view the diagnostic logs for the configuration tools. |
-os |
(Optional) Use this option to view the diagnostic logs for the operating system. |
-ashhtml |
(Optional) Use this option to view the diagnostic logs for the specified component. |
-ashtext |
(Optional) Use this option to view the diagnostic logs for the Appliance. |
-awrhtml |
(Optional) Use this option to view the diagnostic logs for the Appliance. |
-awrtext |
(Optional) Use this option to view the diagnostic logs for the specified component. |
|
(Optional) Use this option to choose to mask sensitive data in the log collection. |
|
(Optional) Use this option to choose to sanitize (redact) sensitive data in the log collection. |
Usage Notes
You can use Trace File Collector (the tfactl
command) to collect all log files for the Oracle Database Appliance components.
You can also use the command odaadmcli manage
diagcollect
, with similar command options, to
collect the same diagnostic information.
For more information about using the -mask
and
-sanitize
options, see the next topic.
Sanitizing Sensitive Information in Diagnostic Collections
Oracle Autonomous Health Framework uses Adaptive Classification and Redaction (ACR) to sanitize sensitive data.
After collecting copies of diagnostic data, Oracle Trace File Analyzer and Oracle ORAchk use Adaptive Classification and Redaction (ACR) to sanitize sensitive data in the collections. ACR uses a Machine Learning based engine to redact a pre-defined set of entity types in a given set of files. ACR also sanitizes or masks entities that occur in path names. Sanitization replaces a sensitive value with random characters. Masking replaces a sensitive value with a series of asterisks ("*").
- Host names
- IP addresses
- MAC addresses
- Oracle Database names
- Tablespace names
- Service names
- Ports
- Operating system user names
ACR also masks user data from the database appearing in block and redo dumps.
Example 18-4 Block dumps before redaction
14A533F40 00000000 00000000 00000000 002C0000 [..............,.]
14A533F50 35360C02 30352E30 31322E37 380C3938 [..650.507.2189.8]
14A533F60 31203433 37203332 2C303133 360C0200 [34 123 7310,...6]
Example 18-5 Block dumps after redaction
14A533F40 ******** ******** ******** ******** [****************]
14A533F50 ******** ******** ******** ******** [****************]
14A533F60 ******** ******** ******** ******** [****************]
Example 18-6 Redo dumps before redaction
col 74: [ 1] 80
col 75: [ 5] c4 0b 19 01 1f
col 76: [ 7] 78 77 06 16 0c 2f 26
Example 18-7 Redo dumps after redaction
col 74: [ 1] **
col 75: [ 5] ** ** ** ** **
col 76: [ 7] ** ** ** ** ** ** **
Redaction of Literal Values in SQL Statements in AWR, ASH and ADDM Reports
Automatic Workload Repository (AWR), Active Session History (ASH), and Automatic Database Diagnostic Monitor (ADDM) reports are HTML files, which contain sensitive entities such as hostnames, database names, and service names in the form of HTML tables. In addition to these sensitive entities, they also contain SQL statements, that can contain bind variables or literal values from tables. These literal values can be sensitive personal information (PI) stored in databases. ACR processes such reports to identify and redact both usual sensitive entities and literal values present in the SQL statements.
Sanitizing Sensitive Information Using odaadmcli Command
odaadmcli manage diagcollect
command to collect
diagnostic logs for Oracle Database Appliance components. During collection, ACR can
be used to redact (sanitize or mask) the diagnostic
logs.odaadmcli manage diagcollect [--dataMask|--dataSanitize]
In the command, the --dataMask
option blocks out the
sensitive data in all collections, for example, replaces myhost1
with *******
. The default is None. The
--dataSanitize
option replaces the sensitive data in all
collections with random characters, for example, replaces myhost1
with orzhmv1
. The default is None.
Sanitizing Sensitive Information in Oracle Trace File Analyzer Collections
You can redact (sanitize or mask) Oracle Trace File Analyzer diagnostic collections.
Enabling Automatic Redaction
To enable automatic redaction, use the command:
tfactl set redact=[mask|sanitize|none]
In the command, the -mask
option blocks out the
sensitive data in all collections, for example, replaces myhost1
with *******
. The -sanitize
option replaces the
sensitive data in all collections with random characters, for example, replaces
myhost1
with orzhmv1
. The
none
option does not mask or sanitize sensitive data in
collections. The default is none
.
Enabling On-Demand Redaction
You can redact collections on-demand, for example, tfactl diagcollect -srdc
ORA-00600 -mask or tfactl diagcollect -srdc ORA-00600 -sanitize
.
- To mask sensitive data in all
collections:
tfactl set redact=mask
- To sanitize sensitive data in all
collections:
tfactl set redact=sanitize
Example 18-8 Masking or Sanitizing Sensitive Data in a Specific Collection
tfactl diagcollect -srdc ORA-00600 -mask
tfactl diagcollect -srdc ORA-00600 -sanitize
Sanitizing Sensitive Information in Oracle ORAchk Output
You can sanitize Oracle ORAchk output.
To sanitize Oracle ORAchk output, include the -sanitize
option, for
example, orachk -profile asm -sanitize
. You can also sanitize post
process by passing in an existing log, HTML report, or a zip file, for example,
orachk -sanitize file_name
.
Example 18-9 Sanitizing Sensitive Information in Specific Collection IDs
orachk -sanitize comma_delimited_list_of_collection_IDs
Example 18-10 Sanitizing a File with Relative Path
orachk -sanitize new/orachk_node061919_053119_001343.zip
orachk is sanitizing
/scratch/testuser/may31/new/orachk_node061919_053119_001343.zip. Please wait...
Sanitized collection is:
/scratch/testuser/may31/orachk_aydv061919_053119_001343.zip
orachk -sanitize ../orachk_node061919_053119_001343.zip
orachk is sanitizing
/scratch/testuser/may31/../orachk_node061919_053119_001343.zip. Please wait...
Sanitized collection is:
/scratch/testuser/may31/orachk_aydv061919_053119_001343.zip
Example 18-11 Sanitizing Oracle Autonomous Health Framework Debug Log
orachk -sanitize new/orachk_debug_053119_023653.log
orachk is sanitizing /scratch/testuser/may31/new/orachk_debug_053119_023653.log.
Please wait...
Sanitized collection is: /scratch/testuser/may31/orachk_debug_053119_023653.log
Example 18-12 Running Full Sanity Check
orachk -localonly -profile asm -sanitize -silentforce
Detailed report (html) -
/scratch/testuser/may31/orachk_node061919_053119_04448/orachk_node061919_053119_04448.html
orachk is sanitizing /scratch/testuser/may31/orachk_node061919_053119_04448.
Please wait...
Sanitized collection is: /scratch/testuser/may31/orachk_aydv061919_053119_04448
UPLOAD [if required] - /scratch/testuser/may31/orachk_node061919_053119_04448.zip
orachk -rmap all|comma_delimited_list_of_element_IDs
You can also use orachk -rmap
to lookup a value sanitized by Oracle
Trace File Analyzer.
Example 18-13 Printing the Reverse Map of Sanitized Elements
orachk -rmap MF_NK1,fcb63u2
________________________________________________________________________________
| Entity Type | Substituted Entity Name | Original Entity Name |
________________________________________________________________________________
| dbname | MF_NK1 | HR_DB1 |
| dbname | fcb63u2 | rac12c2 |
________________________________________________________________________________
orachk -rmap all
Running the Disk Diagnostic Tool
Use the Disk Diagnostic Tool to help identify the cause of disk problems.
The tool produces a list of 14 disk checks for each node. To display details, where n represents the disk resource name, enter the following command:
# odaadmcli stordiag n
# odaadmcli stordiag pd_00
Parent topic: Troubleshooting Oracle Database Appliance
Running the Oracle Database Appliance Hardware Monitoring Tool
The Oracle Database Appliance Hardware Monitoring Tool displays the status of different hardware components in Oracle Database Appliance server.
The tool is implemented with the Trace File Analyzer collector. Use the tool both on bare-metal and on virtualized systems. The Oracle Database Appliance Hardware Monitoring Tool reports information only for the node on which you run the command. The information it displays in the output depend on the component that you select to review.
Bare Metal Platform
You can see the list of monitored components by running the command odaadmcli show -h
To see information about specific components, use the command syntax odaadmcli show component
, where component
is the hardware component that you want to query. For example, the command odaadmcli show power
shows information specifically about the Oracle Database Appliance power supply:
# odaadmcli show power
NAME HEALTH HEALTH_DETAILS PART_NO. SERIAL_NO.
Power_Supply_0 OK - 7079395 476856Z+1514CE056G
(Continued)
LOCATION INPUT_POWER OUTPUT_POWER INLET_TEMP EXHAUST_TEMP
PS0 Present 112 watts 28.000 degree C 34.938 degree C
Virtualized Platform
You can see the list of monitored components by running the command oakcli show -h
To see information about specific components, use the command syntax oakcli show component
, where component
is the hardware component that you want to query. For example, the command oakcli show power
shows information specifically about the Oracle Database Appliance power supply:
# oakcli show power
NAME HEALTH HEALTH_DETAILS PART_NO. SERIAL_NO.
Power Supply_0 OK - 7047410 476856F+1242CE0020
Power Supply_1 OK - 7047410 476856F+1242CE004J
(Continued)
LOCATION INPUT_POWER OUTPUT_POWER INLET_TEMP EXHAUST_TEMP
PS0 Present 88 watts 31.250 degree C 34.188 degree C
PS1 Present 66 watts 31.250 degree C 34.188 degree C
Note:
Oracle Database Appliance Server Hardware Monitoring Tool is enabled during initial startup of ODA_BASE on Oracle Database Appliance Virtualized Platform. When it starts, the tool collects base statistics for about 5 minutes. During this time, the tool displays the message "Gathering Statistics…" message.
Parent topic: Troubleshooting Oracle Database Appliance
Configuring a Trusted SSL Certificate for Oracle Database Appliance
The Browser User Interface and DCS Controller use SSL-based HTTPS protocol for secure communication. Understand the implications of this added security and the options to configure SSL certificates.
The Browser User Interface provides an added layer of security using certificates and encryption, when an administrator interacts with the appliance. Encryption of data ensures that:
- Data is sent to the intended recipient, and not to any malicious third-party.
- When data is exchanged between the server and the browser, data interception cannot occur nor can the data be edited.
When you connect to the Browser User Interface through HTTPS, the DCS Controller presents your browser with a certificate to verify the identity of appliance. If the web browser finds that the certificate is not from a trusted Certificate Authority (CA), then the browser assumes it has encountered an untrusted source, and generates a security alert message. The security alert dialog boxes display because Browser User Interface security is enabled through HTTPS and SSL, but you have not secured your Web tier properly with a trusted matching certificate from a Certificate Authority. It is possible to purchase commercial certificates from a Certificate Authority or create your own and register them with a Certificate Authority. However, the server and browser certificates must use the same public certificate key and trusted certificate to avoid the error message produced by the browser.
There are three options to configure your certificates:
- Create your own key and Java keystore, ensure it is signed by a Certificate Authority (CA) and then import it for use.
- Package an existing Privacy Enhanced Mail (PEM) format key and certificates in a new Java keystore.
- Convert an existing PKCS or PFX keystore to a Java keystore
and configure it for the Browser User Interface.
Note:
For Oracle Database Appliance High-Availability hardware models, run the configuration steps on both nodes.
The following topics explain how to configure these options:
- Option 1: Creating a Key and Java Keystore and Importing a Trusted Certificate
Usekeytool
, a key and certificate management utility, to create a keystore and a signing request. - Option 2: Packaging an Existing PEM-format Key and Certificates in a New Java Keystore
Use the OpenSSL tool to package Privacy Enhanced Mail (PEM) files in a PKCS keystore. - Option 3: Converting an Existing PKCS or PFX Keystore to a Java Keystore
If you have an existing PKCS or PFX keystore for your server's domain, convert it to a Java keystore. - Configuring the DCS Server to Use Custom Keystore
After packaging or converting your keystore into Java keystore, configure the DCS server to use your keystore. - Configuring the DCS Agent for Custom Certificate
After you import the certificate into the keystore, configure the DCS agent to use the same certificate.
Parent topic: Troubleshooting Oracle Database Appliance
Option 1: Creating a Key and Java Keystore and Importing a Trusted Certificate
Use keytool
, a key and certificate management utility, to
create a keystore and a signing request.
Option 2: Packaging an Existing PEM-format Key and Certificates in a New Java Keystore
Use the OpenSSL tool to package Privacy Enhanced Mail (PEM) files in a PKCS keystore.
Option 3: Converting an Existing PKCS or PFX Keystore to a Java Keystore
If you have an existing PKCS or PFX keystore for your server's domain, convert it to a Java keystore.
Configuring the DCS Server to Use Custom Keystore
After packaging or converting your keystore into Java keystore, configure the DCS server to use your keystore.
Disabling the Browser User Interface
You can also disable the Browser User Interface. Disabling the Browser User Interface means you can only manage your appliance through the command-line interface.
Parent topic: Troubleshooting Oracle Database Appliance
Preparing Log Files for Oracle Support Services
If you have a system fault that requires help from Oracle Support Services, then you may need to provide log records to help Oracle support diagnose your issue.
- Use the Bill Of Materials report saved in the
/opt/oracle/dcs/Inventory/
directory, to enable Oracle Support to help troubleshoot errors, if necessary. - You can use Trace File Collector (the
tfactl
command) to collect all log files for the Oracle Database Appliance components. - Use the command
odaadmcli manage diagcollect
to collect diagnostic files to send to Oracle Support Services.
The odaadmcli manage diagcollect
command consolidates information from log files stored on Oracle Database Appliance
into a single log file for use by Oracle Support Services. The location of the file
is specified in the command output.
Example 18-14 Collecting log file information for a time period, masking sensitive data
# odaadmcli manage diagcollect --dataMask --fromTime 2019-08-12 --toTime 2019-08-25
DataMask is set as true
FromTime is set as: 2019-08-12
ToTime is set as: 2019-08-25
TFACTL command is: /opt/oracle/tfa/tfa_home/bin/tfactl
Data mask is set.
Collect data from 2019-08-12
Collect data to 2019-08-25
Parent topic: Troubleshooting Oracle Database Appliance