14 Troubleshooting Oracle Database Appliance
Understand tools you can use to validate changes and troubleshoot Oracle Database Appliance problems.
- Viewing Component Information on the Appliance
View details of all the components installed on the appliance, and the RPM drift information. - Errors When Logging into the Browser User Interface
If you have problems logging into the Browser User Interface, then it may be due to your browser or credentials. - Analyzing the Pre-Checks Report for Operating System Upgrades
Understand the patching pre-checks report and the components checked during the operating system upgrade when patching Oracle Database Appliance. - Analyzing the Post Upgrade Checks Report for Operating System Upgrades
Understand the patching post upgrade checks report after performing the operating system upgrade when patching Oracle Database Appliance. - Creating ODABR Snapshot with Insufficient Space During Operating System Upgrade
Understand how to create ODABR snapshots when space is insufficient during the operating system upgrade when patching Oracle Database Appliance. - Errors when re-imaging Oracle Database Appliance
Understand how to troubleshoot errors that occur when re-imaging Oracle Database Appliance. - Using Oracle Autonomous Health Framework for Running Diagnostics
Oracle Autonomous Health Framework collects and analyzes diagnostic data collected, and proactively identifies issues before they affect the health of your system. - Running the Disk Diagnostic Tool
Use the Disk Diagnostic Tool to help identify the cause of disk problems. - Running the Oracle Database Appliance Hardware Monitoring Tool
The Oracle Database Appliance Hardware Monitoring Tool displays the status of different hardware components in Oracle Database Appliance server. - Configuring a Trusted SSL Certificate for Oracle Database Appliance
The Browser User Interface and DCS Controller use SSL-based HTTPS protocol for secure communication. Understand the implications of this added security and the options to configure SSL certificates. - Disabling the Browser User Interface
You can also disable the Browser User Interface. Disabling the Browser User Interface means you can only manage your appliance through the command-line interface. - Preparing Log Files for Oracle Support Services
If you have a system fault that requires help from Oracle Support Services, then you may need to provide log records to help Oracle support diagnose your issue.
Viewing Component Information on the Appliance
View details of all the components installed on the appliance, and the RPM drift information.
Viewing the Bill of Materials in the Browser User Interface
Use the Appliance tab in the Browser User Interface to view information about your deployment and the installed components. The Advanced Information tab displays information about the following components:
-
Grid Infrastructure Version, and the home directory
-
Database Version, Home location, and Edition
-
Location and details about the databases configured
-
All patches applied to the appliance
-
Firmware Controller and Disks
-
ILOM information
-
BIOS version
-
List of RPMs
In the List of RPMs section, click Show and then click RPM Drift to view the differences between the RPMs installed on the appliance, and the RPMs shipped in the latest Oracle Database Appliance Patch Bundle Update release.
Click Download to save the components report. You can use this report to help diagnose any deployment issues.
Viewing the Bill of Materials from the Command Line
The bill of materials is also available through the command line for bare metal and virtualized platforms deployments. The information about the installed components is collected according to a set schedule, and stored in the location /opt/oracle/dcs/Inventory/
for bare metal deployments and in the /opt/oracle/oak/Inventory/
directory for virtualized platforms. The file is stored in the format oda_bom_TimeStamp.json
. Use the command describe-system
to view the bill of materials on the command line. See the Oracle Database Command-Line Interface chapter for command options and usage notes.
Example 14-1 Example Command to View the Bill of Materials from the Command Line for Bare Metal Deployments
# odacli describe-system -b
ODA Components Information
------------------------------
Component Name Component Details
--------------- -----------------------------------------------------------------------------------------------
NODE Name : oda1
Domain Name : testdomain.com
Time Stamp : April 21, 2020 6:21:15 AM UTC
RPMS Installed RPMS : abrt-2.1.11-55.0.1.el7.x86_64,
abrt-addon-ccpp-2.1.11-55.0.1.el7.x86_64,
abrt-addon-kerneloops-2.1.11-55.0.1.el7.x86_64,
abrt-addon-pstoreoops-2.1.11-55.0.1.el7.x86_64,
abrt-addon-python-2.1.11-55.0.1.el7.x86_64,
abrt-addon-vmcore-2.1.11-55.0.1.el7.x86_64,
abrt-addon-xorg-2.1.11-55.0.1.el7.x86_64,
abrt-cli-2.1.11-55.0.1.el7.x86_64,
abrt-console-notification-2.1.11-55.0.1.el7.x86_64,
abrt-dbus-2.1.11-55.0.1.el7.x86_64,
abrt-libs-2.1.11-55.0.1.el7.x86_64,
abrt-python-2.1.11-55.0.1.el7.x86_64,
abrt-tui-2.1.11-55.0.1.el7.x86_64,
acl-2.2.51-14.el7.x86_64,
adwaita-cursor-theme-3.28.0-1.el7.noarch,
adwaita-icon-theme-3.28.0-1.el7.noarch,
aic94xx-firmware-30-6.el7.noarch,
aide-0.15.1-13.0.1.el7.x86_64,
alsa-firmware-1.0.28-2.el7.noarch,
alsa-lib-1.1.8-1.el7.x86_64,
alsa-tools-firmware-1.1.0-1.el7.x86_64,
at-3.1.13-24.el7.x86_64,
at-spi2-atk-2.26.2-1.el7.x86_64,
at-spi2-core-2.28.0-1.el7.x86_64,
atk-2.28.1-1.el7.x86_64,
attr-2.4.46-13.el7.x86_64,
audit-2.8.5-4.el7.x86_64,
audit-libs-2.8.5-4.el7.x86_64,
audit-libs-python-2.8.5-4.el7.x86_64,
augeas-libs-1.4.0-9.el7.x86_64,
authconfig-6.2.8-30.el7.x86_64,
autogen-libopts-5.18-5.el7.x86_64,
avahi-libs-0.6.31-19.el7.x86_64,
basesystem-10.0-7.0.1.el7.noarch,
bash-4.2.46-33.el7.x86_64,
bash-completion-2.1-6.el7.noarch,
bc-1.06.95-13.el7.x86_64,
bind-export-libs-9.11.4-9.P2.el7.x86_64,
bind-libs-9.11.4-9.P2.el7.x86_64,
bind-libs-lite-9.11.4-9.P2.el7.x86_64,
bind-license-9.11.4-9.P2.el7.noarch,
bind-utils-9.11.4-9.P2.el7.x86_64,
binutils-2.27-41.base.0.7.el7_7.2.x86_64,
biosdevname-0.7.3-2.el7.x86_64,
blktrace-1.0.5-9.el7.x86_64,
bnxtnvm-1.40.10-1.x86_64,
boost-date-time-1.53.0-27.el7.x86_64,
boost-filesystem-1.53.0-27.el7.x86_64,
boost-iostreams-1.53.0-27.el7.x86_64,
....
....
....
Example 14-2 Example Command to View the Bill of Materials from the Command Line for Virtualized Platforms
# oakcli describe-system -b
Example 14-3 Example Command to View the Bill of Materials Report from the Stored Location
# ls -la /opt/oracle/dcs/Inventory/
total 264
-rw-r--r-- 1 root root 83550 Apr 26 05:41 oda_bom_2018-04-26_05-41-36.json
Parent topic: Troubleshooting Oracle Database Appliance
Errors When Logging into the Browser User Interface
If you have problems logging into the Browser User Interface, then it may be due to your browser or credentials.
Note:
Oracle Database Appliance uses self-signed certificates. Your browser determines how you log into the Browser User Interface. Depending on the browser and browser version, you may receive a warning or error that the certificate is invalid or not trusted because it is self-signed, or that the connection is not private. Ensure that you accept the self-signed certificate for the agent and Browser User Interface.Follow these steps to log into the Browser User Interface:
Note:
If you have any issues logging into the Oracle Database Appliance Browser User Interface on browsers such as macOS Catalina and Google Chrome, then you may need to use any workaround as described on the official site for the product.Analyzing the Pre-Checks Report for Operating System Upgrades
Understand the patching pre-checks report and the components checked during the operating system upgrade when patching Oracle Database Appliance.
When you patch your Oracle Database Appliance deployment to release 19.6, you must first upgrade your operating system to Oracle Linux 7. The patching pre-checks report contains a section on the operating system upgrade checks run, similar to the following:
Pre-Check Status Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions Success Validated minimum supported versions.
Validate patching tag Success Validated patching tag: 19.6.0.0.0.
Is patch location available Success Patch location is available.
Validate ODABR is installed Failed ODABR utility is not installed on
node: scaoda804c1n1.
Validate ODABR snapshots exist Failed ODABR utility is not installed on
node: scaoda804c1n1.
Validate LVM free space Failed ODABR utility is not installed on
node: scaoda804c1n1.
Space checks for OS upgrade Success Validated space checks.
Install OS upgrade software Success Extracted OS upgrade patches into
/root/oda-upgrade. Do not remove this
directory untill OS upgrade completes.
Verify OS upgrade by running Success Results stored in:
preupgrade checks '/root/preupgrade-results/
preupg_results-200410092820.tar.gz' .
Read complete report file
'/root/preupgrade/result.html' before
attempting OS upgrade.
Validate custom RPMs installed Success No additional RPMs found installed on
node:scaoda804c1n1.
Scheduled jobs check Failed Scheduled jobs found. Disable
scheduled jobs before attempting OS
upgrade.
–-force
flag,
but it is is recommended that you fix these failures before proceeding with the
operating system upgrade.
- Validate ODABR is installed: Oracle Database Appliance Backup Recovery (ODABR) is required for recovering the system back to the pre-upgrade state should an error render the system unusable.
- Validate ODABR snapshots exist: Once ODABR is installed on the
system, ODACLI automatically creates ODABR snapshots before performing the
operating system upgrade. If snapshots are already present on the system when
odacli create-prepatchreport is run, this precheck fails, because ODACLI expects
to create these snapshots itself. If the user created snapshots or the operating
system upgrade was retried (due to a failure) after it had already created the
snapshots, this precheck will fail. Note that if snapshots already exist,
odacli update-server –c OS
still continues with the upgrade. - Validate LVM free space: This checks for the minimum space required enforced by ODABR to create snapshots.
- Verify operating system upgrade by running preupgrade checks: This precheck runs
the RHEL upgrade tool. It is advised that the user go through the
/root/preupgrade/result.html
file and fix any critical failures before performing the operating system upgrade. - Validate custom RPMs installed: ODACLI handles upgrade of only
those RPMs that are part of the Oracle Database Appliance operating system
image, as listed in the versionlock file, or those included as part of Oracle
Database Appliance software, such as DCS and HMP RPMs, or those required to
perform the operating system upgrade itself. As part of prechecks, any other
Oracle supplied RPMs are listed in
/root/oda-upgrade/rpms-added-from-Oracle
and other third-party RPMs in/root/oda-upgrade/rpms-added-from-Thirdparty
files respectively. For custom RPMs, you must separately upgrade these RPMs on each node after the operating system is successfully upgraded to Oracle Linux 7. These files are preserved in the same location after the operating system upgrade completes. - Scheduled jobs check: Use the
odacli list-schedules
command to list scheduled jobs and run theodacli update-schedule -d -i schedule id
command on both nodes to disable each listed job.
Parent topic: Troubleshooting Oracle Database Appliance
Analyzing the Post Upgrade Checks Report for Operating System Upgrades
Understand the patching post upgrade checks report after performing the operating system upgrade when patching Oracle Database Appliance.
The patching post upgrade checks report contains three categories of components (Operating System, Oracle Grid Infrastructure, RPMs).
Operating System Upgrade (OS upgrade check)
If the operating system is not upgraded, then the following error is displayed:
Comp Pre-Check Status Comments
----- ------------------------------ -------- --------------------------------------
OS OS upgrade check ERROR OS has not been upgraded to OL7
Action Required: Upgrade the node to Oracle Linux 7.
If the operating system is upgraded, then the following message is displayed:
Comp Pre-Check Status Comments
----- ------------------------------ -------- --------------------------------------
OS OS upgrade check SUCCESS OS has been upgraded to OL7
Operating System (ODABR Snapshot)
If ODABR RPM is not installed, then the following message is displayed:
Comp Pre-Check Status Comments
----- ------------------------------ -------- --------------------------------------
OS ODABR snapshot SKIPPED ODABR is not installed
If ODABR RPM is installed but ODABR snapshots do not exist, then the following message is displayed:
Comp Pre-Check Status Comments
----- ------------------------------ -------- --------------------------------------
OS ODABR snapshot SUCCESS No ODABR snapshots found
If ODABR RPM is installed and ODABR snapshots exist, then the following message is displayed:
Comp Pre-Check Status Comments
----- ------------------------------ -------- --------------------------------------
OS ODABR snapshot WARNING ODABR snapshot found. Run 'odabr delsnap' to delete.
Action Required: Run /opt/odabr/odabr delsnap
to delete the
snapshots.
Oracle Grid Infrastructure (GI Upgrade Check)
If Oracle Grid Infrastructure is not installed, then the following message is displayed:
Comp Pre-Check Status Comments
----- ------------------------------ -------- --------------------------------------
GI GI upgrade check SKIPPED Grid software is not installed
If the installed Oracle Grid Infrastructure version is 18.x, then the following message is displayed:
Comp Pre-Check Status Comments
----- ------------------------------ -------- --------------------------------------
GI GI upgrade check INFO GI home needs to update to 19.6.0.0.200114
Action Required: Run the odacli update-server
command to
update all components. Delete any ODABR snaps before updating any components.
If the installed Oracle Grid Infrastructure version is 19.x, then the following message is displayed:
Comp Pre-Check Status Comments
----- ------------------------------ -------- --------------------------------------
GI GI upgrade check SUCCESS GI is currently running at latest version
Oracle Grid Infrastructure (GI Status Check)
If Oracle Grid Infrastructure is not installed, then the following message is displayed:
Comp Pre-Check Status Comments
----- ------------------------------ -------- --------------------------------------
GI GI status check SKIPPED Grid software is not installed
If the installed Oracle Grid Infrastructure stack is not running, then the following message is displayed:
Comp Pre-Check Status Comments
----- ------------------------------ -------- --------------------------------------
GI GI status check FAILED Clusterware is not running on the node
Action Required: Check why Oracle Grid Infrastructure is not running.
If the installed Oracle Grid Infrastructure stack is running, then the following message is displayed:
Comp Pre-Check Status Comments
----- ------------------------------ -------- --------------------------------------
GI GI status check SUCCESS Clusterware is running on the node
RPM (Extra RPM Check)
If no extra RPMS were discovered when the operating system was Oracle Linux 6, then the following message is displayed:
Comp Pre-Check Status Comments
----- ------------------------------ -------- --------------------------------------
RPM Extra RPM check SKIPPED No extra RPMs found when OS was at OL6
If extra RPMS were discovered when the operating system was Oracle Linux 6, then the following message is displayed:
Comp Pre-Check Status Comments
----- ------------------------------ -------- --------------------------------------
RPM Extra RPM check FAILED Extra OL6 RPMs were detected. Install OL7 equivalent
Action Required: Check the directory
/root/oda-upgrade/rpms-added-from-ThirdParty
and
/root/oda-upgrade/rpms-added-from-Oracle
. Download and install
the equivalent Oracle Linux 7 RPMs for all the RPMs listed in these directories.
Sample Post Upgrade Checks Report
Sample output from a system after all upgrading all components.
# odacli update-server-postcheck -v 19.6.0.0.0
Upgrade post-check report
-------------------------
Node Name
---------------
node1
Comp Pre-Check Status Comments
----- ------------------------------ -------- --------------------------------------
OS OS upgrade check SUCCESS OS has been upgraded to OL7
GI GI upgrade check SUCCESS GI is currently running at latest version
GI GI status check SUCCESS Clusterware is running on the node
OS ODABR snapshot SKIPPED ODABR is not installed
RPM Extra RPM check SUCCESS No extra RPMs found when OS was at OL6
Parent topic: Troubleshooting Oracle Database Appliance
Creating ODABR Snapshot with Insufficient Space During Operating System Upgrade
Understand how to create ODABR snapshots when space is insufficient during the operating system upgrade when patching Oracle Database Appliance.
Caution:
If you have taken an ODABR snapshot with reduced LVM snapshot size, and you need to restore the system, then ensure that the status of all snapshots is active prior to performing the restore.Note:
It is recommended that you ensure available space of 20 GB each for/root
, /opt
, and
/u01
volumes for the operating system upgrade. However, this
space requirement may vary based on your deployment configuration.
odabr
command, and specify the size of the
snapshots for the volumes /root
, /opt
, and
/u01
. The snapshot size for /root
must be equal to
the actual size of the /root
volume (30GB).
Run the following command to view the size of the volume:
# df -h /root /opt /u01
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroupSys-LogVolRoot 30G 13G 16G 44% /
/dev/mapper/VolGroupSys-LogVolOpt 59G 54G 2.4G 96% /opt
/dev/mapper/VolGroupSys-LogVolU01 99G 83G 11G 89% /u01
The Size column displays the size of the volume. In the above example, the size of
/root
is 30 GB.
Run the following command to calculate the LVM free space:
/sbin/lvm pvs --units g -S vg_name=VolGroupSys -o vg_free 2> /tmp/lvm.out
VFree
104.00g
The output indicates that there is 104GB of free space available. Since there is less than 190 GB of free space, an error is reported during pre-checks.
/root
, 20 GB for /u01
, and 25 GB for
/opt
, run the following
command:opt/odabr/odabr backup /opt/odabr/odabr backup -snap -rsize 30 -usize 20 -osize 25
/opt/odabr/odabr infosnap
Parent topic: Troubleshooting Oracle Database Appliance
Errors when re-imaging Oracle Database Appliance
Understand how to troubleshoot errors that occur when re-imaging Oracle Database Appliance.
If re-imaging Oracle Database Appliance fails, with old header issues such as errors in storage discovery, or in running GI root scripts, or disk group RECO creation, then use the force mode with cleanup.pl
.
# cleanup.pl -f
To ensure that re-imaging is successful, remove the old headers from the storage disks by running the secure erase tool. Verify that the OAK/ASM headers are removed.
# cleanup.pl -erasedata
# cleanup.pl -checkHeader
Retry the re-imaging operation.
Related Topics
Parent topic: Troubleshooting Oracle Database Appliance
Using Oracle Autonomous Health Framework for Running Diagnostics
Oracle Autonomous Health Framework collects and analyzes diagnostic data collected, and proactively identifies issues before they affect the health of your system.
- About Installing Oracle Autonomous Health Framework
Oracle Autonomous Health Framework is installed automatically when you provision or patch to Oracle Database Appliance release 19.6. - Using the Oracle ORAchk Health Check Tool
Run Oracle ORAchk to audit configuration settings and check system health. - Generating and Viewing Oracle ORAchk Health Check Tool Reports in the Browser User Interface
Generate Oracle ORAchk Health Check Tool reports using the Browser User Interface. - Running Oracle Trace File Analyzer (TFA) Collector Commands
Understand the installed location oftfactl
and the options for the command. - Sanitizing Sensitive Information in Diagnostic Collections
Oracle Autonomous Health Framework uses Adaptive Classification and Redaction (ACR) to sanitize sensitive data. - Sanitizing Sensitive Information in Oracle Trace File Analyzer Collections
You can redact (sanitize or mask) Oracle Trace File Analyzer diagnostic collections. - Sanitizing Sensitive Information in Oracle ORAchk Output
You can sanitize Oracle ORAchk output.
Parent topic: Troubleshooting Oracle Database Appliance
About Installing Oracle Autonomous Health Framework
Oracle Autonomous Health Framework is installed automatically when you provision or patch to Oracle Database Appliance release 19.6.
When you provision or patch your appliance to Oracle Database Appliance
release 19.6, Oracle Autonomous Health Framework is
installed in the path /opt/oracle/dcs/oracle.ahf
.
[root@oak ~]# rpm -q oracle-ahf
oracle-ahf-193000-########.x86_64
Note:
When you provision or patch to Oracle Database Appliance release 19.6, Oracle Autonomous Health Framework automatically provides Oracle ORAchk Health Check Tool and Oracle Trace File Analyzer Collector.- Operating system kernel parameters and packages
- Oracle Database Database parameters, and other database configuration settings
- Oracle Grid Infrastructure, which includes Oracle Clusterware and Oracle Automatic Storage Management
- Encapsulation of diagnostic data collection for all Oracle Grid Infrastructure and Oracle RAC components on all cluster nodes into a single command, which you run from a single node
- Option to "trim" diagnostic files during data collection to reduce data upload size
- Options to isolate diagnostic data collection to a given time period, and to a particular product component, such as Oracle ASM, Oracle Database, or Oracle Clusterware
- Centralization of collected diagnostic output to a single node in Oracle Database Appliance, if desired
- On-Demand Scans of all log and trace files for conditions indicating a problem
- Real-Time Scan Alert Logs for conditions indicating a problem (for example, Database Alert Logs, Oracle ASM Alert Logs, and Oracle Clusterware Alert Logs)
Using the Oracle ORAchk Health Check Tool
Run Oracle ORAchk to audit configuration settings and check system health.
Note:
Before running ORAchk, check for the latest version of Oracle Autonomous Health Framework, and download and install it. See My Oracle Support Note 2550798.1 for more information about downloading and installing the latest verion of Oracle Autonomous Health Framework.Running ORAchk on Oracle Database Appliance 19.6 Baremetal Systems for New Installation
When you provision or upgrade to Oracle Database Appliance 19.6, ORAchk is installed using Oracle Autonomous Framework in the
directory /opt/oracle/dcs/oracle.ahf
.
[root@oak bin]# orachk
When all checks are finished, a detailed report is available. The output displays the location of the report in an HTML format and the location of a zip file if you want to upload the report. For example, you can choose the filter to show failed checks only, show checks with a Fail, Warning, Info, or Pass status, or any combination.
Review the Oracle Database Appliance Assessment Report and system health and troubleshoot any issues that are identified. The report includes a summary and filters that enable you to focus on specific areas.
Running ORAchk on Oracle Database Appliance 19.6 Virtualized Platform
When you provision or upgrade to Oracle Database Appliance 19.6, ORAchk is installed using Oracle Autonomous
Framework in the directory /opt/oracle.ahf
.
orachk
, use the following
command:[root@oak bin]# oakcli orachk
Generating and Viewing Oracle ORAchk Health Check Tool Reports in the Browser User Interface
Generate Oracle ORAchk Health Check Tool reports using the Browser User Interface.
Running Oracle Trace File Analyzer (TFA) Collector Commands
Understand the installed location of tfactl
and the options
for the command.
About Using tfactl to Collect Diagnostic Information
When you provision or upgrade to Oracle Database Appliance 19.6, Oracle Trace File Analyzer (TFA) Collector is
installed in the directory
/opt/oracle.ahf/bin/tfactl
. You can
invoke the command line utility for TFA, tfactl
from the directory /opt/oracle.ahf/bin/tfactl
,
or simply type tfactl
.
You can use the following command options to run tfactl
:
/opt/oracle.ahf/bin/tfactl diagcollect -ips|-oda|-odalite|-dcs|-odabackup|
-odapatching|-odadataguard|-odaprovisioning|-odaconfig|-odasystem|-odastorage|-database|
-asm|-crsclient|-dbclient|-dbwlm|-tns|-rhp|-procinfo|-afd|-crs|-cha|-wls|
-emagent|-oms|-ocm|-emplugins|-em|-acfs|-install|-cfgtools|-os|-ashhtml|-ashtext|
-awrhtml|-awrtext -mask -sanitize
Table 14-1 Command Options for tfactl Tool
Option | Description |
---|---|
-h |
(Optional) Describes all the options for this command. |
-ips |
(Optional) Use this option to view the diagnostic logs for the specified component. |
-oda |
(Optional) Use this option to view the logs for the entire Appliance. |
-odalite |
(Optional) Use this option to view the diagnostic logs for the odalite component. |
-dcs |
(Optional) Use this option to view the DCS log files. |
-odabackup |
(Optional) Use this option to view the diagnostic logs for the backup components for the Appliance. |
-odapatching |
(Optional) Use this option to view the diagnostic logs for patching components of the Appliance. |
-odadataguard |
(Optional) Use this option to view the diagnostic logs for Oracle Data Guard component of the Appliance. |
-odaprovisioning |
(Optional) Use this option to view provisioning logs for the Appliance. |
-odaconfig |
(Optional) Use this option to view configuration-related diagnostic logs. |
-odasystem |
(Optional) Use this option to view system information. |
-odastorage |
(Optional) Use this option to view the diagnostic logs for the Appliance storage. |
-database |
(Optional) Use this option to view database-related log files. |
-asm |
(Optional) Use this option to view the diagnostic logs for the Appliance. |
-crsclient |
(Optional) Use this option to view the diagnostic logs for the Appliance. |
-dbclient |
(Optional) Use this option to view the diagnostic logs for the Appliance. |
-dbwlm |
(Optional) Use this option to view the diagnostic logs for the specified component. |
-tns |
(Optional) Use this option to view the diagnostic logs for TNS. |
-rhp |
(Optional) Use this option to view the diagnostic logs for Rapid Home Provisioning. |
-afd |
(Optional) Use this option to view the diagnostic logs for Oracle ASM Filter Driver. |
-crs |
(Optional) Use this option to view the diagnostic logs for Oracle Clusterware. |
-cha |
(Optional) Use this option to view the diagnostic logs for the Cluster Health Monitor. |
-wls |
(Optional) Use this option to view the diagnostic logs for Oracle WebLogic Server. |
-emagent |
(Optional) Use this option to view the diagnostic logs for the Oracle Enterprise Manager agent. |
-oms |
(Optional) Use this option to view the diagnostic logs for the Oracle Enterprise Manager Management Service. |
-ocm |
(Optional) Use this option to view the diagnostic logs for the specified component. |
-emplugins |
(Optional) Use this option to view the diagnostic logs for Oracle Enterprise Manager plug-ins. |
-em |
(Optional) Use this option to view the diagnostic logs for Oracle Enterprise Manager deployment. |
-acfs |
(Optional) Use this option to view the diagnostic logs for Oracle ACFS storage. |
-install |
(Optional) Use this option to view the diagnostic logs for installation. |
-cfgtools |
(Optional) Use this option to view the diagnostic logs for the configuration tools. |
-os |
(Optional) Use this option to view the diagnostic logs for the operating system. |
-ashhtml |
(Optional) Use this option to view the diagnostic logs for the specified component. |
-ashtext |
(Optional) Use this option to view the diagnostic logs for the Appliance. |
-awrhtml |
(Optional) Use this option to view the diagnostic logs for the Appliance. |
-awrtext |
(Optional) Use this option to view the diagnostic logs for the specified component. |
|
(Optional) Use this option to choose to mask sensitive data in the log collection. |
|
(Optional) Use this option to choose to sanitize (redact) sensitive data in the log collection. |
Usage Notes
You can use Trace File Collector (the tfactl
command) to collect all log files for the Oracle Database Appliance components.
You can also use the command odaadmcli manage
diagcollect
, with similar command options, to
collect the same diagnostic information.
For more information about using the -mask
and
-sanitize
options, see the next topic.
Sanitizing Sensitive Information in Diagnostic Collections
Oracle Autonomous Health Framework uses Adaptive Classification and Redaction (ACR) to sanitize sensitive data.
After collecting copies of diagnostic data, Oracle Trace File Analyzer and Oracle ORAchk use Adaptive Classification and Redaction (ACR) to sanitize sensitive data in the collections. ACR uses a Machine Learning based engine to redact a pre-defined set of entity types in a given set of files. ACR also sanitizes or masks entities that occur in path names. Sanitization replaces a sensitive value with random characters. Masking replaces a sensitive value with a series of asterisks ("*").
- Host names
- IP addresses
- MAC addresses
- Oracle Database names
- Tablespace names
- Service names
- Ports
- Operating system user names
ACR also masks user data from the database appearing in block and redo dumps.
Example 14-4 Block dumps before redaction
14A533F40 00000000 00000000 00000000 002C0000 [..............,.]
14A533F50 35360C02 30352E30 31322E37 380C3938 [..650.507.2189.8]
14A533F60 31203433 37203332 2C303133 360C0200 [34 123 7310,...6]
Example 14-5 Block dumps after redaction
14A533F40 ******** ******** ******** ******** [****************]
14A533F50 ******** ******** ******** ******** [****************]
14A533F60 ******** ******** ******** ******** [****************]
Example 14-6 Redo dumps before redaction
col 74: [ 1] 80
col 75: [ 5] c4 0b 19 01 1f
col 76: [ 7] 78 77 06 16 0c 2f 26
Example 14-7 Redo dumps after redaction
col 74: [ 1] **
col 75: [ 5] ** ** ** ** **
col 76: [ 7] ** ** ** ** ** ** **
Sanitizing Sensitive Information Using odaadmcli Command
odaadmcli manage diagcollect
command to collect
diagnostic logs for Oracle Database Appliance components. During collection, ACR can
be used to redact (sanitize or mask) the diagnostic
logs.odaadmcli manage diagcollect [--dataMask|--dataSanitize]
In the command, the --dataMask
option blocks out the
sensitive data in all collections, for example, replaces myhost1
with *******
. The default is None. The
--dataSanitize
option replaces the sensitive data in all
collections with random characters, for example, replaces myhost1
with orzhmv1
. The default is None.
Sanitizing Sensitive Information in Oracle Trace File Analyzer Collections
You can redact (sanitize or mask) Oracle Trace File Analyzer diagnostic collections.
Enabling Automatic Redaction
To enable automatic redaction, use the command:
tfactl set redact=[mask|sanitize|none]
In the command, the -mask
option blocks out the
sensitive data in all collections, for example, replaces myhost1
with *******
. The -sanitize
option replaces the
sensitive data in all collections with random characters, for example, replaces
myhost1
with orzhmv1
. The
none
option does not mask or sanitize sensitive data in
collections. The default is none
.
Enabling On-Demand Redaction
You can redact collections on-demand, for example, tfactl diagcollect -srdc
ORA-00600 -mask or tfactl diagcollect -srdc ORA-00600 -sanitize
.
- To mask sensitive data in all
collections:
tfactl set redact=mask
- To sanitize sensitive data in all
collections:
tfactl set redact=sanitize
Example 14-8 Masking or Sanitizing Sensitive Data in a Specific Collection
tfactl diagcollect -srdc ORA-00600 -mask
tfactl diagcollect -srdc ORA-00600 -sanitize
Sanitizing Sensitive Information in Oracle ORAchk Output
You can sanitize Oracle ORAchk output.
To sanitize Oracle ORAchk output, include the -sanitize
option, for
example, orachk -profile asm -sanitize
. You can also sanitize post
process by passing in an existing log, HTML report, or a zip file, for example,
orachk -sanitize file_name
.
Example 14-9 Sanitizing Sensitive Information in Specific Collection IDs
orachk -sanitize comma_delimited_list_of_collection_IDs
Example 14-10 Sanitizing a File with Relative Path
orachk -sanitize new/orachk_node061919_053119_001343.zip
orachk is sanitizing
/scratch/testuser/may31/new/orachk_node061919_053119_001343.zip. Please wait...
Sanitized collection is:
/scratch/testuser/may31/orachk_aydv061919_053119_001343.zip
orachk -sanitize ../orachk_node061919_053119_001343.zip
orachk is sanitizing
/scratch/testuser/may31/../orachk_node061919_053119_001343.zip. Please wait...
Sanitized collection is:
/scratch/testuser/may31/orachk_aydv061919_053119_001343.zip
Example 14-11 Sanitizing Oracle Autonomous Health Framework Debug Log
orachk -sanitize new/orachk_debug_053119_023653.log
orachk is sanitizing /scratch/testuser/may31/new/orachk_debug_053119_023653.log.
Please wait...
Sanitized collection is: /scratch/testuser/may31/orachk_debug_053119_023653.log
Example 14-12 Running Full Sanity Check
orachk -localonly -profile asm -sanitize -silentforce
Detailed report (html) -
/scratch/testuser/may31/orachk_node061919_053119_04448/orachk_node061919_053119_04448.html
orachk is sanitizing /scratch/testuser/may31/orachk_node061919_053119_04448.
Please wait...
Sanitized collection is: /scratch/testuser/may31/orachk_aydv061919_053119_04448
UPLOAD [if required] - /scratch/testuser/may31/orachk_node061919_053119_04448.zip
orachk -rmap all|comma_delimited_list_of_element_IDs
You can also use orachk -rmap
to lookup a value sanitized by Oracle
Trace File Analyzer.
Example 14-13 Printing the Reverse Map of Sanitized Elements
orachk -rmap MF_NK1,fcb63u2
________________________________________________________________________________
| Entity Type | Substituted Entity Name | Original Entity Name |
________________________________________________________________________________
| dbname | MF_NK1 | HR_DB1 |
| dbname | fcb63u2 | rac12c2 |
________________________________________________________________________________
orachk -rmap all
Running the Disk Diagnostic Tool
Use the Disk Diagnostic Tool to help identify the cause of disk problems.
The tool produces a list of 14 disk checks for each node. To display details, where n represents the disk resource name, enter the following command:
# odaadmcli stordiag n
# odaadmcli stordiag pd_00
Parent topic: Troubleshooting Oracle Database Appliance
Running the Oracle Database Appliance Hardware Monitoring Tool
The Oracle Database Appliance Hardware Monitoring Tool displays the status of different hardware components in Oracle Database Appliance server.
The tool is implemented with the Trace File Analyzer collector. Use the tool both on bare-metal and on virtualized systems. The Oracle Database Appliance Hardware Monitoring Tool reports information only for the node on which you run the command. The information it displays in the output depend on the component that you select to review.
Bare Metal Platform
You can see the list of monitored components by running the command odaadmcli show -h
To see information about specific components, use the command syntax odaadmcli show component
, where component
is the hardware component that you want to query. For example, the command odaadmcli show power
shows information specifically about the Oracle Database Appliance power supply:
# odaadmcli show power
NAME HEALTH HEALTH_DETAILS PART_NO. SERIAL_NO.
Power_Supply_0 OK - 7079395 476856Z+1514CE056G
(Continued)
LOCATION INPUT_POWER OUTPUT_POWER INLET_TEMP EXHAUST_TEMP
PS0 Present 112 watts 28.000 degree C 34.938 degree C
Virtualized Platform
You can see the list of monitored components by running the command oakcli show -h
To see information about specific components, use the command syntax oakcli show component
, where component
is the hardware component that you want to query. For example, the command oakcli show power
shows information specifically about the Oracle Database Appliance power supply:
# oakcli show power
NAME HEALTH HEALTH_DETAILS PART_NO. SERIAL_NO.
Power Supply_0 OK - 7047410 476856F+1242CE0020
Power Supply_1 OK - 7047410 476856F+1242CE004J
(Continued)
LOCATION INPUT_POWER OUTPUT_POWER INLET_TEMP EXHAUST_TEMP
PS0 Present 88 watts 31.250 degree C 34.188 degree C
PS1 Present 66 watts 31.250 degree C 34.188 degree C
Note:
Oracle Database Appliance Server Hardware Monitoring Tool is enabled during initial startup of ODA_BASE on Oracle Database Appliance Virtualized Platform. When it starts, the tool collects base statistics for about 5 minutes. During this time, the tool displays the message "Gathering Statistics…" message.
Parent topic: Troubleshooting Oracle Database Appliance
Configuring a Trusted SSL Certificate for Oracle Database Appliance
The Browser User Interface and DCS Controller use SSL-based HTTPS protocol for secure communication. Understand the implications of this added security and the options to configure SSL certificates.
The Browser User Interface provides an added layer of security using certificates and encryption, when an administrator interacts with the appliance. Encryption of data ensures that:
- Data is sent to the intended recipient, and not to any malicious third-party.
- When data is exchanged between the server and the browser, data interception cannot occur nor can the data be edited.
When you connect to the Browser User Interface through HTTPS, the DCS Controller presents your browser with a certificate to verify the identity of appliance. If the web browser finds that the certificate is not from a trusted Certificate Authority (CA), then the browser assumes it has encountered an untrusted source, and generates a security alert message. The security alert dialog boxes display because Browser User Interface security is enabled through HTTPS and SSL, but you have not secured your Web tier properly with a trusted matching certificate from a Certificate Authority. It is possible to purchase commercial certificates from a Certificate Authority or create your own and register them with a Certificate Authority. However, the server and browser certificates must use the same public certificate key and trusted certificate to avoid the error message produced by the browser.
There are three options to configure your certificates:
- Create your own key and Java keystore, ensure it is signed by a Certificate Authority (CA) and then import it for use.
- Package an existing Privacy Enhanced Mail (PEM) format key and certificates in a new Java keystore.
- Convert an existing PKCS or PFX keystore to a Java keystore
and configure it for the Browser User Interface.
Note:
For Oracle Database Appliance High-Availability hardware models, run the configuration steps on both nodes.
The following topics explain how to configure these options:
- Option 1: Creating a Key and Java Keystore and Importing a Trusted Certificate
Usekeytool
, a key and certificate management utility, to create a keystore and a signing request. - Option 2: Packaging an Existing PEM-format Key and Certificates in a New Java Keystore
Use the OpenSSL tool to package Privacy Enhanced Mail (PEM) files in a PKCS keystore. - Option 3: Converting an Existing PKCS or PFX Keystore to a Java Keystore
If you have an existing PKCS or PFX keystore for your server's domain, convert it to a Java keystore. - Configuring the DCS Server to Use Custom Keystore
After packaging or converting your keystore into Java keystore, configure the DCS server to use your keystore. - Configuring the DCS Agent for Custom Certificate
After you import the certificate into the keystore, configure the DCS agent to use the same certificate.
Parent topic: Troubleshooting Oracle Database Appliance
Option 1: Creating a Key and Java Keystore and Importing a Trusted Certificate
Use keytool
, a key and certificate management utility, to
create a keystore and a signing request.
Option 2: Packaging an Existing PEM-format Key and Certificates in a New Java Keystore
Use the OpenSSL tool to package Privacy Enhanced Mail (PEM) files in a PKCS keystore.
Option 3: Converting an Existing PKCS or PFX Keystore to a Java Keystore
If you have an existing PKCS or PFX keystore for your server's domain, convert it to a Java keystore.
Configuring the DCS Server to Use Custom Keystore
After packaging or converting your keystore into Java keystore, configure the DCS server to use your keystore.
Disabling the Browser User Interface
You can also disable the Browser User Interface. Disabling the Browser User Interface means you can only manage your appliance through the command-line interface.
Parent topic: Troubleshooting Oracle Database Appliance
Preparing Log Files for Oracle Support Services
If you have a system fault that requires help from Oracle Support Services, then you may need to provide log records to help Oracle support diagnose your issue.
- Use the Bill Of Materials report saved in the
/opt/oracle/dcs/Inventory/
directory, to enable Oracle Support to help troubleshoot errors, if necessary. - You can use Trace File Collector (the
tfactl
command) to collect all log files for the Oracle Database Appliance components. - Use the command
odaadmcli manage diagcollect
to collect diagnostic files to send to Oracle Support Services.
The odaadmcli manage diagcollect
command consolidates information from log files stored on Oracle Database Appliance
into a single log file for use by Oracle Support Services. The location of the file
is specified in the command output.
Example 14-14 Collecting log file information for a time period, masking sensitive data
# odaadmcli manage diagcollect --dataMask --fromTime 2019-08-12 --toTime 2019-08-25
DataMask is set as true
FromTime is set as: 2019-08-12
ToTime is set as: 2019-08-25
TFACTL command is: /opt/oracle/tfa/tfa_home/bin/tfactl
Data mask is set.
Collect data from 2019-08-12
Collect data to 2019-08-25
Parent topic: Troubleshooting Oracle Database Appliance