17 Troubleshooting Oracle Database Appliance

Understand tools you can use to validate changes and troubleshoot Oracle Database Appliance problems.

Resolving Errors When Updating DCS Components During Patching

Understand how to troubleshoot errors when updating DCS components during patching.

.

About DCS Components

When you run the odacli update-dcscomponents command during patching, pre-checks for MySQL installation are automatically verified before update of Zookeeper, MySQL, and DCS components. If any of the pre-checks fail, then the command errors out with a reference to the pre-check report log file location /opt/oracle/dcs/log/jobId-dcscomponentsPreCheckReport.log. Review the pre-check report and take corrective actions and then rerun the odacli update-dcscomponents command. If there are no pre-check errors, then the patching process proceeds with updating Zookeeper, MySQL, and DCS components such as the DCS Agent, DCS CLI, and DCS Controller.

Note:

Run the odacli update-dcsadmin command prior to running the odacli update-dcscomponents command.

When the odacli update-dcscomponents command completes successfully:

The command output is as follows:

# ./odacli update-dcscomponents -v 19.11.0.0.0            
{
  "jobId" : "3ac3667a-fa22-40b6-a832-504a56aa3fdc",
  "status" : "Success",
  "message" : "Update-dcscomponents is successful on all the node(s):DCS-Agent
shutdown is successful. MySQL upgrade is done before. Metadata migration is
successful. Agent rpm upgrade is successful. DCS-CLI rpm upgrade is successful.
DCS-Controller rpm upgrade is succ",
  "reports" : null,
  "createTimestamp" : "February 22, 2021 02:37:37 AM CST",
  "description" : "Update-dcscomponents job completed and is not part of Agent
job list",
  "updatedTime" : "February 22, 2021 02:39:10 AM CST"
}

The pre-check report log file at the location /opt/oracle/dcs/log/jobId-dcscomponentsPreCheckReport.log contains the following:

Pre-check Name: Space check
Status: Success
Comments: Required space 3 GB is available in /opt

Pre-check Name: Port check
Status: Success
Comments: Port 3306 is available for running ODA MySQL

Pre-check Name: ODA MySQL rpm installation dry-run check
Status: Success
Comments: ODA MySQL rpm dry-run passed

Pre-check Name: Check for the existence of MySQL connector/J library
Status: Success
Comments: ODA MySQL connector/J library found

Pre-check Name: Check for the existence of Metadata migration utility
Status: Success
Comments: Metadata migration utility found

When the odacli update-dcscomponents command fails:

On Oracle Database Appliance single-node systems, the command output is as follows:

# ./odacli update-dcscomponents -v 19.11.0.0.0            

DCS-10008:Failed to update DCScomponents: 19.10.0.0.0
Internal error while patching the DCS components :
DCS-10231:Cannot proceed. Pre-checks for update-dcscomponents failed. Refer to
/opt/oracle/dcs/log/jobId-dcscomponentsPreCheckReport.log
on node 0 for details.

On Oracle Database Appliance high-availability systems, the command output is as follows:

# ./odacli update-dcscomponents -v 19.11.0.0.0            

Internal error while patching the DCS components :
DCS-10231:Cannot proceed. Pre-checks for update-dcscomponents failed. Refer to
/opt/oracle/dcs/log/jobId-dcscomponentsPreCheckReport.log
on node 0 and /opt/oracle/dcs/log/jobId-dcscomponentsPreCheckReport.log
on node 1 for details.

The command runs all pre-checks one by one, and errors out at the end if any of the pre-checks is marked as Failed. When a pre-check fails, the error message is displayed on to the console along with the reference to pre-check report log location. The pre-check report log file is at the location /opt/oracle/dcs/log/jobId-dcscomponentsPreCheckReport.log.

Pre-check Name: Space check
Status: Failed
Comments: Available space in /opt is 2 GB but minimum required space in /opt is 3 GB 

Pre-check Name: Port check
Status: Success
Comments: Port 3306 is available for running ODA MySQL

Pre-check Name: ODA MySQL rpm installation dry-run check
Status: Success
Comments: ODA MySQL rpm dry-run passed

Pre-check Name: Check for the existence of MySQL connector/J library
Status: Success
Comments: ODA MySQL connector/J library found

Pre-check Name: Check for the existence of Metadata migration utility
Status: Success
Comments: Metadata migration utility found

When the odacli update-dcscomponents command fails due to space check error:

The pre-check report log contains the following:

Pre-check Name: Space check
Status: Failed
Comments: Available space in /opt is 2 GB but minimum required space in /opt is 3 GB 

Pre-check Name: Port check
Status: Success
Comments: Port 3306 is available for running ODA MySQL

Pre-check Name: ODA MySQL rpm installation dry-run check
Status: Success
Comments: ODA MySQL rpm dry-run passed

Pre-check Name: Check for the existence of MySQL connector/J library
Status: Success
Comments: ODA MySQL connector/J library found

Pre-check Name: Check for the existence of Metadata migration utility
Status: Success
Comments: Metadata migration utility found

When the odacli update-dcscomponents command fails due to port check error:

The pre-check report log contains the following:

Pre-check Name: Space check
Status: Success
Comments: Required space 3 GB is available in /opt

Pre-check Name: Port check
Status: Failed
Comments: No port found in the range ( 3306 to 65535 )

Pre-check Name: ODA MySQL rpm installation dry-run check
Status: Success
Comments: ODA MySQL rpm dry-run passed

Pre-check Name: Check for the existence of MySQL connector/J library
Status: Success
Comments: ODA MySQL connector/J library found

Pre-check Name: Check for the existence of Metadata migration utility
Status: Success
Comments: Metadata migration utility found

When the odacli update-dcscomponents command fails due to MySQL RPM installation dry-run check error:

The pre-check report log contains the following:

Pre-check Name: Space check
Status: Success
Comments: Required space 3 GB is available in /opt

Pre-check Name: Port check
Status: Success
Comments: Port 3306 is available for running ODA MySQL

Pre-check Name: ODA MySQL rpm installation dry-run check
Status: Failed
Comments: ODA MySQL rpm dry-run failed. Failed due to the following error :
Exception details are displayed below

Pre-check Name: Check for the existence of MySQL connector/J library
Status: Success
Comments: ODA MySQL connector/J library found

Pre-check Name: Check for the existence of Metadata migration utility
Status: Success
Comments: Metadata migration utility found

When the odacli update-dcscomponents command fails due to MySQL connector/J library check error:

The pre-check report log contains the following:

Pre-check Name: Space check
Status: Success
Comments: Required space 3 GB is available in /opt

Pre-check Name: Port check
Status: Success
Comments: Port 3306 is available for running ODA MySQL

Pre-check Name: ODA MySQL rpm installation dry-run check
Status: Success
Comments: ODA MySQL rpm dry-run passed

Pre-check Name: Check for the existence of MySQL connector/J library
Status: Failed
Comments: MySQL connector/J library does not exist. Ensure update-repository with latest serverzip bundles ran first without any issues prior to running update-dcscomponents

Pre-check Name: Check for the existence of Metadata migration utility
Status: Success
Comments: Metadata migration utility found

When the odacli update-dcscomponents command fails due to Metadata migration utility check error:

The pre-check report log contains the following:

Pre-check Name: Space check
Status: Success
Comments: Required space 3 GB is available in /opt

Pre-check Name: Port check
Status: Success
Comments: Port 3306 is available for running ODA MySQL

Pre-check Name: ODA MySQL rpm installation dry-run check
Status: Success
Comments: ODA MySQL rpm dry-run passed

Pre-check Name: Check for the existence of MySQL connector/J library
Status: Success
Comments: ODA MySQL connector/J library found

Pre-check Name: Check for the existence of Metadata migration utility
Status: Failed
Comments: Metadata migration utility does not exist. Ensure update-repository with latest serverzip bundles ran first without any issues prior to running update-dcscomponents

Viewing Details About DCS Error Messages

Understand how to view details about DCS errors for troubleshooting them.

About Viewing Information About DCS Errors

To view more details about any errors during DCS operations, use the command dcserr error_code.

# dcserr
dcserr <error code>
 
# dcserr 10001
10001, Internal_Error, "Internal error encountered: {0}."
// *Cause: An internal error occurred.
// *Action: Contact Oracle Support Services for assistance.
/
# dcserr 1001
Unknown error code

Viewing Component Information on the Appliance

View details of all the components installed on the appliance, and the RPM drift information.

Viewing the Bill of Materials in the Browser User Interface

Use the Appliance tab in the Browser User Interface to view information about your deployment and the installed components. The Advanced Information tab displays information about the following components:

  • Grid Infrastructure Version, and the home directory

  • Database Version, Home location, and Edition

  • Location and details about the databases configured

  • All patches applied to the appliance

  • Firmware Controller and Disks

  • ILOM information

  • BIOS version

  • List of RPMs

In the List of RPMs section, click Show and then click RPM Drift to view the differences between the RPMs installed on the appliance, and the RPMs shipped in the latest Oracle Database Appliance Patch Bundle Update release.

Click Download to save the components report. You can use this report to help diagnose any deployment issues.

Viewing the Bill of Materials from the Command Line

The bill of materials is also available through the command line for bare metal and virtualized platforms deployments. The information about the installed components is collected according to a set schedule, and stored in the location /opt/oracle/dcs/Inventory/ for bare metal deployments and in the /opt/oracle/oak/Inventory/ directory for virtualized platforms. The file is stored in the format oda_bom_TimeStamp.json. Use the command describe-system to view the bill of materials on the command line. See the Oracle Database Command-Line Interface chapter for command options and usage notes.

Example 17-1 Example Command to View the Bill of Materials from the Command Line for Bare Metal Deployments

# odacli describe-system -b
ODA Components Information 
------------------------------
Component Name                Component Details                                            
---------------               ----------------------------------------------------------------------------------------------- 
NODE                          Name : oda1 
                              Domain Name : testdomain.com 
                              Time Stamp : April 21, 2020 6:21:15 AM UTC 

  
RPMS                          Installed RPMS : abrt-2.1.11-55.0.1.el7.x86_64,
                                               abrt-addon-ccpp-2.1.11-55.0.1.el7.x86_64,
                                               abrt-addon-kerneloops-2.1.11-55.0.1.el7.x86_64,
                                               abrt-addon-pstoreoops-2.1.11-55.0.1.el7.x86_64,
                                               abrt-addon-python-2.1.11-55.0.1.el7.x86_64,
                                               abrt-addon-vmcore-2.1.11-55.0.1.el7.x86_64,
                                               abrt-addon-xorg-2.1.11-55.0.1.el7.x86_64,
                                               abrt-cli-2.1.11-55.0.1.el7.x86_64,
                                               abrt-console-notification-2.1.11-55.0.1.el7.x86_64,
                                               abrt-dbus-2.1.11-55.0.1.el7.x86_64,
                                               abrt-libs-2.1.11-55.0.1.el7.x86_64,
                                               abrt-python-2.1.11-55.0.1.el7.x86_64,
                                               abrt-tui-2.1.11-55.0.1.el7.x86_64,
                                               acl-2.2.51-14.el7.x86_64,
                                               adwaita-cursor-theme-3.28.0-1.el7.noarch,
                                               adwaita-icon-theme-3.28.0-1.el7.noarch,
                                               aic94xx-firmware-30-6.el7.noarch,
                                               aide-0.15.1-13.0.1.el7.x86_64,
                                               alsa-firmware-1.0.28-2.el7.noarch,
                                               alsa-lib-1.1.8-1.el7.x86_64,
                                               alsa-tools-firmware-1.1.0-1.el7.x86_64,
                                               at-3.1.13-24.el7.x86_64,
                                               at-spi2-atk-2.26.2-1.el7.x86_64,
                                               at-spi2-core-2.28.0-1.el7.x86_64,
                                               atk-2.28.1-1.el7.x86_64,
                                               attr-2.4.46-13.el7.x86_64,
                                               audit-2.8.5-4.el7.x86_64,
                                               audit-libs-2.8.5-4.el7.x86_64,
                                               audit-libs-python-2.8.5-4.el7.x86_64,
                                               augeas-libs-1.4.0-9.el7.x86_64,
                                               authconfig-6.2.8-30.el7.x86_64,
                                               autogen-libopts-5.18-5.el7.x86_64,
                                               avahi-libs-0.6.31-19.el7.x86_64,
                                               basesystem-10.0-7.0.1.el7.noarch,
                                               bash-4.2.46-33.el7.x86_64,
                                               bash-completion-2.1-6.el7.noarch,
                                               bc-1.06.95-13.el7.x86_64,
                                               bind-export-libs-9.11.4-9.P2.el7.x86_64,
                                               bind-libs-9.11.4-9.P2.el7.x86_64,
                                               bind-libs-lite-9.11.4-9.P2.el7.x86_64,
                                               bind-license-9.11.4-9.P2.el7.noarch,
                                               bind-utils-9.11.4-9.P2.el7.x86_64,
                                               binutils-2.27-41.base.0.7.el7_7.2.x86_64,
                                               biosdevname-0.7.3-2.el7.x86_64,
                                               blktrace-1.0.5-9.el7.x86_64,
                                               bnxtnvm-1.40.10-1.x86_64,
                                               boost-date-time-1.53.0-27.el7.x86_64,
                                               boost-filesystem-1.53.0-27.el7.x86_64,
                                               boost-iostreams-1.53.0-27.el7.x86_64,
....
....
....

Example 17-2 Example Command to View the Bill of Materials from the Command Line for Virtualized Platforms

# oakcli describe-system -b

Example 17-3 Example Command to View the Bill of Materials Report from the Stored Location

# ls -la /opt/oracle/dcs/Inventory/
total 264
-rw-r--r-- 1 root root 83550 Apr 26 05:41 oda_bom_2018-04-26_05-41-36.json

Errors When Logging into the Browser User Interface

If you have problems logging into the Browser User Interface, then it may be due to your browser or credentials.

Note:

Oracle Database Appliance uses self-signed certificates. Your browser determines how you log into the Browser User Interface. Depending on the browser and browser version, you may receive a warning or error that the certificate is invalid or not trusted because it is self-signed, or that the connection is not private. Ensure that you accept the self-signed certificate for the agent and Browser User Interface.

Follow these steps to log into the Browser User Interface:

  1. Open a browser window.
  2. Go to the following URL: https://ODA-host-ip-address:7093/mgmt/index.html
  3. Get the security certificate (or certificate), confirm the security exception, and add an exception.
  4. Log in with your Oracle Database Appliance credentials.
    If you have not already set the oda-admin password, then a message is displayed, advising you to change the default password to comply with your system security requirements.
  5. If you have not added an exception for the agent security certificate, then a message about accepting agent certificate is displayed.
  6. Using a different tab in your browser, go to the following URL: https://ODA-host-ip-address:7070/login
  7. Get the security certificate (or certificate), confirm the security exception, and add an exception.
  8. Refresh the Browser User Interface URL : https://ODA-host-ip-address:7093/mgmt/index.html

Note:

If you have any issues logging into the Oracle Database Appliance Browser User Interface on browsers such as macOS Catalina and Google Chrome, then you may need to use any workaround as described on the official site for the product.

Errors when re-imaging Oracle Database Appliance

Understand how to troubleshoot errors that occur when re-imaging Oracle Database Appliance.

If re-imaging Oracle Database Appliance fails, with old header issues such as errors in storage discovery, or in running GI root scripts, or disk group RECO creation, then use the force mode with cleanup.pl.

# cleanup.pl -f

To ensure that re-imaging is successful, remove the old headers from the storage disks by running the secure erase tool. Verify that the OAK/ASM headers are removed.

# cleanup.pl -erasedata
# cleanup.pl -checkHeader

Retry the re-imaging operation.

Connecting to Oracle Database Appliance from Oracle Linux 6 Using SSH or SCP

For Oracle Database Appliance release 19.11, you must specify the key exchange (kex) algorithm to use the SSH or SCP client to connect to Oracle Database Appliance nodes on Oracle Linux 6.

For Oracle Linux 7 clients, you can use the SSH client to connect to Oracle Database Appliance nodes.

For Oracle Linux 6, you must use one of the following methods to specify the key exchange (kex) algorithm and use the SSH client to connect to Oracle Database Appliance nodes.
  • Specify kex algorithm when running SSH or SCP:
    $ ssh  -o KexAlgorithms=diffie-hellman-group14-sha1  oda1
    oracle@oda1's password:
     [oracle@oda1 ~]#
  • Add kex algorithm to the ~/.ssh/config file:
    # cat ~/.ssh/config
       Host *
                KexAlgorithms diffie-hellman-group14-sha1

Using Oracle Autonomous Health Framework for Running Diagnostics

Oracle Autonomous Health Framework collects and analyzes diagnostic data collected, and proactively identifies issues before they affect the health of your system.

About Installing Oracle Autonomous Health Framework

Oracle Autonomous Health Framework is installed automatically when you provision or patch to Oracle Database Appliance release 19.11.

When you provision or patch your appliance to Oracle Database Appliance release 19.11, Oracle Autonomous Health Framework is installed in the path /opt/oracle/dcs/oracle.ahf.

You can verify that Oracle Autonomous Health Framework is installed by running the following command:
[root@oak ~]# rpm -q oracle-ahf
oracle-ahf-193000-########.x86_64

Note:

When you provision or patch to Oracle Database Appliance release 19.11, Oracle Autonomous Health Framework automatically provides Oracle ORAchk Health Check Tool and Oracle Trace File Analyzer Collector.
Oracle ORAchk Health Check Tool performs proactive health checks for the Oracle software stack and scans for known problems. Oracle ORAchk Health Check Tool audits important configuration settings for Oracle RAC deployments in the following categories:
  • Operating system kernel parameters and packages
  • Oracle Database Database parameters, and other database configuration settings
  • Oracle Grid Infrastructure, which includes Oracle Clusterware and Oracle Automatic Storage Management
Oracle ORAchk is aware of the entire system. It checks the configuration to indicate if best practices are being followed.
Oracle Trace File Collector provides the following key benefits and options:
  • Encapsulation of diagnostic data collection for all Oracle Grid Infrastructure and Oracle RAC components on all cluster nodes into a single command, which you run from a single node
  • Option to "trim" diagnostic files during data collection to reduce data upload size
  • Options to isolate diagnostic data collection to a given time period, and to a particular product component, such as Oracle ASM, Oracle Database, or Oracle Clusterware
  • Centralization of collected diagnostic output to a single node in Oracle Database Appliance, if desired
  • On-Demand Scans of all log and trace files for conditions indicating a problem
  • Real-Time Scan Alert Logs for conditions indicating a problem (for example, Database Alert Logs, Oracle ASM Alert Logs, and Oracle Clusterware Alert Logs)

Using the Oracle ORAchk Health Check Tool

Run Oracle ORAchk to audit configuration settings and check system health.

Note:

Before running ORAchk, check for the latest version of Oracle Autonomous Health Framework, and download and install it. See My Oracle Support Note 2550798.1 for more information about downloading and installing the latest verion of Oracle Autonomous Health Framework.

Running ORAchk on Oracle Database Appliance 19.11 Baremetal Systems for New Installation

When you provision or upgrade to Oracle Database Appliance 19.11, ORAchk is installed using Oracle Autonomous Framework in the directory /opt/oracle/dcs/oracle.ahf.

To run orachk, use the following command:
[root@oak bin]# orachk

When all checks are finished, a detailed report is available. The output displays the location of the report in an HTML format and the location of a zip file if you want to upload the report. For example, you can choose the filter to show failed checks only, show checks with a Fail, Warning, Info, or Pass status, or any combination.

Review the Oracle Database Appliance Assessment Report and system health and troubleshoot any issues that are identified. The report includes a summary and filters that enable you to focus on specific areas.

Running ORAchk on Oracle Database Appliance 19.11 Virtualized Platform

When you provision or upgrade to Oracle Database Appliance 19.11, ORAchk is installed using Oracle Autonomous Framework in the directory /opt/oracle.ahf.

To run orachk, use the following command:
[root@oak bin]# oakcli orachk

Generating and Viewing Oracle ORAchk Health Check Tool Reports in the Browser User Interface

Generate Oracle ORAchk Health Check Tool reports using the Browser User Interface.

  1. Log into the Browser User Interface with the oda-admin username and password.
    https://Node0–host-ip-address:7093/mgmt/index.html
  2. Click the Monitoring tab.
  3. In the Monitoring page, on the left navigation pane, click ORAchk Report.
    On the ORAchk Reports page, a list of all the generated ORAchk reports is displayed.
  4. In the Actions menu for the ORAchk report you want to view, click View.
    The Oracle Database Appliance Assessment Report is displayed. It contains details of the health of your deployment, and lists current risks, recommendations for action, and links for additional information.
  5. To create an on-demand ORAchk report: On the ORAchk Reports page, click Create and then click Yes in the confirmation box.
    The job to create an ORAchk report is submitted.
  6. Click the link to view the status of the job. Once the job completes successfully, you can view the Oracle Database Appliance Assessment Report on the ORAchk Reports page.
  7. To delete an ORAchk report: In the Actions menu for the ORAchk report you want to delete, click Delete.

Generating and Viewing Database Security Assessment Reports in the Browser User Interface

Generate and view Database Security Assessment Reports using the Browser User Interface.

  1. Log into the Browser User Interface with the oda-admin username and password.
    https://Node0–host-ip-address:7093/mgmt/index.html
  2. Click the Security tab.
  3. In the Security page, on the left navigation pane, click DBSAT Reports.
    On the Database Security Assessment Reports page, a list of all the generated DBSAT reports is displayed.
  4. In the Actions menu for the ORAchk report you want to view, click View.
    The Oracle Database Security Assessment Report is displayed. It contains details of the health of your deployment, and lists current risks, recommendations for action, and links for additional information.
  5. To create a DBSAT report: On the DBSAT Reports page, click Create and then click Yes in the confirmation box.
    The job to create a DBSAT report is submitted.
  6. Click the link to view the status of the job. Once the job completes successfully, you can view the Oracle Database Appliance Assessment Report on the DBSAT Reports page.
  7. To delete a DBSAT report: In the Actions menu for the DBSAT report you want to delete, click Delete.

Running Oracle Trace File Analyzer (TFA) Collector Commands

Understand the installed location of tfactl and the options for the command.

About Using tfactl to Collect Diagnostic Information

When you provision or upgrade to Oracle Database Appliance 19.11, Oracle Trace File Analyzer (TFA) Collector is installed in the directory /opt/oracle.ahf/bin/tfactl. You can invoke the command line utility for TFA, tfactl from the directory /opt/oracle.ahf/bin/tfactl, or simply type tfactl.

You can use the following command options to run tfactl:

 /opt/oracle.ahf/bin/tfactl diagcollect -ips|-oda|-odalite|-dcs|-odabackup|
-odapatching|-odadataguard|-odaprovisioning|-odaconfig|-odasystem|-odastorage|-database|
-asm|-crsclient|-dbclient|-dbwlm|-tns|-rhp|-procinfo|-afd|-crs|-cha|-wls|
-emagent|-oms|-ocm|-emplugins|-em|-acfs|-install|-cfgtools|-os|-ashhtml|-ashtext|
-awrhtml|-awrtext -mask -sanitize

Table 17-1 Command Options for tfactl Tool

Option Description
-h

(Optional) Describes all the options for this command.

-ips

(Optional) Use this option to view the diagnostic logs for the specified component.

-oda

(Optional) Use this option to view the logs for the entire Appliance.

-odalite

(Optional) Use this option to view the diagnostic logs for the odalite component.

-dcs

(Optional) Use this option to view the DCS log files.

-odabackup

(Optional) Use this option to view the diagnostic logs for the backup components for the Appliance.

-odapatching

(Optional) Use this option to view the diagnostic logs for patching components of the Appliance.

-odadataguard

(Optional) Use this option to view the diagnostic logs for Oracle Data Guard component of the Appliance.

-odaprovisioning

(Optional) Use this option to view provisioning logs for the Appliance.

-odaconfig

(Optional) Use this option to view configuration-related diagnostic logs.

-odasystem

(Optional) Use this option to view system information.

-odastorage

(Optional) Use this option to view the diagnostic logs for the Appliance storage.

-database

(Optional) Use this option to view database-related log files.

-asm

(Optional) Use this option to view the diagnostic logs for the Appliance.

-crsclient

(Optional) Use this option to view the diagnostic logs for the Appliance.

-dbclient

(Optional) Use this option to view the diagnostic logs for the Appliance.

-dbwlm

(Optional) Use this option to view the diagnostic logs for the specified component.

-tns

(Optional) Use this option to view the diagnostic logs for TNS.

-rhp

(Optional) Use this option to view the diagnostic logs for Rapid Home Provisioning.

-afd

(Optional) Use this option to view the diagnostic logs for Oracle ASM Filter Driver.

-crs

(Optional) Use this option to view the diagnostic logs for Oracle Clusterware.

-cha

(Optional) Use this option to view the diagnostic logs for the Cluster Health Monitor.

-wls

(Optional) Use this option to view the diagnostic logs for Oracle WebLogic Server.

-emagent

(Optional) Use this option to view the diagnostic logs for the Oracle Enterprise Manager agent.

-oms

(Optional) Use this option to view the diagnostic logs for the Oracle Enterprise Manager Management Service.

-ocm

(Optional) Use this option to view the diagnostic logs for the specified component.

-emplugins

(Optional) Use this option to view the diagnostic logs for Oracle Enterprise Manager plug-ins.

-em

(Optional) Use this option to view the diagnostic logs for Oracle Enterprise Manager deployment.

-acfs

(Optional) Use this option to view the diagnostic logs for Oracle ACFS storage.

-install

(Optional) Use this option to view the diagnostic logs for installation.

-cfgtools

(Optional) Use this option to view the diagnostic logs for the configuration tools.

-os

(Optional) Use this option to view the diagnostic logs for the operating system.

-ashhtml

(Optional) Use this option to view the diagnostic logs for the specified component.

-ashtext

(Optional) Use this option to view the diagnostic logs for the Appliance.

-awrhtml

(Optional) Use this option to view the diagnostic logs for the Appliance.

-awrtext

(Optional) Use this option to view the diagnostic logs for the specified component.

-mask

(Optional) Use this option to choose to mask sensitive data in the log collection.

-sanitize

(Optional) Use this option to choose to sanitize (redact) sensitive data in the log collection.

Usage Notes

You can use Trace File Collector (the tfactl command) to collect all log files for the Oracle Database Appliance components.

You can also use the command odaadmcli manage diagcollect, with similar command options, to collect the same diagnostic information.

For more information about using the -mask and -sanitize options, see the next topic.

Sanitizing Sensitive Information in Diagnostic Collections

Oracle Autonomous Health Framework uses Adaptive Classification and Redaction (ACR) to sanitize sensitive data.

After collecting copies of diagnostic data, Oracle Trace File Analyzer and Oracle ORAchk use Adaptive Classification and Redaction (ACR) to sanitize sensitive data in the collections. ACR uses a Machine Learning based engine to redact a pre-defined set of entity types in a given set of files. ACR also sanitizes or masks entities that occur in files and directory names. Sanitization replaces a sensitive value with random characters. Masking replaces a sensitive value with a series of asterisks ("*").

ACR currently sanitizes the following entity types:
  • Host names
  • IP addresses
  • MAC addresses
  • Oracle Database names
  • Tablespace names
  • Service names
  • Ports
  • Operating system user names

ACR also masks user data from the database appearing in block and redo dumps.

Example 17-4 Block dumps before redaction

14A533F40 00000000 00000000 00000000 002C0000 [..............,.] 
14A533F50 35360C02 30352E30 31322E37 380C3938 [..650.507.2189.8] 
14A533F60 31203433 37203332 2C303133 360C0200 [34 123 7310,...6] 

Example 17-5 Block dumps after redaction

14A533F40 ******** ******** ******** ******** [****************]
14A533F50 ******** ******** ******** ******** [****************]
14A533F60 ******** ******** ******** ******** [****************] 

Example 17-6 Redo dumps before redaction

col 74: [ 1] 80
col 75: [ 5] c4 0b 19 01 1f
col 76: [ 7] 78 77 06 16 0c 2f 26 

Example 17-7 Redo dumps after redaction

col 74: [ 1] **
col 75: [ 5] ** ** ** ** **
col 76: [ 7] ** ** ** ** ** ** **

Redaction of Literal Values in SQL Statements in AWR, ASH and ADDM Reports

Automatic Workload Repository (AWR), Active Session History (ASH), and Automatic Database Diagnostic Monitor (ADDM) reports are HTML files, which contain sensitive entities such as hostnames, database names, and service names in the form of HTML tables. In addition to these sensitive entities, they also contain SQL statements, that can contain bind variables or literal values from tables. These literal values can be sensitive personal information (PI) stored in databases. ACR processes such reports to identify and redact both usual sensitive entities and literal values present in the SQL statements.

Sanitizing Sensitive Information Using odaadmcli Command

Use the odaadmcli manage diagcollect command to collect diagnostic logs for Oracle Database Appliance components. During collection, ACR can be used to redact (sanitize or mask) the diagnostic logs.
odaadmcli manage diagcollect [--dataMask|--dataSanitize]

In the command, the --dataMask option blocks out the sensitive data in all collections, for example, replaces myhost1 with *******. The default is None. The --dataSanitize option replaces the sensitive data in all collections with random characters, for example, replaces myhost1 with orzhmv1. The default is None.

Enabling Adaptive Classification and Redaction (ACR)

Oracle Database Appliance supports Adaptive Classification and Redaction (ACR) to sanitize sensitive data.

After collecting copies of diagnostic data, Oracle Database Appliance use Adaptive Classification and Redaction (ACR) to sanitize sensitive data in the collections. You can use the commands odacli enable-acr and odacli disable-acr to enable or disable ACR across both nodes, not just on the local node.

Example 17-8 Describing current status of ACR

bash-4.2# odacli describe-acr
Trace File Redaction: Enabled

Example 17-9 Enabling ACR:

bash-4.2# odacli enable-acr

Job details                                                      
----------------------------------------------------------------
                ID:  12bbf784-610a-40a8-b409-e74c58bc35aa
               Description:  Enable ACR job
                Status:  Created
                Created:  April 8, 2021 3:04:13 AM PDT

Example 17-10 Disabling ACR

bash-4.2# odacli disable-acr

Job details                                                      
----------------------------------------------------------------
                ID:  1d69f8b3-3989-4192-bbb9-6518e425061a
               Description:  Disable ACR job
                Status:  Created
                Created:  April 8, 2021 3:04:13 AM PDT

Example 17-11 Enabling ACR during provisioning of the appliance

You can enable ACR during provisioning of the appliance by adding the acr option to the JSON file used for provisioning. Specify true or false for the field acrEnable in the JSON file. If the acr option is not specified, then ACR is disabled.

"acr": {
    "acrEnable": true
}

Sanitizing Sensitive Information in Oracle Trace File Analyzer Collections

You can redact (sanitize or mask) Oracle Trace File Analyzer diagnostic collections.

Enabling Automatic Redaction

To enable automatic redaction, use the command:

tfactl set redact=[mask|sanitize|none] 

In the command, the -mask option blocks out the sensitive data in all collections, for example, replaces myhost1 with *******. The -sanitize option replaces the sensitive data in all collections with random characters, for example, replaces myhost1 with orzhmv1. The none option does not mask or sanitize sensitive data in collections. The default is none.

Enabling On-Demand Redaction

You can redact collections on-demand, for example, tfactl diagcollect -srdc ORA-00600 -mask or tfactl diagcollect -srdc ORA-00600 -sanitize.

  1. To mask sensitive data in all collections:
    tfactl set redact=mask
  2. To sanitize sensitive data in all collections:
    tfactl set redact=sanitize

Example 17-12 Masking or Sanitizing Sensitive Data in a Specific Collection

tfactl diagcollect -srdc ORA-00600 -mask
tfactl diagcollect -srdc ORA-00600 -sanitize

Sanitizing Sensitive Information in Oracle ORAchk Output

You can sanitize Oracle ORAchk output.

To sanitize Oracle ORAchk output, include the -sanitize option, for example, orachk -profile asm -sanitize. You can also sanitize post process by passing in an existing log, HTML report, or a zip file, for example, orachk -sanitize file_name.

Example 17-13 Sanitizing Sensitive Information in Specific Collection IDs

orachk -sanitize comma_delimited_list_of_collection_IDs

Example 17-14 Sanitizing a File with Relative Path

orachk -sanitize new/orachk_node061919_053119_001343.zip 
orachk is sanitizing
/scratch/testuser/may31/new/orachk_node061919_053119_001343.zip. Please wait...

Sanitized collection is:
/scratch/testuser/may31/orachk_aydv061919_053119_001343.zip
orachk -sanitize ../orachk_node061919_053119_001343.zip 
orachk is sanitizing
/scratch/testuser/may31/../orachk_node061919_053119_001343.zip. Please wait...

Sanitized collection is:
/scratch/testuser/may31/orachk_aydv061919_053119_001343.zip

Example 17-15 Sanitizing Oracle Autonomous Health Framework Debug Log

orachk -sanitize new/orachk_debug_053119_023653.log
orachk is sanitizing /scratch/testuser/may31/new/orachk_debug_053119_023653.log.
Please wait...

Sanitized collection is: /scratch/testuser/may31/orachk_debug_053119_023653.log

Example 17-16 Running Full Sanity Check

orachk -localonly -profile asm -sanitize -silentforce

Detailed report (html) - 
/scratch/testuser/may31/orachk_node061919_053119_04448/orachk_node061919_053119_04448.html

orachk is sanitizing /scratch/testuser/may31/orachk_node061919_053119_04448.
Please wait...

Sanitized collection is: /scratch/testuser/may31/orachk_aydv061919_053119_04448

UPLOAD [if required] - /scratch/testuser/may31/orachk_node061919_053119_04448.zip
To reverse lookup a sanitized value, use the command:
orachk -rmap all|comma_delimited_list_of_element_IDs

You can also use orachk -rmap to lookup a value sanitized by Oracle Trace File Analyzer.

Example 17-17 Printing the Reverse Map of Sanitized Elements


orachk -rmap MF_NK1,fcb63u2

________________________________________________________________________________
| Entity Type | Substituted Entity Name | Original Entity Name |
________________________________________________________________________________
| dbname      | MF_NK1               | HR_DB1            |
| dbname      | fcb63u2              | rac12c2           |
________________________________________________________________________________
orachk -rmap all

Running the Disk Diagnostic Tool

Use the Disk Diagnostic Tool to help identify the cause of disk problems.

The tool produces a list of 14 disk checks for each node. To display details, where n represents the disk resource name, enter the following command:

# odaadmcli stordiag n
For example, to display detailed information for NVMe pd_00:
# odaadmcli stordiag pd_00

Running the Oracle Database Appliance Hardware Monitoring Tool

The Oracle Database Appliance Hardware Monitoring Tool displays the status of different hardware components in Oracle Database Appliance server.

The tool is implemented with the Trace File Analyzer collector. Use the tool both on bare-metal and on virtualized systems. The Oracle Database Appliance Hardware Monitoring Tool reports information only for the node on which you run the command. The information it displays in the output depend on the component that you select to review.

Bare Metal Platform

You can see the list of monitored components by running the command odaadmcli show -h

To see information about specific components, use the command syntax odaadmcli show component, where component is the hardware component that you want to query. For example, the command odaadmcli show power shows information specifically about the Oracle Database Appliance power supply:

# odaadmcli show power

NAME            HEALTH  HEALTH_DETAILS   PART_NO.  	SERIAL_NO.
Power_Supply_0  OK            -          7079395     476856Z+1514CE056G

(Continued)
LOCATION    INPUT_POWER   OUTPUT_POWER   INLET_TEMP         EXHAUST_TEMP
PS0         Present       112 watts      28.000 degree C    34.938 degree C

Virtualized Platform

You can see the list of monitored components by running the command oakcli show -h

To see information about specific components, use the command syntax oakcli show component, where component is the hardware component that you want to query. For example, the command oakcli show power shows information specifically about the Oracle Database Appliance power supply:

# oakcli show power

NAME            HEALTH HEALTH_DETAILS PART_NO. SERIAL_NO.          
Power Supply_0  OK      -             7047410   476856F+1242CE0020
Power Supply_1  OK     -              7047410   476856F+1242CE004J

(Continued)

LOCATION  INPUT_POWER OUTPUT_POWER INLET_TEMP         EXHAUST_TEMP
PS0       Present     88 watts     31.250 degree C    34.188 degree C
PS1       Present     66 watts     31.250 degree C    34.188 degree C

Note:

Oracle Database Appliance Server Hardware Monitoring Tool is enabled during initial startup of ODA_BASE on Oracle Database Appliance Virtualized Platform. When it starts, the tool collects base statistics for about 5 minutes. During this time, the tool displays the message "Gathering Statistics…" message.

Configuring a Trusted SSL Certificate for Oracle Database Appliance

The Browser User Interface and DCS Controller use SSL-based HTTPS protocol for secure communication. Understand the implications of this added security and the options to configure SSL certificates.

The Browser User Interface provides an added layer of security using certificates and encryption, when an administrator interacts with the appliance. Encryption of data ensures that:

  • Data is sent to the intended recipient, and not to any malicious third-party.
  • When data is exchanged between the server and the browser, data interception cannot occur nor can the data be edited.

When you connect to the Browser User Interface through HTTPS, the DCS Controller presents your browser with a certificate to verify the identity of appliance. If the web browser finds that the certificate is not from a trusted Certificate Authority (CA), then the browser assumes it has encountered an untrusted source, and generates a security alert message. The security alert dialog boxes display because Browser User Interface security is enabled through HTTPS and SSL, but you have not secured your Web tier properly with a trusted matching certificate from a Certificate Authority. It is possible to purchase commercial certificates from a Certificate Authority or create your own and register them with a Certificate Authority. However, the server and browser certificates must use the same public certificate key and trusted certificate to avoid the error message produced by the browser.

There are three options to configure your certificates:

  • Create your own key and Java keystore, ensure it is signed by a Certificate Authority (CA) and then import it for use.
  • Package an existing Privacy Enhanced Mail (PEM) format key and certificates in a new Java keystore.
  • Convert an existing PKCS or PFX keystore to a Java keystore and configure it for the Browser User Interface.

    Note:

    For Oracle Database Appliance High-Availability hardware models, run the configuration steps on both nodes.

The following topics explain how to configure these options:

Option 1: Creating a Key and Java Keystore and Importing a Trusted Certificate

Use keytool, a key and certificate management utility, to create a keystore and a signing request.

  1. Create the keystore:
    keytool -genkeypair -alias your.domain.com -storetype jks -keystore 
    your.domain.com.jks -validity 366 -keyalg RSA -keysize 4096
  2. The command prompts you for identifying data:
    1. What is your first and last name? your.domain.com
    2. What is the name of your organizational unit? yourunit
    3. What is the name of your organization? yourorg
    4. What is the name of your City or Locality? yourcity
    5. What is the name of your State or Province? yourstate
    6. What is the two-letter country code for this unit? US
  3. Create the certificate signing request (CSR):
    keytool -certreq -alias your.domain.com -file your.domain.com.csr
    -keystore your.domain.com.jks -ext san=dns:your.domain.com
  4. Request a Certificate Authority (CA) signed certificate:
    1. In the directory where you ran Step 1 above, locate the file your.domain.com.csr.
    2. Submit the file to your Certificate Authority (CA).
      Details vary from one CA to another. Typically, you submit your request through a website; then the CA contacts you to verify your identity. CAs can send signed reply files in a variety of formats, and CAs use a variety of names for those formats. The CA's reply must be in PEM or PKCS#7 format.
    3. There may be a waiting period for the CA's reply.
  5. Import the CA's reply. The CA's reply will provide one PKCS file or multiple PEM files.
    1. Copy the CA's files into the directory where you created the keystore in Step 1 above.
    2. Use keytool to export the certificate from the keystore:
      keytool -exportcert -alias your.domain.com -file /opt/oracle/dcs/conf/keystore-cert.crt 
      -keystore your.domain.name.jks
  6. Use keytool to import the keystore certificate and the CA reply files:
    keytool -importcert -trustcacerts -alias your.domain.com 
    -file /opt/oracle/dcs/conf/keystore-cert.crt  -keystore /opt/oracle/dcs/conf/dcs-ca-certs
    To import PKCS file, run the command:
    keytool -importcert -trustcacerts -alias your.domain.com -file 
    CAreply.pkcs -keystore /opt/oracle/dcs/conf/dcs-ca-certs

    CAreply.pkcs is the name of the PKCS file provided by the CA and your.domain.com is the complete domain name of your server.

    If the CA sent PEM files, then there may be one file, but most often there are two or three. Import the files to your keystore with commands in the order shown below, after substituting your values:
    keytool -importcert -alias root -file root.cert.pem -keystore /opt/oracle/dcs/conf/dcs-ca-certs -trustcacerts
    keytool -importcert -alias intermediate -file intermediate.cert.pem /opt/oracle/dcs/conf/dcs-ca-certs -trustcacerts
    keytool -importcert -alias intermediat2 -file intermediat2.cert.pem /opt/oracle/dcs/conf/dcs-ca-certs -trustcacerts
    keytool -importcert -alias your.domain.com -file server.cert.pem /opt/oracle/dcs/conf/dcs-ca-certs -trustcacerts

    root.cert.pem is the name of the root certificate file and intermediate.cert.pem is the name of the intermediate certificate file. The root and intermediate files link the CA's signature to a widely trusted root certificate that is known to web browsers. Most, but not all, CA replies include roots and intermediates. server.cert.pem is the name of the server certificate file. The file links your domain name with your public key and the CA's signature.

Option 2: Packaging an Existing PEM-format Key and Certificates in a New Java Keystore

Use the OpenSSL tool to package Privacy Enhanced Mail (PEM) files in a PKCS keystore.

If you have an existing private key and certificates for your server's domain in PEM format, importing them into a Java keystore requires the OpenSSL tool. OpenSSL can package the PEM files in a PKCS keystore. Java keytool can then convert the PKCS keystore to a Java keystore.
  1. Install OpenSSL.
  2. Copy your private key, server certificate, and intermediate certificate into one directory.
  3. Package the key and certificates into a PKCS keystore as follows:
    openssl pkcs12 -export -in server.cert.pem -inkey private.key.pem -certfile 
    intermediate.cert.pem -name "your.domain.com" -out your.domain.com.p12
    server.cert.pem is the name of the server certificate file, your.domain.com is the complete domain name of your server, private.key.pem is the private counterpart to the public key in server.cert.pem, and intermediate.cert.pem is the name of the intermediate certificate file.
    Convert the resulting PKCS keystore file, your.domain.com.p12 into a Java keystore.

Option 3: Converting an Existing PKCS or PFX Keystore to a Java Keystore

If you have an existing PKCS or PFX keystore for your server's domain, convert it to a Java keystore.

  1. Run the command:
    keytool -importkeystore -srckeystore your.domain.com.p12 -srcstoretype PKCS12 
    -destkeystore /opt/oracle/dcs/conf/dcs-ca-certs -deststoretype jks
    your.domain.com.p12 is the existing keystore file and your.domain.com is the complete domain name of your server.
  2. Configure the DCS server as explained in the topic Configuring the DCS Server to Use Custom Keystore.

Configuring the DCS Server to Use Custom Keystore

After packaging or converting your keystore into Java keystore, configure the DCS server to use your keystore.

  1. Login to the appliance.
    ssh -l root oda-host-name
  2. Generate the obfuscated keystore password:
    java -cp /opt/oracle/dcs/bin/dcs-controller-n.n.n.-SNAPSHOT.jar
     org.eclipse.jetty.util.security.Password keystore-password

    For example:

    [root@oda]# java -cp /opt/oracle/dcs/bin/dcs-controller-2.4.18-SNAPSHOT.jar 
    org.eclipse.jetty.util.security.Password test
    12:46:33.858 [main] DEBUG org.eclipse.jetty.util.log 
    - Logging to Logger[org.eclipse.jetty.util.log] via org.eclipse.jetty.util.log.Slf4jLog
    12:46:33.867 [main] INFO org.eclipse.jetty.util.log 
    - Logging initialized @239ms to org.eclipse.jetty.util.log.Slf4jLog
    test
    OBF:1z0f1vu91vv11z0f
    MD5:098f6bcd4621d373cade4e832627b4f6
    [root@scaoda7s001 conf]#

    Copy the password that starts with OBF:.

  3. Update the DCS controller configuration file.
    cd /opt/oracle/dcs/conf

    Update the following parameters in dcs-controller.json:

    "keyStorePath": "keystore-directory-path/your.domain.com.jks"      
     "trustStorePath": /opt/oracle/dcs/conf/dcs-ca-certs
     "keyStorePassword": "obfuscated keystorepassword"
     "certAlias": "your.domain.com"
  4. Restart the DCS Controller.
    systemctl stop initdcscontroller
    systemctl start initdcscontroller
  5. Access the Browser User Interface at https://oda-host-name:7093/mgmt/index.html.

Configuring the DCS Agent for Custom Certificate

After you import the certificate into the keystore, configure the DCS agent to use the same certificate.

  1. Update the DCS agent configuration file:
    cd /opt/oracle/dcs/conf
    Update the following parameters in the dcs-agent.json file:
    "keyStorePath": "keystore-directory-path/your.domain.com.jks"      
     "trustStorePath": /opt/oracle/dcs/conf/dcs-ca-certs
     "keyStorePassword": "obfuscated keystorepassword"
     "certAlias": "your.domain.com"
  2. Restart the DCS agent:
    systemctl stop initdcsagent
    systemctl start initdcsagent
  3. Access the agent at https://oda-host-name:7070.
  4. Update the CLI certificates.
    cp -f /opt/oracle/dcs/conf/dcs-ca-certs 
    /opt/oracle/dcs/dcscli/dcs-ca-certs
  5. Update the DCS command-line configuration files:
    [root@]# cd /opt/oracle/dcs/dcscli
    Update the following parameters in dcscli-adm.conf and dcscli.conf:
    TrustStorePath=/opt/oracle/dcs/conf/dcs-ca-certs
    TrustStorePassword=keystore_password

Disabling the Browser User Interface

You can also disable the Browser User Interface. Disabling the Browser User Interface means you can only manage your appliance through the command-line interface.

  1. Log in to the appliance:
    ssh -l root oda-host-name
  2. Stop the DCS controller. For High-Availability systems, run the command on both nodes.
    systemctl stop initdcscontroller

Preparing Log Files for Oracle Support Services

If you have a system fault that requires help from Oracle Support Services, then you may need to provide log records to help Oracle support diagnose your issue.

You can collect diagnostic information for your appliance in the following ways:
  • Use the Bill Of Materials report saved in the /opt/oracle/dcs/Inventory/ directory, to enable Oracle Support to help troubleshoot errors, if necessary.
  • You can use Trace File Collector (the tfactl command) to collect all log files for the Oracle Database Appliance components.
  • Use the command odaadmcli manage diagcollect to collect diagnostic files to send to Oracle Support Services.

The odaadmcli manage diagcollect command consolidates information from log files stored on Oracle Database Appliance into a single log file for use by Oracle Support Services. The location of the file is specified in the command output.

Example 17-18 Collecting log file information for a time period, masking sensitive data

# odaadmcli manage diagcollect --dataMask --fromTime 2019-08-12 --toTime 2019-08-25
DataMask is set as true
FromTime is set as: 2019-08-12
ToTime is set as: 2019-08-25
TFACTL command is: /opt/oracle/tfa/tfa_home/bin/tfactl
Data mask is set.
Collect data from 2019-08-12
Collect data to 2019-08-25