1. Oracle Autonomous Health Framework User's Guide
  2. Using Automatic Diagnostic Collections

Using Automatic Diagnostic Collections

Oracle Trace File Analyzer monitors your logs for significant problems, such as internal errors like ORA-00600, or node evictions.

Collecting Diagnostics Automatically

This section explains how Oracle Trace File Analyzer (TFA) automatically detects problems and collects diagnostic information.

When TFA detects a significant problem, it performs the following actions:

  • Runs the required diagnostic tools and collects relevant log data from the time of the problem.
  • Trims log files to collect only the information needed for diagnosis.
  • Collects and packages the diagnostic data from all nodes in the cluster and consolidates the collection on a single node.
  • Stores diagnostic collections in the TFA repository.
  • Sends an email notification to designated recipients when the diagnostic collection is ready for upload to Oracle Support.

TFA includes a mechanism to prevent repeated errors from generating excessive automatic diagnostic collections.

When TFA detects an event, it establishes the start point for the collection and waits five minutes before collecting diagnostic data. This delay allows TFA to capture additional related events in a single collection.

If events continue to occur after five minutes, TFA continues to wait for additional events. It waits for up to 30 seconds with no new events, for a maximum additional wait period of five minutes.

If events continue after the total ten-minute period, TFA proceeds with diagnostic collection.

After the collection completes, TFA sends an email notification containing the collection location to the designated recipients.

If the environment can connect to oracle.com, you can also use TFA to upload the collection to a Service Request (SR).

Automatic diagnostic collection is enabled by default.

To enable or disable automatic diagnostic collection:

tfactl set autodiagcollect=ON|OFF

Log entries that trigger automatic collection

The following log patterns can trigger automatic diagnostic collection:

Log Entry PatternLog Monitored
ORA-297(01|02|03|08|09|10|40)Alert Log - Oracle Database
ORA-00600Alert Log - Oracle Database
ORA-07445Alert Log - Oracle Database / Oracle ASM
ORA-04(69|([7-8][0-9]|9([0-3]|[5-8])))Alert Log - Oracle Database / Oracle ASM Proxy
ORA-32701Alert Log - Oracle Database
ORA-00494Alert Log - Oracle Database / Oracle ASM
ORA-04020Alert Log - Oracle Database
ORA-04021Alert Log - Oracle Database
ORA-01578Alert Log - Oracle Database
ORA-00700Alert Log - Oracle Database
System State dumpedAlert Log - Oracle Database
CRS-016(07|10|11|12)Alert Log - Oracle Clusterware

In addition, when Oracle Cluster Health Advisor detects a problem event, TFA can automatically trigger the corresponding diagnostic collection.

Configuring Email Notification Details

Configure TFA to send an email notification to a registered email address after an automatic diagnostic collection completes.

Email notification must be configured on the system where TFA is running.

To configure email notifications:

  1. Set the notification email address for a specific ORACLE_HOME by specifying the operating system owner and email address:

    tfactl set notificationAddress=<os_user>:<email>

    For example:

    tfactl set notificationAddress=oracle:some.body@example.com

  2. Set the notification email address for any ORACLE_HOME:

    tfactl set notificationAddress=<email>

    For example:

    tfactl set notificationAddress=another.body@example.com

  3. Configure the SMTP server using:

    tfactl set smtp

    Specify the SMTP parameters when prompted.

    ParameterDescription
    smtp.hostSpecifies the SMTP server host name.
    smtp.portSpecifies the SMTP server port.
    smtp.userSpecifies the SMTP user.
    smtp.passwordSpecifies the password for the SMTP user.
    smtp.authSets the authentication flag to true or false.
    smtp.sslSets the SSL flag to true or false.
    smtp.fromSpecifies the sender email address.
    smtp.toSpecifies a comma-delimited list of recipient email addresses.
    smtp.ccSpecifies a comma-delimited list of CC email addresses.
    smtp.bccSpecifies a comma-delimited list of BCC email addresses.
    smtp.debugSets the debug flag to true or false.

    Note:

    Use the following command to view the current SMTP configuration:

    tfactl print smtp
  4. Verify the SMTP configuration by sending a test email: 
    tfactl sendmail <email_address>

    When TFA detects a significant error, it sends an email notification containing information about the diagnostic collection.

  5. After receiving the notification email:

    1. Review the referenced collection information to investigate the root cause.
    2. If you identify the cause and can correct it, resolve the underlying problem.
    3. If you cannot determine the root cause, create a Service Request with Oracle Support and upload the diagnostic collection.

Collecting Problems Detected by Oracle Cluster Health Advisor

Configure Oracle Cluster Health Advisor to automatically collect diagnostics when it detects abnormal events and to send email notifications through TFA.

  1. Enable automatic diagnostic collection for Oracle Cluster Health Advisor events:

    tfactl set chaautocollect=ON

  2. Enable Oracle Cluster Health Advisor notifications through TFA:

    tfactl set chanotification=on

  3. Configure the email address for Oracle Cluster Health Advisor notifications:

    tfactl set notificationAddress=chatfa:john.doe@acompany.com

Sanitizing Sensitive Information in TFA Collections

After collecting diagnostic data, TFA uses Adaptive Classification and Redaction (ACR) to sanitize sensitive information in collections.

Note:

Starting with Oracle Autonomous Health Framework 24.1, the TFA masking feature is deprecated and may be desupported in a future release.

Use the following command to configure how sensitive information is handled:

tfactl set redact=mask|sanitize|none

The available options are:

  • mask - Masks sensitive information in all collections. For example, myhost1 can be replaced with ******.
  • sanitize - Replaces sensitive information in all collections with randomly generated characters. For example, myhost1 can be replaced with orzhmv1.
  • none - Does not mask or sanitize sensitive information. This is the default.

You can also use the -mask and -sanitize options with tfactl diagcollect to process a specific collection.

To mask sensitive information in all collections:

tfactl set redact=mask

To sanitize sensitive information in all collections:

tfactl set redact=sanitize

To mask sensitive information in a specific collection:

tfactl diagcollect -SRDC ORA-00600 -mask

To sanitize sensitive information in a specific collection:

tfactl diagcollect -SRDC ORA-00600 -sanitize

Flood Control for Similar Issues

Flood control reduces resource consumption by limiting repeated diagnostic collections for similar events.

You can use flood control to:

  • Enable or disable flood control.
  • Specify the number of times TFA can collect diagnostics for an event.
  • Configure the time period used to evaluate repeated events.
  • Configure how long TFA pauses collection after the limit is reached.
  • View, update, and clear flood control information.

Flood control data is stored in the Berkeley Database (BDB) and persists across TFA restarts.

Check whether flood control is enabled

tfactl get floodcontrol

Example output:

.----------------------------------------.
|               testhost                 |
+--------------------------------+-------+
| Configuration Parameter        | Value |
+--------------------------------+-------+
| Flood Control ( floodcontrol ) | ON    |
'--------------------------------+-------'

Check the flood control limit

tfactl get fc.limit

Example output:

.------------------------------------------------.
|                   testhost                     |
+----------------------------------------+-------+
| Configuration Parameter                | Value |
+----------------------------------------+-------+
| Flood Control Limit Count ( fc.limit ) | 3     |
'----------------------------------------+-------'

Check the flood control limit time

tfactl get fc.limittime

Example output:

.-------------------------------------------------------------.
|                          testhost                           |
+-----------------------------------------------------+-------+
| Configuration Parameter                             | Value |
+-----------------------------------------------------+-------+
| Flood Control Limit Time (minutes) ( fc.limitTime ) | 60    |
'-----------------------------------------------------+-------'

Check the flood control pause time

tfactl get fc.pausetime

Example output:

.-------------------------------------------------------------.
|                          testhost                           |
+-----------------------------------------------------+-------+
| Configuration Parameter                             | Value |
+-----------------------------------------------------+-------+
| Flood Control Pause Time (minutes) ( fc.pauseTime ) | 120   |
'-----------------------------------------------------+-------'

Display flood control details

tfactl floodcontrol print

Example output:

.----------------------------------------------------------------------------------------------------------------------------------------------------------.
| Event                  | Count | Start Date                   | Last Date                    | Limit | Limit Time | Pause Time | Coll Count | Skip Count |
+------------------------+-------+------------------------------+------------------------------+-------+------------+------------+------------+------------+
| orcl:ORA-00600:user1   |     1 | Thu May 21 09:18:56 UTC 2020 | Thu May 21 09:18:56 UTC 2020 |     3 |         60 |        120 |          1 |          0 |
+------------------------+-------+------------------------------+------------------------------+-------+------------+------------+------------+------------+
| orcl:ORA-00600:user2   |     1 | Thu May 21 09:18:25 UTC 2020 | Thu May 21 09:18:25 UTC 2020 |     3 |         60 |        120 |          4 |          2 |
'------------------------+-------+------------------------------+------------------------------+-------+------------+------------+------------+------------'

Clear flood control information for an event

tfactl floodcontrol clear -event orcl:ORA-00600:user1

Example output:

Successfully cleared Event orcl:ORA-00600:user1

Verify that the event has been cleared:

tfactl floodcontrol print

Example output:

.---------------------------------------------------------------------------------------------------------------------.
| Event                  | Count | Start Date | Last Date | Limit | Limit Time | Pause Time | Coll Count | Skip Count |
+------------------------+-------+------------+-----------+-------+------------+------------+------------+------------+
| orcl:ORA-00600:user1   |     0 | null       | null      |     3 |         60 |        120 |          3 |          2 |
'------------------------+-------+------------+-----------+-------+------------+------------+------------+------------'

Update flood control settings for an event

tfactl floodcontrol update \
  -event orcl:ORA-00600:user1 \
  -limit 10 \
  -limittime 90 \
  -pausetime 180

Example output:

Successfully updated Flood Control Event

Verify the updated settings:

tfactl floodcontrol print -event orcl:ORA-00600:user1

Example output:

.----------------------------------------------------------------------------------------------------------------------------------------------------------.
| Event                  | Count | Start Date                   | Last Date                    | Limit | Limit Time | Pause Time | Coll Count | Skip Count |
+------------------------+-------+------------------------------+------------------------------+-------+------------+------------+------------+------------+
| orcl:ORA-00600:user1   |     1 | Thu May 21 09:18:25 UTC 2020 | Thu May 21 09:18:25 UTC 2020 |    10 |         90 |        180 |          4 |          2 |
'------------------------+-------+------------------------------+------------------------------+-------+------------+------------+------------+------------'