1. User’s Guide
  2. Oracle Trace File Analyzer Command-Line and Shell Options
  3. Running Summary and Analysis Commands

F.2 Running Summary and Analysis Commands

Use these commands to view the summary of deployment and status of Oracle Trace File Analyzer, and changes and events detected by Oracle Trace File Analyzer.

Topics:

  • tfactl summary
    Use the tfactl summary command to view the summary of Oracle Trace File Analyzer deployment.
  • tfactl changes
    Use the tfactl changes command to view the changes detected by Oracle Trace File Analyzer.
  • tfactl events
    Use the tfactl events command to view the events detected by Oracle Trace File Analyzer.
  • tfactl analyze
    Use the tfactl analyze command to obtain analysis of your system by parsing the database, Oracle ASM, and Oracle Grid Infrastructure alert logs, system message logs, OSWatcher Top, and OSWatcher Slabinfo files.
  • tfactl run
    Use the tfactl run  command to run the requested action (can be inventory or scan or any support tool).
  • tfactl toolstatus
    Use the tfactl toolstatus  command to view the status of Oracle Trace File Analyzer Support Tools across all nodes.

F.2.1 tfactl summary

Use the tfactl summary command to view the summary of Oracle Trace File Analyzer deployment.

Syntax

tfactl summary

Example

$ tfactl summary
Output from host : myserver69
------------------------------
=====
Nodes
=====
myserver69
myserver70
myserver71

=====
Homes
=====
.-------------------------------------------------------------------------------------------------------------------------.
| Home                                           | Type | Version    | Database          | Instance             | Patches |
+------------------------------------------------+------+------------+-------------------+----------------------+---------+
| /scratch/app/11.2.0.4/grid                     | GI   | 11.2.0.4.0 |                   |                      |         |
| /scratch/app/oradb/product/11.2.0/dbhome_11204 | DB   | 11.2.0.4.0 | apxcmupg,rdb11204 | apxcmupg_1,rdb112041 |         |
'------------------------------------------------+------+------------+-------------------+----------------------+---------'


Output from host : myserver70
------------------------------
=====
Homes
=====

.--------------------------------------------------------------------------------------------------------------.
| Home                                           | Type | Version    | Database          | Instance  | Patches |
+------------------------------------------------+------+------------+-------------------+-----------+---------+
| /scratch/app/11.2.0.4/grid                     | GI   | 11.2.0.4.0 |                   |           |         |
| /scratch/app/oradb/product/11.2.0/dbhome_11204 | DB   | 11.2.0.4.0 | apxcmupg,rdb11204 | rdb112042 |         |
'------------------------------------------------+------+------------+-------------------+-----------+---------'
Output from host : myserver71
------------------------------

=====
Homes
=====

.--------------------------------------------------------------------------------------------------------------.
| Home                                           | Type | Version    | Database          | Instance  | Patches |
+------------------------------------------------+------+------------+-------------------+-----------+---------+
| /scratch/app/11.2.0.4/grid                     | GI   | 11.2.0.4.0 |                   |           |         |
| /scratch/app/oradb/product/11.2.0/dbhome_11204 | DB   | 11.2.0.4.0 | apxcmupg,rdb11204 | rdb112043 |         |
'------------------------------------------------+------+------------+-------------------+-----------+---------'

F.2.2 tfactl changes

Use the tfactl changes command to view the changes detected by Oracle Trace File Analyzer.

Syntax

tfactl changes

Example

$ tfactl changes


Output from host : myserver69
------------------------------


Output from host : myserver70
------------------------------
Jul/26/2016 10:20:35 : Parameter 'sunrpc.transports' value changed : tcp 1048576 => udp 32768
Jul/26/2016 10:20:35 : Parameter 'sunrpc.transports' value changed : tcp 1048576 => tcp-bc 1048576


Output from host : myserver71
------------------------------
Jul/26/2016 10:21:06 : Parameter 'sunrpc.transports' value changed : tcp 1048576 => udp 32768
Jul/26/2016 10:21:06 : Parameter 'sunrpc.transports' value changed : tcp 1048576 => tcp-bc 1048576
-bash-4.1# tfactl analyze
INFO: analyzing all (Alert and Unix System Logs) logs for the last 60 minutes...  Please wait...
INFO: analyzing host: myserver69

                       Report title: Analysis of Alert,System Logs
                  Report date range: last ~1 hour(s)
         Report (default) time zone: UTC - Coordinated Universal Time
                Analysis started at: 26-Jul-2016 10:36:03 AM UTC
              Elapsed analysis time: 1 second(s).
                 Configuration file: /scratch/app/11.2.0.4/grid/tfa/myserver69/tfa_home/ext/tnt/conf/tnt.prop
                Configuration group: all
                Total message count:         15,261, from 20-Nov-2015 02:06:21 AM UTC to 26-Jul-2016 10:10:58 AM UTC
  Messages matching last ~1 hour(s):              1, from 26-Jul-2016 10:10:58 AM UTC to 26-Jul-2016 10:10:58 AM UTC
        last ~1 hour(s) error count:              0
last ~1 hour(s) ignored error count:              0
 last ~1 hour(s) unique error count:              0

Message types for last ~1 hour(s)
   Occurrences percent  server name          type
   ----------- -------  -------------------- -----
             1  100.0%  myserver69           generic
   ----------- -------
             1  100.0%

Unique error messages for last ~1 hour(s)
   Occurrences percent  server name          error
   ----------- -------  -------------------- -----
   ----------- -------
             0  100.0%

F.2.3 tfactl events

Use the tfactl events command to view the events detected by Oracle Trace File Analyzer.

Syntax

tfactl events

Example

$ tfactl events
Output from host : myserver69
------------------------------
Jul/25/2016 06:25:33 : 
           [crs.myserver69] : [cssd(7513)]CRS-1603:CSSD on node myserver69 shutdown by user.
Jul/25/2016 06:32:41 : 
           [crs.myserver69] : [cssd(5794)]CRS-1601:CSSD Reconfiguration complete. 
Active nodes are myserver69 myserver70 myserver71 .
Jul/25/2016 06:47:37 : 
           [crs.myserver69] : [/scratch/app/11.2.0.4/grid/bin/scriptagent.bin(16233)]
CRS-5818:Aborted command 'start' for resource 'ora.oc4j'. Details at (:CRSAGF00113:) 
{1:32892:193} in /scratch/app/11.2.0.4/grid/log/myserver69/agent/crsd/scriptagent_oragrid/scriptagent_oragrid.log.
Jul/25/2016 06:24:43 : 
           [db.apxcmupg.apxcmupg_1] : Instance terminated by USER, pid = 21581
Jul/25/2016 06:24:43 : 
           [db.rdb11204.rdb112041] : Instance terminated by USER, pid = 18683
Jul/25/2016 06:24:44 : 
           [db.+ASM1] : ORA-15032: not all alterations performed
           [db.+ASM1] : ORA-15001: diskgroup "FRA" does not exist or is not mounted
           [db.+ASM1] : ORA-15032: not all alterations performed
           [db.+ASM1] : ORA-15001: diskgroup "FRA" does not exist or is not mounted
           [db.+ASM1] : ORA-15032: not all alterations performed
           [db.+ASM1] : ORA-15001: diskgroup "FRA" does not exist or is not mounted
           [db.+ASM1] : ORA-15032: not all alterations performed
           [db.+ASM1] : ORA-15001: diskgroup "FRA" does not exist or is not mounted
           [db.+ASM1] : ORA-15032: not all alterations performed
           [db.+ASM1] : ORA-15001: diskgroup "FRA" does not exist or is not mounted
           [db.+ASM1] : ORA-15032: not all alterations performed
           [db.+ASM1] : ORA-15001: diskgroup "DATA" does not exist or is not mounted
           [db.+ASM1] : ORA-15032: not all alterations performed
           [db.+ASM1] : ORA-15001: diskgroup "DATA" does not exist or is not mounted
           [db.+ASM1] : ORA-15032: not all alterations performed
           [db.+ASM1] : ORA-15001: diskgroup "DATA" does not exist or is not mounted
           [db.+ASM1] : ORA-15032: not all alterations performed
           [db.+ASM1] : ORA-15001: diskgroup "DATA" does not exist or is not mounted
           [db.+ASM1] : ORA-15032: not all alterations performed
           [db.+ASM1] : ORA-15001: diskgroup "DATA" does not exist or is not mounted
Jul/25/2016 06:24:53 : 
           [db.+ASM1] : ORA-15032: not all alterations performed
           [db.+ASM1] : ORA-15027: active use of diskgroup "VDATA" precludes its dismount
Jul/25/2016 06:25:22 : 
           [db.+ASM1] : Shutting down instance (immediate)
           [db.+ASM1] : Shutting down instance: further logons disabled

Summary :
=========
INFO   : 2
ERROR   : 26
WARNING   : 1

F.2.4 tfactl analyze

Use the tfactl analyze command to obtain analysis of your system by parsing the database, Oracle ASM, and Oracle Grid Infrastructure alert logs, system message logs, OSWatcher Top, and OSWatcher Slabinfo files.

Filter the output of the command by component, error type, and time.

With the tfactl analyze command, you can choose from the following types of log file analysis:

  • Show the most common messages within the logs: This analysis provides a quick indication of where larger issues are occurring. Oracle Trace File Analyzer takes important messages out of the alert logs and strips the extraneous information from the log messages, organizes the most commonly occurring messages, and displays them in the order from most common to least common. By default, Oracle Trace File Analyzer analyzes error messages, but you can specify a particular type of message for analysis.

  • Search for text within log messages: This is similar to using the grep utility to search, only faster because Oracle Trace File Analyzer checks the time of each message and only shows those matching the last x number of minutes or any interval of time.

  • Analyze the Oracle OSWatcher log statistics: Oracle Trace File Analyzer reads the various statistics available in the OSWatcher log files and provides detailed analysis showing first, highest, lowest, average, and the last three readings of each statistic. Choose any interval down to a specific minute or second. Oracle Trace File Analyzer optionally provides the original data from the OSWatcher logs for each value reported on (data point).

Syntax

tfactl analyze [-search "pattern"] [-comp db | asm | crs | acfs | os | osw | oswslabinfo | all] 
[-type error | warning | generic] [-last nh|d] 
[-from "MMM/DD/YYYY HH24:MI:SS"] [-to "MMM/DD/YYYY HH24:MI:SS"] [-for "MMM/DD/YYYY HH24:MI:SS"] 
[-node all | local | n1,n2,...] [-verbose] [-o file]

Parameters

Table F-5 tfactl analyze Command Parameters

Parameter Description
-search "pattern"

Searches for a pattern enclosed in double quotation marks ("") in system and alert logs within a specified time range. This parameter supports both case-sensitive and case-insensitive search in alert and system message files across the cluster within the specified filters. Default is case insensitive.

If you do not specify the -search parameter, then Oracle Trace File Analyzer provides a summary of messages within specified filters from alert and system log messages across the cluster.

Oracle Trace File Analyzer displays message counts grouped by type (error, warning, and generic) and shows unique messages in a table organized by message type selected for analysis. The generic message type is assigned to all messages which are not either an error or warning message type.

-comp db | asm | crs | acfs | os | osw | oswslabinfo | all

Select which components you want Oracle Trace File Analyzer to analyze. Default is all.

  • db: Database alert logs

  • asm: Oracle ASM alert logs

  • crs: Oracle Grid Infrastructure alert logs

  • acfs: Oracle ACFS alert logs

  • os: System message files

  • osw: OSW Top output

  • oswlabinfo: OSW Slabinfo output

When OSWatcher data is available, OSW and OSWSLABINFO components provide summary views of OSWatcher data.

-type error | warning | generic

Select what type of messages Oracle Trace File Analyzer analyzes. Default is error.

-last n[h|d]

Specify an amount of time, in hours or days, before current time that you want Oracle Trace File Analyzer to analyze.
-from | -to | -for "MMM/DD/YYYY HH24:MI:SS"

Specify a time interval, using the -from and -to parameters together, or a specific time using the -for parameter, that you want Oracle Trace File Analyzer to analyze.

-node all | local | n1,n2,...

Specify a comma-separated list of host names. Use -local to analyze files on the local node. Default is all.

–verbose

Displays verbose output.
–o file

Specify a file where Oracle Trace File Analyzer writes the output instead of displaying on the screen.

-type Parameter Arguments

The tfactl analyze command classifies all the messages into different categories when you specify the -type parameter. The analysis component provides count of messages by the message type you configure and lists all unique messages grouped by count within specified filters. The message type patterns for each argument are listed in the following table.

Table F-6 tfactl analyze -type Parameter Arguments

Argument Description
error

Error message patterns for database and Oracle ASM alert logs:

.*ORA-00600:.*
.*ORA-07445:.*
.*IPC Send timeout detected. Sender: ospid.*
.*Direct NFS: channel id .* path .* to filer .* PING timeout.*
.*Direct NFS: channel id .* path .* to filer .* is DOWN.*
.*ospid: .* has not called a wait for .* secs.*
.*IPC Send timeout to .* inc .* for msg type .* from opid.*
.*IPC Send timeout: Terminating pid.*
.*Receiver: inst .* binc .* ospid.*
.* terminating instance due to error.*
.*: terminating the instance due to error.*
.*Global Enqueue Services Deadlock detected

Error message patterns for Oracle Grid Infrastructure alert logs:

.*CRS-8011:.*,.*CRS-8013:.*,.*CRS-1607:.*,.*CRS-1615:.*,
.*CRS-1714:.*,.*CRS-1656:.*,.*PRVF-5305:.*,.*CRS-1601:.*,
.*CRS-1610:.*,.*PANIC. CRSD exiting:.*,.*Fatal Error from AGFW Proxy:.*
warning

Warning message patterns for database and Oracle ASM alert logs:

NOTE: process .* initiating offline of disk .*
.*WARNING: cache read a corrupted block group.*
.*NOTE: a corrupted block from group FRA was dumped to
generic

Any messages that do not match any of the preceding patterns.

Examples

The following command examples demonstrate how to use Oracle Trace File Analyzer to search collected data:

  • $ tfactl analyze -search "error" -last 2d

    Oracle Trace File Analyzer searches alert and system log files from the past two days for messages that contain the case-insensitive string "error".

  • $ tfactl analyze -comp os -for "Jul/01/2016 11" -search "."

    Oracle Trace File Analyzer displays all system log messages for July 1, 2016 at 11 am.

  • $ tfactl analyze -search "/ORA-/c" -comp db -last 2d

    Oracle Trace File Analyzer searches database alert logs for the case-sensitive string "ORA-" from the past two days.

The following command examples demonstrate how to use Oracle Trace File Analyzer to analyze collected data:

  • $ tfactl analyze -last 5h

    Oracle Trace File Analyzer displays a summary of events collected from all alert logs and system messages from the past five hours.

  • $ tfactl analyze -comp os -last 1d

    Oracle Trace File Analyzer displays a summary of events from system messages from the past day.

  • $ tfactl analyze -last 1h -type generic

    Oracle Trace File Analyzer analyzes all generic messages from the last hour.

The following command examples demonstrate how to use Oracle Trace File Analyzer to analyze OSWatcher Top and Slabinfo:

  • $ tfactl analyze -comp osw -last 6h

    Oracle Trace File Analyzer displays OSWatcher Top summary for the past six hours.

  • $ tfactl analyze -comp oswslabinfo -from "2016-07-01" -to "2016-07-03"

    Oracle Trace File Analyzer displays OSWatcher Slabinfo summary for specified time period.

F.2.5 tfactl run

Use the tfactl run  command to run the requested action (can be inventory or scan or any support tool).

Syntax

tfactl run [inventory | scan | tool]

Parameters

Table F-7 tfactl run Command Parameters

Parameter Description

inventory

Inventory of all trace file directories.

scan

Runs a one off scan.

tool

Runs the desired analysis tool.

Analysis Tools

Table F-8 tfactl run Analysis Tools Parameters

Parameter Description

changes

Prints system changes.

events

Lists all important events in system.

exachk

Runs Oracle EXAchk.

grep

grep  for input string in logs.

history

Lists commands run in current Oracle Trace File Analyzer shell session.

ls

Searches files in Oracle Trace File Analyzer.

orachk

Runs Oracle ORAchk.

oratop

Runs oratop.

oswbb

Runs OSWatcher Analyzer.

param

Prints parameter value.
ps

Finds a process.

pstack

Runs pstack  on a process.

prw

Runs Procwatcher.

sqlt

Runs SQLT.

summary

Prints system summary.

tail

Tails log files.

vi

Searches and opens files in the vi  editor.

Profiling Tools

Table F-9 tfactl run Profiling Tools Parameters

Parameter Description

dbglevel

Sets CRS log and trace levels using profiles.

F.2.6 tfactl toolstatus

Use the tfactl toolstatus  command to view the status of Oracle Trace File Analyzer Support Tools across all nodes.

Syntax

$ tfactl toolstatus

Example

The tfactl toolstatus  command returns output similar to the following, showing which tool is deployed and where the tool is deployed.

Table F-10 tfactl toolstatus Output

Host Tool Status

hostname

alertsummary

DEPLOYED

hostname

exachk

DEPLOYED

hostname

ls

DEPLOYED

hostname

triage

DEPLOYED

hostname

pstack

DEPLOYED

hostname

orachk

DEPLOYED

hostname

sqlt

DEPLOYED

hostname

grep

DEPLOYED

hostname

summary

DEPLOYED

hostname

vi

DEPLOYED

hostname

prw

NOT RUNNING

hostname

tail

DEPLOYED

hostname

param

DEPLOYED

hostname

dbglevel

DEPLOYED

hostname

managelogs

DEPLOYED

hostname

history

DEPLOYED

hostname

oratop

DEPLOYED

hostname

calog

DEPLOYED

hostname

menu

DEPLOYED

hostname

oswbb

RUNNING

hostname

changes

DEPLOYED

hostname

events

DEPLOYED

hostname

ps

DEPLOYED

hostname

srdc

DEPLOYED