Skip Headers
Oracle® Audit Vault Administrator's Guide
Release 10.2.3.1

Part Number E13841-02
Go to Documentation Home
Home
Go to Book List
Book List
Go to Table of Contents
Contents
Go to Index
Index
Go to Feedback page
Contact Us

Go to previous page
Previous
Go to next page
Next
View PDF

A Troubleshooting an Oracle Audit Vault System

This appendix contains:

A.1 Location of Audit Vault Server Log and Error Files

Table A-1 describes the Audit Vault Server log and error files. These files are located in the Audit Vault Server $ORACLE_HOME/av/log directory. They contain important information about the return status of commands and operations. Use this information to diagnose problems.

Table A-1 Name and Description of Audit Vault Server Log and Error Files

File Name Description

agent.err

Contains a log of errors encountered in collection agent initialization. You can delete this file at any time.

agent.out

Contains a log of all primary collection agent-related operations and activity. You can delete this file at any time.

avca.log

Contains a log of all AVCA and AVCTL commands that have been run and the results of running each command. You can only delete this file only after you have shut down the Audit Vault Server.

av_client-%g.log.n

Contains a log of the collection agent operations and any errors returned from those operations. You can delete this file at any time. The %g is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. A concurrent existence of this file is indicated by a .n suffix appended to the file type name, such as av_client-%g.log.n, where n is an integer issued in sequence (for example, av_client-0.log.1).

avorcldb.log

Contains a log of all AVORCLDB commands that have been run and the results of running each command. You can delete this file at any time.

DB2DB-%g.log

Contains a log of all AVDB2DB commands that have been run and the results of running each command. You can delete this file at any time. The %g is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of AVDB2DB commands, restart Oracle Audit Vault from the Audit Vault Server with the log level set to debug, as follows:

avctl stop_av -loglevel debug
avctl start_av -loglevel debug

MSSQLDB-%g.log

Contains a log of all AVMSSQLDB commands that have been run and the results of running each command. You can delete this file at any time. The %g is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of AVMSSQLDB commands, restart Oracle Audit Vault from the Audit Vault Server with the log level set to debug, as follows:

avctl stop_av -loglevel debug
avctl start_av -loglevel debug

SYBDB-%g.log

Contains a log of all AVSYBDB commands that have been run and the results of running each command. You can delete this file at any time. The %g is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of AVSYBDB commands, restart Oracle Audit Vault from the Audit Vault Server with the log level set to debug, as follows:

avctl stop_av -loglevel debug
avctl start_av -loglevel debug

If you need to troubleshoot the Audit Vault Console, enable Oracle Enterprise Manager logging. To do so, modify the emomslogging.properties file (located in the ORACLE_HOME/sysman/config directory) in the Audit Vault Server home. Add the following lines to this file:

log4j.appender.avAppender=org.apache.log4j.RollingFileAppender
log4j.appender.avAppender.File=$ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole__/log/av-application.log 
log4j.appender.avAppender.Append=true
log4j.appender.avAppender.MaxFileSize =20000000
log4j.appender.avAppender.Threshold = DEBUG
log4j.appender.avAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.avAppender.layout.ConversionPattern=%d [%t] %-5p %c{2} %M.%L - %m\n
log4j.category.oracle =DEBUG, avAppender

You can use this information to debug communication problems between the server and the collection agents.

A.2 Location of Audit Vault Collection Agent Log and Error Files

Table A-2 lists the names and a description of the Audit Vault collection agent log and error files located in the $ORACLE_HOME/av/log directory. These files contain important information about the return status of commands and operations. Use this information to diagnose problems.

Table A-2 Name and Description of Audit Vault Collection Agent Log and Error Files

File Name Description

agent.err

Contains a log of all errors encountered in collection agent initialization and operation. You can delete this file at any time.

agent.out

Contains a log of all primary collection agent-related operations and activity. You can delete this file at any time.

avca.log

Contains a log of all AVCA and AVCTL commands that have been run and the results of running each command. You can delete this file at any time.

avorcldb.log

Contains a log of all AVORCLDB commands that have been run and the results of running each command. You can delete this file at any time.

DBAUD-and-OSAUD-collector-name_source-name_source-id.log

Contains a log of collection operations for the Oracle Database DBAUD and OSAUD collectors. This file has no maximum size limit. To delete this log, shut down the collector, delete the log, and then restart the collector.

non-Oracle_collector-name_source-name_collector_name-%g.log

Contains a log of collection operations for the MSSQLDB, SYBDB, and DB2DB collectors. The % symbol refers to the generation number of the log file. The maximum log file size is 100 MB.

You can only delete this file after you have shut down OC4J. For example, to delete the log where %g is 0, you must stop OC4J. To delete the logs where%g is higher than 0, you can do so while OC4J is running.

To increase the log level, restart OC4J on the collection agent side with the appropriate debug level, as in the following example:

avctl stop_oc4j
avctl start_oc4j -loglevel error

See Section 7.12 and Section 7.16 for information about these commands, including the available log levels for OC4J.

agent_client-%g.log.n

Contains a log of the collection agent operations and any errors returned from those operations. The %g is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. A concurrent existence of this file is indicated by a .n suffix appended to the file type name, such as av_client-%g.log.n, where n is an integer issued in sequence, for example, av_client-0.log.1. You can delete this file at any time.

DB2DB-%g.log

Contains a log of all AVDB2DB commands that have been run and the results of running each command. You can delete this file at any time. The %g is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of AVDB2DB commands, restart OC4J on the collection agent side with the log level set to debug, as follows:

avctl stop_oc4j
avctl start_oc4j -loglevel debug

MSSQLDB-%g.log

Contains a log of all AVMSSQLDB commands that have been run and the results of running each command. You can delete this file at any time. The %g is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of AVMSSQLDB commands, restart OC4J on the collection agent side with the log level set to debug, as follows:

avctl stop_oc4j
avctl start_oc4j -loglevel debug

SYBDB-%g.log

Contains a log of all AVSYBDB commands that have been run and the results of running each command. You can delete this file at any time. The %g is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of AVSYBDB commands, restart OC4J on the collection agent side with the log level set to debug, as follows:

avctl stop_oc4j
avctl start_oc4j -loglevel debug

sqlnet.log

Contains a log of SQL*Net information.


The Oracle Audit Vault collection agent $ORACLE_HOME/oc4j/j2ee/home/log contains the logs generated by the collection agent OC4J. In this directory, the file AVAgent-access.log contains a log of requests that the collection agent receives from the Audit Vault Server. Use this information to debug communication problems between the server and the collection agent.

Failed configuration commands are located in the Audit Vault collection agent $ORACLE_HOME/cfgtoollogs directory, which includes the file, configToolFailedCommands. This file contains only the name of the failed command. See the avca.log or avorcldb.log file for additional information, including any associated errors and error messages.

A.3 Troubleshooting Tips

This section contains:

A.3.1 Checking Trace Files for Detailed Information About Oracle Database Errors

For detailed information about the cause of an Oracle Database error message, check the trace files. The trace files also indicate problems that may have occurred with the Audit Vault data warehouse, alert, and some configuration issues. The USER_DUMP_DEST initialization parameter specifies the current location of the trace files. You can find the value of this parameter by issuing SHOW PARAMETER USER_DUMP_DEST in SQL*Plus. See Oracle Database Performance Tuning Guide for more information about trace files

A.3.2 Troubleshooting Audit Vault Server

Problem: Need to find the best way to tune the Audit Vault Server performance for the REDO collector.

The Audit Vault Server installation process sets the STREAMS_POOL_SIZE initialization parameter to 150 MB. If you plan to use the REDO collector, you must tune this parameter to maximize REDO collector performance. In an Oracle Real Application Clusters (Oracle RAC) environment, tune this parameter on all nodes, because it is uncertain where the queue will be after a database instance starts.

Solution:

Typically, after you have configured and started the REDO collector, let it run for a while. This enables the Oracle Database autotuning feature to allocate memory for the best database performance for the STREAMS_POOL_SIZE parameter. Using Automatic Workload Repository (AWR), check if Streams AQ has a flow control problem, such as enqueue being blocked. If the performance is slow (for example, only 500 records are applied per second), you may need to tune the STREAMS_POOL_SIZE parameter.

If you have at least 1 GB of physical memory in your Audit Vault Server system, set the STREAMS_POOL_SIZE parameter to 200 MB using the ALTER SYSTEM SQL statement, as follows:

ALTER SYSTEM SET STREAMS_POOL_SIZE=200;

The record apply rate should be 2000 records per second, which is a typical maximum rate for the REDO collector. Usually, setting the value to 200 MB is sufficient. If you are using Oracle Audit Vault in an Oracle RAC environment, set this parameter value accordingly on all nodes in the cluster, as follows:

ALTER SYSTEM SET STREAMS_POOL_SIZE=200 SID=avn;

Replace avn with the SID for each node in the cluster.

A.3.3 Troubleshooting Audit Vault Collection Agent

Problem: Audit Vault agent status is blank on the Windows Services Panel.

After installing Audit Vault Agent for Microsoft Windows (32-bit), configuring a source and collectors, then starting the agent on the Audit Vault Server side, you notice that the Services Panel on the Windows system where the Audit Vault collection agent resides shows that the status is blank, rather than Started.

Solution:

This is normal behavior for the Audit Vault collection agent on Microsoft Windows systems because the service is a short-lived process. Once the Agent service process completes its task, it exits, so the status of the service will not show as Started. However, the Audit Vault collection agent is running without problems.

Run the avctl show_agent_status command to check the status of the Audit Vault Agent, as follows:

C:\ORACLE_HOME\agent_dir\bin\avctl show_agent_status -agentname agent1
AVCTL started
Getting agent metrics...
--------------------------------
Agent is running
--------------------------------
Metrics retrieved successfully. 

Problem: Need to debug a collection agent problem.

You want to enable debug logging while trying to diagnose an Audit Vault collection agent problem.

Solution:

  1. Run the following set of AVCTL commands on the command line:

    avctl stop_oc4j
    avctl start_oc4j -loglevel debug
    
  2. Check the log output in the Audit Vault collection agent $ORACLE_HOME/av/log directory.

  3. Because debugging creates more logging and writing overhead, remember to disable it when you no longer need it, as follows:

    avctl stop_oc4j
    avctl start_oc4j
    

See Section 7.12 and Section 7.16 for more information about the avctl stop_oc4j and avctl start_oc4j commands.

Problem: The Agent OC4J or Audit Vault Console OC4J fails to start.

After you run the avctlstart_oc4j command, an avctl show_oc4j_status command shows that OC4J is not running. Or, after you issue avctl start_av command, an avctl show_av_status command shows that OC4J is not running.

Solution:

Go to $ORACLE_HOME/av/log/agent.err log file and check the error message that appears in the log.

Or, go to $ORACLE_HOME/oc4j/j2ee/home and issue the following command to find the error message that appears on the console:

java -jar oc4j.jar

This problem is most likely caused by a port conflict. For example, if the problem is caused by an RMI port conflict, you would see a message similar to the following:

C:\oracle\product\10.2\avagentrc3_01\oc4j\j2ee\home>java -jar oc4j.jar

08/05/16 10:39:51 Error starting ORMI-Server.  Unable to bind socket: Address already in use: JVM_Bind

The RMI, JMS, and HTTP ports are necessary for starting OC4J or the Audit Vault Console OC4J. The agent OC4J and Audit Vault Console OC4J can fail to start or the agent service of the Audit Vault Console can become unavailable if these ports have a conflict. If there is a port conflict, you can modify the port settings in the following files at $ORACLE_HOME/oc4j/j2ee/config by selecting a port number not in use:

  • rmi.xml

  • jms.xml

  • http-web-site.xml or (av-agent-web-site.xml)

Problem: The setup command returned an error message that the connection to the source database using the credentials in the wallet was not successful.

This problem is most likely due to entering an incorrect user name or password or both when issuing the setup command using the AVORCLDB, AVMSSQLDB, AVSYBDB, or AVDB2DB command-line utility.

Solution:

Reissue the setup command again using the correct credentials.

A.3.4 Troubleshooting the Audit Vault Collector

Problem: Cannot start the DBAUD collector and the log file shows an error.

The DBAUD collector log file (in the Audit Vault collection agent home directory) shows the following entry:

INFO @ '17/08/2008 15:05:48 02:00': 
Could not call Listener, NS error 12541, 12560, 511, 2, 0

Solution:

Ensure that you completed the last step for configuring the source database and collectors, as described in Section 2.3.6, which describes how to run the avorcldb setup command in the Audit Vault collection agent home. See also Section 2.2.3 and Section 2.2.4.

Follow these steps:

  1. Change directories to the network/admin directory:

    $ cd $ORACLE_HOME/network/admin
    
  2. Perform the cat command on your tnsnames.ora file.

    There should be an entry similar to SRCDB1. If there is no SRCDB1 entry in your tnsnames.ora file, then run the avorcldb setup command as shown in Section 2.3.6.

  3. Try to connect to the source database with the following command, assuming your tnsnames.ora file has an SRCDB2 entry.

    For example:

    $ sqlplus /@SRCDB1
    

    If the connection is successful, then your source database is set up correctly.

  4. Try starting the DBAUD collector using the avctl start_colletor command.

    For example:

    $ avctl start_collector -collname REDO_Collector -srcname ORCLSRC1.EXAMPLE.COM
    

    See Section 7.11 for more information about the avctl start_collector command.

Problem: Not sure if the DBAUD and OSAUD collectors are collecting from the AUD$ table and the OS file, respectively.

After you set up both the DBAUD and OSAUD collectors, you want to check that they are collecting from the AUD$ table and OS file, respectively.

Solution:

To determine if the DBAUD collector is collecting from the AUD$ table, check the contents of the DBAUD log file, located in the $ORACLE_HOME/av/log directory.

To determine if the OSAUD collector is collecting from the OS file, check the contents of the osaud_collector-name_source-name_source-id.log file in the Audit Vault collection agent- home $ORACLE_HOME/av/log directory.

Check each file for entries that indicate that the collector is collecting audit records.

For example, the DBAUD collector log file would have entries similar to the following:

***** Started logging for 'AUD$ Audit Collector' *****
.
.
.
INFO @ '25/10/2008 19:08:42 -8:00': 
     ***** SRC connected OK

INFO @ '25/10/2008 19:08:53 -8:00': 
     ***** SRC data retireved OK
.
.
.

The OSAUD collector log file could have an entry as follows:

File opened for logging source "DBS1.REGRESS.RDBMS.DEV.US.ORACLE.COM"
INFO @ '24/10/2008 18:16:18 -8:00': 
***** Started logging for 'OS Audit Collector' *****

If the log files look correct, then refresh the data warehouse using the avctl refresh_warehouse command in the Audit Vault Server shell. When this operation completes, log in to the Audit Vault Console as the Audit Vault auditor. Examine the graphical summary named Activity by Audit Event Category on the Overview page for the appearance of additional audit records in the various event categories. Increased counts for the various event categories indicate that these collectors are collecting audit records.

Problem: ORA-01017:invalid username/password; logon denied error when starting up the DBAUD_Collector or setting up the REDO_Collector.

When you try to start the DBAUD collector or configure the REDO collector, the following error message appears:

ORA-01017: invalid username/password; logon denied

Solution:

There may be a problem with your user name or password in the password file, or a problem with the wallet credentials. Try re-creating the user name and password. If the problem persists, re-create the password file. If this does not correct the problem, add the source user to the wallet again using the avorcldb setup command. Ensure that this user is the same user name and password that you are using on the source database.

Problem: Collector log for the MSSQLDB, SYBDB, or DB2DB collector indicates that a jar file is missing.

If the following JDBC driver jar files are missing from the Audit Vault collection agent $ORACLE_HOME/jlib library, this error appears in the collector log of the respective collector being used.

  • SQL Server: sqljdbc.jar

  • Sybase ASE: jconn3.jar

  • IBM DB2: db2jcc.jar

Under other circumstances, such as when you use either the AVMSSQLDB, AVSYBDB, or AVDB2DB command-line utilities, the following error is appears when the JDBC driver is not in this directory:

JDBC Driver is missing. Please make sure that the JDBC jar exists in the location specified in Audit Vault documentation.

Solution:

See the following sections:

  • SQL Server: Section 2.4.1 for information about the sqljdbc.jar file

  • Sybase ASE: Section 2.5.1 for information about the jconn3.jar file

  • IBM DB2: Section 2.6.1 for information about the db2jcc.jar files

After you download and copy these JDBC drivers in place, restart OC4J. See Section 7.16 and Section 7.12 for more information about stopping and starting the OC4J agent.

Problem: Unable to connect to source database.

When you try to verify the ORCLDB, MSSQLDB, SYBDB, or DB2DB collector using the verify command, the following error message appears:

Unable to connect to source database

Solution:

This error appears if the source user that you specified in the verify command for the source database does not have sufficient privileges to connect to the source database. Check if the source user has sufficient privileges to connect to the respective database. See the following sections for information about creating a source user with sufficient privileges:

A.3.5 Troubleshooting Oracle Audit Vault Console

Problem: Audit Vault Console does not appear in the Web browser.

When you try to access the Audit Vault Console in a Web browser, it appears to hang, or after a while it times out.

Solution:

This may be happening because the Audit Vault Console is down. To check the status of the Audit Vault Console, issue an avctl show_av_status command in the Audit Vault Server shell. If the status indicates that the Audit Vault Console is down, issue the avctl start_av command in the Audit Vault Server shell to restart it. Then start the Audit Vault Console in the Web browser. The Audit Vault Console should appear and let you log in to the management system of the Audit Vault auditor administrator.

Problem: Need to debug an Audit Vault Console problem.

You want to enable debug logging while trying to diagnose an Audit Vault Console problem.

Solution:

Run the following commands on the command-line:

avctl stop_av
avctl start_av -loglevel debug

Then check the log output in the Audit Vault Server $ORACLE_HOME/av/log directory.

Because debugging creates more logging and writing overhead, remember to disable it when you no longer need it, as follows:

avctl stop_av
avctl start_av

See Section 7.10 and Section 7.14 for more information about these commands.

A.3.6 Troubleshooting Oracle Audit Vault in an Oracle Real Application Clusters Environment

Problem: In an Oracle RAC environment, the avca drop_agent operation fails with an error when this command is issued from one of the Oracle RAC nodes.

When you try to run the avca add_agent command from one of the Oracle RAC nodes, the command fails.

Solution:

In an Oracle RAC environment, you must run the AVCA commands from the node on which Oracle Enterprise Manager resides. This is the same node on which the av.ear file is deployed.

To find where the av.ear file is deployed, locate the $ORACLE_HOME/oc4j/j2ee/oc4j_applications/applications/av/av/WEB-INF/classes/av.properties file is located.

Once you locate the node, run the AVCA and AVCTL commands from that node.

If the node on which the av.ear file is deployed is down, deploy the av.ear file to another node using the avca deploy_av command. See Section 6.4 for more information about this command.

When you run the avca deploy_av command, on the other node Oracle Database creates a wallet containing the default avadmin entries. You must add the other entries, such as the source user credentials, to the wallet by using the setup command for the appropriate utility (AVORCLDB, AVMSSQLDB, AVSYBDB, or AVDB2DB), depending on the collectors being used.

To access the Audit Vault Console from this other node, enter the following URL in the Web browser:

http://host:port/av

In this specification:

  • host is the host name or IP address of the other Oracle RAC node

  • port is the port number for the Oracle RAC node