A Troubleshooting an Oracle Audit Vault System

This appendix contains:

A.1 Location of Audit Vault Server Log and Error Files

Table A-1 describes the Audit Vault Server log and error files. These files are located in the Audit Vault Server $ORACLE_HOME/av/log directory. They contain important information about the return status of commands and operations. Use this information to diagnose problems. You should periodically monitor these files, and delete these files to control the amount of disk space the log files use.

Table A-1  Names and Descriptions of Audit Vault Server Log and Error Files

File Name Description

agent.err

Contains a log of errors encountered in collection agent initialization. You can delete this file at any time.

agent.out

Contains a log of all primary collection agent-related operations and activity. You can delete this file at any time.

alert-email-%g.log.n

Contains a log of the email alert operations and any errors returned from those operations. You can delete this file at any time. The %g is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. A concurrent existence of this file is indicated by a .n suffix appended to the file type name, such as av_client-%g.log.n, where n is an integer issued in sequence (for example, alert-email-0.log.1).

alert_troubleticket-%g.log.n

Contains a log of the trouble ticket alert operations and any errors returned from those operations. You can delete this file at any time. The %g is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. A concurrent existence of this file is indicated by a .n suffix appended to the file type name, such as av_client-%g.log.n, where n is an integer issued in sequence (for example, alert-troubleticket-0.log.1).

avca.log

Contains a log of all AVCA and AVCTL commands that have been run and the results of running each command. You can only delete this file only after you have shut down the Audit Vault Server.

av_client-%g.log.n

Contains a log of the collection agent operations and any errors returned from those operations. You can delete this file at any time. The %g is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. A concurrent existence of this file is indicated by a .n suffix appended to the file type name, such as av_client-%g.log.n, where n is an integer issued in sequence (for example, av_client-0.log.1).

avorcldb.log

Contains a log of all AVORCLDB commands that have been run and the results of running each command. You can delete this file at any time.

DB2DB-%g.log

Contains a log of all AVDB2DB commands that have been run and the results of running each command. You can delete this file at any time. The %g is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of AVDB2DB commands, restart Oracle Audit Vault from the Audit Vault Server with the log level set to debug, as follows:

avctl stop_av -loglevel debug
avctl start_av -loglevel debug

MSSQLDB-%g.log

Contains a log of all AVMSSQLDB commands that have been run and the results of running each command. You can delete this file at any time. The %g is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of AVMSSQLDB commands, restart Oracle Audit Vault from the Audit Vault Server with the log level set to debug, as follows:

avctl stop_av -loglevel debug
avctl start_av -loglevel debug
report-generation-%g.log

Contains a report for both recurring scheduled and manual log messages. You can delete this file at any time. The %g is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit.

SYBDB-%g.log

Contains a log of all AVSYBDB commands that have been run and the results of running each command. You can delete this file at any time. The %g is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of AVSYBDB commands, restart Oracle Audit Vault from the Audit Vault Server with the log level set to debug, as follows:

avctl stop_av -loglevel debug
avctl start_av -loglevel debug

report-generation-0.log

Contains report generation-related log messages. This file is created when Oracle Audit Vault generates PDF reports in the Audit Vault Server. You can delete this file at any time.

policy-0.log

Contains policy-related log messages for Oracle database audit policies created in Audit Vault. This file is created when Oracle Audit Vault fetches audit settings from a source database, or provisions the audit settings back to the source database. You can delete this file at any time.


If you need to troubleshoot the Audit Vault Console, enable Oracle Enterprise Manager logging. To do so, modify the emomslogging.properties file (located in the $ORACLE_HOME/sysman/config directory) in the Audit Vault Server home. Add the following lines to this file:

log4j.appender.avAppender=org.apache.log4j.RollingFileAppender
log4j.appender.avAppender.File=$ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole__/log/av-application.log 
log4j.appender.avAppender.Append=true
log4j.appender.avAppender.MaxFileSize =20000000
log4j.appender.avAppender.Threshold = DEBUG
log4j.appender.avAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.avAppender.layout.ConversionPattern=%d [%t] %-5p %c{2} %M.%L - %m\n
log4j.category.oracle =DEBUG, avAppender

You can use this information to debug communication problems between the server and the collection agents.

A.2 Location of Audit Vault Collection Agent Log and Error Files

Table A-2 lists the names and a description of the Audit Vault collection agent log and error files located in the $ORACLE_HOME/av/log directory. These files contain important information about the return status of commands and operations. Use this information to diagnose problems.

Table A-2 Names and Descriptions of Audit Vault Collection Agent Log and Error Files

File Name Description

agent.err

Contains a log of all errors encountered in collection agent initialization and operation. You can delete this file at any time.

agent.out

Contains a log of all primary collection agent-related operations and activity. You can delete this file at any time.

avca.log

Contains a log of all AVCA and AVCTL commands that have been run and the results of running each command. You can delete this file at any time.

avorcldb.log

Contains a log of all AVORCLDB commands that have been run and the results of running each command. You can delete this file at any time.

collname_srcname_src_id.log

Contains a log of information about what was collected and any collection errors for the DBAUD and OSAUD collectors. This file has no maximum size limit. To delete this log, shut down the collector, delete the log, and then restart the collector.

See also Section A.3.4 for more information troubleshooting collectors.

srcname-collname-#.log

Applies to all collectors, as follows:

  • Oracle Database DBAUD, OSAUD, and REDO collectors. Contains monitoring information, such as whether the collector is active and how many records were sent. For the REDO collector, the Streams framework performs the actual collection, so the Oracle Audit Vault agent has no knowledge of the collection.

  • Non-Oracle Database collectors. Contains a log of all collection operations for the MSSQLDB, SYBDB, and DB2 collectors.

The # symbol refers to the generation number of the log file. The maximum log file size is 100 MB.

A srcname-collname-#.log.lck accompanies the log files. You can safely ignore this file.

You can only delete this file after you have shut down the agent (for collection agents from Release 10.2.3.2 and later) or OC4J (for Release 10.2.3.1 or earlier agents but not yet upgraded). For example, to delete the log where # is 0, you must stop the agent or OC4J. To delete the logs where # is higher than 0, you can do so while the agent or OC4J is running.

To increase the log level, restart the agent or OC4J with the appropriate debug level, as in the following examples:

Release 10.2.3.2 and later:

avctl stop_agent
avctl start_agent -loglevel error

See Section 8.12 and Section 8.9 for information about these commands, including the available log levels.

Release 10.2.3.1 or earlier:

avctl stop_oc4j
avctl start_oc4j -loglevel error

See Section 8.15.3 and Section 8.15.2 for information about these commands, including the available log levels.

agent_client-%g.log.n

Contains a log of the collection agent operations and any errors returned from those operations. The %g is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. A concurrent existence of this file is indicated by a .n suffix appended to the file type name, such as av_client-%g.log.n, where n is an integer issued in sequence, for example, av_client-0.log.1. You can delete this file at any time.

DB2DB-%g.log

Contains a log of all AVDB2DB commands that have been run and the results of running each command. You can delete this file at any time. The %g is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of AVDB2DB commands, restart the agent (for agents from Release 10.2.3.2 and later) or OC4J (for Release 10.2.3.1 or earlier but not yet upgraded) with the log level set to debug, as follows:

Release 10.2.3.2 and later:

avctl stop_agent
avctl start_agent -loglevel error

See Section 8.12 and Section 8.9 for information about these commands, including the available log levels.

Release 10.2.3.1 or earlier:

avctl stop_oc4j
avctl start_oc4j -loglevel error

See Section 8.15.3 and Section 8.15.2 for information about these commands, including the available log levels.

MSSQLDB-%g.log

Contains a log of all AVMSSQLDB commands that have been run and the results of running each command. You can delete this file at any time. The %g is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of AVMSSQLDB commands, restart the agent (for agents from Release 10.2.3.2 and later) or OC4J (for Release 10.2.3.1 or earlier but not yet upgraded) on the collection agent side with the log level set to debug, as follows:

Release 10.2.3.2 and later:

avctl stop_agent
avctl start_agent -loglevel error

See Section 8.12 and Section 8.9 for information about these commands, including the available log levels.

Release 10.2.3.1 or earlier (but not yet upgraded):

avctl stop_oc4j
avctl start_oc4j -loglevel error

See Section 8.15.3 and Section 8.15.2 for information about these commands, including the available log levels.

SYBDB-%g.log

Contains a log of all AVSYBDB commands that have been run and the results of running each command. You can delete this file at any time. The %g is a generation number that starts from 0 (zero) and increases once the file size reaches the 100 MB limit. To enable detailed logging of AVSYBDB commands, restart the agent (for agents from Release 10.2.3.2 and later) or OC4J (for Release 10.3.2.1 or earlier, but not yet upgraded) on the collection agent side with the log level set to debug, as follows:

Release 10.2.3.2 and later:

avctl stop_agent
avctl start_agent -loglevel error

See Section 8.12 and Section 8.9 for information about these commands, including the available log levels.

Release 10.2.3.1 or earlier:

avctl stop_oc4j
avctl start_oc4j -loglevel error

See Section 8.15.3 and Section 8.15.2 for information about these commands, including the available log levels.

sqlnet.log

Contains a log of SQL*Net information.


The Oracle Audit Vault collection agent $ORACLE_HOME/oc4j/j2ee/home/log contains the logs generated by the collection agent OC4J. In this directory, the file AVAgent-access.log contains a log of requests that the collection agent receives from the Audit Vault Server. Use this information to debug communication problems between the server and the collection agent.

Failed configuration commands are located in the Audit Vault collection agent $ORACLE_HOME/cfgtoollogs directory, which includes the file, configToolFailedCommands. This file contains only the name of the failed command. See the avca.log or avorcldb.log file for additional information, including any associated errors and error messages.

A.3 Troubleshooting Tips

This section contains:

A.3.1 Checking Trace Files for Detailed Information About Oracle Database Errors

For detailed information about the cause of an Oracle Database error message, check the trace files. The trace files also indicate problems that may have occurred with the Audit Vault data warehouse, alert, and some configuration issues. The USER_DUMP_DEST initialization parameter specifies the current location of the trace files. You can find the value of this parameter by issuing SHOW PARAMETER USER_DUMP_DEST in SQL*Plus. See Oracle Database Performance Tuning Guide for more information about trace files

A.3.2 Troubleshooting Audit Vault Server

This section contains:

A.3.2.1 Tuning Audit Vault Server Performance for the REDO Collector

Problem: You are concerned that the Audit Vault Server performance for the REDO collector is slow. The Audit Vault Server installation process sets the STREAMS_POOL_SIZE initialization parameter to 150 MB. If you plan to use the REDO collector, you must tune this parameter to maximize REDO collector performance. In an Oracle Real Application Clusters (Oracle RAC) environment, tune this parameter on all nodes, because it is uncertain where the queue will be after a database instance starts

Solution: Typically, after you have configured and started the REDO collector, let it run for a while. This enables the Oracle Database autotuning feature to allocate memory for the best database performance for the STREAMS_POOL_SIZE parameter. Using Automatic Workload Repository (AWR), check if Streams AQ has a flow control problem, such as enqueue being blocked. If the performance is slow (for example, only 500 records are applied per second), you may need to tune the STREAMS_POOL_SIZE parameter.

If you have at least 1 GB of physical memory in your Audit Vault Server system, set the STREAMS_POOL_SIZE parameter to 200 MB using the ALTER SYSTEM SQL statement, as follows:

ALTER SYSTEM SET STREAMS_POOL_SIZE=200;

The record apply rate should be 2000 records per second, which is a typical maximum rate for the REDO collector. Usually, setting the value to 200 MB is sufficient. If you are using Oracle Audit Vault in an Oracle RAC environment, set this parameter value accordingly on all nodes in the cluster, as follows:

ALTER SYSTEM SET STREAMS_POOL_SIZE=200 SID=avn;

Replace avn with the SID for each node in the cluster.

A.3.3 Troubleshooting Audit Vault Collection Agent

This section contains:

A.3.3.1 Blank Status on Windows Services Panel for Audit Vault Agent

Problem: After you install Audit Vault Agent for Microsoft Windows (32-bit), configure a source database and collectors, and then start the agent on the Audit Vault Server side, you notice that the Services Panel on the Windows system where the Audit Vault collection agent resides shows that the status is blank, rather than Started.

Solution: This is normal behavior for the Audit Vault collection agent on Microsoft Windows systems because the service is a short-lived process. Once the Agent service process completes its task, it exits, so the status of the service will not show as Started. However, the Audit Vault collection agent is running without problems.

Run the avctl show_agent_status command to check the status of the Audit Vault Agent, as follows:

C:\ORACLE_HOME\bin\avctl show_oc4j_status

A.3.3.2 Debugging a Collection Agent Problem

Problem: You discover that there is a problem with the Audit Vault collection agents, so you want to enable debug logging while trying to diagnose the problem.

Solution:

  1. Open a shell or command prompt for the Audit Vault collection agent.

    • UNIX: Set the environment variables, as described in Section 2.2.3.

    • Microsoft Windows: Go to the collection agent ORACLE_HOME\bin directory.

  2. Run the following AVCTL commands on the command line:

    • For Release 10.2.3.2 and later agents: Run the following commands:

      avctl stop_agent
      avctl start_agent -loglevel debug
      
    • For Release 10.2.3.1 or earlier agents that have not been upgraded: Run these commands:

      avctl stop_oc4j
      avctl start_oc4j -loglevel debug
      
  3. Check the log output in the Audit Vault collection agent $ORACLE_HOME/av/log directory.

  4. Because debugging creates more logging and writing overhead, remember to disable it when you no longer need it.

    • For Release 10.2.3.2 and later agents: Run these commands:

      avctl stop_agent
      avctl start_agent -loglevel info
      
    • For Release 10.2.3.1 or earlier agents that have not been upgraded: Run these commands:

      avctl stop_oc4j
      avctl start_oc4j -loglevel info
      

A.3.3.3 The Agent OC4J or Audit Vault Console OC4J Failing to Start

Problem: After you run the avctlstart_oc4j command, an avctl show_oc4j_status command shows that OC4J is not running. Or, after you issue avctl start_av command, an avctl show_av_status command shows that OC4J is not running.

Solution: Go to $ORACLE_HOME/av/log/agent.err log file and check the error message that appears in the log.

Or, go to $ORACLE_HOME/oc4j/j2ee/home and issue the following command to find the error message that appears on the console:

java -jar oc4j.jar

This problem is most likely caused by a port conflict. For example, if the problem is caused by an RMI port conflict, you would see a message similar to the following:

C:\oracle\product\10.3\avagentrc3_01\oc4j\j2ee\home>java -jar oc4j.jar

09/05/16 10:39:51 Error starting ORMI-Server.  Unable to bind socket: Address already in use: JVM_Bind

The RMI, JMS, and HTTP ports are necessary for starting OC4J or the Audit Vault Console OC4J. The agent OC4J and Audit Vault Console OC4J can fail to start or the agent service of the Audit Vault Console can become unavailable if these ports have a conflict. If there is a port conflict, see Section 4.9 for information about changing the Oracle Audit Vault port number. See also Section 1.3.4.3 for the default collection agent port numbers.

A.3.3.4 Failed Source Database Connection Due to Invalid Wallet Credentials

Problem: The setup command for the AVORCLDB, AVMSSQLDB, AVSYBDB, or AVDB2DB command-line utility returns an error message saying that the connection to the source database using the credentials in the wallet was not successful. This problem is most likely due to your entering an incorrect user name or password or both when you issued the setup command.

Solution: Reissue the setup command again using the correct credentials. See the following sections:

  • Section 9.9 for Oracle databases

  • Section 10.9 for SQL Server databases

  • Section 11.9 for Sybase ASE databases

  • IBM DB2 databases: The avdb2db utility has no setup command. For IBM DB2 databases, you only need to change the password of the designated user account.

A.3.4 Troubleshooting the Audit Vault Collectors

This section contains:

A.3.4.1 ORA-01031 Error When You Try to Create a an Oracle Database Collector

Problem: You are able to add a collector, but the following error appears in the avorcldb add_collector command output:

ERROR: java.sql.SQLException: ORA-01031: insufficient privileges

Afterward, the collector is added, but it fails to collect any audit data.

Solution: The source database user account that you used to create the collector may not have the correct privileges. To remedy this problem:

  1. Run the zarsspriv.sql script for the source database user account, as described in Section 2.3.1.

  2. Restart the collector, as described in Section 2.8.

Another problem can be the compatibility between versions of the Audit Vault Server and source database. See Oracle Audit Vault Installation Guide for your platform for the requirements.

A.3.4.2 Oracle Source Database DBAUD Log Errors When Starting DBAUD Collector

Problem: For Oracle source databases that use the DBAUD collector, errors similar to the following may appear in the DBAUD log file when the collectors before the collectors have started:

INFO @ '12/08/2009 06:50:01 02:00':
Could not call Listener, NS error 12541, 12560, 511, 2, 0
INFO @ '12/08/2009 06:50:03 02:00':
Could not call Listener, NS error 12541, 12560, 511, 2, 0
INFO @ '12/08/2009 06:51:03 02:00':
Could not call Listener, NS error 12541, 12560, 511, 2, 0
INFO @ '12/08/2009 06:51:11 02:00':
Could not call Listener, NS error 12541, 12560, 511, 2, 0
INFO @ '12/08/2009 06:51:11 02:00':
 
             ***** Started logging for 'AUD$ Audit Collector' *****
 
INFO @ '12/08/2009 06:51:11 02:00':
     ***** Collector Name = DBAUD_Collector
 
INFO @ '12/08/2009 06:51:11 02:00':
     ***** Source Name = HUKSA.EXAMPLE.COM
 
INFO @ '12/08/2009 06:51:11 02:00':
     ***** Av Name = AV
 
INFO @ '12/08/2009 06:51:11 02:00':
     ***** Initialization done OK
 
INFO @ '12/08/2009 06:51:11 02:00':
     ***** Starting CB
 
INFO @ '12/08/2009 06:51:13 02:00':
Getting parameter |AUDAUDIT_DELAY_TIME|, got |20|
 
INFO @ '12/08/2009 06:51:13 02:00':
Getting parameter |AUDAUDIT_SLEEP_TIME|, got |5000|
 
INFO @ '12/08/2009 06:51:13 02:00':
Getting parameter |AUDAUDIT_ACTIVE_SLEEP_TIME|, got |1000|
 
INFO @ '12/08/2009 06:51:13 02:00':
Getting parameter |AUDAUDIT_MAX_PROCESS_RECORDS|, got |1000|
 
INFO @ '12/08/2009 06:51:13 02:00':
     ***** CSDK inited OK + 1
 
INFO @ '12/08/2009 06:51:13 02:00':
     ***** Src alias = SRCDB1
 
INFO @ '12/08/2009 06:51:13 02:00':
     ***** SRC connected OK
 
INFO @ '12/08/2009 06:51:13 02:00':
     ***** SRC data retrieved OK
 
INFO @ '12/08/2009 06:51:13 02:00':
     ***** Recovery done OK

Solution: You can disregard these error messages. To check that the collector has truly started, you can run the avctl show_collector_status command. See Section 8.6.

A.3.4.3 DBAUD Collector Does Not Start and the Listener Is Not Available

Problem: You cannot start the DBAUD collector for an Oracle source database, and the listener is not available. The DBAUD collector log file (in the Audit Vault collection agent home directory) shows the following entry:

INFO @ '17/08/2009 15:05:48 02:00': 
Could not call Listener, NS error 12541, 12560, 511, 2, 0

Solution:

Ensure that you completed the last step for configuring the source database and collectors, as described in Section 2.3.5, which describes how to run the avorcldb setup command in the Audit Vault collection agent home. See also Section 2.2.3 and Section 2.2.4.

Follow these steps:

  1. Change directories to the network/admin directory:

    cd $ORACLE_HOME/network/admin
    
  2. Perform the cat command on your tnsnames.ora file.

    There should be an entry similar to SRCDB1. If there is no SRCDB1 entry in your tnsnames.ora file, then run the avorcldb setup command as shown in Section 2.3.5.

  3. Try to connect to the source database with the following command, assuming your tnsnames.ora file has an SRCDB2 entry.

    For example:

    sqlplus /@SRCDB1
    

    If the connection is successful, then your source database is set up correctly.

  4. Try starting the DBAUD collector using the avctl start_colletor command.

    For example:

    avctl start_collector -collname REDO_Collector -srcname ORCLSRC1.EXAMPLE.COM
    

    See Section 8.11 for more information about the avctl start_collector command.

A.3.4.4 Not Sure if the DBAUD and OSAUD Collectors Are Working

Problem: You are not sure if the DBAUD collector is collecting from the SYS.AUD$ table and the OSAUD collector is collecting from the operating system audit file.

Solution: To determine if the DBAUD collector is collecting from the AUD$ table, check the contents of the DBAUD log file, located in the $ORACLE_HOME/av/log directory.

To determine if the OSAUD collector is collecting from the OS file, check the contents of the osaud_collector-name_source-name_source-id.log file in the Audit Vault collection agent- home $ORACLE_HOME/av/log directory.

Check each file for entries that indicate that the collector is collecting audit records.

For example, the DBAUD collector log file would have entries similar to the following:

     ***** Started logging for 'AUD$ Audit Collector' *****
.
.
.
INFO @ '25/10/2008 19:08:42 -8:00': 
     ***** SRC connected OK

INFO @ '25/10/2008 19:08:53 -8:00': 
     ***** SRC data retireved OK
.
.
.

The OSAUD collector log file could have an entry as follows:

File opened for logging source "DBS1.REGRESS.RDBMS.DEV.US.ORACLE.COM"
INFO @ '24/10/2008 18:16:18 -8:00': 
***** Started logging for 'OS Audit Collector' *****

Log in to the Audit Vault Console as the Audit Vault auditor. Examine the graphical summary named Activity by Audit Event Category on the Overview page for the appearance of additional audit records in the various event categories. Increased counts for the various event categories indicate that these collectors are collecting audit records.

A.3.4.5 ORA-01017 Error When You Try to Start the DBAUD or REDO Collectors

Problem: When you try to start the DBAUD collector or configure the REDO collector, the following error message appears:

ORA-01017: invalid username/password; logon denied

Solution: Ensure that the host that you specified for the -av argument of the avorcldb add_collector command is correct. You can confirm the host name by checking the tnsnames.ora file for the source database.

Alternatively, there may be a problem with your user name or password in the password file, or a problem with the wallet credentials. Try re-creating the user name and password. If the problem persists, re-create the password file. If this does not correct the problem, then add the source user to the wallet again using the avorcldb setup command. Ensure that this user is the same user name and password that you are using on the source database.

A.3.4.6 MSSQLDB, SYBDB, or DB2 Collector Log Indicates Jar File Is Missing

Problem: The collector log for the MSSQLDB, SYBDB, or DB2 collector indicates that a jar file is missing. If the following JDBC driver jar files are missing from the Audit Vault collection agent $ORACLE_HOME/jlib library, then this error message appears in the collector log of the respective collector being used.

  • SQL Server: sqljdbc.jar

  • Sybase ASE: jconn3.jar

  • IBM DB2: db2jcc.jar

Under other circumstances, such as when you use either the AVMSSQLDB, AVSYBDB, or AVDB2DB command-line utilities, the following error appears when the JDBC driver is not in this directory:

JDBC Driver is missing. Please make sure that the JDBC jar exists in the location specified in Audit Vault documentation.

Solution: See the following sections:

  • SQL Server: Section 2.4.1 for information about the sqljdbc.jar file

  • Sybase ASE: Section 2.5.1 for information about the jconn3.jar file

  • IBM DB2: Section 2.6.1 for information about the db2jcc.jar file

After you download and copy these JDBC drivers in place, restart the collection agent. See Section 8.12 and Section 8.9 for more information about stopping and starting the agent for Release 10.2.32 agents. For agents that were created in Release 10.2.3.1 or earlier, restart OC4J by running the avctl stop_oc4j and avctl start_oc4j commands.

A.3.4.7 Collector Unable to Connect to the Source Database

Problem: When you try to verify the ORCLDB, MSSQLDB, SYBDB, or DB2 collector using the verify command, the following error message appears:

Unable to connect to source database

Solution: This error appears if the source user that you specified in the verify command for the source database does not have sufficient privileges to connect to the source database. Check if the source user has sufficient privileges to connect to the respective database. See the following sections for information about creating a source user with sufficient privileges:

A.3.4.8 Failure of the Computer on Which a Collector Resides

Problem: The computer on which you have created a collector fails.

Solution: For collectors that retrieve audit data from the database (and not operating system files), you can collect audit trails from another host computer by moving the collector to a different computer. To do so, see the following sections:

  • Section 9.4, for the avorcldb alter_collector command; applies to the DBAUD collector only

  • Section 10.4, for the avmssqldb alter_collector command; applies to the server-side trace logs only

  • Section 11.4, for the avsybdb alter_collector command

A.3.4.9 DB2 Collector Connection Being Denied Due to Lack of License

Problem: When you run the avdb2db verify command or perform other DB2 collector-related functions, the log file may report that the connection was denied because the license is missing. This can result from having the wrong version of the IBM Data Server Driver for JDBC and SQLJ installed. You must have version 3.50 or later.

Solution: Check the version of the IBM Data Server Driver for JDBC and SQLJ driver, and if necessary, upgrade it.

To check the version of this driver, run the following command on the db2jcc.jar file:

java -cp jar_file_directory_path/db2jcc.jar com.ibm.db2.jcc.DB2Jcc -version

If the version of the driver is earlier than version 3.50, then follow the instructions in Section 2.6.1 to upgrade to the correct version.

A.3.4.10 REDO Collection Agents Terminate But No Errors Are Recorded

Problem: The REDO collection agent terminate unexpectedly, but there are no errors recorded for you to diagnose the problem.

Solution: To create a log of errors, run the Oracle Streams configuration report and health check script. See Oracle Streams Concepts and Administration for more information.

A.3.5 Troubleshooting Oracle Audit Vault Console

This section contains:

A.3.5.1 Audit Vault Console Not Appearing in the Web Browser

Problem: When you try to access the Audit Vault Console in a Web browser, it appears to hang, or after a while it times out.

Solution: This may be happening because the Audit Vault Console is down. To check the status of the Audit Vault Console, issue an avctl show_av_status command in the Audit Vault Server shell or command prompt. If the status indicates that the Audit Vault Console is down, issue the avctl start_av command in the Audit Vault Server shell or command prompt to restart it. Then start the Audit Vault Console in the Web browser. The Audit Vault Console should appear and let you log in to the management system of the Audit Vault auditor administrator.

A.3.5.2 Audit Vault Console Problem Requiring Debugging

Problem: You want to enable debug logging while trying to diagnose an Audit Vault Console problem.

Solution: Run the following commands on the command-line:

avctl stop_av
avctl start_av -loglevel debug

Then check the log output in the Audit Vault Server $ORACLE_HOME/av/log directory.

Because debugging creates more logging and writing overhead, remember to disable it when you no longer need it, as follows:

avctl stop_av
avctl start_av

See Section 8.10 and Section 8.13 for more information about these commands.

A.3.5.3 Oracle RAC Node Containing the Audit Vault Console Becomes Disabled

Problem: In an Oracle RAC environment, the node on which the Audit Vault Console is installed becomes disabled. When this node becomes unavailable, the Console does not automatically fail over to another node as the repository database does. As a result, the Audit Vault Console application is no longer available to users.

Solution: You must manually bring up the Audit Vault Console on another node in the Oracle RAC cluster. See Section 4.6 for more information.

A.3.6 Troubleshooting the Oracle Audit Vault Audit Reports

This section contains:

A.3.6.1 Oracle Audit Vault Reports Not Displaying

Problem: When you select the Reports tab in the Audit Vault Console, an error message appears saying that the report cannot be displayed.

This problem can occur in the following situations:

  • The Oracle Audit Vault installation may not have completed successfully. Check the Audit Vault Server installation log files, which are located in the $ORACLE_HOME/av/log directory. See "Location of Audit Vault Server Log and Error Files" for more information.

  • The database cannot register the Oracle listener XDB HTTP endpoint. Check the Audit Vault Server database listener status:

    lsnrctl status listener_AV_SID
    

    The output should contain the following line of text:

    ...(PORT=5707 ))(Presentation=HTTP)(Session=RAW))
    

    If it does not, then follow these steps:

    1. Log in to SQL*Plus as the XDB user or a user who has been granted the EXECUTE privilege for the DBMS_XDB PL/SQL package.

      sqlplus xdb_admin
      Enter password: password
      
    2. Run the following procedure:

      SQL> EXEC DBMS_XDB.SETHTTPPORT(5707);
      
    3. Connect as user SYS with the SYSDBA privilege, and then run the following ALTER SYSTEM statement:

      SQL> ALTER SYSTEM REGISTER;
      

    Alternatively, follow these steps:

    1. Disable Oracle Database Vault.

      See Oracle Database Vault Administrator's Guide for instructions on disabling (and re-enabling) Oracle Database Vault.

    2. Log in to SQL*Plus as user SYS with the SYSDBA privilege, and then run the following SQL statements:

      SQL> EXEC DBMS_XDB.SETHTTPPORT(5707);
      SQL> ALTER SYSTEM REGISTER;
      
    3. Re-enable Oracle Database Vault.

A.3.6.2 Oracle Audit Vault Reports Not Showing Any Data

Problem: When you generate a report, the Oracle Audit Vault reports are not showing any data, even in cases where the audit trail data can be retrieved from source database.

Solution: A report filter may be incorrectly set, or the collection agent or collectors may not be running. Try the following solutions:

  • Check the filters for the report. For example, the report may have been enabled to show only events that occurred in the last 24 hours. Disable this filter and then check that the warehouse data is being refreshed. See Oracle Audit Vault Auditor's Guide for more information.

  • If the audit trail data cannot be retrieved, ensure that the collectors are enabled. See Section 2.7 and Section 2.8 for more information.

A.3.6.3 Not Sure if Audit Data Is Appearing in the Data Warehouse

Problem: Even though the Oracle Audit Vault data warehouse is refreshed automatically with audit data, you are not sure if the data warehouse is capturing this data.

Solution: If you believe that the audit data is not being updated in the data warehouse, then check the Activity Overview Report, described in Oracle Audit Vault Auditor's Guide. If you find that data is missing, then check the server-side log files (alert and trace logs, located in the $ORACLE_HOME/av/log directory) for errors. If there are errors, then contact Oracle Support.

A.3.6.4 Advanced Alerts Unable to Fire and New Alerts Cannot Be Created

Problem: An advanced alert that uses a user-defined function is no longer able to be fired. Furthermore, you find that you cannot add any new alerts for the affected Audit Vault alert rule set, which is comprised of the category and source database type. For example, you would no longer be able to create or use alerts for the Account Management category of the Oracle Database source type. In fact, none of the alerts for the affected rule set may work. This problem can occur if the function has been changed or dropped, or the privileges for running the function have been changed. As a result, the alert is no longer valid.

Solution: Check that the function has not been modified or its privileges changed. Ensure that the AVREPORTUSER account has been granted the EXECUTE privilege for the function. Then recreate the alert to use the corrected function. Afterward, the alert and other alerts in the rule set should work correctly.

A.3.7 Troubleshooting Oracle Audit Vault in an Oracle Real Application Clusters Environment

This section contains:

A.3.7.1 avca drop_agent Command Failing

Problem: In an Oracle RAC environment, the avca drop_agent operation fails with an error when this command is issued from one of the Oracle RAC nodes.

When you try to run the avca add_agent command from one of the Oracle RAC nodes, the command fails.

Solution: In an Oracle RAC environment, you must run the AVCA commands from the node on which Oracle Enterprise Manager resides. This is the same node on which the av.ear file is deployed.

To find where the av.ear file is deployed, locate the $ORACLE_HOME/oc4j/j2ee/oc4j_applications/applications/av/av/WEB-INF/classes/av.properties file is located.

Once you locate the node, run the AVCA and AVCTL commands from that node.

If the node on which the av.ear file is deployed is down, deploy the av.ear file to another node using the avca deploy_av command. See Section 7.6 for more information about this command.

When you run the avca deploy_av command, on the other node Oracle Database creates a wallet containing the default avadmin entries. You must add the other entries, such as the source user credentials, to the wallet by using the setup command for the appropriate utility (AVORCLDB, AVMSSQLDB, AVSYBDB, or AVDB2DB), depending on the collectors being used.

To access the Audit Vault Console from this other node, enter the following URL in the Web browser:

https://host:port/av

In this specification:

  • host is the host name or IP address of the other Oracle RAC node

  • port is the port number for the Oracle RAC node