| Oracle® Audit Vault Administrator's Guide Release 10.2.3 Part Number E11059-03 |
|
|
View PDF |
| Oracle® Audit Vault Administrator's Guide Release 10.2.3 Part Number E11059-03 |
|
|
View PDF |
This chapter provides troubleshooting information for administering an Audit Vault system. This chapter includes the following sections:
Table 6-1 shows the names and a description of the Audit Vault Server log and error files located in the Audit Vault Server $ORACLE_HOME/av/log directory. These files contain important information regarding the return status of commands and operations that will be useful in diagnosing problems should they occur. Log files can be deleted at any time, except for the avca.log file, which can only be deleted when the Audit Vault Server is shut down.
Table 6-1 Name and Description of Audit Vault Server Log and Error Files
| File Name | Description |
|---|---|
|
|
Contains a log of errors encountered in collection agent initialization. This file can be deleted at any time. |
|
|
Contains a log of all primary collection agent-related operations and activity. This file can be deleted at any time. |
|
|
Contains a log of all AVCA commands that have been run and the results of running each command. This file can only be deleted after Audit Vault Server is shut down. |
|
|
Contains a log of the collection agent operations and any errors returned from those operations. The |
|
|
Contains a log of all AVORCLDB commands that have been run and the results of running each command. This file can be deleted at any time. |
|
|
Contains a log of all AVMSSQLDB commands that have been run and the results of running each command. This file can be deleted at any time. The |
If you, as the Audit Vault Administrator, need to do Audit Vault Console troubleshooting, you must enable Enterprise Manager logging. You must modify the following emomslogging.properties file in the Audit Vault Server home: $ORACLE_HOME/sysman/config/emomslogging.properties on Linux or UNIX systems or ORACLE_HOME\sysman\config\emomslogging.properties on Windows systems and add the following lines of information:
log4j.appender.avAppender=org.apache.log4j.RollingFileAppender
log4j.appender.avAppender.File=<$ORACLE_HOME>/oc4j/j2ee/OC4J_DBConsole__/log/av-application.log
log4j.appender.avAppender.Append=true
log4j.appender.avAppender.MaxFileSize =20000000
log4j.appender.avAppender.Threshold = DEBUG
log4j.appender.avAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.avAppender.layout.ConversionPattern=%d [%t] %-5p %c{2} %M.%L - %m\n
log4j.category.oracle =DEBUG, avAppender
This information can be used to debug communication issues between the server and the collection agents.
Table 6-2 shows the names and a description of the Audit Vault Collection Agent log and error files located in the Audit Vault Collection Agent $ORACLE_HOME/av/log directory. These files contain important information regarding the return status of commands and operations that will be useful in diagnosing problems should they occur.
Table 6-2 Name and Description of Audit Vault Collection Agent Log and Error Files
| File Name | Description |
|---|---|
|
|
Contains a log of all errors encountered in collection agent initialization and operation. This file can be deleted at any time. |
|
|
Contains a log of all primary collection agent-related operations and activity. This file can be deleted at any time. |
|
|
Contains a log of all AVCA commands that have been run and the results of running each command. This file can be deleted at any time. |
|
|
Contains a log of all AVORCLDB commands that have been run and the results of running each command. This file can be deleted at any time. |
|
|
Contains a log of collection operations for the DBAUD, OSAUD, and MSSQLDB collectors. This file can only be deleted after Audit Vault Collection Agent OC4J is shut down. To increase the log level, you must restart the agent OC4J on the Collection Agent side with the appropriate debug level. |
|
|
Contains a log of the collection agent operations and any errors returned from those operations. The |
|
|
Contains a log of all AVMSSQLDB commands that have been run and the results of running each command. This file can be deleted at any time. The |
|
|
Contains a log of SQL*Net information. |
The directory Audit_Vault_Agent_Home/oc4j/j2ee/home/log contains the logs generated by the collection agent OC4J. In this directory, the file AVAgent-access.log contains a log of requests the collection agent receives from the Audit Vault Server. This information can be used to debug communication issues between the server and the collection agent.
Failed configuration commands are located in the Audit Vault Collection Agent $ORACLE_HOME/cfgtoollogs directory, which includes the file, configToolFailedCommands. This file contains just the name of the failed command. See the avca.log or avorcldb.log file for additional information, including any associated errors and error messages.
This section describes a number of troubleshooting scenarios that you might encounter with some of the Audit Vault components and how try to resolve each one. The scenarios are placed in the following groupings:
This section describes Audit Vault Server problems that you might encounter.
Problem: Best way to tune Audit Vault Server performance when using the REDO collector.
Following an Audit Vault Server installation, the streams_pool_size initialization parameter is set to 150 MB. This parameter must be tuned to maximize REDO collector performance if you are going to be using this collector. In an Oracle Real Application Clusters (Oracle RAC) environment, this parameter must be tuned on all nodes because it is uncertain where the queue will be particularly after an instance startup.
Typically, once a REDO collector is configured and started, let it run for a while. This will allow the autotuning feature of Oracle Database to allocate memory for the best database performance for the streams_pool_size parameter. Using AWR, check to see if STREAMS AQ has a flow control issue – enqueue being blocked. In the event that you notice that the performance, for example, is only 500 records being applied per second, it may become necessary to manually tune this parameter.
Assuming that you have at least 1 GB of physical memory in your Audit Vault Server system, set this parameter to 200 MB using the SQL command ALTER SYSTEM SET STREAMS_POOL_SIZE=200;. Monitor the performance again using AWR. You should achieve a record apply rate of 2000 records per second, which is a typical maximum rate for the REDO collector. Usually, setting the value to 200 MB should be sufficient. If you are using Oracle Audit Vault in an Oracle RAC environment, set this parameter value accordingly on all nodes in the cluster. Use the SQL command ALTER SYSTEM SET STREAMS_POOL_SIZE=200 SID=avn;, where n is the number portion of the SID for each node in the cluster, for example, av2, av3, av4, and so forth, if that is your naming convention.
This section describes Audit Vault Collection Agent problems that you might encounter.
Problem: Audit Vault Agent Status is Blank on the Windows Services Panel
After installing Audit Vault Agent for Windows (32-bit), configuring a source and collectors, then starting the agent on the Audit Vault Server side, a check of the Services Panel on the Windows system where the Audit Vault Agent resides shows that the status is blank, rather than Started.
This is normal behavior for the Audit Vault Agent on Windows systems because the service is a short-lived process. Once the Agent service process completes its task, it exits, so the status of the service will not show as "Started", however, the Audit Vault Agent is running fine.
You can run the AVCTL show_agent_status command to check the status of the Audit Vault Agent. For example, on a Windows system:
C:\>avctl show_agent_status -agentname agent1 AVCTL started Getting agent metrics... -------------------------------- Agent is running -------------------------------- Metrics retrieved successfully.
Problem: Need to debug a Collection Agent problem
In trying to diagnose an Audit Vault Collection Agent problem, it would be nice to be able to turn on debug logging.
This can be done by performing the following set of AVCTL commands on the command-line:
avctl stop_oc4j avctl start_oc4j -loglevel debug
Then check the log output in the Audit Vault Agent Oracle_home/av/log directory on Linux and UNIX systems or in the Oracle_home\av\log directory on Windows systems.
Turning on debug logging creates more logging and writing overhead, so be sure to turn off debug logging when you are ready to do so by performing the following set of AVCTL commands on the command-line:
avctl stop_oc4j avctl start_oc4j
See the AVCTL stop_oc4j and start_oc4j commands for more information.
Problem: The Agent OC4J or Audit Vault Console OC4J fails to start
After issuing an AVCTLstart_oc4j command, an AVCTL show_oc4j_status command shows that the agent OC4J is not running. Or, after issuing an AVCTL start_av command, an AVCTLshow_av_status command shows that the Audit Vault Console OC4J is not running.
Go to $ORACLE_HOME/av/log/agent.err log file to see what error message appears in the log.
Or, go to $ORACLE_HOME/oc4j/j2ee/home and issue the following command to see what error message appears on the console:
java -jar oc4j.jar
This problem is most likely caused by a port conflict. For example, if the problem is caused by an RMI port conflict, you would see the following message in the console:
D:\oracle\product\10.2.3\avagentrc3_01\oc4j\j2ee\home>java -jar oc4j.jar 08/05/16 10:39:51 Error starting ORMI-Server. Unable to bind socket: Address already in use: JVM_Bind
Three ports are needed to start the agent OC4J or Audit Vault Console OC4J: RMI, JMS, and HTTP. A port conflict with any of these ports can cause the agent OC4J or Audit Vault Console OC4J to fail to start or the agent service of Audit Vault Console to become unavailable. If there is a port conflict with any of these ports, each of these ports can be modified in the following files at $ORACLE_HOME/oc4j/j2ee/config by selecting a port number not in use:
rmi.xml
jms.xml
http-web-site.xml or (av-agent-web-site.xml)
Problem: The setup command returned an error message that the connection to the source database using the credentials in the wallet was not successful
This problem is most likely due to entering an incorrect user name or password or both when issuing the setup command using either the AVORCLDB or AVMSSQLDB command-line utilities.
Reissue the setup command again using the correct credentials.
This section describes Audit Vault collector problems that you might encounter.
Problem: Cannot start the DBAUD collector and the log file shows an error
The DBAUD Collector log file (in the Audit Vault Collection Agent home) shows the following entry:
INFO @ '17/08/2007 15:05:48 02:00': Could not call Listener, NS error 12541, 12560, 511, 2, 0
In configuring the source and collectors, the last step can be overlooked. This is Step 6 in Section 2.2 about running the avorcldb setup command in the Audit Vault Collection Agent home. Overlooking this step, will cause the DBAUD collector from starting.
To verify that this is the problem, set your environment variables for the Audit Vault Collection Agent shell (ORACLE_HOME, PATH, and LD_LIBRARY_PATH).
Change directories to the network/admin directory:
Perform the cat command on your tnsnames.ora file. There should be an entry something like SRCDB1. If there is no SRCDB1 entry in your tnsnames.ora file, run the avorcldb setup command as shown in Step 6 as previously referenced.
Next, try to connect to the source database with the following command:
sqlplus /@SRCDB1
If the connection is successful, then your source database is set up correctly. Try starting the DBAUD collector (AVCTL start_colletor command).
Problem: Not sure if the DBAUD_Collector or OSAUD_Collector collectors are collecting from the AUD$ table and the OS file, respectively
After you set up both the DBAUD_Collector and OSAUD_Collector collectors, you want to check to see that they are collecting from the AUD$ table and OS file, respectively.
To see if DBAUD_Collector is collecting from the AUD$ table, check the contents of the DBAUD_Collector_<source-name_prefix><source-id>.log file in the Audit Vault Collection Agent home /av/log directory.
To see if OSAUD_Collector is collecting from the OS File, check the contents of the orcldb_osaud_<source name>.log file in the Audit Vault Collection Agent home /av/log directory.
Bring each file into an editor and search for entries that indicate that the collector is collecting audit records.
For example, entries like these would be found in the DBAUD_Collector log file:
***** Started logging for 'AUD$ Audit Collector' *****
.
.
.
INFO @ '25/01/2007 19:08:42 -8:00':
***** SRC connected OK
INFO @ '25/01/2007 19:08:53 -8:00':
***** SRC data retireved OK
.
.
.
For example, an entry like this would be found in the OSAUD_Collector log file:
File opened for logging source "DBS1.REGRESS.RDBMS.DEV.US.ORACLE.COM" INFO @ '24/01/2007 18:16:18 -8:00': ***** Started logging for 'OS Audit Collector' *****
If everything looks OK, close the editor, then refresh the warehouse using the AVCTL refresh_warehouse command in the Audit Vault Server shell. When this operation completes, log in to the Audit Vault Console as the Audit Vault auditor and examine the graphical summary named Activity by Audit Event Category on the Overview page for the appearance of additional audit records in the various event categories. Increased counts for the various event categories indicate that these collectors are collecting audit records.
Problem: ORA-01017:invalid username/password; logon denied error when starting up the DBAUD_Collector or setting up the REDO_Collector
When you try to start up the DBAUD_Collector or set up the REDO_Collector, you get an ORA-01017: invalid username/password; logon denied error.
This error is likely due to a problem with your user name or your password or both in the password file as well as a problem with the wallet. Try re-creating the user name and password. If the problem persists, re-create the password file. If this does not fix the problem, add the source user to the wallet again using the AVORCLDB setup command. Ensure that it is the same user name and password that you are using on the source database.
Problem: Collector log for MSSQLDB collector indicates a jar file is missing
When the JDBC Driver (sqljdbc.jar) for the SQL Server source database, cannot be found in the Collector Agent home/jlib directory, this error will appear in the collector log of the collector being used. Under other circumstances, such as when using the AVMSSQLDB command-line utility, the following error is returned when the JDBC Driver is not located in this directory:
SQLException3, "JDBC Driver is missing. Please make sure that the JDBC jar exists in the location specified in Audit Vault documentation."
See Step 1 in Section 2.3 for information about the sqljdbc.jar (JDBC Driver) for the Microsoft SQL Server source database. Once the JDBC Driver is in place, you must restart the Agent OC4J. See the steps to follow in Stopping and Starting the Agent OC4J in Oracle Audit Vault Collection Agent Installation Guide for more information.
Problem: Unable to connect to source database
When trying to verify either the, ORCLDB or MSSQLDB collector using the verify command, you may be returned the following error "Unable to connect to source database".
This error can be returned if the source user you specified in the verify command for either the Oracle Database source or SQL Server source database source database does not have sufficient privileges to connect to the source database. Check and see if the specified source user has sufficient privileges to connect to the respective database. See Step 2 in Section 2.2 for information about creating and granting an Oracle Database source user sufficient privileges to access the database. See Step 2 in Section 2.3 for information about creating and granting a SQL Server database source user sufficient privileges to access the database.
This section describes Audit Vault Console problems that you might encounter.
Problem: Audit Vault Console does not come up in the Web browser
When you try to bring up the Audit Vault Console in a Web browser, it appears to hang, or after a while it times out.
This may be happening because Audit Vault Console is down. To check the status of Audit Vault Console, issue an AVCTL show_av_status command in the Audit Vault Server shell. If the status indicates that the Audit Vault Console is down, issue an AVCTL start_av command in the Audit Vault Server shell to get it started again. Then start up the Audit Vault Console in the Web browser. The Audit Vault Console should appear and let you log in to the Audit Vault auditor's or administrator's management system.
Problem: Need to debug an Audit Vault Console problem
In trying to diagnose an Audit Vault Console problem on the Audit Vault Server, it would be nice to be able to turn on debug logging.
This can be done by performing the following set of AVCTL commands on the command-line:
avctl stop_av avctl start_av -loglevel debug
Then check the log output in the Audit Vault Server Oracle_home/av/log directory on Linux and UNIX systems or in the Oracle_home\av\log directory on Windows systems.
Turning on debug logging will degrade performance, so be sure to turn off debug logging when you are ready to do so by performing the following set of AVCTL commands on the command-line:
avctl stop_av avctl start_av
See the AVCTL stop_av and start_av commands for more information.
This section describes some problems that you might encounter when you run Audit Vault in an Oracle Real Application Clusters (Oracle RAC) environment.
Problem: In an Oracle RAC environment, the AVCA drop_agent operation fails with an error when this command is issued from one of the Oracle RAC nodes
When you try to issue an AVCA add_agent command from one of the Oracle RAC nodes, the command fails.
In an Oracle RAC environment, AVCA commands must be issued from the node on which Oracle Enterprise Manager resides. This is the same node on which the av.ear file is deployed.
In an Oracle RAC environment, AVCA and AVCTL commands can be issued only from the node where the av.ear file is deployed.
To see where the av.ear file is deployed, check to see where the following file is located: $ORACLE_HOME/oc4j/j2ee/oc4j_applications/applications/av/av/WEB-INF/classes/av.properties
Once you locate the node, run all AVCA and AVCTL commands from that node.
If the node on which the av.ear file is deployed is down, deploy the av.ear file to another node using the AVCA deploy_av command. The command syntax is as follows:
deploy_av -sid <sid> -dbalias <db alias>
-avconsoleport <av console port>
In this example:
-sid <sid> is the Oracle system identifier (SID) for the instance.
-dbalias <db alias> is the database alias.
-avconsoleport <av console port> is the port number for the Audit Vault Console.
Note that when the AVCA deploy_av command is issued, a wallet containing the default avadmin entries is also created on the other node. However, other entries, such as the source user credentials must be added to the wallet using the setup command for the command-line interface (AVORCLDB or AVMSSQLDB) being used that matches the collectors that are in use.
To use the AV Console from this other node, enter its host name or IP address (<host>) and port number (<port>) as you did previously in the Address field of the browser window (http://<host>:<port>/av), but replace the original host name or IP address with that for the other node.
|
 Copyright © 2007, 2008, Oracle. All rights reserved. Legal Notices |
|