E Troubleshooting Oracle Audit Vault and Database Firewall

Topics

Troubleshooting Tips

Partial or No Traffic Seen for an Oracle Database Monitored by Database Firewall

Problem

I see no traffic, or only partial traffic, captured in reports for an Oracle Database monitored by the Database Firewall.

Solutions

Go through the following checks to find the trouble:

  1. In the Audit Vault Server, check that the report filters are set correctly, including the time slot.

  2. Check that the system time on the Database Firewall is synchronized with the time on the Audit Vault Server and the secured target system.

  3. Check that the secured target's network traffic is visible to the Database Firewall using the Live Capture utility on the firewall. See "Viewing and Capturing Network Traffic in a Database Firewall".

  4. Check that the Oracle Database service name or SID is used correctly. If you specified an Oracle Database service name in the Enforcement Point settings for this secured target, you will only see traffic for that service name. To see all traffic, remove the service name from the Enforcement Point settings to see all traffic.

    If you have entered a service name in the Enforcement Point, and see no traffic, check to see that the service name is entered correctly in the Enforcement Point settings.

    For Enforcement Points set to use DAM mode, the Database Firewall may be monitoring traffic for existing client connections to the database. Since these connections were in place before you deployed the Database Firewall, it will not be able to detect the service name you specify in the Enforcement Point. In this case, restart the client connections to the database.

    For information on Enforcement Points, see "Configuring Enforcement Points".

  5. Check that the correct Database Firewall policy is deployed. For information on editing and deploying firewall policies, see Oracle Audit Vault and Database Firewall Auditor's Guide.

RPM Upgrade Failed

Problem

An RPM upgrade failed with the following error:

error: %post(dbfw-mgmtsvr-###) scriptlet failed, exit status 1

Solution

  1. Check that there is at least 10MB of free /tmp space.

  2. Remove the new RPM:

    rpm -e dbfw-mgmtsvr-###

  3. Retry the upgrade.

Agent Activation Request Returns 'host is not registered' Error

Problem

I used the following two commands to register the Audit Vault Agent's host computer (where the agent is deployed), and to request Audit Vault Agent activation:

From the Audit Vault Server:

avcli> register host 'host_name'

From the host computer:

agentctl activate

But the agentctl activate command returns: Agent host is not registered

Solution

Your agent host may be multi-homed. In this case, the agent hostname to IP address resolution may resolve to the NIC/IP that is not used by the agent while connecting to the AV server. To resolve this issue, try to register the agent host using the with ip option and then try activating the agent again.

From the Audit Vault Server, use the following command:

avcli> register host 'host_name' with ip 'host_ip_address'

If you still have issues, try finding the IP address used in the database session when you connect to the Audit Vault server from the agent host, using these commands:

sqlplus username/password@"(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=Audit_Vault_Server_IP)(PORT=1521))(CONNECT_DATA= (SERVICE_NAME=dbfwdb)))"

sqlplus> select SYS_CONTEXT('USERENV','IP_ADDRESS') from dual;

Use the IP address from the above query to register your host.

Unable to Deploy Agent on the Secondary Audit Vault Server

Problem

When I try to deploy the Audit Vault Agent on the secondary Audit Vault Server in a high availability pair, I get an error that the host is not registered.

Cause

After you pair two Audit Vault Servers for high availability, you do all configuration on the primary server in the pair only, including Audit Vault Agent deployment. See "Step 3: Start High Availability Pairing of the Audit Vault Servers".

Operation Fails When I Try to Build Host Monitor or Collect Oracle Database Trail

Problem

This problem may manifest with various symptoms:

  • When I try to build a host monitor, the operation fails or cannot find the correct binaries.

  • When I try to collect audit data from an Oracle Database secured target, the operation fails.

  • The Audit Vault Agent cannot connect to the Audit Vault Server.

  • Audit trail does not start.

Solution

  1. Unset all environment variables except the following:

    • PATH

    • TERM

    • PS1

    • LANG

    • LC_*

    • JAVA_HOME

    Then run the java -jar agent.jar command again on the host machine. For instructions, see "Deploying the Audit Vault Agent on the Host Computer".

  2. If you deployed the Audit Vault Agent in a Linux environment, ensure that the host machine name is present in the /etc/hosts file.

'java -jar agent.jar' Failed on Windows Machine

Problem

The command java -jar agent.jar failed on my Windows secured target machine, and I noticed in the log files that the Audit Vault Agent services installation/un-installation failed.

Solution

  1. Follow the instructions for unregistering the agent in "Registering or Unregistering the Audit Vault Agent as a Windows Service".

    If Method 1 fails, then try Method 2.

  2. Run the java -jar agent.jar command again.

Unable to Un-install the Audit Vault Agent Windows Service

Follow the instructions for unregistering the agent in "Registering or Unregistering the Audit Vault Agent as a Windows Service".

If Method 1 fails, then try Method 2.

Access Denied Error While Installing Agent as a Windows Service

Problem

I got an error during installation of the Audit Vault Agent on Windows, and I noticed the following error in the AGENT_HOME\av\log\av.agent.prunsvr log file:

[2013-05-02 11:55:53] [info] Commons Daemon procrun (1.0.6.0 32-bit) started
[2013-05-02 11:55:53] [error] Unable to open the Service Manager
[2013-05-02 11:55:53] [error] Access is denied.
[2013-05-02 11:55:53] [error] Commons Daemon procrun failed with exit value:
7 (Failed to )
[2013-05-02 11:55:53] [error] Access is denied. 

Solution

The above message means that the logged in user does not have privileges to install the Audit Vault Agent as a Windows Service. If you get the above message, try launching the command shell with the Run As Administrator option, and then execute java -jar agent.jar in that command shell.

Unable to Start the Agent Through the Services Applet On The Control Panel

Problem

I did the following:

  1. Installed the Audit Vault Agent using the java -jar agent.jar command.

  2. Activated the Audit Vault Agent.

  3. Started the Audit Vault Agent using the agentctl start -k key command.

    The agent started up and is in RUNNING state.

  4. Stopped the Audit Vault Agent.

  5. Tried to start the Audit Vault Agent using the Services Applet on the Windows Control Panel.

    The Audit Vault Agent errored out immediately.

Solution

This means that the Audit Vault Agent is configured to use a Windows account that does not have privileges to connect to the Audit Vault Server.

Take the following steps:

  1. Go to Control Panel, then to Services Applet.

  2. Select the Oracle Audit Vault Agent service.

  3. Right click and select the Properties menu.

  4. Click the Log on tab.

  5. Select This account: and then enter a valid account name and password.

  6. Save and exit.

  7. Start the Audit Vault Agent through the Services Applet.

Error When Starting the Agent

Problem

After I installed the Audit Vault Agent, I set the username and password in the OracleAVAgent Windows Service Properties Log On tab. However, when I try to start the OracleAVAgent service, I see the following error in the
Agent_Home\av\log\av.agent.prunsvr.date.log file:

[info]  Commons Daemon procrun (1.0.6.0 32-bit) started
[info]  Running 'OracleAVAgent' Service...
[info]  Starting service...
[error] Failed creating java 
[error] ServiceStart returned 1
[info]  Run service finished.
[info]  Commons Daemon procrun finished

Solution

This means that the OracleAVAgent service is not able to launch the Java process. Try the following:

  1. Uninstall all JDKs and/or JREs in the system.

  2. Reinstall JDK SE or JRE and then start the OracleAVAgent service.

  3. If this doesn't help, you can install 32 bit JDK SE or JRE and then start the OracleAVAgent service.

Error When Running Host Monitor Setup

Problem

I am setting up a Host Monitor. When I run the command bin/hostmonsetup install, the following error is displayed:

[root@dbsec1 av]# bin/hostmonsetup install
/usr/bin/ld: cannot find -lpcap
collect2: ld returned 1 exit status
make: *** [hostmonitor] Error 1
Line 105: Failed to generate executables for Host monitor.

Solution

This means the host computer does not have the required libraries for the host monitor. Install the required libraries listed in "Prerequisites for Host Monitoring".

Alerts on Oracle Database Secured Target are not Triggered for a Long Time

Problem

I configured an Oracle Database secured target to audit to XML files, configured an audit trail in Oracle AVDF of type DIRECTORY, and then configured an alert to trigger on certain events. My alert did not get triggered for a long time.

Solution

This issue can occur if the Oracle Database secured target is not flushing the audit records to the file immediately. Contact Oracle Support in order to access support note 1358183.1 Audit Files Are Not Immediately Flushed To Disk.

Internal capacity exceeded messages seen in the /var/log/messages file

Problem

Not all the expected traffic is being captured or logged by the DBFW, and error messages are present in the /var/log/messages file containing the text "Internal capacity exceeded".

Solution - 1

Increase the processing resources available for the Secured Target on which the issue is observed through the setting of the MAXIMUM_ENFORCEMENT_POINT_THREADS collection attribute. For more information, refer to "Registering Secured Targets"

Solution - 2

The size of the buffer used for inter-process communication on the DBFW can be increased to improve throughput, though at the cost of more memory being allocated by the relevant processes. Please note that this setting is in units of Megabytes, and has a default value of 16. To change the configuration for this value execute the following procedure:

  1. Log in to the DBFW console as the root user.

  2. Edit the file /usr/local/dbfw/etc/dbfw.conf. Look for an entry with the key IPC_PRIMARY_BUF_SIZE_MB. If it exists, this is the line to change. If it does not exist, add a new line beginning with IPC_PRIMARY_BUF_SIZE_MB.

  3. Change the IPC_PRIMARY_BUF_SIZE_MB line to reflect the required buffer size. For example, if you wished to change the buffer size to 24 megabytes, the configuration line should be IPC_PRIMARY_BUF_SIZE_MB="24". Save the changes.

  4. From the command line restart the DBFW processes so that the new setting is used with the command line /etc/init.d/dbfw restart.

There is also a second setting available to alter the maximum size that the inter-process communication buffer can grow to. It's units are in megabytes, and has a default value of 64 megabytes. To change the configuration for this value execute the following procedure:

  1. Log in to the DBFW console as the root user.

  2. Edit the file /var/dbfw/va/N/etc/appliance.conf, where N is the number of the enforcement point in question. Look for an entry with the key IPC_BUF_SIZ_MB. If it exists, this is the line to change. If it does not exist, add a new line beginning with IPC_BUF_SIZ_MB.

  3. Change the IPC_BUF_SIZ_MB to reflect the desired maximum buffer size. For example, if you wished to change the buffer size to 80 megabytes, the configuration line should be IPC_BUF_SIZ_MB="80". Save the changes.

  4. From the command line restart the DBFW processes so that the new setting is used with the command line /etc/init.d/dbfw restart.

If the problem persists and after altering the above settings the Internal capacity exceeded error is still encountered, then further investigation by support is required. Perform the following:

  1. Log in to the DBFW console as the root user.

  2. Edit the file /usr/local/dbfw/etc/logging.conf

  3. Find the line log4j.logger.com.oracle.dbfw.Metrics=ERROR

  4. Comment out this line by placing a # character at the beginning of the line log4j.logger.com.oracle.dbfw.Metrics=ERROR. Save the changes.

  5. From the command line restart the DBFW processes so that the new setting is used with the command line /etc/init.d/dbfw restart

  6. Leave the DBFW running for several hours under load even while the Internal capacity exceeded error is still encountered.

  7. After this period, get the diagnostics output from the DBFW as detailed in MOS note How to Collect Diagnostic Logs From Audit Vault Server (Doc ID 2144813.1). Provide the diagnostics output to support for further analysis.