1. Installation Guide
  2. Troubleshooting Oracle Audit Vault and Database Firewall

A Troubleshooting Oracle Audit Vault and Database Firewall

Oracle Audit Vault and Database Firewall provides troubleshooting advice for a range of installation or upgrade scenarios.

A.1 Information to Provide Support When Filing a Service Request

Review this list of information to provide support when filing a service request.

Note:

Diagnostics data, especially trace files, often contains sensitive information. Protect it accordingly and only gather and send the information that's required.
  • Oracle AVDF version, including any installed bundle patches
  • If virtualization is being used? If so, which one?
  • How much physical memory is available to Audit Vault Server and Database Firewall appliances?
  • How much disk space was available with the initial installation?
  • Did you add any SAN storage and in that case how much disk space?
  • Provide any relevant details about the brand and model of the hardware being used. This is relevant if you have specific issues relating to booting from the installation media.
  • Host OS for the secured target database and version, this is relevant for checking agent compatibility issues.
  • Brand of the secured target database, such as Oracle, MySQL, SQL Server, etc.
  • Version of the secured target database, including PSU and other one-off patches.
  • Upload the alert.log file of the secured target database.
  • From any Oracle secured target database provide the output of:
    • show parameter audit
    • opatch lsinventory -patch -detail
    • If unified auditing was configured (for some versions of Oracle database only)
    • Audit Trail type that is being configured and all relevant attributes
  • Detailed diagnostic information for Audit Vault Server, see Downloading Detailed Diagnostics Reports for Oracle Audit Vault Server
  • If requested by Oracle Support, diagnostic information from Oracle Trace File Analyzer. See Using Oracle Trace File Analyzer (TFA).
  • Information about Database Firewall:
    • Detailed diagnostic info for Database Firewall, see Viewing the Status and Diagnostics Report for Database Firewall
    • How many Network Interface Cards are installed in the database firewall appliance?
    • Is the enforcement point using default password enumeration (DPE) or database activity monitoring (DAM)? If so is it bridge, span, or proxy?
    • Do you use VLAN tagging? There are restrictions for support of VLANs.
  • For installation issues, diagnostic files related to the installation. See Collecting Logs to Debug Installation Failures.

Before contacting support, the Audit Trail Transaction Log should follow these guidelines:

  • The user setup script must be run with the argument REDO_COLL
  • The secured target database must be configured with ARCHIVELOG
  • The streams recommended patches must be applied to the secured target db: Streams Recommended Patches (Doc ID 437838.1)
  • global_name must be fully qualified (select global_name from global_name;)
  • Parameter global_names = true is recommended
  • If errors happen on capture or apply side please check respective alert.logfiles as you would do with any Streams related issue (av log will show only limited information for this audit trail type)

Related Topics

A.2 Error When Installing Audit Vault Server in Releases 20.1 to 20.3

Learn how to resolve an error observed when installing Audit Vault Server 20.1, 20.2, or 20.3.

Problem

An error is observed when installing Audit Vault Server. This is observed only in Oracle AVDF releases 20.1 to 20.3.

Solution

The Audit Vault Server installer (ISO) file is split into 3 parts or files in Oracle AVDF releases 20.1 to 20.3. All the three ISO files have to be concatenated to get a single Audit Vault Server 20.x ISO (avdf-install.iso) before proceeding with installation.

Refer to Downloading and Verifying Oracle AVDF Software for complete information.

Starting with Oracle AVDF 20.4, there is a single Audit Vault Server ISO file and there is no need to concatenate.

A.3 Conflicting Data on Storage Added to Oracle AVDF

Learn how to remove existing conflicting data from storage before adding it to Oracle Audit Vault and Database Firewall (Oracle AVDF).

Problem

The preexisting file system, Logical Volume Manager (LVM), or device mapper metadata may conflict with Oracle AVDF functionality. This may result in patch, upgrade, or installation failure.

Symptoms

The symptoms of any preexisting LVM or other device mapper metadata include, but are not limited to, the following:

  • Two vg_root volume groups.
  • Hard drive devices that become unavailable during patching, upgrade, or installation. This may lead to input or output errors and eventually result in patch, upgrade, or installation failure.

Solution

Caution:

This will erase data from the drive.
  1. Download the latest Oracle Linux 8 ISO image from Oracle Linux Downloads.

  2. Boot into rescue mode.

    1. Load the Oracle Linux 8 ISO onto your appliance and boot.

      The installation menu displays the following options:

      Install Oracle Linux 8.x.x
Test this media & install Oracle Linux 8.x.x
Troubleshooting

    2. Press the Down Arrow to select Troubleshooting, and press Enter.

      The troubleshooting menu displays the following options:

      Install Oracle Linux 8.x.x in basic graphics mode
Rescue a Oracle Linux system

    3. Press the Down Arrow to select Rescue a Oracle Linux system, and press Enter.

      The rescue menu displays the following options:

      1) Continue
2) Read-only mount
3) Skip to shell
4) Quit (Reboot)
    4. Type 3 (Skip to shell), and press Enter.
    5. Press Enter again to open the shell prompt.

  3. To discover the attached storage, enter the lsblk command at the shell prompt.

    For example:

    sh-4.4# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0         7:0    0 745.4M  1 loop
loop1         7:1    0     4G  1 loop
├─live-rw   253:0    0     4G  0 dm   /
└─live-base 253:1    0     4G  1 dm
loop2         7:2    0    32G  0 loop
└─live-rw   253:0    0     4G  0 dm   /
sda           8:0    0   256G  0 disk
└─sda1        8:1    0   256G  0 part
sr0          11:0    1  11.2G  0 rom  /run/install/repo
sr1          11:1    1  1024M  0 rom

  4. To wipe the drive, enter the wipefs command at the shell prompt.

    Enter wipefs --help to see a complete list of options.

    For example, to wipe the /dev/sda drive, enter the following command: 

    sh-4.4# wipefs --all /dev/sda

    The command output lists the changes. For example:

    /dev/sda: 2 bytes were erased at offset 0x000001fe (dos): 55 aa
/dev/sda: calling ioctl to re-read partition table: Success

  5. To safely power off, enter the sync command at the shell prompt, followed by poweroff. 

    sh-4.4# sync
sh-4.4# poweroff
  6. After you wipe the drive, eject the ISO and restart the installation.

A.4 EFI Related Error When Installing Audit Vault Server on VMware

Learn how to resolve EFI related error when installing Audit Vault Server on VMware.

Problem

The following possible errors are observed when attempting to install Audit Vault Server on VMware:


EFI Virtual disk (0.0) … unsuccessful.
EFI VMware Virtual SATA CDROM Drive (0.0) … unsuccessful.
EFI Network …

Solution

There are important prerequisites to be followed while installing Audit Vault Server on VMware:

  • You must set VMX configuration parameter disk.EnableUUID to TRUE. This must be done to enable proper mounting of disks. Without this setting, the Audit Vault Server installation on VMware will fail.

  • You must set your virtual machine to use EFI boot. In some versions of VMware this is done by selecting the VM Options tab, then expanding Boot Options, and then choose EFI in the Firmware field. You must disable secure boot. Do not select the checkbox Enable UEFI secure boot field.

    This EFI boot setting is required only for fresh installation of Audit Vault Server specifically when the disk size is more than 2TB. This setting is not required for upgrade.

Note:

See Installing Audit Vault Server on VMware for complete information.

A.5 Cannot Access the Audit Vault Server Console

Learn the workaround for when you cannot access the Audit Vault server user interface or console.

Problem

The Audit Vault Server console is not accessible.

Solution

There are two remedies that you can perform depending on when this problem occurs:

  • The problem occurs immediately after Audit Vault Server installation.

    In this case, the installation may not have been completed correctly. Perform the installation again.

  • The problem occurs after the system is already running.

    In this case, check that the disk is not full and that the Oracle Audit Vault Server database is running using this command:

    /etc/init.d/dbfwdb status

    To restart the database, use run this command as root:

    /etc/init.d/dbfwdb start

    If you have a problem restarting the database, then contact Oracle Support.

A.6 Collecting Logs to Debug Installation Failures

You can collect logs to debug issues when installing Oracle Audit Vault and Database Firewall.

A.6.1 Collecting Logs for Base Operating System Installation Issues

Use these steps to collect logs for failures that happen during the installation of the base operating system (pre- or post-reboot).

Collecting logs for debugging pre-reboot installation failures

  1. During installation or upgrade, after mounting the .iso file, press Tab to interrupt the normal boot process.
  2. To collect logs, the installer must run with command line access. To enable command line access, remove the noshell from the boot option.

  3. After the failure occurs, use one of the following keyboard shortcuts to access the command line:

    • Starting with Oracle AVDF 20.9 (Oracle Linux 8), press Ctrl+B and then press 2.
    • For installing Oracle AVDF 20.1 to 20.8 (Oracle Linux 7), press Alt+Right Arrow.

  4. Run one of the following commands to start the collection tool:

    • Starting with Oracle AVDF 20.9 (Oracle Linux 8), use the following command:

      /usr/libexec/platform-python /run/install/repo/collect_diagnostics.py

      For Oracle AVDF 20.1 to 20.8 (Oracle Linux 7), use the following command:

      python /run/install/repo/collect_diagnostics.py
  5. Follow the instructions to collect the diagnostics file.

Collecting logs for debugging post-reboot installation failures

  1. Using the password you have previously set, log in as root on the console or using SSH.

  2. Run one of the following commands to start the collection tool:

    • Starting with Oracle AVDF 20.9 (Oracle Linux 8), use the following command:

      /usr/libexec/platform-python /media/avdf-install/collect_diagnostics.py

    • For Oracle AVDF 20.1 to 20.8 (Oracle Linux 7), use the following command:

      python /media/avdf-install/collect_diagnostics.py
  3. Follow the instructions to collect the diagnostics file.

Transferring the log file for analysis

After following the instructions to collect the logs for pre- or post-reboot failures, the collection tool should have created a log or diagnostic file in the following location:

/root/install-diagnostics.tgz

  1. Follow the instructions at the prompt to transfer the log file for analysis. Use the following command:

    scp /root/install-diagnostics.tgz <user>@<Ip address>:<Path>

  2. You may also perform the following steps and commands to configure the network:

    ip addr add <IP address>/<sub net> dev <interface>
    ip link set <interface> up
    ip route add default via <gateway>
  3. Use the information available in the log file to analyze the issue and then try the installation again after addressing the issue.

A.6.2 Collecting Logs for Oracle AVDF Installation Issues

Use these steps to collect logs for failures that happen when installing Oracle AVDF.

  1. At the install start screen, press Tab (and delete the word "noshell").
  2. Press Enter to begin the installation.

  3. After the installation begins, press Ctrl+B and then press 2.

    The login screen should appear even if the installation fails.

  4. Use tar or Gzip to collect the following logs:
    • /var/log
    • /var/lib/oracle/diag
    • /var/lib/oracle/oraInventory/logs
    • /tmp

  5. Collect the following configuration files:

    • /etc/sysconfig/avdf
    • /var/lib/avdf/system_history.yaml
    • /usr/local/dbfw/etc/dbfw.conf

  6. Collect the output from the following commands:

    1. su root
    2. rpm -qa avs
    3. ls -lrt /var/log/installation-*
    4. ls -lrt /var/log/upgrade-*
    5. df -h
    6. du -sh /var/lib/oracle/19.7.0.0.0
    7. du -sh /var/lib/oracle/19.7.0.0.0/grid
    8. cat /proc/meminfo

  7. Collect the output from the following commands:

    1. su root
    2. hostname
    3. cd /var/lib/oracle/diag
    4. ls -lrt
    5. cd crs
    6. ls -lrt
    7. hostname
    8. cd <hostname>
    9. ls -lrt
    10. cd crs
    11. ls -lrt

A.7 Unable to Reach Gateway Error

Learn to fix incorrect Gateway details entered during installation.

Problem

Incorrect or invalid Gateway details entered while installing Audit Vault Sever or Database Firewall. The following error message may be encountered:

Gateway is not reachable from host

Solution

The Gateway details can to be corrected by following these steps:

  1. Log in to Terminal-1 as root user. Alternately, Terminal-1 can be accessed by pressing Ctrl+Alt+Right Arrow Key.
  2. Access and open the dbfw.conf file by executing this command: 
    vi /usr/local/dbfw/etc/dbfw.conf
  3. Set the correct value for the GATEWAY field by overwriting the existing value.
  4. Save and close the file.
  5. Execute the command to apply the modified value: 
    /usr/local/dbfw/bin/priv/configure-networking
  6. Return back to the appliance screen by pressing Ctrl+Alt+Left Arrow Key.

Note:

The network settings entered during installation can be modified, by choosing the Change IP Settings option in the installer or appliance screen.

A.8 Issue with Configuring or Managing Oracle AVDF through Oracle Enterprise Manager Cloud Control

Learn how to solve an issue with configuring or managing Oracle AVDF through Oracle Enterprise Manager Cloud Control.

Problem

Unable to configure or manage Oracle AVDF through Oracle Enterprise Manager Cloud Control.

Solution

Oracle AVDF plug-in is an interface within Oracle Enterprise Manager Cloud Control for administrators to manage and monitor Oracle AVDF components. Refer to System Monitoring Plug-in User's Guide for Audit Vault and Database Firewall in case of any issues when configuring the Oracle EM plug-in.

Refer to Compatibility with Oracle Enterprise Manager to check the supported versions of Oracle Enterprise Manager with Oracle AVDF 20.

A.9 Installation Stops Progressing After Entering the IP Address

Learn what to do when the installation stops progressing.

Problem

When installing Audit Vault Server, the installation stops progressing after you enter the IP address.

Solution

  1. Follow the instructions in Collecting Logs for Oracle AVDF Installation Issues to debug and collect logs for Oracle AVDF 20 installation issues.
  2. File a service request (SR) and attach the collected diagnostic information to the SR.

A.10 No Signal Error During Post-Install Tasks

Learn what to do when you receive a "no signal" error.

Problem

During the installation you receive a "no signal" error with a green screen, and the installation takes a long time to complete.

Solution

  1. Capture the screen content.
  2. Follow the instructions at Collecting Logs for ORacle AVDF Installation Issues to debug and collect logs for Oracle AVDF 20 installation issues.
  3. File a service request (SR) and attach the screen capture and the collected diagnostic information to the SR.

A.11 Pre-upgrade RPM Warnings

While patching or upgrading Oracle Audit Vault and Database Firewall (Oracle AVDF), the pre-upgrade RPM displays warnings to indicate issues that you need to resolve before proceeding with the update.

A.11.1 RPM Upgrade Failed

Read the troubleshooting advice if RPM upgrades fail.

Problem

An RPM upgrade failed with the following error:

error: %post(dbfw-mgmtsvr-###) scriptlet failed, exit status 1

Solution

  1. Check that there is at least 10MB of free /tmp space.

  2. Remove the new RPM:

    rpm -e dbfw-mgmtsvr-###

  3. Retry the upgrade.

A.11.2 Pre-upgrade RPM Failure Due to Insufficient Memory

Learn how to resolve pre-upgrade RPM failure due to insufficient memory.

Problem

Installing the pre-upgrade RPM places the system in a safe state, performs multiple checks, and rearranges free space on the appliance for a safe and successful installation or upgrade of Audit Vault Server and Database Firewall.

The following error may be observed:


 AVDF::Installer::Upgrade::InvalidPreconditions
 Recommended memory is x.yy GB; system only has xx.yy MB available
 ERROR:
 AVDF::Installer::Upgrade::InvalidPreconditions
 Verifying pre-upgrade conditions failed.

Solution

Follow these steps to resolve this issue:

  1. Run the following command to find the exact version of the pre-upgrade RPM:

    rpm -qa |grep avdf-pre*

  2. Run the following command to uninstall and remove the pre-upgrade RPM:

    rpm -e {rpm name}

  3. Power off the host machine.

  4. Increase the memory as per the recommendation.

  5. Power on the host machine.

  6. Re-install the pre-upgrade RPM.

  7. Ensure to check the warnings related to memory are resolved.

  8. Proceed with the upgrade as per Oracle AVDF documentation.

A.11.3 Insufficient Space Error in /var/lib/oracle File System Reported by Pre-upgrade RPM

Learn how to fix insufficient space error issue in /var/lib/oracle (lv_oracle) file system reported by pre-upgrade RPM.

Problem

An error or issue is observed when running pre-upgrade RPM. There is insufficient space in /var/lib/oracle (lv_oracle) file system.

Solution

The /var/lib/oracle file system needs a minimum of 31 GB free space for performing upgrade.

Follow these steps to clear space in /var/lib/oracle and to proceed with the upgrade process:

  1. Run the following command as grid user: 

    /usr/bin/find /var/lib/oracle/grid/rdbms/audit -name '*.aud' -mtime +1 -delete

    This process may take up to one hour to complete.

  2. Create another terminal.

  3. Run the following command as grid user to remove the trc and trm files: 

    rm /var/lib/oracle/diag/asm/+asm/+ASM/trace/*.tr[cm]

  4. As root user check if the /var/lib/oracle/upgrade_iso_file directory exists. Remove the ISO file in case it exists.

  5. As root user check and remove these file in case they exist. 

    rm /var/lib/oracle/software/database.tar.xz
    rm /var/lib/oracle/dbfw/av/grid[12].zip

  6. Run the following command as oracle user and remove the trc and trm files: 

    rm /var/lib/oracle/diag/rdbms/dbfwdb/dbfwdb/trace/*.tr[cm]

  7. Clear diagnostic logs through the Audit Vault Server console. This process may also release some additional space. In case any of the components are set to Debug, then set them to Warning.

    Note:

    Clearing Diagnostic Logs

A.11.4 Insufficient Space Error in / File System Reported by Pre-upgrade RPM

Learn how to fix insufficient space error issue in the / file system reported by pre-upgrade RPM.

Problem

An error similar to the below message is observed when running pre-upgrade RPM. There is insufficient space in the / file system. 
Checking upgrade preconditions
This upgrade requires at least 2.35GiB free on / (actual: 2.29GiB)

    AVDF::Installer::Upgrade::InvalidPreconditions

Precondition: 'space-check.rb'
    Result: 'Please follow the instructions in the Administrator's Guide to add storage, then retry.
    Summary: AVDF::Installer::Upgrade::InvalidPreconditions
        System is not ready for upgrade.

Solution

Extend / using the free space from vg_root:
lvextend --resizefs -L+2.35G /dev/vg_root/lv_ol8root

A.11.5 Pre-upgrade RPM Could Not Stop Certain Processes During Oracle AVDF Upgrade

Learn how to fix warnings or errors pointed by pre-upgrade RPM while upgrading Oracle AVDF.

Problem

The pre-upgrade RPM performs necessary checks to prepare the appliance conducive for upgrade. It stops certain processes running on the appliance in due course. In some cases, some of the processes cannot be stopped by the pre-upgrade RPM. It results in the following errors or warnings:

Not all processes were stopped
target is busy

Solution

Follow these steps:

  1. The pre-upgrade RPM suggests a possible way or solution to figure out the specific processes that are still running. Follow the instructions and stop the specific processes.
  2. Uninstall the pre-upgrade RPM.
  3. Reinstall the pre-upgrade RPM.
  4. Proceed with the upgrade procedure.

A.11.6 Pre-upgrade RPM Fails with "Unable to Stop Observer"

Learn how to resolve the "unable to stop observer" warning in the pre-upgrade RPM.

Problem

The pre-upgrade RPM fails with the "unable to stop observer" warning.

Messages and debug files display one of the following errors when the observer was started:

'DGMGRL:ORA-28000: The account is locked.’ or ‘DGMGRL:ORA-28001: the password has expired’

Solution

This can happen if the sys password has expired or the sys user is locked. To resolve this issue, update the sys user on the primary and standby systems. See Verify That the SYS User Is Unlocked and the Password Is Not Expired for instructions.

A.11.7 Pre-upgrade RPM Check: Alert Queue Space Warning

The pre-upgrade RPM displays a warning if the system doesn't have sufficient space to purge the alert queue during the upgrade.

The following warning appears:

The system does not have sufficient space to purge alert queue. Refer to Installation Guide on how to resolve this.

To resolve this issue, see Ensure That the System Has Sufficient Space to Purge the Alert Queue for instructions.

A.11.8 Pre-upgrade RPM Check: Boot Device Is Greater Than 2 TB

The pre-upgrade RPM warns you if the boot device greater than 2 TB, in which case the upgrade process may fail. Ensure that the boot device is less than 2 TB before upgrading.

To resolve this issue, see Ensure That the Boot Device Is Less Than 2 TB for instructions.

A.11.9 Pre-upgrade RPM Check: Boot Partition Space Warning

The pre-upgrade RPM warns you if there is not enough space in the boot partition, in which case the upgrade process may fail. Ensure that the boot partition has at least 500 MB before upgrading.

To resolve this issue, see Ensure That the Boot Partition Has at Least 500 MB for instructions.

A.11.10 Pre-upgrade RPM Check: Legacy Crypto Warning

If your current Oracle Audit Vault and Database Firewall (Oracle AVDF) 12.2 deployment has Host Monitor Agents or Audit Vault Agents on AIX and you're upgrading to Oracle AVDF 20.4 or later, then the pre-upgrade RPM displays a warning about TLS and encryption.

To resolve this issue, you need to run commands both before and after the upgrade.

Upgrading from Oracle AVDF 12.2.0.11.0 and Earlier

When upgrading from Oracle AVDF 12.2.0.11.0 and earlier, the pre-upgrade RPM displays the following warning. Follow the instructions in the warning to resolve the issue.

If you have deployed Host Monitor Agents (or Audit Vault Agents on AIX) in your environment, TLS 1.1 should be used for encryption instead of the default version of TLS 1.2. Else, Host Monitor Agents (or Audit Vault Agents on AIX) will not upgrade automatically. If you wish to use TLS 1.1 for encryption run the below command before proceeding with the upgrade.

ruby /usr/local/dbfw/bin/upgrade/configure_tls_settings.rb 2

Post Audit Vault Server and Agents upgrade, run the following command as root user:

/usr/local/dbfw/bin/priv/configure-networking --agent-tls-cipher-level 4

Run the following command post upgrade, if it is only displayed on the prompt:

/usr/local/dbfw/bin/priv/send_agent_update_signal.sh

Refer to Oracle AVDF Installation Guide, sections "Pre-upgrade RPM Legacy Crypto Check Warning" and "Post Upgrade TLS Security Hardening" for more details.

Upgrading from Oracle AVDF 12.2.0.12.0 and Later

When upgrading from Oracle AVDF 12.2.0.12.0 and later, the pre-upgrade RPM displays the following warning. Follow the instructions in the warning to resolve the issue.

If you have deployed Audit Vault Agents on AIX in your environment, TLS 1.1 should be used for encryption instead of the default version of TLS 1.2. Else, the Agents on AIX will not upgrade automatically. If you wish to use TLS 1.1 for encryption run the below command before proceeding with the upgrade.

ruby /usr/local/dbfw/bin/upgrade/configure_tls_settings.rb 2

Post Audit Vault Server and Agents upgrade, run the following command as root user:

/usr/local/dbfw/bin/priv/configure-networking --agent-tls-cipher-level 4

Run the following command post upgrade, if it is only displayed on the prompt:

/usr/local/dbfw/bin/priv/send_agent_update_signal.sh

Refer to Oracle AVDF Installation Guide, sections "Pre-upgrade RPM Legacy Crypto Check Warning" and "Post Upgrade TLS Security Hardening" for more details.

Related Topics

A.11.11 Pre-upgrade RPM Fails with "Not All Processes Were Stopped"

Problem

The pre-upgrade RPM fails with the following warning: Not all processes were stopped: 7378,7379.

For example:

rpm -ivh --force avdf-pre-upgrade-20.x.0.0.0-0_NNNNNN.NNNN.x86_64.rpm 
Preparing... ########################################### [100%] 
1:avdf-pre-upgrade ########################################### [100%] 
Checking upgrade preconditions 
/bin/df: '/var/dbfw/upgrade': No such file or directory 
/bin/df: no file systems processed 
Shutting down services. 
Traceback (most recent call last): 
3: from /usr/local/dbfw/bin/pre_upgrade.rb:642:in '<main>' 
2: from /usr/local/dbfw/bin/pre_upgrade.rb:614:in 'process_command_line' 
1: from /usr/local/dbfw/bin/pre_upgrade.rb:503:in 'post_install' 
/usr/local/dbfw/lib/ruby/upgrade/common.rb:621:in 'stop_nonroot_processes': 
Not all processes were stopped: 7378,7379

Cause

This issue could be caused by an idle SSH session, busy devices, or open temporary files.

Solution

  1. Uninstall the RPM as the root user.

    1. Log in to the Audit Vault Server through SSH and switch to the root user.

      See Logging In to Oracle AVDF Appliances Through SSH.

    2. Uninstall the pre-upgrade RPM by using one of the following commands:

      rpm -e avdf-pre-upgrade
      rpm -e avdf-pre-upgrade --noscripts

  2. Check the pre-upgrade RPM listing.

    1. Enter the following command:

      rpm -qa |grep avdf-pre-upgrade
    2. Ensure that there's no entry for avdf-pre-upgrade RPM.
    3. Reboot the Audit Vault Server if it's a STANDALONE system.

  3. Check for other SSH sessions, busy devices, or temporary open files.

    1. Ensure that there are no other SSH sessions that are owned by the support user.

      To do this, identify idle notty (no tty) SSH sessions and try to stop them.

      Use the following commands to check the pid of sshd: support@notty. 

      ps -ef |grep support
      ps -ef |grep notty

      For example:

      support 2480 2427 0 18:31 ? 00:00:00 sshd: support@notty
support 2481 2480 0 18:31 ? 00:00:00 -bash
kill -9 2481
kill -9 2480
    2. Check again for support@notty processes in the system.

    3. Ensure that the system doesn't have any busy devices or open temporary files. To do this, run lsof against /tmp and /usr/local/dbfw/tmp.

      For example:

      lsof /usr/local/dbfw/tmp
lsof /tmp

      Note:

      Ensure that no logs are open when starting the patching or upgrade process.

  4. Try to install the pre-upgrade RPM as the root user.

    1. Log in to the Audit Vault Server through SSH and switch to the root user.

      See Logging In to Oracle AVDF Appliances Through SSH.

    2. Enter the following command:

      rpm -i /root/avdf-pre-upgrade-20.x.0.0.0-0_NNNNNN.NNNN.x86_64.rpm

A.11.12 Pre-upgrade RPM Check: Agent Failure Checks - Upgrade Prerequisites

Starting with Oracle AVDF 20.9, the pre-upgrade RPM verifies that the Audit Vault Agent and Host Monitor Agent configurations are compatible with Oracle AVDF 20.10 or later.

Problem

The agent_prereq_checks_failure_report.txt report indicates that a Audit Vault Agent or Host Monitor Agent doesn't meet the prerequisites to update to Oracle AVDF 20.10 or later. You can find the agent success and failure reports in the following locations:

  • Success report: /opt/avdf/report/agent_prereq_checks_success_report.txt
  • Failure report: /opt/avdf/report/agent_prereq_checks_failure_report.txt

The following example shows a failure message:

Agent/HM Validation Failure statuses are as below :
------------------------------------------------------------------
Agent Name : agent-linux
Agent Validation Status : FAILURE
Agent Failure Checks : Upgrade Prerequisites check jar build with latest version. Please check the minimum java version required. - <Exception Message>
Agent Checks Warning Messages :
Validated at : 2022-12-02 09:11:24.774880

Solution

Resolve the issue that's indicated in the report. For example, update the Audit Vault Agent machine to the minimum Java version that's supported.

You can rerun the failure check scripts individually to verify that the issues are resolved. Run these scripts as the root user. 

/usr/bin/python3 /usr/local/dbfw/bin/upgrade/pre_upgrade_validate_agent.py standalone
/usr/bin/python3 /usr/local/dbfw/bin/upgrade/pre_upgrade_download_agent_validation_status.py standalone

A.11.13 Removing Pre-upgrade RPM for AVDF 20.10 May Not Bring the Services Back Up

Before upgrade, you must install the pre-upgrade RPM to prepare the system for upgrade. However, sometimes rpm -e avdf-pre-upgrade is required to bring the system back to the original state, but it may fail to bring up the Audit Vault Server.

Issue

Before upgrade, you must install the pre-upgrade RPM to prepare the system for upgrade. Sometimes you may need to execute rpm -e avdf-pre-upgrade to remove the pre-upgrade RPM and bring the system back online to fix some issues before doing upgrade. However, sometimes rpm -e avdf-pre-upgrade fails to bring up Audit Vault Server.

If you run the following command as the root user
systemctl status asmdb
and receive this output: Failed to start Start the oracle asm instance, then you are encountering this known issue.

Workaround

Perform the following steps to bring the Audit Vault Server back up:

  1. Run the following as the root user:
    rpm -qa|grep avdf-pre-upgrade
  2. If avdf-pre-upgrade-20.10.0.0.0-0_* is in the output of the previous step, run the following as the root user: 
    rpm -e avdf-pre-upgrade
    1. If rpm -e avdf-pre-upgrade throws the following error:
      error: %preun(avdf-pre-upgrade-20.10.0.0.*) scriptlet failed, exit status 1
error: avdf-pre-upgrade-20.10.0.0.*: erase failed
      Run the following as the root user:
      /bin/bash /usr/local/dbfw/bin/pre_upgrade_set_python.sh
    2. Run the following as the root user: 
      rpm -e avdf-pre-upgrade

    If this brings up the Audit Vault Server and it is accessible through the UI, then you don't need to complete the rest of the steps. Otherwise, continue with these steps.

  3. Run the following as the root user:
    systemctl status asmdb
  4. If the previous step results still shows Failed to start Start the oracle asm instance. Run the following as the root user:
    systemctl stop asmdb
systemctl stop ohasd
systemctl stop oracle-ohasd
  5. Run the following as the root user:
    systemctl start oracle-ohasd
systemctl start ohasd
  6. Switch to the grid user:
    su - grid
  7. Run the following as the grid user:
    csrctl check has
    If ohasd is online with this message CRS-4638: Oracle High Availability Services is online, it is ready for the next step.
  8. Run the following as the grid user:
    crsctl status resource ora.cssd
    Wait until you get the state: STATE=ONLINE on <AVSname>.

    If you get the state STATE=OFFLINE or STATE=UNKNOWN, go to step three and repeat steps three through seven.

  9. Run the following as the root user:
    systemctl start https
  10. Run the following as the root user:
    systemctl isolate avdf-runtime
    This will take some time, wait until it finishes.
  11. Check on the database service by running the following as the root user:
    systemctl status dbfwdb

    If the output of this command includes Started Start the oracle database, then the Audit Vault Server database is up.

  12. Test the Audit Vault Server by accessing the GUI:

    Log in to the Audit Vault Server Console as an administrator.

    The Audit Vault Server will be back up and running as it was before running rpm -i vdf-pre-upgrade-20.10.0.0.0-0_*.

A.12 SSH Becomes Disabled After Updating Oracle AVDF with FIPS Enabled

If SSH becomes disabled after updating Oracle AVDF with FIPS mode enabled, update the SSH keys to be compliant with FIPS.

Problem

After updating Oracle AVDF to release 20.9 with FIPS mode enabled, SSH becomes disabled.

Solution

Before enabling FIPS 140-2, ensure that your SSH keys are compliant with FIPS. If your SSH keys are not compliant with FIPS, the SSH connection with the appliance might be lost after enabling FIPS.

For Oracle AVDF on Oracle Cloud Infrastructure (OCI), before enabling FIPS mode, ensure that the opc user has FIPS-compliant keys registered to /home/opc/.ssh/authorized_keys.

Follow these steps to resolve this issue:

  1. Log into the Audit Vault Server console and disable FIPS mode.

  2. Log back into the appliance through SSH and check or update the user keys for SSH-enabled users in ~/.ssh/authorized_keys to be compliant with FIPS.

    It can take several minutes for the console to become available after enabling or disabling FIPS mode.

  3. Enable FIPS mode.

Related Topics

A.13 SSH Connection Times Out When Uninstalling the Pre-Upgrade RPM

Problem

The SSH connection times out when uninstalling the pre-upgrade RPM.

Cause

The default SSH connection timeout is 10 minutes, and uninstalling the pre-upgrade RPM can take longer than 10 minutes.

Solution

Run the screen command before uninstalling the pre-upgrade RPM. The screen command prevents network disconnections from interrupting the patching or upgrading.

If the session terminates, resume by switching to the root user and then running the screen -r command.

A.14 Installation Pauses After Entering the Root Password

Problem

When you start the installer for Oracle AVDF 20.5, it installs a few packages and prompts you to change the root password. After you enter the new root password, the installer immediately display some unmount commands and returns to the starting installation screen. You're unable to proceed with the installation.

Cause

The ISO file was removed before the installation was completed.

Solution

After you enter the new root password and return to the starting installation screen, complete the following steps:

  • Remove the ISO CD from the CD drive and restart the machine.
  • When you're promoted to log in, log in as ROOT.
  • When promoted for the ISO file, add the ISO file from the media.

A.15 When Upgrading to Oracle AVDF 20.3 ELMIG_POPULATE_CLUSTERS_202 and ELMIG_CONVERT_HASH_202 Are Reported as INVALID in dba_objects Table

Even though the objects are invalid this doesn't have any impact on the system operation and can be ignored.

Problem

When upgrading to Oracle AVDF 20.3 the objects ELMIG_POPULATE_CLUSTERS_202 and ELMIG_CONVERT_HASH_202 are reported as INVALID in dba_objects table.

The following query results in the ELMIG_POPULATE_CLUSTERS_202 and ELMIG_CONVERT_HASH_202 objects. 
elect object_name from dba_objects where status = ‘INVALID’;
OBJECT_NAME

Solution

This doesn't have any impact on the system operation and can be ignored.

A.16 Error Occurred Trying to Format SDAF1 When Installing Oracle AVDF

Problem

During the installation of Audit Vault Server,the following error is encountered:

Error: An error occurred trying to format sdaf1. This problem is serious, and the install cannot continue. Press to reboot your system.

Cause

The server has SAN connectivity.

Solution

Disable the SAN connectivity. ISCSI device should not be attached until Audit Vault Server installation is completed.

A.17 Audit Vault Agent Failed on Startup: OAV-10: Failed to Release Connection to DB

Problem

When installing the Audit Vault Agent error OAV-10: Failed to Release Connection to Database occurred when ./agentctl start -k was executed. The database to Audit Vault Server connection failed.

Cause

The wrong location was used for JAVA_HOME and agentctl picked up a different Java in the path. The connection failed as it does not work with Java that is present in the database home.

Solution

Set the proper location for JAVA_HOME:
JAVA_HOME=/usr/java/jdk1.8.0_361
export PATH=$JAVA_HOME/bin:$PATH

A.18 Upgrade to AVDF 20.4 Failed During upgrade_apex Step

When upgrading to AVDF 20.4, the upgrade_apex step results in ODF-10001: Internal error: FAILED migration: upgrade_apex (as oracle) error.

Problem

The /var/log/messages contains ERROR - ODF-10001: Internal error: FAILED migration: upgrade_apex (as oracle) (applied change) and /var/log/debug contains 
upgrade_apex: error: cannot create /var/lib/oracle/dbfw/apex/images/computer.gif
upgrade_apex: Permission denied
upgrade_apex: error: cannot create /var/lib/oracle/dbfw/apex/images/phone_support.gif
upgrade_apex: Permission denied
In addition, running the following command as the root user: 
/opt/avdf/bin/privmigutl –status
results in 
System state - recovery
Migration set 'AVS' - failed
Last migration 'Upgrading apex20' - failed

Solution

  1. Log in to the Audit Vault Server through SSH and switch to the root user.

    See Logging In to Oracle AVDF Appliances Through SSH.

  2. Run the following command:
    chown -R oracle:oinstall /var/lib/oracle/dbfw/apex/images

  3. Switch to the oracle user.

    su - oracle
  4. Run the following script:
    /usr/local/dbfw/etc/privileged-migrations/upgrade_apex
  5. Run 
    echo $?

    If the result is 2, then the script has completed successfully.

  6. Log in to the Audit Vault Server through SSH and switch to the root user.

    See Logging In to Oracle AVDF Appliances Through SSH.

  7. Resume the upgrade process by running the following: 
    /opt/avdf/bin/privmigutl --resume --confirm

    Make the sure ssh connection to the Oracle AVDF server is reliable and does not terminate while running this command.

  8. Check if the $ORACLE_HOME/apex/images folder and its contents have oracle:oinstall permission, and if not, grant these permissions.

Related Topics