A Troubleshooting Oracle Audit Vault and Database Firewall
Oracle Audit Vault and Database Firewall provides troubleshooting advice for expected issues in the deployment or installation process.
A.1 Information to Provide Support When Filing a Service Request
Review this list of information to provide support when filing a service request.
Note:
Diagnostics data, especially trace files, often contains sensitive information. Protect it accordingly and only gather and send the information that's required.- Oracle AVDF version, including any installed bundle patches
- If virtualization is being used? If so, which one?
- How much physical memory is available to Audit Vault Server and Database Firewall appliances?
- How much disk space was available with the initial installation?
- Did you add any SAN storage and in that case how much disk space?
- Provide any relevant details about the brand and model of the hardware being used. This is relevant if you have specific issues relating to booting from the installation media.
- Host OS for the secured target database and version, this is relevant for checking agent compatibility issues.
- Brand of the secured target database, such as Oracle, MySQL, SQL Server, etc.
- Version of the secured target database, including PSU and other one-off patches.
- Upload the alert.log file of the secured target database.
- From any Oracle secured target database provide the output of:
show parameter audit
opatch lsinventory -patch -detail
- If unified auditing was configured (for some versions of Oracle database only)
- Audit Trail type that is being configured and all relevant attributes
- Detailed diagnostic information for Audit Vault Server, see Downloading Detailed Diagnostics Reports for Oracle Audit Vault Server
- If requested by Oracle Support, diagnostic information from Oracle Trace File Analyzer. See Using Oracle Trace File Analyzer (TFA).
- Information about Database Firewall:
- Detailed diagnostic info for Database Firewall, see Viewing the Status and Diagnostics Report for Database Firewall
- How many Network Interface Cards are installed in the database firewall appliance?
- Is the enforcement point using default password enumeration (DPE) or database activity monitoring (DAM)? If so is it bridge, span, or proxy?
- Do you use VLAN tagging? There are restrictions for support of VLANs.
- For installation issues, diagnostic files related to the installation. See Collecting Logs to Debug Installation Failures.
Before contacting support, the Audit Trail Transaction Log should follow these guidelines:
- The user setup script must be run with the argument
REDO_COLL
- The secured target database must be configured with
ARCHIVELOG
- The streams recommended patches must be applied to the secured target db: Streams Recommended Patches (Doc ID 437838.1)
global_name
must be fully qualified (select global_name from global_name;)- Parameter
global_names = true
is recommended - If errors happen on capture or apply side please check respective alert.logfiles as you would do with any Streams related issue (av log will show only limited information for this audit trail type)
Related Topics
A.2 Error When Installing Audit Vault Server in Releases 20.1 to 20.3
Learn how to resolve an error observed when installing Audit Vault Server 20.1, 20.2, or 20.3.
Problem
An error is observed when installing Audit Vault Server. This is observed only in Oracle AVDF releases 20.1 to 20.3.
Solution
The Audit Vault Server installer (ISO) file is split into 3 parts or
files in Oracle AVDF releases 20.1 to 20.3. All the three ISO files have to be
concatenated to get a single Audit Vault Server 20.x ISO
(avdf-install.iso
) before proceeding with installation.
Refer to Downloading and Verifying Oracle AVDF Software for complete information.
Starting with Oracle AVDF 20.4, there is a single Audit Vault Server ISO file and there is no need to concatenate.
A.3 Conflicting Data on Storage Added to Oracle AVDF
Learn how to remove existing conflicting data from storage before adding it to Oracle Audit Vault and Database Firewall (Oracle AVDF).
Problem
The preexisting file system, Logical Volume Manager (LVM), or device mapper metadata may conflict with Oracle AVDF functionality. This may result in patch, upgrade, or installation failure.
Symptoms
The symptoms of any preexisting LVM or other device mapper metadata include, but are not limited to, the following:
- Two
vg_root
volume groups. - Hard drive devices that become unavailable during patching, upgrade, or installation. This may lead to input or output errors and eventually result in patch, upgrade, or installation failure.
Solution
Caution:
This will erase data from the drive.- Download the latest Oracle Linux 8 ISO image from Oracle Linux Downloads.
-
Boot into rescue mode.
-
Load the Oracle Linux 8 ISO onto your appliance and boot.
The installation menu displays the following options:
Install Oracle Linux 8.x.x Test this media & install Oracle Linux 8.x.x Troubleshooting
-
Press the Down Arrow to select Troubleshooting, and press Enter.
The troubleshooting menu displays the following options:
Install Oracle Linux 8.x.x in basic graphics mode Rescue a Oracle Linux system
-
Press the Down Arrow to select Rescue a Oracle Linux system, and press Enter.
The rescue menu displays the following options:
1) Continue 2) Read-only mount 3) Skip to shell 4) Quit (Reboot)
- Type 3 (Skip to shell), and press Enter.
- Press Enter again to open the shell prompt.
-
-
To discover the attached storage, enter the
lsblk
command at the shell prompt.For example:
sh-4.4# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 745.4M 1 loop loop1 7:1 0 4G 1 loop ├─live-rw 253:0 0 4G 0 dm / └─live-base 253:1 0 4G 1 dm loop2 7:2 0 32G 0 loop └─live-rw 253:0 0 4G 0 dm / sda 8:0 0 256G 0 disk └─sda1 8:1 0 256G 0 part sr0 11:0 1 11.2G 0 rom /run/install/repo sr1 11:1 1 1024M 0 rom
-
To wipe the drive, enter the
wipefs
command at the shell prompt.Enter
wipefs --help
to see a complete list of options.For example, to wipe the
/dev/sda
drive, enter the following command:sh-4.4# wipefs --all /dev/sda
The command output lists the changes. For example:
/dev/sda: 2 bytes were erased at offset 0x000001fe (dos): 55 aa /dev/sda: calling ioctl to re-read partition table: Success
-
To safely power off, enter the
sync
command at the shell prompt, followed bypoweroff
.sh-4.4# sync sh-4.4# poweroff
- After you wipe the drive, eject the ISO and restart the installation.
A.4 EFI Related Error When Installing Audit Vault Server on VMware
Learn how to resolve EFI related error when installing Audit Vault Server on VMware.
Problem
The following possible errors are observed when attempting to install Audit Vault Server on VMware:
EFI Virtual disk (0.0) … unsuccessful.
EFI VMware Virtual SATA CDROM Drive (0.0) … unsuccessful.
EFI Network …
Solution
There are important prerequisites to be followed while installing Audit Vault Server on VMware:
-
You must set VMX configuration parameter disk.EnableUUID to
TRUE
. This must be done to enable proper mounting of disks. Without this setting, the Audit Vault Server installation on VMware will fail. -
You must set your virtual machine to use EFI boot. In some versions of VMware this is done by selecting the VM Options tab, then expanding Boot Options, and then choose
EFI
in the Firmware field. You must disable secure boot. Do not select the checkbox Enable UEFI secure boot field.This EFI boot setting is required only for fresh installation of Audit Vault Server specifically when the disk size is more than 2TB. This setting is not required for upgrade.
Note:
See Installing Audit Vault Server on VMware for complete information.A.5 Cannot Access the Audit Vault Server Console
Learn the workaround for when you cannot access the Audit Vault server user interface or console.
Problem
The Audit Vault Server console is not accessible.
Solution
There are two remedies that you can perform depending on when this problem occurs:
-
The problem occurs immediately after Audit Vault Server installation.
In this case, the installation may not have been completed correctly. Perform the installation again.
-
The problem occurs after the system is already running.
In this case, check that the disk is not full and that the Oracle Audit Vault Server database is running using this command:
/etc/init.d/dbfwdb status
To restart the database, use run this command as
root
:/etc/init.d/dbfwdb start
If you have a problem restarting the database, then contact Oracle Support.
A.6 Collecting Logs to Debug Installation Failures
You can collect logs to debug issues when installing Oracle Audit Vault and Database Firewall.
A.6.1 Collecting Logs for Base Operating System Installation Issues
Use these steps to collect logs for failures that happen during the installation of the base operating system (pre- or post-reboot).
Collecting logs for debugging pre-reboot installation failures
- During installation or upgrade, after mounting the
.iso
file, press Tab to interrupt the normal boot process. - To collect logs, the installer must run with command line access. To
enable command line access, remove the
noshell
from the boot option. -
After the failure occurs, use one of the following keyboard shortcuts to access the command line:
- Starting with Oracle AVDF 20.9 (Oracle Linux 8), press Ctrl+B and then press 2.
- For installing Oracle AVDF 20.1 to 20.8 (Oracle Linux 7), press Alt+Right Arrow.
-
Run one of the following commands to start the collection tool:
-
Starting with Oracle AVDF 20.9 (Oracle Linux 8), use the following command:
/usr/libexec/platform-python /run/install/repo/collect_diagnostics.py
For Oracle AVDF 20.1 to 20.8 (Oracle Linux 7), use the following command:
python /run/install/repo/collect_diagnostics.py
-
- Follow the instructions to collect the diagnostics file.
Collecting logs for debugging post-reboot installation failures
- Using the password you have previously set, log in as
root
on the console or using SSH. -
Run one of the following commands to start the collection tool:
-
Starting with Oracle AVDF 20.9 (Oracle Linux 8), use the following command:
/usr/libexec/platform-python /media/avdf-install/collect_diagnostics.py
-
For Oracle AVDF 20.1 to 20.8 (Oracle Linux 7), use the following command:
python /media/avdf-install/collect_diagnostics.py
-
- Follow the instructions to collect the diagnostics file.
Transferring the log file for analysis
After following the instructions to collect the logs for pre- or post-reboot failures, the collection tool should have created a log or diagnostic file in the following location:
/root/install-diagnostics.tgz
-
Follow the instructions at the prompt to transfer the log file for analysis. Use the following command:
scp /root/install-diagnostics.tgz <user>@<Ip address>:<Path>
-
You may also perform the following steps and commands to configure the network:
ip addr add <IP address>/<sub net> dev <interface>
ip link set <interface> up
ip route add default via <gateway>
- Use the information available in the log file to analyze the issue and then try the installation again after addressing the issue.
A.6.2 Collecting Logs for Oracle AVDF Installation Issues
Use these steps to collect logs for failures that happen when installing Oracle AVDF.
- At the install start screen, press Tab (and delete the word "noshell").
- Press Enter to begin the installation.
-
After the installation begins, press Ctrl+B and then press 2.
The login screen should appear even if the installation fails.
- Use tar or Gzip to collect the following logs:
/var/log
/var/lib/oracle/diag
/var/lib/oracle/oraInventory/logs
/tmp
-
Collect the following configuration files:
/etc/sysconfig/avdf
/var/lib/avdf/system_history.yaml
/usr/local/dbfw/etc/dbfw.conf
-
Collect the output from the following commands:
su root
rpm -qa avs
ls -lrt /var/log/installation-*
ls -lrt /var/log/upgrade-*
df -h
du -sh /var/lib/oracle/19.7.0.0.0
du -sh /var/lib/oracle/19.7.0.0.0/grid
cat /proc/meminfo
-
Collect the output from the following commands:
su root
hostname
cd /var/lib/oracle/diag
ls -lrt
cd crs
ls -lrt
hostname
cd <hostname>
ls -lrt
cd crs
ls -lrt
A.7 Unable to Reach Gateway Error
Learn to fix incorrect Gateway details entered during installation.
Problem
Incorrect or invalid Gateway details entered while installing Audit Vault Sever or Database Firewall. The following error message may be encountered:
Gateway is not reachable from host
Solution
The Gateway details can to be corrected by following these steps:
- Log in to Terminal-1 as root user.
Alternately, Terminal-1 can be accessed by pressing
Ctrl+Alt+Right Arrow Key
. - Access and open the dbfw.conf file by executing this command:
vi /usr/local/dbfw/etc/dbfw.conf
- Set the correct value for the GATEWAY field by overwriting the existing value.
- Save and close the file.
- Execute the command to apply the modified value:
/usr/local/dbfw/bin/priv/configure-networking
- Return back to the appliance screen by pressing
Ctrl+Alt+Left Arrow Key
.
Note:
The network settings entered during installation can be modified, by choosing the Change IP Settings option in the installer or appliance screen.A.8 Issue with Configuring or Managing Oracle AVDF through Oracle Enterprise Manager Cloud Control
Learn how to solve an issue with configuring or managing Oracle AVDF through Oracle Enterprise Manager Cloud Control.
Problem
Unable to configure or manage Oracle AVDF through Oracle Enterprise Manager Cloud Control.
Solution
Oracle AVDF plug-in is an interface within Oracle Enterprise Manager Cloud Control for administrators to manage and monitor Oracle AVDF components. Refer to System Monitoring Plug-in User's Guide for Audit Vault and Database Firewall in case of any issues when configuring the Oracle EM plug-in.
Refer to Compatibility with Oracle Enterprise Manager to check the supported versions of Oracle Enterprise Manager with Oracle AVDF 20.
A.9 Installation Stops Progressing After Entering the IP Address
Learn what to do when the installation stops progressing.
Problem
When installing Audit Vault Server, the installation stops progressing after you enter the IP address.
Solution
- Follow the instructions in Collecting Logs for Oracle AVDF Installation Issues to debug and collect logs for Oracle AVDF 20 installation issues.
- File a service request (SR) and attach the collected diagnostic information to the SR.
A.10 No Signal Error During Post-Install Tasks
Learn what to do when you receive a "no signal" error.
Problem
During the installation you receive a "no signal" error with a green screen, and the installation takes a long time to complete.
Solution
- Capture the screen content.
- Follow the instructions at Collecting Logs for ORacle AVDF Installation Issues to debug and collect logs for Oracle AVDF 20 installation issues.
- File a service request (SR) and attach the screen capture and the collected diagnostic information to the SR.
A.11 Pre-upgrade RPM Warnings
While patching or upgrading Oracle Audit Vault and Database Firewall (Oracle AVDF), the pre-upgrade RPM displays warnings to indicate issues that you need to resolve before proceeding with the update.
A.11.1 RPM Upgrade Failed
Read the troubleshooting advice if RPM upgrades fail.
Problem
An RPM upgrade failed with the following error:
error: %post(dbfw-mgmtsvr-###) scriptlet failed, exit status 1
Solution
-
Check that there is at least 10MB of free
/tmp
space. -
Remove the new RPM:
rpm -e dbfw-mgmtsvr-###
-
Retry the upgrade.
A.11.2 Uninstalling the Pre-Upgrade RPM for AVDF 20.12 and Later Doesn't Remove Filesystem
If you currently have Oracle AVDF 20.11 or earlier and apply the pre-upgrade RPM for AVDF 20.12 or later, and decide to not proceed with the upgrade, the filesystem for Database Firewall doesn't get removed. If you wish to reallocate the space reserved for upgrade, perform the following.
-
Run the following command to find the exact version of the pre-upgrade RPM:
rpm -q avdf-pre-upgrade
-
Run the following command to uninstall and remove the pre-upgrade RPM:
rpm -e {rpm name}
- Run the following command to verify the filesystem remains mounted:
# df
You will see something similar to:
[...] /dev/mapper/vg_root-lv_var_dbfw_upgrade on /var/dbfw/upgrade type ext4(rw,relatime,seclabel) [...]
- Run the following command to unmount the
filesystem:
umount /var/dbfw/upgrade
- Run the following command to remove the logical
volume:
lvremove /dev/vg_root/lv_var_dbfw_upgrade
- Run the following command to confirm the logical volume is unmounted and
removed:
# df # lvs
You will see something similar to:
[no /var/dbfw/upgrade records] [no /var/dbfw/upgrade records]
A.11.3 Pre-upgrade RPM Failure Due to Insufficient Memory
Learn how to resolve pre-upgrade RPM failure due to insufficient memory.
Problem
Installing the pre-upgrade RPM places the system in a safe state, performs multiple checks, and rearranges free space on the appliance for a safe and successful installation or upgrade of Audit Vault Server and Database Firewall.
The following error may be observed:
AVDF::Installer::Upgrade::InvalidPreconditions
Recommended memory is x.yy GB; system only has xx.yy MB available
ERROR:
AVDF::Installer::Upgrade::InvalidPreconditions
Verifying pre-upgrade conditions failed.
Solution
Follow these steps to resolve this issue:
-
Run the following command to find the exact version of the pre-upgrade RPM:
rpm -q avdf-pre-upgrade
-
Run the following command to uninstall and remove the pre-upgrade RPM:
rpm -e {rpm name}
-
Power off the host machine.
-
Increase the memory as per the recommendation.
-
Power on the host machine.
-
Re-install the pre-upgrade RPM.
-
Ensure to check the warnings related to memory are resolved.
-
Proceed with the upgrade as per Oracle AVDF documentation.
A.11.4 Insufficient Space Error in /var/lib/oracle File System Reported by Pre-upgrade RPM
Learn how to fix insufficient space error issue in
/var/lib/oracle
(lv_oracle)
file system reported by pre-upgrade RPM.
Problem
An error or issue is observed when running pre-upgrade RPM. There is
insufficient space in /var/lib/oracle
(lv_oracle)
file system.
Solution
The /var/lib/oracle
file system needs a minimum of
31 GB free space for performing upgrade.
Follow these steps to clear space in /var/lib/oracle
and to proceed with the upgrade process:
-
Run the following command as grid user:
/usr/bin/find /var/lib/oracle/grid/rdbms/audit -name '*.aud' -mtime +1 -delete
This process may take up to one hour to complete.
-
Create another terminal.
-
Run the following command as grid user to remove the
trc
andtrm
files:rm /var/lib/oracle/diag/asm/+asm/+ASM/trace/*.tr[cm]
-
As root user check if the
/var/lib/oracle/upgrade_iso_file
directory exists. Remove the ISO file in case it exists. -
As root user check and remove these file in case they exist.
rm /var/lib/oracle/software/database.tar.xz
rm /var/lib/oracle/dbfw/av/grid[12].zip
-
Run the following command as oracle user and remove the
trc
andtrm
files:rm /var/lib/oracle/diag/rdbms/dbfwdb/dbfwdb/trace/*.tr[cm]
-
Clear diagnostic logs through the Audit Vault Server console. This process may also release some additional space. In case any of the components are set to
Debug
, then set them toWarning
.
A.11.5 Insufficient Space Error in / File System Reported by Pre-upgrade RPM
Learn how to fix insufficient space error issue in the
/
file system reported by pre-upgrade RPM.
Problem
/
file
system.
Checking upgrade preconditions
This upgrade requires at least 2.35GiB free on / (actual: 2.29GiB)
AVDF::Installer::Upgrade::InvalidPreconditions
Precondition: 'space-check.rb'
Result: 'Please follow the instructions in the Administrator's Guide to add storage, then retry.
Summary: AVDF::Installer::Upgrade::InvalidPreconditions
System is not ready for upgrade.
Solution
/
using the free space from
vg_root
:lvextend --resizefs -L+2.35G /dev/vg_root/lv_ol8root
A.11.6 Pre-upgrade RPM Could Not Stop Certain Processes During Oracle AVDF Upgrade
Learn how to fix warnings or errors pointed by pre-upgrade RPM while upgrading Oracle AVDF.
Problem
The pre-upgrade RPM performs necessary checks to prepare the appliance conducive for upgrade. It stops certain processes running on the appliance in due course. In some cases, some of the processes cannot be stopped by the pre-upgrade RPM. It results in the following errors or warnings:
Not all processes were stopped
target is busy
Solution
Follow these steps:
- The pre-upgrade RPM suggests a possible way or solution to figure out the specific processes that are still running. Follow the instructions and stop the specific processes.
- Uninstall the pre-upgrade RPM.
- Reinstall the pre-upgrade RPM.
- Proceed with the upgrade procedure.
A.11.7 Pre-upgrade RPM Fails with "Unable to Stop Observer"
Learn how to resolve the "unable to stop observer" warning in the pre-upgrade RPM.
Problem
The pre-upgrade RPM fails with the "unable to stop observer" warning.
Messages and debug files display one of the following errors when the observer was started:
'DGMGRL:ORA-28000: The account is locked.’ or ‘DGMGRL:ORA-28001: the password has expired’
Solution
This can happen if the sys
password has expired or the
sys
user is locked. To resolve this issue, update the
sys
user on the primary and standby systems. See Verify That the SYS User Is Unlocked
and the Password Is Not Expired for instructions.
A.11.8 Pre-upgrade RPM Check: Alert Queue Space Warning
The pre-upgrade RPM displays a warning if the system doesn't have sufficient space to purge the alert queue during the upgrade.
The following warning appears:
The system does not have sufficient space to purge alert queue. Refer to Installation Guide on how to resolve this.
To resolve this issue, see Ensure That the System Has Sufficient Space to Purge the Alert Queue for instructions.
A.11.9 Pre-upgrade RPM Check: Boot Device Is Greater Than 2 TB
The pre-upgrade RPM warns you if the boot device greater than 2 TB, in which case the upgrade process may fail. Ensure that the boot device is less than 2 TB before upgrading.
To resolve this issue, see Ensure That the Boot Device Is Less Than 2 TB for instructions.
A.11.10 Pre-upgrade RPM Check: Boot Partition Space Warning
The pre-upgrade RPM warns you if there is not enough space in the boot partition, in which case the upgrade process may fail. Ensure that the boot partition has at least 500 MB before upgrading.
To resolve this issue, see Ensure That the Boot Partition Has at Least 500 MB for instructions.
A.11.11 Pre-upgrade RPM Check: Legacy Crypto Warning
If your current Oracle Audit Vault and Database Firewall (Oracle AVDF) 12.2 deployment has Host Monitor Agents or Audit Vault Agents on AIX and you're upgrading to Oracle AVDF 20.4 or later, then the pre-upgrade RPM displays a warning about TLS and encryption.
To resolve this issue, you need to run commands both before and after the upgrade.
Upgrading from Oracle AVDF 12.2.0.11.0 and Earlier
When upgrading from Oracle AVDF 12.2.0.11.0 and earlier, the pre-upgrade RPM displays the following warning. Follow the instructions in the warning to resolve the issue.
If you have deployed Host Monitor Agents (or Audit Vault Agents
on AIX) in your environment, TLS 1.1 should be used for encryption instead of
the default version of TLS 1.2. Else, Host Monitor Agents (or Audit Vault Agents
on AIX) will not upgrade automatically. If you wish to use TLS 1.1 for
encryption run the below command before proceeding with the
upgrade.
ruby /usr/local/dbfw/bin/upgrade/configure_tls_settings.rb 2
Post Audit Vault Server and Agents upgrade, run the following command as root
user:
/usr/local/dbfw/bin/priv/configure-networking --agent-tls-cipher-level 4
Run the following command post upgrade, if it is only displayed
on the prompt:
/usr/local/dbfw/bin/priv/send_agent_update_signal.sh
Refer to Oracle AVDF Installation Guide, sections "Pre-upgrade RPM Legacy
Crypto Check Warning" and "Post Upgrade TLS Security Hardening" for more
details.
Upgrading from Oracle AVDF 12.2.0.12.0 and Later
When upgrading from Oracle AVDF 12.2.0.12.0 and later, the pre-upgrade RPM displays the following warning. Follow the instructions in the warning to resolve the issue.
If you have deployed Audit Vault Agents on AIX in your environment, TLS 1.1
should be used for encryption instead of the default version of TLS 1.2. Else,
the Agents on AIX will not upgrade automatically. If you wish to use TLS 1.1 for
encryption run the below command before proceeding with the
upgrade.
ruby /usr/local/dbfw/bin/upgrade/configure_tls_settings.rb 2
Post Audit Vault Server and Agents upgrade, run the following command as root
user:
/usr/local/dbfw/bin/priv/configure-networking --agent-tls-cipher-level 4
Run the following command post upgrade, if it is only displayed
on the prompt:
/usr/local/dbfw/bin/priv/send_agent_update_signal.sh
Refer to Oracle AVDF Installation Guide, sections "Pre-upgrade RPM Legacy
Crypto Check Warning" and "Post Upgrade TLS Security Hardening" for more
details.
A.11.12 Pre-upgrade RPM Fails with "Not All Processes Were Stopped"
Problem
The pre-upgrade RPM fails with the following warning: Not all processes were stopped: 7378,7379.
For example:
rpm -ivh --force avdf-pre-upgrade-20.x.0.0.0-0_NNNNNN.NNNN.x86_64.rpm
Preparing... ########################################### [100%]
1:avdf-pre-upgrade ########################################### [100%]
Checking upgrade preconditions
/bin/df: '/var/dbfw/upgrade': No such file or directory
/bin/df: no file systems processed
Shutting down services.
Traceback (most recent call last):
3: from /usr/local/dbfw/bin/pre_upgrade.rb:642:in '<main>'
2: from /usr/local/dbfw/bin/pre_upgrade.rb:614:in 'process_command_line'
1: from /usr/local/dbfw/bin/pre_upgrade.rb:503:in 'post_install'
/usr/local/dbfw/lib/ruby/upgrade/common.rb:621:in 'stop_nonroot_processes':
Not all processes were stopped: 7378,7379
Cause
This issue could be caused by an idle SSH session, busy devices, or open temporary files.
Solution
-
Uninstall the RPM as the
root
user.-
Log in to the Audit Vault Server through SSH and switch to the
root
user. -
Uninstall the pre-upgrade RPM by using one of the following commands:
rpm -e avdf-pre-upgrade
rpm -e avdf-pre-upgrade --noscripts
-
-
Check the pre-upgrade RPM listing.
-
Enter the following command:
rpm -qa |grep avdf-pre-upgrade
- Ensure that there's no entry for
avdf-pre-upgrade
RPM. - Reboot the Audit Vault Server if it's a
STANDALONE
system.
-
-
Check for other SSH sessions, busy devices, or temporary open files.
-
Ensure that there are no other SSH sessions that are owned by the
support
user.To do this, identify idle notty (no tty) SSH sessions and try to stop them.
Use the following commands to check the
pid
ofsshd: support@notty
.ps -ef |grep support
ps -ef |grep notty
For example:
support 2480 2427 0 18:31 ? 00:00:00 sshd: support@notty support 2481 2480 0 18:31 ? 00:00:00 -bash kill -9 2481 kill -9 2480
- Check again for
support@notty
processes in the system. -
Ensure that the system doesn't have any busy devices or open temporary files. To do this, run
lsof
against/tmp
and/usr/local/dbfw/tmp
.For example:
lsof /usr/local/dbfw/tmp lsof /tmp
Note:
Ensure that no logs are open when starting the patching or upgrade process.
-
-
Try to install the pre-upgrade RPM as the
root
user.-
Log in to the Audit Vault Server through SSH and switch to the
root
user. -
Enter the following command:
rpm -i /root/avdf-pre-upgrade-20.x.0.0.0-0_NNNNNN.NNNN.x86_64.rpm
-
A.11.13 Pre-upgrade RPM Check: Agent Failure Checks - Upgrade Prerequisites
Starting with Oracle AVDF 20.9, the pre-upgrade RPM verifies that the Audit Vault Agent and Host Monitor Agent configurations are compatible with Oracle AVDF 20.10 or later.
Problem
The agent_prereq_checks_failure_report.txt
report
indicates that a Audit Vault Agent or Host Monitor Agent doesn't meet the
prerequisites to update to Oracle AVDF 20.10 or later. You can find the agent
success and failure reports in the following locations:
- Success report:
/opt/avdf/report/agent_prereq_checks_success_report.txt
- Failure report:
/opt/avdf/report/agent_prereq_checks_failure_report.txt
The following example shows a failure message:
Agent/HM Validation Failure statuses are as below :
------------------------------------------------------------------
Agent Name : agent-linux
Agent Validation Status : FAILURE
Agent Failure Checks : Upgrade Prerequisites check jar build with latest version. Please check the minimum java version required. - <Exception Message>
Agent Checks Warning Messages :
Validated at : 2022-12-02 09:11:24.774880
Solution
Resolve the issue that's indicated in the report. For example, update the Audit Vault Agent machine to the minimum Java version that's supported.
You can rerun the failure check scripts individually to verify that the issues are
resolved. Run these scripts as the root
user.
/usr/bin/python3 /usr/local/dbfw/bin/upgrade/pre_upgrade_validate_agent.py standalone
/usr/bin/python3 /usr/local/dbfw/bin/upgrade/pre_upgrade_download_agent_validation_status.py standalone
A.12 SSH Becomes Disabled After Updating Oracle AVDF with FIPS Enabled
If SSH becomes disabled after updating Oracle AVDF with FIPS mode enabled, update the SSH keys to be compliant with FIPS.
Problem
After updating Oracle AVDF to release 20.9 with FIPS mode enabled, SSH becomes disabled.
Solution
Before enabling FIPS 140-2, ensure that your SSH keys are compliant with FIPS. If your SSH keys are not compliant with FIPS, the SSH connection with the appliance might be lost after enabling FIPS.
For Oracle AVDF on Oracle Cloud Infrastructure (OCI), before
enabling FIPS mode, ensure that the opc
user has FIPS-compliant
keys registered to /home/opc/.ssh/authorized_keys
.
Follow these steps to resolve this issue:
-
Log into the Audit Vault Server console and disable FIPS mode.
-
Log back into the appliance through SSH and check or update the user keys for SSH-enabled users in
~/.ssh/authorized_keys
to be compliant with FIPS.It can take several minutes for the console to become available after enabling or disabling FIPS mode.
-
Enable FIPS mode.
Related Topics
A.13 SSH Connection Times Out When Uninstalling the Pre-Upgrade RPM
Problem
The SSH connection times out when uninstalling the pre-upgrade RPM.
Cause
The default SSH connection timeout is 10 minutes, and uninstalling the pre-upgrade RPM can take longer than 10 minutes.
Solution
Run the screen
command before uninstalling the pre-upgrade RPM. The
screen
command prevents network disconnections from
interrupting the patching or upgrading.
If the session terminates, resume by switching to the root
user and
then running the screen -r
command.
A.14 Installation Pauses After Entering the Root Password
Problem
When you start the installer for Oracle AVDF 20.5, it installs a few packages and prompts you to change the root password. After you enter the new root password, the installer immediately display some unmount commands and returns to the starting installation screen. You're unable to proceed with the installation.
Cause
The ISO file was removed before the installation was completed.
Solution
After you enter the new root password and return to the starting installation screen, complete the following steps:
- Remove the ISO CD from the CD drive and restart the machine.
- When you're promoted to log in, log in as
ROOT
. - When promoted for the ISO file, add the ISO file from the media.
A.15 When Upgrading to Oracle AVDF 20.3
ELMIG_POPULATE_CLUSTERS_202
and ELMIG_CONVERT_HASH_202
Are Reported as INVALID
in dba_objects
Table
Even though the objects are invalid this doesn't have any impact on the system operation and can be ignored.
Problem
When upgrading to Oracle AVDF 20.3 the objects
ELMIG_POPULATE_CLUSTERS_202
and
ELMIG_CONVERT_HASH_202
are reported as INVALID
in dba_objects
table.
ELMIG_POPULATE_CLUSTERS_202
and
ELMIG_CONVERT_HASH_202
objects.
elect object_name from dba_objects where status = ‘INVALID’;
OBJECT_NAME
Solution
This doesn't have any impact on the system operation and can be ignored.
A.16 Error Occurred Trying to Format
SDAF1
When Installing Oracle AVDF
Problem
During the installation of Audit Vault Server,the following error is encountered:
Error: An error occurred trying to format sdaf1. This problem is serious, and
the install cannot continue. Press to reboot your system.
Cause
The server has SAN connectivity.
Solution
Disable the SAN connectivity. ISCSI device should not be attached until Audit Vault Server installation is completed.
A.17 Audit Vault Agent Failed on Startup:
OAV-10: Failed to Release Connection to DB
Problem
When installing the Audit Vault Agent error OAV-10: Failed to Release
Connection to Database
occurred when ./agentctl start
-k
was executed. The database to Audit Vault Server connection
failed.
Cause
The wrong location was used for JAVA_HOME
and
agentctl
picked up a different Java in the path. The connection
failed as it does not work with Java that is present in the database home.
Solution
JAVA_HOME
:JAVA_HOME=/usr/java/jdk1.8.0_361
export PATH=$JAVA_HOME/bin:$PATH
A.18 Upgrade to AVDF 20.4 Failed During
upgrade_apex
Step
When upgrading to AVDF 20.4, the upgrade_apex
step results
in ODF-10001: Internal error: FAILED migration: upgrade_apex (as oracle)
error.
Problem
/var/log/messages
contains ERROR - ODF-10001: Internal
error: FAILED migration: upgrade_apex (as oracle) (applied change)
and
/var/log/debug
contains
upgrade_apex: error: cannot create /var/lib/oracle/dbfw/apex/images/computer.gif
upgrade_apex: Permission denied
upgrade_apex: error: cannot create /var/lib/oracle/dbfw/apex/images/phone_support.gif
upgrade_apex: Permission denied
root
user:
/opt/avdf/bin/privmigutl –status
results in
System state - recovery
Migration set 'AVS' - failed
Last migration 'Upgrading apex20' - failed
Solution
-
Log in to the Audit Vault Server through SSH and switch to the
root
user. - Run the following
command:
chown -R oracle:oinstall /var/lib/oracle/dbfw/apex/images
-
Switch to the
oracle
user.su - oracle
- Run the following
script:
/usr/local/dbfw/etc/privileged-migrations/upgrade_apex
- Run
echo $?
If the result is 2, then the script has completed successfully.
-
Log in to the Audit Vault Server through SSH and switch to the
root
user. - Resume the upgrade process by running the following:
/opt/avdf/bin/privmigutl --resume --confirm
Make the sure ssh connection to the Oracle AVDF server is reliable and does not terminate while running this command.
- Check if the
$ORACLE_HOME/apex/images
folder and its contents haveoracle:oinstall
permission, and if not, grant these permissions.
Related Topics
A.19 Missing "Save as" Option in Web Console After Upgrading Oracle Audit Vault Server
Problem
After upgrading the Audit Vault Server from Oracle AVDF 12.x to 20.x, the "Save As" option is no longer available in the UI Web Console on certain pages. Previously, this option was used to save pages from the Audit Trail Table Listing and Host Table Listing, now making it difficult for users to generate reports in newer versions.
Cause
Customers rely on the "Save As" option to create reports from the Audit Trail and Host Table Listings for audit reviews and management presentations. The new UI in versions 20.1 to 20.4 removed this option, causing a disruption in existing workflows for auditors.
Solution
To work around the missing "Save As" feature, users can manually run SQL queries to extract the necessary data. These queries can then be formatted into HTML or PDF reports using SQL developer, SQL*Plus, or other tools.
- Audit Trail Query: This query can be used to get a table as seen on the Audit Trail page.
Select st.secured_target_name,atv.location,atv.audit_trail_type,atv.host_name,atv.status,stt.secured_target_type_name,atv.error_message,atv.collection_autostart,atv.trail_autostart_attempts,atv.trail_last_start_time, (select checkpoint_time from avsys.checkpoint c where c.audit_Trail_id = atv.audit_Trail_id )Last_Collection_Time from avsys.audit_trail_view atv, avsys.secured_target st,avsys.secured_target_type stt where atv.source_id = st.secured_target_id and st.secured_target_type_id = stt.secured_target_type_id and st.active = 'Y' and atv.active ='Y';
- Host Page Data Query: This query can be used to get table as seen on Host page.
select a.host_name, a.host_ip, a.activation_key, upper(a.status) as agent_status, a.activation_time, a.platform, a.agent_location, a.version, h.HOSTMON_INSTALL_STATE, h.HOSTMON_LOCATION, h.HOSTMON_VERSION, h.HOSTMON_ZIP_GEN_TIMESTAMP, h.HOSTMON_UPDATE_TIMESTAMP, h.AGENT_OS_USER, h.HOSTMON_ERROR_MSG, hs.STATUS_TIMESTAMP as last_connected_At from avsys.agent_view a, avsys.hostmon_view h, avsys.host hs where a.host_ip = h.host_ip and hs.host_ip = h.host_ip and hs.host_ip = a.host_ip and a.deleted_at is null and h.deleted_at is null and hs.deleted_at is null;
- Generating HTML Reports: Users can follow these steps to generate presentable HTML reports using SQL*Plus:
set LINESIZE 4000; set PAGESIZE 4000; column SECURED_TARGET_NAME format a20; column TRAIL_AUTOSTART_ATTEMPTS format 9; column AUDIT_TRAIL_TYPE format a10; column host_name format a25; column status format a12; column TRAIL_AUTOSTART_ATTEMPTS format 9; column location format a32; column TRAIL_LAST_START_TIME format a30; column LAST_COLLECTION_TIME format a30; column ERROR_MESSAGE format a80; set colsep '|' Select st.secured_target_name,atv.location,atv.audit_trail_type,atv.host_name,atv.status,stt.secured_target_type_name, atv.error_message,atv.collection_autostart,atv.trail_autostart_attempts,atv.trail_last_start_time, (select checkpoint_time from avsys.checkpoint c where c.audit_Trail_id = atv.audit_Trail_id ) Last_Collection_Time from avsys.audit_trail_view atv, avsys.secured_target st, avsys.secured_target_type stt where atv.source_id = st.secured_target_id and st.secured_target_type_id = stt.secured_target_type_id and st.active = 'Y' and atv.active ='Y';
Execute the following query to generate the report:
SET ECHO OFF SET PAGESIZE 4000 SET FEEDBACK OFF SET TERMOUT OFF SET MARKUP HTML ON TABLE "" SPOOL AuditTrail.html Select st.secured_target_name,atv.location,atv.audit_trail_type,atv.host_name,atv.status,stt.secured_target_type_name, atv.error_message,atv.collection_autostart,atv.trail_autostart_attempts,atv.trail_last_start_time, (select checkpoint_time from avsys.checkpoint c where c.audit_Trail_id = atv.audit_Trail_id ) Last_Collection_Time from avsys.audit_trail_view atv, avsys.secured_target st, avsys.secured_target_type stt where atv.source_id = st.secured_target_id and st.secured_target_type_id = stt.secured_target_type_id and st.active = 'Y' and atv.active = 'Y'; SPOOL OFF
A.20 Oracle AVDF 20.7 Installation Fails Due to Package Download Error
While installing Oracle AVDF 20.7, the installation failed to complete the Oracle Audit Server Installation.
Problem
The installation of Oracle AVDF 20.7 installation fails with the error:
Installation failed: Failed to complete the Oracle Audit Server Installation
Cause
localhost run-privileged-migrations[22598]:
com.oracle.dbfw.privilegedMigration DEBUG - yum: Error downloading packages:
localhost run-privileged-migrations[22598]:
com.oracle.dbfw.privilegedMigration DEBUG - yum:
avs-grid-<version>.x86_64: [Errno 256] No more mirrors to try.
This occurs when the installation ISO is unavailable during part of the installation or if network issues interrupt the download process.
Solution
- Verify the ISO file:
- Ensure the ISO file is completely downloaded and accessible throughout the installation.
- Verify the download by checking the file size and comparing the SHA-256 checksum. Run the following command on Linux to generate the checksum:
$ sha256sum Vpart_number.iso
- Confirm the checksum matches the value provided in the File Download dialog box.
- Check ISO Accessibility:
- After mounting the ISO to the Audit Vault Server (AVS) as root, confirm it is recognized by the system:
Look for a label likels /dev/disk/by-label
AVS_20_7_0_0_0
. - Mount the ISO with:
mount /dev/disk/by-label/AVS_20_7_0_0_0 /mnt
- Verify that the required package files are present:
find /mnt -name 'avs-grid*' -ls
- After mounting the ISO to the Audit Vault Server (AVS) as root, confirm it is recognized by the system:
- Network Stability Check:
- From the AV server, test the network stability with:
cat 'Copy one of the files from find /mnt -name "avs-grid*" -ls' > /dev/null
- If this command is slow, network slowness may be affecting the installation. Check
/var/log/messages
for additional error logs.
- From the AV server, test the network stability with:
A.21 Calculating Minimum Required In-Memory Size for AVDF to Prevent "Insufficient Memory" Errors
Learn how to find sufficient memory to populate tables to In-Memory area if you receive an "insufficient memory" error.
Problem
The AVDF system may display an "Insufficient memory" error if adequate memory is not allocated to the In-Memory area for storing EVENT_LOG
data.
Solution
EVENT_LOG
data in In-Memory for a one-month period, follow these steps:
- Run the Primary Query
Execute the following SQL query to determine the required memory allocation:
SELECT NVL(MAX((SUM(msize)) / (SELECT EXTRACT(DAY FROM (partition_end - partition_start)) FROM avsys.dw_partition_view WHERE partition_name=pname)), 0) FROM (SELECT s.partition_name pname, (i.inmemory_size + i.bytes_not_populated) msize FROM user_tab_subpartitions s, v$im_user_segments i WHERE s.subpartition_name=i.partition_name AND s.table_name='EVENT_LOG' AND i.segment_name='EVENT_LOG') GROUP BY pname;
- Calculate the Required Memory
- Multiply the output of the above query by
31*1.2
(for a maximum of 31 days in a month and an additional 20% buffer for days with more data). - The resulting value is the minimum required memory in bytes for one month.
- Multiply the output of the above query by
- Alternative Calculation (if the query returns 0)
If the primary query returns 0, run this alternative query to estimate the required memory:
SELECT MAX((SUM(r.bytes)) / (SELECT EXTRACT(DAY FROM (partition_end - partition_start)) FROM avsys.dw_partition_view WHERE partition_name=pname)) FROM (SELECT s.partition_name pname, u.bytes FROM user_tab_subpartitions s, user_segments u WHERE s.subpartition_name=u.partition_name AND u.segment_name='EVENT_LOG' AND s.table_name='EVENT_LOG') r GROUP BY pname;
- Multiply the output of this query by
31*0.8
(accounting for disk data compression by reducing memory by 20%). - This result provides the minimum required memory in bytes for one month.
- Multiply the output of this query by
EVENT_LOG
data.
A.22 Upgrading 20.12 to 20.13 Fails on VMware With Error at Privileged Migrations Step
Learn how to resolve a privileged migrations error when upgrading from Oracle AVDF 20.12 to 20.13.
Problem
run-privileged-migrations ERROR - ODF-10001: Internal error: Fatal error running migrations
Solution
To resolve this issue, first confirm that you are experiencing the same error. If so, follow the subsequent steps:- As the
root
user, check the integrity of the RPM database:
If there are no errors, these instructions do not apply, contact Oracle Support.cd /var/lib/rpm /usr/lib/rpm/rpmd_verify Packages
- If errors are found, execute the following commands to rebuild the RPM database:
cd /var/lib cp -ax --backup=t rpm rpm.old rm -i rpm/__db.??? rpm --rebuilddb
- Once you have rebuilt the RPM database, check the validity of the rebuilt package database:
cd /var/lib/rpm /usr/lib/rpm/rpmdb_verify Packages
- Once confirmed, proceed with the upgrade according to the specific type of RPM database corruption encountered. Follow the appropriate steps based on the scenario experienced:
- Resume the upgrade if the privileged migrations have not yet started:
- Reboot the system.
- Log in as the
root
user. - Run the following command:
systemctl isolate avdf-upgrade.target
- To review the upgrade status, re-log in on the console as the
root
user.
- Resume the upgrade after the privileged migrations have started:
- Apply the AVDF 20.13 update to the recovery utility:
rpm -U /media/avdf-install/bootstrap/Packages/avdf-bootstrap-20.13.0.0.0-*.noarch.rpm
- Check the current status:
/opt/avdf/bin/privmigutl --status
- Review the output to find the failing migration and re-run it manually as the
root
user. - Once the migration has completed successfully, run the following command:
/opt/avdf/bin/privmigutl --resume
- Apply the AVDF 20.13 update to the recovery utility:
- Resume the upgrade if the privileged migrations have not yet started:
A.23 Package Version Mismatch After Patching Leading to Perl Package Update Failure
Learn how to update various Perl packages which are causing issues when patching from Oracle AVDF 20.9 to newer versions.
Problem
After patching to AVDF 20.9, certain systems have an outdated set of Perl packages,
such as perl-interpreter
, perl-libs
, and
perl-Utils
, which do not match versions available in a fresh
install. This mismatch does not introduce CVEs (as per Qualys reports) but could
lead to dependency conflicts, especially when attempting further upgrades.
Solution
Perl-devel
package and any other packages that depend on it, then manually upgrade the Perl packages using the following workaround:
- Run the following command before beginning to patch:
dnf remove perl-devel
- Insert the upgrade ISO.
- Mount it with
mount /dev/sr0 /images
. - Run the update command:
This command erases the packages pre-patching, upgrades all the outdated Perl packages, aligning the system with the expected package set, and successfully resolves dependency conflicts./usr/bin/yum update --exclude=avs,dbfw-mgmtsvr -c /images/upgrade.repo