C.7 Installation and Upgrade Issues

Review these troubleshooting tips for common installation and upgrade issues when working with Oracle Key Vault.

Parent topic: Troubleshooting Oracle Key Vault

C.7.1 Oracle Key Vault Installation Failure

Oracle Key Vault installation is failing.

Probable Cause

Oracle Key Vault Installation may fail due to various reasons listed with solutions in the Solutions section.

Solution

  1. Check if the physical server has RAID enabled software.

    Note:

    Oracle Key Vault does not support the use of Software RAID. Ensure that RAID is configured only at the hardware level.
  2. On the Oracle Key Vault server, check the network settings. Also check if there is a delay in copying the iso files from the NFS location to the disk.
  3. Check system configuration including disk size, number of disks, disk configuration, RAM size, and number of CPU cores. Ensure that the server meets the minimum system configuration requirements of Oracle Key Vault.
  4. Verify if the checksum matches with the local iso file.
    1. Copy the checksum of the iso from the specific location where you downloaded the iso file.
    2. Generate the checksum for the downloaded iso file.
    3. Both the checksum values should be the same. If not, download the correct iso file.
      sha256sum isoname.iso
    4. Check if the issue is resolved.

Parent topic: Installation and Upgrade Issues

C.7.2 Oracle Key Vault Upgrade Failure

Oracle Key Vault upgrade fails with file descriptor (FD) open error.

Probable Cause 1

Oracle Key Vault upgrade version is not supported.

Solution

Verify that the Oracle Key Vault upgrade from the current release to the version you want to upgrade is supported. See, Release Notes of specific Oracle Key Vault release for supported upgrade paths.

Probable Cause 2

Oracle Key Vault certificates have expired.

Solution

  1. Check if the Oracle Key Vault server or node certificates have expired.
    1. Log in to the Oracle Key Vault management console as a user who has the system administrator role.
    2. To check the server or node certificate expiration status, go to System, Status and check the Server or Node Certificate Expiration Date.
    3. Regenerate the server certificate if it has expired. Go to System, Status.
    4. Click Service Certificates and select Manage Server or Node Certificate to regenerate the new server certificate.
  2. Check if the Oracle Key Vault CA certificate has expired.
    1. Log in to the Oracle Key Vault management console as a system administrator.
    2. To check the CA certificate expiration status, navigate to System, Status and check the CA Certificate Expiration Date.
    3. If the CA certificate has expired, contact Oracle support.
  3. Verify if all the pre-upgrade steps are performed as described in the Oracle Key Vault Installation and Upgrade guide.

    Note:

    For Oracle Key Vault 21.5 or higher, see the steps listed inManaging CA Certificate Rotation after it has expired, to rotate the expired CA certificate.

Considerations for Multi-Master Cluster Deployment

  1. Log in to each node's management console, and check if the replication lag or heartbeat lag is high in other nodes. If yes, then resolve this issue first. See,Heartbeat Lag or High Replication Lag in Multi-Master Cluster Environment.
  2. Based on the upgrade scenario, certain restrictions may apply when upgrading nodes of a read/write pair. This includes the order in which the nodes are disabled, upgraded, and then re-enabled after the upgrade. See, Oracle Key Vault Installation and Upgrade guide of the target version for specific instructions.
  3. Verify if the node is not disabled beyond the maximum node disabled duration limit.

Considerations for Primary-Standby Environment

  1. Verify if the upgrade is performed first on standby and then on the primary. See,Upgrading a Pair of Primary-Standby Oracle Key Vault Servers .
  2. Retry the upgrade. If the issue persists, contact Oracle Support.

Parent topic: Installation and Upgrade Issues

C.7.3 Oracle Key Vault Management Console is Not Accessible After Installation

The Oracle Key Vault management console is not accessible after rebooting the Oracle Key Vault server as part of the installation on VMWare virtual machine.

Example

On the Oracle Key Vault server, the /var/log/messages shows the error, 

OAV-46501: invalid IP Address.
java.sql.SQLException: OAV-46501: invalid IP Address.

Probable Cause

The HTTPd server daemon on the Oracle Key Vault server is not started. Oracle Key Vault faced issues in identifying the network adapter on VMware.

Solution
  1. Change the network adapter settings in VMware. Change the Virtual Machine from VMXNET3 to e1000.
  2. Reboot the Oracle Key Vault server.

Parent topic: Installation and Upgrade Issues

C.7.4 Oracle Key Vault Upgrade Failure

Oracle Key Vault upgrade fails with an error.

Example

Oracle Key Vault upgrade fails with,


Open FDs on: Volume: lv_tmp, Process: sshd
Sample Error Output
[root@server ]# /usr/bin/ruby /images/upgrade.rb --confirm
Power loss during upgrade may cause data loss. Do not power
off during upgrade.
Verifying boot partition before upgrade
Verifying upgrade preconditions
The Oracle base has been set to /var/lib/oracle
The Oracle base has been set to /var/lib/oracle
The Oracle base has been set to /var/lib/oracle
The Oracle base has been set to /var/lib/oracle
The Oracle base has been set to /var/lib/oracle
The Oracle base has been set to /var/lib/oracle
The Oracle base has been set to /var/lib/oracle
The Oracle base has been set to /var/lib/oracle
The Oracle base has been set to /var/lib/oracle
The Oracle base has been set to /var/lib/oracle
The Oracle base has been set to /var/lib/oracle

AVDF::Installer::Upgrade::LVOpenFD

Open FDs on: Volume: lv_tmp, Process: sshd, File(s): /tmp/ssh-<xxxxxx>/agent.1662

Failed to apply update: Verifying pre-upgrade conditions failed.
Failed to apply update: /images/upgrade/lib/preconditions.rb:2189:in 'verify_all'
/images/upgrade.rb:203:in 'upgrade'
/images/upgrade.rb:282:in '<main>

Probable Cause

This error is due to a running SSH process.

Solution

  1. Verify if any SSH process is running on the Oracle Key Vault server:
    $ ps -eaf | grep -i "ssh"
  2. Stop the running SSH process and retry the upgrade.

Parent topic: Installation and Upgrade Issues

C.7.5 Unable to boot after installation of Oracle Key Vault on VMWare VM

Unable to boot the Virtual Machine (VM) after Oracle Key Vault installation.

Example

After reboot, the following error message appears on VM console.

PXE-M0F: Exiting Intel PXE ROM
         Operating system not Found

Probable Cause

The virtual disk is set to 2 TB and is not allowing the server to boot.

Solution

  1. Set the virtual disc size to 1.9 TB.
  2. In vSphere 5.5.x and 6.0.x, large capacity virtual disks have these conditions and limitations:
    • The maximum supported VMDK size on an VMFS-5 data store is increased to 62 TB. However, the maximum supported VMDK size on VMFS-3 is still 2 TB.
    • The maximum supported size of a VMDK on NFS is the lesser of 62 TB and 1% less than the maximum file size supported by the NFS filesystem.

Parent topic: Installation and Upgrade Issues

C.7.6 Operation Failed on Network Information Screen After Upgrade from 21.x to 21.5 and Later.

Network information screen is displaying Operation failed error.

Example

The network information screen is displaying Operation failed error while fetching the network interfaces details.

Probable Cause

The network configuration file name is not changed to the supported naming convention on 21.5 or later.

Solution

  1. Go to /etc/sysconfig/network-scripts/.
  2. List all the files and find if the ifcfg* files have d1 appended at the end. For example, /etc/sysconfig/network-scripts/ifcfg-bond0-eno3d1.
  3. Rename the files to the correct name, mv ifcfg-bond0-eno3d1 ifcfg-bond0-eno3.
  4. Reboot the server.

Parent topic: Installation and Upgrade Issues

C.7.7 Upgrade to Oracle Key Vault Failed During the upgrade_apex Step

When upgrading to Oracle Key Vault, the upgrade_apex step results in ODF-10001: Internal error: FAILED migration: upgrade_apex (as oracle) error.

Problem

The /var/log/messages contains ERROR - ODF-10001: Internal error: FAILED migration: upgrade_apex (as oracle) (applied change) and /var/log/debug contains 
upgrade_apex: error: cannot create /var/lib/oracle/dbfw/apex/images/computer.gif
upgrade_apex: Permission denied
upgrade_apex: error: cannot create /var/lib/oracle/dbfw/apex/images/phone_support.gif
upgrade_apex: Permission denied
In addition, running the following command as the root user: 
/opt/avdf/bin/privmigutl –status
results in 
System state - recovery
Migration set 'AVS' - failed
Last migration 'Upgrading apex20' - failed

Solution

  1. Log in to the Oracle Key Vault Server through SSH and switch to the root user.
  2. Run the following command:
    chown -R oracle:oinstall /var/lib/oracle/dbfw/apex/images
  3. Switch to the oracle user.
    su - oracle
  4. Run the following script:
    /opt/avdf/install/privileged_migration/upgrade_apex
  5. Run echo $?.

    If the result is 2, then the script has completed successfully.

  6. Log in to the Oracle Key Vault Server through SSH and switch to the root user.
  7. Resume the upgrade process by running the following command: 
    /opt/avdf/bin/privmigutl --resume --confirm

    Make sure the ssh connection to the Oracle Key Vault server is reliable and does not terminate while running this command.

  8. Check if the $ORACLE_HOME/apex/images folder and its contents have oracle:oinstall permission. If not, grant the permission.

Parent topic: Installation and Upgrade Issues

C.7.8 Upgrading Oracle Key Vault Fails When Rebuilding the Index with UTLRP.SQL

When attempting to upgrade to Oracle Key Vault, it fails while rebuilding the index by executing UTLRP.SQL after setting max_string_size to extended.

Problem

Upgrading to the current version of Oracle Key Vault fails while rebuilding the index with UTLRP.SQL. You may receive the following error logged in /var/log/messages:
HOSTNAME su: (to oracle) root on none
HOSTNAME com.oracle.privilegedMigration.max_string_size_extended: Failed utlrp.sql to setup MAX_STRING_SIZE to extended.
HOSTNAME run-application-migrations[28910]: com.oracle.dbfw.privilegedMigration ERROR - ODF-10001: Internal error: FAILEDmigration: max_string_size_extended (as root) (applied change)

Cause

The upgrade fails while rebuilding the index by executing UTLRP.SQL after setting max_string_size to extended.

Solution

To resolve this issue and resume upgrade, follow these steps:
  1. Connect to Oracle Key Vault database as sys -
    sqlplus / as sysdba
@/var/lib/oracle/dbfw/rdbms/admin/utlrp.sql
DECLARE

    The query fails with the following error: 

    ERROR at line 1: 
ORA-01502: index 'SYS.I_WRI$_OPTSTAT_HH_OBJ_ICOL_ST' or partition of such index is in unusable state
Function created.
PL/SQL procedure successfully completed.
Function dropped.
Warning: XDB now invalid
PL/SQL procedure successfully completed.
  2. Rebuild the package to fix the issue by executing the following query: 
    alter index SYS.I_WRI$_OPTSTAT_HH_OBJ_ICOL_ST rebuild;
  3. Execute the script again:
    @/var/lib/oracle/dbfw/rdbms/admin/utlrp.sql
  4. Run the following commands:
    alter package objowner.object compile body;
grant execute on DBMS_SQL to public;
  5. Check if any other invalid objects are present:
    SELECT owner, object_type, object_name FROM dba_objects WHERE status !='VALID';
  6. Drop the trigger which failed:
    drop trigger avsys.start_el_migration;
  7. Resume the upgrade: 
    /opt/avdf/bin/privmigutl --resume --confirm

Parent topic: Installation and Upgrade Issues