12.3 Troubleshooting Oracle VM Manager

This section describes some problems you may encounter when using Oracle VM Manager, and explains how to resolve them.

12.3.1 Oracle VM Manager Log Files

Oracle VM Manager error messages are displayed in the user interface, in the Jobs tab, in the object's Events list and are also available in log files. Log files are stored in the following directory on the Oracle VM Manager host computer:

/u01/app/oracle/ovm-manager-3/domains/ovm_domain/servers/AdminServer/logs

Oracle VM Manager that you can use for troubleshooting are as follows:

Log file

Description

access.log

Tracks HTTP access to the Web interface of the Oracle VM Manager and to the underlying Oracle WebLogic Server HTTP interface. This log can be used to track access and HTTP operations within Oracle VM Manager to help debug access issues and to audit access to the Oracle VM Manager.

AdminServer.log

Tracks events within the underlying Oracle WebLogic Server framework, including events triggered by Oracle VM Manager. This log can be used to track a variety of issues within Oracle VM Manager including TLS/SSL certificate issues, server availability issues, and any actions performed within Oracle VM Manager which are usually identifiable by searching for items containing the string com.oracle.ovm.mgr. Log in failures resulting from locked accounts (as opposed to incorrect credentials) are also in this file.

AdminServer-diagnostic.log

Tracks exceptions within the underlying Oracle WebLogic Server framework, including particular events triggered by Oracle VM Manager such as log in failures due to incorrect credentials. This log can be used to track Oracle VM Manager behavior that results in an exception or for log in failure, which can be tracked by searching for the string An incorrect username or password was specified.

Because log file format is determined by Oracle WebLogic Server, many of these files may be difficult to read. A log parsing tool is included with Oracle VM Manager to help extract useful information from the actual log files. The log parsing tool is named OvmLogTool.py and is located at:

/u01/app/oracle/ovm-manager-3/ovm_tools/bin

OvmLogTool.py can do the following things:

  • Convert and combine all the AdminServer log files into one easier-to-read file.

  • Create a filtered summary log file that only lists errors.

  • Tail the AdminServer log, applying the filtering on the fly.

Usually analysis of the logs starts by generating an error summary log. The summary file can act as an index into the filtered file to investigate and analyze errors, providing you with time stamps an a shortened summary of each error that may need further investigation. To generate a summary log file, do the following:

# python OvmLogTool.py -s -o summary
processing input file: /u01/app/oracle/ovm-manager-3/domains/ovm_domain/servers/
AdminServer/logs/AdminServer.log00001
processing input file: /u01/app/oracle/ovm-manager-3/domains/ovm_domain/servers/
AdminServer/logs/AdminServer.log

This generates a file named summary in the local directory. You can use this to look for errors that occurred within Oracle VM Manager.

To get a full log of all events and errors within Oracle VM Manager you can do the following:

# python OvmLogTool.py -o filteredlog
processing input file: /u01/app/oracle/ovm-manager-3/domains/ovm_domain/servers/
AdminServer/logs/AdminServer.log00001
processing input file: /u01/app/oracle/ovm-manager-3/domains/ovm_domain/servers/
AdminServer/logs/AdminServer.log

This generates a file named filteredlog in the local directory. You can use this to look for all events that occurred within Oracle VM Manager.

Finally, you can use OvmLogTool.py to filter results on the fly while tailing the log:

# python OvmLogTool.py -t
tailing log file: /u01/app/oracle/ovm-manager-3/domains/ovm_domain/servers/
AdminServer/logs/AdminServer.log

Use Ctrl+C to exit the program when you have finished tailing the log file.

12.3.2 Oracle VM Manager Web Interface Database Synchronization

The Oracle VM Manager Web Interface and the Oracle VM Manager Command Line Interface both use a representation of the Oracle VM data model to to more quickly retrieve data objects, which optimizes the performance of Oracle VM Manager. The representation of the data model is saved to a separate database to the primary Oracle VM Manager database. This separate database relies on events from Oracle VM Manager to stay synchronized with the actual data model.

It is possible, in specific cases, for the representation of the data model to become out of sync with the actual data model. As a result, some objects in the Oracle VM Manager Web Interface do not reflect the actual environment. A typical scenario where this may happen is during a virtual machine clone operation that fails. During this process, the virtual machine is actually created within the data model and the database used by the user interface layer. If a part of the whole operation fails, Oracle VM Manager attempts to clean up and roll back, resulting in the virtual machine being removed from the data model. However, in this case, an event is not generated to notify that the clean up has succeeded, and the virtual machine information remains in the user interface database. The result is that the cloned virtual machine is still shown in the Oracle VM Manager Web Interface and the Oracle VM Manager Command Line Interface even though it does not actually exist in the environment.

The user interface database is not resynchronized automatically when the service is restarted, as this can take a long time for large environments. To force database resynchronization, you must create a file on the Oracle VM Manager host before restarting the service. The following instructions explain how to force resynchronization.

Steps to resynchronize the Oracle VM Manager Web Interface database

  • You must access the shell of the Oracle VM Manager host, either directly or over SSH.

  • Change user to the 'oracle' user, using su.

    # su - oracle
  • Touch a file called /tmp/.resyncUI as the oracle user.

    $ touch /tmp/.resyncUI

    If you are unable to do this as the oracle user, then touch the file as any other user but ensure that its permissions are such that any other user can delete the file:

    # touch /tmp/.resyncUI
    # chmod 666 /tmp/.resyncUI
  • Restart the Oracle VM Manager service as root:

    # service ovmm restart

12.3.3 Increasing the Memory Allocated to Oracle WebLogic Server

In environments that host thousands of virtual machines and where large data sets exist, performance issues can occur with Oracle VM Manager, such as garbage collection taking a long time or out of memory conditions arise. Likewise, Oracle VM Manager can generally be slow to respond and a significant performance degradation occurs. To resolve these performance issues, you can increase the amount of memory that is allocated to Oracle WebLogic Server.

Note

You should not increase the memory allocation to the maximum limit that is available to the host server. As a guideline, you should leave at least 2GB available memory for the host operating system and other services.

To increase the memory allocated to Oracle WebLogic Server, do the following:

  1. Start an ssh session to the Oracle VM Manager host computer as the root user.

  2. Open the following file for editing: /etc/sysconfig/ovmm

  3. Specify the amount of memory allocated to Oracle WebLogic Server as the value for the JVM_MEMORY_MAX property.

    Note

    The default is JVM_MEMORY_MAX=4096m.

  4. Save and close /etc/sysconfig/ovmm.

  5. Restart the Oracle VM Manager service.

    # service ovmm restart

12.3.4 No File Systems Found When Searching a Storage Server

On storage servers that have a very large number of file systems available, the UI may time out while refreshing the list of available file systems, resulting in a No File Systems Available message. This usually means that the time out value is set too low for the number of file systems that the UI needs to refresh. To resolve this, change the settings for the Refresh Timeout Value in the Preferences Pane on the Tools and Resources tab in the Oracle VM Manager Web Interface to increase the timeout value.

See the Preferences section in the Oracle VM Manager Online Help for more information on Oracle VM Manager UI preferences.

12.3.5 Cannot Discover Servers to Oracle VM Manager Due To Time Differences

Oracle VM Manager uses certificate-based authentication to maintain secure communication with the Oracle VM Agent that runs on each Oracle VM Server instance. This means that the system time on each Oracle VM Server must be within the bounds of the certificate valid from and valid to timestamps, or certificate validity is challenged and Oracle VM Manager is unable to authenticate to the Oracle VM Agent.

In most instances, this is not a problem, since servers are automatically configured to use the Oracle VM Manager as an NTP server as soon as ownership is taken. However, during server discovery the Oracle VM Manager uses a password to perform its initial authentication against a server and to provide its certificate for ongoing communication. During this phase of discovery, a check is performed to ensure that the system time on the Oracle VM Server is within the bounds required for certificate authentication to take place. If this is not the case, discovery fails and an error message is returned:

OVMAPI_4024E: Cannot take ownership of server myserver5.virtlab.info because the date 
and time on the server are outside the valid range for the manager's SSL certificate.  
The certificate is valid from 10/24/13 7:37 PM to 10/25/23 7:37 PM, but server's timestamp 
is 9/28/13 8:14 PM.  Please verify the server's NTP and time settings.

In this situation, it is necessary that you access the affected Oracle VM Server directly and update its system time manually before attempting to rediscover the server using Oracle VM Manager. Once Oracle VM Manager has completed discovery and is able to take ownership of the server, its NTP configuration is updated automatically and it remains synchronized with the Oracle VM Manager host.

12.3.6 Cannot Create a Clustered Server Pool on a Disk that already has an OCFS2 File System

If you attempt to create a clustered server pool using a disk located on a storage device that already contains an OCFS2 file system, the Oracle VM Agent on the server detects the file system and refuses to overwrite it. This is normal behavior and protects you from accidentally setting up two OCFS2 file systems with matching UUIDs on the same disk, leading to instability and unexpected behavior.

If you are certain that the existing OCFS2 file system that is already present on the disk is no longer in use by any other server pool or repository, you can clean the disk by connecting to the Oracle VM Server and issuing the following command:

dd if=/dev/zero of=/dev/mapper/360a98000433468704234786f36394763 bs=1M count=256

Replace /dev/mapper/360a98000433468704234786f36394763 with the correct path to the disk device where you are creating the new server pool cluster.

Warning

Using dd is data destructive. Make sure you are certain about the disk device name and that the OCFS2 file system that you are deleting is no longer in use. It is advisable to make backups of any data that exists on the disk that you are editing before proceeding. This operation should be performed by a skilled systems administrator.

12.3.7 Cannot Create a Repository on a Device that has Partitions

A repository cannot be hosted on a physical disk that has pre-existing partitions. If you attempt to create a repository on a disk that already has a partition, an error is generated notifying you that the backing device is not allowed to contain partitions. If you are intent on creating a repository on the selected disk, you must delete any pre-existing partitions. This may require you to directly access the Oracle VM Server where the disk is mounted and to manually remove the partition objects on the disk using the fdisk command. For example:

# fdisk /dev/sdb


WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): p

Disk /dev/sdb: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0001e791

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1        6528    52428800   83  Linux

Command (m for help): d 
Partition number (1): 1

Command (m for help): w
Warning

Using fdisk is data destructive. Make sure you are certain about the disk device name and partitions that you are deleting. It is advisable to make backups of any data that exists on the disk that you are editing before proceeding. This operation should be performed by a skilled systems administrator.

Usually, restarting the affected Oracle VM Server after performing these operations is advisable. In the case of an iSCSI disk, connections to targets need to be re-initiated. In all cases, the storage needs to be refreshed. Simply restarting the Oracle VM Server ensures that all necessary actions are performed.

Once the Oracle VM Server has restarted, you may attempt to recreate the repository.

12.3.8 Removing Oracle VM Template Configuration Packages on Oracle Linux 5 Hosts

If you are running Oracle VM Manager on an Oracle Linux 5 host, removing all ovm-template-config related RPM packages in a single command results in an error and some packages are not successfully removed.

To resolve this, you must use two yum erase operations and ensure that ovm-template-config is removed last, as follows:

#yum erase ovm-template-config-authentication \
            ovm-template-config-datetime \
            ovm-template-config-firewall \
            ovm-template-config-network \
            ovm-template-config-selinux \
            ovm-template-config-ssh \
            ovm-template-config-system \
            ovm-template-config-user

#yum erase ovmd libovmapi xenstoreprovider ovm-template-config

12.3.9 Unable to Manage Oracle VM Server for x86 at Release 3.2.10 or 3.2.11, and Oracle VM Agent for SPARC at Release 3.3.1

As of Oracle VM Release 3.4.5, Oracle VM Manager uses the TLS version 1.2 (TLSv1.2) protocol for all connections for security reasons. As a result, management of Oracle VM Server for x86 at Release 3.2.10 or 3.2.11, and Oracle VM Agent for SPARC at Release 3.3.1, is not possible by default. As a workaround, you must enable the TLSv1 protocol, which is less secure.

For instructions on how to do this, see Enabling the TLS Version 1 Protocol.