B.2 Troubleshooting Oracle VM Server

B.2.1 Debugging Tools
B.2.2 Using DHCP
B.2.3 Setting the Guest's Clock
B.2.4 Wallclock Time Skew Problems
B.2.5 Mouse Pointer Tracking Problems
B.2.6 Hardware Virtualized Guest Stops
B.2.7 Migration of Large Hardware Virtualized Guest Results in CPU Soft Lock
B.2.8 Hardware Virtualized Guest Devices Not Working as Expected
B.2.9 Firewall Blocks NFS Access
B.2.10 Migrating Virtual Machines

This section describes some problems you may encounter when using Oracle VM Server, and explains how to resolve them.

If you need to contact Oracle Support Services, you will be asked to supply the log files mentioned in this section. You may also be required to provide the exact version of each Oracle VM component. To find the version of Oracle VM Manager, click the Help menu, then About. To find the version of Oracle VM Server and Oracle VM Agent, see Section 6.10.15, “Viewing Oracle VM Server Operating System Information and Control Domains”.

B.2.1 Debugging Tools

If virtual machine creation fails, check the Oracle VM Server log files and use the command-line tools to help you find the cause of a problem. There are a number of useful command-line tools, important directories, and log files that you should check when troubleshooting problems with Oracle VM Server. This section discusses:

  • Oracle VM Server directories

  • Oracle VM Server log files

  • Oracle VM Server command-line tools

B.2.1.1 Oracle VM Server Directories

The important Oracle VM Server directories you should check when troubleshooting problems with Oracle VM Server are listed in Table B.3, “Oracle VM Server directories”

Table B.3 Oracle VM Server directories

Directory

Purpose

/etc/xen

Contains Oracle VM Server configuration files for the Oracle VM Server daemon and virtualized guests.

/etc/xen/scripts

Contains networking related scripts.

/var/log

Contains the Oracle VM Agent log file, ovs-agent.log.

Contains the ovmwatch.log, which logs virtual machine life cycle events.

Contains the ovm-consoled.log, which logs remote VNC console access, and all communication with Oracle VM Manager.

/var/log/xen

Contains Oracle VM Server log files.

/var/log/messages

Contains Oracle VM Server messages.


B.2.1.2 Oracle VM Server Log Files

The Oracle VM Server log files you should check when troubleshooting problems with Oracle VM Server are listed in Table B.4, “Oracle VM Server log files”

Table B.4 Oracle VM Server log files

Log File

Purpose

xend.log

Contains a log of all the actions of the Oracle VM Server daemon. Actions are normal or error conditions. This log contains the same information as output using the xm log command. This file is located in the /var/log/xen directory.

xend-debug.log

Contains more detailed logs of the actions of the Oracle VM Server daemon. This file is located in the /var/log/xen directory.

xen-hotplug.log

Contains a log of hotplug events. Hotplug events are logged if a device or network script does not start up or become available. This file is located in the /var/log/xen directory.

qemu-dm.pid.log

Contains a log for each hardware virtualized guest. This log is created by the quemu-dm process. Use the ps command to find the pid (process identifier) and replace this in the file name. This file is located in the /var/log/xen directory.

ovs-agent.log

Contains a log for Oracle VM Agent. This file is located in the /var/log/ directory.

osc.log

Contains a log for Oracle VM Storage Connect Plug-ins. This file is located in the /var/log/ directory.

ovm-consoled.log

Contains a log for the Oracle VM virtual machine console. This file is located in the /var/log/ directory.

ovmwatch.log

Contains a log for the Oracle VM watch daemon. This file is located in the /var/log/ directory.


B.2.1.3 Oracle VM Server Command-Line Tools

The Oracle VM Server command-line tools you should use when troubleshooting problems with Oracle VM Server are listed in Table B.5, “Oracle VM Server command-line tools”.

Table B.5 Oracle VM Server command-line tools

Command-Line Tool

Purpose

xentop

Displays real-time information about Oracle VM Server and domains.

xm dmesg

Displays log information on the hypervisor.

xm log

Displays log information of the Oracle VM Server daemon.


A further set of Oracle VM command line utilities are available for download, separate to the Oracle VM Server in-built utilities. These Oracle VM utilities are a collection of command line scripts that allow you to perform a set of basic management tasks on Oracle VM Servers and virtual machines in an Oracle VM environment. These utilities are particularly useful to administrators who need to execute certain operations quickly and/or repeatedly. Using the command line scripts makes these tasks quicker and easier to perform. See the Oracle VM Utilities Guide for more information on these utilities.

B.2.2 Using DHCP

It is recommended that you install Oracle VM Server on a computer with a static IP address. If your computers uses DHCP you should configure your DHCP server to assign static DHCP addresses. This makes sure your host always receives the same IP address. The behavior of the Oracle VM Server host is undefined if used in an environment where your IP address may change due to DHCP lease expiry.

B.2.3 Setting the Guest's Clock

Paravirtualized guests may perform their own system clock management, for example, using the NTPD (Network Time Protocol daemon), or the hypervisor may perform system clock management for all guests.

You can set paravirtualized guests to manage their own system clocks by setting the xen.independent_wallclock parameter to 1 in the /etc/sysctl.conf file. For example

"xen.independent_wallclock = 1"

If you want to set the hypervisor to manage paravirtualized guest system clocks, set xen.independent_wallclock to 0. Any attempts to set or modify the time in a guest will fail.

You can temporarily override the setting in the /proc file. For example

"echo 1 > /proc/sys/xen/independent_wallclock"
Note

This setting does not apply to hardware virtualized guests.

B.2.4 Wallclock Time Skew Problems

Additional parameters may be needed in the boot loader (grub.conf) configuration file for certain operating system variants after the guest is installed. Specifically, for optimal clock accuracy, Linux guest boot parameters should be specified to ensure that the pit clock source is utilized. Adding clock=pit nohpet nopmtimer for most guests will result in the selection of pit as the clock source for the guest. Published templates for Oracle VM include these additional parameters.

Proper maintenance of virtual time can be tricky. The various parameters provide tuning for virtual time management and supplement, but do not replace, the need for an ntp time service running within guest. Ensure that the ntpd service is running and that the /etc/ntpd.conf configuration file is pointing to valid time servers.

B.2.5 Mouse Pointer Tracking Problems

If your mouse pointer fails to track your cursor in a VNC Viewer session in a hardware virtualized guest, add the following to the Oracle VM Server configuration file located at /etc/xen/xend-config.sxp to force the device model to use absolute (tablet) coordinates:

usbdevice='tablet'

Restart the Oracle VM Server for the changes to take effect. You may need to do this for each Oracle VM Server in the server pool.

B.2.6 Hardware Virtualized Guest Stops

When running hardware virtualized guests, the QEMU process (qemu-dm) may have its memory usage grow substantially, especially under heavy I/O loads. This may cause the hardware virtualized guest to stop as it runs out of memory. If the guest is stopped, increase the memory allocation for dom0, for example from 512MB to 768MB.

B.2.7 Migration of Large Hardware Virtualized Guest Results in CPU Soft Lock

On some hardware, such as the SUN FIRE X4170 M2 Server, migration of very large virtual machines using hardware virtualization can result in a soft lockup causing the virtual machine to become unresponsive. This lock is caused when the migration causes the virtual machine kernel to lose the clock source. Access to the console for the virtual machine shows a series of error messages similar to the following:

BUG: soft lockup - CPU#0 stuck for 315s! [kstop/0:2131]

To resolve this, the virtual machine must be restarted and the clocksource=jiffies option should be added to the HVM guest kernel command line, before rebooting the virtual machine again.

Important

This option should only be used on HVM guest systems that have actually resulted in a CPU soft lock.

B.2.8 Hardware Virtualized Guest Devices Not Working as Expected

Some devices, such as sound cards, may not work as expected in hardware virtualized guests. In a hardware virtualized guest, a device that requires physical memory addresses instead uses virtualized memory addresses, so incorrect memory location values may be set. This is because DMA (Direct Memory Access) is virtualized in hardware virtualized guests.

Hardware virtualized guest operating systems expect to be loaded in memory starting somewhere around address 0 and upwards. This is only possible for the first hardware virtualized guest loaded. Oracle VM Server virtualizes the memory address to be 0 to the size of allocated memory, but the guest operating system is actually loaded at another memory location. The difference is fixed up in the shadow page table, but the operating system is unaware of this.

For example, a sound is loaded into memory in a hardware virtualized guest running Microsoft Windows™ at an address of 100MB may produce garbage through the sound card, instead of the intended audio. This is because the sound is actually loaded at 100MB plus 256MB. The sound card receives the address of 100MB, but it is actually at 256MB.

An IOMMU (Input/Output Memory Management Unit) in the computer's memory management unit would remove this problem as it would take care of mapping virtual addresses to physical addresses, and enable hardware virtualized guests direct access to the hardware.

B.2.9 Firewall Blocks NFS Access

Oracle VM Server blocks NFS access from any external computer (or guest) by default. This may cause problems when trying to create a guest using an NFS connection. To resolve this, disable the firewall with the following command:

# service iptables stop

B.2.10 Migrating Virtual Machines

You cannot migrate virtual machines on computers with hardware that is not identical. To migrate a virtual machines, you must have hardware that is the same make and model and the CPU must be in the same CPU family. You must also have the same Oracle VM Server release number.