Skip Headers
Oracle® VM User's Guide
Release 3.0 for x86

Part Number E18549-03
Go to Documentation Home
Home
Go to Book List
Book List
Go to Table of Contents
Contents
Go to Index
Index
Go to Feedback page
Contact Us

Go to previous page
Previous
Go to next page
Next
PDF · Mobi · ePub

B Troubleshooting

This chapter contains information on using the jobs framework, and troubleshooting Oracle VM, and contains:

For additional information, see the Oracle support-oriented Web sites:

B.1 Working with the Jobs Framework

Oracle VM Manager uses a job operations framework that supports a flexible approach to the configuration of physical and virtual objects. Oracle VM Manager maintains an accurate and consistent view of the virtualization environment while users perform separate and simultaneous jobs. Each configuration change (a transaction performed by a single user) is considered a job.

The following sections describe jobs, and how resources are locked and released at the start and conclusion of each job, and how to manage jobs. This section contains:

B.1.1 Jobs Overview

A job is a configuration change that affects one or more physical or virtual objects. Examples of user operations that can be included in a job are:

  • Adding or deleting a server pool

  • Adding a VNIC to a virtual machine

A single job can contain one or many individual operations. When a job is in progress, a yellow lock appears to the left of the resources included in the job.

B.1.2 Jobs and Resource Locking

A single job can contain one or many individual operations. When a job is in progress, a yellow lock appears to the left of the resources included in the job.

Objects involved in a job are locked to all other Oracle VM Manager users until the job is completed or aborted. Only a user with the same permission level on the object can unlock it. This assures that a consistent and accurate view is maintained for all users.

The state of locked objects cannot be known until the locks are cleared. The state of Oracle VM Manager is always accurately reflected by the state of objects that are not locked.

B.1.3 Locks and Multiple Users

A number of different users may perform jobs simultaneously, provided they are performed on different objects. For example, suppose User A has created a Finance-One server pool and begins a job by moving Oracle VM Servers into another server pool. At the same time, User B modifies the resources of the Commodities server pool. Each user has a separate job pane for jobs, and would see each other's objects as locked. The objects remain locked until the jobs are completed.

Prior to completing a job, a lock can be cleared in two ways:

  • By logging out the user who initiated the lock. This action can be performed by the user, or by an Oracle VM Manager administrator.

  • By direct action of an Oracle VM Manager administrator.

As a job completes, its progress is shown in the Jobs tab. All locks are cleared when a job completes.

B.1.4 Job Failure and Rollback

Job operations are validated by Oracle VM Manager as they are added to the Job tab. The failure of any operation causes the following to happen:

  • The job is cancelled.

  • All operations specified by the job are cancelled.

  • The state of Oracle VM Manager is rolled back to the state it was prior to the start of the job.

  • All locks in the operation are released.

B.1.5 Jobs and Events

When a job operation fails, one or more events may be generated and displayed in Oracle VM Manager. Events are flagged with yellow icons in the navigation tree.

To get information on failed events, click the Events tab in the Jobs management pane.

B.1.6 Job States

A job listed in the Job tab can have any of the states defined in Table B-1.

Table B-1 Job states

Job State Definition

Completed

The job has completed.

Running

The job is in progress.

Aborted

The job has been aborted. Oracle VM Manager has rolled-back to its previous state and all locks have been released.

Failed

The job has failed. Oracle VM Manager has rolled-back to its previous state and all locks have been released.


B.1.7 Starting A Job

A job begins when you make any change in Oracle VM Manager. Each change you make appears in the Job pane as a discrete operation. Job operations can be comparatively minor actions, such as renaming a virtual machine. Operations may also have a wider scope, such as the creation of a new network or storage device. Performing any of these actions changes the configuration of Oracle VM Manager. When a new job is started, information about the job is displayed in the Jobs pane at the bottom of the management pane to show the job's progress.

B.1.8 Aborting A Job

If a job is running or fails to complete, you can abort the job to cancel it. Administrators can abort the jobs of all users.

If you abort a job, all queued operations roll back to the pre-job state. Some job operations, such as renaming an object, complete quickly. Others, such as adjusting the memory used by a virtual machine, take longer.

There are two ways to abort a job:

  • Using the Jobs view

  • Using the Jobs pane in any view, other than the Jobs view.

Both procedures for aborting jobs are listed below.

To abort a job using the Jobs view: 

  1. Select the Jobs view.

  2. Select the Jobs tab.

  3. Select the job in the Jobs table.

  4. Click Abort in the toolbar.

To abort a job using the Jobs pane: 

  1. In any view, other than the Jobs view, select the job in the Jobs pane.

  2. Click Abort in the Abort column.

B.1.9 Determining the Cause of a Job Failure

If a job succeeds, all operations associated with it performed in Oracle VM Manager. A Job Succeeded message appears in the Job Progress area.

If a job fails, the state of Oracle VM Manager returns to its pre-job state. Click Details to see high-level information on all operations in the job.

B.2 Troubleshooting Oracle VM Server

This section describes some problems you may encounter when using Oracle VM Server, and explains how to resolve them. It includes the following topics:

B.2.1 Debugging Tools

If domain creation fails, check the Oracle VM Server log files and use the command-line tools to help you find the cause of a problem. There are a number of useful command-line tools, important directories, and log files that you should check when troubleshooting problems with Oracle VM Server. This section discusses:

  • Oracle VM Server directories

  • Oracle VM Server log files

  • Oracle VM Server command-line tools

B.2.1.1 Oracle VM Server Directories

The important Oracle VM Server directories you should check when troubleshooting problems with Oracle VM Server are listed in Table B-2, "Oracle VM Server directories"

Table B-2 Oracle VM Server directories

Directory Purpose

/etc/xen

Contains Oracle VM Server configuration files for the Oracle VM Server daemon and virtualized guests.

/etc/xen/scripts

Contains networking related scripts.

/var/log

Contains the Oracle VM Agent log file, ovs-agent.log.

Contains the ovmwatch.log, which logs virtual machine life cycle events.

Contains the ovm-consoled.log, which logs remote VNC console access, and all communication with Oracle VM Manager.

/var/log/xen

Contains Oracle VM Server log files.

/var/log/messages

Contains Oracle VM Server messages.


B.2.1.2 Oracle VM Server Log Files

The Oracle VM Server log files you should check when troubleshooting problems with Oracle VM Server are listed in Table B-3, "Oracle VM Server log files"

Table B-3 Oracle VM Server log files

Log File Purpose

xend.log

Contains a log of all the actions of the Oracle VM Server daemon. Actions are normal or error conditions. This log contains the same information as output using the xm log command. This file is located in the /var/log/xen directory.

xend-debug.log

Contains more detailed logs of the actions of the Oracle VM Server daemon. This file is located in the /var/log/xen directory.

xen-hotplug.log

Contains a log of hotplug events. Hotplug events are logged if a device or network script does not start up or become available. This file is located in the /var/log/xen directory.

qemu-dm.pid.log

Contains a log for each hardware virtualized guest. This log is created by the quemu-dm process. Use the ps command to find the pid (process identifier) and replace this in the file name. This file is located in the /var/log/xen directory.


B.2.1.3 Oracle VM Server Command-Line Tools

The Oracle VM Server command-line tools you should use when troubleshooting problems with Oracle VM Server are listed in Table B-4, "Oracle VM Server command-line tools".

Table B-4 Oracle VM Server command-line tools

Command-Line Tool Purpose

xentop

Displays real-time information about Oracle VM Server and domains.

xm dmesg

Displays log information on the hypervisor.

xm log

Displays log information of the Oracle VM Server daemon.


B.2.2 Using DHCP

It is recommended that you install Oracle VM Server on a computer with a static IP address. If your computers uses DHCP you should configure your DHCP server to assign static DHCP addresses. This makes sure your host always receives the same IP address. The behavior of the Oracle VM Server host is undefined if used in an environment where your IP address may change due to DHCP lease expiry.

B.2.3 Guest Console Access

You can connect to a guest's console using Oracle VM Manager. If you do not have access to Oracle VM Manager, you can configure access to a guest's console with VNC (Virtual Network Computing). VNC access to guests requires that VNC access is enabled in the guest's configuration file, vm.cfg. Some VNC parameters (for example, the listening address and password) can be configured in one, either, or both of the following locations:

  • The Oracle VM Server configuration file:

    /etc/xen/xend-config.sxp

  • The guest configuration file in either of the following locations:

    /etc/xen/name

    /OVS/running_pool/name/vm.cfg

Hardware virtualized guests use the vnc=1 parameter in the guest configuration file, for example

vnc=1
vnclisten '0.0.0.0'

Paravirtualized guests use the VNC virtual frame buffer in the guest configuration file, for example

vfb = ['type=vnc,vncunused=1,vnclisten=0.0.0.0,vncpasswd=mypassword']

VNC settings defined in the guest configuration file override the settings in the Oracle VM Server configuration file. For example, if the following is specified in a hardware virtualized guest configuration file:

vnc=1
vnclisten '0.0.0.0'
vncpassword 'mypassword'

The values set in the guest configuration file are used for VNC access, rather than any corresponding values set in the Oracle VM Server configuration.

Note:

Setting vnclisten to 0.0.0.0 sets VNC to allow access to any computer. This may compromise security on the host computer.

If the following is specified in a hardware virtualized guest configuration file:

vnc=1

VNC is enabled in the guest, and the vnclisten parameter is used from the Oracle VM Server configuration file. If vnclisten is not specified in the Oracle VM Server configuration file, a default value of 127.0.0.1 is used. If the following is specified in the hardware virtualized guest configuration file:

vnc=0

VNC access to the guest is disabled.

Setting the default configuration options for VNC access in the Oracle VM Server configuration file enables you to configure access for all guests, and then individually override VNC access by setting the VNC parameters in the guest configuration file.

The following example is a VNC configuration entry in a paravirtualized guest configuration file:

vfb = ['type=vnc,vncunused=1,vnclisten=0.0.0.0,vncpasswd=mypassword']

The following example as a VNC configuration entry in a hardware virtualized guest configuration fie:

vnc = 1                   # vnc=1 enabled, 0=disabled
vncconsole = 1            # vncconsole=1 enables spawning VNC viewer for domain's
                          # console. Default=0
vnclisten = 0.0.0.0       # Address that should be listened on for the VNC server
                          # if VNC is set. Default (if vnc=0) is to use
                          # 'vnc-listen' setting from /etc/xen/xend-config.sxp
vncpasswd = 'mypassword'  # VNC password
vncunused = 1             # vncunused=1 - find an unused port for the VNC server
                          # to listen on. Default=1

In this example, the vncunused=1 parameter allocates a new VNC port number each time a guest is created and assigns it to the guest. Port numbers are allocated starting at the default VNC port number of 5900, so dom1 is allocated port 5900, dom2 is allocated port 5901, dom3 port 5902, and so on.

Connect to the guest on the host computer with the command

# vncviewer -Shared ipaddress:port

The -Shared parameter enables you to share the VNC connection. If you do not include this parameter, another user may destroy your VNC session if they connect at the same time. Connect from a remote computer with a VNC viewer using the connection string:

ipaddress:port

In both examples, ipaddress is the IP address or hostname of the Oracle VM Server, and port is the VNC port number of the guest.

B.2.4 Cannot Display Graphical Installer When Creating Guests

If the graphical installer does not start when creating a guest using the virt-install command-line tool, you should check your X11 configuration. If you are using a console through an ssh (Secure Shell) connection, connect to the console and set the DISPLAY environment variable, for example

# ssh root@example
# export DISPLAY=example:0.0

Alternatively, you can enable connect to a console and enable ssh forwarding using the ssh -X command, for example

# ssh -X root@example

If you use Putty to connect to a console, you must connect from an X11 capable operating system.

B.2.5 Hardware Virtualized Guest Console Not Displayed

If a console is not displayed after you create a hardware virtualized guest, your disk device specification may be incorrect. When you create a hardware virtualized guest, you must specify the VNC console setup. This is not required for a paravirtualized guest.

B.2.6 Setting the Guest's Clock

Paravirtualized guests may perform their own system clock management, for example, using the NTPD (Network Time Protocol daemon), or the hypervisor may perform system clock management for all guests.

You can set paravirtualized guests to manage their own system clocks by setting the xen.independent_wallclock parameter to 1 in the /etc/sysctl.conf file. For example

"xen.independent_wallclock = 1"

If you want to set the hypervisor to manage paravirtualized guest system clocks, set xen.independent_wallclock to 0. Any attempts to set or modify the time in a guest will fail.

You can temporarily override the setting in the /proc file. For example

"echo 1 > /proc/sys/xen/independent_wallclock"

Note:

This setting does not apply to hardware virtualized guests.

B.2.7 Wallclock Time Skew Problems

Oracle VM Release 2.1.1 introduces the use of the timer_mode parameter for hardware virtualized guests. This parameter, when properly applied, can reduce or even eliminate problems with wallclock time skew in most hardware virtualized guests. Wallclock time skew problems do not occur in paravirtualized guests.

Since the application of the correct value of the timer_mode parameter can be difficult to determine, you can pass the os-type and os-variant command-line switches to virt-install to select the best timer_mode value for the guest operating system. When you use these virt-install parameters, the correct timer_mode value is automatically added to the guest configuration file. For example, to create an Oracle Linux 5 64-bit guest, add the following to the virt-install command-line:

# virt-install --hvm ... --os-type=linux --os-variant=el5_64 ...

For best results, additional parameters may be needed in the boot loader (grub.conf) configuration file for certain operating system variants after the guest is installed. Specifically, for optimal clock accuracy, Linux guest boot parameters should be specified to ensure that the pit clock source is utilized. Adding clock=pit nohpet nopmtimer for most guests will result in the selection of pit as the clock source for the guest. Published templates for Oracle VM will include these additional parameters.

Proper maintenance of virtual time can be tricky. The various parameters provide tuning for virtual time management and supplement, but do not replace, the need for an ntp time service running within guest. Ensure that the ntpd service is running and that the /etc/ntpd.conf configuration file is pointing to valid time servers.

B.2.8 Mouse Pointer Tracking Problems

If your mouse pointer fails to track your cursor in a VNC Viewer session in a hardware virtualized guest, add the following to the Oracle VM Server configuration file located at /etc/xen/xend-config.sxp to force the device model to use absolute (tablet) coordinates:

usbdevice='tablet'

Restart the Oracle VM Server for the changes to take effect. You may need to do this for each Oracle VM Server in the server pool.

B.2.9 Hardware Virtualized Guest Stops

When running hardware virtualized guests, the QEMU process (qemu-dm) may have its memory usage grow substantially, especially under heavy I/O loads. This may cause the hardware virtualized guest to stop as it runs out of memory. If the guest is stopped, increase the memory allocation for dom0, for example from 512MB to 768MB.

B.2.10 Hardware Virtualized Guest Devices Not Working as Expected

Some devices, such as sound cards, may not work as expected in hardware virtualized guests. In a hardware virtualized guest, a device that requires physical memory addresses instead uses virtualized memory addresses, so incorrect memory location values may be set. This is because DMA (Direct Memory Access) is virtualized in hardware virtualized guests.

Hardware virtualized guest operating systems expect to be loaded in memory starting somewhere around address 0 and upwards. This is only possible for the first hardware virtualized guest loaded. Oracle VM Server virtualizes the memory address to be 0 to the size of allocated memory, but the guest operating system is actually loaded at another memory location. The difference is fixed up in the shadow page table, but the operating system is unaware of this.

For example, a sound is loaded into memory in a hardware virtualized guest running Windows at an address of 100MB may produce garbage through the sound card, instead of the intended audio. This is because the sound is actually loaded at 100MB plus 256MB. The sound card receives the address of 100MB, but it is actually at 256MB.

An IOMMU (Input/Output Memory Management Unit) in the computer's memory management unit would remove this problem as it would take care of mapping virtual addresses to physical addresses, and enable hardware virtualized guests direct access to the hardware.

B.2.11 CD-ROM Image Not Found

If you create a paravirtualized or hardware virtualized guest using a configuration file, and the CDROM image cannot be found during the installation, you may have the IDE devices in the incorrect order. Putting the IDE devices in order fixes this problem. Check that the disk = [ ... ] parameter is defined as hdc:cdrom and is included before hda, otherwise the usual boot='dc' configuration fails to find the CDROM image.

B.2.12 Firewall Blocks NFS Access

Oracle VM Server blocks NFS access from any external computer (or guest) by default. This may cause problems when trying to create a guest using an NFS connection. To resolve this, disable the firewall with the following command:

# service iptables stop

B.2.13 Migrating Domains

You cannot migrate domains on computers with hardware that is not identical. To migrate a domain, you must have hardware that is the same make and model. You must also have the same Oracle VM Server release.

B.2.14 Attaching to a Console with the Grub Boot Loader

Tracking down startup problems with a hardware virtualized guest may be difficult because you may not be able to attach a console using the xm console command. To workaround this problem, you can include a console in the guest's Grub boot loader, and connect to a console during boot.

To include a console in the Grub boot loader, add the following lines before the first "title ..." line in the /etc/grub.conf file:

serial --unit=0 --speed=9600 --word=8 --parity=no --stop=1
terminal --timeout=10 serial console

B.3 Troubleshooting Oracle VM Manager

This section describes some problems you may encounter when using Oracle VM Manager, and explains how to resolve them. It includes the following topics:

B.3.1 Log Files

Oracle VM Manager messages are displayed in the User Interface, in the Jobs view, or under the object's Events tab.

Table B-5, "Oracle VM Manager Log Files" lists any other log files for Oracle VM Manager.

Table B-5 Oracle VM Manager Log Files

Log File Description

/u01/app/oracle/ovm-manager-3/machine1/base_adf_domain/servers/AdminServer/logs

The Oracle WebLogic application log, which contains Oracle VM Manager messages.


B.3.2 Cannot Start Virtual Machine Console

If you launch the console of a virtual machine in Oracle VM Manager, and an error is displayed, you may not have installed the VNC viewer on the Oracle VM Manager host computer. If you do not have the VNC viewer installed on the Oracle VM Manager host computer, the following error is displayed when you attempt to launch the virtual machine console:

Unable to launch the application.
Name: VncViewer
Publisher: TightVnc
From: URL

To resolve this problem, install a VNC viewer on the Oracle VM Manager host. See the Oracle VM Release Notes for more information.

You can also install a VNC viewer on the client accessing the Oracle VM Manager UI. Oracle recommends you also install a VNC viewer on the Oracle VM Manager host computer so that if a client does not have a VNC viewer, this problem will not occur.

B.3.3 Cannot Create a Virtual Machine from Installation Media

The following message is displayed: "Error: There is no server supporting hardware virtualization in the selected server pool."

To solve this problem, make sure the Virtual Machine Server supports hardware virtualization.

Follow these steps to check:

  1. Run the following command to check if hardware virtualization is supported by the CPU:

    # cat /proc/cpuinfo |grep -E 'vmx|smx'
    

    If any information that contains vmx or smx is displayed, it means that the CPU supports hardware virtualization. Here is an example of the returned message:

    flags : fpu tsc msr pae mce cx8 apic mtrr mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr lahf_lm
    
  2. Make sure you have enabled hardware virtualization in the BIOS.

  3. Run the following command to check if the operating system supports hardware virtualization:

    # xm info |grep hvm
    

    The following is an example of the returned message:

    xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x 
    

If the CPU does not support hardware virtualization, use the paravirtualized method to create the virtual machine. See Section 8.7, "Creating a Virtual Machine".

B.3.4 Cannot Change CD in the Virtual Machine

To change the CD in a virtual machine:

  1. Unmount the first CD:

    # umount mount-point
    
  2. Select the second ISO file, and click Change CD.

  3. Mount the second CD:

    # mount /dev/cdrom mount-point