Part IV Sun xVM Hypervisor

The Sun^TM xVM hypervisor within the OpenSolaris^TM Operating System runs on x64 and x86 machines. The hypervisor is used to execute virtual machine instances called domains. The hypervisor is based on the work of the open source Xen project.

Chapter 37 Sun xVM Hypervisor System

This chapter introduces the Sun^TM xVM hypervisor. The hypervisor is derived from an open source project, Xen.

For more information about using the hypervisor and the xVM architecture, see:

• The OpenSolaris community site, which provides the latest Sun xVM hypervisor documentation, including xVM 3.3 release documentation

• The home of the Xen project

Sun xVM Hypervisor Virtualization System Overview

The Sun xVM hypervisor is a type 1 hypervisor that partitions a single physical machine into multiple virtual machines, to provide server consolidation and utility computing. Existing applications and binaries run unmodified.

The hypervisor presents a virtual machine to guests. The hypervisor forms a layer between the software running in the virtual machine and the hardware. This separation enables the hypervisor to control how guest operating systems running inside a virtual machine use hardware resources.

The hypervisor securely executes multiple virtual machines, or guest domains, simultaneously on a single x64 or x86 compatible computer. Unlike virtualization using zones, each virtual machine runs a full instance of an operating system.

There are two kinds of domains, the control domain and the guest domain. The control domain is also known as domain 0, or dom0. A guest operating system, or unprivileged domain, is also called a domain U or domU.

When working with the xVM software, note that the virsh and virt-install commands are preferred over the use of the legacy xm command whenever possible.

Uniform View of Hardware

A hypervisor provides a uniform view of underlying hardware. Machines from different vendors with different I/O subsystems appear to be the same machine, which means that virtual machines can run on any available supported computer. Thus, administrators can view hardware as a pool of resources that can run arbitrary services on demand. Because the hypervisor also encapsulates a virtual machine's software state, the hypervisor layer can map and remap virtual machines to available hardware resources at any time and also use live migration to move virtual machines across computers. These capabilities can also be used for load balancing among a collection of machines, dealing with hardware failures, and scaling systems. When a computer fails and must go offline or when a new machine comes online, the hypervisor layer can remap virtual machines accordingly. Virtual machines are also easy to replicate, which allows administrators to bring new services online as needed.

The hypervisor virtualizes the system's hardware. A virtualization API and tools are provided by the libvirt and virt-install utilities. The hypervisor transparently shares and partitions the system's CPUs, memory, and NIC resources among the user domains. The hypervisor performs the low-level work required to provide a virtualized platform for operating systems.

The hypervisor assigns one or more virtual CPUs (VCPUs) to each domain, allocated from dom0. The virsh setvcpus and virsh vcpupin commands can be used to dynamically set and pin VCPUs to processors. Each VCPU contains all the state one would typically associate with a physical CPU, such as registers, flags, and timestamps. A VCPU in xVM is an entity that can be scheduled, like a thread in the Solaris^TMsystem. When it is a domain's turn to run on a CPU, xVM loads the physical CPU with the state in the VCPU, and lets it run. The Solaris system treats each VCPU as it would treat a physical CPU. When the hypervisor selects a VCPU to run, it will be running the thread that the Solaris system loaded on the VCPU.

When to Use Domains

Containment

Containment gives administrators a general-purpose undo capability. Administrators can suspend a virtual machine and resume it at any time, or checkpoint a virtual machine and roll it back to a previous execution state. With this capability, systems can more easily recover from crashes or configuration errors. See Recovery.

Containment also supports a very flexible mobility model. Users can copy a suspended virtual machine over a network or store and transport it on removable media. The hypervisor provides total mediation of all interactions between the virtual machine and underlying hardware, thus allowing strong isolation between virtual machines and supporting the multiplexing of many virtual machines on a single hardware platform. The hypervisor can consolidate several physical machines with low rates of utilization as virtual systems on a single computer, thereby lowering hardware costs and space requirements.

Security

Strong isolation is also valuable for reliability and security. Applications that previously ran together on one machine can now be separated on different virtual machines. If one application experiences a fault, the other applications are isolated from this occurrence and will not be affected. Further, if a virtual machine is compromised, the incident is contained to only that compromised virtual machine.

Resource Virtualization to Enable Interoperability

The hypervisor provides a layer between software environments and physical hardware that has the following characteristics:

• Programmable and transparent to the software above it

• Makes efficient use of the hardware below it

Virtualization provides a way to bypass interoperability constraints. Virtualizing a system or component such as a processor, memory, or an I/O device at a given abstraction level maps its interface and visible resources onto the interface and resources of an underlying, possibly different, real system. Consequently, the real system appears as a different virtual system or even as multiple virtual systems.

Hardware Platform for Running Sun xVM Hypervisor

Supported hardware includes x64 and x86 machines.

Determining HVM Support

To run Windows hardware-assisted virtual machine (HVM) domains, an HVM-capable machine that is running a Solaris xVM dom0 is required. A machine is HVM-capable if it is has either an AMD Opteron Revision F or later, or an Intel CPU with VT extensions.

To determine whether a machine is HVM-capable, run the virt-install program with the -v option.

If HVM is not supported, an error message displays:

# virt-install -v
   ERROR    Host does not support virtualization type 'hvm'

Sun xVM Hypervisor Memory Requirements

The current 4–GB default configuration includes memory for the OpenSolaris xVM hypervisor control functions and 10 or fewer guests.

When dom0 is booted, the OpenSolaris xVM hypervisor reserves a small amount of memory for itself and assigns the remainder to dom0. When a domU is created, the Sun xVM hypervisor takes memory from dom0 and assigns it to the new domU.

There is no default configuration or a default amount of memory to assign to dom0, unless the user sets a dom0_mem value in the GRUB menu when booting dom0. The values that can be set are initial memory, minimum memory, and maximum memory.

If you are running known memory hungry apps in dom0, then you need more memory. Examples include cacao processes and Firefox.

The Solaris System and x86 Platforms

With the introduction of the Sun xVM hypervisor, there are now two platform implementations on the x86 architecture. The implementation names are returned by the uname command with the -i option and refer to the same machine, i86pc.

• i86pc refers to the Solaris system directly installed on the physical system, also known as running on bare metal.

• i86xpv refers to the Solaris system running both paravirtualized and fully virtualized on top of an xVM hypervisor.

Applications should use the uname command with the -p option. On x86 platforms, regardless of whether the Solaris system is running under xVM, this option always returns i386. The -p option is equivalent to using the -m option in some other operating systems.

Guests That Are Known to Work

Supported virtual machine configurations in the OpenSolaris 2009.06 release include the OpenSolaris control domain (domain 0), and guest, or domU, operating systems.

Because the control domain must work closely with the hypervisor layer, the control domain is always paravirtualized. A system can have both paravirtualized and fully virtualized domains running simultaneously.

Types of guests include the following:

• Fully virtualized guests, referred to as hardware-assisted virtual machines (HVMs). These are virtual machines that utilize Intel-VT and AMD-V extensions. An HVM guest domain runs an unmodified operating system.

• HVM + PVIO: These guests provide better performance through the use of PVIO drivers.

• Paravirtualized (PV): A PV guest OS is a modified OS that is aware that is operating under a hypervisor, which results in higher performance.

Note that Windows HVM domains can be susceptible to viruses, so make sure you comply with your site's network security policies.

The following information applies to the control domain:

• ISA floppy is not supported.

• For 32–bit, the processor must support physical address extensions (PAE) mode.

• Beginning with build snv_83, the OpenSolaris OS supports all Ethernet-type interfaces, and their data-links can be administered with the dladm command. Prior to OpenSolaris build snv_83, the network interface card (NIC) must support GLDv3. These devices include bge, e1000g, xge, nge, and rge.

The Sun xVM Hypervisor and Domain 0

The hypervisor is responsible for controlling and executing each of the domains and runs with full privileges. The control tools for managing the domains run under the specialized control domain domain 0 (dom0).

The hypervisor virtualizes the system's hardware. The hypervisor transparently shares and partitions the system's CPUs, memory, and NIC resources among the user domains. The hypervisor performs the low-level work required to provide a virtualized platform for operating systems.

The hypervisor relies primarily on the control domain for the following:

• Which guest domains are created.

• Which resources the guest domains can access.

• How much memory a given guest domain can have.

• Which devices a guest domain can access, because xVM does not include any device drivers. The hypervisor delegates control of the machine's physical devices to the control domain, domain 0.

Thus, by default, only the control domain has access to physical devices. The guest domains running on the host are presented with virtualized devices. The domain interacts with the virtualized devices in the same way that the domain would interact with the physical devices. Also see Resource Virtualization to Enable Interoperability.

The following figure shows the Sun xVM hypervisor configuration.

Figure 37–1 Sun xVM Configuration

Sun xVM Hypervisor Scheduler

The hypervisor schedules running domains (including domain 0) onto the set of physical CPUs as configured. The scheduler is constrained by configuration specifications such as the following.

• Domain configuration, such as the number of virtual CPUs allocated.

• Runtime configuration, such as pinning virtual CPUs to physical CPUs. Pinning CPUs ensures that certain VCPUs can only run on certain physical CPUs.

The default domain scheduler for the hypervisor is the credit scheduler. This is a fair-share domain scheduler that balances virtual CPUs of domains across the allowed set of physical CPUs according to workload. No CPU will be idle if a domain has work to do and wants to run.

The scheduler is configured through the xm sched-credit command described in the xm(1M) man page.

The following parameters are used to configure the scheduler:

-d domain, --domain=domain

Domain for which to set scheduling parameters.

-c cap, --cap=cap

The maximum amount of CPU the domain can consume. A value of zero, which is the default, means no maximum is set. The value is expressed in percentage points of a physical CPU. A value of 100 percent corresponds to one full CPU. Thus, a value of 50 specifies a cap of half a physical CPU.

-w weight, --weight=weight

The relative weight, or importance, of the domain. A domain with twice the weight of the other domains receives double the CPU time of those other domains when CPU use is in contention. Valid weights are in the range 1-65536. The default weight is 256.

Example 37–1 xm sched-credit Configuration

The following line configures scheduling parameters for the domain sol1. The domain has a weight of 500 and a cap of 1 CPU.

xm sched-credit -d sol1 -w 500 -c100

If used without the -w and -c options, the current settings for the given domain are shown.

Supported Virtualization Modes

There are two basic types of virtualization, full virtualization and paravirtualization. The hypervisor supports both modes.

Full virtualization allows any x86 operating system, including Solaris, Linux, or Windows systems, to run in a guest domain.

Paravirtualization requires changes to the operating system, with the minimum requirement being modifications to support the virtual device interfaces.

A system can have both paravirtualized and fully virtualized domains running simultaneously.

For paravirtualized mode and for all types of operating systems, the only requirement is that the operating system be modified to support the virtual device interfaces.

Overview of Paravirtualization

In the more lightweight paravirtualization, the operating system is both aware of the virtualization layer and modified to support it, which results in higher performance.

The paravirtualized guest domain operating system is ported to run on top of the hypervisor, and uses virtual network, disk, and console devices.

Since the control domain must work closely with the hypervisor layer, control domain is always paravirtualized. Guest domains can be either paravirtualized or fully virtualized.

Devices and Drivers

Because full paravirtualization requires changes to the OS, only specific operating systems can be hosted in a fully paravirtualized guest domain. Currently, those are limited to Solaris, Linux FreeBSD, and NetBSD, although others might be made available in the future.

Partial paravirtualization describes a mechanism in which an otherwise unmodified OS is augmented with paravirtualized drivers for I/O devices. This can significantly improve the performance of the otherwise unmodified guest domain.

Paravirtualized drivers for I/O devices are implemented as a pair of drivers, one in each of the guest and host domains. This mechanism is often termed split drivers.

A frontend driver runs in the guest domain and communicates with a backend driver running in domain 0. This enables the guest domain to access services available to domain 0.

xVM software in the OpenSolaris release currently supports two main split drivers, one for network I/O and one for disk I/O.

Within the guest domain, the frontend driver appears as a normal device. For network, this is an Ethernet device. For disk, this is a traditional block device.

Within domain 0, the behavior of the backend driver depends on the type of device and the configuration of the domain. Network backend drivers typically extend the physical network connectivity available to domain 0 into the guest domain by using a virtual NIC feature. Disk backend drivers can make disk space available to guest domains by using files, ZFS volumes, and physical devices. Various file formats are supported when files are used to provide storage, for example, VMDK. (For more information on VMDK, see the section Using vdiskadm to Create Virtual Disks .)

The Solaris frontend drivers share the same source code whether they are used in paravirtualized (PV) or partially paravirtualized (HVM+PV) domains. There are #ifdefs in the driver source code to accommodate differences between HVM+PV and PV environments, but otherwise they are the same.

The Windows frontend drivers have different source code than those for Solaris, but the protocol between them and the Solaris backend drivers is the same as that used by the Solaris frontend drivers. This protocol was developed by the Xen open source community and is defined by the source code for the Linux variants of the drivers.

The source code for the Solaris drivers is found in /usr/src/uts/common/xen/io. The network frontend driver is xnf, and the disk frontend driver is xdf. The backend drivers have various names, such as xnb, xnbo, xnbu, xdb, and xpvtap.

In addition to these drivers, the Solaris console is virtualized when the Solaris system is running as a guest domain. The console driver interacts with the xenconsoled(1M) daemon running in domain 0 to provide console access.

Overview of Full Virtualization

In a full virtualization, the operating system is not aware that it is running in a virtualized environment under xVM. A fully virtualized guest domain is referred to as a hardware-assisted virtual machine (HVM). An HVM guest domain runs an unmodified operating system.

Fully-virtualized guest domains are supported under xVM with virtualization extensions available on Intel-VT or AMD Secure Virtual Machine (SVM) processors. These extensions must be present and enabled. Some BIOS versions disable the extensions by default. Note that this that hardware is also needed in HVM+PVIO configurations such as Solaris 10 5/09 (Solaris 10 U7) guest domains.

Full virtualization requires that the hypervisor transparently intercept many operations that an operating system typically performs directly on the hardware. This interception allows the hypervisor to ensure that a domain cannot read or modify another domain's memory, cannot interfere with its device access, and cannot shut down the CPUs it is using.

Virtual Devices

Networks

The physical network consists of both an external physical LAN and the extension of the LAN within the host to a guest's network. Paravirtualized domains use the xnb backend driver in dom0 to communicate with a physical network interface.

The virtual network operates through the underlying physical network infrastructure.

You can create one physical network, also referred to as a switch, for each NIC on the system. If you have more than one physical NIC configured, you might want to configure the default-nic property of the xvm/xend SMF service, as described in the xend(1M).

To view the IP address assigned through DHCP, use the ifconfig command.

See the document New Network Options, Including Limiting Bandwidth and Setting a VLAN ID, for Virtual Network Interfaces Attached to a Guest to learn about conventions for network options used in the virt-install utility.

Virtual NICs

A single physical NIC can be carved into multiple VNICs, which can be assigned to different Solaris xVM instances running on the same system. VNICs are managed using the dladm command line utility described in the dladm(1M) man page. You can use virsh dumpxml output to correlate the domain's network interface with the assigned VNIC.

The dladm show-vnic command can be used to display VNIC information. In the following output, 1 is the domain ID, and 0 is the NIC number for that guest.

# dladm show-vnic
LINK         OVER         SPEED  MACADDRESS           MACADDRTYPE VID
xvm1_0       e1000g0      1000   0:16:3e:64:99:4d     fixed 0

For more information, see OpenSolaris Project: Crossbow: Network Virtualization and Resource Control. Crossbow provides network virtualization and resource control by virtualizing the stack and NIC around any service, protocol, or virtual machine.

Virtual FibreChannel HBAs

Certain FibreChannel HBAs that support N Port ID virtualization (NPIV) can be carved into multiple virtual HBAs, in a manner similar to VNICs. Each virtual HBA has a unique identity, or WWN, on the SAN, which can be used for LUN masking or zoning. Block devices present on these virtual HBAs can be automatically mounted to specific OpenSolaris xVM guest operating systems. When these guest operating systems migrate, the HBA's identity also migrates. NPIV is administered using the fcinfo(1M) and xm(1M) commands.

Virtual FibreChannel HBAs

Virtual block devices can be stored on FibreChannel disk subject to the following limitations:

• Back end block device names (for example, c0t0d0s2) must be the same for migration to succeed. If device names are different, migration will fail.

• For FibreChannel devices, it is likely that the cX number will be different if scsi_vhci, described in scsi_vhci(7D), is not used. If scsi_vhci is used, device names are likely to be identical. If scsi_vhci is enabled and has never been disabled, the controller should be at cX on x86/x64 machines.

NPIV and FibreChannel

NPIV is fully supported for xVM. The xm command can associate all block devices from one NPIV port to a guest domain. NPIV identity, specifically the port WWN and node WWN, will migrate between devices on the SAN. NPIV allows zoning and LUN masking to be used with Xen. Zoning and LUN masking are useful tools for access control on a SAN. Soft zoning rather than hard zoning (grouping by switch port) should be used on the switch. Soft zoning groups devices by the HBA's Port WWN. If there is more than one physical SAN, or if the system uses hard zoning, the administrator must ensure that the physical HBA is connected to the correct SAN. Switch administrative tools can be used for this purpose.

How to Configure NPIV for Hypervisor Use

This procedure uses the fcinfo command described in fcinfo(1M).

1. Identify the FibreChannel HBAs that will be used. If migration will be supported, the HBAs must be identified. The fcinfo command can be used to list the Port WWN.

2. Create NPIV ports in the Dom0 control domain.

3. View devices by usingfcinfo in dom0. Verify that they are visible in the respective guest domains.

Using vdiskadm to Create Virtual Disks

You can use the virt-install command to create disks.

The vdiskadm command described in the vdiskadm(1M) man page creates and manages virtual disks. vdiskadm is implemented as a set of subcommands, some of which have their own options and operands. All operations on the vdisk need to be performed using vdiskadm.

The types of virtual disks are:

• -t vmdk:fixed

• -t vmdk:sparse

• -t raw:fixed

• -t vdi:fixed

• -t vdi:sparse

• -t vhd:fixed

• -t vhd:sparse

vmdk is the native VMware format, vdi is the native Sun VirtualBox format, vhd is the native Hyper-V format, and raw describes a file that looks like a raw disk. A raw disk is always in fixed format so that option can be explicitly set or implicitly understood. If the type is not specified, the default value is vmdk. If the option is not specified, the default value is fixed for type raw and sparse for types vmdk and vdi.

Create a new virtual disk of the specified size and at the location specified by vdname. If vdname includes a path to the virtual disk, the directories from that path will be created during creation of the virtual disk. -t type[:opt],[opt]] specifies the type of virtual disk to be created. Type as one line.

# vdiskadm create -s size [-t type[:opt],[opt]] [-c comment] vdname

You can import a disk image from a block device or file to a vdisk, convert it to a different type of vdisk, and export from a vdisk to a block device or file. This includes the full vhd support (sparse and fixed) and the ability to import a vmdk 1.1 optimized stream file. An optimized stream file is read-only and must be imported to another type (vmdk:sparse by default) in order to be used as a vdisk.

Examples

Creating a Default vmdk:sparse File

A directory of the vdisk name is created and populated with two files. vdisk.vmdk is the file with the disk data. vdisk.xml is the file containing information about the disk, such as creation time, type:option of disk, and snapshot information. Note that vdisk.vmdk has a suffix of the vdisk type.

# vdiskadm create -s 8g root_disk
# ls -l root_disk
total 82
-rw-------   1 root     root     1114112 May  8 16:15 vdisk.vmdk
-rw-r--r--   1 root     root         584 May  8 16:15 vdisk.xml

Creating a vdisk File of Type vhd

The suffix specified is now vhd. Since the option isn't specified with a type, the option has the default of sparse. Note that the disk file, vdisk_vhd, isn't fully populated to 8G.

# vdiskadm create -s 8g -t vhd root_disk_vhd
# ls -l root_disk_vhd
total 44
-rw-------   1 root     root       21504 May  8 16:15 vdisk.vhd
-rw-r--r--   1 root     root         590 May  8 16:15 vdisk.xml

Creating a vmdk:fixed File

Creating a vmdk type vdisk with option fixed takes a minute or more to create since it is creating and initializing 8G of data. The creation time is dependent upon the size of the vdisk.

# vdiskadm create -s 8g -t vmdk:fixed root_disk_fix
# ls -l root_disk_fix
total 16785428
-rw-------   1 root     root     8589934592 May  8 16:18 vdisk-flat.vmdk
-rw-------   1 root     root         638 May  8 16:18 vdisk.vmdk
-rw-r--r--   1 root     root         593 May  8 16:18 vdisk.xml

The contents of the xml file for root_disk_fix are:

# cat root_disk_fix/vdisk.xml
<?xml version="1.0"?>
<!DOCTYPE vdisk PUBLIC "-//Sun Microsystems Inc//DTD xVM Management All//EN" "file:///usr/share/lib/xml/dtd/vdisk.dtd">
<vdisk readonly="false" removable="false" cdrom="false" creation-time-epoch="1241821124" vtype="vmdk" sparse="false" rwcnt="0" rocnt="0">
  <name>root_disk_fix</name>
  <version>1.0</version>
  <parent>none</parent>
  <diskprop>
    <filename>root_disk_fix</filename>
    <vdfile>vdisk.vmdk</vdfile>
    <owner>root</owner>
    <max-size>8589934592</max-size>
    <sectors>16777216</sectors>
    <description>none</description>
  </diskprop>
</vdisk>

This same information can be retrieved from the vdiskadm command by using the subcommand prop-get:

# vdiskadm prop-get -p all root_disk_fix
readonly: false
removable: false
cdrom: false
creation-time-epoch: 1241821124
vtype: vmdk
sparse: false
rwcnt: 0
rocnt: 0
name: root_disk_fix
version: 1.0
parent: none
filename: root_disk_fix
vdfile: vdisk.vmdk
owner: root
max-size: 8589934592
sectors: 16777216
description: none
effective-size: 8589935230
creation-time: Fri May  8 16:18:44 2009
modification-time: Fri May  8 16:18:44 2009
modification-time-epoch: 1241821124

The modification times and effective size are all derived “on the fly,” and are not stored in the xml file. The creation and modification times are shown both in epoch format and in human readable format, for use by both software applications (such as Sun Ops Center) and system administrators.

The rwcnt and rocnt fields shown in the xml file are the reader/writer locks on the vdisk. There can be only one writer at a time, but multiple readers can be using the vdisk. These fields are used to set or reset the reader/writer lock associated with the virtual disk. These fields should not be set or reset by hand; they can only be modified by using vdiskadm ref-inc [-r] vdame or vdiskadm ref-dec vdname. These fields are used by blktap for shared or exclusive use of the virtual disk.

Snapshots

A snapshot is a read-only copy of a virtual disk. Snapshots can be created quickly and initially consume little space. As data within the active virtual disk changes, the snapshot consumes more data than would otherwise be shared with the active virtual disk.

vdisk supports snapshots in a manner similar to ZFS, except that the vdisk cannot be in use during a snapshot. The user can take a snapshot of the vdisk and later rollback to that snapshot, if needed. The user can also take a snapshot and then clone that snapshot into another vdisk.

To see the images are associated with a vdisk, type:

# vdiskadm list vhd_sp
vhd_sp

Take a snapshot of the virtual disk immediately after installing it:

# vdiskadm snapshot /export/home/vdisks/vhd_sp@install

List all images associated with the virtual disk:

# vdiskadm list /export/home/vdisks/vhd_sp
vhd_sp@install
vhd_sp

The original file, vdisk.vhd, has been moved to vdisk@install.vhd. A new file that contains the differences has been created. It is named vdisk.vhd.

# ls -l vhd_sp
total 2717732
-rw-------   1 root     root       17408 May 11 16:41 vdisk.vhd
-rw-r--r--   1 xvm      root         717 May 11 16:41 vdisk.xml
-rw-------   1 root     root     1390768640 May 11 16:41 vdisk@install.vhd 

The vdisk.xml file shows the added snapshot element. When additional snapshots are created, new snapshot elements will be added to the xml description. The snapshot order in the list (and shown with vdiskadm list) shows the order in which the snapshots are loaded.

# cat vhd_sp/vdisk.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE vdisk PUBLIC "-//Sun Microsystems Inc//DTD xVM Management All//EN" "file:///usr/share/lib/xml/dtd/vdisk.dtd">
<vdisk readonly="false" removable="false" cdrom="false" creation-time-epoch="1241643718" vtype="vhd" sparse="true" rwcnt="0" rocnt="0">
  <name>vhd_sp</name>
  <version>1.0</version>
  <parent>none</parent>
  <diskprop>
    <filename>vhd_sp</filename>
    <vdfile>vdisk.vhd</vdfile>
    <owner>xvm</owner>
    <max-size>6442450944</max-size>
    <sectors>12582912</sectors>
    <description>none</description>
  </diskprop>
  <snapshot creation-time-epoch="1242081709">
    <name>@install</name>
    <vdfile>vdisk@install.vhd</vdfile>
  </snapshot>
</vdisk>

Now, take another snapshot after a bfu and list the contents:

# vdiskadm snapshot /export/home/vdisks/vhd_sp@bfu

# vdiskadm list /export/home/vdisks/vhd_sp
vhd_sp@install
vhd_sp@bfu
vhd_sp

To roll back the disk to a point right after the install:

# vdiskadm rollback -r /export/home/vdisks/vhd_sp@install
# vdiskadm list /export/home/vdisks/vhd_sp
vhd_sp@install
vhd_sp

The rollback operation removes vdisk.vhd and any intervening snapshot images after vdisk@install.vhd, and creates a new differences file named vdisk.vhd.

# ls -l vhd_sp
total 2717732
-rw-------   1 root     root       17408 May 11 16:47 vdisk.vhd
-rw-r--r--   1 xvm      root         717 May 11 16:47 vdisk.xml
-rw-------   1 root     root     1390768640 May 11 16:47 vdisk@install.vhd

Clones

A clone is a writable copy of a virtual disk. The default type of clone is a merged (that is, coalesced) copy of the original virtual disk. An example of a merged clone occurs when a virtual disk is comprised of several snapshots; a subsequent clone operation results in a new virtual disk containing no snapshots. A clone will be of the same type as the original virtual disk (that is, vmdk:fixed). When a merged clone is created there is no linkage back to the original virtual disk or to any of its snapshots. This lack of linkage allows the merged clone to be moved to another physical machine.

Create a clone of the specified snapshot or virtual disk. The clone is created with the type and option and the size of the virtual disk being cloned. If clone_vdname includes a path, the subdirectories from that path will be created during creation of the cloned virtual disk. By default, a merged clone image is created:

# vdiskadm clone [-c comment] vdname|snapshot clone_vdname

About Domains

The control domain and the guest domain are separate entities.

Each domain has a name and a UUID. Domains can be renamed, but typically retain the same UUID.

A domain ID is an integer that is specific to a running instance. This ID changes whenever a guest domain is booted. A domain must be running to have a domain ID. UUID 0 is assigned to dom0.

Control Domain 0

For the latest information on Domain 0, see the document dom0 configuration for admins .

The control domain is a version of Solaris modified to run under the xVM hypervisor. When the control domain is running, the control tools are enabled. In most other respects, the control domain 0 instance runs and behaves like an unmodified instance of the Solaris Operating System.

The control domain provides console access to the guest domains it controls, but you cannot otherwise access a guest domain from the control domain unless you use the remote login commands rlogin, telnet, and ssh. A control domain should be reserved for system management work associated with running a hypervisor. This means, for example, that users should not have logins on the control domain. The control domain provides shared access to a physical network interface to the guest domains, which have no direct access to physical devices.

If a control domain crashes with a standard Solaris panic, the dump will include just the control domain. Also see About Crash Dumps.

The following information applies to the control domain:

• ISA floppy is not supported.

• For 32–bit, the processor must support physical address extensions (PAE) mode.

• The OpenSolaris OS supports all Ethernet-type interfaces, and their data-links can be administered with the dladm command.

Guest Domain Space Requirements

Size your domain as you would configure a machine to do the same workload.

The virtual disk requirement is dependent on the guest operating system and software that you install.

Domain States

A domain can be in one of six states. States are shown in virsh list displays.

For example:

#  virsh list
ID    Name        State  
-------------------------
0     Domain-0    running
2     sxc18       paused

The states are:

r, running

The domain is currently running on a CPU.

b, blocked

The domain is blocked, and not running or able to be run. This can be caused because the domain is waiting on IO (a traditional wait state) or it has gone to sleep because there was nothing running in it.

p, paused

The domain has been paused, usually through the administrator running virsh suspend. When in a paused state, the domain still consumes allocated resources like memory, but is not eligible for scheduling by the hypervisor. Run resume domain to place the domain in the running state.

s, in shutdown

The domain is in process of shutting down, but has not completely shutdown or crashed.

s, shutoff

The domain is shut down.

c, crashed

The domain has crashed. Usually this state can only occur if the domain has been configured not to restart on crash. See xmdomain.cfg(5) for more information.

Chapter 38 Booting and Running the Sun xVM Hypervisor

This chapter discusses booting and running the hypervisor.

x86 Boot Architecture

The open source GNU GRand Unified Bootloader (GRUB) is implemented on x86 based systems that are running the Solaris Operating System. GRUB is the boot loader that is responsible for loading a boot archive into a system's memory. The boot archive contains the kernel modules and configuration files that are required to boot the system.

The GRUB main menu is based on the /boot/grub/menu.lst configuration file. For more information on GRUB, see Chapter 11, Modifying Solaris Boot Behavior (Tasks), in System Administration Guide: Basic Administration and grub(5).

Whether to run Solaris as a virtualized control domain or as a standalone operating system is a boot-time decision. To run the Solaris operating system as a standalone system, continue to use the same GRUB menu entries that you use currently.

In the Solaris xVM entry for booting Solaris as a control domain with the hypervisor, the kernel$ line refers to the hypervisor, and there are two module$ lines. The first module$ line must list the path to unix twice, with any arguments. The second module$ line lists the path to the boot archive.

In some situations, the GRUB menu.lst file might not reside in /boot/grub. To determine the location of the active GRUB menu.lst file, use the bootadm command with the list-menu subcommand.

For more information on the GNU GRand Unified Bootloader (GRUB) boot menu and its components, see System Administration Guide: Basic Administration.

Domain 0 Procedures

How to Set up OpenSolaris 2009.06 as a Sun xVM dom0

Use this procedure to set up the OpenSolaris 2008.11 or 2009.06 release as an xVM dom0.

1. Create a separate boot environment, if needed.

   # pfexec beadm create -a -d xvm xvm

2. If you're running a recent version of OpenSolaris, you can install the xvm-gui cluster instead of the individual packages.

   Otherwise, install the required packages.

   # pfexec beadm mount xvm /tmp/xvm-be # pfexec pkg -R /tmp/xvm-be install SUNWvirt-manager SUNWxvm SUNWvdisk SUNWvncviewer # pfexec beadm umount xvm

3. Run the following AWK script over the menu.1st file that controls the GRUB menu. This step is necessary for beadm to reboot into xVM. The menu.1st file is typically located in the /pool-name/boot/grub directory on systems with a ZFS root. The correct menu.1st file is shown in Step 4 of this procedure.

   $ awk ' /^title/ { xvm=0; } /^title.xvm$/ { xvm=1; } /^(splashimage|foreground|background)/ { if (xvm == 1) next } /^kernel\$/ { if (xvm == 1) { print("kernel\$ /boot/\$ISADIR/xen.gz") sub("^kernel\\$", "module$") gsub("console=graphics", "console=text") gsub("i86pc", "i86xpv") $2=$2 " " $2 } } { print }' /rpool/boot/grub/menu.lst >/var/tmp/menu.lst.xvm

4. Verify that the AWK script worked properly.

   $ tail /var/tmp/menu.lst.xvm #============ End of LIBBE entry ============= title xvm findroot (pool_rpool,0,a) bootfs rpool/ROOT/xvm kernel$ /boot/$ISADIR/xen.gz module$ /platform/i86xpv/kernel/$ISADIR/unix /platform/i86xpv/kernel/$ISADIR/unix -B $ZFS-BOOTFS,console=text module$ /platform/i86pc/$ISADIR/boot_archive #============ End of LIBBE entry =============

5. Create a new GRUB entry in /rpool/boot/grub/menu.1st.xvm and then reboot.

6. Enable the xVM services.

   # svcadm enable -r xvm/virtd ; svcadm enable -r xvm/domains

You should now be able to install guest domains.

How to View Domains on the System

Run the virsh list, xm list or the xm top commands to view the domains on the system. These commands provide details of running domains. The display from any of these commands should show the single control domain, called Domain-0.

1. Become superuser, or assume the appropriate role.

2. Run virsh list.

   # virsh list Id Name State ---------------------------------- 0 Domain-0 running

See Also

See the xentop(1M) man page.

Creating Guest Domain Environments by Using File Systems

Prior to installing guest domains (domain Us), you must decide how to create the guest domain environment.

• Use the Solaris ZFS file system and volume datasets to enable the rapid provisioning of guest domains.

• Use normal files to store guest domain disk images. Live migrations are done using Network File System (NFS).

How to Configure ZFS for Use With Solaris xVM

First, create a new storage pool using the zpool command described in zpool(1M). The name of the disk can be specified as a full device path, such as /dev/dsk/c0t0d0s0 , or as a disk name, such as c1t0d0. Multiple disks can be specified by using disk names separated by a space, to use disk striping.

Then, create a Solaris ZFS volume to store the guest domain master image. Note that the Solaris ZFS volume is a dataset, however, it represents a block device and can be used like traditional UNIX block devices.

1. Become superuser, or assume the appropriate role.

2. Create a new storage pool named xpool that includes the disk c1t0d0.

   # zpool create xpool c1t0d0

   The -f option can be used to force the action.

3. Verify that the storage pool is created:

   # zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT xpool 34G 156K 34.0G 0% ONLINE -L

   You can use the zpool iostat command to view information about the I/O throughput of the newly created storage pool.

4. Create an 8–Gbyte Solaris ZFS volume to store the guest domain master image.

   # zfs create -V 8gb xpool/domU-master

5. List the Solaris ZFS volume block device nodes, which are identified as devices in the /dev/zvol/dsk and /dev/zvol/rdsk directories.

   # ls -l /dev/zvol/dsk/xpool total 2 lrwxrwxrwx 1 root 35 Apr 19 10:24 domu.master ->../../ll/ll/devices/pseudo/zfs@0:1c

How to Enable Migration to This Machine

This procedure is used to allow migration to this machine from all other control domains.

1. Become superuser, or assume the appropriate role.

2. Edit the xend-relocation-address, xend-relocation-hosts-allow, and xend-relocation-server properties of xvm/xend.

See Also

See dom0 configuration for admins

Chapter 39 Creating and Managing Guest Domains

You can create, monitor, manage, and configure guests within a given OpenSolaris^TM xVM hypervisor instance.

About Guest Installation

For the latest information on using the virt-install command-line utility to install guest domains, see Using virt-install to Install a Guest. This document provides xVM 3.3 examples.

To create a guest by using the virt-install utility, you must specify an installation source, disk storage, networking, and other parameters. After the guest OS is installed, it can be managed through the virsh utility, as described in Managing Guests.

Sizing Guests

The number of guests that can be created is determined primarily by the amount of memory and the disk space available.

Size your domain as you would configure a machine to do the same workload. The virtual disk requirement is dependent on the guest operating system and software that you install.

Installation Types

Types of virt-install installations that can be performed include the following:

• Interactive (not available in xVM 3.3)

• Command line, with options supplied

• Netinstall

• ISO image

• Solaris JumpStart

About Network Installations

After you configure the installation server, you can run the virt-install command described in the virt-install(1M) from dom0. Use the -d option to add_install_client to specify that the client use DHCP. If the -d option is not specified, the client uses bootparams. For xVM paravirtualized guests, both approaches work. Use your site-specific tool for setting up the appropriate DHCP parameters for the client.

Network Installation

To do a network installation, use the -l option and provide a path to the network installation image. When giving a machine name or IP address, the domU must be able to get to that install machine directly, not go through a router to another net. For example:

-l nfs:install:/export/xvm/xvmgate-70i72-nd

You can also use an IP address instead of a machine name. For example:

-l nfs:172.20.25.12:/export/xvm/xvmgate-70i72-nd

Example 39–1 Network Installation

virt-install -n gath-01 -r 1000 --nographics -f /dev/dsk/c1t0d0s3 \
-m "aa:04:03:35:a8:06" -p \
-l nfs:install48:/export/xvm/xvmgate-70i72-nd

ISO Image Installation

To use the ISO image, use the -l option with a full path to the ISO image. If a full path is given instead of the nfs:mach_name:path format of a network installation, then virt-install assumes that this is an ISO image:

-l /net/install/export/xvm/solarisdvd.iso

Example 39–2 ISO Image

virt-install -n gath-01 -r 1000 --nographics -f /dev/dsk/c1t0d0s3 \
-m aa:04:03:35:a8:06 -p \
-l /net/install48/export/xvm/solarisdvd.iso

You can quote arguments to options. While arguments, such as the path to an ISO image, are generally not quoted on the command line, quotes might be used in scripts.

-l "/net/install48/export/xvm/solarisdvd.iso"

JumpStart Installation

JumpStart configuration files are manually created and managed. You can initiate a custom JumpStart through network installation after setting up the server. When you create a profile server, you must ensure that systems can access the JumpStart directory on the profile server during a custom JumpStart installation. Each time that you add a system for network installation, use the add_install_client command to specify the profile server. You use the add_install_client command to create the /etc/bootparams entry for the domU.

To do a JumpStart with virt-install, use the --autocf option. For example:

--autocf nfs:install:/export/jumpstart/jump-user/x86

You cannot use a full path such as:

--autocf /net/install/export/jumpstart/jump-user/x86

Example 39–3 JumpStart

virt-install -n gath-01 -r 1000 --nographics -f /dev/dsk/c1t0d0s3 \
-m aa:04:03:35:a8:06 -p \
-l /net/install48/export/xvm/xvmgate-70i72/solarisdvd.iso
--autocf nfs:install:/export/jumpstart/jump-user/x86

Required Information

You will need to supply the guest domain information listed below.

name

Name for the guest domain. Each guest domain must have a unique name. This name serves as the label of the guest operating system. The name must be a real hostname for network installations to work.

image location

Location of the installation software. Installation must be over a network (which includes an NFS share from the local host operating system) or be an ISO install.

For example:

--location nfs:my.nfs.server.com:/home/install/test/mydomain

For HVM, an ISO or CDROM device should be given instead of an image location.

Installations using http or ftp, as shown in the following examples, are supported for Linux paravirtualized domain installations only:

http://my.http.server.com:/install/test/mydomain ftp://my.ftp.server.com:/install/test/mydomain

vcpus

The number of CPUs for the guest domain. The default is 1. You can assign specific CPUs. If undefined, the hypervisor makes the selection.

memory

Amount of RAM to be allocated to the guest, in megabytes. A running domain should use a minimum of 512 megabytes. However, to install the guest domain, 1 Gbyte (1024 megabytes) is required.

graphics

Graphical console. Default is graphics. The nographics option applies to paravirtual guests only. If you intend to enable graphics support, you must decide whether the graphical installer should be used.

(Optional) Virtual network interface MAC address

This is the MAC address of the dom0's network interface that you want the domU to use to send and receive internet traffic. By default, the hypervisor tools uses the first available network interface card (NIC) when creating guest domains.

The default values for the action to be taken on a domU shutdown, reboot, or crash are set by virt-install. You currently cannot change these defaults.

The complete list of supported virt-install options are listed in the virt-install(1M) man page.

Creating Guest Domains

Using virt-install to Install a Guest Domain

The virt-install program can be run as a command-line utility, with parameters specified through options, or interactively, in response to a series of prompts.

The default values for the action to be taken on a domU shutdown, reboot, or crash are set by virt-install. You currently cannot change these defaults.

Install Using virt-install With Options

This example uses virt-install with options to install a Solaris domU from the command line using an ISO image. The command line options specify for virt-install to create an 18-Gbyte root disk image file /xvm/domu-x16.img. The option --nographics is used because this is a Solaris paravirtualized configuration. If you invoke virt-install with command line options but do not supply all required information, the tool prompts you for the needed information.

machine:root> virt-install --nographics  -n domu-x16 --paravirt \
 -f /xvm/domu-x16.img -r 1011 \
 -l /net/inst-server/export/xVM/x_iso/63-0419-nd.iso
Starting install...
Creating domain...
SunOS Release 5.11 Version 64-bit
Copyright 1983-2007 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Configuring /dev
Solaris Interactive Text (Console session)
Using install cd in /dev/dsk/c0d1p0
Using RPC Bootparams for network configuration information.
Attempting to configure interface xnf0...
Skipped interface xnf0
Setting up Java. Please wait...
Beginning system identification...
Searching for configuration file(s)...
Search complete.
Discovering additional network configuration...

When the domain creation completes, sysidcfg runs to complete the system identification.

Example: How to Install Open Solaris 2008.11 or Later in Paravirtualized Mode

Use this procedure to set up the OpenSolaris 2008.11 or later release as a paravirtual guest. You must be running a Solaris dom0 on your system.

1. To start the installation of the OpenSolaris 2008.11 or later release, run the following commands:

   # zfs create rpool/zvol # zfs create -V 10G rpool/zvol/domu-220-root # virt-install --nographics --paravirt --ram 1024 --name domu-220 -f /dev/zvol/dsk/rpool/zvol/domu-220-root -l /isos/osol-2008.11.iso

   This procedure assumes that your server is set up to assign dynamic addresses. If you want to assign static addresses, specify the address with the mac property of the -w/--network option. See limiting bandwidth and setting a VLAN ID.

2. Choose the defaults on the console for the two questions regarding the server setup.

3. After the OpenSolaris 2008.11 Live CD or OpenSolaris 2009.06 release has finished booting, a VNC session is available from within the guest domain. You can connect to the guest domain's VNC session as follows:

   # domid=`virsh domid domu-220` # ip=`/usr/lib/xen/bin/xenstore-read /local/domain/$domid/ipaddr/0` # port=`/usr/lib/xen/bin/xenstore-read /local/domain/$domid/guest/vnc/port` # /usr/lib/xen/bin/xenstore-read /local/domain/$domid/guest/vnc/passwd DJP9tYDZ # vncviewer $ip:$port

4. Enter the given password at the VNC password prompt to bring up a VNC session.

   VNC sessions are not secure. However, because you need elevated privileges to read the VNC password from XenStore, the VNC sessions are secure as long as you run the VNC viewer locally on dom0, or via SSH tunnelling or another secure method.

5. Enable the post-install VNC viewer.

   By default, the VNC session is not enabled after the installation. You can change the default configuration as follows:

   # svccfg -s x11-server setprop options/xvm_vnc = "true" # svcadm restart xvm/vnc-config # svcadm restart gdm

Example: How to Install and Boot a Red Hat Enterprise Linux 5.3 PV Guest on an OpenSolaris 2009.06 dom0

The following sequence of commands installs a Red Hat Enterprise Linux guest over NFS using the text installer:

# mount -F hsfs /rhel.iso /mnt
# share -o ro /mnt
# virt-install -n pv-rhel -r 1024 -l nfs:mydom0:/mnt \
 --os-type=linux os-variant=rhel5.3 \
 -f /dev/zvol/dsk/pv-rhel.zvol -p --nographics

The following command installs a Red Hat Enterprise Linux guest using the media in the dom0 optical drive (CD-ROM/DVD) , utilizing the RedHat Linux version 5 kickstart feature to automate the installation process.

# virt-install  \
        --name rhat \
        --ram 500 \
        --file /dev/zvol/dsk/rhat.zvol \
        --paravirt \
        --location /dev/dsk/c2t0d0s2 \
        --os-type=linux os-variant=rhel5 \
        --extra-args "ks=/export/install/rhat/ks.cfg

Because of a standard Linux restriction that PV guests cannot be installed from a CD, the CD must be mounted in a location (usually dom0) exported over NFS, and then an NFS Linux installation done. Often it is much easier to do an HTTP Linux install.

Additional virt-install Examples

Example 39–4 Solaris PV Guest

virt-install -n solarisPV --paravirt -r 1024 \ 
  --nographics -f /export/solarisPV/root.img -s 16 \ 
  -l /ws/xvm-gate/public/isos/72-0910/solarisdvd.iso

Example 39–5 Solaris HVM Guest

virt-install -n solarisHVM --hvm -r 1024 --vnc \ 
  -f /export/solarisHVM/root.img -s 16 \ 
  -c /ws/xvm-gate/public/isos/72-0910/solarisdvd.iso

For this version of virt-install, ISO, physical media, and Preboot Execution Environment (PXE) pxe network installations are supported for HVM. If a physical CD is used, remember to unmount it after use.

Example 39–6 Windows HVM Guest

# virt-install -n winxp --hvm -r 1024 --vnc \ 
  -f /export/winxp/root.img -s 16 -c /windows/media.iso

For this version of virt-install, ISO, physical CD, and PXE network installations are supported for HVM. If a physical CD is used, remember to unmount it after use.

Example 39–7 Install Microsoft Windows Using a Local File as a Root Disk

A normal file is used to store the contents of the guest domain disk image, as opposed to using a ZFS volume, for example.

virt-install --name windows1 --ram 1024 \
--cdrom /en_winxp_pro_with_sp2.iso --file /guests/windows1-disk
--file-size 10 --vnc
 

Example 39–8 Install Solaris Using Network Install and JumpStart onto a ZFS Volume

zfs create -V 8G pool/solaris1-disk
virt-install --name solaris1 --ram 1024 --nographics \
--file /dev/zvol/dsk/pool/solaris1-disk \
--location nfs:install.domain.com:/export/solaris/nv75 \
--autocf nfs:install.domain.com:/export/jumpstart/solaris1

How to Complete the Solaris DomU sysidcfg Configuration

1. After the domain is created, the sysidcfg is initiated and you are prompted to answer a series of questions. Your screen will look similar to this:

   SunOS Release 5.11 Version 64-bit Copyright 1983-2007 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Hostname: my-zone Loading smf(5) service descriptions: 114/114 Select a Language 1. English 2. French 3. German 4. Italian 5. Japanese 6. Korean 7. Simplified Chinese 8. Spanish 9. Swedish 10. Traditional Chinese Please make a choice (1 - 10), or press h or ? for help: What type of terminal are you using? 1) ANSI Standard CRT 2) DEC VT52 3) DEC VT100 4) Heathkit 19 5) Lear Siegler ADM31 6) PC Console 7) Sun Command Tool 8) Sun Workstation 9) Televideo 910 10) Televideo 925 11) Wyse Model 50 12) X Terminal Emulator (xterms) 13) CDE Terminal Emulator (dtterm) 14) Other Type the number of your choice and press Return: . . .

   For more information on the sysidcfg file, see the sysidcfg(4) man page. For an example sysidcfg file, see Understanding the Solaris xVM Server Architecture, Part No 820-3089-102.

Managing Guests

The main interface for command and control of both xVM control domains and guest domains is the virsh(1M) utility. Users should use virsh wherever possible to control virtualized operating systems. Some xVM operations are not yet implemented by the virsh utility. In those cases, the legacy utility xm(1M) can be used for detailed control.

The following actions can be performed:

• Start Selected Guest

  # virsh start sdomu

• Suspend Selected Guest

  # virsh suspend sdomu

  You can also use the destroy subcommand to bring the system to the shutoff state, but this can result in damage to data.

• Resume Selected Guest

  # virsh resume sdomu

• Shutdown Selected Guest

  # virsh shutdown sdomu

• Reboot Selected Guest

  # virsh reboot sdomu

• Undefine Selected Guest

  # virsh undefine sdomu

  Undefine the configuration for an inactive domain which is specified by either its domain name or UUID.

  To delete a guest, undefine the guest and then remove any associated disk resources.

• Connect Selected Guest to Network

  # virsh attach-interface sdomu bridge e1000g0

• Take Snapshot of Active Domain

  # virsh save sdomu /domains/sdomusnap

  The domain will be in the shut down state when the save operation completes. The resources, such as memory, allocated for the domain will be freed for use by other running domains.

  To restore:

  # virsh restore sdomu /domains/sdomusnap

  Note that network connections present before the save operation might be severed, since TCP timeouts might have expired.

• Pin vCPU

  # virsh vcpupin domain vcpu cpulist

  Pin domain's virtual CPUs to the host's physical CPUs. The domain parameter is the domain name, ID, or UUID. The vcpu parameter is the VCPU number. The cpulist parameter is a list of host CPU numbers, separated by commas.

• Restore Selected Guest

  # virsh restore sdomu

  Restore a domain that is in the saved state.

• Delete Selected Guest

  Deleting the guest will cause its image and snapshot to be deleted from the system.

To view the state of a domain, use the virsh list command.

For example:

#  virsh list
ID    Name        State  
-------------------------
0     Domain-0    running
2     sxc18       running

By default, only running domains are displayed. Use virsh list --inactive to display only non-running domains.

Live Migration

The following prerequisites apply:

• Both the source machine and the target host must be on the same subnet.

• The host and the target must each have the same CPU type (AMD or Intel).

• Both systems must be running the same release of the xVM software.

• There must be sufficient CPU and memory resources on the target to host the domain.

• The target dom0 should have the same network interface as the source dom0 network interface used by the domU. For example, if the domU to be migrated has a VNIC that is bridged over the e1000g0 interface on the source dom0, then the target dom0 must also have the e1000g0 interface.

Enabling Live Migration on a Target Host

By default, xend listens only on the loopback address for requests from the localhost. The target host must be configured to accept the migration of a guest domain. The following example configures the xend SMF service on the target machine to accept guest migration from a system named host1. The caret (^) and dollar sign ($) are pattern-matching characters to ensure that the entire host name matches. The host1 name must match the name the target thinks the machine is called, which could be a host name, but could also be a fully qualified domain name (FQDN).

# svccfg -s svc:system/xvm/xend
svc:/system/xvm/xend> setprop config/xend-relocation-address = ""
svc:/system/xvm/xend> setprop config/xend-relocation-hosts-allow = "^host1\.?.*$ ^localhost$"
svc:/system/xvm/xend> end
# svcadm refresh svc:system/xvm/xend:default && \
svcadm restart svc:system/xvm/xend:default

You can test the connection by using:

host1# telnet target-host 8002

If connection fails, check the /var/log/xen/xend.log file on the target system.

Configuring the Guest Domain to Be Migrated

In addition to configuring the target system to accept migrations, you must also configure the domain that will be migrated so that the domain's storage is accessible from both the source and the target systems. The domain's accessible disks must reside on some form of shared storage, such as NFS files or iSCSI volumes. This document uses the NFS method available in the OpenSolaris 2009.06 release.

Prepare the NFS Storage on the NFS Server

On the NFS server, share the directory:

# sharectl set -p nfsmapid_domain=sun.com nfs
# svcadm restart svc:/network/nfs/mapid:default
# share -F nfs -o "sec=sys,root=host1:host2,rw" /vdisks

On both the host1 source system and the host2target system, also execute the sharectl to set the NFS mapid name to sun.com, and the svcadm command restart the xend service.

Create a PV Guest on the Source Machine (host1)

# virt-install -p --nographics -n domain -r 1024 -l /isos/os0906/os0906.iso -f /net/hostname_of_nfs_server/vdisks/testpv

The virt-install command then creates the virtual disk on the NFS server and starts the guest installation process.

iscsi Volume Method, Available in a Later Build

This method should be available in a build after the OpenSolaris 2009.06 release.

An iscsi-backed guest would be created with this base command:

# virt-install -n <name> -r <ram> -p --nographics -l /path/to/iso \
-m <mac address> \
--disk path=/static/<iscsi target ip address>/<lun>/<iqnnumber>,driver=phy,subdriver=iscsi 

An example would be:

# virt-install -n ubuntu -r 1024 -p --nographics -l /net/xvm-4200m2-03/isos/ubuntu-7.04-32.iso \ 
--disk path=/static/172.20.26.10/0/iqn.1986-03.com.sun:02:52ac879e-788e-e0ea-bf5c-f86b2b63258a,driver=phy,subdriver=iscsi

In addition to setting up the xend relocation host allow field as described above in the section "Enabling Live Migration on a Target Host,” also issue the following command to enable static discovery on both dom0s:

# iscsiadm modify discovery -s enable 

Migrating the Guest Domain

Use the following command to migrate the guest.

host1# virsh migrate domain --live xen:// xenmigr://target-host

While the use of the virsh command is preferred, you can try the following command if the virsh command appears to fail.

host1# xm migrate -l domain target-host

You can observe the migration while it occurs by monitoring the domain status on both machines using virsh list.

Note that the domain definition remains on the system on which it was created. You can start the domain on that system's dom0 with the virsh start domain command.

Assigning a CD-ROM Device to a Guest

Assigning the CD-ROM Device to an HVM Guest

You cannot add temporary access to a device for an HVM domU because you cannot dynamically add IDE CD-ROM devices. To make a CD-ROM available for use in an HVM guest, you must add the device before you boot the guest. You can then change the media by using virsh attach-disk.

Note that in an HVM domU, you must use eject in the domU to unlock a removable-media device, such as a CD device, before running the attach-disk subcommand.

The attach-disksubcommand is also used to change the media in a CD drive.

For additional explanation, see the virsh(1M) man page, including “Example 4 Changing a CD in a Solaris HVM Guest Domain.”

Assigning the CD-ROM Device to a PV Guest

In PV guest domains, you can use virsh attach-disk to add a CD device and detach-disk to remove it. These commands can be used on halted or running domUs. If you want to change the CD after attaching the device, you must use virshdetach-disk and then virsh attach-disk with the new media.

About Xvnc

Virtual network computing (VNC) is a remote control software product that allows you to view and fully interact with one computer desktop, the Xvnc server, by using the VNC viewer on another computer desktop. The two computers do not have to be running the same type of operating system. VNC provides a guest domain graphical login.

By default, consoles for HVM guests are graphics consoles. You can use VNC to view a Windows guest domain from a Solaris dom0. You only need to set the address and password for VNC to work with HVM guests. HVM installs may specify either VNC (--vnc) or Simple DirectMedia Layer (SDL) (--sdl) for graphics support. You can later configure the OS to use the serial console as the main console.

Use the vncpasswd command to set the password used to access VNC desktops. The password is stored on the server. For more information, see vncpasswd(1).

Xvnc displays to a VNC viewer over the network. The VNC server display number is the same as the X server display number. For example, snoopy:2 refers to display 2on machine snoopy for both VNC and an X server.

For the latest information on VNC setup, see .

How to Set Up VNC to Provide a Solaris Guest Graphical Login with CDE

1. Become superuser, or assume the appropriate role.

2. Enable XDMCP connections:.

   # svccfg -s cde-login svc:/application/graphical-login/cde-login> setprop dtlogin/args=""

3. (Optional) If you are not running vncviewer locally on the control domain, set X11-server to listen to the tcp port:

   # svccfg -s x11-server svc:/application/x-11/x11-server> setprop options/tcp_listen=true

   The VNC listen facility should be used with caution due to security considerations.

4. Enable the Xvnc inetd services.

   # svcadm enable xvnc-inetd

5. Connect from another machine and verify that you see the login screen and can log in to a desktop session.

   # vncviewer domU:0

How to Set Up VNC to Provide a Solaris Guest Graphical Login with GDM

1. Become superuser, or assume the appropriate role.

2. Enable XDMCP for GDM:

   # printf '[xdmcp]\nEnable=true\n' >>/etc/X11/gdm/custom.conf # svcadm restart gdm

3. Make sure that GDM is running:

   # svcadm enable -s gdm

4. Set the X11-server to listen to the tcp port:

   # svccfg -s x11-server svc:/application/x-11/x11-server> setprop options/tcp_listen=true

5. Enable the Xvnc inetd services:

   # svcadm enable xvnc-inetd

6. Connect from another machine and verify that you see the login screen and can log in to a desktop session.

   # vncviewer domU:0

How to Start VNC at System Boot

This procedure starts VNC at system boot from the dtlogin, displaying the dtlogin login screen.

1. Become superuser, or assume the appropriate role.

2. Add an instance of x11-server service called display1 for configuration, and configure it to run Xvnc.

   svccfg -s application/x11/x11-server add display1 svccfg -s application/x11/x11-server:display1 addpg options application svccfg -s application/x11/x11-server:display1 addpropvalue options/server astring: "/usr/X11/bin/Xvnc" svccfg -s application/x11/x11-server:display1 addpropvalue options/server_args astring: '"SecurityTypes=None"'

3. Configure dtlogin to start it.

   mkdir -p /etc/dt/config cp /usr/dt/config/Xservers /etc/dt/config/Xservers echo " :1 Local local_uid@none root /usr/X11/bin/Xserver :1" >> /etc/dt/config/Xservers pkill -HUP dtlogin

4. Connect from another machine and verify that you see the login screen and can log in to a desktop session.

   # vncviewer domU:0

How to Start a GNOME Session

1. Use the following to start the GNOME session.

   # /bin/sh # mkdir <your homedir>/.vnc # echo "#!/bin/sh\n/usr/bin/dbus-launch /usr/bin/gnome-session" > <your homedir>/.vnc/xstartup

How to View the VNC Man Pages

You can use the man command to view the man pages.

• man Xvnc

• man vncviewer

• man vncpasswd

Live links to these man pages cannot be made from this book.

The MANPATH variable is normally set for you in desktop login sessions. If the entry is not found, check your MANPATH environment variable and add the path to the X11 man pages if necessary.

1. View the MANPATH:

   echo $MANPATH /usr/share/man:/usr/dt/man:/usr/openwin/share/man

2. If necessary, add the path to the X11 man pages.

   setenv MANPATH /usr/share/man:/usr/dt/man:/usr/openwin/man:/usr/X11/man

Chapter 40 xVM System Administration

This chapter covers xVM system administration topics.

Printing Kernel and Machine Information

Use uname to determine the kernel you are running.

hostname% uname -a
SunOS hostname 5.11 snv_80 i86pc i386 i86xpv

Use the isainfo command to print the basic application environments supported by the currently running system.

hostname% isainfo -x
amd64: sse2 sse fxsr amd_3dnowx amd_3dnow amd_mmx mmx cmov cx8 tsc fpu
i386: ahf sse2 sse fxsr amd_3dnowx amd_3dnow amd_mmx mmx cmov cx8 tsc fpu

Use the psrinfo command to display information about processors.

hostname% psrinfo -vp
The physical processor has 1 virtual processor (0)
x86 (Authentic AMD family 15 model 5 step 10 clock 2200 MHz)
Dual Core AMD Opteron(tm) Processor 275

Configuring the Serial Console to Be the Main Console

After an HVM guest domain has been installed, you can configure the OS to use the serial console as the main console.

How to Configure the Serial Console as the Main Console

1. Type:

   # eeprom console=ttya

2. Reboot the HVM guest domain.

virsh Command and Domain Management

The main command interface used to control both Solaris xVM and guest domains is the virsh command. virsh provides a generic and stable interface for controlling virtualized operating systems. Use virsh instead of xm wherever possible.

Many virsh commands act asynchronously. This means that the system prompt can return before the operation has completed.

If you modify CPUs or memory by using the virsh command, these changes will be saved in the configuration file and persist across reboots.

virsh Command Structure

Most virsh commands follow the format:

# virsh subcommand domain-id | name | uuid [options]

subcommand

One of the subcommands listed in the virsh(1M) man page

domain-id, name, or uuid

A domain identifier

options

An option to a subcommand

Example 40–1 Using a virsh Command

This line connects to a domU named sxc18.

# virsh console sxc18

virshCommand

The virsh is used to manage domains. You must run the commands as the root user or by assuming the appropriate role account on the host operating system. The commands cannot be run in the guest domain.

# virsh nodeinfo
CPU model:           i86pc
CPU(s):              2
CPU frequency:       2391 MHz
CPU socket(s):       2
Core(s) per socket:  1
Thread(s) per core:  1
NUMA cell(s):        1
Memory size:         4127744 kB

Ethernet-Type Interface Support

The OpenSolaris^TM OS supports all Ethernet-type interfaces, and their data-links can be administered with the dladm command.

Suspend and Resume Functions and Commands

Some xVM operations are not yet implemented in the virsh command. In those cases, the equivalent xm command can be used. Subcommand terminology differs between the xmand virsh commands. In particular, the suspend and resume commands have different meanings.

Cloning ZFS-Based Solaris Domains

If you are using a ZFS volume as the root disk for a domU, you can use the ZFS snapshot facilities to clone another domU with the same configuration. By taking a clone of the root disk, you can quickly provision similar domains.

For example, you might install Solaris as a guest domain, run sys-unconfig(1M), then clone that disk image for use in new Solaris domains. Installing a Solaris domain in this way requires only the configuration step, rather than a full install. The only extra storage used for the cloned domain is the amount needed for the differences between the domains.

You also have the capability to revert to a previous configuration if necessary.

Any clones created from a snapshot must be destroyed before the snapshot can be destroyed.

How to Use ZFS Snapshot to Clone a Solaris DomU

If you use a ZFS volume as the virtual disk for your guest domain, you can take a snapshot of the storage. The snapshot is used to create clones.

Note that you might want to use the sys-unconfig command described in sys-unconfig(1M) in the domain before you take the snapshot. The resulting clones would not have host names or name services configured, which is also known as "as-manufactured." When it comes up, the new clone displays the configuration screens.

1. Become superuser, or assume the appropriate role.

2. (Optional) To create a snapshot to produce domains that require sysidcfg to complete system identification, use the sys-unconfig command in a domain named domain1.

3. Shut down domain1.

   # virsh shutdown domain1

4. Take a snapshot of the root disk used by domain1.

   # zfs snapshot pool/domain1-root@clone

5. Create a clone named domain2 from the snapshot domain1-root@clone.

   # zfs clone pool/domain1-root@clone pool/domain2-root

6. (Optional) Display the snapshot and clone.

   # zfs list NAME USED AVAIL REFER MOUNTPOINT pool 92.0K 67.0G 9.5K /pool pool/domain1 8K 67.0G 8K /pool/domain1 pool/domain2 8K 67.0G 8K /pool/domain2

7. Dump the configuration of domain1.

   # virsh dumpxml domain1 >domain1.xml

8. Copy the configuration file domain1.xml to a file named domain2.xml.

   # cp domain1.xml domain2.xml

9. Make the following changes in the domain2.xml file.

   1. Replace domain1 in this line:

      <name>domain1</name>

      With the new name, domain2:

      <name>domain2</name>

   2. So that virsh will generate a new domain configuration, remove the UUID line, which looks like this:

      <uuid>72bb96b6e6cf13594fb0cd1290610611</uuid>

   3. Point to the new disk by editing the following line:

      <source dev='/dev/zvol/dsk/export/domain1-root'/>

      Change domain1–root to domain2–root so that the line appears as follows:

      <source dev='/dev/zvol/dsk/export/domain2-root'/>

10. Inform virsh about the new domain:

    # virsh define domain2.xml

11. Boot the cloned domain.

More Information on ZFS Snapshot Features

Also see Chapter 7, Working With ZFS Snapshots and Clones, in Solaris ZFS Administration Guide.

Recovery

You can keep snapshots of the guest domain OS installations that are known to be good images, and use ZFS rollback to revert to a snapshot if the domain has a problem. For more information, see Rolling Back a ZFS Snapshot in Solaris ZFS Administration Guide.

Communication From xVM Hypervisor to Dom0 Using xm

Although the hypervisor and dom0 work closely together to manage a running system, the dom0 operating system has little direct visibility into the hypervisor. The hypervisor's entire address space is inaccessible to the dom0.

The only source of information is provided by the xm command, a user-space tool that communicates with the hypervisor via hypercalls.

Some of the commonly used xm commands are:

xm info

Report static information about the machine, such as number of CPUs, total memory, and xVM version.

# xm info host : test release : 5.11 version : onnv-userj machine : i86pc nr_cpus : 2 nr_nodes : 1 sockets_per_node : 2 cores_per_socket : 1 threads_per_core : 1 cpu_mhz : 2391 hw_caps : 078bfbff:e1d3fbff:00000000:00000010 total_memory : 4031 free_memory : 1953 xen_major : 3 xen_minor : 1 xen_extra : .2-xvm xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : Thu Dec 20 20:11:49 2007 -0800 15623:41d827ccece7 cc_compiler : gcc version 3.4.3 (csl-sol210-3_4-20050802) cc_compile_by : userj cc_compile_domain : lab.sun.com cc_compile_date : Thu Dec 20 20:24:36 PST 2007 xend_config_format : 4

xm list

List all domains and some high-level information.

xm top

Analogous to the Linux top command, but it reports domain information instead of process information. Information about the xVM system and domains is displayed in a continuously updating manner through the xentop command. See xentop.

xm log

Display the contents of the xend log.

xm help

List all the available commands.

xentrace

Capture trace buffer data from xVM.

xentop

Display information about the xVM system and domains in a continuously updating manner. See xm top.

xm start domain

Start a managed domain that was created by virt-install.

If you modify guest domain CPUs or memory by using the xm command, these changes will be saved in the configuration file and persist across reboots.

See the xm(1M) man page for more information.

About Crash Dumps

Domain 0 and Hypervisor Crashes

On a running system, the hypervisor's memory is completely off-limits to dom0. If the hypervisor crashes, however, the resulting panic dump will generate a core file that provides a unified view of both xVM and dom0. In this core file, xVM appears as a Solaris kernel module named xpv. For example:

 > $c
                xpv`panic+0xbf()
                xpv`do_crashdump_trigger+0x19()
                xpv`keypress_softirq+0x35()
                xpv`do_softirq+0x54()
                xpv`idle_loop+0x55()

The following applies to crash dumps:

• If a dom0 crashes with a standard Solaris panic, the dump will include only the dom0.

• When the hypervisor itself panics, the resulting dump includes the xVM state as well as the dom0 state.

How to Force a Crash Dump of a Guest

You can use the following command to force a crash dump of an OpenSolaris guest in the event of problems within the guest:

# virsh dump domain

The crash dump file created can be analyzed through /bin/mdb. The xvm user must be able to write to the location specified.

Chapter 41 Troubleshooting Miscellaneous Sun xVM Problems

This chapter contains troubleshooting information for Sun xVM.

Dom0 Configuration Requirements

At this time, you should note the following issues when configuring dom0:

ZFS ARC limitation needed

ZFS must be prevented from taking too much memory from dom0. Limit the ARC by adding the following line to /etc/system:

set zfs:zfs_arc_max = 0x10000000

Reboot the system.

Limit dom0 memory

Limit dom0 to a specific amount of memory by using the Xen dom0_memoption. For example, if you have 4Gb of memory, give dom0 1Gb in /rpool/boot/grub/menu.lst:

... kernel$ /boot/$ISADIR/xen.gz dom0_mem=1024M ...

If Text-Only Boot Was not Selected in OpenSolaris 2008.11 Installation

When installing OpenSolaris 2008.11, the "text-only" boot option from the GRUB menu should be selected. This is the second entry. However, the following workaround can be used if the text-only boot option is not selected.

1. When the GNU GRand Unified Bootloader (GRUB) boot menu appears on the console, type e for edit before the boot selection times out and starts booting the default entry.

2. The edit menu screen that pops up will have several lines on it. Use the down arrow key to scroll down to the line foreground, and type d for delete.

3. Repeat the Step 2 process for the line background. Use the down arrow key to scroll down to the line that says background, and type d for delete.

4. Using the arrow keys, scroll to the kernel line that contains the boot arguments, and type e for edit. Then, delete the entry for console=graphics.

5. Press Return to exit line edit and save the changes, and then type b to continue the boot.

Verifying System Elements

How to Verify Devices in /dev/xen

1. Change directories to /dev/xen and list the contents:

   # cd /dev/xen # ls balloon domcaps evtchn privcmd xenbus

How to Verify That the xvm Hypervisor Services Are Started

1. Become superuser, or assume the appropriate role.

2. Verify that the xvm services are running.

   # svcs -a | grep xvm

   If the system displays the following, the services are not running:

   disabled Dec_07 svc:/system/xvm/store:default disabled Dec_07 svc:/system/xvm/xend:default disabled Dec_07 svc:/system/xvm/console:default disabled Dec_07 svc:/system/xvm/domains:default

3. If the services are not running, verify that you booted an i86xpv kernel.

   # uname -i i86xpv

   Reboot if necessary.

4. If the correct kernel is running, enable the services.

   # svcadm enable xvm/store # svcadm enable xvm/xend # svcadm enable xvm/console # svcadm enable xvm/domains

   You are now ready to create guest domains (domUs).

Tracing the Interaction Between dom0 and the Hypervisor

Due to the isolation of the hypervisor from dom0, there is currently no way to apply DTrace directly to the hypervisor. There is an xpv DTrace provider that allows you to trace the interaction between dom0 and the hypervisor.

To list the available DTrace probes, use the following command line:

# dtrace -l -i 'xpv:::'

To enable all probes, use the following command line

# dtrace -n 'xpv::: {}'

Guest Issues

Known OpenSolaris 2009.06 Guest Issues

• If you are running an OpenSolaris 2009.06 HVM guest on Hyper-V SP2 (RC build), you'll need to update your OpenSolaris 2009.06 to at least snv_113to get the fix for CR 6768204, “dnet interface takes a long time to resume after plumb/unplumb in Hyper-V virtual machine.” Without the update, the network interface doesn't work for several minutes after the first interface unplumb/plumb operation. Obtain the update from the OpenSolaris Packaging Repository .

• If you're running an OpenSolaris 2009.06 HVM guest on OpenSolaris 2009.06 dom0, you'll need to apply the workaround for CR 6828707 for XServer to start on the guest.

  The issue is how long gdm should wait before it assumes a started Xserver is defunct. The default set in /usr/share/gdm/factory-defaults.conf is:

  GdmXserverTimeout=10

  The default should be increased. To override the default and set the timeout to the recommended value, add the following to /etc/X11/gdm/custom.conf:

  GdmXserverTimeout=30

  If the workaround is not used, the user will not see the GUI login window. If booted in text mode, only a console login window will be available. If booted in graphics mode, only the OpenSolaris startup screen will be seen.

Known Issues With Running Solaris 10 Guests on Other Operating Systems

• CR 6810371 HCTS CPU test fail with "can not fork: too many processes" if the virtual machine uses a dynamic hard disk.

  This problem affects Solaris 10 5/09 HVM guests. The workaround is to either change hard disk of the virtual machine to fixed-size, or to create a big file and point system swap to use that file.

For More Information

For additional documentation and more information about the hypervisor feature, visit the OpenSolaris community. The FAQ at this site will be updated to contain the latest issues.

For the latest information on dom0, see the document dom0 configuration for admins . See the document New Network Options, Including Limiting Bandwidth and Setting a VLAN ID, for Virtual Network Interfaces Attached to a Guest at this site to learn about new conventions for network options in virt-install.

xVM 3.3 documentation can be found on the OpenSolaris Community: Xen site.