C H A P T E R  4

Setting Up Services and Logical Domains

This chapter describes the procedures necessary to set up default services, your control domain, and guest domains.


Output Messages

You receive different output messages from the commands you use to create default services and to set up the control (primary) domain depending on your platform:

Sun UltraSPARC T1 Processors

You receive the following notice after the setup commands for the primary domain if you are using a server with a Sun UltraSPARC T1 processor:


Notice: the LDom Manager is running in configuration mode. Any 
configuration changes made will only take effect after the machine
configuration is downloaded to the system controller and the host
is reset.

Sun UltraSPARC T2 Processors

First Operation – You receive the following message after the first operation on any device or for any service on the primary domain if you are using a server with a Sun UltraSPARC T2 processor:


Initiating delayed reconfigure operation on LDom primary. All
configuration changes for other LDoms are disabled until the
LDom reboots, at which time the new configuration for LDom
primary will also take effect.

Subsequent Operations Until Reboot – You receive the following notice after every subsequent operation on the primary domain until reboot if you are using a server with a Sun UltraSPARC T2 processor:


Notice: LDom primary is in the process of a delayed
reconfiguration. Any changes made to this LDom will only take
effect after it reboots.


Creating Default Services

You must create the following virtual default services initially to be able to use them later:

procedure icon  To Create Default Services

  1. Create a virtual disk server (vds) to allow importing virtual disks into a logical domain.

    For example, the following command adds a virtual disk server (primary-vds0) to the control domain (primary).


    primary$ ldm add-vds primary-vds0 primary
    

  2. Create a virtual console concentrator service (vcc) for use by the virtual network terminal server daemon (vntsd) and as a concentrator for all logical domain consoles.

    For example, the following command would add a virtual console concentrator service (primary-vcc0) with a port range from 5000 to 5100 to the control domain (primary).


    primary$ ldm add-vcc port-range=5000-5100 primary-vcc0 primary
    

  3. Create a virtual switch service (vsw) to enable networking between virtual network (vnet) devices in logical domains. Assign a GLDv3-compliant network adapter to the virtual switch if each of the logical domains needs to communicate outside the box through the virtual switch.

    For example, the following command would add a virtual switch service (primary-vsw0) on network adapter driver e1000g0 to the control domain (primary).


    primary$ ldm add-vsw net-dev=e1000g0 primary-vsw0 primary
    

    This command automatically allocates a MAC address to the virtual switch. You can specify your own MAC address as an option to the ldm add-vsw command. However, in that case, it is your responsibility to ensure that the MAC address specified does not conflict with an already existing MAC address.

    If the virtual switch being added replaces the underlying physical adapter as the primary network interface, it must be assigned the MAC address of the physical adapter, so that the Dynamic Host Configuration Protocol (DHCP) server assigns the domain the same IP address. See Enabling Networking Between the Control/Service Domain and Other Domains.


    primary$ ldm add-vsw mac-addr=2:04:4f:fb:9f:0d net-dev=e1000g0 primary-vsw0 primary
    

  4. Verify the services have been created by using the list-services subcommand. Your output should look similar to the following.


    primary$ ldm list-services primary
    VDS
        NAME             VOLUME         OPTIONS          DEVICE
        primary-vds0
     
    VCC
        NAME             PORT-RANGE
        primary-vcc0     5000-5100
     
    VSW
        NAME             MAC               NET-DEV   DEVICE     MODE
        primary-vsw0     02:04:4f:fb:9f:0d e1000g0   switch@0   prog,promisc
    


Initial Configuration of the Control Domain

Initially, all system resources are allocated to the control domain. To allow the creation of other logical domains, you must release some of these resources.



Note - The notices that the LDoms Manager is running in configuration mode in the output in the following examples apply only to the Sun UltraSPARC T1 processors.



procedure icon  To Set Up the Control Domain



Note - This procedure contains examples of resources to set for your control domain. These numbers are examples only, and the values used might not be appropriate for your control domain.



  1. Assign cryptographic resources to the control domain.



    Note - If you have any cryptographic devices in the control domain, you cannot dynamically reconfigure CPUs. So if you are not using cryptographic devices, set-mau to 0.



    The following example would assign one cryptographic resource to the control domain, primary. This leaves the remainder of the cryptographic resources available to a guest domain.


    primary$ ldm set-mau 1 primary
    

  2. Assign virtual CPUs to the control domain.

    For example, the following command would assign 4 virtual CPUs to the control domain, primary. This leaves the remainder of the virtual CPUs available to a guest domain.


    primary$ ldm set-vcpu 4 primary
    

  3. Assign memory to the control domain.

    For example, the following command would assign 1 gigabyte of memory to the control domain, primary. This leaves the remainder of the memory available to a guest domain.


    primary$ ldm set-memory 1G primary
    



    Note - If you are not using ZFS to deliver disk services, 1 gigabyte of memory should be adequate. If you are using ZFS to deliver disk services, assign a complete core of 4 virtual CPUs and at least 4 gigabyte of memory. You may need to assign additional complete cores for heavier I/O loads.



  4. Add a logical domain machine configuration to the system controller (SC).

    For example, the following command would add a configuration called initial.


    primary$ ldm add-config initial
    



    Note - Currently, there is a limit of 8 configurations that can be saved on the SC, not including the factory-default configuration.



  5. Verify that the configuration is ready to be used at the next reboot.


    primary$ ldm list-config
    factory-default [current]
    initial [next]
    

    This list subcommand shows that the factory-default configuration set is currently being used and the initial configuration set will be used once you reboot.


Rebooting to Use Logical Domains

You must reboot the control/service domain for the configuration changes to take effect and the resources to be released for other logical domains to use.

procedure icon  To Reboot to Use Logical Domains


Enabling Networking Between the Control/Service Domain and Other Domains

By default, networking between the control/service domain and other domains in the system is disabled. To enable this, the virtual switch device should be configured as a network device. The virtual switch can either replace the underlying physical device (e1000g0 in this example) as the primary interface or be configured as an additional network interface in the domain.



Note - Perform the following configuration steps from the domain’s console, as the procedure could temporarily disrupt network connectivity to the domain.



procedure icon  To Configure the Virtual Switch as the Primary Interface

  1. Print out the addressing information for all interfaces.


    primary# ifconfig -a
    

  2. Plumb the virtual switch. In this example, vsw0 is the virtual switch being configured.


    primary# ifconfig vsw0 plumb
    

  3. (Optional) To obtain the list of all virtual switch instances in a domain, you can list them.


    primary# /usr/sbin/dladm show-link | grep vsw
    vsw0            type: non-vlan  mtu: 1500       device: vsw0
    

  4. Unplumb the physical network device assigned to the virtual switch (net-dev), which is e1000g0 in this example.


    primary# ifconfig e1000g0 down unplumb
    

  5. To migrate properties of the physical network device (e1000g0) to the virtual switch (vsw0) device, do one of the following:

    • If networking is configured using a static IP address, reuse the IP address and netmask of e1000g0 for vsw0.


      primary# ifconfig vsw0 IP_of_e1000g0 netmask netmask_of_e1000g0 broadcast + up
      

    • If networking is configured using DHCP, enable DHCP for vsw0.


      primary# ifconfig vsw0 dhcp start
      

  6. Make the required configuration file modifications to make this change permanent.


    primary# mv /etc/hostname.e1000g0 /etc/hostname.vsw0
    primary# mv /etc/dhcp.e1000g0 /etc/dhcp.vsw0
    



    Note - If necessary, you can also configure the virtual switch as well as the physical network device. In this case, plumb the virtual switch as in Step 2, and do not unplumb the physical device (skip Step 4). The virtual switch must then be configured with either a static IP address or obtain a dynamic IP address from a DHCP server.




Enabling the Virtual Network Terminal Server Daemon

You must enable the virtual network terminal server daemon (vntsd) to provide access to the virtual console of each logical domain. Refer to the Solaris 10 OS Reference Manual collection or the vntsd(1M) man page for information about how to use this daemon.

procedure icon  To Enable the Virtual Network Terminal Server Daemon



Note - Be sure you have created the default service vconscon on the control domain before you enable vntsd. See Creating Default Services for more information.



  1. Use the svcadm(1M) command to enable the virtual network terminal server daemon, vntsd(1M).


    # svcadm enable vntsd
    

  2. Use the svcs(1) command to verify that the vntsd is enabled.


    # svcs -l vntsd
    fmri         svc:/ldoms/vntsd:default
    enabled      true
    state        online
    next_state   none
    state_time   Sat Jan 27 03:14:17 2007
    logfile      /var/svc/log/ldoms-vntsd:default.log
    restarter    svc:/system/svc/restarter:default
    contract_id  93
    dependency   optional_all/error svc:/milestone/network (online)
    dependency   optional_all/none svc:/system/system-log (online)
    


Creating and Starting a Guest Domain

The guest domain must run an operating system that understands both the sun4v platform and the virtual devices presented by the hypervisor. Currently, this is the Solaris 10 11/06 OS at a minimum. Refer to the Logical Domains (LDoms) 1.0.2 Release Notes for any specific patches that might be necessary. Once you have created default services and reallocated resources from the control domain, you can create and start a guest domain.

procedure icon  To Create and Start a Guest Domain

  1. Create a logical domain.

    For example, the following command would create a guest domain named ldg1.


    primary$ ldm add-domain ldg1
    

  2. Add CPUs to the guest domain.

    For example, the following command would add four virtual CPUs to guest domain ldg1.


    primary$ ldm add-vcpu 4 ldg1
    

  3. Add memory to the guest domain.

    For example, the following command would add 512 megabytes of memory to guest domain ldg1.


    primary$ ldm add-memory 512m ldg1
    

  4. Add a virtual network device to the guest domain.

    For example, the following command would add a virtual network device with these specifics to the guest domain ldg1.


    primary$ ldm add-vnet vnet1 primary-vsw0 ldg1
    

    Where:

    • vnet1 is a unique interface name to the logical domain, assigned to this virtual network device instance for reference on subsequent set-vnet or remove-vnet subcommands.

    • primary-vsw0 is the name of an existing network service (virtual switch) to which to connect.

  5. Specify the device to be exported by the virtual disk server as a virtual disk to the guest domain.

    You can export a physical disk, disk slice, volumes, or file as a block device. Exporting loopback (lofi) devices as block devices is not supported in this release of Logical Domains software. The following examples show a physical disk and a file.

    • Physical Disk Example. The first example adds a physical disk with these specifics.


      primary$ ldm add-vdsdev /dev/dsk/c0t0d0s2 vol1@primary-vds0
      

      Where:

      • /dev/dsk/c0t0d0s2 is the path name of the actual physical device. When adding a device, the path name must be paired with the device name.

      • vol1 is a unique name you must specify for the device being added to the virtual disk server. The device name must be unique to this virtual disk server instance, because this name is exported by this virtual disk server to the clients for adding. When adding a device, the device name must be paired with the path name of the actual device.

      • primary-vds0 is the name of the virtual disk server to which to add this device.

    • File Example. This second example is exporting a file as a block device.


      primary$ ldm add-vdsdev path-to-file/filename vol1@primary-vds0
      

      Where:

    • path-to-file/filename is the path name of the actual file exported as a block device. When adding a device, the path name must be paired with the device name.

    • vol1 is a unique name you must specify for the device being added to the virtual disk server. The device name must be unique to this virtual disk server instance, because this name is exported by this virtual disk server to the clients for adding. When adding a device, the device name must be paired with the path name of the actual device.

    • primary-vds0 is the name of the virtual disk server to which to add this device.

  6. Add a virtual disk to the guest domain.

    The following example adds a virtual disk to the guest domain ldg1.


    primary$ ldm add-vdisk vdisk1 vol1@primary-vds0 ldg1
    

    Where:

    • vdisk1 is the name of the virtual disk.

    • vol1 is the name of the existing virtual disk server device to which to connect.

    • primary-vds0 is the name of the existing virtual disk server to which to connect.



    Note - The virtual disks are generic block devices that are backed by different types of physical devices, volumes, or files. A virtual disk is not synonymous with a SCSI disk and, therefore, excludes the target ID in the disk label. Virtual disks in a logical domain have the following format: cNdNsN, where cN is the virtual controller, dN is the virtual disk number, and sN is the slice.



  7. Set auto-boot and boot-device variables for the guest domain.

    The first example command sets auto-boot\? to true for guest domain ldg1.


    primary$ ldm set-var auto-boot\?=true ldg1
    

    The second example command sets boot-device to vdisk for the guest domain ldg1.


    primary$ ldm set-var boot-device=vdisk ldg1
    

  8. Bind resources to the guest domain ldg1 and then list the domain to verify that it is bound.


    primary$ ldm bind-domain ldg1
    primary$ ldm list-domain ldg1
    NAME          STATE    FLAGS  CONS   VCPU MEMORY   UTIL  UPTIME
    ldg1          bound    -----  5001   4    512M 
    

  9. To find the console port of the guest domain, you can look at the output of the preceding list-domain subcommand.

    You can see under the heading Cons that logical domain guest 1 (ldg1) has its console output bound to port 5001.

  10. Start the guest domain ldg1.


    primary$ ldm start-domain ldg1
    

  11. Connect to the console of a guest domain. There are several ways you can do this.

    • You can log into the control domain and connect directly to the console port on the local host:


      $ ssh admin@controldom.domain
      $ telnet localhost 5001
      

    • You can also connect to a guest console over a network if it is enabled in the vntsd(1M) SMF manifest. For example:


      $ telnet host-name 5001
      

      A Service Management Facility manifest is an XML file that describes a service. For more information about creating an SMF manifest, refer to the Solaris 10 System Administrator Collection.



      Note - To access a non-English OS in a guest domain through the console, the terminal for the console must be in the locale required by the OS.




Jump-Starting a Guest Domain

If you are jump-starting a guest domain, you would use a normal JumpStart procedure with the following profile syntax changes from a regular Solaris OS JumpStart procedure to a JumpStart procedure specific to LDoms as shown in the following two examples.

Normal JumpStart Profile


filesys c1t1d0s0 free /
filesys c1t1d0s1 2048 swap
filesys c1t1d0s5 120 /spare1
filesys c1t1d0s6 120 /spare2

Virtual disk device names in a logical domain differ from physical disk device names in that they do not contain a target ID (tN) in the device name. Instead of the normal cNtNdNsN format, virtual disk device names are of the format cNdNsN, where cN is the virtual controller, dN is the virtual disk number, and sN is the slice. Modify your JumpStart profile to reflect this change as in the following profile example.

Actual Profile Used for a Logical Domain


filesys c0d0s0 free /
filesys c0d0s1 2048 swap
filesys c0d0s5 120 /spare1
filesys c0d0s6 120 /spare2