Logical Domains 1.2 Administration Guide

Chapter 5 Setting Up I/O Domains

This chapter describes how to set up additional I/O domains in a Logical Domains environment.

This chapter covers the following topics:

I/O Domains and PCI EXPRESS Buses

An I/O domain is a domain that has direct ownership of and direct access to physical I/O devices. It can be created by assigning a PCI EXPRESS (PCI-E) bus to a domain. PCI-E buses that are present on a server are identified with names such as pci@780 (bus_a) or pci@400 (pci_0). Use the ldm command to assign each bus to a separate domain.

The maximum number of I/O domains that you can create depends on the number of PCI-E buses available on the server. A Sun UltraSPARC T1 based server has up to two PCI-E buses, so it can have up to two I/O domains. A Sun UltraSPARC T2 Plus based server has up to four PCI-E buses, so it can have up to four I/O domains.

Note –

Sun UltraSPARC T2 based servers, such as the Sun SPARC Enterprise T5120 and T5220 servers, only have one PCI-E bus. So, such servers cannot configure more than one domain with direct access to physical devices. However, you can assign a Network Interface Unit (NIU) to a domain rather than configuring a new I/O domain. See Using NIU Hybrid I/O.

When a server is initially configured with Logical Domains or is using the factory-default configuration, the control domain has access to all the physical device resources. This means that the control (primary) domain is the only I/O domain configured on the system and that it owns all the PCI-E buses.

ProcedureCreate a New I/O Domain

This example procedure shows how to create a new I/O domain from an initial configuration where several buses are owned by the primary domain. By default the primary domain owns all buses present on the system. This example is for a Sun Fire T2000 server. This procedure can also be used on other servers. The instructions for different servers might vary slightly from these, but you can obtain the basic principles from this example.

First, you must retain the bus that has the primary domain's boot disk. Then, remove another bus from the primary domain and assign it to another domain.

Caution – Caution –

All internal disks on the supported servers are connected to a single PCI bus. If a domain is booted from an internal disk, do not remove that bus from the domain. Also, ensure that you are not removing a bus with devices (such as network ports) that are used by a domain. If you remove the wrong bus, a domain might not be able to access the required devices and could become unusable. To remove a bus that has devices that are used by a domain, reconfigure that domain to use devices from other buses. For example, you might have to reconfigure the domain to use a different onboard network port or a PCI card from a different PCI slot.

In this example, the primary domain only uses a disk (c1t0d0) and network interface (e1000g2). If the primary domain uses more devices, repeat Steps 2-4 for each device to ensure that none are located on the bus that will be removed.

  1. Verify that the primary domain owns several PCI buses.

    primary# ldm list-bindings primary
        DEVICE           PSEUDONYM        OPTIONS
        pci@780          bus_a
        pci@7c0          bus_b
  2. Determine the device path of the boot disk, which needs to be retained.

    primary# df /
    /                  (/dev/dsk/c1t0d0s0 ): 1309384 blocks   457028 files
  3. Determine the physical device to which the block device is linked.

    The following example uses block device c1t0d0s0:

    primary# ls -l /dev/dsk/c1t0d0s0
    lrwxrwxrwx   1 root     root          65 Feb  2 17:19 /dev/dsk/c1t0d0s0 -> ../

    In this example, the physical device for the primary domain's boot disk is connected to bus pci@7c0, which corresponds to the earlier listing of bus_b. This means that you can assign bus_a (pci@780) to another domain.

  4. Determine the network interface that is used by the system.

    primary# dladm show-phys
    LINK         MEDIA                STATE      SPEED  DUPLEX    DEVICE
    e1000g0      Ethernet             unknown    0      half      e1000g0
    e1000g1      Ethernet             unknown    0      half      e1000g1
    e1000g2      Ethernet             up         100    full      e1000g2
    e1000g3      Ethernet             unknown    0      half      e1000g3

    Interfaces that are in the unknown state are not configured, so they are not used. In this example, only the e1000g2 interface is used.

  5. Determine the physical device to which the network interface is linked.

    The following command uses the e1000g2 network interface:

    primary# ls -l /dev/e1000g2
    lrwxrwxrwx   1 root     root          48 Sep 25  2008 /dev/e1000g2
      -> ../devices/pci@7c0/pci@0/pci@2/network@0:e1000g2

    In this example, the physical device for the network interface used by the primary domain is under bus pci@7c0, which corresponds to the earlier listing of bus_a. This means that the boot disk and the network interface of the primary domain are on the same bus, bus_b (pci@7c0). So, bus_a (pci@780) can safely be assigned to another domain because it is not used by the primary domain.

    If the network interface used by the primary domain was on a bus that you want to assign to another domain, the primary domain would need to be reconfigured to use a different network interface.

  6. Remove the bus that does not contain the boot disk from a domain.

    In this example, bus pci@780 is being removed from the primary domain.

    primary# ldm remove-io pci@780 primary
  7. Save this configuration to the service processor.

    In this example, the configuration is io-domain.

    primary# ldm add-config io-domain

    This configuration, io-domain, is also set as the next configuration to be used after the reboot.

    Note –

    Currently, there is a limit of 8 configurations that can be saved on the SP, not including the factory-default configuration.

  8. Reboot the primary domain so that the change takes effect.

    primary# shutdown -i6 -g0 -y
  9. Stop the domain to which you want to add the PCI bus.

    The following example stops the ldg1 domain:

    primary# ldm stop ldg1
    primary# ldm unbind ldg1
  10. Add the available bus to the domain that needs direct access.

    The available bus is pci@780 and the domain is ldg1.

    primary# ldm add-io pci@780 ldg1

    If you have an Infiniband card, you might need to enable the bypass mode on the pci@780 bus. See Enabling the I/O MMU Bypass Mode on a PCI Bus for information on whether you need to enable the bypass mode.

  11. Restart the domain so that the change takes affect.

    The following commands restart the ldg1 domain:

    primary# ldm bind ldg1
    primary# ldm start ldg1
  12. Confirm that the correct bus is still assigned to the primary domain and the correct bus is assigned to domain ldg1.

    primary# ldm list-bindings primary ldg1
    primary       active  -n-cv  SP     4     4G      0.4%  18h 25m
        DEVICE           PSEUDONYM        OPTIONS
        pci@7c0          bus_b
    ldg1          active  -n---  5000   4     2G      10%   35m
        DEVICE           PSEUDONYM        OPTIONS
        pci@780          bus_a

    This output confirms that the PCI-E bus bus_b and the devices below it are assigned to domain primary, and bus_a and its devices are assigned to ldg1.

Enabling the I/O MMU Bypass Mode on a PCI Bus

If you have an Infiniband Host Channel Adapter (HCA) card, you might need to turn the I/O memory management unit (MMU) bypass mode on. By default, Logical Domains software controls PCI-E transactions so that a given I/O device or PCI-E option can only access the physical memory assigned within the I/O domain. Any attempt to access memory of another guest domain is prevented by the I/O MMU. This provides a higher level of security between the I/O domain and all other domains. However, in the rare case where a PCI-E or PCI-X option card does not load or operate with the I/O MMU bypass mode off, this option allows you to turn the I/O MMU bypass mode on. However, if you turn the bypass mode on, there no longer is a hardware-enforced protection of memory accesses from the I/O domain.

The bypass=on option turns on the I/O MMU bypass mode. This bypass mode should be enabled only if the respective I/O domain and I/O devices within that I/O domain are trusted by all guest domains. This example turns on the bypass mode.

primary# ldm add-io bypass=on pci@780 ldg1

The output shows bypass=on under OPTIONS.