Go to main content

Oracle® VM Server for SPARC 3.6 Administration Guide

Exit Print View

Updated: September 2019
 
 

Creating an I/O Domain by Assigning PCIe Endpoint Devices

You can assign an individual PCIe endpoint (or direct I/O-assignable) device to a domain. This use of PCIe endpoint devices increases the granularity of the device assignment to I/O domains. This capability is delivered by means of the direct I/O (DIO) feature.

The DIO feature enables you to create more I/O domains than the number of PCIe buses in a system. The possible number of I/O domains is now limited only by the number of PCIe endpoint devices.

    A PCIe endpoint device can be one of the following:

  • A PCIe card in a slot

  • An on-board PCIe device that is identified by the platform


Note - Because root domains cannot have dependencies on other root domains, a root domain that owns a PCIe bus cannot have its PCIe endpoint devices or SR-IOV virtual functions assigned to another root domain. However, you can assign a PCIe endpoint device or virtual function from a PCIe bus to the root domain that owns that bus.

The following diagram shows that the PCIe endpoint device, PCIE3, is assigned to an I/O domain. Both bus pci_0 and the switch in the I/O domain are virtual. The PCIE3 endpoint device is no longer accessible in the primary domain.

In the I/O domain, the pci_0 block and the switch are a virtual root complex and a virtual PCIe switch, respectively. This block and switch are similar to the pci_0 block and the switch in the primary domain. In the primary domain, the devices in slot PCIE3 are a “shadow” form of the original devices and are identified as SUNW,assigned.


Caution

Caution  - You cannot use Oracle Solaris hot-plug operations to hot-remove a PCIe endpoint device after that device is removed from the primary domain by using the ldm remove-io command. For information about replacing or removing a PCIe endpoint device, see Making PCIe Hardware Changes.


Figure 5  Assigning a PCIe Endpoint Device to an I/O Domain

image:Shows how to assign a PCIe endpoint device to an I/O domain.

Use the ldm list-io command to list the PCIe endpoint devices.

Though the DIO feature permits any PCIe card in a slot to be assigned to an I/O domain, only certain PCIe cards are supported. See Direct I/O Hardware and Software Requirements.


Caution

Caution  - PCIe cards that have a bridge are not supported. PCIe function-level assignment is also not supported. Assigning an unsupported PCIe card to an I/O domain might result in unpredictable behavior.


    The following items describe important details about the DIO feature:

  • This feature is enabled only when all the software requirements are met. See Direct I/O Hardware and Software Requirements.

  • Only PCIe endpoints that are connected to a PCIe bus assigned to a root domain can be assigned to another domain with the DIO feature.

  • I/O domains that use DIO have access to the PCIe endpoint devices only when the root domain is running.

  • Rebooting the root domain affects I/O domains that have PCIe endpoint devices. See Rebooting the Root Domain With PCIe Endpoints Configured. The root domain also performs the following tasks:

    • Initializes and manages the PCIe bus.

    • Handles all bus errors that are triggered by the PCIe endpoint devices that are assigned to I/O domains. Note that only the primary domain receives all PCIe bus-related errors.