Creating an I/O Domain by Assigning PCIe Endpoint Devices
You can assign an individual PCIe endpoint (or direct I/O-assignable) device to a domain. This use of PCIe endpoint devices increases the granularity of the device assignment to I/O domains. This capability is delivered by means of the direct I/O (DIO) feature.
The DIO feature enables you to create more I/O domains than the number of PCIe buses in a system. The possible number of I/O domains is now limited only by the number of PCIe endpoint devices.
A PCIe endpoint device can be one of the following:
-
A PCIe card in a slot
-
An on-board PCIe device that is identified by the platform
Note:
Because root domains cannot have dependencies on other root domains, a root domain that owns a PCIe bus cannot have its PCIe endpoint devices or SR-IOV virtual functions assigned to another root domain. However, you can assign a PCIe endpoint device or virtual function from a PCIe bus to the root domain that owns that bus.The following diagram shows that the PCIe endpoint device, PCIE3
, is assigned to an I/O domain. Both bus pci_0
and the switch in the I/O domain are virtual. The PCIE3
endpoint device is no longer accessible in the primary
domain.
In the I/O domain, the pci_0
block and the switch are a virtual root complex and a virtual PCIe switch, respectively. This block and switch are similar to the pci_0
block and the switch in the primary
domain. In the primary
domain, the devices in slot PCIE3
are a “shadow” form of the original devices and are identified as SUNW,assigned
.
Caution:
You cannot use Oracle Solaris hot-plug operations to hot-remove a PCIe endpoint device after that device is removed from theprimary
domain by using the ldm remove-io
command. For information about replacing or removing a PCIe endpoint device, see Making PCIe Hardware Changes.
Assigning a PCIe Endpoint Device to an I/O Domain
![Shows how to assign a PCIe endpoint device to an I/O domain. Shows how to assign a PCIe endpoint device to an I/O domain.](img/l10n_pcie_device_assignment.jpg)
Use the ldm list-io
command to list the PCIe endpoint devices.
Though the DIO feature permits any PCIe card in a slot to be assigned to an I/O domain, only certain PCIe cards are supported. See Direct I/O Hardware and Software Requirements.
Caution:
PCIe cards that have a bridge are not supported. PCIe function-level assignment is also not supported. Assigning an unsupported PCIe card to an I/O domain might result in unpredictable behavior.The following items describe important details about the DIO feature:
-
This feature is enabled only when all the software requirements are met. See Direct I/O Hardware and Software Requirements.
-
Only PCIe endpoints that are connected to a PCIe bus assigned to a root domain can be assigned to another domain with the DIO feature.
-
I/O domains that use DIO have access to the PCIe endpoint devices only when the root domain is running.
-
Rebooting the root domain affects I/O domains that have PCIe endpoint devices. See Rebooting the Root Domain With PCIe Endpoints Configured. The root domain also performs the following tasks:
-
Initializes and manages the PCIe bus.
-
Handles all bus errors that are triggered by the PCIe endpoint devices that are assigned to I/O domains. Note that only the
primary
domain receives all PCIe bus-related errors.
-