Assigning PCIe Endpoint Devices

Starting with the Oracle VM Server for SPARC 2.0 release and the Oracle Solaris 10 9/10 OS, you can assign an individual PCIe endpoint (or direct I/O-assignable) device to a domain. This use of PCIe endpoint devices increases the granularity of the device assignment to I/O domains. This capability is delivered by means of the direct I/O (DIO) feature.

The DIO feature enables you to create more I/O domains than the number of PCIe buses in a system. The possible number of I/O domains is now limited only by the number of PCIe endpoint devices.

A PCIe endpoint device can be one of the following:

A PCIe card in a slot
An on-board PCIe device that is identified by the platform

The following diagram shows that the PCIe endpoint device, PCIE3, is assigned to an I/O domain. Both bus pci_0 and the switch in the I/O domain are virtual. The PCIE3 endpoint device is no longer accessible in the primary domain.

In the I/O domain, the pci_0 block and the switch are a virtual root complex and a virtual PCIe switch, respectively. This block and switch are very similar to the pci_0 block and the switch in the primary domain. In the primary domain, the devices in slot PCIE3 are a shadow form of the original devices and are identified as SUNW,assigned.

Caution - You cannot use Oracle Solaris hot-plug operations to “hot remove” a PCIe endpoint device after that device is removed from the primary domain by using the ldm rm-io command. For information about replacing or removing a PCIe endpoint device, see Making PCIe Hardware Changes.

Figure 6-2 Assigning a PCIe Endpoint Device to an I/O Domain

image:Diagram shows how to assign a PCIe endpoint device to an I/O domain.

Use the ldm list-io command to list the PCIe endpoint devices.

Though the DIO feature permits any PCIe card in a slot to be assigned to an I/O domain, only certain PCIe cards are supported. See Direct I/O Hardware and Software Requirements in Oracle VM Server for SPARC 3.0 Release Notes.

Caution - PCIe cards that have a bridge are not supported. PCIe function-level assignment is also not supported. Assigning an unsupported PCIe card to an I/O domain might result in unpredictable behavior.

The following are a few important details about the DIO feature:

This feature is enabled only when all the software requirements are met. See Direct I/O Hardware and Software Requirements in Oracle VM Server for SPARC 3.0 Release Notes.
Only PCIe endpoints that are connected to a PCIe bus assigned to the primary domain can be assigned to another domain with the DIO feature.
I/O domains using DIO have access to the PCIe endpoint devices only when the primary domain is running.
Rebooting the root domain affects I/O domains that have PCIe endpoint devices. See Rebooting the Root Domain. The root domain also performs the following tasks:
- Initializes the PCIe bus and manages the bus.
- Handles all bus errors that are triggered by the PCIe endpoint devices that are assigned to I/O domains. Note that only the primary domain receives all PCIe bus-related errors.

Direct I/O Hardware and Software Requirements

To successfully use the DIO feature, you must be running the appropriate software and assign only the PCIe cards that are supported by the DIO feature to I/O domains. For the hardware and software requirements, see Direct I/O Hardware and Software Requirements in Oracle VM Server for SPARC 3.0 Release Notes.

Note - All PCIe cards that are supported on a platform are supported in the primary domain. See the documentation for your platform for the list of supported PCIe cards. However, only direct I/O-supported PCIe cards can be assigned to I/O domains.

To add or remove PCIe end-point devices by using the direct I/O feature, you must first enable I/O virtualization on the PCIe bus itself.

You can use the ldm set-io or ldm add-io command to set the iov property to on. You can also use the ldm add-domain or ldm set-domain command to set the rc-add-policy property to iov. See the ldm(1M) man page.

Rebooting the root domain affects direct I/O, so carefully plan your direct I/O configuration changes to maximize the direct I/O-related changes to the primary domain and to minimize primary domain reboots.

Current Direct I/O Feature Limitations

For information about how to work around the following limitations, see Planning PCIe Endpoint Device Configuration.

Assignment or removal of a PCIe endpoint device to any other domain is only permitted when that domain is either stopped or inactive.

Planning PCIe Endpoint Device Configuration

Carefully plan ahead when you assign or remove PCIe endpoint devices to avoid primary domain downtime. The reboot of the primary domain not only affects the services that are available on the primary domain itself, but it also affects the I/O domains that have PCIe endpoint devices assigned. Though the changes to each I/O domain do not affect the other domains, planning ahead helps to minimize the consequences on the services that are provided by that domain.

When in a delayed reconfiguration, you can continue to add or remove more devices, and then reboot the primary domain only one time to make all the changes take effect.

For an example, see How to Create an I/O Domain by Assigning a PCIe Endpoint Device.

The following describes the general steps you must take to plan and perform a DIO device configuration:

Understand and record your system hardware configuration.

Specifically, record information about the part numbers and other details of the PCIe cards in the system.

Use the ldm list-io -l and prtdiag -v commands to obtain the information and save it for future reference.
Determine which PCIe endpoint devices are required to be in the primary domain.

For example, determine the PCIe endpoint devices that provide access to the following:
- Boot disk device
- Network device
- Other devices that the primary domain offers as services
Remove all PCIe endpoint devices that you might use in I/O domains.

This step helps you to avoid performing subsequent reboot operations on the primary domain, as reboots affect I/O domains.

Use the ldm rm-io command to remove the PCIe endpoint devices. Use pseudonyms rather than device paths to specify the devices to the rm-io and add-io subcommands.

Note - After you have removed all the devices you want during a delayed reconfiguration, you only need to reboot the primary domain one time to make all the changes take effect.
Save this configuration to the service processor (SP).

Use the ldm add-config command.
Reboot the root domain to release the PCIe endpoint devices that you removed in Step 3.
Confirm that the PCIe endpoint devices you removed are no longer assigned to the primary domain.

Use the ldm list-io -l command to verify that the devices you removed appear as SUNW,assigned-device in the output.
Assign an available PCIe endpoint device to a guest domain to provide direct access to the physical device.

After you make this assignment, you can no longer migrate the guest domain to another physical system by means of the domain migration feature.
Add or remove a PCIe endpoint device to or from a guest domain.

Use the ldm add-io command.

Minimize the changes to I/O domains by reducing the reboot operations and by avoiding downtime of services offered by that domain.
(Optional) Make changes to the PCIe hardware.

See Making PCIe Hardware Changes.

Rebooting the Root Domain

The primary domain is the owner of the PCIe bus and is responsible for initializing and managing the bus. The primary domain must be active and running a version of the Oracle Solaris OS that supports the DIO feature. Shutting down, halting, or rebooting the primary domain interrupts access to the PCIe bus. When the PCIe bus is unavailable, the PCIe devices on that bus are affected and might become unavailable.

The behavior of I/O domains with PCIe endpoint devices is unpredictable when the primary domain is rebooted while those I/O domains are running. For instance, I/O domains with PCIe endpoint devices might panic during or after the reboot. Upon reboot of the primary domain, you would need to manually stop and start each domain.

To work around these issues, perform one of the following steps:

Manually shut down any domains on the system that have PCIe endpoint devices assigned to them before you shut down the primary domain.

This step ensures that these domains are cleanly shut down before you shut down, halt, or reboot the primary domain.

To find all the domains that have PCIe endpoint devices assigned to them, run the ldm list-io command. This command enables you to list the PCIe endpoint devices that have been assigned to domains on the system. So, use this information to help you plan. For a detailed description of this command output, see the ldm(1M) man page.

For each domain found, stop the domain by running the ldm stop command.
Configure a domain dependency relationship between the primary domain and the domains that have PCIe endpoint devices assigned to them.

This dependency relationship ensures that domains with PCIe endpoint devices are automatically restarted when the primary domain reboots for any reason.

Note that this dependency relationship forcibly resets those domains, and they cannot cleanly shut down. However, the dependency relationship does not affect any domains that were manually shut down.
```
# ldm set-domain failure-policy=reset primary
# ldm set-domain master=primary ldom
```

Making PCIe Hardware Changes

The following steps help you avoid misconfiguring the PCIe endpoint assignments. For platform-specific information about installing and removing specific hardware, see the documentation for your platform.

No action is required if you are installing a PCIe card into an empty slot. This PCIe card is automatically owned by the domain that owns the PCIe bus.

To assign the new PCIe card to an I/O domain, use the ldm rm-io command to first remove the card from the primary domain. Then, use the ldm add-io command to assign the card to an I/O domain.
No action is required if a PCIe card is removed from the system and assigned to the primary domain.
To remove a PCIe card that is assigned to an I/O domain, first remove the device from the I/O domain. Then, add the device to the primary domain before you physically remove the device from the system.
To replace a PCIe card that is assigned to an I/O domain, verify that the new card is supported by the DIO feature.

If so, no action is required to automatically assign the new card to the current I/O domain.

If not, first remove that PCIe card from the I/O domain by using the ldm rm-io command. Next, use the ldm add-io command to reassign that PCIe card to the primary domain. Then, physically replace the PCIe card you assigned to the primary domain with a different PCIe card. These steps enable you to avoid a configuration that is unsupported by the DIO feature.

How to Create an I/O Domain by Assigning a PCIe Endpoint Device

Plan all DIO deployments ahead of time to minimize downtime.

Caution - The primary domain loses access to the on-board DVD device if you assign the /SYS/MB/SASHBA1 slot on a SPARC T3-1 or a SPARC T4-1 system to a DIO domain.

The SPARC T3-1 and SPARC T4-1 systems include two DIO slots for on-board storage, which are represented by the /SYS/MB/SASHBA0 and /SYS/MB/SASHBA1 paths. In addition to hosting multi-headed on-board disks, the /SYS/MB/SASHBA1 slot hosts the on-board DVD device. So, if you assign /SYS/MB/SASHBA1 to a DIO domain, the primary domain loses access to the on-board DVD device.

The SPARC T3-2 and SPARC T4-2 systems have a single SASHBA slot that hosts all on-board disks as well as the on-board DVD device. So if you assign SASHBA to a DIO domain, the on-board disks and the on-board DVD device are loaned to the DIO domain and unavailable to the primary domain.

For an example of adding a PCIe endpoint device to create an I/O domain, see Planning PCIe Endpoint Device Configuration.

Note - In this release, it is best use the DefaultFixed NCP to configure datalinks and network interfaces on Oracle Solaris 11 systems.

The Oracle Solaris 11 OS includes the following NCPs:

DefaultFixed. Enables you to use the dladm or ipadm command to manage networking.
Automatic. Enables you to use the netcfg or netadm command to manage networking.

Ensure that the DefaultFixed NCP is enabled by using the netadm list command. See Chapter 7, Using Datalink and Interface Configuration Commands on Profiles, in Oracle Solaris Administration: Network Interfaces and Network Virtualization.

Identify and archive the devices that are currently installed on the system.

The output of the ldm list-io -l command shows how the I/O devices are currently configured. You can obtain more detailed information by using the prtdiag -v command.

Note - After the devices are assigned to I/O domains, the identity of the devices can only be determined in the I/O domains.

# ldm list-io -l
NAME                                      TYPE   BUS      DOMAIN   STATUS   
----                                      ----   ---      ------   ------   
niu_0                                     NIU    niu_0    primary           
[niu@480]
niu_1                                     NIU    niu_1    primary           
[niu@580]
pci_0                                     BUS    pci_0    primary           
[pci@400]
pci_1                                     BUS    pci_1    primary           
[pci@500]
/SYS/MB/PCIE0                             PCIE   pci_0    primary  OCC      
[pci@400/pci@2/pci@0/pci@8]
    SUNW,emlxs@0/fp/disk
    SUNW,emlxs@0/fp/tape
    SUNW,emlxs@0/fp@0,0
    SUNW,emlxs@0,1/fp/disk
    SUNW,emlxs@0,1/fp/tape
    SUNW,emlxs@0,1/fp@0,0
/SYS/MB/PCIE2                             PCIE   pci_0    primary  OCC      
[pci@400/pci@2/pci@0/pci@4]
    pci/scsi/disk
    pci/scsi/tape
    pci/scsi/disk
    pci/scsi/tape
/SYS/MB/PCIE4                             PCIE   pci_0    primary  OCC      
[pci@400/pci@2/pci@0/pci@0]
    ethernet@0
    ethernet@0,1
    SUNW,qlc@0,2/fp/disk
    SUNW,qlc@0,2/fp@0,0
    SUNW,qlc@0,3/fp/disk
    SUNW,qlc@0,3/fp@0,0
/SYS/MB/PCIE6                             PCIE   pci_0    primary  EMP      
[pci@400/pci@1/pci@0/pci@8]
/SYS/MB/PCIE8                             PCIE   pci_0    primary  EMP      
[pci@400/pci@1/pci@0/pci@c]
/SYS/MB/SASHBA                            PCIE   pci_0    primary  OCC      
[pci@400/pci@2/pci@0/pci@e]
    scsi@0/iport@1
    scsi@0/iport@2
    scsi@0/iport@4
    scsi@0/iport@8
    scsi@0/iport@80/cdrom@p7,0
    scsi@0/iport@v0
/SYS/MB/NET0                              PCIE   pci_0    primary  OCC      
[pci@400/pci@1/pci@0/pci@4]
    network@0
    network@0,1
/SYS/MB/PCIE1                             PCIE   pci_1    primary  OCC      
[pci@500/pci@2/pci@0/pci@a]
    SUNW,qlc@0/fp/disk
    SUNW,qlc@0/fp@0,0
    SUNW,qlc@0,1/fp/disk
    SUNW,qlc@0,1/fp@0,0
/SYS/MB/PCIE3                             PCIE   pci_1    primary  OCC      
[pci@500/pci@2/pci@0/pci@6]
    network@0
    network@0,1
    network@0,2
    network@0,3
/SYS/MB/PCIE5                             PCIE   pci_1    primary  OCC      
[pci@500/pci@2/pci@0/pci@0]
    network@0
    network@0,1
/SYS/MB/PCIE7                             PCIE   pci_1    primary  EMP      
[pci@500/pci@1/pci@0/pci@6]
/SYS/MB/PCIE9                             PCIE   pci_1    primary  EMP      
[pci@500/pci@1/pci@0/pci@0]
/SYS/MB/NET2                              PCIE   pci_1    primary  OCC      
[pci@500/pci@1/pci@0/pci@5]
    network@0
    network@0,1
    ethernet@0,80
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary           
[pci@400/pci@1/pci@0/pci@4/network@0]
    maxvfs = 7
/SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary           
[pci@400/pci@1/pci@0/pci@4/network@0,1]
    maxvfs = 7
/SYS/MB/PCIE5/IOVNET.PF0                  PF     pci_1    primary           
[pci@500/pci@2/pci@0/pci@0/network@0]
    maxvfs = 63
/SYS/MB/PCIE5/IOVNET.PF1                  PF     pci_1    primary           
[pci@500/pci@2/pci@0/pci@0/network@0,1]
    maxvfs = 63
/SYS/MB/NET2/IOVNET.PF0                   PF     pci_1    primary           
[pci@500/pci@1/pci@0/pci@5/network@0]
    maxvfs = 7
/SYS/MB/NET2/IOVNET.PF1                   PF     pci_1    primary           
[pci@500/pci@1/pci@0/pci@5/network@0,1]
    maxvfs = 7

Determine the device path of the boot disk that must be retained.
See Step 2 in How to Create an I/O Domain by Assigning a PCIe Bus.
Determine the physical device to which the block device is linked.
See Step 3 in How to Create an I/O Domain by Assigning a PCIe Bus.
Determine the network interface that is used by the system.
See Step 4 in How to Create an I/O Domain by Assigning a PCIe Bus.
Determine the physical device to which the network interface is linked.
The following command uses the igb0 network interface:
```
primary# ls -l /dev/igb0
lrwxrwxrwx   1 root     root          46 Jul 30 17:29 /dev/igb0 ->
../devices/pci@500/pci@0/pci@8/network@0:igb0
```
In this example, the physical device for the network interface used by the primary domain is connected to the PCIe endpoint device (pci@500/pci@0/pci@8), which corresponds to the listing of MB/NET0 in Step 1. So, you do not want to remove this device from the primary domain. You can safely assign all other PCIe devices to other domains because they are not used by the primary domain.
If the network interface used by the primary domain is on a bus that you want to assign to another domain, the primary domain would need to be reconfigured to use a different network interface.

Remove the PCIe endpoint devices that you might use in I/O domains.

In this example, you can remove the PCIE2, PCIE3, PCIE4, and PCIE5 endpoint devices because they are not being used by the primary domain.

Remove the PCIe endpoint devices.

Caution - Do not remove the devices that are used or required by the primary domain.

If you mistakenly remove the wrong devices, use the ldm cancel-reconf primary command to cancel the delayed reconfiguration on the primary domain.

You can remove multiple devices at one time to avoid multiple reboots.

# ldm start-reconf primary
primary# ldm set-io iov=on pci_1
All configuration changes for other domains are disabled until the primary domain reboots,
at which time the new configuration for the primary domain will also take effect.
primary# ldm remove-io /SYS/MB/PCIE1 primary
------------------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------
primary# ldm remove-io /SYS/MB/PCIE3 primary
------------------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------
primary# ldm remove-io /SYS/MB/PCIE5 primary
------------------------------------------------------------------------------
Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------

Save the new configuration to the service processor (SP).
The following command saves the configuration in a file called dio:
```
# ldm add-config dio
```
Reboot the system to reflect the removal of the PCIe endpoint devices.
```
# reboot -- -r
```

Log in to the primary domain and verify that the PCIe endpoint devices are no longer assigned to the domain.

# ldm list-io
NAME                                      TYPE   BUS      DOMAIN   STATUS   
----                                      ----   ---      ------   ------   
niu_0                                     NIU    niu_0    primary           
niu_1                                     NIU    niu_1    primary           
pci_0                                     BUS    pci_0    primary           
pci_1                                     BUS    pci_1    primary  IOV      
/SYS/MB/PCIE0                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE2                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE4                             PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE6                             PCIE   pci_0    primary  EMP      
/SYS/MB/PCIE8                             PCIE   pci_0    primary  EMP      
/SYS/MB/SASHBA                            PCIE   pci_0    primary  OCC      
/SYS/MB/NET0                              PCIE   pci_0    primary  OCC      
/SYS/MB/PCIE1                             PCIE   pci_1             OCC      
/SYS/MB/PCIE3                             PCIE   pci_1             OCC      
/SYS/MB/PCIE5                             PCIE   pci_1             OCC      
/SYS/MB/PCIE7                             PCIE   pci_1    primary  EMP      
/SYS/MB/PCIE9                             PCIE   pci_1    primary  EMP      
/SYS/MB/NET2                              PCIE   pci_1    primary  OCC      
/SYS/MB/NET0/IOVNET.PF0                   PF     pci_0    primary           
/SYS/MB/NET0/IOVNET.PF1                   PF     pci_0    primary           
/SYS/MB/NET2/IOVNET.PF0                   PF     pci_1    primary           
/SYS/MB/NET2/IOVNET.PF1                   PF     pci_1    primary

Note - The ldm list-io -l output might show SUNW,assigned-device for the PCIe endpoint devices that were removed. Actual information is no longer available from the primary domain, but the domain to which the device is assigned has this information.

Assign a PCIe endpoint device to a domain.
1. Add the PCIE3 device to the ldg1 domain.
```
# ldm add-io /SYS/MB/PCIE3 ldg1
```
2. Bind and start the ldg1 domain.
```
# ldm bind ldg1
# ldm start ldg1
LDom ldg1 started
```

Log in to the ldg1 domain and verify that the device is available for use.

Verify that the network device is available and then configure the network device for use in the domain.

Oracle Solaris 10. Run the following command:

# dladm show-dev
nxge0           link: unknown   speed: 0     Mbps       duplex: unknown
nxge1           link: unknown   speed: 0     Mbps       duplex: unknown
nxge2           link: unknown   speed: 0     Mbps       duplex: unknown
nxge3           link: unknown   speed: 0     Mbps       duplex: unknown

Oracle Solaris 11. Run the following command:

# dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net0              Ethernet             unknown    0      unknown   nxge0
net1              Ethernet             unknown    0      unknown   nxge1
net2              Ethernet             unknown    0      unknown   nxge2
net3              Ethernet             unknown    0      unknown   nxge3

Skip Navigation Links
Exit Print View
	Oracle VM Server for SPARC 3.0 Administration Guide Oracle VM Server for SPARC