Making PCIe Hardware Changes

Language:

The following steps help you avoid misconfiguring the PCIe endpoint assignments. For platform-specific information about installing and removing specific hardware, see the documentation for your platform.

No action is required if you are installing a PCIe card into an empty slot. This PCIe card is automatically owned by the domain that owns the PCIe bus.

To assign the new PCIe card to an I/O domain, use the ldm remove-io command to first remove the card from the root domain. Then, use the ldm add-io command to assign the card to an I/O domain.
No action is required if a PCIe card is removed from the system and assigned to the root domain.
To remove a PCIe card that is assigned to an I/O domain, first remove the device from the I/O domain. Then, add the device to the root domain before you physically remove the device from the system.
To replace a PCIe card that is assigned to an I/O domain, verify that the new card is supported by the DIO feature.

If so, no action is required to automatically assign the new card to the current I/O domain.

If not, first remove that PCIe card from the I/O domain by using the ldm remove-io command. Next, use the ldm add-io command to reassign that PCIe card to the root domain. Then, physically replace the PCIe card you assigned to the root domain with a different PCIe card. These steps enable you to avoid a configuration that is unsupported by the DIO feature.

Minimizing Guest Domain Outages When Removing a PCIe Card

While you remove or replace a PCIe card from a system that runs the Oracle VM Server for SPARC software, the domains that depend on this hardware are unavailable. To minimize such guest domain outages, you must prepare your system to use the hotplug capabilities to physically remove the card.

How to Minimize Guest Domain Outages When Removing a PCIe Card

This procedure enables you to avoid an outage to a guest domain that does not have direct I/O or SR-IOV device assigned to it and that has multiple paths configured. Note that this procedure requires two reboots of the primary domain.

Note - This procedure does not apply when the PCIe card is on a root complex owned by a non-primary root domain. Instead, see How to Replace PCIe Direct I/O Cards Assigned to an Oracle VM Server for SPARC Guest Domain (Doc ID 1684273.1) (https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=226878266536565&id=1684273.1&_adf.ctrl-state=bo9fbmr1n_49).

Stop the guest domain that has the PCIe slot assigned to it.
```
primary# ldm stop domain-name
```

Remove the PCIe slot from the guest domain.

primary# ldm remove-io PCIe-slot domain-name

Stop the guest domains that have PCIe slots and SR-IOV virtual functions assigned to them.
```
primary# ldm stop domain-name
```
Note - You do not need to stop guest domains that have PCIe buses assigned to them because they might be providing alternate paths to network and disk devices to the guest domains.
Initiate a delayed reconfiguration on the primary domain so that you can assign this slot to it.
```
primary# ldm start-reconf primary
```

Add the PCIe slot to the primary domain.

primary# ldm add-io PCIe-slot domain-name

Reboot the primary domain.
```
primary# shutdown -i6 -g0 -y
```
Use the hotplug commands to replace the PCIe card.
For information about Oracle Solaris OS hotplug capabilities, see Chapter 2, Dynamically Configuring Devices in Managing Devices in Oracle Solaris 11.3.
After the card is replaced, perform the following steps if you must reassign this same PCIe slot to the guest domain:
1. Initiate a delayed reconfiguration on the primary domain.
```
primary# ldm start-reconf primary
```
2. Remove the PCIe slot from the primary domain.
```
primary# ldm remove-io PCIe-slot domain-name
```
3. Reboot the primary domain to cause the removal of the PCIe slot to take effect.
```
primary# shutdown -i6 -g0 -y
```
4. Reassign the PCIe slot to the guest domain.
```
primary# ldm add-io PCIe-slot domain-name
```
5. Start the guest domains to which you want to assign PCIe slots and SR-IOV virtual functions.
```
primary# ldm start domain-name
```