PCIe Device Root Complex Failover Behavior

Each PCIe and EMS slot can be associated to either a primary root complex or a secondary root complex, depending on the configuration of the DCU. When the DCU contains four healthy CMUs, all of the PCIe and EMS slots will be associated to their primary root complexes.

When a CMP or CMU fails, or when a CMU is removed, the primary root complexes on that CMP or CMU will no longer be available. Depending on how you set the Oracle ILOM ioreconfigure property on the PDomain host, the PCIe or EMS slots will either be rerouted to their secondary root complexes or the slots will no longer be available.

The ioreconfigure property signals under what conditions the PCIe I/O fabric paths from the root complexes to the slots will be created. After the server is powered on for the first time, the ioreconfigure property will be set to true. During the initial power on, the PCIe I/O fabric paths will be created for all of the PCIe and EMS slots in the server.

If you set the ioreconfigure property to false, the PCIe I/O fabric paths will not be recreated when a CMU or CMP fails or is removed. The PCIe and EMS slots routed to the primary root complexes on the missing CMU or CMP will no longer be available.

If you set the ioreconfigure property to true, the PCIe I/O fabric paths will be recreated when you next restart the PDomain. If the primary root complexes for PCIe or EMS slots are no longer available and you have set the ioreconfigure property to true, the PCIe I/O fabrics for these slots will be rerouted to their secondary root complexes and the virtual switches will be merged to function from a single root port.

In the following illustration, CMU3 has either failed or has been removed from DCU0. Since the ioreconfigure property is set to false, one EMS slot and four PCIe slots are no longer accessible.

The following illustration displays what happens when you set the ioreconfigure property to true. The PCIe and EMS slots that were once routed to the primary root complexes on CMU3 have been rerouted to their secondary root complexes on CMU2.

For example, PCIe slot 9 has been rerouted from its primary root complex, pci_15, to its secondary root complex, pci_5. The OpenBoot PROM device path for PCIe slot 9 has changed from the primary path on CMU3/CMP1 (pci@6c0):

/pci@6c0/pci@1/pci@0/pci@8

To the secondary path on CMU2/CMU0 (pci@440):

/pci@440/pci@1/pci@0/pci@8

In the following illustration, DCU0 contains only CMU0 and CMU1. Since the primary and secondary root complexes on CMU2 and CMU3 are no longer available, you cannot access EMS slots 3 and 4 and PCIe slots 9 through 16.

When neither the primary nor secondary root complex of a PCIe or EMS slot is available, you cannot access the slot as there is no electrical path from a working root complex through the PCIe I/O switch fabric to the slot.

Related Information

• PCIe Communication and Paths

• Fully-Populated Default Configuration

• Half-Populated Configuration

• Understanding PCIe Slot Root Complex Names and Device Paths

• Manage I/O Path Reconfiguration Settings

• Identify the Root Complex of a Device