1. Exadata Server X10M Service Manual
  2. Servicing PCIe Cards

11 Servicing PCIe Cards

This section describes how to service PCIe cards and PCIe Risers.

PCIe cards and PCIe risers are replaceable components that require you to power off the server before servicing. Foot 1

For more information about replaceable components, see Illustrated Parts Breakdown and Replaceable Components. For complete software and cabling information about your card, refer to your PCIe card documentation. For a complete list of supported PCIe cards, see Product Description.

Caution:

These procedures require that you handle components that are sensitive to electrostatic discharge. This sensitivity can cause the components to fail. To avoid damage, ensure that you follow antistatic practices as described in Electrostatic Discharge Safety.

Caution:

Ensure that all power is removed from the server before removing or installing PCIe cards. You must disconnect all power cables from the system before performing these procedures.

PCIe Slot Locations

The server is designed to support both full height and low profile PCIe cards. Using PCIe risers, the full height PCI slot configuration supports up to 4 full height and 2 half height PCIe cards. The 9-slot PCIe half height configuration supports up to 9 half height PCIe cards. See Back Panel Components for PCIe slot numbering and supported PCIe configurations.

  • All card slots support Half Height, Half Length (HHHL) PCIe cards (Low profile short) at PCIe Gen4 or Gen5.
  • Four riser card slots support Full Height, Full Length (FHFL) PCIe cards at PCIe Gen5.
  • All PCIe card slots are 75W capable slots. All of the PCIe slots comply with the PCI Express 5.0 specification and can accommodate 75 Watt PCIe cards.
  • All PCIe card slots support persistent power.
  • All Add-in-Card (AIC) slots support Hot-Plug. Foot 2
  • PCIe card slots 1 through 4 are nonfunctional in single-processor systems.
  • The Full Height card chassis configuration uses two PCIe riser cards supporting two Full height 10.5 inch double width GPU cards at x16 PCIe Gen4.
  • Each riser also supports a PCIe Gen3 Pensando Ortano/Elba Server Adapter, DSC-200V, 2x200GbE, Dual QSFP56 Smart NIC card.
  • PCIe Gen5 card slots can be up to four x16 PCIe slots, or up to six x8 PCIe slots, depending upon the PCIe card slots 6 and 7 motherboard cabling option.
  • Full Height PCIe card risers support a 150W SmartNIC in slots 4 and 8, and a 400W GPU in slots 2 and 6.

Remove a Half Height PCIe Card

Before replacing or installing a PCIe card, refer to the card documentation for specific installation and cabling instructions.
  1. Prepare the server for service.
    1. Power off the server.
    2. Disconnect the power cords from the power supplies.
    3. Extend the server to the maintenance position.
    4. Attach an antistatic wrist strap.
    5. Remove the server top cover.
  2. Locate the PCIe card that you want to remove.

    PCIe slots 1 through 4 are nonfunctional in single-processor systems.

    See Back Panel Components.

  3. If necessary, record where the PCIe cards are installed.
  4. Unplug all data cables from the PCIe card [1].

    Note the cable connections so that it's easier to properly reconnect the cables.

    If it is difficult to disconnect the cables from the PCIe card when the card is inserted into the PCIe slot, consider performing the procedure in Step 5 before completing this step.

  5. Rotate the PCIe card locking mechanism in to an upright position [1], and then lift up on the PCIe card to disengage it from the motherboard connectors [2].
    Figure showing a Half Height PCIe card being removed from the server.
  6. Place the PCIe card on an antistatic mat.

    Caution:

    If you are not immediately inserting a replacement PCIe card into the empty slot, insert a PCIe filler panel in the slot to reduce the possibility of radiated electromagnetic interference (EMI). See Remove and Install Filler Panels.

Install a Half Height PCIe Card

If you are installing an x16 PCIe card in slot 6, you must install the motherboard PCIe Flyover cable in to slot 6 to enable support for a PCIe x16 electrical interface; otherwise, slot 6 supports only a PCIe x8 electrical interface. If you are installing a PCIe card in slot 7, you must install the motherboard PCIe Flyover cable in slot 7 to support a PCIe x8 electrical interface; otherwise, PCIe slot 7 is non-functional. See Configure PCIe Electrical Interfaces Using the PCIe Flyover Cable.
  1. Unpack the replacement PCIe card, and place it on an antistatic mat.
    Before replacing or installing a PCIe card, refer to the card documentation for specific installation and cabling instructions.
  2. Locate the proper PCIe slot for the card you are replacing.
    PCIe slots 1 through 4 are nonfunctional in single-processor systems. See Back Panel Components.
  3. If necessary, remove the PCIe filler panel from the slot.
    Save this filler panel in case you need to remove the PCIe card from the system.
  4. Insert the PCIe card into the correct slot [1], and rotate the PCIe locking mechanism downward to secure the PCIe card in place [2].

    You hear an audible click when the PCIe card is secured into the slot.

    If it is difficult to reconnect the cables to the PCIe card when the card is inserted into the PCIe slot, consider performing the procedure in Step 5 before completing this step.

    Figure showing a Half Height PCIe card being installed into the server.
  5. Reconnect the cables to the PCIe card that you unplugged during the removal procedure [2].
    To ensure proper SAS cable connections to the Broadcom 9500-16i, 12GB SAS PCIe, 16 Port, Internal card, see the SAS cabling reference table in Step 3 in Install SAS Storage Drive Cables.
  6. Return the server to operation.
    1. Install the top cover.
    2. Return the server to the normal rack position.
    3. Connect any required data cables to the PCIe card.

      (Optional) Route data cables through the Cable Management Arm (CMA).

    4. Reconnect the power cords to the power supplies and power on the server.
    5. Verify that the System OK LED is lit.
  7. Use Oracle ILOM to clear any server PCIe card faults.

    If a PCIe card fault message in Oracle ILOM is not cleared under Open Problems, you must manually clear the fault using Oracle ILOM. Refer to "Clear Faults for Undetected Replaced or Repaired Hardware Components" in the Oracle ILOM User's Guide for System Monitoring and Diagnostics at Oracle ILOM Documentation.

  8. To determine whether additional steps are required to complete the installation of the PCIe card, refer to the server product notes for the type of PCIe card you installed.

Remove a Full Height PCIe Card

This task also applies to half-height PCIe cards that are being removed from PCIe Risers.

Note:

Before replacing or installing a PCIe card, refer to the card documentation for specific installation and cabling instructions.
  1. Locate the PCIe card that you want to remove, and if necessary, record the PCIe slot where the PCIe card is installed.
    PCIe slots 1 through 4 are nonfunctional in single-processor systems. See Back Panel Components.
  2. Remove the PCIe Riser that contains the PCIe card you want to remove.
  3. Using a Torx (6 lobe) T10 screwdriver, remove the screw that secures the PCIe card bracket to the PCIe riser.
    Figure showing screws that secure the PCIe card bracket being removed.
  4. While holding the PCIe Riser in one hand, use your other hand to carefully pull the PCIe card connector out of the PCIe Riser slot.
    Figure showing a Full Height PCIe card being removed from the server.
  5. If required, unplug any data cables from the PCIe card.
    Note the cable connections so that it's easier to properly reconnect the cables.
  6. Place the PCIe card on an antistatic mat.

    Caution:

    If you are not immediately inserting a replacement PCIe card into the empty slot, insert a PCIe filler panel in the slot to reduce the possibility of radiated electromagnetic interference (EMI). See Remove and Install Filler Panels.

Install a Full Height PCIe Card

This task also applies to half-height PCIe cards that are being installed in PCIe Risers.

If you are installing a x16 PCIe card in Riser 6, you must install the motherboard PCIe Flyover cable in to slot 6 to enable support for a PCIe x16 electrical interface; otherwise, slot 6 supports only a PCIe x8 electrical interface. See Configure PCIe Electrical Interfaces Using the PCIe Flyover Cable.

  1. Unpack the replacement PCIe card, and place it on an antistatic mat.
  2. Locate the proper PCIe slot for the card you are replacing.
    PCIe slots 1 through 4 are nonfunctional in single-processor systems. See Back Panel Components.
  3. If necessary, remove the PCIe filler panel from the PCIe Riser slot.
    Save this filler panel in case you need to remove the PCIe card from the system.
  4. While holding the PCIe Riser in one hand, use your other hand to insert the PCIe card into the correct slot on the PCIe Riser.
    Figure showing a full height PCIe card being installed into the server.

    You hear an audible click when the PCIe card is secured into the slot.

  5. Using a Torx (6 lobe) T10 screwdriver, install and tighten the screw to 8 in-lb (0.9 Nm/9.2 kg-cm) that secures the PCIe card bracket to the PCIe Riser.
    Figure showing screws that secure the PCIe card bracket being tightened.
  6. Reconnect the cables to the PCIe card that you unplugged during the removal procedure.
    If it is difficult to reconnect the cables to the PCIe card when the card is secured into the PCIe slot, consider performing this procedure prior to inserting the PCIe card in to the PCIe Riser slot.
  7. Install the PCIe riser that contains the PCIe card.

Remove a PCIe Riser

This task applies only to the back panel Riser configuration.
  1. Prepare the server for service.
    1. Power off the server.
    2. Disconnect the power cords from the power supplies.
    3. Remove rear panel cables.
      See Full Height PCIe Configuration in Back Panel Components.
    4. Using a Torx (6 lobe) T25 screwdriver, loosen the rear panel Riser(s) attaching screw [2].
    5. Extend the server into the maintenance position.
    6. Attach an antistatic wrist strap to your wrist, and then to a metal area on the chassis.
    7. Remove the server top cover.
  2. Identify the proper back panel PCIe slot (Riser 2 or Riser 6) for the Riser you are replacing.
    See Full Height PCIe Configuration in Back Panel Components.
  3. Disconnect the data and power cables from the adjacent PCIe card in PCIe slot 1 or 5. Then remove the adjacent PCIe card.
  4. Disconnect the PCIe Riser signal cable from the side of the riser you are replacing [1].
  5. Using a Torx (6 lobe) T25 screwdriver, loosen one screw that secures the PCIe riser to the back panel [2].
  6. Using a Torx (6 lobe) T25 screwdriver, loosen the two screws that secure the PCIe riser to the motherboard [3].
  7. Lift the PCIe riser from the slot and place it on an antistatic mat [4].

    Figure showing a PCIe Riser being removed from the server.

  8. If required, remove the installed PCIe card from the Riser PCIe slot.
    Hold the Riser in one hand and use your other hand to carefully pull the PCIe card connector out of the PCIe riser slot.

    Leave the PCIe data and power cables attached to the PCIe card and place the PCIe card on an antistatic mat. You will need to install the PCIe card in the replacement PCIe riser.

    See also Remove a Full Height PCIe Card and Remove a Half Height PCIe Card.

Install a PCIe Riser

This task applies only to the back panel Riser configuration.
  1. Unpack the replacement PCIe Riser, and place it on an antistatic mat.
  2. Locate the proper PCIe slot (Riser 2 or Riser 6) for the Riser you are replacing.
  3. If necessary, remove the PCIe filler panel from the PCIe slot on the PCIe riser.
    Save this filler panel in case you need to remove the PCIe Riser from the system.
  4. While holding the PCIe riser, insert supported full height or half height PCIe cards into PCIe slots on the PCIe riser.
  5. Reconnect the cables to the PCIe card that you unplugged during the removal procedure.
    If it is difficult to reconnect the cables to the PCIe card when the card is inserted into the PCIe slot, consider performing this procedure prior to inserting the PCIe card in to the PCIe Riser slot.
  6. Using a Torx (6 lobe) T10 screwdriver, install and tighten the screw to 8 in-lb (0.9 Nm/9.2 kg-cm) that secures the PCIe card bracket to the PCIe riser.
  7. Lower the PCIe riser in to the proper PCIe slot and press down until the riser is firmly seated in the PCIe connector [1].

    You hear an audible click when the PCIe card is secured into the slot.


    Figure showing a PCIe Riser being installed in the server.

  8. Using a Torx (6 lobe) T25 screwdriver, tighten two captive screws to 8 in-lb (0.9 Nm/9.2 kg-cm) to secure the PCIe riser to the motherboard [2] and the rear panel of the server [3].
  9. Reconnect the PCIe Riser signal cable between the motherboard and PCIe riser [4].
  10. If required, reinstall the adjacent half height PCIe card that was removed from PCIe slot 1 or 5.
  11. Return the server to operation.
    1. Install the top cover.
    2. Return the server to the normal rack position.
    3. Connect any required data cables to the PCIe card.

      Route data cables through the Cable Management Arm.

    4. Reconnect the power cords to the power supplies and power on the server.
    5. Verify that the System OK LED is lit. Server Status Indicator LEDs
  12. Use Oracle ILOM to clear any server PCIe card faults.

    If a PCIe card fault message in Oracle ILOM is not cleared under Open Problems, you must manually clear the fault using Oracle ILOM. Refer to "Clear Faults for Undetected Replaced or Repaired Hardware Components" in the Oracle ILOM User's Guide for System Monitoring and Diagnostics at Oracle ILOM Documentation.

  13. To determine whether additional steps are required to complete the installation of the PCIe card, refer to the server product notes for the type of PCIe card you installed.

Configure PCIe Electrical Interfaces Using the PCIe Flyover Cable

Using the PCIe Flyover cable, server PCIe slots 6 and 7 can be configured to support either a PCIe x16 (slot 6) or PCIe x8 (slot 7) electrical interface. PCIe slot 6 supports a PCIe x8 electrical interface and PCIe slot 7 is non-functional when the PCIe Flyover PCIe cable is not installed in the server. Installing the cable in slot 6 configures the slot as a PCIe x16 electrical interface. Installing the cable in slot 7 configures the slot as a PCIe x8 electrical interface.

Note:

The PCIe Flyover cable is factory installed in either PCIe slot 6 or slot 7. You must switch the cable to the desired PCIe slot prior to installing a PCIe card or PCIe Riser 6.
  1. Determine where you want to install the PCIe card or Riser.

    See Rules For Configuring PCIe Slots 6 and 7.

    Rules For Configuring PCIe Slots 6 and 7

    • If you are installing an x16 PCIe card in slot 6, you must install the PCIe Flyover cable in to slot 6 to enable support for a PCIe x16 electrical interface; otherwise, slot 6 supports only a PCIe x8 electrical interface.

    • If you are installing a PCIe card in slot 7, you must install the PCIe Flyover cable in slot 7 to support a PCIe x8 electrical interface; otherwise, PCIe slot 7 is non-functional.

    • If you are installing Riser 6 and it contains a PCIe x16 card, you must install the PCIe Flyover cable in to slot 6 to enable a PCIe x16 electrical interface for the Riser; otherwise, slot 6 supports only a PCIe x8 electrical interface.

  2. Perform one of the following.

    • Connect the PCIe Flyover cable between the motherboard connector and PCIe slot 6.

    • Connect the PCIe Flyover cable between the motherboard connector and PCIe slot 7.

  3. Press and hold the button on the connector to disengage the locking mechanism, then gently pull the connector from the motherboard [1].
    Figure showing the PCIe Flyover cable being installed in PCIe slot 6.
    Callout Description

    1

    PCIe Flyover cable

    2

    PCIe slot 7 electrical interface connector

    3

    PCIe slot 6 electrical interface connector
  4. Install the connector in the desired slot and press down on the connector until it is fully seated in the slot [2].
  5. Continue by installing the PCIe card or Riser 6.

Remove the PCIe Flyover Cable

Before removing the PCIe Flyover cable, review the rules for configuring the PCIe electrical interfaces. See Configure PCIe Electrical Interfaces Using the PCIe Flyover Cable.
  1. Prepare the server for service.
    1. Power off the server.
    2. Disconnect the power cords from the power supplies.
    3. Extend the server to the maintenance position.
    4. Attach an antistatic wrist strap.
    5. Remove the server top cover.
  2. Locate the PCIe Flyover cable in the server.

    The PCIe Flyover cable is installed on the rear left side of the system, between the motherboard handle and behind PCIe slots 6 and 7. The PCIe Flyover cable is connected to either the PCIe slot 6 or slot 7 electrical interface connector. Note which PCIe electrical interface is being used.

  3. If necessary, remove PCI Riser 6 to ease access to the PCIe Flyover cable.
  4. Disconnect and remove the PCIe Flyover cable.

    Figure showing the PCIe Flyover cable being removed from the server.

    At each end of the cable, press and hold the metal clip to disengage the locking mechanism, then gently lift the PCIe Flyover cable from the electrical interface connector.

Install the PCIe Flyover Cable

Before installing the PCIe Flyover cable, review the rules for configuring the PCIe electrical interfaces. See Configure PCIe Electrical Interfaces Using the PCIe Flyover Cable.
  1. Unpack the replacement PCIe Flyover cable and place it on an antistatic mat.
  2. Connect the PCIe Flyover cable to the motherboard connector and to either the PCIe slot 6 or slot 7 PCIe electrical interface connector.

    For information on which PCIe slot to connect the PCIe Flyover cable, see Configure PCIe Electrical Interfaces Using the PCIe Flyover Cable.


    Figure showing the PCIe Flyover cable being installed in the server.

    Plug each end of the cable into its connector until you hear an audible click.

  3. If necessary, reinstall PCI Riser 6.
  4. Return the server to operation.
    1. Install the top cover.
    2. Return the server to the normal rack position.
    3. Reconnect the power cords to the power supplies and power on the server.
    4. Verify that the System OK LED is lit.

Footnote Legend

Footnote 1: Hot-Plug support is limited to Exadata storage node configurations only.
Footnote 2: Hot-Plug support is limited to Exadata storage node configurations only.