4.2 Servicing PCIe Cards (CRU)

This section describes how to remove and install PCIe cards.

For the location of the PCIe slots, see 1.3 Server Back Panel Features.

4.2.1 PCIe Card Configuration Rules

The server supports up to ten low-profile PCIe 2.0 cards. Refer to the Sun Fire X4470 Server Product Notes (821-0704) for details about supported PCIe cards, along with card quantity and slot restrictions.

The following PCIe card slots are available on the server:

• Slots 0 and 9: x4 electrical interface

• Slots 1, 2, 4, 6, 7, and 8: x8 electrical interface

• Slots 3 and 5: x8 or x16 electrical interface (x16 connector)

  ---

  Note - Slots 3 and 5 operate as x16 interfaces only when an x16 capable card is installed and the adjacent slot (4 or 6) is unpopulated.

  ---

To determine the slot in which to install a PCIe card, follow these guidelines:

• First consider any cooling considerations that require a card to be installed in a certain slot. For information about cooling considerations for PCIe cards, refer to the Sun Fire X4470 Server Product Notes (821-0704).

• If your configuration includes an internal SAS/SATA Host Bus Adapter (HBA), check to see if it has an integrated battery pack. Install a battery-backed internal HBA in slot 4, for best cooling; otherwise, put the simpler, battery-less SAS/SATA HBA in slot 2. Slot 2 is convenient for cabling, and using Slots 3 or 4 limits the availability of x16 electrical interfaces.

• For high bandwidth cards, install the cards so that the load on the server is balanced between the server’s two I/O hubs (IOH), each of which connects to five of the available PCIe slots. (IOH0 connects to slots 0–4 and IOH1 to slots 5–9). Avoid slots 9 and 0 for high-bandwidth cards since those are the x4 slots. Balance the load as follows:

  • IOH0 (which connects to slots 0–4), might already be using some bandwidth for the ICH10 southbridge devices, so install the first high-bandwidth card in one of slots 5–8.

  • Install the next high-bandwidth card in one of slots 1–4. (If an internal HBA is already installed in slot 0–4, skip this step and install the next card in another of slots 5–8.)

  • Continue to alternate the high-bandwidth cards between slots 1–4 and 5–8.

• Install lower-speed cards (for example, 1-Gbit/s Ethernet and 8-Gbit/s Fibre Channel adapters) in any remaining slots, since these cards perform well in any location, including the x4 slots, 0 and 9.

4.2.2 PCIe Cards With Bootable Devices

If the server has several PCIe cards installed that provide bootable devices, disable the Option ROM on the PCIe slots not used for booting so that resources are available for the slots used for booting. By default, the BIOS enables Option ROMs for PCIe slots 2 and 4, and the four Gigabit Ethernet ports. Refer to Configure Option ROM Settings for information about disabling Option ROM.

4.2.3 Avoiding PCI Resource Exhaustion Errors

With serveral Sun x4 PCI Express Quad Gigabit Ethernet Low Profile Adapter PCIe cards (4446A-Z-N or X4446A-Z-N) installed in server slots 0 through 4, or even one of these cards installed in slots 5 through 9, BIOS might log a "PCI resource exhaustion" event. This error occurs because BIOS allocates 8 Kbytes of I/O space for each of the Quad Gigabit Ethernet cards (and only 4 Kbytes of I/O space for a typical card).

The following is a common form of the error that might be logged:

6491 Tue Dec 7 14:19:57 2010 IPMI Log minor ID = a5a9 : 12/07/2010 : 14:19:57 : System Firmware Error : sensor number = 0x00 : PCI resource exhaustion: Bus 147 Device 0 Func 0 6490 Tue Dec 7 14:19:57 2010 IPMI Log minor ID = a5a8 : 12/07/2010 : 14:19:57 : System Firmware Error : sensor number = 0x00 : PCI resource exhaustion: Bus 147 Device 0 Func 1

To eliminate the "PCI resource exhaustion" event, change the BIOS settings to disable I/O allocation for any slot where the Quad Gigabit Ethernet card is installed. Refer to 7.8 Performing Common BIOS Procedures for instructions on how to access the BIOS Setup Utility in order to disable I/O allocation from the PCIPnP menu. The exception to this workaround is that if PXE network booting will use a Quad Gigabit Ethernet card, you should leave I/O allocation enabled for that slot.Since BIOS does not allocate I/O space for empty slots, maximizing the number of empty PCIe slots in the server might also help to avoid the "PCI resource exhaustion" event.

4.2.4 Removing and Installing PCIe Cards and PCIe Card Filler Panels

Use the following procedures to remove and install PCIe cards and PCIe card filler panels.

Remove a PCIe Card Filler Panel

---

Caution - This procedure requires that you handle components that are sensitive to static discharge. This sensitivity can cause the component to fail. To avoid damage, ensure that you follow antistatic practices as described in 2.3 Performing Electrostatic Discharge and Static Prevention Measures.

---

---

Caution - Ensure that all power is removed from the server before removing or installing expansion cards. You must disconnect the power cables before performing this procedure.

---

1. Prepare for servicing:
   1. Attach an antistatic wrist strap.

      See 2.3 Performing Electrostatic Discharge and Static Prevention Measures.

   2. Power off the server and disconnect all power cords from the server power supplies.

      See 2.6 Powering Off the Server.

   3. Extend the server to the maintenance position.

      See 2.4 Positioning the Server for Maintenance.

   4. Remove the top cover.

      See 2.7 Removing the Server Top Cover

2. Locate the PCIe card filler panel that you want to remove.

   See 1.3 Server Back Panel Features for information about PCIe slots and their locations.

3. Disengage the PCIe slot crossbar from its locked position and rotate the crossbar into an upright position (Disengaging the Crossbar).
4. Remove the PCIe card filler panel from the PCIe slot.

   ---

   Dikkat - Whenever you remove a PCIe card filler panel, you should replace it with another filler panel or a PCIe card; otherwise, the server might overheat due to improper airflow.

   ---

Remove a PCIe Card

---

Caution - This procedure requires that you handle components that are sensitive to static discharge. This sensitivity can cause the component to fail. To avoid damage, ensure that you follow antistatic practices as described in 2.3 Performing Electrostatic Discharge and Static Prevention Measures.

---

---

Caution - Ensure that all power is removed from the server before removing or installing expansion cards. You must disconnect the power cables before performing this procedure.

---

1. Prepare for servicing:
   1. Attach an antistatic wrist strap.

      See 2.3 Performing Electrostatic Discharge and Static Prevention Measures.

   2. Power off the server and disconnect all power cords from the server power supplies.

      See 2.6 Powering Off the Server.

   3. Extend the server to the maintenance position.

      See 2.4 Positioning the Server for Maintenance.

   4. Remove the top cover.

      See 2.7 Removing the Server Top Cover

2. Locate the PCIe card that you want to remove.

   See 1.3 Server Back Panel Features for information about PCIe slots and their locations.

3. If necessary, make a note of where the PCIe cards are installed.
4. Unplug all data cables from the PCIe card.

   Note the location of all cables for reinstallation later.

5. Disengage the PCIe card slot crossbar from its locked position and rotate the crossbar into an upright position (Disengaging the Crossbar).
6. Carefully remove the PCIe card from the PCIe card slot.
   

   Figure 4-9 Disengaging the Crossbar

   

   ---

   Dikkat - Whenever you remove a PCIe card, you should replace it with another PCIe card or a filler panel; otherwise, the server might overheat due to improper airflow.

   ---

Install a PCIe Card

Refer to Installing a PCIe Card when completing the following procedure.

---

Caution - This procedure requires that you handle components that are sensitive to static discharge. This sensitivity can cause the component to fail. To avoid damage, ensure that you follow antistatic practices as described in 2.3 Performing Electrostatic Discharge and Static Prevention Measures.

---

---

Caution - Ensure that all power is removed from the server before removing or installing PCIe cards. You must disconnect the power cables before performing this procedure.

---

1. Unpack the PCIe card and place it on an antistatic mat.
2. Ensure that the server is powered off and all power cords are disconnected from the server power supplies.

   See 2.6 Powering Off the Server.

3. Install the PCIe card into the PCIe card slot.
   

   Figure 4-10 Installing a PCIe Card

   
4. Return the PCIe card slot crossbar to its closed and locked position.
5. Return the server to operation:
   1. Install the top cover.

      See 6.1 Replacing the Server Top Cover.

   2. Return the server to the normal rack position.

      See 6.2 Returning the Server to the Normal Rack Position.

   3. Reconnect all power cords to the server power supplies and power on the server.

      See 6.3 Powering On the Server.

6. If the PCIe card being installed is replacing a faulty PCIe card, manually clear the PCIe card fault using Oracle Integrated Lights Out Manager (ILOM).

   For instructions on clearing server faults, refer to the Oracle Integrated Lights Out Manager (ILOM) 3.0 Supplement for Sun Fire X4470 Server (821-0702).

7. Refer to the documentation shipped with the PCIe card for information about configuring the PCIe card, including installing required operating systems.

   To create or recover RAID configurations, refer to the LSI MegaRAID SAS Software User’s Guide, which is available at the following location: http://www.lsi.com/support/sun

Install a PCIe Card Filler Panel

Refer to Installing a PCIe Card when completing the following procedure.

---

Caution - This procedure requires that you handle components that are sensitive to static discharge. This sensitivity can cause the component to fail. To avoid damage, ensure that you follow antistatic practices as described in 2.3 Performing Electrostatic Discharge and Static Prevention Measures.

---

---

Caution - Ensure that all power is removed from the server before removing or installing PCIe card filler panels. You must disconnect the power cables before performing this procedure.

---

1. Ensure that the server is powered off and all power cords are disconnected from the server power supplies.

   See 2.6 Powering Off the Server.

2. Install the PCIe card filler panel into the PCIe card slot.
3. Return the PCIe card slot crossbar to its closed and locked position.
4. Return the server to operation:
   1. Install the top cover.

      See 6.1 Replacing the Server Top Cover.

   2. Return the server to the normal rack position.

      See 6.2 Returning the Server to the Normal Rack Position.

   3. Reconnect all power cords to the server power supplies and power on the server.

      See 6.3 Powering On the Server.