Sun Blade 6048 InfiniBand
QDR Switched Network Express Module User’s Guide | 
   |
| C H A P T E R 2 |
This chapter describes how to replace a Sun Blade 6048 InfiniBand QDR Switched Network Express Module (IB-QNEM) in a powered-on Sun Blade 6048 Series Chassis. This chapter also includes instructions to verify that the IB-QNEM has been installed correctly.
This chapter contains the following sections:
If an IB-QNEM fails or if you choose to change the I/O configuration, you must replace the IB-QNEM. You can replace an IB-QNEM in a powered-on Sun Blade 6048 Series Chassis using a hot-plug operation.
If you are removing but not replacing the IB-QNEM, you must install both NEM slot filler panels to meet FCC limits for electromagnetic interference (EMI) and to ensure proper airflow and cooling.
If you encounter a problem replacing the IB-QNEM, see Managing the IB-QNEM.
Note - If you are installing a IB-QNEM in a Sun Blade 6048 Series Chassis that has not been powered on, see the Sun Blade 6048 Series Installation Guide (820-2312). |
The IB-QNEMs are customer-replaceable units (CRUs).
Identify which IB-QNEM to replace.
If the amber Service Action Required indicator is lit on a specific IB-QNEM, there is a problem with that IB-QNEM. Otherwise, you can choose any IB-QNEM to replace if, for example, you want to change the I/O configuration.
Squeeze the two tabs on both ejector levers to unlatch the levers from the IB-QNEM rear panel.
Pull both ejector levers simultaneously out to the sides until the IB-QNEM disengages from the chassis midplane.
 | Caution - Be careful not to over rotate the ejector levers and damage the chassis. |
Once the IB-QNEM is disconnected from the midplane, the green OK led will turn off.
Pull both ejector levers away from the NEM slot to slide the IB-QNEM out until you can comfortably remove the assembly with both hands.
| Caution - Do not support the IB-QNEM by the two ejector levers. |
Inspect the midplane connectors in the chassis NEM slots for bent or damaged pins before installing the IB-QNEM.
Do not install the IB-QNEM if you find any bent or damaged pins.
Align the IB-QNEM with the top NEM slot and the chassis guidance system.
Ensure that the IB-QNEM aligns with the chassis guidance system. Failure to align the IB-QNEM correctly can result in damage to the IB-QNEM's connections to the chassis midplane.
Slide the IB-QNEM into the NEM guide rails until the ejector levers touch the guide rails.
Pull both ejector levers out to the sides and position them into their latch on the rails.
The levers can be closed slightly to ensure that they are aligned correctly.
With both hands on the rear panel, push the IB-QNEM firmly into the midplane connectors.
| Caution - Do not push the IB-QNEM in at an angle or slam it into the midplane connectors. Such actions might result in bent or damaged connector pins. |
Push both ejector levers simultaneously toward the center so that the IB-QNEM engages fully with the connectors on the midplane
The levers should eventually lock into their latch openings on the IB-QNEM rear panel.
| Caution - Locking one lever at a time might result in seating problems that can lead to symbol errors. |
The IB-QNEM will automatic power on once the IB-QNEM is fully seated in the chassis. The green OK LED will blink during power up and eventually remain lit once the power on completes successfully.
If the amber Service Action Required indicator is on, try re-installing the IB-QNEM. If the problem persists, initiate a service call.
Connect the InfiniBand cables to the IB-QNEM port connectors.
Ensure that the connectors are properly engaged. The connectors click when locked.
| Caution - Avoid putting unnecessary stress on the connection. Do not bend or twist the cable near the connectors, and avoid sharp cable bends of more than 90 degrees. |
Verify that the IB-QNEM is working properly.
See To Verify Hardware Installation.
If you have not installed the IB-QNEM in the chassis and connected it to an operational InfiniBand switch or HCA, do so before you attempt to verify the installation. The InfiniBand switch should automatically recognize InfiniBand servers when the servers are connected to the fabric.
Once you have physically installed the IB-QNEM and ensured that the cables are connected to the IB-QNEM and other IB devices (switches or HCA), ensure that an IB Subnet Manager is running on the connected InfiniBand fabric (network).
If the green Physical Link indicator is illuminated, you have successfully completed the hardware installation and you can proceed to verification through the ILOM interfaces (Step 2). The green Physical Link indicator lights to show that the port is enabled. That is, a physical link to a remote switch (or, possibly an HCA) has been established. There might be a short delay before the indicator lights.
If the port LEDs are not illuminated, one possible cause might be that the InfiniBand drivers are not installed. You cannot verify a complete installation on Linux until you install these drivers.
Examine hardware status through one of the ILOM (Integrated Lights Out Manager) interfaces.
Use one of the following procedures:
For a description of the possible states of the IB-QNEM LEDs, see To Verify Component Status Using the LEDs.
Log in to the ILOM web interface using the IP address of the CMM.
The initial page of the ILOM web interface appears, providing visual verification of successful hardware installation. Note the image of the installed IB-QNEM in the view of the back of the chassis.
Select the System Information tab and then select the Components tab.
The Component Management page appears.
Scroll down to /CH/NEM1 and select the NEM1 component name.
The ILOM page showing the IB-QNEM status details appears.
If you are physically near the IB-QNEM, you can examine its LEDs to verify that it has returned the expected feedback.
Find the NEM1 in your system. type:
> show /CH |
To verify that the NEM1 is installed, type:
If you are physically near the IB-QNEM, examine its LEDs to verify that it has returned the expected feedback.
Because the IB-QNEMs are shared resources, all Sun Blade server modules must respond favorably to the PCI hot-remove request. However, a blade might not relinquish the link to a IB-QNEM if, for instance, there are busy NFS mounted volumes, file transfers, and so on.
To determine the state of the NEM-to-blade connections, you can use the ILOM command-line interface, as described in the following procedure.
Log in to the ILOM CMM CLI interface. For example:.
# ssh root@cmm-ip-address The authenticity of host ’xx.xx.xx.xxx (xx.xx.xx.xxx)’ can’t be established. RSA key fingerprint is a2:71:bd:bc:ce:74:4c:24:50:5f:fb:c5:c3:01:e6:58. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added ’10.60.32.197’ (RSA) to the list of known hosts. Password: Sun(TM) Integrated Lights Out Manager Version 2.0.3.13 Copyright 2008 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Warning: password is set to factory default. -> |
Start the CLI by typing the command shown in the example.
Type the show command to display the IB-QNEM hardware and software information:
-> show
NEM1
Targets:
Properties:
fpga_version = 03
cpld_version = 08
cpld_id = 5C
full_serial_no = 0110SJC-0904
S/N = CA0037
P/N = 5413378
HW_rev = 01
FRUID_specno = 885-1445-01
ENG_change_tag =
mfg_time = Mon Jan 19 19:12:59 2009
cable_present =
0 1 2 3 4 5 6 7 8 9
1 1 1 1 1 1 1 1 1 1
ports_A =
External 4x IB ports on switch A:
---------------------------------
A0 A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 A11 A12 A13 A14
Up Up Up Up Up Up Up Up Up Up Up Up Up Up Up
ports belonging to server blades on switch A:
---------------------------------------------
BL0#0 BL0#1 BL1#0 BL1#1 BL2#0 BL2#1 BL3#0 BL3#1 BL4#0 BL4#1 BL5#0 BL5#1
down down down down down down down down Up Up Up down
ports on switch A connected to the peer switch:
-----------------------------------------------
#28 #29 #30 #31 #32 #33 #34 #35 #36
Up Up Up Up Up Up Up Up Up
Logical p# Phys p# State
---------- ------- -----
1 ( 1) Up
2 ( 2) Up
3 ( 3) Up
4 ( 4) Up
5 ( 5) Up
6 ( 6) Up
7 ( 7) Up
8 ( 8) Up
9 ( 9) Up
10 (10) Up
11 (11) Up
12 (12) Up
13 (18) Up
14 (17) Up
15 (16) Up
16 (19) down
17 (20) down
18 (21) down
19 (22) down
20 (23) down
21 (24) down
22 (25) down
23 (26) down
24 (27) Up
25 (28) Up
26 (29) Up
27 (30) down
28 (15) Up
29 (14) Up
30 (13) Up
31 (31) Up
32 (32) Up
33 (33) Up
34 (34) Up
35 (35) Up
36 (36) Up
ports_B =
External 4x IB ports on switch B:
---------------------------------
B0 B1 B2 B3 B4 B5 B6 B7 B8 B9 B10 B11 B12 B13 B14
Up Up Up Up Up Up Up Up Up Up Up Up Up Up Up
ports belonging to server blades on switch B:
---------------------------------------------
BL6#0 BL6#1 BL7#0 BL7#1 BL8#0 BL8#1 BL9#0 BL9#1 BL10#0 BL10#1 BL11#0 BL11#1
down down down down down down down down down down down down
ports on switch B connected to the peer switch:
-----------------------------------------------
#28 #29 #30 #31 #32 #33 #34 #35 #36
Up Up Up Up Up Up Up Up Up
Logical p# Phys p# State
---------- ------- -----
1 ( 4) Up
2 ( 5) Up
3 ( 6) Up
4 ( 7) Up
5 ( 8) Up
6 ( 9) Up
7 (10) Up
8 (11) Up
9 (12) Up
10 (18) Up
11 (17) Up
12 (16) Up
13 (15) Up
14 (14) Up
15 (13) Up
16 (25) down
17 (26) down
18 (27) down
19 (28) down
20 (29) down
21 (30) down
22 (36) down
23 (35) down
24 (34) down
25 (33) down
26 (32) down
27 (31) down
28 ( 3) Up
29 ( 2) Up
30 ( 1) Up
31 (19) Up
32 (20) Up
33 (21) Up
34 (22) Up
35 (23) Up
36 (24) Up
temperature = 34 degrees
leds =
Locate : Off
OK : On
Service: Off
R2R : Off
ipass0 = 2612BER-0904XB0006 10m, 84 Circuit iPass Stacked Active Optical Cable
ipass1 = 2612BER-0905XA0011 10m, 84 Circuit iPass Stacked Active Optical Cable
ipass2 = 2612BER-0905XB0003 10m, 84 Circuit iPass Stacked Active Optical Cable
ipass3 = 2612BER-0905XB0004 10m, 84 Circuit iPass Stacked Active Optical Cable
ipass4 = 2612BER-0904XB0025 10m, 84 Circuit iPass Stacked Active Optical Cable
ipass5 = 2612BER-0904XA0006 10m, 84 Circuit iPass Stacked Active Optical Cable
ipass6 = 2612BER-0905XB0011 10m, 84 Circuit iPass Stacked Active Optical Cable
ipass7 = 2612BER-0905XA0003 10m, 84 Circuit iPass Stacked Active Optical Cable
ipass8 = 2612BER-0905XA0004 10m, 84 Circuit iPass Stacked Active Optical Cable
ipass9 = 2612BER-0904XA0025 10m, 84 Circuit iPass Stacked Active Optical Cable
Commands:
show
set
reset
resetswitch
version
help
exit
->
|
|
Sun Blade 6048 InfiniBand
QDR Switched Network Express Module User’s Guide | 820-6705-10 |
|
Copyright © 2009, Sun Microsystems, Inc. All rights reserved.
To Remove the IB-QNEM in
a Powered-On Chassis