| Skip Navigation Links | |
| Exit Print View | |
|
Sun Network QDR InfiniBand Gateway Switch Product Notes for Firmware Version 2.0 |
| Skip Navigation Links | |
| Exit Print View | |
|
|
Sun Network QDR InfiniBand Gateway Switch Product Notes for Firmware Version 2.0 |
1. Sun InfiniBand QDR Gateway Switch Product Notes
General Information and Issues
Software Information and Issues
Main Board, Management Controller, and Chassis Serial Numbers
High Availability in Partitions
Firmware Update Considerations
Symbol Errors on Internal Ports
Commands Available to Restricted Linux Shells
Hardware Information and Issues
Documentation Information and Issues
smpartition Command Guidelines
VNIC Configuration on Linux Hosts in Gateway Manual Mode
Corrections to Creating Virtual I/O Adapters (Oracle Solaris)
Configure IP Addresses on the Data Link (Oracle Solaris)
Features and Functionality Documented
Acquire the ConnectX-2 Firmware
Upgrading the Gateway Firmware
When partitioning the InfiniBand fabric with the smpartition command, consider these guidelines during the configuration session:
Valid P_Keys for partitioning are 0x1 through 0x7fff. However, both 0x1 and 0x7fff are predefined P_Keys and cannot be created or deleted with the smpartition command.
Default entries (ports) in the predefined partitions cannot be removed. However, when you add ports into the predefined partitions using the smpartition add command, the added ports take precedence over the default ALL_SWITCHES and ALL_CAS port groupings.
The SELF and ALL port groupings are not supported by the smpartition command. To specify all ports for a partition use the ALL_SWITCHES port grouping in conjuction with the ALL_CAS port grouping in the smpartition add command.
To remove the ipoib flag from a partition, use the smpartition modify -flag command without any additional arguments. The command removes the ipoib flag successfully, but erroneously issues this warning:
Missing port parameter
Could not modify partition.
If you use EoIB and VNIC functionality in a nondefault partition, you must specify the appropriate BridgeX chips' InfiniBand port GUIDs to the partition.
When you know the VNIC's Ethernet connector, this table provides the respective BridgeX chip's portname.
|
When you know the Portname, you can look in the output of the showgwports command for the respective PortGUID. Using the table and the showgwports command together, when you know the Ethernet connectors used for your VNICs, you can find the respective BridgeX chip's InfiniBand port GUIDs to add to the nondefault partition.
If you are creating VNICs in Gateway Manual Mode, you must follow these conditions.
VNIC names appear frequently in the documentation. For example, the VNIC-specific configuration file contains the DEVICE parameter, which format is described as:
DEVICE=name
where name is suggested to be ethXX, and XX is a user-specified sequence number. VNIC names of eth0, eth1, ... to eth7 have been problematic with Linux kernels.
A solution is to use this format for the VNIC name:
ethgateway_instance_VNIC_instance
where:
gateway_instance is the gateway instance number as returned by the showgwconfig command.
VNIC_instance is a sequential VNIC ID number assigned to the VNIC. You can view the VNIC ID numbers with the showvnics command.
An example VNIC name might be eth03_03.
Using this new format, the filename for the VNIC-specific configuration file becomes ifcfg-ethgateway_instance_VNIC_instance. For the previous example, the configuration filename is ifcfg-eth03_03.
For consistency, the VNIC-specific configuration file must contain these two parameters:
DEVICE=name HWADDR=mac
where:
name is ethgateway_instance_VNIC_instance as described previously.
mac is the MAC assigned to the VNIC.
In Gateway Administration, there are several errors in the procedures for creating VIOAs in Oracle Solaris. Those procedures are corrected here.
Note - If the VIOA is to be assigned to a VLAN, you cannot create the VIOA from the gateway. Instead, use the procedure in Gateway Administration, creating Oracle Solaris VNICs.
For this procedure, the Oracle Solaris 11 host name is solaris01.
# dladm show-ib LINK HCAGUID PORTGUID PORT STATE PKEYS ibp0 21280001A0A590 21280001A0A591 1 up FFFF ibp1 21280001A0A590 21280001A0A592 2 up FFFF #
In this example, the HCA GUID is 21280001A0A590, and the port GUIDs are 21280001A0A591 and 21280001A0A592.
FabMan@gateway_name->ibhosts Ca : 0x0021280001A0A590 ports 2 "solaris01" Ca : 0x00212856cd22c040 ports 2 "SUN IB QDR GW switch mnm34-97 Bridge 1" Ca : 0x0002c903000891aa ports 2 "mnm34-54 HCA-1" Ca : 0x00212800013ece9e ports 2 "mnm34-55 HCA-1" Ca : 0x0003ba000100e370 ports 2 "mnm34-60 HCA-1" . . . FabMan@gateway_name->
The port GUIDs are the HCA GUID +1 (port 1) and HCA GUID +2 (port 2), respectively.
If not, check the cabling between the gateway and the Oracle Solaris 11 host.
FabMan@gateway_name->showvnics ID STATE FLG IOA_GUID NODE IID MAC VLN PKEY GW --- -------- --- ----------------------- ---------- ---- ----------------- --- ---- -------- 4 UP N 00:03:BA:00:01:00:E3:71 mnm34-60 0000 02:02:02:02:02:03 NO ffff 0A-ETH-1 5 UP N 00:03:BA:00:01:00:E3:71 mnm34-60 0002 02:02:02:02:02:04 NO ffff 0A-ETH-1 3 DISABLED N 00:03:BA:00:01:00:E3:71 mnm34-60 0000 02:02:02:02:02:02 NO ffff 0A-ETH-1 6 UP N 00:03:BA:00:01:00:E3:72 mnm34-60 0000 02:02:02:02:02:05 NO ffff 0A-ETH-1 FabMan@gateway_name->
Note - Until the IP interface is created on the data link (using the ipadm command) at the Oracle Solaris host, the STATE of the VNIC is WAIT-IOA.
STATE – The state of the VIOA.
IOA_GUID – The port GUID of the VIOA. The GUID might be listed several times.
NODE – The host name of the node hosting the VIOA.
MAC – The MAC address assigned to the VIOA. There might be several MAC addresses.
PKEY – The partition key assigned.
GW – The physical connector cabled to the node or host.
If no port GUID of the Oracle Solaris 11 host is listed, or there are no MAC addresses assigned to the port GUID, go to Step 10.
Otherwise, go to Configure IP Addresses on the Data Link (Oracle Solaris).
Note - Each MAC address must be unique and not 00:00:00:00:00:00.
Note - Only even numbers are supported for the most significant byte of the MAC address (unicast).
FabMan@gateway_name->createvnic connector -guid guid -mac mac_address -pkey default
where:
connector is the name of the connector (0A-ETH-1 to 0A-ETH-4 and 1A-ETH-1 to 1A-ETH-4).
guid is the global unique identifier of the target port on the host associated with the VIOA.
mac_address is the MAC address to be assigned to the VIOA.
For example, to assign the MAC address of a0:a5:91:95:30:9a to the VIOA for port GUID 21280001A0A591 through connector 0A-ETH-2:
FabMan@gateway_name->createvnic 0A-ETH-2 -guid 0021280001A0A591 -mac a0:a5:91:95:30:9a -pkey default VNIC created FabMan@gateway_name->
The VIOA is created and an Ethernet over InfiniBand data link is bound to the VIOA.
FabMan@gateway_name->showvnics ID STATE FLG IOA_GUID NODE IID MAC VLN PKEY GW --- -------- --- ----------------------- ---------- ---- ----------------- --- ---- -------- 7 UP N 00:21:28:00:01:A0:A5:91 solaris01 0000 a0:a5:91:95:30:9a NO ffff 0A-ETH-2 4 UP N 00:03:BA:00:01:00:E3:71 mnm34-60 0000 02:02:02:02:02:03 NO ffff 0A-ETH-1 5 UP N 00:03:BA:00:01:00:E3:71 mnm34-60 0002 02:02:02:02:02:04 NO ffff 0A-ETH-1 3 DISABLED N 00:03:BA:00:01:00:E3:71 mnm34-60 0000 02:02:02:02:02:02 NO ffff 0A-ETH-1 6 UP N 00:03:BA:00:01:00:E3:72 mnm34-60 0000 02:02:02:02:02:05 NO ffff 0A-ETH-1 FabMan@gateway_name->
Note - Until the IP interface is created on the data link (using the ipadm command) at the Oracle Solaris host, the STATE of the VNIC is WAIT-IOA.
See Configure IP Addresses on the Data Link (Oracle Solaris).
For example:
# dladm show-phys LINK MEDIA STATE SPEED DUPLEX DEVICE vnet0 Ethernet up 0 unknown vnet0 ibp0 Infiniband up 32000 unknown ibp0 ibp1 Infiniband down 2000 unknown ibp1 net5 Ethernet up 10000 full eoib0 net4 Ethernet up 10000 full eoib1 net6 Ethernet unknown 10000 full eoib2 #
The dladm show-phys command displays all data links corresponding to all hardware devices.
Similarly, the dladm show-link command displays the class, MTU, and state for all data links. For example:
# dladm show-link LINK CLASS MTU STATE OVER vnet0 phys 1500 up -- ibp0 phys 65520 up -- ibp1 phys 65520 down -- net5 phys 1500 up -- net4 phys 1500 up -- net6 phys 1500 unknown -- #
# ipadm create-ip link
where link is the name of the data link. For example:
# ipadm create-ip net6 #
Note - When you create the IP interface on the data link, the respective VNIC STATE (as seen with the showvnics command) goes from WAIT-IOA to UP.
# ipadm show-if IFNAME CLASS STATE ACTIVE OVER eoib2 ip down no -- #
Note - You must create the IP interface before an IP address is assigned to it.
# ipadm create-addr -T static -a IP_address/24 IP_interface/v4
where:
IP_address is the IP address to be assigned to the IP interface.
IP_interface is the interface corresponding to the data link and VIOA.
For example, to assign the IP address of 192.168.17.24 to IP interface eoib2:
# ipadm create-addr -T static -a 192.168.17.24/24 eoib2/v4 #
# ipadm show-addr IP_interface/v4
where IP_interface is the interface corresponding to the data link and VIOA. For example:
# ipadm show-addr eoib2/v4 ADDROBJ TYPE STATE ADDR eoib2/v4 static ok 192.168.17.24/24 #
In sections of Gateway Remote Management, authentication and privacy passwords are described for SNMP service user accounts and SNMP V3 protocol commands. For the user accounts and commands, the MD5 and SHA authentication passwords are 8 to 12 characters in length, and the DES privacy password must be exactly 8 characters long.
The features and functionality described in the gateway documentation has been updated to reflect the firmware starting with version 2.0. Upgrading your gateway firmware to the most current version helps increase gateway functionality. See Upgrading the Gateway Firmware.
Oracle's My Oracle Support page is displayed.
The dashboard page is displayed.
Note - If you do not have an account, you must register.
The Patches & Updates page is displayed.
The Patch Search window updates.
Possible products are suggested.
The Release Is field might autopropogate with the most current version.
For example, BridgeX OFED 1.5.1.
The Patch Search window expands with the search results.
For example, 12621910. The Patch Search window reformats.
The File Download window opens.
For example, p12621910_151_Linux-x86-64.zip.
The file is downloaded and saved.
The BXOFED software is in the BXOFED-1.5.1-version_for Oracle.tgz file. There are also readme, release notes, installation guide and user manual files in the filename.zip file.
For your host to properly interface with the gateway, the firmware of the ConnectX-2 chip in the HCA must be updated to version 2.7.000 or higher.
Oracle's My Oracle Support page is displayed.
The dashboard page is displayed.
Note - If you do not have an account, you must register.
The Patches & Updates page is displayed.
The Patch Search window updates.
For example, Sun Dual Port 4x QDR InfiniBand (IB) HCA PCIe ExpressModule.
For example, FW25408 v2.7.8130.
For example, Oracle Solaris on x86-64 (64-bit).
The Patch Search window expands with the search results.
For example, 12610332. The Patch Search window reformats.
The File Download window opens.
For example, p12610332__Solaris86-64.zip.
The file is downloaded and saved.
The ConnectX-2 firmware is in the fw-ConnectX2-rel-2_7_version.bin file. For example, fw-ConnectX2-rel-2_7_8130-375-3697-01.bin.
In Gateway Remote Management, firmware version numbers are provided as x.y, x.y.z, and x.y.z-w. Currently, these numbers are 2.0, 2.0.8, and 2.0.8-1 respectively. The following two procedures describe how to acquire and upgrade the firmware through the Oracle ILOM CLI.
Oracle's My Oracle Support page is displayed.
The dashboard page is displayed.
Note - If you do not have an account, you must register.
The Patches and Updates page is displayed.
The Patch Search window updates.
The Patch Search window updates.
Where x.y.z is the version number of the firmware package to be acquired. For example, 2.0.8.
The Patch Search window expands with the search results.
For example, 16295011. The Patch Search window reformats.
The File Download window opens.
For example, p16295011_208_Generic.zip.
The file is downloaded and saved.
The firmware is in the SUN_DCS_gw_x.y.z.tar.gz file.
The README file contains the latest information about the firmware release.
$ gtar -zxvf SUN_DCS_gw_x.y.z.tar.gz
The extracted files are displayed.
Note - If you are going to downgrade the firmware to a version earlier than 2.0, you must remove user partitions and depopulate the Subnet Manager nodes list. Refer to Gateway Administration, removing partitions for firmware downgrade.
For example:
% ssh -l root gateway_name root@gateway_name's password: password #
where gateway_name is the host name of the management controller. Initially, the password is changeme.
# disablesm Stopping partitiond daemon. [ OK ] Stopping IB Subnet Manager.. [ OK ] #
# df -h /tmp Filesystem Size Used Avail Use% Mounted on tmpfs 250M 240K 249M 1% /tmp #
In this example, there are 249 MB available. If not enough space is available, you must delete files from the /tmp directory.
# df -h /config Filesystem Size Used Avail Use% Mounted on /dev/hda2 16M 3.6M 11M 25% /config #
In this example, there are 11 MB available. If not enough space is available, you must delete files from the /config directory.
# df -h /var/log Filesystem Size Used Avail Use% Mounted on /dev/hda3 16M 3.6M 11M 25% /var/log #
In this example, there are 11 MB available. If not enough space is available, you must delete files from the /var/log directory.
# free -m
total used free shared buffers cached
Mem: 498 104 393 0 12 47
-/+ buffers/cache: 45 453
Swap: 0 0 0
#In the -/+ buffers/cache: row of the free column, there should be at least 150 MB free memory. In this example, there are 453 MB available. If not enough memory is available, you must exit nonessential applications that are running.
# spsh Oracle(R) Integrated Lights Out Manager Version ILOM 3.0 r47111 Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved. ->
You are now in the Oracle ILOM shell.
You can use the exit command to return to the Linux shell.
-> load -source URI/pkgname
where:
URI is the uniform resource indicator for the host where the gateway firmware package is located. The FTP and HTTP protocols are supported.
pkgname is the name of the firmware package in the transfer directory.
For example, using the FTP protocol:
-> load -source ftp://root:changeme@123.45.67.99//tmp/sundcs_gw_repository_2.0.8_1.pkg Downloading firmware image. This will take a few minutes.
Note - If you are upgrading from firmware version 2.0.X, you can use the -force option to disable version number checking, and force the upgrade.
The firmware is downloaded. The upgrade begins. A caution is displayed, and you are asked to commit to the upgrade.
NOTE: Firmware upgrade will upgrade firmware on SUN DCS gw Kontron module,
I4 and BridgeX. Upgrade takes few minutes to complete.
ILOM will enter a special mode to load new firmware. No other tasks
should be performed in ILOM until the firmware upgrade is complete.
Are you sure you want to load the specified file (y/n)?The upgrade begins.
Setting up environment for firmware upgrade. This will take few minutes. Starting SUN DCS gw FW update ========================== Performing operation: I4 A ========================== I4 fw upgrade from 7.3.0(INI:1) to 7.4.1010(INI:1): Upgrade started... Upgrade completed. INFO: I4 fw upgrade from 7.3.0(INI:1) to 7.4.1010(INI:1) succeeded ========================== Performing operation: BX A ========================== BX fw upgrade from 8.3.3166(INI:4) to 8.4.2812(INI:6): Upgrade started... Upgrade completed. INFO: BX fw upgrade from 8.3.3166(INI:4) to 8.4.2812(INI:6) succeeded ========================== Performing operation: BX B ========================== BX fw upgrade from 8.3.3166(INI:4) to 8.4.2812(INI:6): Upgrade started... Upgrade completed. INFO: BX fw upgrade from 8.3.3166(INI:4) to 8.4.2812(INI:6) succeeded =========================== Summary of Firmware update =========================== I4 status : FW UPDATE - SUCCESS I4 update succeeded on : A I4 already up-to-date on : none I4 update failed on : none BX status : FW UPDATE - SUCCESS BX update succeeded on : A, B BX already up-to-date on : none BX update failed on : none ========================================= Performing operation: SUN DCS gw firmware update ========================================= SUN DCS gw Kontron module fw upgrade from 1.3.2-1 to 2.0.8-1: Please reboot the system to enable firmware update of Kontron module. The download of the Kontron firmware image happens during reboot. After system reboot, Kontron FW update progress can be monitored in browser using URL [http://GWsystem] OR at OS command line prompt by using command [telnet GWsystem 1234] where GWsystem is the hostname or IP address of SUN DCS GW. Firmware update is complete. ->
-> exit exit #
Refer to Gateway Administration, restarting the entire gateway.
Note - The restart process takes between 4 to 5 minutes to complete.
You can monitor the update progress through:
Web browser – http://gateway_name
CLI – telnet gateway_name 1234
where gateway_name is the host name or IP address of the management controller.
Note - The Oracle ILOM stack requires at least 2 minutes to become operational after a reboot.
The next time you log in to the gateway, this message is displayed:
FW upgrade completed successfully on Mon Oct 17 18:36:14 IST 2012. Please run the "fwverify" CLI command to verify the new image. This message will be cleared on next reboot.
% ssh -l ilom-admin gateway_name
ilom-admin@gateway_name's password: password
-> show /SP/Fabric_Mgmt
NOTE: show on Fabric_Mgmt will launch a restricted Linux shell.
User can execute switch diagnosis, SM Configuration and IB
monitoring commands in the shell. To view the list of commands,
use "help" at rsh prompt.
Use exit command at rsh prompt to revert back to
ILOM shell.
FabMan@gateway_name->enablesm
Starting IB Subnet Manager. [ OK ]
Starting partitiond daemon. [ OK ]
FabMan@gateway_name->FabMan@gateway_name->version SUN DCS gw version: 2.0.8-1 Build time: Feb 6 2013 09:47:52 FPGA version: 0x33 SP board info: Manufacturing Date: 2009.06.24 Serial Number: "NCD3R0853" Hardware Revision: 0x0006 Firmware Revision: 0x0103 BIOS version: NOW1R113 BIOS date: 07/20/2009 FabMan@gateway_name->
In the first line of the output is SUN DCS gw version x.y.z-w, where x.y.z-w is the version of the firmware upgraded (or downgraded). For example, 2.0.8-1.
FabMan@gateway_name->fwverify Checking all present packages: ........................................................................... OK Checking if any packages are missing: ........................................................................... OK Verifying installed files: ........................................................................... OK FabMan@gateway_name->
|