Sun Blade 6000 Virtualized Multi-Fabric 10GbE Network Express Module Product Notes

These product notes contain important and late-breaking information about Oracle’s Sun Bladetrademark 6000 Virtualized Multi-Fabric 10GbE Network Express Module (Virtualized NEM).

The following topics are covered in this document:


Supported Operating Systems

The following operating systems are supported for the Sun Blade 6000 Virtualized Multi-Fabric 10GbE NEM.


Firmware and Software Updates

This section covers the following topics:

Software Version 2.2

The following updates are included in software version 2.2:

You can download SW 2.2 as follows:

1. Navigate to the following site:

http://www.oracle.com/goto/blades

2. Click on Drivers and Firmware in the Download box on the right side of the page.

The Sun Blade Systems - Firmware page is displayed.

3. Click on Network Express Modules.

The Sun Blade Systems Netwok Express Modules page is displayed.

4. Navigate to “Sun Blade 6000 Virtualized Multi-Fabric 10GbE Network Express Module Software 2.2”.

5. Click the download button, and log in as instructed, to access the firmware download.

Tools and Drivers CD

Tools and Drivers CD updates are available at:

http://www.sun.com/servers/blades/downloads.jsp

Refer to the Sun Blade Virtualized Mult-Fabric 10GbE Network Express Module User’s Guide for instructions on instlling the Virtualized NEM drivers.

The following sections contain information about Tools and Drivers CD versions:

Tools and Drivers CD 2.1

The Tools and Drivers CD version 2.1 (SW 2.1) contains the following updates:

Tools and Drivers CD 2.0

The Tools and Drivers CD version 2.0 (SW 2.0) contains the following updates:

ILOM version 2.0.3.12 build17 r44665 is also available with SW 2.0.

For issues fixed in SW 2.0, see Issues Fixed in SW 2.0.

Tools and Drivers CD 1.0.1

The Tools and Drivers CD version 1.0.1 (SW 1.0.1) contains the following updates:


Blade, FEM, and REM Compatibility

The following table lists the server blades, Fabric Express Modules (FEMs) and RAID expansion modules (REMs) that are supported with the Virtualized NEM. server blades that do not have FEMs or REMs listed have onboard network controllers or SAS expanders.


Sun Blade

FEM

REM

X6620

n/a

n/a

X6240

X4263A

X4620A or X4607A

X6250

X4681A

X4620A

X6270

X4263A

X4620A or X4607A

X6270 M2

X4263A

SG-SAS6-R-REM-Z

X6275-GbE

n/a

n/a

X6440

X4263A

X4620A or X4607A

X6450

X4681A

X4620A or X4607A

T6330

n/a

n/a

T6320

X4835A

X4607A or SG-SAS6-REM-Z (SAS-2)

T6340

X4835A

X4607A or SG-SAS6-REM-Z (SAS-2)



Known Issues

The following are known issues for the Sun Bladetrademark 6000 Virtualized Multi-Fabric 10GbE Network Express Module (Virtualized NEM).

Hardware and Networking Issues

The following are known hardware issues and networking issues for the Virtualized NEM:

Main 12V Power Sequencer Update (CR 6915085)

On rare occasions, the Virtualized NEM might shutdown unexpectedly.

To prevent this occurrence, you need to update the Virtualized NEM main 12V power sequencer firmware.

Before You Begin

Contact Oracle Service to gain access to the sunservice account for the CMM.



Note - System damage can occur. Perform the firmware update on an offline NEM and do not attempt this procedure without first contacting Oracle Service.


This section contains the following procedures:


procedure icon  To Update the Main Power Firmware

1. Download the following file from the Virtualized NEM SW 2.2 download site: NEMHydraPwrUpd10.zip.

See Software Version 2.2 for information on how to locate the download site.

2. Unzip the NEMHydraPwrUpd10.zip file.

You will need the following files for this procedure:



Note - Make sure that you have execute permissions on these files. If you do not have execute permissions, use the chmod +x command on the files before copying them to the CMM.


3. Copy the files listed in Step 2 into the CMM’s /coredump directory.

You will need information from Oracle Service to complete this step. If you have not already contacted Oracle Service, do so now.

4. Log into the CMM ILOM CLI.

5. Navigate to the CMM /coredump directory:

cd /coredump

6. Update the NEM in slot 0, with the following command:

./update_nem0_mainpwr.sh

7. If necessary, update the NEM in slot 1:

./update_nem1_mainpwr.sh



Note - Each command may take more than two minutes to complete.


8. Power cycle the Virtualized NEM.

9. Verify that the correct firmware version is installed with the following command:

./nem_pwr_ver.sh

The following output displays if the firmware has been correctly updated:

04

03

11

10. Exit the CMM.


procedure icon  To Revert to the Original Main Power Firmware

1. Download the following file from the Virtualized NEM SW 2.2 download site: NEMHydraPwrUpd10.zip.

See Software Version 2.2 for information on how to locate the download site.

2. Unzip the NEMHydraPwrUpd10.zip file.

You will need the following files for this procedure:



Note - Make sure that you have execute permissions on these files. If you do not have execute permissions, use the chmod +x command on the files before copying them to the CMM.


3. Copy the files listed in Step 2 into the CMM’s /coredump directory.

You will need information from Oracle Service to complete this step. If you have not already contacted Oracle Service, do so now.

4. Log into the CMM ILOM CLI.

5. Navigate to the CMM /coredump directory:

cd /coredump

6. Restore the NEM in slot 0, with the following command:

./restore_nem0_mainpwr.sh

7. If necessary, restore the NEM in slot 1:

./restore_nem1_mainpwr.sh



Note - Each command may take more than two minutes to complete.


8. Power cycle the Virtualized NEM.

9. Verify that the correct firmware version is installed with the following command:

./nem_pwr_ver.sh

The following output displays if the firmware has been correctly updated:

00

00

00

10. Exit the CMM.

 

Server Modules With SAS-2 REMs Might Hang When Inserted Into a Chassis With SAS-1 Devices

If a server module with a SAS-2 RAID Expansion Module (REM) is inserted into a Sun Blade 6000 modular system chassis that has SAS-1 NEMs and a Sun Blade 6000 disk module, the server blade’s SAS-2 REM might hang.

Workaround

To prevent this problem, you need to upgrade the firmware of your SAS-1 components (SAS-1 NEMs and disk modules) to a firmware version that supports SAS-1/SAS-2 coexistence. This upgrade must be done before you insert the SAS-2 server module into the chassis.

At a minimum, all SAS expanders for SAS-1 NEMs and Sun Blade 6000 disk modules must be upgraded to firmware revision 5.04.03 to allow SAS-1/SAS-2 device coexistence in the Sun Blade 6000 modular system chassis. SAS expander firmware version 5.04.03 is available in the Virtualized NEM software release 2.2.

See the SAS-1/SAS-2 Compatibility Upgrade Guide (821-1800) for details on obtaining the firmware and performing the upgrade.

Virtualized NEM Support for Sun Blade T6320 and T6340

The Sun Blade T6320 and T6340 server modules require the Sun Blade T63X0 PCIe Pass-Through Fabric Expansion Module (FEM) in order to use the 10 GbE functionality of the Virtualized NEM.

Refer to the Sun Blade T63x0 PCI Pass-Through Fabric Expansion Module User’s Guide for information on the release of this FEM and additional software and firmware requirements for these server modules.

Packet Drop Might Occur With 100 Meter or Longer Cables (6796609)

On-board Ethernet devices might see an excessive rate of packet drop using cables 100 meters or longer when link speed is negotiated at 1GB.

Workarounds

IPMP Configuration Might Cause Interface Failover (6775751)

When configured with IPMP, hxge interfaces might failover to the alternative interface without any real faults occurring. This is primarily due to IPMP’s path probes failing. Path probes may fail when either the system or the given network interface is over utilized.

Workarounds

Link Aggregation Is Not Supported on the Virtualized NEM (6749109)

Link aggregation is not supported on the virtualized NEM. If multiple server modules are performing independent link aggregations, this will lead to unpredictable results. This is due to limitations in the link aggregation standard from the IEEE.802.1AX, specifically the Link Aggregation Control Protocol (LACP).

Errors Displayed During PXE Boot (6806643)

When using the Virtualized NEM for PXE booting over the 10 GbE interface, rbr_empty and UDP checksum errors might be displayed during the boot process (See FIGURE 1). This should not affect the PXE installation.

FIGURE 1 Errors Displayed During PXE Boot


Errors displayed during PXE boot

Only One Virtualized NEM Device Appears in Boot List (6752737)

When you press the F8 key or view the BIOS Setup Utility boot priority list, only one virtualized NEM device will appear.

To boot from the NEM that is not listed, do the following.

1. Choose the NEM listed in the boot device list.

FIGURE 2 and FIGURE 3 show the boot device list that appears when you either press F8 or use the boot list in the BIOS Setup Utility.

FIGURE 2 F8 Boot List


Graphic showing F8 boot list

FIGURE 3 BIOS Setup Utility Boot List


Graphic showing BIOS Setup Utility Boot List

2. Select the Virtualized NEM shown in the list.

The system will start to boot from the first Virtualized NEM device with matching device and vendor ids.

3. Press the ESC key to move to the next virtualized NEM device and boot from it.

ILOM Issues

The following are known ILOM issues for the Virtualized NEM:

ILOM Proxy show /NEM0/state Command Displays an Incorrect Power Value (6779318)

If you use the ILOM proxy show /NEM0/state command to view NEM power state, the value property will display Power Off, even though the NEM is powered on.

Workaround

Ignore the properties value given by the proxy show /NEM0/state command.

Do one of the following to read the correct power state:

a. Log on to the Virtualized NEM SP from the chassis CMM:

-> start /CH/NEMx/SP/cli

b. Use the following command to show the NEM properties:

-> show /NEM/OK

A Benign Error Message Appears When Resetting the NEM Using the ILOM Proxy (6757019)

After using the reset command to reset the Virtualized NEM using the ILOM proxy, you might receive a warning message, as shown in the following example output:


WARNING:  This action will interrupt service to NEM1
This command should only be run if you are sure the device is not in use
Press ’y’ to continue.  Press any other key to cancel this action
reset of NEM1 in progress...
board reset command sent.  It may take a minute before it completes
reset of NEM1 failed with 255

The failure message can be safely ignored.

Setting the LED Using the ILOM Proxy Does Not Work (6760490)

If you try to set the NEM LED behavior using the ILOM proxy, the command will appear to work in the CLI, but it will not change the LED behavior.

Workaround

You can use the standard CMM ILOM or NEM SP to set the LED behavior.

Refer to the ILOM Supplement section of the Sun Blade 6000 Virtualized Multi-Fabric 10 GbE Network Express Module User’s Guide.

Operating System Issues

The following are known operating system issues for the Virtualized NEM:

System Panic When Changing MTU to Jumbo Frames Size With ESX 4.0 (6880558)

When the hxge driver is installed on a sever running ESX 4.0, a system panic can occur after changing the maximum transmisstion unit (MTU) value to a jumbo frame size. This issue occurs due to a bug in vmkernel.

This issue is VMware PR 518560.

Workaround

Change the MTU value to a jumbo frames size when there is no current network activity on the server.

SLES 11 hxge Driver Fails to Load (6909431)

When attempting to load the hxge driver on SLES11, it fails to load with the following error:


FATAL: module ’/lib/modules/2.6.27.19-5-default/kernel/drivers/net/hxge.ko’ is unsupported
Use --allow-unsupported or set allow_unsupported_modules to 1 in /etc/modprobe.d/unsupported-modules

Workaround

Load the driver with the --allow-unsupported flag as follows:


modprobe hxge --allow-unsupported

Blade Server Running Solaris OS Might Hang During Heavy Stress Test (6871113)

A blade server running Solaris OS and recovering from RBR empty might fail the test when under heavy stress.

The Windows Server 2008 Device Drivers Appear To Not Be Digitally Signed in Device Manager (6815500)

The message "Not digitally signed" will display in Windows Server 2008 device manager for each instance of the Sun Blade 6000 Virtualized Multi-Fabric 10GbE Network Express Module. This is a known issue in Windows Server 2008.

You can ignore this message. The Sun Blade 6000 Virtualized Multi-Fabric 10Gbe Network Express Module network drivers are signed by Microsoft.

 

Windows NTttcp Application Might Stop When Network Cable is Disconnected (6777024)

When using the NTttcp application in Windows Server 2003 or 2008 OS, and the network cable is disconnected and reconnected within a short period of time, the NTttcp might stop transmitting traffic and exit.

Workaround

Restart the NTttcp network application.

Network Device Naming Instances Might Change on a Solaris OS Server Module (6769339)

When using the Solaris OS, customers might experience changes to their network device naming instances (hxge0 and hxge1) on a server module when moving from one slot to an alternative slot.

Workaround

Use the following workaround to identify the NEM network device:

1. Note the MAC address label assignment on the Sun Blade 6000 Virtualized 10Gb NEM located on the inner left hand side of the NEM module.

2. Compare this MAC Address with the network device instances on the server module using ifconfig -a, and note down the first 5 octets that match.

For example, if the NEM module MAC address label has 00:14:4F:00:00:00, the server module slot zero will be assigned MAC addresses 00:14:4F:00:00:01 through 00:14:4F:00:00:04.

This is the normal behavior in the way Solaris performs its enumeration of network device instances.

Driver Fails to Start When Jumbo Frames are Enabled in a Sun Blade X6440 System Running Windows (6776566)

After Jumbo Frames is enabled or disabled for the Virtualized NEM (Sun Blade 6000 10Gbe Networking Controller) on a Sun Blade X6440 server module running Windows, the network connection for Virtualized NEM could become disabled.

FIGURE 4 shows a yellow error icon next to the disabled Sun Blade 6000 10GbE Networking Controller in the Device Manager display.

FIGURE 4 Disabled Sun Blade 6000 10GbE Networking Controller


Graphic showing disabled networking controller

Workaround

Reboot the server module to recover the network connection. After the reboot, the network connection will be up and running with the desired Jumbo Frame setting.

FIGURE 5 shows the yellow error icon removed from the Sun Blade 6000 10GbE Networking Controller, indicating that the network connection is enabled in the Device Manager display.

FIGURE 5 Enabled Sun Blade 6000 10GbE Networking Controller


Graphic showing enabled networking controller

ethtool Shows Incorrect NEM Speed With RHEL 5.2 64-bit (6788773)

When running ethtool on a RHEL 5.2 64-bit system to determine the speed of the Virtualized NEM, ethtool will report an unknown speed. This is because RHEL 5.2 64-bit has an older version of ethtool.

 

A Benign Warning Message Appears When Unloading the RHEL 4.7 hxge Driver (6762810)

When unloading the RHEL 4.7 version of the Virtualized NEM Linux driver, the following type of warning message will appear:


Oct 22 23:16:26 ban25c15bl9 kernel: Trying to free nonexistent resource <bddf0000-bddfffff>

This is a benign message and can be ignored.

Operating Systems Cannot be Installed Over the 10GbE Interface (6759944)

The hxge driver is not bundled with any of the server module operating systems and cannot be installed over a 10GbE interface.



Note - Support for operating system installation over the 10 Gigabit Ethernet interface is planned for future OS releases.


Workaround

Create a customized boot archive with hxge driver support.

The following document shows how to customize the boot archive for the Solaris operating system.

http://www.sun.com/blueprints/0806/819-7546.html

For Linux or Windows systems, refer to the documentation from your OS distribution to create a customized boot archive with the hxge driver.


Fixed Issues

The following issues have been fixed. Unless otherwise noted, the software updates mentioned are for the Virtualized NEM.

Issues Fixed in SW 2.1

Issues With Re-enabling Windows 2008 10GbE Driver (6834080)

After disabling and re-enabling one or both of the 10GbE network devices, one device will sometimes disappear from device manager. The device might also flag problem code 10 (cannot start).

Also, if driver is re-loaded manually, then disabled and re-enable d, the same failure might occur.

Workaround

Reinitialize the bus.

Issues Fixed in SW 2.0

The software updates included in SW 2.0 fix the following issues:

When you update to ILOM version 2.0.3.12 build17 r44665 (SW 2.0), the Fcode firmware will be updated to version 1.4, which fixes CR 6861451.

Sun Blade X6220 Server Module Cannot be Booted Using the F12 Key (6775898)

This issue is fixed in Sun Blade X6220 server SW 3.0 (BIOS 114).

When using a Sun Blade X6220 server module to perform a network boot with Sun Blade 6000 Virtualized 10GbE Multi-Fabric NEM, the network boot will not work if you use the F12 key.

Workaround

Select F8 during BIOS Post and select PXE:SUN 10GNemV 1.8 GPL etherboot.

Windows ipv6 NTttcp failure issue (6776512)

This issue is fixed with the Windows Server 2008 driver update in SW 1.0.1.

Lost Network Link Might Not Be Recognized in Windows Server 2003 (6805038)

This issue is fixed with the Windows Server 2003 driver update in SW 1.1.

On a Windows Server 2003 OS server module with Sun Blade 6000 Virtualized 10GbE NEM in non-failover mode, if the network link is lost (for example, the network cable is unplugged) there might not be an indication in the Windows “Control Panel->Network Connections" window.

Workaround

single-step bullet  Observe the link LED light above the SFP+ ports on Oracle’s Sun Blade 6000 Virtualized 10 GbE NEM.

If the LED is not illuminated, then the network link is down.

 

 


Copyright

If this document has been previously published, replace “XXXX” with the first year of publication. Otherwise, use only 2010 as the copyright year. Remove this paragraph and text.

This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.

The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing.

If this is software or related software documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, the following notice is applicable:

U.S. GOVERNMENT RIGHTS. Programs, software, databases, and related documentation and technical data delivered to U.S. Government customers are "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, duplication, disclosure, modification, and adaptation shall be subject to the restrictions and license terms set forth in the applicable Government contract, and, to the extent applicable by the terms of the Government contract, the additional rights set forth in FAR 52.227-19, Commercial Computer Software License (December 2007). Oracle America, Inc., 500 Oracle Parkway, Redwood City, CA 94065.

This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications which may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications.

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. UNIX is a registered trademark licensed through X/Open Company, Ltd.

This software or hardware and documentation may provide access to or information on content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services.


Copyright © 2009, 2010, Oracle et/ou ses affiliés. Tous droits réservés.

Ce logiciel et la documentation qui l’accompagne sont protégés par les lois sur la propriété intellectuelle. Ils sont concédés sous licence et soumis à des restrictions d’utilisation et de divulgation. Sauf disposition de votre contrat de licence ou de la loi, vous ne pouvez pas copier, reproduire, traduire, diffuser, modifier, breveter, transmettre, distribuer, exposer, exécuter, publier ou afficher le logiciel, même partiellement, sous quelque forme et par quelque procédé que ce soit. Par ailleurs, il est interdit de procéder à toute ingénierie inverse du logiciel, de le désassembler ou de le décompiler, excepté à des fins d’interopérabilité avec des logiciels tiers ou tel que prescrit par la loi.

Les informations fournies dans ce document sont susceptibles de modification sans préavis. Par ailleurs, Oracle Corporation ne garantit pas qu’elles soient exemptes d’erreurs et vous invite, le cas échéant, à lui en faire part par écrit.

Si ce logiciel, ou la documentation qui l’accompagne, est concédé sous licence au Gouvernement des Etats-Unis, ou à toute entité qui délivre la licence de ce logiciel ou l’utilise pour le compte du Gouvernement des Etats-Unis, la notice suivante s’applique :

U.S. GOVERNMENT RIGHTS. Programs, software, databases, and related documentation and technical data delivered to U.S. Government customers are "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, duplication, disclosure, modification, and adaptation shall be subject to the restrictions and license terms set forth in the applicable Government contract, and, to the extent applicable by the terms of the Government contract, the additional rights set forth in FAR 52.227-19, Commercial Computer Software License (December 2007). Oracle America, Inc., 500 Oracle Parkway, Redwood City, CA 94065.

Ce logiciel ou matériel a été développé pour un usage général dans le cadre d’applications de gestion des informations. Ce logiciel ou matériel n’est pas conçu ni n’est destiné à être utilisé dans des applications à risque, notamment dans des applications pouvant causer des dommages corporels. Si vous utilisez ce logiciel ou matériel dans le cadre d’applications dangereuses, il est de votre responsabilité de prendre toutes les mesures de secours, de sauvegarde, de redondance et autres mesures nécessaires à son utilisation dans des conditions optimales de sécurité. Oracle Corporation et ses affiliés déclinent toute responsabilité quant aux dommages causés par l’utilisation de ce logiciel ou matériel pour ce type d’applications.

Oracle et Java sont des marques déposées d’Oracle Corporation et/ou de ses affiliés.Tout autre nom mentionné peut correspondre à des marques appartenant à d’autres propriétaires qu’Oracle.

AMD, Opteron, le logo AMD et le logo AMD Opteron sont des marques ou des marques déposées d’Advanced Micro Devices. Intel et Intel Xeon sont des marques ou des marques déposées d’Intel Corporation. Toutes les marques SPARC sont utilisées sous licence et sont des marques ou des marques déposées de SPARC International, Inc. UNIX est une marque déposée concédée sous licence par X/Open Company, Ltd.

Ce logiciel ou matériel et la documentation qui l’accompagne peuvent fournir des informations ou des liens donnant accès à des contenus, des produits et des services émanant de tiers. Oracle Corporation et ses affiliés déclinent toute responsabilité ou garantie expresse quant aux contenus, produits ou services émanant de tiers. En aucun cas, Oracle Corporation et ses affiliés ne sauraient être tenus pour responsables des pertes subies, des coûts occasionnés ou des dommages causés par l’accès à des contenus, produits ou services tiers, ou à leur utilisation.