Sun Quad GbE UTP x8 PCIe ExpressModule Release Notes

This document contains material not available when the Sun Quad GbE UTP x8 PCIe ExpressModule User’s Guide was completed. Read this document before installing and configuring your card.

You can find the latest updates for the nxge driver software at this web site:

http://www.sun.com/download

Select Networking.

You can find the latest card documentation at this web site:

http://docs.sun.com/app/docs/prod/net.inter.crds


Known Issues With Sun SPARC and Solaris x86 Systems

This section describes the known issues and workarounds for the nxge driver on Sun SPARC and Solaris x86 systems at the time of this release.

Issue: nxge Driver for Specific Ethernet Cards Might Report Spurious Errors (CRs 6572623, 6573600, 6623776, and 6625536)

This issue is fixed in the following patch releases for the SUNWnxge 11.10 package version.

Installation Details:

1. Determine the current package version.

Obtain the SUNWnxge package version as follows:


# pkginfo -l SUNWnxge | grep VERSION

If the unbundled SUNWnxge package is installed, the version is 1.0. For example:


VERSION=1.0,REV=2007.01.12.10.0

This version was delivered as a standalone package, not bundled with the Solaris OS.

If the bundled SUNWnxge package is installed, the version is 11.10. For example:


VERSION:  11.10.0,REV=2007.07.08.17.44

This package is bundled to the Solaris 10 8/07 release.

2. Upgrade SUNWnxge version 1.0.

If version 1.0 (unbundled SUNWnxge package) is installed, the SUNWnxge package and the Solaris kernel should be upgraded to Solaris 10 8/07 level.



Note - This step is not needed if version 11.0 is already installed.


To upgrade the SUNWnxge package, follow the workaround instructions in Issue: Upgrading Solaris OS to Solaris 10 8/07 to Fix a Panic Problem Seen With Solaris 10 11/06 (CRs 6613003 and 6591438).

After this step, install the SUNWnxge version 11.0 and the following kernel update patches:

3. Install relief patches.

Install the following Kernel Update Patches

Issue: Upgrading Solaris OS to Solaris 10 8/07 to Fix a Panic Problem Seen With Solaris 10 11/06
(CRs 6613003 and 6591438)

On a host with Solaris 10 11/06 OS installed, if you install the latest recommended patch cluster, which includes KU 120011-14, the system panics if you install an nxge Ethernet adapter (with nxge package version 1.0 and driver version 1.x).

This issue occurs when the add_drv command in the postinstall script of the SUNWnxge package is run (while adding the package), the system enters a panic cycle. The system does not panic if KU 120011-14 is removed.

The latest nxge driver packages are now bundled with Solaris 10 8/07 and later OS releases. This issue does not occur on systems with the Solaris 10 8/07 or later OS.

Note the following nxge patch dependencies:

Workaround: Perform One of the Following

Perform one of the following steps (Step 1 or Step 2).

1. Upgrade to Solaris 10 8/07.

Before upgrading, check if the unbundled nxge driver packages are installed. For example:


% pkginfo -l SUNWnxge | grep 1.0,REV=2007.01.12.10.0

If the packages are installed, and you have modified the nxge.conf file (on a Solaris 10 11/06 system), make a backup of the nxge.conf file before upgrading or reinstalling Solaris 10 8/07 or later releases. For Solaris 10 11/06 only (unbundled), the location of the nxge.conf file is /kernel/drv/nxge.conf for both SPARC and x86 platforms.

For Solaris 10 8/07 (nxge bundled), the locations of the nxge.conf file are as follows:



Note - Do not install the older unbundled nxge driver packages (downloaded for Solaris 10 11/06) if the system is upgraded or freshly installed with the Solaris 10 8/07 or later OSs.


2. Or, on Solaris 10 11/06 systems, remove the unbundled nxge driver packages, and install the latest bundled packages from the Solaris 10 8/07 media.

a. Backup the nxge.conf file if you have modified it (see Step 1).

b. Remove the old nxge driver package version:


# pkgrm SUNWnxge

c. Install KU patches for Solaris 10 8/07 level:

d. Add the new bundled version of the nxge driver package from the Solaris 10 8/07 media.


# pkgadd SUNWnxge

Issue: Installing the Correct Versions of nxge Data Integrity Patches for Your System

SunAlert 103076 was created to fix data integrity issues with the nxge driver.



Note - There are two separate sets and versions of patches to fix the data integrity issues. Installing the wrong patch results in an actual failure rather than a standard "does not apply" exit.


Workaround: Install the Following Patches

For Solaris 10 11/06, install Data Integrity Patch 125670-04 or later.



caution icon Caution - Do not install Patches 125670/125671 or later if you have KU 120011/120012-14 or later installed.


For Solaris 10 8/07, install Data Integrity Patch 127741-01 or later.



caution icon Caution - Do not install Patches 127741/127742-01 or later if you have KUs 118833/118855-36 or later or 125100/125101-10 or later installed.



Known Issues With Sun SPARC Systems

This section describes the known issues and workarounds for the nxge driver on Sun SPARC systems at the time of this release.

Issue: MSI or SoftRing Settings Might Need Site-Specific Adjustment

The default MSI or SoftRing settings might need to be adjusted for your site.

Workaround: Refer to the Tuning Web Site

Refer to the following tuning web site for information on tunable parameters and how to adjust tunable MSI and SoftRing parameters:

http://www.solarisinternals.com/wiki/index.php/Networks

Issue: nxge.conf Missing Hex Code (0x) Prefix for “class-opt-ipv4-tcp = fe0” Value

When a Sun Quad GbE UTP x8 PCI Express Card or ExpressModule is installed in a Sun SPARC Enterprise M9000 server, and the "class-opt-ipv4-tcp = fe0;" value is uncommented (set) in the /platform/sun4u/kernel/drv/nxge.conf configuration file, the following messages are output during system boot:


WARNING: Unexpected token ’fe0’ on line 112 of /platform/sun4u/kernel/drv/nxge.conf
WARNING: Unexpected end of line on line 112 of /platform/sun4u/kernel/drv/nxge.conf

Hex values are expressed with a prefix of 0x. The correct format should be as follows:


“class-opt-ipv4-tcp = 0xfe0;”


Known Issues With Solaris x86 Systems

This section describes the known issues and workarounds for the nxge driver on Solaris x86 systems at the time of this release.

Issue: Interrupts Might be Missed

Interrupt migration feature in the Solaris x86 kernel may lead to interrupts being missed.

Workaround: To Avoid Missing Interrupts

You must add the following lines to the /etc/system file to avoid missing interrupts that can lead to an interface hang.


set ddi_msix_alloc_limit=1 
set pcplusmp:apic_enable_dynamic_migration=0 
set pcplusmp:apic_intr_policy=1

Reboot the system after adding these lines:


# boot -r

Issue: Detaching Driver Can Kill Other Devices

Detaching or unloading a driver can kill other devices and lead to a system panic.

Workaround: Do Not Unplumb a Configured Interface

Avoid unplumbing an interface after it has been configured. To stop using a configured interface, issue the following command:


# ifconfig nxgeX down 

where X is the number of the interface you want to stop using.

Issue: Older BIOS Versions Might Hang

Older BIOS versions show intermittent system hang issues.

Workaround: Upgrade the System BIOS

Refer to:

http://www.sun.com/download/index.jsp?cat= Hardware%20Drivers&tab=3&subcat= x64%20Servers%20%26%20Workstations

Issue: SoftRing Mode is Disabled by Default

Default SoftRing mode is disabled under the Solaris x86 Operating System.

Workaround: Enable SoftRing Mode

Add the following line in /etc/system file and reboot.


set ip:ip_squeue_fanout=1 
set ip_squeue_soft_ring=1

Then add the following to a startup script, or use the ndd utility before plumbing the interface:


# ndd -set /dev/ip ip_soft_rings_cnt 8

Issue: Operating System Cannot be Installed Over the 10 Gigabit Ethernet Interface

The nxge driver is not bundled with Solaris x86 Operating System and cannot be installed over a 10 Gigabit Ethernet interface.



Note - Support for operating system installation over the 10 Gigabit Ethernet interface is planned for a future Solaris release.


Workaround: Create a Custom Boot Archive with nxge Driver Support

Create a customized boot archive with nxge driver support. The following document shows how to customize the boot archive for the Solaris Operating System.

http://www.sun.com/blueprints/0806/819-7546.html

For Linux systems, refer to the documentation from your Linux distribution to create a customized boot archive with the nxge driver.

Issue: Limited Space for Option ROMs

POST may report "Not Enough Space to copy PCI Option ROM" when Sun 10Gigabit cards are plugged in the system due to limited space available for the Option ROMs.

Workaround: Disable Slots or Interfaces


procedure icon  To Disable Slots and Interfaces from Option ROM Probing

If there is not enough space available for the Option ROMs, you will see the following error message during system boot:


1. Press F2 to get to the BIOS Setup screen:


2. Press the right arrow ket to get to the PCIPnP menu and disable the OPROM scanning for the slot that has the Sun Quad Gigabit Ethernet UTP PCI Express Card.


This example shows PCIe slot 0.

3. Save your changes.



Known Issues With Linux Systems

This section describes the known issues and workarounds for the nxge driver on Linux systems at the time of this release.

Issue: Card Fails on RHEL4U3 LargeSMP Kernel With Over 8 CPUs

The Sun x8 Express 10 Gigabit Ethernet Fiber XFP Low Profile adapter will fail to function on RHEL4U3 LargeSMP Kernel with more than 8 CPUs.

Workaround: Reduce CPUs to 8 or Upgrade to RHEL4U4

This is an issue with interrupt distribution. You can use the following two workarounds:

Add the following line at the end of the kernel parameters in the menu.1st file:


 maxcpus=8

The following is an examle entry in a /boot/grub/menu.lst file:


title Red Hat Enterprise Linux Server (2.6.18-53.el5)
  root (hd0,0)
  kernel /vmlinuz-2.6.18-53.el5 ro root=/dev/VolGroup00/LogVol00 maxcpus=8 console=ttyS0,9600n8 rhgb quiet
  initrd /initrd-2.6.18-53.el5.img 

Issue: System Panics When irqbalance Is Enabled

System panics observed when irqbalance is enabled.

Workaround: Disable irqbalance

You must disable irqbalance to avoid system crashes. Use the following procedure:

1. Boot the system and configure the Sun x8 Express 10 Gigabit Ethernet interfaces as usual.

2. Disable irqbalance by doing any of the following methods:

Issue: ethtool Utility

The ethtool utility, as of version 5, does not provide full support of 10 Gigabit Ethernet network interfaces. Consequently, the ethtool utility does not support bonding on Sun x8 Express 10 Gigabit Ethernet interfaces.

Workaround: See Support URL

Wait for the fix to be applied to the ethtool source. For more information, see the 1170813 Patch ID for correctly reporting 10Gb/s support at:

http://sourceforge.net/projects/gkernel/