Sun Quad GbE UTP x8 PCIe ExpressModule Release Notes

This document contains material not available when the Sun Quad GbE UTP x8 PCIe ExpressModule User’s Guide was completed. Read this document before installing and configuring your card.

You can find the latest updates for the nxge driver software at this web site:

http://www.sun.com/download

Select Networking.

You can find the latest card documentation at this web site:

http://docs.sun.com/app/docs/prod/net.inter.crds

Known Issues With Sun SPARC and Solaris x86 Systems

This section describes the known issues and workarounds for the nxge driver on Sun SPARC and Solaris x86 systems at the time of this release.

Issue: `nxge` Driver for Specific Ethernet Cards Might Report Spurious Errors (CRs 6572623, 6573600, 6623776, and 6625536)

This issue is fixed in the following patch releases for the SUNWnxge 11.10 package version.

SPARC platforms: 127111-08 or higher

x86 platforms: 127112-08 or higher

Installation Details:

1. Determine the current package version.

Obtain the SUNWnxge package version as follows:

# pkginfo -l SUNWnxge | grep VERSION

If the unbundled SUNWnxge package is installed, the version is 1.0. For example:

VERSION=1.0,REV=2007.01.12.10.0

This version was delivered as a standalone package, not bundled with the Solaris OS.

If the bundled SUNWnxge package is installed, the version is 11.10. For example:

VERSION:  11.10.0,REV=2007.07.08.17.44

This package is bundled to the Solaris 10 8/07 release.

2. Upgrade SUNWnxge version 1.0.

If version 1.0 (unbundled SUNWnxge package) is installed, the SUNWnxge package and the Solaris kernel should be upgraded to Solaris 10 8/07 level.

Note - This step is not needed if version 11.0 is already installed.

To upgrade the SUNWnxge package, follow the workaround instructions in Issue: Upgrading Solaris OS to Solaris 10 8/07 to Fix a Panic Problem Seen With Solaris 10 11/06 (CRs 6613003 and 6591438).

After this step, install the SUNWnxge version 11.0 and the following kernel update patches:

SPARC platforms: 120011-14

x86 platforms: 120012-14

3. Install relief patches.

Install the following Kernel Update Patches

SPARC platforms: 127111-08 or higher

x86 platforms: 127112-08 or higher

Issue: Upgrading Solaris OS to Solaris 10 8/07 to Fix a Panic Problem Seen With Solaris 10 11/06
(CRs 6613003 and 6591438)

On a host with Solaris 10 11/06 OS installed, if you install the latest recommended patch cluster, which includes KU 120011-14, the system panics if you install an nxge Ethernet adapter (with nxge package version 1.0 and driver version 1.x).

This issue occurs when the add_drv command in the postinstall script of the SUNWnxge package is run (while adding the package), the system enters a panic cycle. The system does not panic if KU 120011-14 is removed.

The latest nxge driver packages are now bundled with Solaris 10 8/07 and later OS releases. This issue does not occur on systems with the Solaris 10 8/07 or later OS.

Note the following nxge patch dependencies:

Patch 127741-01:

Requires Patch 120011-14

Is incompatible with Patches 118833-36 and 125100-10

Patch 127742-01:

Requires Patch 120012-14

Is incompatible with Patches 118855-36 and 125101-10

Patch 125670-04

Requires Patch 118833-36

Is incompatible with Patch 12011-14

Patch 125671-04

Requires Patch 118855-36

Is incompatible with Patch 120012-14

Workaround: Perform One of the Following

Perform one of the following steps (Step 1 or Step 2).

1. Upgrade to Solaris 10 8/07.

Before upgrading, check if the unbundled nxge driver packages are installed. For example:

% pkginfo -l SUNWnxge | grep 1.0,REV=2007.01.12.10.0

If the packages are installed, and you have modified the nxge.conf file (on a Solaris 10 11/06 system), make a backup of the nxge.conf file before upgrading or reinstalling Solaris 10 8/07 or later releases. For Solaris 10 11/06 only (unbundled), the location of the nxge.conf file is /kernel/drv/nxge.conf for both SPARC and x86 platforms.

For Solaris 10 8/07 (nxge bundled), the locations of the nxge.conf file are as follows:

/platform/sun4u/kernel/drv/nxge.conf

/platform/sun4v/kernel/drv/nxge.conf

/platform/i86pc/kernel/drv/nxge.conf

Note - Do not install the older unbundled nxge driver packages (downloaded for Solaris 10 11/06) if the system is upgraded or freshly installed with the Solaris 10 8/07 or later OSs.

2. Or, on Solaris 10 11/06 systems, remove the unbundled nxge driver packages, and install the latest bundled packages from the Solaris 10 8/07 media.

a. Backup the nxge.conf file if you have modified it (see Step 1).

b. Remove the old nxge driver package version:

# pkgrm SUNWnxge

c. Install KU patches for Solaris 10 8/07 level:

SPARC platforms: 120011-14

x86 platforms: 120012-14

d. Add the new bundled version of the nxge driver package from the Solaris 10 8/07 media.

# pkgadd SUNWnxge

Issue: Installing the Correct Versions of `nxge` Data Integrity Patches for Your System

SunAlert 103076 was created to fix data integrity issues with the nxge driver.

Note - There are two separate sets and versions of patches to fix the data integrity issues. Installing the wrong patch results in an actual failure rather than a standard "does not apply" exit.

Workaround: Install the Following Patches

For Solaris 10 11/06, install Data Integrity Patch 125670-04 or later.

Caution - Do not install Patches 125670/125671 or later if you have KU 120011/120012-14 or later installed.

For Solaris 10 8/07, install Data Integrity Patch 127741-01 or later.

Caution - Do not install Patches 127741/127742-01 or later if you have KUs 118833/118855-36 or later or 125100/125101-10 or later installed.

Known Issues With Sun SPARC Systems

This section describes the known issues and workarounds for the nxge driver on Sun SPARC systems at the time of this release.

Issue: MSI or SoftRing Settings Might Need Site-Specific Adjustment

The default MSI or SoftRing settings might need to be adjusted for your site.

Workaround: Refer to the Tuning Web Site

Refer to the following tuning web site for information on tunable parameters and how to adjust tunable MSI and SoftRing parameters:

http://www.solarisinternals.com/wiki/index.php/Networks

Issue: `nxge.conf` Missing Hex Code (`0x`) Prefix for `“class-opt-ipv4-tcp = fe0”` Value

When a Sun Quad GbE UTP x8 PCI Express Card or ExpressModule is installed in a Sun SPARC Enterprise M9000 server, and the "class-opt-ipv4-tcp = fe0;" value is uncommented (set) in the /platform/sun4u/kernel/drv/nxge.conf configuration file, the following messages are output during system boot:

WARNING: Unexpected token ’fe0’ on line 112 of /platform/sun4u/kernel/drv/nxge.conf
WARNING: Unexpected end of line on line 112 of /platform/sun4u/kernel/drv/nxge.conf

Hex values are expressed with a prefix of 0x. The correct format should be as follows:

“class-opt-ipv4-tcp = 0xfe0;”

Known Issues With Solaris x86 Systems

This section describes the known issues and workarounds for the nxge driver on Solaris x86 systems at the time of this release.

Issue: Interrupts Might be Missed

Interrupt migration feature in the Solaris x86 kernel may lead to interrupts being missed.

Workaround: To Avoid Missing Interrupts

You must add the following lines to the /etc/system file to avoid missing interrupts that can lead to an interface hang.

set ddi_msix_alloc_limit=1 
set pcplusmp:apic_enable_dynamic_migration=0 
set pcplusmp:apic_intr_policy=1

Reboot the system after adding these lines:

# boot -r

Issue: Detaching Driver Can Kill Other Devices

Detaching or unloading a driver can kill other devices and lead to a system panic.

Workaround: Do Not Unplumb a Configured Interface

Avoid unplumbing an interface after it has been configured. To stop using a configured interface, issue the following command:

# ifconfig nxgeX down

where X is the number of the interface you want to stop using.

Issue: Older BIOS Versions Might Hang

Older BIOS versions show intermittent system hang issues.

Workaround: Upgrade the System BIOS

Minimum BIOS required for X4600 systems is 40

Minimum BIOS required for X4100/X4200 systems is 29

Refer to:

http://www.sun.com/download/index.jsp?cat= Hardware%20Drivers&tab=3&subcat= x64%20Servers%20%26%20Workstations

Issue: SoftRing Mode is Disabled by Default

Default SoftRing mode is disabled under the Solaris x86 Operating System.

Workaround: Enable SoftRing Mode

Add the following line in /etc/system file and reboot.

set ip:ip_squeue_fanout=1 
set ip_squeue_soft_ring=1

Then add the following to a startup script, or use the ndd utility before plumbing the interface:

# ndd -set /dev/ip ip_soft_rings_cnt 8

Issue: Operating System Cannot be Installed Over the 10 Gigabit Ethernet Interface

The nxge driver is not bundled with Solaris x86 Operating System and cannot be installed over a 10 Gigabit Ethernet interface.

Note - Support for operating system installation over the 10 Gigabit Ethernet interface is planned for a future Solaris release.

Workaround: Create a Custom Boot Archive with `nxge` Driver Support

Create a customized boot archive with nxge driver support. The following document shows how to customize the boot archive for the Solaris Operating System.

http://www.sun.com/blueprints/0806/819-7546.html

For Linux systems, refer to the documentation from your Linux distribution to create a customized boot archive with the nxge driver.

Issue: Limited Space for Option ROMs

POST may report "Not Enough Space to copy PCI Option ROM" when Sun 10Gigabit cards are plugged in the system due to limited space available for the Option ROMs.

Workaround: Disable Slots or Interfaces

If the Sun 10 Gigabit Ethernet port will not be used for installing the operating system, then the slots with the Sun x8 Express 10 Gigabit Ethernet Fiber XFP Low Profile adapter must be disabled from Option ROM probing.

If the Sun 10 Gigabit Ethernet port will be used for installing the operating system, then disable all the other slots as well as the on-board interfaces from Option ROM probing.

To Disable Slots and Interfaces from Option ROM Probing

If there is not enough space available for the Option ROMs, you will see the following error message during system boot:

1. Press F2 to get to the BIOS Setup screen:

2. Press the right arrow ket to get to the PCIPnP menu and disable the OPROM scanning for the slot that has the Sun Quad Gigabit Ethernet UTP PCI Express Card.

This example shows PCIe slot 0.

3. Save your changes.

Known Issues With Linux Systems

This section describes the known issues and workarounds for the nxge driver on Linux systems at the time of this release.

Issue: Card Fails on RHEL4U3 LargeSMP Kernel With Over 8 CPUs

The Sun x8 Express 10 Gigabit Ethernet Fiber XFP Low Profile adapter will fail to function on RHEL4U3 LargeSMP Kernel with more than 8 CPUs.

Workaround: Reduce CPUs to 8 or Upgrade to RHEL4U4

This is an issue with interrupt distribution. You can use the following two workarounds:

Reduce the number of CPUs to 8.

% vi /boot/grub/menu.lst

Add the following line at the end of the kernel parameters in the menu.1st file:

 maxcpus=8

The following is an examle entry in a /boot/grub/menu.lst file:

title Red Hat Enterprise Linux Server (2.6.18-53.el5)
  root (hd0,0)
  kernel /vmlinuz-2.6.18-53.el5 ro root=/dev/VolGroup00/LogVol00 maxcpus=8 console=ttyS0,9600n8 rhgb quiet
  initrd /initrd-2.6.18-53.el5.img

Upgrade the operating system to RHEL4U4.

Issue: System Panics When `irqbalance` Is Enabled

System panics observed when irqbalance is enabled.

Workaround: Disable `irqbalance`

You must disable irqbalance to avoid system crashes. Use the following procedure:

1. Boot the system and configure the Sun x8 Express 10 Gigabit Ethernet interfaces as usual.

2. Disable irqbalance by doing any of the following methods:

Kill the irqbalance process:

# ps -ef | grep irqbalance 
# kill -9 pid

By stopping the irqbalance service SuSE10 or RHEL4U4:

# /etc/init.d/irq_balancer stop

Issue: `ethtool` Utility

The ethtool utility, as of version 5, does not provide full support of 10 Gigabit Ethernet network interfaces. Consequently, the ethtool utility does not support bonding on Sun x8 Express 10 Gigabit Ethernet interfaces.

Workaround: See Support URL

Wait for the fix to be applied to the ethtool source. For more information, see the 1170813 Patch ID for correctly reporting 10Gb/s support at:

http://sourceforge.net/projects/gkernel/