Installing the Sun 10-Gigabit Ethernet Device Driver on Linux Platforms

A P P E N D I X D

This appendix explains how to install the Sun 10-Gigabit Ethernet PCI-X device driver in a Linux environment. This appendix contains the following sections:

Setting Up the Driver in a Linux Platform

Tuning for Maximum Performance

Setting Driver Parameters in a Linux Environment

Setting Up the Driver in a Linux Platform

If your system uses the Linux operating system you will need to perform the following procedure to be sure the Sun 10-Gigabit Ethernet device driver is properly installed and loaded.

To Set Up the Driver in a Linux Platform

1. Build the 10-Gigabit Ethernet driver:

For RedHat, use the following command:

# rpmbuild --rebuild sun-10-gigabit-kernel-1.0.version.src.rpm

where the version is the minor version number such as:
sun-10-gigabit-kernel-1.0.82.6-1.src.rpm

2. Build the man pages (optional):

For RedHat, use the following command:

# rpmbuild --rebuild sun-10-gigabit-1.0.version.src.rpm

3. Change to the rpm directory:

For RedHat, use the following command:

# cd /usr/src/redhat/RPMS/arch

4. Install the 10-Gigabit Ethernet rpms, using the same command for both RedHat and Suse:

# rpm -ivh sun-10-gigabit-kernel-1.0.version.arch.rpm

5. Use the depmod command to register the Sun 10-Gigabit Ethernet module:

# depmod

6. Load ixge driver for all instances:

# modprobe ixge

7. Verify that the driver is loaded.

In the following example, the ixge driver is shown in bold italic. On your system the driver will be indistinguishable from the other modules.

# lsmod
 
Module                  Size  Used by    	Not tainted
ixge                   53592   0  (unused)
ide-cd                 36616   0  (autoclean)
ipv6                  285176  -1  (autoclean)
key                    78344   0  (autoclean) [ipv6]
st                     31380   0  (autoclean) (unused)
sr_mod                 16812   0  (autoclean) (unused)
cdrom                  34664   0  (autoclean) [ide-cd sr_mod]
sg                     40384   0  (autoclean)
joydev                  7136   0  (unused)
evdev                   6496   0  (unused)
input                   7488   0  [joydev evdev]
usb-ohci               23728   0  (unused)
usbcore                87648   1  [usb-ohci]
bcm5700               194456   1 
lvm-mod                81248   0  (autoclean)
reiserfs              248344   1 
mptscsih               44144   2 
mptbase                46400   3  [mptscsih]
sd_mod                 16192   4 
scsi_mod              133028   5  [st sr_mod sg mptscsih sd_mod]

8. Run the dmesg command to see which devices the ixge driver was mapped to:

# dmesg
Sun(R) 10GE StreamStor River - version 1.0.82.5
Copyright (c) 1999-2004 Intel Corp., Sun Corp.
divert: allocating divert_blk for eth2
eth2: Sun(R) 10GE StreamStor River Network Connection
divert: allocating divert_blk for eth3

9. Add the ixge interfaces to the /etc/modules.conf file.

alias eth2 ixge
 
alias eth3 ixge

10. Use the ethtool command to check the parameter configurations that apply to the ixge driver.

# ethtool eth3
 
Settings for eth3:
Supported ports: [ FIBRE ]
Supported link modes:   10000baseT/Full 
Supports auto-negotiation: Yes
Advertised link modes:  10000baseT/Full 
Advertised auto-negotiation: Yes
Speed: 10000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Current message level: 0x000000ff (255)
Link detected: yes

Tuning for Maximum Performance

The following tunings will improve the performance of the Sun 10-Gigabit Ethernet device driver on a system running the Linux operating system.

1. Create a performance tuning script (for example, ixge_perf.sh), and edit this file to add the interface or device ID of Sun 10-Gigabit Ethernet adapter.

#!/bin/bash
echo "configuring network performance"
# set mmrbc to 4k reads, modify only Sun 10-Gigabit Ethernet device IDs
 
# replace 1a48 with appropriate Sun 10-Gigabit Ethernet device’s ID
# installed on the system, if needed.
setpci -d 8086:1048 e6.b=2e
 
# set the MTU (max transmission unit) - it requires your switch and
# clients to change too and should support the value you provide here.
# Also set the txqueuelen on your Sun 10-Gigabit Ethernet adapter.
# If your adapter was configured as eth2, you would the following:
ifconfig eth2 mtu 16110 txqueuelen 1000 up
 
# call the sysctl utility to modify /proc/sys entries; update the
# location of the sysctl_ixge.conf file as appropriate
sysctl -p ./sysctl_ixge.conf

2. Create the conf file (for example, sysctl_ixge.conf) that will be called by the sysctl utility.

### IPV4 specific settings
# turns TCP timestamp support off, default 1, reduces CPU use
net.ipv4.tcp_timestamps = 0
# turn SACK support off, default on systems with a VERY fast bus ->
# memory interface this is the big gainer
net.ipv4.tcp_sack = 0
# sets min/default/max TCP read buffer, default 4096 87380 174760
net.ipv4.tcp_rmem = 10000000 10000000 10000000
# sets min/pressure/max TCP write buffer, default 4096 16384 131072
net.ipv4.tcp_wmem = 10000000 10000000 10000000
# sets min/pressure/max TCP buffer space, default 31744 32256 32768
net.ipv4.tcp_mem = 10000000 10000000 10000000
 
### CORE settings (mostly for socket and UDP effect)
# maximum receive socket buffer size, default 131071
net.core.rmem_max = 524287
# maximum send socket buffer size, default 131071
net.core.wmem_max = 524287
# default receive socket buffer size, default 65535
net.core.rmem_default = 524287
# default send socket buffer size, default 65535
net.core.wmem_default = 524287
# maximum amount of option memory buffers, default 10240
net.core.optmem_max = 524287
# number of unprocessed input packets before kernel starts dropping
# them, default 300
net.core.netdev_max_backlog = 300000

Setting Driver Parameters in a Linux Environment

In a Linux operating system, the driver parameters can only be set at the time the driver is loaded. If you have already loaded the driver and have not set the parameters at the same time. Remove the driver and reinstall it.

To Set `ixge` Parameters in a Linux Environment

1. Remove the Sun 10-Gigabit Ethernet PCI-X device driver, if it is already loaded.

# rmmod ixge

2. Load the Sun 10-Gigabit Ethernet PCI-X device driver:

# modprobe ixge option=VAL1, VAL2, . . .

or you can use the insmod command:

# insmod ixge option=VAL1, VAL2, . . .

where VAL1 applies to the first ixge interface and VAL2 to the next interface and so on.

TABLE D-1 lists the tunable ixge driver parameters for Linux operating systems, and describes their function.

TABLE D-1 Tunable `ixge` Driver Parameters for Linux Operating Systems
Keyword	Description
FlowControl	Valid Range: 0-3 (0=none, 1=Rx only, 2=Tx only, 3=Rx&Tx) Default: Read from the EEPROM If EEPROM is not detected, default is 3. This parameter controls the automatic generation (Tx) and response (Rx) to Ethernet PAUSE frames.
RxDescriptiors	Valid Range: 64-4096 Default Value: 1024 This value is the number of receive descriptors allocated by the driver. Increasing this value allows the driver to buffer more incoming packets. Each descriptor is 16 bytes. A receive buffer is also allocated for each descriptor and can be either 2048, 4056, 8192, or 16384 bytes, depending on the MTU setting. When the MTU size is 1500 or less, the receive buffer size is 2048 bytes. When the MTU is greater than 1500 the receive buffer size will be either 4056, 8192, or 16384 bytes. The maximum MTU size is 16114.
RxIntDelay	Valid Range: 0-65535 (0=off) Default Value: 72 This value delays the generation of receive interrupts in units of 0.8192 microseconds. Receive interrupt reduction can improve CPU efficiency if properly tuned for specific network traffic. Increasing this value adds extra latency to frame reception and can end up decreasing the throughput of TCP traffic. If the system is reporting dropped receives, this value may be set too high, causing the driver to run out of available receive descriptors.
TxDescriptors	Valid Range: 64-4096 Default Value: 256 This value is the number of transmit descriptors allocated by the driver. Increasing this value allows the driver to queue more transmits. Each descriptor is 16 bytes.
XsumRX	Valid Range: 0-1 Default Value: 1 A value of 1 indicates that the driver should enable IP checksum offload for received packets (both UDP and TCP) to the adapter hardware.

To Configure VLANs in a Linux Environment

1. Ensure that the ixge module is loaded:

# modprobe ixge

2. Plumb the Sun 10-Gigabit Ethernet PCI-X interface:

# ifconfig eth2 xxx.xxx.xx.xxx up

where xxx.xxx.xx.xxx = the IP address of the interface.

3. Insert the vlan module:

# /sbin/insmod 8021q

4. Add vlan instance (VID):

# sbin/vconfig add eth2 5

5. Configure ixge vlan (eth2 in this example):

# ifconfig eth2.5 xxx.xxx.xx.xxx up

where xxx.xxx.xx.xxx = the IP address of the interface.

To Configure Bonding for Multiple `ixge` Interfaces

1. Modify /etc/modules.conf file by adding these lines:

alias bond0 bonding
options bonding max_bonds=2 mode=4 miimon=1000

Where:

bond0 is the bonding device.
max_bonds is the number of bond interfaces to be created.
mode specifies the bonding policies.
miimon is the frequency in milliseconds that MII link monitoring will occur.

Refer to Linux documentation for more information.

2. Load the bonding driver:

# modprobe bonding

3. Configure bond0 interface.

In this example, bond0 is the master of the two interfaces eth2 and eth3.

# ip addr add 192.12.38.64/24 brd + dev bond0
# ip link set dev bond0 up
# ifenslave  bond0 eth2 eth3