Sun Quad GbE UTP x8 PCIe ExpressModule User’s Guide

The nxge device driver controls the Sun Quad Gigabit Ethernet interfaces. You can manually set the nxge driver parameters to customize each device in your system.

This chapter lists the available device driver parameters and describes how you can set these parameters.

• nxge Hardware and Software Overview
• Setting nxge Driver Parameters on a Solaris Platform
• Setting Parameters Using the ndd Utility
• Setting Parameters Using the nxge.conf File

---

nxge Hardware and Software Overview

The Sun Quad GbE UTP x8 PCIe ExpressModule provides four Gigabit full-duplex networking interfaces. The device driver automatically sets the link speed to 1000 Mbit/sec and conforms to the IEEE 802.3ad Ethernet standard. Each interface has 4 receive DMA channels and 6 transmit DMA channels to allow for parallel processing of the packets.

The Sun Quad GbE UTP x8 PCIe ExpressModule extends CPU and OS parallelism to networking with its support for hardware-based flow classification and multiple DMAs. Using CPU thread affinity to bind a given flow to a specific CPU thread, the ExpressModule enables a one-to-one correlation of Rx and Tx packets across the same TCP connection. This can help avoid cross-calls and context switching to deliver greater performance while reducing the need for CPU resources to support I/O processing. The Sun Quad GbE UTP x8 PCIe ExpressModule uses the Sun innovative MAC controller.

---

Setting nxge Driver Parameters on a Solaris Platform

You can set the nxge device driver parameters in two ways:

• Using the ndd utility
• Using the nxge.conf file

If you use the ndd utility, the parameters are valid only until you reboot the system. This method is good for testing parameter settings.

To set parameters so they remain in effect after you reboot the system, perform the following procedure:

1. Create an nxge.conf file:

/platform/sun4u/kernel/drv/nxge.conf

2. Add parameter values to this file when you need to set a particular parameter for a device in the system.

---

Setting Parameters Using the ndd Utility

Use the ndd utility to configure parameters that are valid until you reboot the system.

The following sections describe how you can use the nxge driver and the ndd utility to modify (with the -set option) or display (without the -set option) the parameters for each nxge device.

Noninteractive and Interactive Modes

You can use the ndd utility in two modes:

• Noninteractive
• Interactive

In Noninteractive mode, you invoke the utility to execute a specific command. Once the command is executed, you exit the utility. In Interactive mode, you can use the utility to get or set more than one parameter value. Refer to the ndd(1M) man page for more information.

Before you use the ndd utility to get or set a parameter for a nxge device, you must specify the device instance for the utility.

1. Check the /etc/path_to_inst file to identify the instance associated with a particular device.

# grep nxge /etc/path_to_inst
"/pci@7c0/pci@0/pci@9/network@0" 0 "nxge"
"/pci@7c0/pci@0/pci@9/network@0,1" 1 "nxge"
"/pci@7c0/pci@0/pci@9/network@0,2" 2 "nxge"
"/pci@7c0/pci@0/pci@9/network@0,3" 3 "nxge"

This section describes how to modify and display parameter values.

1. Use the -set option to modify a parameter.

If you invoke the ndd utility with the -set option, the utility passes value, which must be specified, down to the named /dev/nxgedriver-instance, and assigns the value to the parameter:

# ndd -set /dev/nxgenumber parameter-value

where number is the driver instance, for example /dev/nxge0, /dev/nxge1.

2. Display the value of a parameter by specifying the parameter name and omitting the value.

When you omit the -set option, the utility queries the named driver instance, retrieves the value associated with the specified parameter, and prints it:

# ndd /dev/nxgenumber parameter

1. Modify a parameter value in interactive mode by specifying ndd /dev/nxgenumber:

# ndd /dev/nxge0
name to get/set? (Enter the parameter name or ? to view all parameters)

After you enter the parameter name, the ndd utility prompts you for the parameter value.

2. List all the parameters supported by the nxge driver, by typing ?.

# ndd /dev/nxge1 name to get/set ?
?                             (read only)
function_number               (read only)
adv_autoneg_cap               (read and write)
adv_10gfdx_cap                (read and write)
adv_1000fdx_cap               (read and write)
adv_100fdx_cap                (read and write)
adv_10fdx_cap                 (read and write)
adv_pause_cap                 (read and write)
accept_jumbo                  (read and write)
rxdma_intr_time               (read and write)
rxdma_intr_pkts               (read and write)
class_opt_ipv4_tcp            (read and write)
class_opt_ipv4_udp            (read and write)
class_opt_ipv4_ah             (read and write)
class_opt_ipv4_sctp           (read and write)
class_opt_ipv6_tcp            (read and write)
class_opt_ipv6_udp            (read and write)
class_opt_ipv6_ah             (read and write)
class_opt_ipv6_sctp           (read and write)
 

---

Setting Parameters Using the nxge.conf File

Specify the driver parameter properties for each device by creating a nxge.conf file in the /platform/sun4v/kernel/drv/sparcv9/ directory. Use a nxge.conf file when you need to set a particular parameter for a device in the system.

The man pages for prtconf(1M) and driver.conf(4) include additional details. The next procedure shows an example of setting parameters in a nxge.conf file.

  To access any man page, type the man command plus the name of the man page.

For example, to access man pages for prtconf(1M), type:

% man prtconf

1. Obtain the hardware path names for the nxge devices in the device tree.

a. Check the /etc/driver_aliases file to identify the name associated with a particular device:

# grep nxge /etc/driver_aliases
nxge "pciex108e,abcd"

b. Locate the path names and the associated instance numbers in the
/etc/path_to_inst file.

# grep nxge /etc/path_to_inst
"/pci@7c0/pci@0/pci@9/network@0" 0 "nxge"
"/pci@7c0/pci@0/pci@9/network@0,1" 1 "nxge"
"/pci@7c0/pci@0/pci@9/network@0,2" 2 "nxge"
"/pci@7c0/pci@0/pci@9/network@0,3" 3 "nxge"

In this example:

• The first part within the double quotes specifies the hardware node name in the device tree.
• The number not enclosed in quotes is the instance number (shown in bold italics for emphasis).
• The last part in double quotes is the driver name.

To identify a PCIe device unambiguously in the nxge.conf file, use the name, parent name, and the unit-address for the device. Refer to the pci(4) man page for more information about the PCIe device specification.

In this example:

• name = "pciex108e,abcd"
• parent = "/pci@7c0/pci@0/pci@9/"
• unit-address = "0"

2. Set the parameters for the nxge devices in the /platform/sun4u/kernel/drv/nxge.conf file.

• The following parameters can be set using the /platform/sun4u/kernel/drv/nxge.conf file.


# #---------------Link Configuration ---------------------- # The link parameters depend on the type of the card # and the port. # 10 gigabit related parameters ( i.e adv_10gfdx_cap) # apply only to 10gigabit ports. # Half duplex is not supported on any NIU card. # # adv-autoneg-cap # Advertise auto-negotiation capability. # default is 1 # adv-autoneg-cap = 1; # # adv_10gfdx_cap # Advertise 10gbps Full duplex capability. # default is 1 # adv_10gfdx_cap = 1; # # adv_1000fdx_cap # Advertise 1gbps Full duplex capability. # default is 1 # adv_1000fdx_cap = 1; # # adv_100fdx_cap # Advertise 100mbps Full duplex capability. # default is 1 # adv_100fdx_cap = 1; # # adv_10fdx_cap # Advertise 10mbps Full duplex capability. # default is 1 # adv_10fdx_cap = 1; # # adv_asmpause_cap # Advertise Asymmetric pause capability. # default is 0 # adv_asmpause_cap = 0; # # adv_pause_cap # Advertise pause capability. # default is 1 # adv_pause_cap = 1; # # #------- Jumbo frame support --------------------------------- # To enable jumbo support for all nxge interfaces, # accept_jumbo = 1; # # To disable jumbo support for all nxge interfaces, # accept_jumbo = 0; # # Default is 0. See the example at the end of this file for # enabling or disabling jumbo for a particular nxge interface. # # #------- Receive DMA Configuration ---------------------------- # # rxdma-intr-time # Interrupts after this number of NIU hardware ticks have # elapsed since the last packet was received. # A value of zero means no time blanking (Default = 8). # # rxdma-intr-pkts # Interrupt after this number of packets have arrived since # the last packet was serviced. A value of zero indicates # no packet blanking (Default = 20). # # Default Interrupt Blanking parameters. # # rxdma-intr-time = 8; # rxdma-intr-pkts = 20; # # #------- Classification and Load Distribution Configuration ------ # # class-opt-****-*** # These variables define how each IP class is configured. # Configuration options range from whether TCAM lookup ie # is enabled to flow hash generation. # This parameters also control how the flow template is # constructed and how packet is distributed within RDC # groups. # # supported classes: # class-opt-ipv4-tcp class-opt-ipv4-udp class-opt-ipv4-sctp # class-opt-ipv4-ah class-opt-ipv6-tcp class-opt-ipv6-udp # class-opt-ipv6-sctp class-opt-ipv6-ah # # Configuration bits (The following bits will be decoded # by the driver as hex format). # # 0010: use MAC Port (for flow key) # 0020: use L2DA (for flow key) # 0040: use VLAN (for flow key) # 0080: use proto (for flow key) # 0100: use IP src addr (for flow key) # 0200: use IP dest addr (for flow key) # 0400: use Src Port (for flow key) # 0800: use Dest Port (for flow key) # # class-opt-ipv4-tcp = fe0; #


• The following parameters operate on a per port basis and can be set using the /platform/sun4u/kernel/drv/nxge.conf file.


# # ------- How to set parameters for a particular interface -------- # The example below shows how to locate the device path and set a # parameter for a particular nxge interface. (Using jumbo support as # an example) # # Use the following command to find out the device paths for nxge, # more /etc/path_to_inst | grep nxge # # For example, if you see, # "/pci@7c0/pci@0/pci@8/network@0" 0 "nxge" # "/pci@7c0/pci@0/pci@8/network@0,1" 1 "nxge" # "/pci@7c0/pci@0/pci@8/network@0,2" 2 "nxge" # "/pci@7c0/pci@0/pci@8/network@0,3" 3 "nxge" # # then you can enable jumbo for ports 0 and 1 and disable jumbo for ports 2 # and 3 as follows, # # name = "pciex108e,abcd" parent = "/pci@7c0/pci@0/pci@8/" unit-address = "0" # accept_jumbo = 1; # name = "pciex108e,abcd" parent = "/pci@7c0/pci@0/pci@8/" unit-address = "0,1" # accept_jumbo = 1; # name = "pciex108e,abcd" parent = "/pci@7c0/pci@0/pci@8/" unit-address = "0,2" # accept_jumbo = 0; # name = "pciex108e,abcd" parent = "/pci@7c0/pci@0/pci@8/" unit-address = "0,3" # accept_jumbo = 0;


• In the following example, the ports of all the Sun Quad GbE UTP x8 PCIe ExpressModules are being set for load balancing Rx traffic based on IP source address. The default value is F80, indicating Rx load balancing based on IP 5-tuple. Notice the semicolon at the end of the last parameter.


class-opt-ipv4-tcp = 100; class-opt-ipv4-udp = 100;

The following example shows ports on two different cards being set. Only one node needs to be specified.

name = "pciex108e,abcd" parent = "/pci@780/pci@0/pci@8/" unit-address = "0" class-opt-ipv4-tcp = 0x100;
 
name = "pciex108e,abcd" parent = "/pci@7c0/pci@0/pci@9/" unit-address = "0" class-opt-ipv4-tcp = 0x40;

3. Save the nxge.conf file.

Sun Quad GbE UTP x8 PCIe ExpressModule User’s Guide

820-1609-12

Copyright © 2008 Sun Microsystems, Inc. All Rights Reserved.