Configuring ixge Driver Parameters for SPARC Platforms

C H A P T E R 3

Configuring ixge Driver Parameters for SPARC Platforms

This chapter describes how to configure the driver parameters used by the Sun 10-Gigabit Ethernet PCI-X ixge adapter for SPARC platforms. This chapter contains the following sections:

10-Gigabit Ethernet Device Driver Parameters

Setting ixge Driver Parameters

10-Gigabit Ethernet Driver Operating Statistics

10-Gigabit Ethernet Device Driver Parameters

The ixge device driver controls the Sun 10-Gigabit Ethernet PCI-X adapter. The ixge driver is attached to the UNIX pci name property pci8086,1048 for the Sun 10-Gigabit Ethernet PCI-X adapter (8086 is the vendor ID and 1048 is the PCI device ID).

You can manually configure the ixge device driver parameters to customize each Sun 10-Gigabit Ethernet adapter device in your system. This section provides an overview of the capabilities of the 10-Gigabit Ethernet driver used in the adapter, lists the available ixge device driver parameters, and describes how to configure these parameters.

10-Gigabit Ethernet Driver Parameter Values and Definitions

TABLE 3-1 describes the parameters and settings for the ixge device driver.

TABLE 3-1 `ixge` Driver Parameter, Status, and Descriptions
Parameter	Status	Description
`instance`	Read and write	Device instance
`adv-asmpause-cap`	Read and write	Flow control parameter
`adv-pause-cap`	Read and write	Flow control parameter
`rx-intr-pkts`	Read and write	Receive interrupt blanking values
`rx-intr-time`	Read and write	Receive interrupt blanking values
`accept-jumbo`	Read and write	Enable jumbo frames

Flow Control Parameters

The ixge device sources (transmits) and terminates (receives) pause frames conforming to the IEEE 802.3x Frame Based Link Level Flow Control Protocol. In response to received flow control frames, the ixge device can slow down its transmit rate. The ixge device is capable of sourcing flow control frames by requesting the link partner to slow down, provided that the link partner supports this feature. By default, the driver advertises both transmit and receive pause capability during autonegotiation.

TABLE 3-2 provides flow control keywords and describes their function.

TABLE 3-2 Read-Write Flow Control Keyword Descriptions
Keyword	Description
`adv_asmpause_cap`	The adapter supports asymmetric pause, which means it can pause only in one direction. 0=Off (default) 1=On
`adv_pause_cap`	This parameter has two meanings, each depending on the value of `adv_asmpause_cap`. (Default=0) If `adv_asmpause_cap` = 1 while a`dv_pause_cap` = 1, pauses are received. If `adv_asmpause_cap` = 1 while a`dv_pause_cap` = 0, pauses are transmitted. If `adv_asmpause_cap` = 0 while a`dv_pause_cap` = 1, pauses are sent and received. If `adv_asmpause_cap` = 0, then a`dv_pause_cap` determines whether Pause capability is on or off.

Interrupt Parameters

TABLE 3-3 describes the receive interrupt blanking values.

TABLE 3-3 RX Blanking Register for Alias Read
Field Name	Values	Description
rx_intr_pkts	0 to 511	Interrupt after this number of packets have arrived since the last packet was serviced. A value of zero indicates no packet blanking. (Default=3)
rx_intr_time	0 to 32768	Interrupt after 4.5 US ticks have elapsed since the last packet was serviced. A value of zero indicates no time blanking. (Default=0)

Jumbo Frames

The variable accept-jumbo enables the driver to instruct the upper layers that the driver is operating in Jumbo-frame mode.

TABLE 3-4 `accept-jumbo` Parameters
Parameter	Values	Description
`accept-jumbo`	1	Enables Jumbo-frame mode and sets MTU to greater than 1500.
`accept-jumbo`	0	Jumbo frames is not enabled. MTU is at 1500.

Refer to the documentation that came with your switch for exact commands to configure jumbo frames support.

To Configure Jumbo Frames Using ndd

1. Unplumb the interface you want to configure Jumbo Frames on.

2. Kill any snoop or netstat processes running over the interface to be configured.

3. Set the instance number:

% ndd -set /dev/ixge instance 0

In this example, the instance number set is 0.

4. Set the instance to accept jumbo frames:

% ndd -set /dev/ixge accept-jumbo 1

5. Plumb the interface up:

% ifconfig ixge0 plumb xxx.xxx.xx.xxx up

where xxx.xxx.xx.xxx = the IP address of the interface.

To Configure Jumbo Frames in a Linux Environment

1. Ensure that the ixge software is installed.

2. Plumb the Sun 10-Gigabit Ethernet PCI-X interface:

# modprobe ixge
# ifconfig eth2 xxx.xxx.xx.xxx up
# ifconfig eth2 mtu 8170

where xxx.xxx.xx.xxx = the IP address of the interface.

Setting `ixge` Driver Parameters

You can set the ixge device driver parameters in two ways:

Using the ndd utility

Using the ixge.conf file

If you use the ndd utility, the parameters are valid only until you reboot the system. This method is good for testing parameter settings.

To set parameters so they remain in effect after you reboot the system, create a
/platform/sun4u/kernel/drv/ixge.conf file and add parameter values to this file when you need to set a particular parameter for a device in the system.

Setting Parameters Using the `ndd` Utility

Use the ndd utility to configure parameters that are valid until you reboot the system. The ndd utility supports any networking driver that implements the Data Link Provider Interface (DLPI).

The following sections describe how you can use the ixge driver and the ndd utility to modify (with the -set option) or display (without the -set option) the parameters for each ixge device.

To Specify Device Instances for the `ndd` Utility

Before you use the ndd utility to get or set a parameter for a ixge device, you must specify the device instance for the utility.

1. Check the /etc/path_to_inst file to identify the instance associated with a particular device.

# grep ixge /etc/path_to_inst
"/pci@9,600000/ethernet@1" 0 "ixge"

In this example, the 10-Gigabit Ethernet instance is from the installed adapter. The instance number is in bold italics for clarity.

2. Use the instance number to select the device.

# ndd -set /dev/ixge instance number

The device remains selected until you change the selection.

Noninteractive and Interactive Modes

You can use the ndd utility in two modes:

Noninteractive

Interactive

In noninteractive mode, you invoke the utility to execute a specific command. Once the command is executed, you exit the utility. In interactive mode, you can use the utility to get or set more than one parameter value. Refer to the ndd(1M) man page for more information.

To Use the ndd Utility in Noninteractive Mode

This section describes how to modify and display parameter values.

To modify a parameter value, use the -set option.

If you invoke the ndd utility with the -set option, the utility passes value, which must be specified, down to the named /dev/ixge driver instance, and assigns the value to the parameter:

# ndd -set /dev/ixge parameter value

When you change any parameter, a message similar to the following appears:

ixge0: xcvr addr:0x01 - link up 10000 Mbps full duplex

To display the value of a parameter, specify the parameter name and omit the value.

When you omit the -set option, the utility queries the named driver instance, retrieves the value associated with the specified parameter, and prints it:

# ndd /dev/ixge parameter

To Use the ndd Utility in Interactive Mode

To modify a parameter value in interactive mode, specify ndd /dev/ixge:

# ndd /dev/ixge
name to get/set? (Enter the parameter name or ? to view all parameters)

After you enter the parameter name, the ndd utility prompts you for the parameter value (see TABLE 3-1 through TABLE 3-6).

To list all the parameters supported by the ixge driver, type ?.

See TABLE 3-1 through TABLE 3-6 for parameter descriptions.

# ndd -get /dev/ixge \?
?                             (read only)
instance                      (read and write)
adv_asmpause_cap              (read and write)
adv_pause_cap                 (read and write)
rx_intr_pkts                  (read and write)
rx_intr_time                  (read and write)
accept_jumbo                  (read and write)
laggr_multistream             (read and write)#

Note - By default, autonegotiation is set to off. This adapter operates only at full-duplex.

Setting Parameters Using the `ixge.conf` File

Specify the driver parameter properties for each device by creating a ixge.conf file in the /platform/sun4u/kernel/drv directory. Use a ixge.conf file when you need to set a particular parameter for a device in the system. The parameters you can set are the read and write parameters listed in 10-Gigabit Ethernet Driver Parameter Values and Definitions.

Note - Configuring the parameters by putting ndd commands in rcX.d scripts is not supported.

The man pages for prtconf(1M) and driver.conf(4) include additional details. The next procedure shows an example of setting parameters in a ixge.conf file.

To access any man page, type the man command plus the name of the man page. For example, to access man pages for prtconf(1M), type:

% man prtconf

To Set Driver Parameters Using a `ixge.conf` File

1. Obtain the hardware path names for the ixge devices in the device tree.

a. Check the /etc/driver_aliases file to identify the name associated with a particular device:

# grep ixge /etc/driver_aliases
ixge "pci8086,1048.108e.7036"

Note - ixge "pci8086,1048.108e.7036" is used by the Sun 10-Gigabit Ethernet adapter.

b. Locate the path names and the associated instance numbers in the
/etc/path_to_inst file.

# grep ixge /etc/path_to_inst
"/pci@9,600000/ethernet@1" 0 "ixge"

In this example:

The first part within the double quotes specifies the hardware node name in the device tree.

The number not enclosed in quotes is the instance number (shown in bold italics).

The last part in double quotes is the driver name.

In the hardware node name, the last component after the last / character and before the @ character is the device name.

The path name before the last component is the parent name.

The number after the final @ character within quotes is the unit-address.

To identify a PCI device unambiguously in the ixge.conf file, use the name, parent name, and the unit-address for the device. Refer to the pci(4) man page for more information about the PCI device specification.

2. Set the parameters for the above devices in the
/platform/sun4u/kernel/drv/ixge.conf file.

In the following example, the accept-jumbo parameter is set to enable jumbo frames for all Sun 10-Gigabit Ethernet devices. See the driver.conf(4) man page for more information.

 accept-jumbo=1;

In the following example, the accept-jumbo parameter is set to enable jumbo frames is set for a single instance of the Sun 10-Gigabit Ethernet device.

In this example:

parent = "/pci@8,600000"

unit-address = "0"

name="pci8086,1048.108e.7036" parent="pci@8,600000" unit-address="0" accept-jumbo=1;

Note - If you omit the name=, parent=, and unit-address= definitions, the settings become global to all Sun 10-Gigabit Ethernet instances.

3. Save the ixge.conf file.

10-Gigabit Ethernet Driver Operating Statistics

These statistics are part of the statistics presented by the kstat command.

TABLE 3-5 describes the read-only Media Independent Interface (MII) capabilities. These parameters define the capabilities of the hardware. The Gigabit Media Independent Interface (GMII) supports all of the following capabilities.

TABLE 3-5 Read-Only `ixge` Device Capabilities
Parameter	Description (Local interface Capabilities)
`cap_autoneg`	0 = Not capable of autonegotiation 1 = Autonegotiation capable
`cap_10gfdx`	Local interface full-duplex capability 0 = Not 10000 Mbit/sec full-duplex capable 1 = 10000 Mbit/sec full-duplex capable
`cap_10ghdx`	Local interface half-duplex capability 0 = Not 10000 Mbit/sec half-duplex capable 1 = 10000 Mbit/sec half-duplex capable
`cap_1000fdx`	Local interface full-duplex capability 0 = Not 1000 Mbit/sec full-duplex capable 1 = 1000 Mbit/sec full-duplex capable
`cap_1000hdx`	Local interface half-duplex capability 0 = Not 1000 Mbit/sec half-duplex capable 1 = 1000 Mbit/sec half-duplex capable
`cap_100fdx`	Local interface full-duplex capability 0 = Not 100 Mbit/sec full-duplex capable 1 = 100 Mbit/sec full-duplex capable
`cap_100hdx`	Local interface half-duplex capability 0 = Not 100 Mbit/sec half-duplex capable 1 = 100 Mbit/sec half-duplex capable
`cap_10fdx`	Local interface full-duplex capability 0 = Not 10 Mbit/sec full-duplex capable 1 = 10 Mbit/sec full-duplex capable
`cap_10hdx`	Local interface half-duplex capability 0 = Not 10 Mbit/sec half-duplex capable 1 = 10 Mbit/sec half-duplex capable
`cap_asm_pause`	Local interface flow control capability 0 = Not asymmetric pause capable 1 = Asymmetric pause (from the local device) capable
`cap_pause`	Local interface flow control capability 0 = Not Symmetric pause capable 1 = Symmetric pause capable

Table describes the read-only ixge device capabilities.

TABLE 3-6 describes the kstat ixge:1 receive and transmit parameters:

TABLE 3-6 Receive and Transmit Parameters
Parameter	Description
lb_mode	Copy of the Loopback mode the device is in, if any.
rev_id	Revision ID of the 10-Gigabit Ethernet device useful for recognition of device being used in the field.
rx_allocb_fail	Number of times the driver failed to allocate OS receive buffer.
rx_hdr_drops	Number of times an entire page of packets less than 256 bytes each was dropped because the driver was unable to map a new page to replace it.
rx_hdr_pkts	Number of packets received that were less than 256 bytes.
rx_inits	Number of packets received
rx_jumbo_pkts	Number of jumbo packets received.
rx_len_mm	Receive length error count.
rx_msgdup_fail	Number of times driver failed to duplicate a packet on receipt.
rx_mtu_drops	Number of times an entire page of packets greater than 256 bytes and less than 1514 each was dropped because the driver was unable to map a new page to replace it.
rx_new_hdr_pgs	Number of pages that were filled with packets less than 256 bytes that got replaced during reception.
rx_new_mtu_pgs	Number of pages that were filled with packets greater than 256 bytes and less than 1514 that got replaced during reception.
rx_new_nxt_pgs	Number of pages that contained packets that were split across pages that got replaced during reception.
rx_new_pages	Number of pages that got replaced during reception.
rx_no_buf	Number of times the hardware cannot receive data because there is no more receive buffer space.
rx_nocanput	Number of packets dropped due to full STREAMS message queue.
rx_no_comp_wb	Number of times the hardware cannot post completion entries for received data.
rx_nxt_drops	Number of times a page with a split packet was dropped because the driver was unable to map a new one to replace it.
rx_ov_flow	Number of packets dropped due to full h/w FIFO.
rx_pkts_dropped	Number of packets dropped due to service FIFO.
rx_reused_pgs	Number of times driver able to reuse a system page.
rx_split_pkts	Number of packets that were split across two pages.
snaptime	System snapshot time in seconds.
trunk_mode	A value of 1 indicates that a link is aggregated.
tx_allocb_fail	Number of times driver failed to duplicate a packet on transmit.
tx_ddi_pkts	Number of packets that used DDI_DMA.
tx_dma_bind_fail	Number of times a page table entry was not available to enable the driver to map the kernel memory to device-accessible memory for transmission.
tx_dvma_pkts	Number of packets that used dvma.
tx_hdr_pkts	Number of packets that used bcopy (buffer copy).
tx_jumbo_pkts	Number of packets greater that Ethernet MTU of 1522.
tx_max_pend	Maximum number of transmits pending on any of the four queues.
tx_msgdup_fail	Number of times driver failed to duplicate a packet for promiscuous stream.
tx_no_desc	Number of tx packets dropped due to full descriptor ring.
`tx`_`starts`	Number of times that the driver attempted to transmit a packet.
xcvr_addr	GMII/MII physical layer device address for management interface.
xcvr_id	GMII/MII physical layer device Identification Decimal copy of MII registers 2 and 3.
xcvr_inits	Number of physical layer re-initializations incremented each time link parameters are changed using NDD.
xcvr_inuse	Type of Transceiver/SERDES which is always PCS (3).

To Check Link Partner Settings

Type the kstat command as superuser:

CODE EXAMPLE 3-1 Output from `kstat` Command
# `kstat ixge:1` module: ixge instance: 1 name: ixge1 class: net brdcstrcv 0 brdcstxmt 0 cap_1000fdx 0 cap_1000hdx 0 cap_100fdx 0 cap_100hdx 0 cap_100T4 0 cap_10fdx 0 cap_10gfdx 1 cap_10ghdx 0 cap_10hdx 0 cap_asmpause 0 cap_autoneg 0 cap_pause 0 crc_err 0 crtime 147.15892392 ierrors 2 ifspeed 10000000000 ipackets 0 ipackets64 0 lb_mode 0 link_asmpause 0 link_duplex 2 link_pause 0 link_speed 10000 link_T4 0 link_up 1 local_faults 1 mac_mtu 0 multircv 0 multixmt 0 norcvbuf 0 noxmtbuf 0 obytes 0 obytes64 0 oerrors 0 opackets 0 opackets64 0 pci_bad_ack_err 0 pci_bus_speed 100 pci_dmarz_err 0 pci_dmawz_err 0 pci_drto_err 0 pci_err 0 pci_parity_err 0 pci_rma_err 0 pci_rta_err 0 pcix_bus_mode 1 promisc off qos_mode 0 rbytes 0 rbytes64 0 remote_faults 1 rev_id 28726 rx_allocb_fail 0 rx_hdr_drops 0 rx_hdr_pkts 0 rx_inits 0 rx_jumbo_pkts 0 rx_len_mm 0 rx_msgdup_fail 0 rx_mtu_drops 0 rx_mtu_pkts 0 rx_new_hdr_pgs 0 rx_new_mtu_pgs 0 rx_new_nxt_pgs 0 rx_new_pages 0 rx_no_buf 0 rx_nocanput 0 rx_no_comp_wb 0 rx_nxt_drops 0 rx_ov_flow 0 rx_pkts_dropped 0 rx_reused_pgs 0 rx_split_pkts 0 rx_tag_err 0 rx_taskq_waits 0 snaptime 7883.09014984 trunk_mode 0 tx_allocb_fail 0 tx_ddi_pkts 0 tx_dma_bind_fail 0 tx_dvma_pkts 0 tx_hdr_pkts 0 tx_inits 0 tx_jumbo_pkts 0 tx_max_pend 0 tx_msgdup_fail 0 tx_nocanput 0 tx_no_desc 0 tx_starts 0 tx_uflo 0 xcvr_addr 0 xcvr_id 0 xcvr_inits 1 xcvr_inuse 3

CODE EXAMPLE 3-1 Output from kstat Command


# kstat ixge:1
module: ixge                            instance: 1
name:   ixge1                           class:    net
        brdcstrcv                       0
        brdcstxmt                       0
        cap_1000fdx                     0
        cap_1000hdx                     0
        cap_100fdx                      0
        cap_100hdx                      0
        cap_100T4                       0
        cap_10fdx                       0
        cap_10gfdx                      1
        cap_10ghdx                      0
        cap_10hdx                       0
        cap_asmpause                    0
        cap_autoneg                     0
        cap_pause                       0
        crc_err                         0
        crtime                          147.15892392
        ierrors                         2
        ifspeed                         10000000000
        ipackets                        0
        ipackets64                      0
        lb_mode                         0
        link_asmpause                   0
        link_duplex                     2
        link_pause                      0
        link_speed                      10000
        link_T4                         0
        link_up                         1
        local_faults                    1
        mac_mtu                         0
        multircv                        0
        multixmt                        0
        norcvbuf                        0
        noxmtbuf                        0
        obytes                          0
        obytes64                        0
        oerrors                         0
        opackets                        0
        opackets64                      0
        pci_bad_ack_err                 0
        pci_bus_speed                   100
        pci_dmarz_err                   0
        pci_dmawz_err                   0
        pci_drto_err                    0
        pci_err                         0
        pci_parity_err                  0
        pci_rma_err                     0
        pci_rta_err                     0
        pcix_bus_mode                   1
        promisc                         off
        qos_mode                        0
        rbytes                          0
        rbytes64                        0
        remote_faults                   1
        rev_id                          28726
        rx_allocb_fail                  0
        rx_hdr_drops                    0
        rx_hdr_pkts                     0
        rx_inits                        0
        rx_jumbo_pkts                   0
        rx_len_mm                       0
        rx_msgdup_fail                  0
        rx_mtu_drops                    0
        rx_mtu_pkts                     0
        rx_new_hdr_pgs                  0
        rx_new_mtu_pgs                  0
        rx_new_nxt_pgs                  0
        rx_new_pages                    0
        rx_no_buf                       0
        rx_nocanput                     0
        rx_no_comp_wb                   0
        rx_nxt_drops                    0
        rx_ov_flow                      0
        rx_pkts_dropped                 0
        rx_reused_pgs                   0
        rx_split_pkts                   0
        rx_tag_err                      0
        rx_taskq_waits                  0
        snaptime                        7883.09014984
        trunk_mode                      0
        tx_allocb_fail                  0
        tx_ddi_pkts                     0
        tx_dma_bind_fail                0
        tx_dvma_pkts                    0
        tx_hdr_pkts                     0
        tx_inits                        0
        tx_jumbo_pkts                   0
        tx_max_pend                     0
        tx_msgdup_fail                  0
        tx_nocanput                     0
        tx_no_desc                      0
        tx_starts                       0
        tx_uflo                         0
        xcvr_addr                       0
        xcvr_id                         0
        xcvr_inits                      1
        xcvr_inuse                      3

To Discover Device Capabilities

Use the kstat command to discover device capabilities.

# kstat ixge:1 | grep cap_
        cap_1000fdx                     0
        cap_1000hdx                     0
        cap_100fdx                      0
        cap_100hdx                      0
        cap_100T4                       0
        cap_10fdx                       0
        cap_10gfdx                      1
        cap_10ghdx                      0
        cap_10hdx                       0
        cap_asmpause                    0
        cap_autoneg                     0
        cap_pause                       0

To Discover Link Settings

Use the kstat command to discover link settings.

# kstat ixge:1 | grep link
        link_asmpause                   0
        link_duplex                     2
        link_pause                      0
        link_speed                      10000
        link_T4                         0
        link_up                         1

Configuration Checking

The jumbo frame configuration checking occurs at Layer 2 or Layer 3, depending on the configuration method.

Layer 2 Configuration Checking

You can view the MTU configuration of a ixge instance at any time using the kstat command. The kstat mac_mtu variable represents the complete size of the Ethernet frame, which includes the Ethernet header, maximum payload, and crc. This value is consistent with the definition of MTU presented by switch vendors.

Use the following command to get the information:.

# kstat ixge:0 | grep mac_mtu
        mac_mtu                         8192

Layer 3 Configuration Checking

You can check the Layer 3 configuration by invoking the ifconfig command on a preconfigured interface as shown in the previous examples:.

# ifconfig ixge138000
ixge138000: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu
8170 index 5
        inet 192.12.38.69 netmask ffffff00 broadcast 192.12.38.255
        ether 0:3:ba:c4:f7:1d