JavaScript is required to for searching.
Skip Navigation Links
Exit Print View
Sun Blade 6000 Virtualized 40 GbE Network Express Module

User's Guide

search filter icon
search icon

Document Information

Using This Documentation

Related Documentation

Feedback

Support and Accessibility

Understanding the NEM

Task Overview

Product Features

Understanding the Components

SAS Connections

Ethernet Connections

10GbE NIC Virtualization

Fabric Express Modules

Processor

Identifying Ports and LEDs

Connector Locations

External NEM Ports

RJ-45 Ethernet Connector Port LEDs

NEM Front Panel and LEDs

Performing Hot-plug Insertion and Removal

Performing PCIe Hot-plug

Understanding PCIe Hot-plug

Prepare the PCIe Interface for Hot-plug Removal

Performing SAS Interface Hot-plug

Prepare SAS Interfaces for Hot-plug Removal

Restoring SAS Configurations After NEM Installation

Saving the Zoning Configuration to a Backup File

Recovering Zoning Configurations

Installing or Replacing a NEM

Before Installing a NEM

Determine the CMM Part Number Using ILOM 3.0x Web Interface

Determine the CMM Part Number Using ILOM 3.1x Web Interface

Determine the CMM Part Number Using ipmitool

Find the CMM Part Number on the CMM Label

Install a NEM

Verifying NEM Installation

Verify Installation (CMM ILOM Web)

Verify Installation (CMM ILOM CLI)

Remove a NEM

Replace a NEM

Installing and Removing SFP+ Optical Transceiver Modules

Install an SFP+ Optical Transceiver Module

Remove an Optical Transceiver SFP+ Module

SFP+ Connector Cabling Options

Enabling Jumbo Frames

Configuring Jumbo Frames (Solaris)

Jumbo Frames Overview

Jumbo Frames Configurations and Driver Statistics

Enable Jumbo Frames (Solaris)

Check Layer 2 Configuration

Check Layer 3 Configuration

Configuring Jumbo Frames (Linux)

Temporarily Configure Jumbo Frames (Linux)

Permanently Enable Jumbo Frames (Linux)

Enable Jumbo Frames (Windows)

Configuring Jumbo Frames (VMware ESX)

Create a Jumbo Frames-Enabled vSwitch (VMWare)

Create a Jumbo Frames-Enabled VMkernel Interface (VMWare

Configuring Static Link Aggregation

Example (Oracle Solaris 11 OS)

Example (Oracle Solaris 10 OS)

Example (Linux OS)

Booting an x86 Server Over the Network

Booting the NEM on an x86 Blade Server Over the Network

Set Up the BIOS for Booting From the NEM

Use the F12 Key to Initiate the Network Boot

Use the F8 Key to Boot Off the PXE Server

Boot a SPARC Blade Over the Network

Installing Drivers on Oracle Solaris OS SPARC or x86 Platform

Download the sxge Driver

Configure the Network Host Files

Configure the sxge Device Driver Parameters

Installing Drivers on a Linux Platform

Downloading, Installing, and Removing Drivers

Download and Install the Drivers

Remove the Drivers

Configuring the Network Interface

Temporary Network Interface Configuration

Permanent Network Interface Configuration

Bring the Interface Online Manually

Configure the Network Interface File Automatically for Red Hat Linux

Configure the Network Interface Automatically for SUSE Linux

Device Checking and Testing

Configure the Syslog Parameter

Installing Drivers on a Windows Platform

Installing Drivers on a Windows Platform

Installing and Uninstalling the Network Controller

Installing and Uninstalling the Sun Blade 6000 40 GbE VLAN Driver (Optional)

Adding and Removing VLANs

Install the Enclosure Device on a Windows Server 2008 System

Dynamic and NIC Teaming

Installing Drivers on a VMware ESX Server Platform

Installing ESX Server Drivers

Install a NEM Driver for a New ESX4.0 Installation

Install a NEM Driver on an Existing ESX 4.0 Server

Install a NEM Driver on an ESX/ESXi Server

Configure the NEM Network Adapters

Uninstalling the Drivers

Configuring Blade and Port Parameters and Managing the NEM

Configuring and Viewing Parameters

Defaults and Options

Setting Blade Parameters

Setting Port Parameters

Saving User Configurations or Resetting to Defaults

Viewing Parameter Settings, Firmware Versions, and Configuration File

Configuring NEM for Oracle VM Para-Virtualization

Configure Para-Virtualization Without Live Migration

Configure Para-Virtualization With Live Migration

Managing the NEM Using CMM ILOM

ILOM Documentation and Updates

Start NEM CLI from CMM ILOM CLI

Updating the NEM Firmware

Configuring Hot Plug Service

Verify and Enable Hot Plug (Oracle Solaris OS)

Verify and Enable Hot Plug (Linux OS)

Disabling Interfaces

Updating Firmware (Web)

Obtain and Apply the Newest Firmware

Reset the NEM (Web)

Updating the Firmware (CLI)

Update NEM ILOM Firmware (CLI)

Reset the NEM (CLI)

Glossary

Index

Download and Install the Drivers

The following instructions apply to both drivers, sxge and sxgevf. When downloading and installing, specify which driver you want.

To use the sxgevf driver, you need to have the sxge driver already installed.

  1. Use the ifconfig command to obtain a list of the current Ethernet network interfaces.
    host #> ifconfig -a | grep eth
    eth0      Link encap:Ethernet  HWaddr 00:14:4F:CA:15:68  
    eth1      Link encap:Ethernet  HWaddr 00:14:4F:CA:15:69  

    After you have installed the NEM and installed and loaded a driver, a new eth device is displayed. This is the eth device for the NEM.

  2. Access the sxge drivers through one of the following methods:
    • Locate and copy the sxge device driver.zip or sxgevf device driver.zip file from the Tools and Drivers CD for the server module operating system on which you want to install the driver.

      For Oracle Enterprise Linux, use the corresponding Red Hat Enterprise Linux driver.

      The drivers are located in the following directories:

      /linux/drivers/rhel5.4

      /linux/drivers/sles10sp3

      /linux/drivers/sles11


      Note - Linux source files are available at /linux/drivers/src.


    • Download latest driver packages from the the following web site:

      http://support.oracle.com

      1. At the My Oracle Support site, log in.

        An account is required to download software. You must register if you do not have an account.

      2. Click the Patches & Updates tab.
      3. Click “Product or Family” (Advance search).
      4. In the Product field, type Sun Blade 6000 Virtualized 40GbE NEM.
      5. In the Release field, select the newest release, then click the arrow next to the Sun Blade 6000 Virtualized 40GbE NEM folder.
      6. Click the Search button.
      7. Download the latest available firmware package, and extract it to an accessible folder on the network.
  3. If you are downloading the sxgevf driver, make sure that SR-IOV mode is enabled before installing the driver. (See Setting Blade Parameters.)
  4. Once the Linux drivers package is downloaded, uncompress and unpack the driver and other files.

    In the following example, the download directory is tge10:

    host #>mkdir tge10
    host #>cp xxx/p12918641_100_Generic.zip
    host #>cd tge10
    host #>unzip p12918641_100_Generic.zip
    host #>ls
    readme.html
    Sun_Blade_6000_Virtualized_40GbE_NEM_1_0_4.xml
    Sun_Blade_6000_Virtualized_40GbE_NEM_1_0_4.zip
    host #>unzip Sun_Blade_6000_Virtualized_40GbE_NEM_1_0_4.zip
    host #>ls
    host #>firmware fpga  license  linux  solaris  windows  release_note
  5. Select and install the appropriate OS driver package.

    In the following example, RHEL6.0 Linux driver is selected for REHL version 6.0.

    1. Navigate to the Linux directory that contains the driver package that is appropriate for your OS.
      host #> cd linux/RHL6.0
      host #>ls
      kmod-sxge-1.0-0.x86_64.rpm sxge-debuginfo-1.0-0.x86_64.rpm
      kmod-sxgevf-1.0-0.x86_64.rpm  sxgevf-debuginfo-1.0-0.x86_64.rpm

      In the following example, the file kmod-sxgevf-1.0-0.x86_64.rpm is the package that will be installed.

      If you are not sure which release your system is running, use the lsb_release command to show information about your host's operating system.

      host #> lsb_release -a
      LSB Version:
      core-4.0-intel64:core-4.0-noarch:graphics-4.0-intel64:graphics-
      4.0-noarch:printing-4.0-intel64:printing-4.0-noarch
      Distributor ID: RedHatEnterpriseServer
      Description: Red Hat Enterprise Linux Server release 6.0 (Santiago)
      Release: 6.0
      Codename: Santiago
      host #> rpm -q kmod-sxgevf-1.0-0.x86_64
      package kmod-sxgevf-1.0-0.x86_64 is not installed
    2. Identify the OS-specific subdirectory, and verify that the driver you want to install (sxge or sxgevf) is not currently installed.

      Note - If the driver is already installed, uninstall the driver to avoid complications. See Remove the Drivers for instructions. The update command (rpm -u) is not supported for updating the drivers.


    3. Install the appropriate package (.rpm) file.
      host #> rpm -ivh --nodeps kmod-sxge-1.0-0.x86_64.rpm  
      Preparing...                ########################################### [100%]
           1:kmod-sxge                   ########################################### [100%]
      post Install Done
      host #> rpm -ivh --nodeps kmod-sxgevf-1.0-0.x86_64.rpm  
      Preparing...                ########################################### [100%]
           1:kmod-sxgevf ######################################### [100%]
      post Install Done

      Note - To load and use the sxgevf driver, you must have the sxge driver already installed.


  6. Load the driver.
    1. Verify that the NEM is available to the system (that is, it is actively on the PCIe I/O bus).

      The following command sample has been edited for brevity. Lines marked [....] denote output that has been removed.

      host #> lspci 
      [...]
      13:00.0 PCI bridge: Oracle Corporation Unknown Device 2070
      14:02.0 PCI bridge: Oracle Corporation Unknown Device 2070
      14:03.0 PCI bridge: Oracle Corporation Unknown Device 2070
      15:00.0 Ethernet controller: Sun Microsystems Computer Corp. 
      Unknown device 207a (rev 01)
      15:00.2 Ethernet controller: Sun Microsystems Computer Corp. 
      Unknown device 207b (rev 01)
      15:00.3 Ethernet controller: Sun Microsystems Computer Corp. 
      Unknown device 207b (rev 01)
      16:00.0 Ethernet controller: Sun Microsystems Computer Corp. 
      Unknown device 207a (rev 01)
      16:00.1 Ethernet controller: Sun Microsystems Compute Corp. 
      Unknown device 207b (rev 01)
      16:00.2 Ethernet controller: Sun Microsystems Computer Corp. 
      Unknown device 207b (rev 01)
      16:00.3 Ethernet controller: Sun Microsystems Computer Corp. 
      Unknown device 207b (rev 01)
      [...]
      host #> lspci
      [...]
      19:00.0 PCI bridge: Sun Microsystems Computer Corp. 
      Unknown device 2070 (rev 01)
      1a:02.0 PCI bridge: Sun Microsystems Computer Corp. 
      Unknown device 2070 (rev 01)
      1a:03.0 PCI bridge: Sun Microsystems Computer Corp. 
      Unknown device 2070 (rev 01)
      1b:00.0 Ethernet controller: Sun Microsystems Computer Corp. 
      Unknown device 2078 (rev 01)
      1c:00.0 Ethernet controller: Sun Microsystems Computer Corp. 
      Unknown device 2078 (rev 01)
      [...]
      • Device code 0x2078 is the NEM device when used as a regular PCI device.

      • Device code 0x207a is the PF (using sxge driver) NEM device when SR-IOV is enabled.

      • Device code 0x207b is the VF (using sxgevf driver) NEM device when SR-IOV is enabled.

        Presence of this line indicates the NEM is visible and available to the system.

    2. Manually load the driver.
      host #> modprobe sxge or sxgevf
    3. Verify the driver is loaded.
      host #> lsmod | grep sxge
      sxge 71304 0  
      host#> modinfo sxge
      filename:       /lib/modules/2.6.18-238.el5/extra/sxge/sxge.ko
      version:        0.06032011
      license:        GPL
      description:    SXGE ethernet driver
      author: 
      srcversion:     73779771D7EA5154342838B
      alias:          pci:v0000108Ed0000207Asv*sd*bc*sc*i*
      alias:          pci:v0000108Ed00002078sv*sd*bc*sc*i*
      depends:        
      vermagic:       2.6.18-238.el5 SMP mod_unload gcc-4.1
      parm:           debug:SXGE debug level (int)
      [etc.]
      host #> lsmod | grep sxgevf
      sxge (or sxgevf) 71304 0  
      host#> modinfo sxgevf
      filename:       /lib/modules/2.6.18-238.el5/extra/sxgevf/sxgevf.ko
      version:        0.06032011
      license:        GPL
      description:    SXGEVF ethernet driver
      author: 
      srcversion:     73779771D7EA5154342838B
      alias:          pci:v0000108Ed0000207Asv*sd*bc*sc*i*
      alias:          pci:v0000108Ed00002078sv*sd*bc*sc*i*
      depends:        
      vermagic:       2.6.18-238.el5 SMP mod_unload gcc-4.1
      parm:           debug:SXGEVF debug level (int)
      [etc.]
      • If you get the output shown in the previous examples, the drivers are loaded into memory and actively running.
      • If the modprobe command fails, you will see the following output:
        host #> modprobe sxge or sxgevf
        FATAL: Module sxge not found. 
        FATAL: Module sxgevf not found. 

        This output indicates that you probably installed the wrong driver version. Uninstall the driver, then install the correct package for your Linux release.

        If you are running a custom or patched kernel, you might have to build a custom driver to match your custom kernel.

  7. Identify the NEM eth device.

    After the NEM has been correctly installed, and the driver has been successfully installed and loaded, the new NEM eth device will be visible.

    1. Execute the following command to view the available eth devices.
      host #> ifconfig -a | grep eth 
      eth0      Link encap:Ethernet  HWaddr 00:14:4F:CA:15:68  
      eth1      Link encap:Ethernet  HWaddr 00:14:4F:CA:15:69  
      eth2      Link encap:Ethernet  HWaddr 00:14:4F:29:00:1D  

      In this example, eth0 and eth1 were previously present. The eth2 device is for a new NEM. You can identify each eth device (NEM0 or NEM1) by matching the Ethernet MAC address with the one you recorded and saved when you physically installed the NEM into the chassis in Install a NEM.

    2. Make sure that the eth2 driver is the correct Ethernet driver for the NEM.
      host #> ethtool -i eth2 
      driver: sxge
      version: 0.06032011
      firmware-version: 
      bus-info: 0000:1b:00.0
      driver: sxgevf
      version: 0.01202011
      firmware-version: 
      bus-info: 0000:1c:00.0
    3. For more detail on eth3, use the ifconfig command.
      host #> ifconfig eth2 
      Link encap:Ethernet  HWaddr FA:CE:00:00:FC:0F 
      BROADCAST MULTICAST  MTU:1500  Metric:1
      RX packets:0 errors:0 dropped:0 overruns:0 frame:0
      TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1000 
      RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
      Interrupt:194

      The eth2 device is active and available to the system, but has yet to be configured (assigned an IP address). See Configuring the Network Interface for details on configuring the NEM for the Linux OS.

  8. Load the VF driver to the guest OS.

    Refer to the Oracle VM Server for x86 XEN documentation.

    http://docs.oracle.com/cd/E15458_01/index.htm

Related Information