SAS Connections

Each Virtualized Multi-Fabric 10GbE NEM has 10 SAS x2 connections to the modules in the chassis through the chassis mid-plane. There is one SAS x2 connection for each module slot. External SAS connections are not currently supported.

One of the functions of the Virtualized Multi-Fabric 10GbE NEM SAS expander is to provide zoning for the disks that are visible to the server modules in the chassis. This includes disks on the server modules, disks on disk modules, and external disks. Firmware on the Virtualized Multi-Fabric 10GbE NEM SAS expander controls zoning (that is, which disks can be seen by each individual server module).

The default zoning is such that the server modules and disk modules must be placed in the chassis in pairs. The pairs can be in slots 0+1, 2+3, 4+5, 6+7, or 8+9. Other pairings, such as 1+2 or 5+6, do not couple the server and storage modules.

Thus, in the recommended use of default zoning, a server module in an even-numbered slot, n, sees all the disks in slot n+1, but no other disks. FIGURE 1-1 shows the default zoning configuration for server modules and disk modules.

The disk module that connects to a server module in slot n (where n = 0, 2, 4, 6, or 8) must be in slot n+1.

For more information on the Sun Blade 6000 disk module, refer to the documentation site at: (http://docs.sun.com/app/docs/prod/blade.6000disk#hic)

Ethernet Connections

The Virtualized Multi-Fabric 10GbE NEM provides the magnetics and RJ-45 connectors for the ten 10/100/1000 BASE-T Ethernet interfaces that come from Sun Fire 6000 server modules through the midplane. The Sun Blade 6000 Virtualized Multi-Fabric 10GbE NEM has one 10/100/1000BASE-T Ethernet port for each server module slot. There is no active circuitry on the Virtualized Multi-Fabric 10GbE NEM for these GbE ports.

The Virtualized Multi-Fabric 10GbE NEM also provides a 10 GbE connection for each server module. Each server module appears to have its own 10 GbE NIC through the Virtualized NEM ASICs. All the server modules share two physical small form-factor pluggable (SFP+) 10 GbE ports via a dual channel 10 GbE Serializer/Deserializer (SerDes) per ASIC. In a Sun Blade 6000 chassis, five server modules connect to a single Virtualized NEM ASIC and share its 10 GbE port. Two Virtualized NEM ASICs can be configured so that the ten server modules share one 10 GbE port for simplified cable aggregation.

See 10 GbE NIC Virtualization for further information on the NEM 10 GbE virtualization.

FIGURE 1-2 shows a schematic of the NEM Ethernet connections.

10 GbE NIC Virtualization

This section provides information on the 10 GbE NIC functionality for the Virtualized Multi-Fabric 10GbE NEM.

This section covers the following topics:

• Virtualized NEM ASIC

• Enabling or Disabling the IAL

• Fabric Express Modules

Virtualized NEM ASIC

The Virtualized NEM ASIC allows up to five hosts to share a single 10 GbE network port, with a dedicated PCIe endpoint for each host.

Shared I/O allows each server module to work as if a dedicated NIC connects the server module to the network. Each server module owns a virtual MAC that provides per-server-module statistics on Rx/Tx traffic. The MAC that interfaces to the 10 GbE network port is shared and is hidden from the server modules. Only the service processor can access and configure this port.

A Virtualized Multi-Fabric 10GbE NEM contains two Virtualized NEM ASICs, which can operate in two different modes:

• Bandwidth Mode

• Connectivity Mode

Bandwidth Mode

The Virtualized NEM ASICs can operate in bandwidth mode where the Virtualized NEM ASICs act without knowledge of each other or connectivity (IAL) mode where all server modules share a single 10GbE uplink. Each ASIC provides 10 GbE network access for the five hosts attached to it.

FIGURE 1-3 shows how the ASICs operate in bandwidth mode.

FIGURE 1-3   Blade Network Topology With 2 x 10 GbE

Connectivity Mode

Two Virtualized NEM ASICs can also be interconnected so that a single 10 GbE port can act as the shared I/O for 10 server modules (connectivity mode). The inter-ASIC link (IAL) extends the virtual Ethernet segment to all the server modules. An illustration of this configuration is shown in FIGURE 1-4.

FIGURE 1-4   Blade Network Topology WIth 1 x 10 GbE

Because of the symmetry of the IAL, either Virtualized NEM ASIC can be connected to the external network. This configuration can be used for higher aggregation per 10 GbE port (1x10 GbE for 10 server modules).

Service Processor

The Virtualized Multi-Fabric 10GbE NEM has an Aspeed AST2000 controller as its service processor (SP), which is responsible for the control and management of the Virtualized NEM ASICs. The NEM also has a 10/100BASE-T Ethernet management port for connectivity to the Sun Blade 6000 CMM.

The following list describes the features of the service processor:

• The AST2000 controller has a 200MHz ARM9 CPU core and a rich set of features and interfaces. The BCM5241 receives a 25MHz reference clock.

• A Broadcom BCM5241 10/100BASE-T Ethernet PHY connected to the AST2000 controller’s 10/100M Fast Ethernet MAC provides the Ethernet management interface to the CMM.

• The SP can update Sun Blade 6000 Virtualized Multi-Fabric 10GbE NEM FPGA and firmware using GPIOs connected to the JTAG port of the FPGA.

Refer to Appendix B for more information on the Integrated Lights Out Manager (ILOM) server management application for the SP.

External Virtualized NEM Ports

This section describes the external ports on the NEM.

RJ45 Ethernet Connector Port LEDs

Each RJ45 Ethernet port has two LEDs. The left LED is green and lights to show that a link has been established. It blinks off randomly whenever there is network activity on that port.

On the RJ-45 connectors, the right LED is bi-color (amber and green) and indicates the speed of connection by the color it displays. When the port is operating at 100 megabits per second, the right LED displays one color. When operating at 1000 megabits per second, it displays the other color. When operating at 10 megabits per second, the right LED is off. The green/amber color scheme varies from one server blade to another. TABLE 1-6 provides a chart for interpreting the link-speed relationships.

When an Ethernet port is connected to an x64 server blade (server blades with a model number that starts with X) that has been put in Wake-on-LAN (WOL) mode, the Link LED indicates when the system is in standby mode. It does this by blinking in a repeating, non-random pattern. It flashes ON for 0.1 second and OFF for 2.9 seconds. In standby mode, the system is functioning at a minimal level and is ready to resume full activity.