| Sun Netra CP3260 Blade Server User’s Guide |    
|
| C H A P T E R 5 |
|
Hardware Functional Descriptions |
This chapter contains the following sections:
The Netra CP3260 blade server is an ATCA node board based on the UltraSPARC T2 processor. It is hot-swappable to an ATCA midplane and supports dual 10/100/1000BASE-T Ethernet interfaces as Base interface, and 10-Gb XAUI or 1-Gb SERDES Ethernet interfaces as Fabric interfaces to support a redundant Dual Star topology.
FIGURE 5-1 is a block diagram of the Netra CP3260 blade server. The blade server functions can be broken down into subsystems consisting of:
FIGURE 5-1 Netra CP3260 Blade Server Block Diagram
The UltraSPARC T2 processor has eight SPARC physical processor cores. Each SPARC physical processor core has full hardware support for eight strands, two integer execution pipelines, one floating-point execution pipeline, and one memory pipeline. The eight SPARC cores are connected through a crossbar to an on-chip unified 4-Mbyte, 16 way associative L2 cache.
There are four on-chip memory controllers that interface directly to FB-DIMM memory and include eight FB-DIMM slots (one per channel, or two per memory controller). In addition, there are an on-chip PCI-Express I/O interface and two 10-Gb Ethernet ports. The UltraSPARC T2 processor is a highly integrated processor that implements the 64-bit SPARC V9 architecture.
All I/O is provided via the UltraSPARC T2 PCI-Express interface. A PLX switch provides the PCI-E interfaces. Dual Intel NICs (or network adapters) supply the two Ethernet interfaces to the front panel and the ARTM for management interfaces.
The Base interface extends from the Intel 82571 NIC connected to the PCI-E switch via a x4 connection. The PCI-E switch then connects to the Power PC and to the Base interface.
The Fabric interface uses the processor’s XAUI interface and connects to the midplane fabric interface.
A PCI-E-to-PCI bridge provides an on-board local PCI bus to a PCI-to-USB device that provides the USB ports. Two USB ports connect to the front panel and one USB port connects to an USB-to-IDE bridge. The USB-to-IDE bridge supplies the IDE interface for the Compact Flash.
The RMII port on the Power PC connects to the Base interface through an Ethernet switch.
ARTM support is routed to the ARTM through the Zone 3 connectors. Typical ARTM support includes:
Because Netra CP32x0 Advanced RTMs (ARTM) provide additional functionality, the following interfaces are also routed to the ARTM.
Power for the ARTMs is provided through the Zone 3 RTM power connector.
This section provides a brief description of the Netra CP3260 hardware modules.
The UltraSPARC T2 multicore processor is the basis of the Netra CP3260 blade server. The UltraSPARC T2 processor is based on chip multithreading (CMT) technology that is optimized for highly threaded transactional processing. The UltraSPARC T2 processor improves throughput while using less power and dissipating less heat than conventional processor designs. It is high performance, highly integrated processor that implements the 64-bit SPARC V9 architecture. On the Netra CP3260 blade server, the UltraSPARC T2 processor operates at 1.2 GHz and contains 16 Kbytes of instruction cache per core and 8 Kbytes of data cache per core (FIGURE 5-2).
Depending on the model purchased, the processor has six or eight UltraSPARC cores. Each core equates to a 64-bit execution pipeline capable of running eight threads. The result is that the 8-core processor handles up to 64 active threads concurrently and a 6-core processor handles up to 48 active threads concurrently.
FIGURE 5-2 UltraSPARC T2 Multicore Processor Block Diagram
The Electronic Fuse (Efuse) block within the UltraSPARC T2 processor contains configuration information that is electronically burned in manufacturing.
The UltraSPARC T2 processor provides 8 physical SPARC processor cores and each physical core is capable of supporting 8 threads for a total of 64 threads.
The UltraSPARC T2 provides a total of 4 Mbytes of L2 cache banked 8 ways.
The UltraSPARC T2 supports 4 FB-DIMM memory controllers, each controller capable of supporting 2 FB-DIMM memory channels. The UltraSPARC T2 connects directly to 8 DIMMS (one DIMM per channel) providing for a total of 8 DIMM slots.
See Section 5.2.2, Memory Subsystem for details on the Netra CP3260 memory design.
The UltraSPARC T2 provides the following I/O interfaces
The UltraSPARC T2 provides a PCI Express Unit (PEU) that implements the root complex behavior of the PCI -Express Base specification 1.0A.
The PEU supports x1, x2, x4, and x8 configuration at the data rate of 2.5Bb/s in each direction. The PEU also supports the lane reversal feature thus easing blade server routing restrictions.
The XAUI interface is a high speed point-to-point serial interface with four differential pairs for transmit (TX) and four differential pairs for receive (RX), operating at 3.125 Gbps.
The XAUI interface is the Fabric interface to the ATCA midplane. The XAUI interface is routed to the Zone 2 connector on the midplane (see Section A.4, Connectors and Pinout for details on board connector pinout assignments.
The heart of the Memory subsystem is the FB-DIMM memory. Data from the FB-DIMM memory controller is brought out on a high-speed serial bus that connects to a single memory DIMM per channel. The FB-DIMM provides an Advanced Memory Buffer (AMB) which converts this serial interface into the standard memory interface used by DDR2 memory.
The UltraSPARC T2 provides 4 FB-DIMM memory controllers with each controller supporting two memory channels.
The Netra CP3260 provides up to 8 FB-DIMM slots, two per memory controller. This limits the maximum memory to 16 Gbytes if 2-Gbyte DIMMs are used, or 8 GBytes if 1-Gbyte DIMMs are used.
The target speed for FB-DIMMS is 667 MHz DDR2 RAMs which results in a channeled speed of 4.0 Gbps.
The Netra CP3260 I/O subsystem comprises of the following main features:
The Netra CP3260 uses the PLX PCI switch that encompasses a 32-lane PCI Express switch with six configurable ports (x1, x2, x4, x8, x16).
The Netra CP3260 provides dual redundant 10/100/1000BASE-T Ethernet links to connect to the midplane Base interface channels. A x4 lane port from the PCI Express switch connects to a dual gigabit controller.
The dual XAUI ports from UltraSPARC T2 processor are multiplexed between the midplane Zone 2 connector (that is, the fabric interface). The XAUI interface operate at 10-Gbps or 1-Gbps.
The Netra CP3260 provides a Zone 3 interface to a common ARTM which essentially appears as an AMC card. The common ARTM supports the AMC.1 and AMC.2 compliant interfaces.
An X8 PCI Express channel is routed to the RTM slot.
The ARTM interface provides an Ethernet SERDES interface for AMC.2 compliant functionality. These ports are sourced from the dual Gigabit Ethernet controller that connects to the PCI Express switch using a x4 lane PCI-E channels. (See Section 5.2.4, Other ARTM Interfaces for other non-AMC interfaces routed to the ARTM.)
The common ARTMs require two voltages:
Power management is handled by the ATCA shelf manager’s Intelligent Platform Management Controller (IPMC).
In addition to the AMC-type interface to the ARTM, additional interfaces are routed to the ARTM to provide for rear access to the Netra CP3260 I/O.
The console port from the PPC is routed to the ARTM for rear access. When a serial device is connected into the ARTM’s serial port, the console data is output to that port. When the ARTM serial port is not used, console data is routed to the H8 for netconsole or to the front panel serial port provided a serial device is connected into the port.
A single 10/100/1000BASE-T Ethernet port is routed from the Intel 82671 Dual Gigabit Ethernet controller to rear panel RJ-45 connectors. A second port goes to the front panel RJ-45 management port.
The front panel I/O consists of the following:
A single 10/100/1000BASE-T Ethernet port is routed from the Intel 82671 Dual Gigabit Ethernet controller to front panel RJ-45 connector. A second management port goes to the ARTM.
The serial console port from the PPC is routed to the front panel and terminated in an RJ-45 connector.
Dual USB ports are 2.0 compliant and routed from the USB hub to the front panel.
The Netra CP3260 blade server provides a Type I/II compact flash socket for removal media. The Compact Flash socket interface is derived from the USB-to-IDE controller.
The Netra CP3260 blade server includes a MPC885 service processor subsystem used for Logical Domains (LDoms) configuration and Hypervisor interaction, host system reset, and boot support. The Netra CP3260 blade server uses the MPC885 to run the vBSC firmware (on VxWorks).
The following I/O interfaces provided by MPC885 are used by the Netra CP3260 blade server:
The Field-Programmable Gate Array (FPGA) serves as a gateway between the UltraSPARC T2 and the MPC subsystem, and provides support for the IPMC. The FPGA provides the following functionality:
The FPGA configuration is performed after an FPGA reset when the configuration is downloaded from the PROM.
The Renesas H8S/2166 provides the IPM controller (IPMC) function on the Netra CP3260 blade server. The IPMC provides PICMG 3.0 blade server management functionality, and it interfaces to the host CPU through a serial interface. The IPMC subsystem is powered from the standby power.
The IPMC is responsible for the following:
The BMR-H8S provides dual buffered Intelligent Platform Management Bus (IPMB) interfaces to the IPMB-0 bus on the PICMG 3.0 midplane. The I2C channels on the H8S are connected the IPMB-A and IPMB-B through the I2C buffers. The I2C buffers allow the blade server I2C to be isolated from the midplane until the blade server is fully seated and the I2C bus on the midplane is idle.
The BMR-H8S provides one serial payload interface to the PPC. This interface supports hardware flow control, RTS (Request To Send) and CTS (Clear To Send).
The H8S provides a local IPMI interface wired to the ARTM’s MMC (IPMB-L). The MMC IPMI interface is isolated through FET and controlled by H8 GPIO pins.
Hot-swap is supported by monitoring of the hot-swap handle switch. The handle switch goes directly to one of the GPIO pins on the H8S (PE5).
The hardware supports both non-graceful shutdown and graceful shutdown of payload in response to a hot-swap event. In case of non-graceful shutdown, firmware on the IPMC disables the voltage rails before turning the blue LED on.
Optionally the IPMC can send a message across the serial payload interface to the PPC.
The Netra CP3260 blade server supports three LEDs compliant with ATCA specification:
The LEDs are controlled by H8.
The BMR-H8S is able to control (enable/disable) power rails to the payload.
The Analog Devices ADM1026 is used for system monitoring functions. The ADM1026 is interfaced with the IPMC, and the IPMC firmware is responsible for monitoring these sensors.
The ADM1026 measures most of the blade server voltages. The ADM1026 has one temperature sensor embedded in the device itself and supports two remote sensing channels that require external diodes for temperature sensing. These two remote sensors are used for measuring the CPU thermal diode temperature.
The ADM1026 measures the voltages listed in TABLE 5-1. The ADM1206 and H8 firmware monitors these voltages. When the voltages are within regulations, the blade server functions with no warnings. When any voltage goes out of regulation beyond approximately ± 7%, a Critical Warning is generated and presented to the shelf manager.
The ADM1026 also monitors the CPU diode temperature on the Netra CP3260 blade server. The ADM1206 and H8 firmware reports a minor, major, or critical alarm when the temperature of the CPU goes beyond the thresholds listed in TABLE 5-2. Also, if the temperature of the CPU goes beyond the Emergency H8 Shutdown value listed in TABLE 5-2, the H8 initiates a event/warning blade server to the shelf manager and shuts down.
The H8 temperature alarm equivalents are:
Minor Alarm = Upper Non Critical (UNC)
Major Alarm = Upper Critical (UC)
Critical Alarm = Upper Non Recoverable (UNR)
Emergency H8 Shutdown (EMR) = UNR
|
Caution - These voltage and temperature thresholds should not be changed under normal operating conditions. |
There are two FRUID PROMs on the Sun Netra CP3260 and both are 64-Kbyte SEEPROMs. One contains Sun FRU information and the other contains IPMI FRU information. Each device stores static information, such as the part number and manufacturing date, and dynamic data that can be updated by the host system, such as operational statistics and failure information.
The UltraSPARC T2 provides a PCI Express reset out on the PEX_RESET_L pin. This pin connects to the FPGA and in turn, this is used by the FPGA to reset the I/O subsystem which includes the PLC PCI Express switch.
The PCIe switch in turn provides an inband reset (output pin) which, via the FPGA, resets the:
In addition to the processor resetting the I/O subsystem, each I/O component has reset software write-able by the PPC to reset individual components.
The Ethernet switch is reset-able via software by the PPC. This allows the Ethernet path between the PPC and the Base to remain active in the event the processor is being reset.
A reset mask register is provided in the FPGA to allow the masking of resets to individual I/O components. For example, a PEX_RESET_L reset from the UltraSPARC T2 processor resets only those I/O subcomponents not masked by the reset mask register.
The Netra CP3260 blade server uses the Artesyn ATCA power module solution. The Artesyn power module provides an integrated ATCA power solution that meets PICMG 3.0 requirements, including dual bus input, DC isolation, hold up, hot-plug, and management power (3.3V standby). It provides a 12V intermediate bus as backend power. Some of the salient features of the module are:
The TOD clock battery must be type CR1632, with a minimum of 4ma abnormal charging current rating (for example; a Renata CR1632).
|
Caution - Risk of explosion if battery is replaced by an incorrect type. |
Dispose of batteries properly in accordance with manufacturer’s instructions and local regulations.
See Section 2.5.3, Adding or Replacing TOD Clock Battery for battery location and installation instructions.
| Sun Netra CP3260 Blade Server User’s Guide | 820-0457-11 |
|
Copyright © 2009 Sun Microsystems, Inc. All rights reserved.