Connecting Multiple SPARC SuperCluster T4-4 Systems

SPARC SuperCluster T4-4 Owner's Guide

Document Information

Using This Documentation

Product Documentation Library

Feedback

Understanding the System

SPARC SuperCluster T4-4 Overview

Spares Kit Components

SPARC SuperCluster T4-4 Restrictions

Identifying Hardware Components

Full Rack Components

Half Rack Components

Understanding the Hardware Components and Connections

Understanding the Hardware Components

SPARC T4-4 Servers

Exadata Storage Servers

Sun ZFS Storage 7320 Appliance

Sun Datacenter InfiniBand Switch 36 Switches

Cisco Catalyst 4948 Ethernet Management Switch

Power Distribution Units

Understanding the Physical Connections

SPARC T4-4 Server Physical Connections

Card and Port Locations (SPARC T4-4 Servers)

InfiniBand Private Network Physical Connections (SPARC T4-4 Servers)

Oracle ILOM Management Network Physical Connections (SPARC T4-4 Servers)

1 GbE Host Management Network Physical Connections (SPARC T4-4 Servers)

10 GbE Client Access Network Physical Connections (SPARC T4-4 Servers)

Exadata Storage Server Physical Connections

InfiniBand Private Network Physical Connections (Exadata Storage Servers)

Oracle ILOM Management Network Physical Connections (Exadata Storage Servers)

1 GbE Host Management Network Physical Connections (Exadata Storage Servers)

Sun ZFS Storage 7320 Appliance Physical Connections

InfiniBand Private Network Physical Connections (Sun ZFS Storage 7320 Appliance)

Oracle ILOM Management Network Physical Connections (Sun ZFS Storage 7320 Appliance)

1 GbE Host Management Network Physical Connections (Sun ZFS Storage 7320 Appliance)

SAS Physical Connections (Sun ZFS Storage 7320 Appliance)

Cluster Physical Connections (Sun ZFS Storage 7320 Appliance)

Power Distribution Units Physical Connections

Understanding the SPARC SuperCluster T4-4 Configurations

Understanding Logical Domains and the Express Module Slots

Understanding the Management Network

Understanding the 10 GbE Client Access Network

Understanding the InfiniBand Network

InfiniBand Network Data Paths for a Database Domain

InfiniBand Network Data Paths for an Application Domain

Understanding Clustering Software

Cluster Software for the Database Domain

Cluster Software for the Oracle Solaris Application Domains

Example SPARC SuperCluster T4-4 Configuration

Database Domain

CPU, Memory, and PCI Root Complex Resource Allocation

InfiniBand Connections

At the SPARC T4-4 Server Level

At the SPARC SuperCluster T4-4 Level

10 GbE Client Access Connections

Application Domain Running Oracle Solaris 10

CPU, Memory, and PCI Root Complex Resource Allocation

InfiniBand Connections

At the SPARC T4-4 Server Level

At the SPARC SuperCluster T4-4 Level

10 GbE Client Access Connections

Understanding the Network Requirements

Network Requirements Overview

Network Connection Requirements for SPARC SuperCluster T4-4

Default IP Addresses (SPARC SuperCluster T4-4)

Default IP Addresses

Default Host Names and IP Addresses (Full Rack)

Default Host Names and IP Addresses for the Oracle ILOM and Host Management Networks (Full Rack)

Default Host Names and IP Addresses for the InfiniBand and 10 GbE Client Access Networks (Full Rack)

Default Host Names and IP Addresses (Half Rack)

Default Host Names and IP Addresses for the Oracle ILOM and Host Management Networks (Half Rack)

Default Host Names and IP Addresses for the InfiniBand and 10 GbE Client Access Networks (Half Rack)

Preparing the Site

Cautions and Considerations

Reviewing System Specifications

Physical Specifications

Installation and Service Area

Rack and Floor Cutout Dimensions

Reviewing Power Requirements

System Power Consumption

Facility Power Requirements

Grounding Requirements

PDU Power Requirements

Preparing for Cooling

Environmental Requirements

Heat Dissipation and Airflow Requirements

Perforated Floor Tiles

Preparing the Unloading Route and Unpacking Area

Shipping Package Dimensions

Loading Dock and Receiving Area Requirements

Access Route Guidelines

Unpacking Area

Preparing the Network

Network Connection Requirements

Network IP Address Requirements

Prepare DNS for the System

Installing the System

Installation Overview

Oracle Safety Information

Unpacking the System

Tools for Installation

Find the Unpacking Instructions

Unpack and Inspect the Machine

Moving the Rack Into Place

Move the SPARC SuperCluster T4-4

Install a Ground Cable (Optional)

Adjust the Leveling Feet

Powering on the System the First Time

Connect Power Cords to the Rack

Power On the SPARC SuperCluster T4-4

Connect a Laptop to the SPARC SuperCluster T4-4

Connect to a 10 GbE Client Access Network

Using an Optional Fiber Channel Express Module

Maintaining the System

Cautions and Warnings

Powering Off the System

Graceful Power Off Sequence

Shut Down the Exadata Storage Servers

Power Off the Exadata Storage Servers

Gracefully Shut Down LDoms

Emergency Power Off

Power On the System

Managing Oracle Solaris 11 Boot Environments

Advantages to Maintaining Multiple Boot Environments

Create a Boot Environment

Mount to a Different Build Environment

Reboot to the Original Boot Environment

Remove Unwanted Boot Environments

Using Dynamic Intimate Shared Memory

DISM Restrictions

Disable DISM

Component-Specific Service Procedures

Maintaining Exadata Storage Servers

Monitor Write-through Caching Mode

Shut Down an Exadata Storage Server

Drop an Exadata Storage Server

Tuning SPARC SuperCluster

Monitor ssctuner

Turn Off ssctuner

Disable Individual ssctuner Components

Configuring CPU and Memory Allocation

CPU/Memory Tool Overview

Supported CPU/Memory Allocations

Change CPU/Memory Allocations

Revert to a Previous CPU/Memory Configuration

Enabling and Disabling Capacity-on-Demand on SPARC SuperCluster

Monitoring the System

Monitoring the System Using Auto Service Request

ASR Overview

Preparing to Configure ASR

Prepare the ASR Environment

Software Requirements: ASR Manager

Software Requirements: SPARC SuperCluster T4-4

Installing ASR Manager Components

Install the Oracle Automated Service Manager (OASM) Package

Install Service Tools Bundle (STB) for Solaris Only

Install the Oracle Auto Service Request (ASR) Package

Verify ASR Manager

Configure SNMP Trap Destinations for the Exadata Storage Servers

Configure ASR on the Sun ZFS Storage 7320 Storage Appliance

Configure ASR on the SPARC T4-4 Servers: Oracle ILOM

Configuring ASR on the SPARC T4-4 Servers: Oracle Solaris 11

Enable the HTTP Receiver on the ASR Manager

Enable HTTPS on the ASR Manager (Optional)

Approve and Verify ASR Activation for SPARC SuperCluster T4-4 Assets

Monitoring the System Using OCM

OCM Overview

Install Oracle Configuration Manager on the SPARC T4-4 Servers

Monitoring the System Using EM Exadata Plug-in

Configuring Exalogic Software

Overview of Exalogic-Specific Enhancements

Exalogic Software Prerequisites

Enable Domain-Level Enhancements

Enable Cluster-Level Session Replication Enhancements

Configuring Grid Link Data Source for Dept1_Cluster1

Grid Link Data Source

Fast Connection Failover

Runtime Connection Load Balancing

XA Affinity

SCAN Addresses

Secure Communication using Oracle Wallet

Create a GridLink Data Source on Dept1_Cluster1

Configuring SDP-Enabled JDBC Drivers for Dept1_Cluster1

Configure the Database to Support Infiniband

Enable SDP Support for JDBC

Monitor SDP Sockets Using netstat on Oracle Solaris

Configuring SDP InfiniBand Listener for Exalogic Connections

Create an SDP Listener on the InfiniBand Network

Understanding Internal Cabling

Ethernet Management Switch

Ethernet Management Switch Connections in a Full Rack

Ethernet Management Switch Connections in a Half Rack

Sun ZFS Storage 7320 Appliance

Single-Phase PDUs

Three-Phase PDUs

Connecting Multiple SPARC SuperCluster T4-4 Systems

Multi-Rack Cabling Overview

Connecting Expansion Racks

Oracle Exadata Storage Expansion Rack Components

Preparing for Installation

Reviewing System Specifications

Physical Specifications

Reviewing Power Requirements

System Power Consumption

PDU Power Requirements

Preparing for Cooling

Environmental Requirements

Heat Dissipation and Airflow Requirements

Perforated Floor Tiles

Preparing the Unloading Route and Unpacking Area

Shipping Package Dimensions

Preparing the Network

Network Requirements Overview

Network Connection and IP Address Requirements

Installing the Oracle Exadata Storage Expansion Rack

Default IP Addresses

Cabling Tables (Internal)

Front and Rear Views of the Oracle Exadata Storage Expansion Rack

Oracle ILOM Cabling Tables

Administrative Gigabit Ethernet Port Cabling Tables

Single Phase Power Distribution Unit Cabling Tables

Three Phase Power Distribution Unit Cabling Tables

InfiniBand Network Cabling Tables

Connecting an Oracle Exadata Storage Expansion Rack to the SPARC SuperCluster T4-4

InfiniBand Switch Information for the SPARC SuperCluster T4-4

InfiniBand Switch Information for the Oracle Exadata Storage Expansion Rack

Connecting an Oracle Exadata Storage Expansion Quarter Rack to the SPARC SuperCluster T4-4

Connecting an Oracle Exadata Storage Expansion Half Rack or Oracle Exadata Storage Expansion Full Rack to the SPARC SuperCluster T4-4

Language:

These topics provide instructions for connecting one SPARC SuperCluster T4-4 to one or more SPARC SuperCluster T4-4 systems.

Multi-Rack Cabling Overview
Two-Rack Cabling
Three-Rack Cabling
Four-Rack Cabling
Five-Rack Cabling
Six-Rack Cabling
Seven-Rack Cabling
Eight-Rack Cabling