Identifying InfiniBand Fabric Connections - Oracle® SuperCluster T5-8 Owner's Guide

Oracle^® SuperCluster T5-8 Owner's Guide

Document Information

Using This Documentation

Product Documentation Library

Feedback

Understanding the System

Understanding Oracle SuperCluster T5-8

Spares Kit Components

Oracle SuperCluster T5-8 Restrictions

Identifying Hardware Components

Full Rack Components

Half Rack Components

Understanding the Hardware Components and Connections

Understanding the Hardware Components

SPARC T5-8 Servers

Exadata Storage Servers

ZFS Storage Appliance

Sun Datacenter InfiniBand Switch 36 Switches

Cisco Catalyst 4948 Ethernet Management Switch

Power Distribution Units

Understanding the Physical Connections

Understanding the SPARC T5-8 Server Physical Connections

PCIe Slots (SPARC T5-8 Servers)

Card Locations (SPARC T5-8 Servers)

NET MGT and NET0-3 Port Locations (SPARC T5-8 Servers)

InfiniBand Private Network Physical Connections (SPARC T5-8 Servers)

Oracle ILOM Management Network Physical Connections (SPARC T5-8 Servers)

1-GbE Host Management Network Physical Connections (SPARC T5-8 Servers)

10-GbE Client Access Network Physical Connections (SPARC T5-8 Servers)

Understanding the Exadata Storage Server Physical Connections

InfiniBand Private Network Physical Connections (Exadata Storage Servers)

Oracle ILOM Management Network Physical Connections (Exadata Storage Servers)

1-GbE Host Management Network Physical Connections (Exadata Storage Servers)

Understanding the ZFS Storage Appliance Physical Connections

InfiniBand Private Network Physical Connections (ZFS Storage Appliance)

Oracle ILOM Management Network Physical Connections (ZFS Storage Appliance)

1-GbE Host Management Network Physical Connections (ZFS Storage Appliance)

SAS Physical Connections (ZFS Storage Appliance)

Cluster Physical Connections (ZFS Storage Appliance)

Power Distribution Units Physical Connections

Understanding the Software Configurations

Understanding Domains

Dedicated Domains

Understanding SR-IOV Domain Types

Root Domains

I/O Domains

Understanding General Configuration Information

Logical Domains and the PCIe Slots Overview

Management Network Overview

10-GbE Client Access Network Overview

InfiniBand Network Overview

InfiniBand Network Data Paths for a Database Domain

InfiniBand Network Data Paths for an Application Domain

Understanding Half Rack Configurations

Logical Domain Configurations and PCIe Slot Mapping (Half Rack)

Understanding Large Domains (Half Rack)

Percentage of CPU and Memory Resource Allocation

Management Network

10-GbE Client Access Network

InfiniBand Network

Understanding Medium Domains (Half Rack)

Percentage of CPU and Memory Resource Allocation

Management Network

10-GbE Client Access Network

InfiniBand Network

Understanding Small Domains (Half Rack)

Percentage of CPU and Memory Resource Allocation

Management Network

10-GbE Client Access Network

InfiniBand Network

Understanding Full Rack Configurations

Logical Domain Configurations and PCIe Slot Mapping (Full Rack)

Understanding Giant Domains (Full Rack)

Percentage of CPU and Memory Resource Allocation

Management Network

10-GbE Client Access Network

InfiniBand Network

Understanding Large Domains (Full Rack)

Percentage of CPU and Memory Resource Allocation

Management Network

10-GbE Client Access Network

InfiniBand Network

Understanding Medium Domains (Full Rack)

Percentage of CPU and Memory Resource Allocation

Management Network

10-GbE Client Access Network

InfiniBand Network

Understanding Small Domains (Full Rack)

Percentage of CPU and Memory Resource Allocation

Management Network

10-GbE Client Access Network

InfiniBand Network

Understanding Clustering Software

Cluster Software for the Database Domain

Cluster Software for the Oracle Solaris Application Domains

Understanding the Network Requirements

Network Requirements Overview

Network Connection Requirements for Oracle SuperCluster T5-8

Understanding Default IP Addresses

Default IP Addresses

Default Host Names and IP Addresses

Default Host Names and IP Addresses for the Oracle ILOM and Host Management Networks

Default Host Names and IP Addresses for the InfiniBand and 10-GbE Client Access Networks

Preparing the Site

Cautions and Considerations

Reviewing System Specifications

Physical Specifications

Installation and Service Area

Rack and Floor Cutout Dimensions

Reviewing Power Requirements

Power Consumption

Facility Power Requirements

Grounding Requirements

PDU Power Requirements

PDU Thresholds

PDU Thresholds (Quarter-Populated Racks)

PDU Thresholds (Half-Populated Racks)

PDU Thresholds (Fully-Populated Racks)

Preparing for Cooling

Environmental Requirements

Heat Dissipation and Airflow Requirements

Perforated Floor Tiles

Preparing the Unloading Route and Unpacking Area

Shipping Package Dimensions

Loading Dock and Receiving Area Requirements

Access Route Guidelines

Unpacking Area

Preparing the Network

Network Connection Requirements

Network IP Address Requirements

Prepare DNS for the System

Installing the System

Installation Overview

Oracle Safety Information

Unpacking the System

Tools for Installation

Find the Unpacking Instructions

Unpack and Inspect the System

Moving the Rack Into Place

Move Oracle SuperCluster T5-8

Install a Ground Cable (Optional)

Adjust the Leveling Feet

Powering on the System for the First Time

Connect Power Cords to the Rack

Power On the System

Connect a Laptop to the System

Connect to a 10-GbE Client Access Network

Using an Optional Fibre Channel PCIe Card

Maintaining the System

Cautions and Warnings

Powering Off the System

Graceful Power Off Sequence

Shut Down the Exadata Storage Servers

Power Off the Exadata Storage Servers

Gracefully Shut Down LDoms

Emergency Power Off

Power On the System

Identify the Version of SuperCluster Software

SuperCluster Tools

Managing Oracle Solaris 11 Boot Environments

Advantages to Maintaining Multiple Boot Environments

Create a Boot Environment

Mount to a Different Build Environment

Reboot to the Original Boot Environment

Remove Unwanted Boot Environments

Using Dynamic Intimate Shared Memory

DISM Restrictions

Disable DISM

Component-Specific Service Procedures

Maintaining Exadata Storage Servers

Monitor Write-through Caching Mode

Shut Down or Reboot an Exadata Storage Server

Drop an Exadata Storage Server

Tuning the System (ssctuner)

ssctuner Overview

Monitor ssctuner Activity

View Log Files

Configure the EMAIL_ADDRESS Property

Change ssctuner Properties and Disable Features

Configure ssctuner to Run compliance(1M) Benchmarks

Monitor and View the Compliance Benchmark

Install ssctuner

Enable ssctuner

Configuring CPU and Memory Resources (osc-setcoremem)

osc-setcoremem Overview

Minimum and Maximum Resources (Dedicated Domains)

Supported Domain Configurations

Plan CPU and Memory Allocations

Display the Current Domain Configuration (osc-setcoremem)

Display the Current Domain Configuration (ldm)

Change CPU/Memory Allocations (Socket Granularity)

Change CPU/Memory Allocations (Core Granularity)

Park Cores and Memory

Access osc-setcoremem Log Files

View the SP Configuration

Revert to a Previous CPU/Memory Configuration

Remove a CPU/Memory Configuration

Monitoring the System

Monitoring the System Using Auto Service Request

ASR Overview

ASR Resources

ASR Installation Overview

Configure SNMP Trap Destinations for Exadata Storage Servers

Configure ASR on the ZFS Storage Appliance

Configure ASR on SPARC T5-8 Servers (Oracle ILOM)

Configuring ASR on the SPARC T5-8 Servers (Oracle Solaris 11)

Enable the HTTP Receiver on the ASR Manager

Enable HTTPS on ASR Manager (Optional)

Approve and Verify ASR Asset Activation

Monitoring the System Using OCM

OCM Overview

Install Oracle Configuration Manager on SPARC T5-8 Servers

Monitoring the System Using EM Exadata Plug-in

System Requirements

Known Issues With EM Exadata Plug-in

Configuring Exalogic Software

Exalogic Software Overview

Exalogic Software Prerequisites

Enable Domain-Level Enhancements

Enable Cluster-Level Session Replication Enhancements

Configuring Grid Link Data Source for Dept1_Cluster1

Grid Link Data Source

Fast Connection Failover

Runtime Connection Load Balancing

XA Affinity

SCAN Addresses

Secure Communication using Oracle Wallet

Create a GridLink Data Source on Dept1_Cluster1

Configuring SDP-Enabled JDBC Drivers for Dept1_Cluster1

Configure the Database to Support Infiniband

Enable SDP Support for JDBC

Monitor SDP Sockets Using netstat on Oracle Solaris

Configuring SDP InfiniBand Listener for Exalogic Connections

Create an SDP Listener on the InfiniBand Network

Understanding Internal Cabling

Connector Locations

Identifying InfiniBand Fabric Connections

IB Spine Switch

IB Leaf Switch No. 1

IB Leaf Switch No. 2

Ethernet Management Switch Connections

ZFS Storage Appliance Connections

Single-Phase PDU Cabling

Three-Phase PDU Cabling

Connecting Multiple Oracle SuperCluster T5-8 Systems

Multi-Rack Cabling Overview

Connecting Expansion Racks

Oracle Exadata Storage Expansion Rack Components

Preparing for Installation

Reviewing System Specifications

Physical Specifications

Reviewing Power Requirements

System Power Consumption

PDU Power Requirements

Preparing for Cooling

Environmental Requirements

Heat Dissipation and Airflow Requirements

Perforated Floor Tiles

Preparing the Unloading Route and Unpacking Area

Shipping Package Dimensions

Preparing the Network

Network Requirements Overview

Network Connection and IP Address Requirements

Installing the Oracle Exadata Storage Expansion Rack

Expansion Rack Default IP Addresses

Understanding Expansion Rack Internal Cabling

Front and Rear Expansion Rack Layout

Oracle ILOM Cabling

Administrative Gigabit Ethernet Port Cabling Tables

Single-Phase PDU Cabling

Three-Phase Power Distribution Unit Cabling

InfiniBand Network Cabling

Connecting an Expansion Rack to Oracle SuperCluster T5-8

InfiniBand Switch Information for Oracle SuperCluster T5-8

InfiniBand Switch Information for the Oracle Exadata Storage Expansion Rack

Connecting an Oracle Exadata Storage Expansion Quarter Rack to Oracle SuperCluster T5-8

Connecting an Oracle Exadata Storage Expansion Half Rack or Oracle Exadata Storage Expansion Full Rack to Oracle SuperCluster T5-8

Thirteen-Rack Cabling

Fourteen-Rack Cabling

Fifteen-Rack Cabling

Sixteen-Rack Cabling

Seventeen-Rack Cabling

Eighteen-Rack Cabling