Contents

Title and Copyright Information

Preface

• Audience
• Document Change Summary
• Documentation Accessibility
• Related Documents
• Conventions

1 Introduction to Exalogic Machine

• 1.1 About Oracle Exalogic
• 1.2 Exalogic Hardware Configurations
• 1.3 Features of Exalogic
• 1.4 Hardware Components of Exalogic
• 1.5 Exalogic Machine Rack Layout
  • 1.5.1 Exalogic Machine Full Rack Layout
  • 1.5.2 Exalogic Machine Half Rack Layout
  • 1.5.3 Exalogic Machine Quarter Rack Layout
  • 1.5.4 Exalogic Machine Eighth Rack Layout
• 1.6 Operational Procedures for Exalogic Machine
  • 1.6.1 Non-emergency Power Procedure
    • 1.6.1.1 Power On Sequence
    • 1.6.1.2 Power Off Sequence
  • 1.6.2 Emergency Power-off Considerations
    • 1.6.2.1 Emergency Power-off Procedure
    • 1.6.2.2 Emergency Power-off Switch
  • 1.6.3 Cautions and Warnings

2 Site Requirements for Exalogic Machine

• 2.1 Environmental Requirements
• 2.2 Space Requirements
  • 2.2.1 Receiving and Unpacking Requirements
  • 2.2.2 Maintenance Access Requirements
• 2.3 Flooring Requirements
• 2.4 Electrical Power Requirements
  • 2.4.1 Facility Power Requirements
  • 2.4.2 Circuit Breaker Requirements
  • 2.4.3 Grounding Guidelines
• 2.5 Temperature and Humidity Requirements
• 2.6 Ventilation and Cooling Requirements

3 Installation Planning and Procedure

• 3.1 Installation Overview
• 3.2 Preparing to Install an Exalogic Machine
  • 3.2.1 Before You Begin
  • 3.2.2 Reviewing Safety Guidelines
  • 3.2.3 Unpacking Exalogic Machine Rack
  • 3.2.4 Tools for Installation
  • 3.2.5 Preparing the Installation Site
  • 3.2.6 Placing Exalogic Machine in Its Allocated Space
    • 3.2.6.1 Moving an Exalogic Machine
    • 3.2.6.2 Stabilizing the Exalogic Machine
      • 3.2.6.2.1 Stabilize Exalogic Machine with Leveling Feet
      • 3.2.6.2.2 Stabilize Exalogic Machine with Mounting Brackets
    • 3.2.6.3 Attaching a Ground Cable (Optional)
• 3.3 Powering on the System the First Time
  • 3.3.1 Inspecting the Machine After It Is in Place
  • 3.3.2 Connecting Power Cords
  • 3.3.3 Powering On the Exalogic Machine
• 3.4 Connecting a Laptop to Exalogic Machine
• 3.5 What Next?

4 Default IP Addresses and Ports Used in Exalogic Machine

• 4.1 Exalogic Machine Full Rack
• 4.2 Exalogic Machine Half Rack
• 4.3 Exalogic Machine Quarter Rack
• 4.4 Exalogic Machine Eighth Rack

5 Adding Exalogic Machine to Your Network

• 5.1 Default State of the Exalogic Machine Network Configuration
• 5.2 Verifying Factory Configuration
• 5.3 Prerequisites for Connecting Exalogic Machine to External Ethernet Network
• 5.4 Initial Network Configuration of Exalogic Machine
  • 5.4.1 Configuring the Cisco Ethernet Switch

6 Understanding Network Requirements and Configuration

• 6.1 Overview of Network Requirements
• 6.2 Naming Conventions
• 6.3 Important Notes for Oracle Solaris Users
• 6.4 Network Connection and IP Address Requirements
  • 6.4.1 Network Connection Requirements
  • 6.4.2 IP Address Requirements
• 6.5 Default InfiniBand Bonded IP Addresses
• 6.6 Introduction to Oracle Exalogic Network Configuration
  • 6.6.1 InfiniBand Fabric
  • 6.6.2 InfiniBand Switches
  • 6.6.3 Default Bonded Interfaces
  • 6.6.4 IPMP Overview for Oracle Solaris Users
    • 6.6.4.1 IPMP Components
    • 6.6.4.2 IPMP Groups
  • 6.6.5 Connectivity Between Exalogic Compute Nodes
  • 6.6.6 Connectivity Between Exalogic Machine and External LAN Through Sun Network QDR InfiniBand Gateway Switch
    • 6.6.6.1 Ethernet Device Requirements
    • 6.6.6.2 Network Interface Configuration for Compute Nodes
    • 6.6.6.3 Transceiver and Cable Requirements
  • 6.6.7 Additional InfiniBand Network Requirements and Specifications
• 6.7 Preparing to Reconfigure the Networking of Exalogic Machine
• 6.8 Subnet Manager Requirements for Connecting Exalogic to Exadata
• 6.9 Network Configuration Worksheets
  • 6.9.1 General Network Configuration Worksheet (Required)
  • 6.9.2 Management Network Configuration Worksheet (Required)
  • 6.9.3 Client Access Network Configuration Worksheet (Required)
  • 6.9.4 Private InfiniBand Network Configuration Worksheet (Required)

7 Setting Up Oracle Integrated Lights Out Manager (ILOM)

• 7.1 ILOM Overview
  • 7.1.1 ILOM Interfaces
• 7.2 Important Notes Before You Begin
• 7.3 Management Network Diagram for Exalogic Machine
• 7.4 ILOM IP Addresses for Exalogic Machine Components
• 7.5 Connecting to ILOM via the Network
  • 7.5.1 Connecting to the CLI
  • 7.5.2 Connecting to the Web GUI
  • 7.5.3 Launching a Remote KVM Session
• 7.6 Connecting to ILOM via a Serial Connection
• 7.7 Reconfiguring Network Access
  • 7.7.1 Configuring ILOM IP Addresses Manually
• 7.8 What Next?

8 Configuring the Sun ZFS Storage 7320 appliance

• 8.1 Prerequisites
• 8.2 Getting Started
• 8.3 Sun ZFS Storage 7320 Appliance Overview
  • 8.3.1 Introduction to Projects
  • 8.3.2 Introduction to Shares
• 8.4 Configuration Overview
  • 8.4.1 Initial Configuration
  • 8.4.2 Connecting Storage Heads to the Management Network and Accessing the Web Interface
  • 8.4.3 Cluster Network Configuration
  • 8.4.4 Network Configuration Options
    • 8.4.4.1 Option 1: ASR Support and Separate Paths for Management and Disaster Recovery
    • 8.4.4.2 Option 2: ASR Support and Shared Path for Management and Disaster Recovery, with Single Management URL
    • 8.4.4.3 Option 3: ASR Support and No Disaster Recovery, But with Single Management URL
  • 8.4.5 Default Storage Configuration
  • 8.4.6 Custom Configuration
• 8.5 Creating Custom Projects
• 8.6 Creating Custom Shares
• 8.7 Using the Phone Home Service to Manage the Storage Appliance
  • 8.7.1 Registering Your Storage Appliance

9 Configuring NFS Version 4 (NFSv4) on Exalogic

• 9.1 Overview
• 9.2 Verifying NIS Setting on Exalogic
• 9.3 Configuring the ZFS Storage 7320
  • 9.3.1 NFS Service
  • 9.3.2 NIS Service
• 9.4 Configuring an Exalogic Linux Compute Node to Use NFSv4
• 9.5 Creating NFSv4 Mount Points on Oracle Linux

10 Configuring Ethernet Over InfiniBand

• 10.1 Introduction to Virtual NIC (VNIC)
• 10.2 Setting Up Ethernet Over InfiniBand (EoIB) on Oracle Linux
• 10.3 Setting Up Ethernet Over InfiniBand (EoIB) on Oracle Solaris

11 Virtual LANs (VLANs)

• 11.1 Introduction to VLAN
• 11.2 Example Scenario
• 11.3 Tagging Ethernet Connectors With a VLAN Identifier
• 11.4 Oracle Linux: Creating VNICs and Associating Them with VLANs
• 11.5 Oracle Solaris: Creating VNICs and Associating Them with VLANs

12 Using the InfiniBand Gateway Switches and Managing the InfiniBand Network Using Subnet Manager

• 12.1 Using Sun Network QDR InfiniBand Gateway Switches
  • 12.1.1 Physical Specifications
  • 12.1.2 Accessing the Command-Line Interface (CLI) of a Gateway Switch
  • 12.1.3 Verifying the Status of a Gateway Switch
  • 12.1.4 Starting the Subnet Manager Manually
  • 12.1.5 Checking Link Status
  • 12.1.6 Verifying the InfiniBand Fabric
    • 12.1.6.1 Discovering the InfiniBand Network Topology
    • 12.1.6.2 Performing Diagnostics on the InfiniBand Fabric
    • 12.1.6.3 Validating and Checking Errors in the InfiniBand Fabric
  • 12.1.7 Monitoring a Gateway Switch Using Web Interface
• 12.2 Understanding Administrative Commands
  • 12.2.1 Hardware Command Overview
  • 12.2.2 InfiniBand Command Overview
• 12.3 Managing InfiniBand Network Using Subnet Manager
  • 12.3.1 Overview of Subnet Manager
  • 12.3.2 Running the Subnet Manager in Different Rack Configurations
  • 12.3.3 Monitoring the Subnet Manager
    • 12.3.3.1 Displaying the Subnet Manager Status
    • 12.3.3.2 Displaying Recent Subnet Manager Activity
  • 12.3.4 Controlling the Subnet Manager
    • 12.3.4.1 Identifying the Location of Master Subnet Manager
    • 12.3.4.2 Relocating Master Subnet Manager
    • 12.3.4.3 Enabling Subnet Manager on a Switch
    • 12.3.4.4 Disabling Subnet Manager on a Switch
• 12.4 Working with the Default Rack-Level InfiniBand Partition
  • 12.4.1 Partition in Exalogic Machine
  • 12.4.2 Verifying the Default Partition
• 12.5 What Next?

13 Using Sun Datacenter InfiniBand Switch 36 in Multirack Configurations

• 13.1 Physical Specifications
• 13.2 Accessing the Command-Line Interface (CLI) of a Sun Datacenter InfiniBand Switch 36
• 13.3 Verifying the Switch Status
• 13.4 Starting the Subnet Manager in Multirack Configuration Scenarios
• 13.5 Checking Link Status
• 13.6 Verifying the InfiniBand Fabric in a Multirack Configuration
  • 13.6.1 Discovering the InfiniBand Network Topology in a Multirack Configuration
  • 13.6.2 Performing Diagnostics on the InfiniBand Fabric in a Multirack Configuration
  • 13.6.3 Validating and Checking Errors in the InfiniBand Fabric in a Multirack Configuration
• 13.7 Monitoring the Spine Switch Using Web Interface
• 13.8 What Next?

14 Monitoring and Controlling the InfiniBand Fabric

• 14.1 Monitoring the InfiniBand Fabric
  • 14.1.1 Identifying All Switches in the Fabric
  • 14.1.2 Identifying All HCAs in the Fabric
  • 14.1.3 Displaying the InfiniBand Fabric Topology
  • 14.1.4 Displaying a Route Through the Fabric
  • 14.1.5 Displaying the Link Status of a Node
  • 14.1.6 Displaying Counters for a Node
  • 14.1.7 Displaying Data Counters for a Node
  • 14.1.8 Displaying Low-Level Detailed Information for a Node
  • 14.1.9 Displaying Low-Level Detailed Information for a Port
  • 14.1.10 Mapping LIDs to GUIDs
  • 14.1.11 Performing Comprehensive Diagnostics for the Entire Fabric
  • 14.1.12 Performing Comprehensive Diagnostics for a Route
  • 14.1.13 Determining Changes to the InfiniBand Topology
  • 14.1.14 Determining Which Links Are Experiencing Significant Errors
  • 14.1.15 Checking All Ports
• 14.2 Controlling the InfiniBand Fabric
  • 14.2.1 Clearing Error Counters
  • 14.2.2 Clearing Data Counters
  • 14.2.3 Resetting a Port
  • 14.2.4 Setting Port Speed
  • 14.2.5 Disabling a Port
  • 14.2.6 Enabling a Port
• 14.3 For More Information

15 Using InfiniBand Partitions in Exalogic Physical Environments

• 15.1 Partitioning Overview
• 15.2 Before You Begin
  • 15.2.1 Verifying InfiniBand Switch Firmware
  • 15.2.2 Gathering Port GUIDs of Compute Nodes and BridgeX Ports of Gateway Switches
  • 15.2.3 Identifying All InfiniBand Switches in the Fabric
  • 15.2.4 Determining the SM Priority on an InfiniBand Switch
  • 15.2.5 Logging In to the InfiniBand Switch That Runs Master SM
• 15.3 Understanding Partition Keys
• 15.4 Moving from a Default Partition to a Custom Partition
• 15.5 Creating an IPoIB Partition and Adding Ports
• 15.6 Deleting a Partition
• 15.7 Creating a Partition for EoIB and Associating the pkey with a VNIC and VLAN
• 15.8 Post-Configuration Steps
• 15.9 Important Notes for Combined Exalogic-Exadata Fabric Users
• 15.10 Partitioning Limitations

16 Monitoring Exalogic Machine Using Oracle Enterprise Manager 11g Ops Center

• 16.1 Overview
• 16.2 Key Features
• 16.3 Prerequisites
• 16.4 Accessing Oracle Enterprise Manager Ops Center Documentation
• 16.5 Launching Oracle Enterprise Manager Ops Center
• 16.6 Understanding the Workflow
• 16.7 Managing Users and Roles
• 16.8 Discovering and Managing Exalogic Machine Hardware
• 16.9 Grouping Exalogic Machine Hardware Assets
  • 16.9.1 Prerequisites
  • 16.9.2 Creating the Exalogic Top-Level Group
  • 16.9.3 Creating a Sub-Group for Exalogic Compute Nodes
  • 16.9.4 Creating a Sub-Group for the Sun ZFS Storage 7320 appliance
  • 16.9.5 Creating a Sub-Group for InfiniBand Switches
  • 16.9.6 Adding Assets to a Group
• 16.10 Viewing Exalogic Compute Nodes
• 16.11 Viewing InfiniBand Switches
• 16.12 Viewing the Sun ZFS Storage 7320 appliance
• 16.13 Viewing the InfiniBand Fabric and Its Nodes
• 16.14 About Problem Management
• 16.15 Using Monitoring Profiles and Rules
  • 16.15.1 Creating a Monitoring Profile
  • 16.15.2 Adding a Monitoring Rule from the Asset View
• 16.16 Using Reports in Oracle Enterprise Manager Ops Center
• 16.17 Using Oracle Services in Oracle Enterprise Manager Ops Center
  • 16.17.1 Prerequisites for Using Oracle Services in Oracle Enterprise Manager Ops Center
  • 16.17.2 Viewing Service Requests
  • 16.17.3 Filing a Service Request

17 Installing Auto Service Request (ASR) Software

• 17.1 About Oracle Auto Service Request (ASR)
• 17.2 Recommended Configuration
• 17.3 Before You Begin
• 17.4 Prerequisites for Installing ASR Manager
• 17.5 Installing ASR Manager on a Standalone System
  • 17.5.1 Installing Service Tags for Oracle Linux
  • 17.5.2 Installing SASM Package
  • 17.5.3 Installing ASR Package
• 17.6 Registering the ASR Manager
• 17.7 Activating ILOM for Exalogic Compute Nodes
• 17.8 Activating the Sun ZFS Storage 7320 Appliance
• 17.9 Approving and Verifying ASR Activation for Exalogic Machine Assets
  • 17.9.1 Approving Exalogic Machine Assets in My Oracle Support
  • 17.9.2 Viewing and Verifying ASR Assets

A Configuring Exalogic Machine Using ECU

• A.1 Overview
• A.2 Important Notes Before You Begin
• A.3 Configuration Tasks

B Site Checklists

• B.1 System Components Checklist
• B.2 Data Center Room Checklist
• B.3 Data Center Environment Checklist
• B.4 Access Route Checklist
• B.5 Facility Power Checklist
• B.6 Power Checklist
• B.7 Safety Checklist
• B.8 Logistics Checklist
• B.9 Network Specification Checklist

C Network Connectivity Diagrams

• C.1 Single Rack Scenarios
  • C.1.1 Single Exalogic Eighth Rack
  • C.1.2 Single Exalogic Quarter Rack
  • C.1.3 Single Exalogic Half Rack
  • C.1.4 Single Exalogic Full Rack
• C.2 Multirack Scenarios: All Exalogic
  • C.2.1 Multirack Scenario: Two Exalogic Eighth Racks
  • C.2.2 Multirack Scenario: Exalogic Eighth Rack and Exalogic Quarter Rack
  • C.2.3 Multirack Scenario: Exalogic Eighth Rack and Exalogic Half Rack
  • C.2.4 Multirack Scenario: Exalogic Eighth Rack and Exalogic Full Rack
  • C.2.5 Multirack Scenario: Two Exalogic Quarter Racks
  • C.2.6 Multirack Scenario: Exalogic Quarter Rack and Exalogic Half Rack
  • C.2.7 Multirack Scenario: Exalogic Quarter Rack and Exalogic Full Rack
  • C.2.8 Multirack Scenario: Two Exalogic Half Racks
  • C.2.9 Multirack Scenario: Exalogic Half Rack and Exalogic Full Rack
  • C.2.10 Multirack Scenario: Two Exalogic Full Racks
  • C.2.11 Multirack Scenario: Eight Exalogic Full Racks
• C.3 Multirack Scenarios: Exalogic and Exadata
  • C.3.1 Multirack Scenario: Exalogic Eighth Rack and Exadata Quarter Rack
  • C.3.2 Multirack Scenario: Exalogic Eighth Rack and Exadata Half Rack
  • C.3.3 Multirack Scenario: Exalogic Eighth Rack and Exadata Full Rack
  • C.3.4 Multirack Scenario: Exalogic Quarter Rack and Exadata Quarter Rack
  • C.3.5 Multirack Scenario: Exalogic Quarter Rack and Exadata Half Rack
  • C.3.6 Multirack Scenario: Exalogic Quarter Rack and Exadata Full Rack
  • C.3.7 Multirack Scenario: Exalogic Half Rack and Exadata Quarter Rack
  • C.3.8 Multirack Scenario: Exalogic Half Rack and Exadata Half Rack
  • C.3.9 Multirack Scenario: Exalogic Half Rack and Exadata Full Rack
  • C.3.10 Multirack Scenario: Exalogic Full Rack and Exadata Quarter Rack
  • C.3.11 Multirack Scenario: Exalogic Full Rack and Exadata Half Rack
  • C.3.12 Multirack Scenario: Exalogic Full Rack and Exadata Full Rack

D Replacement Units

• D.1 Rack-Level FRUs
• D.2 Parts for Sun ZFS Storage 7320 appliance
• D.3 Parts for Sun Server X3-2 Compute Nodes
• D.4 Parts for X4170 M2 Compute Nodes
• D.5 Parts for Sun Network QDR InfiniBand Gateway Switch
• D.6 Parts for Sun Datacenter InfiniBand Switch 36
• D.7 Parts for the Cisco Catalyst 4948E-F-S Switch
• D.8 Parts for the Cisco Catalyst 4948 Switch

E Cabling Tables for Exalogic

• E.1 Administrative Gigabit Ethernet Network-Cabling Tables
• E.2 InfiniBand Network Cabling Tables
  • E.2.1 Exalogic Machine Full Rack
  • E.2.2 Exalogic Machine Half Rack
  • E.2.3 Exalogic Machine Quarter Rack
  • E.2.4 Exalogic Machine Eighth Rack