JavaScript is required to for searching.
Skip Navigation Links
Exit Print View
Oracle SuperCluster M6-32 HTML Owner’s Guide
search filter icon
search icon

Document Information

Using This Documentation

Product Documentation Library

Feedback

Overview

Determining SuperCluster M6-32 Configurations

Configuration Flowchart

Determine the Number of Compute Servers

Determine the Number of DCUs in Each Compute Server

Determine the Number of CMUs in Each DCU

CMU Overview

Determine the Amount of Memory in Each DCU

Determine the PDomain Configuration on Each Compute Server

Determine the LDom Configuration for Each PDomain

Determining the Best Configuration for Your Situation

Understanding PDomain Configurations

Understanding High Redundancy

Allocating CPU Resources for LDoms

Allocating Memory Resources for LDoms

Understanding PCIe Cards and Slots for LDoms

Understanding Storage for LDoms

Understanding the Hardware

Understanding SuperCluster M6-32

Identifying SuperCluster M6-32 Components

Understanding DCUs

DCU Overview

Understanding DCU Configurations

Understanding Half-Populated DCU Root Complexes

Understanding Fully-Populated DCU Root Complexes

Understanding PDomains

PDomains Overview

PDomain Guidelines

Extended Configuration PDomain Overview

Understanding Extended Configuration PDomains

Base Configuration Overview

Understanding Base Configuration PDomains

Understanding LDoms

Understanding Compute Server Hardware and Networks

CPU and Memory Resources Overview

LDoms and the PCIe Slots Overview

Management Network Overview

10GbE Client Access Network Overview

Understanding the IB Network

IB Network Overview

IB Network Data Paths for a Database Domain

IB Network Data Paths for an Application Domain

Understanding Domains

Dedicated Domains

Understanding SR-IOV Domain Types

Root Domains

I/O Domains

Understanding LDom Configurations for Extended Configuration PDomains

Understanding LDom Configurations for Fully-Populated DCUs (Extended Configuration PDomains)

LDom Configurations for Fully-Populated DCUs (Extended Configuration PDomains)

E4-1 LDom Configuration

E4-2 LDom Configuration

E4-3 LDom Configuration

E4-4 LDom Configuration

Understanding LDom Configurations for Half-Populated DCUs (Extended Configuration PDomains)

LDom Configurations for Half-Populated DCUs (Extended Configuration PDomains)

E2-1 LDom Configuration

E2-2 LDom Configuration

E2-3 LDom Configuration

E2-4 LDom Configuration

Understanding LDom Configurations for Base Configuration PDomains

Understanding LDom Configurations for Fully-Populated DCUs (Base Configuration PDomains)

LDom Configurations for Fully-Populated DCUs (Base Configuration PDomains)

B4-1 LDom Configuration

B4-2 LDom Configuration

B4-3 LDom Configuration

B4-4 LDom Configuration

Understanding LDom Configurations for Half-Populated DCUs (Base Configuration PDomains)

LDom Configurations for Half-Populated DCUs (Base Configuration PDomains)

B2-1 LDom Configuration

B2-2 LDom Configuration

B2-3 LDom Configuration

B2-4 LDom Configuration

Understanding Clustering Software

Cluster Software Overview

Cluster Software for the Database Domain

Cluster Software for the Oracle Solaris Application Domains

Understanding System Administration Resources

Oracle ILOM Overview

Understanding Platform-Specific Oracle ILOM Features

Oracle Solaris OS Overview

OpenBoot Overview

Oracle ILOM Remote Console Plus Overview

Oracle Hardware Management Pack Overview

Time Synchronization and NTP Service

SNMP Service

Multidomain Extensions to Oracle ILOM MIBs

LDAP/SSL

Active Directory

Hardware Installation

Installing SuperCluster M6-32

Hardware Installation Overview

Weight Specifications

Hardware Installation Task Overview

Hardware Installation Documents

Preparing the Site (Storage Rack and Expansion Racks)

Prepare the Site for the Racks

Storage Rack Components

Physical Specifications

Reviewing Power Requirements

Preparing for Cooling

Preparing the Network

Network Topology

Network Infrastructure Requirements

Compute Server Default Host Names and IP Addresses

Install Cable Drops

Prepare DNS

Cabling SuperCluster M6-32

Compute Server Network Components

Storage Rack Network Components

Cable the ZFS Storage Appliance

ZFS Appliance Power Cord Connection Reference

ZFS Storage Appliance Cabling Reference

Cable IB Switches

Leaf Switch 1 Cabling Reference

Leaf Switch 2 Cabling Reference

IB Switch-to-Switch Cabling Reference

Cable the Ethernet Management Switch

Ethernet Management Switch Cabling Reference

Connect SuperCluster M6-32 to the Facility Networks

Connecting Expansion Racks

Expansion Rack Overview

Expansion Rack Components

Expansion Rack Layouts

Installing Expansion Racks

One Expansion Rack Cabling

Two Expansion Racks Cabling

Three Expansion Racks Cabling

Four Expansion Racks Cabling

Five Expansion Racks Cabling

Six Expansion Racks Cabling

Seven Expansion Racks Cabling

Expansion Rack Default IP Addresses

Understanding Internal Cabling (Expansion Rack)

Administration

Understanding SuperCluster Software

Identify the Version of SuperCluster Software

SuperCluster Tools

Controlling SuperCluster M6-32

Cautions

Power On SuperCluster M6-32

Powering Off SuperCluster M6-32 Gracefully

Power Off SuperCluster M6-32 in an Emergency

Monitoring SuperCluster M6-32 (OCM)

OCM Overview

Access OCM Documentation

Monitoring the System With ASR

ASR Overview

ASR Resources

ASR Installation Overview

Configure ASR on the Compute Servers (Oracle ILOM)

Configure SNMP Trap Destinations for Storage Servers

Configure ASR on the ZFS Storage Appliance

Configuring ASR on the Compute Servers (Oracle Solaris 11)

Approve and Verify ASR Asset Activation

Tuning SuperCluster M6-32

ssctuner Overview

Monitor ssctuner Activity

View Log Files

Change ssctuner Properties and Disable Features

Install ssctuner

Enable ssctuner

Configuring CPU and Memory Resources (osc-setcoremem)

osc-setcoremem Overview

Minimum and Maximum Resources (Dedicated Domains)

Supported Domain Configurations

Plan CPU and Memory Allocations

Display the Current Domain Configuration (osc-setcoremem)

Display the Current Domain Configuration (ldm)

Change CPU/Memory Allocations (Socket Granularity)

Change CPU/Memory Allocations (Core Granularity)

Park Cores and Memory

Access osc-setcoremem Log Files

View the SP Configuration

Revert to a Previous CPU/Memory Configuration

Remove a CPU/Memory Configuration

Obtaining the EM Exadata Plug-in

Confirm System Requirements

Known Issues With the EM Exadata Plug-in

Configuring the Exalogic Software

Exalogic Software Overview

Prepare to Configure the Exalogic Software

Enable Domain-Level Enhancements

Enable Cluster-Level Session Replication Enhancements

Configuring Grid Link Data Source for Dept1_Cluster1

Configuring SDP-Enabled JDBC Drivers for Dept1_Cluster1

Create an SDP Listener on the IB Network

Administering Oracle Solaris 11 Boot Environments

Advantages to Maintaining Multiple Boot Environments

Create a Boot Environment

Mount to a Different Build Environment

Reboot to the Original Boot Environment

Create a Snapshot of a Boot Environment

Remove Unwanted Boot Environments

Administering DISM

DISM Restrictions

Disable DISM

Administering Storage Servers

Monitor Write-through Caching Mode

Shut Down or Reboot a Storage Server

Drop a Storage Server

Glossary

Index

I/O Domains

An I/O Domain is an SR-IOV domain that owns its own VFs, each of which is a virtual device based on a PF in one of the Root Domains. Root domains function solely as a provider of VFs to the I/O Domains, based on the physical I/O devices associated with each Root Domain. Applications and zones are supported only in I/O Domains, not in Root Domains.

You can create multiple I/O Domains using the I/O Domain Creation tool. As part of the domain creation process, you also associate one of the following SuperCluster-specific domain types to each I/O Domain:

Note that only Database Domains that are dedicated domains can host database zones. Database I/O Domains cannot host database zones.

The CPU cores and memory resources owned by an I/O Domain are assigned from the CPU and memory repositories (the cores and memory released from Root Domains on the system) when an I/O Domain is created, as shown in the following graphic.

image:Graphic showing the I/O Domains getting resources from the CPU and memory repositories.

You use the I/O Domain Creation tool to assign the CPU core and memory resources to the I/O Domains, based on the amount of CPU core and memory resources that you want to assign to each I/O Domain and the total amount of CPU core and memory resources available in the CPU and memory repositories.

Similarly, the IB VFs and 10GbE VFs owned by the I/O Domains come from the IB VF and 10GbE VF repositories (the IB VFs and 10GbE VFs released from Root Domains on the system), as shown in the following graphic.

image:Graphic showing the I/O Domains getting resources from the IB VF and 10GbE VF repositories.

Again, you use the I/O Domain Creation tool to assign the IB VFs and 10GbE VFs to the I/O Domains using the resources available in the IB VF and 10GbE VF repositories. However, because VFs are created on each EMS and IB HCA, the VFs assigned to an I/O Domain will always come from the specific Root Domain that is associated with the EMSs and IB HCAs that contain those VFs.

The number and size of the I/O Domains that you can create depends on several factors, including the amount of CPU core and memory resources that are available in the CPU and memory repositories and the amount of CPU core and memory resources that you want to assign to each I/O Domain. However, while it is useful to know the total amount of resources are that are parked in the repositories, it does not necessarily translate into the maximum number of I/O Domains that you can create for your system. In addition, you should not create an I/O Domain that uses more than one socket's worth of resources.

For example, assume that you have 32 cores parked in the CPU repository and 1472 GB of memory parked in the memory repository. You could therefore create I/O Domains in any of the following ways:

When you go through the process of creating I/O Domains, at some point, the I/O Domain Creation tool will inform you that you cannot create additional I/O Domains. This could be due to several factors, such as reaching the limit of total CPU core and memory resources in the CPU and memory repositories, reaching the limit of resources available specifically to you as a user, or reaching the limit on the number of I/O Domains allowable for this system.


Note - The following examples describe how resources might be divided up between domains using percentages to make the conceptual information easier to understand. However, you actually divide CPU core and memory resources between domains at a socket granularity or core granularity level. Refer to the Oracle SuperCluster M6-32 Owner's Guide: Administration for more information.


As an example configuration showing how you might assign CPU and memory resources to each domain, assume that you have a domain configuration where one of the domains is a Root Domain, and the other three domains are dedicated domains, as shown in the following figure.

image:Graphic showing a single Root Domain providing resources from the IB VF and 10GbE VF repositories.

Even though dedicated domains and Root Domains are all shown as equal-sized domains in the preceding figure, that does not mean that CPU core and memory resources must be split evenly across all four domains (where each domain would get 25% of the CPU core and memory resources). Using information that you provide in the configuration worksheets, you can request different sizes of CPU core and memory resources for each domain when your SuperCluster M6-32 is initially installed.

For example, you could request that each dedicated domain have 30% of the CPU core and memory resources (for a total of 90% of the CPU cores and memory resources allocated to the three dedicated domains), and the remaining 10% allocated to the single Root Domain. Having this configuration would mean that only 10% of the CPU core and memory resources are available for I/O Domains to pull from the CPU and memory repositories. However, you could also request that some of the resources from the dedicated domains be parked at the time of the initial installation of your system, which would further increase the amount of CPU core and memory resources available for I/O Domains to pull from the repositories.

You could also use the CPU/Memory tool after the initial installation to resize the amount of CPU core and memory resources used by the existing domains, depending on the configuration that you chose at the time of your initial installation:

Refer to the Oracle SuperCluster M6-32 Owner's Guide: Administration for more information.

Assume you have a mixture of dedicated domains and Root Domains as mentioned earlier, where each dedicated domain has 30% of the CPU core and memory resources (total of 90% resources allocated to dedicated domains), and the remaining 10% allocated to the Root Domain. You could then make the following changes to the resource allocation, depending on your situation: