7 Connecting Oracle Exalytics In-Memory Machine to Oracle Exadata Database Machine

This chapter describes how to connect an Oracle Exalytics In-Memory Machine and an Oracle Exadata Database Machine together on the same InfiniBand fabric.

It contains the following topics:

7.1 Understanding InfiniBand Card, Cable, and Switch Firmware Requirements

This section contains the following topics:

7.1.1 InfiniBand Card

Each Oracle Oracle Exalytics In-Memory Machine includes an InfiniBand card, such as the 2-port QDR InfiniBand Host Channel Adapter (HCA).

This InfiniBand card is installed in the Oracle Oracle Exalytics In-Memory Machine in its PCIe Slot 6.

7.1.2 InfiniBand Cables

You may use any spare 5-meter InfiniBand cables included in your Oracle Exadata Database Machine. If you need assistance, contact Oracle support personnel. You need these cables for connecting your Oracle Oracle Exalytics In-Memory Machine to an Oracle Exadata Database Machine on the same InfiniBand fabric.

7.1.3 InfiniBand Switches in the Oracle Exadata Database Machine

Oracle Exadata Database Machine includes two 36-port InfiniBand leaf switches (Sun Datacenter QDR InfiniBand Switch 36). These two switches located in the middle of the rack are referred to as leaf switches. A spine switch is included in unit 1 of Oracle Exadata Database Machine Full Rack and Half Rack configurations.

Note:

In this document, the leaf switch located in the lower unit of the Oracle Exadata Database Machine rack is referred to as leaf 1. The leaf switch located above leaf 1 in the Oracle Exadata Database Machine rack is referred to as leaf 2.

For more information about Oracle Exadata Database Machine's rack layout and hardware components, see the Oracle Exadata Database Machine Owner's Guide.

7.2 Physically Connecting the Network

This section describes the following topics:

7.2.1 Physical Layout Considerations

It is assumed that the rack containing the Oracle Oracle Exalytics In-Memory Machine and the Oracle Exadata Database Machine rack are placed side by side in the data center. Consider the length of the connecting InfiniBand cables when placing the racks in your data center. Do not place the Oracle Oracle Exalytics In-Memory Machine in an Oracle Exadata Database Machine rack.

7.2.2 Scenario 1: Single Exalytics In-Memory Machine with Oracle Exadata Database Machine

Figure 7-1 shows how a single (non-clustered) Oracle Oracle Exalytics In-Memory Machine is connected to an Oracle Exadata Database Machine rack.

Figure 7-1 Non-Clustered Oracle Exalytics In-Memory Machine Connected to Oracle Exadata Database Machine

Description of Figure 7-1 follows
Description of "Figure 7-1 Non-Clustered Oracle Exalytics In-Memory Machine Connected to Oracle Exadata Database Machine"

For high availability purposes, connect the two ports of the InfiniBand card, which is installed in the Oracle Oracle Exalytics In-Memory Machine, to separate leaf switches of the Oracle Exadata Database Machine. For example, as shown in Figure 7-1, you can connect port 1 of the Oracle Oracle Exalytics In-Memory Machine's InfiniBand card to the leaf 1 switch (Sun Datacenter InfiniBand Switch 36 installed in the lower unit of the Oracle Exadata Database Machine). In addition, you should connect port 2 of the Oracle Oracle Exalytics In-Memory Machine's InfiniBand card to the leaf 2 switch (Sun Datacenter InfiniBand Switch 36 installed in the upper unit of the Oracle Exadata Database Machine). Port 1 can function as the active port, and port 2 can function as the passive port.

Note:

On the leaft switches of the Oracle Exadata Database Machine, the following free ports can be used:

Ports 5B, 6A, 6B, 7A, 7B, and 12A

7.2.3 Scenario 2: Clustered Exalytics In-Memory Machine with Oracle Exadata Database Machine

Figure 7-2 shows how a cluster of Oracle Oracle Exalytics In-Memory Machines is connected to an Oracle Exadata Database Machine rack.

Figure 7-2 Clustered Oracle Exalytics In-Memory Machine Connected to Oracle Exadata Database Machine

Description of Figure 7-2 follows
Description of "Figure 7-2 Clustered Oracle Exalytics In-Memory Machine Connected to Oracle Exadata Database Machine"

For high availability purposes, connect the active port of the InfiniBand card included in each of the Oracle Oracle Exalytics In-Memory Machines to the leaf 1 switch (Sun Datacenter InfiniBand Switch 36 installed in the lower unit of the Oracle Exadata Database Machine). In addition, you should connect the passive port of the InfiniBand card included in each of the Oracle Oracle Exalytics In-Memory Machines to the leaf 2 switch (Sun Datacenter InfiniBand Switch 36 installed in the upper unit of the Oracle Exadata Database Machine). Port 1 can function as the active port, and port 2 can function as the passive port.

Note:

On the leaft switches of the Oracle Exadata Database Machine, the following free ports can be used:

Ports 5B, 6A,6B,7A,7B, and 12A

7.3 Choosing a Subnet Manager

Each InfiniBand network requires a Subnet Manager (SM). You can use the SM running on one of the Oracle Exadata Database Machine's leaf switches in your fabric.

For more information about using the InfiniBand switch, see the following documents:

7.4 Loading the Kernel Modules

The kernel modules required for InfiniBand network support are loaded by the Oracle Linux base operating system, which is pre-installed on the Oracle Oracle Exalytics In-Memory Machine.

7.5 Monitoring the InfiniBand Fabric

You can monitor many aspects of the InfiniBand fabric by running InfiniBand commands on the Sun Datacenter InfiniBand Switch 36, which is included in the Oracle Exadata Database Machine.

For more information about these commands, see the Sun Datacenter InfiniBand Switch 36 Command Reference.

7.6 Checking Network Health

After establishing InfiniBand network connectivity, you can verify the status of the InfiniBand link by using the ibstat command on the CLI of the switch.

The ibstat command displays the basic status, as in the following example:

# ibstat
Switch 'is4_0'
Switch type: MT48436
Number of ports: 0
Firmware version: 7.3.0
Hardware version: a0
Node GUID: 0x00212856cfe2c0a0
System image GUID: 0x00212856cfe2c0a3
Port 0:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 16
LMC: 0
SM lid: 16
Capability mask: 0x4250084a
Port GUID: 0x00212856cfe2c0a0

The physical state of connected ports is shown as LinkUp.

7.7 Testing the Extended Network Connectivity

After verifying the basic InfiniBand network connectivity, you can run the ibhosts, ibswitches, and ibnetdiscover commands on the switch CLI.

The ibhosts command displays host nodes in the InfiniBand fabric, the ibswitches command displays the InfiniBand switch nodes in the fabric, and the ibnetdiscover command discovers the InfiniBand topology.

For more information such monitoring and diagnostic commands, see the Sun Datacenter InfiniBand Switch 36 Command Reference.