This chapter describes how to connect an Oracle Exalytics In-Memory Machine and an Oracle Exadata Database Machine together on the same InfiniBand fabric.
It contains the following topics:
This section contains the following topics:
Each Oracle Oracle Exalytics In-Memory Machine includes an InfiniBand card, such as the 2-port QDR InfiniBand Host Channel Adapter (HCA).
This InfiniBand card is installed in the Oracle Oracle Exalytics In-Memory Machine in its PCIe Slot 6.
You may use any spare 5-meter InfiniBand cables included in your Oracle Exadata Database Machine. If you need assistance, contact Oracle support personnel. You need these cables for connecting your Oracle Oracle Exalytics In-Memory Machine to an Oracle Exadata Database Machine on the same InfiniBand fabric.
Oracle Exadata Database Machine includes two 36-port InfiniBand leaf switches (Sun Datacenter QDR InfiniBand Switch 36). These two switches located in the middle of the rack are referred to as leaf switches. A spine switch is included in unit 1 of Oracle Exadata Database Machine Full Rack and Half Rack configurations.
Note:
In this document, the leaf switch located in the lower unit of the Oracle Exadata Database Machine rack is referred to as leaf 1. The leaf switch located above leaf 1 in the Oracle Exadata Database Machine rack is referred to as leaf 2.
For more information about Oracle Exadata Database Machine's rack layout and hardware components, see the Oracle Exadata Database Machine Owner's Guide.
This section describes the following topics:
Scenario 1: Single Exalytics In-Memory Machine with Oracle Exadata Database Machine
Scenario 2: Clustered Exalytics In-Memory Machine with Oracle Exadata Database Machine
It is assumed that the rack containing the Oracle Oracle Exalytics In-Memory Machine and the Oracle Exadata Database Machine rack are placed side by side in the data center. Consider the length of the connecting InfiniBand cables when placing the racks in your data center. Do not place the Oracle Oracle Exalytics In-Memory Machine in an Oracle Exadata Database Machine rack.
Figure 7-1 shows how a single (non-clustered) Oracle Oracle Exalytics In-Memory Machine is connected to an Oracle Exadata Database Machine rack.
Figure 7-1 Non-Clustered Oracle Exalytics In-Memory Machine Connected to Oracle Exadata Database Machine
For high availability purposes, connect the two ports of the InfiniBand card, which is installed in the Oracle Oracle Exalytics In-Memory Machine, to separate leaf switches of the Oracle Exadata Database Machine. For example, as shown in Figure 7-1, you can connect port 1 of the Oracle Oracle Exalytics In-Memory Machine's InfiniBand card to the leaf 1 switch (Sun Datacenter InfiniBand Switch 36 installed in the lower unit of the Oracle Exadata Database Machine). In addition, you should connect port 2 of the Oracle Oracle Exalytics In-Memory Machine's InfiniBand card to the leaf 2 switch (Sun Datacenter InfiniBand Switch 36 installed in the upper unit of the Oracle Exadata Database Machine). Port 1 can function as the active port, and port 2 can function as the passive port.
Note:
On the leaft switches of the Oracle Exadata Database Machine, the following free ports can be used:
Ports 5B, 6A, 6B, 7A, 7B, and 12A
Figure 7-2 shows how a cluster of Oracle Oracle Exalytics In-Memory Machines is connected to an Oracle Exadata Database Machine rack.
Figure 7-2 Clustered Oracle Exalytics In-Memory Machine Connected to Oracle Exadata Database Machine
For high availability purposes, connect the active port of the InfiniBand card included in each of the Oracle Oracle Exalytics In-Memory Machines to the leaf 1 switch (Sun Datacenter InfiniBand Switch 36 installed in the lower unit of the Oracle Exadata Database Machine). In addition, you should connect the passive port of the InfiniBand card included in each of the Oracle Oracle Exalytics In-Memory Machines to the leaf 2 switch (Sun Datacenter InfiniBand Switch 36 installed in the upper unit of the Oracle Exadata Database Machine). Port 1 can function as the active port, and port 2 can function as the passive port.
Note:
On the leaft switches of the Oracle Exadata Database Machine, the following free ports can be used:
Ports 5B, 6A,6B,7A,7B, and 12A
Each InfiniBand network requires a Subnet Manager (SM). You can use the SM running on one of the Oracle Exadata Database Machine's leaf switches in your fabric.
For more information about using the InfiniBand switch, see the following documents:
To configure the InfiniBand connection between Exalytics and Exadata, do the following:
Enable automatic RDS kernel module loading on Exalytics.
Edit the /etc/infiniband/openib.conf
file and set SDP_LOAD=yes
and RDS_LOAD=yes
.
Reboot the nodes.
Create InfiniBand Listener on Exadata compute nodes.
Edit the /etc/infiniband/openib.conf
file and set SDP_LOAD=yes
and RDS_LOAD=yes
.
Edit the /etc/ofed/libsdp.conf
file and find the following lines:
use both server * : use both client * :
Change these lines to read as follows:
use tcp server * *:* use tcp client * *:*
use tcp
indicates that TCP is the default protocol and SDP will be used from specific applications using their local configuration.
Edit the /etc/modprobe.conf
file and add this line at the end:
options ib_sdp sdp_zcopy_thresh=0 recv_poll=0
Reboot each compute node.
Edit the /etc/hosts
file on each Exalytics machine and Exadata Database Machine compute node to add the virtual IP addresses for use on the InfiniBand network, for example, Exadata Database Machine quarter rack (2 compute nodes). Ensure that these IP addresses are not in use on the private network.
192.168.10.21 dm01db01-ibvip.mycompany.com dm01db01-ibvip 192.168.10.22 dm01db02-ibvip.mycompany.com dm01db02-ibvip
On one of the Exadata Database Machine compute nodes, as the root
user execute the following commands to create a network resource for the InfiniBand network, and add the starting virtual IP addresses for each compute node on it. Make appropriate substitutions for the network and subnet mask in use on your private network (it may not be 192.168.10.0/255.255.255.0). Ask your admin or use ipcalc
, based on the network mask to calculate the network.
srvctl add network -k 2 -S 192.168.10.0/255.255.255.0/bondib0 srvctl add vip -n dm01db01 -A dm01db01-ibvip/255.255.255.0/bondib0 -k 2 srvctl add vip -n dm01db02 -A dm01db02-ibvip/255.255.255.0/bondib0 -k 2 srvctl start vip -i dbm01db01-ibvip srvctl start vip -i dbm01db02-ibvip
On one of the Exadata Database Machine compute nodes, as oracle
user (or owner of the Grid Infrastructure ORACLE_HOME
) execute the following command:
srvctl add listener -l LISTENER_IB -k 2 -p TCP:1522/SDP:1522
On each Exadata Database Machine compute node, as oracle
user (or owner of the Grid Infrastructure ORACLE_HOME
), create or append to tnsnames.ora
in GRID_HOME/network/admin
with the following new entries (note that the *IBREMOTE and *IBLOCAL entries will be slightly different on each node to reference the proper remote and local VIP names—you cannot copy the entries from one node to another without some modifications):
DBM = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = dm01-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = dbm) )) DBM_IB = (DESCRIPTION = (LOAD_BALANCE=on) (ADDRESS = (PROTOCOL = TCP)(HOST = dm01db01-ibvip)(PORT = 1522)) (ADDRESS = (PROTOCOL = TCP)(HOST = dm01db02-ibvip)(PORT = 1522)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = dbm) )) LISTENER_IBREMOTE = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = dm01db02-ibvip.mycompany.com)(PORT = 1522)) )) LISTENER_IBLOCAL = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = dm01db01-ibvip.mycompany.com)(PORT = 1522)) (ADDRESS = (PROTOCOL = SDP)(HOST = dm01db01-ibvip.mycompany.com)(PORT = 1522)) )) LISTENER_IPLOCAL = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = dm0101-vip.mycompany.com)(PORT = 1521)) )) LISTENER_IPREMOTE = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = dm01-scan.mycompany.com)(PORT = 1521)) ))
On one Exadata Database Machine compute node, connect to the database instance as sysdba
:
SQLPLUS> alter system set listener_networks='((NAME=network2) (LOCAL_LISTENER=LISTENER_IBLOCAL)(REMOTE_LISTENER=LISTENER_IBREMOTE))', '((NAME=network1)(LOCAL_LISTENER=LISTENER_IPLOCAL) (REMOTE_LISTENER=LISTENER_IPREMOTE))' scope=both;
On one Exadata Database Machine compute node, as oracle user (or owner of the Grid Infrastructure ORACLE_HOME
) restart the listener named LISTENER_IB
by running the following commands:
srvctl stop listener -l LISTENER_IB srvctl start listener -l LISTENER_IB
On the Exalytics machine, edit sys.odbc.ini
file to make an entry that points to the Exadata DB host:
OracleNetServiceName=DWH.ESS.LOCAL_IB
You can monitor many aspects of the InfiniBand fabric by running InfiniBand commands on the Sun Datacenter InfiniBand Switch 36, which is included in the Oracle Exadata Database Machine.
For more information about these commands, see the Sun Datacenter InfiniBand Switch 36 Command Reference.
After establishing InfiniBand network connectivity, you can verify the status of the basic InfiniBand network connectivity with rds-ping
, rds-info -l
by using the ibstat
command on the Exalytics machine.
The ibstat
command displays the basic status, as in the following example:
# ibstat CA 'mlx4_0' CA type: MT26428 Number of ports: 2 Firmware version: 2.7.8130 Hardware version: b0 Node GUID: 0x0021280001cf577e System image GUID: 0x0021280001cf5781 Port 1: State: Active Physical state: LinkUp Rate: 40 Base lid: 106 LMC: 0 SM lid: 54 Capability mask: 0x02510868 Port GUID: 0x0021280001cf577f Link layer: IB Port 2: State: Active Physical state: LinkUp Rate: 40 Base lid: 95 LMC: 0 SM lid: 54 Capability mask: 0x02510868 Port GUID: 0x0021280001cf5780 Link layer: IB
The physical state of connected ports is shown as LinkUp
.
After verifying the basic InfiniBand network connectivity, you can run the ibhosts
, ibswitches
, and ibnetdiscover
commands on the switch CLI.
The ibhosts
command displays host nodes in the InfiniBand fabric, the ibswitches
command displays the InfiniBand switch nodes in the fabric, and the ibnetdiscover
command discovers the InfiniBand topology.
For more information such monitoring and diagnostic commands, see the Sun Datacenter InfiniBand Switch 36 Command Reference.