8 Connecting Oracle Exalytics In-Memory Machine to Oracle Big Data Appliance

This chapter describes how to connect Oracle Exalytics In-Memory Machine and an Oracle Big Data Appliance together on the same InfiniBand fabric.

It contains the following sections:

Section 8.1, "Hardware Requirements"
Section 8.2, "Software Requirements"
Section 8.3, "Physical Connectivity"
Section 8.4, "InfiniBand Connection Validation (Layer-2)"
Section 8.5, "IPoIB Configuration (Layer-3)"
Section 8.6, "IPoIB Validation"

8.1 Hardware Requirements

Oracle Big Data Appliance has three in-built InfiniBand switches. Use the two NM2-GW leaf switches to connect the Oracle Exalytics machine. The Oracle Exalytics machine has one InfiniBand HCA card, but it does not include any InfiniBand switch or any instance of InfiniBand Subnet Manager.

To connect Oracle Exalytics to Oracle Big Data Appliance, you need a pair of InfiniBand cables depending on the length requirements at the datacenter. If the total distance between Exalytics and Oracle Big Data Appliance leaf switch ports is less than 10 meters, you can use copper cables. If the length is more than 10 meters, you must use fiber optic cable for InfiniBand with appropriate transceivers.

8.2 Software Requirements

The base images of both Oracle Big Data Appliance and Oracle Exalytics include software support for InfiniBand connectivity.

8.2.1 Exalytics Version Details

To view software details of the Oracle Exalytics machine, run the following commands:

Note:

From the following commands, use imageinfo command for 1.0.0.2 systems and earlier. Use exalytics_imageinfo command for 1.0.0.3 systems and later.

[root@exalytics root]# exalytics_imageinfo
Exalogic 1.0.0.5 (build:r)
 
Image version : 1.0.0.5
Image build version :
Creation timestamp : Wed 19 Jun 2013 12:26:49 PM BST
Kernel version : 2.6.32-100.23.1.el5
Image activated : 2013-09-03 20:49:13 -0400
Image status : SUCCESS
 
[root@exalytics root]# ofed_info | head -1
BXOFED-1.5.2-1.3.8000:
 
[root@exalytics root]# lsmod | grep ipoib
ib_ipoib 67547 1
ib_cm 32459 2 rdma_cm,ib_ipoib
ib_sa 36903 5 rdma_ucm,rdma_cm,mlx4_vnic,ib_ipoib,ib_cm
ib_core 61642 14 rdma_ucm,rds_rdma,ib_sdp,rdma_cm,iw_cm,ib_uverbs,ib_umad,mlx4_vnic,mlx4_ib,ib_mthca,ib_ipoib,ib_cm,ib_sa,ib_mad
ipv6 286932 222 ib_sdp,ib_addr,ib_ipoib,bonding
 
 
[root@exalytics root]# lsmod | grep rds
rds_tcp 9460 0
rds_rdma 72457 1
rds 91561 2 rds_tcp,rds_rdma
rdma_cm 29058 3 rdma_ucm,rds_rdma,ib_sdp
ib_core 61642 14 rdma_ucm,rds_rdma,ib_sdp,rdma_cm,iw_cm,ib_uverbs,ib_umad,mlx4_vnic,mlx4_ib,ib_mthca,ib_ipoib,ib_cm,ib_sa,ib_mad
[root@exalytics root#

8.2.2 BDA Version Details

To view software support of the Oracle Big Data Appliance, run the following commands:

[root@bda10 ~]# imageinfo
Big Data Appliance Image Info
 
IMAGE_CREATION_DATE : Tue Aug 20 23:36:32 PDT 2013
IMAGE_LABEL : BDA_2.2.1_LINUX.X64_130820
IMAGE_VERSION : 2.2.1
KERNEL_VERSION : 2.6.32-200.21.1.el5uek
BDA_RPM_VERSION : bda-2.2.1-1
OFA_RPM_VERSION : ofa-2.6.32-200.21.1.el5uek-1.5.5-4.0.55.4
JDK_VERSION : jdk-1.6.0_51-fcs
HADOOP_VERSION : 2.0.0-cdh4.3.0
 
[root@bda10 ~]# ofed_info | head -1
OFED-IOV-1.5.5-1.0.0120:
 
[root@bda10 ~]# lsmod | grep ipoib
ib_ipoib 67931 1
ib_cm 35714 2 rdma_cm,ib_ipoib
ipv6 290974 5 bonding,cnic,ib_sdp,ib_addr,ib_ipoib
ib_sa 39126 6 rdma_ucm,rdma_cm,ib_ipoib,ib_cm,mlx4_vnic,mlx4_ib
ib_core 66876 12 rdma_ucm,ib_sdp,rdma_cm,iw_cm,ib_ipoib,ib_cm,ib_uverbs,ib_umad,mlx4_vnic,mlx4_ib,ib_sa,ib_mad
 
[root@bda10 ~]# lsmod | grep rds
rds 94503 0
[root@bda10 ~]#

8.3 Physical Connectivity

On both GW switches of Oracle Big Data Appliance, four ports are available for external connections-5A, 5B, 6A, 6B. Oracle recommends that you use ports 5A on both leaf switches. If these ports are not available, use any other available port on the leaf switches.

Note:

Never connect a host (HCA) system to the spine switch inside a rack.

If the Oracle Big Data Appliance rack is connected to other Oracle Engineered Systems, port 5-6[A|B] may not be available. In such a case, it is recommended to use any other available port on the InfiniBand leaf switch of the Oracle Big Data Appliance rack. If no free ports are available, contact Oracle Support.

Figure 8-1 shows how to connect an Exalytics to a standalone BDA rack.

Figure 8-1 Physical Connection

Description of "Figure 8-1 Physical Connection"

8.4 InfiniBand Connection Validation (Layer-2)

Once the cables are connected and you can see green LEDs on the ports, you must validate the InfiniBand connection.

Log in to the Oracle Big Data Appliance compute node or Exalytics machine.
From the Linux root shell, run the ibhosts command to verify if the newly connected (Exalytics/Oracle Big Data Appliance) machine is visible.

On the Exalytics machine, run the ibstat command to see the local ports' status. If an active Subnet Manager is present, then the ports should be Active.

[root@exalytics ~]# ibstat

CA 'mlx4_0'
CA type: MT26428
Number of ports: 2
Firmware version: 2.7.8130
Hardware version: b0
Node GUID: 0x0021280001efe984
System image GUID: 0x0021280001efe987
Port 1:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 209
LMC: 0
SM lid: 187
Capability mask: 0x02510868
Port GUID: 0x0021280001efe985
Link layer: IB
Port 2:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 73
LMC: 0
SM lid: 187
Capability mask: 0x02510868
Port GUID: 0x0021280001efe986
Link layer: IB
 
[root@exalytics ~]# sminfo

sminfo: sm lid 187 sm guid 0x10e00c16c0c0a0, activity count 124743184 priority 14 state 3 SMINFO_MASTER

Run the following command to make sure all the links are at 4X QDR:
```
[root@exalytics ~]# ibdiagnet -ls 10 -lw 4x
```

8.5 IPoIB Configuration (Layer-3)

Make sure that a unique IP address is available in the IPoIB subnet used in the Oracle Big Data Appliance rack. Never reuse an IP address that is already assigned, or allocated for Big Data Appliance or other purposes in this subnet. To identify a unique and available IP address with associated subnet mask, refer to the Oracle Big Data Appliance configuration..

To identify IPoIB network information from an existing Oracle Big Data Appliance compute node, run the following command:

[root@bda10 ~]# egrep '(IPADDR|NETMASK|NETWORK|BROADCAST)' /etc/sysconfig/network-scripts/ifcfg-bondib0
 
IPADDR=192.168.41.10
NETMASK=255.255.248.0
NETWORK=192.168.40.0
BROADCAST=192.168.47.255

Using the information obtained, you can request for a free and unique IP address belonging to Oracle Big Data Appliance's IPoIB subnet.

To find out the CIDR prefix, run the ipcalc command as follows:

[root@bda10 ~]# ipcalc -p 192.168.41.10 255.255.248.0
PREFIX=21

Note that in the example, the possible IP address host range is from 192.168.40.1 to 192.168.47.254. In the following step, IP 192.168.44.100/21 will be used as an IP address for Exalytics' IPoIB bond0 interface.

Run the exalytics_configure_network.sh script on Exalytics as root. When prompted, configure the IPoIB (InfiniBand) network interface. It is a bonded interface named bond0 using ib0 and ib1.

[root@exalytics root]# /opt/exalytics/bin/exalytics_configure_network.sh
Network Configuration Script Version exalytics-scripts-1.0.0.5ps3-52
Do you want to configure InfiniBand for Exadata connection? (y/n) [n]: y
Enter bond0 IP address:192.168.44.100
Use 192.168.44.162 for bond0 ip address (y/n) [y]:
Enter bond0 netmask address:255.255.248.0
Use 255.255.248.0 for bond0 netmask (y/n) [y]:

8.6 IPoIB Validation

To validate, ping an Oracle Big Data Appliance node's IPoIB address from Exalytics. If the ping is successful, you can use the connection for applications. You can test the application usage between the Exalytics and BDA. It may be necessary to install appropriate plugins and/or clients for Hadoop on Exalytics Machine. For further information, refer to product owner's guide.