This chapter describes how to connect Oracle Exalytics In-Memory Machine and an Oracle Big Data Appliance together on the same InfiniBand fabric.
It contains the following sections:
Oracle Big Data Appliance has three in-built InfiniBand switches. Use the two NM2-GW leaf switches to connect the Oracle Exalytics machine. The Oracle Exalytics machine has one InfiniBand HCA card, but it does not include any InfiniBand switch or any instance of InfiniBand Subnet Manager.
To connect Oracle Exalytics to Oracle Big Data Appliance, you need a pair of InfiniBand cables depending on the length requirements at the datacenter. If the total distance between Exalytics and Oracle Big Data Appliance leaf switch ports is less than 10 meters, you can use copper cables. If the length is more than 10 meters, you must use fiber optic cable for InfiniBand with appropriate transceivers.
The base images of both Oracle Big Data Appliance and Oracle Exalytics include software support for InfiniBand connectivity.
To view software details of the Oracle Exalytics machine, run the following commands:
Note:
From the following commands, use imageinfo
command for 1.0.0.2 systems and earlier. Use exalytics_imageinfo
command for 1.0.0.3 systems and later.
[root@exalytics root]# exalytics_imageinfo Exalogic 1.0.0.5 (build:r) Image version : 1.0.0.5 Image build version : Creation timestamp : Wed 19 Jun 2013 12:26:49 PM BST Kernel version : 2.6.32-100.23.1.el5 Image activated : 2013-09-03 20:49:13 -0400 Image status : SUCCESS [root@exalytics root]# ofed_info | head -1 BXOFED-1.5.2-1.3.8000: [root@exalytics root]# lsmod | grep ipoib ib_ipoib 67547 1 ib_cm 32459 2 rdma_cm,ib_ipoib ib_sa 36903 5 rdma_ucm,rdma_cm,mlx4_vnic,ib_ipoib,ib_cm ib_core 61642 14 rdma_ucm,rds_rdma,ib_sdp,rdma_cm,iw_cm,ib_uverbs,ib_umad,mlx4_vnic,mlx4_ib,ib_mthca,ib_ipoib,ib_cm,ib_sa,ib_mad ipv6 286932 222 ib_sdp,ib_addr,ib_ipoib,bonding [root@exalytics root]# lsmod | grep rds rds_tcp 9460 0 rds_rdma 72457 1 rds 91561 2 rds_tcp,rds_rdma rdma_cm 29058 3 rdma_ucm,rds_rdma,ib_sdp ib_core 61642 14 rdma_ucm,rds_rdma,ib_sdp,rdma_cm,iw_cm,ib_uverbs,ib_umad,mlx4_vnic,mlx4_ib,ib_mthca,ib_ipoib,ib_cm,ib_sa,ib_mad [root@exalytics root#
To view software support of the Oracle Big Data Appliance, run the following commands:
[root@bda10 ~]# imageinfo Big Data Appliance Image Info IMAGE_CREATION_DATE : Tue Aug 20 23:36:32 PDT 2013 IMAGE_LABEL : BDA_2.2.1_LINUX.X64_130820 IMAGE_VERSION : 2.2.1 KERNEL_VERSION : 2.6.32-200.21.1.el5uek BDA_RPM_VERSION : bda-2.2.1-1 OFA_RPM_VERSION : ofa-2.6.32-200.21.1.el5uek-1.5.5-4.0.55.4 JDK_VERSION : jdk-1.6.0_51-fcs HADOOP_VERSION : 2.0.0-cdh4.3.0 [root@bda10 ~]# ofed_info | head -1 OFED-IOV-1.5.5-1.0.0120: [root@bda10 ~]# lsmod | grep ipoib ib_ipoib 67931 1 ib_cm 35714 2 rdma_cm,ib_ipoib ipv6 290974 5 bonding,cnic,ib_sdp,ib_addr,ib_ipoib ib_sa 39126 6 rdma_ucm,rdma_cm,ib_ipoib,ib_cm,mlx4_vnic,mlx4_ib ib_core 66876 12 rdma_ucm,ib_sdp,rdma_cm,iw_cm,ib_ipoib,ib_cm,ib_uverbs,ib_umad,mlx4_vnic,mlx4_ib,ib_sa,ib_mad [root@bda10 ~]# lsmod | grep rds rds 94503 0 [root@bda10 ~]#
On both GW switches of Oracle Big Data Appliance, four ports are available for external connections-5A, 5B, 6A, 6B. Oracle recommends that you use ports 5A on both leaf switches. If these ports are not available, use any other available port on the leaf switches.
Note:
Never connect a host (HCA) system to the spine switch inside a rack.
If the Oracle Big Data Appliance rack is connected to other Oracle Engineered Systems, port 5-6[A|B] may not be available. In such a case, it is recommended to use any other available port on the InfiniBand leaf switch of the Oracle Big Data Appliance rack. If no free ports are available, contact Oracle Support.
Figure 8-1 shows how to connect an Exalytics to a standalone BDA rack.
Once the cables are connected and you can see green LEDs on the ports, you must validate the InfiniBand connection.
Log in to the Oracle Big Data Appliance compute node or Exalytics machine.
From the Linux root shell, run the ibhosts
command to verify if the newly connected (Exalytics/Oracle Big Data Appliance) machine is visible.
On the Exalytics machine, run the ibstat
command to see the local ports' status. If an active Subnet Manager is present, then the ports should be Active.
[root@exalytics ~]# ibstat CA 'mlx4_0' CA type: MT26428 Number of ports: 2 Firmware version: 2.7.8130 Hardware version: b0 Node GUID: 0x0021280001efe984 System image GUID: 0x0021280001efe987 Port 1: State: Active Physical state: LinkUp Rate: 40 Base lid: 209 LMC: 0 SM lid: 187 Capability mask: 0x02510868 Port GUID: 0x0021280001efe985 Link layer: IB Port 2: State: Active Physical state: LinkUp Rate: 40 Base lid: 73 LMC: 0 SM lid: 187 Capability mask: 0x02510868 Port GUID: 0x0021280001efe986 Link layer: IB [root@exalytics ~]# sminfo sminfo: sm lid 187 sm guid 0x10e00c16c0c0a0, activity count 124743184 priority 14 state 3 SMINFO_MASTER
Run the following command to make sure all the links are at 4X QDR:
[root@exalytics ~]# ibdiagnet -ls 10 -lw 4x
Make sure that a unique IP address is available in the IPoIB subnet used in the Oracle Big Data Appliance rack. Never reuse an IP address that is already assigned, or allocated for Big Data Appliance or other purposes in this subnet. To identify a unique and available IP address with associated subnet mask, refer to the Oracle Big Data Appliance configuration..
To identify IPoIB network information from an existing Oracle Big Data Appliance compute node, run the following command:
[root@bda10 ~]# egrep '(IPADDR|NETMASK|NETWORK|BROADCAST)' /etc/sysconfig/network-scripts/ifcfg-bondib0 IPADDR=192.168.41.10 NETMASK=255.255.248.0 NETWORK=192.168.40.0 BROADCAST=192.168.47.255
Using the information obtained, you can request for a free and unique IP address belonging to Oracle Big Data Appliance's IPoIB subnet.
To find out the CIDR prefix, run the ipcalc
command as follows:
[root@bda10 ~]# ipcalc -p 192.168.41.10 255.255.248.0 PREFIX=21
Note that in the example, the possible IP address host range is from 192.168.40.1 to 192.168.47.254. In the following step, IP 192.168.44.100/21 will be used as an IP address for Exalytics' IPoIB bond0 interface.
Run the exalytics_configure_network.sh
script on Exalytics as root. When prompted, configure the IPoIB (InfiniBand) network interface. It is a bonded interface named bond0 using ib0 and ib1.
[root@exalytics root]# /opt/exalytics/bin/exalytics_configure_network.sh Network Configuration Script Version exalytics-scripts-1.0.0.5ps3-52 Do you want to configure InfiniBand for Exadata connection? (y/n) [n]: y Enter bond0 IP address:192.168.44.100 Use 192.168.44.162 for bond0 ip address (y/n) [y]: Enter bond0 netmask address:255.255.248.0 Use 255.255.248.0 for bond0 netmask (y/n) [y]:
To validate, ping an Oracle Big Data Appliance node's IPoIB address from Exalytics. If the ping is successful, you can use the connection for applications. You can test the application usage between the Exalytics and BDA. It may be necessary to install appropriate plugins and/or clients for Hadoop on Exalytics Machine. For further information, refer to product owner's guide.