7 Connecting Oracle Exalytics In-Memory Machine to Oracle Exadata Database Machine

This chapter describes how to connect an Oracle Exalytics In-Memory Machine and an Oracle Exadata Database Machine together on the same InfiniBand fabric.

It contains the following topics:

7.1 Understanding InfiniBand Card, Cable, and Switch Firmware Requirements

This section contains the following topics:

7.1.1 InfiniBand Card

Each Oracle Oracle Exalytics In-Memory Machine includes an InfiniBand card, such as the 2-port QDR InfiniBand Host Channel Adapter (HCA).

This InfiniBand card is installed in the Oracle Oracle Exalytics In-Memory Machine in its PCIe Slot 6.

7.1.2 InfiniBand Cables

You may use any spare 5-meter InfiniBand cables included in your Oracle Exadata Database Machine. If you need assistance, contact Oracle support personnel. You need these cables for connecting your Oracle Oracle Exalytics In-Memory Machine to an Oracle Exadata Database Machine on the same InfiniBand fabric.

7.1.3 InfiniBand Switches in the Oracle Exadata Database Machine

Oracle Exadata Database Machine includes two 36-port InfiniBand leaf switches (Sun Datacenter QDR InfiniBand Switch 36). These two switches located in the middle of the rack are referred to as leaf switches. A spine switch is included in unit 1 of Oracle Exadata Database Machine Full Rack and Half Rack configurations.

Note:

In this document, the leaf switch located in the lower unit of the Oracle Exadata Database Machine rack is referred to as leaf 1. The leaf switch located above leaf 1 in the Oracle Exadata Database Machine rack is referred to as leaf 2.

For more information about Oracle Exadata Database Machine's rack layout and hardware components, see the Oracle Exadata Database Machine Owner's Guide.

7.2 Physically Connecting the Network

This section describes the following topics:

7.2.1 Physical Layout Considerations

It is assumed that the rack containing the Oracle Oracle Exalytics In-Memory Machine and the Oracle Exadata Database Machine rack are placed side by side in the data center. Consider the length of the connecting InfiniBand cables when placing the racks in your data center. Do not place the Oracle Oracle Exalytics In-Memory Machine in an Oracle Exadata Database Machine rack.

7.2.2 Scenario 1: Single Exalytics In-Memory Machine with Oracle Exadata Database Machine

Figure 7-1 shows how a single (non-clustered) Oracle Oracle Exalytics In-Memory Machine is connected to an Oracle Exadata Database Machine rack.

Figure 7-1 Non-Clustered Oracle Exalytics In-Memory Machine Connected to Oracle Exadata Database Machine

Description of Figure 7-1 follows
Description of "Figure 7-1 Non-Clustered Oracle Exalytics In-Memory Machine Connected to Oracle Exadata Database Machine"

For high availability purposes, connect the two ports of the InfiniBand card, which is installed in the Oracle Oracle Exalytics In-Memory Machine, to separate leaf switches of the Oracle Exadata Database Machine. For example, as shown in Figure 7-1, you can connect port 1 of the Oracle Oracle Exalytics In-Memory Machine's InfiniBand card to the leaf 1 switch (Sun Datacenter InfiniBand Switch 36 installed in the lower unit of the Oracle Exadata Database Machine). In addition, you should connect port 2 of the Oracle Oracle Exalytics In-Memory Machine's InfiniBand card to the leaf 2 switch (Sun Datacenter InfiniBand Switch 36 installed in the upper unit of the Oracle Exadata Database Machine). Port 1 can function as the active port, and port 2 can function as the passive port.

Note:

On the leaft switches of the Oracle Exadata Database Machine, the following free ports can be used:

Ports 5B, 6A, 6B, 7A, 7B, and 12A

7.2.3 Scenario 2: Clustered Exalytics In-Memory Machine with Oracle Exadata Database Machine

Figure 7-2 shows how a cluster of Oracle Oracle Exalytics In-Memory Machines is connected to an Oracle Exadata Database Machine rack.

Figure 7-2 Clustered Oracle Exalytics In-Memory Machine Connected to Oracle Exadata Database Machine

Description of Figure 7-2 follows
Description of "Figure 7-2 Clustered Oracle Exalytics In-Memory Machine Connected to Oracle Exadata Database Machine"

For high availability purposes, connect the active port of the InfiniBand card included in each of the Oracle Oracle Exalytics In-Memory Machines to the leaf 1 switch (Sun Datacenter InfiniBand Switch 36 installed in the lower unit of the Oracle Exadata Database Machine). In addition, you should connect the passive port of the InfiniBand card included in each of the Oracle Oracle Exalytics In-Memory Machines to the leaf 2 switch (Sun Datacenter InfiniBand Switch 36 installed in the upper unit of the Oracle Exadata Database Machine). Port 1 can function as the active port, and port 2 can function as the passive port.

Note:

On the leaft switches of the Oracle Exadata Database Machine, the following free ports can be used:

Ports 5B, 6A,6B,7A,7B, and 12A

7.3 Choosing a Subnet Manager

Each InfiniBand network requires a Subnet Manager (SM). You can use the SM running on one of the Oracle Exadata Database Machine's leaf switches in your fabric.

For more information about using the InfiniBand switch, see the following documents:

7.4 Configuring the InfiniBand Connection

To configure the InfiniBand connection between Exalytics and Exadata, do the following:

  1. Enable automatic RDS kernel module loading on Exalytics.

    1. Edit the /etc/infiniband/openib.conf file and set SDP_LOAD=yes and RDS_LOAD=yes.

    2. Reboot the nodes.

  2. Create InfiniBand Listener on Exadata compute nodes.

    1. Edit the /etc/infiniband/openib.conf file and set SDP_LOAD=yes and RDS_LOAD=yes.

    2. Edit the /etc/ofed/libsdp.conf file and find the following lines:

      use both server * :
      use both client * :
      

      Change these lines to read as follows:

      use tcp server * *:*
      use tcp client * *:*
      

      use tcp indicates that TCP is the default protocol and SDP will be used from specific applications using their local configuration.

    3. Edit the /etc/modprobe.conf file and add this line at the end:

      options ib_sdp sdp_zcopy_thresh=0 recv_poll=0
      
    4. Reboot each compute node.

    5. Edit the /etc/hosts file on each Exalytics machine and Exadata Database Machine compute node to add the virtual IP addresses for use on the InfiniBand network, for example, Exadata Database Machine quarter rack (2 compute nodes). Ensure that these IP addresses are not in use on the private network.

      192.168.10.21 dm01db01-ibvip.mycompany.com dm01db01-ibvip
      192.168.10.22 dm01db02-ibvip.mycompany.com dm01db02-ibvip
      
    6. On one of the Exadata Database Machine compute nodes, as the root user execute the following commands to create a network resource for the InfiniBand network, and add the starting virtual IP addresses for each compute node on it. Make appropriate substitutions for the network and subnet mask in use on your private network (it may not be 192.168.10.0/255.255.255.0). Ask your admin or use ipcalc, based on the network mask to calculate the network.

      srvctl add network -k 2 -S 192.168.10.0/255.255.255.0/bondib0
      srvctl add vip -n dm01db01 -A dm01db01-ibvip/255.255.255.0/bondib0 -k 2
      srvctl add vip -n dm01db02 -A dm01db02-ibvip/255.255.255.0/bondib0 -k 2
      srvctl start vip -i dbm01db01-ibvip
      srvctl start vip -i dbm01db02-ibvip
      
    7. On one of the Exadata Database Machine compute nodes, as oracle user (or owner of the Grid Infrastructure ORACLE_HOME) execute the following command:

      srvctl add listener -l LISTENER_IB -k 2 -p TCP:1522/SDP:1522
      
    8. On each Exadata Database Machine compute node, as oracle user (or owner of the Grid Infrastructure ORACLE_HOME), create or append to tnsnames.ora in GRID_HOME/network/admin with the following new entries (note that the *IBREMOTE and *IBLOCAL entries will be slightly different on each node to reference the proper remote and local VIP names—you cannot copy the entries from one node to another without some modifications):

      DBM =
      (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = dm01-scan)(PORT = 1521))
      (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = dbm)
      ))
      DBM_IB =
      (DESCRIPTION =
      (LOAD_BALANCE=on)
      (ADDRESS = (PROTOCOL = TCP)(HOST = dm01db01-ibvip)(PORT = 1522))
      (ADDRESS = (PROTOCOL = TCP)(HOST = dm01db02-ibvip)(PORT = 1522))
      (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = dbm)
      ))
      LISTENER_IBREMOTE =
      (DESCRIPTION =
      (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = dm01db02-ibvip.mycompany.com)(PORT = 1522))
      ))
      LISTENER_IBLOCAL =
      (DESCRIPTION =
      (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = dm01db01-ibvip.mycompany.com)(PORT = 1522))
      (ADDRESS = (PROTOCOL = SDP)(HOST = dm01db01-ibvip.mycompany.com)(PORT = 1522))
      ))
      LISTENER_IPLOCAL =
      (DESCRIPTION =
      (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = dm0101-vip.mycompany.com)(PORT = 1521))
      ))
      LISTENER_IPREMOTE =
      (DESCRIPTION =
      (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = dm01-scan.mycompany.com)(PORT = 1521))
      ))
      
    9. On one Exadata Database Machine compute node, connect to the database instance as sysdba:

      SQLPLUS> alter system set listener_networks='((NAME=network2) 
      (LOCAL_LISTENER=LISTENER_IBLOCAL)(REMOTE_LISTENER=LISTENER_IBREMOTE))',
       '((NAME=network1)(LOCAL_LISTENER=LISTENER_IPLOCAL)
      (REMOTE_LISTENER=LISTENER_IPREMOTE))' scope=both;
      
    10. On one Exadata Database Machine compute node, as oracle user (or owner of the Grid Infrastructure ORACLE_HOME) restart the listener named LISTENER_IB by running the following commands:

      srvctl stop listener -l LISTENER_IB
      srvctl start listener -l LISTENER_IB
      
  3. On the Exalytics machine, edit sys.odbc.ini file to make an entry that points to the Exadata DB host:

    OracleNetServiceName=DWH.ESS.LOCAL_IB
    

7.5 Monitoring the InfiniBand Fabric

You can monitor many aspects of the InfiniBand fabric by running InfiniBand commands on the Sun Datacenter InfiniBand Switch 36, which is included in the Oracle Exadata Database Machine.

For more information about these commands, see the Sun Datacenter InfiniBand Switch 36 Command Reference.

7.6 Checking Network Health

After establishing InfiniBand network connectivity, you can verify the status of the basic InfiniBand network connectivity with rds-ping, rds-info -l by using the ibstat command on the Exalytics machine.

The ibstat command displays the basic status, as in the following example:

# ibstat

CA 'mlx4_0'
        CA type: MT26428
        Number of ports: 2
        Firmware version: 2.7.8130
        Hardware version: b0
        Node GUID: 0x0021280001cf577e
        System image GUID: 0x0021280001cf5781
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 106
                LMC: 0
                SM lid: 54
                Capability mask: 0x02510868
                Port GUID: 0x0021280001cf577f
                Link layer: IB
        Port 2:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 95
                LMC: 0
                SM lid: 54
                Capability mask: 0x02510868
                Port GUID: 0x0021280001cf5780
                Link layer: IB

The physical state of connected ports is shown as LinkUp.

7.7 Testing the Extended Network Connectivity

After verifying the basic InfiniBand network connectivity, you can run the ibhosts, ibswitches, and ibnetdiscover commands on the switch CLI.

The ibhosts command displays host nodes in the InfiniBand fabric, the ibswitches command displays the InfiniBand switch nodes in the fabric, and the ibnetdiscover command discovers the InfiniBand topology.

For more information such monitoring and diagnostic commands, see the Sun Datacenter InfiniBand Switch 36 Command Reference.