9 Configuring Ingest Over InfiniBand

The ingest network connects the protected Oracle Database servers to Recovery Appliance for backup within the same data center. You can optionally configure the ingest network over InfiniBand instead of or in addition to a private, high-speed Ethernet network.

Ingest Over InfiniBand is supported only with compute servers running release March 2015 or later with database patch 12.1.0.2 DBBP4 applied.

This chapter contains the following sections:

9.1 About Ingest Over InfiniBand

Oracle recommends that the Recovery Appliance receives backups from all Oracle Exadata Database Machine racks over an Ethernet network. This configuration, shown in "Example of Network Connections for Recovery Appliance", supports thousands of protected databases equally. Nonetheless, you can configure local Oracle Exadata Database Machine racks to back up to Recovery Appliance over the InfiniBand network. This configuration provides the local Exadata racks with priority access to Recovery Appliance. Redo and SQL*Net control information from the protected databases are still sent over the 10 Gb Ethernet network.

Note:

Do not attempt to use the InfiniBand network for redo transport as this configuration is not supported.

To configure Ingest Over InfiniBand:

Ensure that all racks in the InfiniBand fabric are in the same subnet and have unique IP addresses. The IP addresses can be registered in the DNS, but it is not mandatory.

When deploying a new Recovery Appliance rack, identify the same subnet as Oracle Exadata Database Machine and specify a unique range of IP addresses in Oracle Exadata Deployment Assistant.

After deployment, correct any issues with the subnet or IP addresses. See Oracle Exadata Database Machine Maintenance Guide for instructions on changing InfiniBand IP addresses and host names.
Edit the /etc/hosts file on all servers in the InfiniBand fabric to list all IP addresses and host names.
Create a VIP IP address for each compute server in the Recovery Appliance rack.

See "Adding VIPs to the Recovery Appliance InfiniBand Network".
Cable the Recovery Appliance and Oracle Exadata Database Machine racks together.

9.2 Cabling Recovery Appliance to Oracle Exadata Database Machine

When configuring Ingest Over InfiniBand, you can connect Recovery Appliance to one or more local Oracle Exadata Database Machine racks using one of these methods:

If the Recovery Appliance rack has a spine switch, then use the multirack cabling tables for the appropriate number of racks. See Multirack Cabling Tables.
If the Recovery Appliance rack has no spine switch, then you can cable it in a way that is similar to cabling together multiple Oracle Exadata Database Machine quarter racks.

Figure 9-1 shows the cable connections between a Recovery Appliance rack and an Oracle Exadata Database Machine quarter rack. The leaf switches within each rack retain their existing seven connections. The leaf switches interconnect the racks with two links, each using the ports reserved for external connectivity.

Figure 9-1 Recovery Appliance Connected to an Exadata Quarter Rack

Description of "Figure 9-1 Recovery Appliance Connected to an Exadata Quarter Rack"

Figure 9-2 shows the cable connections from Recovery Appliance to an Oracle Exadata Database Machine half rack or full rack. The leaf switches within each rack retain their existing seven connections. The leaf switches interconnect the racks with two links, each using the ports reserved for external connectivity.

Figure 9-2 Recovery Appliance Connected to an Exadata Half or Full Rack

Description of "Figure 9-2 Recovery Appliance Connected to an Exadata Half or Full Rack"

Figure 9-3 shows the cable connections from Recovery Appliance to two or more Exadata racks. All Oracle Exadata Database Machine racks must be half or full racks, and must be interconnected using a fat-tree topology.

Each Recovery Appliance leaf switch has two links to the spine switches in the Exadata racks. When connecting five or more racks, use one link to the spine switches.

Figure 9-3 Recovery Appliance Connected to Multiple Exadata Racks

Description of "Figure 9-3 Recovery Appliance Connected to Multiple Exadata Racks"

9.3 Adding VIPs to the Recovery Appliance InfiniBand Network

After cabling the Recovery Appliance rack with one or more Oracle Exadata Database Machine racks, you must add the virtual IP addresses (VIPs) to the Recovery Appliance InfiniBand network.

To add the VIPs:

Verify the subnet information for the Recovery Appliance InfiniBand network:

$ oifcfg iflist -p -n | grep PRIVATE
eth0 10.10.40.0 PRIVATE 255.255.248.0
ib0 192.168.40.0 PRIVATE 255.255.248.0
ib1 192.168.40.0 PRIVATE 255.255.248.0
bondeth0 10.10.62.0 PRIVATE 255.255.255.0
bondeth1 10.10.241.192 PRIVATE 255.255.255.192

Change to the root user.
Add a new network to Oracle Cluster Ready Services (CRS):
```
# srvctl add network -k 3 -S 192.168.40.0/255.255.248.0/ib0\|ib1
```
Note:
You must specify network #3 (use the "-k 3" option) when you configure Ingest Over InfiniBand.

Verify that the new network is configured correctly:

# srvctl config network -k 3 
Network 3 exists 
Subnet IPv4: 192.168.40.0/255.255.248.0/ib0:ib1, static 
Subnet IPv6: 
Ping Targets: 
Network is enabled 
Network is individually enabled on nodes: 
Network is individually disabled on nodes:

Add the VIPs to the new CRS network:

# srvctl add vip -n raadm01 -A raadm01-priv-vip.example.com/255.255.248.0/ib0\|ib1 -k 3 
#
# srvctl add vip -n raadm02 -A raadm02-priv-vip.example.com/255.255.248.0/ib0\|ib1 -k 3

Start the VIPs on the Recovery Appliance InfiniBand network:

# srvctl start vip -i raadm01-priv-vip.example.com 
# srvctl start vip -i raadm02-priv-vip.example.com

Change to the oracle user.

Verify that the VIPs are running:

$ srvctl status vip -vip raadm01-priv-vip.example.com
VIP 192.168.40.79 is enabled
VIP 192.168.40.79 is running on node: raadm01

$ srvctl status vip -vip raadm02-priv-vip.example.com
VIP 192.168.40.80 is enabled
VIP 192.168.40.80 is running on node: raadm02

9.4 Updating Recovery Appliance With the New VIPs for InfiniBand

Recovery Appliance recognizes a preferred network and a default network for receiving backups into the appliance. The deployment procedure configures the 10 Gb Ethernet network as the default network. The InfiniBand network becomes the preferred network. The protected databases use the preferred network if they can contact it; otherwise, they use the default network.

The following procedure updates a table in Oracle Database with the VIPs created in "Adding VIPs to the Recovery Appliance InfiniBand Network".

To update Recovery Appliance with the Virtual IPs:

While logged in to a compute server in the Recovery Appliance rack, open SQL*Plus and connect as RASYS.

View the current values in the RAI_HOST table:

SQL> SELECT * FROM rai_host

NODE_NAME            ADMIN_IP_ADDRESS  BACKUP_IP_ADDRESS  
-------------------  ----------------  --------------------------  
raadm01.example.com  10.10.40.171      10.10.62.139
raadm02.example.com  10.10.40.172      10.10.62.140
 
 
REPLICATION_IP_ADDRESS
-----------------------
10.10.241.204
10.10.241.205

Update the BACKUP_IP_ADDRESS column with the VIP associated with the InfiniBand network for each host. Use the appropriate values for the IP addresses and compute server (node) names for your installation:

SQL> UPDATE rai_host
     SET backup_ip_address='192.168.40.79,'|| backup_ip_address 
     WHERE node_name = 'raadm01.example.com';
 
SQL> UPDATE rai_host
     SET backup_ip_address='192.168.40.80,'||backup_ip_address 
     WHERE node_name = raadm02.example.com'

Save your changes to the table:
```
SQL> commit;
```

Query RAI_HOST to verify that the settings are correct:

SQL> SELECT * FROM rai_host;
 
NODE_NAME            ADMIN_IP_ADDRESS  BACKUP_IP_ADDRESS  
-------------------  ----------------  --------------------------  
raadm01.example.com  10.10.40.171      192.168.40.79,10.10.62.139
raadm02.example.com  10.10.40.172      192.168.40.80,10.10.62.140

REPLICATION_IP_ADDRESS
-----------------------
10.10.241.204
10.10.241.205

9.5 Update the Database Initialization Parameters

Recovery Appliance processes database backup and restore requests using dispatchers, which prevent an excessive number of dedicated connections to the appliance database.

The init.ora file defines the dispatchers for the default ingest network as shown in this example:

*.dispatchers=''
ra1.dispatchers='(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=raingest01-vip.example.com))(SDU=65536))(SERVICE=ZDLRAXDB)(DISPATCHERS=4)'
ra2.dispatchers='(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=raingest02-vip.example.com))(SDU=65536))(SERVICE=ZDLRAXDB)(DISPATCHERS=4)'

You can use the SQL ALTER SYSTEM command to add the dispatchers for the InfiniBand network to the init.ora parameter settings.

To add the dispatchers for InfiniBand:

While still connected to Oracle Database as RASYS, configure the dispatchers for the InfiniBand VIPs as shown in these examples:

SQL> ALTER SYSTEM SET dispatchers=
'(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)
(HOST=raadm01-priv-vip.example.com))(SDU=65536))
(SERVICE=ZDLRAXDB)(DISPATCHERS=4)' SCOPE=BOTH SID='ra1';
 
SQL> ALTER SYSTEM SET dispatchers=
'(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)
(HOST=raadm02-priv-vip.example.com))(SDU=65536))
(SERVICE=ZDLRAXDB)(DISPATCHERS=4)' SCOPE=BOTH SID='ra2';