1. Owner's Guide
  2. Installation and Configuration
  3. Expanding a Recovery Appliance Rack with Additional Storage Servers
  4. Preparing to Extend Recovery Appliance

Preparing to Extend Recovery Appliance

Before expanding a rack, review the cabling requirements and collect information about the current rack. This section contains the following topics:

Estimating RDMA Network Fabric Cable Path Lengths

Ensure that you have the required number and lengths of the RDMA Network Fabric cables for your installation. Cable paths should be as short as possible. After you calculate the length of a cable path, consider the following ways that the length might increase. Then select the shortest cable to satisfy the length requirement.

Consider the following:

  • Bends in the cable path increase the required length of the cable. A cable rarely travels in a straight line from connector to connector. Bends in the cable path are necessary, and each bend increases the total length.

  • Bundling increases the required length of the cables. Bundling causes one or more cables to follow a common path. However, the bend radius is different in different parts of the bundle. If the bundle is large and unorganized, and there are many bends, one cable might bend only on the inner radius, while another cable might bend only on the outer radius. In this situation, the difference in the required cable lengths is substantial.

  • If you are routing the RDMA Network Fabric cable under the floor, consider the height of the raised floor when calculating the cable path length.

Obtaining the Current Configuration Information

Use the current configuration of the Recovery Appliance rack to plan patching requirements, configure new IP addresses, and so on.

Log into the first compute server in the rack and collect the following information:

  • The exachk report for the rack. See My Oracle Support Doc ID 1070954.1 for information about the exachk utility.

  • Image history information. Use the following command: 

    # dcli -g ~/all_group -l root "imagehistory" > imagehistory.txt

  • Current IP addresses of all storage servers and compute servers. Use the following command: 

    # dcli -g ~/all_group -l root "ifconfig" > ifconfig_all.txt

  • Configuration of the storage servers, disks, flash logs, and IORM plans. Use the following commands: 

    # dcli -g ~/cell_group -l root "cellcli -e list cell detail" > cell_detail.txt

# dcli -g ~/cell_group -l root "cellcli -e list physicaldisk detail" >   \
physicaldisk_detail.txt

# dcli -g ~/cell_group -l root "cellcli -e list griddisk attributes name,offset,size,status,asmmodestatus,asmdeactivationoutcome" > griddisk.txt

# dcli -g ~/cell_group -l root "cellcli -e list flashcache detail" > fc_detail.txt

# dcli -g ~/cell_group -l root "cellcli -e list flashlog detail" > fl_detail.txt

# dcli -g ~/cell_group -l root "cellcli -e list iormplan detail" > iorm_detail.txt

  • HugePages memory configuration on the compute servers. Use the following command: 

    dcli -g ~/dbs_group -l root "cat /proc/meminfo | grep 'HugePages'" > hugepages.txt

  • InfiniBand switch information. Use the following command: 

    # ibswitches > ibswitches.txt

  • Firmware version of the Sun Datacenter InfiniBand Switch 36 switches. Use the nm2version command on each switch.

  • The following network files on the first compute server in the rack:

    • /etc/resolv.conf

    • /etc/ntp.conf

    • /etc/network

    • /etc/sysconfig/network-scripts/ifcfg-*

  • Any users, user identifiers, groups, and group identifiers created for cluster-managed services that must be created on the new servers:

    • /etc/passwd

    • /etc/group

  • The current cluster status. Use the following command:

    # crsctl stat res -t > crs_stat.txt

  • Patch information from the Grid Infrastructure and Oracle homes. Use the following commands as the Grid Infrastructure home owner and the Oracle home owner:

    /u01/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch lsinventory -oh    \
GRID_HOME -detail -all_nodes > opatch_grid.txt

/u01/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch lsinventory -oh    \
ORACLE_HOME -detail -all_nodes >> opatch_oracle.txt

    In the preceding commands, GRID_HOME is the path of the Grid Infrastructure home directory, and ORACLE_HOME is the path of the Oracle home directory.

Generating the Network Configuration Files

Use the Oracle Exadata Deployment Assistant to list the IP addresses for the new servers. Specify only the number of new servers. Define all other items, such as the Oracle home location and owner, the same as the existing rack configuration. In particular, the bonding configuration of the new servers must match the existing servers in the rack. The Oracle Exadata Deployment Assistant InfiniBand Network Configuration Page enables you to select the type of bonding.

You must install the Recovery Appliance software in the new storage servers, after installing them in the rack.

See Also:

Moving the Audit and Diagnostic Files

Move to delete files in the $GRID_HOME/rdbms/audit directory and the $GRID_HOME/log/diagnostics directory before extending a cluster. Oracle recommends doing this task a day or two before the planned extension to allow sufficient time.

Synchronizing Release and Patch Levels

The servers probably have a later release or patch level than the rack where they will be added. You can either update the rack to the later release or reimage the new servers to match the current rack. In either case, ensure that all servers, and all Sun Datacenter InfiniBand Switch 36 switches, are at the same patch level.

Also consider how to handle the Grid Infrastructure and database home releases and bundle patch updates. If you plan to apply the new patches, then Oracle recommends upgrading the existing servers, so that the new servers inherit the releases as part of the extension procedure. Then fewer servers are patched. Apply the patches to the existing servers in advance, thereby reducing the total amount of work required during the extension.