1 Preparing to Extend Oracle Exadata Database Machine

Before extending any rack hardware, review the safety precautions and cabling information, and collect information about the current rack in this section.

1.1 About Extending Oracle Engineered System

You can extend Oracle Engineered System either by adding servers to the current configuration or by cabling together multiple racks.

Here are considerations when extending the Oracle Engineered System:

  • Elastic configuration: You can extend Oracle Engineered System from a fixed or custom configuration to another custom configuration by adding any combination of database or storage servers up to the allowed maximum.

  • You can cable together any combination of Oracle Engineered System half racks and full racks.

  • XM8 racks: You must order Cisco Nexus 9336C-FX2 Switch and cables before extending Oracle Engineered System X8M racks.

  • X4-2 racks: You must order a Sun Datacenter InfiniBand Switch 36 switch and cables before extending Oracle Engineered System X4-2 racks.

Note:

  • The cable lengths shown in Multi-Rack Cabling Tables assume the racks are adjacent to each other. If the racks are not adjacent or use overhead cabling trays, then they may require longer cables lengths. Up to 100 meters is supported.

    Only optical cables are supported for lengths greater than 5 meters.

  • Only X8M racks can extend other X8M racks.

  • Earlier Oracle Engineered System Racks can be extended with later Oracle Engineered System Racks up to X8, but not X8M.

  • When extending Oracle Engineered System Eighth Rack with Oracle Exadata Storage Expansion Rack, ensure there are two separate disk groups. One disk group is for the drives in the Oracle Engineered System Eighth Rack, and the other disk group is for the drives in Oracle Exadata Storage Expansion Rack.

Multiple Oracle Engineered Systems can be run as separate environments, and connect through the RDMA Network Fabric. If you are planning to utilize multiple Oracle Engineered Systems in this manner, then note the following:

  • All servers on the RDMA Network Fabric must have a unique IP address. When Oracle Engineered System is deployed, the default network is 192.168.10.1. You must modify the IP addresses before re-configuring the RDMA Network Fabric. Failure to do so causes duplicate IP addresses.

  • After modifying the network, run the appropriate verification tools:

    • X8M: InfiniBand Transport Layer systems based on a RoCE Network Layer

      Run infinicheck commands to verify the network is working properly. You will create a file that contains IP addresses for Exadata Storage Servers, such as combined_cellip.ora.

    • X8 and earlier: InfiniBand Transport Layer systems based on an InfiniBand Network Layer

      Run the verify-topology and infinicheck commands to verify the network is working properly. You will create a file that contains IP addresses for Exadata Storage Servers, such as combined_cellip.ora. Here is an example:

      # cd /opt/oracle.SupportTools/ibdiagtools
      # ./verify-toplogy -t fattree
      # ./infinicheck -c /tmp/combined_cellip.ora -b
      
  • When Oracle Engineered Systems run in separate clusters, do not modify the cellip.ora files. The cellip.ora file on a database server should only include the IP addresses for the storage servers used with that database server.

  • Cells with disk types different from what is already installed can be added, but the disk types cannot be mixed in the same Oracle Automatic Storage Management (Oracle ASM) disk group. For example, if the existing disk groups all use high performance disks, and cells with high capacity disks are being added, then it is necessary to create new disk groups for the high capacity disks.

    When adding the same type of disk, ensure that the grid disk sizes are exactly the same even if the new disks are larger than the existing ones. For example, if the existing disks are 3 TB, and the additional disks are 4 TB, then it is necessary to create grid disks that match the size on the 3 TB disks. A new disk group can be created using the extra 1 TB of disk space.

  • In order to access Exadata Storage Servers in one Oracle Engineered System by another Oracle Engineered System when they are not running as a single cluster, Exadata Storage Servers must have unique Oracle ASM disk group and failure group names on each Oracle Engineered System.

    For example, for two Oracle Engineered Systems cabled together but run as separate clusters, the following names should be unique:

    • Cell name
    • Cell disk name
    • Grid disk name
    • Oracle ASM failure group name
  • All equipment receives a Customer Support Identifier (CSI). Any new equipment for the Oracle Engineered System has a new CSI. Contact Oracle Support Services to reconcile the new CSI with the existing Oracle Engineered System CSI. Have the original instance numbers or serial numbers available, as well as the new numbers when contacting Oracle Support Services.

For X8M: InfiniBand Transport Layer systems based on a RoCE Network Layer

The RDMA Network Fabric can be used for external connectivity. The external connectivity ports in the Cisco Nexus 9336C-FX2 Switch can connect to media servers for tape backup, data loading, and client and application access. Use the available ports on the leaf switches for external connectivity. There are 12 ports per rack. The available ports are 1, 2, 3, 34, 35 and 36 in each leaf switch. For high availability connections, connect one port to one leaf switch and the other port to the second leaf switch. The validated InfiniBand cable lengths are:

  • Up to 5 meters passive copper QSFP cables or QSFP breakout to SFP Cables
  • Up to 100 meters fiber optic QSFP cables or QSFP breakout to SFP Cables

For X8 and earlier: InfiniBand Transport Layer systems based on an InfiniBand Network Layer

The RDMA Network Fabric can be used for external connectivity. The external connectivity ports in the Sun Datacenter InfiniBand Switch 36 switches can connect to media servers for tape backup, data loading, and client and application access. Use the available ports on the leaf switches for external connectivity. There are 12 ports per rack. The available ports are 5B, 6A, 6B, 7A, 7B, and 12A in each leaf switch. For high availability connections, connect one port to one leaf switch and the other port to the second leaf switch. The validated InfiniBand cable lengths are:

  • Up to 5 meters passive copper 4X QDR QSFP cables
  • Up to 100 meters fiber optic 4X D=QDR QSFP cables

Related Topics

1.2 Reviewing the Safety Precautions

Before upgrading Oracle Exadata Database Machines, read Important Safety Information for Sun Hardware Systems included with the rack.

Note:

Contact a service representative or Oracle Advanced Customer Support to confirm that Oracle has qualified your equipment for installation and use in Oracle Exadata Database Machine. Oracle is not liable for any issues when you install or use non-qualified equipment.

1.3 Reviewing the Cable Precautions

Review the following RDMA Network Fabric cable precautions before working with the cables:

  • Fiber optic cables with laser transceivers must be of type Class 1.

  • Do not allow any copper core cable to bend to a radius tighter than 127 mm (5 inches). Tight bends can damage the cables internally.

  • Do not allow any optical cable to bend to a radius tighter than 85 mm (3.4 inches). Tight bends can damage the cables internally.

  • Do not use zip ties to bundle or support cables. The sharp edges of the ties can damage the cables internally. Use hook-and-loop straps.

  • Do not allow any cable to experience extreme tension. Do not pull or drag the cables, which can damage them internally.

  • Unroll a cable for its length.

  • Do not twist cables more than one revolution for their entire length. Twisting a cable might damage it internally.

  • Do not route cables where they can be stepped on, or experience rolling loads. A crushing effect can damage the cable internally.

1.4 Estimating Cable Path Lengths

Cable paths should be as short as possible. When the length of a cable path has been calculated, select the shortest cable to satisfy the length requirement. When specifying a cable, consider the following:

  • Bends in the cable path increase the required length of the cable. Rarely does a cable travel in a straight line from connector to connector. Bends in the cable path are necessary, and each bend increases the total length.

  • Bundling increases the required length of the cables. Bundling causes one or more cables to follow a common path. However, the bend radius is different in different parts of the bundle. If the bundle is large and unorganized, and there are many bends, one cable might experience only the inner radius of bends, while another cable might experience the outer radius of bends. In this situation, the differences of the required lengths of the cables is quite substantial.

  • When calculating the cable path length and route is under the floor, take into consideration the height of the raised floor.

1.5 Bundling Cables

When bundling RDMA Network Fabric cables in groups, use hook-and-loop straps to keep cables organized. If possible, use color-coordinated straps to help identify cables and their routing. Splitter and C-4 copper conductor cables are fairly thick and heavy for their length. Consider the retention strength of the hook-and-loop straps when supporting cables. Bundle as few cables as reasonably possible. If the cables break free of their straps and fall free, the cables might break internally when they strike the floor or from sudden changes in tension.

Bundle the cables using many hook-and-loop straps. Oracle recommends that no more than eight cables be bundled together.

Place the hook-and-loop straps as close together as reasonably possible, for example, one strap every foot (0.3 m). If a cable breaks free from a strap, then the cable cannot fall far before it is retained by another strap.

1.5.1 Floor and Underfloor Delivery of Cables

The RDMA Network Fabric switches accept cables from floor or underfloor delivery. Floor and underfloor delivery limits the tension in the cable to the weight of the cable for the rack height of the switch.

Note:

Overhead cabling details are not included in this guide. For details on overhead cabling, contact a certified service engineer.

1.6 Reviewing the Cable Management Arm Guidelines

Review the following cable management arm (CMA) guidelines before routing the cables.

  • Remove all required cables from the packaging, and allow cables to acclimate or reach operating temperature, if possible. The acclimation period is usually 24 hours. This improves the ability to manipulate the cables.

  • Label both ends of each cable using a label stock that meets the ANSI/TIA/EIA 606-A standard, if possible.

  • Begin the installation procedure in ascending order.

  • Only slide out one server at a time. Sliding out more than one server can cause cables to drop cause problems when sliding the servers back.

  • Separate the installation by dressing cables with the least stringent bend radius requirements first. The following bend radius requirements are based on EIA/TIA 568-x standards, and may vary from the manufacturer's requirements:

    • CAT5e UTP: 4 x diameter of the cable or 1 inch; 25.4 mm minimum bend radius

    • AC power cables: 4 x diameter of the cable or 1 inch; 25.4 mm minimum bend radius

    • For X8M: InfiniBand Transport Layer systems based on a RoCE Network Layer

      30 AWG: Single cable diameter of 4.5 +/- 0.2 mm and lengths from 1 to 3 meters; 21 mm single bend minimum bend radius or 45 mm repeated bends.

      26 AWG: Single cable diameter of 5.8 +0.3 mm/-1.0 mm and lengths from 2.5 to 5 meters; 29 mm single bend minimum bend radius or 58 mm repeated bends.

    • For X8 and earlier: InfiniBand Transport Layer systems based on an InfiniBand Network Layer

      TwinAx: 5 x diameter of the cable or 1.175 inch; 33 mm minimum bend radius.

    • Quad Small Form-factor Pluggable (QSFP) cable: 6 x diameter of the cable or 2 inch; 55 mm minimum bend radius.

    • Fiber core cable: 10 x diameter of the cable or 1.22 inch; 31.75 mm minimum bend radius for a 0.125 cable.

  • Install the cables with the best longevity rate first.

1.7 Obtaining Current Configuration Information

The current configuration information is used to plan patching requirements, configure new IP addresses, and so on. The following information should be collected as described before extending the rack:

  • The Oracle EXAchk report for the current rack.

  • Image history information using the following command:

    dcli -g ~/all_group -l root "imagehistory" > imagehistory.txt
    
  • Current IP addresses defined for all Exadata Storage Servers and database servers using the following command:

    dcli -g ~/all_group -l root "ifconfig" > ifconfig_all.txt
    
  • Information about the configuration of the cells, cell disks, flash logs, and IORM plans using the following commands:

    dcli -g ~/cell_group -l root "cellcli -e list cell detail" > cell_detail.txt
    
    dcli -g ~/cell_group -l root "cellcli -e list physicaldisk detail" >   \
    physicaldisk_detail.txt
    
    dcli -g ~/cell_group -l root "cellcli -e list griddisk attributes      \
    name,offset,size,status,asmmodestatus,asmdeactivationoutcome" > griddisk.txt
    
    dcli -g ~/cell_group -l root "cellcli -e list flashcache detail" >     \
    fc_detail.txt
    
    dcli -g ~/cell_group -l root "cellcli -e list flashlog detail" > fl_detail.txt
    
    dcli -g ~/cell_group -l root "cellcli -e list iormplan detail" >       \
    iorm_detail.txt
    
  • HugePages memory configuration on the database servers using the following command:

    dcli -g ~/dbs_group -l root "cat /proc/meminfo | grep 'HugePages'" >    \
    hugepages.txt
    
  • For X8M: InfiniBand Transport Layer systems based on a RoCE Network Layer, you should have a list of the switch names from the initial configuration with OEDA. Put the names in a file, for example, roceswitches.txt, with each switch name on a separate line.

  • For X8 and earlier: InfiniBand Transport Layer systems based on an InfiniBand Network Layer, use the following command:

    ibswitches > ibswitches.txt
    

    Use the nm2version on each Sun Datacenter InfiniBand Switch 36 switch to get its firmware version.

  • The following network files from the first database server in the rack:

    • /etc/resolv.conf

    • /etc/ntp.conf

    • /etc/network

    • /etc/sysconfig/network-scripts/ifcfg-*

  • Any users, user identifiers, groups and group identifiers created for cluster-managed services that need to be created on the new servers, such as Oracle GoldenGate.

    • /etc/passwd

    • /etc/group

  • Output of current cluster status using the following command:

    crsctl stat res -t > crs_stat.txt
    
  • Patch information from the Grid Infrastructure and Oracle homes using the following commands. The commands must be run as Grid Infrastructure home owner, and the Oracle home owner.

    /u01/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch lsinventory -oh    \
    GRID_HOME -detail -all_nodes > opatch_grid.txt
    
    /u01/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch lsinventory -oh    \
    ORACLE_HOME -detail -all_nodes >> opatch_oracle.txt
    

    In the preceding commands, GRID_HOME is the path for the Grid Infrastructure home directory, and ORACLE_HOME is the path for the Oracle home directory.

1.8 Preparing the Network Configuration

When adding additional servers to your rack, you will need IP address and the current network configuration settings.

When adding additional servers or rack to an existing rack, the IP addresses for the new servers are obtained using Oracle Exadata Deployment Assistant (OEDA). If adding additional servers to an existing rack, then the application should only include the new servers. If adding an additional rack, then the new rack should use its own OEDA. The exact Oracle Automatic Storage Management (Oracle ASM) disk group configuration currently in use may not be reflected by the application. This is not an issue, as the grid disks and disk groups are configured manually. All other items, such as the Oracle home location and owner, should be defined exactly as the existing configuration.

When adding Oracle Exadata X4-2 Database Server or later or Oracle Exadata Storage Server X4-2L or later, the bonding configuration must match the existing servers in the rack. The OEDA InfiniBand configuration page has an option to select the type of bonding. Select the option for active-active bonding, or deselect the option for active-passive bonding.

The configuration file generated by the application is used by OEDA. After using OEDA, use the checkip.sh and dbm.dat files to verify the network configuration. The only errors that should occur are from the ping command to the SCAN addresses, Cisco switch, and Sun Datacenter InfiniBand Switch 36 switches.

1.9 Moving Audit and Diagnostic Files

The files in the $GRID_HOME/rdbms/audit directory and the $GRID_HOME/log/diagnostics directory should be moved or deleted before extending a cluster. Oracle recommends moving or deleting the files a day or two before the planned extension because it may take time.

1.10 Reviewing Release and Patch Levels

When adding new servers to a rack, you must match the installed operating system version and software releases.

The new rack or servers most-likely have a later release or patch level than the current rack. In some cases, you may want to update the current rack release to the later release. In other cases, you may want to stay at your current release, and choose to reimage the new rack to match the current rack. Whatever you choose to do, ensure that the existing and new servers and Sun Datacenter InfiniBand Switch 36 switches are at the same patch level.

Tip:

Check My Oracle Support note 888828.1 for latest information on minimum releases.

Older servers in a rack may need to be patched to a later release to meet the minimum required software release. In addition, older database servers might use Oracle Linux release 5.3. Those servers need to be updated to a newer Oracle Linux release.

Additional patching considerations include the Oracle Grid Infrastructure and Oracle Database home releases and bundle patch updates. If new patches will be applied, then Oracle recommends changing the existing servers so that the new servers will inherit the releases as part of the extension procedure. This way, the number of servers being patched is lower. Any patching of the existing servers should be performed in advance so they are at the desired level when the extension work is scheduled, thereby reducing the total amount of work required during the extension.

1.11 Performing Preliminary Checks

Perform a visual check of Oracle Exadata Database Machine physical systems before extending the hardware.

  1. Check the rack for damage.

  2. Check the rack for loose or missing screws.

  3. Check Oracle Exadata Database Machine for the ordered configuration.

  4. Check that all cable connections are secure and well seated.

  5. Check power cables.

  6. Ensure the correct connectors have been supplied for the data center facility power source.

  7. Check network data cables.

  8. Check the site location tile arrangement for cable access and airflow.

  9. Check the data center airflow into the front of Oracle Exadata Database Machine.

1.12 Preparing to Add Servers

Perform the following tasks before adding the servers:

  1. Unpack the Oracle Exadata Database Machine expansion kit.

  2. Unpack all Oracle Exadata Database Machine server components from the packing cartons. The following items should be packaged with the servers:

    • Oracle Database servers or Exadata Storage Server

    • Power cord, packaged with country kit

    • Cable management arm with installation instructions

    • Rackmount kit containing rack rails and installation instructions

    • (Optional) Sun server documentation and media kit

    Note:

    If you are extending Oracle Exadata Database Machine X4-2, Oracle Exadata Database Machine X3-8 Full Rack, or Oracle Exadata Database Machine X2-2 (with X4170 and X4275 servers) half rack, then order the expansion kit that includes a Sun Datacenter InfiniBand Switch 36 switch.

    Figure 1-1 shows the components in the server expansion kit.

    Figure 1-1 Server Components for Upgrade

    Description of Figure 1-1 follows
    Description of "Figure 1-1 Server Components for Upgrade"
  3. Lay out the cables for the servers.

  4. Unroll the cables and stretch them to remove the bends.

  5. Apply the cable labels. Oracle recommends labeling all cables before installation.

  6. Install the servers.

  7. Cable the servers.

See Also: