Preparing to Extend Oracle Exadata Database Machine

1.1 About Extending Oracle Exadata

You can extend Oracle Exadata either by adding servers to the current configuration or by cabling together multiple racks.

Here are considerations when extending Oracle Exadata:

You can extend Oracle Exadata from a fixed or custom configuration to another configuration by adding any combination of database or storage servers up to the allowed maximum.
You can cable together multiple Oracle Exadata racks subject to the following:
- You can cable together different rack models. For example, you can cable together an X8-2 rack and an X7-2 rack.
- All racks that are cabled together in a multi-rack configuration must use the same RDMA Network Fabric. That is, all racks must use RoCE Network Fabric, or all racks must use InfiniBand Network Fabric.
  
  You cannot have a mixture of racks using RoCE Network Fabric and InfiniBand Network Fabric. For example, you cannot cable together an X8-2 rack and an X9M-2 rack.
- All racks that are cabled together in a multi-rack configuration have the same database server hardware architecture. That is, all racks must use 2-socket database servers, or all racks must use 8-socket database servers.
  
  You cannot have a mixture of racks using 2-socket and 8-socket database servers. For example, you cannot cable together an X9M-2 rack and an X9M-8 rack.
Prior to extending a system across multiple racks, you must acquire the appropriate RDMA Network Fabric switches and transceivers.
When extending Oracle Exadata Eighth Rack with Oracle Exadata Storage Expansion Rack, Oracle recommends using separate disk groups for the disks in each rack.

Multiple Oracle Exadata racks can run as separate environments while sharing the RDMA Network Fabric. If you are planning to utilize multiple Oracle Exadata racks in this manner, then note the following:

All servers on the RDMA Network Fabric must have a unique IP address. When Oracle Exadata is deployed, the default network is 192.168.10.1. You must modify the IP addresses before re-configuring the RDMA Network Fabric. Failure to do so causes duplicate IP addresses.

After modifying the network, run the appropriate verification tools:

For X8M and later, with RoCE Network Fabric:

Run the infinicheck command to verify the network. You should supply a file that contains a list of all the database server host names or RoCE Network Fabric IP addresses, and another file that lists all of the RoCE Network Fabric IP addresses for the storage servers. For example:

# /opt/oracle.SupportTools/ibdiagtools/infinicheck -g hosts -c cells

                        INFINICHECK
                [Network Connectivity, Configuration and Performance]

                    ####  FABRIC TYPE TESTS  ####

System type identified: RoCE

Verifying User Equivalance of user=root from all DBs to all CELLs.

                ####  RoCE CONFIGURATION TESTS  ####
        Checking for presence of RoCE devices on all DBs and CELLs

[SUCCESS].... RoCE devices on all DBs and CELLs look good

        Checking for RoCE Policy Routing settings on all DBs and CELLs

[SUCCESS].... RoCE Policy Routing settings look good

        Checking for RoCE DSCP ToS mapping on all DBs and CELLs

[SUCCESS].... RoCE DSCP ToS settings look good

        Checking for RoCE PFC settings and DSCP mapping on all DBs and CELLs

[SUCCESS].... RoCE PFC and DSCP settings look good

        Checking for RoCE interface MTU settings. Expected value : 2300

[SUCCESS].... RoCE interface MTU settings look good

        Verifying switch advertised DSCP on all DBs and CELLs ports ( ~ 2 min )

[SUCCESS].... Advertised DSCP settings from RoCE switch looks good


                    ####  CONNECTIVITY TESTS  ####
                    [COMPUTE NODES -> STORAGE CELLS]
                           (60 seconds approx.)
                   (Will walk through QoS values: 0-6)
[SUCCESS]..............Results OK

[SUCCESS]....... All  can talk to all storage cells

                    [COMPUTE NODES -> COMPUTE NODES]
                           (60 seconds approx.)
                   (Will walk through QoS values: 0-6)
[SUCCESS]..............Results OK

[SUCCESS]....... All hosts can talk to all other nodes

        Verifying Subnet Masks on all nodes
[SUCCESS] ......... Subnet Masks is same across the network

If user equivalence for password-less SSH is not configured, then you must first run infinicheck with the -s option. For example:

# /opt/oracle.SupportTools/ibdiagtools/infinicheck -g hosts -c cells -s

For X8 and earlier, with InfiniBand Network Fabric:

Run the verify-topology (or InfiniBand commands like showtopology and ibdiagnet) and infinicheck commands to verify the network is working properly. For example:
```
# cd /opt/oracle.SupportTools/ibdiagtools
# ./verify-toplogy -t fattree
# ./infinicheck -g hosts -c cells
```

When Oracle Exadata racks run in separate clusters, do not modify the cellip.ora files. The cellip.ora file on a database server should only include the IP addresses for the storage servers used with that database server.
Cells with disk types different from what is already installed can be added, but the disk types cannot be mixed in the same Oracle Automatic Storage Management (Oracle ASM) disk group. For example, if the existing disk groups all use high performance disks, and cells with high capacity disks are being added, then it is necessary to create new disk groups for the high capacity disks.

When adding the same type of disk, ensure that the grid disk sizes are exactly the same even if the new disks are larger than the existing ones. For example, if the existing disks are 3 TB, and the additional disks are 4 TB, then it is necessary to create grid disks that match the size on the 3 TB disks. A new disk group can be created using the extra 1 TB of disk space.
In order to access Exadata Storage Servers in one Oracle Exadata rack by another Oracle Exadata rack when they are not running as a single cluster, Exadata Storage Servers must have unique Oracle ASM disk group and failure group names on each Oracle Exadata.

For example, for two Oracle Exadata racks cabled together but run as separate clusters, the following names should be unique:
- Cell name
- Cell disk name
- Grid disk name
- Oracle ASM failure group name
All equipment receives a Customer Support Identifier (CSI). Any new equipment for the Oracle Exadata has a new CSI. Contact Oracle Support Services to reconcile the new CSI with the existing Oracle Exadata CSI. Have the original instance numbers or serial numbers available, as well as the new numbers when contacting Oracle Support Services.
For X8M and later, with RoCE Network Fabric:

You can use the RDMA Network Fabric for limited external connectivity. The external connectivity ports in the RoCE Network Fabric switches can connect to Oracle ZFS Storage Appliance or Oracle Zero Data Loss Recovery Appliance to provide a backup solution.

For details about the recommended connectivity options, see the following solution briefs:
- Exadata X8M and ZFS Storage 25Gb Backup Solution - Configuration of Oracle Exadata X8M with Oracle ZFS Storage Appliance ZS7-2 using dedicated backup switches
- 100Gb Backup Solution - Oracle Exadata Database Machine X8M with Oracle ZFS Storage Appliance ZS7-2
For X8 and earlier, with InfiniBand Network Fabric:

The RDMA Network Fabric can be used for external connectivity. The external connectivity ports in the Sun Datacenter InfiniBand Switch 36 switches can connect to media servers for tape backup, data loading, and client and application access. Use the available ports on the leaf switches for external connectivity. There are 12 ports per rack. The available ports are 5B, 6A, 6B, 7A, 7B, and 12A in each leaf switch. For high availability connections, connect one port to one leaf switch and the other port to the second leaf switch. The validated InfiniBand cable lengths are:
- Up to 5 meters for passive copper 4X QDR QSFP cables
- Up to 100 meters for fiber optic 4X QDR QSFP cables

Related Topics

Elastic Configurations

Parent topic: Preparing to Extend Oracle Exadata Database Machine

1.2 Reviewing the Safety Precautions

Before upgrading Oracle Exadata Database Machines, read Important Safety Information for Sun Hardware Systems included with the rack.

Note:

Contact a service representative or Oracle Advanced Customer Support to confirm that Oracle has qualified your equipment for installation and use in Oracle Exadata Database Machine. Oracle is not liable for any issues when you install or use non-qualified equipment.

1.3 Reviewing the Cable Precautions

Review the following RDMA Network Fabric cable precautions before working with the cables:

Fiber optic cables with laser transceivers must be of type Class 1.
Do not allow any copper core cable to bend to a radius tighter than 127 mm (5 inches). Tight bends can damage the cables internally.
Do not allow any optical cable to bend to a radius tighter than 85 mm (3.4 inches). Tight bends can damage the cables internally.
Do not use zip ties to bundle or support cables. The sharp edges of the ties can damage the cables internally. Use hook-and-loop straps.
Do not allow any cable to experience extreme tension. Do not pull or drag the cables, which can damage them internally.
Unroll a cable for its length.
Do not twist cables more than one revolution for their entire length. Twisting a cable might damage it internally.
Do not route cables where they can be stepped on, or experience rolling loads. A crushing effect can damage the cable internally.

Parent topic: Preparing to Extend Oracle Exadata Database Machine

1.4 Estimating Cable Path Lengths

Cable paths should be as short as possible. When the length of a cable path has been calculated, select the shortest cable to satisfy the length requirement. When specifying a cable, consider the following:

Bends in the cable path increase the required length of the cable. Rarely does a cable travel in a straight line from connector to connector. Bends in the cable path are necessary, and each bend increases the total length.
Bundling increases the required length of the cables. Bundling causes one or more cables to follow a common path. However, the bend radius is different in different parts of the bundle. If the bundle is large and unorganized, and there are many bends, one cable might experience only the inner radius of bends, while another cable might experience the outer radius of bends. In this situation, the differences of the required lengths of the cables is quite substantial.
When calculating the cable path length and route is under the floor, take into consideration the height of the raised floor.

Parent topic: Preparing to Extend Oracle Exadata Database Machine

1.5 Bundling Cables

When bundling RDMA Network Fabric cables in groups, use hook-and-loop straps to keep cables organized. If possible, use color-coordinated straps to help identify cables and their routing. Splitter and C-4 copper conductor cables are fairly thick and heavy for their length. Consider the retention strength of the hook-and-loop straps when supporting cables. Bundle as few cables as reasonably possible. If the cables break free of their straps and fall free, the cables might break internally when they strike the floor or from sudden changes in tension.

Bundle the cables using many hook-and-loop straps. Oracle recommends that no more than eight cables be bundled together.

Place the hook-and-loop straps as close together as reasonably possible, for example, one strap every foot (0.3 m). If a cable breaks free from a strap, then the cable cannot fall far before it is retained by another strap.

Floor and Underfloor Delivery of Cables

Parent topic: Preparing to Extend Oracle Exadata Database Machine

1.5.1 Floor and Underfloor Delivery of Cables

The RDMA Network Fabric switches accept cables from floor or underfloor delivery. Floor and underfloor delivery limits the tension in the cable to the weight of the cable for the rack height of the switch.

Note:

Overhead cabling details are not included in this guide. For details on overhead cabling, contact a certified service engineer.

Parent topic: Bundling Cables

1.6 Reviewing the Cable Management Arm Guidelines

Review the following cable management arm (CMA) guidelines before routing the cables.

Remove all required cables from the packaging, and allow cables to acclimate or reach operating temperature, if possible. The acclimation period is usually 24 hours. This improves the ability to manipulate the cables.
Label both ends of each cable using a label stock that meets the ANSI/TIA/EIA 606-A standard, if possible.
Begin the installation procedure in ascending order.
Only slide out one server at a time. Sliding out more than one server can cause cables to drop cause problems when sliding the servers back.
Separate the installation by dressing cables with the least stringent bend radius requirements first. The following bend radius requirements are based on EIA/TIA 568-x standards, and may vary from the manufacturer's requirements:
- CAT5e UTP: 4 x diameter of the cable or 1 inch; 25.4 mm minimum bend radius
- AC power cables: 4 x diameter of the cable or 1 inch; 25.4 mm minimum bend radius
- For X8M and later, with RoCE Network Fabric:
  - 30 AWG: Single cable diameter of 4.5 +/- 0.2 mm and lengths from 1 to 3 meters; 21 mm single bend minimum bend radius or 45 mm repeated bends.
  - 26 AWG: Single cable diameter of 5.8 +0.3 mm/-1.0 mm and lengths from 2.5 to 5 meters; 29 mm single bend minimum bend radius or 58 mm repeated bends.
- For X8 and earlier, with InfiniBand Network Fabric:
  - TwinAx: 5 x diameter of the cable or 1.175 inch; 33 mm minimum bend radius.
  - Quad Small Form-factor Pluggable (QSFP) cable: 6 x diameter of the cable or 2 inch; 55 mm minimum bend radius.
  - Fiber core cable: 10 x diameter of the cable or 1.22 inch; 31.75 mm minimum bend radius for a 0.125 cable.
Install the cables with the best longevity rate first.

Parent topic: Preparing to Extend Oracle Exadata Database Machine

1.7 Obtaining Current Configuration Information

The current configuration information is used to plan patching requirements, configure new IP addresses, and so on. The following information should be collected as described before extending the rack:

The Oracle EXAchk report for the current rack.

Image history information using the following command:

dcli -g ~/all_group -l root "imagehistory" > imagehistory.txt

Current IP addresses defined for all Exadata Storage Servers and database servers using the following command:
```
dcli -g ~/all_group -l root "ifconfig" > ifconfig_all.txt
```

Information about the configuration of the cells, cell disks, flash logs, and IORM plans using the following commands:

dcli -g ~/cell_group -l root "cellcli -e list cell detail" > cell_detail.txt

dcli -g ~/cell_group -l root "cellcli -e list physicaldisk detail" >   \
physicaldisk_detail.txt

dcli -g ~/cell_group -l root "cellcli -e list griddisk attributes      \
name,offset,size,status,asmmodestatus,asmdeactivationoutcome" > griddisk.txt

dcli -g ~/cell_group -l root "cellcli -e list flashcache detail" >     \
fc_detail.txt

dcli -g ~/cell_group -l root "cellcli -e list flashlog detail" > fl_detail.txt

dcli -g ~/cell_group -l root "cellcli -e list iormplan detail" >       \
iorm_detail.txt

HugePages memory configuration on the database servers using the following command:

dcli -g ~/dbs_group -l root "cat /proc/meminfo | grep 'HugePages'" >    \
hugepages.txt

For X8M and later, with RoCE Network Fabric:

You should have a list of the switch names from the initial configuration with OEDA. Put the names in a file, for example, roceswitches.txt, with each switch name on a separate line.
For X8 and earlier, with InfiniBand Network Fabric:

Use the following command:
```
ibswitches > ibswitches.txt
```
Use the nm2version on each Sun Datacenter InfiniBand Switch 36 switch to get its firmware version.
The following network files from the first database server in the rack:
- /etc/resolv.conf
- /etc/ntp.conf
- /etc/network
- /etc/sysconfig/network-scripts/ifcfg-*
Any users, user identifiers, groups and group identifiers created for cluster-managed services that need to be created on the new servers, such as Oracle GoldenGate.
- /etc/passwd
- /etc/group
Output of current cluster status using the following command:
```
crsctl stat res -t > crs_stat.txt
```
Patch information from the Grid Infrastructure and Oracle homes using the following commands. The commands must be run as Grid Infrastructure home owner, and the Oracle home owner.
```
/u01/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch lsinventory -oh    \
GRID_HOME -detail -all_nodes > opatch_grid.txt

/u01/app/oracle/product/11.2.0/dbhome_1/OPatch/opatch lsinventory -oh    \
ORACLE_HOME -detail -all_nodes >> opatch_oracle.txt
```
In the preceding commands, GRID_HOME is the path for the Grid Infrastructure home directory, and ORACLE_HOME is the path for the Oracle home directory.

1.8 Preparing the Network Configuration

When adding additional servers to your rack, you will need IP address and the current network configuration settings.

When adding additional servers or rack to an existing rack, the IP addresses for the new servers are obtained using Oracle Exadata Deployment Assistant (OEDA). If adding additional servers to an existing rack, then the application should only include the new servers. If adding an additional rack, then the new rack should use its own OEDA. The exact Oracle Automatic Storage Management (Oracle ASM) disk group configuration currently in use may not be reflected by the application. This is not an issue, as the grid disks and disk groups are configured manually. All other items, such as the Oracle home location and owner, should be defined exactly as the existing configuration.

When adding Oracle Exadata X4-2 Database Server or later or Oracle Exadata Storage Server X4-2L or later, the bonding configuration must match the existing servers in the rack. The OEDA InfiniBand configuration page has an option to select the type of bonding. Select the option for active-active bonding, or deselect the option for active-passive bonding.

If you want to configure network isolation for groups of virtual database servers and storage servers so that network traffic of one Oracle RAC cluster is not accessible to another Oracle RAC cluster, then you can use OEDA to configuration InfiniBand Partitioning.

The configuration file generated by the application is used by OEDA. After using OEDA, use the checkip.sh and dbm.dat files to verify the network configuration. The only errors that should occur are from the ping command to the SCAN addresses, Cisco switch, and Sun Datacenter InfiniBand Switch 36 switches.

1.9 Moving Audit and Diagnostic Files

The files in the $GRID_HOME/rdbms/audit directory and the $GRID_HOME/log/diagnostics directory should be moved or deleted before extending a cluster. Oracle recommends moving or deleting the files a day or two before the planned extension because it may take time.

Parent topic: Preparing to Extend Oracle Exadata Database Machine

1.10 Reviewing Release and Patch Levels

When adding new servers to a rack, you must match the installed operating system version and software releases.

The new rack or servers most-likely have a later release or patch level than the current rack. In some cases, you may want to update the current rack release to the later release. In other cases, you may want to stay at your current release, and choose to re-image the new rack to match the current rack. Whatever you choose to do, ensure that the existing and new servers and switches are at the same patch level.

Tip:

Check My Oracle Support note 888828.1 for latest information on minimum releases.

Older servers in a rack may need to be patched to a later release to meet the minimum required software release. In addition, older database servers might use Oracle Linux release 5.3. Those servers need to be updated to a newer Oracle Linux release.

Additional patching considerations include the Oracle Grid Infrastructure and Oracle Database home releases and updates. If new patches will be applied, then Oracle recommends changing the existing servers so that the new servers will inherit the releases as part of the extension procedure. This way, the number of servers being patched is lower. Any patching of the existing servers should be performed in advance so they are at the desired level when the extension work is scheduled, thereby reducing the total amount of work required during the extension.

1.11 Performing Preliminary Checks

Perform a visual check of Oracle Exadata Database Machine physical systems before extending the hardware.

Check the rack for damage.
Check the rack for loose or missing screws.
Check Oracle Exadata Database Machine for the ordered configuration.
Check that all cable connections are secure and well seated.
Check power cables.
Ensure the correct connectors have been supplied for the data center facility power source.
Check network data cables.
Check the site location tile arrangement for cable access and airflow.
Check the data center airflow into the front of Oracle Exadata Database Machine.

Parent topic: Preparing to Extend Oracle Exadata Database Machine

1.12 Preparing to Add Servers

Perform the following tasks before adding the servers:

Unpack the Oracle Exadata Database Machine expansion kit.
Unpack all Oracle Exadata Database Machine server components from the packing cartons. The following items should be packaged with the servers:
- Oracle Database servers or Exadata Storage Server
- Power cord, packaged with country kit
- Cable management arm with installation instructions
- Rackmount kit containing rack rails and installation instructions
- (Optional) Sun server documentation and media kit
Note:

If you are extending Oracle Exadata Database Machine X4-2, Oracle Exadata Database Machine X3-8 Full Rack, or Oracle Exadata Database Machine X2-2 (with X4170 and X4275 servers) half rack, then order the expansion kit that includes a Sun Datacenter InfiniBand Switch 36 switch.

Figure 1-1 shows the components in the server expansion kit.

Figure 1-1 Server Components for Upgrade

Description of "Figure 1-1 Server Components for Upgrade"
Lay out the cables for the servers.
Unroll the cables and stretch them to remove the bends.
Apply the cable labels. Oracle recommends labeling all cables before installation.
Install the servers.
Cable the servers.