Performing Initial Elastic Configuration of Oracle Exadata

6.13 Performing Initial Elastic Configuration of Oracle Exadata

This topic provides background information on elastic configuration and describes how to perform the initial configuration of Oracle Exadata.

You need the configuration file generated by Oracle Exadata Deployment Assistant (OEDA).

Elastic configuration is new in Oracle Exadata System Software release 12.1.2.1.0. It applies to all rack configurations, such as the following:

Racks ordered with a standard number of database nodes and cells, for example, quarter or half racks
Rack configurations that feature additional database nodes and cells
Oracle Exadata Two-Socket X5 generation and later storage servers
Oracle Exadata Racks that use an InfiniBand Network Layer or a RoCE Network Layer.

Starting with the Oracle Exadata Two-Socket X5 generation of storage servers, the same process can also be used to add additional database servers or storage servers to an existing configuration.

The process allows initial IP addresses to be assigned to database servers and storage servers, regardless of the exact configuration you ordered. Your specific configuration can then be applied to the nodes. The procedure can be run from any database node (usually the first node,) and also from a storage server in cases where expansion involves adding only storage servers.

Every Oracle Exadata has a predefined method for the cabling of nodes to the RDMA Network Fabric switch ports. Therefore, there is a fixed mapping from each node's location in the rack to the ports of the RDMA Network Fabric switches. Assuming the rack is always populated following this map, a node's rack unit location can be identified by querying the RDMA Network Fabric to determine the RDMA Network Fabric switch port the node is connected to. Once found, that information is used to determine the rack unit location of the node. With this information, nodes can be allocated initial IP addresses based on their rack unit location, with nodes lower in the rack getting lower IP addresses.

Note:

Disk space for unused system partitions must be reclaimed before installing software on all database servers with InfiniBand Network Fabric. Use the following command to reclaim the disk space: /opt/oracle.SupportTools/reclaimdisks.sh -free -reclaim. This command is not required on RoCE-based Exadata database servers.
If a RAID rebuild/reconstruction (that is, a disk replacement or expansion) is in progress, wait until it is finished before performing the steps in this section. The reason is that the reclaimdisks.sh script is blocked from running until the RAID rebuild/reconstruction is complete.

Open a console session to the first database server. The first database server is the lowest database server in the rack, which is rack position U16. In the case of an expansion involving only cells, the procedure should be run from a storage server instead of a database node.

Note:
The network connection and configuration for Oracle Exadata has not occurred yet.
Log in as the root user on the first database server.

Note:
If you do not have the password for the root user, then contact Oracle Support Services.

On systems with InfiniBand Network Fabric only, verify all servers are configured with the 172.16.* IP address for the eth0 interface using the following command:

# ibhosts


Ca  : 0x0010e00001486fb8 ports 2 "node10 elasticNode 172.16.2.46,172.16.2.46 ETH0"
Ca  : 0x0010e00001491228 ports 2 "node9 elasticNode 172.16.2.45,172.16.2.45 ETH0"
Ca  : 0x0010e000014844f8 ports 2 "node8 elasticNode 172.16.2.44,172.16.2.44 ETH0"
Ca  : 0x0010e00001488218 ports 2 "node4 elasticNode 172.16.2.40,172.16.2.40 ETH0"
Ca  : 0x0010e000014908b8 ports 2 "node2 elasticNode 172.16.2.38,172.16.2.38 ETH0"
Ca  : 0x0010e0000148ca68 ports 2 "node1 elasticNode 172.16.2.37,172.16.2.37 ETH0"
Ca  : 0x0010e00001485fd8 ports 2 "node3 elasticNode 172.16.2.39,172.16.2.39 ETH0"

Each server description should be hostname elasticNode. ipaddress ETH0.

If using virtualization on the rack, configure each database server as follows:

Note:

Starting with Oracle Exadata Database Machine X8M-2, Oracle Linux KVM is the virtualization technology for systems that use RoCE Network Fabric. Otherwise, Oracle VM Server (OVM) is a Xen-based virtualization technology, which is used across Oracle Exadata Database Machine systems that use InfiniBand Network Fabric.

This step applies to both virtualization technologies.
```
# /opt/oracle.SupportTools/switch_to_ovm.sh
```
The database servers will reboot during this step.

Note:
It is not possible or supported to switch between physical and virtualized configurations.
On systems with InfiniBand Network Fabric only, reclaim the unused disk space on the database servers.

Disk space for unused system partitions must be reclaimed before installing software on all database servers with InfiniBand Network Fabric. Use the following command to reclaim the disk space: /opt/oracle.SupportTools/reclaimdisks.sh -free -reclaim. This command is not required on RoCE-based Exadata database servers.

reclaimdisks.sh performs partition reconfiguration and a few other necessary configuration changes that take typically less than a minute to complete.
On systems with InfiniBand Network Fabric only, verify the disk configuration on the database servers.
```
# /opt/oracle.SupportTools/reclaimdisks.sh -check
```
If the system is a physical (bare metal) deployment, the last line of output will be the following:
```
Valid. Booted: Linux. Layout: Linux.
```
If the system is an Oracle VM deployment, the last line of output will be the following:
```
Valid. Booted: DOM0. Layout: DOM0.
```
Copy the OEDA configuration files to the database server using one of the following methods. If adding only storage servers, the steps can be run from a storage server.
- Using a USB flash drive:
  1. Copy the configuration file to a USB flash drive.
  2. Connect the USB drive to the database server or storage server.
  3. Locate the USB drive using the following command:
```
for x in `ls -1 /sys/block`; do udevadm info --attribute-walk --path=/sys/block/$x | grep \
-iq 'DRIVERS=="usb-storage"'; if [ $? -eq 0 ] ; then echo /dev/${x}1; \
fi ; done
```
    The expected output is similar to:
```
/dev/sdb1
```
    Note:
    
    If an Oracle Exadata Storage Server X6 or X5 Extreme Flash (EF) is used, the command will also return the two internal USB drives, usually /dev/sda1 and /dev/sdb1. The newly inserted USB drive will be the additional device, usually /dev/sdc1.
    
    The expected output is similar to:
```
/dev/sda1
/dev/sdb1
/dev/sdc1
```
    The internal USBs can be identified by viewing mdstat:
```
# grep md4 /proc/mdstat | awk '{print $5,$6;}'
```
  4. Create a directory on the database server or storage server using the following command:
```
# mkdir /mnt/usb
```
  5. Create a directory for the installation using the following command:
```
# mkdir /opt/oracle.SupportTools/onecommand
```
    If the directory already exists, then rename the existing one to onecommand.old, and then re-create the empty directory.
    
    Note:
    Do not use a directory under /u01 or /EXAVMIMAGES because reclaimdidsks.sh will fail if there is content in those directories.
  6. Mount the device. Use the device name given in step 7.c. The following is an example of the command.
```
# mount -t vfat /dev/sdb1 /mnt/usb
```
  7. Copy OEDA and the configuration files from the USB flash drive to the onecommand directory on the database server or storage server using the following command:
```
# cp /mnt/usb/path_to_file/*.* /opt/oracle.SupportTools/onecommand
```
    In the preceding command, path_to_file is the directory path to the configuration file on the USB flash drive.
  8. Unmount the USB flash drive using the following command:
```
# umount /mnt/usb
```
  9. Remove the USB flash drive from the database server or storage server.
- Using a temporary network connection:
  
  This option is only needed if there were IP address conflicts, and Oracle Exadata is not already connected to the network.
  1. Connect a network cable to the Ethernet port labeled NET0 on the back of the first database server.
  2. Bring up the eth0:1 interface as follows:
```
# ifconfig eth0:1 real_ip netmask real_netmask up
```
    In the preceding command, real_ip is the IP address to assign to the database server, and real_netmask is the subnet mask to assign to the database server.
  3. Add the default route as follows:
```
# route add -net 0.0.0.0 netmask 0.0.0.0 gw real_gateway
```
    In the preceding command, real_gateway is the default gateway to use for the database server.
  4. Copy the configuration files to the /opt/oracle.SupportTools/onecommand directory on the database server or storage server using a file transfer utility such as scp.
  5. Disconnect the network cable from Ethernet port NET0.
    
    Note:
    It is not necessary to reconfigure the network because the network is reconfigured automatically during the restart step.
Download and unzip the OEDA archive to a host on the same network as the servers in the rack (172.16.*).

The applyElasticConfig.sh script is not configured by default on the servers in the rack.
Run the applyElasticConfig.sh script.

For example, if you unzipped the OEDA ZIP file into the /opt/oracle.SupportTools/onecommand/linux-x64 directory, then use the following commands:
```
# cd /opt/oracle.SupportTools/onecommand/linux-x64

# ./applyElasticConfig.sh -cf customer_name-configFile.xml
```
The applyElasticConfig.sh script performs network configuration for all database servers and storage servers. All servers restart at the end of the process.
Connect the corporate network cable for the management network to the Management Network Switch.
Connect the corporate network cables for the client access network to the database servers.
Restart all database servers and storage servers.
Log in to the first database server (or storage server, if only storage servers were added for expansion) to verify network connectivity using the following commands:
```
# cd /opt/oracle.SupportTools/onecommand/linux-x64

#./checkip.sh -cf configuration_file
```
In the preceding command, configuration_file is the name of the configuration file from OEDA, and includes the full path.
Note:
- The checkip.sh script can be run on a server outside of Oracle Exadata to verify network connectivity to Oracle Exadata. Copy the checkip.sh file to the external system, and run the command as shown in this step.
- If the command is run from a Microsoft Windows machine, then the command is checkip.cmd.
- Download the version of OEDA for the relevant platform.
Use OEDA to install and configure the software for Oracle Exadata.