You can install and configure one or more Big Data Appliance High Capacity Nodes plus InfiniBand Infrastructure to add up to 12 new servers on a starter rack. This chapter explains the configuration steps for adding new servers on a starter rack.
Oracle Big Data Appliance supports clusters with mixed X5–2L, X6–2L, and X7–2L servers. However if you include X7–2L servers in the cluster, then you must first upgrade the cluster to Oracle Big Data Appliance release v4.10.0 or higher.
This chapter consists of the following sections:
Installing New Servers to an Oracle Big Data Appliance Starter Rack
Installing Oracle Big Data Appliance Software on the New Servers
Note:
Oracle recommends that an Oracle field engineer performs the procedures described in this chapter.
Also note that physically adding servers to a partially-filled rack requires downtime. Extending a cluster onto new nodes in the same rack should not require downtime.
An Oracle Big Data Appliance starter rack has the same hardware configuration as a full rack, except that it comes with 6 servers instead of 18. All switches and power supplies are included in the starter rack, and do not need to be upgraded or supplemented to support additional servers.
You can add up to 12 new servers on a starter rack by using a High Capacity node plus InfiniBand Infrastructure. You can either add all the new servers together or add some of the new servers at a later time on a starter rack.
All Oracle Big Data Appliance expansion nodes come with the in-rack cables required for power as well as connections to the administrative and InfiniBand networks. Adding servers to a rack requires no changes to multi-rack InfiniBand connections, since that is a separate cabling task from switch to switch.
Before starting the installation, ensure that you received all components, cables, and accessories for the High Capacity node plus InfiniBand Infrastructure as shown in the following tables.
Table 8-1 lists the components and accessories.
Table 8-1 A High Capacity Node plus Infiniband Infrastructure Components and Accessories
| Quantity | Description |
|---|---|
|
1 |
Oracle Big Data Appliance High Capacity Node plus InfiniBand Infrastructure |
|
2 |
2U snap-in slide rail kit |
|
1 |
1U-2U slim-rail CMA kit |
|
1 |
Velcro 25-foot roll |
|
1 |
Cable labeling kit |
|
1 |
Server documentation and license packages |
Table 8-2 lists the cables.
Table 8-2 A High Capacity Node plus Infiniband Infrastructure Cables
| Length | Quantity | Description |
|---|---|---|
|
10 feet |
1 |
Red Ethernet Cat 5 or Cat 5e |
|
10 feet |
1 |
Ethernet Black Cat 5 or Cat 5e |
|
3 meters |
2 |
InfiniBand QSFP |
|
2 meters |
2 |
Power |
Ensure that the starter rack is in a good working condition before you install the High Capacity node plus InfiniBand Infrastructure:
Check the LEDs on all components to verify a normal operating state. Resolve any issues before proceeding.
Verify that the Sun Rack II PDU firmware is at or above the minimum version of 1.06. See "Updating the PDU Firmware Version."
You can also add one or more Oracle Big Data Appliance servers to an Oracle Big Data Appliance X4-2 starter rack by using a High Capacity node plus InfiniBand Infrastructure.
To prepare the starter rack for expanding the hardware:
Upgrade the starter rack to Oracle Big Data Appliance 4.1 or later.
Note:
Starter rack hardware for the Oracle Big Data Appliance versions earlier to 4.1 cannot be expanded using a High Capacity node plus InfiniBand Infrastructure.
The following instructions describe how to physically install new servers on a starter rack.
Caution:
Two people are needed to install new servers on a starter rack.
To install new servers on a starter rack:
Unpack all power cables, Ethernet cables, and InfiniBand cables, and mark them with the labels provided in the kit:
Mark each end of a cable with the appropriate label. Take care to put the "from" and "to" labels on the correct end.
Select the appropriate PDU power cord labels for your installation, either single-phase or three-phase.
Plug in the cables at both ends. Refer to the cabling tables in the appendixes.
Install the server rack mount kits in the rack. Start at the first empty unit at the bottom and work up toward the top of the rack.
Install the servers in the rack rails. After installing each server, ensure that the server and the unit below it can slide in and out normally.
Install the cable arms and cables into each server, starting with the first new server at the bottom and working up to the top unit. Ensure that all InfiniBand cables are plugged in to the correct ports on both ends. Keep the power cables out of the way while you route the InfiniBand and Ethernet cables.
Caution:
Do not plug the power cables into the PDU strips until you are ready to turn on the PDU and configure it. Plug the power cables only into the servers now.
After installing the new servers in the rack, you can configure them as described in the following steps.
To configure the new servers:
At the <password> placeholder in these instructions, enter the password provided by the Oracle Install Coordinator.
Disconnect the Cisco switch uplink cable (port 48) from the network, in case the factory IP addresses duplicate those on the local network.
Plug the power cords into the appropriate outlets. Route the cables to PDU-A last. When you are done, Oracle ILOM starts, the service processor LED is solid green, and the system is in Standby mode with a flashing OK LED.
Press the power button on the servers to turn them on. The system automatically detects the factory IP addresses. See "Factory Network Settings."
Using a laptop connected to the first new server, log in as root using SSH. The default password is provided by the Oracle Install Coordinator.
Set up passwordless SSH for the root user. For range, for example, enter [7:10] when you install four new servers.
# setup-root-ssh -j "eth0_ips[range]" -p <password>
See "setup-root-ssh."
Confirm that passwordless SSH works properly across the range of new servers. Use the same value for range as you did in step 5.
# dcli -j "eth0_ips[range]" "hostname; date"
Compare the output of the following command with the system serial number (S/N) sticker on the front of each system.
# dcli -j "eth0_ips[range]" "dmidecode -s chassis-serial-number"
192.168.1.7: 1143FMM064
.
.
.
192.168.1.10: 1144FMM1A0
If you find inconsistencies, then verify that the InfiniBand cables are plugged into the correct ports on the gateway switches for each node.
Set the rack master serial number in the ILOM to the S/N on the front left rack sticker (also displayed on the rear of the rack near PDU-A):
# dcli -j "eth0_ips[range]" "ipmitool sunoem cli 'set /SP \system_identifier="\"Oracle Big Data Appliance rack-master-serial-number\""'"
This example sets the range to [7:10] for the four new servers, and sets the serial number to AK00086253:
# dcli -j "eth0_ips[7:10]" "ipmitool sunoem cli 'set /SP \system_identifier="\"Oracle Big Data Appliance AK00086253\""'"
The system identifier has a 40-character limit.
Verify that the rack master serial number is set correctly. This example uses the range for the four new servers:
# dcli -j "eth0_ips[7:10]" "ipmitool sunoem cli 'show /SP system_identifier'" | grep =
192.168.1.7: system_identifier = Oracle Big Data Appliance AK00086253
.
.
.
192.168.1.10: system_identifier = Oracle Big Data Appliance AK00086253
Complete the physical hardware checks beginning with Step 10 in "Configuring the Oracle Big Data Appliance Servers." In all dcli commands, be sure to enter the range, using the eth0_ips[range] syntax shown previously.
The following topics explain how to add the new servers to the Ethernet administrative network:
See Also:
These instructions use a serial connection to Oracle ILOM. If you use an SSH connection instead, then you lose the console connection when the IP addresses change from the factory settings to the custom settings in "Configuring the Network."
Follow these steps to copy the configuration files to the first new server in the rack.
To copy the configuration files:
At the <password> placeholder in these instructions, enter the password provided by the Oracle Install Coordinator.
Plug your laptop into the Oracle ILOM "SER MGT" port of the first new server.
Log in as root with the default welcome1 password:
login as: root Using keyboard-interactive authentication. Password: welcome1 Oracle(R) Integrated Lights Out Manager Version 3.1.2.12 r74388 Copyright (c) 2012, Oracle and/or its affiliates. All rights reserved. ->
Open the console and log in with the same credentials:
-> start /SP/console Are you sure you want to start /SP/console (y/n)? y Serial console started. To stop, type ESC ( bda7 login: User: root Password: <password> #
If you do not see a prompt, press Enter.
Copy the original Oracle Big Data Appliance configuration files from the first server in the rack (server 1) to the new server where you are logged in. Replace IP_address with the real IP address of server 1:
# scp root@IP_address:/opt/oracle/bda/\{rack-network.json,cluster-network.json\} /opt/oracle/bda
Alternatively, use a USB flash drive to copy the file between servers.
Copy the new networkexpansion.json configuration file from a USB flash drive to the same /opt/oracle/dba directory:
# mkdir /mnt/usb # /opt/oracle/bda/bin/showusb /dev/sdm1 # mount /dev/sdm1 /mnt/usb # cp /mnt/usb/networkexpansion.json /opt/oracle/bda
See "Copying the Configuration Files to Oracle Big Data Appliance" for general instructions on using a flash drive.
Complete the following procedure to configure the network.
Run the rack-networksetup script:
# cd /opt/oracle/bda # cd network # ./rack-networksetup
After this step completes, the servers have their custom IP addresses.
Reconnect the Cisco switch uplink cable to the local network.
If you lost your SSH connection when the IP addresses changed, then reconnect:
Use SSH to connect to Oracle ILOM of the first server, using the custom IP address.
Log in as the root user with the default welcome1 password.
If you use a serial connection, you may still be connected when rack-networksetup completes. If not, type Esc+( to get the Oracle ILOM prompt before reconnecting.
Verify the InfiniBand topology by running bdacheckib —m on the rack configuration file. For example, if the rack name is rack01:
# cd /opt/oracle/bda # bdacheckib -m rack01-network.json
Verify that passwordless SSH still works:
# cd /opt/oracle/bda # dcli -j "eth0_ips[range]" "hostname ; date"
If you are prompted for a password, type Ctrl+c several times. If the dcli command fails, then repeat Steps 5 and 6 of the first procedure to configure the servers.
Use the output from the date utility in Step 5 to verify that the clocks for all servers and switches are synchronized. The times should be within a few seconds.
If they are not synchronized, then you can manually change the time, restart the device, or check the routing through the Cisco switch to the NTP server. Issue this command to check the status of an NTP client:
# ntpq -p 127.0.0.1
# ./cluster-networksetup | tee /tmp/bdanet_exp.out
The tee utility captures the output in a file while still allowing it to appear on your screen.
Log out of the server, unless it is the first server in the cluster.
You can add the new servers to an existing cluster, or configure them as a new cluster.
To add the servers to an existing cluster:
Complete the instructions in "Adding Servers to a Cluster."
To configure the servers as a separate cluster:
Run Oracle Big Data Appliance Configuration Generation Utility, and generate the configuration files. See "Generating the Configuration Files."
Run the Mammoth utility to install the software. See "Installing the Software on a New Rack."