You can install and configure up to two in-rack expansion kits in an Oracle Big Data Appliance starter rack. This chapter explains how to configure the hardware provided in an expansion kit.
This chapter consists of the following sections:
Installing the Servers in the Oracle Big Data Appliance Rack
Installing Oracle Big Data Appliance Software on the New Servers
Note:
Oracle recommends that an Oracle field engineer performs the procedures described in this chapter.An Oracle Big Data Appliance starter rack has the same hardware configuration as a full rack, except that it comes with six servers instead of 18. All switches and power supplies are included in the starter rack, and do not need to be upgraded or supplemented to support additional servers.
An in-rack expansion kit provides six more servers and the components needed to install them. You can install one expansion kit to form a 12-server rack, or two expansion kits to form a full, 18-server rack.
If you install the second rack at a later time, then you must repeat the instructions provided in this chapter. Otherwise, you can install and configure all 12 new servers together.
Before starting the installation, ensure that you received all components, cables, and accessories, as shown in the following tables.
Table 8-1 lists the components and accessories.
Table 8-1 In-Rack Expansion Kit Components and Accessories
| Part Number | Quantity | Description |
|---|---|---|
|
250-2020 |
1 |
Velcro roll |
|
7071129 |
1 |
Cable labeling kit |
|
7078043 or 7067086 |
6 |
Sun Server X4-2L or Sun Server X3-2L |
|
7046514 |
12 |
2U snap-in slide rail kit |
|
7042273 |
6 |
1U-2U slim-rail CMA kit |
|
1 |
Server documentation and license packages |
Table 8-2 lists the cables.
Ensure that the starter rack is in good working order before you install the in-rack expansion kit:
Check the LEDs on all components to verify a normal operating state. Resolve any issues before proceeding.
Verify that the Sun Rack II PDU firmware is at or above the minimum version of 1.06. See "Updating the PDU Firmware Version."
The following instructions describe how to physically install the new servers in the starter rack.
Caution:
Two people are needed to install the servers in the rack.To install the servers in the rack:
Unpack all power cables, Ethernet cables, and InfiniBand cables, and mark them with the labels provided in the kit:
Mark each end of a cable with the appropriate label. Take care to put the "from" and "to" labels on the correct end.
Select the appropriate PDU power cord labels for your installation, either single-phase or three-phase.
Plug in the cables at both ends. Refer to the cabling tables in Appendix C.
Install the server rack mount kits in the rack. Start at the first empty unit at the bottom and work up toward the top of the rack.
Install the servers in the rack rails. After installing each server, ensure that both it and the unit below it can slide in and out normally.
Install the cable arms and cables into each server, starting with the first new server at the bottom (server 7 or 13) and working up to the top unit. Ensure that all InfiniBand cables are plugged in to the correct ports on both ends. Keep the power cables out of the way while you route the InfiniBand and Ethernet cables.
Caution:
Do not plug the power cables into the PDU strips until you are ready to turn on the PDU and configure it. Plug the power cables only into the servers now.After installing the new servers in the rack, you can configure them.
To configure the new servers:
Disconnect the Cisco switch uplink cable (port 48) from the network, in case the factory IP addresses duplicate those on the local network.
Plug the power cords into the appropriate outlets. Route the cables to PDU-A last. When you are done, Oracle ILOM starts, the service processor LED is solid green, and the system is in Standby mode with a flashing OK LED.
Press the power button on the servers to turn them on. The system automatically detects the factory IP addresses. See "Factory Network Settings."
Using a laptop connected to the first new server (server 7 or 13), log in as root using SSH. The default password is welcome1.
Set up passwordless SSH for the root user. For range, enter [7:12] when you install the first expansion kit, [13-18] for the second expansion kit, or [7:18] for both kits.
# setup-root-ssh -j "eth0_ips[range]" -p welcome1
Confirm that passwordless SSH works properly across the range of new servers. Use the same value for range as you did in step 5.
# dcli -j "eth0_ips[range]" "hostname; date"
Compare the output of the following command with the system serial number (S/N) sticker on the front of each system. Verify the order of each server assignment, where the range for the first expansion kit is 7 to 12, and for the second expansion kit is 13 to 18.
[# dcli -j "eth0_ips[7:12]" "dmidecode -s chassis-serial-number"
192.168.1.7: 1143FMM064
.
.
.
192.168.1.12: 1144FMM1A0
If you find inconsistencies, then verify that the InfiniBand cables are plugged into the correct ports on the gateway switches for each node.
Set the rack master serial number in the ILOM to the S/N on the front left rack sticker (also displayed on the rear of the rack near PDU-A):
# dcli -j "eth0_ips[range]" "ipmitool sunoem cli 'set /SP \system_identifier="\"Oracle Big Data Appliance rack-master-serial-number\""'"
This example sets the range to [7:12] for the first expansion kit, and sets the serial number to AK00086253:
# dcli -j "eth0_ips[7:12]" "ipmitool sunoem cli 'set /SP \system_identifier="\"Oracle Big Data Appliance AK00086253\""'"
The system identifier has a 40-character limit.
Verify that the rack master serial number is set correctly. This example uses the range for the first expansion rack:
# dcli -j "eth0_ips[7:12]" "ipmitool sunoem cli 'show /SP system_identifier'" | grep =
192.168.1.7: system_identifier = Oracle Big Data Appliance AK00086253
.
.
.
192.168.1.12: system_identifier = Oracle Big Data Appliance AK00086253
Complete the physical hardware checks beginning with Step 7 in "Configuring the Oracle Big Data Appliance Servers." In all dcli commands, be sure to enter the range, using the eth0_ips[range] syntax shown previously.
The following topics explain how to add the new servers to the Ethernet administrative network:
See Also:
"Configuring the Network."These instructions use a serial connection to Oracle ILOM. If you use an SSH connection instead, then you lose the console connection when the IP addresses change from the factory settings to the custom settings in "Configuring the Network."
Follow these steps to copy the configuration files to the first new server in the rack.
To copy the configuration files:
Plug your laptop into the Oracle ILOM "SER MGT" port of the first new server (7 or 13).
Log in as root with the default welcome1 password:
login as: root Using keyboard-interactive authentication. Password: welcome1 Oracle(R) Integrated Lights Out Manager Version 3.1.2.12 r74388 Copyright (c) 2012, Oracle and/or its affiliates. All rights reserved. ->
Open the console and log in with the same credentials:
-> start /SP/console Are you sure you want to start /SP/console (y/n)? y Serial console started. To stop, type ESC ( bda7 login: User: root Password: welcome1 #
If you do not see a prompt, press Enter.
Copy the original Oracle Big Data Appliance configuration file from the first server in the rack (server 1) to the new server where you are logged in. Replace IP_address with the real IP address of server 1:
# scp root@IP_address:/opt/oracle/bda/BdaDeploy.json /opt/oracle/bda
Alternatively, use a USB flash drive to copy the file between servers.
Copy the new BdaExpansion.json configuration file from a USB flash drive to the same /opt/oracle/dba directory:
# mkdir /mnt/usb # /opt/oracle/bda/bin/showusb /dev/sdm1 # mount /dev/sdm1 /mnt/usb # cp /mnt/usb/BdaExpansion.json /opt/oracle/bda
See "Copying the Configuration Files to Oracle Big Data Appliance" for general instructions on using a flash drive.
Complete the following procedure to configure the network.
Run the networksetup-one script:
# cd /opt/oracle/bda # cd network # ./networksetup-one
After this step completes, the servers have their custom IP addresses.
Reconnect the Cisco switch uplink cable to the local network.
If you lost your SSH connection when the IP addresses changed, then reconnect:
Use SSH to connect to Oracle ILOM of the first server, using the custom IP address.
Log in as the root user with the default welcome1 password.
If you use a serial connection, you may still be connected when networksetup-one completes. If not, type Esc+( to get the Oracle ILOM prompt before reconnecting.
Verify the InfiniBand topology:
# cd /opt/oracle/bda # bdacheckib BdaDeploy.json
Verify that passwordless SSH still works:
# cd /opt/oracle/bda [# dcli "hostname ; date"
If you are prompted for a password, type Ctrl+c several times. If the dcli command fails, then repeat Steps 5 and 6 of the first procedure to configure the servers.
Use the output from the date utility in Step 5 to verify that the clocks for all servers and switches are synchronized. The times should be within a few seconds.
If they are not synchronized, then you can manually change the time, restart the device, or check the routing through the Cisco switch to the NTP server. Issue this command to check the status of an NTP client:
# ntpq -p
# ./networksetup-two | tee /tmp/bdanet_exp.out
The tee utility captures the output in a file while still allowing it to appear on your screen.
Log out of the server, unless it is the first server in the cluster.
Complete the following procedure to ensure that the network is working properly
Log in to the first server in the cluster.
Verify that the network and all network services are working properly:
# bdachecknet | tee -a /tmp/bdanet_exp.out
bdachecknet: check syntax and static semantics of opt/oracle/bda/BdaDeploy.json
bdachecknet: passed
bdachecknet: ping test private infiniband ips (bondib0 40gbs)
bdachecknet: passed
bdachecknet: ping test admin ips (eth0 1gbs)
bdachecknet: passed
.
.
.
If the systems are not able to resolve and use the DNS names, then manually correct the /etc/resolv.conf files and verify the reverse lookup.
Submit an install base update service request in My Oracle Support to update your assets.
You can add the new servers to an existing cluster, or configure them as a new cluster.
To add the servers to an existing cluster:
Complete the instructions in "Adding Servers to a Cluster."
To configure the servers as a separate cluster:
Run Oracle Big Data Appliance Configuration Generation Utility, and generate the configuration files. See "Generating the Configuration Files."
Run the Mammoth to install the software. See "Installing the Software on a New Rack."