This chapter consists of the following sections:
Note:Oracle recommends that an Oracle field engineer performs the procedures described in this chapter.
An Oracle Big Data Appliance starter rack has the same hardware configuration as a full rack, except that it comes with six servers instead of 18. All switches and power supplies are included in the starter rack, and do not need to be upgraded or supplemented to support additional servers.
An in-rack expansion kit provides six more servers and the components needed to install them. You can install one expansion kit to form a 12-server rack, or two expansion kits to form a full, 18-server rack.
If you install the second in-rack expansion kit at a later time, then you must repeat the instructions provided in this chapter. Otherwise, you can install and configure all 12 new servers together.
Table 8-1 lists the components and accessories.
Cable labeling kit
7078043 or 7067086
Sun Server X4-2L or Sun Server X3-2L
2U snap-in slide rail kit
1U-2U slim-rail CMA kit
Server documentation and license packages
Table 8-2 lists the cables.
Ensure that the starter rack is in good working order before you install the in-rack expansion kit:
Check the LEDs on all components to verify a normal operating state. Resolve any issues before proceeding.
Verify that the Sun Rack II PDU firmware is at or above the minimum version of 1.06. See "Updating the PDU Firmware Version."
Caution:Two people are needed to install the servers in the rack.
To install the servers in the rack:
Unpack all power cables, Ethernet cables, and InfiniBand cables, and mark them with the labels provided in the kit:
Mark each end of a cable with the appropriate label. Take care to put the "from" and "to" labels on the correct end.
Select the appropriate PDU power cord labels for your installation, either single-phase or three-phase.
Plug in the cables at both ends. Refer to the cabling tables in Appendix C.
Install the server rack mount kits in the rack. Start at the first empty unit at the bottom and work up toward the top of the rack.
Install the servers in the rack rails. After installing each server, ensure that both it and the unit below it can slide in and out normally.
Install the cable arms and cables into each server, starting with the first new server at the bottom (server 7 or 13) and working up to the top unit. Ensure that all InfiniBand cables are plugged in to the correct ports on both ends. Keep the power cables out of the way while you route the InfiniBand and Ethernet cables.
Caution:Do not plug the power cables into the PDU strips until you are ready to turn on the PDU and configure it. Plug the power cables only into the servers now.
To configure the new servers:
Disconnect the Cisco switch uplink cable (port 48) from the network, in case the factory IP addresses duplicate those on the local network.
Plug the power cords into the appropriate outlets. Route the cables to PDU-A last. When you are done, Oracle ILOM starts, the service processor LED is solid green, and the system is in Standby mode with a flashing OK LED.
Press the power button on the servers to turn them on. The system automatically detects the factory IP addresses. See "Factory Network Settings."
Using a laptop connected to the first new server (server 7 or 13), log in as
root using SSH. The default password is
Set up passwordless SSH for the
root user. For range, enter [7:12] when you install the first expansion kit, [13-18] for the second expansion kit, or [7:18] for both kits.
# setup-root-ssh -j "eth0_ips[range]" -p welcome1
Confirm that passwordless SSH works properly across the range of new servers. Use the same value for range as you did in step 5.
# dcli -j "eth0_ips[range]" "hostname; date"
Compare the output of the following command with the system serial number (S/N) sticker on the front of each system. Verify the order of each server assignment, where the range for the first expansion kit is 7 to 12, and for the second expansion kit is 13 to 18.
# dcli -j "eth0_ips[7:12]" "dmidecode -s chassis-serial-number" 192.168.1.7: 1143FMM064 . . . 192.168.1.12: 1144FMM1A0
If you find inconsistencies, then verify that the InfiniBand cables are plugged into the correct ports on the gateway switches for each node.
Set the rack master serial number in the ILOM to the S/N on the front left rack sticker (also displayed on the rear of the rack near PDU-A):
# dcli -j "eth0_ips[range]" "ipmitool sunoem cli 'set /SP \system_identifier="\"Oracle Big Data Appliance rack-master-serial-number\""'"
This example sets the range to [7:12] for the first expansion kit, and sets the serial number to AK00086253:
# dcli -j "eth0_ips[7:12]" "ipmitool sunoem cli 'set /SP \system_identifier="\"Oracle Big Data Appliance AK00086253\""'"
The system identifier has a 40-character limit.
Verify that the rack master serial number is set correctly. This example uses the range for the first expansion rack:
# dcli -j "eth0_ips[7:12]" "ipmitool sunoem cli 'show /SP system_identifier'" | grep = 192.168.1.7: system_identifier = Oracle Big Data Appliance AK00086253 . . . 192.168.1.12: system_identifier = Oracle Big Data Appliance AK00086253
Complete the physical hardware checks beginning with Step 7 in "Configuring the Oracle Big Data Appliance Servers." In all
dcli commands, be sure to enter the range, using the
[range] syntax shown previously.
The following topics explain how to add the new servers to the Ethernet administrative network:
See Also:"Configuring the Network."
These instructions use a serial connection to Oracle ILOM. If you use an SSH connection instead, then you lose the console connection when the IP addresses change from the factory settings to the custom settings in "Configuring the Network."
To copy the configuration files:
Plug your laptop into the Oracle ILOM "SER MGT" port of the first new server (7 or 13).
Log in as
root with the default
login as: root Using keyboard-interactive authentication. Password: welcome1 Oracle(R) Integrated Lights Out Manager Version 126.96.36.199 r74388 Copyright (c) 2012, Oracle and/or its affiliates. All rights reserved. ->
Open the console and log in with the same credentials:
-> start /SP/console Are you sure you want to start /SP/console (y/n)? y Serial console started. To stop, type ESC ( bda7 login: User: root Password: welcome1 #
If you do not see a prompt, press Enter.
Copy the original Oracle Big Data Appliance configuration file from the first server in the rack (server 1) to the new server where you are logged in. Replace IP_address with the real IP address of server 1:
# scp root@IP_address:/opt/oracle/bda/BdaDeploy.json /opt/oracle/bda
Alternatively, use a USB flash drive to copy the file between servers.
# mkdir /mnt/usb # /opt/oracle/bda/bin/showusb /dev/sdm1 # mount /dev/sdm1 /mnt/usb # cp /mnt/usb/BdaExpansion.json /opt/oracle/bda
See "Copying the Configuration Files to Oracle Big Data Appliance" for general instructions on using a flash drive.
Complete the following procedure to configure the network.
# cd /opt/oracle/bda # cd network # ./networksetup-one
After this step completes, the servers have their custom IP addresses.
Reconnect the Cisco switch uplink cable to the local network.
If you lost your SSH connection when the IP addresses changed, then reconnect:
Use SSH to connect to Oracle ILOM of the first server, using the custom IP address.
Log in as the
root user with the default
If you use a serial connection, you may still be connected when networksetup-one completes. If not, type Esc+( to get the Oracle ILOM prompt before reconnecting.
Verify the InfiniBand topology:
# cd /opt/oracle/bda # bdacheckib BdaDeploy.json
Verify that passwordless SSH still works:
# cd /opt/oracle/bda # dcli -j "eth0_ips[7:12]" "hostname ; date"
Use the output from the
date utility in Step 5 to verify that the clocks for all servers and switches are synchronized. The times should be within a few seconds.
If they are not synchronized, then you can manually change the time, restart the device, or check the routing through the Cisco switch to the NTP server. Issue this command to check the status of an NTP client:
# ntpq -p
# ./networksetup-two | tee /tmp/bdanet_exp.out
Log out of the server, unless it is the first server in the cluster.
Complete the following procedure to ensure that the network is working properly
Log in to the first server in the cluster.
Verify that the network and all network services are working properly:
Submit an install base update service request in My Oracle Support to update your assets.
You can add the new servers to an existing cluster, or configure them as a new cluster.
To add the servers to an existing cluster:
Complete the instructions in "Adding Servers to a Cluster."
To configure the servers as a separate cluster: