8 Expanding an Oracle Big Data Appliance Starter Rack

You can install and configure up to 12 servers on a starter rack.

Oracle Big Data Appliance supports clusters with homogenous or mixed X5–2L, X6–2L, X7–2L, and X8-2L clusters. However, if you include X8-2L or X7–2L servers in the cluster, then you must first upgrade the cluster to Oracle Big Data Appliance release v4.10.0 or higher.

All Big Data Appliances support mixed server hardware within a cluster, but all nodes within a cluster must run the same Big Data Appliance software release and must also uniformly run either Oracle Linux 6 or Oracle Linux 7.


Oracle recommends that an Oracle field engineer performs the procedures described in this chapter.

Also note that physically adding servers to a partially-filled rack requires downtime. Extending a cluster onto new nodes in the same rack should not require downtime.

8.1 About Expanding an Oracle Big Data Appliance Starter Rack

An Oracle Big Data Appliance starter rack has the same hardware configuration as a full rack, except that it comes with 6 servers instead of 18. All switches and power supplies are included in the starter rack, and do not need to be upgraded or supplemented to support additional servers.

You can add up to 12 new servers on a starter rack by using a High Capacity node plus InfiniBand Infrastructure. You can either add all the new servers together or add some of the new servers at a later time on a starter rack.

All Oracle Big Data Appliance expansion nodes come with the in-rack cables required for power as well as connections to the administrative and InfiniBand networks. Adding servers to a rack requires no changes to multi-rack InfiniBand connections, since that is a separate cabling task from switch to switch.

8.1.1 Checking the Contents of a High Capacity Node plus InfiniBand Infrastructure

Before starting the installation, ensure that you received all components, cables, and accessories for the High Capacity node plus InfiniBand Infrastructure as shown in the following tables.

Table 8-1 lists the components and accessories.

Table 8-1 A High Capacity Node plus InfiniBand Infrastructure Components and Accessories

Quantity Description


Oracle Big Data Appliance High Capacity Node plus InfiniBand Infrastructure


2U snap-in slide rail kit


1U-2U slim-rail CMA kit


Velcro 25-foot roll


Cable labeling kit


Server documentation and license packages

Table 8-2 lists the cables.

Table 8-2 A High Capacity Node plus Infiniband Infrastructure Cables

Length Quantity Description

10 feet


Red Ethernet Cat 5 or Cat 5e

10 feet


Ethernet Black Cat 5 or Cat 5e

3 meters


InfiniBand QSFP

2 meters



8.1.2 Checking the Status of a Starter Rack

Ensure that the starter rack is in a good working condition before you install the High Capacity node plus InfiniBand Infrastructure:

  • Check the LEDs on all components to verify a normal operating state. Resolve any issues before proceeding.

  • Verify that the Sun Rack II PDU firmware is at or above the minimum version of 1.06. See "Updating the PDU Firmware Version."

8.2 Installing New Servers on an Oracle Big Data Appliance Starter Rack

The following describes how to physically install new servers on a starter rack.


Two people are needed to install new servers on a starter rack.

To install new servers on a starter rack:

  1. Unpack all power cables, Ethernet cables, and InfiniBand cables, and mark them with the labels provided in the kit:

    • Mark both ends of cable with the appropriate label. Take care to put the "from" and "to" labels on the correct end.

    • Select the appropriate PDU power cord labels for your installation, either single-phase or three-phase.

  2. Plug in the cables at both ends. Refer to the cabling tables in the appendixes.

  3. Install the server rack mount kits in the rack. Start at the first empty unit at the bottom and work up toward the top of the rack.

  4. Install the servers in the rack rails. After installing each server, ensure that the server and the unit below it can slide in and out normally.

  5. Install the cable arms and cables into each server, starting with the first new server at the bottom and working up to the top unit. Ensure that all InfiniBand cables are plugged in to the correct ports on both ends. Keep the power cables out of the way while you route the InfiniBand and Ethernet cables.


    Do not plug the power cables into the PDU strips until you are ready to turn on the PDU and configure it. Plug the power cables only into the servers now.

8.3 Configuring the New Oracle Big Data Appliance Servers

After installing the new servers in the rack, you can configure them as described in the following steps.

To configure the new servers:

At the <password> placeholder in these instructions, enter the password provided by the Oracle Install Coordinator.

  1. Disconnect the Cisco switch uplink cable (port 48) from the network, in case the factory IP addresses duplicate those on the local network.

  2. Plug the power cords into the appropriate outlets. Route the cables to PDU-A last. When you are done, Oracle ILOM starts, the service processor LED is solid green, and the system is in Standby mode with a flashing OK LED.

  3. Press the power button on the servers to turn them on. The system automatically detects the factory IP addresses. See "Factory Network Settings."

  4. Using a laptop connected to the first new server, log in as root using SSH. The default password is provided by the Oracle Install Coordinator.

  5. Set up passwordless SSH for the root user. For range, for example, enter [7:10] when you install four new servers.

    # setup-root-ssh -j "eth0_ips[range]" -p <password> 

    See "setup-root-ssh."

  6. Confirm that passwordless SSH works properly across the range of new servers. Use the same value for range as you did in step 5.

    # dcli -j "eth0_ips[range]" "hostname; date"
  7. Compare the output of the following command with the system serial number (S/N) sticker on the front of each system.

    # dcli -j "eth0_ips[range]" "dmidecode -s chassis-serial-number" 1143FMM064
         . 1144FMM1A0

    If you find inconsistencies, then verify that the InfiniBand cables are plugged into the correct ports on the gateway switches for each node.

  8. Set the rack master serial number in the ILOM to the S/N on the front left rack sticker (also displayed on the rear of the rack near PDU-A):

    # dcli -j "eth0_ips[range]" "ipmitool sunoem cli 'set /SP \system_identifier="\"Oracle Big Data Appliance rack-master-serial-number\""'"

    This example sets the range to [7:10] for the four new servers, and sets the serial number to AK00086253:

    # dcli -j "eth0_ips[7:10]" "ipmitool sunoem cli 'set /SP \system_identifier="\"Oracle Big Data Appliance AK00086253\""'"

    The system identifier has a 40-character limit.

  9. Verify that the rack master serial number is set correctly. This example uses the range for the four new servers:

    # dcli -j "eth0_ips[7:10]" "ipmitool sunoem cli 'show /SP system_identifier'" | grep = system_identifier = Oracle Big Data Appliance AK00086253
         . system_identifier = Oracle Big Data Appliance AK00086253
  10. Complete the physical hardware checks beginning with Step 10 in "Configuring the Oracle Big Data Appliance Servers." In all dcli commands, be sure to enter the range, using the eth0_ips[range] syntax shown previously.

8.4 Setting Up the Network

The following topics explain how to add the new servers to the Ethernet administrative network:

8.4.1 Copying the Configuration Files

These instructions use a serial connection to Oracle ILOM. If you use an SSH connection instead, then you lose the console connection when the IP addresses change from the factory settings to the custom settings in "Configuring the Network."

Follow these steps to copy the configuration files to the first new server in the rack.

To copy the configuration files:

At the <password> placeholder in these instructions, enter the password provided by the Oracle Install Coordinator.

  1. Plug your laptop into the Oracle ILOM "SER MGT" port of the first new server.

  2. Log in as root with the password provided to you:

    login as: root
    Using keyboard-interactive authentication.
    Password: password
    Oracle(R) Integrated Lights Out Manager
    Version r74388
    Copyright (c) 2012, Oracle and/or its affiliates. All rights reserved.
  3. Open the console and log in with the same credentials:

    -> start /SP/console
    Are you sure you want to start /SP/console (y/n)? y
    Serial console started. To stop, type ESC (
    bda7 login:
    User: root
    Password:  <password> 
  4. If you do not see a prompt, press Enter.

  5. Copy the original Oracle Big Data Appliance configuration files from the first server in the rack (server 1) to the new server where you are logged in. Replace IP_address with the real IP address of server 1:

    # scp root@IP_address:/opt/oracle/bda/\{rack-network.json,cluster-network.json\} /opt/oracle/bda

    Alternatively, use a USB flash drive to copy the file between servers.

  6. Copy the new Copy the new <rackname>-rack-expansion.json and <clustername>-cluster-expansion.json configuration files from a USB flash drive to the same /opt/oracle/dba directory. In the copy command, rename the files as shown in the example below.

    # mkdir /mnt/usb
    # /opt/oracle/bda/bin/showusb /dev/sdm1
    # mount /dev/sdm1 /mnt/usb
    # cp /mnt/usb/ <clustername>-cluster-expansion.json /opt/oracle/bda/cluster-expansion.json
    # cp /mnt/usb/ <rackname>-rack-expansion.json /opt/oracle/bda/rack-expansion.json

    See "Copying the Configuration Files to Oracle Big Data Appliance" for general instructions on using a flash drive.

8.4.2 Configuring the Network

Complete the following procedure to configure the network.

  1. Run the rack-networksetup script:

    # cd /opt/oracle/bda
    # cd network
    # ./rack-networksetup

    After this step completes, the servers have their custom IP addresses.

  2. Reconnect the Cisco switch uplink cable to the local network.

  3. If you lost your SSH connection when the IP addresses changed, then reconnect:

    1. Use SSH to connect to Oracle ILOM of the first server, using the custom IP address.

    2. Log in as the root user with the password provided to you by the Oracle installation coordinator.

    If you use a serial connection, you may still be connected when rack-networksetup completes. If not, type Esc+( to get the Oracle ILOM prompt before reconnecting.

  4. Verify that the administrative network is properly set up. Run the following command and check that the IP addresses reported back match the addresses in the configuration files.

    # dcli -j "eth0_ips[range]" "ifconfig eth0 | grep 'inet '"
  5. Verify the InfiniBand topology by running bdacheckib —m on the rack configuration file:

    # cd /opt/oracle/bda
    # bdacheckib -m rack-network.json
  6. Verify that passwordless SSH still works:

    # cd /opt/oracle/bda
    # dcli  -j "eth0_ips[range]" "hostname ; date"

    If you are prompted for a password, type Ctrl+c several times. If the dcli command fails, then repeat Steps 5 and 6 of the first procedure to configure the servers.

  7. Use the output from the date utility in Step 6 to verify that the clocks for all servers and switches are synchronized. The times should be within a few seconds.

    If they are not synchronized, then you can manually change the time, restart the device, or check the routing through the Cisco switch to the NTP server. Issue this command to check the status of an NTP client:

    # ntpq -p
  8. Run cluster-networksetup:

    # ./cluster-networksetup | tee /tmp/bdanet_exp.out

    The tee utility captures the output in a file while still allowing it to appear on your screen.

  9. Log out of the server, unless it is the first server in the cluster.

8.4.3 Verifying the Network Configuration

Complete the following procedure to ensure that the network is working properly

  1. Log in to the first server in the cluster.
  2. Verify that the network and all network services are working properly:
    # bdachecknet-cluster | tee -a /tmp/bdanet_exp.out
    bdachecknet-cluster: do basic sanity checks on /opt/oracle/bda/rack-network.json and /opt/oracle/bda/cluster-network.json
    bdachecknet-cluster: passed
    bdachecknet-cluster: checking for  rack-expansion.json
    bdachecknet-cluster: ping test private infiniband ips (bondib0 40gbs)
    bdachecknet-cluster: passed
    bdachecknet-cluster: ping test admin ips (eth0 1gbs)
    bdachecknet-cluster: passed
    bdachecknet-cluster: test client network (eoib) resolve and reverse resolve
    bdachecknet-cluster: passed
  3. If the systems are not able to resolve and use the DNS names, then manually correct the /etc/resolv.conf files and verify the reverse lookup.
  4. Submit an install base update service request in My Oracle Support to update your assets.

8.5 Installing Oracle Big Data Appliance Software on the New Servers

You can add the new servers to an existing cluster, or configure them as a new cluster.

To add the servers to an existing cluster:

To configure the servers as a separate cluster:

  1. Run Oracle Big Data Appliance Configuration Generation Utility, and generate the configuration files. See "Generating the Configuration Files."

  2. Run the Mammoth utility to install the software. See "Installing the Software on a New Rack."