7 Configuring an Oracle Big Data Appliance Full Rack or Starter Rack
This chapter describes how to configure the system, accounts, and software for an Oracle Big Data Appliance full rack or starter rack.
This chapter contains the following sections:
See Also:
Note:
-
Oracle recommends that you have the
bda-install-preview.htmlfile available while completing these procedures. See "About the Configuration Files." -
Oracle recommends that an Oracle field engineer performs the configuration steps described in this chapter.
7.1 Configuring the KVM Switch
A KVM switch is provided only in Oracle Big Data Appliance racks with Sun Fire X4270 M2 servers.
7.1.1 Starting the KVM Switch
-
Ensure that all connected components are powered off.
-
Pull the KVM tray out from the front of the rack, and open it using the handle.
-
Touch the touch pad.
-
Toggle between the host and KVM interface by pressing the Ctrl key on the left side twice, similar to double-clicking with a mouse. You see the main user interface page.
-
In the navigator on the left, select Target Devices under Unit View. In the main display area, verify that 18 target devices are listed with Action set to KVM Session.
The sessions are numbered from the bottom of the rack to the top.
-
If 18 sessions are not shown:
-
In the navigator under Appliances, expand Ports, and then select IQ Adaptors.
-
In the main display area, choose the Port table heading to sort the sessions by port number.
-
Note any missing sessions, so that you can fix them later.
-
In the navigator, choose Target Devices to return to the Target Devices page.
-
7.1.2 Connecting the KVM Switch to the Management Network
- In the navigator under User Accounts, select Local.
- Under Users, choose Admin.
- Set the password for the Admin account to the password provided by the Oracle Install Coordinator and then choose Save. Do not modify any other parameters.
- Under Appliance Settings, expand Network, and then choose IPv4. The Network Information page appears.
- Enter values for Address, Subnet, and Gateway, and then choose Save.
- Under Appliance Settings, choose DNS to display the DNS Information page.
- Enter the IP addresses of the DNS servers, and then choose Save.
- Under Network, choose General to display the Appliance General Network Settings page.
- Connect the KVM LAN1 Ethernet port to the management network.
- To verify that the port has been configured correctly, ensure that the Media Access Control (MAC) address on the Network Settings page matches the label next to the LAN1/LAN2 ports at the rear of the KVM switch.
- Under Users, select Overview to display the Unit Maintenance page.
- Enter a name for the KVM switch, and then choose Save.
- To restart the KVM switch, choose Reboot under Overview and Yes to confirm.
7.1.3 Configuring the KVM Switch to Access the Servers
- Under Unit View, select Target Devices to display the Target Devices page.
- Start up the server. The power button is on the front panel.
- Click the server name in the Name column to display the Unit Overview page.
- Click Overview and overwrite the name with the Oracle standard naming format of customer prefix, node type, and number. For example,
bda1node03identifies the third server from the bottom of the bda1 rack. - Click Save.
- Repeat Steps 2 through 5 for each server in the rack. Each server boots through BIOS, and boots the operating system with the default factory IP configuration.
7.2 Configuring the Ethernet Switch
Oracle Big Data X7 racks are now shipped with the Cisco Nexus 9348GC-FXP Ethernet Switch.
Previously X7 racks shipped with the Cisco Nexus 93108-1G. The configuration steps are the same for both versions of the Nexus switch. Older Racks were shipped with the Cisco Catalyst 4948 Ethernet Switch.
7.2.1 Configuring the Cisco Nexus 93108-1G or 9348 Ethernet Switch
The Cisco Nexus 93108-1G or 9348 Ethernet switch supplied with Oracle Big Data Appliance rack is minimally configured during installation.
Note that the Cisco Nexus 93108-1G or 9348 switch supports multiple uplinks to the customer network by utilizing the QSFP+ ports. This is a more complicated switch setup due to the redundant connectivity, and should be performed by the customer's network administrator.
Whether you are configuring the switch for the first time, or configuring a replacement switch, use the following procedures:
7.2.1.1 Performing the Initial Switch Configuration for the Cisco Nexus 93108-1G Ethernet Switch
During the initial configuration, you reset the switch and use the Basic System Configuration Dialog to configure the switch.
Before configuring the switch, note the following:
-
The Cisco Ethernet switch should not be connected until the running configuration has been verified, and any necessary changes have been made by the network administrator.
-
The Cisco Ethernet switch should not be connected to the customer network until the IP addresses on all components have been configured in Oracle Big Data Appliance rack. This is to prevent any duplicate IP address conflicts which are possible due to the default addresses set in the components when shipped.
Configuration should be done with the network administrator.
7.2.2 Configuring the Cisco Catalyst 4948 Ethernet Switch
The Cisco Catalyst 4948 Ethernet switch supplied with Oracle Big Data Appliance has IPBASEK9-MZ firmware. The switch is minimally configured during installation. These procedures configure the Cisco Ethernet switch into one large virtual LAN.
The Cisco Ethernet switch configuration consists of these topics and procedures:
7.2.2.1 Scope of the Configuration
This configuration disables IP routing and sets the following:
-
Host name
-
IP address
-
Subnet mask
-
Default gateway
-
Domain name
-
Name server
-
NTP server
-
Time
-
Time zone
7.2.2.2 Prerequisites for Configuring the Ethernet Switch
To avoid disrupting the customer network, observe these prerequisites:
-
Do not connect the Cisco Ethernet switch until the network administrator has verified the running configuration and made any necessary changes.
-
Do not connect the Cisco Ethernet switch to the customer network until the IP addresses on all components have been configured in Oracle Big Data Appliance. This sequence prevents any duplicate IP address conflicts, which are possible due to the default addresses set in the components when shipped.
-
Configure the Cisco Ethernet switch with the network administrator.
See Also:
-
Oracle Big Data Appliance Site Checklists
7.2.2.3 Configuring the Ethernet Switch on the Customer Network
At the <password> placeholder in these instructions, enter the password provided by the Oracle Install Coordinator.
To configure the Ethernet switch on the customer network:
-
Connect a serial cable from the Cisco switch console to a laptop or similar device. An RJ45 to DB9 serial cable is included with the Cisco documentation package.
The status lights turn on and this message is displayed:
Switch con0 is now available Press RETURN to get started.
-
If the Ethernet switch displays status lights and a welcome message, then continue. If it does not, then go to "Correcting the Autoboot Settings."
-
Ensure that the terminal session is recorded on the laptop by logging the output. You can use the output as a record that the switch has been configured correctly. The default serial port speed is 9600 baud, 8 bits, no parity, 1 stop bit, and no handshake.
-
Change to
enablemode using the following command.Switch> enable Password: <
password> -
Configure the network for a single VLAN. The following is an example of the configuration:
Switch# configure terminal Enter configuration commands, one per line. End with CNTL/Z. Switch(config)# interface vlan 1 Switch(config-if)# ip address 10.7.7.34 255.255.255.0 Switch(config-if)# end Switch# *Jan 23 15:54:00.506: %SYS-5-CONFIG_I:Configured from console by console Switch# write memory Building configuration... Compressed configuration from 2474 bytes to 1066 bytes [OK ]
-
If the network does not require IP routing on the switch, then disable the default IP routing setting and configure the default gateway. This method is preferred. Consult the network administrator if in doubt.
Switch# configure terminal Enter configuration commands, one per line. End with CNTL/Z. Switch(config)# no ip routing Switch(config)# ip default-gateway 10.17.7.1 Switch(config)# end *Jan 23 15:54:00.506: %SYS-5-CONFIG_I:Configured from console by console Switch# write memory Building configuration... Compressed configuration from 3600 bytes to 1305 bytes[OK]]
-
If the network requires IP routing on the switch, and then keep the default IP routing setting and configure the default gateway as follows:
Switch# configure terminal Enter configuration commands, one per line. End with CNTL/Z. Switch(config)# ip route 0.0.0.0 0.0.0.0 10.7.7.1 Switch(config)# end *Jan 23 15:55:02.506: %SYS-5-CONFIG_I:Configured from console by console Switch# write memory Building configuration... Compressed configuration from 2502 bytes to 1085 bytes [OK ]
-
Set the host name of the switch to the name specified in the installation preview. This example uses the name bda1sw-ip.
Switch# configure terminal Enter configuration commands, one per line. End with CNTL/Z. Switch(config)# hostname bda1sw-ip bda1sw-ip(config)# end *Jan 23 15:57:50.886: %SYS-5-CONFIG_I: Configured from console by console bda1sw-ip# write memory Building configuration... Compressed configuration from 3604 bytes to 1308 bytes[OK] bda1sw-ip#
The system host name appears in the prompt.
-
Configure up to three DNS servers, using the IP addresses specified in the installation preview instead of the examples shown here. If no DNS service is available, then set the domain name anyway, so that the SSH keys are configurable.
bda1sw-ip# configure terminal Enter configuration commands, one per line. End with CNTL/Z. bda1sw-ip(config)# ip domain-name mydomain.com bda1sw-ip(config)# ip name-server 10.7.7.3 bda1sw-ip(config)# ip name-server 129.148.5.5 bda1sw-ip(config)# ip name-server 10.8.160.1 bda1sw-ip(config)# end *Sep 15 14:26:37.045: %SYS-5-CONFIG_I: Configured from console by console bda1sw-ip# write memory Building configuration... Compressed configuration from 2603 bytes to 1158 bytes[OK]
Correcting the Autoboot Settings
If the Cisco Ethernet switch does not display any link or status lights, then the incorrect autoboot settings may prevent it from starting. If the serial console port shows the rommon prompt, then complete the following procedure.
Correcting the autoboot settings of the Cisco switch:
-
Verify that the IPBASE and IPBASEK9 firmware reboot files are available:
rommon 1 >dir bootflash: File Size (Bytes) File Name ------------------------------ 25213107 cat4500e-ipbase-mz.150-2.SG2.bin 32288280 cat4500e-ipbasek9-mz.151-1.SG.bin Total space = 128165888 bytes, Available = 60643328 bytes -
Restart the switch manually using the default image:
rommon 2 >boot Rommon reg: 0x00004780 Reset2Reg: 0x00009EFF ## Fortooine controller 0x0B5A753B..0x0B70B6CD original size:0x001C3EA5 ################## diagsk5 version 5.0.1 . . . -
Press Enter to get the Switch prompt.
-
Enter enable mode. Enter the password provided by the Oracle Install Coordinator, just press Enter:
Switch>enable Password: The default password is <
password> Switch# -
Verify that the boot variable is not set:
Switch# show bootvar BOOT variable = CONFIG_FILE variable does not exist BOOTLDR variable does not exist Configuration register is 0x2102 -
Verify that IPBASEK9 is available in the boot file system:
Switch# dir bootflash: Directory of bootflash:/ 6 -rw- 25213107 Dec 3 2012 17:27:38 +00:00 cat4500eipbase-mz.150-2.SG2.bin 7 -rw- 32288280 Apr 1 2013 11:41:54 +00:00 cat4500eipbasek9-mz.151-1.SG.bin 128165888 bytes total (60643328 bytes free) -
If IPBASEK9 is not listed in the
bootflashdirectory, then you must install it before continuing. See My Oracle Support Document ID 1415044.1. -
Set the
BOOTvariable as shown in this example:Switch# config term Enter configuration commands, one per line. End with CNTL/Z. Switch(config)# config-register 0x2102 Switch(config)# no boot system Switch(config)# boot system bootflash:cat4500e-ipbasek9-mz.151-1.SG.bin Switch(config)# end
-
Verify the new
BOOTsetting:Switch# show running-config | include boot boot-start-marker boot system bootflash:cat4500e-ipbasek9-mz.151-1.SG.bin boot-end-marker -
Save the changes to the startup configuration:
Switch# copy running-config startup-config all Destination filename [startup-config]? Enter Switch# write memory Building configuration... Compressed configuration from 3628 bytes to 1281 bytes[OK]
-
Reload the firmware.
Switch# reload Proceed with reload? [confirm] y *Apr 24 09:31:32.887: %SYS-5-RELOAD: Reload requested by console. Reload reason: Reload command
The switch automatically restarts and then continues the reload process.
7.2.2.4 Setting Up Telnet Access on the Ethernet Switch
Oracle Big Data Appliance ships with a version of the Cisco Ethernet switch software that supports both telnet and SSH. Telnet access is optional. The following procedure describes how to enable and disable remote telnet access.
Note:
Telnet access is not secure.
To set up telnet access to the Ethernet switch:
At the <password> placeholder in these instructions, enter the password provided by the Oracle Install Coordinator.
-
Set the password for telnet access if necessary; it should already be set when you receive Oracle Big Data Appliance.
bda1sw-ip# configure terminal Enter configuration commands, one per line. End with CNTL/Z. bda1sw-ip(config)# enable password <
passwordbda1sw-ip(config)# enable secret <passwordThe enable secret you have chosen is the same as your enable password. This is not recommended. Re-enter the enable secret. bda1sw-ip(config)# end bda1sw-ip# write memory *Jan 23 15:57:50.886: %SYS-5-CONFIG_I:Configured from console by console Building configuration... Compressed configuration from 2502 bytes to 1085 bytes [OK ] -
Set up telnet access. In this example, the first
loginoutput shows that the password is not set and telnet access is disabled. If thelogincommand returns nothing, then the password is set and telnet access is available.Switch# configure terminal Enter configuration commands, one per line. End with CNTL/Z. bda1sw-ip(config)# line vty 0 15 bda1sw-ip(config-line)# login %Login disabled on line 1,until 'password' is set %Login disabled on line 2,until 'password' is set %Login disabled on line 3,until 'password' is set ... bda1sw-ip(config-line)# password <
password> bda1sw-ip(config-line)# login bda1sw-ip(config-line)# end bda1sw-ip# write memory *Jan 23 15:58:53.630: %SYS-5-CONFIG_I: Configured from console by console Building configuration... Compressed configuration from 3604 bytes to 1308 bytes[OK] -
To disable telnet access and prevent remote access, follow this example:
Switch# configure terminal Enter configuration commands, one per line. End with CNTL/Z. bda1sw-ip(config)# line vty 0 15 bda1sw-ip(config-line)# no password bda1sw-ip(config-line)# login %Login disabled on line 1, until 'password' is set %Login disabled on line 2, until 'password' is set %Login disabled on line 3, until 'password' is set ... bda1sw-ip(config-line)# end bda1sw-ip# write memory *Jan 23 15:58:53.630: %SYS-5-CONFIG_I: Configured from console by console Building configuration... Compressed configuration from 3786 bytes to 1468 bytes[OK]
7.2.2.5 Configuring SSH on the Ethernet Switch
The current firmware supports remote connections using SSH. To set up SSH on the Cisco Ethernet switch, follow the example below.
Where the <password> placeholder occurs in this example, enter the password provided by the Oracle Install Coordinator.
bda1sw-ip# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
bda1sw-ip(config)# crypto key generate rsa
% You already have RSA keys defined named bda1sw-ip.mydomain.com.
% Do you really want to replace them? [yes/no]: yes
Choose the size of the key modulus in the range of 360 to 2048 for
your General Purpose Keys. Choosing a key modulus greater than 512
may take a few minutes.
How many bits in the modulus [512]: 768
% Generating 768 bit RSA keys, keys will be non-exportable...[OK]
bda1sw-ip(config)# username admin password 0 <password>
bda1sw-ip(config)# line vty 0 15
bda1sw-ip(config-line)# transport input ssh
bda1sw-ip(config-line)# exit
bda1sw-ip(config)# aaa new-model
bda1sw-ip(config)# ip ssh time-out 60
bda1sw-ip(config)# ip ssh authentication-retries 3
Oracle Internal and Approved Partners Only Page 11 of 47 V 1.0 Created: 21. Dec. 2012
Task Comment Check
bda1sw-ip(config)# ip ssh version 2
bda1sw-ip(config)# end
*Sep 15 14:26:37.045: %SYS-5-CONFIG_I: Configured from console by console
bda1sw-ip# write memory
Building configuration...
Compressed configuration from 2603 bytes to 1158 bytes[OK]7.2.2.6 Setting the Clock and Time Zone on the Ethernet Switch
The Cisco Ethernet switch keeps internal time in coordinated universal time (UTC) format.
To set the local time and time zone, ordering is important. The following is an example of setting the local time to the U.S. Eastern time zone:
bda1sw-ip# configure terminal Enter configuration commands, one per line. End with CNTL/Z. bda1sw-ip(config)# clock timezone EST -5 bda1sw-ip(config)# clock summer-time EDT recurring bda1sw-ip(config)# end bda1sw-ip# clock set 15:00:00 January 23 2012 bda1sw-ip# write memory Building configuration... Compressed configuration from 3778 bytes to 1433 bytes[OK] bda1sw-ip# show clock 15:00:18.819 EST Mon Jan 23 2012
Clock Commands
The following are descriptions of the commands for setting the clock and time zone:
-
To use UTC, enter this command:
no clock timezone global configuration
-
To use a time zone:
clock timezone
zonehours-offset[minutes-offset]In this command, zone is the time zone to display when standard time is in effect, hours-offset is the hours offset from UTC, and minutes-offset is the minutes offset from UTC.
-
clock summer-time
zonerecurring [week day month hh:mm week day month\hh:mm[offset]]In this command, zone is the time zone to be displayed when summer time (daylight savings time) is in effect, week is the week of the month (1 to 5 or last), day is the day of the week, month is the month, hh:mm is the time in 24-hour format, and offset is the number of minutes to add during summer time. The default offset is 60 minutes. Summer time is disabled by default.
-
To manually set the clock to any time:
clock sethh:mm:ss month day yearIn this command, hh:mm:ss is the hour, month, and second in 24-hour format, day is the day of the month, month is the month, and year is the year. The time specified is relative to the configured time zone.
7.2.2.7 Configuring the NTP Servers for the Ethernet Switch
Configure up to two NTP servers. Use IP addresses instead of names, which fail when the switch is not connected to the network. If you must use names, then configure the NTP servers after the switch is connected to the network and DNS is resolving names correctly.
The following example shows the NTP server synchronized to local time when the Cisco switch is connected to the network and has access to NTP.
bda1sw-ip# configure terminal Enter configuration commands, one per line. End with CNTL/Z. bda1sw-ip(config)# ntp server 10.196.23.254 prefer bda1sw-ip(config)# ntp server 192.168.9.19 bda1sw-ip(config)# end Jan 23 20:00:41.235: %SYS-5-CONFIG_I:Configured from console by console bda1sw-ip# write memory Building configuration... Compressed configuration from 3870 bytes to 1487 bytes [OK ] bda1sw-ip# show ntp status output varies by network bda1sw-ip# show clock 15:00:57.919 EST Mon Jan 23 2012
A symbol may precede the output of the show clock command:
- * (star): The time is not authoritative.
- . (dot) The time is authoritative, but NTP is not synchronized.
- (blank) The time is authoritative, as shown in the example.
7.2.2.8 Disabling the Spanning Tree
By default, spanning tree is enabled on Cisco switches, but it is disabled in the factory on all the server connection ports 1 to 47.
Adding a switch with spanning tree enabled can cause problems on networks where it is disabled intentionally. As a precaution, disable spanning tree on the uplink port's VLAN before connecting to the customer network.
If spanning tree must be turned on before or after the connection, then the network administrator can specify the appropriate protocol settings, as described in the Cisco switch configuration documentation.
To disable the spanning tree:
-
If the port must be disabled, then enter these commands:
bda1sw-ip# configure terminal Enter configuration commands, one per line. End with CNTL/Z. Switch(config)# no spanning-tree vlan 1 Switch(config)# end Jan 23 20:01:15.083: %SYS-5-CONFIG_I: Configured from console by console bda1sw-ip# write memory Building configuration... Compressed configuration from 2654 bytes to 1163 bytes[OK]
-
To verify the disabling of the spanning tree:
bda1sw-ip# show spanning-tree vlan 1 Spanning tree instance(s) for vlan 1 does not exist. -
To re-enable the default protocol settings, if necessary:
Switch# configure terminal Enter configuration commands, one per line. End with CNTL/Z. Switch(config)# spanning-tree vlan 1 Switch(config)# end Switch# write memory
7.3 Configuring the InfiniBand Leaf and Spine Switches
Oracle Big Data Appliance has two Sun Network QDR InfiniBand Gateway leaf switches and one Sun Datacenter InfiniBand Switch 36 spine switch. To configure the switches, follow these procedures for each one:
Note:
Only Oracle Big Data Appliance servers may be connected to the Oracle Big Data Appliance InfiniBand NM2 switches. Also, it is not permitted to interconnect systems other than Oracle Engineered System servers (such as Oracle Big Data Appliance and Exadata Database Machine) to the InfiniBand fabric. The InfiniBand fabric is intimately connected with the operating system and the kernel and is designed to support Oracle Engineered System servers only.7.3.1 Configuring an InfiniBand Switch
Before you Start:
-
For details on remote Ethernet connections to
192.168.1.201,192.168.1.202,or192.168.1.203, see "Connecting to Oracle ILOM Using the Network". -
For details on local serial connections, see "Connecting to Oracle ILOM Using a Serial Connection".
To configure an InfiniBand switch:
7.3.3 Checking the Health of an InfiniBand Switch
To check the health of an InfiniBand leaf or spine switch:
-
Open the Fabric Management shell:
-> show /SYS/Fabric_Mgmt
The prompt changes from -> to FabMan@hostname->
-
Check the firmware version. You can find the current version and related information at Oracle Support note 1534255.1.
FabMan@bda1sw-02->version SUN DCS gw version: 2.0.5-2 Build time: Nov 29 2011 16:05:05 FPGA version: 0x34 SP board info: Manufacturing Date: 2011.05.31 Serial Number: "NCD6Q0126" Hardware Revision: 0x0006 Firmware Revision: 0x0000 BIOS version: SUN0R100 BIOS date: 06/22/2010 FabMan@bda1sw-02-> -
Check the overall health of the switch and correct any issues:
FabMan@bda1sw-ib2-> showunhealthy OK - No unhealthy sensors -
Check the environment. Ensure that all tests return OK and PASSED, and correct any issues before continuing. This example shows a problem with PSU1 caused by a loose power cord. See the line starting with WARNING PSU.
FabMan@bda1sw-ib2-> env_test Environment test started: Starting Environment Daemon test: Environment daemon running Environment Daemon test returned OK Starting Voltage test Voltage ECB OK Measured 3.3V Main = 3.28 Measured 3.3V Standby = 3.39 V Measured 12V = 11.97 V Measured 5V = 4.99 V Measured VBAT = 3.09 V Measured 1.0V = 1.01 V Measured I4 1.2V = 1.22 V Measured 2.5V = 2.52 V Measured V1P2 DIG = 1.19 V Measured V1P2 ANG = 1.18 V Measured 1.2V BridgeX = 1.22 V Measured 1.8V = 1.78 V Measured 1.2V Standby = 1.20 V Voltage test returned OK Starting PSU test: PSU 0 present OK WARNING PSU 1 present AC Loss PSU test returned 1 faults Starting Temperature test: Back temperature 30 Front temperature 29 SP temperature 36 Switch temperature 52, . . . -
Verify a priority setting of 5 for the InfiniBand Gateway leaf switches or 8 for the InfiniBand Switch 36 spine switch:
FabMan@bda1sw-ib2-> setsmpriority list Current SM settings: smpriority 5 controlled_handover TRUE subnet_prefix 0xfe80000000000000If
smpriorityis correct, then you can skip the next step. -
To correct the priority setting:
-
Stop the InfiniBand Subnet Manager:
FabMan@bda1sw-ib2-> disablesm -
Set the priority to 5 for the InfiniBand Gateway leaf switches or 8 for the InfiniBand Switch 36 spine switch. This example is for a leaf switch:
FabMan@bda1sw-ib2-> setsmpriority 5 -
Restart the InfiniBand Subnet Manager:
FabMan@bda1sw-ib2-> enablesm
-
-
Exit the Fabric Management shell:
FabMan@bda1sw-ib2-> exit -> -
Exit the Oracle ILOM shell:
-> exit
-
Log in to the switch as
rootand restart it to ensure that all changes take effect:# reboot
-
Repeat these steps for the other InfiniBand switches.
7.3.4 Checking the Sun Network QDR InfiniBand Gateway Switches
You must check all gateway switch instance numbers (NM2-GW) in the fabric to ensure that the automatic setup assigned unique instance numbers to each switch, both within each rack and among all racks in a multirack environment.
If two switches in the fabric share the same instance number, then the IB-10GbE ports might not work correctly.
The switches must not use consecutive numbers. Oracle recommends even numbers only. From the factory, the Oracle Big Data Appliance gateway switches use the numbers 10 and 20.
If Oracle Big Data Appliance racks are connected to Oracle Exalogic Elastic Cloud Machine racks, then verify that the instance numbers are even and unique to each switch in the fabric.
Note:
Firmware for all Sun Network QDR InfiniBand Gateway switches in the fabric should be at least version 2.1.8-1. Version 2.2.7-1 is recommended. Version 2.2.9 is also validated for Oracle Big Data Appliance racks, but for large production clusters currently running much older firmware, a more conservative upgrade to version 2.1.8-1 or 2.2.7-1 is recommended.
To check the gateway switches:
At the <password> placeholder in these instructions, enter the password provided by the Oracle Install Coordinator.
-
Login as user
ilom-adminon rack 1 gateway switch 1:localhost: ilom-admin password: <
password> -
Enter the Fabric Management shell:
-> show /SYS/Fabric_Mgmt NOTE: show on Fabric_Mgmt will launch a restricted Linux shell. User can execute switch diagnosis, SM Configuration and IB monitoring commands in the shell. To view the list of commands, use "help" at rsh prompt. Use exit command at rsh prompt to revert back to ILOM shell. FabMan@hostname-> -
Display the current number:
FabMan@bda1sw-iba->setgwinstance --list No BXM system name set, using 6 last bits of the ip-address, value: 24If you see
No BXM system name set(as shown here), or a system name of 0, then you must set the instance number manually. -
If rack 1 gateway switch 1 is not set to 10:
FabMan@bda1sw-iba->setgwinstance 10 Stopping Bridge Manager..-. [ OK ] Starting Bridge Manager. [ OK ] FabMan@bda1sw-iba->setgwinstance --list BXM system name set to 10
-
Repeat Steps 1 to 4 for gateway switch 2. Set it to 20 instead of 10.
-
If multiple racks are cabled together (Oracle Big Data Appliance racks and Oracle Exalogic Elastic Cloud Machine racks), then repeat Steps 1 to 5 on each rack. Increment the instance numbers from the previous rack by 2 to get a unique number:
-
Set rack 2 gateway switch 1 to 12 (10+2).
-
Set rack 2 gateway switch 2 to 22 (20+2).
Continue in this way, assigning even numbers between 0 and 63.
-
7.4 Configuring the Power Distribution Units
The power distribution unit (PDU) configuration consists of these procedures:
7.4.1 Connecting the PDUs to the Network
The power distribution units (PDUs) are configured with a static IP address to connect to the network for monitoring. Ensure that you have the following network information before connecting the PDUs:
-
Static IP address
-
Subnet mask
-
Default gateway
To connect the PDUs to the network:
-
Use a web browser to access the PDU metering unit by entering the factory default IP address for the unit. The address of PDU A is
192.168.1.210, and the address of PDU B is192.168.1.211.The Current Measurement page opens. If the PDUs do not respond, then they probably have different IP addresses:
-
Press and hold the red RESET button.
-
When the display starts to count down in about 10 to 15 seconds, release the button and then press it again without holding it down.
-
When the monitor is restarting, note the current IP setting, and whether it is a dynamic host configuration protocol (DHCP) or static address. Use those addresses to connect.
-
-
Log in as the
adminuser on the PDU metering unit. The default password isadm1noradmin. Change this password after configuring the network. -
To check the firmware version, select Module Info. The version should be 1.06. You can upgrade the firmware for a lower version. See "Updating the PDU Firmware Version."
-
Click Net Configuration in the upper left of the page.
-
Confirm that the DHCP Enabled option is not selected.
-
Enter the following network settings for the PDU metering unit:
-
IP address
-
Subnet mask
-
Default gateway
-
-
Click Submit to set the network settings and reset the PDU metering unit.
7.4.2 Updating the PDU Firmware Version
To update the PDU firmware version:
-
Download the latest firmware version from My Oracle Support:
-
Log in at
https://support.oracle.com. -
Select the Patches & Updates tab.
-
For Patch Search, click Product or Family (Advanced).
-
For Product, select Sun Rack II PDU.
-
For Release, select Sun Rack II PDU 1.0.6.
-
Click Search to see the Patch Search Results page.
-
Click the patch name, such as 16523441.
-
Download the file.
-
-
Unzip the file on your local system. It has a name such as p16523441_1060_Generic.zip. Then unzip the PDU file, which has a name like OraclePDU_v1.0.6.zip.
-
In the PDU web interface, select Net Configuration and log in as
adminwith passwordadm1n. -
Scroll down to Firmware Update.
-
Click Browse, select the MKAPP_V1.06.DL file, and click Submit.
-
Click Browse, select the HTML_V1.06.DL file, and click Submit.
-
Click Module Info to verify the firmware version number.
-
Click Net Configuration, and then click Logout.
7.5 Configuring the Oracle Big Data Appliance Servers
Before configuring the network, ensure that the Oracle Big Data Appliance servers are set up correctly.
Note:
If you must connect to the Oracle ILOM serial management port, then the baud rate setting on the servers changes from the default Oracle ILOM setting of 9600 to 115200 baud, 8 bits, no parity, and 1 stop bit. To display the server console, use this command:
start HOST/console
To check the Oracle Big Data Appliance servers:
-
Power on all servers by pressing the power button on the front panel of each server. The servers take 5 to 10 minutes to run through the normal startup tests.
-
Use a laptop to connect to a server:
-
Open an SSH connection using PuTTY or a similar utility.
-
Enter the default IP address of the server.
-
-
Log in as the
rootuser to the first server. The password is provided by the Oracle Install Coordinator.. -
Set up passwordless SSH for
rootby entering thesetup-root-sshcommand, as described in "Setting Up Passwordless SSH".For starter racks, with or without expansion, you must use the
-jparameter. Seesetup-root-ssh.# setup-root-ssh -j "etho0_ips[range]" -
Verify that SSH keys are distributed across the rack:
# cd /opt/oracle/bda # dcli "hostname ; date"
-
If prompted for a password, enter Ctrl+C several times. This prompt confirms distribution of the keys, so that you can continue to the next step. Otherwise, generate the root SSH keys across the rack, replacing password with a valid password and using the
-jparameter for a starter rack:# setup-root-ssh -p password [-j "eth0_ips[range]"]
Enter the
dclicommand in Step 5 again to verify the keys. -
Compare the system serial numbers (S/N) on the front sticker to the server assignments from the bottom (1) to the top (18) of the rack, to ensure that they match:
# dcli "dmidecode -s "chassis-serial-number" 192.168.10.1: 1143FMM064 . . . 192.168.10.18: 1144FMM1A0 -
Compare the rack master serial number on the rack front sticker to the rack assignment, to ensure that they match:
# dcli "ipmitool sunoem cli 'show /SP system_identifier'" | grep = 192.168.10.1: system_identifier = Oracle Big Data Appliance AK00024695 . . . 192.168.10.18: system_identifier = Oracle Big Data Appliance AK00024695 -
If the rack master serial number is incorrect, then reset it in the ILOM on every system. Enter the following command on one line, with no breaks, replacing R-MSN with the rack master serial number (such as AK00024695):
# dcli -l root \"ipmitool sunoem cli' set /SP system_identifier=\"\"Oracle Big Data Appliance R-MSN\""'" \> /tmp/set-rack-csn.outIn the previous syntax, the
system_identifierhas a 40-character limit. -
Verify that the InfiniBand ports are up, two on each server (36 total for a full rack of 18 servers).
# dcli ibstatus | grep phys 192.168.10.1: phys state: 5: LinkUp 192.168.10.1: phys state: 5: LinkUp .. 192.168.10.18: phys state: 5: LinkUp 192.168.10.18: phys state: 5: LinkUp -
Verify that the InfiniBand ports are running at 40 Gbps (4X QDR):
# dcli ibstatus | grep rate | more 192.168.10.1: rate: 40 Gb/sec (4X QDR) 192.168.10.1: rate: 40 Gb/sec (4X QDR) .. 192.168.10.18: rate: 40 Gb/sec (4X QDR) 192.168.10.18: rate: 40 Gb/sec (4X QDR) -
Verify that Oracle ILOM does not detect any faults:
# dcli 'ipmitool sunoem cli "show faulty"'
The output should appear as follows for each server:
bda1node02-adm.example.com: Connected. Use ^D to exit. bda1node02-adm.example.com: -> show faulty bda1node02-adm.example.com: Target | Property | Value bda1node02-adm.example.com:-------------+---------------------+----------- bda1node02-adm.example.com: bda1node02-adm.example.com: -> Session closed bda1node02-adm.example.com: Disconnected
-
Power up each server and verify that
BDA_IMAGING_SUCCEEDEDandBDA_REBOOT_SUCCEEDEDare in the/rootdirectory of every server:# dcli 'ls /root/BDA*'
-
Save the hardware profile output from each system in a file for review, replacing filename with a file name of your choice:
# dcli bdacheckhw > filename
-
Check the hardware profile output file using commands like the following. In these example, the file name is
all-bdahwcheck.out.-
To verify that there are no failures in the hardware profile:
grep -vi SUCCESS ~/all-bdahwcheck.out
This command returns all lines that do not show
SUCCESS, so no returned lines means no failures. -
To verify 32 cores:
grep cores ~/all-bdahwcheck.out
-
To verify 63 GB of memory:
grep memory ~/all-bdahwcheck.out
-
To verify four fans:
grep fans ~/all-bdahwcheck.out
-
To verify that the status is OK for both power supplies:
grep supply ~/all-bdahwcheck.out
-
To verify that disks 0 to 11 are all the same model, online, spun up, and no alert:
grep disk ~/all-bdahwcheck.out | grep "model\|status" | more
-
To verify that the host channel adapter model is Mellanox Technologies MT26428 ConnectX VPI PCIe 2.0:
grep Host ~/all-bdahwcheck.out | grep model | more
-
-
Save the RAID configuration in a file, replacing filename with a file name of your choice:
dcli MegaCli64 -ldinfo -lall -a0 | grep "Virtual Drive\|State" > filename
-
Verify that 12 virtual drives (0 to 11) are listed for each server. In this example, the RAID configuration is stored in a file named
all-ldstate.out.less ~/all-ldstate.out
-
Save the software profile output from each system into a file for review, replacing filename with a file name of your choice:
dcli bdachecksw > filename
-
Verify that the partition setup and software versions are correct. In this example, the software profile is stored in a file named all-bdaswcheck.out.
less ~/all-bdaswcheck.out
-
Verify the system boots in this order: USB, RAID Slot 0, PXE:
# dcli "ubiosconfig export all -x /tmp/bios.xml" [# dcli "grep -m1 -A20 boot_order /tmp/bios.xml" | more [root@bda01 bda]# dcli "ubiosconfig export all -x /tmp/bios.xml" [root@bda01 bda]# dcli "grep -m1 -A20 boot_order /tmp/bios.xml" | more <boot_order> <boot_device> <description>USB:USBIN:ORACLE UNIGEN-UFD PMAP</description> <instance>1</instance> </boot_device> <boot_device> <description>RAID:PCIE6:(Bus 50 Dev 00)PCI RAID Adapter</description> <instance>1</instance> </boot_device> <boot_device> <description>PXE:NET0:IBA XE Slot 4000 v2196</description> <instance>1</instance> </boot_device> <boot_device> <description>PXE:NET1:IBA XE Slot 4001 v2196</description> <instance>1</instance> </boot_device> <boot_device> <description>PXE:NET2:IBA XE Slot 8800 v2196</description> <instance>1</instance> </boot_device> -
If the boot order is incorrect:
-
Log in to the ILOM for that server and force the next restart into BIOS setup:
# set /HOST boot_device=bios
-
Restart the server and correct the BIOS.
-
7.6 Configuring the Network
Oracle Big Data Appliance Configuration Generation Utility generates the cluster-network.json and rack-network.json files, which are used to configure all three network interfaces and all Oracle ILOMs. See "Generating the Configuration Files" if you do not have this file.
The network configuration consists of these procedures:
7.6.1 Copying the Configuration Files to Oracle Big Data Appliance
To copy the configuration files to Oracle Big Data Appliance:
Note:
A valid configuration file is required on all nodes of the cluster. The same file (and the same version of the file) should be deployed to this path on all nodes of the cluster – /opt/oracle/bda/install/state/config.json.
In addition to this, there should be a valid set of network configuration files on all nodes of the cluster. There are there two alternate valid sets of configuration files:
network.jsonIf this is present it should be the same on all nodes of a rack and contain all network information for all nodes of the rack. If this is present then rack-network.json and cluster-network.json should not be present
- Both
rack-network.jsonandcluster-network.json.If these are present then
network.jsonshould not be present.rack-network.jsonshould be the same on all nodes in a rack and should contain only admin network information.cluster-network.jsonshould be the same on all nodes of the cluster and should contain client and private network information
7.6.2 Starting the Network Configuration
Note:
If you have installed Oracle Big Data Appliance releases previously, note that the setup scriptsnetworksetup-one and networksetup-two are deprecated.
Run rack-networksetup in place of networksetup-one. Run cluster-networksetup in place of networksetup-two.
The rack-networksetup script reads rack-network.json and sets up the host names and Oracle ILOM names for all servers and configures the administrative network and the private InfiniBand network. The procedure varies slightly depending on whether you are configuring a new rack (either a full rack or a starter rack) , or, are configuring additional servers after deploying the starter rack.
To start the network configuration on a new rack:
At the <password> placeholder in these instructions, enter the password provided by the Oracle Install Coordinator.
-
Log in as the
rootuser on the first server. Use the password provided by the Oracle Install Coordinator.# ssh root@192.168.10.1 Password: <password> -
For a new rack, verify that
/opt/oracle/bdacontains the newrack-network.jsonandcluster-network.jsonfiles.For the additional servers over the starter rack, verify that
/opt/oracle/bdacontains bothrack-network.jsonandcluster-network.json(ornetwork.json) from an existing node and the newnetworkexpansion.jsonfile. -
Verify that the InfiniBand network is healthy:
# cd /opt/oracle/bda # bdacheckib -s LINK bdasw-ib1.0B ... bdasw-ib3.8B UP LINK bdasw-ib1.1B ... bdasw-ib2.8B UP LINK bdasw-ib3.15A ... bda2.HCA-1.2 UP LINK bdasw-ib3.15B ... bda1.HCA-1.2 UP LINK bdasw-ib3.14A ... bda4.HCA-1.2 UP LINK bdasw-ib3.14B ... bda3.HCA-1.2 UP LINK bdasw-ib3.13A ... bda6.HCA-1.2 UP . . .
The InfiniBand network is used extensively during software installation. To perform more in-depth checks of the fabric, run the
iblinkinfoutility. -
Remove passwordless SSH. See "remove-root-ssh."
-
Reconnect to the server using its ILOM instead of SSH:
# ssh root@192.168.1.101 Password: <
password> -> start -f /HOST/console bda1 login: User: root Password: <password>The
rack-networksetupscript restarts the network services, so you do not see all of the output using an SSH connection. -
Begin the network configuration:
# cd /opt/oracle/bda/network # ./rack-networksetup
Example 7-1 Sample Output from rack-networksetup
# ./rack-networksetup rack-networksetup: do basic sanity checks on /opt/oracle/bda/rack-network.json and /opt/oracle/bda/cluster -network.json rack-networksetup: passed rack-networksetup: Found 6 node(s) accessible rack-networksetup: If this number is not the total number of nodes, rack-networksetup: please make sure that all nodes are ON and connected correctly rack-networksetup: Type 'r' if you want to retry (If you are waiting for nodes to boot and become accessib le) rack-networksetup: Type 'n' to abort network setup rack-networksetup: Type 'y' to continue configuring 6 node(s) Continue using 6 nodes? [r/n/y]: y rack-networksetup: checking for rack-expansion.json rack-networksetup: ping servers on ship admin network by ip rack-networksetup: passed rack-networksetup: test ssh to servers on ship admin network by ip rack-networksetup: passed rack-networksetup: verify servers on ship admin are BDA nodes rack-networksetup: passed rack-networksetup: copy /opt/oracle/bda/rack-network.json and /opt/oracle/bda/cluster-network.json to servers rack-networksetup: verify transfer of /opt/oracle/bda/rack-network.json and /opt/oracle/bda/cluster-network.json rack-networksetup: passed rack-networksetup: executing network settings on all servers rack-networksetup: wait a few seconds for the network to restart on <IP address> rack-networksetup: wait a few seconds for the network to restart on <IP address> rack-networksetup: rack-networksetup: bdanetworktool.main: Start time = Sun Feb 6 14:33:30 PST 2017 rack-networksetup: rack-networksetup: bdanetworktool.main: Host = bda02 rack-networksetup: rack-networksetup: bdanetworktool.main: Using deploy files /opt/oracle/bda/cluster-network.json and /opt/oracle/bda/rack-network.json rack-networksetup: rack-networksetup: bdanetworktool.main: Using node nr = 2 rack-networksetup: rack-networksetup: bdanetworktool.main: Using local files = false rack-networksetup: rack-networksetup: bdanetworktool.main: Apply nws settings on usb = false rack-networksetup: rack-networksetup: bdanetworktool.main: Apply dns settings = false rack-networksetup: rack-networksetup: bdanetworktool.main: Apply client nw settings = false rack-networksetup: rack-networksetup: bdanetworktool.main: Remove client nw settings = false rack-networksetup: rack-networksetup: bdanetworktool.main: Apply ntp settings = false rack-networksetup: rack-networksetup: bdanetworktool.main: Apply extra ib settings = false rack-networksetup: rack-networksetup: bdanetworktool.main: Apply ssh settings = true rack-networksetup: rack-networksetup: bdanetworktool.main: Apply bondib0 settings = false rack-networksetup: rack-networksetup: bdanetworktool.main: Apply eth0 settings = true rack-networksetup: rack-networksetup: bdanetworktool.main: Apply ilom settings = true rack-networksetup: rack-networksetup: bdanetworktool.main: Apply /etc/hosts settings = true rack-networksetup: rack-networksetup: bdanetworktool.main: Set timezone = true rack-networksetup: rack-networksetup: bdanetworktool.main: Set host name = true rack-networksetup: rack-networksetup: bdanetworktool.main: Creating = /etc/sysconfig/network-scripts/ifcfg-eth0 rack-networksetup: rack-networksetup: bdanetworktool.main: Creating = /etc/sysconfig/network rack-networksetup: rack-networksetup: bdanetworktool.main: Removing all from /etc/hosts rack-networksetup: rack-networksetup: bdanetworktool.main: Removing known ip entries from /etc/hosts . . .
7.6.3 Connecting to the Administrative Network
Before completing the network configuration, you must connect Oracle Big Data Appliance to the 1 GbE administrative (management) network.
To connect Oracle Big Data Appliance to the administrative network:
-
Connect port 48 of the Cisco Ethernet switch to the administrative network.
-
To use the small form-factor pluggable (SFP) ports for a fiber uplink in port 48 (optional), reconfigure the port:
bda1sw-ip# configure terminal Enter configuration commands, one per line. End with CNTL/Z. bda1sw-ip(config)# interface gigabitEthernet 1/48 bda1sw-ip(config-if)# media-type sfp bda1sw-ip(config-if)# end bda1sw-ip# *Sep 15 14:12:06.309: %SYS-5-CONFIG_I: Configured from console by console bda1sw-ip# write memory bda1sw-ip# copy running-config startup-config
See Also:
For SFP port configuration details, see theCisco Catalyst 4948 Install Guide at
http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst4900/4948/4948_in/05modcfg.html
7.6.4 Connecting to the Client Network
You must also connect Oracle Big Data Appliance to the 10 GbE client network.
To connect Oracle Big Data Appliance to the client network:
-
Connect the two Sun Network QDR InfiniBand Gateway leaf switches to the data center.
-
After the cables are routed, the network administrator may need to alter the network switch end configuration to recognize the links.
-
Use SSH to connect to a Sun Network QDR InfiniBand Gateway switch, and log in as the
ilom-adminuser. -
Enter the Fabric Management shell:
-> show /SYS/Fabric_Mgmt
-
Ensure that the Bridge entries have active links from the leaf switches to the customer network switch:
FabMan@bda1sw-iba-> listlinkupThis output shows four ports connected to each switch:
Connector 0A-ETH Present Bridge-0 Port 0A-ETH-1 (Bridge-0-2) up (Enabled) Bridge-0 Port 0A-ETH-2 (Bridge-0-2) down (Enabled) Bridge-0 Port 0A-ETH-3 (Bridge-0-1) up (Enabled) Bridge-0 Port 0A-ETH-4 (Bridge-0-1) down (Enabled) Connector 1A-ETH Present Bridge-1 Port 1A-ETH-1 (Bridge-1-2) up (Enabled) Bridge-1 Port 1A-ETH-2 (Bridge-1-2) down (Enabled) Bridge-1 Port 1A-ETH-3 (Bridge-1-1) up (Enabled) Bridge-1 Port 1A-ETH-4 (Bridge-1-1) down (Enabled)
7.6.5 Completing the Network Configuration
Note:
If you have installed Oracle Big Data Appliance releases previously, note that the setup scriptsnetworksetup-one and networksetup-two are deprecated.
To complete the configuration, run cluster-networksetup in place of networksetup-two.
The cluster-networksetup script reads cluster-network.json . It completes some steps started by rack-networksetup (steps that require a network connection). It also configures the default VLAN and all required VNICs for the 10 GbE client network. It then verifies all network connections and displays a message if it discovers any unexpected ones, including those caused by cabling mistakes.
The 10 GbE ports of the Sun Network QDR InfiniBand Gateway switches must be connected to the data center.
To complete the network configuration:
-
Ensure that both the administrative network and the client network are connected to Oracle Big Data Appliance.
Note:
This procedure fails if the networks are not connected. See "Connecting to the Administrative Network".
-
Run the following script to complete the network setup:
./cluster-networksetup | tee /tmp/network-conf.out
The previous command captures the output in a file named
/tmp/network-conf.out.
Example 7-2 shows sample output from the script.
Example 7-2 Sample Output from cluster-networksetup
# ./cluster-networksetup
cluster-networksetup: do basic sanity checks on /opt/oracle/bda/rack-network.json and /opt/oracle/bda/cluster-network.json
cluster-networksetup: passed
cluster-networksetup: checking for rack-expansion.json
cluster-networksetup: ping server ips on admin network
cluster-networksetup: passed
cluster-networksetup: test ssh server ips on admin network
cluster-networksetup: passed
cluster-networksetup: run connected network post script on each server
cluster-networksetup: post network setup for 10.100.100.101
cluster-networksetup: post network setup for 10.100.100.102
cluster-networksetup: post network setup for 10.100.100.103
cluster-networksetup: post network setup for 10.100.100.104
cluster-networksetup: post network setup for 10.100.100.105
cluster-networksetup: post network setup for this node
cluster-networksetup: passed
cluster-networksetup: verify infiniband topology
cluster-networksetup: passed
cluster-networksetup: start setup client network (10gigE over Infiniband)
cluster-networksetup: ping both gtw leaf switches
cluster-networksetup: passed
cluster-networksetup: confirm switches occur in ib fabric
cluster-networksetup: switch crosscheck failed matching all switches:
cluster-networksetup: -> BdaDeploy switch key values: leaf1: bda1sw-ib2, leaf2: bda1sw-ib3, spine: bda1sw-ib1
cluster-networksetup: -> ib switch values: bda1sw-ib1 bdasw-ib3 bda1sw-ib2
cluster-networksetup: passed
cluster-networksetup: verify existence of gateway ports
cluster-networksetup: passed
cluster-networksetup: ping server ips on admin network
cluster-networksetup: passed
cluster-networksetup: ping servers by name on admin network
cluster-networksetup: passed
cluster-networksetup: test ssh server ips on admin network
cluster-networksetup: passed
cluster-networksetup: test unique gw instance id
cluster-networksetup: passed
cluster-networksetup: wait a few seconds for the network to restart on 10.100.100.101
cluster-networksetup: wait a few seconds for the network to restart on 10.100.100.102
.
.
.
7.6.6 Checking the Virtual Interface Cards (VNICs)
Verify that the VNICs were created correctly on both gateway switches.
To check the VNICs:
-
Use SSH to connect to a Sun Network QDR InfiniBand Gateway switch, and log in as the
ilom-adminuser. -
Enter the Fabric Management shell:
-> show /SYS/Fabric_Mgmt
-
Verify that the active ports are assigned to the default VLAN (0):
FabMan@bda1sw-ib2-> showvlan Connector/LAG VLN PKEY ------------- --- ---- 0A-ETH-1 0 ffff 0A-ETH-3 0 ffff 1A-ETH-1 0 ffff 1A-ETH-3 0 ffff -
Verify that the VNICs were created round-robin on each server and 10 GbE interface:
FabMan@hostname-> showvnics ID STATE FLG IOA_GUID NODE IID MAC VLN PKEY GW --- -------- --- ----------------------- -------------------------------- ---- ----------------- --- ---- -------- 561 UP N 0021280001CF4C23 bda1node12 BDA 192.168.41.31 0000 CE:4C:23:85:2B:0A NO ffff 0A-ETH-1 . . . -
Ensure that you can ping in to and out from the 10 GB interfaces.
7.6.7 Verifying Access to the Network Services
Ensure that all network services are running correctly.
To verify access to the network services:
-
Verify that the dcli command is working:
# cd /opt/oracle/bda # dcli "hostname ; date"
-
If you are prompted for a password, then enter Ctrl+c several times. Otherwise, generate new SSH keys with the setup-root-ssh utility. See "setup-root-ssh."
-
Use the output from the
datecommand to check that the servers are synchronized within a few seconds. If they are not, then restart each device. Time differences can cause the software installation to fail.You can also check the NTP clients:
# ntpq -p 127.0.0.1
-
Verify network connectivity:
[root@node01 network]# bdachecknet-rack bdachecknet-rack: do basic sanity checks on /opt/oracle/bda/rack-network.json and /opt/oracle/bda/cluster-network.json bdachecknet-rack: passed bdachecknet-rack: checking for rack-expansion.json bdachecknet-rack: ping test private infiniband ips (bondib0 40gbs) bdachecknet-rack: passed bdachecknet-rack: ping test admin ips (eth0 1gbs) bdachecknet-rack: passed bdachecknet-rack: test client network (eoib) resolve and reverse resolve bdachecknet-rack: passed bdachecknet-rack: test client name array matches ip array bdachecknet-rack: passed bdachecknet-rack: ping servers on client network by ip bdachecknet-rack: passed bdachecknet-rack: test ntp servers bdachecknet-rack: passed bdachecknet-rack: ping client gateway bdachecknet-rack: passed bdachecknet-rack: test arp -a bdachecknet-rack: passed bdachecknet-rack: all checks succeeded -
If the network is not able to resolve both DNS names and IP addresses, then correct the
/etc/resolv.conffiles. Otherwise, the software installation will fail. -
Check the accessibility of the PDUs by entering the IP address or host name as the address in a browser.
If the connection is successful, you see the Current Measurement page.
7.7 Configuring Multiple Oracle Big Data Appliance Racks
The general process when installing multiple racks is to configure the individual racks, cable the racks together, and validate the multirack network.
7.7.1 Configuring and Cabling Multiple Racks
Complete one of the following procedures. They differ depending on whether you are setting up a new installation or adding racks to an operational installation, and whether there are other engineered systems, such as Oracle Exadata Database Machine.
To install multiple racks:
-
Complete the configuration steps for each rack individually:
-
Cable the Oracle Big Data Appliance racks together. See Connecting Multiple Oracle Big Data Appliance Racks.
-
Revalidate the InfiniBand network. See "Validating a Multirack InfiniBand Network."
To add one or more racks to an operational Oracle Big Data Appliance rack:
-
Complete the configuration steps for each new Oracle Big Data Appliance rack individually:
-
Unplug the power to the InfiniBand switches on the new racks.
-
Cable the Oracle Big Data Appliance racks together. See Connecting Multiple Oracle Big Data Appliance Racks.
-
Plug in power to the InfiniBand switches. Wait five minutes between switches while the subnet manager fabric is updated.
-
Revalidate the InfiniBand network. See "Validating a Multirack InfiniBand Network."
To connect Oracle Big Data Appliance racks to a new Oracle Exadata Database Machine or Oracle Exalogic Elastic Cloud Machine rack:
-
Complete the configuration steps for each Oracle Big Data Appliance rack individually:
-
Cable the Oracle Big Data Appliance racks together. See Connecting Multiple Oracle Big Data Appliance Racks.
-
Revalidate the InfiniBand network. See "Validating a Multirack InfiniBand Network."
To add one or more racks to an operational Oracle Exadata Database Machine or Oracle Exalogic Elastic Cloud Machine rack:
-
Complete the configuration steps for each new Oracle Big Data Appliance rack individually:
-
Unplug the power to the InfiniBand switches on the new racks.
-
Cable the Oracle Big Data Appliance, Oracle Exadata Database Machine, or Oracle Exalogic Elastic Cloud Machine racks together. See Connecting Multiple Oracle Big Data Appliance Racks.
-
Plug in power to the InfiniBand switches. Wait five minutes between switches while the subnet manager fabric is updated.
-
Ensure that the subnet manager only runs on the switches with the highest firmware versions:
For example, if Oracle Big Data Appliance has the highest firmware version, then make its spine switch the master and its gateway switches the failover. On the racks that have lower firmware versions than the Oracle Big Data Appliance InfiniBand switches (both spine and gateway), disable the subnet manager.
-
Log into the Exadata master switch as
root. -
Verify that the Oracle Big Data Appliance switches are visible:
# ibswitches
-
Disable the subnet manager on Oracle Exadata Database Machine:
# disablesm
-
Log in to each additional lower-version InfiniBand switch and verify that the master switch changes to the Oracle Big Data Appliance spine switch:
# getmaster
-
Disable subnet manager on each lower-version InfiniBand switch:
# disablesm
-
-
Revalidate the InfiniBand network. See "Validating a Multirack InfiniBand Network."
7.7.2 Validating a Multirack InfiniBand Network
After cabling the racks together, verify that the InfiniBand network is operating correctly.
Note:
The bdacheckib utility validates connections among Oracle Big Data Appliance and Oracle Exadata Database Machine racks. It does not check other types of racks, such as Oracle Exalogic Elastic Cloud Machine.
To validate the InfiniBand network:
-
Generate a topology file for the network:
# bdacheckib -g
The name of the generated file is sample-multi-rack.json
-
Open sample-multi-rack.json in a text editor.
-
Reorder the elements to start with the rack at the left and end with the rack on the right. Save the file.
-
Validate the configuration:
# bdacheckib sample-multi-rack.json Verifing rack #1 leaf: bda1sw-ib2 LINK ... to rack2 UP LINK ... to rack2 UP LINK ... to rack1 UP LINK ... to rack2 UP LINK ... to rack3 UP LINK ... to rack3 UP LINK ... to rack1 UP LINK ... to rack1 UP leaf: bda1sw-ib3 LINK ... to rack2 UP LINK ... to rack2 UP LINK ... to rack1 UP . . .
Example 7-3 shows a sample configuration file.
Example 7-3 Multirack Configuration File sample-multi-rack.json
# cat sample-multi-rack.json
# This json multirack spec is generated. The array elements are sorted
# alphabetically. A proper arranged json spec representing racks from left to right
# can be used as input mr.json to bdacheckib (bdacheckib mr.json)
# Commas separating rack elements are optional.
[{"SPINE_NAME": "bda1sw-ib1", "LEAF1_NAME": "bda1sw-ib2", "LEAF2_NAME": "bda1sw-ib3"}
{"SPINE_NAME": "bda2sw-ib1", "LEAF1_NAME": "bda2sw-ib2", "LEAF2_NAME": "bda2sw-ib3"}
{"SPINE_NAME": "exa1sw-ib1", "LEAF1_NAME": "exa1sw-ib2", "LEAF2_NAME": "exa01sw-ib3"}]