This chapter contains the following topics:
Note:
For ease of reading, the name "Oracle Exadata Rack" is used when information refers to both Oracle Exadata Database Machine and Oracle Exadata Storage Expansion Rack.
All procedures in this chapter are applicable to Oracle Exadata Database Machine and Oracle Exadata Storage Expansion Rack.
Power distribution units (PDUs) can be replaced while Oracle Exadata Rack is online. PDU-A is on the left, and PDU-B is on the right when viewing the rack from the rear.
Before replacing a PDU, the following guidelines should be reviewed to ensure the procedure is safe and does not disrupt availability:
Unlatching the InfiniBand cables while removing or inserting PDU-A may cause a loss of service due to nodes being removed from the cluster. This could cause the rack to be unavailable. Care should be taken when handling the InfiniBand cables, which are normally latched securely. Do not place excessive tension on the InfiniBand cables by pulling them.
Unhooking the wrong power feeds causes the rack to shut down. Trace the power cables running from the PDU that will be replaced to the power source, and only unplug those feeds.
Allow time to unpack and repack the PDU replacement parts. Note how the power cords are coiled in the packaging so the failed unit can be repacked the same way.
Removal of the side panel lessens the amount of time needed to replace the PDU. However, it is not necessary to remove the side panel to replace the PDU.
Use of a cordless drill or power screwdriver lessens the amount of time needed to replace the PDU. Allow more time for the replacement if using the hand wrench tool provided with the replacement rack. If using a screwdriver, then ensure that there are Torx T30 and T25 bits.
It may be necessary to remove the server cable arms to move the power cables. If that is the case, then twist the plug connection and flex the cable arm connector to avoid having to unclip the cable arm. If it is necessary to unclip the cable arm, then support the cables with one hand, remove the power cord, and then clip the cable arm. Do not leave the cable arm hanging.
When removing the T30 screws from the L-bracket, do not remove the T25 screws or nuts that attach the PDU to the bracket until the PDU is out of the rack.
The Integrated Lights Out Manager (ILOM) may become unresponsive. If this happens, then manual intervention is needed to reset the Service Processor (SP) on the ILOM. The following procedures describe how to reset the ILOM:
See Also:
Oracle Integrated Lights Out Manager (ILOM) Documentation at http://www.oracle.com/goto/ilom/docs
The following procedure describes how to reset the ILOM by connecting to it using SSH:
If it is not possible to connect to the ILOM using SSH, then log in to the ILOM remote console. The following procedure describes how to reset the ILOM using the remote console.
If you could not connect to the ILOM using SSH or the remote console, then log in to the local host or another host on the ILOM network, and use IPMItool. The following procedure describes how to reset the ILOM using IPMItool:
If you could not connect to the ILOM using SSH, the remote console, or IPMItool on the Oracle Exadata Database Machine X2-2 server or Exadata Storage Server, then press the SP reset pin. The following procedure describes how to reset the ILOM using the SP reset pin.
If you could not reset the ILOM on the Sun Fire X4800 Oracle Database Server or Sun Server X2-8 Oracle Database Server using SSH, the remote console or IPMItool, then remove the service processor (SP) from the server, and put it back.
Messages are displayed at the operating system level. These messages can be ignored. The fans will speed up because there is no fan control.
http://docs.oracle.com/cd/E19140-01/html/821-0282/gjfvy.html#scrolltoc
The following procedure describes how to configure the service processor (SP) and ILOM network settings:
Ensure you are using the correct link speed for Oracle Exadata Database Machine X7-2 and Oracle Exadata Database Machine X7-8 compute nodes.
Note:
You should configure the client network ports using Oracle Exadata Deployment Assistant (OEDA) during the deployment of the X7-2 and X7-8 Systems. See the OEDA Customer Network Configuration Page.The following steps may be necessary to configure an X7-2 or X7-8 client access port if the OEDA deployment was not performed or was performed incorrectly.
1 GbE network connections can be changed to 10 GbE connections.
This procedure applies to Oracle Exadata Database Machine models X6 and earlier.
When changing the connections, note the following:
To prevent a single point of failure for a bonded 10 GbE interface on Oracle Exadata Database Machine X2-8, use different ports on the Network Express Modules (NEMs) on the two cards, such as NEM0 NET1 and NEM1 NET0.
The 10 GbE interfaces are identified as eth4 and eth5 on Sun Fire X4170 M2 Oracle Database Servers, and as eth8 through eth15 on Sun Fire X4800 Oracle Database Servers. Oracle recommends using following on Oracle Exadata Database Machine X2-8:
BONDETH0 using interfaces eth9 and eth15
10 GbE NEM0(left)/NET1
10 GbE NEM1(right)/NET3
Oracle Clusterware is shut down, and the database server is restarted during the procedure.
This section contains the following tasks:
Verify the functionality of the ping
command before any changes using the following commands. By verifying the ping
command before any changes, you know what is the results should be after changing the interfaces. Similar commands can be used to check other servers that connect to Oracle Exadata Database Machine.
# grep "^nameserver" /etc/resolv.conf nameserver ip_address_1 nameserver ip_address_2 # ping -c 2 ip_address_1 PING ip_address_1 (ip_address_1) 56(84) bytes of data. 64 bytes from ip_address_1: icmp_seq=1 ttl=57 time=1.12 ms 64 bytes from ip_address_1: icmp_seq=2 ttl=57 time=1.05 ms --- ip_address_1 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1002ms rtt min/avg/max/mdev = 1.054/1.087/1.120/0.033 ms
If the test is not successful, showing 100% packet loss, then you should expect similar results when this same verification is run in "Task 4: Verify the 10 GbE Interfaces". If the test is successful, showing 0% packet loss, then you must see similar results after changing the 10 GbE connections.
The following procedure describes how to back up the current interface files:
The following procedure describes how to edit the ifcfg
configuration files:
The InfiniBand network connects the database servers and Exadata Storage Servers through the BONDIB0 interface to the InfiniBand switches in the rack. This section describes how to perform maintenance on the InfiniBand switches.
This section contains the following topics:
The procedure for backing up and restoring switch settings depends on the firmware on the switch. The 1.1.3-2 firmware has Integrated Lights Out Manager (ILOM) which provides backup and restore capability. The 1.0.1 firmware does not have ILOM. You can either upgrade to the 1.1.3-2 firmware and then use the procedure in "Backing Up Settings on a Switch with 2.1.3-4 Firmware", or you can manually perform the backup and restore of individual files.
This section contains the following topics:
See Also:
Oracle Integrated Lights Out Manager (ILOM) Documentation at http://www.oracle.com/goto/ilom/docs
The following procedure describes how to back up a switch with 2.1.3-4 firmware. The backup only needs to be done once after the switch has been initially configured with the right settings.
The following procedure describes how to back up a switch with 1.1.3-2 firmware. The backup only needs to be done once after the switch has been initially configured with the right settings.
The following procedure describes how to back up the settings on a switch with 1.0.1 firmware:
The following procedure describes how to restore the settings on a switch with 2.1.3-4 firmware:
The following procedure describes how to restore the settings on a switch with 1.1.3-2 firmware:
During the InfiniBand switch firmware upgrade to 2.2.2, the management server may raise a critical alert regarding the InfiniBand ports being disabled and in the "down" state. The alert is expected and will be cleared quickly, usually within six minutes. No action is required from users.
The following is an example of the alert sequence:
27_1 2016-06-16T07:43:48-07:00 critical "InfiniBand Port HCA-1:1 may require attention. State:Down, Physical State:Disabled."
27_2 2016-06-16T07:48:40-07:00 clear "InfiniBand Port HCA-1:1 status is OK."
InfiniBand bonding (BONDIB0 through BONDIB3) for database servers in Oracle Exadata Database Machine X3-8 Full Rack, and Oracle Exadata Database Machine X2-8 Full Rack use both ports on the same card for each of the four InfiniBand cards. If both ports on a single card are disabled, such as the ports fail or cables are removed, then the Oracle Clusterware stack halts. The following procedure describes how to isolate the card, and then restart Oracle Clusterware after correcting the InfiniBand card problem:
Isolate the failed InfiniBand card as follows:
Stop Oracle Clusterware as the root
user using the following command. If Oracle Clusterware is already down, then go to step 1.b.
# crsctl stop crs
Edit the cellinit.ora
file to remove the affected IP address.
Start Oracle Clusterware as the root
user using the following command:
# crsctl start crs
Correct the problem with the InfiniBand card.
Return the card to service as follows:
Stop Oracle Clusterware as the root
user using the following command:
# crsctl stop crs
Add the IP address to the cellinit.ora
file.
Start Oracle Clusterware as the root
user using the following command:
# crsctl start crs
This procedure describes how to replace a failed Sun Datacenter InfiniBand Switch 36 switch.
See Also:
Oracle Exadata Database Machine System Overview for information on cabling
Sun Datacenter InfiniBand Switch 36 Firmware Version 2.1 Documentation at http://docs.oracle.com/cd/E36265_01/index.html
See Sun Datacenter InfiniBand Switch 36 User's Guide at http://docs.oracle.com/cd/E19197-01/835-0784-05/gentextid-226.html
Oracle Exadata Database Machine Extending and Multi-Rack Cabling Guide for information about installing a Sun Datacenter InfiniBand Switch 36 switch
Oracle Exadata Database Machine Installation and Configuration Guide for more information about setting the subnet manager master
This procedure describes how to verify the InfiniBand network configuration.
See Also:
"Using the verify-topology Utility" for additional information about the verify-topology utility
Exadata Database Machine and Exadata Storage Server Supported Versions (My Oracle Support Doc ID 888828.1) for the current releases, and instructions about how to check the installed release
Oracle Exadata Database Machine includes the verify-topology utility. This utility can be used to identify the following network connection problems:
Missing InfiniBand cable
Missing InfiniBand connection
Incorrectly-seated cable
Cable connected to the wrong endpoint
The utility is available in the ibdiagtools
directory on all servers. To view the options for the verify-topology utility, use the following command:
./verify-topology -h [ DB Machine Infiniband Cabling Topology Verification Tool ] Usage: ./verify-topology [-v|--verbose] [-r|--reuse (cached maps)] [-m|--mapfile] [-ibn|--ibnetdiscover (specify location of ibnetdiscover output)] [-ibh|--ibhosts (specify location of ibhosts output)] [-ibs|--ibswitches (specify location of ibswitches output)] [-t|--topology [torus | fattree | halfrack] default is fattree]
The following is an example of the output when using the verify-topology utility. In the example, the error shows the cables are connected incorrectly. Both cables from the server are going to same InfiniBand switch. If the switch fails, then the server loses connectivity to InfiniBand network.
[ DB Machine Infiniband Cabling Topology Verification Tool ] Bad link:Switch 0x21283a8371a0a0 Port 11A - Sun Port 11B Reason : 2.5 Gbps Speed found. Could be 10 Gbps Possible cause : Cable isn't fully seated in Bad link:Switch 0x21283a89eba0a0 Port 11B - Sun Port 11A Reason : 2.5 Gbps Speed found. Could be 10 Gbps Possible cause : Cable isn't fully seated in Is every external switch connected to every internal switch..........[SUCCESS] Are any external switches connected to each other....................[SUCCESS] Are any hosts connected to spine switch..............................[SUCCESS] Check if all hosts have 2 CAs to different switches..................[ERROR] Node trnA-db01 has 1 endpoints. (Should be 2) Port 2 of this node is not connected to any switch --------fattree End Point Cabling verifation failed----- Leaf switch check: cardinality and even distribution.................[ERROR] Internal QDR Switch 0x21283a8371a0a0 has fewer than 4 compute nodes It has only 3 links belonging to compute nodes Check if each rack has an valid internal ring........................[SUCCESS]
If hardware maintenance has taken place with any component in the InfiniBand network, including replacing an InfiniBand HCA on a server, an InfiniBand switch, or an InfiniBand cable, or if operation of the InfiniBand network is suspected to be substandard, then verify the InfiniBand network is operating properly. The following procedure describes how to verify network operation:
Note:
The following procedure can be used any time the InfiniBand network is performing below expectations.
The Subnet Manager manages all operational characteristics of the InfiniBand network.
The operational characteristics include:
Discover the network topology
Assign a local identifier to all ports connected to the network
Calculate and program switch forwarding tables
Monitor changes in the fabric
The InfiniBand network can have more than one Subnet Manager, but only one Subnet Manager is active at a time. The active Subnet Manager is the Master Subnet Manager. The other Subnet Managers are the Standby Subnet Managers. If a Master Subnet Manager is shut down or fails, then a Standby Subnet Manager automatically becomes the Master Subnet Manager.
Each Subnet Manager has a priority that can be configured. When there is more than one Subnet Manager on the InfiniBand network, the Subnet Manager with the highest priority becomes the Master Subnet Manager. On Oracle Exadata Database Machine, the Subnet Managers on leaf switches should be configured as priority 5, and the Subnet Managers on spine switches should be configured as priority 8.
The following guidelines determine where Subnet Managers run on Oracle Exadata Database Machine:
Only run Subnet Managers on the InfiniBand switches specified for use in Oracle Exadata Database Machine, Oracle Exalogic Elastic Cloud, Oracle Big Data Appliance, and Oracle SuperCluster. Running Subnet Manager on any other device is not supported.
In Exadata-only configurations, when the InfiniBand network consists of one, two, or three racks cabled together, all switches should run Subnet Manager. The Master Subnet Manager should be run on a spine switch. If the network has only leaf switches, as in Oracle Exadata Database Machine Quarter Racks, then Subnet Manager Master runs on a leaf switch. When the InfiniBand network consists of four or more racks cabled together, then only spine switches should run Subnet Manager. The leaf switches should disable Subnet Manager.
In multirack configurations using different types of racks such as Exadata plus Exalogic, see My Oracle Support note 1682501.1.
See Also:
Sun Datacenter InfiniBand Switch 36 Firmware Version 2.1 Documentation at http://docs.oracle.com/cd/E36265_01/index.html
Oracle Exadata Database Machine Installation and Configuration Guide for information about setting Subnet Manager priority
The patchmgr utility is used to upgrade and downgrade the InfiniBand switches. The minimum switch firmware release that can use the patchmgr utility is release 1.3.3-2. Switch firmware is upgraded in a rolling manner. If a spine switch is present in the rack, then the spine switch is upgraded first. If a spine switch is not in the rack, then upgrade the switch that is running the subnet manager. If the subnet manager is not running on the switches, then perform the upgrade in any order.
Create a file that lists the InfiniBand switches to be updated, one switch per line. The following is an example of the file:
# cat ibswitches.lst myibswitch-01 myibswitch-02
To upgrade the InfiniBand switches, the switch firmware must be at release 1.3.3-2 or later. If the switch firmware is at an earlier release, then it is necessary to upgrade the firmware to release 1.3.3-2 using the instructions in My Oracle Support note 888828.1.
The only included downgrade is to release 2.1.6-2. Use the following commands to downgrade the firmware:
# ./patchmgr -ibswitches ibswitches.lst -downgrade -ibswitch_precheck [-force] [-unkey] # ./patchmgr -ibswitches ibswitches.lst -downgrade [-force] [-unkey]
Configuring InfiniBand partitioning is described in Implementing InfiniBand Partitioning across Oracle VM Oracle RAC Clusters on Oracle Exadata Database Machine. You can use InfiniBand partitioning with or without Oracle VM.
It may be necessary to change the InfiniBand network information on an existing Oracle Exadata Rack. The change may be needed to support a media server with multiple InfiniBand cards, or keep InfiniBand traffic on a distinct InfiniBand network such as having production, test and QA environments in the same rack.
All InfiniBand addresses must be in the same subnet, with a minimum subnet mask of 255.255.240.0 (or /20). The subnet mask chosen should be wide enough to accommodate possible future expansion of the Oracle Exadata Rack and InfiniBand network.
See Also:
Oracle Fusion Middleware Exalogic Enterprise Deployment Guide for information about configuring the SDP listener to connect to an Oracle Exalogic system
This procedure describes how to change the InfiniBand network information.
The procedure described in this section is based on the following assumptions:
All changes should be done as the ilom-admin
user using the Integrated Lights Out Manager (ILOM) interface.
Channel bonding is used for the client access network, such that the NET1 and NET2 interfaces are bonded to create BONDETH0. If channel bonding is not used, then replace BONDETH0 with NET1 in the procedure.
On Oracle Exadata Database Machine X4-2 and later hardware, as of Oracle Exadata System Software release 11.2.3.3.0, the name used for InfiniBand bonding changed from BONDIB0 to IB0 and IB1. These interfaces are changed the same way as the ifcfg-bondib0
interface.
As of Oracle Exadata System Software release 11.2.2.1.0, the names used for bonding changed. The names are BONDIB0 for the InfiniBand bonding and BONDETH0 for Ethernet bonding. In earlier releases, the names were BOND0 and BOND1, respectively.
The procedure uses the dcli
utility and the root
user. This significantly reduces the overall time to complete the procedure by running the commands in parallel on the database servers.
The dcli
utility requires SSH user-equivalence. If SSH user-equivalence is not configured, then some commands must be run explicitly on each database server.
The database group file, dbs_group
, must exist and be located in the /root
directory.
Ensure recent backups of the Oracle Cluster Registry (OCR) exist before changing the InfiniBand network information. OCR backups are located in the $Grid_home/cdata/cluster-name
directory, where Grid_home represents the location of your Oracle Grid Infrastructure software installation.
Starting with Oracle Grid Infrastructure 11g release 2 (11.2) , the private network configuration is stored in the Grid Plug and Play (gpnp) profile as well as the OCR. If the gpnp definition is not correct, then Oracle Clusterware CRS does not start. Take a backup of the gpnp profile on all nodes before changing the InfiniBand network information using the following commands:
$ cd $Grid_home/gpnp/hostname/profiles/peer/ $ cp -p profile.xml profile.xml.bk
See Also:
Oracle Exadata System Software User's Guide for additional information about the ipconf
command
Oracle Real Application Clusters Administration and Deployment Guide for additional information about Server Control Utility (SRVCTL) commands
Oracle Exadata Database Machine exachk or HealthCheck (My Oracle Support Doc ID 1070954.1)
Cluster Verification Utility (CLUVFY) FAQ (My Oracle Support Doc ID 316817.1)
There are three logical network interfaces configured on the database servers. The interfaces are the management network (eth0), the client access network (BOND1 or BONDETH0), and the private InfiniBand network (BOND0, BONDIB0, or IB0 and IB1).
Note:
The tasks in this section are for Oracle Exadata Database Servers that were configured prior to Oracle Exadata System Software release 11.2.3.2.1.
Starting with Oracle Exadata System Software release 11.2.2.3.0, connections that come in on the management network have their responses sent out on the management network interface, and connections on the client access network have their responses sent out on the client access network interface. The private InfiniBand network traffic is direct communication between the two endpoints, and no routers are involved in the communication.
For Oracle Exadata System Software releases earlier than release 11.2.2.3.0, the default route for outbound traffic not destined for an IP address on the management or private InfiniBand network is sent out using the client access network. The tasks in this section modify the routing such that traffic that comes in on the management network has the responses sent out on the management network. Similarly, traffic coming in on the client network has the responses sent out on the client network.
The tasks for network routing are for boot-time routing or real-time routing. The following apply to both types of routing:
These tasks are for database servers running a release earlier than Oracle Exadata System Software release 11.2.2.3.0.
The following sample IP addresses, netmasks, and gateways are used in the tasks:
Management network has IP address 10.149.49.12, netmask 255.255.252.0 (network 10.149.48.0/22), and gateway 10.149.48.1.
Client access network has IP address 10.204.78.15, netmask 255.255.255.0 (network 10.204.78.0/24), and gateway 10.1.78.1.
Note:
If the database server has additional networks configured, then files should be set up for the additional networks.
To configure network routing for boot-time routing, rule and routing files must be created for each database server. The rule and routing files must be located in the /etc/sysconfig/network-scripts
directory on each database server. For each Ethernet interface on the management network that has a configured IP address, the database server must have route-eth
n
and rule-eth
n
files. For each bonded Ethernet interface, the database server must have route-bondeth
n
and rule-bondeth
n
files. The following are examples of the content in the files:
File | Content |
---|---|
|
from 10.149.49.12 table 220 to 10.149.49.12 table 220 |
|
10.149.48.0/22 dev eth0 table 220 default via 10.149.48.1 dev eth0 table 220 |
|
from 10.204.78.0/24 table 210 to 10.204.78.0/24 table 210 |
|
10.204.78.0/24 dev bondeth0 table 210 default via 10.204.78.1 dev bondeth0 table 210 |
To configure the rules on a running system, use the /sbin/ip
command to create the same configuration that is performed at startup. The following commands result in the same configuration as the boot-time files:
/sbin/ip rule add from 10.149.49.12 table 220 /sbin/ip rule add to 10.149.49.12 table 220 /sbin/ip route add 10.149.48.0/22 dev eth0 table 220 /sbin/ip route add default via 10.149.48.1 dev eth0 table 220 /sbin/ip rule add from 10.204.78.0/24 table 210 /sbin/ip rule add to 10.204.78.0/24 table 210 /sbin/ip route add 10.204.78.0/24 dev bondeth0 table 210 /sbin/ip route add default via 10.204.78.1 dev bondeth0 table 210
Oracle recommends restarting the database server after running the commands to validate that the boot-time configuration is correct.
Use the following command to verify the network routing rules. The command output shows all the rules on the system.
# /sbin/ip rule list 0: from all lookup 255 32762: from all to 10.204.78.0/24 lookup 210 32763: from 10.204.78.0/24 lookup 210 32764: from all to 10.149.49.12 lookup 220 32765: from 10.149.49.12 lookup 220 32766: from all lookup main 32767: from all lookup default
The default routing table is not changed because two new routing tables are created during the preceding tasks. The new routing tables are used when the rules dictate their use. The following commands show how to check the default and new routing tables:
To check the default routing table. The following is an example of the command and output.
# /sbin/ip route list 10.204.78.0/24 dev bondeth0 proto kernel scope link src 10.204.78.15 192.168.10.0/24 dev bondib0 proto kernel scope link src 192.168.10.8 10.149.48.0/22 dev eth0 proto kernel scope link src 10.149.49.12 default via 10.149.52.1 dev bondeth0
To check that the supplemental tables include the table name with the command. The following is an example of the command and output.
# /sbin/ip route list table 220 10.149.48.0/22 dev eth0 scope link default via 10.149.48.1 dev eth0 root@dbhost# ip route list table 210 10.204.78.0/24 dev bondeth0 scope link default via 10.204.78.1 dev bondeth0
The network routing configuration can be removed to configure or troubleshoot Oracle Exadata Database Machine. Use the following commands to remove the rules and routes:
/sbin/ip route del default via 10.149.48.1 dev eth0 table 220 /sbin/ip route del 10.149.48.0/22 dev eth0 table 220 /sbin/ip rule del to 10.149.49.12 table 220 /sbin/ip rule del from 10.149.49.12 table 220 /sbin/ip route del default via 10.204.78.1 dev bondeth0 table 210 /sbin/ip route del 10.204.78.0/24 dev bondeth0 table 210 /sbin/ip rule del to 10.204.78.0/24 table 210 /sbin/ip rule del from 10.204.78.0/24 table 210
To return to the default network routing, delete the supplemental files from the /etc/sysconfig/network-scripts
directory, and then restart the server. The following is an example of the commands to remove the files, and restart the server:
/bin/rm -f /etc/sysconfig/network-scripts/rule-eth0 /bin/rm -f /etc/sysconfig/network-scripts/route-eth0 /bin/rm -f /etc/sysconfig/network-scripts/rule-bondeth0 /bin/rm -f /etc/sysconfig/network-scripts/route-bondeth0 reboot
The configuration settings for the Domain Name System (DNS) servers can be changed after initial setup.
All servers and switches in Oracle Exadata Database Machine should reference the same DNS servers. All domains that Oracle Exadata Database Machine references should be resolvable through each individual DNS server. The following topics contain the tasks and procedures for setting the Oracle Exadata Database Machine servers and switches to the same DNS servers. Oracle recommends changing the servers one at a time.
All configuration procedures should be done as the ilom-admin
user using the Integrated Lights Out Manager (ILOM) interface. Use one of the following procedures to change the DNS server, depending on firmware release:
This procedure describes how to change the DNS server address on the Cisco Ethernet switch.
This procedure describes how to change the DNS server address on the database servers.
This procedure describes how to change the DNS server on Oracle Exadata Storage Servers.
Note:
The NTP settings can also be set during this procedure.This procedure describes how to change the DNS server configuration using the KVM switch.
Note:
The KVM switch is only available in Oracle Exadata Database Machine X2-2 racks and Oracle Exadata Storage Expansion Racks with Oracle Exadata Storage Server with Sun Fire X4270 M2 Servers.
The KVM switch does not support NTP.
The configuration settings for the Network Time Protocol (NTP) servers can be changed after initial setup.
All servers and switches in Oracle Exadata Database Machine should reference the same NTP servers so that the servers are synchronized to the same time. The following topics contain the tasks and procedures for setting the Oracle Exadata Database Machine servers and switches to the same NTP server addresses. Oracle recommends changing the servers one at a time.
Task 2: Set the NTP Server Address on the Sun Datacenter InfiniBand Switch 36 Switch
Task 3: Set the NTP Server Address on the Cisco Ethernet Switch
Note:
These procedures assume that there is not a large time discrepancy between the two NTP servers. Use the command ntpq -p
to see if the system is healthy first before performing the NTP server update.
Up to two NTP servers can be configured for use with Oracle Exadata Database Machine.
You can set or change the Network Time Protocol (NTP) server address on the database server of Oracle Exadata Database Machine.
You can set or change the Network Time Protocol (NTP) server address on the Sun Datacenter InfiniBand Switch 36 switch.
Note:
Do not manually edit the files on the InfiniBand switches.
You can set or change the Network Time Protocol (NTP) server on the Cisco Ethernet switch.
You can set or change the Network Time Protocol (NTP) server on Exadata Storage Servers.
Note:
The DNS settings can also be set during this procedure.
WARNING:
If you do not complete the steps to ensure taking the grid disks offline will not impact Oracle ASM operation, then you can cause a database outage.
Related Topics
You can change the time zones on Oracle Exadata Database Machine after initial configuration and deployment.
The following components need to be modified when changing the time zone settings:
Exadata Storage Servers
Database servers
InfiniBand switches
Cisco Ethernet switch
Note:
Cell services and Oracle Clusterware services must be stopped before changing the time zone settings.
The following tasks describe how to change the time zone settings on the components:
The following procedure describes how to change the time zone setting on Exadata Storage Servers. Complete the setting changes to all storage servers before changing the settings on the database servers.
After modifying the time zone setting on the storage cells, you can change the time zone setting on the database servers.
Before starting this procedure, you should have already stopped the Oracle Clusterware stack and modified the time zone on the storage cells, as described in Task 1: Change Time Zone Settings on Exadata Storage Servers.
You can change the time zone setting on the Sun Datacenter InfiniBand Switch 36 switches.
You can change the time zone setting on the Cisco switch.
enable
command to enter privileged mode.configure terminal
command to begin configuration.Example 4-1 Setting the Time Zone on the Cisco Switch
The following is an example of setting the time zone to US Eastern time with summer time enabled:
$ telnet dmcisco-ip
Connected to switch name
Escape character is '^]'.
User Access Verification
Password:
dmcisco-ip>enable
Password:
dmcisco-ip#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
dmcisco-ip(config)#clock timezone EST -5
dmcisco-ip(config)#clock summer-time EDT recurring
dmcisco-ip(config)#end
dmcisco-ip#write memory
Building configuration...
Compressed configuration from 6421 bytes to 2041 bytes[OK]
dmcisco-ip#show clock
12:03:43.516 EDT Wed May 12 2012
dmcisco-ip#
This procedure describes how to configure the KVM (Keyboard, Video, Mouse) switch.
The switch is configured with all the connected components powered off.
Note:
The KVM switch is only available in Oracle Exadata Database Machine X2-2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers.
Pull the KVM tray out from the front of the rack, and open it using the handle.
Touch the touch pad.
Toggle between the host and KVM interface by pressing the Ctrl
key on the left side twice, similar to a double-click on a mouse.
Select Target Devices from the Unit View of the user interface. The number of sessions shown should be 22 for Oracle Exadata Database Machine Full Rack, 11 for Oracle Exadata Database Machine Half Rack, and 5 for Oracle Exadata Database Machine Quarter Rack. The number of sessions should be 18 for Oracle Exadata Storage Expansion Full Rack, 9 for Oracle Exadata Storage Expansion Half Rack, and 4 for Oracle Exadata Storage Expansion Quarter Rack.
Note:
If all sessions are not shown, then select IQ Adaptors from the Ports heading. Click the table heading, and then Port, to sort the sessions by port number. Note any missing items. The sessions are numbered from the bottom of the rack to the top.
Return to the Target Devices screen.
Select Local from User Accounts.
Click Admin under Users.
Set a password for the Admin account. Do not modify any other parameters.
Click Save.
Select Network from Appliance Settings. The Network Information screen appears.
Select IPv4 or IPv6.
Enter the values for Address, Subnet, Gateway, and the IP addresses of the DNS servers.
Click Save.
Connect the KVM LAN1 Ethernet port to the management network.
Verify the port has been configured correctly by checking the MAC address on the Network Information screen. The address should match the label next to the LAN1/LAN2 ports on the rear of the KVM switch.
Select Overview from Appliance.
Enter a name for the KVM switch.
Click Save.
Restart the KVM switch by selecting Reboot under Overview.
Examine the firmware version of the switch by selecting Versions from Appliance Settings. There are two version numbers shown, Application and Boot, as shown in the following:
Required version is: Application 1.2.10.15038 Boot 1.6.15020
Note:
The recommended firmware version is 1.2.8 or later.
If the firmware is 1.2.3 or earlier, then it can be upgraded from a network browser. If it is version 1.2.3 or later, then it can be upgraded from the local keyboard using a flash drive plugged in to the KVM USB port. To upgrade the firmware, do the following:
Select Overview from Appliance.
Select Upgrade Firmware from the Tools list.
Select the method to upgrade.
Click Upgrade.
Confirm the firmware version.
See Also:
Avocent Web site for information about KVM switch Management Information Base (MIB) at https://www.vertivco.com/en-us/support/software-download/it-management/avocent-mergepoint-unity-switches-software-downloads/
The following procedure describes how to configure the KVM switch to access the servers:
Note:
The KVM switch is only available in Oracle Exadata Database Machine X2-2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers.
trnacel03
has the prefix trna
, and is storage cell 3 from the bottom of the rack, and trnadb02
has the prefix trna
, and is database server 2 from the bottom of the rack.The following procedure describes how to access a server using the KVM switch:
Note:
The KVM switch is only available in Oracle Exadata Database Machine X2-2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers.