This chapter describes how to maintain the Recovery Appliance rack components. It contains the following topics:
See Also:
When maintaining the Recovery Appliance hardware, observe the following precautions:
WARNING:
Do not touch the parts of this product that use high-voltage power. Touching them might result in serious injury.
Caution:
Do not power off Recovery Appliance unless there is an emergency. In that case, follow "Emergency Power-Off Procedure".
Keep the front and rear cabinet doors closed. Failure to do so might cause system failure or result in damage to the hardware components.
Keep the top, front, and back of the cabinets clear to allow proper airflow and to prevent the components from overheating.
Use the following command to determine the model of a compute server or a storage server:
dmidecode -s system-product-name
This section includes the following topics:
In an emergency, halt power to Recovery Appliance immediately. The following emergencies might require powering off Recovery Appliance:
Natural disasters, such as earthquake, flood, hurricane, tornado, or cyclone
Abnormal noise, smell, or smoke coming from the system
Threat to human safety
In an emergency, do one of the following:
Turn off power at the circuit breaker.
Pull the emergency power-off switch in the computer room.
After the emergency, contact Oracle Support Services about restoring power to the system.
You can use the emergency power-off (EPO) switch to remove power from Recovery Appliance.
EPO switches are required when computer equipment contains batteries capable of supplying more than 750 volt-amperes for more than five minutes. Systems that have these batteries include internal EPO hardware for connecting to a site EPO switch or relay.
Under normal, nonemergency conditions, you can power down the software services and hardware gracefully.
Stop all software services before shutting down the rack components.
You must stop the Recovery Appliance services, Oracle Database File System, Oracle Database, and the cluster services.
To stop the Recovery Appliance services:
Log in as oracle to either Recovery Appliance compute server.
Open a SQL connection to Oracle Database as the rasys user:
$ sqlplus rasys
Check the status of the services:
SQL> SELECT state FROM ra_server;
STATE
------------
ON
Shut down Recovery Appliance services:
SQL> exec dbms_ra.shutdown;
Disconnect from Oracle Database:
SQL> exit
If Oracle Secure Backup is configured:
Switch to the root user.
Check the current status of Oracle Secure Backup:
# $GRID_HOME/bin/crsctl status res osbadmin
NAME=osbadmin
TYPE=cluster_resource
TARGET=ONLINE
STATE=ONLINE on example01adm04
If Oracle Secure Backup is online, then stop it:
# $GRID_HOME/bin/crsctl stop res osbadmin
Switch back to the oracle user.
Check the status of the Oracle Database File System (DBFS) mounts:
$ $GRID_HOME/bin/crsctl status res ob_dbfs rep_dbfs
NAME=ob_dbfs
TYPE=local_resource
TARGET=ONLINE , ONLINE
STATE=ONLINE on radb07, ONLINE on radb08
NAME=rep_dbfs
TYPE=local_resource
TARGET=ONLINE , ONLINE
STATE=ONLINE on radb07, ONLINE on radb08
Stop DBFS:
$ $GRID_HOME/bin/crsctl stop res ob_dbfs rep_dbfs
CRS-2673: Attempting to stop 'ob_dbfs' on 'radb08'
CRS-2673: Attempting to stop 'rep_dbfs' on 'radb07'
CRS-2673: Attempting to stop 'ob_dbfs' on 'radb07'
CRS-2673: Attempting to stop 'rep_dbfs' on 'radb08'
CRS-2677: Stop of 'rep_dbfs' on 'radb07' succeeded
CRS-2677: Stop of 'ob_dbfs' on 'radb07' succeeded
CRS-2677: Stop of 'rep_dbfs' on 'radb08' succeeded
CRS-2677: Stop of 'ob_dbfs' on 'radb08' succeeded
Verify that the DBFS mounts are offline:
$ $GRID_HOME/bin/crsctl status res ob_dbfs rep_dbfs
NAME=ob_dbfs
TYPE=local_resource
TARGET=OFFLINE, OFFLINE
STATE=OFFLINE, OFFLINE
NAME=rep_dbfs
TYPE=local_resource
TARGET=OFFLINE, OFFLINE
STATE=OFFLINE, OFFLINE
Check the status of Oracle Database:
$ srvctl status database -d zdlra5
Instance zdlra51 is running on node radb07
Instance zdlra52 is running on node radb08
Stop Oracle Database:
$ srvctl stop database -d zdlra5
Verify that Oracle Database is stopped:
$ srvctl status database -d zdlra5
Instance zdlra51 is not running on node radb07
Instance zdlra52 is not running on node radb08
Switch to the root user.
Stop the Oracle Clusterware stack on all nodes in the cluster:
# $GRID_HOME/bin/crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on 'zdlradb07'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on
'zdlradb07'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'zdlradb07'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'zdlradb07'
.
.
.
#
If the command fails, reenter it with the -f option.
On each compute server, run the following command to stop the Oracle Cluster Ready Services (CRS):
# $GRID_HOME/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'radb08'
CRS-2673: Attempting to stop 'ora.crf' on 'radb08'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'radb08'
.
.
.
CRS-2677: Stop of 'ora.crf' on 'radb08' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'radb08' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'radb08' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Shut down or reboot the hardware as required, in the following order:
Compute servers
Storage servers
Rack and switches
Before powering down a server, stop the services running on it, as described in "Shutting Down Recovery Appliance".
To shut down a compute server or a storage server:
Example 12-1 Powering Off Recovery Appliance Using the dcli Utility
Stop Oracle Clusterware on all compute servers:
# GRID_HOME/grid/bin/crsctl stop cluster -all
Shut down the other compute server in the rack:
# dcli -l root -g ra-adm02 shutdown -h -y now
In the preceding command, ra01adm02 is the name of the second compute server.
Shut down all storage servers:
# dcli -l root -g cell_group shutdown -h -y now
In the preceding command, cell_group is a file that lists all storage servers.
Shut down the local compute server:
shutdown -h -y now
Power off the rack.
Use the dcli utility to run the shutdown command on multiple servers simultaneously. Do not run dcli from a server that will be powered off by the command.
The following example shuts down a group of storage servers listed in a file named cell_group:
# dcli -l root -g cell_group shutdown -h -y now
Example 12-1 shows the power off procedure for the rack when using the dcli utility to shut down multiple servers simultaneously. The commands run from a compute server.
Turn on the rack components first, then start the software services.
To power on the rack components, use one of the following methods:
Press the power button on the front of the component.
Log in to Oracle ILOM and apply power to the system. See "Powering On Servers Remotely".
Power on the rack components in this sequence:
Rack and switches
Allow the switches a few minutes to initialize, before you start the storage servers.
Storage servers
Allow five to 10 minutes for the storage servers to start all services. Ensure that they finish initializing before you start the compute servers.
Compute servers
When a compute server is powered on, the operating system and Oracle Clusterware start automatically. Oracle Clusterware then starts all resources that are configured to start automatically.
You can use the Oracle ILOM interface to power on the Recovery Appliance servers remotely. To access Oracle ILOM, use the web console, the command-line interface (CLI), intelligent platform management interface (IPMI), or simple network management protocol (SNMP).
For example, to apply power to server ra01cel01 using IPMI, you use its Oracle ILOM with a command like the following:
# ipmitool -H ra01cel01-ilom -U root chassis power on
IPMItool must be installed on the server where you use the command.
See Also:
Oracle Integrated Lights Out Manager (ILOM) 3.0 documentation for additional information about using Oracle ILOM to power on the servers:
Log in as root to a Recovery Appliance compute server.
Confirm that Oracle Cluster Ready Services (CRS) is running:
# $GRID_HOME/bin/crsctl status server
NAME=radb07
STATE=ONLINE
NAME=radb08
STATE=ONLINE
If CRS is not running, then start it:
# $GRID_HOME/bin/crsctl start cluster -all
CRS-2672: Attempting to start 'ora.evmd' on 'radb07'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'radb07'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'radb08'
.
.
.
#
Switch to the oracle user.
Verify that Oracle Database is running:
$ srvctl status database -d zdlra5
Instance zdlra51 is not running on node radb07
Instance zdlra52 is not running on node radb08
If Oracle Database is not running:
Start Oracle Database:
$ srvctl start database -d zdlra5
Confirm that Oracle Database is running:
$ srvctl status database -d zdlra5
Instance zdlra51 is running on node radb07
Instance zdlra52 is running on node radb08
Verify that the Database File System (DBFS) mounts are online:
$ $GRID_HOME/bin/crsctl status res ob_dbfs rep_dbfs
NAME=ob_dbfs
TYPE=local_resource
TARGET=OFFLINE, OFFLINE
STATE=OFFLINE, OFFLINE
NAME=rep_dbfs
TYPE=local_resource
TARGET=OFFLINE, OFFLINE
STATE=OFFLINE, OFFLINE
If DBFS is offline, then start it:
$ $GRID_HOME/bin/crsctl start res ob_dbfs rep_dbfs
CRS-2672: Attempting to start 'rep_dbfs' on 'zdlradb07'
CRS-2672: Attempting to start 'ob_dbfs' on 'zdlradb07'
CRS-2672: Attempting to start 'ob_dbfs' on 'zdlradb08'
.
.
.
$
Connect to Oracle Database as the RASYS user:
$ sqlplus rasys
Check the status of Recovery Appliance services:
SQL> SELECT state FROM ra_server;
STATE
------------
OFF
If the services are off, then start them:
SQL> exec dbms_ra.startup;
Confirm that the services are started:
SQL> /
STATE
------------
ON
The disk controllers in storage servers and compute servers have battery-backed write cache to accelerate write performance. If the battery charge capacity degrades, so that the battery cannot protect the cached data for a power loss of 48 hours or more, then the write cache is disabled and the disk controller switches to write-through mode. Write performance is reduced, but no data is lost.
The battery charge capacity degrades over time, and its life expectancy is inversely proportional to the operating temperature. Table 12-1 shows the worst case life expectancy of a battery in Recovery Appliance.
Table 12-1 Battery Life Expectancy
| Inlet Ambient Temperature | Battery Lifetime |
|---|---|
|
< 25 degrees Celsius (77 degrees Fahrenheit) |
3 years |
|
< 32 degrees Celsius (89.6 digresses Fahrenheit) |
2 years |
To monitor the battery change capacity in the compute servers:
# /opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -a0 | grep "Full Charge" -A5 | sort \ | grep Full -A1
The following is an example of the output from the command:
Full Charge Capacity: 1357 mAh Max Error: 2 %
You should proactively replace batteries that have a capacity less than 800 milliampere hour (mAh) and a maximum error less than 10%. Immediately replace any battery that has less than 674 mAh or a maximum error greater than 10%.
To monitor the battery temperature:
/opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -a0 | grep BatteryType; \ /opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -a0 | grep -i temper
The following is an example of the output from the command:
BatteryType: iBBU08 Temperature: 38 C Temperature : OK Over Temperature : No
If the battery temperature is greater than or equal to 55 degrees Celsius, then determine the cause, and correct the problem.
Note:
Storage servers generate an alert when the battery charge capacity is insufficient, the temperature is high, or the battery should be replaced.
Oracle replaces the failed batteries at no extra charge under these conditions:
The battery charge capacity in the disk controllers falls below the minimum threshold
The system is covered either by the Oracle Premier Support for Systems or occurs during the warranty period.
For customers with Premier Support for Systems, Oracle attempts to proactively replace the batteries in Recovery Appliance before the end of the estimated lifetime, on a best effort basis.
A power distribution unit (PDU) can be replaced while Recovery Appliance is online. The second PDU in the rack maintains the power to all components in the rack. PDU-A is on the left, and PDU-B is on the right, when viewing the rack from the rear.
Before replacing a PDU, review the following guidelines to ensure that you can perform the procedure safely and without disrupting availability:
Unlatching the InfiniBand cables while removing or inserting PDU-A might remove servers from the cluster and thus make the rack unavailable. Be careful when handling the InfiniBand cables, which are normally latched securely. Do not place excessive tension on the InfiniBand cables by pulling them.
Unhooking the wrong power feeds shuts down the rack. Trace the power cables that will be replaced from the PDU to the power source, and only unplug those feeds.
Allow time to unpack and repack the PDU replacement parts. Notice how the power cords are coiled in the packaging, so you can repack the failed unit the same way.
Removing the side panel decreases the time needed to replace the PDU. However, removing the side panel is optional.
Using a cordless drill or power screwdriver decreases the time needed to replace the PDU. Allow more time for the replacement if you use a hand wrench. A screwdriver requires Torx T30 and T25 bits.
You might need to remove the server cable arms to move the power cables. In that case, twist the plug connection and flex the cable arm connector, to avoid having to unclip the cable arm. If you must unclip the cable arm, then support the cables with one hand, remove the power cord, and then clip the cable arm. Do not leave the cable arm hanging.
When you remove the T30 screws from the L-bracket, do not remove the T25 screws or nuts that attach the PDU to the bracket, until the PDU is out of the rack.
To replace a PDU:
Restart the PDU monitor to identify the network settings:
Press the reset button for 20 seconds, until it starts to count down from 5 to 0. While it is counting, release the button, and then press it once.
Record the network settings, firmware version, and so on, displayed on the LCD screen as the monitor restarts.
If the PDU monitor is not working, then retrieve the network settings by connecting to the PDU over the network, or from the network administrator.
Turn off all PDU breakers.
Unplug the PDU power plugs from the AC outlets.
If the rack is on a raised floor, then move the power cords out through the floor cutout. You might need to maneuver the rack over the cutout first.
WARNING:
If the power cords use overhead routing, then put them in a location where they will not fall or hit anyone.
For replacing PDU-B when there is no side panel access, and the rack does not have an InfiniBand cable harness:
Note:
Do not unstrap any cables attached to the cable arms.
Unscrew the T25 screws holding the square cable arms to the rack.
Move the InfiniBand cables to the middle, out of the way.
Unplug all power cables that connect the servers and switches to the PDU. Keep the power cables together in group bundles.
Remove the T30 screws from the top and bottom of the L-bracket, and note where the screws are used.
Note where the PDU sits in the rack frame.
The PDU is typically an inch back from the rack frame, to allow access to the breaker switches.
Angle and maneuver the PDU out of the rack.
Hold the PDU or put it down, if there is enough room, while maneuvering the AC power cords through the rack. You might need to cut the cable ties that hold the AC cord flush with the bottom side of the PDU.
Pull the cords as near to the bottom or top of the rack as possible. There is more room between the servers to guide the outlet plug through the routing hole.
Remove the smaller Torx T25 screws, and loosen the nut on the top and bottom to remove the PDU from the L-bracket. You do not need to remove the nut.
Attach the L-bracket to the new PDU.
Put the new PDU next to the rack.
Route the AC cords through the rack to the outlets.
Note:
Do not cable-tie the AC cord to the new PDU.
Place the new PDU in the rack by angling and maneuvering it until the L-brackets rest on the top and bottom rails.
Line up the holes and slots so that the PDU is about an inch back from the rack frame.
Attach the power cords, using the labels on the cords as a guide. For example, G5-0 indicates PDU group 5 outlet 0 on the PDU.
Attach the InfiniBand cable holders, if you removed them in step 4. Oracle recommends that you first screw in the holders by hand to avoid stripping the screws.
Attach the AC power cords to the outlets.
Turn on the breakers.
Cable and program the PDU monitor for the network, as needed.
See Also:
Oracle Sun Rack II Power Distribution Units User's Guide for information about programming the PDU monitor at
The Oracle Integrated Lights Out Manager (Oracle ILOM) might become unresponsive. If this happens, then you must manually reset the Service Processor (SP) on Oracle ILOM.
The following procedures describe how to reset Oracle ILOM:
See Also:
Oracle Integrated Lights Out Manager (ILOM) 3.0 documentation at
http://docs.oracle.com/cd/E19860-01/E21549/bbgiedhj.html#z4000b491400243
If you cannot connect to Oracle ILOM using SSH, then log in to the remote console.
To reset Oracle ILOM using the remote console:
If you cannot connect to Oracle ILOM using either SSH or the remote console, then use IPMItool.
To reset Oracle ILOM using IPMItool:
You do not need to shut down a compute server in Recovery Appliance to repair the physical disks. No downtime of the rack is required; however, individual servers might require downtime, and you might need to take them out of the cluster temporarily.
An LSI MegaRAID SAS 9261-8i disk controller manages the disk drives in each compute server. The disks have a RAID-5 configuration. Each compute server has four disk drives. One virtual drive comprises the RAID set.
See Also:
"LED Status Descriptions" for information about the LEDs
"Parts for Compute Servers" for the repair procedures
Oracle recommends that you periodically verify the status of the compute server RAID devices. The impact is minimal. In contrast, the impact of corrective action varies depending on the specific issue uncovered, and can range from simple reconfiguration to an outage.
Log in to each compute server as root and perform the following procedure.
To verify the RAID status:
If a compute server is irretrievably damaged, then you must replace it and reimage the replacement server. During the reimaging procedure, the other compute servers in the cluster are available. When adding the new server to the cluster, you copy the software from a working compute server to the new server.
The following tasks describe how to reimage a compute server:
Open a support request with Oracle Support Services. The support engineer identifies the failed server and sends you a replacement. The support engineer also asks for the output from the imagehistory command, run from a working compute server. The output provides a link to the computeImageMaker file that was used to image the original compute server, and is used to restore the system.
You must remove the failed compute server from Oracle Real Application Clusters (Oracle RAC).
In these steps, working_server is a working compute server in the cluster, failed_server is the compute server being replaced, and replacement_server is the new server.
To remove a failed compute server from the Oracle RAC cluster:
Log in to working_server as the oracle user.
Disable the listener that runs on the failed server:
$ srvctl disable listener -n failed_server $ srvctl stop listener -n failed_server
Delete the Oracle home directory from the inventory:
$ cd $ORACLE_HOME/oui/bin $ ./runInstaller -updateNodeList ORACLE_HOME= \ /u01/app/oracle/product/12.1.0/dbhome_1 "CLUSTER_NODES=list_of_working_servers"
In the preceding command, list_of_working_servers is a list of the compute servers that are still working in the cluster, such as ra01db02, ra01db03, and so on.
Verify that the failed server was deleted—that is, unpinned—from the cluster:
$ olsnodes -s -t
ra01db01 Inactive Unpinned
ra01db02 Active Unpinned
Stop and delete the virtual IP (VIP) resources for the failed compute server:
# srvctl stop vip -i failed_server-vip PRCC-1016 : failed_server-vip.example.com was already stopped # srvctl remove vip -i failed_server-vip Please confirm that you intend to remove the VIPs failed_server-vip (y/[n]) y
Delete the compute server from the cluster:
# crsctl delete node -n failed_server CRS-4661: Node failed_server successfully deleted.
If you receive an error message similar to the following, then relocate the voting disks.
CRS-4662: Error while trying to delete node ra01db01. CRS-4000: Command Delete failed, or completed with errors.
To relocate the voting disks:
Determine the current location of the voting disks. The sample output shows that the current location is DBFS_DG.
# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ----------
1. ONLINE 123456789abab (o/192.168.73.102/DATA_CD_00_ra01cel07) [DBFS_DG]
2. ONLINE 123456789cdcd (o/192.168.73.103/DATA_CD_00_ra01cel08) [DBFS_DG]
3. ONLINE 123456789efef (o/192.168.73.100/DATA_CD_00_ra01cel05) [DBFS_DG]
Located 3 voting disk(s).
Move the voting disks to another disk group:
# ./crsctl replace votedisk +DATA
Successful addition of voting disk 2345667aabbdd.
...
CRS-4266: Voting file(s) successfully replaced
Return the voting disks to the original location. This example returns them to DBFS_DG:
# ./crsctl replace votedisk +DBFS_DG
Repeat the crsctl command to delete the server from the cluster.
Update the Oracle inventory:
$ cd $ORACLE_HOME/oui/bin $ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/12.1.0/grid \ "CLUSTER_NODES=list_of_working_servers" CRS=TRUE
Verify that the server was deleted successfully:
$ cluvfy stage -post nodedel -n failed_server -verbose Performing post-checks for node removal Checking CRS integrity... The Oracle clusterware is healthy on node "ra01db02" CRS integrity check passed Result: Node removal check passed Post-check for node removal was successful.
See Also:
Oracle Real Application Clusters Administration and Deployment Guide for information about deleting a compute server from a cluster
Use a USB flash drive to copy the image to the new compute server.
To prepare the USB flash drive for use:
Before you perform the following procedure, replace the failed compute server with the new server. See Expanding a Recovery Appliance Rack with Additional Storage Servers.
To load the image onto the replacement server:
Insert the USB flash drive into the USB port on the replacement server.
Log in to the console through the service processor to monitor progress.
Power on the compute server either by physically pressing the power button or by using Oracle ILOM.
If you replaced the motherboard:
Press F2 during BIOS
Select BIOS Setup
Set the USB flash drive first, and then the RAID controller.
Otherwise, press F8 during BIOS, select the one-time boot selection menu, and choose the USB flash drive.
Allow the system to start.
As the system starts, it detects the CELLUSBINSTALL media. The imaging process has two phases. Let both phases complete before proceeding to the next step.
The first phase of the imaging process identifies any BIOS or firmware that is out of date, and upgrades the components to the expected level for the image. If any components are upgraded or downgraded, then the system automatically restarts.
The second phase of the imaging process installs the factory image on the replacement compute server.
Remove the USB flash drive when the system prompts you.
Press Enter to power off the server.
The replacement compute server does not have a host names, IP addresses, DNS, or NTP settings. This task describes how to configure the replacement compute server.
The information must be the same on all compute servers in Recovery Appliance. You can obtain the IP addresses from the DNS. You should also have a copy of the Installation Template from the initial installation.
To configure the replacement compute server:
Note:
If the compute server does not use all network interfaces, then the configuration process stops with a warning that some network interfaces are disconnected. It prompts whether to retry the discovery process. Respond with yes or no, as appropriate for the environment.
If bonding is used for the ingest network, then it is now set in the default active-passive mode.
The initial installation of Recovery Appliance modified various files.
To modify the files on the replacement compute server:
Replicate the contents of the following files from a working compute server in the cluster:
Copy the /etc/security/limits.conf file.
Merge the contents of the /etc/hosts files.
Copy the /etc/oracle/cell/network-config/cellinit.ora file.
Update the IP address with the IP address of the BONDIB0 interface on the replacement compute server.
Copy the /etc/oracle/cell/network-config/cellip.ora file.
Configure additional network requirements, such as 10 GbE.
Copy the /etc/modprobe.conf file.
Copy the /etc/sysctl.conf file.
Restart the compute server, so the network changes take effect.
Set up the Oracle software owner on the replacement compute server by adding the user name to one or more groups. The owner is usually the oracle user.
Obtain the current group information from a working compute server:
# id oracle
uid=1000(oracle) gid=1001(oinstall) groups=1001(oinstall),1002(dba),1003(oper),1004(asmdba)
Use the groupadd command to add the group information to the replacement compute server. This example adds the groups identified in the previous step:
# groupadd –g 1001 oinstall # groupadd –g 1002 dba # groupadd –g 1003 oper # groupadd –g 1004 asmdba
Obtain the current user information from a working compute server:
# id oracle uid=1000(oracle) gid=1001(oinstall) \ groups=1001(oinstall),1002(dba),1003(oper),1004(asmdba)
Add the user information to the replacement compute server. This example adds the group IDs from the previous step to the oracle user ID:
# useradd -u 1000 -g 1001 -G 1001,1002,1003,1004 -m -d /home/oracle -s \ /bin/bash oracle
Create the ORACLE_BASE and Grid Infrastructure directories. This example creates /u01/app/oracle and /u01/app/12.1.0/grid:
# mkdir -p /u01/app/oracle # mkdir -p /u01/app/12.1.0/grid # chown -R oracle:oinstall /u01/app
Change the ownership of the cellip.ora and cellinit.ora files. The owner is typically oracle:dba.
# chown -R oracle:dba /etc/oracle/cell/network-config
Secure the restored compute server:
$ chmod u+x /opt/oracle.SupportTools/harden_passwords_reset_root_ssh $ /opt/oracle.SupportTools/harden_passwords_reset_root_ssh
The compute server restarts.
Log in as the root user. When you are prompted for a new password, set it to match the root password of the other compute servers.
Set the password for the Oracle software owner. The owner is typically oracle.
# passwd oracle
Set up SSH for the oracle account:
Change to the oracle account on the replacement compute server:
# su - oracle
Create the dcli group file on the replacement compute server, listing the servers in the Oracle cluster.
Run the setssh-Linux.sh script on the replacement compute server. This example runs the script interactively:
$ /opt/oracle.SupportTools/onecommand/setssh-Linux.sh -s
The script prompts for the oracle password on the servers. The -s option causes the script to run in silent mode.
Change to the oracle user on the replacement compute server:
# su - oracle
Verify SSH equivalency:
$ dcli -g dbs_group -l oracle date
Set up or copy any custom login scripts from the working compute server to the replacement compute server:
$ scp .bash* oracle@replacement_server:.
In the preceding command, replacement_server is the name of the new server, such as ra01db01.
Oracle periodically releases software patch bundles for Recovery Appliance. If the working compute server has a patch bundle that is later than the release of the computeImageMaker file, then you must apply the patch bundle to the replacement compute server.
To determine if a patch bundle was applied, use the imagehistory command. Compare information on the replacement compute server to information on the working compute server. If the working database has a later release, then apply the storage server patch bundle to the replacement compute server.
The following procedure describes how to clone the Oracle Grid infrastructure onto the replacement compute server. In the commands, working_server is a working compute server, and replacement_server is the replacement compute server.
To clone the Oracle Grid infrastructure:
Log in as root to a working compute server in the cluster.
Verify the hardware and operating system installation using the cluster verification utility (cluvfy):
$ cluvfy stage -post hwos -n replacement_server,working_server –verbose
The phrase Post-check for hardware and operating system setup was successful should appear at the end of the report.
Verify peer compatibility:
$ cluvfy comp peer -refnode working_server -n replacement_server \ -orainv oinstall -osdba dba | grep -B 3 -A 2 mismatched
The following is an example of the output:
Compatibility check: Available memory [reference node: ra01db02] Node Name Status Ref. node status Comment ------------ ----------------------- ----------------------- ---------- ra01db01 31.02GB (3.2527572E7KB) 29.26GB (3.0681252E7KB) mismatched Available memory check failed Compatibility check: Free disk space for "/tmp" [reference node: ra01db02] Node Name Status Ref. node status Comment ------------ ----------------------- ---------------------- ---------- ra01db01 55.52GB (5.8217472E7KB) 51.82GB (5.4340608E7KB) mismatched Free disk space check failed
If the only failed components are related to the physical memory, swap space, and disk space, then it is safe for you to continue.
Perform the requisite checks for adding the server:
Ensure that the GRID_HOME/network/admin/samples directory has permissions set to 750.
Validate the addition of the compute server:
$ cluvfy stage -ignorePrereq -pre nodeadd -n replacement_server \
-fixup -fixupdir /home/oracle/fixup.d
If the only failed component is related to swap space, then it is safe for you to continue.
You might get an error about a voting disk similar to the following:
ERROR:
PRVF-5449 : Check of Voting Disk location "o/192.168.73.102/ \
DATA_CD_00_ra01cel07(o/192.168.73.102/DATA_CD_00_ra01cel07)" \
failed on the following nodes:
Check failed on nodes:
ra01db01
ra01db01:No such file or directory
...
PRVF-5431 : Oracle Cluster Voting Disk configuration check failed
If this error occurs, then use the -ignorePrereq option when running the addnode script in the next step.
Add the replacement compute server to the cluster:
$ cd /u01/app/12.1.0/grid/addnode/
$ ./addnode.sh -silent "CLUSTER_NEW_NODES={replacement_server}" \
"CLUSTER_NEW_VIRTUAL_HOSTNAMES={replacement_server-vip}"[-ignorePrereq]
The addnode script causes Oracle Universal Installer to copy the Oracle Clusterware software to the replacement compute server. A message like the following is displayed:
WARNING: A new inventory has been created on one or more nodes in this session. However, it has not yet been registered as the central inventory of this system. To register the new inventory please run the script at '/u01/app/oraInventory/orainstRoot.sh' with root privileges on nodes 'ra01db01'. If you do not register the inventory, you may not be able to update or patch the products you installed. The following configuration scripts need to be executed as the "root" user in each cluster node: /u01/app/oraInventory/orainstRoot.sh #On nodes ra01db01 /u01/app/12.1.0/grid/root.sh #On nodes ra01db01
Run the configuration scripts:
Open a terminal window.
Log in as the root user.
Run the scripts on each cluster server.
After the scripts are run, the following message is displayed:
The Cluster Node Addition of /u01/app/12.1.0/grid was successful. Please check '/tmp/silentInstall.log' for more details.
Run the orainstRoot.sh and root.sh scripts:
# /u01/app/oraInventory/orainstRoot.sh Creating the Oracle inventory pointer file (/etc/oraInst.loc) Changing permissions of /u01/app/oraInventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /u01/app/oraInventory to oinstall. The execution of the script is complete. # /u01/app/12.1.0/grid/root.sh
Check the log files in /u01/app/12.1.0/grid/install/ for the output of the root.sh script. The output file reports that the listener resource on the replaced compute server failed to start. This is an example of the expected output:
/u01/app/12.1.0/grid/bin/srvctl start listener -n ra01db01 \ ...Failed /u01/app/12.1.0/grid/perl/bin/perl \ -I/u01/app/12.1.0/grid/perl/lib \ -I/u01/app/12.1.0/grid/crs/install \ /u01/app/12.1.0/grid/crs/install/rootcrs.pl execution failed
Reenable the listener resource that you stopped in "Removing the Failed Compute Server from the Cluster".
# GRID_HOME/grid/bin/srvctl enable listener -l LISTENER \ -n replacement_server # GRID_HOME/grid/bin/srvctl start listener -l LISTENER \ -n replacement_server
See Also:
Oracle Real Application Clusters Administration and Deployment Guide for information about cloning
This section describes how to perform maintenance on the storage servers. It contains the following topics:
When performing maintenance on a storage server, you might need to power down or restart the server. Before shutting down a storage server, verify that taking a server offline does not impact Oracle ASM disk group and database availability. Continued database availability depends on the level of Oracle ASM redundancy used on the affected disk groups, and the current status of disks in other storage servers that have mirror copies of the same data.
Caution:
If a disk in a different cell fails while the cell undergoing maintenance is not completely back in service on the Recovery Appliance, a double disk failure can occur. If the Recovery Appliance is deployed with NORMAL redundancy for the DELTA disk group and if this disk failure is permanent, you will lose all backups on the Recovery Appliance.
Ensure that the cell undergoing maintenance is not offline for an extended period of time. Otherwise, a rebalance operation will occur and this will cause issues because of insufficient space for the operation to complete. By default, the rebalance operation begins 24 hours after the cell goes offline.
To power down a storage server:
See Also:
My Oracle Support Doc ID 1188080.1, "Steps to shut down or reboot an Exadata storage cell without affecting ASM."
You might need to use the diagnostics ISO to access a storage server that fails to restart normally. After starting the server, you can copy files from the ISO to the server, replacing the corrupt files.
The ISO is located on all Recovery Appliance servers at /opt/oracle.SupportTools/diagnostics.iso.
Caution:
Use the diagnostics ISO only after other restart methods, such as using the USB drive, have failed. Contact Oracle Support for advise and guidance before starting this procedure.
To use the diagnostics ISO:
This section contains the following topics:
See Also:
Oracle Maximum Availability Architecture (MAA) website at http://www.oracle.com/goto/maa for additional information about maintenance best practices
The first two disks of storage servers are system disks. Storage server software system software resides on a portion of each of the system disks. These portions on both system disks are referred to as the system area. The nonsystem area of the system disks, referred to as data partitions, is used for normal data storage. All other disks in a storage server are called data disks.
You can monitor a physical disk by checking its attributes with the CellCLI LIST PHYSICALDISK command. For example, a physical disk with a status of failed or warning - predictive failure is having problems and probably must be replaced. The disk firmware maintains the error counters, and marks a drive with Predictive Failure when internal thresholds are exceeded. The drive, not the server software, determines if it needs replacement.
The following list identifies the storage server physical disk statuses.
Physical Disk Status for Storage Servers
Oracle ASM performs bad extent repair for read errors caused by hardware errors. The disks stay online, and no alerts are sent.
When a disk fails:
The Oracle ASM disks associated with it are dropped automatically with the FORCE option, and then an Oracle ASM rebalance restores data redundancy.
The blue LED and the amber LED are turned on for the drive, indicating that disk replacement can proceed. The drive LED stays on solid. See "LED Status Descriptions" for information about LED status lights during predictive failure and poor performance.
The server generates an alert, which includes specific instructions for replacing the disk. If you configured the system for alert notifications, then the alert is sent by email to the designated address.
When a disk has a faulty status:
The Oracle ASM disks associated with the grid disks on the physical drive are dropped automatically.
An Oracle ASM rebalance relocates the data from the predictively failed disk to other disks.
The blue LED is turned on for the drive, indicating that disk replacement can proceed.
When Oracle ASM gets a read error on a physically-addressed metadata block, it does not have mirroring for the blocks:
Oracle ASM takes the disk offline.
Oracle ASM drops the disk with the FORCE option.
The storage server software sends an alert stating that the disk can be replaced.
ASR automatically identifies and removes a poorly performing disk from the active configuration. Recovery Appliance then runs a set of performance tests. When CELLSRV detects poor disk performance, the cell disk status changes to normal - confinedOnline, and the physical disk status changes to warning - confinedOnline. Table 12-2 describes the conditions that trigger disk confinement:
Table 12-2 Alerts Indicating Poor Disk Performance
| Alert Code | Cause |
|---|---|
|
CD_PERF_HANG |
Disk stopped responding |
|
CD_PERF_SLOW_ABS |
High service time threshold (slow disk) |
|
CD_PERF_SLOW_RLTV |
High relative service time threshold (slow disk) |
|
CD_PERF_SLOW_LAT_WT |
High latency on writes |
|
CD_PERF_SLOW_LAT_RD |
High latency on reads |
|
CD_PERF_SLOW_LAT_RW |
High latency on reads and writes |
|
CD_PERF_SLOW_LAT_ERR |
Frequent very high absolute latency on individual I/Os |
|
CD_PERF_IOERR |
I/O errors |
If the problem is temporary and the disk passes the tests, then it is brought back into the configuration. If the disk does not pass the tests, then it is marked poor performance, and ASR submits a service request to replace the disk. If possible, Oracle ASM takes the grid disks offline for testing. Otherwise, the cell disk status stays at normal - confinedOnline until the disks can be taken offline safely. See "Removing an Underperforming Physical Disk".
The disk status change is recorded in the server alert history:
MESSAGE ID date_time info "Hard disk entered confinement status. The LUN n_m changed status to warning - confinedOnline. CellDisk changed status to normal - confinedOnline. Status: WARNING - CONFINEDONLINE Manufacturer: name Model Number: model Size: size Serial Number: serial_number Firmware: fw_release Slot Number: m Cell Disk: cell_disk_name Grid Disk: grid disk 1, grid disk 2 . . . Reason for confinement: threshold for service time exceeded"
These messages are entered in the storage cell alert log:
CDHS: Mark cd health state change cell_disk_name with newState HEALTH_BAD_
ONLINE pending HEALTH_BAD_ONLINE ongoing INVALID cur HEALTH_GOOD
Celldisk entering CONFINE ACTIVE state with cause CD_PERF_SLOW_ABS activeForced: 0
inactiveForced: 0 trigger HistoryFail: 0, forceTestOutcome: 0 testFail: 0
global conf related state: numHDsConf: 1 numFDsConf: 0 numHDsHung: 0 numFDsHung: 0
.
.
.
After you replace the physical disk, you must re-create the grid disks and cell disks that existed on the previous disk in that slot. If those grid disks were part of an Oracle ASM group, then add them back to the disk group, and rebalance the data, based on the disk group redundancy and the ASM_POWER_LIMIT parameter.
Oracle ASM rebalance occurs when dropping or adding a disk. To check the status of the rebalance:
Did the rebalance operation run successfully?
Check the Oracle ASM alert logs.
Is the rebalance operation currently running?
Check the GV$ASM_OPERATION view.
Did the rebalance operation fail?
Check the V$ASM_OPERATION.ERROR view.
You can perform rebalance operations from multiple disk groups on different Oracle ASM instances in the same cluster, if the failed physical disk contained ASM disks from multiple disk groups. One Oracle ASM instance can run one rebalance operation at a time. If all Oracle ASM instances are busy, then the rebalance operations are queued.
The hard disk controller on each storage server periodically performs a discharge and charge of the controller battery. During the operation, the write cache policy changes from write-back caching to write-through caching. Write-through cache mode is slower than write-back cache mode. However, write-back cache mode risks data loss if the storage server loses power or fails. The operation occurs every three months, for example, at 01:00 on the 17th day of January, April, July and October.
This example shows an informational alert that a storage server generates about the status of the caching mode for its logical drives:
HDD disk controller battery on disk contoller at adapter 0 is going into a learn cycle. This is a normal maintenance activity that occurs quarterly and runs for approximately 1 to 12 hours. The disk controller cache might go into WriteThrough caching mode during the learn cycle. Disk write throughput might be temporarily lower during this time. The message is informational only, no action is required.
Use the following commands to manage changes to the periodical write cache policy:
To change the start time for the learn cycle, use a command like the following example:
CellCLI> ALTER CELL bbuLearnCycleTime="2013-01-22T02:00:00-08:00"
The time reverts to the default learn cycle time after the cycle completes.
To see the time for the next learn cycle:
CellCLI> LIST CELL ATTRIBUTES bbuLearnCycleTime
To view the status of the battery:
# /opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -GetBbuStatus -a0 BBU status for Adapter: 0 BatteryType: iBBU08 Voltage: 3721 mV Current: 541 mA Temperature: 43 C BBU Firmware Status: Charging Status : Charging Voltage : OK Temperature : OK Learn Cycle Requested : No Learn Cycle Active : No Learn Cycle Status : OK Learn Cycle Timeout : No I2c Errors Detected : No Battery Pack Missing : No Battery Replacement required : No Remaining Capacity Low : Yes Periodic Learn Required : No Transparent Learn : No Battery state: GasGuageStatus: Fully Discharged : No Fully Charged : No Discharging : No Initialized : No Remaining Time Alarm : Yes Remaining Capacity Alarm: No Discharge Terminated : No Over Temperature : No Charging Terminated : No Over Charged : No Relative State of Charge: 7 % Charger System State: 1 Charger System Ctrl: 0 Charging current: 541 mA Absolute state of charge: 0 % Max Error: 0 % Exit Code: 0x00
A physical disk outage can reduce performance and data redundancy. Therefore, you should replace a failed disk with a new disk as soon as possible.
See Also:
Oracle Database Reference about the V$ASM_OPERATION view
You might need to replace a physical disk because its status is warning - predictive failure. This status indicates that the physical disk will fail soon, and you should replace it at the earliest opportunity.
If the drive fails before you replace it, then see "Replacing a Failed Physical Disk".
To replace a disk before it fails:
See Also:
Oracle Database Reference for information about the V$ASM_OPERATION view
A bad physical disk can degrade the performance of other good disks. You should remove the bad disk from the system.
To remove a physical disk after identifying the bad disk:
See Also:
Recovery Appliance rejects a physical disk when it is in the wrong slot.
Caution:
Reenabling a physical disk removes all data stored on it.
To reenable a rejected physical disk, replace hard_disk_name and hard_disk_id with the appropriate values in this command:
CellCLI> ALTER PHYSICALDISK hard_disk_name/hard_disk_id reenable force Physical disk hard_disk_name/hard_disk_id was reenabled.
This section describes how to perform maintenance on flash disks. It contains the following topics:
Recovery Appliance mirrors data across storage servers, and sends write operations to at least two storage servers. If a flash card in one storage server has problems, then Recovery Appliance services the read and write operations using the mirrored data in another storage server. Service is not interrupted.
If a flash card fails, then the storage server software identifies the data in the flash cache by reading the data from the surviving mirror. It then writes the data to the server with the failed flash card. When the failure occurs, the software saves the location of the data lost in the failed flash cache. Resilvering then replaces the lost data with the mirrored copy. During resilvering, the grid disk status is ACTIVE -- RESILVERING WORKING.
Each storage server has four PCIe cards. Each card has four flash disks (FDOMs) for a total of 16 flash disks. The four PCIe cards are located in PCI slot numbers 1, 2, 4, and 5.
To identify a failed flash disk, use the following command:
CellCLI> LIST PHYSICALDISK WHERE DISKTYPE=flashdisk AND STATUS=failed DETAIL
name: FLASH_5_3
diskType: FlashDisk
luns: 5_3
makeModel: "Sun Flash Accelerator F40 PCIe Card"
physicalFirmware: TI35
physicalInsertTime: 2012-07-13T15:40:59-07:00
physicalSerial: 5L002X4P
physicalSize: 93.13225793838501G
slotNumber: "PCI Slot: 5; FDOM: 3"
status: failed
The card name and slotNumber attributes show the PCI slot and the FDOM number.
When the server software detects a failure, it generates an alert that indicates that the flash disk, and the LUN on it, failed. The alert message includes the PCI slot number of the flash card and the exact FDOM number. These numbers uniquely identify the field replaceable unit (FRU). If you configured the system for alert notification, then the alert is sent to the designated address in an email message.
A flash disk outage can reduce performance and data redundancy. Replace the failed disk at the earliest opportunity. If the flash disk is used for flash cache, then the effective cache size for the server is reduced. If the flash disk is used for flash log, then the flash log is disabled on the disk, thus reducing the effective flash log size. If the flash disk is used for grid disks, then the Oracle ASM disks associated with them are automatically dropped with the FORCE option from the Oracle ASM disk group, and an Oracle ASM rebalance starts to restore the data redundancy.
See Also:
"Parts for Storage Servers" for part number information and a link to the service guide
Oracle Database Reference for information about the V$ASM_OPERATION view
Sun Flash Accelerator F80 PCIe Card User's Guide at
The following status indicators generate an alert. The alert includes specific instructions for replacing the flash disk. If you configured the system for alert notifications, then the alerts are sent by email message to the designated address.
One of the flash disks on the same Sun Flash Accelerator PCIe card failed or has a problem. For example, if FLASH5_3 fails, then FLASH5_0, FLASH5_1, and FLASH5_2 have peer failure status:
CellCLI> LIST PHYSICALDISK
36:0 L45F3A normal
36:1 L45WAE normal
36:2 L45WQW normal
.
.
.
FLASH_5_0 5L0034XM warning - peer failure
FLASH_5_1 5L0034JE warning - peer failure
FLASH_5_2 5L002WJH warning - peer failure
FLASH_5_3 5L002X4P failed
The flash disk will fail soon, and should be replaced at the earliest opportunity. If the flash disk is used for flash cache, then it continues to be used as flash cache. If the flash disk is used for grid disks, then the Oracle ASM disks associated with these grid disks are automatically dropped, and Oracle ASM rebalance relocates the data from the predictively failed disk to other disks.
When one flash disk has predictive failure status, then the data is copied. If the flash disk is used for write back flash cache, then the data is flushed from the flash disks to the grid disks.
The flash disk demonstrates extremely poor performance, and should be replaced at the earliest opportunity. If the flash disk is used for flash cache, then flash cache is dropped from this disk, thus reducing the effective flash cache size for the storage server. If the flash disk is used for grid disks, then the Oracle ASM disks associated with the grid disks on this flash disk are automatically dropped with the FORCE option, if possible. If DROP...FORCE cannot succeed because of offline partners, then the grid disks are dropped normally, and Oracle ASM rebalance relocates the data from the poor performance disk to the other disks.
The capacitors used to support data cache on the PCIe card failed, and the card should be replaced as soon as possible.
To identify a flash disk with a particular health status, use the LIST PHYSICALDISK command. This example queries for the warning - predictive failure status:
CellCLI> LIST PHYSICALDISK WHERE DISKTYPE=flashdisk AND STATUS= \ 'warning - predictive failure' DETAIL name: FLASH_5_3 diskType: FlashDisk luns: 5_3 makeModel: "Sun Flash Accelerator F40 PCIe Card" physicalFirmware: TI35 physicalInsertTime: 2012-07-13T15:40:59-07:00 physicalSerial: 5L002X4P physicalSize: 93.13225793838501G slotNumber: "PCI Slot: 1; FDOM: 2" status: warning - predictive failure
ASR automatically identifies and removes a poorly performing disk from the active configuration. Recovery Appliance then runs a set of performance tests. When CELLSRV detects poor disk performance, the cell disk status changes to normal - confinedOnline, and the physical disk status changes to warning - confinedOnline. Table 12-2 describes the conditions that trigger disk confinement. The conditions are the same for both physical and flash disks.
If the problem is temporary and the disk passes the tests, then it is brought back into the configuration. If the disk does not pass the tests, then it is marked poor performance, and ASR submits a service request to replace the disk. If possible, Oracle ASM takes the grid disks offline for testing. Otherwise, the cell disk status stays at normal - confinedOnline until the disks can be taken offline safely.
The disk status change is recorded in the server alert history:
MESSAGE ID date_time info "Hard disk entered confinement status. The LUN n_m changed status to warning - confinedOnline. CellDisk changed status to normal - confinedOnline. Status: WARNING - CONFINEDONLINE Manufacturer: name Model Number: model Size: size Serial Number: serial_number Firmware: fw_release Slot Number: m Cell Disk: cell_disk_name Grid Disk: grid disk 1, grid disk 2 ... Reason for confinement: threshold for service time exceeded"
These messages are entered in the storage cell alert log:
CDHS: Mark cd health state change cell_disk_name with newState HEALTH_BAD_
ONLINE pending HEALTH_BAD_ONLINE ongoing INVALID cur HEALTH_GOOD
Celldisk entering CONFINE ACTIVE state with cause CD_PERF_SLOW_ABS activeForced: 0
inactiveForced: 0 trigger HistoryFail: 0, forceTestOutcome: 0 testFail: 0
global conf related state: numHDsConf: 1 numFDsConf: 0 numHDsHung: 0 numFDsHung: 0
.
.
.
When the server software detects a predictive or peer failure in a flash disk used for write back flash cache, and only one FDOM is bad, then the server software resilvers the data on the bad FDOM, and flushes the data on the other three FDOMs. If there are valid grid disks, then the server software initiates an Oracle ASM rebalance of the disks. You cannot replace the bad disk until the tasks are completed and an alert indicates that the disk is ready.
An alert is sent when the Oracle ASM disks are dropped, and you can safely replace the flash disk. If the flash disk is used for write-back flash cache, then wait until none of the grid disks are cached by the flash disk.
Caution:
The PCIe cards are not hot pluggable; you must power down a storage server before replacing the flash disks or cards.
Before you perform the following procedure, shut down the server. See "Shutting Down a Storage Server".
To replace a failed flash disk:
See Also:
"Parts for Storage Servers" for part numbers and links to the service guide
Oracle Database Reference for information about the V$ASM_OPERATION view
Sun Flash Accelerator F80 PCIe Card User's Guide at
Caution:
The PCIe cards are not hot pluggable; you must power down a storage server before replacing the flash disks or cards.
Before you perform the following procedure, review the "When Is It Safe to Replace a Faulty Flash Disk?" topic.
To replace a faulty flash disk:
The system automatically uses the new flash disk, as follows:
If the flash disk is used for flash cache, then the effective cache size increases.
If the flash disk is used for grid disks, then the grid disks are re-created on the new flash disk.
If the grid disks were part of an Oracle ASM disk group, then they are added back to the disk group. The data is rebalanced on them, based on the disk group redundancy and the ASM_POWER_LIMIT parameter.
A bad flash disk can degrade the performance of other good flash disks. You should remove a bad flash disk. See "Identifying Underperforming Flash Disks".
To remove an underperforming flash drive:
If the flash disk is used for flash cache:
Ensure that data not synchronized with the disk (dirty data) is flushed from flash cache to the grid disks:
CellCLI> ALTER FLASHCACHE ... FLUSH
Disable the flash cache and create a new one. Do not include the bad flash disk when creating the flash cache.
CellCLI > DROP FLASHCACHE CellCLI > CREATE FLASHCACHE CELLDISK='fd1,fd2,fd3,fd4, ...'
If the flash disk is used for grid disks, then direct Oracle ASM to stop using the bad disk immediately:
SQL> ALTER DISKGROUP diskgroup_name DROP DISK asm_disk_name FORCE
Offline partners might cause the DROP command with the FORCE option to fail. If the previous command fails, do one of the following:
Restore Oracle ASM data redundancy by correcting the other server or disk failures. Then retry the DROP...FORCE command.
Direct Oracle ASM to rebalance the data off the bad disk:
SQL> ALTER DISKGROUP diskgroup_name DROP DISK asm_disk_name NOFORCE
Wait until the Oracle ASM disks associated with the bad flash disk are dropped successfully. The storage server software automatically sends an alert when it is safe to replace the flash disk.
Stop the services:
CellCLI> ALTER CELL SHUTDOWN SERVICES ALL
The preceding command checks if any disks are offline, in predictive failure status, or must be copied to its mirror. If Oracle ASM redundancy is intact, then the command takes the grid disks offline in Oracle ASM, and stops the services.
The following error indicates that stopping the services might cause redundancy problems and force a disk group to dismount:
Stopping the RS, CELLSRV, and MS services... The SHUTDOWN of ALL services was not successful. CELL-01548: Unable to shut down CELLSRV because disk group DATA, RECO may be forced to dismount due to reduced redundancy. Getting the state of CELLSRV services... running Getting the state of MS services... running Getting the state of RS services... running
If this error occurs, then restore Oracle ASM disk group redundancy. Retry the command when the status is normal for all disks.
Shut down the server. See "Shutting Down a Storage Server".
Remove the bad flash disk, and replace it with a new flash disk.
Power up the server. The services are started automatically. As part of the server startup, all grid disks are automatically online in Oracle ASM.
Add the new flash disk to flash cache:
CellCLI> DROP FLASHCACHE CellCLI> CREATE FLASHCACHE ALL
Verify that all grid disks are online:
CellCLI> LIST GRIDDISK ATTRIBUTES asmmodestatus
Wait until asmmodestatus shows ONLINE or UNUSED for all grid disks.
The flash disks are added as follows:
If the flash disk is used for grid disks, then the grid disks are re-created on the new flash disk.
If these grid disks were part of an Oracle ASM disk group and DROP...FORCE was used in Step 2, then they are added back to the disk group and the data is rebalanced on based on disk group redundancy and the ASM_POWER_LIMIT parameter.
If DROP...NOFORCE was used in Step 2, then you must manually add the grid disks back to the Oracle ASM disk group.
The disk controller battery backup unit (disk controller BBU) resides on a drive tray in the compute and storage servers. You can replace the disk controller BBU without downtime. The following procedures describe how to replace the disk controller BBU:
Note:
The procedures in this section do not apply to on-controller battery backup units. Replacement of those units require a system shutdown, because the system must be opened to access the controller card.
The following procedure describes how to replace a disk controller BBU on a compute server:
Each storage server maintains a copy of the software on the USB stick. Whenever the system configuration changes, the server updates the USB stick. You can use this USB stick to recover the server after a hardware replacement or a software failure. You restore the system when the system disks fail, the operating system has a corrupt file system, or the boot area is damaged. You can replace the disks, cards, CPU, memory, and so forth, and recover the server. You can insert the USB stick in a different server, and it will duplicate the old server.
If only one system disk fails, then use CellCLI commands to recover. In the rare event that both system disks fail simultaneously, then use the rescue functionality provided on the storage server CELLBOOT USB flash drive.
This section contains the following topics:
Before rescuing a storage server, you must take steps to protect the data that is stored on it. Those steps depend on whether the system is set up with normal redundancy or high redundancy.
If you are using normal redundancy, then the server has one mirror copy. The data could be irrecoverably lost, if that single mirror also fails during the rescue procedure.
Oracle recommends that you duplicate the mirror copy:
Make a complete backup of the data in the mirror copy.
Take the mirror copy server offline immediately, to prevent any new data changes to it before attempting a rescue.
This procedure ensures that all data residing on the grid disks on the failed server and its mirror copy is inaccessible during the rescue procedure.
The Oracle ASM disk repair timer has a default repair time of 3.6 hours. If you know that you cannot perform the rescue procedure within that time frame, then use the Oracle ASM rebalance procedure to rebalance the disks until you can do the rescue procedure.
See Also:
Oracle Exadata Storage Server Software User's Guide for information about resetting the timer
When the server has high redundancy disk groups, so that Oracle ASM has multiple mirror copies for all the grid disks of the failed server, then take the failed cell offline. After Oracle ASM times out, it automatically drops the grid disks on the failed server, and starts rebalancing the data using mirror copies.
The default time out is two hours. If the server rescue takes more than two hours, then you must re-create the grid disks on the rescued cells in Oracle ASM.
Note the following before using the rescue procedure:
The rescue procedure can rewrite some or all of the disks in the cell. If this happens, then you might lose all the content of those disks without the possibility of recovery. Ensure that you complete the appropriate preliminary steps before starting the rescue. See "If the Server Has Normal Redundancy" or "If the Server Has High Redundancy".
Use extreme caution when using this procedure, and pay attention to the prompts. Ideally, use the rescue procedure only with assistance from Oracle Support Services, and when you can afford to lose the data on some or all of the disks.
The rescue procedure does not destroy the contents of the data disks or the contents of the data partitions on the system disks, unless you explicitly choose to do so during the rescue procedure.
The rescue procedure restores the storage server software to the same release, including any patches that existed on the server during the last successful boot.
The rescue procedure does not restore these configuration settings:
Server configurations, such as alert configurations, SMTP information, administrator email address
ILOM configuration. However, ILOM configurations typically remain undamaged even when the server software fails.
The recovery procedure does restore these configuration settings:
The rescue procedure does not examine or reconstruct data disks or data partitions on the system disks. If there is data corruption on the grid disks, then do not use this rescue procedure. Instead, use the rescue procedures for Oracle Database and Oracle ASM.
After a successful rescue, you must reconfigure the server. If you want to preserve the data, then import the cell disks. Otherwise, you must create new cell disks and grid disks.
See Also:
Oracle Exadata Storage Server Software User's Guide for information on configuring cells, cell disks, and grid disks using the CellCLI utility
Caution:
Follow the rescue procedure with care to avoid data loss.
To rescue a server using the CELLBOOT USB flash drive:
Note:
Additional options might be available that allow you to enter a rescue mode Linux login shell with limited functionality. Then you can log in to the shell as the root user with the password supplied by Oracle Support Services, and manually run additional diagnostics and repairs on the server. For complete details, contact your Oracle Support Services representative.
After a successful rescue, you must configure the server. If the data partitions were preserved, then the cell disks are imported automatically during the rescue procedure.
See Also:
Oracle Exadata Storage Server Software User's Guide for information about IORM plans and metric thresholds