2 Maintaining Database Servers of Oracle Exadata Database Machine
This chapter contains the following topics:
-
Verifying and Modifying the Link Speed on the Client Network Ports for X7
-
Adding and Configuring an Extra Network Card on Oracle Exadata X6-2 and X7-2
-
Migrating a Bare Metal Oracle RAC Cluster to an Oracle RAC Cluster in Oracle VM
-
Managing Oracle VM Domains on Oracle Exadata Database Machine
-
Creating a Snapshot-Based Backup of Oracle Linux Database Server
-
Recovering Oracle Linux Database Servers Using a Snapshot-Based Backup
-
Changing Existing Elastic Configurations for Database Servers
Note:
For ease of reading, the name "Oracle Exadata Rack" is used when information refers to both Oracle Exadata Database Machine and Oracle Exadata Storage Expansion Rack.
2.1 Management Server on Database Servers
Management Server (MS) running on database servers provides monitoring, alerting, and other administrative capabilities. It also provides the DBMCLI command-line administration tool.
See Also:
-
"Management Server on Database Servers" in the Oracle Exadata Database Machine System Overview guide
2.2 Maintaining the Hard Disks of Oracle Database Servers
Repair of the hard disks does not require the database server in Oracle Exadata Database Machine to be shut down.
No downtime of the rack is required, however individual servers may require downtime, and be taken outside of the cluster temporarily.
2.2.1 Verifying the Database Server Configuration
The disks are configured RAID-5 configuration.
The disk drives in each database server are controlled by a MegaRAID SAS disk controller.
Table 2-1 Disk Configurations for Exadata Database Machine Two-Socket Systems
| Server Type | RAID Controller | Disk Configuration |
|---|---|---|
|
Oracle Exadata Database Machine X7-2 |
MegaRAID SAS 9361-16i |
4 disk drives in each database server |
|
Oracle Exadata Database Machine X6-2 |
MegaRAID SAS 9361-8i |
4 disk drives in each database server |
|
Oracle Exadata Database Machine X5-2 |
MegaRAID SAS 9361-8i |
4 disk drives in each database server |
|
Oracle Exadata Database Machine X4-2 |
MegaRAID SAS 9261-8i |
4 disk drives in each database server |
|
Oracle Exadata Database Machine X3-2 |
MegaRAID SAS 9261-8i |
4 disk drives in each database server |
|
Oracle Exadata Database Machine X2-2 |
MegaRAID SAS 9261-8i |
4 disk drives in each database server |
Table 2-2 Disk Configurations for Exadata Database Machine Eight-Socket Systems
| Server Type | RAID Controller | Disk Configuration |
|---|---|---|
|
Oracle Exadata Database Machine X7-8 |
MegaRAID SAS 9361-16i |
Two NVMe flash accelerator cards in each database server |
|
Oracle Exadata Database Machine X5-8 |
MegaRAID SAS 9361-8i |
8 disk drives in each database server with one virtual drive created across the RAID set |
|
Oracle Exadata Database Machine X4-8 |
MegaRAID SAS 9261-8i |
7 disk drives in each database server configured as one 6-disk RAID-5 with one global hot spare drive by default |
|
Oracle Exadata Database Machine X3-8 |
MegaRAID SAS 9261-8i |
8 disk drives in each database server with one virtual drive created across the RAID set |
Oracle recommends verifying the status of the database server RAID devices to avoid possible performance impact, or an outage. The impact of validating the RAID devices is minimal. The impact of corrective actions will vary depending on the specific issue uncovered, and may range from simple reconfiguration to an outage.
2.2.1.1 Verifying Disk Controller Configuration
Checking the Disk Controller
Use the following command to verify the database server disk controller configuration on all systems except Oracle Exadata Database Machine X7-8:
if [[ -d /proc/xen && ! -f /proc/xen/capabilities ]]
then
echo -e "\nThis check will not run in a user domain of a virtualized environment. Execute this check in the management domain.\n"
else
if [ -x /opt/MegaRAID/storcli/storcli64 ]
then
export CMD=/opt/MegaRAID/storcli/storcli64
else
export CMD=/opt/MegaRAID/MegaCli/MegaCli64
fi
RAW_OUTPUT=$($CMD AdpAllInfo -aALL -nolog | grep "Device Present" -A 8);
echo -e "The database server disk controller configuration found is:\n\n$RAW_OUTPUT";
fi;
Note:
This check is not applicable to Oracle Exadata Database Machine X7-8 database servers because they do not have any conventional disk drives.The following is an example of the output from the command for Oracle Exadata Database Machine 2-socket system (X2-2 or later) without the disk expansion kit.
Device Present
================
Virtual Drives : 1
Degraded : 0
Offline : 0
Physical Devices : 5
Disks : 4
Critical Disks : 0
Failed Disks : 0
The following is an example of the output from the command for Oracle Exadata Database Machine X4-8 Full Rack:
Device Present
================
Virtual Drives : 1
Degraded : 0
Offline : 0
Physical Devices : 8
Disks : 7
Critical Disks : 0
Failed Disks : 0
The following is an example of the output from the command for Oracle Exadata Database Machine X5-8 or X6-8 Full Rack:
Device Present
================
Virtual Drives : 1
Degraded : 0
Offline : 0
Physical Devices : 9
Disks : 8
Critical Disks : 0
Failed Disks : 0
For Oracle Exadata Database Machine X4-2, Oracle Exadata Database Machine X3-2, and Oracle Exadata Database Machine X2-2, the expected output is one virtual drive, none degraded or offline, five physical devices (one controller and four disks), four disks, and no critical or failed disks.
For Oracle Exadata Database Machine X3-8 Full Rack and Oracle Exadata Database Machine X2-8 Full Rack, the expected output is one virtual drive, none degraded or offline, 11 physical devices (one controller, two SAS2 expansion ports, and eight disks), eight disks, and no critical or failed disks.
If your output is different, then investigate and correct the problem. Degraded virtual drives usually indicate absent or failed physical disks. Critical disks should be replaced immediately to avoid the risk of data loss if the number of failed disks in the node exceed the count needed to sustain the operations of the system. Failed disks should also be replaced quickly.
Note:
If additional virtual drives or a hot spare are present, then it may be that the procedure to reclaim disks was not performed at deployment time or that a bare metal restore procedure was performed without using the dualboot=no qualifier. Contact Oracle Support Services and reference My Oracle Support note 1323309.1 for additional information and corrective steps.
When upgrading a database server that has a hot spare to Oracle Exadata System Software release 11.2.3.2.0 or later, the hot spare is removed, and added as an active drive to the RAID configuration. The database servers have the same availability in terms of RAID5 redundancy, and can survive the loss of one drive. When a drive failure happens, Oracle Auto Service Request (ASR) sends out a notification to replace the faulty drive at the earliest opportunity.
Checking the Disk Controller on Oracle Exadata Database Machine X7-8
Query mdstat to view the database server disk controller configuration on Oracle Exadata Database Machine X7-8:
[root@dbnode01adm01 ~]# cat /proc/mdstat
Personalities : [raid1]
md34 : active raid1 nvme3n1[1] nvme1n1[0]
3125613568 blocks super external:/md126/0 [2/2] [UU]
md24 : active raid1 nvme2n1[1] nvme0n1[0]
262144000 blocks super external:/md127/0 [2/2] [UU]
md25 : active raid1 nvme2n1[1] nvme0n1[0]
2863467520 blocks super external:/md127/1 [2/2] [UU]
md126 : inactive nvme3n1[1](S) nvme1n1[0](S)
6306 blocks super external:imsm
md127 : inactive nvme2n1[1](S) nvme0n1[0](S)
6306 blocks super external:imsm
unused devices: <none>
If the output is different, then investigate and correct the problem. Degraded virtual drives usually indicate absent or failed physical disks. Critical disks should be replaced immediately to avoid the risk of data loss if the number of failed disks in the node exceed the count needed to sustain the operations of the system. Failed disks should be replaced quickly.
2.2.1.2 Verifying Virtual Drive Configuration
To verify the virtual drive configuration, use the following command to verify the virtual drive configuration:
/opt/MegaRAID/MegaCli/MegaCli64 CfgDsply -aALL | grep "Virtual Drive:"; \ /opt/MegaRAID/MegaCli/MegaCli64 CfgDsply -aALL | grep "Number Of Drives"; \ /opt/MegaRAID/MegaCli/MegaCli64 CfgDsply -aALL | grep "^State"
The following is an example of the output for Oracle Exadata Database Machine X4-2, Oracle Exadata Database Machine X3-2 and Oracle Exadata Database Machine X2-2. The virtual device 0 should have four drives, and the state is Optimal.
Virtual Drive : 0 (Target Id: 0) Number Of Drives : 4 State : Optimal
The expected output for Oracle Exadata Database Machine X3-8 Full Rack and Oracle Exadata Database Machine X2-8 Full Rack displays the virtual device has eight drives and a state of optimal.
Note:
If a disk replacement was performed on a database server without using the dualboot=no option, then the database server may have three virtual devices. Contact Oracle Support and reference My Oracle Support note 1323309.1 for additional information and corrective steps.
2.2.1.3 Verifying Physical Drive Configuration
Check your system for critical, degraded, or failed disks.
To verify physical drive configuration, use the following command to verify the database server physical drive configuration:
/opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL | grep "Firmware state"
The following is an example of the output for Oracle Exadata Database Machine X4-2, Oracle Exadata Database Machine X3-2, and Oracle Exadata Database Machine X2-2:
Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firmware state: Online, Spun Up
The drives should show a state of Online, Spun Up. The order of the output is not important. The output for Oracle Exadata Database Machine X3-8 Full Rack or Oracle Exadata Database Machine X2-8 Full Rack should be eight lines of output showing a state of Online, Spun Up.
If your output is different, then investigate and correct the problem.
Degraded virtual drives usually indicate absent or failed physical disks. Critical disks should be replaced immediately to avoid the risk of data loss if the number of failed disks in the node exceed the count needed to sustain the operations of the system. Failed disks should be replaced quickly.
2.2.2 Monitoring a Database Server RAID Set Rebuilding
If a drive in a database server RAID set is replaced, then the progress of the RAID set rebuild should be monitored.
Use the following command on the database server that has the replaced disk. The command is run as the root user.
/opt/MegaRAID/MegaCli/MegaCli64 -pdrbld -showprog -physdrv \ [disk_enclosure:slot_number] -a0
In the preceding command, disk_enclosure and slot_number indicate the replacement disk identified by the MegaCli64 -PDList command. The following is an example of the output from the command:
Rebuild Progress on Device at Enclosure 252, Slot 2 Completed 41% in 13 Minutes.
2.2.3 Reclaiming a Hot Spare Drive After Upgrading to Oracle Exadata System Software Release 12.1.2.1.0 or Later
Oracle Exadata Database Machines upgraded to Oracle Exadata System Software release 12.1.2.1.0 or later that have a hot spare drive cannot use the reclaimdisks.sh script to reclaim the drive. The following procedure describes how to manually reclaim the drive:
Note:
During the procedure, the database server is restarted twice. The steps in the procedure assume that the Oracle Grid Infrastructure restart is disabled after the server restart.
The sample output shows Oracle Exadata Database Machine X2-2 database server with four disks. The enclosure identifier, slot number, and such may be different for your system.
2.2.4 Understanding Automated File Deletion Policy
Management Server (MS) includes a file deletion policy for the / (root) directory on the database servers that is triggered when file system utilization is high. Deletion of files is triggered when file utilization is 80 percent, and an alert is sent before the deletion begins. The alert includes the name of the directory, and space usage for the subdirectories. In particular, the deletion policy is as follows:
Files in the following directories are deleted using a policy based on the file modification time stamp.
-
/opt/oracle/dbserver/log -
/opt/oracle/dbserver/dbms/deploy/config/metrics -
/opt/oracle/dbserver/dbms/deploy/log
Files that are older than the number of days set by the metricHistoryDays attribute are deleted first, then successive deletions occur for earlier files, down to files with modification time stamps older than or equal to 10 minutes, or until file system utilization is less than 75 percent. The metricHistoryDays attribute applies to files in /opt/oracle/dbserver/dbms/deploy/config/metrics. For the other log and trace files, use the diagHistoryDays attribute.
Starting with Oracle Exadata System Software release 12.1.2.2.0, the maximum amount of space for ms-odl.trc and ms-odl.log files is 100 MB (twenty 5 MB files) for *.trc files and 100 MB (twenty 5 MB files) for *.log files. Previously, it was 50 MB (ten 5 MB files) for both *.trc and *.log files.
ms-odl generation files are renamed when they reach 5 MB, and the oldest are deleted when the files use up 100 MB of space.
2.3 Maintaining Flash Disks on Exadata Database Servers
Starting with Exadata Database Machine X7-8, the database servers contain flash devices instead of hard disks. These flash devices can be replaced without shutting down the server.
2.3.1 Monitoring the Status of Flash Disks
You can monitor the status of a flash disk on the Exadata Database Machine by checking its attributes with the DBMCLI LIST PHYSICALDISK command.
For example, a flash disk status equal to failed is probably having problems and needs to be replaced.
The Exadata Database Server flash disk statuses are as follows:
-
normal
-
normal - dropped for replacement
-
failed
-
failed - dropped for replacement
-
failed - rejected due to incorrect disk model
-
failed - rejected due to incorrect disk model - dropped for replacement
-
failed - rejected due to wrong slot
-
failed - rejected due to wrong slot - dropped for replacement
-
warning - peer failure
-
warning - predictive failure, write-through caching
-
warning - predictive failure
-
warning - predictive failure - dropped for replacement
-
warning - write-through caching
2.4 Adding Disk Expansion Kit to Database Servers
You can add a disk expansion kit to some Oracle Exadata Database Servers.
Note the following restrictions:
-
The disk expansion kit is supported on Oracle Exadata Database Machine X5-2, X6-2, and X7-2 systems only.
-
Oracle Exadata System Software release 12.1.2.3.0 or later is required.
2.5 Adding Memory Expansion Kit to Database Servers
Additional memory can be added to database servers. The following procedure describes how to add the memory:
Note:
-
Memory for Sun Server X4-2 Oracle Database Servers and Sun Server X3-2 Oracle Database Servers can be expanded to a maximum of 512 GB with the memory expansion kit.
-
Memory for Sun Fire X4170 Oracle Database Servers can be expanded to a maximum of 144 GB by removing the existing memory, and replacing it with three X2-2 Memory Expansion Kits.
-
Sun Fire X4170 M2 Oracle Database Servers ship from the factory with 96 GB of memory, with 12 of the 18 DIMM slots populated with 8 GB DIMMs. The optional X2-2 Memory Expansion Kit can be used to populate the remaining 6 empty slots with 16 GB DIMMs to bring the total memory to 192 GB (12 x 8 GB and 6 x 16GB).
The memory expansion kit is primarily for consolidation workloads where many databases are run on each database server. In this scenario, the CPU usage is often low while the memory usage is very high.
However, there is a downside to populating all the memory slots as the frequency of the memory DIMMs drop to 800 MHz from 1333 MHz. The performance effect of the slower memory appears as increased CPU utilization. The average measured increase in CPU utilization is typically between 5% and 10%. The increase varies greatly by workload. In test workloads, several workloads had almost zero increase, while one workload had as high as a 20% increase.
-
When adding memory to Oracle Exadata Database Machines running Oracle Linux, Oracle recommends updating the
/etc/security/limits.conffile with the following:oracle soft memlock 75% oracle hard memlock 75%
2.6 Adding and Configuring an Extra Network Card on Oracle Exadata X6-2 and X7-2
You can add an additional network card on Oracle Exadata X6-2 and X7-2 systems.
Prerequisites
Ensure you are using the correct link speed for Oracle Exadata Database Machine X7-2 and Oracle Exadata Database Machine X7-8 compute nodes. Complete the steps in Verifying and Modifying the Link Speed on the Client Network Ports for X7.
Oracle Exadata Database Machine X6-2
Oracle Exadata Database Machine X6-2 database server offers highly available copper 10G network on the motherboard, and an optical 10G network via a PCI card on slot 2. Oracle offers an additional Ethernet card for customers that require additional connectivity. The additional card provides either dual port 10GE copper connectivity (part number 7100488) or dual port 10GE optical connectivity (part number X1109A-Z). You install this card in PCIe slot 1 on the Oracle Exadata X6-2 database server.
After you install the card and connect it to the network, the Oracle Exadata System Software automatically recognizes the new card and configures the two ports as eth6 and eth7 interfaces on the X6-2 database server. You can use these additional ports to provide an additional client network, or to create a separate backup or data recovery network. On a database server that runs virtual machines, you could use this to isolate traffic from two virtual machines.
Oracle Exadata Database Machine X7-2
Oracle Exadata Database Machine X7-2 database server offers 2 copper (RJ45) or 2 optical (SFP28) network connections on the motherboard plus 2 optical (SFP28) network connections in PCIe card slot 1. Oracle offers an additional 4 copper (RJ45) 10G network connections for customers that require additional connectivity. The additional card is the Oracle Quad Port 10GBase-T card (part number 7111181). You install this card in PCIe slot 3 on the X7-2 database server.
After you install the card and connect it to the network, the Oracle Exadata System Software automatically recognizes the new card and configures the four ports as eth5 to eth8 interfaces on the X7-2 database server. You can use these additional ports to provide an additional client network, or to create a separate backup or data recovery networks. On a database server that runs virtual machines, you could use this to isolate traffic from two virtual machines.
After you have added the card to the database server, you need to configure the card. See:
2.6.1 Viewing the Network Interfaces
To view the network interfaces, you can run the ipconf.pl command.
Example 2-1 Viewing the default network interfaces for an Oracle Exadata Database Machine X6-2 database server
The following example shows the output for an Oracle Exadata Database Machine X6-2 database server without the additional network card. The output shows two network cards:
-
A quad port 10Gb card, on eth0 to eth3
-
A dual port 10Gb card, on eth4 and eth5
# cd /opt/oracle.cellos/ # ./ipconf.pl Logging started to /var/log/cellos/ipconf.log Interface ib0 is Linked. hca: mlx4_0 Interface ib1 is Linked. hca: mlx4_0 Interface eth0 is Linked. driver/mac: ixgbe/00:10:e0:8b:24:b6 Interface eth1 is ..... Linked. driver/mac: ixgbe/00:10:e0:8b:24:b7 Interface eth2 is ..... Linked. driver/mac: ixgbe/00:10:e0:8b:24:b8 Interface eth3 is ..... Linked. driver/mac: ixgbe/00:10:e0:8b:24:b9 Interface eth4 is Linked. driver/mac: ixgbe/90:e2:ba:ac:20:ec (slave of bondeth0) nterface eth5 is Linked. driver/mac: ixgbe/90:e2:ba:ac:20:ec (slave of bondeth0)
Example 2-2 Viewing the default network interfaces for an Oracle Exadata Database Machine X7-2 database server
The following example shows the output for an Oracle Exadata Database Machine X7-2 database server without the additional network card. The output shows three network cards:
-
A single port 10Gb card, on eth0
-
A dual port 10 or 25Gb card, on eth1 and eth2
-
A dual port 25Gb card, on eth3 and eth4
# /opt/oracle.cellos/ipconf.pl Logging started to /var/log/cellos/ipconf.log Interface ib0 is Linked. hca: mlx4_0 Interface ib1 is Linked. hca: mlx4_0 Interface eth0 is Linked. driver/mac: igb/00: 10:e0:c3:ba:72 Interface eth1 is Linked. driver/mac: bnxt_en /00:10:e0:c3:ba:73 Interface eth2 is Linked. driver/mac: bnxt_en /00:10:e0:c3:ba:74 Interface eth3 is Linked. driver/mac: bnxt_en /00:0a:f7:c3:14:a0 (slave of bondeth0) Interface eth4 is Linked. driver/mac: bnxt_en /00:0a:f7:c3:14:a0 (slave of bondeth0)
2.6.2 Configuring the Additional Network Card for a Non-Oracle VM Environment
You can configure the additional network card on an Oracle Exadata Database Machine X6-2 or later database server for a non-Oracle VM environment.
This procedure assumes that you have already installed the network card in the Oracle Exadata Database Machine database server but have not yet completed the configuration with Oracle Exadata Deployment Assistant (OEDA).
WARNING:
If you have already installed Oracle Grid Infrastructure on Oracle Exadata Database Machine, then refer to the Oracle Clusterware documentation. Use caution when changing the network interfaces for the cluster.2.6.3 Configuring the Additional Network Card for an Oracle VM Environment
You can configure the additional network card on an Oracle Exadata Database Machine X6-2 and later database server for an Oracle VM environment.
This procedure assumes that you have already installed the network card in the Oracle Exadata Database Machine database server but have not yet completed the configuration with Oracle Exadata Deployment Assistant (OEDA).
Caution:
Do not attempt this procedure if you have already installed Oracle Grid Infrastructure on Oracle Exadata Database Machine.2.7 Increasing the Number of Active Cores on Database Servers
You can increase the number of active cores on Oracle Exadata Database Machine using capacity-on-demand.
The number of active cores on the database servers on Oracle Exadata Database Machine X4-2 and newer systems can be reduced during installation. The number of active cores can be increased when additional capacity is needed. This is known as capacity-on-demand.
Additional cores are increased in 2-core increments on Oracle Exadata Database Machine X4-2 and newer systems, and in 8-core increments on Oracle Exadata Database Machine X4-8 Full Rack and newer systems. The following table lists the capacity-on-demand core processor configurations.
Table 2-3 Capacity-on-Demand Core Processor Configurations
| Oracle Exadata Database Machine | Eligible Systems | Minimum Cores per Server | Maximum Cores per Server | Core Increments |
|---|---|---|---|---|
|
Oracle Exadata Database Machine X7-2 |
Any configuration except Eighth Rack |
14 |
48 |
From 14 to 48, in multiples of 2: 14, 16, 18, ..., 46, 48 |
|
Oracle Exadata Database Machine X7-2 |
Eighth rack |
8 |
24 |
From 8 to 24, in multiples of 2: 8, 10, 12, ..., 22, 24 |
|
Oracle Exadata Database Machine X6-2 |
Any configuration except Eighth Rack |
14 |
44 |
From 14 to 44, in multiples of 2: 14, 16, 18, ..., 42, 44 |
|
Oracle Exadata Database Machine X6-2 |
Eighth rack |
8 |
22 |
From 8 to 22, in multiples of 2: 8, 10, 12, ..., 20, 22 |
|
Oracle Exadata Database Machine X5-2 |
Any configuration except Eighth Rack |
14 |
36 |
From 14 to 36, in multiples of 2: 14, 16, 18, ..., 34, 36 |
|
Oracle Exadata Database Machine X5-2 |
Eighth rack |
8 |
18 |
From 8 to 18, in multiples of 2: 8, 10, 12, ..., 16, 18 |
|
Oracle Exadata Database Machine X4-2 |
Full rack Half rack Quarter rack |
12 |
24 |
From 12 to 24, in multiples of 2: 12, 14, 16, ..., 22, 24 |
|
Oracle Exadata Database Machine X7-8 |
Any configuration |
56 |
192 |
From 56 to 192, in multiples of 8: 56, 64, 72,..., 184, 192 |
|
Oracle Exadata Database Machine X6-8 and X5-8 |
Any configuration |
56 |
144 |
From 56 to 144, in multiples of 8: 56, 64, 72, ..., 136, 144 |
|
Oracle Exadata Database Machine X4-8 |
Full rack |
48 |
120 |
From 48 to 120, in multiples of 8: 48, 56, 64, ..., 112, 120 |
Note:
Oracle recommends licensing the same number of cores on each server, in case of failover.
Database servers can be added one at a time, and capacity-on-demand can be applied to the individual database servers. This option includes Oracle Exadata Database Machine X5-2 Eighth Racks.
The database server must be restarted after enabling additional cores. If the database servers are part of a cluster, then they can be enabled in a rolling fashion.
2.8 Migrating a Bare Metal Oracle RAC Cluster to an Oracle RAC Cluster in Oracle VM
Note:
This topic applies only to two-socket x86 servers. It does not apply to eight-socket servers such as Oracle Exadata Database Machine X5-8.The migration of a bare metal Oracle RAC cluster to an Oracle RAC cluster in Oracle VM can be achieved in the following ways:
-
Migrate to Oracle RAC cluster in Oracle VM using the existing bare metal Oracle RAC cluster with zero downtime.
-
Migrate to Oracle RAC cluster in Oracle VM by creating a new Oracle RAC cluster in Oracle VM with minimal downtime.
-
Migrate to Oracle RAC cluster in Oracle VM using Oracle Data Guard with minimal downtime.
-
Migrate to Oracle RAC cluster in Oracle VM using Oracle Recovery Manager (RMAN) backup and restore with complete downtime.
The conversion of a bare metal Oracle RAC cluster to an Oracle RAC cluster in Oracle VM has the following implications:
-
Each of the database servers will be converted to an Oracle VM Server on which a management domain (dom0) is created along with one or more user domains, depending on the number of Oracle RAC clusters being deployed. Each user domain on a database server will belong to a particular Oracle RAC cluster.
-
As part of the conversion procedure, the bare metal Oracle RAC cluster will be converted to one Oracle RAC cluster in Oracle VM to start with. There will be one user domain per database server.
-
At the end of the conversion, the cell disk and grid disk configuration of the storage cells are the same as they were at the beginning of the conversion.
-
The management domain will use a small portion of the system resources on each database server. Typically a management domain uses 8 GB of memory and 4 virtual CPUs. This has to be taken into consideration while sizing the SGA of the databases running on the Oracle RAC cluster in Oracle VM.
- Refer to My Oracle Support note 2099488.1 for the complete instructions.
2.9 Managing Oracle VM Domains on Oracle Exadata Database Machine
Oracle VM user domains on Oracle Exadata Database Machine are managed using the xm(1) command run from the management domain (also known as domain-0, or dom0). For a full list of Oracle VM administrative commands, run the xm help command.
Note:
The following xm subcommands are not supported on Oracle Exadata Database Machine:
mem-set mem-max migrate restore resume save suspend sched-* cpupool-* tmem-*
This section contains the following topics:
-
Shutting Down a User Domain From Within the Management Domain
-
Backing Up and Restoring Oracle Databases on Oracle VM User Domains
-
Modifying the Number of Virtual CPUs Allocated to a User Domain
-
Expanding /EXAVMIMAGES on Management Domain on Releases 12.2.x or Later
-
Creating a User Domain Without Oracle Grid Infrastructure and Oracle Database
Note:
Unless otherwise noted, all commands run in the preceding procedures are run as the root user.
2.9.6 Shutting Down a User Domain From Within the Management Domain
The following procedure describes how to shut down a user domain from within a management domain:
2.9.7 Backing Up and Restoring Oracle Databases on Oracle VM User Domains
Backing up and restoring Oracle databases on Oracle VM user domains is the same as backing up and restoring Oracle databases on physical nodes.
2.9.8 Modifying the Memory Allocated to a User Domain
The following procedure describes how to modify the memory allocated to a user domain:
Note:
If you are decreasing the amount of memory allocated to a user domain, you must first review and adjust the SGA size of databases running in the user domain and the corresponding huge pages operating system configuration. Failing to do so may result in user domain that cannot start because too much memory is reserved for huge pages when the Linux operating system attempts to boot. See My Oracle Support note 361468.1 for details.
Note:
This operation requires user domain restart. It is not supported to modify memory allocation using the xm mem-set command.
2.9.9 Modifying the Number of Virtual CPUs Allocated to a User Domain
-
All actions to modify the number of vCPUs allocated to a user domain are performed in the management domain.
-
The number of vCPUs allowed for a user domain may be changed dynamically to a lower value or to a higher value provided it does not exceed the setting of
maxvcpusparameter for the user domain. -
It is possible to over-commit vCPUs such that the total number of vCPUs assigned to all domains exceeds the number of physical CPUs on the system. However, over-committing CPUs should be done only when competing workloads for oversubscribed resources are well understood and concurrent demand does not exceed physical capacity.
The following procedure describes how to modify the number of virtual CPUs allocated to a user domain:
-
Determine the number of physical CPUs as follows:
-
Use the following command in the management domain:
# xm info | grep -A3 nr_cpus nr_cpus : 24 nr_nodes : 2 cores_per_socket : 6 threads_per_core : 2
In the output, note that the
nr_nodesline refers to the number of sockets. The Exadata database server where the command is run is a 2-socket 6 cores per socket processor, resulting in 24 physical CPU threads (2 sockets x 6 cores/socket = 12 cores. 12 cores x 2 threads per core = 24 CPU threads). -
Use the following command to determine the current setting of vCPUs configured and online for a user domain:
# xm list DomainName -l | grep vcpus (vcpus 4) (online_vcpus 2)In the preceding command, DomainName is the name of the user domain. The output from the command indicates the maximum number of vCPUs for the user domain is 4, and the current number of online vCPUs is 2. This user domain may have the number of online vCPUs adjusted to any value not greater than the
vcpusparameter while the user domain remains online. The user domain must be taken offline to increase the number of online vCPUs to a value higher than thevcpusparameter.
-
-
Reduce or increase the number of vCPUs as follows:
-
To reduce the number of vCPUs:
-
Determine the currently allocated number of vCPUs for the user domain using the following command:
# xm list DomainName -
Reduce the currently allocated number of vCPUs using the following command:
# xm vcpu-set DomainName vCPUs_preferred
In the preceding command, vCPUs_preferred is the value of the preferred number of vCPUs
-
-
To increase the number of vCPUs
-
Determine the current settings of the
vcpusparameter using the following command:# xm list DomainName -l | grep vcpus (vcpus 4) (online_vcpus 2) -
If the preferred number of vCPUs is less than or equal to the value of the
vcpusparameter, then run the following command to increase the number of online vCPUs.# xm vcpu-set DomainName vCPUs_preferred
In the preceding command, vCPUs_preferred is the value of the preferred number of vCPUs
-
If the preferred number of vCPUs is greater than the value of the
vcpusparameter, then the user domain must be taken offline to increase the number of online vCPUs to a value higher than thevcpusparameter. Do the following:i. Shut down the user domain.
ii. Create a backup copy of the
/EXAVMIMAGES/GuestImages/DomainName/vm.cfgfile.iii. Edit the
/EXAVMIMAGES/GuestImages/DomainName/vm.cfgfile to set thevcpusparameter to the desired number of vCPUs.Note: By default a user domain will online the number of vCPUs configured via the
vcpusparameter. If you want a user domain to start with some vCPUs offline, then add themaxvcpusparameter tovm.cfg, setting it to the maximum number of vCPUs the user domain is permitted to have online. Set thevcpusparameter to the number of vCPUs to online when the user domain starts. For example, to start a user domain with 2 vCPUs online and to allow an additional 6 vCPUs to be added to the user domain while it remains online, use the following settings invm.cfg:maxvcpus=8 vcpus=2
iv. Start the user domain.
-
-
2.9.10 Increasing the Disk Space in a User Domain
The procedures in this section describe how to increase the size of Logical Volume Manager (LVM) partitions, swap space, and file systems in a user domain. This section contains the following topics:
2.9.10.1 Adding a New LVM Disk to a User Domain
This procedure describes how to add a new LVM disk to a user domain to increase the amount of usable LVM disk space in a user domain. This procedure is done so that the size of a file system or swap LVM partition can be increased. This procedure is performed while the system remains online.
Note:
This procedure requires steps be run in the management domain (Domain-0), and in the user domain.
Run all steps in this procedure as the root user.
2.9.10.2 Increasing the Size of the root File System
This procedure describes how to increase the size of the system partition and / (root) file system.
This procedure is performed while the file system remains online.
Note:
There are two system partitions, LVDbSys1 and LVDbSys2. One partition is active and mounted. The other partition is inactive and used as a backup location during upgrade. The size of both system partitions must be equal.
Keep at least 1 GB of free space in the VGExaDb volume group. The free space is used for the LVM snapshot created by the dbnodeupdate.sh utility during software maintenance. If you make snapshot-based backups of the / (root) and /u01 directories as described in "Creating a Snapshot-Based Backup of Oracle Linux Database Server," then keep at least 6 GB of free space in the VGExaDb volume group.
Related Topics
2.9.10.3 Increasing the Size of the /u01 File System
This procedure describes how to increase the size of the /u01 file system.
This procedure is performed while the file system remains online.
Note:
Keep at least 1 GB of free space in the VGExaDb volume group. The free space is used for the LVM snapshot created by the dbnodeupdate.sh utility during software maintenance. If you make snapshot-based backups of the / (root) and /u01 directories as described in "Creating a Snapshot-Based Backup of Oracle Linux Database Server," then keep at least 6 GB of free space in the VGExaDb volume group
Related Topics
2.9.10.4 Increasing the Size of the Grid Infrastructure Home or Database Home File System
The Grid Infrastructure software home and the Oracle Database software home are created as separate disk image files in the management domain. The disk image files are located in the /EXAVMIMAGES/GuestImages/DomainName/ directory. The disk image files are attached to the user domain automatically during virtual machine startup, and mounted as separate, non-LVM file systems in the user domain.
The procedure in this section describes how to increase the size of the database home file system in a user domain. The same procedure can be used to increase the size of the grid home.
2.9.11 Expanding /EXAVMIMAGES After Adding the Database Server Disk Expansion Kit
With the addition of a disk expansion kit to the database server, it is important to follow proper procedures to add this additional space to the /EXAVMIMAGES file system.
2.9.11.1 Expanding /EXAVMIMAGES on Management Domain on Releases 12.2.x or Later
If you are using a release of Oracle Exadata System Software release 12.2.x or later, then use this procedure to expand the /EXAVMIMAGES file system on the management domain following the addition of a disk expansion kit.
During deployment, all available disk space on a database server will be allocated in the management domain (dom0) with the majority of the space allocated to /EXAVMIMAGES for user domain storage. The /EXAVMIMAGES file system is created on /dev/VGExaDb/LVDbExaVMImages.
In the example below, dm01db01 is the name of the management domain, and dm01db01vm01 is a user domain.
2.9.11.2 Expanding /EXAVMIMAGES on Management Domain on Releases Earlier than 12.2.x
If you are using a release of Oracle Exadata System Software that is release 12.1.x or earlier, then use this procedure to expand the /EXAVMIMAGES directory on the management domain following the addition of a disk expansion kit.
During deployment, all available disk space on a database server will be allocated in the management domain with the majority of the space allocated to /EXAVMIMAGES for user domain storage. The /EXAVMIMAGES file system is created on /dev/sda3.
In the example below, dm01db01 is the name of the management domain, and dm01db01vm01 is a user domain.
2.9.12 Creating Oracle VM Oracle RAC Clusters
This procedure creates Oracle VM Oracle RAC clusters using Oracle Exadata Deployment Assistant configuration tool and deployment tool.
The requirements for adding an Oracle VM Oracle RAC cluster are as follows:
-
The system has already been deployed with one or more Oracle VM Oracle RAC clusters.
-
System has available resources, such as memory, CPU, local disk space, and Oracle Exadata Storage Server disk space.
-
Oracle Exadata Deployment Assistant deployment files used for initial system configuration are available.
2.9.13 Expanding an Oracle VM Oracle RAC Cluster on Exadata
You can expand an existing Oracle RAC cluster on Oracle VM by adding guest domains using Oracle Exadata Deployment Assistant (OEDA).
Note:
During the execution of this procedure, the existing Oracle RAC cluster nodes along with their database instances incur zero downtime.
Use cases for this procedure include:
-
You have an existing Oracle RAC cluster that uses only a subset of the database servers of an Exadata rack, and now the nodes not being used by the cluster have become candidates for use.
-
You have an existing Oracle RAC cluster on Exadata that recently got inter-racked with a new Oracle VM Server node or a separate Exadata rack.
- Use OEDA to build a new XML file containing information about the new guest domains (virtual machines) to be added on the dom0’s that will host the new guest domains.
Save the generated XML file in
unzipped_OEDA_location/ExadataConfigurations.If you are adding a single guest domain, the XML file will portray a single node Oracle RAC cluster.
All the screens before the "Define Clusters" screen will have the same data as the XML file of the existing cluster.
- Select the appropriate Oracle VM Server in the Define Clusters screen. The following figure shows a single node addition to a cluster.
- Enter cluster detail such as IP addresses and pool size in the cluster details screen. The following figure shows a single node addition to a cluster.
- Select the appropriate Oracle VM Server in the Define Clusters screen. The following figure shows a single node addition to a cluster.
2.9.14 Creating a User Domain Without Oracle Grid Infrastructure and Oracle Database
A user domain can be created without Oracle Grid Infrastructure and Oracle Database installed on the system. The new user domain has the following characteristics:
-
Operating system image is Oracle Linux
-
Access to the management, client, and InfiniBand networks
-
No Oracle Grid Infrastructure and Oracle Database is installed
The following procedure creates a user domain without Oracle Grid Infrastructure and Oracle Database installed:
-
Allocate new, unused, IP addresses and host names for the new user domain. IP addresses and host names are needed for the management network, client (SCAN) network, and the private InfiniBand network.
Note:
Ensure the intended InfiniBand network IP addresses are unused by using the
pingcommand for each address. Theibhostscommand cannot be used to determine all InfiniBand network IP addresses in use because it does not contain entries for user domains. -
If necessary, obtain an updated user domain (domU) system image file.
The
exadata.img.domu_makercommand that you will run later in this procedure to create a user domain requires the user domain (domU) system image fileSystem.first.boot.version.imgin/EXAVMIMAGES, whereversionmatches the management domain Exadata software version as determined by running the "imageinfo -ver" command in the management domain.For example, when
exadata.img.domu_makeris run to create a new user domain and the management domain Exadata software version is 12.1.2.1.1.150316.2, the user domain (domU) system image file/EXAVMIMAGES/System.first.boot.12.1.2.1.1.150316.2.imgmust exist.# imageinfo -ver 12.1.2.1.1.150316.2 # ls -l /EXAVMIMAGES/System.first.boot.12.1.2.1.1.150316.2.img -rw-r--r-- 1 root root 13958643712 Mar 23 12:25 /EXAVMIMAGES/System.first.boot.12.1.2.1.1.150316.2.img
If the user domain (domU) system image file does not exist, then it must be obtained from My Oracle Support and placed in
/EXAVMIMAGESin the management domain. See My Oracle Support note 888828.1 for additional information. -
In the management domain, copy an existing XML configuration file from a deployed user domain to a new file name using the following command:
# cp /EXAVMIMAGES/conf/existingDomainName-vm.xml /EXAVMIMAGES/conf/newDomainName-vm.xml
In the preceding command,
existingDomainName-vm.xmlis the XML configuration file of the deployed user domain, andnewDomainName-vm.xmlis the name of the new file.In the following example, the configuration file for user domain "dm01db01vm01" is copied to
nondbdomain-vm.xml.# cp /EXAVMIMAGES/conf/dm01db01vm01-vm.xml /EXAVMIMAGES/conf/nondbdomain-vm.xml
-
In the management domain, edit the new XML file as follows:
-
Change all
<Hostname>tags to match the new host names for the respective networks. -
Change all
<IP_address>tags to match the new IP addresses for the respective networks. -
Change the
<virtualMachine>tag to contain the new host name. -
Change the
<hostName>tag to contain the new host name. -
Delete the entire
<disk id="disk_2">and<disk id="disk_3">elements, including all their sub-elements. You must delete the entire entry between the starting<disk>tag to the corresponding closing</disk>.
-
-
In the management domain, allocate InfiniBand network GUIDs for the new user domain using the
/opt/exadata_ovm/exadata.img.domu_makercommand.# /opt/exadata_ovm/exadata.img.domu_maker allocate-guids \ /EXAVMIMAGES/conf/newDomainName-vm.xml \ /EXAVMIMAGES/conf/final-newDomainName-vm.xml -
In the management domain, create the new user domain using the
/opt/exadata_ovm/exadata.img.domu_makercommand.# /opt/exadata_ovm/exadata.img.domu_maker start-domain \ /EXAVMIMAGES/conf/final-newDomainName-vm.xml
2.9.15 Moving a User Domain to a Different Database Server
User domains can move to different database servers.
The target Oracle Exadata Database Server must meet the following requirements:
-
The target database server must have the same Oracle Exadata System Software release installed with Oracle VM.
-
The target database server must have the same network visibility.
-
The target database server must have access to the same Oracle Exadata Storage Servers.
-
The target database server must have sufficient free resources (CPU, memory, and local disk storage) to operate the user domain.
-
It is possible to over-commit virtual CPUs such that the total number of virtual CPUs assigned to all domains exceeds the number of physical CPUs on the system. Over-committing CPUs can be done only when the competing workloads for over-subscribed resources are well understood and the concurrent demand does not exceed physical capacity.
-
It is not possible to over-commit memory.
-
Copying disk images to the target database server may increase space allocation of the disk image files because the copied files are no longer able to benefit from the disk space savings gained by using OCFS2 reflinks.
-
-
The user domain name must not be already in use on the target database server.
The following procedure moves a user domain to a new database server in the same Oracle Exadata System Software configuration. All steps in this procedure are performed in the management domain.
2.9.16 Removing an Oracle RAC Cluster Running in Oracle VM
You can remove all Oracle RAC nodes of an Oracle VM cluster, including the databases running within the cluster and all data stored on the Oracle Exadata Storage Server used by those databases.
To remove only a subset of user domains of a Oracle VM cluster, refer to the next section.
There are two main steps to remove a Oracle VM cluster:
-
Remove the user domain files from the management domain.
-
Remove the unused Oracle Exadata grid disks.
Note:
If the Oracle Exadata Deployment Assistant xml configuration files are to be reused later, then they will be not synchronized because the definition for the removed user domain still exists in Oracle Exadata Deployment Assistant files.
2.9.17 Deleting a User Domain from an Oracle VM Oracle RAC Cluster
You can remove a single Oracle RAC node from an Oracle VM cluster.
The Oracle Exadata grid disks remain in use by the remaining nodes in the cluster, and must not be dropped.
Note:
If Oracle Exadata Deployment Assistant xml configuration files are to be reused later, then they will be not synchronized because the definition for the removed user domain still exists in Oracle Exadata Deployment Assistant files.
Related Topics
2.9.18 Implementing Tagged VLAN Interfaces
This topic describes the implementation of tagged VLAN interfaces in Oracle VM environments on Exadata.
Oracle databases running in Oracle VM guests on Oracle Exadata Database Machine are accessed through the client Ethernet network defined in the Oracle Exadata Deployment Assistant (OEDA) configuration tool. Client network configuration in both the management domain (dom0) and user domains (domU's) is done automatically when the OEDA installation tool creates the first user domain during initial deployment.
The following figure shows a default bonded client network configuration:
Figure 2-3 NIC Layout in an Oracle Virtual Environment
Description of "Figure 2-3 NIC Layout in an Oracle Virtual Environment"
The network has the following configuration:
-
In the dom0, eth slave interfaces (for example, eth1 and eth2, or eth4 and eth5) that allow access to the domU client network defined in OEDA are discovered, configured, and brought up, but no IP is assigned.
-
In the dom0, bondeth0 master interface is configured and brought up, but no IP is assigned.
-
In the dom0, bridge interface vmbondeth0 is configured, but no IP is assigned.
-
In the dom0, one virtual backend interface (vif) per domU that maps to that particular domU's bondeth0 interface is configured and brought up, but no IP is assigned. These vifs are configured on top of the bridge interface vmbondeth0, and the mapping between the dom0 vif interface and its corresponding user domain interface bondeth0 is defined in the user domain configuration file called
vm.cfg, located in/EXAVMIMAGES/GuestImages/user domain name.
For default installations, a single bondeth0 and a corresponding vmbondeth0 bridge interface is configured in the dom0 as described above. This bondeth0 interface is based on the default Access Virtual Local Area Network (Access VLAN). The ports on the switch used by the slave interfaces making up bondeth0 are configured for Access VLAN.
Using VLAN Tagging
If there is a need for virtual deployments on Exadata to access additional VLANs on the client network, such as enabling network isolation across user domains, then 802.1Q-based VLAN tagging is a solution. The following figure shows a client network configuration with VLAN tagging.
Figure 2-4 NIC Layout for Oracle Virtual Environments with VLAN Tagging
Description of "Figure 2-4 NIC Layout for Oracle Virtual Environments with VLAN Tagging"
For instructions on how to configure and use such additional VLAN tagged interfaces on the client network, see My Oracle Support note 2018550.1. The Access VLAN must stay working and configured before and after these instructions are followed. At no time is the Access VLAN to be disabled.
2.9.19 Implementing InfiniBand Partitioning across Oracle VM Oracle RAC Clusters on Oracle Exadata Database Machine
One of the key requirements of consolidated systems from a security standpoint is network isolation across the multiple environments within a consolidated system. For consolidations achieved using Oracle VM Oracle Real Application Clusters (Oracle RAC) clusters on Oracle Exadata, this means isolation across the different Oracle RAC clusters such that network traffic of one Oracle RAC cluster is not accessible to another Oracle RAC cluster. For the Ethernet networks, this is accomplished using VLAN tagging as described in My Oracle Support DocID 2018550.1. For the InfiniBand network, this is accomplished using custom InfiniBand partitioning, dedicated partition keys, and partitioned tables.
An InfiniBand partition defines a group of InfiniBand nodes or members that are allowed to communicate with one another. With InfiniBand partitioning, partitions identified by unique partition keys are created and are managed by the master subnet manager. Members are then assigned to these custom partitions. Members within a partition can only communicate among themselves (depending on the membership as explained in the Appendix 1 of My Oracle Support DocID 2018550.1). A member of one partition cannot communicate with a member of a different partition regardless of the membership. Continuing along these lines, the Oracle VM Oracle RAC nodes of one particular cluster are assigned one dedicated partition for the clusterware communication and one partition for communication with the storage cells. This way, the nodes of one Oracle RAC cluster will not be able to communicate with the nodes of another Oracle RAC cluster that belong to a different partition. The nodes in each Oracle RAC cluster have different partition keys assigned to them.
By default, the InfiniBand subnet manager provides a single partition that is identified by the partition key 0x7FFF (limited membership) or 0xFFFF (full membership). In Oracle VM deployments on Oracle Exadata Database Machine where custom InfiniBand partitioning is not used, the partition key 0xFFFF is used across all the user domains.
Figure 2-5 Oracle VM Oracle RAC Clusters without InfiniBand Network Isolation Across Clusters
Description of "Figure 2-5 Oracle VM Oracle RAC Clusters without InfiniBand Network Isolation Across Clusters"
With non-default custom partitions in place for implementing isolation across the Oracle VM Oracle RAC clusters, the configuration changes to what is shown in Figure 2-6. New interfaces clib0, clib1 (for the cluster pkey) and stib0, stib1 (for the storage pkey) exist in each of the user domains (domU's).
There is no change to InfiniBand interfaces in the management domain (dom0).
Figure 2-6 Oracle VM Oracle RAC Clusters with InfiniBand Network Isolation Across Clusters Using InfiniBand Partitioning
Description of "Figure 2-6 Oracle VM Oracle RAC Clusters with InfiniBand Network Isolation Across Clusters Using InfiniBand Partitioning"
2.9.19.1 Implementing InfiniBand Partitioning across OVM RAC Clusters: Requirements
Before configuring InfiniBand partitioning, ensure that:
-
You have configured OVM on your Exadata system.
-
All the user domains and storage cells are using the default partition key 0xFFFF.
-
You have set up passwordless secure shell (ssh) access for the root user from one of the management domains (dom0 node) to all the OVM RAC cluster nodes, storage cells, and InfiniBand switches.
-
InfiniBand switches are installed with firmware versions 2.0.4 or above.
-
You have an understanding of InfiniBand partitioning.
2.9.19.2 Implementing InfiniBand Partitioning across Oracle VM RAC Clusters: Steps
To implement InfiniBand partitioning, you perform the following high-level steps:
-
On the InfiniBand switches, create a dedicated partition (cluster pkey) for every Oracle VM RAC cluster to be used by the clusterware.
-
On the InfiniBand switches, create one partition (storage pkey) to be used by all the Oracle VM RAC clusters and the storage cells for communication between the Oracle RAC cluster nodes and the storage cells.
-
Each partition requires a new IPoIB network interface. On the Oracle VM RAC nodes and on the storage cells, generate all the relevant network configuration files for the new IPoIB interfaces. (clib0,clib1,stib0,stib1 in the above diagram).
-
Modify the Oracle VM RAC cluster nodes and the Grid Infrastructure configuration of every Oracle VM RAC cluster to use the newly created IPoIB interfaces.
-
In the management domains (dom0s), modify the user domain configuration file for each user domain to use the partition key applicable to that user domain
-
Modify the storage cells to use the newly created IPoIB interfaces.
-
Restart the storage cells and the Oracle RAC clusters and make sure the Grid Infrastructure and the databases are functional. In this procedure, the Oracle RAC clusters incur a minimal downtime. This step is when the downtime occurs.
This section describes the steps in detail.
-
Allocate IP addresses to be used by the pkey interfaces.
Plan and allocate sets of IP addresses and netmasks for each Oracle VM RAC cluster that will be used by the cluster pkey interfaces and the storage pkey interfaces when InfiniBand partitioning gets implemented in the cluster.
Within an Oracle VM RAC cluster, cluster pkey IP address and netmask should be on a separate subnet from the storage pkey IP address and netmask.
The tables below can be used as reference for one particular RAC cluster:
Table 2-4 Existing Configuration
Interface Name IP Address Netmask ib0
192.168.12.153
255.255.248.0
ib1
192.168.12.154
255.255.248.0
The following table shows the new IP addresses and netmasks required by the pkey interfaces while executing the steps of this document for that one Oracle RAC cluster.
Table 2-5 New IP Addresses and Netmasks Required by the pkey Interfaces
Interface Name IP Address Netmask clib0
192.168.112.1
255.255.248.0
clib1
192.168.112.2
255.255.248.0
stib0
192.168.114.1
255.255.240.0
stib1
192.168.114.2
255.255.240.0
-
Create and configure the partition keys on the InfiniBand switches.
-
Enable password-less ssh equivalence for the root user from any one of the dom0 nodes to all the switches on the InfiniBand fabric. This is the node that you will use for running all the scripts in this procedure. This node will be called the “driver dom0” in this procedure.
You can do this by running the following
dclicommand from the dom0 node that will be used to run the scripts:# dcli –g ib_switch_list -l root –kib_switch_list refers to a file that contains the list of all the InfiniBand switches on the fabric, each one on a separate line.
-
From "Implementing InfiniBand Partitioning across OVM RAC clusters on Exadata (My Oracle Support Doc ID 2075398.1)", download and untar
create_pkeys.tarto the driver dom0 node. When you untar the file, you should get three files:-
create_pkeys_on_switch.sh -
run_create_pkeys.sh -
create_pkey_files.sh
-
-
Run the script
create_pkeys_on_switch.shfrom driver dom0 to create and configure the partition keys on the InfiniBand switches.Note:
One execution of
create_pkeys_on_switch.shcreates exactly one partition. You must run the script once for each partition to be created. For example, an environment that contains two Oracle VM RAC clusters will have a total of three partitions: one storage partition and two cluster partitions (one per Oracle RAC cluster). In this example, you will need to runcreate_pkeys_on_switch.shthree times.The script needs to be executed from only one node (the driver dom0). It will create the partitions in all the switches provided as input during the execution of the script.
Verify the partitions got created on all the switches after the script completes:
# /usr/local/sbin/smpartition list active no-page
At this stage ensure that you have created all the required partitions.
-
-
Configure the Oracle VM RAC cluster nodes for using partition keys.
Note:
This step makes the following changes on the Oracle VM RAC cluster nodes:
-
Modifies these files:
-
/etc/sysconfig/network-scripts/ifcfg-ib0 -
/etc/sysconfig/network-scripts/ifcfg-ib1
-
-
Removes these files:
-
/etc/sysconfig/network-scripts/rule-ib0 -
/etc/sysconfig/network-scripts/rule-ib1 -
/etc/sysconfig/network-scripts/route-ib0 -
/etc/sysconfig/network-scripts/route-ib1
-
-
Before modifying or removing the files above, the script backs them up in the
/etc/sysconfig/network-scripts/backup-for-pkeysdirectory. If this step fails, restore all the files from/etc/sysconfig/network-scripts/backup-for-pkeysto/etc/sysconfig/network-scriptsbefore rerunning this step. -
Creates the following new files in
/etc/sysconfig/network-scripts:-
ifcfg-clib0 -
ifcfg-clib1 -
rule-clib0 -
rule-clib1 -
route-clib0 -
route-clib1 -
ifcfg-stib0 -
ifcfg-stib1 -
rule-stib0 -
rule-stib1 -
route-stib0 -
route-stib1
-
If this step fails, and the files above have been created, then you need to remove them before rerunning this step.
-
Make sure password-less ssh is set up from the driver dom0 node used in step 1 to all the Oracle RAC cluster nodes and the storage cells that need to be configured for partition keys.
-
Make sure
run_create_pkeys.shandcreate_pkey_files.share executable and they are in the same directory on the driver dom0 node. -
Run
run_create_pkeys.sh. The syntax for this script is:run_create_pkeys.sh <node_name> <interface_name> <pkey_id> <node_type> <pkey_ipaddr> <pkey_netmask> <pkey_interfaceType>
<node_name> specifies the
<interface_name> is either
ib0orib1.<pkey_id> specifies the pkey without the “
0x” prefix.<node_type> is either
computeorcell.<pkey_ipaddr> specifies the IP address.
<pkey_netmask> specifies the netmask in CIDR format, for example,
/21.<pkey_interfaceType> is
clusterorstoragefor “compute” node types, orstoragefor “cell” node types.Note:
The pkey_ipaddr and pkey_netmask of the cluster pkey interface must be on a different subnet from the pkey_ipaddr and pkey_netmask of the storage pkey interface.
For cluster nodes, you need to run the script a total of four times for every cluster node with a node type value of compute.
<pkey_id> is the cluster partition key derived from the cluster pkey_id value entered in step 2.
You can use the command below to derive the partition key values to be entered in the command (as shown above) from the pkey_id value entered in step 2.
FinalHexValue=$(echo "obase=16;ibase=2;$(expr 1000000000000000 + $(echo "obase=2;ibase=16;$(echo $HexValue|tr [:lower:] [:upper:])"|bc))"|bc|tr [:upper:] [:lower:])FinalHexValue is the value that will be entered in the command here and HexValue is the value entered in step 2 for pkey_id.
The following table describes the four runs:
Table 2-6 Four Runs for Cluster Nodes
Run Interface Name pkey_id node_type pkey_ipaddress pkey_netmask pkey_interfaceType 1
ib0
Cluster pkey ID
compute
Cluster pkey IP address for ib0
Cluster pkey netmask for ib0
cluster
2
ib1
Cluster pkey ID
compute
Cluster pkey IP address for ib1
Cluster pkey netmask for ib1
cluster
3
ib0
Storage pkey ID
compute
Storage pkey IP address for ib0
Storage pkey netmask for ib0
storage
4
ib1
Storage pkey ID
compute
Storage pkey IP address for ib1
Storage pkey netmask for ib1
storage
Examples:
# ./run_create_pkeys.sh vm-guest-1 ib0 a000 compute 192.168.12.153 /21 cluster # ./run_create_pkeys.sh vm-guest-1 ib1 a000 compute 192.168.12.154 /21 cluster # ./run_create_pkeys.sh vm-guest-1 ib0 aa00 compute 192.168.114.15 /20 storage # ./run_create_pkeys.sh vm-guest-1 ib1 aa00 compute 192.168.114.16 /20 storage
At this stage all the required networking files listed below have been created for the new pkey-enabled network interfaces on the Oracle VM RAC cluster nodes:
-
/etc/sysconfig/network-scripts/ifcfg-clib0 -
/etc/sysconfig/network-scripts/ifcfg-clib1 -
/etc/sysconfig/network-scripts/ifcfg-stib0 -
/etc/sysconfig/network-scripts/ifcfg-stib1 -
/etc/sysconfig/network-scripts/rule-clib0 -
/etc/sysconfig/network-scripts/rule-clib1 -
/etc/sysconfig/network-scripts/rule-stib0 -
/etc/sysconfig/network-scripts/rule-stib1 -
/etc/sysconfig/network-scripts/route-clib0 -
/etc/sysconfig/network-scripts/route-clib1 -
/etc/sysconfig/network-scripts/route-stib0 -
/etc/sysconfig/network-scripts/route-stib1
Oracle Grid Infrastructure has also been modified to make use of the new network interfaces upon restart. "
$GRID_HOME/bin/oifcfg getif" should list clib0 and clib1 in the list of interfaces to be used for the cluster_interconnect. -
-
-
Modify Oracle ASM and Oracle RAC
CLUSTER_INTERCONNECTSparameter.-
Login to each of the Oracle ASM instances in the Oracle RAC cluster using SQL*Plus as SYS, and run the following command:
alter system set cluster_interconnects='<cluster pkey IP address of ib0>:<cluster pkey IP address of ib1>' scope=spfile sid='<name of the current ASM instance>';
For example:
alter system set cluster_interconnects='192.168.12.153:192.168.12.154' scope=spfile sid='<name of the current ASM instance>';
-
Login to each of the database instances in the Oracle RAC cluster using SQL*Plus, and run the following command:
alter system set cluster_interconnects='<cluster pkey IP address of ib0>:<cluster pkey IP address of ib1>' scope=spfile sid='<name of the current RDBMS instance>';
For example:
alter system set cluster_interconnects='192.168.12.153:192.168.12.154' scope=spfile sid='<name of the current RDBMS instance>';
-
Shut down and disable CRS auto-start on all the Oracle RAC cluster nodes.
# $GRID_HOME/bin/crsctl stop crs # $GRID_HOME/bin/crsctl disable crs
At this stage Oracle Grid Infrastructure, the Oracle ASM instances, and the Oracle Database instances have been modified to make use of the newly created network interfaces.
-
-
Modify
cellip.oraandcellinit.oraon all the cluster nodes (user domains).-
Manually edit
/etc/oracle/cell/network-config/cellip.oraon all the Oracle RAC cluster nodes to replace the existing IP address with the two storage pkey IP addresses of every storage cell that will be set up in step 6. Separate the two IP addresses with a “;”.Here is an example of a
/etc/oracle/cell/network-config/cellip.orafile:cell="192.168.114.1;192.168.114.2" cell="192.168.114.3;192.168.114.4" cell="192.168.114.5;192.168.114.6" cell="192.168.114.7;192.168.114.8" cell="192.168.114.9;192.168.114.10" cell="192.168.114.11;192.168.114.12" cell="192.168.114.13;192.168.114.14"
The steps below are recommended to achieve the objective of this step. You perform these steps on any one database server node of the cluster (user domain for an Oracle VM RAC cluster):
-
Run the following commands:
# cd /etc/oracle/cell/network-config # cp cellip.ora cellip.ora-bak
-
Make the modifications to
cellip.ora-bak. -
Make sure ssh equivalence is set up for the root user to all the cluster nodes from this cluster node.
-
Run the following commands below.
<cluster_nodes>refers to a file containing the names of all the Oracle RAC cluster nodes of the Oracle VM RAC cluster, each node on a separate line.# /usr/local/bin/dcli -g <cluster_nodes> –l root "/bin/cp /etc/oracle/cell/network-config/cellip.ora /etc/oracle/cell/network-config/cellip-orig.ora" # /usr/local/bin/dcli -g <cluster_nodes> –l root –f cellip.ora-bak –d /etc/oracle/cell/network-config/cellip.ora
-
-
Back up
/etc/oracle/cell/network-config/cellinit.orato/etc/oracle/cell/network-config/cellinit.ora-bak. -
Manually edit
/etc/oracle/cell/network-config/cellinit.orato replace the existing IP addresses/netmask with the two storage pkey IP addresses/netmask of the cluster node which was used in step 3. The IP address and netmask were used in the third and fourth run of step 3.
-
-
Modify all the relevant
vm.cfgfiles in the dom0. This step is applicable only for Oracle VM environments.Login to all the dom0s and manually edit
/EXAVMIMAGES/GuestImages/<user domain name>/vm.cfgto include the partition keys created in step 2. See example below.Modify the line:
ib_pkeys = [{'pf':'40:00.0','port':'1','pkey':['0xffff',]},{'pf':'40:00.0','port':'2','pkey':['0xffff',]},]to:
ib_pkeys = [{'pf':'40:00.0','port':'1','pkey':['0xa000','0xaa00',]},{'pf':'40:00.0','port':'2','pkey':['0xa000','0xaa00',]},]In the example above,
0xa000is the cluster partition key derived from the cluster pkey_id value entered in step 2, and0xaa00is the storage partition key derived from the storage pkey_id value.You can use the command below to derive the partition key values to be entered in
vm.cfg(as shown above) from the pkey_id values entered in step 2.FinalHexValue=$(echo "obase=16;ibase=2;$(expr 1000000000000000 + $(echo "obase=2;ibase=16;$(echo $HexValue|tr [:lower:] [:upper:])"|bc))"|bc|tr [:upper:] [:lower:])
FinalHexValue is the value that will be entered in
vm.cfgand HexValue is the value entered in step 2 for pkey_id.Note:
If your environment has multiple Oracle VM RAC clusters, step 7 and step 8 need to be performed only once AFTER steps 3 through 6 have been executed for all the Oracle VM RAC clusters.
-
Configure the storage cells for using partition keys.
-
Make sure
run_create_pkeys.shandcreate_pkey_files.shhave been made available from step 2 and that they are in the same directory on the same dom0 node used in step 2. -
Make sure password-less ssh is set up from the dom0 node used in step 2 to all the storage cells that need to be configured for partition keys.
-
Execute
run_create_pkeys.shfrom that dom0 node. The syntax for this script for storage cells is:run_create_pkeys.sh <node_name> <interfaceName> <pkey_id> <node_type> <pkey_ipaddr> <pkey_netmask> <pkey_interfaceType>
<node_name> specifies the
<interface_name> is either
ib0orib1.<pkey_id> specifies the pkey without the “
0x” prefix.<node_type> is either
computeorcell.<pkey_ipaddr> specifies the IP address.
<pkey_netmask> specifies the netmask in CIDR format, for example,
/21.<pkey_interfaceType> is
clusterorstoragefor “compute” node types, orstoragefor “cell” node types.Note:
<pkey_id> in the command above is the storage partition key derived from the storage pkey_id value entered in step 2.
The command below can be used to derive the partition key values to be entered in the command (as shown above) from the pkey_id value entered in step 2.
FinalHexValue=$(echo "obase=16;ibase=2;$(expr 1000000000000000 + $(echo "obase=2;ibase=16;$(echo $HexValue|tr [:lower:] [:upper:])"|bc))"|bc|tr [:upper:] [:lower:])FinalHexValue is the value that will be entered in the command and HexValue is the value entered in step 2 for pkey_id.
For storage cells, you need to run the script twice for every storage cell with a node_type of “cell”, as shown in the following table:
Table 2-7 Two Runs for Storage Cells
Run Interface Name pkey_id node_type pkey_ipaddress pkey_netmask pkey_interfaceType 1
ib0
Storage pkey ID
cell
Storage pkey IP address for ib0
Storage pkey netmask for ib0
storage
2
ib1
Storage pkey ID
cell
Storage pkey IP address for ib1
Storage pkey netmask for ib1
storage
Examples:
# ./run_create_pkeys.sh cell-1 ib0 aa00 cell 192.168.114.1 /20 storage # ./run_create_pkeys.sh cell-1 ib1 aa00 cell 192.168.114.2 /20 storage
Note:
You can ignore the following messages from the script. The reboot of the storage cells in step 9 will take care of these items:
Network configuration altered. Please issue the following commands as root to restart the network and open IB stack:service openibd restartservice network restartA restart of all services is required to put new network configuration into effect. MS-CELLSRV communication may be hampered until restart.At this stage the storage cells have been modified to use the new network interfaces upon reboot.
-
-
Modify
/opt/oracle.cellos/cell.confon the storage cells and reboot the storage cells.-
Back up
/opt/oracle.cellos/cell.conf.# cp /opt/oracle.cellos/cell.conf /opt/oracle.cellos/cell.conf-prepkey
-
Change the following lines in
/opt/oracle.cellos/cell.conf.Change this line:
<Pkeyconfigured>no</Pkeyconfigured>
to:
<Pkeyconfigured>yes</Pkeyconfigured>
Change this line for the 2 private interfaces ib0 and ib1:
<IP_enabled>yes</IP_enabled>
to:
<IP_enabled>no</IP_enabled>
-
Make sure Grid Infrastructure is stopped on all Oracle VM RAC nodes.
-
Reboot all the storage cell servers.
-
After the storage cells come back up, check the new pkey-enabled network interfaces are in use:
# cellcli -e list cell detail | egrep 'interconnect|ipaddress'
This should show the new pkey-enabled interfaces (stib0 and stib1) along with the new set of IP addresses.
-
-
Restart the cluster nodes.
-
Login to the corresponding dom0 of each of the user domain nodes.
-
Run the following commands:
# xm shutdown <user domain name> # xm create /EXAVMIMAGES/GuestImages/<user domain name>/vm.cfg
-
-
Start and verify the Oracle Grid Infrastructure stack is fully up on all the cluster nodes.
-
Start and enable auto-start of the Grid Infrastructure stack on all the Oracle RAC cluster nodes
# $GRID_HOME/bin/crsctl start crs # $GRID_HOME/bin/crsctl enable crs
-
Once Oracle Grid Infrastructure is started up on all the nodes, verify the
cluster_interconnectsparameter is set to use the newly configured pkey interfaces:SQL> select inst_id, value from gv$parameter where name = 'cluster_interconnects'
-
Remove the old cluster interconnect interfaces from OCR.
# $GRID_HOME/bin/oifcfg delif –global ib0/<old subnet> # $GRID_HOME/bin/oifcfg delif –global ib1/<old subnet>
-
2.9.19.3 Implementing InfiniBand Partitioning across OVM RAC Clusters: Setting up Limited Membership
The 12.1.0.2 October 2016 Database Bundle Patch introduces a security enhancement feature where the GUIDs of the database nodes can be assigned to the storage pkey with limited membership instead of full membership, as was the case prior to the 12.1.0.2 October 2016 Bundle Patch. This addresses a security concern where one RAC node from one RAC cluster could talk to a RAC node from another RAC cluster using the storage pkey interfaces.
Full Membership and Limited Membership
An InfiniBand partition defines a group of InfiniBand nodes that are allowed to communicate with one another. With InfiniBand partitioning, you define custom or unique partition keys that are managed by the master subnet manager, and assign members to the custom partition keys. Members with the same partition key can only communicate amongst themselves. A member of one partition key cannot communicate with a member that has a different partition key, regardless of membership type. The OVM RAC cluster nodes of one cluster are assigned one partition key for clusterware communication and another partition key for communication with storage cells. This way, the nodes of one RAC cluster will not be able to communicate with the nodes of another RAC cluster, which have a different partition key assigned to them. This is very similar conceptually to tagged VLANs in the Ethernet world.
Partition keys (pkeys) are 15-bit integers and have a value of 0x1 to 0x7FFF. An additional bit, the membership bit, identifies the membership of a member of the partition. Memberships can be:
-
Full: The membership bit is set to 1. Members with full membership can communicate with each other as well as members with limited membership within same the partition key.
-
Limited: The membership bit is set to 0. Members with limited membership within a partition cannot communicate with each other. However they can communicate with other members with full membership within the same partition.
Combined together, the pkey and the membership bit comprise a 16-bit integer. The most significant bit is the membership bit.
By default, the InfiniBand subnet manager provides a single partition and it is identified by the partition key 0x7FFF (limited membership) or 0xFFFF (full membership).
An HCA port can participate in a maximum of 128 partitions. Each partition key provides a new IPoIB network interface. For example, InfiniBand port 1 with partition key 0xa001 will result in a new network interface. These interfaces are named with meaningful names through the ifcfg-<interface> file parameters.
An InfiniBand node can be a member of multiple partitions. When a packet arrives at a database node, the partition key (pkey) of the packet is matched with the Subnet Manager configuration. This validation prevents a database node from communicating with another database node outside of the partitions of which it is a member.
Every node within the infiniBand fabric has a partition key table which you can see in /sys/class/infiniband/mlx4_0/ports/[1-2]/pkeys. Every Queue Pair (QP) of the node has an index (pkey) associated with it that maps to an entry in that table. Whenever a packet is sent from the QP’s send queue, the indexed pkey is attached with it. Whenever a packet is received on the QP’s receive queue, the indexed pkey is compared with that of the incoming packet. If it does not match, the packet is silently discarded. The receiving Channel Adapter does not know it arrived and the sending Channel Adapter gets no acknowledgement as well that it was received. The sent packet simply gets manifested as a lost packet. It is only when the pkey of the incoming packet matches the indexed pkey of the QP’s receive queue, a handshake is made and the packet is accepted and an acknowledgment is sent to the sending channel adapter. This is how only members of the same partition are able to communicate with each other and not with hosts that are not members of that partition (which means those hosts that does not have that pkey in their partition table).
The steps below describe how to set up this enhancement on a pkey-enabled environment that has the 12.1.0.2 October 2016 Database Bundle Patch applied. There are two possible scenarios, as described below:
Case 1. Implementing the feature on a pkey-enabled environment in a rolling manner
In this case, you have already applied the 12.1.0.2 October 2016 Database Bundle Patch.
Perform the steps below on one node at a time.
-
Shut down the Grid Infrastructure on the node.
# $GI_HOME/bin/crsctl stop crs
-
Determine the two port GUIDs of the dom0 (control domain) which manages this user domain OVM RAC cluster node.
# /usr/sbin/ibstat | grep Port
-
Login to the Infiniband Switch where the SM master is running as root.
-
Run the commands below on the InfiniBand switch.
# /usr/local/sbin/smpartition start # /usr/local/sbin/smpartition modify -n <storage pkey name> -port <Port GUID1 of the dom0 from step 2> -m limited # /usr/local/sbin/smpartition modify -n <storage pkey name> -port <Port GUID2 of the dom0 from step 2> -m limited # /usr/local/sbin/smpartition commit
-
Modify the
vm.cfgfile for this OVM RAC user domain node in the dom0.-
Login to the dom0 as root.
-
Edit
/EXAVMIMAGES/GuestImages/<user domain name>/vm.cfgand modify the partition keys as shown in the example below.Modify this line:
ib_pkeys = [{'pf':'40:00.0','port':'1','pkey':[ '0xclpkey','0x<stpkey>',]},{'pf':'40:00.0','port':'2','pkey':[ '0xclpkey','0x<stpkey>',]},]to this:
ib_pkeys = [{'pf':'40:00.0','port':'1','pkey':[ '0xclpkey','0x<mod_stpkey>',]},{'pf':'40:00.0','port':'2','pkey':[ '0xclpkey','0x<mod_stpkey>',]},]<mod_stpkey>is derived from<stpkey>using the formula below:mod_stpkey=$(echo "obase=16;ibase=2;$(expr $(echo "obase=2;ibase=16;$(echo $stpkey|tr [:lower:] [:upper:])"|bc) - 1000000000000000)"|bc|tr [:upper:] [:lower:])Note that
<stpkey>and<mod_stpkey>in the formula above are specified without the "0x" prefix.
-
-
Modify the
/etc/sysconfig/network-scripts/ifcfg-stib*files on the user domain RAC nodes.Edit the PKEY_ID in those files using the formula below:
mod_stpkey=$(echo "obase=16;ibase=2;$(expr $(echo "obase=2;ibase=16;$(echo $stpkey|tr [:lower:] [:upper:])"|bc) - 1000000000000000)"|bc|tr [:upper:] [:lower:])mod_stpkey is the new PKEY_ID, and stpkey is the old PKEY_ID.
Note that
<stpkey>and<mod_stpkey>in the formula above are specified without the "0x" prefix. -
Modify
/opt/oracle.cellos/pkey.confon the user domain RAC nodes.Edit the Pkey for the storage network pkey interfaces (stib*):
Change:
<Pkey>0xstpkey</Pkey>to:
<Pkey>0xmod_stpkey</Pkey>mod_stpkey is derived from stpkey using the formula below:
mod_stpkey=$(echo "obase=16;ibase=2;$(expr $(echo "obase=2;ibase=16;$(echo $stpkey|tr [:lower:] [:upper:])"|bc) - 1000000000000000)"|bc|tr [:upper:] [:lower:])stpkeyandmod_stpkeyused in the formula above are specified without the "0x" prefix. -
Restart the OVM RAC user domain node.
-
Login to the dom0 as root.
-
Run the following commands:
# xm shutdown <user domain name> # xm create /EXAVMIMAGES/GuestImages/<user domain name>/vm.cfg
-
-
Verify the Grid Infrastructure stack is fully up on the cluster node.
-
Repeat the steps on the remaining cluster nodes, one node at a time.
Case 2. Implementing the feature on a pkey-enabled environment while you apply the 12.1.0.2 October 2016 Database Bundle Patch in a rolling manner
Perform the steps below on one node at a time.
-
Apply the 12.1.0.2 October 2016 Database Bundle Patch on the cluster node.
-
Run the steps 1 through 10 from Case 1 above on the node where the patch was applied.
-
Move on to the next cluster node and repeat steps 1 and 2 above.
Note:
Once the dom0 GUIDs are converted to limited membership, deployment of any new cluster will have the October 2016 Database Bundle Patch as a prerequisite.
2.9.20 Running Oracle EXAchk in Oracle VM Environments
Oracle EXAchk version 12.1.0.2.2 and higher supports virtualization on Oracle Exadata Database Machine.
To perform the complete set of Oracle EXAchk audit checks in an Oracle Exadata Database Machine Oracle VM environment, Oracle EXAchk must be installed in and run from multiple locations, as follows:
-
From one management domain (dom0)
-
From one user domain (domU) in each Oracle VM Oracle Real Application Clusters (Oracle RAC) cluster
For example, an Oracle Exadata Database Machine Quarter Rack with 2 database servers containing 4 Oracle VM Oracle RAC clusters (2 nodes per cluster for a total of 8 domU's across both database servers) requires running Oracle EXAchk 5 separate times, as follows:
-
Run Oracle EXAchk in the first user domain (domU) for the first cluster.
-
Run Oracle EXAchk in the first user domain (domU) for the second cluster.
-
Run Oracle EXAchk in the first user domain (domU) for the third cluster.
-
Run Oracle EXAchk in the first user domain (domU) for the fourth cluster.
-
Run Oracle EXAchk in the first management domain (dom0).
The audit checks performed by Oracle EXAchk are specified in the following table:
Table 2-8 Audit Checks Performed by Oracle EXAchk
| Where to Install and Run Oracle EXAchk | Audit Checks Performed |
|---|---|
|
Management domain (dom0) |
Hardware and operating system level checks for:
|
|
User domain (domU) |
Operating system level checks for user domains, and checks for Oracle Grid Infrastructure and Oracle Database |
Oracle EXAchk Command Line Options
Oracle EXAchk requires no special command line options. It automatically detects that it is running in an Oracle Exadata Database Machine Oracle VM environment and whether it is running in a management domain or user domain and performs the applicable audit checks. For example, in the simplest case, you can run Oracle EXAchk with no command line options:
./exachk
When Oracle EXAchk is run in the management domain, it performs audit checks on all database servers, storage servers, and InfiniBand switches accessible through the InfiniBand network.
To run Oracle EXAchk on a subset of servers or switches, use the following command line options:
Table 2-9 Command Line Options for Oracle EXAchk
| Option | Description |
|---|---|
|
|
Specifies a comma-separated list of database servers. |
|
|
Specifies a comma-separated list of storage servers. |
|
|
Specifies a comma-separated list of InfiniBand switches. |
For example, for an Oracle Exadata Database Machine Full Rack where only the first Quarter Rack is configured for virtualization, but all components are accessible through the InfiniBand network, you can run a command similar to the following from the database server dm01adm01:
./exachk -clusternodes dm01adm01,dm01adm02
-cells dm01celadm01,dm01celadm02,dm01celadm03
-ibswitches dm01swibs0,dm01sw-iba0,dm01sw-ibb0
Related Topics
2.10 Extending LVM Partitions
Logical Volume Manager (LVM) provides flexibility to reorganize the partitions in the database servers.
Note:
-
Keep at least 1 GB of free space in the
VGExaDbvolume group. This space is used for the LVM snapshot created by thedbnodeupdate.shutility during software maintenance. -
If you make snapshot-based backups of the
/(root) and/u01directories by following the steps in Creating a Snapshot-Based Backup of Oracle Linux Database Server, then keep at least 6 GB of free space in theVGExaDbvolume group.
This section contains the following topics:
2.10.1 Extending the root LVM Partition
The procedure for extending the root LVM partition depends on your Oracle Exadata System Software release.
The following sections contain the procedures, based on the Oracle Exadata System Software release:
2.10.1.1 Extending the root LVM Partition on Systems Running Oracle Exadata System Software Release 11.2.3.2.1 or Later
The following procedure describes how to extend the size of the root (/) partition on systems running Oracle Exadata System Software release 11.2.3.2.1 or later:
Note:
-
This procedure does not require an outage on the server.
-
For management domain systems, the active and inactive Sys LVM's are
LVDbSys2andLVDbSys3instead ofLVDbSys1andLVDbSys2. -
Make sure that
LVDbSys1andLVDbSys2are sized the same.
2.10.1.2 Extending the root LVM Partition on Systems Running Oracle Exadata System Software Earlier than Release 11.2.3.2.1
The following procedure describes how to extend the size of the root (/) partition on systems running Oracle Exadata System Software earlier than release 11.2.3.2.1:
Note:
-
This procedure requires the system to be offline and restarted.
-
Keep at least 1 GB of free space in the
VGExaDbvolume group to be used for the LVM snapshot created by thedbnodeupdate.shutility during software maintenance. If you make snapshot-based backups of the/(root) and/u01directories by following the steps in "Creating a Snapshot-Based Backup of Oracle Linux Database Server", then keep at least 6 GB of free space in theVGExaDbvolume group. -
For management domain systems, active and inactive Sys LVM's are
LVDbSys2andLVDbSys3instead ofLVDbSys1andLVDbSys2. -
Make sure
LVDbSys1andLVDbSys2are sized the same.
- Restart the system in diagnostic mode.
- Click Open.
A message similar to the following appears on the console.
- Click Open.
2.10.2 Resizing a Non-root LVM Partition
The procedure for resizing a non-root LVM partition depends on your Oracle Exadata System Software release.
The following sections contain the procedures, based on the Oracle Exadata System Software release:
2.10.2.1 Extending a Non-root LVM Partition on Systems Running Oracle Exadata System Software Release 11.2.3.2.1 or Later
This procedure describes how to extend the size of a non-root (/u01) partition on systems running Oracle Exadata System Software release 11.2.3.2.1 or later.
This procedure does not require an outage on the server.
2.10.2.2 Extending a Non-root LVM Partition on Systems Running Oracle Exadata System Software Earlier than Release 11.2.3.2.1
This procedure describes how to extend the size of a non-root (/u01) partition on systems running Oracle Exadata System Software earlier than release 11.2.3.2.1.
In this procedure, /dev/VGExaDb/LVDbOra1 is mounted at /u01.
Note:
-
Keep at least 1 GB of free space in the
VGExaDbvolume group. This space is used for the LVM snapshot created by thedbnodeupdate.shutility during software maintenance. -
If you make snapshot-based backups of the
/(root) and/u01directories by following the steps in Creating a Snapshot-Based Backup of Oracle Linux Database Server, then keep at least 6 GB of free space in theVGExaDbvolume group.
2.10.2.3 Reducing a Non-root LVM Partition on Systems Running Oracle Exadata System Software Release 11.2.3.2.1 or Later
You can reduce the size of a non-root (/u01) partition on systems running Oracle Exadata System Software release 11.2.3.2.1 or later.
Note:
-
This procedure does not require an outage on the server.
-
It is recommended that you back up your file system before performing this procedure.
2.10.3 Extending the Swap Partition
This procedure describes how to extend the size of the swap (/swap) partition.
Note:
This procedure requires the system to be offline and restarted.
Keep at least 1 GB of free space in the VGExaDb volume group to be used for the Logical Volume Manager (LVM) snapshot created by the dbnodeupdate.sh utility during software maintenance. If you make snapshot-based backups of the / (root) and /u01 directories by following the steps in "Creating a Snapshot-Based Backup of Oracle Linux Database Server", then keep at least 6 GB of free space in the VGExaDb volume group.
2.11 Creating a Snapshot-Based Backup of Oracle Linux Database Server
A backup should be made before and after every significant change to the software on the database server. For example, a backup should be made before and after the following procedures:
-
Application of operating system patches
-
Application of Oracle patches
-
Reconfiguration of significant operating parameters
-
Installation or reconfiguration of significant non Oracle software
This section contains the following topics:
2.11.1 Creating a Snapshot-Based Backup of Oracle Linux Database Server with Uncustomized Partitions
This procedure describes how to take a snapshot-based backup. The values shown in the procedure are examples.
If you have not customized the database server partitions from their original shipped configuration, then use the procedures in this section to take a backup and use the backup to restore the database server using the backup.
Note:
-
The recovery procedure restores the exact partitions, including the name and sizes, as they were originally shipped. If you modified the partitions in any way, then you cannot use this procedure. Modifications include changing sizes, renaming, addition or removal of partitions.
-
All steps must be performed as the
rootuser.
2.11.2 Creating a Snapshot-Based Backup of Oracle Linux Database Server with Customized Partitions
When you have customized the partitions, the procedure to back up is the same as the procedure used for non-customized database servers, with the following exceptions:
-
You must add any additional partitions similar to
/u01to the backup. -
If any partitions were renamed, then use the names defined for your environment. For example, if
/u01was renamed to/myown_u01, then use/myown_u01in the commands.
2.12 Backing up the Management Domain (dom0) and User Domains (domU) in an Oracle Virtual Server Deployment
In an Oracle Virtual Server deployment, you need to back up the management domain, dom0, and the user domains (domU's):
2.12.1 Backing up the Management Domain dom0 Using Snapshot-Based Backup
This procedure describes how to take a snapshot-based backup of the management domain, dom0.
The logical volume /dev/VGExaDb/LVDoNotRemoveOrUse is a placeholder to make sure there is always free space available to create a snapshot. If you run dbserver_backup.sh, then the placeholder LVM is removed by the script, the free space is used for a snapshot, and the LVM is re-created after the snapshot is created. If you follow the manual procedure described here, then you have to perform all these tasks manually.
The values shown in the steps below are examples. All steps must be performed as the root user.
2.12.2 Backing up the User Domains
There are two ways to back up the user domains:
-
Method 1: Back up the storage repository using Oracle Cluster File System (OCFS) reflinks to get a consistent backup
This method backs up the storage repository that is the
/EXAVMIMAGESocfs2 file system.You can back up the entire
/EXAVMIMAGES, which will back up all the user domains managed by the dom0, or you can select which user domains you want to back up. The user domains are located in the/EXAVMIMAGES/GuestImages/userdirectories.This method provides a more robust and a comprehensive backup than method 2. Method 2 provides a quicker and an easier backup method, especially in role separated environments.
Method 1 is ideal where a dom0 administrator is responsible for user domain backups.
-
Method 2: Back up a user domain using snapshot-based backup
This method backs up a single user domain using snapshot-based backup from inside the user domain.
Method 2 is ideal where a user domain administrator is responsible for the user domain backups.
2.12.2.1 Method 1: Back up All the User Domains
You can back up the storage repository that is the /EXAVMIMAGES OCFS2 file system
The procedure shown here backs up all the user domains.
The sample commands below may be used as a reference for the operations listed in steps 2 through 6 above.
## Create the Backup Directory
find /EXAVMIMAGES -type d|grep -v 'lost+found'|awk '{print "mkdir -p /EXAVMIMAGES/Backup"$1}'|sh
## Pause user domains
xm list|egrep -v '^Domain-0|^Name'|awk '{print "xm pause",$2}'|sh
## Create Reflinks for all the files to be backed up
find /EXAVMIMAGES/ -type f|awk '{print "reflink",$0,"/EXAVMIMAGES/Backup"$0}'|sh
## Unpause user domains
xm list|egrep -v '^Domain-0|^Name'|awk '{print "xm unpause",$2}'|sh;
## Backup from the reflinks
pushd /EXAVMIMAGES/Backup/EXAVMIMAGES
tar -Spcvf /remote_FS/exavmimages-sparse.tar * 1> /tmp/Backup.log 2> /tmp/Backup.err
popd
rm -rf /EXAVMIMAGES/Backup/*
2.12.2.2 Method 2: Back up a User Domain from Inside the User Domain
The following procedure describes how to take a snapshot-based backup of a user domain from inside the user domain.
Note:
This method of backing up a user domain from inside the user domain using LVM snapshots will have limited usage in terms of recovery. Such a backup can only be used for recovery purposes when the user domain is still bootable and allows login as the root user. This means the damage is such that some files have been lost or damaged but can be restored from the tar backup after the user domain has booted up and the / (root) partition and the boot partitions are mounted. If that is not the case and the damage is such that the user domain does not boot, then you need a backup taken using method 1 above to recover the user domain, and you need to perform the recovery procedure at the user domain level using the recovery procedure described below.
All steps are performed from inside the user domain.
The procedure backs up the following:
-
LVDbSys1 lvm
-
LVDbOra1 lvm
-
/bootpartition -
Grid Infrastructure home
-
RDBMS home
All steps must be performed as the root user.
-
Prepare a destination to hold the backup.
# mkdir -p /remote_FS # mount -t nfs -o rw,intr,soft,proto=tcp,nolock ip_address:/nfs_location/ /remote_FS
ip_address is the IP address of the NFS server, and nfs_location is the NFS location holding the backups.
-
Take a snapshot-based backup of the file systems containing / (root) and the /u01 directories, as follows:
-
Create a snapshot named LVDbSys1_snap for the file system containing the root directory. The volume group must have at least 1 GB of free space for the command to succeed.
# lvcreate -L1G -s -n LVDbSys1_snap /dev/VGExaDb/LVDbSys1
-
Label the snapshot.
# e2label /dev/VGExaDb/LVDbSys1_snap DBSYS_SNAP
-
Mount the snapshot.
# mkdir /root/mnt # mount /dev/VGExaDb/LVDbSys1_snap /root/mnt -t ext4
-
Create a snapshot named u01_snap for the
/u01directory.# lvcreate -L256M -s -n u01_snap /dev/VGExaDb/LVDbOra1
-
Label the snapshot.
# e2label /dev/VGExaDb/u01_snap DBORA_SNAP
-
Mount the snapshot.
# mkdir -p /root/mnt/u01 # mount /dev/VGExaDb/u01_snap /root/mnt/u01 -t ext4
-
Change to the directory for the backup.
# cd /root/mnt
-
Create the backup file to back up the two snapshots taken above, the
/bootpartition, the RDBMS home, and the Grid Infrastructure home.# tar -pjcvf /remote_FS/mybackup.tar.bz2 * /boot /u01/app/12.1.0.2/grid /u01/app/oracle/product/12.1.0.2/dbhome_1 > /tmp/backup_tar.stdout 2> /tmp/backup_tar.stderr
-
Check the
/tmp/backup_tar.stderrfile for any significant errors. Errors about failing to tar open sockets, and other similar errors, can be ignored.
-
-
Unmount and remove the snapshots for the file system containing the root directories.
# cd / # umount /root/mnt/u01 # umount /root/mnt # /bin/rmdir /root/mnt # lvremove /dev/VGExaDb/u01_snap # lvremove /dev/VGExaDb/LVDbSys1_snap
-
Unmount the NFS share.
# umount /remote_FS
2.13 Recovering Oracle Linux Database Servers Using a Snapshot-Based Backup
This section describes how to recover a database server file systems running Oracle Linux using a snapshot-based backup after severe disaster conditions happen for the database server, or when the server hardware is replaced to such an extent that it amounts to new hardware. For example, replacing all hard disks leaves no trace of original software on the system. This is similar to replacing the complete system as far as the software is concerned. In addition, it provides a method for disaster recovery of the database servers using an LVM snapshot-based backup taken when the database server was healthy before the disaster condition.
The recovery procedures use the diagnostics.iso image as a virtual CD-ROM to restart the database server in rescue mode using the ILOM. The general workflow includes the following tasks:
-
Recreate the following:
-
Boot partitions
-
Physical volumes
-
Volume groups
-
Logical volumes
-
File system
-
Swap partition
-
-
Activate the swap partition.
-
Ensure the
/bootpartition is the active boot partition. -
Restore the data.
-
Reconfigure GRUB.
-
Restart the server.
The recovery procedures described in this section do not include backup or recovery of Oracle Exadata Storage Servers or database data. Oracle recommends testing the backup and recovery procedures on a regular basis. This section contains the following topics:
-
Recovering Oracle Linux Database Server with Uncustomized Partitions
-
Recovering Exadata X7 Database Servers with Customized Partitions
-
Recovering Exadata X6 or Earlier Database Servers with Customized Partitions
-
Configuring Oracle Exadata Database Machine Eighth Rack Oracle Linux Database Server After Recovery
Note:
Restoring files from tape may require additional drives to be loaded, and is not covered in this chapter. Oracle recommends backing up files to an NFS location, and using existing tape options to back up and recover from the NFS host.
2.13.1 Recovering Oracle Linux Database Server with Uncustomized Partitions
You can recover the Oracle Linux database server from a snapshot-based backup when using uncustomized partitions.
This procedure is applicable when the layout of the partitions, logical volumes, file systems, and their sizes are equal to the layout when the database server was initially deployed.
Caution:
All existing data on the disks is lost during the procedure.- Answer as follows when prompted by the system. The responses are shown in bold.
Note that for Oracle Exadata System Software release 12.1.2.2.0 or later, DHCP is used and you do not have to manually set up the network.
-
If you are using Oracle Exadata System Software release 18.1 or later, running on Oracle Exadata Database Machine X7, then the prompt looks like the following:
Description of the illustration GUID-6FAD894E-2510-4C37-B651-7EF2C9DEC7D2-default.jpg -
If you are using Oracle Exadata System Software release 18.1 or later and restoring through one of the 10GbE Ethernet SFP+ ports on Oracle Exadata Database Machine X3-2, X4-2, X5-2 or X6-2, then the prompt looks like the following:
------------------------------------------------------------------------------ Choose from the following by typing letter in '()': (e)nter interactive diagnostics shell. Use diagnostics shell password to login as root user (reboot or power cycle to exit the shell), (r)estore system from NFS backup archive, Select: r Continue (y/n) [n]: y Rescue password: [INFO ] Enter path to the backup file on the NFS server in format: Enter path to the backup file on the NFS server in format: <ip_address_of_the_NFS_share>:/<path>/<archive_file> For example, 10.124.1.15:/export/nfs/share/backup.2010.04.08.tar.bz2 NFS line: <nfs_ip>:/export/mybackup.tar.bz2 [INFO ] The backup file could be created either from LVM or non-LVM based COMPUTE node [INFO ] Versions below 11.2.1.3.0 do not support LVM based partitioning Use LVM based scheme. (y/n) [y]: y Configure network settings on host via DHCP. (y/n) [y]: n Configure bonded network interface. (y/n) [y]: y IP Address of bondeth0 on this host: <IP address of the DB host> Netmask of bondeth0 on this host: <netmask for the above IP address> Bonding mode:active-backup or 802.3ad [802.3ad]: active-backup Slave interface1 for bondeth0 (ethX) [eth4]: eth4 Slave interface2 for bondeth0 (ethX) [eth5]: eth5 ... [ 354.619610] bondeth0: first active interface up! [ 354.661427] ixgbe 0000:13:00.1 eth5: NIC Link is Up 10 Gbps, Flow Control: RX/TX [ 354.724414] bondeth0: link status definitely up for interface eth5, 10000 Mbps full duplex Default gateway: <Gateway for the above IP address> ------------------------------------------------------------------------------ -
If you are using Oracle Exadata System Software release 12.1.x or 12.2.x, then the prompts look like the following:
------------------------------------------------------------------------------ Use diagnostics shell password to login as root user (reboot or power cycle to exit the shell), (r)estore system from NFS backup archive. Select: r Continue (y/n) [n]: y Rescue password: [INFO: ] Enter path to the backup file on the NFS server in format: Enter path to the backup file on the NFS server in format: <ip_address_of_the_NFS_share>:/<path>/<archive_file> For example, 10.124.1.15:/export/nfs/share/backup.2010.04.08.tar.bz2 NFS line: <nfs_ip>:/export/mybackup.tar.bz2 [INFO: ] The backup file could be created either from LVM or non-LVM based COMPUTE node [INFO: ] Versions below 11.2.1.3.0 do not support LVM based partitioning Use LVM based scheme. (y/n) [y]: y ------------------------------------------------------------------------------ -
If you are using Oracle Exadata System Software release earlier than 12.1.2.2.0, then the prompts look like the following
------------------------------------------------------------------------------ Choose from following by typing letter in '()': (e)nter interactive diagnostics shell. Must use credentials from Oracle support to login (reboot or power cycle to exit the shell), (r)estore system from NFS backup archive, Select:r Are you sure (y/n) [n]:y The backup file could be created either from LVM or non-LVM based compute node versions below 11.2.1.3.1 and 11.2.2.1.0 or higher do not support LVM based partitioning use LVM based scheme(y/n):y Enter path to the backup file on the NFS server in format: ip_address_of_the_NFS_share:/path/archive_file For example, 10.10.10.10:/export/operating_system.tar.bz2 NFS line:<nfs_ip>:/export/mybackup.tar.bz2 IP Address of this host:IP address of the DB host Netmask of this host:netmask for the above IP address Default gateway:Gateway for the above IP address. If there is no default gateway in your network, enter 0.0.0.0. ------------------------------------------------------------------------------
When the recovery completes, the log in screen appears.
-
2.13.2 Recovering Exadata X7 Database Servers with Customized Partitions
This procedure describes how to recover an Oracle Exadata Database Machine X7-2 or later Oracle Linux database server from a snapshot-based backup when using customized partitions.
Note:
This task assumes you are running Oracle Exadata System Software release (18.1.0) or greater.2.13.3 Recovering Exadata X6 or Earlier Database Servers with Customized Partitions
This procedure describes how to recover Oracle Exadata Database Servers for Oracle Exadata Database Machine X6-2 or earlier running Oracle Linux from a snapshot-based backup when using customized partitions.
The steps are the same as restoring non-customized partitions until you are prompted to choose from enter interactive diagnostics shell and restore system from NFS backup archive options after booting the database server using diagnostics.iso.
2.13.4 Configuring Oracle Exadata Database Machine Eighth Rack Oracle Linux Database Server After Recovery
After the Oracle Linux database server in Oracle Exadata Database Machine Eighth Rack has been re-imaged, restored, or rescued, you can then reconfigure the eighth rack. Use one of the following procedures:
2.13.4.1 Configuring Eighth Rack On X3-2 or Later Machines Running Oracle Exadata Storage Server Release 12.1.2.3.0 or Later
The following procedure should be performed after Oracle Linux database server in Oracle Exadata Database Machine Eighth Rack has been re-imaged, restored, or rescued.
For X3-2 systems, use this method only if you are running Oracle Exadata Storage Server Release 12.1.2.3.0 or later.
2.14 Recovering in an Oracle VM Server Deployment
The following procedures describe how to recover an Oracle VM Server from a snapshot-based backup when severe disaster conditions damage the Oracle VM Server, or when the server hardware is replaced to such an extent that it amounts to new hardware. For example, replacing all hard disks leaves no trace of original software on the system. This is similar to replacing the complete system as far as the software is concerned. In addition, it provides a method for disaster recovery of the database servers using an LVM snapshot-based backup taken when the database server was healthy before the disaster condition.
The recovery procedures use the diagnostics.iso image as a virtual CD-ROM to restart the Oracle VM Server in rescue mode using the Integrated Lights Out Manager (ILOM). At a high-level, the steps look like this:
-
Re-create the following:
-
Boot partitions
-
Physical volumes
-
Volume groups
-
Logical volumes
-
File system
-
Swap partition
-
-
Activate the swap partition.
-
Ensure the
/bootpartition is the active boot partition. -
Restore the data.
-
Reconfigure GRUB.
-
Restart the server.
The recovery procedures described in this section do not include backup or recovery of Oracle Exadata Storage Servers or Oracle Database data. Oracle recommends testing the backup and recovery procedures on a regular basis.
2.14.1 Scenario 1: Recovering an Oracle VM Server Along with All Its User Domains from Backup
You can recover the Oracle VM Server and all its user domains from a backup.
The following procedures step you through the recovery process. Chose one of the following procedures, based on the version of Oracle Exadata System Software that is installed on your system.
Note:
All existing data on the disks is lost during these procedures.
2.14.1.1 Recovering an Oracle Virtual Server Along with All Its User Domains (Releases Prior to 12.2.1.1.0)
You can recover an Oracle Virtual Server (OVS) from a snapshot-based backup when severe disaster conditions damage the OVS, or when the server hardware is replaced to such an extent that it amounts to new hardware.
- Attach the
/opt/oracle.SupportTools/diagnostics.isofile from any healthy database server as virtual media to the ILOM of the Oracle Virtual Server to be restored.The following example shows how to set up a virtual CD-ROM using the ILOM interface:- Open the
diagnostics.isofile.
- Open the
At this point all the user domains should come up along with Oracle Grid Infrastructure and the database instances, and the database instances should join the Oracle RAC cluster formed by the other surviving Oracle Virtual Server nodes.
2.14.1.2 Recovering an Oracle Virtual Server Along with All Its User Domains (Releases 12.2.1.1.0 and Later)
You can recover an Oracle Virtual Server (OVS) from a snapshot-based backup when severe disaster conditions damage the OVS, or when the server hardware is replaced to such an extent that it amounts to new hardware.
- Attach the
/opt/oracle.SupportTools/diagnostics.isofile from any healthy database server as virtual media to the ILOM of the Oracle Virtual Server to be restored.The following example shows how to set up a virtual CD-ROM using the ILOM interface:- Open the
diagnostics.isofile.
- Open the
At this point all the user domains should come up along with Oracle Grid Infrastructure and the database instances, and the database instances should join the Oracle RAC cluster formed by the other surviving Oracle Virtual Server nodes.
2.14.2 Scenario 2: Reimaging dom0 and Restoring User Domains from Backups
The following procedure can be used when the OVS/dom0 is damaged beyond repair and no backup exists for the dom0, but there is a backup available of the storage repository (/EXAVMIMAGES file system) housing all the user domains. This procedure reimages the dom0 and reconstructs all the user domains.
-
Re-image the OVS with the image used in the other OVS/dom0's in the rack using the procedure described in "Re-Imaging Oracle Exadata Database Servers".
-
Run the following commands:
# /opt/oracle.SupportTools/switch_to_ovm.sh # /opt/oracle.SupportTools/reclaimdisks.sh –free –reclaim
-
If the recovery is on Oracle Exadata Database Machine eighth rack, then perform the procedure described in "Configuring Oracle Exadata Database Machine Eighth Rack Oracle Linux Database Server After Recovery".
-
Rebuild the ocfs2 file system on the
/dev/sda3partition.# umount /EXAVMIMAGES # mkfs -t ocfs2 -L ocfs2 -T vmstore --fs-features=local /dev/sda3 --force
-
Mount the ocfs2 partition
/dev/sda3on/EXAVMIMAGES.# mount -t ocfs2 /dev/sda3 /EXAVMIMAGES
-
Mount the backup NFS server to restore the
/EXAVMIMAGESfile system which holds the user domain images:# mkdir -p /remote_FS # mount -t nfs -o ro,intr,soft,proto=tcp,nolock nfs_ip:/location_of_backup /remote_FS
-
Restore the
/EXAVMIMAGESfile system.# tar -Spxvf /remote_FS/backup-of-exavmimages.tar -C /EXAVMIMAGESNote:
The restore process of storage repository restores the user domain specific files (files under
/EXAVMINAGES/GuestImages/<user_domain>/) as regular files and not as ocfs2 reflinks which is what these files in the storage repository originally got created as at the time of user domain creation. Consequently, the space usage in/EXAVMINAGESmay go up after the restoration process compared to the original space usage at the time of the backup. -
Set up the network bridges manually.
-
Determine the version of the ovmutils rpm:
# rpm -qa|grep ovmutils
-
If the version of the ovmutils rpm is earlier than 12.1.2.2.0, perform these steps:
-
Back up
/opt/exadata_ovm/exadata.img.domu_maker. You will need the backup copy later.# cp /opt/exadata_ovm/exadata.img.domu_maker /opt/exadata_ovm/exadata.img.domu_maker-orig
-
Open the
/opt/exadata_ovm/exadata.img.domu_makerfile in a text editor such as vi, and search for "g_do_not_set_bridge=yes". It should be located a few lines below the case statement option "network-discovery)".Change it to "g_do_not_set_bridge=no".
Save and exit
/opt/exadata_ovm/exadata.img.domu_maker. -
Run
/opt/exadata_ovm/exadata.img.domu_makermanually for every xml file in the/EXAVMIMAGES/confdirectory.# cd /EXAVMIMAGES/conf # ls -1|while read file; do /opt/exadata_ovm/exadata.img.domu_maker network-discovery $file /tmp/netdisc-$file; done
-
Restore
/opt/exadata_ovm/exadata.img.domu_makerfrom the backup copy.# cp /opt/exadata_ovm/exadata.img.domu_maker-orig /opt/exadata_ovm/exadata.img.domu_maker
-
-
If the version of the ovmutils rpm is 12.1.2.2.0 or later, run the following command:
# /opt/exadata_ovm/exadata.img.domu_maker add-bonded-bridge-dom0 vmbondeth0 eth4 eth5
-
-
For each user domain directory in the
/EXAVMIMAGES/GuestImagesdirectory, perform the steps below.-
Get the UUID of the user domain.
# grep ^uuid /EXAVMIMAGES/GuestImages/<user domain hostname>/vm.cfg|awk -F"=" '{print $2}'|sed s/"'"//g|sed s/" "//gThe command returns the
uuidvalue, which is used in the commands below. -
# mkdir -p /OVS/Repositories/uuid -
# ln -s /EXAVMIMAGES/GuestImages/user_domain_hostname/vm.cfg/OVS/Repositories/uuid/vm.cfg -
# ln -s /OVS/Repositories/uuid/vm.cfg /etc/xen/auto/user_domain_hostname.cfg -
# mkdir VirtualDisks -
# cd VirtualDisks -
Create four symbolic links in this directory using the four disk image names in the
vm.cfgfile, pointing to the four ".img" files in/EXAVMIMAGES/GuestImages/user_domain_hostnamedirectory.For example, below is a sample disk entry in a sample
vm.cfgfile in a/OVS/Repositories/uuiddirectory:disk = ['file:/OVS/Repositories/6e7c7109c1bc4ebba279f84e595e0b27/VirtualDisks/dfd641a1c6a84bd69643da704ff98594.img,xvda,w','file:/OVS/Repositories/6e7c7109c1bc4ebba279f84e595e0b27/VirtualDisks/d349fd420a1e49459118e6a6fcdbc2a4.img,xvdb,w','file:/OVS/Repositories/6e7c7109c1bc4ebba279f84e595e0b27/VirtualDisks/8ac470eeb8704aab9a8b3adedf1c3b04.img,xvdc,w','file:/OVS/Repositories/6e7c7109c1bc4ebba279f84e595e0b27/VirtualDisks/333e7ed2850a441ca4d2461044dd0f7c.img,xvdd,w']
You can list the four ".img" files in the
/EXAVMIMAGES/GuestImages/user_domain_hostnamedirectory:ls /EXAVMIMAGES/GuestImages/user_domain_name/*.img/EXAVMIMAGES/GuestImages/user_domain_name/System.img /EXAVMIMAGES/GuestImages/user_domain_name/grid12.1.0.2.2.img /EXAVMIMAGES/GuestImages/user_domain_name/db12.1.0.2.2-3.img /EXAVMIMAGES/GuestImages/user_domain_name/pv1_vgexadb.img
In this case, the commands below may be used to create the four symbolic links where dbm01db08vm01 is the user domain hostname:
# ln -s /EXAVMIMAGES/GuestImages/dbm01db08vm01/System.img $(grep ^disk /EXAVMIMAGES/GuestImages/dbm01db08vm01/vm.cfg|awk -F":" '{print $2}'|awk -F"," '{print $1}'|awk -F"/" '{print $6}') # ln -s /EXAVMIMAGES/GuestImages/dbm01db08vm01/grid12.1.0.2.2.img $(grep ^disk /EXAVMIMAGES/GuestImages/dbm01db08vm01/vm.cfg|awk -F":" '{print $3}'|awk -F"," '{print $1}'|awk -F"/" '{print $6}') # ln -s /EXAVMIMAGES/GuestImages/dbm01db08vm01/db12.1.0.2.2-3.img $(grep ^disk /EXAVMIMAGES/GuestImages/dbm01db08vm01/vm.cfg|awk -F":" '{print $4}'|awk -F"," '{print $1}'|awk -F"/" '{print $6}') # ln -s /EXAVMIMAGES/GuestImages/dbm01db08vm01/pv1_vgexadb.img $(grep ^disk /EXAVMIMAGES/GuestImages/dbm01db08vm01/vm.cfg|awk -F":" '{print $5}'|awk -F"," '{print $1}'|awk -F"/" '{print $6}')
-
-
Bring up each user domain.
# xm create /EXAVMIMAGES/GuestImages/user_domain_hostname/vm.cfg
At this point all the user domain should come up along with the Grid Infrastructure and the database instances in them and join the Oracle RAC cluster formed by the other surviving OVS nodes.
2.14.3 Scenario 3: Restoring and Recovering User Domains from Snapshot Backups
Use this procedure to restore lost or damaged files of a user domain using a snapshot-based user domain backup taken from inside a user domain. The user domain backup was created using the procedure described in "Method 2: Back up a User Domain from Inside the User Domain".
2.15 Updating Oracle Linux Database Servers
Starting with Oracle Exadata System Software release 12.1.2.2.0, you can run patchmgr to orchestrate updates.
Starting with Oracle Exadata System Software release 11.2.3.1.0, the database server update procedure uses the Unbreakable Linux Network (ULN) for distribution of database operating system and firmware updates. The updates are distributed as multiple rpms. The dbnodeupdate.sh utility is used to apply the updates. The utility performs the following tasks:
-
Uses the built-in
dbserver_backup.shscript to perform a backup of the file system hosting the operating system before updating the server when the database servers use a Logical Volume Manager (LVM) configuration. -
Automates all preparation, update, and validation steps including the stopping of the databases and Oracle Grid Infrastructure stack, Oracle Enterprise Manager Cloud Control agents, and unmounting remote network mounts that might exist.
-
Applies Oracle best practices and workarounds for the latest known issues.
-
Verifies that the update was successful, relinks the Oracle binaries, and starts the Oracle stack.
For environments that cannot use ULN, an ISO repository for each Oracle Exadata System Software release is available.
The dbnodeupdate.sh utility is available from My Oracle Support note 1553103.1. The utility supports all hardware generations, as well as Oracle Exadata Database Servers running Oracle VM Server (dom0) and Oracle VM (domU). The dbnodeupdate.sh utility also supports updates from Oracle Linux 5 to Oracle Linux 6. The Oracle Exadata System Software patch README files specify if the patch itself is applicable for a particular hardware generation. Always review the support note to ensure that the latest release of the utility is used for the update.
Note:
The ISO image method is not intended to be a complete replacement of a local ULN mirror. The ISO image contains only the Oracle Exadata channel content. No other content from other ULN channels is in the ISO image. If your environment uses other Linux packages for which updates are not contained in the Oracle Exadata channel, then you should also update those packages using ULN channels, as needed.
This section contains the following topics:
2.15.1 Paths for Updating Oracle Exadata Database Servers
Oracle Exadata Database Servers running Oracle Linux 5.5 or later and Oracle Exadata System Software release 11.2.2.4.2 or later must use the dbnodeupdate.sh utility to update the database servers.
Oracle Linux database servers running Oracle Exadata System Software releases earlier than 11.2.2.4.2 should first update to Oracle Linux 5.5 and Oracle Exadata System Software 11.2.2.4.2 or later before using the dbnodeupdate.sh utility.
The following table shows the update paths for the database servers based on the Oracle Exadata System Software and Oracle Linux releases.
Note:
If output from the uname -r command on the database server shows a kernel release that is earlier than or equal to release 2.6.18-128.1.16.0.1, then the database server is running Oracle Linux 5.3.
Table 2-10 Update Paths for Database Servers
| Currently Installed Oracle Exadata System Software Release | Currently Installed Oracle Linux Release | Recommended Action |
|---|---|---|
|
Release 11.2.2.4.2 or later |
Release 5.5 or later |
Update to the target release as described in Overview of Oracle Exadata Database Servers Updates. |
|
Releases earlier than 11.2.2.4.2. |
Release 5.5 or later |
|
|
Releases earlier than 11.2.2.4.2 |
Releases earlier than 5.5 |
|
Note:
-
Oracle Exadata Database Servers currently running a non-Unbreakable Enterprise Kernel (UEK) kernel on releases 11.2.3.2.0 and later, updating to a release earlier than Oracle Exadata System Software release 11.2.3.3.0, the database servers will remain on a non-UEK kernel when using the
dbnodeupdate.shutility. -
When there is a requirement to use the UEK kernel in a release earlier than release 11.2.3.3.0, and when currently on a non-UEK kernel, then use the
dbnodeupdate.shutility to do the update and later convert manually to UEK after the update is complete. Refer to My Oracle Support note 1524411.1 for additional information. -
Starting with Oracle Exadata System Software release 11.2.3.3.0 only UEK kernels are supported. For this reason, the
dbnodeupdate.shutility will update any non-UEK system directly to the UEK2 kernel when updating to Oracle Exadata System Software release 11.2.3.3.0 or later. -
Regular Oracle Exadata Database Server updates as described in My Oracle Support note 888828.1 can only be applied when the date-stamp of the target release is newer than the date-stamp of the release currently running. Updates from images running a release with a date-stamp newer than the target release will be blocked.
For example: Updating Oracle Exadata Database Server image 12.1.1.1.2.150411 to image 12.1.2.1.0.141206.1 is blocked because the 12.1.1.1.2.1 image (with date-stamp 150411) has a date-stamp newer than image 12.1.2.1.0 (141206), even though release 12.1.2.1.0 seems to be more recent than release 12.1.1.1.2 based on the release number. In situations like this, you should update to a maintenance release that has a newer date-stamp for the same major release.
Updating individual non-Exadata operating system packages
If you want to update individual non-Exadata operating system packages, you can set up a local yum mirror of the Oracle Linux "latest" channel for your system and run "yum update <packagename>".
Always be careful of any other packages that might be pulled in as well.
You may have to remove the exadata-sun-computenode-exact rpm first:
rpm -e exadata-sun-computenode-exact
If you do not have a local yum mirror, then download the individual packages from ULN or public-yum, and update the specific package using:
rpm -uvh package-names
Caution:
Never run the yum updatecommand without specifying any package name, because this will cause many packages and dependencies to be pulled in, making unintentional updates, which can make your Oracle Exadata Database Machine unusable and make it much harder or impossible to update to the next Oracle Exadata System Software release.
Related Topics
- Exadata 11.2.2.4.2 release and patch (13513611) for Exadata 11.1.3.3, 11.2.1.2.x, 11.2.2.2.x, 11.2.2.3.[0-5], 11.2.2.4.[0-1]. (My Oracle Support Doc ID 1388400.1)
- Updating key software components on database hosts to match those on the cells (My Oracle Support Doc ID 1284070.1)
- Reconfigure an Exadata Database Server to Use UEK Kernel After Previously Choosing to use non-UEK Kernel (My Oracle Support Doc ID 1524411.1)
- Exadata Database Machine and Exadata Storage Server Supported Versions (My Oracle Support Doc ID 888828.1)
- Customizing Database Server Packages and Understanding Minimum and Exact Dependencies
- Responses to common Exadata security scan findings (My Oracle Support Doc ID 1405320.1)
2.15.2 Overview of Oracle Exadata Database Servers Updates
This topic provides a high-level overview of the procedure to update the Oracle Exadata Database Servers to a new release.
The information in this section applies to database servers that meet the following requirements:
-
Oracle Exadata System Software release 11.2.2.4.2 or later is installed.
-
Oracle Linux 5.5 or later is installed. Having Oracle Exadata System Software release 11.2.3.1.0 and later satisfies this requirement.
If the database servers do not meet the preceding requirements, then refer to Table 2-10 for the prerequisite steps that must be done to meet the requirements.
The following procedure describes how to update the Oracle Linux database servers to a new release:
-
Prepare and populate the yum repository with the Oracle Exadata channel content for the target release. The ULNchannel name to use for the release is listed in the specific README file for the patch. When not using a ULN channel, an ISO image can be used.
-
Perform the prerequisite check, update, and post-patching steps, depending on the current Oracle Exadata System Software release as follows:
-
Oracle Database running with Oracle Exadata System Software release 11.2.3.1.0 and later: Updating Database Servers Previously Patched or Imaged to Oracle Exadata System Software Release 11.2.3.n.n or 12.1.n.n.n.
-
Oracle Database running with Oracle Exadata System Software release 11.2.2.4.2: Updating Database Servers Running Oracle Exadata System Software Release 11.2.2.4.2.
-
2.15.3 Preparing and Populating the yum Repository with the Oracle Exadata Channel Content
The following methods can be used to create and use a yum repository containing the Oracle Exadata channel content. The dbnodeupdate.sh utility then uses the yum repository to update the database servers in Oracle Exadata Database Machine. Each method is described in detail in the following sections.
2.15.3.1 Setting Up a Local Mirror of the ULN Oracle Exadata Channel on a Separate Server
This method is recommended for the following conditions:
-
There is a large number of database servers to update.
-
The single repository server is accessible to all the database servers.
-
An infrastructure exists for building a local ULN mirror on a separate Linux server.
-
Database servers have customized software that requires updates from other ULN channels.
The mirror is a separate Linux server that holds the downloaded channel data with the Oracle Exadata release from ULN that you want to update to. The database servers connect to the local mirror (repository) to retrieve updates.
Caution:
Do not use a database server as the local ULN mirror. Doing so may lead to system failures due to dependency conflicts between the packages required by ULN to construct the repository, and the packages installed or updated on the database server. If a separate server is not available, then use the ISO image method.
When setting up a local repository for the first time, the additional required Linux subscriptions depend on the Enterprise Linux or Oracle Linux release being used on the server hosting the repository. The server should subscribe to the following additional ULN channels in order for the repositories to be built:
-
For 64-bit Enterprise Linux 4 systems:
el4_x86_64_latest, andel4_x86_64_addonsare required. -
For 64-bit Enterprise Linux 5 systems:
el5_x86_64_latest, andel5_x86_64_addonsare required. -
For 64-bit Oracle Linux 5 systems:
ol5_x86_64_latest, andel5_x86_64_addonsare required. -
For 64-bit Oracle Linux 6 systems:
ol6_x86_64_latest, andol6_x86_64_addonsare required.
2.15.3.1.1 Creating and Maintaining a Local ULNMirror
This procedure describes how to create and maintain a local ULN mirror as the yum repository.
Related Topics
2.15.3.1.2 Verifying the Local ULN Mirror
The yum mirror creation procedure described in the previous section provides a script that is used to populate the local mirror. Running the script starts the download of rpms from ULN to a local directory.
The web server's Document Root directory should be configured to point to this location on disk so that the Oracle Exadata channel content is made available to the Oracle Exadata Database Servers using the HTTP web server.
The URL specified as the repository location to the dbnodeupdate.sh utility should always be at the same level as the repodata directory, as shown in the following graphic:
The following command shows how to verify the repository as the root user:
[root@dm01 ]# ./dbnodeupdate.sh -u -l http://yum-repo/yum/EngineeredSystems/exadata/dbserver/12.1.1.1.0/base/x86_64/ -v
Note:
The database node must be running Oracle Exadata System Software release 12.1.2.2.0 or later to access yum repositories listening only on IPv6 addresses.
2.15.3.2 Downloading the ISO Image of the ULN Oracle Exadata Channel and Passing the Location of the Compressed Image to the dbnodeupdate.sh Utility
Starting with Oracle Exadata System Software release 11.2.3.1.0 and later, an ISO image of the Oracle Exadata channel is available as a downloadable patch from My Oracle Support.
The location of the compressed file can be passed directly to the dbnodeupdate.sh utility. This method is recommend for the following conditions:
-
There is no separate server available to be used as a repository server.
-
The database servers do not have customized software that requires updates from additional ULN channels.
-
The simplest method is preferred.
Note:
The ISO image method is not intended to be a complete replacement of a local ULN mirror. The ISO image contains only the Oracle Exadata channel content. No other content from other ULN channels is in the ISO image. If your environment uses other Linux package for which updates are not contained in the Oracle Exadata channel, then you should also update those packages using ULN channels, as needed.
The following procedure describes how to download and verify the ISO image:
2.15.3.3 Downloading the ISO Image and Making It Available on a Web Server
Starting with Oracle Exadata System Software release 11.2.3.1.0 and later, an ISO image of the Oracle Exadata channel is available as a downloadable patch from My Oracle Support.
The ISO image can be downloaded, uncompressed, and put on web server that has HTTP connectivity to every Oracle Exadata Database Server. Refer to My Oracle Support note 888828.1 for links to the release-specific patch information. This method is recommend for the following conditions:
-
There is a large number of database servers to update.
-
The single repository server is accessible to all the database servers.
-
The database servers do not have customized software that requires updates from additional ULN channels.
The following procedure describes how to make the ISO image of the Oracle Exadata channel content available on a Linux-based server running Apache HTTP Server. Other operating systems can be used, but the instructions are not included in this document. In the procedure, the ISO image is copied to the /u01/app/oracle/stage directory, and Document Root entry in the web server document is /var/www/html.
Note:
Oracle recommends that a separate server be used as the web server. An Oracle Exadata Database Server can be used, but the httpd package is not a standard Oracle Exadata Database Machine package. If a database server will be used, then Apache HTTP Server must be installed on the server before performing the following procedure.
2.15.4 Reducing Downtime of Database Updates
Downtime can be reduced by making a backup before and running the prerequisite checks before starting the update.
When the actual update occurs, the backup can be skipped. Backups can only be made when no remote network mounts, such as NFS, are active. The prerequisite checks should always be run days before starting any downtime for the actual update.
Note:
When updating Logical Volume Manager (LVM)-enabled systems from Oracle Linux 5 to Oracle Linux 6, a backup is done as part of the actual update. The backup cannot be skipped.
All warnings and failures should be addressed before taking the downtime. Note the following when updating the database servers:
-
Updating database servers previously patched or imaged to release 11.2.3.n.n or 12.1.n.n.n:
Only minimum-dependency failures, and not having a backup of the current image before updating to Oracle Linux 6 will block the update from proceeding. When the check for minimum-dependencies fails, check the
dbnodeupdate.logfile in the/var/log/cellosdirectory for more details on why the dependencies fail. Dependency checks are not be performed when updating to Oracle Linux 6. After resolving the warnings and failures, re-run the prerequisite check. -
Updating database servers running release 11.2.2.4.2:
Dependency checks are not performed when updating from release 11.2.2.4.2, also
obsolete.lstandexclusion.lstlist functionality is disabled. After resolving the warnings and failures, re-run the prerequisite check.
2.15.5 Updating Database Servers Previously Patched or Imaged to Oracle Exadata System Software Release 11.2.3.n.n or 12.1.n.n.n
The dbnodeupdate.sh utility is used to update the database servers that are running Oracle Exadata System Software release 11.2.3.n.n, and release 12.1.n.n.n. The procedures in this section assume that the database servers are running an Oracle Exadata System Software release later than 11.2.2.4.2, and that the update is available either as local ULN mirror or as a local ISO image.
Note:
Two components are required when using the dbnodeupdate.sh utility:
-
The ULN mirror or compressed ISO file of the patch
-
The
dbnodeupdate.shutility
This section contains the following topics:
2.15.5.1 Customizing Database Server Packages and Understanding Minimum and Exact Dependencies
System administrators are allowed to customize the operating system on the database servers by updating existing packages supplied by Oracle Exadata System Software to newer releases or by installing additional packages — as long as the customization does not impair Oracle Exadata System Software functionality.
Note:
Kernel and InfiniBand packages should not be updated or removed unless instructed by Oracle Support Services.
When updating to any release equal to or later than Oracle Exadata System Software release 11.2.3.3.0, the yum update performed using the dbnodeupdate.sh utility works with minimum and exact dependencies. These dependencies are enforced during prerequisite checking and during the update phase to help administrators stay exactly at or close to the original Oracle Exadata System Software release when customizing the system.
The dependencies are enforced by the exadata-sun-computenode-exact RPM and the exadata-sun-computenode-minimum RPM as follows:
-
The
exadata-sun-computenode-exactRPM ensures that a specific release of Oracle-supplied packages is allowed during the update. -
The
exadata-sun-computenode-minimumRPM ensures that a specific or later release of Oracle-supplied packages is allowed during the update.
By having the exact dependencies enforced by exadata-sun-computenode-exact RPM, the system appear as if it was freshly-imaged to the newer release because the Oracle Exadata System Software packages are exactly as the same release.
The exadata-sun-computenode-minimum RPM sets the minimum dependencies, and enforces that all packages are installed but at the same time also allow packages to be at a later version.
By default, the dbnodeupdate.sh utility updates attempt to enforce the exact-dependencies when updating to a next release. When exact-dependencies conflict and cannot be enforced, the utility attempts to apply the exadata-sun-computenode-minimum RPM during the update to enforce minimum dependencies. Missing or not updating to the exact dependencies is allowed and not a problem. If the system needs to be updated to the exact dependencies, then the conflict needs to be resolved. Check the dbnodeupdate.log file to see what packages conflict, remove them cautiously, and then re-run the dbnodeupdate.sh utility.
Use the following command to see the exact-dependencies enforced by the Oracle Exadata System Software release:
[root@dm01 ]# rpm -qR exadata-sun-computenode-exact | grep '=' | grep -v '('
The output from the command shows the exact-dependencies.
For example, the output for Oracle Exadata System Software release 12.1.1.1.1 shows that the sendmail package must be exactly 8.13.8-8.1.el5_7.
Use the following command to see the minimum dependencies enforced by the release:
[root@dm01 ]# rpm -qR exadata-sun-computenode-minimum | grep '>=' | grep -v '('
The output from the command shows the minimum dependencies.
For example, the output for Oracle Exadata System Software release 12.1.1.1.1 shows that the minimum sendmail package is sendmail >= 8.13.8-8.1.el5_7. This means that later releases of sendmail are allowed when not enforcing exact dependencies.
The dbnodeupdate.sh utility confirmation screen always shows what dependencies will be enforced during the update. When both exact and minimum dependencies cannot be applied, the update cannot proceed. For such cases the dbnodeupdate.log file should be consulted to find out what caused the dependencies to fail. After removing the failed dependency, the dbnodeupdate.sh utility should be re-run to validate dependencies and ensure that the least minimum-dependencies can be enforced.
Oracle recommends running the dbnodeupdate.sh utility with the -v flag to verify prerequisites several days before the scheduled maintenance. The dbnodeupdate.sh utility log file reports dependency problems that must be resolved before proceeding with the update and taking the downtime. When dependency errors occur during the prerequisite check or before the update starts, do the following to resolve the problem:
-
Analyze the requirement or yum error.
-
For updates starting from Oracle Exadata System Software release 11.2.3.1.0 and later, consult the files listed on the screen, including the
/var/log/cellos/dbnodeupdate.logfile. -
For updates starting from Oracle Exadata System Software release 11.2.2.4.2, consult the
/var/log/exadata.computenode.post.log,/var/log/exadata.computenode.post.log, and/var/log/cellos/bootstrap.uln.logfiles.
-
-
Depending on the issue, it may be necessary to de-install, install or update the RPM packages causing the dependency issue or conflict. The
dbnodeupdate.logfile lists the failed dependencies. -
Perform the update using the
dbnodeupdate.shutility. Only use the-nflag to skip the backup if a backup was already made in an earlier attempt. By default, existing backups for the same image are overwritten.When updating LVM-enabled systems from Oracle Linux 5 to Oracle Linux 6, a backup is done as part of the actual update. The
-nflag is ignored. -
Re-install custom RPM packages that had been de-installed after the update.
Note:
When manually updating individual packages on your system, for example, due to security requirements. Dependencies enforced by exadata-sun-computenode-exact may conflict. When this happens, it is supported to deinstall the exadata-sun-computenode-exact package in order to allow the installation of later packages. This can be done by running the following command:
[root@dm01 ]# rpm -e exadata-sun-computenode-exact
When a later Oracle Exadata System Software update includes the newer package, the exact-dependency is automatically reinstalled by the dbnodeupdate.sh utility.
It is not supported to remove the exadata-sun-computenode-minimum package. Packages should not be forced in using the rpm -Uvh --nodeps command unless directed by Oracle Support Services.
2.15.5.2 Preparing to Run the dbnodeupdate.sh Utility
The dbnodeupdate.sh utility is available from My Oracle Support from “dbnodeupdate.sh and dbserver.patch.zip: Updating Exadata Database Server Software using the DBNodeUpdate Utility and patchmgr (Doc ID 1553103.1)”. The following procedure describes how to download and prepare the utility on the server.
2.15.5.3 Reviewing the List of Obsolete Packages, List of Excluded Packages, and Automatic Package Removal
When updating to Oracle Exadata System Software release 11.2.3.3.0 and later, some packages on the Oracle Exadata Database Server become obsolete.
While updating a database server, the dbnodeupdate.sh utility displays the excluded RPM list and obsolete RPM list on the confirmation screen. The following is an example of the confirmation screen. In this example, an exclusion list has not yet been created by the user.
Active Image version : 11.2.3.2.1.130302
Active Kernel version : 2.6.32-400.21.1.el5uek
Active LVM Name : /dev/mapper/VGExaDb-LVDbSys2
Inactive Image version : 11.2.3.2.1.130109
Inactive LVM Name : /dev/mapper/VGExaDb-LVDbSys1
Current user id : root
Action : upgrade
Upgrading to : 11.2.3.3.0.131014.1
Baseurl : http://mynode.us.example.com/yum/exadata_dbserver_11.2.3.3.0_x86_64_base/x86_64/ (http)
Create a backup : Yes
Shutdown stack : No (Currently stack is up)
RPM exclusion list : Not in use (add rpms to /etc/exadata/yum/exclusion.lst and restart dbnodeupdate.sh)
RPM obsolete list : /etc/exadata/yum/obsolete.lst (lists rpms to be removed by the update)
: RPM obsolete list is extracted from exadata-sun-computenode-11.2.3.3.0.131014.1-1.x86_64.rpm
Logfile : /var/log/cellos/dbnodeupdate.log (runid: 021213024645)
Diagfile : /var/log/cellos/dbnodeupdate.021213024645.diag
Server model : SUN FIRE X4170 M2 SERVER
Remote mounts exist : Yes (dbnodeupdate.sh will try unmounting)
dbnodeupdate.sh rel. : 2.20 (always check MOS 1553103.1 for the latest release)
Automatic checks incl. : Issue - Yum rolling update requires fix for 11768055 when Grid Infrastructure is below 11.2.0.2 BP12
Manual checks todo : Issue - Database Server upgrades to 11.2.2.3.0 or higher may hit network routing issues after the upgrade
Note : After upgrading and rebooting run './dbnodeupdate.sh -c' to finish post steps
Continue ? [Y/n]
To see what packages will become obsolete, review the contents of the obsolete.lst file, enter N when prompted by the script. The location of the obsolete.lst file is displayed on the confirmation screen. The obsolete.lst file lists only those packages that will be removed during the update when no action is taken. Packages added manually to this list are ignored. The following is an example of the obsolete.lst file:
[root@dm01 ]# cat /etc/exadata/yum/obsolete.lst # Generated by dbnodeupdate.sh runid: 021213024645 at.x86_64 java-*-openjdk rhino.noarch jline.noarch jpackage-utils.noarch giflib.x86_64 alsa-lib.x86_64 xorg-x11-fonts-Type1.noarch prelink.x86_64 biosconfig biosconfig_expat qlvnictools ibvexdmtools opensm.x86_64 ofed-scripts ibibverbs-devel-static infiniband-diags-debuginfo libibverbs-debuginfo librdmacm-debuginfo libibumad-debuginfo libibmad-debuginfo ibutils-debuginfo libmlx4-debuginfo libsdp-debuginfo mstflint-debuginfo perftest-debuginfo qperf-debuginfo libibcm-debuginfo compat-dapl-utils.x86_64 compat-dapl.x86_64 dapl-utils.x86_64 dapl.x86_64 libgfortran.x86_64 mdadm.x86_64 mpi-selector.noarch pygobject2.x86_64 specspo.noarch time.x86_64 tree.x86_64 unix2dos.x86_64 usbutils.x86_64 words.noarch
To prevent a package listed in the obsolete.lst file from being removed, create the /etc/exadata/yum/exclusion.lst file, and add the RPM name or wildcard for the RPMs to keep in the exclusion list.
The following example shows a package added to the exclusion list:
[root@dm01 ]# cat /etc/exadata/yum/exclusion.lst java-*-openjdk
After adding an entry to the exclusion.lst file, and re-running the dbnodeupdate.sh utility, the exclusion list is detected by the utility. The RPM packages on the exclusion list are still shown in the obsolete.lst file, but the listed packages in the exclusion.lst file are not removed during the update. The following example of the confirmation screen shows the exclusion.lst file is in use after an entry has been added to it.
Active Image version : 11.2.3.2.1.130302
Active Kernel version : 2.6.32-400.21.1.el5uek
Active LVM Name : /dev/mapper/VGExaDb-LVDbSys2
Inactive Image version : 11.2.3.2.1.130109
Inactive LVM Name : /dev/mapper/VGExaDb-LVDbSys1
Current user id : root
Action : upgrade
Upgrading to : 11.2.3.3.0.131014.1
Baseurl : http://mynode.us.example.com/yum/exadata_dbserver_11.2.3.3.0_x86_64_base/x86_64/ (http)
Create a backup : Yes
Shutdown stack : No (Currently stack is up)
RPM exclusion list : In use (rpms listed in /etc/exadata/yum/exclusion.lst)
RPM obsolete list : /etc/exadata/yum/obsolete.lst (lists rpms to be removed by the update)
: RPM obsolete list is extracted from exadata-sun-computenode-11.2.3.3.0.131014.1-1.x86_64.rpm
Logfile : /var/log/cellos/dbnodeupdate.log (runid: 021213024900)
Diagfile : /var/log/cellos/dbnodeupdate.021213024900.diag
Server model : SUN FIRE X4170 M2 SERVER
Remote mounts exist : Yes (dbnodeupdate.sh will try unmounting)
dbnodeupdate.sh rel. : 2.20 (always check MOS 1553103.1 for the latest release)
Automatic checks incl. : Issue - Yum rolling update requires fix for 11768055 when Grid Infrastructure is below 11.2.0.2 BP12
Manual checks todo : Issue - Database Server upgrades to 11.2.2.3.0 or higher may hit network routing issues after the upgrade
Note : After upgrading and rebooting run './dbnodeupdate.sh -c' to finish post steps
Continue ? [Y/n]
Note:
Neither the dbnodeupdate.sh utility or Oracle Exadata System Software upgrade process remove RPM packages not listed in the obsolete.lst file. Only files listed in the obsolete.lst file can be added to the exclusion.lst file. Custom packages are not removed. The obsolete.lst and exclusion.lst functionality is not available when updating from release 11.2.2.4.2, or when updating to Oracle Linux 6.
2.15.5.4 Running the dbnodeupdate.sh Utility in Update Mode
The following procedure shows how to run the dbnodeupdate.sh utility in update mode.
2.15.5.5 Example of Running the dbnodeupdate.sh Utility
This topic provides an example of using the dbnodeupdate.sh utility.
In this example:
-
The system is currently running Oracle Exadata System Software release 11.2.3.3.1 (shown in the Active Image version entry).
-
The system is using kernel 2.6.39-400.128.17.el5uek (Active Kernel version).
-
The current active system partition is LVDbSys2 (Active Logical Volume Manager (LVM) name).
-
The update will update the system to Oracle Exadata System Software release 12.1.1.1.1.140712 (Upgrading to).
-
The update will use the ISO image provided in the compressed file (Iso file).
The following actions are performed, as noted in the output of the dbnodeupdate.sh utility:
-
A backup is created (Create a backup), which is put on
LVDbSys1(Inactive LVM Name). -
The current system has the database stack running (Shutdown Stack), however, the
-sflag was specified, and the stack will be shut down before proceeding the backup and the update. -
The
obsolete.lstfile (RPM obsolete list) specifies what RPMs will be removed as part of the update. -
No
exclusion.lstfile (RPM exclusion list) was provided. - The update to Oracle Exadata System Software release 12.1.1.1.1 is only possible for updates with
exadata-computenode-minimumRPM dependencies because exact dependencies check will fail. For updates to minimal dependency restrictions there is no conflict (Minimum dependencies). -
The log of the
dbnodeupdate.shutility process is appended to thedbnodeupdate.logfile (Logfile). Thediagfilefile (Diagfile) captures important system information before the update starts. Each use of the utility produces thediagfilefile. This file is used for troubleshooting when required. -
Active remote network mounts such a NFS are detected (Remote mounts exist), and the
dbnodeupdate.shutility will try to unmount them. When that is not possible, the utility lists the blocking sessions and quits. -
The
dbnodeupdate.sh rel.entry specifies the currentdbnodeupdate.shrelease. -
As the utility checks the system, an on-screen list displays the known issues the
dbnodeupdate.shutility has identified for the Oracle Exadata System Software source and target release. Issues that can be automatically corrected by the utility are listed first, and followed by any manual checks that need to be performed on the system.
Example 2-3 Running the dbnodeupdate.sh Utility
[root@dm01 ]# ./dbnodeupdate.sh -u -l p18889969_121111_Linux-x86-64.zip -s
Active Image version : 11.2.3.3.1.140529.1
Active Kernel version : 2.6.39-400.128.17.el5uek
Active LVM Name : /dev/mapper/VGExaDb-LVDbSys2
Inactive Image version : 11.2.3.2.0.120713
Inactive LVM Name : /dev/mapper/VGExaDb-LVDbSys1
Current user id : root
Action : upgrade
Upgrading to : 12.1.1.1.1.140712 (to exadata-sun-computenode-minimum)
Baseurl : file:///var/www/html/yum/unknown/EXADATA/dbserver/121114021722/x86_64/ (iso)
Iso file : /u01/dbnodeupdate/iso.stage.121114021722/121111_base_repo_140712.iso
Create a backup : Yes
Shutdown stack : Yes (Currently stack is up)
RPM exclusion list : Not in use (add rpms to /etc/exadata/yum/exclusion.lst and restart dbnodeupdate.sh)
RPM obsolete list : /etc/exadata/yum/obsolete.lst (lists rpms to be removed by the update)
: RPM obsolete list is extracted from exadata-sun-computenode-12.1.1.1.1.140712-1.x86_64.rpm
Exact dependencies : Will fail on a next update. Update to 'exact' will be not possible. Falling back to 'minimum'
: See /var/log/exadatatmp/121114021722/exact_conflict_report.txt for more details
: Update target switched to 'minimum'
Minimum dependencies : No conflicts
Logfile : /var/log/cellos/dbnodeupdate.log (runid: 121114021722)
Diagfile : /var/log/cellos/dbnodeupdate.121114021722.diag
Server model : SUN FIRE X4170 SERVER
Remote mounts exist : Yes (dbnodeupdate.sh will try unmounting)
dbnodeupdate.sh rel. : 3.79b (always check MOS 1553103.1 for the latest release of dbnodeupdate.sh)
Note : After upgrading and rebooting run './dbnodeupdate.sh -c' to finish post steps.
The following known issues will be checked for and automatically corrected by dbnodeupdate.sh:
(*) - Issue - Adjust kernel.rcu_delay in /etc/sysctl.conf. See MOS 1916147.1
(*) - Issue - Fix for CVE-2014-6271 and CVE-2014-7169 (Shell-Shock)
The following known issues will be checked for but require manual follow-up:
(*) - Issue - Yum rolling update requires fix for 11768055 when Grid Infrastructure is below 11.2.0.2 BP12
Continue ? [y/n]
y
(*) 2014-11-12 02:21:19: Verifying GI and DB's are shutdown
(*) 2014-11-12 02:21:19: Shutting down GI and db
(*) 2014-11-12 02:23:17: Successfully unmounted network mount /mnt/rene
(*) 2014-11-12 02:23:17: Unmount of /boot successful
(*) 2014-11-12 02:23:17: Check for /dev/sda1 successful
(*) 2014-11-12 02:23:17: Mount of /boot successful
(*) 2014-11-12 02:23:17: Disabling stack from starting
(*) 2014-11-12 02:23:17: Performing filesystem backup to /dev/mapper/VGExaDb-LVDbSys1. Avg. 30
minutes (maximum 120) depends per environment........
(*) 2014-11-12 02:30:29: Backup successful
(*) 2014-11-12 02:30:36: ExaWatcher stopped successful
(*) 2014-11-12 02:30:36: Capturing service status and file attributes. This may take a while...
(*) 2014-11-12 02:30:42: Service status and file attribute report in: /etc/exadata/reports
(*) 2014-11-12 02:30:43: Validating the specified source location.
(*) 2014-11-12 02:30:44: Cleaning up the yum cache.
(*) 2014-11-12 02:30:47: Performing yum update. Node is expected to reboot when finished.
(*) 2014-11-12 02:31:03: Waiting for post rpm script to finish. Sleeping another 60 seconds (60 / 900)
Remote broadcast message (Wed Nov 12 02:31:08 2014):
Exadata post install steps started.
It may take up to 15 minutes.
Remote broadcast message (Wed Nov 12 02:31:53 2014):
Exadata post install steps completed.
(*) 2014-11-12 02:32:03: Waiting for post rpm script to finish. Sleeping another 60 seconds (120 / 900)
(*) 2014-11-12 02:33:03: All post steps are finished.
(*) 2014-11-12 02:33:03: System will reboot automatically for changes to take effect
(*) 2014-11-12 02:33:03: After reboot run "./dbnodeupdate.sh -c" to complete the upgrade
(*) 2014-11-12 02:33:05: Cleaning up iso and temp mount points
(*) 2014-11-12 02:33:05: Rebooting now...
Broadcast message from root (pts/0) (Wed Nov 12 02:33:05 2014):
The system is going down for reboot NOW!
2.15.6 Updating Database Servers Running Oracle Exadata System Software Release 11.2.2.4.2
Updates for database servers running Oracle Exadata System Software release 11.2.2.4.2 include an update that prepares the servers to use yum.
The steps to prepare the servers are done using the dbnodeupdate.sh utility. For this reason, updates from Oracle Exadata System Software release 11.2.2.4.2 require two runs of the dbnodeupdate.sh utility with different arguments. The utility provides instructions about which command to run and when to it.
When updating the database servers, the following assumptions are made:
-
The Oracle Exadata System Software release is 11.2.2.4.2.
-
The database servers are running Oracle Linux 5.5 or later.
-
The database server update is available as a local ULN mirror or as a local ISO image.
This section contains the following topics:
Related Topics
2.15.6.1 Preparing to Use the dbnodeupdate.sh Utility on Database Servers with Release 11.2.2.4.2
This procedure describes how to download and prepare the utility on the server.
The dbnodeupdate.sh utility is available from My Oracle Support from Doc ID 1553103.1. The dbnodeupdate.sh utility can update releases running 11.2.2.4.2 directly to Oracle Linux 6.
Related Topics
- dbnodeupdate.sh and dbserver.patch.zip: Updating Exadata Database Server Software using the DBNodeUpdate Utility and patchmgr (My Oracle Support Doc ID 1553103.1)
- Setting Up a Local Mirror of the ULN Oracle Exadata Channel on a Separate Server
- Downloading the ISO Image and Making It Available on a Web Server
- Preparing and Populating the yum Repository with the Oracle Exadata Channel Content
2.15.7 Rolling Back Updates on the Database Servers
The database server updates modify the Oracle Exadata System Software, Oracle Linux system software, and firmware.
The updates do not modify the Oracle Grid Infrastructure home, Oracle Database home (other than relinking during the dbnodeupdate.sh -c step), or customer-installed software. The database server updates are performed on the active system disk partition.
To review the active system disk partition information, run the imageinfo command. The database servers using Logical Volume Manager (LVM) have at least one system partition. The following is an example of the output from the imageinfo command:
Kernel version: 2.6.39-400.238.0.el6uek.x86_64 #1 SMP Fri Aug 15 14:27:21 PDT 2014 x86_64 Image version: 12.1.2.1.0.141022 Image activated: 2014-10-22 22:11:12 -0700 Image status: success System partition on device: /dev/mapper/VGExaDb-LVDbSys1
When the dbnodeupdate.sh utility performs a rollback, the inactive system disk partition becomes active, and the active system disk partition becomes inactive. Rolling back to the previous release by activating the inactive system partition does not roll back the firmware. The Oracle Exadata System Software releases support later firmware releases.
The ability to roll back database server updates require a backup to be made prior to updating the database servers. The dbnodeupdate.sh utility creates a backup on the inactive system disk partition when the following conditions are met:
-
The system uses LVM.
-
The size of the inactive partition is equal to the active partition. If only
LVDbSys1partition exists, then there must be as much free space in the volume group as the inactive partition to be created. -
If both system disk partitions,
LVDbSys1andLVDbSys2, exist, then the volume group must have at least 1 GB free space for the temporary snapshot created during the backup.
If the preceding conditions are not met, then the dbnodeupdate.sh utility reports this, and disables the automatic backup for the process. If you proceed with the update, then you will not be able to roll back the update.
Note:
-
For systems being updated to Oracle Linux 6, a backup must be performed before proceeding with the update. The backup is automatic when updating LVM-enabled systems from Oracle Linux 5 to Oracle Linux 6.
-
Database servers running as Oracle VM Server (dom0) switch between
LVDbSys2andLVDbSys3as the active system partition when rolling back. -
Database servers running as Oracle VM (domU) have smaller sizes for
LVDbSys1and c compared to physical hardware deployments.
2.15.8 Updating Database Nodes with patchmgr
Starting with Oracle Exadata System Software release 12.1.2.2.0, Oracle Exadata Database Servers (running releases later than 11.2.2.4.2), Oracle VM Server nodes (dom0), and Oracle VM nodes (domU) can be updated, rolled back, and backed up using patchmgr.
You can still run dbnodeupdate.sh in standalone mode, but performing the updates by running patchmgr enables you to run a single command to update multiple nodes at the same time; you do not need to run dbnodeupdate.sh separately on each node. patchmgr can update the nodes in a rolling or non-rolling fashion.
For patchmgr to do this orchestration, you run it from a database node that will not be updated itself. That node needs to be updated later by running patchmgr from a node that has already been updated or by running dbnodeupdate.sh standalone on that node itself.
Similar to updating cells, you create a file that specifies a list of database nodes (including domU and dom0) to be updated. patchmgr then updates the specified nodes in a rolling or non-rolling fashion. The default is non-rolling, which is similar to updating cells. Rollbacks are done in a rolling fashion.
If you are running dbnodeupdate.sh standalone, you can just follow the standard steps for running dbnodeupdate.sh without the patchmgr component. Always check My Oracle Support note 1553103.1 for the latest release of dbnodeupdate.sh.
2.15.8.1 Getting and Installing dbserver.patch.zip
Starting with Oracle Exadata System Software release 12.1.2.2.0, a new dbserver.patch.zip file is available for running dbnodeupdate from patchmgr.
If you are using a local yum repository in the form of the zipped ISO, there is a separate zipped ISO file for regular hardware, user domains (domU), and management domains (dom0). When using the ISO file for the update, it is recommended that you put the ISO file in the same directory where you have dbnodeupdate.zip.
2.15.8.2 Behavior for Non-Rolling Upgrades
The behavior for non-rolling upgrades is as follows:
-
If a node fails at the pre-check stage, the whole process fails.
-
If a node fails at the patch stage or reboot stage, patchmgr skips further steps for the node. The upgrade process continues for the other nodes.
-
The pre-check stage is done serially.
-
The patch/reboot and complete stages are done in parallel.
The notification alert sequence is:
-
Started (All nodes in parallel)
-
Patching (All nodes in parallel)
-
Reboot (All nodes in parallel)
-
Complete Step (Each node serially)
-
Succeeded (Each node serially)
2.15.8.3 Update Procedure
The steps you perform depending on what type of update you are performing.
If you are updating database nodes that will also go from Oracle Linux 5 to Oracle Linux 6:
-
Run the prerequisites check using patchmgr.
-
Perform a backup.
-
Run an update with mandatory backup.
If you are doing any update other than the Oracle Linux 5 to Oracle Linux 6:
-
Run the prerequisites check.
-
(Optional) Perform a backup.
-
Run an update with optional backup.
2.15.8.4 patchmgr Options
You can use a variety of options with the patchmgr command.
The following options are supported when updating a database node with patchmgr:
Table 2-11 Options for patchmgr: Required Options
| Option | Description |
|---|---|
|
|
Specifies the name of the database node list file. The file has one host name or IP address per line. |
Table 2-12 Options for patchmgr: Action Options
| Option | Description |
|---|---|
|
|
Runs the pre-update validation checks on the database nodes specified in the list file, without actually doing the update. |
|
|
Upgrades the database nodes specified in the list file. |
|
|
Backs up the database nodes specified in the list file. |
|
|
Rolls back the database nodes specified in the list file. |
|
|
Removes all temporary content on the database nodes specified in the host list in a non-rolling fashion. This option is typically used after an upgrade or rollback operation. |
|
|
Returns information about the specified property. Property can be:
|
Table 2-13 Options for patchmgr: Supporting Options
| Option | Description |
|---|---|
|
|
Specifies the path to a zipped ISO file. It is recommended that the file be in the same directory as the This option or the This option cannot be used with the |
|
|
Specifies the URL to a yum HTTP repository. This option or the This option cannot be used with the |
|
|
Specifies the absolute path to the log directory, or you can specify Specifying the |
|
|
Specifies that a backup on the database nodes for an upgrade action should not be performed. |
|
|
Specifies the full release version being updated to. You can find this information in the README file for the patch. |
|
|
Removes passwordless SSH access to the database nodes before exiting. |
|
|
Specifies that the update is to be done in a rolling fashion. If not specified, the update is done in a non-rolling fashion. Note: Rollbacks are always done in a rolling fashion, even if this option is not specified. Note: Prechecks and cleanups are always done in a non-rolling fashion, even if the |
|
|
Allows This is equivalent to the |
|
|
Removes any custom RPMs when the database node is updated from Oracle Linux 5 to Oracle Linux 6. This is equivalent to the |
|
|
No RPM changes at prerequisite check. This is equivalent to the |
Table 2-14 Options for patchmgr: Mail Options
| Option | Description |
|---|---|
|
|
Specifies the from email address for the patchmgr notification. |
|
|
Specifies the to email addresses for the patchmgr notification. |
|
|
Specifies that the same from address in Return-Path: mail header should be used. |
2.15.8.5 Checking Prerequisites
The -precheck option simulates the update without actually doing the update.
You use the -precheck option to catch and resolve any errors in advance of the actual update.
Usage:
# ./patchmgr -dbnodes dbnode_list -precheck -iso_repo iso_file -target_version version
# ./patchmgr -dbnodes dbnode_list -precheck -yum_repo yum_http -target_version version
Example 2-4 Using patchmgr with a yum HTTP repository
Example using yum HTTP repository, where date_stamp specifies the release date of the version, for example, 150731:
# ./patchmgr -dbnodes dbnode_list -precheck -yum_repo http://yum-repo/yum/ol6/EXADATA/ dbserver/12.1.2.2.0/base/x86_64/ -target_version 12.1.2.2.0.date_stamp
Example 2-5 Using patchmgr with a zipped yum ISO repository
Example using zipped yum ISO repository, where date_stamp specifies the release date of the version, for example, 150731:
# ./patchmgr -dbnodes dbnode_list -precheck -iso_repo ./repo.zip -target_version 12.1.2.2.0.date_stamp
2.15.8.6 Updating the (Database) Nodes
The -upgrade option performs the actual upgrade of the database node(s).
If you need to update all database nodes in a system, you must run the command from a driving node, which cannot be one of the database nodes being updated.
Usage:
# ./patchmgr -dbnodes dbnode_list -upgrade -iso_repo iso_file -target_version version
# ./patchmgr -dbnodes dbnode_list -upgrade -yum_repo yum_http -target_version version
Example 2-6 Rolling update using yum http repository
Example of a rolling update using yum HTTP repository where date_stamp specifies the release date of the version, for example, 150731.
# ./patchmgr -dbnodes dbnode_list -upgrade -yum_repo http://yum-repo/yum/ol6/EXADATA/ dbserver/12.1.2.1.0/base/x86_64/ -target_version 12.1.2.2.0.date_stamp -rolling
Example 2-7 Non-rolling update using zipped yum ISO repository
Example of a rolling update using zipped yum ISO repository where date_stamp specifies the release date of the version, for example, 150731.
# ./patchmgr -dbnodes dbnode_list -upgrade -iso_repo ./repo.zip -target_version 12.1.2.2.0.date_stamp
2.15.8.7 Rolling Back the Update
Use the -rollback option to rollback an update.
The -rollback option switches active and inactive Logical Volume Managers (LVM), restores /boot contents, reinstalls Grub, and reboots the database nodes so that it reverses the effects of the upgrade. Note that this is only available on LVM-enabled systems.
Note:
Rollbacks can only be done in a rolling fashion, even if -rolling is not specified.
Usage:
# ./patchmgr -dbnodes dbnode_list -rollback
Related Topics
2.15.8.8 Troubleshooting Failures When Updating Database Nodes with patchmgr
Review these notes if you run into problems running patchmgr.
For the correct syntax for using patchmgr to update database nodes, see the patchmgr online help.
Updating database nodes with the patchmgr tool is less verbose in that it prints only minimal information to the screen. If additional information is required, you can view the patchmgr logs and the dbnodeupdate.sh logs that patchmgr copied over from the failing node, if available.
As with updating storage cells, patchmgr requires SSH equivalence to run.
As always, be sure you download the latest dbserver.patch.zip from My Oracle Support note 1553103.1.
In rare cases, patchmgr may be unable to determine the status of an update, whether the update was successful or not. In such cases, it displays a message that the update failed. However, it is possible that the update still completed successfully. To determine the actual status of the update:
-
Check the image status of the (database) node. You can do this by running the
imageinfocommand. TheImage statusline displays the status. -
Check the version of the Exadata software. This can also be determined from the
imageinfocommand.
If the image status is success, and the Exadata version is the new expected version, then the update was successful and you can ignore the update failed message. Then:
-
Run
dbnodeupdate.sh -cmanually on the particular node to perform the completion steps of the update. -
Remove the completed node from the (database) node file.
-
Rerun patchmgr to perform the update on the remaining nodes.
Updating database nodes using patchmgr is optional. You can still run dbnodeupdate.sh manually. If you get any blocking errors from patchmgr when updating critical systems, it is recommended that you perform the update by running dbnodeupdate.sh manually.
You should file a service request with Oracle Support to analyze why the patchmgr orchestration failed.
2.15.9 Running dbnodeupdate.sh and patchmgr (and dbnodeupdate Orchestration) Using sudo
2.15.9.1 Running dbnodeupdate.sh Using sudo
Perform the following steps to set up the /etc/sudoers file for running dbnodeupdate.sh using sudo:
To verify that the setup is correct, run dbnodeupdate in prereq check mode as the oracle user:
[oracle]$ cd /u01/stage/patch/dbnodeupdate [oracle]$ sudo ./dbnodeupdate.sh -u -l http://my-yum-repo/yum/EngineeredSystems/exadata/dbserver/12.1.2.1.3/base/x86_64/ -v
dbnodeupdate would exit if it is run without root privileges.
Note:
-
The above setup requires that everything in
/u01/stage/patch/dbnodeupdatebe owned by root. -
A new version of dbnodeupdate would need to be placed in the same location as specified by
sudoers.
2.15.9.2 Running patchmgr (and dbnodeupdate Orchestration) Using sudo
You can run patchmgr (which is packaged in dbserver.patch.zip) using sudo to perform any of patchmgr's functionalities, such as patching cells, patching InfiniBand switches, or orchestrating dbnodeupdate execution.
Perform the following steps to set up the /etc/sudoers file for running patchmgr using sudo:
Note:
-
patchmgr expects root SSH equivalence on all database nodes that will be updated, even when run using sudo.
-
The above setup requires that the entire contents of
/u01/stage/patch/dbserverpatchbe owned by root. -
A new version of
dbserver.patch.zipwould need to be placed in the same location as specified bysudoers.
To verify that the setup is correct, run patchmgr in prereq check mode as the oracle user:
[oracle]$ cd /u01/stage/patch/dbserverpatch/ [oracle]$ sudo ./patchmgr -dbnodes dbgroup -precheck \ -yum_repo http://my-yum-repo/yum/EngineeredSystems/exadata/dbserver/12.1.2.1.3/base/x86_64/ \ -target_version 12.1.2.1.3.151021
2.16 Re-Imaging Oracle Exadata Database Servers
The re-image procedure is necessary when an Oracle Linux database server has been irretrievably damaged.
You replace the damaged database server with a new database server. You also use this procedure if there are multiple disk failures causing local disk storage failure and there is no database server backup.
During the re-imaging procedure the other database servers on Oracle Exadata Database Machine are available. When the new server is added to the cluster, the software is copied from an existing database server to the new server. It is your responsibility to restore scripting, CRON jobs, maintenance actions, and non-Oracle software.
Note:
The procedures in this section assume the database is Oracle Database 12c Release 1 (12.1) or Oracle Database 11g Release 2 (11.2). If the database is Oracle Database 11g Release 1 (11.1), then refer to the documentation for that release for information about adding and deleting a server from a cluster.The following tasks describes how to re-image an Oracle Exadata Database Server running Oracle Linux:
2.16.1 Contact Oracle Support Services
Open a support request with Oracle Support Services. The support engineer will identify the failed server, and send a replacement. The support engineer will ask for the output from the imagehistory command run from a surviving database server. The output provides a link to the computeImageMaker file that was used to image the original database server, and provides a means to restore the system to the same level.
2.16.2 Download Latest Release of Cluster Verification Utility
The latest release of the cluster verification utility (cluvfy) is available from My Oracle Support.
See My Oracle Support note 316817.1 for download instructions and other information.
2.16.3 Remove Failed Database Server from the Cluster
The failed database server must be removed from the cluster before you can reimage the server.
The steps in this task are performed using a working database server in the cluster. In the following commands, working_server is a working database server, and replacement_server is the replacement database server.
2.16.4 Image the Replacement Database Server
Once the database server has been replaced, you can image the new database server. You can use installation media on a USB thumb drive, or a touchless option using PXE or ISO attached to the ILOM. See Imaging a New System in Oracle Exadata Database Machine Installation and Configuration Guide for the details.
2.16.5 Configure the Replacement Database Server
The replacement database server does not have any host names, IP addresses, DNS or NTP settings. The steps in this task describe how to configure the replacement database server. You need the following information prior to configuring the replacement database server:
-
Name servers
-
Time zone, such as Americas/Chicago
-
NTP servers
-
IP address information for the management network
-
IP address information for the client access network
-
IP address information for the InfiniBand network
-
Canonical host name
-
Default gateway
The information should be the same on all database servers in Oracle Exadata Database Machine. The IP addresses can be obtained from DNS. In addition, a document with the information should have been provided when Oracle Exadata Database Machine was installed.
The following procedure describes how to configure the replacement database server:
- Power on the replacement database server. When the system boots, it automatically runs the Configure Oracle Exadata routine, and prompts for information.
- Enter the information when prompted, and confirm the settings. The startup process will continue.
Note:
-
If the database server does not use all network interfaces, then the configuration process stops, and warns that some network interfaces are disconnected. It prompts whether to retry the discovery process. Respond with
yesorno, as appropriate for the environment. -
If bonding is used for the client access network, then it is set in the default active-passive mode at this time.
2.16.6 Prepare the Replacement Database Server for the Cluster
During initial installation of Oracle Exadata Database Machine, certain files were modified by the installation process. The following procedure describes how to ensure the changes made during initial installation are done to the replacement database server:
-
Copy or merge the contents of the following files using files on a working database server as reference:
-
Copy the contents of the
/etc/security/limits.conffile. -
Merge the contents of the
/etc/hostsfiles. -
Copy the
/etc/oracle/cell/network-config/cellinit.orafile. -
Update the
/etc/oracle/cell/network-config/cellinit.orafile with the IP_ADDRESS of theifcfg-bondib0interface (in case of active/passive bonding) orib0andib1interfaces (in case of active/active bonding) of the replacement server. -
Copy the
/etc/oracle/cell/network-config/cellip.orafile. The content of thecellip.orafile should be the same on all database servers. -
Configure additional network requirements, such as 10 GbE.
-
Copy the
/etc/modprobe.conffile. The contents of the file should be the same on all database servers. -
Copy the
/etc/sysctl.conffile. The contents of the file should be the same on all database servers. -
Restart the database server so the network changes take effect.
-
-
Set up the users for the software owners on the replacement database server by adding groups. If you are using role-separated management, then the users are usually
oracleandgrid. If you use a single software owner, then the user is usuallyoracle. The group information is available on a working database server.-
Obtain the current group information from a working database server using the following command:
# id oracle uid=1000(oracle) gid=1001(oinstall) groups=1001(oinstall),1002(dba),1003(oper),1004(asmdba)
-
Use the
groupaddcommand to add the group information to the replacement database server using the following commands:# groupadd -g 1001 oinstall # groupadd -g 1002 dba # groupadd -g 1003 oper # groupadd -g 1004 asmdba
-
Obtain the current user information from a working database server using the following command:
# id oracle uid=1000(oracle) gid=1001(oinstall) \ groups=1001(oinstall),1002(dba),1003(oper),1004(asmdba)
-
Add the user information to the replacement database server using the following command:
# useradd -u 1000 -g 1001 -G 1001,1002,1003,1004 -m -d /home/oracle -s \ /bin/bash oracle
-
Create the Oracle Base and Grid home directories, such as
/u01/app/oracleand/u01/app/12.2.0.1/gridusing the following commands:# mkdir -p /u01/app/oracle # mkdir -p /u01/app/12.2.0.1/grid # chown -R oracle:oinstall /u01/app
-
Change the ownership on the
cellip.oraandcellinit.orafiles using the following command. The ownership is usuallyoracle:dba.# chown -R oracle:dba /etc/oracle/cell/network-config
-
Secure the restored database server using the following commands:
$ chmod u+x /opt/oracle.SupportTools/harden_passwords_reset_root_ssh $ /opt/oracle.SupportTools/harden_passwords_reset_root_ssh
Note:
The database server restarts. Log in as therootuser when prompted by the system. You are prompted for a new password. Set the password to match therootpassword of the other database servers. -
Set the password for the Oracle software owner using the following command. The owner is usually
oracle.# passwd oracle
-
-
Set up SSH for the
oracleaccount as follows:-
Log in to the
oracleaccount on the replacement database server using the following command:# su - oracle
-
Create the
dcligroup file on the replacement database server listing the servers in the Oracle cluster. -
Run the following command on the replacement database server.
$ dcli -g dbs_group -l oracle -k
-
Log in as the
oracleuser on the replacement database server using the following command:# su - oracle
-
Verify SSH equivalency using the following command:
$ dcli -g dbs_group -l oracle date
-
-
Set up or copy any custom login scripts from a working database server to the replacement database server using the following command:
$ scp .bash* oracle@replacement_server:.In the preceding command, replacement_server is the name of the new server, such as
dm01db01.
2.16.7 Apply Oracle Exadata System Software Patch Bundles to the Replacement Database Server
Oracle periodically releases Oracle Exadata System Software patch bundles for Oracle Exadata Database Machine.
If a patch bundle has been applied to the working database servers that was later than the release of the computeImageMaker file, then the patch bundle must be applied to the replacement Oracle Exadata Database Server. Determine the if a patch bundle has been applied as follows:
-
Prior to Oracle Exadata System Software release 11.2.1.2.3, the database servers did not maintain version history information. To determine the release number, log in to Oracle Exadata Storage Server, and run the following command:
imageinfo -ver
If the command shows a different release than the release used by the
computeImageMakerfile, then Oracle Exadata System Software patch has been applied to Oracle Exadata Database Machine and must be applied to the replacement Oracle Exadata Database Server. -
Starting with Oracle Exadata System Software release 11.2.1.2.3, the
imagehistorycommand exists on the Oracle Exadata Database Server. Compare information on the replacement Oracle Exadata Database Server to information on a working Oracle Exadata Database Server. If the working database has a later release, then apply the Oracle Exadata Storage Server patch bundle to the replacement Oracle Exadata Database Server.
2.16.8 Clone Oracle Grid Infrastructure to the Replacement Database Server
This procedure describes how to clone Oracle Grid Infrastructure to the replacement database server.
In the following commands, working_server is a working database server, and replacement_server is the replacement database server. The commands in this procedure are run from a working database server as the Grid home owner. When the root user is needed to run a command, it will be called out.
2.16.9 Clone Oracle Database Homes to the Replacement Database Server
The following procedure describes how to clone the Oracle Database homes to the replacement server.
Run the commands from a working database server as the oracle user. When the root user is needed to run a command, it will be called out.
2.17 Changing Existing Elastic Configurations for Database Servers
This section describes how to perform the following changes to existing elastic configurations.
For changes involving storage cells, see "Changing Existing Elastic Configurations for Storage Cells".
2.17.1 Adding a New Database Server to the Cluster
You can add a new database server to an existing Oracle Real Application Clusters (Oracle RAC) cluster running on Oracle Exadata Database Machine.
2.17.2 Moving an Existing Database Server to a Different Cluster
You can repurpose an existing database server and move it to a different cluster within the same Oracle Exadata Rack.
2.18 Managing Quorum Disks for High Redundancy Disk Groups
This section contains the following subsections:
2.18.1 Overview of Quorum Disk Manager
The Quorum Disk Manager utility, introduced in Oracle Exadata release 12.1.2.3.0, helps you to manage the quorum disks.
The utility enables you to create an iSCSI quorum disk on two of the database nodes and store a voting file on those two quorum disks. These two additional voting files are used to meet the minimum requirement of five voting files for a high redundancy disk group. This feature is only applicable to Oracle Exadata racks that meet the following requirements:
-
The Oracle Exadata rack has fewer than five storage servers.
-
The Oracle Exadata rack has at least two database nodes.
-
The Oracle Exadata rack has at least one high redundancy disk group.
The feature allows for the voting files to be stored in a high redundancy disk group on Oracle Exadata racks with fewer than five storage servers due to the presence of two extra failure groups.
Without this feature, voting files are stored in a normal redundancy disk group on Exadata racks with fewer than five storage servers. This makes Oracle Grid Infrastructure vulnerable to a double partner storage server failure that results in the loss of the voting file quorum, in turn resulting in a complete cluster and database outage. Refer to My Oracle Support note 1339373.1 for restarting the clusterware and databases in this scenario.
The iSCSI quorum disk implementation has high availability because the IP addresses on ib0 and ib1 are highly available using RDS. The multipathing feature ensures the iSCSI quorum disk implementation will work seamlessly if a more flexible or isolated internal network configuration is implemented in the future.
Each iSCSI device shown in the figure below corresponds to a particular path to the iSCSI target. Each path corresponds to an InfiniBand port on the database node. For each multipath quorum disk device in an active–active system, there are two iSCSI devices, one for ib0 and the other for ib1.
Figure 2-7 Multipath Device Connects to Both iSCSI Devices in an Active-Active System
Description of "Figure 2-7 Multipath Device Connects to Both iSCSI Devices in an Active-Active System"
The feature is applicable to bare metal Oracle RAC clusters as well as Oracle VM Oracle RAC clusters. For Oracle VM Oracle RAC clusters, the quorum disk devices reside in the Oracle RAC cluster nodes which are Oracle VM user domains as shown in the following figure.
Figure 2-8 Quorum Disk Devices on Oracle VM Oracle RAC Cluster
Description of "Figure 2-8 Quorum Disk Devices on Oracle VM Oracle RAC Cluster"
Note that for pkey-enabled environments, the interfaces used for discovering the targets should be the pkey interfaces used for the Oracle Clusterware communication. These interfaces are listed using the following command:
Grid_home/bin/oifcfg getif | grep cluster_interconnect | awk '{print $1}'
The Quorum Disk Manager utility (quorumdiskmgr) is used to create and manage all the necessary components including the iSCSI configuration, the iSCSI targets, the iSCSI LUNs, and the iSCSI devices for implementing this feature.
2.18.2 Software Requirements for Quorum Disk Manager
To use this feature, the following releases are required:
-
Oracle Exadata software release 12.1.2.3.0 and above
-
Patch 23200778 for all Database homes
-
Oracle Grid Infrastructure 12.1.0.2.160119 with patches 22722476 and 22682752, or Oracle Grid Infrastructure 12.1.0.2.160419 and above
Note that for new deployments, OEDA installs the patches automatically.
2.18.3 Adding Quorum Disks to Database Nodes
You can add quorum disks to database nodes on an Oracle Exadata rack that contains a high redundancy disk group with fewer than 5 storage servers.
The example in this section creates quorum disks for a quarter rack with two database nodes: db01 and db02.
This is an active-active system. On both db01 and db02 there are two InfiniBand ports: ib0 and ib1.
The network interfaces to be used for communication with the iSCSI devices can be found using the following command:
$ oifcfg getif | grep cluster_interconnect | awk '{print $1}'
The IP address of each interface can be found using the following command:
ip addr show interface_name
The InfiniBand IP addresses for this example are as follows:
On db01:
-
Network interface: ib0, IP address: 192.168.10.45
-
Network interface: ib1, IP address: 192.168.10.46
On db02:
-
Network interface: ib0, IP address: 192.168.10.47
-
Network interface: ib1, IP address: 192.168.10.48
The Oracle ASM disk group to which the quorum disks will be added is DATAC1. The Oracle ASM owner is grid, and the user group is dba.
Initially, the voting files reside on a normal redundancy disk group RECOC1:
$ Grid_home/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 21f5507a28934f77bf3b7ecf88b26c47 (o/192.168.76.187;192.168.76.188/RECOC1_CD_00_celadm12) [RECOC1]
2. ONLINE 387f71ee81f14f38bfbdf0693451e328 (o/192.168.76.189;192.168.76.190/RECOC1_CD_00_celadm13) [RECOC1]
3. ONLINE 6f7fab62e6054fb8bf167108cdbd2f64 (o/192.168.76.191;192.168.76.192/RECOC1_CD_00_celadm14) [RECOC1]
Located 3 voting disk(s).
2.18.4 Recreating Quorum Disks
In certain circumstances, you might need to recreate a quorum disk.
-
When recreating a guest domU
-
If you deleted the quorum disks without first dropping the quorum disks from the Oracle ASM disk group
Related Topics
2.18.5 Use Cases
2.18.5.1 New Deployments on Oracle Exadata 12.1.2.3.0 or Later
For new deployments on Oracle Exadata release 12.1.2.3.0 and above, OEDA implements this feature by default when all of the following requirements are satisfied:
-
The system has at least two database nodes and fewer than five storage servers.
-
You are running OEDA release February 2016 or later.
-
You meet the software requirements listed in Software Requirements for Quorum Disk Manager.
-
Oracle Database is 11.2.0.4 and above.
-
The system has at least one high redundancy disk group.
If the system has three storage servers in place, then two quorum disks will be created on the first two database nodes of the cluster picked by OEDA.
If the system has four storage servers in place, then one quorum disk will be created on the first database node picked by OEDA.
2.18.5.2 Upgrading to Oracle Exadata Release 12.1.2.3.0 or Later
If the target Exadata system has fewer than five storage servers, at least one high redundancy disk group, and two or more database nodes, you can implement this feature manually using quorumdiskmgr.
Related Topics
2.18.5.3 Downgrading to a Pre-12.1.2.3.0 Oracle Exadata Release
Rolling back to a pre-12.1.2.3.0 Oracle Exadata release, which does not support quorum disks, from a release that supports quorum disks, which is any release 12.1.2.3.0 and later, requires quorum disk configuration to be removed if the environment has quorum disk implementation in place. You need to remove the quorum disk configuration before performing the Exadata software rollback.
To remove quorum disk configuration, perform these steps:
-
Ensure there is at least one normal redundancy disk group in place. If not, create one.
-
Relocate the voting files to a normal redundancy disk group:
$GI_HOME/bin/crsctl replace votedisk +normal_redundancy_diskgroup
-
Drop the quorum disks from ASM. Run the following command for each quorum disk:
SQL> alter diskgroup diskgroup_name drop quorum disk quorum_disk_name force;
Wait for the rebalance operation to complete. You can tell it is complete when
v$asm_operationreturns no rows for the disk group. -
Delete the quorum devices. Run the following command from each database node that has quorum disks in place:
/opt/oracle.SupportTools/quorumdiskmgr --delete --device [--asm-disk-group asm_disk_group] [--host-name host_name]
-
Delete the targets. Run the following command from each database node that has quorum disks in place:
/opt/oracle.SupportTools/quorumdiskmgr --delete --target [--asm-disk-group asm_disk_group]
-
Delete the configuration. Run the following command from each database node that has quorum disks in place:
/opt/oracle.SupportTools/quorumdiskmgr --delete –config
2.18.5.4 Changing Elastic Configurations
2.18.5.4.1 Adding a Database Node
If the existing Oracle RAC cluster has fewer than two database nodes and fewer than five storage servers, and the voting files are not stored in a high redundancy disk group, then Oracle recommends adding quorum disks to the database node(s) and relocating the voting files to a high redundancy disk group.
Note:
The requirements listed in "Software Requirements for Quorum Disk Manager" must be met.
If the existing Oracle RAC cluster already has quorum disks in place, the quorum disks need to be made visible to the newly added node prior to adding the node to the Oracle RAC cluster using the addnode.sh procedure. Follow the steps below for making the quorum disks visible to the node being added:
Related Topics
2.18.5.4.2 Removing a Database Node
If the database node being removed did not host a quorum disk, then no action is required.
If database node being removed hosted a quorum disk containing a voting file and there are fewer than five storage servers in the RAC cluster, then a quorum disk must be created on a different database node before the database node is removed. Follow these steps:
-
Create a quorum disk on a database node that does not currently host a quorum disk.
-
Log into db01 and db02 as root.
-
Run the quorumdiskmgr command with the
--create --configoptions to create quorum disk configurations on both db01 and db02.# /opt/oracle.SupportTools/quorumdiskmgr --create --config --owner=grid --group=dba --network-iface-list="ib0, ib1"
-
Run the quorumdiskmgr command with the
--list --configoptions to verify that the configurations have been successfully created on both db01 and db02.# /opt/oracle.SupportTools/quorumdiskmgr --list --config
The output should look like:
Owner: grid Group: dba ifaces: exadata_ib1 exadata_ib0
-
Run the quorumdiskmgr command with the
--create --targetoptions to create a target on both db01 and db02 for ASM disk group DATAC1 and make the target visible to both db01 and db02.# /opt/oracle.SupportTools/quorumdiskmgr --create --target --asm-disk-group=datac1 --visible-to="192.168.10.45, 192.168.10.46, 192.168.10.47, 192.168.10.48"
-
Run the quorumdiskmgr command with the
--list --targetoptions to verify the target has been successfully created on both db01 and db02.# /opt/oracle.SupportTools/quorumdiskmgr --list --target
On db01, the output should look like:
Name: iqn.2015-05.com.oracle:QD_DATAC1_DB01 Size: 128 MB Host name: DB01 ASM disk group name: DATAC1 Visible to: 192.168.10.45, 192.168.10.46, 192.168.10.47, 192.168.10.48 Discovered by:
On db02, the output should be:
Name: iqn.2015-05.com.oracle:QD_DATAC1_DB02 Size: 128 MB Host name: DB02 ASM disk group name: DATAC1 Visible to: 192.168.10.45, 192.168.10.46, 192.168.10.47, 192.168.10.48 Discovered by:
-
Run the quorumdiskmgr command with the
--create --deviceoptions to create devices on both db01 and db02 from targets on both db01 and db02.# /opt/oracle.SupportTools/quorumdiskmgr --create --device --target-ip-list="192.168.10.45, 192.168.10.46, 192.168.10.47, 192.168.10.48"
-
Run the quorumdiskmgr command with the --list --device options to verify the devices have been successfully created on both db01 and db02.
# /opt/oracle.SupportTools/quorumdiskmgr --list --device
On both db01 and db02, the output should look like:
Device path: /dev/exadata_quorum/QD_DATAC1_DB01 Size: 128 MB Host name: DB01 ASM disk group name: DATAC1 Device path: /dev/exadata_quorum/QD_DATAC1_DB02 Size: 128 MB Host name: DB02 ASM disk group name: DATAC1
-
Add the two quorum disk devices to a high redundancy ASM disk group.
If there is no high redundancy disk group, create a high redundancy disk group and include the two new quorum disks. For example:
SQL> create diskgroup DATAC1 high redundancy quorum failgroup db01 disk '/dev/exadata_quorum/QD_ DATAC1_DB01' quorum failgroup db02 disk '/dev/exadata_quorum/QD_ DATAC1_DB02' ...
If a high redundancy disk group already exists, add the two new quorum disks. For example:
SQL> alter diskgroup datac1 add quorum failgroup db01 disk '/dev/exadata_quorum/QD_DATAC1_DB02' quorum failgroup db02 disk '/dev/exadata_quorum/QD_DATAC1_DB01';
-
-
Once the database node is removed, its voting file will get relocated automatically to the quorum disk added in step 1.
2.18.5.4.3 Adding an Oracle Exadata Storage Server and Expanding an Existing High Redundancy Disk Group
When you add a storage server that uses quorum disks, Oracle recommends relocating a voting file from a database node to the newly added storage server.
-
Add the Exadata storage server. See Adding a Cell Node for details.
In the example below, the new storage server added is called "celadm04".
-
After the storage server is added, verify the new fail group from
v$asm_disk.SQL> select distinct failgroup from v$asm_disk; FAILGROUP ------------------------------ ADM01 ADM02 CELADM01 CELADM02 CELADM03 CELADM04
-
Verify at least one database node has a quorum disk containing a voting file.
$ crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 834ee5a8f5054f12bf47210c51ecb8f4 (o/192.168.12.125;192.168.12.126/DATAC5_CD_00_celadm01) [DATAC5] 2. ONLINE f4af2213d9964f0bbfa30b2ba711b475 (o/192.168.12.127;192.168.12.128/DATAC5_CD_00_celadm02) [DATAC5] 3. ONLINE ed61778df2964f37bf1d53ea03cd7173 (o/192.168.12.129;192.168.12.130/DATAC5_CD_00_celadm03) [DATAC5] 4. ONLINE bfe1c3aa91334f16bf78ee7d33ad77e0 (/dev/exadata_quorum/QD_DATAC5_ADM01) [DATAC5] 5. ONLINE a3a56e7145694f75bf21751520b226ef (/dev/exadata_quorum/QD_DATAC5_ADM02) [DATAC5] Located 5 voting disk(s).
The example above shows there are two quorum disks with voting files on two database nodes.
-
Drop one of the quorum disks.
SQL> alter diskgroup datac5 drop quorum disk QD_DATAC5_ADM01;
The voting file on the dropped quorum disk will be relocated automatically to the newly added storage server by the Grid Infrastructure as part of the voting file refresh. You can verify this as follows:
$ crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 834ee5a8f5054f12bf47210c51ecb8f4 (o/192.168.12.125;192.168.12.126/DATAC5_CD_00_celadm01) [DATAC5] 2. ONLINE f4af2213d9964f0bbfa30b2ba711b475 (o/192.168.12.127;192.168.12.128/DATAC5_CD_00_celadm02) [DATAC5] 3. ONLINE ed61778df2964f37bf1d53ea03cd7173 (o/192.168.12.129;192.168.12.130/DATAC5_CD_00_celadm03) [DATAC5] 4. ONLINE a3a56e7145694f75bf21751520b226ef (/dev/exadata_quorum/QD_DATAC5_ADM02) [DATAC5] 5. ONLINE ab5aefd60cf84fe9bff6541b16e33787 (o/192.168.12.131;192.168.12.132/DATAC5_CD_00_celadm04) [DATAC5]
2.18.5.4.4 Removing an Oracle Exadata Storage Server
If removing a storage server results in the number of storage servers being used by the Oracle RAC cluster to fewer than five, and the voting files reside in a high redundancy disk group, then Oracle recommends adding quorum disks to the database nodes, if not in place already.
Prior to removing the storage server, add the quorum disks so that five copies of the voting files are available immediately after removing the storage server.
Related Topics
2.18.6 quorumdiskmgr Reference
The quorum disk manager utility (quorumdiskmgr) runs on each database server to enable you to create and manage iSCSI quorum disks on database servers. You use quorumdiskmgr to create, list, alter, and delete iSCSI quorum disks on database servers. The utility is installed on database servers when they are shipped.
This reference section contains the following topics:
2.18.6.1 Syntax for the Quorum Disk Manager Utility
The quorum disk manager utility is a command-line tool. It has the following syntax:
quorumdiskmgr --verb --object [--options]
verb is an action performed on an object. It is one of: alter, create, delete, list.
object is an object on which the command performs an action.
options extend the use of a command combination to include additional parameters for the command.
When using the quorumdiskmgr utility, the following rules apply:
-
Verbs, objects, and options are case-sensitive except where explicitly stated.
-
Use the double quote character around the value of an option that includes spaces or punctuation.
2.18.6.2 quorumdiskmgr Objects
| Object | Description |
|---|---|
|
|
The quorum disk configurations include the owner and group of the ASM instance to which the iSCSI quorum disks will be added, and the list of network interfaces through which local and remote iSCSI quorum disks will be discovered. |
|
|
A target is an endpoint on each database server that waits for an iSCSI initiator to establish a session and provides required IO data transfer. |
|
|
A device is an iSCSI device created by logging into a local or remote target. |
2.18.6.3 Creating a Quorum Disk Configuration (--create --config)
The --create --config action creates a quorum disk configuration. The configuration must be created before any targets or devices can be created.
Syntax
quorumdiskmgr --create --config [--owner owner --group group] --network-iface-list network-iface-list
Parameters
The following table lists the parameters for the --create --config action:
| Parameter | Description |
|---|---|
|
owner |
Specifies the owner of the ASM instance to which the iSCSI quorum disks will be added. This is an optional parameter. The default value is |
|
group |
Specifies the group of the ASM instance to which the iSCSI quorum disks will be added. This is an optional parameter. The default value is |
|
network-iface-list |
Specifies the list of network interface names through which the local and remote targets will be discovered. |
Example
quorumdiskmgr --create --config --owner=oracle --group=dba --network-iface-list="ib0, ib1"
2.18.6.4 Creating a Target (--create --target)
The --create --target action creates a target that can be accessed by database servers with an InfiniBand IP address in the specified InfiniBand IP address list and will be used to create devices that will be added to the specified ASM disk group.
After a target is created, its asm-disk-group, host-name, and size attributes cannot be changed.
Syntax
quorumdiskmgr --create --target --asm-disk-group asm_disk_group --visible-to infiniband_ip_list [--host-name host_name] [--size size]
Parameters
| Parameter | Description |
|---|---|
|
asm-disk-group |
Specifies the ASM disk group to which the device created from the target will be added. The value of asm-disk-group is not case-sensitive. |
|
visible-to |
Specifies a list of InfiniBand IP addresses. Database servers with an InfiniBand IP address in the list will have access to the target. |
|
host-name |
Specifies the host name of the database server on which quorumdiskmgr runs. The total length of asm-disk-group and host-name cannot exceed 26 characters. If the host name is too long, a shorter host name can be specified as long as a different host name is specified for each database server in the quarter rack. This is an optional parameter. The default value is the host name of the database server on which quorumdiskmgr runs. The value of host-name is not case-sensitive. |
|
size |
Specifies the size of the target. This is an optional parameter. The default value is 128 MB. |
Example
quorumdiskmgr --create --target --size=128MB --asm-disk-group=datac1 --visible-to="192.168.10.45, 192.168.10.46" --host-name=db01
2.18.6.5 Creating a Device (--create --device)
The --create --device action creates devices by discovering and logging into targets on database servers with an InfiniBand IP address in the specified list of IP addresses.
The created devices will be automatically discovered by the ASM instance with the owner and group specified during configuration creation.
Syntax
quorumdiskmgr --create --device --target-ip-list target_ip_list
Parameters
| Parameter | Description |
|---|---|
|
target-ip-list |
Specifies a list of InfiniBand IP addresses. quorumdiskmgr will discover targets on database servers with an InfiniBand IP address in the list and log into the targets to create devices. |
Example
quorumdiskmgr --create --device --target-ip-list="192.168.10.45, 192.168.10.46"
2.18.6.6 Listing Quorum Disk Configurations (--list --config)
The --list --config action lists the quorum disk configurations.
Syntax
quorumdiskmgr --list --config
Sample Output
Owner: grid Group: dba ifaces: exadata_ib1 exadata_ib0
2.18.6.7 Listing Targets (--list --target)
The --list --target action lists the attributes of targets, including target name, size, host name, ASM disk group name, the list of IP addresses (a visible-to IP address list) indicating which database servers have access to the target, and the list of IP addresses (a discovered-by IP address list) indicating which database servers have logged into the target.
If an ASM disk group name is specified, the action lists all local targets created for the specified ASM disk group. Otherwise, the action lists all local targets created for quorum disks.
Syntax
quorumdiskmgr --list --target [--asm-disk-group asm_disk_group]
parameters
| Parameter | Description |
|---|---|
|
asm-disk-group |
Specifies the ASM disk group. quorumdiskmgr displays all local targets for this ASM disk group. The value of asm-disk-group is not case-sensitive. |
Sample Output
Name: iqn.2015-05.com.oracle:QD_DATAC1_DB01 Size: 128 MB Host name: DB01 ASM disk group name: DATAC1 Visible to: 192.168.10.48, 192.168.10.49, 192.168.10.46, 192.168.10.47 Discovered by: 192.168.10.47, 192.168.10.46
2.18.6.8 Listing Devices (--list --device)
The --list --device action lists the attributes of devices, including device path, size, host name and ASM disk group name.
If only an ASM disk group name is specified, the action lists all the devices that have been added to the ASM disk group.
If only a host name is specified, the action lists all the devices created from the targets on the host.
If both an ASM disk group name and a host name are specified, the action lists a single device created from the target on the host and that has been added to the ASM disk group.
If neither an ASM disk group name nor a host name is specified, the action lists all quorum disk devices.
Syntax
quorumdiskmgr --list --device [--asm-disk-group asm_disk_group] [--host-name host_name]
parameters
| Parameter | Description |
|---|---|
|
asm-disk-group |
Specifies the ASM disk group to which devices have been added. The value of asm-disk-group is not case-sensitive. |
|
host-name |
Specifies the host name of the database server from whose targets devices are created. The value of host-name is not case-sensitive. |
Sample Output
Device path: /dev/exadata_quorum/QD_DATAC1_DB01 Size: 128 MB Host name: DB01 ASM disk group name: DATAC1 Device path: /dev/exadata_quorum/QD_DATAC1_DB02 Size: 128 MB Host name: DB02 ASM disk group name: DATAC1
2.18.6.9 Deleting Configurations (--delete --config)
The --delete --config action deletes quorum disk configurations. The configurations can only be deleted when there are no targets or devices present.
Syntax
quorumdiskmgr --delete --config
2.18.6.10 Deleting Targets (--delete --target)
The --delete --target action deletes the targets created for quorum disks on database servers.
If an ASM disk group name is specified, the action deletes all local targets created for the specified ASM disk group. Otherwise, the action deletes all local targets created for quorum disks.
Syntax
quorumdiskmgr --delete --target [--asm-disk-group asm_disk_group]
Parameters
| Parameter | Description |
|---|---|
|
asm-disk-group |
Specifies the ASM disk group. Local targets created for this disk group will be deleted. The value of asm-disk-group is not case-sensitive. |
Example
quorumdiskmgr --delete --target --asm-disk-group=datac1
2.18.6.11 Deleting Devices (--delete --device)
The --delete --device action deletes quorum disk devices.
If only an ASM disk group name is specified, the action deletes all the devices that have been added to the ASM disk group.
If only a host name is specified, the action deletes all the devices created from the targets on the host.
If both an ASM disk group name and a host name are specified, the action deletes a single device created from the target on the host and that has been added to the ASM disk group.
If neither an ASM disk group name nor a host name is specified, the action deletes all quorum disk devices.
Syntax
quorumdiskmgr --delete --device [--asm-disk-group asm_disk_group] [--host-name host_name]
Parameters
| Parameter | Description |
|---|---|
|
asm-disk-group |
Specifies the ASM disk group whose device you want to delete. The value of asm-disk-group is not case-sensitive. |
|
host-name |
Specifies the host name of the database server. Devices created from targets on this host will be deleted. The value of host-name is not case-sensitive. |
Example
quorumdiskmgr --delete --device --host-name=db01
2.18.6.12 Changing Owner and Group Values (--alter --config)
The --alter --config action changes the owner and group configurations.
Syntax
quorumdiskmgr --alter --config --owner owner --group group
Parameters
| Parameter | Description |
|---|---|
|
owner |
Specifies the new owner for the quorum disk configuration. This parameter is optional. If not specified, the owner is unchanged. |
|
group |
Specifies the new group for the quorum disk configuration. This parameter is optional. If not specified, the group is unchanged. |
Example
quorumdiskmgr --alter --config --owner=grid --group=dba
2.18.6.13 Changing the InfiniBand IP Addresses (--alter --target)
The --alter --target action changes the InfiniBand IP addresses of the database servers that have access to the local target created for the specified ASM disk group.
Syntax
quorumdiskmgr --alter --target --asm-disk-group asm_disk_group --visible-to infiniband_ip_list
Parameters
| Parameter | Description |
|---|---|
|
asm-disk-group |
Specifies the ASM disk group to which the device created from the target will be added. The value of asm-disk-group is not case-sensitive. |
|
visible-to |
Specifies a list of InfiniBand IP addresses. Database servers with an InfiniBand IP address in the list will have access to the target. |
Example
quorumdiskmgr --alter --target --asm-disk-group=datac1 --visible-to="192.168.10.45, 192.168.10.47"
2.19 Using vmetrics
The vmetrics package enables you to display system statistics gathered by the vmetrics service. You can access the system statistics from dom0 or domU. The vmetrics service runs on dom0, collects the statistics, and pushes them to the xenstore. This allows the domU's to access the statistics.
System statistics collected by the vmetrics service are shown below, with sample values:
com.sap.host.host.VirtualizationVendor=Oracle Corporation;
com.sap.host.host.VirtProductInfo=Oracle VM 3;
com.sap.host.host.PagedInMemory=0;
com.sap.host.host.PagedOutMemory=0;
com.sap.host.host.PageRates=0;
com.sap.vm.vm.uuid=2b80522b-060d-47ee-8209-2ab65778eb7e;
com.sap.host.host.HostName=scac10adm01.us.oracle.com;
com.sap.host.host.HostSystemInfo=scac10adm01;
com.sap.host.host.NumberOfPhysicalCPUs=24;
com.sap.host.host.NumCPUs=4;
com.sap.host.host.TotalPhyMem=98295;
com.sap.host.host.UsedVirtualMemory=2577;
com.sap.host.host.MemoryAllocatedToVirtualServers=2577;
com.sap.host.host.FreeVirtualMemory=29788;
com.sap.host.host.FreePhysicalMemory=5212;
com.sap.host.host.TotalCPUTime=242507.220000;
com.sap.host.host.Time=1453150151;
com.sap.vm.vm.PhysicalMemoryAllocatedToVirtualSystem=8192;
com.sap.vm.vm.ResourceMemoryLimit=8192;
com.sap.vm.vm.TotalCPUTime=10160.1831404;
com.sap.vm.vm.ResourceProcessorLimit=4;
2.19.1 Installing and Starting the vmetrics Service
To install the vmetrics service, run the install.sh script as the root user on dom0:
[root@scac10adm01]# cd /opt/oracle.SupportTools/vmetrics [root@scac10adm01]# ./install.sh
The install.sh script verifies that it is running on dom0, stops any vmetrics services currently running, copies the package files to /opt/oracle.vmetrics, and copies vmetrics.svc to /etc/init.d.
To start the vmetrics service on dom0, run the following command as the root user on dom0:
[root@scac10adm01 vmetrics]# service vmetrics.svc start
The commands to gather the statistics are run every 30 seconds.
2.19.2 Files in the vmetrics Package
The vmetrics package contains the following files:
| File | Description |
|---|---|
|
|
This file installs the package. |
|
|
This script reads the statistics from the xenstore and displays them in XML format. |
|
|
This Python script runs the system commands and uploads them to the xenstore. The system commands are listed in the |
|
|
This XML file specifies the metrics that the dom0 should push to the xenstore, and the system commands to run for each metric. |
|
|
The |
2.19.3 Displaying the Statistics
Once the statistics have been pushed to the xenstore, you can view the statistics on dom0 and domU by running either of the following commands:
Note:
On domU's, ensure that the xenstoreprovider and ovmd packages are installed.
xenstoreprovider is the library which communicates with the ovmapi kernel infrastructure.
ovmd is a daemon that handles configuration and reconfiguration events and provides a mechanism to send/receive messages between the VM and the Oracle VM Manager.
The following command installs the necessary packages on Oracle Linux 5 and 6 to support the Oracle VM API.
# yum install ovmd xenstoreprovider
-
The
/usr/sbin/ovmd -g vmhostcommand displays the statistics on one line. Thesedcommand breaks up the line into multiple lines, one statistic per line. You need to run this command as the root user.root@scac10db01vm04 ~]# /usr/sbin/ovmd -g vmhost |sed 's/; */;\n/g;s/:"/:"\n/g' com.sap.host.host.VirtualizationVendor=Oracle Corporation; com.sap.host.host.VirtProductInfo=Oracle VM 3; com.sap.host.host.PagedInMemory=0; com.sap.host.host.PagedOutMemory=0; com.sap.host.host.PageRates=0; com.sap.vm.vm.uuid=2b80522b-060d-47ee-8209-2ab65778eb7e; com.sap.host.host.HostName=scac10adm01.us.oracle.com; com.sap.host.host.HostSystemInfo=scac10adm01; com.sap.host.host.NumberOfPhysicalCPUs=24; com.sap.host.host.NumCPUs=4; ...
-
The
vm-dump-metricscommand displays the metrics in XML format.[root@scac10db01vm04 ~]# ./vm-dump-metrics <metrics> <metric type='real64' context='host'> <name>TotalCPUTime</name> <value>242773.600000</value> </metric> <metric type='uint64' context='host'> <name>PagedOutMemory</name> <value>0</value> </metric> ...
Note that you have copy the
vm-dump-metricscommand to the domU's from which you want to run the command.