2 Maintaining Database Servers of Oracle Exadata Database Machine

This chapter contains the following topics:

• Management Server on Database Servers

• Maintaining the Hard Disks of Oracle Database Servers

• Maintaining Flash Disks on Exadata Database Servers

• Adding Disk Expansion Kit to Database Servers

• Adding Memory Expansion Kit to Database Servers

• Verify the Link Speed on Network Ports

• Adding and Configuring an Extra Network Card on Oracle Exadata X6-2 and X7-2

• Increasing the Number of Active Cores on Database Servers

• Migrating a Bare Metal Oracle RAC Cluster to an Oracle RAC Cluster in Oracle VM

• Managing Oracle VM Domains on Oracle Exadata Database Machine

• Extending LVM Partitions

• Creating a Snapshot-Based Backup of Oracle Linux Database Server

• Backing up the Management Domain (dom0) and User Domains (domU) in an Oracle Virtual Server Deployment

• Recovering Oracle Linux Database Server Using a Snapshot-Based Backup

• Recovering in an Oracle Virtual Server Deployment

• Updating Oracle Linux Database Servers

• Re-Imaging Oracle Linux Database Servers

• Changing Existing Elastic Configurations for Database Servers

• Managing Quorum Disks for High Redundancy Disk Groups

• Using vmetrics

Note:

For ease of reading, the name "Oracle Exadata Rack" is used when information refers to both Oracle Exadata Database Machine and Oracle Exadata Storage Expansion Rack.

2.1 Management Server on Database Servers

Management Server (MS) running on database servers provides monitoring, alerting, and other administrative capabilities. It also provides the DBMCLI command-line administration tool.

See Also:

• "Management Server on Database Servers" in the Oracle Exadata Database Machine System Overview guide

• Using the DBMCLI Utility

2.2 Maintaining the Hard Disks of Oracle Database Servers

Repair of the hard disks does not require the database server in Oracle Exadata Database Machine to be shut down.

No downtime of the rack is required, however individual servers may require downtime, and be taken outside of the cluster temporarily.

• Verifying the Database Server Configuration

• Monitoring a Database Server RAID Set Rebuilding

• Reclaiming a Hot Spare Drive After Upgrading to Oracle Exadata Storage Server Software Release 12.1.2.1.0 or Later

• Understanding Automated File Deletion Policy

2.2.1 Verifying the Database Server Configuration

The disks are configured RAID-5 configuration.

The disk drives in each database server are controlled by a MegaRAID SAS disk controller.

Table 2-1 Disk Configurations for Exadata Database Machine Two-Socket Systems

Server Type	RAID Controller	Disk Configuration
Oracle Exadata Database Machine X7-2	MegaRAID SAS 9361-16i	4 disk drives in each database server
Oracle Exadata Database Machine X6-2	MegaRAID SAS 9361-8i	4 disk drives in each database server
Oracle Exadata Database Machine X5-2	MegaRAID SAS 9361-8i	4 disk drives in each database server
Oracle Exadata Database Machine X4-2	MegaRAID SAS 9261-8i	4 disk drives in each database server
Oracle Exadata Database Machine X3-2	MegaRAID SAS 9261-8i	4 disk drives in each database server
Oracle Exadata Database Machine X2-2	MegaRAID SAS 9261-8i	4 disk drives in each database server

Table 2-2 Disk Configurations for Exadata Database Machine Eight-Socket Systems

Server Type	RAID Controller	Disk Configuration
Oracle Exadata Database Machine X7-8	MegaRAID SAS 9361-16i	Two NVMe flash accelerator cards in each database server
Oracle Exadata Database Machine X5-8	MegaRAID SAS 9361-8i	8 disk drives in each database server with one virtual drive created across the RAID set
Oracle Exadata Database Machine X4-8	MegaRAID SAS 9261-8i	7 disk drives in each database server configured as one 6-disk RAID-5 with one global hot spare drive by default
Oracle Exadata Database Machine X3-8	MegaRAID SAS 9261-8i	8 disk drives in each database server with one virtual drive created across the RAID set

Oracle recommends verifying the status of the database server RAID devices to avoid possible performance impact, or an outage. The impact of validating the RAID devices is minimal. The impact of corrective actions will vary depending on the specific issue uncovered, and may range from simple reconfiguration to an outage.

2.2.1.1 Verifying Disk Controller Configuration

Use the following command to verify the database server disk controller configuration:

/opt/MegaRAID/MegaCli/MegaCli64 -AdpAllInfo -aALL | grep "Device Present" -A 8

The following is an example of the output from the command for Oracle Exadata Database Machine X3-2 or Oracle Exadata Database Machine X2-2.

                Device Present
                ================
Virtual Drives    : 1 
  Degraded        : 0 
  Offline         : 0 
Physical Devices  : 5 
  Disks           : 4 
  Critical Disks  : 0 
  Failed Disks    : 0 

The following is an example of the output from the command for Oracle Exadata Database Machine X4-8 Full Rack:

                Device Present
                ================
Virtual Drives    : 1
  Degraded        : 0
  Offline         : 0
Physical Devices  : 8
  Disks           : 7
  Critical Disks  : 0
  Failed Disks    : 0

The following is an example of the output from the command for Oracle Exadata Database Machine X5-8 Full Rack:

                Device Present
                ================
Virtual Drives   : 1
  Degraded       : 0
  Offline        : 0
Physical Devices : 9
  Disks          : 8
  Critical Disks : 0
  Failed Disks   : 0

For Oracle Exadata Database Machine X4-2, Oracle Exadata Database Machine X3-2, and Oracle Exadata Database Machine X2-2, the expected output is one virtual drive, none degraded or offline, five physical devices (one controller + four disks), four disks, and no critical or failed disks.

For Oracle Exadata Database Machine X3-8 Full Rack and Oracle Exadata Database Machine X2-8 Full Rack, the expected output is one virtual drive, none degraded or offline, 11 physical devices (one controller + two SAS2 expansion ports+ eight disks), eight disks, and no critical or failed disks.

If the output is different, then investigate and correct the problem. Degraded virtual drives usually indicate absent or failed physical disks. Critical disks should be replaced immediately to avoid the risk of data loss if the number of failed disks in the node exceed the count needed to sustain the operations of the system. Failed disks should also be replaced quickly.

Note:

If additional virtual drives or a hot spare are present, then it may be that the procedure to reclaim disks was not performed at deployment time or that a bare metal restore procedure was performed without using the dualboot=no qualifier. Refer to My Oracle Support note 1323309.1 for additional information and corrective steps.

When upgrading a database server that has a hot spare to Oracle Exadata Storage Server Software release 11.2.3.2.0 or later, the hot spare is removed, and added as an active drive to the RAID configuration. The database servers have the same availability in terms of RAID 5 redundancy, and can survive the loss of one drive. When a drive failure happens Auto Service Request sends out a notification to replace the faulty drive at the earliest opportunity.

2.2.1.2 Verifying Virtual Drive Configuration

To verify the virtual drive configuration, use the following command to verify the virtual drive configuration:

/opt/MegaRAID/MegaCli/MegaCli64 CfgDsply -aALL | grep "Virtual Drive:";    \
/opt/MegaRAID/MegaCli/MegaCli64 CfgDsply -aALL | grep "Number Of Drives";  \
/opt/MegaRAID/MegaCli/MegaCli64 CfgDsply -aALL | grep "^State" 

The following is an example of the output for Oracle Exadata Database Machine X4-2, Oracle Exadata Database Machine X3-2 and Oracle Exadata Database Machine X2-2. The virtual device 0 should have four drives, and the state is Optimal.

Virtual Drive                 : 0 (Target Id: 0)
Number Of Drives              : 4
State                         : Optimal

The expected output for Oracle Exadata Database Machine X3-8 Full Rack and Oracle Exadata Database Machine X2-8 Full Rack displays the virtual device has eight drives and a state of optimal.

Note:

If a disk replacement was performed on a database server without using the dualboot=no option, then the database server may have three virtual devices. Refer to My Oracle Support note 1323309.1 for additional information and corrective steps.

2.2.1.3 Verifying Physical Drive Configuration

To verify physical drive configuration, use the following command to verify the database server physical drive configuration:

For Linux, use the following command to verify the database server physical drive configuration:

/opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL | grep "Firmware state"

The following is an example of the output for Oracle Exadata Database Machine X4-2, Oracle Exadata Database Machine X3-2, and Oracle Exadata Database Machine X2-2. The drives should be Online, Spun Up. The order of the output is not important. The output for Oracle Exadata Database Machine X3-8 Full Rack or Oracle Exadata Database Machine X2-8 Full Rack should be eight lines of output showing a state of Online, Spun Up.

Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up

If the output is different, then investigate and correct the problem.

Degraded virtual drives usually indicate absent or failed physical disks. Critical disks should be replaced immediately to avoid the risk of data loss if the number of failed disks in the node exceed the count needed to sustain the operations of the system. Failed disks should also be replaced quickly.

2.2.2 Monitoring a Database Server RAID Set Rebuilding

If a drive in a database server RAID set is replaced, then the progress of the RAID set rebuild should be monitored.

Use the following command on the database server that has the replaced disk. The command is run as the root user.

/opt/MegaRAID/MegaCli/MegaCli64 -pdrbld -showprog -physdrv \
[disk_enclosure:slot_number] -a0

In the preceding command, disk_enclosure and slot_number indicate the replacement disk identified by the MegaCli64 -PDList command. The following is an example of the output from the command:

Rebuild Progress on Device at Enclosure 252, Slot 2 Completed 41% in 13 Minutes.

2.2.3 Reclaiming a Hot Spare Drive After Upgrading to Oracle Exadata Storage Server Software Release 12.1.2.1.0 or Later

Oracle Exadata Database Machines upgraded to Oracle Exadata Storage Server Software release 12.1.2.1.0 or later that have a hot spare drive cannot use the reclaimdisks.sh script to reclaim the drive. The following procedure describes how to manually reclaim the drive:

Note:

During the procedure, the database server is restarted twice. The steps in the procedure assume that the Grid Infrastructure restart is disabled after the server restart.

The sample output shows Oracle Exadata Database Machine X2-2 database server with four disks. The enclosure identifier, slot number, and such may be different for your system.

1. Identify the hot spare drive using the following command:

   # /opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL
   

   The following is an example of the output from the command for the hot spare drive:

   ...
   Enclosure Device ID: 252
   Slot Number: 3
   Enclosure position: N/A
   Device Id: 8
   WWN: 5000CCA00A9FAA5F
   Sequence Number: 2
   Media Error Count: 0
   Other Error Count: 0
   Predictive Failure Count: 0
   Last Predictive Failure Event Seq Number: 0
   PD Type: SAS
   Hotspare Information:
   Type: Global, with enclosure affinity, is revertible
    
   Raw Size: 279.396 GB [0x22ecb25c Sectors]
   Non Coerced Size: 278.896 GB [0x22dcb25c Sectors]
   Coerced Size: 278.464 GB [0x22cee000 Sectors]
   Sector Size: 0
   Logical Sector Size: 0
   Physical Sector Size: 0
   Firmware state: Hotspare, Spun down
   Device Firmware Level: A2A8
   Shield Counter: 0
   Successful diagnostics completion on : N/A
   ...
   

   The command identified the hot spare drive on enclosure identifier 252, slot 3.

2. Obtain the virtual drive information using the following command:

   # /opt/MegaRAID/MegaCli/MegaCli64 -LDInfo -Lall -Aall
   

   The following is an example of the output from the command:

   Adapter 0 -- Virtual Drive Information:
   Virtual Drive: 0 (Target Id: 0)
   Name :DBSYS
   RAID Level : Primary-5, Secondary-0, RAID Level Qualifier-3
   Size : 556.929 GB
   Sector Size : 512
   Is VD emulated : No
   Parity Size : 278.464 GB
   State : Optimal
   Strip Size : 1.0 MB
   Number Of Drives : 3
   Span Depth : 1
   Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
   Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
   Default Access Policy: Read/Write
   Current Access Policy: Read/Write
   Disk Cache Policy : Disabled
   Encryption Type : None
   Is VD Cached: No
   

   The command identified a RAID 5 configuration for virtual drive 0 on adapter 0.

3. Remove the hot spare drive using the following command:

   # /opt/MegaRAID/MegaCli/MegaCli64 -PDHSP -Rmv -PhysDrv[252:3] -a0
   

4. Add the drive as an active RAID 5 drive using the following command:

   # /opt/MegaRAID/MegaCli/MegaCli64 -LDRecon -Start -r5     \
     -Add -PhysDrv[252:3] -L0 -a0
   
   Start Reconstruction of Virtual Drive Success.
   Exit Code: 0x00
   

   Note:

   If the message Failed to Start Reconstruction of Virtual Drive is displayed, then follow the instructions in My Oracle Support note 1505157.1, Failed to Start Reconstruction of Virtual Drive - Adding a hot-spare manually.

5. Watch the progress of the RAID reconstruction using the following command:

   # /opt/MegaRAID/MegaCli/MegaCli64 -LDRecon -ShowProg -L0 -a0
   
   Reconstruction on VD #0 (target id #0) Completed 1% in 2 Minutes.
   

   The following output shows the output of the command after the hot spare drive is added to the RAID 5 configuration, and the reconstruction is finished:

   Reconstruction on VD #0 is not in Progress.
   

6. Verify the number of drives using the following command:

   # /opt/MegaRAID/MegaCli/MegaCli64 -LDInfo -Lall -Aall
   

   The following is an example of the output from the command:

   Adapter 0 -- Virtual Drive Information:
   Virtual Drive: 0 (Target Id: 0)
   Name :DBSYS
   RAID Level : Primary-5, Secondary-0, RAID Level Qualifier-3
   Size : 835.394 GB
   Sector Size : 512
   Is VD emulated : No
   Parity Size : 278.464 GB
   State : Optimal
   Strip Size : 1.0 MB
   Number Of Drives : 4
   Span Depth : 1
   Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
   Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
   Default Access Policy: Read/Write
   Current Access Policy: Read/Write
   Disk Cache Policy : Disabled
   Encryption Type : None
   Is VD Cached: No
   

7. Check the size of the RAID using the following command:

   # parted /dev/sda print
   
   Model: LSI MR9261-8i (scsi)
   Disk /dev/sda: 598GB
   Sector size (logical/physical): 512B/4096B
   Partition Table: msdos
    
   Number Start End Size Type File system Flags
   1 32.3kB 132MB 132MB primary ext3 boot
   2 132MB 598GB 598GB primary lvm 
   

8. Restart the server in order for the changes to take effect.
9. Check the size of the RAID using the following command:

   # parted /dev/sda print
   
   Model: LSI MR9261-8i (scsi)
   Disk /dev/sda: 897GB
   Sector size (logical/physical): 512B/4096B
   Partition Table: msdos
    
   Number Start End Size Type File system Flags
   1 32.3kB 132MB 132MB primary ext3 boot
   2 132MB 598GB 598GB primary lvm
   

   The increased RAID size allows for extending the volume group. In order to extend the volume group, an additional partition must be added to the drive.

10. Obtain the new size, in sectors, using the following command:

    # parted /dev/sda
    
    GNU Parted 2.1
    Using /dev/sda
    Welcome to GNU Parted! Type 'help' to view a list of commands.
    (parted) unit s
    (parted) print
    Model: LSI MR9261-8i (scsi)
    Disk /dev/sda: 1751949312s
    Sector size (logical/physical): 512B/4096B
    Partition Table: msdos
     
    Number Start End Size Type File system Flags
    1 63s 257039s 256977s primary ext3 boot
    2 257040s 1167957629s 1167700590s primary lvm
    

    In the preceding example, a third partition can be created starting at sector 1167957630, and ending at the end of the disk at sector 1751949311.

11. Create an additional partition on the drive using the following command:

    # parted /dev/sda
    
    GNU Parted 2.1
    Using /dev/sda
    Welcome to GNU Parted! Type 'help' to view a list of commands.
    (parted) unit s
     
    (parted) mkpart
     
    Partition type? primary/extended? primary
    File system type? [ext2]? ext2 
    Start? 1167957630
    End? 1751949311
    Warning: The resulting partition is not properly aligned for best performance.
    Ignore/Cancel? Ignore
    Warning: WARNING: the kernel failed to re-read the partition table on /dev/sda (Device or resource busy). As a
    result, it may not reflect all of your changes until after reboot.
    (parted)
     
    (parted) print
    Model: LSI MR9261-8i (scsi)
    Disk /dev/sda: 1751949312s
    Sector size (logical/physical): 512B/4096B
    Partition Table: msdos
     
    Number Start End Size Type File system Flags
    1 63s 257039s 256977s primary ext3 boot
    2 257040s 1167957629s 1167700590s primary lvm
    3 1167957630s 1751949311s 583991682s primary
     
    (parted) set 3 lvm on 
     
    Warning: WARNING: the kernel failed to re-read the partition table on /dev/sda (Device or resource busy). As a
    result, it may not reflect all of your changes until after reboot.
    (parted) print
    Model: LSI MR9261-8i (scsi)
    Disk /dev/sda: 1751949312s
    Sector size (logical/physical): 512B/4096B
    Partition Table: msdos
     
    Number Start End Size Type File system Flags
    1 63s 257039s 256977s primary ext3 boot
    2 257040s 1167957629s 1167700590s primary lvm
    3 1167957630s 1751949311s 583991682s primary lvm
    

12. Restart the database server.
13. Create the physical volume using the following command:

    # pvcreate /dev/partition_name
    

14. Add the physical volume to the existing volume group using the following command:

    # vgextend volume_group /dev/partition_name
     
    Volume group "volume_name" successfully extended 
    

15. Resize the logical volume and file systems as described in "Extending LVM Partitions."

2.2.4 Understanding Automated File Deletion Policy

Management Server (MS) includes a file deletion policy for the / (root) directory on the database servers that is triggered when file system utilization is high. Deletion of files is triggered when file utilization is 80 percent, and an alert is sent before the deletion begins. The alert includes the name of the directory, and space usage for the subdirectories. In particular, the deletion policy is as follows:

Files in the following directories are deleted using a policy based on the file modification time stamp.

• /opt/oracle/dbserver/log

• /opt/oracle/dbserver/dbms/deploy/config/metrics

• /opt/oracle/dbserver/dbms/deploy/log

Files that are older than the number of days set by the metricHistoryDays attribute are deleted first, then successive deletions occur for earlier files, down to files with modification time stamps older than or equal to 10 minutes, or until file system utilization is less than 75 percent. The metricHistoryDays attribute applies to files in /opt/oracle/dbserver/dbms/deploy/config/metrics. For the other log and trace files, use the diagHistoryDays attribute.

Starting with 12.1.2.2.0, the maximum amount of space for ms-odl.trc and ms-odl.log files is 100 MB (twenty 5 MB files) for trc files and 100 MB (twenty 5 MB files) for log files. Previously, it was 50 MB (ten 5 MB files) for both trc and log files.

ms-odl generation files are renamed when they reach 5 MB, and the oldest are deleted when the files use up 100 MB of space.

2.3 Maintaining Flash Disks on Exadata Database Servers

Starting with Exadata Database Machine X7-8, the database servers contain flash devices instead of hard disks. These flash devices can be replaced without shutting down the server.

• Monitoring the Status of Flash Disks

• Performing a Hot-Pluggable Replacement of a Flash Disk

2.3.1 Monitoring the Status of Flash Disks

You can monitor the status of a flash disk on the Exadata Database Machine by checking its attributes with the DBMCLI LIST PHYSICALDISK command.

For example, a flash disk status equal to failed is probably having problems and needs to be replaced.

• Use the DBMCLI command LIST PHSYICALDISK to determine the status of a flash disk:

  DBMCLI> LIST PHYSICALDISK WHERE disktype=flashdisk AND status!=normal DETAIL
           name:               FLASH_1_1
           deviceName:         /dev/nvme0n1
           diskType:           FlashDisk
           luns:               1_1
           makeModel:          "Oracle Flash Accelerator F640 PCIe Card"
           physicalFirmware:   QDV1RD09
           physicalInsertTime: 2017-08-11T12:25:00-07:00
           physicalSerial:     PHLE6514003R6P4BGN-1
           physicalSize:       2.910957656800747T
           slotNumber:         "PCI Slot: 1; FDOM: 1"
           status:             failed - dropped for replacement
  

The Exadata Database Server flash disk statuses are as follows:

• normal

• normal - dropped for replacement

• failed

• failed - dropped for replacement

• failed - rejected due to incorrect disk model

• failed - rejected due to incorrect disk model - dropped for replacement

• failed - rejected due to wrong slot

• failed - rejected due to wrong slot - dropped for replacement

• warning - peer failure

• warning - predictive failure, write-through caching

• warning - predictive failure

• warning - predictive failure - dropped for replacement

• warning - write-through caching

2.3.2 Performing a Hot-Pluggable Replacement of a Flash Disk

Starting with Exadata Database Machine X7-8, the database server uses hot-pluggable flash disks instead of hard disk drives.

1. Determine if the flash disk is ready to be replaced.
   When performing a hot-pluggable replacement of a flash device on an Exadata X7-8 database server, the disk status should be Dropped for replacement, which indicates the flash disk is ready for online replacement.

   DBMCLI> LIST PHYSICALDISK WHERE DISKTYPE=flashdisk AND STATUS LIKE '.*dropped 
   for replacement.*' DETAIL
   
            name:               FLASH_1_1
            deviceName:         /dev/nvme0n1
            diskType:           FlashDisk
            luns:               1_1
            makeModel:          "Oracle Flash Accelerator F640 PCIe Card"
            physicalFirmware:   QDV1RD09
            physicalInsertTime: 2017-08-11T12:25:00-07:00
            physicalSerial:     PHLE6514003R6P4BGN-1
            physicalSize:       2.910957656800747T
            slotNumber:         "PCI Slot: 1; FDOM: 1"
            status:             failed - dropped for replacement
   

2. Locate the failed flash disk based on the PCI number and FDOM number.
   A white server LED is lit to help locate the affected database server. An amber attention LED is lit to identify the affected flash card.
3. Make sure the power LED is off on the card.

   Caution:

   Removing a card with power LED on could result in a system crash. If a failed disk has a status of Failed – dropped for replacement but the power LED is still on, contact Oracle Support.
4. Remove and replace the failed flash disk.
   1. Slide out the Dual PCIe Card Carrier (DPCC) and replace the flash card inside.
   2. Slide the DPCC carrier back to the slot.
5. Use a stylus to press both ATTN buttons on the front of the DPCC.

   • If only a single PCIe card is present, press only the corresponding ATTN button.

   • If you are not performing a hot-pluggable replacement, then this step is not necessary.

   The buttons alert the system to a request to bring the devices online. When the system acknowledges the request, it lights the OK indicators on the DPCC. If you do not press the ATTN buttons, then the flash disk will not be detected by the operating system.

2.4 Adding Disk Expansion Kit to Database Servers

You can add a disk expansion kit to your Exadata database server with the following restrictions:

• The disk expansion kit is supported on Oracle Exadata Database Machine X5-2, X6-2, and X7-2 systems only.

• Oracle Exadata software 12.1.2.3.0 or later is required.

1. Replace the plastic filler with the four drives in the disk expansion kit.
2. An alert will be raised to indicate the disk expansion kit is detected and the drives have been automatically added to the existing RAID5 configuration and reconstruction of the corresponding virtual drive is started.
3. Another alert will be raised when the virtual drive reconstruction is completed.
4. Run /opt/oracle.SupportTools/reclaimdisks.sh -extend-vgexadb to extend the VGExaDb volume group to the rest of the /dev/sda system disk.

   Note:

   • reclaimdisks.sh works only during initial deployment, before the database software is installed. If you have already been running Oracle Exadata Database Machine for a while, you do not need to run it because you should have already run it during initial deployment. In the event that it was not run during initial deployment, you cannot run it at this time because too many changes to /u01 have been made. reclaimdisks.sh returns an error message if it discovers changes to the /u01 file system. By default, the /u01 file system is empty on new systems.

   • Run this command only after the virtual drive reconstruction is complete. If you run it before reconstruction is complete, you will see the following messages:

     [WARNING ] Reconstruction of the logical drive 0 is in progress: Completed: 14%. Left: 5 Hours 32 Minutes
     [WARNING ] Continue after reconstruction is complete
     

     If this occurs, wait until the virtual drive reconstruction is complete, then re-run the command.

   If prompted to fix the GPT (GUID Partition Table) or to continue with the current settings, enter F to fix the GPT.

   [root@dbnode01 ~]# /opt/oracle.SupportTools/reclaimdisks.sh -extend-vgexadb
   Model is ORACLE SERVER X6-2
   Number of LSI controllers: 1
   Physical disks found: 8 (252:0 252:1 252:2 252:3 252:4 252:5 252:6 252:7)
   Logical drives found: 1
   Linux logical drive: 0
   RAID Level for the Linux logical drive: 5
   Physical disks in the Linux logical drive: 8 (252:0 252:1 252:2 252:3 252:4 252:5 252:6 252:7)
   Dedicated Hot Spares for the Linux logical drive: 0
   Global Hot Spares: 0
   Valid. Disks configuration: RAID5 from 8 disks with no global and dedicated hot spare disks.
   Valid. Booted: Linux. Layout: Linux + DOM0.
   [INFO     ] Size of system block device /dev/sda: 4193GB
   [INFO     ] Last partition on /dev/sda ends on: 1797GB
   [INFO     ] Unused space detected on the system block device: /dev/sda
   [INFO     ] Label of partition table on /dev/sda: gpt
   [INFO     ] Adjust the partition table to use all of the space on /dev/sda
   [INFO     ] Respond to the following prompt by typing 'F'
   Warning: Not all of the space available to /dev/sda appears to be used, you can fix the GPT to use all of the space (an extra 4679680000 blocks) or
   continue with the current setting?
   Fix/Ignore? F
   Model: LSI MR9361-8i (scsi)
   Disk /dev/sda: 4193GB
   Sector size (logical/physical): 512B/512B
   Partition Table: gpt
    
   Number  Start   End     Size    File system  Name     Flags
    1      32.8kB  537MB   537MB   ext4         primary  boot
    2      537MB   123GB   122GB                primary  lvm
    3      123GB   1690GB  1567GB               primary
    4      1690GB  1797GB  107GB                primary  lvm
    
   [INFO     ] Check for Linux with inactive DOM0 system disk
   [INFO     ] Valid Linux with inactive DOM0 system disk is detected
   [INFO     ] Number of partitions on the system device /dev/sda: 4
   [INFO     ] Higher partition number on the system device /dev/sda: 4
   [INFO     ] Last sector on the system device /dev/sda: 8189440000
   [INFO     ] End sector of the last partition on the system device /dev/sda: 3509759000
   [INFO     ] Unmount /u01 from /dev/mapper/VGExaDbOra-LVDbOra1
   [INFO     ] Remove inactive system logical volume /dev/VGExaDb/LVDbSys3
   [INFO     ] Remove xen files from /boot
   [INFO     ] Remove logical volume /dev/VGExaDbOra/LVDbOra1
   [INFO     ] Remove volume group VGExaDbOra
   [INFO     ] Remove physical volume /dev/sda4
   [INFO     ] Remove partition /dev/sda4
   [INFO     ] Remove device /dev/sda4
   [INFO     ] Remove partition /dev/sda3
   [INFO     ] Remove device /dev/sda3
   [INFO     ] Create primary partition 3 using 240132160 8189439966
   [INFO     ] Set lvm flag for the primary partition 3 on device /dev/sda
   [INFO     ] Add device /dev/sda3
   [INFO     ] Primary LVM partition /dev/sda3 has size 7949307807 sectors
   [INFO     ] Create physical volume on partition /dev/sda3
   [INFO     ] LVM Physical Volume /dev/sda3 has size 3654340511 sectors
   [INFO     ] Size of LVM physical volume less than size of device /dev/sda3
   [INFO     ] Remove LVM physical volume /dev/sda3
   [INFO     ] Reboot is required to apply the changes in the partition table
   

5. If required, reboot the node to apply the changes in the partition table. The message at the end of the previous command will tell you if a reboot is required.

   [root@dbnode01 ~]# reboot
   

   If a reboot is not required, skip to step 7.

6. Run /opt/oracle.SupportTools/reclaimdisks.sh -extend-vgexadb.

   This step depends on what you want to do with the additional space. For example, you can increase the size of existing volume groups, or you can create and mount new VGs to make use of the additional space.

   Note that this command may return an error, as shown below. This error can be ignored.

   [root@dbnode01 ~]# /opt/oracle.SupportTools/reclaimdisks.sh -extend-vgexadb
   Model is ORACLE SERVER X6-2
   Number of LSI controllers: 1
   Physical disks found: 8 (252:0 252:1 252:2 252:3 252:4 252:5 252:6 252:7)
   Logical drives found: 1
   Linux logical drive: 0
   RAID Level for the Linux logical drive: 5
   Physical disks in the Linux logical drive: 8 (252:0 252:1 252:2 252:3 252:4 252:5 252:6 252:7)
   Dedicated Hot Spares for the Linux logical drive: 0
   Global Hot Spares: 0
   Valid. Disks configuration: RAID5 from 8 disks with no global and dedicated hot spare disks.
   Valid. Booted: Linux. Layout: Linux.
   [INFO     ] Check for Linux system disk
   [INFO     ] Number of partitions on the system device /dev/sda: 4
   [INFO     ] Higher partition number on the system device /dev/sda: 4
   [INFO     ] Last sector on the system device /dev/sda: 8189440000
   [INFO     ] End sector of the last partition on the system device /dev/sda: 8189439966
   [INFO     ] Next free available partition on the system device /dev/sda:
   [INFO     ] Primary LVM partition /dev/sda4 has size 4679680000 sectors
   [INFO     ] Create physical volume on partition /dev/sda4
   [INFO     ] LVM Physical Volume /dev/sda4 has size 4679680000 sectors
   [INFO     ] Size of LVM physical volume matches size of primary LVM partition /dev/sda4
   [INFO     ] Extend volume group VGExaDb with physical volume on /dev/sda4
   [INFO     ] Create 100Gb logical volume for DBORA partition in volume group VGExaDb
   [WARNING  ] Failed command at attempt: lvm lvcreate -L 100GB -n LVDbOra1 VGExaDb at 1/1
   [ERROR    ] Failed command: lvm lvcreate -L 100GB -n LVDbOra1 VGExaDb
   [ERROR    ] Unable to create logical volume LVDbOra1 in volume group VGExaDb
   [ERROR    ] Unable to reclaim all disk space
   

7. Run /opt/oracle.SupportTools/reclaimdisks.sh for confirmation.

   [root@dbnode01 ~]# /opt/oracle.SupportTools/reclaimdisks.sh
   Model is ORACLE SERVER X6-2
   Number of LSI controllers: 1
   Physical disks found: 8 (252:0 252:1 252:2 252:3 252:4 252:5 252:6 252:7)
   Logical drives found: 1
   Linux logical drive: 0
   RAID Level for the Linux logical drive: 5
   Physical disks in the Linux logical drive: 8 (252:0 252:1 252:2 252:3 252:4 252:5 252:6 252:7)
   Dedicated Hot Spares for the Linux logical drive: 0
   Global Hot Spares: 0
   Valid. Disks configuration: RAID5 from 8 disks with no global and dedicated hot spare disks.
   Valid. Booted: Linux. Layout: Linux.
   

8. Resize the logical volume and file systems as described in "Extending LVM Partitions."

2.5 Adding Memory Expansion Kit to Database Servers

Additional memory can be added to database servers. The following procedure describes how to add the memory:

1. Power down the database server.
2. Replace the plastic fillers with the DIMMs.

   • X7-2: See "Servicing the DIMMs (CRU)" in Oracle Server X7-2 Service Manual at http://docs.oracle.com/cd/E72435_01/html/E72445/gqvkr.html#scrolltoc

   • X7-8: See "Servicing DIMMs" in Oracle Server X7-8 Service Manual at http://docs.oracle.com/cd/E71925_01/html/E71936/grfoz.html#scrolltoc

   • X6-2: See "Servicing the DIMMs (CRU)" in Oracle Server X6-2 Service Manual at http://docs.oracle.com/cd/E62159_01/html/E62171/z40003f01425075.html#scrolltoc

   • X5-2: See "Servicing the DIMMs (CRU)" in Oracle Server X5-2 Service Manual at http://docs.oracle.com/cd/E41059_01/html/E48320/z40003f01425075.html

   • X5-8: See "Replace a Failed DIMM" in Oracle Server X5-8 Service Manual at https://docs.oracle.com/cd/E56301_01/html/E56311/gnsdu.html#scrolltoc

   • X4-8: See "Replace a Failed DIMM" in Sun Server X4-8 Service Manual at http://docs.oracle.com/cd/E40591_01/html/E40317/gnsdu.html#scrolltoc

   • X4-2: See "Servicing the DIMMs (CRU)" in Sun Server X4-2 Service Manual at http://docs.oracle.com/cd/E36975_01/html/E38045/z40003f01425075.html#scrolltoc

   • X3-2: See "Servicing the DIMMs (CRU)" in Sun Server X3-2 (formerly Sun Fire X4170 M3) Service Manual at http://docs.oracle.com/cd/E22368_01/html/E27242/z40003f01425075.html#scrolltoc

   • X2-8: See "Removing and Installing DIMMs (CRU)" in Sun Server X2-8 (formerly Sun Fire X4800 M2) Service Manual at http://docs.oracle.com/cd/E20815_01/html/E20819/gjfvw.html#scrolltoc

   • Sun Fire X4170 M2: See "Servicing DDR3 DIMMs" in Sun Fire X4170 M2 Server Service Manual at http://docs.oracle.com/cd/E19762-01/E22369-02/motherboard_x4170.html#50581523_71526

3. Power on the database server.
4. Add the database server back to the cluster.

Note:

• Memory for Sun Server X4-2 Oracle Database Servers and Sun Server X3-2 Oracle Database Servers can be expanded to a maximum of 512 GB with the memory expansion kit.

• Memory for Sun Fire X4170 Oracle Database Servers can be expanded to a maximum of 144 GB by removing the existing memory, and replacing it with three X2-2 Memory Expansion Kits.

• Sun Fire X4170 M2 Oracle Database Servers ship from the factory with 96 GB of memory, with 12 of the 18 DIMM slots populated with 8 GB DIMMs. The optional X2-2 Memory Expansion Kit can be used to populate the remaining 6 empty slots with 16 GB DIMMs to bring the total memory to 192 GB (12 x 8 GB and 6 x 16GB).

  The memory expansion kit is primarily for consolidation workloads where many databases are run on each database server. In this scenario, the CPU usage is often low while the memory usage is very high.

  However, there is a downside to populating all the memory slots as the frequency of the memory DIMMs drop to 800 MHz from 1333 MHz. The performance effect of the slower memory appears as increased CPU utilization. The average measured increase in CPU utilization is typically between 5% and 10%. The increase varies greatly by workload. In test workloads, several workloads had almost zero increase, while one workload had as high as a 20% increase.

• When adding memory to Oracle Exadata Database Machines running Oracle Linux, Oracle recommends updating the /etc/security/limits.conf file with the following:

  oracle    soft     memlock 75%
  oracle    hard     memlock 75%
  

2.6 Verify the Link Speed on Network Ports

Ensure you are using the correct link speed for Oracle Exadata Database Machine X7-2 compute nodes.

On Oracle Exadata Database Machine X7-2 compute nodes you might experience issues when connecting to 10GbE switches. These issues include links not being detected or being unable to connect to the gateway.

1. For each 10GbE network interface (designate by x) that does not have the link detected, run the following commands:

   # ifdown ethx
   # ethtool -s ethx 10000 duplex full autoneg off
   # ifup ethx
   # ethtool ethx
   

2. Confirm that the output from the ethtool command shows yes for Link detected.

           Link detected: yes
   

3. Edit the appropriate files in /etc/sysconfig/network-scripts, where x is the number associated with the network interface.
   1. Locate the /etc/sysconfig/network-scripts/ifcfg-ethx file. Add the following line, if it is not already present in the file:

      ETHTOOL_OPTS="speed 10000 duplex full autoneg off"
      

   2. Repeat the previous step for all network interfaces that do not have the ETHTOOL_OPTS setting in the associated ifcfg-ethx file and are connected to 10GbE switches.

   The network interface should now show the link as detected. These changes are persistent, and do not need to be repeated after a server reboot.

2.7 Adding and Configuring an Extra Network Card on Oracle Exadata X6-2 and X7-2

You can add an additional network card on Oracle Exadata X6-2 and X7-2 systems.

Oracle Exadata Database Machine X6-2

Oracle Exadata Database Machine X6-2 database server offers highly available copper 10G network on the motherboard, and an optical 10G network via a PCI card on slot 2. Oracle offers an additional Ethernet card for customers that require additional connectivity. The additional card provides either dual port 10GE copper connectivity (part number 7100488) or dual port 10GE optical connectivity (part number X1109A-Z). You install this card in PCIe slot 1 on the Oracle Exadata X6-2 database server.

After you install the card and connect it to the network, the Oracle Exadata System Software automatically recognizes the new card and configures the two ports as eth6 and eth7 interfaces on the X6-2 database server. You can use these additional ports to provide an additional client network, or to create a separate backup or data recovery network. On a database server that runs virtual machines, you could use this to isolate traffic from two virtual machines.

Oracle Exadata Database Machine X7-2

Oracle Exadata Database Machine X7-2 database server offers 2 copper (RJ45) or 2 optical (SFP28) network connections on the motherboard plus 2 optical (SFP28) network connections in PCIe card slot 1. Oracle offers an additional 4 copper (RJ45) 10G network connections for customers that require additional connectivity. The additional card is the Oracle Quad Port 10GBase-T card (part number 7111181). You install this card in PCIe slot 3 on the X7-2 database server.

After you install the card and connect it to the network, the Oracle Exadata System Software automatically recognizes the new card and configures the four ports as eth5 to eth8 interfaces on the X7-2 database server. You can use these additional ports to provide an additional client network, or to create a separate backup or data recovery networks. On a database server that runs virtual machines, you could use this to isolate traffic from two virtual machines.

After you have added the card to the database server, you need to configure the card. See:

• Viewing the Network Interfaces

• Configuring the Additional Network Card for a Non-Oracle VM Environment

• Configuring the Additional Network Card for an Oracle VM Environment

2.7.1 Viewing the Network Interfaces

To view the network interfaces, you can run the ipconf.pl command.

Example 2-1 Viewing the default network interfaces for an Oracle Exadata Database Machine X6-2 database server

The following example shows the output for an Oracle Exadata Database Machine X6-2 database server without the additional network card. The output shows two network cards:

• A quad port 10Gb card, on eth0 to eth3

• A dual port 10Gb card, on eth4 and eth5

# cd /opt/oracle.cellos/

# ./ipconf.pl
Logging started to /var/log/cellos/ipconf.log
Interface ib0   is                      Linked.    hca: mlx4_0
Interface ib1   is                      Linked.    hca: mlx4_0
Interface eth0  is                      Linked.    driver/mac: ixgbe/00:10:e0:8b:24:b6
Interface eth1  is .....                Linked.    driver/mac: ixgbe/00:10:e0:8b:24:b7
Interface eth2  is .....                Linked.    driver/mac: ixgbe/00:10:e0:8b:24:b8
Interface eth3  is .....                Linked.    driver/mac: ixgbe/00:10:e0:8b:24:b9
Interface eth4  is                      Linked.    driver/mac: ixgbe/90:e2:ba:ac:20:ec (slave of bondeth0)
Interface eth5  is                      Linked.    driver/mac: ixgbe/90:e2:ba:ac:20:ec (slave of bondeth0)

Example 2-2 Viewing the default network interfaces for an Oracle Exadata Database Machine X7-2 database server

The following example shows the output for an Oracle Exadata Database Machine X7-2 database server without the additional network card. The output shows three network cards:

• A single port 10Gb card, on eth0

• A dual port 10 or 25Gb card, on eth1 and eth2

• A dual port 25Gb card, on eth3 and eth4

# /opt/oracle.cellos/ipconf.pl
Logging started to /var/log/cellos/ipconf.log 
Interface ib0   is                      Linked.    hca: mlx4_0 
Interface ib1   is                      Linked.    hca: mlx4_0 
Interface eth0  is                      Linked.    driver/mac: igb/00:
10:e0:c3:ba:72 
Interface eth1  is                      Linked.    driver/mac: bnxt_en
/00:10:e0:c3:ba:73 
Interface eth2  is                      Linked.    driver/mac: bnxt_en
/00:10:e0:c3:ba:74 
Interface eth3  is                      Linked.    driver/mac: bnxt_en
/00:0a:f7:c3:14:a0 (slave of bondeth0) 
Interface eth4  is                      Linked.    driver/mac: bnxt_en
/00:0a:f7:c3:14:a0 (slave of bondeth0)

2.7.2 Configuring the Additional Network Card for a Non-Oracle VM Environment

You can configure the additional network card on an Oracle Exadata Database Machine X6-2 or later database server for a non-Oracle VM environment.

This procedure assumes that you have already installed the network card in the Oracle Exadata Database Machine database server but have not yet completed the configuration with Oracle Exadata Deployment Assistant (OEDA).

WARNING:

If you have already installed Oracle Grid Infrastructure on Oracle Exadata Database Machine, then refer to the Oracle Clusterware documentation. Use caution when changing the network interfaces for the cluster.

1. Ensure you have the following information for the new network card.
   You will need to input this information when you run ipconf.pl.
   • IP address
   • Netmask
   • Gateway
2. Run the ipconf.pl script to configure the card.

   The following example shows a sample ipconf.pl session. The output shows three network cards:

   • A quad port 10Gb card, on eth0 to eth3

   • A dual port 10Gb card, on eth4 and eth5, with only one port cabled

   • A dual port 10Gb card, on eth6 and eth7, with only one port cabled. This is the new network card.

   # cd /opt/oracle.cellos/
   # ./ipconf.pl
   
   Logging started to /var/log/cellos/ipconf.log
   Interface ib0   is                      Linked.    hca: mlx4_0
   Interface ib1   is                      Linked.    hca: mlx4_0
   Interface eth0  is                      Linked.    driver/mac: 
   ixgbe/00:10:e0:8b:22:e8 (slave of vmeth0)
   Interface eth1  is                      Linked.    driver/mac: 
   ixgbe/00:10:e0:8b:22:e9 (slave of bondeth0)
   Interface eth2  is                      Linked.    driver/mac: 
   ixgbe/00:10:e0:8b:22:e9 (slave of bondeth0)
   Interface eth3  is                      Linked.    driver/mac: 
   ixgbe/00:10:e0:8b:22:eb
   Interface eth4  is                      Linked.    driver/mac: 
   ixgbe/90:e2:ba:ac:1d:e4
   Interface eth5  is .................... Unlinked.  driver/mac: 
   ixgbe/90:e2:ba:ac:1d:e5
   Interface eth6  is ...                  Linked.    driver/mac: 
   ixgbe/90:e2:ba:78:d0:10
   Interface eth7  is .................... Unlinked.  driver/mac: 
   ixgbe/90:e2:ba:78:d0:11
   
   bondeth0 eth1,eth2 UP      vmbondeth0 10.128.1.169  255.255.240.0
   10.128.0.1  SCAN       test08client02.example.com
   bondeth1 None      UNCONF 
   bondeth2 None      UNCONF 
   bondeth3 None      UNCONF 
   Select interface name to configure or press Enter to continue: eth6
   Selected interface. eth6
   IP address or up or none: 10.129.19.34
   Netmask: 255.255.248.0
   Gateway (IP address or none) or none: 10.129.16.0
   
   Select network type for interface from the list below
   1: Management
   2: SCAN
   3: Other
   Network type: 3
   
   Fully qualified hostname or none: test08adm02-bkup.example.com
   Continue configuring or re-configuring interfaces? (y/n) [y]: n
   ...
   Do you want to configure basic ILOM settings (y/n) [y]: n
   [Info]: Custom changes have been detected in /etc/sysconfig/network-script
   s/ifcfg-eth6
   [Info]: Original file /etc/sysconfig/network-scripts/ifcfg-eth6 will be 
   saved in /opt/oracle.cellos/conf/network-scripts/backup_by_Exadata_ipconf
   [Info]: Original file /etc/ssh/sshd_config will be saved in /etc/ssh/sshd_
   config.backupbyExadata
   [Info]: Generate /etc/ssh/sshd_config with ListenAddress(es) 10.128.18.106, 
   10.129.19.34, 10.128.1.169, 192.168.18.44, 192.168.18.45
   Stopping sshd:                                             [  OK  ]
   Starting sshd:                                             [  OK  ]
   [Info]: Save /etc/sysctl.conf in /etc/sysctl.conf.backupbyExadata
   [Info]: Adjust settings for IB interfaces in /etc/sysctl.conf
   Re-login using new IP address 10.128.18.106 if you were disconnected after 
   following commands
   ip addr show vmbondeth0
   ip addr show bondeth0
   ip addr show vmeth0
   ip addr show eth0
   ifup eth6
   sleep 1
   ifup vmeth6
   sleep 1
   ip addr show vmeth6
   ip addr show eth6
   sleep 4
   service sshd condrestart
   

3. If you need to set up the network card with VLAN, perform these steps:
   1. Add the VLAN ID to the /opt/oracle.cellos/cell.conf file.

      • Locate the Ethernet interface in the file. For example:

        <Interfaces>
          <Gateway>10.129.16.0</Gateway>
          <Hostname>test08adm02-bkup.example.com</Hostname>
          <IP_address>10.129.19.34</IP_address>
          <IP_enabled>yes</IP_enabled>
          <IP_ssh_listen>enabled</IP_ssh_listen>
          <Inet_protocol>IPv4</Inet_protocol>
          <Name>eth6</Name>
          <Net_type>Other</Net_type>
          <Netmask>255.255.248.0</Netmask>
          <State>1</State>
          <Status>UP</Status>
          <Vlan_id>0</Vlan_id>
        </Interfaces>
        

      • Add the VLAN ID to the <Vlan_id> element. The following example shows the interface configured with VLAN ID of 2122.

        <Interfaces>
          <Gateway>10.129.16.0</Gateway>
          <Hostname>test08adm02-bkup.example.com</Hostname>
          <IP_address>10.129.19.34</IP_address>
          <IP_enabled>yes</IP_enabled>
          <IP_ssh_listen>enabled</IP_ssh_listen>
          <Inet_protocol>IPv4</Inet_protocol>
          <Name>eth6</Name>
          <Net_type>Other</Net_type>
          <Netmask>255.255.248.0</Netmask>
          <State>1</State>
          <Status>UP</Status>
          <Vlan_id>2122</Vlan_id>
        </Interfaces>
        

   2. Run the following command to configure the network interface using the modified cell.conf file:

      # /opt/oracle.cellos/ipconf.pl -init -force
      

   3. Validate the interface has the VLAN configured by checking that the /etc/sysconfig/network-scripts directory contains files with the VLAN ID in the filename. For example, if the VLAN ID is 2122, you should see the following files:

      # ls -ltr /etc/sysconfig/network-scripts/*2122*
      
      -rw-r----- 1 root root 250 Sep  7 14:39 /etc/sysconfig/network-scripts/ifcfg-eth6.2122
      -rw-r----- 1 root root  85 Sep  7 14:39 /etc/sysconfig/network-scripts/route-eth6.2122
      -rw-r----- 1 root root  56 Sep  7 14:39 /etc/sysconfig/network-scripts/rule-eth6.2122
      

4. Reboot the database server for the changes to take effect.
5. Check that the network is working by pinging the gateway. For example:

   # ping 10.129.16.0
   

2.7.3 Configuring the Additional Network Card for an Oracle VM Environment

You can configure the additional network card on an Oracle Exadata Database Machine X6-2 and later database server for an Oracle VM environment.

This procedure assumes that you have already installed the network card in the Oracle Exadata Database Machine database server but have not yet completed the configuration with Oracle Exadata Deployment Assistant (OEDA).

Caution:

Do not attempt this procedure if you have already installed Oracle Grid Infrastructure on Oracle Exadata Database Machine.

1. Add a section for the new network in the /EXAVMIMAGES/conf/virtual_machine_config_file configuration file in dom0.

   The following example assumes the bridge is called vmeth6, and the interface is called eth1. The virtual machine configuration file name is /EXAVMIMAGES/conf/test08adm01vm01.example.com-vm.xml.

   <Interfaces>
     <Bridge>vmeth6</Bridge>
     <Gateway>10.129.16.0</Gateway>
     <Hostname>test08adm02-bkup.example.com</Hostname>
     <IP_address>10.129.19.34</IP_address>
     <Name>eth1</Name>
     <IP_enabled>yes</IP_enabled>
     <IP_ssh_listen>disabled</IP_ssh_listen>
     <Net_type>Other</Net_type>
     <Netmask>255.255.248.0</Netmask>
     <Vlan_id>0</Vlan_id>
     <State>1</State>
     <Status>UP</Status>
   </Interfaces>
   

   If you are using VLANs, enter the appropriate VLAN ID [1-4095] in the <Vlan_id> element.

2. Create the bridge.
   1. To create an unbonded bridge named vmeth6:

      # /opt/exadata_ovm/exadata.img.domu_maker add-single-bridge-dom0 vmeth6
      

   2. To create an bonded bridge, use a command similar to the following:

      # /opt/exadata_ovm/exadata.img.domu_maker add-bonded-bridge-dom0 bridge_name slave1 slave2 [vlan]
      

      slave1 and slave2 are the names of the bonded interfaces.

      For example:

      # /opt/exadata_ovm/exadata.img.domu_maker add-bonded-bridge-dom0 vmbondeth1 eth6 eth7
      

3. Allocate the InfiniBand GUIDs:

   # /opt/exadata_ovm/exadata.img.domu_maker allocate-guids virtual_machine_config_file virtual_machine_config_file_final
   

   The virtual machine configuration files are located in the /EXAVMIMAGES/conf directory. For example:

   # /opt/exadata_ovm/exadata.img.domu_maker allocate-guids /EXAVMIMAGES/conf/
   test08adm01vm01.example.com-vm.xml /EXAVMIMAGES/conf/final-test08adm01vm01
   .example.com-vm.xml
   

4. Shut down the guest and restart it.

   # /opt/exadata_ovm/exadata.img.domu_maker remove-domain /EXAVMIMAGES/conf
   /final-test08adm01vm01.example.com-vm.xml
   
   # /opt/exadata_ovm/exadata.img.domu_maker start-domain /EXAVMIMAGES/conf
   /final-test08adm01vm01.example.com-vm.xml
   

5. Once the guest is running, use the ip addr command to verify the interface is valid.

   The following example checks the eth1 interface.

   # ip addr show eth1
   eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
     link/ether 00:16:3e:53:56:00 brd ff:ff:ff:ff:ff:ff
     inet 10.129.19.34/21 brd 10.129.23.255 scope global eth1
        valid_lft forever preferred_lft forever
   

2.8 Increasing the Number of Active Cores on Database Servers

You can increase the number of active cores on Oracle Exadata Database Machine using capacity-on-demand.

The number of active cores on the database servers on Oracle Exadata Database Machine X4-2 and newer systems can be reduced during installation. The number of active cores can be increased when additional capacity is needed. This is known as capacity-on-demand.

Additional cores are increased in 2-core increments on Oracle Exadata Database Machine X4-2 and newer systems, and in 8-core increments on Oracle Exadata Database Machine X4-8 Full Rack and newer systems. The following table lists the capacity-on-demand core processor configurations.

Table 2-3 Capacity-on-Demand Core Processor Configurations

Oracle Exadata Database Machine	Eligible Systems	Minimum Cores per Server	Maximum Cores per Server	Core Increments
Oracle Exadata Database Machine X7-2	Any configuration except Eighth Rack	14	48	From 14 to 48, in multiples of 2: 14, 16, 18, ..., 46, 48
Oracle Exadata Database Machine X7-2	Eighth rack	8	24	From 8 to 24, in multiples of 2: 8, 10, 12, ..., 22, 24
Oracle Exadata Database Machine X6-2	Any configuration except Eighth Rack	14	44	From 14 to 44, in multiples of 2: 14, 16, 18, ..., 42, 44
Oracle Exadata Database Machine X6-2	Eighth rack	8	22	From 8 to 22, in multiples of 2: 8, 10, 12, ..., 20, 22
Oracle Exadata Database Machine X5-2	Any configuration except Eighth Rack	14	36	From 14 to 36, in multiples of 2: 14, 16, 18, ..., 34, 36
Oracle Exadata Database Machine X5-2	Eighth rack	8	18	From 8 to 18, in multiples of 2: 8, 10, 12, ..., 16, 18
Oracle Exadata Database Machine X4-2	Full rack Half rack Quarter rack	12	24	From 12 to 24, in multiples of 2: 12, 14, 16, ..., 22, 24
Oracle Exadata Database Machine X7-8	Any configuration	56	192	From 56 to 192, in multiples of 8: 56, 64, 72,..., 184, 192
Oracle Exadata Database Machine X6-8 and X5-8	Any configuration	56	144	From 56 to 144, in multiples of 8: 56, 64, 72, ..., 136, 144
Oracle Exadata Database Machine X4-8	Full rack	48	120	From 48 to 120, in multiples of 8: 48, 56, 64, ..., 112, 120

Note:

Oracle recommends licensing the same number of cores on each server, in case of failover.

Database servers can be added one at a time, and capacity-on-demand can be applied to the individual database servers. This option includes Oracle Exadata Database Machine X5-2 Eighth Racks.

The database server must be restarted after enabling additional cores. If the database servers are part of a cluster, then they can be enabled in a rolling fashion.

1. Verify the number of active physical cores using the following command:

   DBMCLI> LIST DBSERVER attributes coreCount
   

2. Use the following command to increase the number of active physical cores:

   DBMCLI> ALTER DBSERVER pendingCoreCount = new_number_of_active_physical_cores
   

3. Verify the pending number of active physical cores using the following command:

   DBMCLI> LIST DBSERVER attributes pendingCoreCount
   

4. Restart the server.
5. Verify the number of active physical cores using the following command:

   DBMCLI> LIST DBSERVER attributes coreCount
   

2.9 Migrating a Bare Metal Oracle RAC Cluster to an Oracle RAC Cluster in Oracle VM

Note:

This section applies only to two-socket x86 servers. It does not apply to eight-socket servers or SPARC servers.

The migration of a bare metal Oracle RAC cluster to an Oracle RAC Cluster in Oracle VM can be achieved in the following ways:

• Migrate to Oracle RAC cluster in Oracle VM using the existing bare metal Oracle RAC cluster with zero downtime

• Migrate to Oracle RAC cluster in Oracle VM by creating a new Oracle RAC cluster in Oracle VM with minimal downtime

• Migrate to Oracle RAC cluster in Oracle VM using Oracle Data Guard with minimal downtime

• Migrate to Oracle RAC cluster in Oracle VM using RMAN backup and restore with complete downtime

The conversion of a bare metal Oracle RAC cluster to an Oracle RAC cluster in Oracle VM has the following implications:

• Each of the database servers will be converted to an Oracle Virtual Server on which a management domain will be created along with one or more user domains, depending on the number of Oracle RAC clusters being deployed. Each user domain on a database server will belong to a particular Oracle RAC cluster.

• As part of the conversion procedure, the bare metal Oracle RAC cluster will be converted to one Oracle RAC cluster in Oracle VM to start with. There will be one user domain per database server.

• At the end of the conversion, the cell disk and the grid disk configuration of the storage cells will be the same as they were at the beginning of the conversion.

• The management domain will use a small portion of the system resources on each database server. Typically a management domain will use 8 GB of memory and 4 virtual CPUs. This has to be taken into consideration while sizing the SGA of the database(s) running on the Oracle RAC cluster in Oracle VM.

2.10 Managing Oracle VM Domains on Oracle Exadata Database Machine

Oracle VM user domains on Oracle Exadata Database Machine are managed using the xm(1) command run from the management domain (also known as domain-0, or dom0). For a full list of Oracle VM administrative commands, run the xm help command.

Note:

The following xm subcommands are not supported on Oracle Exadata Database Machine:

mem-set
mem-max
migrate
restore
resume
save
suspend
sched-*
cpupool-*
tmem-*

This section contains the following topics:

• Showing Running Domains

• Monitoring a User Domain Console

• Starting a User Domain

• Disabling User Domain Automatic Start

• Shutting Down a User Domain From Within the User Domain

• Shutting Down a User Domain From Within the Management Domain

• Backing Up and Restoring Oracle Databases on Oracle VM User Domains

• Modifying the Memory Allocated to a User Domain

• Modifying the Number of Virtual CPUs Allocated to a User Domain

• Increasing the Disk Space in a User Domain

• Expanding /EXAVMIMAGES on Management Domain after Database Server Disk Expansion

• Creating Oracle VM Oracle RAC Clusters

• Expanding an Oracle VM Oracle RAC Cluster on Exadata

• Creating a User Domain Without Oracle Grid Infrastructure and Oracle Database

• Moving a User Domain to a Different Database Server

• Removing an Oracle RAC Cluster Running in Oracle VM

• Deleting a User Domain from an Oracle VM Oracle RAC Cluster

• Implementing Tagged VLAN Interfaces

• Implementing InfiniBand Partitioning across Oracle VM Oracle RAC Clusters on Oracle Exadata

• Running Oracle EXAchk

Note:

Unless otherwise noted, all commands run in the preceding procedures are run as the root user.

2.10.1 Showing Running Domains

The following procedure describes how to show running domains:

1. Connect to the management domain (domain zero, or dom0).
2. Run the xm list command. The following is an example of the output:

   Example
   # xm list
   Name                         ID   Mem   VCPUs      State   Time(s)
   Domain-0                      0   8192     4       r-----  409812.7
   dm01db01vm01                  8   8192     2       -b---- 156610.6
   dm01db01vm02                  9   8192     2       -b---- 152169.8
   dm01db01vm03                 10  10240     4       -b---- 150225.9
   dm01db01vm04                 16  12288     8       -b---- 113519.3
   dm01db01vm05                 12  12288     8       -b---- 174101.6
   dm01db01vm06                 13  12288     8       -b---- 169115.9
   dm01db01vm07                 14   8192     4       -b---- 175573.0
   

2.10.2 Monitoring a User Domain Console

The following procedure describes how to monitor a user domain console:

1. Connect as the root user to the management domain.
2. Obtain the domain name using the xm list command.
3. Use the following command to attach to the user domain console:

   # xm console DomainName
   

   In the preceding command, DomainName is the name of the domain.

4. Press CTRL+] to disconnect from the console.

2.10.3 Starting a User Domain

The following procedure describes how to start a user domain:

1. Use the following command to start the user domain:

   # xm create /EXAVMIMAGES/GuestImages/DomainName/vm.cfg
   Using config file "/EXAVMIMAGES/GuestImages/dm01db01vm04/vm.cfg".
   Started domain dm01db01vm04 (id=23)
   

   In the preceding command, DomainName is the name of the domain.

   Note:

   To see Oracle Linux boot messages during user domain startup, connect to the console during startup using the -c option. To disconnect from the console after startup is complete, press CTRL+].

2.10.4 Disabling User Domain Automatic Start

The following procedure describes how to disable a user domain from automatically starting when the management domain is started:

1. Connect to the management domain.
2. Remove the symbolic link to the user domain configuration file in the /etc/xen/auto directory using the following command:

   # rm /etc/xen/auto/DomainName.cfg
   

   In the preceding command, DomainName is the name of the domain.

2.10.5 Shutting Down a User Domain From Within the User Domain

The following procedure describes how to shut down a user domain from within a user domain:

1. Connect as the root user to the user domain.
2. Use the following command to shut down the domain:

   # shutdown -h now
   

2.10.6 Shutting Down a User Domain From Within the Management Domain

The following procedure describes how to shut down a user domain from within a management domain:

1. Connect as the root user to the management domain.
2. Use the following command to shut down the domain:

   # xm shutdown DomainName -w
   

   In the preceding command, DomainName is the name of the domain.

   Note:

   Use the -w option so that the xm command waits until the domain shutdown completes before returning. The xm shutdown command performs the same orderly shutdown as runningshutdown -h now within the user domain.

   To shut down all user domains within the management domain, use the following command:

   # xm shutdown -a -w
   

   The following is an example of the output:

   Domain dm01db01vm04 terminated
   All domains terminated
   

2.10.7 Backing Up and Restoring Oracle Databases on Oracle VM User Domains

Backing up and restoring Oracle databases on Oracle VM user domains is the same as backing up and restoring Oracle databases on physical nodes.

• When backing up Oracle databases on Oracle VM user domains you must set the following four parameters in the /etc/sysctl.conf file on the database nodes (user domains). If you are using Oracle Exadata storage to hold the backups, the parameters need to be set in the /etc/sysctl.conf file on the Exadata storage cells as well.

  net.core.rmem_default = 4194304
  net.core.wmem_default = 4194304
  net.core.rmem_max = 4194304
  net.core.wmem_max = 4194304
  

• If you are using Exadata storage, each Oracle VM RAC cluster requires its own Oracle Automatic Storage Management (Oracle ASM) disk group to be designated as the fast recovery area (FRA) such as +RECO. Refer to the "Exadata Database Machine Backup and Restore Configuration and Operational Best Practices" white paper for details.
• If you are using Oracle ZFS Storage Appliance, refer to the "Protecting Exadata Database Machine with the Oracle ZFS Storage Appliance: Configuration Best Practices" white paper for details.

2.10.8 Modifying the Memory Allocated to a User Domain

The following procedure describes how to modify the memory allocated to a user domain:

Note:

If you are decreasing the amount of memory allocated to a user domain, you must first review and adjust the SGA size of databases running in the user domain and the corresponding huge pages operating system configuration. Failing to do so may result in user domain that cannot start because too much memory is reserved for huge pages when the Linux operating system attempts to boot. See My Oracle Support note 361468.1 for details.

Note:

This operation requires user domain restart. It is not supported to modify memory allocation using the xm mem-set command.

1. Connect to the management domain.
2. Use the following command to determine the amount of free memory available, when increasing the allocation:

   # xm info | grep free_memory
   

   Note:

   When assigning free memory to a user domain, approximately 1 to 2 percent of free memory is used for metadata and control structures. Therefore, the amount of memory increase possible is 1 to2 percent less than free memory value.

3. Shut down the user domain gracefully using the name obtained from the xm list command. Use the -w option so the xm command waits until the domain is shut down before returning.

   # xm shutdown DomainName -w
   

   In the preceding command, DomainName is the name of the domain.

4. Create a backup copy of the/EXAVMIMAGES/GuestImages/DomainName/vm.cfg file.
5. Edit the memory and maxmem settings in the /EXAVMIMAGES/GuestImages/DomainName/vm.cfg file using a text editor. The memory and maxmem settings must be identical values.

   Note:

   If the memory and maxmem parameters are not identical values, then InfiniBand network interfaces are not configured during user domain start, which prevents proper Oracle CRS and database startup.

6. Use the following command to start the user domain:

   # xm create /EXAVMIMAGES/GuestImages/DomainName/vm.cfg
   

   Note:

   To see Oracle Linux boot messages during user domain startup, connect to the console during startup using the -c option. To disconnect from the console after startup is complete, press CTRL+].

2.10.9 Modifying the Number of Virtual CPUs Allocated to a User Domain

Note the following about modifying the number of virtual CPUs (vCPUs):

• All actions to modify the number of vCPUs allocated to a user domain are performed in the management domain.

• The number of vCPUs allowed for a user domain may be changed dynamically to a lower value or to a higher value provided it does not exceed the setting of maxvcpus parameter for the user domain.

• It is possible to over-commit vCPUs such that the total number of vCPUs assigned to all domains exceeds the number of physical CPUs on the system. However, over-committing CPUs should be done only when competing workloads for oversubscribed resources are well understood and concurrent demand does not exceed physical capacity.

The following procedure describes how to modify the number of virtual CPUs allocated to a user domain:

1. Determine the number of physical CPUs as follows:

   1. Use the following command in the management domain:

      # xm info | grep -A3 nr_cpus
      nr_cpus                : 24
      nr_nodes               : 2
      cores_per_socket       : 6
      threads_per_core       : 2
      

      In the output, note that the nr_nodes line refers to the number of sockets. The Exadata database server where the command is run is a 2-socket 6 cores per socket processor, resulting in 24 physical CPU threads (2 sockets x 6 cores/socket = 12 cores. 12 cores x 2 threads per core = 24 CPU threads).

   2. Use the following command to determine the current setting of vCPUs configured and online for a user domain:

      # xm list DomainName -l | grep vcpus
      
          (vcpus 4)
          (online_vcpus 2)
      

      In the preceding command, DomainName is the name of the user domain. The output from the command indicates the maximum number of vCPUs for the user domain is 4, and the current number of online vCPUs is 2. This user domain may have the number of online vCPUs adjusted to any value not greater than the vcpus parameter while the user domain remains online. The user domain must be taken offline to increase the number of online vCPUs to a value higher than the vcpus parameter.

2. Reduce or increase the number of vCPUs as follows:

   • To reduce the number of vCPUs:

     1. Determine the currently allocated number of vCPUs for the user domain using the following command:

        # xm list DomainName
        

     2. Reduce the currently allocated number of vCPUs using the following command:

        # xm vcpu-set DomainName vCPUs_preferred
        

        In the preceding command, vCPUs_preferred is the value of the preferred number of vCPUs

   • To increase the number of vCPUs

     1. Determine the current settings of the vcpus parameter using the following command:

        # xm list DomainName -l | grep vcpus
            (vcpus 4)
            (online_vcpus 2)
        

     2. If the preferred number of vCPUs is less than or equal to the value of the vcpus parameter, then run the following command to increase the number of online vCPUs.

        # xm vcpu-set DomainName vCPUs_preferred
        

        In the preceding command, vCPUs_preferred is the value of the preferred number of vCPUs

     3. If the preferred number of vCPUs is greater than the value of the vcpus parameter, then the user domain must be taken offline to increase the number of online vCPUs to a value higher than the vcpus parameter. Do the following:

        i. Shut down the user domain.

        ii. Create a backup copy of the /EXAVMIMAGES/GuestImages/DomainName/vm.cfg file.

        iii. Edit the /EXAVMIMAGES/GuestImages/DomainName/vm.cfg file to set the vcpus parameter to the desired number of vCPUs.

        Note: By default a user domain will online the number of vCPUs configured via the vcpus parameter. If you want a user domain to start with some vCPUs offline, then add the maxvcpus parameter to vm.cfg, setting it to the maximum number of vCPUs the user domain is permitted to have online. Set the vcpus parameter to the number of vCPUs to online when the user domain starts. For example, to start a user domain with 2 vCPUs online and to allow an additional 6 vCPUs to be added to the user domain while it remains online, use the following settings in vm.cfg:

        maxvcpus=8
        vcpus=2
        

        iv. Start the user domain.

2.10.10 Increasing the Disk Space in a User Domain

The procedures in this section describe how to increase the size of Logical Volume Manager (LVM) partitions, swap space, and file systems in a user domain. This section contains the following topics:

• Adding a New LVM Disk to a User Domain

• Increasing the Size of the root File System

• Increasing the Size of the /u01 File System

• Increasing the Size of the Grid Infrastructure Home or Database Home File System

• Increasing the Size of the Swap Area

2.10.10.1 Adding a New LVM Disk to a User Domain

This procedure describes how to add a new LVM disk to a user domain to increase the amount of usable LVM disk space in a user domain. This procedure is done so that the size of a file system or swap LVM partition can be increased. This procedure is performed while the system remains online.

Note:

This procedure requires steps be run in the management domain (Domain-0), and in the user domain.

Run all steps in this procedure as the root user.

1. In the management domain, verify the free disk space in /EXAVMIMAGES using the following command:

   # df -h /EXAVMIMAGES
   

   The following is an example of the output from the command:

   Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda3            721G  111G  611G  16% /EXAVMIMAGES
   

2. In the management domain, select a name for the new disk image, and verify that the name is not already used in the user domain.

   # ls -l /EXAVMIMAGES/GuestImages/DomainName/new_disk_image_name
   
   ls: /EXAVMIMAGES/GuestImages/DomainName/new_disk_image_name: No such file or \
   directory
   

   In the preceding command, DomainName is the name of the domain, and new_disk_image_name is the new disk image name.

3. In the management domain, create a new disk image.

   # qemu-img create /EXAVMIMAGES/GuestImages/DomainName/new_disk_image_name size
   

   In the following example of the command, the new disk image name is pv2_vgexadb.img, and the image size is 10 GB.

   # qemu-img create /EXAVMIMAGES/GuestImages/DomainName/pv2_vgexadb.img 10G
   

4. In the user domain, determine an available disk name. In the following example, disk names xvda through xvdd are used, and disk name xvde is unused.

   # lsblk -id
   NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
   xvda 202:0    0  13G  0 disk 
   xvdb 202:16   0  20G  0 disk /u01/app/12.1.0.2/grid
   xvdc 202:32   0  20G  0 disk /u01/app/oracle/product/12.1.0.2/dbhome_1
   xvdd 202:48   0  41G  0 disk
   

5. In the management domain, attach the new disk image to the user domain in read/write mode. In the following example, the new disk image is presented in the user domain as device /dev/xvde.

   # xm block-attach DomainName     \
   file:/EXAVMIMAGES/GuestImages/DomainName/new_disk_image_name /dev/xvde w
   

6. In the user domain, verify the disk device is available. In the following example, disk name xvde is available in the user domain.

   # lsblk -id
   NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
   xvda 202:0    0  13G  0 disk 
   xvdb 202:16   0  20G  0 disk /u01/app/12.1.0.2/grid
   xvdc 202:32   0  20G  0 disk /u01/app/oracle/product/12.1.0.2/dbhome_1
   xvdd 202:48   0  41G  0 disk 
   xvde 202:64   0  10G  0 disk
   

7. In the user domain, partition the new disk device. In the following example, disk device /dev/xvde is partitioned.

   # parted /dev/xvde mklabel gpt
   # parted -s /dev/xvde mkpart primary 0 100%
   # parted -s /dev/xvde set 1 lvm on
   

   The parted mkpart command may report the following message. This message can be ignored:

   Warning: The resulting partition is not properly aligned for best performance.
   

8. In the user domain, create an LVM physical volume on the new disk partition. In the following example, an LVM physical volume is created on disk partition /dev/xvde1.

   # pvcreate /dev/xvde1
   

9. In the user domain, extend the volume group and verify the additional space in the volume group. In the following example, disk name xvde is now available in the user domain.

   # vgextend VGExaDb /dev/xvde1
   # vgdisplay -s
   

10. In the management domain, make a backup of the user domain configuration file vm.cfg.

    # cp /EXAVMIMAGES/GuestImages/DomainName/vm.cfg   \
         /EXAVMIMAGES/GuestImages/DomainName/vm.cfg.backup
    

11. In the management domain, obtain the UUID of the user domain using the following command:

    # grep ^uuid /EXAVMIMAGES/GuestImages/DomainName/vm.cfg
    

    In the following example, the user domain UUID is 49ffddce4efe43f5910d0c61c87bba58.

    # grep ^uuid /EXAVMIMAGES/GuestImages/dm01db01vm01/vm.cfg
    uuid = '49ffddce4efe43f5910d0c61c87bba58'
    

12. In the management domain, generate a UUID for the new disk image using the following command:

    # uuidgen | tr -d '-'
    

    In the following example, the new disk UUID is 0d56da6a5013428c97e73266f81c3404.

    # uuidgen | tr -d '-'
    0d56da6a5013428c97e73266f81c3404
    

13. In the management domain, create a symbolic link from /OVS/Repositories to the new disk image using the following command:

    # ln -s /EXAVMIMAGES/GuestImages/DomainName/newDiskImage.img    \
     /OVS/Repositories/user_domain_uuid/VirtualDisks/new_disk_uuid.img
    

    In the following example, a symbolic link is created to the new disk image file pv2_vgexadb.img for user domain dm01db01vm01. The UUID for user domain dm01db01vm01 is 49ffddce4efe43f5910d0c61c87bba58. The UUID for the new disk image is 0d56da6a5013428c97e73266f81c3404.

    # ln -s /EXAVMIMAGES/GuestImages/dm01db01vm01/pv2_vgexadb.img \
    /OVS/Repositories/49ffddce4efe43f5910d0c61c87bba58/VirtualDisks/   \
    0d56da6a5013428c97e73266f81c3404.img
    

14. In the management domain, append an entry for the new disk to the disk parameter in the user domain configuration file vm.cfg. This makes the new disk image attach automatically to the user domain during the next startup. The new entry matches the following format:

    'file:/OVS/Repositories/user_domain_uuid/VirtualDisks/new_disk_uuid.img,disk_device,w'
    

    The following is an example of an original disk parameter entry in the vm.cfg file:

    disk=['file:/OVS/Repositories/49ffddce4efe43f5910d0c61c87bba58/VirtualDisks/  \
    76197586bc914d3d9fa9d4f092c95be2.img,xvda,w',                                 \
    'file:/OVS/Repositories/49ffddce4efe43f591 0d0c61c87bba58/VirtualDisks/       \
    78470933af6b4253b9ce27814ceddbbd.img,xvdb,w',                                 \
    'file:/OVS/Repositories/49ffddce4efe43f5910d0c61c87bba58/VirtualDisks/        \
    20d5528f5f9e4fd8a96f151a13d2006b.img,xvdc,w',                                 \
    'file:/OVS/Repositories/49ffddce4efe43f5910d0c61c87bba58/VirtualDisks/        \
    058af368db2c4f27971bbe1f19286681.img,xvdd,w']
    

    The following example shows an entry appended to the disk parameter for a new disk image that is accessible within the user domain as disk device /dev/xvde:.

    disk=['file:/OVS/Repositories/49ffddce4efe43f5910d0c61c87bba58/VirtualDisks/  \
    76197586bc914d3d9fa9d4f092c95be2.img,xvda,w',                                 \
    'file:/OVS/Repositories/49ffddce4efe43f591 0d0c61c87bba58/VirtualDisks/       \
    78470933af6b4253b9ce27814ceddbbd.img,xvdb,w',                                 \
    'file:/OVS/Repositories/49ffddce4efe43f5910d0c61c87bba58/VirtualDisks/        \
    20d5528f5f9e4fd8a96f151a13d2006b.img,xvdc,w',                                 \
    'file:/OVS/Repositories/49ffddce4efe43f5910d0c61c87bba58/VirtualDisks/        \
    058af368db2c4f27971bbe1f19286681.img,xvdd,w',                                 \
    'file:/OVS/Repositories/49ffddce4efe43f5910d0c61c87bba58/VirtualDisks/        \
    0d56da6a5013428c97e73266f81c3404.img,xvde,w']
    

2.10.10.2 Increasing the Size of the root File System

This procedure describes how to increase the size of the system partition and / (root) file system.

This procedure is performed while the file system remains online.

Note:

There are two system partitions, LVDbSys1 and LVDbSys2. One partition is active and mounted. The other partition is inactive and used as a backup location during upgrade. The size of both system partitions must be equal.

Keep at least 1 GB of free space in the VGExaDb volume group. The free space is used for the LVM snapshot created by the dbnodeupdate.sh utility during software maintenance. If you make snapshot-based backups of the / (root) and /u01 directories as described in "Creating a Snapshot-Based Backup of Oracle Linux Database Server," then keep at least 6 GB of free space in the VGExaDb volume group.

1. Collect information about the current environment.
   1. Use the df command to identify the amount of free and used space in the root partition (/)

      # df -h /
      

      The following is an example of the output from the command:

      Filesystem            Size  Used Avail Use% Mounted on
      /dev/mapper/VGExaDb-LVDbSys1
                             12G  5.1G  6.2G  46% / 
      

      Note:

      The active root partition may be either LVDbSys1 or LVDbSys2, depending on previous maintenance activities.

   2. Use the lvs command to display the current logical volume configuration.

      # lvs -o lv_name,lv_path,vg_name,lv_size
      

      The following is an example of the output from the command:

      LV        Path                   VG      LSize 
      LVDbOra1  /dev/VGExaDb/LVDbOra1  VGExaDb 10.00g
      LVDbSwap1 /dev/VGExaDb/LVDbSwap1 VGExaDb  8.00g
      LVDbSys1  /dev/VGExaDb/LVDbSys1  VGExaDb 12.00g
      LVDbSys2  /dev/VGExaDb/LVDbSys2  VGExaDb 12.00g 
      

2. Verify there is available space in the volume group VGExaDb using the vgdisplay command.

   # vgdisplay VGExaDb -s
   

   The following is an example of the output from the command:

   "VGExaDb" 53.49 GiB [42.00 GiB used / 11.49 GiB free]
   

   The volume group must contain enough free space to increase the size of both system partitions, and maintain at least 1 GB of free space for the LVM snapshot created by the dbnodeupdate.sh utility during upgrade. If there is not sufficient free space in the volume group, then add a new disk to LVM.

3. Resize both LVDbSys1 and LVDbSys2 logical volumes using the lvextend command.

   # lvextend -L +size /dev/VGExaDb/LVDbSys1
   # lvextend -L +size /dev/VGExaDb/LVDbSys2
   

   In the preceding command, size is the amount of space to be added to the logical volume. The amount of space added to each system partition must be the same.

   The following example extends the logical volumes by 10 GB:

   # lvextend -L +10G /dev/VGExaDb/LVDbSys1
   # lvextend -L +10G /dev/VGExaDb/LVDbSys2
   

4. Resize the file system within the logical volume using the resize2fs command.

   # resize2fs /dev/VGExaDb/LVDbSys1
   # resize2fs /dev/VGExaDb/LVDbSys2
   

5. Verify the space was extended for the active system partition using the df command.

   # df -h /
   

2.10.10.3 Increasing the Size of the /u01 File System

This procedure describes how to increase the size of the /u01 file system.

This procedure is performed while the file system remains online.

Note:

Keep at least 1 GB of free space in the VGExaDb volume group. The free space is used for the LVM snapshot created by the dbnodeupdate.sh utility during software maintenance. If you make snapshot-based backups of the / (root) and /u01 directories as described in "Creating a Snapshot-Based Backup of Oracle Linux Database Server," then keep at least 6 GB of free space in the VGExaDb volume group

1. Collect information about the current environment.
   1. Use the df command to identify the amount of free and used space in the /u01 partition.

      # df -h /u01
      

      The following is an example of the output from the command:

      Filesystem            Size  Used Avail Use% Mounted on
      /dev/mapper/VGExaDb-LVDbOra1
                            9.9G  1.7G  7.8G  18% /u01
      

   2. Use the lvs command to display the current logical volume configuration used by the /u01 file system.

      # lvs -o lv_name,lv_path,vg_name,lv_size /dev/VGExaDb/LVDbOra1
      

      The following is an example of the output from the command:

      LV        Path                  VG       LSize 
      LVDbOra1 /dev/VGExaDb/LVDbOra1  VGExaDb 10.00g
      

2. Verify there is available space in the volume group VGExaDb using the vgdisplay command.

   # vgdisplay VGExaDb -s
   

   The following is an example of the output from the command:

   "VGExaDb" 53.49 GiB [42.00 GiB used / 11.49 GiB free]
   

   If the output shows there is less than 1 GB of free space, then neither the logical volume nor file system should be extended. Maintain at least 1 GB of free space in the VGExaDb volume group for the LVM snapshot created by the dbnodeupdate.sh utility during an upgrade. If there is not sufficient free space in the volume group, then add a new disk to LVM.

3. Resize the logical volume using the lvextend command.

   # lvextend -L +sizeG /dev/VGExaDb/LVDbOra1
   

   In the preceding command, size is the amount of space to be added to the logical volume.

   The following example extends the logical volume by 10 GB:

   # lvextend -L +10G /dev/VGExaDb/LVDbOra1
   

4. Resize the file system within the logical volume using the resize2fs command.

   # resize2fs /dev/VGExaDb/LVDbOra1
   

5. Verify the space was extended using the df command.

   # df -h /u01
   

2.10.10.4 Increasing the Size of the Grid Infrastructure Home or Database Home File System

The Grid Infrastructure software home and the Oracle Database software home are created as separate disk image files in the management domain. The disk image files are located in the /EXAVMIMAGES/GuestImages/DomainName/ directory. The disk image files are attached to the user domain automatically during virtual machine startup, and mounted as separate, non-LVM file systems in the user domain.

The procedure in this section describes how to increase the size of the database home file system in a user domain. The same procedure can be used to increase the size of the grid home.

1. Connect to the user domain, and check the file system size using the df command.

   # df -h /u01/app/oracle/product/12.1.0.2/dbhome_1
   

   The following is an example of the output from the command:

   Filesystem  Size  Used Avail Use% Mounted on
    /dev/xvdc    20G  6.5G   13G  35% /u01/app/oracle/product/12.1.0.2/dbhome_1
   

2. Connect to the management domain, and then shut down the user domain using the xm command and specifying the name of the domain.

   # xm shutdown DomainName
   

3. Create an OCFS reflink to serve as a backup of the disk image that will be increased.

   # cd /EXAVMIMAGES/GuestImages/DomainName
   # reflink db12.1.0.2.1-3.img before_resize.db12.1.0.2.1-3.img
   

4. Create an empty disk image using the qemu-img command, and append it to the database home disk image.

   The empty disk image size is the size to extend the file system. The last command removes the empty disk image after appending to the database home disk image.

   # qemu-img create emptyfile 10G
   # cat emptyfile >> db12.1.0.2.1-3.img
   # rm emptyfile
   

5. Check the file system using the e2fsck command.

   # e42fsck -f db12.1.0.2.1-3.img
   

6. Resize the file system using the resize2fs command.

   # resize2fs db12.1.0.2.1-3.img
   

7. Start the user domain.

   # xm create /EXAVMIMAGES/GuestImages/DomainName/vm.cfg
   

8. Connect to the user domain, and verify the file system size was increased.

   # df -h /u01/app/oracle/product/12.1.0.2/dbhome_1
   

   The following is an example of the output from the command:

   Filesystem      Size  Used Avail Use% Mounted on
   /dev/xvdc        30G  6.5G   22G  23% /u01/app/oracle/product/12.1.0.2/dbhome_1
   

9. Connect to the management domain, and remove the backup image.

   # cd /EXAVMIMAGES/GuestImages/DomainName
   # rm back_up_image.img
   

   In the preceding command, back_up_image.img is the name of the backup image file.

2.10.10.5 Increasing the Size of the Swap Area

This procedure describes how to increase the amount of swap configured in a user domain.

1. Verify there is available space in the volume group VGExaDb using the vgdisplay command.

   # vgdisplay VGExaDb -s
   

   The following is an example of the output from the command:

   "VGExaDb" 53.49 GiB [42.00 GiB used / 11.49 GiB free]
   

   If the command shows that there is less than 1 GB of free space, then neither the logical volume nor file system should be extended. Maintain at least 1 GB of free space in the VGExaDb volume group for the LVM snapshot created by the dbnodeupdate.sh utility during an upgrade. If there is not sufficient free space in the volume group, then add a new disk to LVM.

2. Create a new logical volume of the size to increase swap space using the lvcreate command.

   In the following example, a new 8 GB logical volume named LVDbSwap2 is created.

   # lvcreate -L 8G -n LVDbSwap2 VGExaDb
   

3. Setup the new logical volume as a swap device with a unique label, such as SWAP2, using the mkswap command. The unique label is a device LABEL entry that is currently unused in the /etc/fstab file.

   # mkswap -L SWAP2 /dev/VGExaDb/LVDbSwap2
   

4. Enable the new swap device using the swapon command.

   # swapon -L SWAP2
   

5. Verify the new swap device is enabled using the swapon command.

   # swapon -s
   

   The following is an example of the output from the command:

   Filename         Type            Size      Used     Priority
   /dev/dm-3        partition       8388604   306108   -1
   /dev/dm-4        partition       8388604   0         -2
   

6. Edit the /etc/fstab file to add the new swap device by copying the existing swap entry, and then changing the LABEL value in the new entry to the label used to create the new swap device. In the following example, the new swap device was added to the /etc/fstab file as LABEL=SWAP2.

   # cat /etc/fstab
   LABEL=DBSYS   /                       ext4    defaults        1 1
   LABEL=BOOT    /boot                   ext4    defaults,nodev        1 1
   tmpfs         /dev/shm                tmpfs   defaults,size=7998m 0
   devpts        /dev/pts                devpts  gid=5,mode=620  0 0
   sysfs         /sys                    sysfs   defaults        0 0
   proc          /proc                   proc    defaults        0 0
   LABEL=SWAP    swap                    swap    defaults        0 0
   LABEL=SWAP2   swap                    swap    defaults        0 0
   LABEL=DBORA   /u01                    ext4    defaults        1 1
   /dev/xvdb     /u01/app/12.1.0.2/grid  ext4    defaults        1 1
   /dev/xvdc       /u01/app/oracle/product/12.1.0.2/dbhome_1       ext4   defaults        1 1
   

2.10.11 Expanding /EXAVMIMAGES on Management Domain after Database Server Disk Expansion

With the addition of a disk expansion kit to the database server, it is important to follow proper procedures to add this additional space to the /EXAVMIMAGES file system.

During deployment, all available disk space on a database server will be allocated in dom0 with the majority of the space allocated to /EXAVMIMAGES for user domain storage.

In the example below, dm01db01 is the name of the dom0 management domain, and dm01db01vm01 is a guest user domain.

1. Ensure reclaimdisks.sh has been run by using the -check option.

   Note that the last line reads "Layout: DOM0". If reclaimdisks.sh was not run, it would read "Layout: DOM0 + Linux".

   [root@dm01db01 ~]# /opt/oracle.SupportTools/reclaimdisks.sh -check
   Model is ORACLE SERVER X5-2
   Number of LSI controllers: 1
   Physical disks found: 4 (252:0 252:1 252:2 252:3)
   Logical drives found: 1
   Linux logical drive: 0
   RAID Level for the Linux logical drive: 5
   Physical disks in the Linux logical drive: 4 (252:0 252:1 252:2 252:3)
   Dedicated Hot Spares for the Linux logical drive: 0
   Global Hot Spares: 0
   Valid. Disks configuration: RAID5 from 4 disks with no global and dedicated hot spare disks.
   Valid. Booted: DOM0. Layout: DOM0.
   

2. Add the disk expansion kit to the database server.
   The kit consists of 4 additional hard drives to be installed in the 4 available slots. Remove the filler panels and install the drives. The drives may be installed in any order.
3. Verify that the RAID reconstruction is completed by seeing the warning and clear messages in the alerthistory.

   This may take several hours to complete. The example below shows that it took approximately 7 hours. Once the clear message (message 1_2 below) is present, the reconstruction is completed and it is safe to proceed.

   [root@dm01db01 ~]# dbmcli -e list alerthistory
   
            1_1     2016-02-15T14:01:00-08:00       warning         "A disk
    expansion kit was installed. The additional physical drives were automatically
    added to the existing RAID5 configuration, and reconstruction of the
    corresponding virtual drive was automatically started."
   
            1_2     2016-02-15T21:01:01-08:00       clear           "Virtual drive
    reconstruction due to disk expansion was completed."
   

4. Collect information about the current environment.

   [root@dm01db01 ~]# cat /proc/partitions |grep sda
      8        0 4094720000 sda
      8        1     524288 sda1
      8        2  119541760 sda2
      8        3 1634813903 sda3
   
   [root@dm01db01 ~]# df -h /EXAVMIMAGES
   Filesystem            Size  Used Avail Use% Mounted on
   /dev/sda3             1.6T   44G  1.5T   3% /EXAVMIMAGES
   
   [root@dm01db01 ~]# xm list
   Name                                        ID   Mem VCPUs      State   Time(s)
   Domain-0                                     0  8192     4     r-----  94039.1
   dm01db01vm01.us.oracle.com                4 16384     2     -b----   3597.3
   

5. Stop all user domain guests by running the command xm shutdown –a –w from dom0.

   After all user domain guests are shut down, only Domain-0 (dom0) should be listed.

   [root@dm01db01 ~]# xm shutdown –a -w
   Domain dm01db01vm01.us.oracle.com terminated 
   All domains terminated
   
   [root@dm01db01 ~]# xm list
   Name                                        ID   Mem VCPUs      State   Time(s)
   Domain-0                                     0  8192     4     r-----  94073.4
   

6. Run parted to verify the partition size.

   Use the syntax shown below to avoid unnecessary prompts.

   [root@dm01db01 ~]# parted -s /dev/sda print
   Model: LSI MR9361-8i (scsi)
   Disk /dev/sda: 4193GB
   Sector size (logical/physical): 512B/512B
   Partition Table: gpt
   
   Number  Start   End     Size    File system  Name     Flags
    1      32.8kB  537MB   537MB   ext3         primary  boot 
    2      537MB   123GB   122GB                primary  lvm  
    3      123GB   1797GB  1674GB               primary       
   

   The partition table shown above lists partition 3 as 1674 GB. This is the partition that will be modified. Note that the start of this partition is at 123 GB and that the disk size is 4193 GB, shown in the "Disk" line above the partition table output. You will use these values in step 8.
7. Remove partition 3.

   [root@dm01db01 ~]# parted /dev/sda 
   (parted) rm 3
   (parted) quit
   

   This command produces no output.

8. Re-create the partition specifying the same starting point, then print the resulting updated partition table.

   [root@dm01db01 ~]# parted /dev/sda
   (parted) mkpart primary 123gb 4193gb 
   (parted) print 
   Model: LSI MR9361-8i (scsi)
   Disk /dev/sda: 4193GB
   Sector size (logical/physical): 512B/512B
   Partition Table: gpt
   
   Number  Start   End     Size    File system  Name     Flags
    1      32.8kB  537MB   537MB   ext3         primary  boot 
    2      537MB   123GB   122GB                primary  lvm  
    3      123GB   4193GB  4070GB               primary       
   
   (parted)  quit
   

9. Mount the /EXAVMIMAGES partition again.

   [root@dm01db01 ~]# mount /EXAVMIMAGES
   
   [root@dm01db01 ~]# df -h /EXAVMIMAGES
   Filesystem            Size  Used Avail Use% Mounted on
   /dev/sda3             1.6T   44G  1.5T   3% /EXAVMIMAGES
   

10. Note that the size of the file system is still the same, 1.6 TB, as in step 4.
11. Verify that the partition table as seen by the kernel shows the updated size for partition 3.

    The output for sda3 should now be larger compared to the output observed earlier in step 4.

    [root@dm01db01 ~]# cat /proc/partitions |grep sda
       8        0 4094720000 sda
       8        1     524288 sda1
       8        2  119541760 sda2
       8        3 3974653903 sda3
    

12. Expand the file system.

    You can do this while the file system is mounted and processes are running. Note the updated file system size, compared to the value in step 4. The tunefs.ocfs2 command typically runs very quickly and should have no output normally.

    [root@dm01db01 ~]# tunefs.ocfs2 -S /dev/sda3
    
    [root@dm01db01 ~]# df -h /EXAVMIMAGES
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda3             3.8T   44G  3.7T   2% /EXAVMIMAGES
    

13. Restart user domains.

2.10.12 Creating Oracle VM Oracle RAC Clusters

This procedure creates Oracle VM Oracle RAC clusters using Oracle Exadata Deployment Assistant configuration tool and deployment tool.

The requirements for adding an Oracle VM Oracle RAC cluster are as follows:

• The system has already been deployed with one or more Oracle VM Oracle RAC clusters.

• System has available resources, such as memory, CPU, local disk space, and Oracle Exadata Storage Server disk space.

• Oracle Exadata Deployment Assistant deployment files used for initial system configuration are available.

1. Verify there are sufficient resources to add a new user domain in the management domain.

   If you are creating an Oracle VM Oracle RAC cluster, then verify resources in all management domains where you are creating a new user domain.

2. Use the following command to verify the Oracle Exadata Storage Server disk space:

   # dcli -l celladmin -g cell_group "cellcli -e 'list celldisk attributes name, \
    diskType, freeSpace where freeSpace>0'"
   

3. By default, database servers in Oracle Exadata Database Machine contain only packages required to run Oracle Database, and are not capable of running Oracle Exadata Deployment Assistant configuration tool.
4. Download the latest Oracle Exadata Deployment Assistant from My Oracle Support note 888828.1, and place it on a system capable of running a graphic-based program.

   By default, database servers in Oracle Exadata Database Machine contain only packages required to run Oracle Database, and are not capable of running Oracle Exadata Deployment Assistant configuration tool.

5. Obtain the Oracle Exadata Deployment Assistant template files used to deploy the system.
6. Run the Oracle Exadata Deployment Assistant configuration tool as follows:
   1. Click Import.
   2. Select and open the XML file used to deploy the system with the name CustomerName-NamePrefix.xml.
   3. Click Next as needed to get to the Define Clusters page, and verify the IP address and host name information as you navigate the pages. If there have been no networking changes since the initial deployment, then no changes are needed.
   4. Increment the number of clusters on the Define Clusters page.
   5. Select the new cluster tab to edit the cluster information. Do not change any other clusters.
   6. Enter a unique cluster name for the cluster.
   7. Select the Oracle Virtual Server (OVS) and CELL components for the new cluster, and then click Add.

      Note:

      The recommended practice for best performance and simplest administration is to select all cells.

   8. Click Next as needed to get to the new cluster page. Do not change any other clusters.
   9. Enter the information for the new cluster. Information includes the virtual guest size, disk group details, and database name. The database name must be unique for all databases that use the same Oracle Exadata Storage Servers.
   10. Click Next to get to the Review and Edit page, and verify the information for the new cluster.
   11. Click Next as needed to get to the Generate page.
   12. Click Next to generate the new configuration files.
   13. Select the destination directory for the configuration files.
   14. Click Save.

       Note:

       If the Oracle VM Defaults were altered for this new cluster, then configuration details for existing clusters will be re-written to match the new template settings. For example, if you previously deployed vm01 as SMALL with memory=8GB, and then change the SMALL template to memory=10GB for this new VM, then the new Oracle Exadata Deployment Assistant XML files show vm01 with memory=10GB even though there was no intent to change vm01.

   15. Click Installation Template on the Finish page to review the details of the new cluster.
   16. Click Finish to exit the configuration tool.
7. Verify the XML file for the new cluster exists and has the name CustomerName-NamePrefix-ClusterName.xml in the destination folder.
8. Obtain the deployment files for the Oracle Grid Infrastructure and Oracle Database releases selected, and place them in the Oracle Exadata Deployment Assistant WorkDir directory.
9. Run the Oracle Exadata Deployment Assistant Deployment Tool using the -cf option to specify the XML file for the new cluster, and the -l option to list the steps using the following command:

   $ ./install.sh -cf    \
   ExadataConfigurations/CustomerName-NamePrefix-ClusterName.xml -l
   

   You should see output similar to the following:

   Initializing 
   ||||| 
   1. Validate Configuration File 
   2. Update Nodes for Eighth Rack 
   3. Create Virtual Machine 
   4. Create Users 
   5. Setup Cell Connectivity 
   6. Calibrate Cells 
   7. Create Cell Disks 
   8. Create Grid Disks 
   9. Configure Alerting 
   10. Install Cluster Software 
   11. Initialize Cluster Software 
   12. Install Database Software 
   13. Relink Database with RDS 
   14. Create ASM Diskgroups 
   15. Create Databases 
   16. Apply Security Fixes 
   17. Install Exachk 
   18. Create Installation Summary 
   19. Resecure Machine
   

10. Skip the following steps when adding new Oracle VM clusters in an existing Oracle VM environment on Exadata:

    • (For Eight Rack systems only) 2. Update Nodes for Eighth Rack

    • 6. Calibrate Cells

    • 7. Create Cell Disks

    • 19. Resecure Machine

    Note:

    The step numbers change based on the selected hardware configuration. Use the step names to identify the correct steps on your system.

    For example, to execute step 1, run the following command:

    $ ./install.sh -cf \
    ExadataConfigurations/CustomerName-NamePrefix-ClusterName.xml -s 1
    

    To make OEDA run only a subset of the steps, you can specify a range, for example:

    $ ./install.sh -cf \
    ExadataConfigurations/CustomerName-NamePrefix-ClusterName.xml –r 3–5
    

11. For all other systems, run all steps except for the Configure Alerting step using the XML file for the new cluster.

    To run an individual step, use a command similar to the following, which executes the first step:

    $ ./install.sh -cf \
    ExadataConfigurations/CustomerName-NamePrefix-ClusterName.xml -s 1
    

2.10.13 Expanding an Oracle VM Oracle RAC Cluster on Exadata

You can expand an existing Oracle RAC cluster on Oracle VM by adding guest domains using Oracle Exadata Deployment Assistant (OEDA).

Note:

During the execution of this procedure, the existing Oracle RAC cluster nodes along with their database instances incur zero downtime.

Use cases for this procedure include:

• You have an existing Oracle RAC cluster that uses only a subset of the database servers of an Exadata rack, and now the nodes not being used by the cluster have become candidates for use.

• You have an existing Oracle RAC cluster on Exadata that recently got inter-racked with a new Oracle Virtual Server (OVS) node or a separate Exadata rack.

1. Use OEDA to build a new XML file containing information about the new guest domains (virtual machines) to be added on the dom0’s that will host the new guest domains.

   Save the generated XML file in unzipped_OEDA_location/ExadataConfigurations.

   If you are adding a single guest domain, the XML file will portray a single node Oracle RAC cluster.

   All the screens before the "Define Clusters" screen will have the same data as the XML file of the existing cluster.

   1. Select the appropriate Oracle Virtual Server (OVS) in the Define Clusters screen. The following figure shows a single node addition to a cluster.

      Figure 2-1 Define Clusters page in OEDA

      
      Description of "Figure 2-1 Define Clusters page in OEDA"
   2. Enter cluster detail such as IP addresses and pool size in the cluster details screen. The following figure shows a single node addition to a cluster.

      Figure 2-2 Cluster page in OEDA

      
      Description of "Figure 2-2 Cluster page in OEDA"
2. Run OEDA (install.sh) using the XML file built in step 1 and execute OEDA for the following steps:

   • Create Virtual Machine

   • Create Users

   • Setup Cell Connectivity

   The step numbers of the above steps can be listed using :

   ./install.sh -cf ./ExadataConfigurations/xml_file_generated_in_step_1 -l
   

   Note:

   The step numbers change based on the selected hardware configuration so look them up carefully using the command above.

   To make OEDA run only a subset of the steps, you can specify a range, for example: –r 2–4

   cd unzipped_OEDA_location
   ./install.sh -cf ./ExadataConfigurations/xml_file_generated_in_step_1 -r range_of_step_numbers_for_the_above_steps
   

3. Clean up the guest domains to be added to the cluster and prepare it to be used by addnode.sh later.
   1. Log into the newly created guest domains as root and run the following:

      # cd /
      # chown -R oracle:oinstall /u01 
      # cd /u01/app/12.1.0.2/grid  (cd to the Grid_home)
      # rm -rf *
      # cd /u01/app/oracle/product/12.1.0.2/dbhome_1  (cd to the Oracle_home)
      # rm -rf *
      

   2. Make sure oraInventory does not exist inside /u01/app.
   3. Make sure /etc/oraInst.loc does not exist.
4. Set up secure shell (SSH) equivalence from the newly added guest domains to all the other nodes of the cluster.
   1. Login as oracle to each guest domain belonging to the cluster including the newly created guest domains.
   2. Create a file called db_group in /home/oracle containing all the guest domain names belonging to the cluster including the newly added guest domains.
   3. Run the following command as oracle from all the guest domains of the cluster including the newly added guest domains and answer all prompts.

      dcli -g db_group -l oracle -k
      

   4. Validate SSH equivalence has been setup correctly across all the nodes of the cluster.
5. Sync up the files across all the nodes of the cluster.

   Copy or merge the contents of the following files in the newly added nodes using files from an existing cluster node such that they are same across all the nodes in the cluster:

   • /etc/security/limits.conf

   • /etc/hosts

     The /etc/hosts file should be first modified on one node to include all IP address of the cluster nodes and cell IP addresses and then that file should be copied over to all cluster nodes.

   • /etc/oracle/cell/network-config/cellip.ora

   • /etc/sysctl.conf

6. Reboot all the newly added nodes (guest domains).
7. Change ownership of /etc/oracle/cell/network-config/cellip.ora in the newly added guest domains.

   # chown oracle:oinstall /etc/oracle/cell/network-config/cellip.ora
   

8. Verify the integrity of the cluster and new nodes by running the following command from an existing node of the cluster. This node will be called the init-node in the remaining steps.

   $ cluvfy stage -pre nodeadd -n comma_delimited_node_names [-fixup] [-verbose]
   

   The –fixup and –verbose options are optional.

9. To extend the Oracle Grid Infrastructure home to the new nodes, navigate to the Grid_home/addnode directory on init-node and run the addnode.sh script as the user that installed Oracle Clusterware.

   $ cd Grid_home/addnode
   
   $ ./addnode.sh -silent "CLUSTER_NEW_NODES={comma_delimited_node_names}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={comma_delimited_node_vips)}"
   

10. When prompted, run orainstRoot.sh as root on all the newly added nodes sequentially.

    # /opt/oracle/oraInventory/orainstRoot.sh
    

11. Disable HAIP and then run Grid_home/root.sh as root on all the newly added nodes sequentially.

    As the root user, set the HAIP_UNSUPPORTED operating system parameter to TRUE to disable HAIP on the new nodes.

    # export HAIP_UNSUPPORTED=true
    # /u01/app/12.1.0.2/grid/root.sh
    

12. Extend the database home to the newly added node.

    Navigate to the $ORACLE_HOME/addnode directory on init-node and run addnode.sh as the user that installed Oracle RAC:

    $ cd $ORACLE_HOME/addnode
    
    $ ./addnode.sh -silent "CLUSTER_NEW_NODES={comma_delimited_node_names}"
    

13. Run $ORACLE_HOME/root.sh as root on the newly added nodes when prompted.
14. Use the Oracle Database Configuration Assistant (DBCA) to add the database instance to the newly added nodes in silent mode or using the graphical user interface.

    If using the silent mode, run the following command:

    $ $ORACLE_HOME/bin/dbca -addInstance -gdbName global database name
          -nodeName node_name_for_new_instance
          -instanceName instance_name_for_new_instance
          -sysDBAUserName SYS -sysDBAPassword password -silent
    

15. Set the cluster_interconnects parameter in the newly added Oracle ASM instances and the database instances.

    From each newly added node, perform these steps for the new Oracle ASM instance and the new database instance on that node:

    1. Start up SQL*Plus and connect to the Oracle ASM instance.
    2. Set the cluster_interconnects parameter for the new Oracle ASM instance.

       alter system set cluster_interconnects=IP_address_of_private_interconnects scope=spfile sid=newly_added_ASM_instance;
       

    3. Start up SQL*Plus and connect to the database instance.
    4. Set the cluster_interconnects parameter for the new database instance.

       alter system set cluster_interconnects=IP_address_of_private_interconnects scope=spfile sid=newly_added_DB_instance;
       

16. Restart the Oracle Grid Infrastructure stack on the newly added nodes.

2.10.14 Creating a User Domain Without Oracle Grid Infrastructure and Oracle Database

A user domain can be created without Oracle Grid Infrastructure and Oracle Database installed on the system. The new user domain has the following characteristics:

• Operating system image is Oracle Linux

• Access to the management, client, and InfiniBand networks

• No Oracle Grid Infrastructure and Oracle Database is installed

The following procedure creates a user domain without Oracle Grid Infrastructure and Oracle Database installed:

1. Allocate new, unused, IP addresses and host names for the new user domain. IP addresses and host names are needed for the management network, client (SCAN) network, and the private InfiniBand network.

   Note:

   Ensure the intended InfiniBand network IP addresses are unused by using the ping command for each address. The ibhosts command cannot be used to determine all InfiniBand network IP addresses in use because it does not contain entries for user domains.

2. If necessary, obtain an updated user domain (domU) system image file.

   The exadata.img.domu_maker command that you will run later in this procedure to create a user domain requires the user domain (domU) system image file System.first.boot.version.img in /EXAVMIMAGES, where version matches the management domain Exadata software version as determined by running the "imageinfo -ver" command in the management domain.

   For example, when exadata.img.domu_maker is run to create a new user domain and the management domain Exadata software version is 12.1.2.1.1.150316.2, the user domain (domU) system image file /EXAVMIMAGES/System.first.boot.12.1.2.1.1.150316.2.img must exist.

   # imageinfo -ver
   12.1.2.1.1.150316.2
   
   # ls -l /EXAVMIMAGES/System.first.boot.12.1.2.1.1.150316.2.img
   -rw-r--r-- 1 root root 13958643712 Mar 23 12:25 /EXAVMIMAGES/System.first.boot.12.1.2.1.1.150316.2.img
   

   If the user domain (domU) system image file does not exist, then it must be obtained from My Oracle Support and placed in /EXAVMIMAGES in the management domain. See My Oracle Support note 888828.1 for additional information.

3. In the management domain, copy an existing XML configuration file from a deployed user domain to a new file name using the following command:

   # cp /EXAVMIMAGES/conf/existingDomainName-vm.xml /EXAVMIMAGES/conf/newDomainName-vm.xml
   

   In the preceding command, existingDomainName-vm.xml is the XML configuration file of the deployed user domain, and newDomainName-vm.xml is the name of the new file.

   In the following example, the configuration file for user domain "dm01db01vm01" is copied to nondbdomain-vm.xml.

   # cp /EXAVMIMAGES/conf/dm01db01vm01-vm.xml /EXAVMIMAGES/conf/nondbdomain-vm.xml
   

4. In the management domain, edit the new XML file as follows:

   1. Change all <Hostname> tags to match the new host names for the respective networks.

   2. Change all <IP_address> tags to match the new IP addresses for the respective networks.

   3. Change the <virtualMachine> tag to contain the new host name.

   4. Change the <hostName> tag to contain the new host name.

   5. Delete the entire <disk id="disk_2"> and <disk id="disk_3"> elements, including all their sub-elements. You must delete the entire entry between the starting <disk> tag to the corresponding closing </disk>.

5. In the management domain, allocate InfiniBand network GUIDs for the new user domain using the /opt/exadata_ovm/exadata.img.domu_maker command.

   # /opt/exadata_ovm/exadata.img.domu_maker allocate-guids \
        /EXAVMIMAGES/conf/newDomainName-vm.xml              \
        /EXAVMIMAGES/conf/final-newDomainName-vm.xml
   

6. In the management domain, create the new user domain using the /opt/exadata_ovm/exadata.img.domu_maker command.

   # /opt/exadata_ovm/exadata.img.domu_maker start-domain \
        /EXAVMIMAGES/conf/final-newDomainName-vm.xml
   

2.10.15 Moving a User Domain to a Different Database Server

User domains can move to different database servers. The target database server must meet the following requirements:

• The target database server must have the same Oracle Exadata Storage Server Software release installed with Oracle VM.

• The target database server must have the same network visibility.

• The target database server must have access to the same Exadata Storage Servers.

• The target database server must have sufficient free resources (CPU, memory, and local disk storage) to operate the user domain.

  • It is possible to over-commit vCPUs such that the total number of vCPUs assigned to all domains exceeds the number of physical CPUs on the system. Over-committing CPU can be done only when competing workloads for over-subscribed resources are well understood and concurrent demand does not exceed physical capacity.

  • It is not possible to over-commit memory.

  • Copying disk images to the target database server may increase space allocation of the disk image files because the copied files are no longer able to benefit from the disk space savings gained by using OCFS2 reflinks.

• The user domain name must not be already in use on the target database server.

The following procedure moves a user domain to a new database server in the same Oracle Exadata Storage Server Software configuration. All steps in this procedure are performed in the management domain.

1. Shut down the user domain using the following command:

   # xm shutdown DomainName -w
   

2. Copy the user domain disk image and configuration files to the target database server using the following command:

   # scp -r /EXAVMIMAGES/GuestImages/DomainName/ target:/EXAVMIMAGES/GuestImages
   

   In the preceding command, DomainName is the name of the domain.

3. Obtain the UUID of the user domain using the following command:

   # grep ^uuid /EXAVMIMAGES/GuestImages/DomainName/vm.cfg
   

   In the following example the user domain UUID is 49ffddce4efe43f5910d0c61c87bba58.

   # grep ^uuid /EXAVMIMAGES/GuestImages/DomainName/vm.cfg
   uuid = '49ffddce4efe43f5910d0c61c87bba58'
   

4. Using the UUID of the user domain, copy the user domain symbolic links from /OVS/Repositories to the target database server using the following command:

   # tar cpvf - /OVS/Repositories/UUID/ | ssh target "tar xpvf - -C /"
   

5. Start the user domain on the target database server using the following command:

   # xm create /EXAVMIMAGES/GuestImages/DomainName/xm.cfg
   

2.10.16 Removing an Oracle RAC Cluster Running in Oracle VM

You can remove all Oracle RAC nodes of an Oracle VM cluster, including the databases running within the cluster and all data stored on the Oracle Exadata Storage Server used by those databases.

To remove only a subset of user domains of a Oracle VM cluster, refer to the next section.

There are two main steps to remove a Oracle VM cluster:

• Remove the user domain files from the management domain.

• Remove the unused Oracle Exadata grid disks.

Note:

If the Oracle Exadata Deployment Assistant xml configuration files are to be reused later, then they will be not synchronized because the definition for the removed user domain still exists in Oracle Exadata Deployment Assistant files.

1. Run the following example script as the grid software owner on any user domain to be removed.

   The example shell script generates two scripts, list_griddisk.sh and drop_griddisk.sh, that are run later in this procedure. Do not run the generated scripts until instructed.

   #!/bin/bash
    
   # Run this script as the Grid Infrastructure software owner.
   #
   # This script identifies griddisks used by this cluster and the cells to
   # which they belong, then creates two shell scripts - the list script to
   # show the current status, and the drop script to drop the griddisks.
   #
   # In order for the drop script to succeed, the griddisks must not be in use,
   # meaning databases and CRS are down, and the list script returns no output.
   #
   # The generated scripts are designed to run via dcli -x
    
   ORACLE_SID=$(awk -F: '/^+ASM/{print $1}' /etc/oratab)
   ORAENV_ASK=NO . oraenv >/dev/null
    
   listGriddiskScript=list_griddisk.sh
   dropGriddiskScript=drop_griddisk.sh
    
   rm -f $listGriddiskScript $dropGriddiskScript
    
   gridDiskList=$(asmcmd lsdsk --suppressheader | awk -F'/' '{print $NF}')
   if [[ ${PIPESTATUS[0]} != 0 ]]; then echo "asmcmd failed - exiting"; exit 1; fi
    
   cellList=$(echo "$gridDiskList" | awk -F_ '{print $NF}' | sort -u)
    
   for cell in $cellList; do
     myGriddisks=$(echo "$gridDiskList" | grep ${cell}$ | tr '\n' ',')
     echo "[[ \$(hostname -s) == ${cell} ]] && cellcli -e 'LIST GRIDDISK \
           ${myGriddisks%,} attributes name, asmDiskGroupName, asmModeStatus \
           where asmModeStatus != UNKNOWN'" >> $listGriddiskScript
     echo >> $listGriddiskScript
   done
    
   chmod +x $listGriddiskScript
    
   echo
   echo "Run the following command to list griddisks in use by this cluster:"
   echo
   echo "# dcli -l celladmin -c ${cellList//$'\n'/,} -x $listGriddiskScript"
   echo
    
   for cell in $cellList; do
     myGriddisks=$(echo "$gridDiskList" | grep ${cell}$ | tr '\n' ',')
     echo "[[ \$(hostname -s) == ${cell} ]] && cellcli -e 'DROP GRIDDISK \
           ${myGriddisks%,}'" >> $dropGriddiskScript
     echo >> $dropGriddiskScript
   done
    
   chmod +x $dropGriddiskScript
    
   echo
   echo "Stop CRS on all nodes in this cluster, then run the following"
   echo "command to drop all griddisks used by this cluster:"
   echo
   echo "# dcli -l celladmin -c ${cellList//$'\n'/,} -x $dropGriddiskScript"
   echo
    
   exit 
   

2. Shut down the databases and Oracle Grid Infrastructure in all user domains that will be removed:

    # Grid_home/bin/crsctl stop crs -f
   

3. Run the list_griddisk.sh script generated earlier from any user domain that will be removed.

   Note:

   • Run the script using the dcli command to connect as the celladmin user to all Oracle Exadata Storage Servers in the configuration.

   • Before running the dcli command, set up a passwordless SSH connection between the grid software owner on the database server and the celladmin user on the cells. Otherwise, the command will keep prompting you to enter the password.

   The following is an example of the command:

   $ dcli -l celladmin -c dm01celadm01,dm01celadm02,dm01celadm03  \
   -x list_griddisk.sh
   

   The list_griddisk.sh script should not output any grid disks. Grid disks returned from the list_griddisk.sh script are considered still in use.

   Do not proceed until the list_griddisk.sh script returns empty output indicating no grid disks are in use. Verify that Oracle Grid Infrastructure and the databases are shut down on all user domains to be dropped.

4. Run the drop_griddisk.sh script generated earlier from any user domain that you want to remove.

   Run the script using the dcli command to connect as the celladmin user to all Oracle Exadata Storage Servers in the configuration.

   $ dcli -l celladmin -c dm01celadm01,dm01celadm02,dm01celadm03 \
   -x drop_griddisk.sh
   

5. Run the exadata.img.domu_maker command from the management domain of each user domain you want to remove.

   This command removes the user domains, where DomainName is the name of the user domain.

   # /opt/exadata_ovm/exadata.img.domu_maker remove-domain DomainName
   

   In the following example, the commands remove the two user domains for a two-node Oracle VM RAC cluster in which the user domain dm01db01vm04 runs on the management domain dm01db01, and the user domain dm01db02vm04 runs on the management domain dm01db02.

   [root@dm01db01 ~] # /opt/exadata_ovm/exadata.img.domu_maker \
   remove-domain dm01db01vm04
   [INFO] Start with command line: /opt/exadata_ovm/exadata.img.domu_maker \
    remove-domain dm01db01vm04
   [INFO] Shutting down DomU dm01db01vm04
   [INFO] Autostart link for dm01db01vm04 deleted from /etc/xen/auto
   [INFO] Deleted OVM repository /OVS/Repositories/7bfd49d6bd5a4b2db2e46e8234788067 for DomU dm01db01vm04
   [INFO] Deleted guest vm /EXAVMIMAGES/GuestImages/dm01db01vm04 for \
   DomU dm01db01vm04
    
   [root@dm01db02 ~]# /opt/exadata_ovm/exadata.img.domu_maker \
   remove-domain dm01db02vm04
   [INFO] Start with command line: /opt/exadata_ovm/exadata.img.domu_maker \
   remove-domain dm01db02vm04
   [INFO] Shutting down DomU dm01db02vm04
   [INFO] Autostart link for dm01db02vm04 deleted from /etc/xen/auto
   [INFO] Deleted OVM repository /OVS/Repositories/1d29719ff26a4a17aca99b2f89fd8032 for DomU dm01db02vm04
   [INFO] Deleted guest vm /EXAVMIMAGES/GuestImages/dm01db02vm04  \
   for DomU dm01db02vm04
   

2.10.17 Deleting a User Domain from an Oracle VM Oracle RAC Cluster

You can remove a single Oracle RAC node from an Oracle VM cluster.

The Oracle Exadata grid disks remain in use by the remaining nodes in the cluster, and must not be dropped.

Note:

If Oracle Exadata Deployment Assistant xml configuration files are to be reused later, then they will be not synchronized because the definition for the removed user domain still exists in Oracle Exadata Deployment Assistant files.

1. Delete the cluster node.
2. Use the following command to shut down and remove the user domain, where DomainName is the name of the domain:

   # /opt/exadata_ovm/exadata.img.domu_maker remove-domain DomainName
   

   This command removes the user domain files from the management domain.

2.10.18 Implementing Tagged VLAN Interfaces

This topic describes the implementation of tagged VLAN interfaces in Oracle VM environments on Exadata.

Oracle databases running in Oracle VM guests on Oracle Exadata Database Machine are accessed through the client Ethernet network defined in the Oracle Exadata Deployment Assistant (OEDA) configuration tool. Client network configuration in both the management domain (dom0) and user domains (domU's) is done automatically when the OEDA installation tool creates the first user domain during initial deployment.

The following figure shows a default bonded client network configuration:

Figure 2-3 NIC Layout in an Oracle Virtual Environment

Description of "Figure 2-3 NIC Layout in an Oracle Virtual Environment"

The network has the following configuration:

1. In the dom0, eth slave interfaces (for example, eth1 and eth2, or eth4 and eth5) that allow access to the domU client network defined in OEDA are discovered, configured, and brought up, but no IP is assigned.

2. In the dom0, bondeth0 master interface is configured and brought up, but no IP is assigned.

3. In the dom0, bridge interface vmbondeth0 is configured, but no IP is assigned.

4. In the dom0, one virtual backend interface (vif) per domU that maps to that particular domU's bondeth0 interface is configured and brought up, but no IP is assigned. These vifs are configured on top of the bridge interface vmbondeth0, and the mapping between the dom0 vif interface and its corresponding user domain interface bondeth0 is defined in the user domain configuration file called vm.cfg, located in /EXAVMIMAGES/GuestImages/user domain name.

For default installations, a single bondeth0 and a corresponding vmbondeth0 bridge interface is configured in the dom0 as described above. This bondeth0 interface is based on the default Access Virtual Local Area Network (Access VLAN). The ports on the switch used by the slave interfaces making up bondeth0 are configured for Access VLAN.

Using VLAN Tagging

If there is a need for virtual deployments on Exadata to access additional VLANs on the client network, such as enabling network isolation across user domains, then 802.1Q-based VLAN tagging is a solution. The following figure shows a client network configuration with VLAN tagging.

Figure 2-4 NIC Layout for Oracle Virtual Environments with VLAN Tagging

Description of "Figure 2-4 NIC Layout for Oracle Virtual Environments with VLAN Tagging"

For instructions on how to configure and use such additional VLAN tagged interfaces on the client network, see My Oracle Support note 2018550.1. The Access VLAN must stay working and configured before and after these instructions are followed. At no time is the Access VLAN to be disabled.

2.10.19 Implementing InfiniBand Partitioning across Oracle VM Oracle RAC Clusters on Oracle Exadata

One of the key requirements of consolidated systems from a security standpoint is network isolation across the multiple environments within a consolidated system. For consolidations achieved using Oracle VM Oracle RAC clusters on Oracle Exadata, this means isolation across the different Oracle RAC clusters such that network traffic of one Oracle RAC cluster is not accessible to another Oracle RAC cluster. For the Ethernet networks, this is accomplished using VLAN tagging as described in My Oracle Support DocID 2018550.1. For the InfiniBand network, this is accomplished using custom InfiniBand partitioning, dedicated partition keys, and partitioned tables.

An InfiniBand partition defines a group of InfiniBand nodes or members that are allowed to communicate with one another. With InfiniBand partitioning, partitions identified by unique partition keys are created and are managed by the master subnet manager. Members are then assigned to these custom partitions. Members within a partition can only communicate among themselves (depending on the membership as explained in the Appendix 1 of My Oracle Support DocID 2018550.1). A member of one partition cannot communicate with a member of a different partition regardless of the membership. Continuing along these lines, the Oracle VM Oracle RAC nodes of one particular cluster are assigned one dedicated partition for the clusterware communication and one partition for communication with the storage cells. This way, the nodes of one Oracle RAC cluster will not be able to communicate with the nodes of another Oracle RAC cluster that belong to a different partition. The nodes in each Oracle RAC cluster have different partition keys assigned to them.

By default, the InfiniBand subnet manager provides a single partition that is identified by the partition key 0x7FFF (limited membership) or 0xFFFF (full membership). In Oracle VM deployments on Oracle Exadata where custom InfiniBand partitioning is not used, the partition key 0xFFFF is used across all the user domains.

Figure 2-5 Oracle VM Oracle RAC Clusters on Oracle Exadata without InfiniBand Network Isolation Across Clusters

Description of "Figure 2-5 Oracle VM Oracle RAC Clusters on Oracle Exadata without InfiniBand Network Isolation Across Clusters"

With non-default custom partitions in place for implementing isolation across the Oracle VM Oracle RAC clusters, the configuration changes to what is shown in Figure 2-6. New interfaces clib0, clib1 (for the cluster pkey) and stib0, stib1 (for the storage pkey) exist in each of the user domains (domU's).

There is no change to InfiniBand interfaces in the management domain (dom0).

Figure 2-6 Oracle VM Oracle RAC Clusters on Oracle Exadata with InfiniBand Network Isolation Across Clusters Using InfiniBand Partitioning

Description of "Figure 2-6 Oracle VM Oracle RAC Clusters on Oracle Exadata with InfiniBand Network Isolation Across Clusters Using InfiniBand Partitioning"

2.10.19.1 Implementing InfiniBand Partitioning across OVM RAC Clusters: Requirements

Before configuring InfiniBand partitioning, ensure that:

• You have configured OVM on your Exadata system.

• All the user domains and storage cells are using the default partition key 0xFFFF.

• You have set up passwordless secure shell (ssh) access for the root user from one of the management domains (dom0 node) to all the OVM RAC cluster nodes, storage cells, and InfiniBand switches.

• InfiniBand switches are installed with firmware versions 2.0.4 or above.

• You have an understanding of InfiniBand partitioning.

2.10.19.2 Implementing InfiniBand Partitioning across Oracle VM RAC Clusters: Steps

To implement InfiniBand partitioning, you perform the following high-level steps:

1. On the InfiniBand switches, create a dedicated partition (cluster pkey) for every Oracle VM RAC cluster to be used by the clusterware.

2. On the InfiniBand switches, create one partition (storage pkey) to be used by all the Oracle VM RAC clusters and the storage cells for communication between the Oracle RAC cluster nodes and the storage cells.

3. Each partition requires a new IPoIB network interface. On the Oracle VM RAC nodes and on the storage cells, generate all the relevant network configuration files for the new IPoIB interfaces. (clib0,clib1,stib0,stib1 in the above diagram).

4. Modify the Oracle VM RAC cluster nodes and the Grid Infrastructure configuration of every Oracle VM RAC cluster to use the newly created IPoIB interfaces.

5. In the management domains (dom0s), modify the user domain configuration file for each user domain to use the partition key applicable to that user domain

6. Modify the storage cells to use the newly created IPoIB interfaces.

7. Restart the storage cells and the Oracle RAC clusters and make sure the Grid Infrastructure and the databases are functional. In this procedure, the Oracle RAC clusters incur a minimal downtime. This step is when the downtime occurs.

This section describes the steps in detail.

1. Allocate IP addresses to be used by the pkey interfaces.

   Plan and allocate sets of IP addresses and netmasks for each Oracle VM RAC cluster that will be used by the cluster pkey interfaces and the storage pkey interfaces when InfiniBand partitioning gets implemented in the cluster.

   Within an Oracle VM RAC cluster, cluster pkey IP address and netmask should be on a separate subnet from the storage pkey IP address and netmask.

   The tables below can be used as reference for one particular RAC cluster:

   Table 2-4 Existing Configuration

   Interface Name	IP Address	Netmask
   ib0	192.168.12.153	255.255.248.0
   ib1	192.168.12.154	255.255.248.0

   The following table shows the new IP addresses and netmasks required by the pkey interfaces while executing the steps of this document for that one Oracle RAC cluster.

   Table 2-5 New IP Addresses and Netmasks Required by the pkey Interfaces

   Interface Name	IP Address	Netmask
   clib0	192.168.112.1	255.255.248.0
   clib1	192.168.112.2	255.255.248.0
   stib0	192.168.114.1	255.255.240.0
   stib1	192.168.114.2	255.255.240.0

2. Create and configure the partition keys on the InfiniBand switches.

   1. Enable password-less ssh equivalence for the root user from any one of the dom0 nodes to all the switches on the InfiniBand fabric. This is the node that you will use for running all the scripts in this procedure. This node will be called the “driver dom0” in this procedure.

      You can do this by running the following dcli command from the dom0 node that will be used to run the scripts:

      # dcli –g ib_switch_list -l root –k
      

      ib_switch_list refers to a file that contains the list of all the InfiniBand switches on the fabric, each one on a separate line.

   2. From "Implementing InfiniBand Partitioning across OVM RAC clusters on Exadata (My Oracle Support Doc ID 2075398.1)", download and untar create_pkeys.tar to the driver dom0 node. When you untar the file, you should get three files:

      • create_pkeys_on_switch.sh

      • run_create_pkeys.sh

      • create_pkey_files.sh

   3. Run the script create_pkeys_on_switch.sh from driver dom0 to create and configure the partition keys on the InfiniBand switches.

      Note:

      One execution of create_pkeys_on_switch.sh creates exactly one partition. You must run the script once for each partition to be created. For example, an environment that contains two Oracle VM RAC clusters will have a total of three partitions: one storage partition and two cluster partitions (one per Oracle RAC cluster). In this example, you will need to run create_pkeys_on_switch.sh three times.

      The script needs to be executed from only one node (the driver dom0). It will create the partitions in all the switches provided as input during the execution of the script.

      Verify the partitions got created on all the switches after the script completes:

      # /usr/local/sbin/smpartition list active no-page
      

      At this stage ensure that you have created all the required partitions.

3. Configure the Oracle VM RAC cluster nodes for using partition keys.

   Note:

   This step makes the following changes on the Oracle VM RAC cluster nodes:

   • Modifies these files:

     • /etc/sysconfig/network-scripts/ifcfg-ib0

     • /etc/sysconfig/network-scripts/ifcfg-ib1

   • Removes these files:

     • /etc/sysconfig/network-scripts/rule-ib0

     • /etc/sysconfig/network-scripts/rule-ib1

     • /etc/sysconfig/network-scripts/route-ib0

     • /etc/sysconfig/network-scripts/route-ib1

   • Before modifying or removing the files above, the script backs them up in the /etc/sysconfig/network-scripts/backup-for-pkeys directory. If this step fails, restore all the files from /etc/sysconfig/network-scripts/backup-for-pkeys to /etc/sysconfig/network-scripts before rerunning this step.

   • Creates the following new files in /etc/sysconfig/network-scripts:

     • ifcfg-clib0

     • ifcfg-clib1

     • rule-clib0

     • rule-clib1

     • route-clib0

     • route-clib1

     • ifcfg-stib0

     • ifcfg-stib1

     • rule-stib0

     • rule-stib1

     • route-stib0

     • route-stib1

   If this step fails, and the files above have been created, then you need to remove them before rerunning this step.

   1. Make sure password-less ssh is set up from the driver dom0 node used in step 1 to all the Oracle RAC cluster nodes and the storage cells that need to be configured for partition keys.

   2. Make sure run_create_pkeys.sh and create_pkey_files.sh are executable and they are in the same directory on the driver dom0 node.

   3. Run run_create_pkeys.sh. The syntax for this script is:

      run_create_pkeys.sh <node_name> <interface_name> <pkey_id> <node_type> <pkey_ipaddr> <pkey_netmask> <pkey_interfaceType>
      

      <node_name> specifies the

      <interface_name> is either ib0 or ib1.

      <pkey_id> specifies the pkey without the “0x” prefix.

      <node_type> is either compute or cell.

      <pkey_ipaddr> specifies the IP address.

      <pkey_netmask> specifies the netmask in CIDR format, for example, /21.

      <pkey_interfaceType> is cluster or storage for “compute” node types, or storage for “cell” node types.

      Note:

      The pkey_ipaddr and pkey_netmask of the cluster pkey interface must be on a different subnet from the pkey_ipaddr and pkey_netmask of the storage pkey interface.

      For cluster nodes, you need to run the script a total of four times for every cluster node with a node type value of compute.

      <pkey_id> is the cluster partition key derived from the cluster pkey_id value entered in step 2.

      You can use the command below to derive the partition key values to be entered in the command (as shown above) from the pkey_id value entered in step 2.

      FinalHexValue=$(echo "obase=16;ibase=2;$(expr 1000000000000000 + $(echo "obase=2;ibase=16;$(echo $HexValue|tr [:lower:] [:upper:])"|bc))"|bc|tr [:upper:] [:lower:])

      FinalHexValue is the value that will be entered in the command here and HexValue is the value entered in step 2 for pkey_id.

      The following table describes the four runs:

      Table 2-6 Four Runs for Cluster Nodes

      Run	Interface Name	pkey_id	node_type	pkey_ipaddress	pkey_netmask	pkey_interfaceType
      1	ib0	Cluster pkey ID	compute	Cluster pkey IP address for ib0	Cluster pkey netmask for ib0	cluster
      2	ib1	Cluster pkey ID	compute	Cluster pkey IP address for ib1	Cluster pkey netmask for ib1	cluster
      3	ib0	Storage pkey ID	compute	Storage pkey IP address for ib0	Storage pkey netmask for ib0	storage
      4	ib1	Storage pkey ID	compute	Storage pkey IP address for ib1	Storage pkey netmask for ib1	storage

      Examples:

      # ./run_create_pkeys.sh vm-guest-1 ib0 a000 compute 192.168.12.153 /21 cluster
      
      # ./run_create_pkeys.sh vm-guest-1 ib1 a000 compute 192.168.12.154 /21 cluster
      
      # ./run_create_pkeys.sh vm-guest-1 ib0 aa00 compute 192.168.114.15 /20 storage
      
      # ./run_create_pkeys.sh vm-guest-1 ib1 aa00 compute 192.168.114.16 /20 storage
      

      At this stage all the required networking files listed below have been created for the new pkey-enabled network interfaces on the Oracle VM RAC cluster nodes:

      • /etc/sysconfig/network-scripts/ifcfg-clib0

      • /etc/sysconfig/network-scripts/ifcfg-clib1

      • /etc/sysconfig/network-scripts/ifcfg-stib0

      • /etc/sysconfig/network-scripts/ifcfg-stib1

      • /etc/sysconfig/network-scripts/rule-clib0

      • /etc/sysconfig/network-scripts/rule-clib1

      • /etc/sysconfig/network-scripts/rule-stib0

      • /etc/sysconfig/network-scripts/rule-stib1

      • /etc/sysconfig/network-scripts/route-clib0

      • /etc/sysconfig/network-scripts/route-clib1

      • /etc/sysconfig/network-scripts/route-stib0

      • /etc/sysconfig/network-scripts/route-stib1

      Oracle Grid Infrastructure has also been modified to make use of the new network interfaces upon restart. "$GRID_HOME/bin/oifcfg getif" should list clib0 and clib1 in the list of interfaces to be used for the cluster_interconnect.

4. Modify Oracle ASM and Oracle RAC CLUSTER_INTERCONNECTS parameter.

   1. Login to each of the Oracle ASM instances in the Oracle RAC cluster using SQL*Plus as SYS, and run the following command:

      alter system set cluster_interconnects='<cluster pkey IP address of ib0>:<cluster pkey IP address of ib1>'
        scope=spfile  sid='<name of the current ASM instance>';
      

      For example:

      alter system set cluster_interconnects='192.168.12.153:192.168.12.154'
        scope=spfile  sid='<name of the current ASM instance>';
      

   2. Login to each of the database instances in the Oracle RAC cluster using SQL*Plus, and run the following command:

      alter system set cluster_interconnects='<cluster pkey IP address of ib0>:<cluster pkey IP address of ib1>'
        scope=spfile  sid='<name of the current RDBMS instance>';
      

      For example:

      alter system set cluster_interconnects='192.168.12.153:192.168.12.154'
        scope=spfile  sid='<name of the current RDBMS instance>';
      

   3. Shut down and disable CRS auto-start on all the Oracle RAC cluster nodes.

      # $GRID_HOME/bin/crsctl stop crs
      
      # $GRID_HOME/bin/crsctl disable crs
      

   At this stage Oracle Grid Infrastructure, the Oracle ASM instances, and the Oracle Database instances have been modified to make use of the newly created network interfaces.

5. Modify cellip.ora and cellinit.ora on all the cluster nodes (user domains).

   1. Manually edit /etc/oracle/cell/network-config/cellip.ora on all the Oracle RAC cluster nodes to replace the existing IP address with the two storage pkey IP addresses of every storage cell that will be set up in step 6. Separate the two IP addresses with a “;”.

      Here is an example of a /etc/oracle/cell/network-config/cellip.ora file:

      cell="192.168.114.1;192.168.114.2"
      
      cell="192.168.114.3;192.168.114.4"
      
      cell="192.168.114.5;192.168.114.6"
      
      cell="192.168.114.7;192.168.114.8"
      
      cell="192.168.114.9;192.168.114.10"
      
      cell="192.168.114.11;192.168.114.12"
      
      cell="192.168.114.13;192.168.114.14"
      

      The steps below are recommended to achieve the objective of this step. You perform these steps on any one database server node of the cluster (user domain for an Oracle VM RAC cluster):

      1. Run the following commands:

         # cd /etc/oracle/cell/network-config
         # cp cellip.ora cellip.ora-bak
         

      2. Make the modifications to cellip.ora-bak.

      3. Make sure ssh equivalence is set up for the root user to all the cluster nodes from this cluster node.

      4. Run the following commands below. <cluster_nodes> refers to a file containing the names of all the Oracle RAC cluster nodes of the Oracle VM RAC cluster, each node on a separate line.

         # /usr/local/bin/dcli -g <cluster_nodes> –l root "/bin/cp /etc/oracle/cell/network-config/cellip.ora /etc/oracle/cell/network-config/cellip-orig.ora"
         
         # /usr/local/bin/dcli -g <cluster_nodes> –l root –f cellip.ora-bak –d /etc/oracle/cell/network-config/cellip.ora
         

   2. Back up /etc/oracle/cell/network-config/cellinit.ora to /etc/oracle/cell/network-config/cellinit.ora-bak.

   3. Manually edit /etc/oracle/cell/network-config/cellinit.ora to replace the existing IP addresses/netmask  with the two storage pkey IP addresses/netmask of the cluster node which was used in step 3. The IP address and netmask were used in the third and fourth run of step 3.

6. Modify all the relevant vm.cfg files in the dom0. This step is applicable only for Oracle VM environments.

   Login to all the dom0s and manually edit /EXAVMIMAGES/GuestImages/<user domain name>/vm.cfg to include the partition keys created in step 2. See example below.

   Modify the line:

   ib_pkeys = [{'pf':'40:00.0','port':'1','pkey':['0xffff',]},{'pf':'40:00.0','port':'2','pkey':['0xffff',]},]
   

   to:

   ib_pkeys = [{'pf':'40:00.0','port':'1','pkey':['0xa000','0xaa00',]},{'pf':'40:00.0','port':'2','pkey':['0xa000','0xaa00',]},]
   

   In the example above, 0xa000 is the cluster partition key derived from the cluster pkey_id value entered in step 2, and 0xaa00 is the storage partition key derived from the storage pkey_id value.

   You can use the command below to derive the partition key values to be entered in vm.cfg (as shown above) from the pkey_id values entered in step 2.

   FinalHexValue=$(echo "obase=16;ibase=2;$(expr 1000000000000000 + $(echo "obase=2;ibase=16;$(echo $HexValue|tr [:lower:] [:upper:])"|bc))"|bc|tr [:upper:] [:lower:])
   

   FinalHexValue is the value that will be entered in vm.cfg and HexValue is the value entered in step 2 for pkey_id.

   Note:

   If your environment has multiple Oracle VM RAC clusters, step 7 and step 8 need to be performed only once AFTER steps 3 through 6 have been executed for all the Oracle VM RAC clusters.

7. Configure the storage cells for using partition keys.

   1. Make sure run_create_pkeys.sh and create_pkey_files.sh have been made available from step 2 and that they are in the same directory on the same dom0 node used in step 2.

   2. Make sure password-less ssh is set up from the dom0 node used in step 2 to all the storage cells that need to be configured for partition keys.

   3. Execute run_create_pkeys.sh from that dom0 node. The syntax for this script for storage cells is:

      run_create_pkeys.sh <node_name> <interfaceName> <pkey_id> <node_type> <pkey_ipaddr> <pkey_netmask> <pkey_interfaceType> 
      

      <node_name> specifies the

      <interface_name> is either ib0 or ib1.

      <pkey_id> specifies the pkey without the “0x” prefix.

      <node_type> is either compute or cell.

      <pkey_ipaddr> specifies the IP address.

      <pkey_netmask> specifies the netmask in CIDR format, for example, /21.

      <pkey_interfaceType> is cluster or storage for “compute” node types, or storage for “cell” node types.

      Note:

      <pkey_id> in the command above is the storage partition key derived from the storage pkey_id value entered in step 2.

      The command below can be used to derive the partition key values to be entered in the command (as shown above) from the pkey_id value entered in step 2.

      FinalHexValue=$(echo "obase=16;ibase=2;$(expr 1000000000000000 + $(echo "obase=2;ibase=16;$(echo $HexValue|tr [:lower:] [:upper:])"|bc))"|bc|tr [:upper:] [:lower:])

      FinalHexValue is the value that will be entered in the command and HexValue is the value entered in step 2 for pkey_id.

      For storage cells, you need to run the script twice for every storage cell with a node_type of “cell”, as shown in the following table:

      Table 2-7 Two Runs for Storage Cells

      Run	Interface Name	pkey_id	node_type	pkey_ipaddress	pkey_netmask	pkey_interfaceType
      1	ib0	Storage pkey ID	cell	Storage pkey IP address for ib0	Storage pkey netmask for ib0	storage
      2	ib1	Storage pkey ID	cell	Storage pkey IP address for ib1	Storage pkey netmask for ib1	storage

      Examples:

      # ./run_create_pkeys.sh cell-1 ib0 aa00 cell 192.168.114.1 /20 storage
      
      # ./run_create_pkeys.sh cell-1 ib1 aa00 cell 192.168.114.2 /20 storage
      

      Note:

      You can ignore the following messages from the script. The reboot of the storage cells in step 9 will take care of these items:

      Network configuration altered. Please issue the following commands as root to restart the network and open IB stack:

      service openibd restart

      service network restart

      A restart of all services is required to put new network configuration into effect. MS-CELLSRV communication may be hampered until restart.

      At this stage the storage cells have been modified to use the new network interfaces upon reboot.

8. Modify /opt/oracle.cellos/cell.conf on the storage cells and reboot the storage cells.

   1. Back up /opt/oracle.cellos/cell.conf.

      # cp /opt/oracle.cellos/cell.conf /opt/oracle.cellos/cell.conf-prepkey
      

   2. Change the following lines in /opt/oracle.cellos/cell.conf.

      Change this line:

      <Pkeyconfigured>no</Pkeyconfigured>
      

      to:

      <Pkeyconfigured>yes</Pkeyconfigured>
      

      Change this line for the 2 private interfaces ib0 and ib1:

      <IP_enabled>yes</IP_enabled>
      

      to:

      <IP_enabled>no</IP_enabled>
      

   3. Make sure Grid Infrastructure is stopped on all Oracle VM RAC nodes.

   4. Reboot all the storage cell servers.

   5. After the storage cells come back up, check the new pkey-enabled network interfaces are in use:

      # cellcli -e list cell detail | egrep 'interconnect|ipaddress'
      

      This should show the new pkey-enabled interfaces (stib0 and stib1) along with the new set of IP addresses.

9. Restart the cluster nodes.

   1. Login to the corresponding dom0 of each of the user domain nodes.

   2. Run the following commands:

      # xm shutdown <user domain name>
      
      # xm create /EXAVMIMAGES/GuestImages/<user domain name>/vm.cfg
      

10. Start and verify the Oracle Grid Infrastructure stack is fully up on all the cluster nodes.

    1. Start and enable auto-start of the Grid Infrastructure stack on all the Oracle RAC cluster nodes

       # $GRID_HOME/bin/crsctl start crs
       
       # $GRID_HOME/bin/crsctl enable crs
       

    2. Once Oracle Grid Infrastructure is started up on all the nodes, verify the cluster_interconnects parameter is set to use the newly configured pkey interfaces:

       SQL> select inst_id, value from gv$parameter where name = 'cluster_interconnects'
       

    3. Remove the old cluster interconnect interfaces from OCR.

       # $GRID_HOME/bin/oifcfg delif –global ib0/<old subnet>
       
       # $GRID_HOME/bin/oifcfg delif –global ib1/<old subnet>
       

2.10.19.3 Implementing InfiniBand Partitioning across OVM RAC Clusters: Setting up Limited Membership

The 12.1.0.2 October 2016 Database Bundle Patch introduces a security enhancement feature where the GUIDs of the database nodes can be assigned to the storage pkey with limited membership instead of full membership, as was the case prior to the 12.1.0.2 October 2016 Bundle Patch. This addresses a security concern where one RAC node from one RAC cluster could talk to a RAC node from another RAC cluster using the storage pkey interfaces.

Full Membership and Limited Membership

An InfiniBand partition defines a group of InfiniBand nodes that are allowed to communicate with one another. With InfiniBand partitioning, you define custom or unique partition keys that are managed by the master subnet manager, and assign members to the custom partition keys. Members with the same partition key can only communicate amongst themselves. A member of one partition key cannot communicate with a member that has a different partition key, regardless of membership type. The OVM RAC cluster nodes of one cluster are assigned one partition key for clusterware communication and another partition key for communication with storage cells. This way, the nodes of one RAC cluster will not be able to communicate with the nodes of another RAC cluster, which have a different partition key assigned to them. This is very similar conceptually to tagged VLANs in the Ethernet world.

Partition keys (pkeys) are 15-bit integers and have a value of 0x1 to 0x7FFF. An additional bit, the membership bit, identifies the membership of a member of the partition. Memberships can be:

• Full: The membership bit is set to 1. Members with full membership can communicate with each other as well as members with limited membership within same the partition key.

• Limited: The membership bit is set to 0. Members with limited membership within a partition cannot communicate with each other. However they can communicate with other members with full membership within the same partition.

Combined together, the pkey and the membership bit comprise a 16-bit integer. The most significant bit is the membership bit.

By default, the InfiniBand subnet manager provides a single partition and it is identified by the partition key 0x7FFF (limited membership) or 0xFFFF (full membership).

An HCA port can participate in a maximum of 128 partitions. Each partition key provides a new IPoIB network interface. For example, InfiniBand port 1 with partition key 0xa001 will result in a new network interface. These interfaces are named with meaningful names through the ifcfg-<interface> file parameters.

An InfiniBand node can be a member of multiple partitions. When a packet arrives at a database node, the partition key (pkey) of the packet is matched with the Subnet Manager configuration. This validation prevents a database node from communicating with another database node outside of the partitions of which it is a member.

Every node within the infiniBand fabric has a partition key table which you can see in /sys/class/infiniband/mlx4_0/ports/[1-2]/pkeys. Every Queue Pair (QP) of the node has an index (pkey) associated with it that maps to an entry in that table. Whenever a packet is sent from the QP’s send queue, the indexed pkey is attached with it. Whenever a packet is received on the QP’s receive queue, the indexed pkey is compared with that of the incoming packet. If it does not match, the packet is silently discarded. The receiving Channel Adapter does not know it arrived and the sending Channel Adapter gets no acknowledgement as well that it was received. The sent packet simply gets manifested as a lost packet. It is only when the pkey of the incoming packet matches the indexed pkey of the QP’s receive queue, a handshake is made and the packet is accepted and an acknowledgment is sent to the sending channel adapter. This is how only members of the same partition are able to communicate with each other and not with hosts that are not members of that partition (which means those hosts that does not have that pkey in their partition table).

The steps below describe how to set up this enhancement on a pkey-enabled environment that has the 12.1.0.2 October 2016 Database Bundle Patch applied. There are two possible scenarios, as described below:

Case 1. Implementing the feature on a pkey-enabled environment in a rolling manner

In this case, you have already applied the 12.1.0.2 October 2016 Database Bundle Patch.

Perform the steps below on one node at a time.

1. Shut down the Grid Infrastructure on the node.

   # $GI_HOME/bin/crsctl stop crs
   

2. Determine the two port GUIDs of the dom0 (control domain) which manages this user domain OVM RAC cluster node.

   # /usr/sbin/ibstat | grep Port
   

3. Login to the Infiniband Switch where the SM master is running as root.

4. Run the commands below on the InfiniBand switch.

   # /usr/local/sbin/smpartition start
   
   # /usr/local/sbin/smpartition modify -n <storage pkey name> -port <Port GUID1 of the dom0 from step 2> -m limited
   
   # /usr/local/sbin/smpartition modify -n <storage pkey name> -port <Port GUID2 of the dom0 from step 2> -m limited
   
   # /usr/local/sbin/smpartition commit
   

5. Modify the vm.cfg file for this OVM RAC user domain node in the dom0.

   1. Login to the dom0 as root.

   2. Edit /EXAVMIMAGES/GuestImages/<user domain name>/vm.cfg and modify the partition keys as shown in the example below.

      Modify this line:

      ib_pkeys = [{'pf':'40:00.0','port':'1','pkey':[ '0xclpkey','0x<stpkey>',]},{'pf':'40:00.0','port':'2','pkey':[ '0xclpkey','0x<stpkey>',]},]

      to this:

      ib_pkeys = [{'pf':'40:00.0','port':'1','pkey':[ '0xclpkey','0x<mod_stpkey>',]},{'pf':'40:00.0','port':'2','pkey':[ '0xclpkey','0x<mod_stpkey>',]},]

      <mod_stpkey> is derived from <stpkey> using the formula below:

      mod_stpkey=$(echo "obase=16;ibase=2;$(expr $(echo "obase=2;ibase=16;$(echo $stpkey|tr [:lower:] [:upper:])"|bc) - 1000000000000000)"|bc|tr [:upper:] [:lower:])

      Note that <stpkey> and <mod_stpkey> in the formula above are specified without the "0x" prefix.

6. Modify the /etc/sysconfig/network-scripts/ifcfg-stib* files on the user domain RAC nodes.

   Edit the PKEY_ID in those files using the formula below:

   mod_stpkey=$(echo "obase=16;ibase=2;$(expr $(echo "obase=2;ibase=16;$(echo $stpkey|tr [:lower:] [:upper:])"|bc) - 1000000000000000)"|bc|tr [:upper:] [:lower:])

   mod_stpkey is the new PKEY_ID, and stpkey is the old PKEY_ID.

   Note that <stpkey> and <mod_stpkey> in the formula above are specified without the "0x" prefix.

7. Modify /opt/oracle.cellos/pkey.conf on the user domain RAC nodes.

   Edit the Pkey for the storage network pkey interfaces (stib*):

   Change:

   <Pkey>0xstpkey</Pkey>

   to:

   <Pkey>0xmod_stpkey</Pkey>

   mod_stpkey is derived from stpkey using the formula below:

   mod_stpkey=$(echo "obase=16;ibase=2;$(expr $(echo "obase=2;ibase=16;$(echo $stpkey|tr [:lower:] [:upper:])"|bc) - 1000000000000000)"|bc|tr [:upper:] [:lower:])

   stpkey and mod_stpkey used in the formula above are specified without the "0x" prefix.

8. Restart the OVM RAC user domain node.

   1. Login to the dom0 as root.

   2. Run the following commands:

      # xm shutdown <user domain name>
      
      # xm create /EXAVMIMAGES/GuestImages/<user domain name>/vm.cfg
      

9. Verify the Grid Infrastructure stack is fully up on the cluster node.

10. Repeat the steps on the remaining cluster nodes, one node at a time.

Case 2. Implementing the feature on a pkey-enabled environment while you apply the 12.1.0.2 October 2016 Database Bundle Patch in a rolling manner

Perform the steps below on one node at a time.

1. Apply the 12.1.0.2 October 2016 Database Bundle Patch on the cluster node.

2. Run the steps 1 through 10 from Case 1 above on the node where the patch was applied.

3. Move on to the next cluster node and repeat steps 1 and 2 above.

Note:

Once the dom0 GUIDs are converted to limited membership, deployment of any new cluster will have the October 2016 Database Bundle Patch as a prerequisite.

2.10.20 Running Oracle EXAchk

Oracle EXAchk version 12.1.0.2.2 and higher supports virtualization on Exadata.

To perform the complete set of Oracle EXAchk audit checks in an Oracle Exadata Oracle VM environment, Oracle EXAchk must be installed in and run from multiple locations, as follows:

• From one management domain (dom0)

• From one user domain (domU) in each Oracle VM Oracle RAC cluster

For example, an Oracle Exadata Database Machine Quarter Rack with 2 database servers containing 4 Oracle VM Oracle RAC clusters (2 nodes per cluster for a total of 8 domU's across both database servers) requires running Oracle EXAchk 5 separate times, as follows:

1. Run Oracle EXAchk in the first user domain (domU) for cluster #1.

2. Run Oracle EXAchk in the first user domain (domU) for cluster #2.

3. Run Oracle EXAchk in the first user domain (domU) for cluster #3.

4. Run Oracle EXAchk in the first user domain (domU) for cluster #4.

5. Run Oracle EXAchk in the first management domain (dom0).

The audit checks performed by Oracle EXAchk are specified in the following table:

Table 2-8 Audit Checks Performed by Oracle EXAchk

Where to Install and Run Oracle EXAchk	Audit Checks Performed
Management domain (dom0)	Hardware and operating system level checks for: Database servers (management domains) Storage servers InfiniBand fabric InfiniBand switches
User domain (domU)	Operating system level checks for user domains, and checks for Oracle Grid Infrastructure and Oracle Database

Oracle EXAchk Command Line Options

Oracle EXAchk requires no special command line options. It automatically detects that it is running in an Oracle Exadata Oracle VM environment and whether it is running in a management domain or user domain and performs the applicable audit checks. For example, in the simplest case, you can run Oracle EXAchk with no command line options:

./exachk

When Oracle EXAchk is run in the management domain, it performs audit checks on all database servers, storage servers, and InfiniBand switches accessible through the InfiniBand network.

To run Oracle EXAchk on a subset of servers or switches, use the following command line options:

Table 2-9 Command Line Options for Oracle EXAchk

Option	Description
-clusternodes	Specifies a comma-separated list of database servers.
-cells	Specifies a comma-separated list of storage servers.
-ibswitches	Specifies a comma-separated list of InfiniBand switches.

For example, for an Oracle Exadata Database Machine Full Rack where only the first Quarter Rack is configured for virtualization, but all components are accessible through the InfiniBand network, you can run a command similar to the following from the database server dm01adm01:

./exachk -clusternodes dm01adm01,dm01adm02
         -cells dm01celadm01,dm01celadm02,dm01celadm03
         -ibswitches dm01swibs0,dm01sw-iba0,dm01sw-ibb0

2.11 Extending LVM Partitions

This section provides information about Logical Volume Manager (LVM) partitions in the database servers. LVM provides flexibility to reorganize the partitions.

Note:

Keep at least 1 GB of free space in the VGExaDb volume group to be used for the LVM snapshot created by the dbnodeupdate.sh utility during software maintenance. If you make snapshot-based backups of the / (root) and /u01 directories by following the steps in "Creating a Snapshot-Based Backup of Oracle Linux Database Server", then keep at least 6 GB of free space in the VGExaDb volume group.

This section contains the following topics:

• Extending the root LVM Partition

• Resizing a Non-root LVM Partition

• Extending the Swap Partition

2.11.1 Extending the root LVM Partition

The procedure for extending the root LVM partition depends on the Oracle Exadata Storage Server Software release. The following sections contain the procedures, based on release:

• Extending the root LVM Partition on Systems Running Oracle Exadata Storage Server Software Release 11.2.3.2.1 or Later

• Extending the root LVM Partition on Systems Running Oracle Exadata Storage Server Software Earlier than Release 11.2.3.2.1

2.11.1.1 Extending the root LVM Partition on Systems Running Oracle Exadata Storage Server Software Release 11.2.3.2.1 or Later

The following procedure describes how to extend the size of the root (/) partition on systems running Oracle Exadata Storage Server Software release 11.2.3.2.1 or later:

Note:

This procedure does not require an outage on the server.

For dom0 systems, active and inactive Sys LVM's are LVDbSys2 and LVDbSys3 instead of LVDbSys1 and LVDbSys2.

Make sure that LVDbSys1 and LVDbSys2 are sized the same.

1. Collect information about the current environment as follows:

   1. Use the df command to identify the amount of free and used space in the root partition (/) as follows:

      # df -h /
       
      

      The following is an example of the output from the command:

      Filesystem            Size  Used Avail Use% Mounted on
      /dev/mapper/VGExaDb-LVDbSys1
                             30G  22G  6.2G  79% / 
      

      Note:

      The active root partition may be either LVDbSys1 or LVDbSys2, depending on previous maintenance activities.

   2. Use the lvs command to display the current volume configuration.

      # lvs -o lv_name,lv_path,vg_name,lv_size
      

      The following is an example of the output from the command:

      LV                 Path                            VG       LSize
      LVDbOra1           /dev/VGExaDb/LVDbOra1           VGExaDb  100.00g
      LVDbSwap1          /dev/VGExaDb/LVDbSwap1          VGExaDb  24.00g
      LVDbSys1           /dev/VGExaDb/LVDbSys1           VGExaDb  30.00g
      LVDbSys2           /dev/VGExaDb/LVDbSys2           VGExaDb  30.00g
      LVDoNotRemoveOrUse /dev/VGExaDb/LVDoNotRemoveOrUse VGExaDb  1.00g
      
      

2. Use the tune2fs command to check the online resize option.

   tune2fs -l /dev/mapper/vg_name-lv_name | grep resize_inode
   
   

   For example:

   tune2fs -l /dev/mapper/VGExaDb-LVDbSys1 | grep resize_inode
   
   

   The resize_inode option should be listed in the output from the command. If the option is not listed, then the file system must be unmounted before resizing the partition. Refer to "Extending the root LVM Partition on Systems Running Oracle Exadata Storage Server Software Earlier than Release 11.2.3.2.1" to resize the partition.

3. Verify there is available space in the volume group VGExaDb using the vgdisplay command as follows:

   # vgdisplay -s
   

   The following is an example of the output from the command:

   "VGExaDb" 834.89 GB [184.00 GB used / 650.89 GB free]
   

   The volume group must contain enough free space to increase the size of both system partitions, and maintain at least 1 GB of free space for the LVM snapshot created by the dbnodeupdate.sh utility during upgrade.

   If there is not enough free space, then verify that the reclaimdisks.sh utility has been run. If the utility has not been run, then use the following command to reclaim disk space:

   # /opt/oracle.SupportTools/reclaimdisks.sh -free -reclaim 
   

   If the utility has been run and there is not enough free space, then the LVM cannot be resized.

   Note:

   reclaimdisks.sh cannot run at the same time as a RAID rebuild (that is, a disk replacement or expansion). Wait until the RAID rebuild is complete, then run reclaimdisks.sh.

4. Resize both LVDbSys1 and LVDbSys2 logical volumes using the lvextend command as follows:

   # lvextend -L +XG --verbose /dev/VGExaDb/LVDbSys1
   # lvextend -L +XG --verbose /dev/VGExaDb/LVDbSys2
   

   In the preceding command, XG is the amount of GB the logical volume will be extended. The amount of space added to each system partition must be the same.

   The following example extends the logical volumes by 10 GB:

   # lvextend -L +10G /dev/VGExaDb/LVDbSys1
   # lvextend -L +10G /dev/VGExaDb/LVDbSys2
   

5. Resize the file system within the logical volume using the resize2fs command as follows:

   # resize2fs /dev/VGExaDb/LVDbSys1
   # resize2fs /dev/VGExaDb/LVDbSys2
   

6. Verify the space was extended for the active system partition using the df command as follows:

   # df -h /
   

2.11.1.2 Extending the root LVM Partition on Systems Running Oracle Exadata Storage Server Software Earlier than Release 11.2.3.2.1

The following procedure describes how to extend the size of the root (/) partition on systems running Oracle Exadata Storage Server Software earlier than release 11.2.3.2.1:

Note:

This procedure requires the system to be offline and restarted.

Keep at least 1 GB of free space in the VGExaDb volume group to be used for the LVM snapshot created by the dbnodeupdate.sh utility during software maintenance. If you make snapshot-based backups of the / (root) and /u01 directories by following the steps in "Creating a Snapshot-Based Backup of Oracle Linux Database Server", then keep at least 6 GB of free space in the VGExaDb volume group.

For dom0 systems, active and inactive Sys LVM's are LVDbSys2 and LVDbSys3 instead of LVDbSys1 and LVDbSys2.

Make sure LVDbSys1 and LVDbSys2 are sized the same.

1. Collect information about the current environment as follows:

   1. Use the df command to identify the mount points for the root partition (/) and the non-root partition (/u01), and their respective LVMs. The following is an example of the output from the command:

      # df
      Filesystem                    1K-blocks   Used    Available Use% Mounted on
      /dev/mapper/VGExaDb-LVDbSys1 30963708   21867152   7523692  75%    /
      /dev/sda1                      126427      16355    103648  14%    /boot
      /dev/mapper/VGExaDb-LVDbOra1 103212320  67404336  30565104  69%    /u01
      tmpfs                         84132864   3294608  80838256   4%    /dev/shm
      

      The file system name in the df command output is in the following format:

      /dev/mapper/VolumeGroup-LogicalVolune
      

      The full logical volume name of the root file system in the preceding example is /dev/VGExaDb/LVDbSys1.

   2. Use the lvscan command to display logical volumes as follows:

      #lvm lvscan
      
      ACTIVE            '/dev/VGExaDb/LVDbSys1'  [30.00 GB]  inherit
      ACTIVE            '/dev/VGExaDb/LVDbSwap1' [24.00 GB]  inherit
      ACTIVE            '/dev/VGExaDb/LVDbOra1'  [100.00 GB] inherit
      

   3. Use the lvdisplay command to display the current logical volume and the volume group configuration.

      #lvm lvdisplay /dev/VGExaDb/LVDbSys1
      
      --- Logical volume ---
      LV Name               /dev/VGExaDb/LVDbSys1
      VG Name               VGExaDb
      LV UUID               GScpD7-lKa8-gLg9-oBo2-uWaM-ZZ4W-Keazih
      LV Write Access       read/write
      LV Status             available
      # open                1
      LV Size               30.00 GB
      Current LE            7680
      Segments              1
      Allocation            inherit
      Read ahead sectors    auto
      - currently set to    256
      Block device          253:0
      

   4. Verify there is available space in the volume group VGExaDb so the logical volume can be extended. If the command shows there is zero free space, then neither the logical volume or the file system can be extended.

      # lvm vgdisplay VGExaDb -s
      "VGExaDb" 556.80 GB [154.00 GB used / 402.80 GB free]
      

2. Restart the system in diagnostic mode as follows:

   1. Copy the /opt/oracle.SupportTools/diagnostics.iso file to a directory on the machine using the ILOM interface.

   2. Log in to the ILOM web interface.

   3. Select the Remote Control tab.

   4. Select the Redirection tab.

   5. Click Launch Remote Console. The ILOM Remote Console window is displayed.

   6. Select the Devices menu in the ILOM Remote Console window.

   7. Click CD-ROM image.

   8. Navigate to the location of the diagnostics.iso file on the local machine in the File Open dialog box.

   9. Select the diagnostics.iso file.

   10. Click Open. A message similar to the following will appear on the console.

       
       Description of the illustration GUID-CEED053C-285D-49D0-A5F6-367AC90089DA-default.png

   11. Select Host Control from the Remote Control tab.

   12. Select CDROM as the next boot device from the list of values.

   13. Click Save. When the system is booted, the diagnostics.iso image is used. The system reverts to the default after the next restart.

   14. Log in as the root user in the ILOM Remote Console window.

   15. Restart the server using the following command:

       # shutdown -r now
       

       The system starts using the diagnostics.iso image.

3. Enter e to enter the diagnostic shell as follows:

   Choose from following by typing letter in '()':
   (e)nter interactive diagnostics shell. Must use credentials from Oracle
   support to login (reboot or power cycle to exit the shell),
   (r)estore system from NFS backup archive,
   Select:e
   

4. Log in to the system as the root user. You will be prompted for the password.

   localhost login: root
   Password: *********
   -sh-3.1# 
   

5. Unmount the root file system using the following command.

   # cd /
   # umount /mnt/cell
   

6. Verify the logical volume name using the following command:

   # lvm lvscan
   ACTIVE '/dev/VGExaDb/LVDbSys1' [30.00 GB] inherit
   ACTIVE '/dev/VGExaDb/LVDbSwap1' [24.00 GB] inherit
   ACTIVE '/dev/VGExaDb/LVDbOra1' [100.00 GB] inherit
   

7. Resize the LVDbSys1 and LVDbSys2 holding the current and backup root file system using the following commands:

   # lvm lvextend -L+XG --verbose /dev/VGExaDb/LVDbSys1
   # lvm lvextend -L+XG --verbose /dev/VGExaDb/LVDbSys2
   

   In the preceding commands, XG is the amount of GB the logical volume will be extended. For example, if the logical volume is expanded 5 GB, then the commands would be:

   # lvm lvextend -L+5G --verbose /dev/VGExaDb/LVDbSys1
   # lvm lvextend -L+5G --verbose /dev/VGExaDb/LVDbSys2
   

8. Verify the file system is valid using e2fsck.

   # e2fsck -f /dev/VGExaDb/LVDbSys1
   # e2fsck -f /dev/VGExaDb/LVDbSys2
   
   

9. Resize the file system using the following command:

   # resize2fs -p /dev/VGExaDb/LVDbSys1
   # resize2fs -p /dev/VGExaDb/LVDbSys2
   

10. Restart the system in normal mode as follows:

    # reboot
    

11. Log in to the system.

12. Verify the root file system mount mounts without issues with the new size.

2.11.2 Resizing a Non-root LVM Partition

The procedure for resizing a non-root LVM partition depends on the Oracle Exadata Storage Server Software release. The following sections contain the procedures, based on release:

• Extending a Non-root LVM Partition on Systems Running Oracle Exadata Storage Server Software Release 11.2.3.2.1 or Later

• Extending a Non-root LVM Partition on Systems Running Oracle Exadata Storage Server Software Earlier than Release 11.2.3.2.1

• Reducing a Non-root LVM Partition on Systems Running Oracle Exadata Storage Server Software Release 11.2.3.2.1 or Later

2.11.2.1 Extending a Non-root LVM Partition on Systems Running Oracle Exadata Storage Server Software Release 11.2.3.2.1 or Later

This procedure describes how to extend the size of a non-root (/u01) partition on systems running Oracle Exadata Storage Server Software release 11.2.3.2.1 or later.

This procedure does not require an outage on the server.

1. Collect information about the current environment.
   1. Use the df command to identify the amount of free and used space in the /u01 partition.

      # df -h /u01
      

      The following is an example of the output from the command:

      Filesystem            Size  Used Avail Use% Mounted on
      /dev/mapper/VGExaDb-LVDbOra1
                            99G   25G  70G   26% /u01
      

   2. Use the lvs command to display the current logical volume configuration used by the /u01 file system.

      # lvs -o lv_name,lv_path,vg_name,lv_size
      

      The following is an example of the output from the command:

       LV        Path                   VG      LSize
       LVDbOra1  /dev/VGExaDb/LVDbOra1  VGExaDb 100.00G
       LVDbSwap1 /dev/VGExaDb/LVDbSwap1 VGExaDb  24.00G
       LVDbSys1  /dev/VGExaDb/LVDbSys1  VGExaDb  30.00G
       LVDbSys2  /dev/VGExaDb/LVDbSys2  VGExaDb  30.00G
      

2. Use the tune2fs command to check the online resize option.

   tune2fs -l /dev/mapper/vg_name | grep resize_inode
   

   The resize_inode option should be listed in the output from the command. If the option is not listed, then the file system must be unmounted before resizing the partition. Refer to "Extending a Non-root LVM Partition on Systems Running Oracle Exadata Storage Server Software Earlier than Release 11.2.3.2.1" to resize the partition.

3. Verify there is available space in the volume group VGExaDb using the vgdisplay command.

   # vgdisplay -s
   

   The following is an example of the output from the command:

   "VGExaDb" 834.89 GB [184.00 GB used / 650.89 GB free]
   

   If the output shows there is less than 1 GB of free space, then neither the logical volume nor file system should be extended. Maintain at least 1 GB of free space in the VGExaDb volume group for the LVM snapshot created by the dbnodeupdate.sh utility during an upgrade.

   If there is not enough free space, then verify that the reclaimdisks.sh utility has been run. If the utility has not been run, then use the following command to reclaim disk space:

   # /opt/oracle.SupportTools/reclaimdisks.sh -free -reclaim 
   

   If the utility has been run and there is not enough free space, then the LVM cannot be resized.

   Note:

   • reclaimdisks.sh cannot run at the same time as a RAID rebuild (that is, a disk replacement or expansion). Wait until the RAID rebuild is complete, then run reclaimdisks.sh.

4. Resize the logical volume using the lvextend command.

   # lvextend -L +sizeG /dev/VGExaDb/LVDbOra1
   

   In the preceding command, size is the amount of space to be added to the logical volume.

   The following example extends the logical volume by 10 GB:

   # lvextend -L +10G /dev/VGExaDb/LVDbOra1
   

5. Resize the file system within the logical volume using the resize2fs command.

   # resize2fs /dev/VGExaDb/LVDbOra1
   

6. Verify the space was extended using the df command

   # df -h /u01
   

2.11.2.2 Extending a Non-root LVM Partition on Systems Running Oracle Exadata Storage Server Software Earlier than Release 11.2.3.2.1

This procedure describes how to extend the size of a non-root (/u01) partition on systems running Oracle Exadata Storage Server Software earlier than release 11.2.3.2.1.

In this procedure, /dev/VGExaDb/LVDbOra1 is mounted at /u01.

Note:

Keep at least 1 GB of free space in the VGExaDb volume group to be used for the LVM snapshot created by the dbnodeupdate.sh utility during software maintenance. If you make snapshot-based backups of the / (root) and /u01 directories by following the steps in "Creating a Snapshot-Based Backup of Oracle Linux Database Server", then keep at least 6 GB of free space in the VGExaDb volume group.

1. Collect information about the current environment.
   1. Use the df command to identify the mount points for the root partition (/) and the non-root partition (/u01), and their respective LVMs.

      # df
      Filesystem                    1K-blocks   Used    Available Use% Mounted on
      /dev/mapper/VGExaDb-LVDbSys1 30963708   21867152   7523692  75%    /
      /dev/sda1                      126427      16355    103648  14%    /boot
      /dev/mapper/VGExaDb-LVDbOra1 103212320  67404336  30565104  69%    /u01
      tmpfs                         84132864   3294608  80838256   4%    /dev/shm
      

   2. Use the lvm lvscan command to display logical volumes.

      ACTIVE            '/dev/VGExaDb/LVDbSys1'  [30.00 GB]  inherit
      ACTIVE            '/dev/VGExaDb/LVDbSwap1' [24.00 GB]  inherit
      ACTIVE            '/dev/VGExaDb/LVDbOra1'  [100.00 GB] inherit
      

   3. Use the lvdisplay command to display the current volume group configuration.

      # lvdisplay /dev/VGExaDb/LVDbOra1
      
      --- Logical volume ---
      LV Name               /dev/VGExaDb/LVDbOra1
      VG Name               VGExaDb
      LV UUID               vzoIE6-uZrX-10Du-UD78-314Y-WXmz-f7SXyY
      LV Write Access       read/write
      LV Status             available
      # open                1
      LV Size               100.00 GB
      Current LE            25600
      Segments              1
      Allocation            inherit
      Read ahead sectors    auto
      - currently set to    256
      Block device          253:2
      

   4. Verify there is available space in the volume group VGExaDb so the logical drive can be extended.

      If the command shows there is zero free space, then neither the logical volume or file system can be extended.

      # lvm vgdisplay VGExaDb -s
      
      "VGExaDb" 556.80 GB [154.00 GB used / 402.80 GB free]
      

2. Shut down any software that uses /u01.

   The following software typically uses /u01:

   • Oracle Clusterware and Oracle Database

     # Grid_home/bin/crsctl stop crs
     

   • Trace File Analyzer

     # Grid_home/bin/tfactl stop
     

   • OS Watcher (releases earlier than release 11.2.3.3.0)

     # /opt/oracle.oswatcher/osw/stopOSW.sh
     

   • ExaWatcher (release 11.2.3.3.0 and later)

     # /opt/oracle.ExaWatcher/ExaWatcher.sh --stop
     

   • Oracle Enterprise Manager agent

     (oracle)$ agent_home/bin/emctl stop agent
     

3. Unmount the partition as the root user.

   # umount /u01
   

   Note:

   If the umount command reports that the file system is busy, then use the fuser(1) command to identify processes still accessing the file system that must be stopped before the umount command will succeed.

   # umount /u01
   umount: /u01: device is busy
   umount: /u01: device is busy
    
   # fuser -mv /u01
    
           USER      PID ACCESS COMMAND
   /u01:   root     6788 ..c..  ssh
           root     8422 ..c..  bash
           root    11444 ..c..  su
           oracle  11445 ..c..  bash
           oracle  11816 ....m  mgr
           root    16451 ..c..  bash
   

4. Verify the file system.

   # e2fsck -f /dev/VGExaDb/LVDbOra1
   

5. Extend the partition.

   In this example, the logical volume is expanded to 80% of the physical volume size. At the same time, the file system is resized with the command.

   # lvextend -L+XG --verbose /dev/VGExaDb/LVDbOra1
   

   In the preceding command, XG is the amount of GB the logical volume will be extended. The following example shows how to extend the logical volume by an additional 200 GB:

   # lvextend -L+200G --verbose /dev/VGExaDb/LVDbOra1
   

   Caution:

   Use extreme caution when reducing the size. The new size must be large enough to hold all the original content of the partition. To reduce the size, use a command similar to the following:

   lvreduce -L60G --resizefs --verbose /dev/VGExaDb/LVDbOra1
   

   In the preceding command, the size of /u01 is reduced to 60 GB.

6. Check the /u01 file system using the e2fsck command.

   # e2fsck -f /dev/VGExaDb/LVDbOra1
   

7. Resize the /u01 file system.

   # resize2fs -p /dev/VGExaDb/LVDbOra1
   

8. Mount the partition.

   # mount -t ext3 /dev/VGExaDb/LVDbOra1 /u01
   

9. Verify the space was extended.

   $ df -h /u01
   

10. Restart any software that was stopped in step 2.

    • Oracle Clusterware and Oracle Database

      # Grid_home/bin/crsctl start crs
      

    • Trace File Analyzer

      # Grid_home/bin/tfactl start
      

    • OS Watcher (releases earlier than release 11.2.3.3.0)

      # /opt/oracle.cellos/vldrun -script oswatcher
      

    • ExaWatcher (release 11.2.3.3.0 and later)

      # /opt/oracle.ExaWatcher/ExaWatcher.sh
      

    • Oracle Enterprise Manager agent

      (oracle)$ agent_home/bin/emctl start agent
      

2.11.2.3 Reducing a Non-root LVM Partition on Systems Running Oracle Exadata Storage Server Software Release 11.2.3.2.1 or Later

The following procedure describes how to reduce the size of a non-root (/u01) partition on systems running Oracle Exadata Storage Server Software release 11.2.3.2.1 or later.

Note:

This procedure does not require an outage on the server. It is recommended that you back up your file system before performing the procedure.

1. Use the df command to determine the amount of free and used space in the /u01 partition:

   # df -h /u01
   

   Example output:

   Filesystem                    Size  Used Avail Use% Mounted on
   /dev/mapper/VGExaDb-LVDbOra1  193G   25G  159G  14% /u01
   

2. Use the lvm command to display the current logical volume configuration used by the /u01 file system.

   # lvm vgdisplay VGExaDb -s
     "VGExaDb" 271.82 GB [250.04 GB used / 21.79 GB free]
   
   # lvm lvscan
     ACTIVE            '/dev/VGExaDb/LVDbSys1' [30.00 GB] inherit
     ACTIVE            '/dev/VGExaDb/LVDbSwap1' [24.00 GB] inherit
     ACTIVE            '/dev/VGExaDb/LVDbOra1' [196.04 GB] inherit
   

   In this case, the size of the LVDbOra1 partition needs to be reduced such that LVDbSys2 (30.00 GB in size) can be created by the dbserver_backup.sh script.

3. Shut down any software that uses /u01. The following software typically uses /u01:

   • Oracle Clusterware and Oracle Database

     # GI_HOME/bin/crsctl stop crs
     

   • Trace File Analyzer

     # GI_HOME/bin/tfactl stop
     

   • OS Watcher (releases earlier than 11.2.3.3.0)

     # /opt/oracle.oswatcher/osw/stopOSW.sh
     

   • ExaWatcher (release 11.2.3.3.0 and later)

     # /opt/oracle.ExaWatcher/ExaWatcher.sh --stop
     

   • Oracle Enterprise Manager agent

     (oracle)$ agent_home/bin/emctl stop agent
     

4. Unmount the partition as the root user:

   # umount /u01
   

   Note:

   If the umount command reports that the file system is busy, then use the fuser(1) command to identify the processes still accessing the file system that must be stopped before the umount command will succeed.

   # umount /u01
   umount: /u01: device is busy
   umount: /u01: device is busy
   
   # fuser -mv /u01
   
           USER      PID ACCESS COMMAND
   /u01:   root     6788 ..c..  ssh
           root     8422 ..c..  bash
           root    11444 ..c..  su
           oracle  11445 ..c..  bash
           oracle  11816 ....m  mgr
           root    16451 ..c..  bash
   

5. Verify the file system:

   # e2fsck -f /dev/VGExaDb/LVDbOra1
   
   fsck 1.39 (29-May-2006)
   e2fsck 1.39 (29-May-2006)
   Pass 1: Checking inodes, blocks, and sizes
   Pass 2: Checking directory structure
   Pass 3: Checking directory connectivity
   Pass 4: Checking reference counts
   Pass 5: Checking group summary information
   DBORA: 72831/25706496 files (2.1% non-contiguous), 7152946/51389440 blocks
   

6. Resize the file system to the required size (120G in the example below).

   # resize2fs /dev/VGExaDb/LVDbOra1 120G
   resize2fs 1.39 (29-May-2006)
   Resizing the filesystem on /dev/VGExaDb/LVDbOra1 to 26214400 (4k) blocks.
   The filesystem on /dev/VGExaDb/LVDbOra1 is now 26214400 blocks long.
   

7. Resize the LVM to the desired size.

   # lvm lvreduce -L 120G --verbose /dev/VGExaDb/LVDbOra1
       Finding volume group VGExaDb
     WARNING: Reducing active logical volume to 120.00 GB
     THIS MAY DESTROY YOUR DATA (filesystem etc.)
   Do you really want to reduce LVDbOra1? [y/n]: y
       Archiving volume group "VGExaDb" metadata (seqno 8).
     Reducing logical volume LVDbOra1 to 120.00 GB
       Found volume group "VGExaDb"
       Found volume group "VGExaDb"
       Loading VGExaDb-LVDbOra1 table (253:2)
       Suspending VGExaDb-LVDbOra1 (253:2) with device flush
       Found volume group "VGExaDb"
       Resuming VGExaDb-LVDbOra1 (253:2)
       Creating volume group backup "/etc/lvm/backup/VGExaDb" (seqno 9).
     Logical volume LVDbOra1 successfully resized
   

8. Mount the partition using the following command:

   # mount -t ext3 /dev/VGExaDb/LVDbOra1 /u01
   

9. Verify the space was reduced using the following command:

   # df -h /u01
   Filesystem                    Size  Used Avail Use% Mounted on
   /dev/mapper/VGExaDb-LVDbOra1  119G   25G   88G  22% /u01
   
   # lvm vgdisplay -s
     "VGExaDb" 271.82 GB [174.00 GB used / 97.82 GB free]
   

10. Restart any software that was stopped in step 3.

    • Oracle Clusterware and Oracle Database

      # GI_HOME/bin/crsctl start crs
      

    • Trace File Analyzer

      # GI_HOME/bin/tfactl start
      

    • OS Watcher (releases earlier than 11.2.3.3.0)

      # /opt/oracle.cellos/vldrun -script oswatcher
      

    • ExaWatcher (release 11.2.3.3.0 and later)

      # /opt/oracle.cellos/vldrun -script oswatcher
      

    • Oracle Enterprise Manager agent

      (oracle)$ agent_home/bin/emctl start agent
      

2.11.3 Extending the Swap Partition

This procedure describes how to extend the size of the swap (/swap) partition.

Note:

This procedure requires the system to be offline and restarted.

Keep at least 1 GB of free space in the VGExaDb volume group to be used for the Logical Volume Manager (LVM) snapshot created by the dbnodeupdate.sh utility during software maintenance. If you make snapshot-based backups of the / (root) and /u01 directories by following the steps in "Creating a Snapshot-Based Backup of Oracle Linux Database Server", then keep at least 6 GB of free space in the VGExaDb volume group.

1. Collect information about the current environment.
   1. Use the swapon command to identify the swap partition.

      # swapon -s
      Filename    Type        Size       Used   Priority
      /dev/dm-2   partition   25165816   0      -1
      

   2. Use the lvm lvscan command to display the logical volumes.

      # lvm lvscan
      ACTIVE '/dev/VGExaDb/LVDbSys1' [30.00 GiB] inherit
      ACTIVE '/dev/VGExaDb/LVDbSys2' [30.00 GiB] inherit
      ACTIVE '/dev/VGExaDb/LVDbSwap1' [24.00 GiB] inherit
      ACTIVE '/dev/VGExaDb/LVDbOra1' [103.00 GiB] inherit
      ACTIVE '/dev/VGExaDb/LVDoNotRemoveOrUse' [1.00 GiB] inherit
      

   3. Use the vgdisplay command to display the current volume group configuration.

      # vgdisplay
        --- Volume group ---
        VG Name               VGExaDb
        System ID            
        Format                lvm2
        Metadata Areas        1
        Metadata Sequence No  4
        VG Access             read/write
        VG Status             resizable
        MAX LV                0
        Cur LV                3
        Open LV               3
        Max PV                0
        Cur PV                1
        Act PV                1
        VG Size               556.80 GB
        PE Size               4.00 MB
        Total PE              142541
        Alloc PE / Size       39424 / 154.00 GB
        Free  PE / Size       103117 / 402.80 GB
        VG UUID               po3xVH-9prk-ftEI-vijh-giTy-5chm-Av0fBu
      

   4. Use the pvdisplay command to display the name of the physical device created by LVM and used with the operating system.

      # pvdisplay
        --- Physical volume ---
        PV Name               /dev/sda2
        VG Name               VGExaDb
        PV Size               556.80 GB / not usable 2.30 MB
        Allocatable           yes
        PE Size (KByte)       4096
        Total PE              142541
        Free PE               103117
        Allocated PE          39424
        PV UUID               Eq0e7e-p1fS-FyGN-zrvj-0Oqd-oUSb-55x2TX
      

2. For x86 machines: Attach the /opt/oracle.SupportTools/diagnostics.iso file from any healthy database server as virtual media to the ILOM of the database server to be restored.

   The following is an example of how to set up a virtual CD-ROM using the ILOM interface:

   1. Copy the /opt/oracle.SupportTools/diagnostics.iso file to a directory on the machine using the ILOM interface.
   2. Log in to the ILOM web interface.
   3. Select Remote Console from the Remote Control tab. This will start the console.
   4. Select the Devices menu.
   5. Select the CD-ROM image option.
   6. Navigate to the location of the diagnostics.iso file.
   7. Open the diagnostics.iso file.
   8. Select Host Control from the Remote Control tab.
   9. Select CDROM as the next boot device from the list of values.
   10. Click Save.
3. Restart the server in diagnostic mode.
4. Verify the file system is valid.

   Use the following command:

   #fsck -f /dev/VGExaDb/LVDbSwap1
   

5. Extend the partition.

   In this example, the logical volume is expanded to 80% of the physical volume size. At the same time, the file system is resized with this command. In the following command, the value for LogicalVolumePath is obtained by the lvm lvscan command, and the value for PhysicalVolumePath is obtained by the pvdisplay command.

   #lvextend -l+80%PVS --resizefs --verbose LogicalVolumePath PhysicalVolumePath
   

6. Restart the system in normal mode.

2.12 Creating a Snapshot-Based Backup of Oracle Linux Database Server

A backup should be made before and after every significant change to the software on the database server. For example, a backup should be made before and after the following procedures:

• Application of operating system patches

• Application of Oracle patches

• Reconfiguration of significant operating parameters

• Installation or reconfiguration of significant non Oracle software

This section contains the following topics:

• Creating a Snapshot-Based Backup of Oracle Linux Database Server with Uncustomized Partitions

• Creating a Snapshot-Based Backup of Oracle Linux Database Server with Customized Partitions

2.12.1 Creating a Snapshot-Based Backup of Oracle Linux Database Server with Uncustomized Partitions

This procedure describes how to take a snapshot-based backup. The values shown in the procedure are examples.

If you have not customized the database server partitions from their original shipped configuration, then use the procedures in this section to take a backup and use the backup to restore the database server using the backup.

Note:

• The recovery procedure restores the exact partitions, including the name and sizes, as they were originally shipped. If you modified the partitions in any way, then you cannot use this procedure. Modifications include changing sizes, renaming, addition or removal of partitions.

• All steps must be performed as the root user.

1. Prepare a destination to hold the backup.

   The destination can be a large, writable NFS location. The NFS location should be large enough to hold the backup tar files. For uncustomized partitions, 145 GB should be adequate.

   1. Create a mount point for the NFS share.

      mkdir -p /root/tar
      

   2. Mount the NFS location.

      In the following command, ip_address is the IP address of the NFS server, and nfs_location is the NFS location.

      mount -t nfs -o rw,intr,soft,proto=tcp,nolock
      ip_address:/nfs_location/ /root/tar
      

2. Take a snapshot-based backup of the / (root), /u01, and /boot directories.
   1. Create a snapshot named root_snap for the root directory.

      LVDbSys1 is used in the example below, but you should use the value based on the output of imageinfo. If the active image is on LVDbSys2, then the command would be: lvcreate -L1G -s -c 32K -n root_snap /dev/VGExaDb/LVDbSys2.

      lvcreate -L1G -s -c 32K -n root_snap /dev/VGExaDb/LVDbSys1
      

   2. Label the snapshot.

      e2label /dev/VGExaDb/root_snap DBSYS_SNAP
      

   3. Determine the file system type of the / (root) and /u01 directories.

      Database servers running 12.1.2.1.0 or later use the "ext4" file system type, while older systems use "ext3". Systems older than X5 with cells updated to 12.1.2.1.0 or later also use "ext3".

      # mount -l
      

   4. Mount the snapshot.

      In the mount command below, filesystem_type_of_/u01_directory is a placeholder for the file system type as determined in the previous step.

      mkdir /root/mnt
      mount /dev/VGExaDb/root_snap /root/mnt -t filesystem_type_of_/u01_directory
      

   5. Create a snapshot named u01_snap for the /u01 directory.

      lvcreate -L5G -s -c 32K -n u01_snap /dev/VGExaDb/LVDbOra1
      

   6. Label the snapshot.

      e2label /dev/VGExaDb/u01_snap DBORA_SNAP
      

   7. Mount the snapshot.

      In the mount command below, filesystem_type_of_/u01_directory is a placeholder for the file system type as determined in step 2.c above.

      mkdir -p /root/mnt/u01
      mount /dev/VGExaDb/u01_snap /root/mnt/u01 -t filesystem_type_of_/u01_directory
      

   8. Change to the directory for the backup.

      cd /root/mnt
      

   9. Create the backup file using one of the following commands:

      • System does not have NFS mount points:

        # tar -pjcvf /root/tar/mybackup.tar.bz2 * /boot --exclude \
        tar/mybackup.tar.bz2 > /tmp/backup_tar.stdout 2> /tmp/backup_tar.stderr
        

      • System has NFS mount points:

        In the following command, nfs_mount_points are the NFS mount points. Excluding the mount points prevents the generation of large files and long backup times.

        # tar -pjcvf /root/tar/mybackup.tar.bz2 * /boot --exclude \
        tar/mybackup.tar.bz2 --exclude nfs_mount_points >         \
        /tmp/backup_tar.stdout 2> /tmp/backup_tar.stderr
        

   10. Check the /tmp/backup_tar.stderr file for any significant errors.
       Errors about failing to tar open sockets, and other similar errors, can be ignored.
3. Unmount the snapshots and remove the snapshots for the / (root) and /01 directories.

   cd /
   umount /root/mnt/u01
   umount /root/mnt
   /bin/rm -rf /root/mnt
   lvremove /dev/VGExaDb/u01_snap
   lvremove /dev/VGExaDb/root_snap
   

4. Unmount the NFS share.

   umount /root/tar
   

2.12.2 Creating a Snapshot-Based Backup of Oracle Linux Database Server with Customized Partitions

When you have customized the partitions, the procedure to back up is the same as the procedure used for non-customized database servers, with the following exceptions:

• You must add any additional partitions similar to /u01 to the backup.

• If any partitions were renamed, then use the names defined for your environment. For example, if /u01 was renamed to /myown_u01, then use /myown_u01 in the commands.

2.13 Backing up the Management Domain (dom0) and User Domains (domU) in an Oracle Virtual Server Deployment

In an Oracle Virtual Server deployment, you need to back up the management domain, dom0, and the user domains (domU's):

• Backing up the Management Domain dom0 Using Snapshot-Based Backup

• Backing up the User Domains

2.13.1 Backing up the Management Domain dom0 Using Snapshot-Based Backup

This procedure describes how to take a snapshot-based backup of the management domain, dom0.

The logical volume /dev/VGExaDb/LVDoNotRemoveOrUse is a placeholder to make sure there is always free space available to create a snapshot. If you run dbserver_backup.sh, then the placeholder LVM is removed by the script, the free space is used for a snapshot, and the LVM is re-created after the snapshot is created. If you follow the manual procedure described here, then you have to perform all these tasks manually.

The values shown in the steps below are examples. All steps must be performed as the root user.

1. Prepare a destination to hold the backup.

   The destination should reside outside of the local machine, such as a writable NFS location, and be large enough to hold the backup tar file(s). For un-customized partitions, the space needed for holding the backup is around 60 GB.

   The following commands may be used to prepare the backup destination.

   # mkdir -p /remote_FS
   
   # mount -t nfs -o rw,intr,soft,proto=tcp,nolock ip_address:/nfs_location/ /remote_FS
   

   ip_address is the IP address of the NFS server, and nfs_location is the NFS location holding the backups.

2. Take a snapshot-based backup of the file system hosting the / (root) directory.
   1. Check for the existence of the LVDoNotRemoveOrUse logical volume.

      If this volume is present, then remove the volume to make space for the snapshot. Execute the script below to check for the existence of the LVDoNotRemoveOrUse logical volume and remove it if present.

      lvm lvdisplay --ignorelockingfailure /dev/VGExaDb/LVDoNotRemoveOrUse
      if [ $? -eq 0 ]; then
        # LVDoNotRemoveOrUse logical volume exists.
        lvm lvremove -f /dev/VGExaDb/LVDoNotRemoveOrUse
        if [ $? -ne 0 ]; then
             echo "Unable to remove logical volume: LVDoNotRemoveOrUse. Unable to proceed with backup"
        fi
      fi
      

      If the LVDoNotRemoveOrUse logical volume does not exist, then investigate the reason and do not proceed with the steps below.

   2. Create a snapshot named LVDbSys3_snap for the file system hosting the / (root) directory.

      This example assumes LVDbSys3 is the active partition.

      # lvcreate -L1G -s -n LVDbSys3_snap /dev/VGExaDb/LVDbSys3
      

   3. Label the snapshot.

      # e2label /dev/VGExaDb/LVDbSys3_snap DBSYSOVS_SNAP
      

   4. Mount the snapshot.

      # mkdir /root/mnt
      
      # mount /dev/VGExaDb/LVDbSys3_snap /root/mnt -t ext4
      

   5. Change to the directory for the backup.

      # cd /root/mnt
      

   6. Create the backup file.

      # tar -pjcvf /remote_FS/mybackup.tar.bz2 * /boot > /tmp/backup_tar.stdout 2> /tmp/backup_tar.stderr
      

   7. Check the /tmp/backup_tar.stderr file for any significant errors.

      Errors about failing to tar open sockets, and other similar errors, can be ignored.

3. Unmount the snapshot and remove the snapshot for the root directory.

   # cd /
   # umount /root/mnt
   # /bin/rmdir /root/mnt
   # lvremove /dev/VGExaDb/LVDbSys3_snap
   

4. Unmount the NFS share.

   # umount /remote_FS
   

2.13.2 Backing up the User Domains

There are two ways to back up the user domains:

• Method 1: Back up the storage repository using Oracle Cluster File System (OCFS) reflinks to get a consistent backup

  This method backs up the storage repository that is the /EXAVMIMAGES ocfs2 file system.

  You can back up the entire /EXAVMIMAGES, which will back up all the user domains managed by the dom0, or you can select which user domains you want to back up. The user domains are located in the /EXAVMIMAGES/GuestImages/user directories.

  This method provides a more robust and a comprehensive backup than method 2. Method 2 provides a quicker and an easier backup method, especially in role separated environments.

  Method 1 is ideal where a dom0 administrator is responsible for user domain backups.

• Method 2: Back up a user domain using snapshot-based backup

  This method backs up a single user domain using snapshot-based backup from inside the user domain.

  Method 2 is ideal where a user domain administrator is responsible for the user domain backups.

2.13.2.1 Method 1: Back up All the User Domains

You can back up the storage repository that is the /EXAVMIMAGES OCFS2 file system

The procedure shown here backs up all the user domains.

1. Prepare a destination to hold the backup.

   The destination should reside outside of the local machine such as writable NFS location, and be large enough to hold the backup. The space needed for holding the backup will be proportional to the number of Oracle VMs deployed on the system, up to a maximum of around 1.6 TB.

   This guide assumes there will be no more than 15 user domains per management domain.

   The following commands may be used to prepare the backup destination.

   # mkdir -p /remote_FS
   
   # mount -t nfs -o rw,intr,soft,proto=tcp,nolock ip_address:/nfs_location/ /remote_FS
   

   ip_address is the IP address of the NFS server, and nfs_location is the NFS location holding the backups.

   Note:

   Steps 2 through 4 below should be completed within the CSS misscount time, which is set to 60s by default. If the workload is high such that the pause of user domains along with the reflink creation takes more than about 45s, it is recommended to shut down the VMs before taking the snapshot using reflinks to avoid Oracle RAC node evictions. This can be done in a rolling fashion where all the user domains in one OVS/dom0 is shut down and backed up, leaving the other user domains of the Oracle RAC clusters up.

   Refer to My Oracle Support note 294430.1 for details on CSS misscount.

2. Pause the user domains.

   # xm pause domain_id
   

3. Create reflinks for all the files in /EXAVMIMAGES.
4. Unpause the user domains.

   # xm unpause domain_id
   

5. Back up the snapshot, which are the reflink files created in step 3, as a .tar file.
6. Remove the reflinks created in step 3.

The sample commands below may be used as a reference for the operations listed in steps 2 through 6 above.

##  Create the Backup Directory
find /EXAVMIMAGES -type d|grep -v 'lost+found'|awk '{print "mkdir -p /EXAVMIMAGES/Backup"$1}'|sh

## Pause user domains
xm list|egrep -v '^Domain-0|^Name'|awk '{print "xm pause",$2}'|sh

##  Create Reflinks for all the files to be backed up
find /EXAVMIMAGES/ -type f|awk '{print "reflink",$0,"/EXAVMIMAGES/Backup"$0}'|sh

## Unpause user domains
xm list|egrep -v '^Domain-0|^Name'|awk '{print "xm unpause",$2}'|sh;

##  Backup from the reflinks
pushd /EXAVMIMAGES/Backup/EXAVMIMAGES
tar -Spcvf /remote_FS/exavmimages-sparse.tar * 1> /tmp/Backup.log 2> /tmp/Backup.err
popd
rm -rf /EXAVMIMAGES/Backup/*

2.13.2.2 Method 2: Back up a User Domain from Inside the User Domain

The following procedure describes how to take a snapshot-based backup of a user domain from inside the user domain.

Note:

This method of backing up a user domain from inside the user domain using LVM snapshots will have limited usage in terms of recovery. Such a backup can only be used for recovery purposes when the user domain is still bootable and allows login as the root user. This means the damage is such that some files have been lost or damaged but can be restored from the tar backup after the user domain has booted up and the / (root) partition and the boot partitions are mounted. If that is not the case and the damage is such that the user domain does not boot, then you need a backup taken using method 1 above to recover the user domain, and you need to perform the recovery procedure at the user domain level using the recovery procedure described below.

All steps are performed from inside the user domain.

The procedure backs up the following:

• LVDbSys1 lvm

• LVDbOra1 lvm

• /boot partition

• Grid Infrastructure home

• RDBMS home

All steps must be performed as the root user.

1. Prepare a destination to hold the backup.

   # mkdir -p /remote_FS
   
   # mount -t nfs -o rw,intr,soft,proto=tcp,nolock ip_address:/nfs_location/ /remote_FS
   

   ip_address is the IP address of the NFS server, and nfs_location is the NFS location holding the backups.

2. Take a snapshot-based backup of the file systems containing / (root) and the /u01 directories, as follows:

   1. Create a snapshot named LVDbSys1_snap for the file system containing the root directory. The volume group must have at least 1 GB of free space for the command to succeed.

      # lvcreate -L1G -s -n LVDbSys1_snap /dev/VGExaDb/LVDbSys1
      

   2. Label the snapshot.

      # e2label /dev/VGExaDb/LVDbSys1_snap DBSYS_SNAP
      

   3. Mount the snapshot.

      # mkdir /root/mnt
      
      # mount /dev/VGExaDb/LVDbSys1_snap /root/mnt -t ext4
      

   4. Create a snapshot named u01_snap for the /u01 directory.

      # lvcreate -L256M -s -n u01_snap /dev/VGExaDb/LVDbOra1
      

   5. Label the snapshot.

      # e2label /dev/VGExaDb/u01_snap DBORA_SNAP
      

   6. Mount the snapshot.

      # mkdir -p /root/mnt/u01
      
      # mount /dev/VGExaDb/u01_snap /root/mnt/u01 -t ext4
      

   7. Change to the directory for the backup.

      # cd /root/mnt
      

   8. Create the backup file to back up the two snapshots taken above, the /boot partition, the RDBMS home, and the Grid Infrastructure home.

      # tar -pjcvf /remote_FS/mybackup.tar.bz2 * /boot /u01/app/12.1.0.2/grid /u01/app/oracle/product/12.1.0.2/dbhome_1 > /tmp/backup_tar.stdout 2> /tmp/backup_tar.stderr
      

   9. Check the /tmp/backup_tar.stderr file for any significant errors. Errors about failing to tar open sockets, and other similar errors, can be ignored.

3. Unmount and remove the snapshots for the file system containing the root directories.

   # cd /
   # umount /root/mnt/u01
   # umount /root/mnt
   # /bin/rmdir /root/mnt
   # lvremove /dev/VGExaDb/u01_snap
   # lvremove /dev/VGExaDb/LVDbSys1_snap 
   

4. Unmount the NFS share.

   # umount /remote_FS
   

2.14 Recovering Oracle Linux Database Server Using a Snapshot-Based Backup

This section describes how to recover a database server file systems running Oracle Linux using a snapshot-based backup after severe disaster conditions happen for the database server, or when the server hardware is replaced to such an extent that it amounts to new hardware. For example, replacing all hard disks leaves no trace of original software on the system. This is similar to replacing the complete system as far as the software is concerned. In addition, it provides a method for disaster recovery of the database servers using an LVM snapshot-based backup taken when the database server was healthy before the disaster condition.

The recovery procedures use the diagnostics.iso image as a virtual CD-ROM to restart the database server in rescue mode using the ILOM. The general workflow includes the following tasks:

1. Recreate the following:

   • Boot partitions

   • Physical volumes

   • Volume groups

   • Logical volumes

   • File system

   • Swap partition

2. Activate the swap partition.

3. Ensure the /boot partition is the active boot partition.

4. Restore the data.

5. Reconfigure GRUB.

6. Restart the server.

The recovery procedures described in this section do not include backup or recovery of Exadata Storage Servers or database data. Oracle recommends testing the backup and recovery procedures on a regular basis. This section contains the following topics:

• Recovering Oracle Linux Database Server with Uncustomized Partitions

• Recovering Oracle Linux Database Server with Customized Partitions

• Configuring Oracle Exadata Database Machine Eighth Rack Oracle Linux Database Server After Recovery

Note:

Restoring files from tape may require additional drives to be loaded, and is not covered in this chapter. Oracle recommends backing up files to an NFS location, and using existing tape options to back up and recover from the NFS host.

2.14.1 Recovering Oracle Linux Database Server with Uncustomized Partitions

You can recover the Oracle Linux database server from a snapshot-based backup when using uncustomized partitions.

This procedure is applicable when the layout of the partitions, logical volumes, file systems, and their sizes are equal to the layout when the database server was initially deployed.

Caution:

All existing data on the disks is lost during the procedure.

1. Prepare NFS server to host the backup archive mybackup.tar.bz2.

   The NFS server must be accessible by IP address. For example, on an NFS server with the IP address nfs_ip, where the directory /export is exported from NFS mounts, put the mybackup.tar.bz2 file in the /export directory.

2. For x86 machines: Attach the /opt/oracle.SupportTools/diagnostics.iso file from any healthy database server as virtual media to the ILOM of the database server to be restored.

   The following is an example of how to set up a virtual CD-ROM using the ILOM interface:

   1. Copy the /opt/oracle.SupportTools/diagnostics.iso file to a directory on the machine using the ILOM interface.
   2. Log in to the ILOM web interface.
   3. Select Remote Console from the Remote Control tab. This will start the console.
   4. Select the Devices menu.
   5. Select the CD-ROM image option.
   6. Navigate to the location of the diagnostics.iso file.
   7. Open the diagnostics.iso file.
   8. Select Host Control from the Remote Control tab.
   9. Select CDROM as the next boot device from the list of values.
   10. Click Save.
3. For SPARC machines: Attach the /opt/oracle.SupportTools/diagnostics.iso file from any healthy database server as virtual media to the ILOM of the database server to be restored.
   1. Copy the /opt/oracle.SupportTools/diagnostics.iso file to the NFS location hosting the backup archive mybackup.tar.bz2.
   2. Log in to the ILOM console.
   3. Run the following command:

      set /SP/services/kvms/host_storage_device/ mode=remote
      

   4. Specify the server URI for the diagnostic iso.

      set /SP/services/kvms/host_storage_device/remote/ server_URI=nfs://ip-number:/path/to/diagnostics.iso
      

4. Restart the system from the ISO file.

   Choose the CD-ROM as boot device during startup. You can also preset the boot device using the ipmitool command from any other machine that can reach the ILOM of the database server to be restored, instead of selecting the boot device manually during boot up using the following commands:

   ipmitool -H ILOM_ip_address_or_hostname \
   -U root_user chassis bootdev cdrom
   
   ipmitool -H ILOM_ip_address_or_hostname \
   -U root_user chassis power cycle
   

5. Answer as follows when prompted by the system. The responses are shown in bold.

   Note that for Exadata software release 12.1.2.2.0 or later, DHCP is used and you do not have to manually set up the network.

   • If you are using Exadata software release 12.1.2.2.0 or later, the prompts look like the following:

      Use diagnostics shell password to login as root user
                 (reboot or power cycle to exit the shell),
               (r)estore system from NFS backup archive.
     Select: r
     Continue (y/n) [n]: y
     Rescue password:
     [INFO: ] Enter path to the backup file on the NFS server in format:
            Enter path to the backup file on the NFS server in format:
            <ip_address_of_the_NFS_share>:/<path>/<archive_file>
            For example, 10.124.1.15:/export/nfs/share/backup.2010.04.08.tar.bz2
     NFS line: 10.196.101.250:/srv/file/test/OARDC/fuji0506/mybackup06.tar.bz2
     [INFO: ] The backup file could be created either from LVM or non-LVM based COMPUTE node
     [INFO: ] Versions below 11.2.1.3.0 do not support LVM based partitioning
     Use LVM based scheme. (y/n) [y]:
     [INFO: ] Starting DHCP client
     ixgbe 0000:88:00.0: eth2: NIC Link is Up 1 Gbps, Flow Control: None
     ixgbe 0000:20:00.1: eth1: NIC Link is Up 1 Gbps, Flow Control: None
     ixgbe 0000:20:00.0: eth0: NIC Link is Up 1 Gbps, Flow Control: RX/TX
     [INFO: ] DHCP settings are in use
     [INFO: ] Interface: eth0
     [INFO: ] IP address: 10.196.103.123
     [INFO: ] Netmask: 255.255.252.0
     [INFO: ] Gateway: 10.196.100.4
     RPC: Registered named UNIX socket transport module.
     RPC: Registered udp transport module.
     RPC: Registered tcp transport module.
     RPC: Registered tcp NFSv4.1 backchannel transport module.
     FS-Cache: Loaded
     FS-Cache: Netfs 'nfs' registered for caching
     NET: Registered protocol family 10
     ADDRCONF(NETDEV_UP): eth4: link is not ready
     ADDRCONF(NETDEV_UP): eth5: link is not ready
     ADDRCONF(NETDEV_UP): eth3: link is not ready
     

     When the recovery completes, the login screen appears.

   • If you are using Exadata software release earlier than 12.1.2.2.0, the prompts look like the following

     Choose from following by typing letter in '()':
     (e)nter interactive diagnostics shell. Must use credentials from Oracle
     support to login (reboot or power cycle to exit the shell),
     (r)estore system from NFS backup archive,
     Select:r
     Are you sure (y/n) [n]:y
      
     The backup file could be created either from LVM or non-LVM based compute node
     versions below 11.2.1.3.1 and 11.2.2.1.0 or higher do not support LVM based partitioning
     use LVM based scheme(y/n):y
      
     Enter path to the backup file on the NFS server in format:
     ip_address_of_the_NFS_share:/path/archive_file
     For example, 10.10.10.10:/export/operating_system.tar.bz2
     NFS line:nfs_ip:/export/mybackup.tar.bz2
     IP Address of this host:IP address of the DB host
     Netmask of this host:netmask for the above IP address
     Default gateway:Gateway for the above IP address. If there is no default gateway in your network, enter 0.0.0.0.
     

   When the recovery completes, the login screen appears.

6. Log in as the root user.
   If you do not have the password for the root user, then contact Oracle Support Services.
7. Use the reboot command to restart the system.
   The restoration process is complete.
8. Verify that all Oracle software can start and function by logging in to the database server.
   The /usr/local/bin/imagehistory command indicates that the database server was reconstructed.

   The following is an example of the output:

   imagehistory
   
   Version                  : 11.2.2.1.0
   Image activation date    : 2010-10-13 13:42:58 -0700
   Imaging mode             : fresh
   Imaging status           : success
   
   Version                  : 11.2.2.1.0
   Image activation date    : 2010-10-30 19:41:18 -0700
   Imaging mode             : restore from nfs backup
   Imaging status           : success
   

9. If the recovery was on Oracle Exadata Database Machine Eighth Rack, then perform the procedure described in "Configuring Oracle Exadata Database Machine Eighth Rack Oracle Linux Database Server After Recovery."

2.14.2 Recovering Oracle Linux Database Server with Customized Partitions

This procedure describes how to recover an Oracle Linux database server from a snapshot-based backup when using customized partitions.

The steps are same as restoring non-customized partitions until you are prompted to choose from enter interactive diagnostics shell and restore system from NFS backup archive options after booting the database server using diagnostics.iso.

1. Choose to enter the diagnostics shell, and log in as the root user.

   If you do not have the password for the root user, then contact Oracle Support.

2. If required, use /opt/MegaRAID/MegaCli/MegaCli64 to configure the disk controller to set up the disks.
3. Ensure you create a primary boot partition of size at least 128 MB to be mounted at /boot.
   The boot area cannot be a LVM partition.
4. Create the boot partition.

   umount /mnt/cell
   parted /dev/sda
   

   The interactive shell appears. The following procedure describes how to respond to the system prompts:

   1. Assign a disk label.

      • If you are running Oracle Exadata System Software release 11.2.3.3.0 or later:

        (parted) mklabel gpt
        

      • If you are running a release earlier than Oracle Exadata System Software release 11.2.3.3.0:

        (parted) mklabel msdos
        

   2. Set the unit size as sector.

      (parted) unit s
      

   3. Check the partition table by displaying the existing partitions.

      (parted) print
      

   4. Remove the partitions that will be re-created.

      (parted) rm <part#>
      

   5. Create a new first partition.

      (parted) mkpart primary 63 1048639
      

   6. Specify this is a bootable partition.

      (parted) set 1 boot on
      

5. Create an additional primary (LVM) partition.
   • If using Oracle Exadata System Software release 18.1.0.0.0 or later — Create second primary (bios_grub) and third primary (LVM) partitions:

     1. Enter mkpart primary 1048640 1050687 to create a new second partition.

     2. Enter set 2 bios_grub on to specify this is a GRUB BIOS partition.

     3. Enter mkpart primary 1050688 1751949278 to create a new third partition.

     4. Enter set 3 lvm on to specify this is a physical volume.

     5. Enter quit to write the information to disk, then quit.

   • If using a release earlier than Oracle Exadata System Software release 18.1.0.0.0:

     1. Enter mkpart primary 1048640 -1 to create a new second partition.

     2. Enter set 2 lvm on to specify this is a physical volume.

     3. Enter quit to write the information to disk, then quit.

6. Use the /sbin/lvm command to re-create the customized LVM partitions and mkfs to create file systems.
   1. Create the physical volume, volume group, and the logical volumes.

      lvm pvcreate /dev/sda2
      lvm vgcreate VGExaDb /dev/sda2
      

   2. Create the logical volume for the / (root) directory, a file system, and label it.

      • Create the logical volume:

        lvm lvcreate -n LVDbSys1 -L40G VGExaDb
        

      • If using Oracle Exadata System Software release 12.1.2.2.0 or later, then create the logical volume for the reserved partition.

        # lvm lvcreate -n LVDoNotRemoveOrUse –L1G VGExaDb
        

        Note:

        Do not create any file system on this logical volume.

      • Create the file system.

        • If you previously had an ext4 file system, use the mkfs.ext4 command:

          mkfs.ext4 /dev/VGExaDb/LVDbSys1
          

        • If you previously had an ext3 file system, use the mkfs.ext3 command:

          mkfs.ext3 /dev/VGExaDb/LVDbSys1
          

      • Label the file system.

        e2label /dev/VGExaDb/LVDbSys1 DBSYS
        

   3. Create the logical volume for the swap directory, and label it.

      lvm lvcreate -n LVDbSwap1 -L24G VGExaDb
      mkswap -L SWAP /dev/VGExaDb/LVDbSwap1
      

   4. Create the logical volume for the root/u01 directory, and label it.

      • Create the logical volume:

        lvm lvcreate -n LVDbOra1 -L100G VGExaDb
        

      • Create the file system.

        • If you previously had an ext4 file system, use the mkfs.ext4 command:

          mkfs.ext4 /dev/VGExaDb/LVDbOra1
          

        • If you previously had an ext3 file system, use the mkfs.ext3 command:

          mkfs.ext3 /dev/VGExaDb/LVDbOra1
          

      • Label the file system.

        e2label /dev/VGExaDb/LVDbOra1 DBORA
        

   5. Create a file system on the /boot partition, and label it.

      • Create the file system.

        • If you previously had an ext4 file system, use the mkfs.ext4 command:

          mkfs.ext4 /dev/sda1
          

        • If you previously had an ext3 file system, use the mkfs.ext3 command:

          mkfs.ext3 /dev/sda1
          

      • Label the file system:

        e2label /dev/sda1 BOOT
        

      Note:

      For customized file system layouts, additional logical volumes can be created at this time. For customized layouts, different sizes may be used.

7. Create mount points for all the partitions to mirror the original system, and mount the respective partitions.

   For example, assuming /mnt is used as the top level directory for this, the mounted list of partition may look like the following:

   /dev/VGExaDb/LVDbSys1 on /mnt
   /dev/VGExaDb/LVDbOra1 on /mnt/u01
   /dev/sda1 on /mnt/boot
   

   Note:

   For customized file system layouts with additional logical volumes, additional mount points need to be created during this step.

   The following is an example of how to mount the root file system, and create two mount points. In the commands below, filesystem_type_of_/_directory specifies the file system of the / (root) directory: it is either ext3 or ext4.

   mount /dev/VGExaDb/LVDbSys1 /mnt -t filesystem_type_of_/_directory
   mkdir /mnt/u01 /mnt/boot
   
   mount /dev/VGExaDb/LVDbOra1 /mnt/u01 -t filesystem_type_of_/u01_directory
   mount /dev/sda1 /mnt/boot -t filesystem_type_of_/boot_directory
   

8. Bring up the network.
   • If the operating system is Oracle Linux 6 or later:

     ip address add ip_address_for_eth0/netmask_for_eth0 dev eth0
     ip link set up eth0
     ip route add default via gateway_address dev eth0
     

   • If the operating system is Oracle Linux 5:

     ifconfig eth0 ip_address_for_eth0 netmask netmask_for_eth0 up
     

9. Mount the NFS server where you have the backup.

   Mount the NFS server with IP address nfs_ip and the export it as /export to the backup location.

   mkdir -p /root/mnt
   mount -t nfs -o ro,intr,soft,proto=tcp,nolock nfs_ip:/export /root/mnt
   

10. Restore from backup.

    tar -pjxvf /root/mnt/mybackup.tar.bz2 -C /mnt
    

11. Unmount the restored file systems, and remount the /boot partition.

    umount /mnt/u01
    umount /mnt/boot
    umount /mnt
    mkdir /boot
    mount /dev/sda1 /mnt/boot -t filesystem_type_of_/boot_directory
    

12. Set up the boot loader.

    In the following instructions, /dev/sda1 is the /boot area.

    • If using Oracle Exadata System Software release 18.1.0.0.0 or later:

      grub2-install /dev/sda
      
      Installing for i386-pc platform.
      Installation finished. No error reported.
      

    • If using a release earlier than Oracle Exadata System Software release 18.1.0.0.0:

      grub
      find /I_am_hd_boot     (1)
      root (hdX,0)
      setup (hdX)
      quit
      

      In the preceding commands, (1) finds the hard disk hdX that has the file I_am_hd_boot, such as (hd0,0).

13. Detach the diagnostics.iso file.
14. Unmount the /boot partition.

    umount /boot
    

15. Restart the system.

    reboot
    

    This completes the restoration procedure for the server.

16. If the recovery was on Oracle Exadata Database Machine Eighth Rack, then perform the procedure described in "Configuring Oracle Exadata Database Machine Eighth Rack Oracle Linux Database Server After Recovery".

2.14.3 Configuring Oracle Exadata Database Machine Eighth Rack Oracle Linux Database Server After Recovery

After the Oracle Linux database server in Oracle Exadata Database Machine Eighth Rack has been re-imaged, restored, or rescued, you can then reconfigure the eighth rack. Use one of the following procedures:

• Configuring Eighth Rack On X3-2 or Later Machines Running Oracle Exadata Storage Server Release 12.1.2.3.0 or Later

• Configuring Eighth Rack On X3-2 Machines Running Oracle Exadata Storage Server Release 12.1.2.2.3 or Earlier

2.14.3.1 Configuring Eighth Rack On X3-2 or Later Machines Running Oracle Exadata Storage Server Release 12.1.2.3.0 or Later

The following procedure should be performed after Oracle Linux database server in Oracle Exadata Database Machine Eighth Rack has been re-imaged, restored, or rescued.

For X3-2 systems, use this method only if you are running Oracle Exadata Storage Server Release 12.1.2.3.0 or later.

1. On the recovered server, check that the resourcecontrol utility exists in the /opt/oracle.SupportTools directory. If not, copy it from another database server to the recovered server.
2. Ensure proper permissions are set on the resourcecontrol utility.

   # chmod 740 /opt/oracle.SupportTools/resourcecontrol
   

3. Verify the current configuration.

   # dbmcli -e LIST DBSERVER ATTRIBUTES coreCount
   

   See Table 2-3 for the number of cores allowed for each machine configuration. If the correct value is shown, then no configuration changes are necessary. If that value is not shown, then continue to step 4 of this procedure.

4. Change the enabled core configuration.

   # dbmcli -e ALTER DBSERVER pendingCoreCount=new_core_count FORCE
   

   new_core_count for an Eighth Rack is:

   • X7-2 : 24

   • X6-2 : 22

   • X5-2 : 18

   • X4-8 : 60

   • X4-2 : 12

5. Restart the server.

   # reboot
   

6. Verify the changes to the configuration.

   # dbmcli -e LIST DBSERVER ATTRIBUTES coreCount
   

2.14.3.2 Configuring Eighth Rack On X3-2 Machines Running Oracle Exadata Storage Server Release 12.1.2.2.3 or Earlier

The following procedure should be performed after Oracle Linux database server in Oracle Exadata Database Machine Eighth Rack has been re-imaged, restored, or rescued.

1. Copy the /opt/oracle.SupportTools/resourcecontrol utility from another database server to the /opt/oracle.SupportTools/resourcecontrol directory on the recovered server.
2. Ensure proper permissions are set on the utility.

   # chmod 740 /opt/oracle.SupportTools/resourcecontrol
   

3. Verify the current configuration.

   The output from the command is shown in this example.

   # /opt/oracle.SupportTools/resourcecontrol -show
   
     Validated hardware and OS. Proceed.
     Number of cores active: 8
   

   For an eighth rack configuration, eight cores should be enabled. If that value is shown, then no configuration changes are necessary. If that value is not shown, then continue to step 4 of this procedure.

   Note:

   If there is an error similar to the following after running the utility, then restarting the server one or more times usually clears the error:

   Validated hardware and OS. Proceed.
   Cannot get ubisconfig export. Cannot Proceed. Exit.
   

4. Change the configuration for enabled cores.

   # /opt/oracle.SupportTools/resourcecontrol -cores 8
   

5. Restart the server.

   # reboot
   

6. Verify the changes to the configuration.

   # /opt/oracle.SupportTools/resourcecontrol -show
   

   The following is an example of the expected output from the command for the database server:

   This is a Linux database server.
   Validated hardware and OS. Proceed.
   Number of cores active per socket: 4
   

2.15 Recovering in an Oracle Virtual Server Deployment

The following procedures describe how to recover an Oracle Virtual Server (OVS) from a snapshot-based backup when severe disaster conditions damage the OVS, or when the server hardware is replaced to such an extent that it amounts to new hardware. For example, replacing all hard disks leaves no trace of original software on the system. This is similar to replacing the complete system as far as the software is concerned. In addition, it provides a method for disaster recovery of the database servers using an LVM snapshot-based backup taken when the database server was healthy before the disaster condition.

The recovery procedures use the diagnostics.iso image as a virtual CD-ROM to restart the Oracle Virtual Server in rescue mode using the ILOM. At a high-level, the steps look like this:

1. Re-create the following:

   • Boot partitions

   • Physical volumes

   • Volume groups

   • Logical volumes

   • File system

   • Swap partition

2. Activate the swap partition.

3. Ensure the /boot partition is the active boot partition.

4. Restore the data.

5. Reconfigure GRUB.

6. Restart the server.

The recovery procedures described in this section do not include backup or recovery of Exadata Storage Servers or database data. Oracle recommends testing the backup and recovery procedures on a regular basis.

2.15.1 Scenario 1: Recovering an Oracle Virtual Server Along with All Its User Domains from Backup

You can recover the Oracle Virtual Server and all its user domains from a backup.

The following procedures step you through the recovery process. Chose one of the following procedures, based on the version of Oracle Exadata Storage Server Software that is installed on your system.

Note:

All existing data on the disks is lost during these procedures.

• Recovering an Oracle Virtual Server Along with All Its User Domains (Releases Prior to 12.2.1.1.0)

• Recovering an Oracle Virtual Server Along with All Its User Domains (Releases 12.2.1.1.0 and Later)

2.15.1.1 Recovering an Oracle Virtual Server Along with All Its User Domains (Releases Prior to 12.2.1.1.0)

You can recover an Oracle Virtual Server (OVS) from a snapshot-based backup when severe disaster conditions damage the OVS, or when the server hardware is replaced to such an extent that it amounts to new hardware.

1. Prepare an NFS server to host the backup archive mybackup.tar.bz2.

   The NFS server must be accessible by IP address. For example, on an NFS server with the IP address nfs_ip, where the directory /export is exported from NFS mounts, put the mybackup.tar.bz2 file in the /export directory

2. Attach the /opt/oracle.SupportTools/diagnostics.iso file from any healthy database server as virtual media to the ILOM of the Oracle Virtual Server to be restored.
   The following example shows how to set up a virtual CD-ROM using the ILOM interface:
   1. Copy the diagnostics.iso file to a directory on the machine that will be using the ILOM interface.
   2. Log in to the ILOM web interface.
   3. In the Oracle ILOM web interface, click Remote Control , and then click Redirection.
   4. Select Use Video Redirection.
   5. After the console launches, click Storage in the KVMS menu.
   6. To add a storage image, such as a DVD image, to the Storage Devices dialog box, click Add.
   7. Open the diagnostics.iso file.
      
      
      Description of the illustration GUID-A42F1AC6-8400-41D6-AE83-E778530740FE-default.jpg
      
   8. To redirect storage media from the Storage Device dialog box, select the storage media and click Connect.

      After a connection to the device has been established, the label on the Connect button in the Storage Device dialog box changes to Disconnect.

   9. Select Host Control from the Host Management tab.
   10. Select CDROM as the next boot device from the list of values.
   11. Click Save.

       When the system is booted, the diagnostics.iso image is used.

3. Restart the system from the ISO image file.

   You can restart the system using one of the following methods:

   • Choose the CD-ROM as the boot device during startup

   • Preset the boot device by running the ipmitool command from any other machine that can reach the ILOM of the Oracle Virtual Server to be restored:

     # ipmitool -H ILOM_ip_address_or_hostname -U root chassis bootdev cdrom
     # ipmitool -H ILOM_ip_address_or_hostname -U root chassis power cycle
     

4. Login to the diagnostics shell as the root user.

   When the system displays the following:

   Choose from following by typing letter in '()':
   (e)nter interactive diagnostics shell. Must use credentials from Oracle support to login (reboot or power cycle to exit the shell),
   (r)estore system from NFS backup archive,
   

   Type e to enter the diagnostics shell, and log in as the root user.

   Note:

   If you do not have the password for the root user, then contact Oracle Support Services.
5. If required, use /opt/MegaRAID/MegaCli/MegaCli64 to configure the disk controller to set up the disks.
6. Remove the logical volumes, the volume group, and the physical volume, in case they still exist after the disaster.

   # lvm vgremove VGExaDb --force
   # lvm pvremove /dev/sda2 --force
   

7. Remove the existing partitions and clean up the drive.

   # parted
   GNU Parted 2.1
   Using /dev/sda
   Welcome to GNU Parted! Type 'help' to view a list of commands.
   (parted) rm 1 
   sda: sda2 sda3
   (parted) rm 2 
   sda: sda3
   (parted) rm 3 
   sda:
   (parted) q
   
   # dd if=/dev/zero of=/dev/sda bs=64M count=2
   

8. Create the three partitions on /dev/sda.
   1. Get the end sector for the disk /dev/sda from a surviving dom0 and store it in a variable:

      # end_sector=$(parted -s /dev/sda unit s print|perl -ne '/^Disk\s+\S+:\s+(\d+)s/ and print $1')
      

   2. Create the boot partition, /dev/sda1.

      # parted -s /dev/sda mklabel gpt mkpart primary 64s 1048639s set 1 boot on
      

   3. Create the partition that will hold the LVMs, /dev/sda2.

      # parted -s /dev/sda mkpart primary 1048640s 240132159s set 2 lvm on
      

   4. Create the OCFS2 storage repository partition, /dev/sda3.

      # parted -s /dev/sda mkpart primary 240132160s ${end_sector}s set 3
      

9. Use the /sbin/lvm command to re-create the logical volumes and mkfs to create file systems.
   1. Create the physical volume and the volume group.

      # lvm pvcreate /dev/sda2
      # lvm vgcreate VGExaDb /dev/sda2
      

   2. Create the logical volume for the file system that will contain the / (root) directory and label it.

      # lvm lvcreate -n LVDbSys3 -L30G VGExaDb
      # mkfs.ext4 /dev/VGExaDb/LVDbSys3
      # e2label /dev/VGExaDb/LVDbSys3 DBSYSOVS
      

   3. Create the logical volume for the swap directory, and label it.

      # lvm lvcreate -n LVDbSwap1 -L24G VGExaDb
      # mkswap -L SWAP /dev/VGExaDb/LVDbSwap1
      

   4. Create the logical volume for the backup partition, and build a file system on top of it.

      # lvm lvcreate -n LVDbSys2 -L30G VGExaDb
      # mkfs.ext4 /dev/VGExaDb/LVDbSys2
      

   5. Create the logical volume for the reserved partition.

      # lvm lvcreate -n LVDoNotRemoveOrUse –L1G VGExaDb
      

      Note:

      Do not create any file system on this logical volume.
   6. Create a file system on the /dev/sda1 partition, and label it.

      In the mkfs.ext3 command below, the -I 128 option is needed to set the inode size to 128.

      # mkfs.ext3 -I 128 /dev/sda1
      # tune2fs -c 0 -i 0 /dev/sda1
      # e2label /dev/sda1 BOOT
      

10. Create mount points for all the partitions, and mount the respective partitions.

    For example, if /mnt is used as the top level directory, the mounted list of partitions may look like:

    • /dev/VGExaDb/LVDbSys3 on /mnt

    • /dev/sda1 on /mnt/boot

    The following example mounts the root file system, and creates two mount points:

    # mount /dev/VGExaDb/LVDbSys3 /mnt -t ext4
    # mkdir /mnt/boot
    # mount /dev/sda1 /mnt/boot -t ext3
    

11. Bring up the network on eth0 and assign the host's IP address and netmask to it.

    # ifconfig eth0 ip_address_for_eth0 netmask netmask_for_eth0 up
    # route add -net 0.0.0.0 netmask 0.0.0.0 gw gateway_ip_address
    

12. Mount the NFS server holding the backups.

    # mkdir -p /root/mnt
    # mount -t nfs -o ro,intr,soft,proto=tcp,nolock nfs_ip:/location_of_backup /root/mnt
    

13. From the backup which was created in "Backing up the Management Domain dom0 Using Snapshot-Based Backup", restore the root directory (/) and the boot file system.

    # tar -pjxvf /root/mnt/backup-of-root-and-boot.tar -C /mnt
    

14. Unmount the restored /dev/sda1 partition, and remount it on /boot.

    # umount /mnt/boot
    # mkdir /boot
    # mount /dev/sda1 /boot -t ext3
    

15. Set up the grub boot loader using the command below:

    # grub --device-map=/boot/grub/device.map << DOM0_GRUB_INSTALL
    root (hd0,0)
    setup (hd0)
    quit
    DOM0_GRUB_INSTALL
    

16. Unmount the /boot partition.

    # umount /boot
    

17. Detach the diagnostics.iso file.

    This can be done by clicking Disconnect on the ILOM web interface console, where you clicked Connect in step 2.h  to attach the DVD ISO image.

18. Check the restored /etc/fstab file and remove any reference to /EXAVMIMAGES and /dev/sda3.

    # cd /mnt/etc
    

    Comment out any line that references /EXAVMIMAGES or /dev/sda3.

19. Restart the system.

    # reboot
    

    This completes the restoration procedure for the Oracle VS/dom0.

20. Convert to eighth rack, if required.

    If the recovery is on an Oracle Exadata Database Machine eighth rack, then perform the procedure described in "Configuring Oracle Exadata Database Machine Eighth Rack Oracle Linux Database Server After Recovery"".

21. When the server comes back up, build an OCFS2 file system on the /dev/sda3 partition.

    # mkfs -t ocfs2 -L ocfs2 -T vmstore --fs-features=local /dev/sda3 --force
    

22. Mount the OCFS2 partition /dev/sda3 on /EXAVMIMAGES.

    # mount -t ocfs2 /dev/sda3 /EXAVMIMAGES
    

23. In /etc/fstab, uncomment the commented out references to /EXAVMIMAGES and /dev/sda3 that was performed in step 18.
24. Mount the backup NFS server that holds the storage repository (/EXAVMIMAGES) backup to restore the /EXAVMIMAGES file system which holds all the user domain images.

    # mkdir -p /root/mnt
    # mount -t nfs -o ro,intr,soft,proto=tcp,nolock nfs_ip:/location_of_backup /root/mnt
    

25. Restore the /EXAVMIMAGES file system.

    # tar -Spxvf /root/mnt/backup-of-exavmimages.tar -C /EXAVMIMAGES
    

26. Bring up each user domain.

    # xm create /EXAVMIMAGES/GuestImages/user_domain_hostname/vm.cfg
    

At this point all the user domains should come up along with Oracle Grid Infrastructure and the database instances, and the database instances should join the Oracle RAC cluster formed by the other surviving Oracle Virtual Server nodes.

2.15.1.2 Recovering an Oracle Virtual Server Along with All Its User Domains (Releases 12.2.1.1.0 and Later)

You can recover an Oracle Virtual Server (OVS) from a snapshot-based backup when severe disaster conditions damage the OVS, or when the server hardware is replaced to such an extent that it amounts to new hardware.

1. Prepare an NFS server to host the backup archive mybackup.tar.bz2.

   The NFS server must be accessible by IP address. For example, on an NFS server with the IP address nfs_ip, where the directory /export is exported from NFS mounts, put the mybackup.tar.bz2 file in the /export directory

2. Attach the /opt/oracle.SupportTools/diagnostics.iso file from any healthy database server as virtual media to the ILOM of the Oracle Virtual Server to be restored.
   The following example shows how to set up a virtual CD-ROM using the ILOM interface:
   1. Copy the diagnostics.iso file to a directory on the machine that will be using the ILOM interface.
   2. Log in to the ILOM web interface.
   3. In the Oracle ILOM web interface, click Remote Control , and then click Redirection.
   4. Select Use Video Redirection.
   5. After the console launches, click Storage in the KVMS menu.
   6. To add a storage image, such as a DVD image, to the Storage Devices dialog box, click Add.
   7. Open the diagnostics.iso file.
      
      
      Description of the illustration GUID-A42F1AC6-8400-41D6-AE83-E778530740FE-default.jpg
      
   8. To redirect storage media from the Storage Device dialog box, select the storage media and click Connect.

      After a connection to the device has been established, the label on the Connect button in the Storage Device dialog box changes to Disconnect.

   9. Select Host Control from the Host Management tab.
   10. Select CDROM as the next boot device from the list of values.
   11. Click Save.

       When the system is booted, the diagnostics.iso image is used.

3. Restart the system from the ISO image file.

   You can restart the system using one of the following methods:

   • Choose the CD-ROM as the boot device during startup

   • Preset the boot device by running the ipmitool command from any other machine that can reach the ILOM of the Oracle Virtual Server to be restored:

     # ipmitool -H ILOM_ip_address_or_hostname -U root chassis bootdev cdrom
     # ipmitool -H ILOM_ip_address_or_hostname -U root chassis power cycle
     

4. Login to the diagnostics shell as the root user.

   When the system displays the following:

   Choose from following by typing letter in '()':
   (e)nter interactive diagnostics shell. Must use credentials from Oracle support to login (reboot or power cycle to exit the shell),
   (r)estore system from NFS backup archive,
   

   Type e to enter the diagnostics shell, and log in as the root user.

   Note:

   If you do not have the password for the root user, then contact Oracle Support Services.
5. If required, use /opt/MegaRAID/MegaCli/MegaCli64 to configure the disk controller to set up the disks.
6. Remove the logical volumes, the volume group, and the physical volume, in case they still exist after the disaster.

   # lvm vgremove VGExaDb --force
   # lvm pvremove /dev/sda2 --force
   

7. Remove the existing partitions and clean up the drive.

   # parted
   GNU Parted 2.1
   Using /dev/sda
   Welcome to GNU Parted! Type 'help' to view a list of commands.
   (parted) rm 1 
   [12064.253824] sda: sda2
   (parted) rm 2 
   [12070.579094] sda: 
   (parted) q
   
   # dd if=/dev/zero of=/dev/sda bs=64M count=2
   

8. Create the two partitions on /dev/sda.
   1. Get the end sector for the disk /dev/sda from a surviving dom0 and store it in a variable:

      # end_sector_logical=$(parted -s /dev/sda unit s print|perl -ne '/^Disk\s+\S+:\s+(\d+)s/ and print $1')
      # end_sector=$( expr $end_sector_logical - 34 )
      

      The values for the start and end sectors in the commands below were taken from a surviving dom0. Because these values can change over time, it is recommended that these values are checked from a surviving dom0 using the following command:

      # parted -s /dev/sda unit S print
      

   2. Create the boot partition, /dev/sda1.

      # parted -s /dev/sda mklabel gpt mkpart primary 64s 1048639s set 1 boot on
      

   3. Create the partition that will hold the LVMs, /dev/sda2.

      # parted -s /dev/sda mkpart primary 1048640s 3509759966s set 2 lvm on
      

9. Use the /sbin/lvm command to re-create the logical volumes and mkfs to create file systems.
   1. Create the physical volume and the volume group.

      # lvm pvcreate /dev/sda2
      # lvm vgcreate VGExaDb /dev/sda2
      

   2. Create the logical volume for the file system that will contain the / (root) directory and label it.

      # lvm lvcreate -n LVDbSys3 -L30G VGExaDb
      # mkfs -t ext4 –b 4096 /dev/VGExaDb/LVDbSys3
      # e2label /dev/VGExaDb/LVDbSys3 DBSYSOVS
      

   3. Create the logical volume for the swap directory, and label it.

      # lvm lvcreate -n LVDbSwap1 -L24G VGExaDb
      # mkswap -L SWAP /dev/VGExaDb/LVDbSwap1
      

   4. Create the logical volume for the backup partition, and build a file system on top of it.

      # lvm lvcreate -n LVDbSys2 -L30G VGExaDb
      # mkfs -t ext4 –b 4096 /dev/VGExaDb/LVDbSys2
      

   5. Create the logical volume for the reserved partition.

      # lvm lvcreate -n LVDoNotRemoveOrUse –L1G VGExaDb
      

      Note:

      Do not create any file system on this logical volume.
   6. Create the logical volume for the guest storage repository.

      # lvm lvcreate -l 100%FREE -n LVDbExaVMImages VGExaDb
      

   7. Create a file system on the /dev/sda1 partition, and label it.

      In the mkfs.ext3 command below, the -I 128 option is needed to set the inode size to 128.

      # mkfs.ext3 -I 128 /dev/sda1
      # tune2fs -c 0 -i 0 /dev/sda1
      # e2label /dev/sda1 BOOT
      

10. Create mount points for all the partitions, and mount the respective partitions.

    For example, if /mnt is used as the top-level directory, the mounted list of partitions might look like:

    • /dev/VGExaDb/LVDbSys3 on /mnt

    • /dev/sda1 on /mnt/boot

    The following example mounts the root file system, and creates two mount points:

    # mount /dev/VGExaDb/LVDbSys3 /mnt -t ext4
    # mkdir /mnt/boot
    # mount /dev/sda1 /mnt/boot -t ext3
    

11. Bring up the network on eth0 and assign the host's IP address and netmask to it.

    # ifconfig eth0 ip_address_for_eth0 netmask netmask_for_eth0 up
    # route add -net 0.0.0.0 netmask 0.0.0.0 gw gateway_ip_address
    

12. Mount the NFS server holding the backups.

    # mkdir -p /root/mnt
    # mount -t nfs -o ro,intr,soft,proto=tcp,nolock nfs_ip:/location_of_backup /root/mnt
    

13. From the backup which was created in "Backing up the Management Domain dom0 Using Snapshot-Based Backup", restore the root directory (/) and the boot file system.

    # tar -pjxvf /root/mnt/backup-of-root-and-boot.tar -C /mnt
    

14. Unmount the restored /dev/sda1 partition, and remount it on /boot.

    # umount /mnt/boot
    # mkdir -p /boot
    # mount /dev/sda1 /boot -t ext3
    

15. Set up the grub boot loader using the command below:

    # grub --device-map=/boot/grub/device.map << DOM0_GRUB_INSTALL
    root (hd0,0)
    setup (hd0)
    quit
    DOM0_GRUB_INSTALL
    

16. Unmount the /boot partition.

    # umount /boot
    

17. Detach the diagnostics.iso file.

    This can be done by clicking Disconnect on the ILOM web interface console, where you clicked Connect in step 2.h to attach the DVD ISO image.

18. Check the restored /etc/fstab file and remove any reference to /EXAVMIMAGES.

    # cd /mnt/etc
    

    Comment out any line that references /EXAVMIMAGES.

19. Restart the system.

    # reboot
    

    This completes the restoration procedure for the Oracle VS/dom0.

20. Convert to eighth rack, if required.

    If the recovery is on an Oracle Exadata Database Machine eighth rack, then perform the procedure described in "Configuring Oracle Exadata Database Machine Eighth Rack Oracle Linux Database Server After Recovery"".

21. When the server comes back up, build an OCFS2 file system on the LVDbExaVMImages logical volume, which was created in step 9.f.

    # mkfs -t ocfs2 -L ocfs2 -T vmstore --fs-features=local /dev/VGExaDb/LVDbExaVMImages --force
    

22. Mount the OCFS2 partition on /EXAVMIMAGES.

    # mount -t ocfs2 /dev/VGExaDb/LVDbExaVMImages /EXAVMIMAGES
    

23. In /etc/fstab, uncomment the commented out references to /EXAVMIMAGES and /dev/mapper/VGExaDb-LVDbExaVMImages that was performed in step 18.
24. Mount the backup NFS server that holds the storage repository (/EXAVMIMAGES) backup to restore the /EXAVMIMAGES file system.

    # mkdir -p /root/mnt
    # mount -t nfs -o ro,intr,soft,proto=tcp,nolock nfs_ip:/location_of_backup /root/mnt
    

25. Restore the /EXAVMIMAGES file system.

    # tar -Spxvf /root/mnt/backup-of-exavmimages.tar -C /EXAVMIMAGES
    

26. Bring up each user domain.

    # xm create /EXAVMIMAGES/GuestImages/user_domain_hostname/vm.cfg
    

At this point all the user domains should come up along with Oracle Grid Infrastructure and the database instances, and the database instances should join the Oracle RAC cluster formed by the other surviving Oracle Virtual Server nodes.

2.15.2 Scenario 2: Reimaging dom0 and Restoring User Domains from Backups

The following procedure can be used when the OVS/dom0 is damaged beyond repair and no backup exists for the dom0, but there is a backup available of the storage repository (/EXAVMIMAGES file system) housing all the user domains. This procedure reimages the dom0 and reconstructs all the user domains.

1. Re-image the OVS with the image used in the other OVS/dom0's in the rack using the procedure described in "Re-Imaging Oracle Linux Database Servers".

2. Run the following commands:

   # /opt/oracle.SupportTools/switch_to_ovm.sh
   
   # /opt/oracle.SupportTools/reclaimdisks.sh –free –reclaim
   

3. If the recovery is on Oracle Exadata Database Machine eighth rack, then perform the procedure described in "Configuring Oracle Exadata Database Machine Eighth Rack Oracle Linux Database Server After Recovery".

4. Rebuild the ocfs2 file system on the /dev/sda3 partition.

   # umount /EXAVMIMAGES
   
   # mkfs -t ocfs2 -L ocfs2 -T vmstore --fs-features=local /dev/sda3 --force
   

5. Mount the ocfs2 partition /dev/sda3 on /EXAVMIMAGES.

   # mount -t ocfs2 /dev/sda3 /EXAVMIMAGES
   

6. Mount the backup NFS server to restore the /EXAVMIMAGES file system which holds the user domain images:

   # mkdir -p /remote_FS
   
   # mount -t nfs -o ro,intr,soft,proto=tcp,nolock nfs_ip:/location_of_backup /remote_FS
   

7. Restore the /EXAVMIMAGES file system.

   # tar -Spxvf /remote_FS/backup-of-exavmimages.tar -C /EXAVMIMAGES
   

   Note:

   The restore process of storage repository restores the user domain specific files (files under /EXAVMINAGES/GuestImages/<user_domain>/) as regular files and not as ocfs2 reflinks which is what these files in the storage repository originally got created as at the time of user domain creation. Consequently, the space usage in /EXAVMINAGES may go up after the restoration process compared to the original space usage at the time of the backup.

8. Set up the network bridges manually.

   1. Determine the version of the ovmutils rpm:

      # rpm -qa|grep ovmutils
      

   2. If the version of the ovmutils rpm is earlier than 12.1.2.2.0, perform these steps:

      • Back up /opt/exadata_ovm/exadata.img.domu_maker. You will need the backup copy later.

        # cp /opt/exadata_ovm/exadata.img.domu_maker /opt/exadata_ovm/exadata.img.domu_maker-orig
        

      • Open the /opt/exadata_ovm/exadata.img.domu_maker file in a text editor such as vi, and search for "g_do_not_set_bridge=yes". It should be located a few lines below the case statement option "network-discovery)".

        Change it to "g_do_not_set_bridge=no".

        Save and exit /opt/exadata_ovm/exadata.img.domu_maker.

      • Run /opt/exadata_ovm/exadata.img.domu_maker manually for every xml file in the /EXAVMIMAGES/conf directory.

        # cd /EXAVMIMAGES/conf
        # ls -1|while read file; do /opt/exadata_ovm/exadata.img.domu_maker network-discovery $file /tmp/netdisc-$file; done
        

      • Restore /opt/exadata_ovm/exadata.img.domu_maker from the backup copy.

        # cp /opt/exadata_ovm/exadata.img.domu_maker-orig /opt/exadata_ovm/exadata.img.domu_maker
        

   3. If the version of the ovmutils rpm is 12.1.2.2.0 or later, run the following command:

      # /opt/exadata_ovm/exadata.img.domu_maker add-bonded-bridge-dom0 vmbondeth0 eth4 eth5
      

9. For each user domain directory in the /EXAVMIMAGES/GuestImages directory, perform the steps below.

   1. Get the UUID of the user domain.

      # grep ^uuid /EXAVMIMAGES/GuestImages/<user domain hostname>/vm.cfg|awk -F"=" '{print $2}'|sed s/"'"//g|sed s/" "//g
      

      The command returns the uuid value, which is used in the commands below.

   2. # mkdir -p /OVS/Repositories/uuid

   3. # ln -s /EXAVMIMAGES/GuestImages/user_domain_hostname/vm.cfg /OVS/Repositories/uuid/vm.cfg

   4. # ln -s /OVS/Repositories/uuid/vm.cfg /etc/xen/auto/user_domain_hostname.cfg

   5. # mkdir VirtualDisks

   6. # cd VirtualDisks

   7. Create four symbolic links in this directory using the four disk image names in the vm.cfg file, pointing to the four ".img" files in /EXAVMIMAGES/GuestImages/user_domain_hostname directory.

      For example, below is a sample disk entry in a sample vm.cfg file in a /OVS/Repositories/uuid directory:

      disk =  ['file:/OVS/Repositories/6e7c7109c1bc4ebba279f84e595e0b27/VirtualDisks/dfd641a1c6a84bd69643da704ff98594.img,xvda,w','file:/OVS/Repositories/6e7c7109c1bc4ebba279f84e595e0b27/VirtualDisks/d349fd420a1e49459118e6a6fcdbc2a4.img,xvdb,w','file:/OVS/Repositories/6e7c7109c1bc4ebba279f84e595e0b27/VirtualDisks/8ac470eeb8704aab9a8b3adedf1c3b04.img,xvdc,w','file:/OVS/Repositories/6e7c7109c1bc4ebba279f84e595e0b27/VirtualDisks/333e7ed2850a441ca4d2461044dd0f7c.img,xvdd,w']
      

      You can list the four ".img" files in the /EXAVMIMAGES/GuestImages/user_domain_hostname directory:

      ls /EXAVMIMAGES/GuestImages/user_domain_name/*.img
      

      /EXAVMIMAGES/GuestImages/user_domain_name/System.img
      /EXAVMIMAGES/GuestImages/user_domain_name/grid12.1.0.2.2.img
      /EXAVMIMAGES/GuestImages/user_domain_name/db12.1.0.2.2-3.img
      /EXAVMIMAGES/GuestImages/user_domain_name/pv1_vgexadb.img
      

      In this case, the commands below may be used to create the four symbolic links where dbm01db08vm01 is the user domain hostname:

      # ln -s /EXAVMIMAGES/GuestImages/dbm01db08vm01/System.img $(grep ^disk /EXAVMIMAGES/GuestImages/dbm01db08vm01/vm.cfg|awk -F":" '{print $2}'|awk -F"," '{print $1}'|awk -F"/" '{print $6}')
      
      # ln -s /EXAVMIMAGES/GuestImages/dbm01db08vm01/grid12.1.0.2.2.img $(grep ^disk /EXAVMIMAGES/GuestImages/dbm01db08vm01/vm.cfg|awk -F":" '{print $3}'|awk -F"," '{print $1}'|awk -F"/" '{print $6}')
      
      # ln -s /EXAVMIMAGES/GuestImages/dbm01db08vm01/db12.1.0.2.2-3.img $(grep ^disk /EXAVMIMAGES/GuestImages/dbm01db08vm01/vm.cfg|awk -F":" '{print $4}'|awk -F"," '{print $1}'|awk -F"/" '{print $6}')
      
      # ln -s /EXAVMIMAGES/GuestImages/dbm01db08vm01/pv1_vgexadb.img $(grep ^disk /EXAVMIMAGES/GuestImages/dbm01db08vm01/vm.cfg|awk -F":" '{print $5}'|awk -F"," '{print $1}'|awk -F"/" '{print $6}')
      

10. Bring up each user domain.

    # xm create /EXAVMIMAGES/GuestImages/user_domain_hostname/vm.cfg
    

At this point all the user domain should come up along with the Grid Infrastructure and the database instances in them and join the Oracle RAC cluster formed by the other surviving OVS nodes.

2.15.3 Scenario 3: Restoring and Recovering User Domains from Snapshot Backups

Use this procedure to restore lost or damaged files of a user domain using a snapshot-based user domain backup taken from inside a user domain. The user domain backup was created using the procedure described in "Method 2: Back up a User Domain from Inside the User Domain".

1. Log in to the user domain as the root user.
2. Mount the backup NFS server to restore the damaged or lost files.

   # mkdir -p /root/mnt
   
   # mount -t nfs -o ro,intr,soft,proto=tcp,nolock nfs_ip:/location_of_backup /root/mnt
   

3. Extract the damaged or lost files from the backup to a staging area.

   Prepare a staging area to hold the extracted files. The backup LVM LVDbSys2 can be used for this:

   # mkdir /backup-LVM
   
   # mount /dev/mapper/VGExaDb-LVDbSys2 /backup-LVM
   
   # mkdir /backup-LVM/tmp_restore
   
   # tar -pjxvf /root/mnt/tar_file_name -C /backup-LVM/tmp_restore absolute_path_of_file_to_be_restored
   

4. Restore the damaged or lost files from the temporary staging area as needed.
5. Reboot the user domain.

2.16 Updating Oracle Linux Database Servers

Starting with Oracle Exadata Storage Server Software release 11.2.3.1.0, the database server update procedure uses the Unbreakable Linux Network (ULN) for distribution of database operating system and firmware updates. Starting with Exadata Storage Server Software release 12.1.2.2.0, you can run patchmgr to orchestrate updates.

The updates are distributed as multiple rpms. The dbnodeupdate.sh utility is used to apply the updates. The utility performs the following tasks:

• Uses the built-in dbserver_backup.sh script to perform a backup of the file system hosting the operating system before updating the server when the database servers use a Logical Volume Manager (LVM) configuration.

• Automates all preparation, update, and validation steps including the stopping of the databases and Grid Infrastructure stack, Oracle Enterprise Manager Cloud Control agents, and unmounting remote network mounts that might exist.

• Applies Oracle best practices and workarounds for the latest known issues.

• Verifies that the update was successful, relinks the Oracle binaries, and starts the Oracle stack.

For environments that cannot use ULN, an ISO repository for each Oracle Exadata Storage Server Software release is available.

The dbnodeupdate.sh utility is available from My Oracle Support note 1553103.1. The utility supports all hardware generations, Oracle Exadata database servers running Oracle Virtual Server (dom0) and Exadata Virtual Machines (domU). The dbnodeupdate.sh utility also supports updates from Oracle Linux 5 to Oracle Linux 6. The Oracle Exadata Storage Server Software patch README files specify if the patch itself is applicable for a particular hardware generation. Always review the support note to ensure that the latest release of the utility is used for the update.

Note:

The ISO image method is not intended to be a complete replacement of a local ULN mirror. The ISO image contains only the Oracle Exadata channel content. No other content from other ULN channels is in the ISO image. If your environment uses other Linux package for which updates are not contained in the Oracle Exadata channel, then you should also update those packages using ULN channels, as needed.

This section contains the following topics:

• Paths for Updating Database Servers

• Overview of Oracle Linux Database Servers Updates

• Preparing and Populating the yum Repository with the Oracle Exadata Channel Content

• Reducing Downtime of Database Updates

• Updating Database Servers Previously Patched or Imaged to Release 11.2.3.n.n or 12.1.n.n.n

• Updating Database Servers Running Release 11.2.2.4.2

• Rolling Back Updates on the Database Servers

• Updating Database Nodes with patchmgr

• Running dbnodeupdate.sh and patchmgr (and dbnodeupdate Orchestration) Using sudo

2.16.1 Paths for Updating Database Servers

Database servers running Oracle Linux 5.5 or later, and Oracle Exadata Storage Server Software release 11.2.2.4.2, or later must use the dbnodeupdate.sh utility to update the database servers.

Oracle Linux database servers running Oracle Exadata Storage Server Software releases earlier than 11.2.2.4.2 should first update to Oracle Linux 5.5 and Oracle Exadata Storage Server Software 11.2.2.4.2 or later before using the dbnodeupdate.sh utility.

The following table shows the update paths for the database servers based on the Oracle Exadata Storage Server Software and Oracle Linux releases.

Note:

If output from the uname -r command on the database server shows a kernel release that is earlier than or equal to release 2.6.18-128.1.16.0.1, then the database server is running Oracle Linux 5.3.

Table 2-10 Update Paths for Database Servers

Currently Installed Oracle Exadata Storage Server Software Release	Currently Installed Oracle Linux Release	Recommended Action
Release 11.2.2.4.2 or later	Release 5.5. or later	Update to the target release as described in "Overview of Oracle Linux Database Servers Updates."
Releases earlier than 11.2.2.4.2.	Release 5.5 or later	Use patch 13513611 to update the database servers to release 11.2.2.4.2, using the steps in section 8 of the patch README. Update to a later release as described in "Overview of Oracle Linux Database Servers Updates."
Releases earlier than 11.2.2.4.2	Releases earlier than 5.5	Use My Oracle Support note 1284070.1 to update to release 11.2.3.2.0. Update to a later release as described in "Overview of Oracle Linux Database Servers Updates."

Note:

• Oracle Exadata database servers currently running a non-UEK kernel on releases 11.2.3.2.0 and later, updating to a release earlier than release 11.2.3.3.0, the database servers will remain on a non-UEK kernel when using the dbnodeupdate.sh utility.

• When there is a requirement to use the UEK kernel in a release earlier than release 11.2.3.3.0, and when currently on a non-UEK kernel, then use the dbnodeupdate.sh utility to do the update and later convert manually to UEK after the update is complete. Refer to My Oracle Support note 1524411.1 for additional information.

• Starting with Oracle Exadata release 11.2.3.3.0 only UEK kernels are supported. For this reason, the dbnodeupdate.sh utility will update any non-UEK system directly to the UEK2 kernel when updating to 11.2.3.3.0 or later.

• Regular Oracle Exadata Database Server updates as described in My Oracle Support note 888828.1 can only be applied when the date-stamp of the target release is newer than the date-stamp of the release currently running. Updates from images running a release with a date-stamp newer than the target release will be blocked.

  For example: Updating 12.1.1.1.2.150411 to 12.1.2.1.0.141206.1 is blocked because 12.1.1.1.2.1 (with date-stamp 150411) has a date-stamp newer than 12.1.2.1.0 (141206), although Exadata release 12.1.2.1.0 seems to be more recent than 12.1.1.1.2 based on the release number. In situations like this, you should update to a maintenance release that has a newer date-stamp for the same major release.

Updating individual non-Exadata operating system packages

If you want to update individual non-Exadata operating system packages, you can set up a local yum mirror of the Oracle Linux "latest" channel for your system and run "yum update <packagename>".

Always be careful of any other packages that might be pulled in as well.

You may have to remove the exadata-sun-computenode-exact rpm first:

rpm -e exadata-sun-computenode-exact

If you do not have a local yum mirror, then download the individual packages from ULN or public-yum, and update the specific package using:

rpm -uvh package-names

Caution:

Never run the yum updatecommand without specifying any package name, because this will cause many packages and dependencies to be pulled in, making unintentional updates, which can make the Exadata system unusable and make it much harder or impossible to update to the next Exadata release.

2.16.2 Overview of Oracle Linux Database Servers Updates

This topic provides a high-level overview of the procedure to update the Oracle Linux database servers to a new release.

The information in this section applies to database servers that meet the following requirements:

• Oracle Exadata Storage Server Software 11g Release 2 (11.2) 11.2.2.4.2 or later is installed.

• Oracle Linux 5.5 or later is installed. Having Oracle Exadata Storage Server Software 11g Release 2 (11.2) 11.2.3.1.0 and later satisfies this requirement.

If the database servers do not meet the preceding requirements, then refer to Table 2-10 for the prerequisite steps that must be done to meet the requirements.

The following procedure describes how to update the Oracle Linux database servers to a new release:

1. Prepare and populate the yum repository with the Oracle Exadata channel content for the target release. The ULN channel name to use for the release is listed in the specific README file for the patch. When not using a ULN channel, an ISO image can be used.

2. Perform the prerequisite check, update, and post-patching steps, depending on the current Oracle Exadata Storage Server Software release as follows:

   • Oracle Database running with Oracle Exadata Storage Server Software release 11.2.3.1.0 and later: "Updating Database Servers Previously Patched or Imaged to Release 11.2.3.n.n or 12.1.n.n.n."

   • Oracle Database running with Oracle Exadata Storage Server Software release 11.2.2.4.2: "Updating Database Servers Running Release 11.2.2.4.2."

2.16.3 Preparing and Populating the yum Repository with the Oracle Exadata Channel Content

The following methods can be used to create and use a yum repository containing the Oracle Exadata channel content. The dbnodeupdate.sh utility then uses the yum repository to update the database servers in Oracle Exadata Database Machine. Each method is described in detail in the following sections.

• Setting Up a Local Mirror of the ULN Oracle Exadata Channel on a Separate Server

• Downloading the ISO Image of the ULN Oracle Exadata Channel and Passing the Location of the Compressed Image to the dbnodeupdate.sh Utility

• Downloading the ISO Image and Making It Available on a Web Server

2.16.3.1 Setting Up a Local Mirror of the ULN Oracle Exadata Channel on a Separate Server

This method is recommended for the following conditions:

• There is a large number of database servers to update.

• The single repository server is accessible to all the database servers.

• An infrastructure exists for building a local ULN mirror on a separate Linux server.

• Database servers have customized software that requires updates from other ULN channels.

The mirror is a separate Linux server that holds the downloaded channel data with the Oracle Exadata release from ULN that you want to update to. The database servers connect to the local mirror (repository) to retrieve updates.

Caution:

Do not use a database server as the local ULN mirror. Doing so may lead to system failures due to dependency conflicts between the packages required by ULN to construct the repository, and the packages installed or updated on the database server. If a separate server is not available, then use the ISO image method.

When setting up a local repository for the first time, the additional required Linux subscriptions depend on the Enterprise Linux or Oracle Linux release being used on the server hosting the repository. The server should subscribe to the following additional ULN channels in order for the repositories to be built:

• For 64-bit Enterprise Linux 4 systems:

  el4_x86_64_latest, and el4_x86_64_addons are required.

• For 64-bit Enterprise Linux 5 systems:

  el5_x86_64_latest, and el5_x86_64_addons are required.

• For 64-bit Oracle Linux 5 systems:

  ol5_x86_64_latest, and el5_x86_64_addons are required.

• For 64-bit Oracle Linux 6 systems:

  ol6_x86_64_latest, and ol6_x86_64_addons are required.

2.16.3.1.1 Creating and Maintaining a Local ULN Mirror

This procedure describes how to create and maintain a local ULN mirror as the yum repository.

1. Review the prerequisites and server setup procedure as described in "How to create a local Unbreakable Linux Network mirror".
2. Log in to ULN at Unbreakable Linux Network.
3. Click the Systems tab.
4. Navigate to the registered yum repository servers.
5. Click Manage Subscriptions.
6. Add the channels named in the Oracle Exadata patch README, such as exadata_dbserver_12.1.1.1.0_x86_64_base.

   Note:

   The Oracle Exadata channel to subscribe to depends on the Oracle Exadata Storage Server Software release you want to update to. The patch README files for that release always have the details.

7. Click Save.

2.16.3.1.2 Verifying the Local ULN Mirror

The yum mirror creation procedure described in the previous section provides a script that is used to populate the local mirror. Running the script starts the rpms download from ULN to a local directory.

The web server's Document Root directory should be configured to point to this location on disk so that the Oracle Exadata channel content is made available to the Oracle Exadata Linux Database Servers using the HTTP web server.

The URL specified as the repository location to the dbnodeupdate.sh utility should always be at the same level as the repodata directory, as shown in the following graphic:

Description of the illustration GUID-B6201246-BE86-45AC-B10C-FD0738B75567-default.png

The following command shows how to verify the repository as the root user:

[root@dm01 ]# ./dbnodeupdate.sh -u -l http://yum-repo/yum/EngineeredSystems/exadata/dbserver/12.1.1.1.0/base/x86_64/ -v

Note:

The database node must be running Exadata release 12.1.2.2.0 or later to access yum repositories listening only on IPv6 addresses.

2.16.3.2 Downloading the ISO Image of the ULN Oracle Exadata Channel and Passing the Location of the Compressed Image to the dbnodeupdate.sh Utility

Starting with release 11.2.3.1.0 and later, an ISO image of the Oracle Exadata channel is available as a downloadable patch from My Oracle Support. The location of the compressed file can be passed directly to the dbnodeupdate.sh utility. This method is recommend for the following conditions:

• There is no separate server available to be used as a repository server.

• The database servers do not have customized software that requires updates from additional ULN channels.

• The simplest method is preferred.

Note:

The ISO image method is not intended to be a complete replacement of a local ULN mirror. The ISO image contains only the Oracle Exadata channel content. No other content from other ULN channels is in the ISO image. If your environment uses other Linux package for which updates are not contained in the Oracle Exadata channel, then you should also update those packages using ULN channels, as needed.

The following procedure describes how to download and verify the ISO image:

1. Download the ISO image from My Oracle Support for the Oracle Exadata release you want to update to.
2. Copy the ISO image, in compressed file format, to every database server to be updated. The path to the compressed file is given as an argument to the dbnodeupdate.sh utility. Do not uncompress the file.
3. Run the following command as the root user to verify the repository:

   [root@dm01 ]# ./dbnodeupdate.sh -u -l                     \
   /u01/dbnodeupdate/p17997668_121110_Linux-x86-64.zip -v
   

2.16.3.3 Downloading the ISO Image and Making It Available on a Web Server

Starting with release 11.2.3.1.0 and later, an ISO image of the Oracle Exadata channel is available as a downloadable patch from My Oracle Support. It can be downloaded, uncompressed, and put on web server that has HTTP connectivity to every Oracle Exadata Database Machine database server. Refer to My Oracle Support note 888828.1 for links to the release-specific patch information. This method is recommend for the following conditions:

• There is a large number of database servers to update.

• The single repository server is accessible to all the database servers.

• The database servers do not have customized software that requires updates from additional ULN channels.

The following procedure describes how to make the ISO image of the Oracle Exadata channel content available on a Linux-based server running Apache HTTP Server. Other operating systems can be used, but the instructions are not included in this document. In the procedure, the ISO image is copied to the /u01/app/oracle/stage directory, and Document Root entry in the web server document is /var/www/html.

Note:

Oracle recommends that a separate server be used as the web server. A database server in Oracle Exadata Database Machine can be used, but the httpd package is not a standard Oracle Exadata package. If a database server will be used, then Apache HTTP Server must be installed on the server before performing the following procedure.

1. Download the compressed ISO image from My Oracle Support for the Oracle Exadata release to be updated. The steps show an update to release 12.1.1.1.0, but the steps are also applicable to releases 11.2.3.1.0 and later.

2. Copy the ISO image to the web server.

3. Create the ISO image mount point as the root user using the following command:

   [root@dm01 ]# mkdir -p /var/www/html/yum/unknown/EXADATA/dbserver/12.1.1.1.0/base
   

4. Uncompress and mount the ISO image as the root user using the following commands:

   [root@dm01 ]# cd /u01/dbnodeupdate
   [root@dm01 ]# unzip p17997668_121110_Linux-x86-64.zip
   [root@dm01 ]# mount -o loop /u01/dbnodeupdate/121110_base_repo.iso /var/www/html/yum/unknown/EXADATA/dbserver/12.1.1.1.0/base
   

5. Start the httpd service as the root user as follows:

   1. Add and enable the httpd server using the following commands:

      [root@dm01 ]# chkconfig --add httpd 
      [root@dm01 ]# chkconfig httpd on
      

      Note:

      It is not required to enable automatic start up of the httpd service.

   2. Start the httpd service using the following command:

      [root@dm01 ]# service httpd start
      

   3. Verify the httpd service is enabled using the following command:

      [root@dm01 ]# chkconfig --list httpd
      

6. Identify and test the repository URL by connecting to it using a web browser. The following is an example of the repository URL.

   http://yum-repo/yum/unknown/EXADATA/dbserver/12.1.1.1.0/base/x86_64/
   

7. Run the following command as the root user to verify the repository:

   [root@dm01 ]# ./dbnodeupdate.sh -u -l  \
   http://yum-repo/yum/EngineeredSystems/exadata/dbserver/12.1.1.1.0/base/x86_64/ -v
   

2.16.4 Reducing Downtime of Database Updates

Downtime can be reduced by making a backup before and running the prerequisite checks before starting the update.

When the actual update occurs, the backup can be skipped. Backups can only be made when no remote network mounts, such as NFS, are active. The prerequisite checks should always be run days before starting any downtime for the actual update.

Note:

When updating Logical Volume Manager (LVM)-enabled systems from Oracle Linux 5 to Oracle Linux 6, a backup is done as part of the actual update. The backup cannot be skipped.

All warnings and failures should be addressed before taking the downtime. Note the following when updating the database servers:

• Updating database servers previously patched or imaged to release 11.2.3.n.n or 12.1.n.n.n:

  Only minimum-dependency failures, and not having a backup of the current image before updating to Oracle Linux 6 will block the update from proceeding. When the check for minimum-dependencies fails, check the dbnodeupdate.log file in the /var/log/cellos directory for more details on why the dependencies fail. Dependency checks are not be performed when updating to Oracle Linux 6. After resolving the warnings and failures, re-run the prerequisite check.

• Updating database servers running release 11.2.2.4.2:

  Dependency checks are not performed when updating from release 11.2.2.4.2, also obsolete.lst and exclusion.lst list functionality is disabled. After resolving the warnings and failures, re-run the prerequisite check.

2.16.4.1 Performing a Backup and Prerequisite Checks Before Updating Database Servers

The database server backup and prerequisite checks can be done several days before updating the servers. The following procedure describes how to back up the active system partition and run the prerequisite check:

1. Use the following command to create a backup:

   [root@dm01 ]# ./dbnodeupdate.sh -u -l repo -b
   

   In the preceding command, repo is the HTTP location of the ULN mirror or the location of the compressed ISO image.

   Note:

   When performing the actual update, the backup step in the dbnodeupdate.sh utility can be skipped if a backup was made earlier. The following command skips backup creation:

   [root@dm01 ]# ./dbnodeupdate.sh -u -l repo -n
   

   In the preceding command, repo is the HTTP location of the ULN mirror or the location of the compressed ISO image.

   When updating LVM-enabled systems from Oracle Linux 5 to Oracle Linux 6, a backup is done as part of the actual update. The -n flag is ignored.

2. Use the following command to run the prerequisite checks:

   [root@dm01 ]# ./dbnodeupdate.sh -u -l  repo -v
   

   In the preceding command, repo is the HTTP location of the ULN mirror or the location of the compressed ISO image.

2.16.5 Updating Database Servers Previously Patched or Imaged to Release 11.2.3.n.n or 12.1.n.n.n

The dbnodeupdate.sh utility is used to update the database servers that are running release 11.2.3.n.n, and release 12.1.n.n.n. The procedures in this section assume that the database servers are running a release later than 11.2.2.4.2, and that the update is available either as local ULN mirror or as a local ISO image.

Note:

Two components are required when using the dbnodeupdate.sh utility:

• The ULN mirror or compressed ISO file of the patch

• The dbnodeupdate.sh utility

This section contains the following topics:

• Customizing Database Server Packages and Understanding Minimum and Exact Dependencies

• Preparing to Run the dbnodeupdate.sh Utility

• Reviewing the List of Obsolete Packages, List of Excluded Packages, and Automatic Package Removal

• Running the dbnodeupdate.sh Utility in Update Mode

2.16.5.1 Customizing Database Server Packages and Understanding Minimum and Exact Dependencies

System administrators are allowed to customize the operating system on the database servers by updating existing Oracle Exadata-supplied packages to newer releases or by installing additional packages as long as the customization does not impair Oracle Exadata Storage Server Software functionality.

Note:

Kernel and InfiniBand packages should not be updated or removed unless instructed by Oracle Support Services.

When updating to any release equal to or later than Oracle Exadata Storage Server Software release 11.2.3.3.0, the yum update performed using the dbnodeupdate.sh utility works with minimum and exact dependencies. These dependencies are enforced during prerequisite checking and during the update phase to help administrators stay exactly at or close to the original Exadata release when customizing the system.

The dependencies are enforced by the exadata-sun-computenode-exact rpm and the exadata-sun-computenode-minimum rpm as follows:

• The exadata-sun-computenode-exact rpm ensures that a specific release of Oracle-supplied packages is allowed during the update.

• The exadata-sun-computenode-minimum rpm ensures that a specific or later release of Oracle-supplied packages is allowed during the update.

By having the exact dependencies enforced by exadata-sun-computenode-exact rpm, the system appear as if it was freshly-imaged to the newer release because the Oracle Exadata packages are exactly as the same release.

The exadata-sun-computenode-minimum rpm sets the minimum dependencies, and enforces that all packages are installed but at the same time also allow packages to be at a later version.

By default, the dbnodeupdate.sh utility updates attempt to enforce the exact-dependencies when updating to a next release. When exact-dependencies conflict and cannot be enforced, the utility attempts to apply the exadata-sun-computenode-minimum rpm during the update to enforce minimum dependencies. Missing or not updating to the exact dependencies is allowed and not a problem. If the system needs to be updated to the exact dependencies, then the conflict needs to be resolved. Check the dbnodeupdate.log file to see what packages conflict, remove them cautiously, and then re-run the dbnodeupdate.sh utility.

Use the following command to see the exact-dependencies enforced by the release:

[root@dm01 ]# rpm -qR exadata-sun-computenode-exact   | grep '=' | grep -v '('

The output from the command shows the exact-dependencies.

For example, the output for Oracle Exadata Storage Server Software release 12.1.1.1.1 shows that the sendmail package must be exactly 8.13.8-8.1.el5_7.

Use the following command to see the minimum dependencies enforced by the release:

[root@dm01 ]# rpm -qR exadata-sun-computenode-minimum  | grep '>=' | grep -v '(' 

The output from the command shows the minimum dependencies.

For example, the output for Oracle Exadata Storage Server Software release 12.1.1.1.1 shows that the minimum sendmail package is sendmail >= 8.13.8-8.1.el5_7. This means that later releases of sendmail are allowed when not enforcing exact-dependencies.

The dbnodeupdate.sh utility confirmation screen always shows what dependencies will be enforced during the update. When both exact and minimum dependencies cannot be applied, the update cannot proceed. For such cases the dbnodeupdate.log file should be consulted to find out what caused the dependencies to fail. After removing the failed dependency, the dbnodeupdate.sh utility should be re-run to validate dependencies and ensure that the least minimum-dependencies can be enforced.

Oracle recommends running the dbnodeupdate.sh utility with the -v flag to verify prerequisites several days before the scheduled maintenance. The dbnodeupdate.sh utility log file reports dependency problems that must be resolved before proceeding with the update and taking the downtime. When dependency errors occur during the prerequisite check or before the update starts, do the following to resolve the problem:

• Analyze the requirement or yum error. For updates starting from release 11.2.3.1.0 and later, consult the files listed on the screen, including the /var/log/cellos/dbnodeupdate.log file. For updates starting from release 11.2.2.4.2, consult the /var/log/exadata.computenode.post.log, /var/log/exadata.computenode.post.log, and /var/log/cellos/bootstrap.uln.log files.

• Depending on the issue, it may be necessary to de-install, install or update the rpm packages causing the dependency issue or conflict. The dbnodeupdate.log file lists the failed dependencies.

• Perform the update using the dbnodeupdate.sh utility. Only use the -n flag to skip the backup if a backup was already made in an earlier attempt. By default, existing backups for the same image are overwritten.

  When updating LVM-enabled systems from Oracle Linux 5 to Oracle Linux 6, a backup is done as part of the actual update. The -n flag is ignored.

• Re-install custom rpm packages that had been de-installed after the update.

Note:

When manually updating individual packages on your system, for example, due to security requirements. Dependencies enforced by exadata-sun-computenode-exact may conflict. When this happens, it is supported to deinstall the exadata-sun-computenode-exact package in order to allow the installation of later packages. This can be done by running the following command:

[root@dm01 ]# rpm -e exadata-sun-computenode-exact

When a later Oracle Exadata update includes the newer package, the exact-dependency is automatically reinstalled by the dbnodeupdate.sh utility.

It is not supported to remove the exadata-sun-computenode-minimum package. Packages should not be forced in using the rpm -Uvh --nodeps command unless directed by Oracle Support Services.

2.16.5.2 Preparing to Run the dbnodeupdate.sh Utility

The dbnodeupdate.sh utility is available from My Oracle Support from “dbnodeupdate.sh and dbserver.patch.zip: Updating Exadata Database Server Software using the DBNodeUpdate Utility and patchmgr (Doc ID 1553103.1)”. The following procedure describes how to download and prepare the utility on the server.

1. Download the dbnodeupdate package file from My Oracle Support.
2. Place the zip file in the /u01/dbnodeupdate directory on the database server to be updated.

   Note:

   This step assumes the ULN repository is available as described by one of the options in "Preparing and Populating the yum Repository with the Oracle Exadata Channel Content."

3. Uncompress the dbnodeupdate package, p16486998_12xxxx_Linux-x86-64.zip, as the root user using the following command:

   [root@dm01 ]# unzip p16486998_12xxxx_Linux-x86-64.zip
   

   Do not uncompress the dbupdate-helpers.zip file.

   Note:

   A user with sudo privileges can use sudo to run the dbnodeupdate.sh utility.

2.16.5.3 Reviewing the List of Obsolete Packages, List of Excluded Packages, and Automatic Package Removal

When updating to release 11.2.3.3.0 and later, some packages on the database server become obsolete. While updating a database server, the dbnodeupdate.sh utility displays the excluded rpm list and obsolete rpm list on the confirmation screen. The following is an example of the confirmation screen. In this example, an exclusion list has not yet been created by the user.

Active Image version   : 11.2.3.2.1.130302
Active Kernel version  : 2.6.32-400.21.1.el5uek
Active LVM Name        : /dev/mapper/VGExaDb-LVDbSys2
Inactive Image version : 11.2.3.2.1.130109
Inactive LVM Name      : /dev/mapper/VGExaDb-LVDbSys1
Current user id        : root
Action                 : upgrade
Upgrading to           : 11.2.3.3.0.131014.1
Baseurl                : http://mynode.us.example.com/yum/exadata_dbserver_11.2.3.3.0_x86_64_base/x86_64/ (http)
Create a backup        : Yes
Shutdown stack         : No (Currently stack is up)
RPM exclusion list     : Not in use (add rpms to /etc/exadata/yum/exclusion.lst and restart dbnodeupdate.sh)
RPM obsolete list      : /etc/exadata/yum/obsolete.lst (lists rpms to be removed by the update)
                       : RPM obsolete list is extracted from exadata-sun-computenode-11.2.3.3.0.131014.1-1.x86_64.rpm
Logfile                : /var/log/cellos/dbnodeupdate.log (runid: 021213024645)
Diagfile               : /var/log/cellos/dbnodeupdate.021213024645.diag
Server model           : SUN FIRE X4170 M2 SERVER
Remote mounts exist    : Yes (dbnodeupdate.sh will try unmounting)
dbnodeupdate.sh rel.   : 2.20 (always check MOS 1553103.1 for the latest release)
Automatic checks incl. : Issue - Yum rolling update requires fix for 11768055 when Grid Infrastructure is below 11.2.0.2 BP12
Manual checks todo     : Issue - Database Server upgrades to 11.2.2.3.0 or higher may hit network routing issues after the upgrade
Note                   : After upgrading and rebooting run './dbnodeupdate.sh -c' to finish post steps
 
Continue ? [Y/n]

To see what packages will become obsolete, review the contents of the obsolete.lst file, enter N when prompted by the script. The location of the obsolete.lst file is displayed on the confirmation screen.The obsolete.lst file lists only those packages that will be removed during the update when no action is taken. Packages added manually to this list are ignored. The following is an example of the obsolete.lst file:

[root@dm01 ]# cat /etc/exadata/yum/obsolete.lst
# Generated by dbnodeupdate.sh runid: 021213024645
at.x86_64
java-*-openjdk
rhino.noarch
jline.noarch
jpackage-utils.noarch
giflib.x86_64
alsa-lib.x86_64
xorg-x11-fonts-Type1.noarch
prelink.x86_64
biosconfig
biosconfig_expat
qlvnictools
ibvexdmtools
opensm.x86_64
ofed-scripts
ibibverbs-devel-static
infiniband-diags-debuginfo
libibverbs-debuginfo
librdmacm-debuginfo
libibumad-debuginfo
libibmad-debuginfo
ibutils-debuginfo
libmlx4-debuginfo
libsdp-debuginfo
mstflint-debuginfo
perftest-debuginfo
qperf-debuginfo
libibcm-debuginfo
compat-dapl-utils.x86_64
compat-dapl.x86_64
dapl-utils.x86_64
dapl.x86_64
libgfortran.x86_64
mdadm.x86_64
mpi-selector.noarch
pygobject2.x86_64
specspo.noarch
time.x86_64
tree.x86_64
unix2dos.x86_64
usbutils.x86_64
words.noarch

To prevent a package listed in the obsolete.lst file from being removed, create the /etc/exadata/yum/exclusion.lst file, and put the rpm name, or wildcard, for the rpms to keep in the exclusion list.

The following example shows a package added to the exclusion list:

[root@dm01 ]# cat /etc/exadata/yum/exclusion.lst 
java-*-openjdk

After adding an entry to the exclusion.lst file, and re-running the dbnodeupdate.sh utility, the exclusion list is detected by the utility. The rpm packages on the exclusion list are still shown in the obsolete.lst file, but the listed packages in the exclusion.lst file are not removed during the update. The following example of the confirmation screen shows the exclusion.lst file is in use after an entry has been added to it.

Active Image version   : 11.2.3.2.1.130302
Active Kernel version  : 2.6.32-400.21.1.el5uek
Active LVM Name        : /dev/mapper/VGExaDb-LVDbSys2
Inactive Image version : 11.2.3.2.1.130109
Inactive LVM Name      : /dev/mapper/VGExaDb-LVDbSys1
Current user id        : root
Action                 : upgrade
Upgrading to           : 11.2.3.3.0.131014.1
Baseurl                : http://mynode.us.example.com/yum/exadata_dbserver_11.2.3.3.0_x86_64_base/x86_64/ (http)
Create a backup        : Yes
Shutdown stack         : No (Currently stack is up)
RPM exclusion list     : In use (rpms listed in /etc/exadata/yum/exclusion.lst)
RPM obsolete list      : /etc/exadata/yum/obsolete.lst (lists rpms to be removed by the update)
                       : RPM obsolete list is extracted from exadata-sun-computenode-11.2.3.3.0.131014.1-1.x86_64.rpm
Logfile                : /var/log/cellos/dbnodeupdate.log (runid: 021213024900)
Diagfile               : /var/log/cellos/dbnodeupdate.021213024900.diag
Server model           : SUN FIRE X4170 M2 SERVER
Remote mounts exist    : Yes (dbnodeupdate.sh will try unmounting)
dbnodeupdate.sh rel.   : 2.20 (always check MOS 1553103.1 for the latest release)
Automatic checks incl. : Issue - Yum rolling update requires fix for 11768055 when Grid Infrastructure is below 11.2.0.2 BP12
Manual checks todo     : Issue - Database Server upgrades to 11.2.2.3.0 or higher may hit network routing issues after the upgrade
Note                   : After upgrading and rebooting run './dbnodeupdate.sh -c' to finish post steps
 
Continue ? [Y/n]

Note:

Neither the dbnodeupdate.sh utility or Oracle Exadata Storage Server Software upgrade process remove rpm packages not listed in the obsolete.lst file. Only files listed in the obsolete.lst file can be added to the exclusion.lst file. Custom packages are not removed. The obsolete.lst and exclusion.lst functionality is not available when updating from release 11.2.2.4.2, or when updating to Oracle Linux 6.

2.16.5.4 Running the dbnodeupdate.sh Utility in Update Mode

The following procedure shows how to run the dbnodeupdate.sh utility in update mode.

1. Run the dbnodeupdate.sh utility using one of the following commands, depending on the location of the repository:

   • Using a local ULN mirror:

     [root@dm01 ]# ./dbnodeupdate.sh -u -l http://my.repository.url/x86_64
     

   • Using the compressed ISO image on the database servers:

     [root@dm01 ]# ./dbnodeupdate.sh -u -l /u01/my_iso_file_zipped.zip
     

     In the preceding command, my_iso_file_zipped is the name of the compressed ISO file.

   Note:

   If the database and Grid Infrastructure stack are active, then they need to be shut down before updating the database server. Add the additional -s flag to the update command to have the dbnodeupdate.sh utility shut down the stack during the update process. The following shows the command:

   [root@dm01 ]# ./dbnodeupdate.sh -u -l repo -s
   

2. Complete the update by running the post-completion steps when the server has finished rebooting using the following command:

   [root@dm01 ]# ./dbnodeupdate.sh -c
   

   The utility proceeds with the post-completion steps.

   Note:

   The update process might still be running when the dbnodeupdate.sh -c command is entered. When this happens, the utility waits until it can determine the image status. The system may restart while waiting to make pending updates effective. If this happens, then re-enter the dbnodeupdate.sh -c command when the system is back online.

2.16.5.4.1 Example of Running the dbnodeupdate.sh Utility

This topic provides an example of using the dbnodeupdate.sh utility.

In this example:

• The system is currently running Oracle Exadata Storage Server Software release 11.2.3.3.1 (shown in the Active Image version entry).

• The system is using kernel 2.6.39-400.128.17.el5uek (Active Kernel version).

• The current active system partition is LVDbSys2 (Active Logical Volume Manager (LVM) name).

• The update will update the system to release 12.1.1.1.1.140712 (Upgrading to).

• The update will use the ISO image provided in the compressed file (Iso file).

The following actions are performed, as noted in the output of the dbnodeupdate.sh utility:

• A backup is created (Create a backup), which is put on LVDbSys1 (Inactive LVM Name).

• The current system has the database stack running (Shutdown Stack), however, the -s flag was specified, and the stack will be shut down before proceeding the backup and the update.

• The obsolete.lst file (RPM obsolete list) specifies what RPMs will be removed as part of the update by Oracle.

• No exclusion.lst file (RPM exclusion list) was provided.

• The update to release 12.1.1.1.1 is only possible for updates with exadata-computenode-minimum RPM dependencies because Exact dependencies check will fail. For updates to minimal dependency restrictions there is no conflict (Minimum dependencies).

• The log of the dbnodeupdate.sh utility process is appended to the dbnodeupdate.log file (Logfile). The diagfile file (Diagfile) captures important system information before the update starts. Each use of the utility produces the diagfile file. This file is used for troubleshooting when required.

• Active remote network mounts such a NFS are detected (Remote mounts exist), and the dbnodeupdate.sh utility will try to unmount them. When that is not possible, the utility lists the blocking sessions and quits.

• The dbnodeupdate.sh rel. entry specifies the current dbnodeupdate.sh release.

• As the utility checks the system, an on-screen list displays the known issues the dbnodeupdate.sh utility has identified for the Oracle Exadata source and target release. Issues that can be automatically corrected by the utility are listed first, and followed by any manual checks that need to be performed on the system.

Example 2-3 Running the dbnodeupdate.sh Utility

[root@dm01 ]# ./dbnodeupdate.sh -u -l p18889969_121111_Linux-x86-64.zip -s
 
Active Image version   : 11.2.3.3.1.140529.1
Active Kernel version  : 2.6.39-400.128.17.el5uek
Active LVM Name        : /dev/mapper/VGExaDb-LVDbSys2
Inactive Image version : 11.2.3.2.0.120713
Inactive LVM Name      : /dev/mapper/VGExaDb-LVDbSys1
Current user id        : root
Action                 : upgrade 
Upgrading to           : 12.1.1.1.1.140712 (to exadata-sun-computenode-minimum)
Baseurl                : file:///var/www/html/yum/unknown/EXADATA/dbserver/121114021722/x86_64/ (iso)
Iso file               : /u01/dbnodeupdate/iso.stage.121114021722/121111_base_repo_140712.iso
Create a backup        : Yes 
Shutdown stack         : Yes (Currently stack is up)
RPM exclusion list     : Not in use (add rpms to /etc/exadata/yum/exclusion.lst and restart dbnodeupdate.sh)
RPM obsolete list      : /etc/exadata/yum/obsolete.lst (lists rpms to be removed by the update)
                       : RPM obsolete list is extracted from exadata-sun-computenode-12.1.1.1.1.140712-1.x86_64.rpm
Exact dependencies     : Will fail on a next update. Update to 'exact' will be not possible. Falling back to 'minimum'
                       : See /var/log/exadatatmp/121114021722/exact_conflict_report.txt for more details
                       : Update target switched to 'minimum'
Minimum dependencies   : No conflicts
Logfile                : /var/log/cellos/dbnodeupdate.log (runid: 121114021722)
Diagfile               : /var/log/cellos/dbnodeupdate.121114021722.diag
Server model           : SUN FIRE X4170 SERVER
Remote mounts exist    : Yes (dbnodeupdate.sh will try unmounting)
dbnodeupdate.sh rel.   : 3.79b (always check MOS 1553103.1 for the latest release of dbnodeupdate.sh)
Note                   : After upgrading and rebooting run './dbnodeupdate.sh -c' to finish post steps.
 
 
 
The following known issues will be checked for and automatically corrected by dbnodeupdate.sh:
  (*) - Issue - Adjust kernel.rcu_delay in /etc/sysctl.conf. See MOS 1916147.1
  (*) - Issue - Fix for CVE-2014-6271 and CVE-2014-7169 (Shell-Shock)
 
The following known issues will be checked for but require manual follow-up:
  (*) - Issue - Yum rolling update requires fix for 11768055 when Grid Infrastructure is below 11.2.0.2 BP12
 
 
Continue ? [y/n] 
y

(*) 2014-11-12 02:21:19: Verifying GI and DB's are shutdown
  (*) 2014-11-12 02:21:19: Shutting down GI and db
  (*) 2014-11-12 02:23:17: Successfully unmounted network mount /mnt/rene
  (*) 2014-11-12 02:23:17: Unmount of /boot successful
  (*) 2014-11-12 02:23:17: Check for /dev/sda1 successful
  (*) 2014-11-12 02:23:17: Mount of /boot successful
  (*) 2014-11-12 02:23:17: Disabling stack from starting
  (*) 2014-11-12 02:23:17: Performing filesystem backup to /dev/mapper/VGExaDb-LVDbSys1. Avg. 30
                           minutes (maximum 120) depends per environment........
  (*) 2014-11-12 02:30:29: Backup successful
  (*) 2014-11-12 02:30:36: ExaWatcher stopped successful
  (*) 2014-11-12 02:30:36: Capturing service status and file attributes. This may take a while...
  (*) 2014-11-12 02:30:42: Service status and file attribute report in: /etc/exadata/reports
  (*) 2014-11-12 02:30:43: Validating the specified source location.
  (*) 2014-11-12 02:30:44: Cleaning up the yum cache.
  (*) 2014-11-12 02:30:47: Performing yum update. Node is expected to reboot when finished.
  (*) 2014-11-12 02:31:03: Waiting for post rpm script to finish. Sleeping another 60 seconds (60 / 900)
 
Remote broadcast message (Wed Nov 12 02:31:08 2014):
 
Exadata post install steps started.
It may take up to 15 minutes.
 
Remote broadcast message (Wed Nov 12 02:31:53 2014):
 
Exadata post install steps completed.
  (*) 2014-11-12 02:32:03: Waiting for post rpm script to finish. Sleeping another 60 seconds (120 / 900)
  (*) 2014-11-12 02:33:03: All post steps are finished.
  (*) 2014-11-12 02:33:03: System will reboot automatically for changes to take effect
  (*) 2014-11-12 02:33:03: After reboot run "./dbnodeupdate.sh -c" to complete the upgrade
  (*) 2014-11-12 02:33:05: Cleaning up iso and temp mount points
 
  (*) 2014-11-12 02:33:05: Rebooting now...
 
Broadcast message from root (pts/0) (Wed Nov 12 02:33:05 2014):
 
The system is going down for reboot NOW!

2.16.6 Updating Database Servers Running Release 11.2.2.4.2

Updates for database servers running Oracle Exadata Storage Server Software release 11.2.2.4.2 include an update that prepares the servers to use yum.

The steps to prepare the servers are done using the dbnodeupdate.sh utility. For this reason, updates from Oracle Exadata release 11.2.2.4.2 require two runs of the dbnodeupdate.sh utility with different arguments. The utility provides instructions about which command to run and when to it.

When updating the database servers, the following assumptions are made:

• The Oracle Exadata Storage Server Software release is 11.2.2.4.2.

• The database servers are running Oracle Linux 5.5 or later.

• The database server update is available as a local ULN mirror or as a local ISO image.

This section contains the following topics:

• Preparing to Use the dbnodeupdate.sh Utility on Database Servers with Release 11.2.2.4.2

• Running the dbnodeupdate.sh Utility

2.16.6.1 Preparing to Use the dbnodeupdate.sh Utility on Database Servers with Release 11.2.2.4.2

This procedure describes how to download and prepare the utility on the server.

The dbnodeupdate.sh utility is available from My Oracle Support from Doc ID 1553103.1. The dbnodeupdate.sh utility can update releases running 11.2.2.4.2 directly to Oracle Linux 6.

1. Download the dbnodeupdate.sh utility from My Oracle Support.
2. Log in as the root user on the database server.
3. Put the compressed file in the /u01/dbnodeupdate directory on the database server. Create this directory if it does not exist.
4. Uncompress the p16486998_12xxxx_Linux-x86-64.zip package file in the /u01/dbnodeupdate directory using the following command:

   [root@dm01 ]# unzip p16486998_12xxxx_Linux-x86-64.zip
   

   Note:

   • When using a compressed ISO, upload it to the database server and place it in the /u01/dbnodeupdate directory, provided there is free space available. Create this directory if it does not exist.

   • When using a local ULN mirror, make sure the HTTP location is available.

   • When making ISO contents available on a web server, upload the contents to the web server, mount it, and verify the URL.

   • A user with sudo privileges can use sudo to run the dbnodeupdate.sh utility.

2.16.6.2 Running the dbnodeupdate.sh Utility

The following procedure describes how to use the dbnodeupdate.sh utility on database servers running release 11.2.2.4.2.

1. Log in as the root user on the database server.
2. Run the dbnodeupdate.sh utility using the following command. The database server restarts automatically.

   [root@dm01 ]# ./dbnodeupdate.sh -u -l repo
   

   In the preceding command, repo is the HTTP location of the ULN mirror or the location of the compressed ISO image.

   Note:

   The database server restarts automatically. Just before the restart, instructions for the next step are provided onscreen.

3. Run the following command after the restart mentioned in step 2. The utility automatically remounts the ISO image when a compressed ISO image is used.

   [root@dm01 ]# ./dbnodeupdate.sh -u -p 2 
   

4. Run the following command to complete the update.

   [root@dm01 ]# ./dbnodeupdate.sh -c
   

   During the completion process, the utility performs post-update checks, applies best practices, relinks the Oracle homes, and starts the stack.

   Note:

   The update process might still be running when the dbnodeupdate.sh -c command is entered. When this happens, the utility waits until it can determine the image status. The system may restart while waiting to make pending updates effective. If this happens, then re-enter the dbnodeupdate.sh -c command when the system is back online.

2.16.7 Rolling Back Updates on the Database Servers

The database server updates modify the Oracle Exadata Storage Server Software, Oracle Linux system software, and firmware. The updates do not modify the Grid Infrastructure home, database home (other than relinking during the dbnodeupdate.sh -c step), or customer-installed software. The database server updates are performed on the active system disk partition.

To review the active system disk partition information, run the imageinfo command. The database servers using Logical Volume Manager (LVM) have at least one system partition. The following is an example of the output from the imageinfo command:

Kernel version: 2.6.39-400.238.0.el6uek.x86_64 #1 SMP Fri Aug 15 14:27:21 PDT 2014 x86_64
Image version: 12.1.2.1.0.141022
Image activated: 2014-10-22 22:11:12 -0700
Image status: success
System partition on device: /dev/mapper/VGExaDb-LVDbSys1

When the dbnodeupdate.sh utility performs a rollback, the inactive system disk partition becomes active, and the active system disk partition becomes inactive. Rolling back to the previous release by activating the inactive system partition does not roll back the firmware. The Oracle Exadata Storage Server Software releases support later firmware releases.

The ability to roll back database server updates require a backup to be made prior to updating the database servers. The dbnodeupdate.sh utility creates a backup on the inactive system disk partition when the following conditions are met:

• The system uses LVM.

• The size of the inactive partition is equal to the active partition. If only LVDbSys1 partition exists, then there must be as much free space in the volume group as the inactive partition to be created.

• If both system disk partitions, LVDbSys1 and LVDbSys2, exist, then the volume group must have at least 1 GB free space for the temporary snapshot created during the backup.

If the preceding conditions are not met, then the dbnodeupdate.sh utility reports this, and disables the automatic backup for the process. If you proceed with the update, then you will not be able to roll back the update.

Note:

For systems being updated to Oracle Linux 6, a backup must be performed before proceeding with the update. The backup is automatic when updating LVM-enabled systems from Oracle Linux 5 to Oracle Linux 6.

Database servers running as Oracle Virtual Server (dom0) switch between LVDbSys2 and LVDbSys3 as the active system partition when rolling back.

Database servers running as Oracle Virtual machine (domU) have smaller sizes for LVDbSys1 and LVDbSys2 compared to physical hardware deployments.

To verify that a backup will be made by the utility, check the dbnodeupdate.sh confirmation dialog in the prerequisite or update processes. The Create a backup entry should list Yes.

The following command rolls back the update:

[root@dm01 ]# ./dbnodeupdate.sh -r -s

If during the process of updating the database server from Oracle Linux 5 to Oracle Linux 6 an error occurs after the first reboot, then a rollback is initiated automatically for LVM-enabled systems.

After rolling back, the post-patching steps using the following command should be run to relink and start the stack:

[root@dm01 ]# ./dbnodeupdate.sh -c 

2.16.8 Updating Database Nodes with patchmgr

Starting with Exadata release 12.1.2.2.0, Oracle Exadata database nodes (running releases later than 11.2.2.4.2), Oracle Exadata Virtual Server nodes (dom0), and Oracle Exadata Virtual Machines (domU) can be updated, rolled back, and backed up using patchmgr.

You can still run dbnodeupdate.sh in standalone mode, but performing the updates by running patchmgr enables you to run a single command to update multiple nodes at the same time; you do not need to run dbnodeupdate.sh separately on each node. patchmgr can update the nodes in a rolling or non-rolling fashion.

For patchmgr to do this orchestration, you run it from a database node that will not be updated itself. That node needs to be updated later by running patchmgr from a node that has already been updated or by running dbnodeupdate.sh standalone on that node itself.

Similar to updating cells, you create a file that specifies a list of database nodes (including domU and dom0) to be updated. patchmgr then updates the specified nodes in a rolling or non-rolling fashion. The default is non-rolling, which is similar to updating cells. Rollbacks are done in a rolling fashion.

If you are running dbnodeupdate.sh standalone, you can just follow the standard steps for running dbnodeupdate.sh without the patchmgr component. Always check My Oracle Support note 1553103.1 for the latest release of dbnodeupdate.sh.

2.16.8.1 Getting and Installing dbserver.patch.zip

Starting with Oracle Exadata Storage Server software release 12.1.2.2.0, a new dbserver.patch.zip file is available for running dbnodeupdate from patchmgr.

The zip file contains patchmgr and dbnodeupdate.zip. Unzip the dbserver.patch.zip file and run patchmgr from there. Do not unzip dbnodeupdate.zip. Always check My Oracle Support note 1553103.1 for the latest release of dbserver.patch.zip.

For those using a local yum repository in the form of the zipped ISO, there is a separate zipped ISO file for regular hardware, domU, and dom0. When using the ISO file for the update, it is recommended that you put the ISO file in the same directory where you have dbnodeupdate.zip.

2.16.8.2 Behavior for Non-Rolling Upgrades

The behavior for non-rolling upgrades is as follows:

• If a node fails at the pre-check stage, the whole process fails.

• If a node fails at the patch stage or reboot stage, patchmgr skips further steps for the node. The upgrade process continues for the other nodes.

• The pre-check stage is done serially.

• The patch/reboot and complete stages are done in parallel.

The notification alert sequence is:

1. Started (All nodes in parallel)

2. Patching (All nodes in parallel)

3. Reboot (All nodes in parallel)

4. Complete Step (Each node serially)

5. Succeeded (Each node serially)

2.16.8.3 Update Procedure

The steps you perform depending on what type of update you are performing.

If you are updating database nodes that will also go from Oracle Linux 5 to Oracle Linux 6:

1. Run the prerequisites check using patchmgr.

2. Perform a backup.

3. Run an update with mandatory backup.

If you are doing any update other than the Oracle Linux 5 to Oracle Linux 6:

1. Run the prerequisites check.

2. (Optional) Perform a backup.

3. Run an update with optional backup.

2.16.8.4 patchmgr Options

You can use a variety of options with the patchmgr command.

The following options are supported when updating a database node with patchmgr:

Table 2-11 Options for patchmgr: Required Options

Option	Description
-dbnodeslist_file	Specifies the name of the database node list file. The file has one host name or IP address per line.

Table 2-12 Options for patchmgr: Action Options

Option	Description
-precheck	Runs the pre-update validation checks on the database nodes specified in the list file, without actually doing the update.
-upgrade	Upgrades the database nodes specified in the list file.
-backup	Backs up the database nodes specified in the list file.
-rollback	Rolls back the database nodes specified in the list file.
-cleanup	Removes all temporary content on the database nodes specified in the host list in a non-rolling fashion. This option is typically used after an upgrade or rollback operation.
-get property	Returns information about the specified property. Property can be: log_dir Returns the log directory.

Table 2-13 Options for patchmgr: Supporting Options

Option	Description
-iso_repo repo_location	Specifies the path to a zipped ISO file. It is recommended that the file be in the same directory as the dbnodeupdate.zip file. This option or the -yum_repo option must be specified for -backup, -precheck and -upgrade actions. This option cannot be used with the -yum_repo option.
-yum_repo repo_location	Specifies the URL to a yum HTTP repository. This option or the -iso_repo option must be specified for -backup, -precheck and -upgrade actions. This option cannot be used with the -iso_repo option.
-log_dir log_directory_or_auto	Specifies the absolute path to the log directory, or you can specify auto to use a log directory that is based on the launch directory and the content of the nodes list file. Specifying the -log_dir option enables you to run multiple patch manager invocations and also to run patch manager as a non-root user.
-nobackup	Specifies that a backup on the database nodes for an upgrade action should not be performed.
-target_version version	Specifies the full release version being updated to. You can find this information in the README file for the patch.
-unkey	Removes passwordless SSH access to the database nodes before exiting.
-rolling	Specifies that the update is to be done in a rolling fashion. If not specified, the update is done in a non-rolling fashion. Note: Rollbacks are always done in a rolling fashion, even if this option is not specified. Note: Prechecks and cleanups are always done in a non-rolling fashion, even if the -rolling option is specified.
-allow_active_network_mounts	Allows dbnodeupdate to run with active NFS or SMB mounts. This is equivalent to the dbnodeupdate.sh -A command.
-force_remove_custom_rpms	Removes any custom RPMs when the database node is updated from Oracle Linux 5 to Oracle Linux 6. This is equivalent to the dbnodeupdate.sh -R command.
-nomodify_at_prereq	No RPM changes at prerequisite check. This is equivalent to the dbnodeupdate.sh -N command.

Table 2-14 Options for patchmgr: Mail Options

Option	Description
-smtp_from "from_email"	Specifies the from email address for the patchmgr notification.
-smtp_to "to_email1 to_email2 to_email3 ..."	Specifies the to email addresses for the patchmgr notification.
-smtp_set_envelope_sender	Specifies that the same from address in Return-Path: mail header should be used.

2.16.8.5 Checking Prerequisites

The -precheck option simulates the update without actually doing the update.

You use the -precheck option to catch and resolve any errors in advance of the actual update.

Usage:

# ./patchmgr -dbnodes dbnode_list -precheck -iso_repo iso_file -target_version version

# ./patchmgr -dbnodes dbnode_list -precheck -yum_repo yum_http -target_version version

Example 2-4 Using patchmgr with a yum HTTP repository

Example using yum HTTP repository, where date_stamp specifies the release date of the version, for example, 150731:

# ./patchmgr -dbnodes dbnode_list -precheck -yum_repo http://yum-repo/yum/ol6/EXADATA/
dbserver/12.1.2.2.0/base/x86_64/ -target_version 12.1.2.2.0.date_stamp

Example 2-5 Using patchmgr with a zipped yum ISO repository

Example using zipped yum ISO repository, where date_stamp specifies the release date of the version, for example, 150731:

# ./patchmgr -dbnodes dbnode_list -precheck -iso_repo ./repo.zip 
-target_version 12.1.2.2.0.date_stamp

2.16.8.6 Updating the (Database) Nodes

The -upgrade option performs the actual upgrade of the database node(s).

If you need to update all database nodes in a system, you must run the command from a driving node, which cannot be one of the database nodes being updated.

Usage:

# ./patchmgr -dbnodes dbnode_list -upgrade -iso_repo iso_file -target_version version

# ./patchmgr -dbnodes dbnode_list -upgrade -yum_repo yum_http -target_version version

Example 2-6 Rolling update using yum http repository

Example of a rolling update using yum HTTP repository where date_stamp specifies the release date of the version, for example, 150731.

# ./patchmgr -dbnodes dbnode_list -upgrade -yum_repo http://yum-repo/yum/ol6/EXADATA/
dbserver/12.1.2.1.0/base/x86_64/ -target_version 12.1.2.2.0.date_stamp -rolling

Example 2-7 Non-rolling update using zipped yum ISO repository

Example of a rolling update using zipped yum ISO repository where date_stamp specifies the release date of the version, for example, 150731.

# ./patchmgr -dbnodes dbnode_list -upgrade -iso_repo ./repo.zip 
-target_version 12.1.2.2.0.date_stamp

2.16.8.7 Rolling Back the Update

Use the -rollback option to rollback an update.

The -rollback option switches active and inactive Logical Volume Managers (LVM), restores /boot contents, reinstalls Grub, and reboots the database nodes so that it reverses the effects of the upgrade. Note that this is only available on LVM-enabled systems.

Note:

Rollbacks can only be done in a rolling fashion, even if -rolling is not specified.

Usage:

# ./patchmgr -dbnodes dbnode_list -rollback

2.16.8.8 Troubleshooting Failures When Updating Database Nodes with patchmgr

Review these notes if you run into problems running patchmgr.

For the correct syntax for using patchmgr to update database nodes, see the patchmgr online help.

Updating database nodes with the patchmgr tool is less verbose in that it prints only minimal information to the screen. If additional information is required, you can view the patchmgr logs and the dbnodeupdate.sh logs that patchmgr copied over from the failing node, if available.

As with updating storage cells, patchmgr requires SSH equivalence to run.

As always, be sure you download the latest dbserver.patch.zip from My Oracle Support note 1553103.1.

In rare cases, patchmgr may be unable to determine the status of an update, whether the update was successful or not. In such cases, it displays a message that the update failed. However, it is possible that the update still completed successfully. To determine the actual status of the update:

• Check the image status of the (database) node. You can do this by running the imageinfo command. The Image statusline displays the status.

• Check the version of the Exadata software. This can also be determined from the imageinfo command.

If the image status is success, and the Exadata version is the new expected version, then the update was successful and you can ignore the update failed message. Then:

• Run dbnodeupdate.sh -c manually on the particular node to perform the completion steps of the update.

• Remove the completed node from the (database) node file.

• Rerun patchmgr to perform the update on the remaining nodes.

Updating database nodes using patchmgr is optional. You can still run dbnodeupdate.sh manually. If you get any blocking errors from patchmgr when updating critical systems, it is recommended that you perform the update by running dbnodeupdate.sh manually.

You should file a service request with Oracle Support to analyze why the patchmgr orchestration failed.

2.16.9 Running dbnodeupdate.sh and patchmgr (and dbnodeupdate Orchestration) Using sudo

2.16.9.1 Running dbnodeupdate.sh Using sudo

Perform the following steps to set up the /etc/sudoers file for running dbnodeupdate.sh using sudo:

1. Log in as root and edit /etc/sudoers using "visudo".

   # visudo
   

2. Add the following entry (all on one line) to the bottom of the sudoers file to allow non-root users, such as the oracle user, to run dbnodeupdate as root:

   Note:

   The first field in the line specifies the non-root user who is granted sudo access for the dbnodeupdate.sh command. The line below uses the oracle user as an example. You can specify a different user if necessary.

   oracle  ALL=(ALL)    NOPASSWD:SETENV: /u01/stage/patch/dbnodeupdate/dbnodeupdate.sh
   

3. As root, create the /u01/stage/patch/dbnodeupdate directory and unzip dbnodeupdate.zip:

   # mkdir -p /u01/stage/patch/dbnodeupdate
   # cp dbnodeupdate.zip /u01/stage/patch/dbnodeupdate
   # cd /u01/stage/patch/dbnodeupdate
   # unzip dbnodeupdate.zip
   

To verify that the setup is correct, run dbnodeupdate in prereq check mode as the oracle user:

[oracle]$ cd /u01/stage/patch/dbnodeupdate

[oracle]$ sudo ./dbnodeupdate.sh -u -l http://my-yum-repo/yum/EngineeredSystems/exadata/dbserver/12.1.2.1.3/base/x86_64/ -v

dbnodeupdate would exit if it is run without root privileges.

Note:

• The above setup requires that everything in /u01/stage/patch/dbnodeupdate be owned by root.

• A new version of dbnodeupdate would need to be placed in the same location as specified by sudoers.

2.16.9.2 Running patchmgr (and dbnodeupdate Orchestration) Using sudo

You can run patchmgr (which is packaged in dbserver.patch.zip) using sudo to perform any of patchmgr's functionalities, such as patching cells, patching InfiniBand switches, or orchestrating dbnodeupdate execution.

Perform the following steps to set up the /etc/sudoers file for running patchmgr using sudo:

1. Log in as root and edit /etc/sudoers using "visudo".

   # visudo
   

2. Add the following entry to the bottom of the sudoers file to allow non-root users, such as the oracle user, to run patchmgr as root.

   Note:

   The first field in the line specifies the non-root user who is granted sudo access for the patchmgr command. The line below uses the oracle user as an example. You can specify a different user if necessary.

   oracle  ALL=(ALL)    NOPASSWD:SETENV: /u01/stage/patch/dbserverpatch/patchmgr
   

3. As root, create the /u01/stage/patch/dbserverpatch directory and unzip dbserver.patch.zip:

   # mkdir -p /u01/stage/patch/dbserverpatch/
   # cp dbserver.patch.zip /u01/stage/patch/dbserverpatch/
   # cd /u01/stage/patch/dbserverpatch/
   # unzip dbserver.patch.zip
   

4. Move everything under the /u01/stage/patch/dbserverpatch/dbserver_patch_x.yymmdd directory to /u01/stage/patch/dbserverpatch/.

   # mv /u01/stage/patch/dbserverpatch/dbserver_patch_x.yymmdd/* /u01/stage/patch/dbserverpatch/
   

Note:

• patchmgr expects root SSH equivalence on all database nodes that will be updated, even when run using sudo.

• The above setup requires that the entire contents of /u01/stage/patch/dbserverpatch be owned by root.

• A new version of dbserver.patch.zip would need to be placed in the same location as specified by sudoers.

To verify that the setup is correct, run patchmgr in prereq check mode as the oracle user:

[oracle]$ cd /u01/stage/patch/dbserverpatch/

[oracle]$ sudo ./patchmgr -dbnodes dbgroup -precheck     \
   -yum_repo http://my-yum-repo/yum/EngineeredSystems/exadata/dbserver/12.1.2.1.3/base/x86_64/  \
   -target_version 12.1.2.1.3.151021

2.17 Re-Imaging Oracle Linux Database Servers

The re-image procedure is necessary when an Oracle Linux database server has been irretrievably damaged.

You replace the damaged Oracle Linux database server with a new database server. You also use this procedure if there are multiple disk failures causing local disk storage failure and there is no database server backup. During the re-imaging procedure the other database servers on Oracle Exadata Database Machine are available. When the new server is added to the cluster, the software is copied from an existing database server to the new server. It is your responsibility to restore scripting, CRON jobs, maintenance actions, and non-Oracle software.

Note:

The procedures in this section assume the database is Oracle Database 12c Release 1 (12.1) or Oracle Database 11g Release 2 (11.2). If the database is Oracle Database 11g Release 1 (11.1), then refer to the documentation for that release for information about adding and deleting a server from a cluster.

The following tasks describes how to re-image an Oracle Linux database server:

• Task 1: Contact Oracle Support Services

• Task 2: Download Latest Release of Cluster Verification Utility

• Task 3: Remove Failed Database Server from the Cluster

• Task 4: Prepare USB Flash Drive for Imaging

• Task 5: Image Replacement Database Server

• Task 6: Configure Replacement Database Server

• Task 7: Prepare Replacement Database Server for the Cluster

• Task 8: Apply Oracle Exadata Storage Server Software Patch Bundles to Replacement Database Server

• Task 9: Clone Oracle Grid Infrastructure to Replacement Database Server

• Task 10: Clone Oracle Database Homes to Replacement Database Server

2.17.1 Task 1: Contact Oracle Support Services

Open an Oracle support request with Oracle Support Services. The support engineer will identify the failed server, and send a replacement. The support engineer will ask for the output from the imagehistory command run from a surviving database server. The output provides a link to the computeImageMaker file that was used to image the original database server, and provides a means to restore the system to the same level.

2.17.2 Task 2: Download Latest Release of Cluster Verification Utility

The latest release of the cluster verification utility (cluvfy) is available from My Oracle Support.

See My Oracle Support note 316817.1 for download instructions and other information.

2.17.3 Task 3: Remove Failed Database Server from the Cluster

The failed database server must be removed from the cluster before you can reimage the server.

The steps in this task are performed using a working database server in the cluster. In the following commands, working_server is a working database server, and replacement_server is the replacement database server.

1. Log in as the oracle user on a database server in the cluster.
2. Disable the listener that runs on the failed server.

   $ srvctl disable listener -n failed_server
   $ srvctl stop listener -n failed_server
   

3. Delete the Oracle home from the Oracle inventory.

   In the following command, list_of_working_servers is a list of the servers that are still working in the cluster, such as dm01db02, dm01db03, and so on.

   $ cd $ORACLE_HOME/oui/bin
   $ ./runInstaller -updateNodeList ORACLE_HOME= \
   /u01/app/oracle/product/12.1.0.2/dbhome_1 "CLUSTER_NODES=list_of_working_servers"
   

4. Verify the failed server is unpinned.

   $ olsnodes -s -t
   

   The following is an example of the output from the command:

   dm01db01        Inactive        Unpinned
   dm01db02        Active          Unpinned
   

5. Stop and delete the VIP resources for the failed database server.

   # srvctl stop vip -i failed_server-vip
   PRCC-1016 : failed_server-vip.example.com was already stopped
   
   # srvctl remove vip -i failed_server-vip
   Please confirm that you intend to remove the VIPs failed_server-vip (y/[n]) y
   

6. Delete the server from the cluster.

   # crsctl delete node -n failed_server
   CRS-4661: Node dm01db01 successfully deleted.
   

   If you receive an error message similar to the following, then relocate the voting disks.

   CRS-4662: Error while trying to delete node dm01db01.
   CRS-4000: Command Delete failed, or completed with errors.
   

   To relocate the voting disks use the following steps:

   1. Determine the current location of the voting disks.

      # crsctl query css votedisk
      

      The following is an example of the output from the command. The current location is DBFS_DG.

      ##  STATE    File Universal Id          File Name                Disk group
      --  -----    -----------------          ---------                ----------
      1. ONLINE   123456789abab (o/192.168.73.102/DATA_CD_00_dm01cel07) [DBFS_DG]
      2. ONLINE   123456789cdcd (o/192.168.73.103/DATA_CD_00_dm01cel08) [DBFS_DG]
      3. ONLINE   123456789efef (o/192.168.73.100/DATA_CD_00_dm01cel05) [DBFS_DG]
      Located 3 voting disk(s).
      

   2. Relocate the voting disks to another disk group.

      # ./crsctl replace votedisk +DATA
      
      Successful addition of voting disk 2345667aabbdd.
      ...
      CRS-4266: Voting file(s) successfully replaced
      

   3. Relocate the voting disks to the original location using a command similar to the following:

      # ./crsctl replace votedisk +DBFS_DG
      

   4. Delete the server from the cluster.
7. Update the Oracle inventory.

   $ cd $ORACLE_HOME/oui/bin
   $ ./runInstaller -updateNodeList ORACLE_HOME=/u01/app/12.1.0.2/grid \
     "CLUSTER_NODES=list_of_working_servers" CRS=TRUE
   

8. Verify the server was deleted successfully.

   $ cluvfy stage -post nodedel -n failed_server -verbose
   

   The following is an example of the output from the command:

   Performing post-checks for node removal
   Checking CRS integrity...
   The Oracle clusterware is healthy on node "dm01db02"
   CRS integrity check passed
   Result:
   Node removal check passed
   Post-check for node removal was successful.
   

2.17.4 Task 4: Prepare USB Flash Drive for Imaging

A USB flash drive is used to restore the image to the new database server.

The following procedures describe how to prepare the USB flash drive for use.

Note:

For releases 12.1.2.1.0 and later, the system used to create the image must be running Oracle Linux 6.

For Releases 12.1.2.2.0 and Later

1. Insert a blank USB flash drive into a working database server in the cluster.

2. Log in as the root user.

3. Prepare the USB drive.

   Use a command similar to the following, where /dev/sdd is the name of the inserted USB drive. The exact name can be found from looking at /var/log/messages after inserting the USB drive.

   # dd if=/dev/zero of=/dev/sdd bs=1M count=100 oflag=direct
   

4. Write the computeImageMaker .img file to the USB drive. This may take 15 minutes or more and no output is shown during the operation.

   Use a command similar to the following, where /dev/sdd is the name of the inserted USB drive.

   # dd if=filename.img of=/dev/sdd bs=1M oflag=direct
   

5.  Rescan the partition table on Linux to recognize the new partition.

   # partprobe
   

6. Verify that the USB is mountable, as long as the system supports ext4 file system. This must be an Oracle Linux 6 system for this step even if the previous dd command was run on an Oracle Linux 5 system.

   Use a command similar to the following, where /dev/sdd is the name of the inserted USB drive.

   mount /dev/sdd1 /mnt
   

7. Prepare and place the preconf.csv file on the USB drive.

   The file name must be preconf.csv on the USB. The preconf.csv file must contain MAC addresses for each node to be used during the image. If no preconf.csv file is used during imaging, the node prompts for its network configuration the first time it boots up.

   # cp /path_to_preconf_file/preconf.csv /mnt/preconf.csv
   
   # umount /mnt   ### this also ensures that the filesystem is synchronized
   

For Releases Earlier Than 12.1.2.2.0:

1. Insert a blank USB flash drive into a working database server in the cluster.
2. Log in as the root user.
3. Unzip the computeImage file.

   Use a command similar to the following, where exadata_release is the release number for Oracle Exadata Storage Server Software, such as 11.2.1.2.0, release_date is the date of the release, such as 091102, and platform is the platform for the release, such as x86_64.

   # unzip computeImageMaker_exadata_release_LINUX.X64_ \
     release_date.platform.tar.zip
   
   # tar -xvf computeImageMaker_exadata_release_LINUX.X64_ \
     release_date.platform.tar
   

4. Load the image onto the USB flash drive using the following commands:

   # cd dl360
   # ./makeImageMedia.sh -factory -stit -reboot-on-success -nodisktests [-preconf path_to_preconf_file]
   

   The makeImageMedia.sh script prompts for information.

5. Remove the USB flash drive from the database server.
6. Remove the unzipped compute directory and the computeImageMaker file from the working database server. The directory and file require about 2 GB of space.

2.17.5 Task 5: Image Replacement Database Server

Once the database server has been replaced, the USB flash drive is used to put the image on the new database server. The following procedure describes how to load the image on the new database server:

1. Insert the USB flash drive into the USB port on the replacement database server.
2. Log in to the console through the service processor, or by using the KVM switch to monitor progress.
3. Power on the database server using either the service processor interface or by physically pressing the power button.
4. Press F2 during BIOS and select BIOS Setup to configure boot order, or press F8 and select the one-time boot selection menu to choose the USB flash drive.
5. Configure the BIOS boot order if the motherboard was replaced. The boot order should be USB flash drive, then RAID controller.
6. Allow the system to start. As the system starts, it detects the CELLUSBINSTALL media. The imaging process has two phases. Let each phase complete before preceding to the next step.

   The first phase of the imaging process identifies any BIOS or firmware that is out of date, and upgrades the components to the expected level for the image. If any components need to be upgraded or downgraded, then the system automatically restarts.

   The second phase of the imaging process installs the factory image on the replacement database server.

7. Remove the USB flash drive when prompted by the system.
8. Press Enter to power off the server after removing the USB flash drive.

2.17.6 Task 6: Configure Replacement Database Server

The replacement database server does not have any host names, IP addresses, DNS or NTP settings. The steps in this task describe how to configure the replacement database server. You need the following information prior to configuring the replacement database server:

• Name servers

• Time zone, such as Americas/Chicago

• NTP servers

• IP address information for the management network

• IP address information for the client access network

• IP address information for the InfiniBand network

• Canonical host name

• Default gateway

The information should be the same on all database servers in Oracle Exadata Database Machine. The IP addresses can be obtained from DNS. In addition, a document with the information should have been provided when Oracle Exadata Database Machine was installed.

The following procedure describes how to configure the replacement database server:

1. Power on the replacement database server. When the system boots, it automatically runs the Configure Oracle Exadata routine, and prompts for information.
2. Enter the information when prompted, and confirm the settings. The startup process will continue.

Note:

• If the database server does not use all network interfaces, then the configuration process stops, and warns that some network interfaces are disconnected. It prompts whether to retry the discovery process. Respond with yes or no, as appropriate for the environment.

• If bonding is used for the client access network, then it is set in the default active-passive mode at this time.

2.17.7 Task 7: Prepare Replacement Database Server for the Cluster

During initial installation of Oracle Exadata Database Machine, certain files were modified by the installation process.The following procedure describes how to ensure the changes made during initial installation are done to the replacement database server:

1. Copy or merge the contents of the following files using files on a working database server as reference:

   1. Copy the contents of the /etc/security/limits.conf file.

   2. Merge the contents of the /etc/hosts files.

   3. Copy the /etc/oracle/cell/network-config/cellinit.ora file.

   4. Update the /etc/oracle/cell/network-config/cellinit.ora file with the IP_ADDRESS of the ifcfg-bondib0 interface (in case of active/passive bonding) or ib0 and ib1 interfaces (in case of active/active bonding) of the replacement server.

   5. Copy the /etc/oracle/cell/network-config/cellip.ora file. The content of the cellip.ora file should be the same on all database servers.

   6. Configure additional network requirements, such as 10 GbE.

   7. Copy the /etc/modprobe.conf file. The contents of the file should be the same on all database servers.

   8. Copy the /etc/sysctl.conf file. The contents of the file should be the same on all database servers.

   9. Restart the database server so the network changes take effect.

2. Set up the oracle user on the replacement database server by adding group, or groups, for the Oracle software owner. The owner is usually oracle. The group information is available on a working database server.

   1. Obtain the current group information from a working database server using the following command:

      # id oracle
      uid=1000(oracle) gid=1001(oinstall) groups=1001(oinstall),1002(dba),1003(oper),1004(asmdba)
      

   2. Use the groupadd command to add the group information to the replacement database server using the following commands:

      # groupadd -g 1001 oinstall
      # groupadd -g 1002 dba
      # groupadd -g 1003 oper
      # groupadd -g 1004 asmdba
      

   3. Obtain the current user information from a working database server using the following command:

      # id oracle uid=1000(oracle) gid=1001(oinstall) \
        groups=1001(oinstall),1002(dba),1003(oper),1004(asmdba)
      

   4. Add the user information to the replacement database server using the following command:

      # useradd -u 1000 -g 1001 -G 1001,1002,1003,1004 -m -d /home/oracle -s \
        /bin/bash oracle
      

   5. Create the ORACLE_BASE and Grid Infrastructure directories, such as /u01/app/oracle and /u01/app/12.1.0.2/grid using the following commands:

      # mkdir -p /u01/app/oracle
      # mkdir -p /u01/app/12.1.0.2/grid
      # chown -R oracle:oinstall /u01/app
      

   6. Change the ownership on the cellip.ora and cellinit.ora files using the following command. The ownership is usually oracle:dba.

      # chown -R oracle:dba /etc/oracle/cell/network-config
      

   7. Secure the restored database server using the following commands:

      $ chmod u+x /opt/oracle.SupportTools/harden_passwords_reset_root_ssh
      $ /opt/oracle.SupportTools/harden_passwords_reset_root_ssh
      

      Note:

      The database server restarts. Log in as the root user when prompted by the system. You are prompted for a new password. Set the password to match the root password of the other database servers.

   8. Set the password for the Oracle software owner using the following command. The owner is usually oracle.

      # passwd oracle
      

3. Set up SSH for the oracle account as follows:

   1. Log in to the oracle account on the replacement database server using the following command:

      # su - oracle
      

   2. Create the dcli group file on the replacement database server listing the servers in the Oracle cluster.

   3. Run the following command on the replacement database server.

      $ dcli -g dbs_group -l oracle -k
      

   4. Log in as the oracle user on the replacement database server using the following command:

      # su - oracle
      

   5. Verify SSH equivalency using the following command:

      $ dcli -g dbs_group -l oracle date
      

4. Set up or copy any custom login scripts from a working database server to the replacement database server using the following command:

   $ scp .bash* oracle@replacement_server:. 
   

   In the preceding command, replacement_server is the name of the new server, such as dm01db01.

2.17.8 Task 8: Apply Oracle Exadata Storage Server Software Patch Bundles to Replacement Database Server

Oracle periodically releases Oracle Exadata Storage Server Software patch bundles for Oracle Exadata Database Machine.

If a patch bundle has been applied to the working database servers that was later than the release of the computeImageMaker file, then the patch bundle must be applied to the replacement database server. Determine the if a patch bundle has been applied as follows:

• Prior to Oracle Exadata Storage Server Software release 11.2.1.2.3, the database servers did not maintain version history information. To determine the release number, log in to Exadata Storage Server, and run the following command:

  imageinfo -ver
  

  If the command shows a different release than the release used by the computeImageMaker file, then Oracle Exadata Storage Server Software patch has been applied to Oracle Exadata Database Machine and must be applied to the replacement database server.

• Starting with Oracle Exadata Storage Server Software release 11.2.1.2.3, the imagehistory command exists on the database server. Compare information on the replacement database server to information on a working database server. If the working database has a later release, then apply the Exadata Storage Server patch bundle to the replacement database server.

2.17.9 Task 9: Clone Oracle Grid Infrastructure to Replacement Database Server

This procedure describes how to clone the Grid Infrastructure to the replacement database server.

In the following commands, working_server is a working database server, and replacement_server is the replacement database server. The commands in this procedure are run from a working database server as the Grid home owner. When the root user is needed to run a command, it will be called out.

1. Verify the hardware and operating system installation using the cluster verification utility (cluvfy).

   $ cluvfy stage -post hwos -n replacement_server,working_server -verbose
   

   The phrase Post-check for hardware and operating system setup was successful should appear at the end of the report.

2. Verify peer compatibility.

   $ cluvfy comp peer -refnode working_server -n replacement_server  \
     -orainv oinstall -osdba dba | grep -B 3 -A 2 mismatched
   

   The following is an example of the output:

   Compatibility check: Available memory [reference node: dm01db02]
   Node Name Status Ref. node status Comment
   ------------ ----------------------- ----------------------- ----------
   dm01db01 31.02GB (3.2527572E7KB) 29.26GB (3.0681252E7KB) mismatched
   Available memory check failed
   Compatibility check: Free disk space for "/tmp" [reference node: dm01db02]
   Node Name Status Ref. node status Comment
   ------------ ----------------------- ---------------------- ----------
   dm01db01 55.52GB (5.8217472E7KB) 51.82GB (5.4340608E7KB) mismatched
   Free disk space check failed
   

   If the only failed components are related to the physical memory, swap space and disk space, then it is safe to continue.

3. Perform the requisite checks for adding the server.
   1. Ensure the GRID_HOME/network/admin/samples directory has permissions set to 750.
   2. Validate the addition of the database server.

      Run the following command as the oracle user. The command prompts for the password of the root user.

      $ cluvfy stage -pre nodeadd -n adczardb03 -fixup -method root -verbose
      Enter "ROOT" password:
      

      If the only failed component is related to swap space, then it is safe to continue.

      If the command returns an error, then set the following environment variable and rerun the command:

      $ export IGNORE_PREADDNODE_CHECKS=Y
      

4. Add the replacement database server to the cluster.

   $ cd /u01/app/12.1.0.2/grid/addnode
   
   $ ./addnode.sh -silent "CLUSTER_NEW_NODES={replacement_server}" \
        "CLUSTER_NEW_VIRTUAL_HOSTNAMES={replacement_server-vip}"
   

   The second command causes Oracle Universal Installer to copy the Oracle Clusterware software to the replacement database server. A message similar to the following is displayed:

   WARNING: A new inventory has been created on one or more nodes in this session.
   However, it has not yet been registered as the central inventory of this
   system. To register the new inventory please run the script at
   '/u01/app/oraInventory/orainstRoot.sh' with root privileges on nodes
   'dm01db01'. If you do not register the inventory, you may not be able to update
   or patch the products you installed.
   
   The following configuration scripts need to be executed as the "root" user in
   each cluster node:
   
   /u01/app/oraInventory/orainstRoot.sh #On nodes dm01db01
   
   /u01/app/12.1.0.2/grid/root.sh #On nodes dm01db01
   

5. Run the configuration scripts in a terminal window as follows:
   1. Open a terminal window.
   2. Log in as the root user.
   3. Run the scripts on each cluster node.

   After the scripts are run, the following message is displayed:

   The Cluster Node Addition of /u01/app/12.1.0.2/grid was successful.
   Please check '/tmp/silentInstall.log' for more details.
   

6. As the root user, disable HAIP then run the orainstRoot.sh and root.sh scripts on the replacement database server using the following commands. Sample output is shown.

   # export HAIP_UNSUPPORTED=true
   # /u01/app/oraInventory/orainstRoot.sh
   Creating the Oracle inventory pointer file (/etc/oraInst.loc)
   Changing permissions of /u01/app/oraInventory.
   Adding read,write permissions for group.
   Removing read,write,execute permissions for world.
   Changing groupname of /u01/app/oraInventory to oinstall.
   The execution of the script is complete.
    
   # /u01/app/12.1.0.2/grid/root.sh
   

   Note:

   Check /u01/app/12.1.0.2/grid/install/ log files for the output of root.sh script.

7. Check the cluster.

   $ /u01/app/12.1.0.2/grid/bin/crsctl check cluster -all
   
   **************************************************************
   node1:
   CRS-4537: Cluster Ready Services is online
   CRS-4529: Cluster Synchronization Services is online
   CRS-4533: Event Manager is online
   **************************************************************
   node2:
   CRS-4537: Cluster Ready Services is online
   CRS-4529: Cluster Synchronization Services is online
   CRS-4533: Event Manager is online
   **************************************************************
   node3:
   CRS-4537: Cluster Ready Services is online
   CRS-4529: Cluster Synchronization Services is online
   CRS-4533: Event Manager is online
   

2.17.10 Task 10: Clone Oracle Database Homes to Replacement Database Server

The following procedure describes how to clone the Oracle Database homes to the replacement server.

Run the commands from a working database server as the oracle user. When the root user is needed to run a command, it will be called out.

1. Add the Oracle Database ORACLE_HOME to the replacement database server using the following commands:

   $ cd /u01/app/oracle/product/12.1.0.2/dbhome_1/addnode
   
   $ ./addnode.sh -silent "CLUSTER_NEW_NODES={replacement_server}"
   

   The second command causes Oracle Universal Installer to copy the Oracle Database software to the replacement database server.

   WARNING: The following configuration scripts need to be executed as the "root"
   user in each cluster node.
   /u01/app/oracle/product/12.1.0.2/dbhome_1/root.sh #On nodes dm01db01
   To execute the configuration scripts:
   Open a terminal window.
   Log in as root.
   Run the scripts on each cluster node.
    
   

   After the scripts are finished, the following messages appear:

   The Cluster Node Addition of /u01/app/oracle/product/12.1.0.2/dbhome_1 was successful.
   Please check '/tmp/silentInstall.log' for more details.
   

2. Run the following script on the replacement database server:

   # /u01/app/oracle/product/12.1.0.2/dbhome_1/root.sh
    
   

   Check the /u01/app/orcale/product/12.1.0.2/dbhome_1/install/root_replacement_server.com_date.log file for the output of the script.

3. Run DBCA in interactive mode to add database instances to the target nodes.

   1. Start up DBCA.

      $ cd /u01/app/oracle/product/12.1.0.2/dbhome_1/bin
      
      $ ./dbca
      

   2. On the “Database Operation” screen, select “Instance Management”. Click Next.

   3. On the “Instance Operation” screen, select “Add an instance”. Click Next.

   4. On the “Database List” screen, select the cluster database to which you want to add an instance.

   5. The “List Instance” screen displays the current instances. Click Next to add a new instance.

   6. The “Add Instance” screen displays the default name and the newly added node to the cluster. Accept the defaults and click Next.

   7. On the “Summary” screen, verify the plan and click Finish.

   8. On the “Progress” screen, watch for 100% completion.

   9. On the “Finish” screen, acknowledge the confirmation that the new instance was successfully added.

   Verify that the instance has been added:

   $ srvctl config database -db dbm01
   

   Verify the administrative privileges on the target node:

   $ cd /u01/app/oracle/product/12.1.0.2/dbhome_1/bin
   
   $ ./cluvfy comp admprv -o db_config -d /u01/app/oracle/product/12.1.0.2/dbhome_1 -n new_node
   

4. Ensure the instance parameters are set for the replaced database instance. The following is an example for the CLUSTER_INTERCONNECTS parameter.

   SQL> SHOW PARAMETER cluster_interconnects
   
   NAME                                 TYPE        VALUE
   ------------------------------       --------    -------------------------
   cluster_interconnects                string
    
   SQL> ALTER SYSTEM SET cluster_interconnects='192.168.73.90' SCOPE=spfile SID='dbm1';
   

5. Validate the configuration files as follows:

   • The ORACLE_HOME/dbs/initSID.ora file points to the spfile in the Oracle ASM shared storage.

   • The password file that is copied in the ORACLE_HOME/dbs directory has been changed to orapwSID.

6. Check that any services that incorporated this instance before and ensure the services are updated to include this replacement instance.
7. If this procedure was performed on Oracle Exadata Database Machine Eighth Rack, then perform the procedure described in "Configuring Oracle Exadata Database Machine Eighth Rack Oracle Linux Database Server After Recovery".

2.18 Changing Existing Elastic Configurations for Database Servers

This section describes how to perform the following changes to existing elastic configurations.

• Adding a New Database Server to the Cluster

• Moving an Existing Database Server to a Different Cluster

• Dropping a Database Server from Oracle RAC

For changes involving storage cells, see "Changing Existing Elastic Configurations for Storage Cells".

2.18.1 Adding a New Database Server to the Cluster

In this scenario, you want to add a new database server to an existing Oracle RAC on Exadata.

1. Determine if the new database server needs to be reimaged.

   Check the image label of the database servers in the cluster to which you want to add the new database server. If reimage of the database server is required to match the image label of the existing database servers in the cluster, then reimage the database server by following tasks 1, 2, 4, 5, and 6 in "Re-Imaging Oracle Linux Database Servers".

   If an upgrade is required, the upgrade may be performed using dbnodeupdate.sh. See Updating Oracle Linux Database Servers for details.

2. Add the database server to the cluster.

   To do this, perform the tasks starting with task 7 in "Re-Imaging Oracle Linux Database Servers".

3. Download and run the latest exachk to ensure that the resulting configuration implements the latest best practices for Oracle Exadata.

2.18.2 Moving an Existing Database Server to a Different Cluster

In this scenario, you want to repurpose an existing database server and move it to a different cluster within the same Exadata rack.

1. Remove the database server from the existing cluster.

   1. Stop the Grid Infrastructure on the database server.

      $GI_HOME/bin/crstl stop crs
      

   2. Remove the database server from the cluster by following task 3 in "Re-Imaging Oracle Linux Database Servers".

2. Determine if the database server that is being repurposed needs to be reimaged.

   Check the image label of the database servers in the cluster to which you want to add the database server that is being repurposed. If a reimage of the database server being re-purposed is required to match the image label of the existing database servers in the cluster, then image the database server by following tasks 1, 2, 4, 5, and 6 in "Re-Imaging Oracle Linux Database Servers".

   If an upgrade is required, the upgrade may be performed using dbnodeupdate.sh. See My Oracle Support note 1553103.1 for details.

3. Add the database server to the cluster.

   To do this, perform the tasks starting with task 7 in "Re-Imaging Oracle Linux Database Servers".

4. Download and run the latest exachk to ensure that the resulting configuration implements the latest best practices for Oracle Exadata.

2.18.3 Dropping a Database Server from Oracle RAC

In this scenario, you want to remove a database server from the Oracle RAC that it is a part of.

1. Stop the Grid Infrastructure on the database server to be removed.

   $GI_HOME/bin/crstl stop crs
   

2. Remove the database server from the cluster by following "Task 3: Remove Failed Database Server from the Cluster" of "Re-Imaging Oracle Linux Database Servers".
3. Download and run the latest ExaCHK to ensure that the resulting configuration implements the latest best practices for Oracle Exadata.

2.19 Managing Quorum Disks for High Redundancy Disk Groups

This section contains the following subsections:

• Overview of Quorum Disk Manager

• Software Requirements for Quorum Disk Manager

• Adding Quorum Disks to Database Nodes

• Recreating Quorum Disks

• Use Cases

• quorumdiskmgr Reference

2.19.1 Overview of Quorum Disk Manager

The Quorum Disk Manager utility, introduced in Oracle Exadata release 12.1.2.3.0, helps you to manage the quorum disks.

The utility enables you to create an iSCSI quorum disk on two of the database nodes and store a voting file on those two quorum disks. These two additional voting files are used to meet the minimum requirement of five voting files for a high redundancy disk group. This feature is only applicable to Oracle Exadata racks that meet the following requirements:

• The Oracle Exadata rack has fewer than five storage servers.

• The Oracle Exadata rack has at least two database nodes.

• The Oracle Exadata rack has at least one high redundancy disk group.

The feature allows for the voting files to be stored in a high redundancy disk group on Oracle Exadata racks with fewer than five storage servers due to the presence of two extra failure groups.

Without this feature, voting files are stored in a normal redundancy disk group on Exadata racks with fewer than five storage servers. This makes Oracle Grid Infrastructure vulnerable to a double partner storage server failure that results in the loss of the voting file quorum, in turn resulting in a complete cluster and database outage. Refer to My Oracle Support note 1339373.1 for restarting the clusterware and databases in this scenario.

The iSCSI quorum disk implementation has high availability because the IP addresses on ib0 and ib1 are highly available using RDS. The multipathing feature ensures the iSCSI quorum disk implementation will work seamlessly if a more flexible or isolated internal network configuration is implemented in the future.

Each iSCSI device shown in the figure below corresponds to a particular path to the iSCSI target. Each path corresponds to an InfiniBand port on the database node. For each multipath quorum disk device in an active–active system, there are two iSCSI devices, one for ib0 and the other for ib1.

Figure 2-7 Multipath Device Connects to Both iSCSI Devices in an Active-Active System

Description of "Figure 2-7 Multipath Device Connects to Both iSCSI Devices in an Active-Active System"

The feature is applicable to bare metal Oracle RAC clusters as well as Oracle VM Oracle RAC clusters. For Oracle VM Oracle RAC clusters, the quorum disk devices reside in the Oracle RAC cluster nodes which are Oracle VM user domains as shown in the following figure.

Figure 2-8 Quorum Disk Devices on Oracle VM Oracle RAC Cluster

Description of "Figure 2-8 Quorum Disk Devices on Oracle VM Oracle RAC Cluster"

Note that for pkey-enabled environments, the interfaces used for discovering the targets should be the pkey interfaces used for the Oracle Clusterware communication. These interfaces are listed using the following command:

Grid_home/bin/oifcfg getif | grep cluster_interconnect | awk '{print $1}'

The Quorum Disk Manager utility (quorumdiskmgr) is used to create and manage all the necessary components including the iSCSI configuration, the iSCSI targets, the iSCSI LUNs, and the iSCSI devices for implementing this feature.

2.19.2 Software Requirements for Quorum Disk Manager

To use this feature, the following releases are required:

• Oracle Exadata software release 12.1.2.3.0 and above

• Patch 23200778 for all Database homes

• Oracle Grid Infrastructure 12.1.0.2.160119 with patches 22722476 and 22682752, or Oracle Grid Infrastructure 12.1.0.2.160419 and above

  Note that for new deployments, OEDA installs the patches automatically.

2.19.3 Adding Quorum Disks to Database Nodes

You can add quorum disks to database nodes on an Oracle Exadata rack that contains a high redundancy disk group with fewer than 5 storage servers.

The example in this section creates quorum disks for a quarter rack with two database nodes: db01 and db02.

This is an active-active system. On both db01 and db02 there are two InfiniBand ports: ib0 and ib1.

The network interfaces to be used for communication with the iSCSI devices can be found using the following command:

$ oifcfg getif | grep cluster_interconnect | awk '{print $1}'

The IP address of each interface can be found using the following command:

ip addr show interface_name

The InfiniBand IP addresses for this example are as follows:

On db01:

• Network interface: ib0, IP address: 192.168.10.45

• Network interface: ib1, IP address: 192.168.10.46

On db02:

• Network interface: ib0, IP address: 192.168.10.47

• Network interface: ib1, IP address: 192.168.10.48

The Oracle ASM disk group to which the quorum disks will be added is DATAC1. The Oracle ASM owner is grid, and the user group is dba.

Initially, the voting files reside on a normal redundancy disk group RECOC1:

$ Grid_home/bin/crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   21f5507a28934f77bf3b7ecf88b26c47 (o/192.168.76.187;192.168.76.188/RECOC1_CD_00_celadm12) [RECOC1]
 2. ONLINE   387f71ee81f14f38bfbdf0693451e328 (o/192.168.76.189;192.168.76.190/RECOC1_CD_00_celadm13) [RECOC1]
 3. ONLINE   6f7fab62e6054fb8bf167108cdbd2f64 (o/192.168.76.191;192.168.76.192/RECOC1_CD_00_celadm14) [RECOC1]
Located 3 voting disk(s).

1. Log into db01 and db02 as root.
2. Run the quorumdiskmgr command with the --create --config options to create quorum disk configurations on both db01 and db02.

   # /opt/oracle.SupportTools/quorumdiskmgr --create --config --owner=grid --group=dba --network-iface-list="ib0, ib1"
   

3. Run the quorumdiskmgr command with the --list --config options to verify that the configurations have been successfully created on both db01 and db02.

   # /opt/oracle.SupportTools/quorumdiskmgr --list --config
   

   The output should look like:

   Owner: grid
   Group: dba
   ifaces: exadata_ib1 exadata_ib0
   

4. Run the quorumdiskmgr command with the --create --target options to create a target on both db01 and db02 for Oracle ASM disk group DATAC1 and make the target visible to both db01 and db02.

   # /opt/oracle.SupportTools/quorumdiskmgr --create --target --asm-disk-group=datac1 --visible-to="192.168.10.45, 192.168.10.46, 192.168.10.47, 192.168.10.48"
   

5. Run the quorumdiskmgr command with the --list --target options to verify the target has been successfully created on both db01 and db02.

   # /opt/oracle.SupportTools/quorumdiskmgr --list --target
   

   On db01, the output should look like:

   Name: iqn.2015-05.com.oracle:QD_DATAC1_DB01 
   Size: 128 MB 
   Host name: DB01
   ASM disk group name: DATAC1 
   Visible to: 192.168.10.45, 192.168.10.46, 192.168.10.47, 192.168.10.48
   Discovered by:
   

   On db02, the output should be:

   Name: iqn.2015-05.com.oracle:QD_DATAC1_DB02 
   Size: 128 MB 
   Host name: DB02
   ASM disk group name: DATAC1 
   Visible to: 192.168.10.45, 192.168.10.46, 192.168.10.47, 192.168.10.48
   Discovered by:
   

6. Run the quorumdiskmgr command with the --create --device options to create devices on both db01 and db02 from targets on both db01 and db02.

   # /opt/oracle.SupportTools/quorumdiskmgr --create --device --target-ip-list="192.168.10.45, 192.168.10.46, 192.168.10.47, 192.168.10.48"
   

7. Run the quorumdiskmgr command with the --list --device options to verify the devices have been successfully created on both db01 and db02.

   # /opt/oracle.SupportTools/quorumdiskmgr --list --device
   

   On both db01 and db02, the output should look like:

   Device path: /dev/exadata_quorum/QD_DATAC1_DB01 
   Size: 128 MB 
   Host name: DB01
   ASM disk group name: DATAC1 
   
   Device path: /dev/exadata_quorum/QD_DATAC1_DB02 
   Size: 128 MB 
   Host name: DB02
   ASM disk group name: DATAC1
   

8. Switch to the grid user on either db01 or db02.
9. Set up Oracle ASM environments.
10. Alter the asm_diskstring initialization parameter and add /dev/exadata_quorum/* to the existing string

    SQL> alter system set asm_diskstring='o/*/DATAC1_*','o/*/RECOC1_*','/dev/exadata_quorum/*' scope=both sid='*';
    

11. Verify the two quorum disk devices have been automatically discovered by Oracle ASM.

    SQL> set linesize 200
    SQL> col path format a50
    SQL> select inst_id, label, path, mode_status, header_status
    from gv$asm_disk where path like '/dev/exadata_quorum/%';
    

    The output should look like:

    INST_ID LABEL          PATH                                MODE_STATUS HEADER_STATUS
    ------- -------------- ----------------------------------  ----------- ---------
          1 QD_DATAC1_DB01 /dev/exadata_quorum/QD_DATAC1_DB01  ONLINE      CANDIDATE
          1 QD_DATAC1_DB02 /dev/exadata_quorum/QD_DATAC1_DB02  ONLINE      CANDIDATE
          2 QD_DATAC1_DB01 /dev/exadata_quorum/QD_DATAC1_DB01  ONLINE      CANDIDATE
          2 QD_DATAC1_DB02 /dev/exadata_quorum/QD_DATAC1_DB02  ONLINE      CANDIDATE
    

12. Add the two quorum disk devices to a high redundancy Oracle ASM disk group.

    If there is no high redundancy disk group, create a high redundancy disk group and include the two new quorum disks. For example:

    SQL> create diskgroup DATAC1 high redundancy quorum failgroup db01 disk '/dev/exadata_quorum/QD_ DATAC1_DB01' quorum failgroup db02 disk '/dev/exadata_quorum/QD_ DATAC1_DB02' ...
    

    If a high redundancy disk group already exists, add the two new quorum disks. For example:

    SQL> alter diskgroup datac1 add quorum failgroup db01 disk '/dev/exadata_quorum/QD_DATAC1_DB01' quorum failgroup db02 disk '/dev/exadata_quorum/QD_DATAC1_DB02';
    

13. Relocate the existing voting files from the normal redundancy disk group to the high redundancy disk group.

    $ Grid_home/bin/crsctl replace votedisk +DATAC1
    

14. Verify the voting disks have been successfully relocated to the high redundancy disk group and that five voting files exist.

    crsctl query css votedisk
    

    The output should show 3 voting disks from storage servers and 2 voting disks from database nodes:

    ## STATE File Universal Id File Name Disk group
    -- ----- ----------------- --------- ---------
    1. ONLINE ca2f1b57873f4ff4bf1dfb78824f2912 (o/192.168.10.42/DATAC1_CD_09_celadm12) [DATAC1]
    2. ONLINE a8c3609a3dd44f53bf17c89429c6ebe6 (o/192.168.10.43/DATAC1_CD_09_celadm13) [DATAC1]
    3. ONLINE cafb7e95a5be4f00bf10bc094469cad9 (o/192.168.10.44/DATAC1_CD_09_celadm14) [DATAC1]
    4. ONLINE 4dca8fb7bd594f6ebf8321ac23e53434 (/dev/exadata_quorum/QD_ DATAC1_DB01) [DATAC1]
    5. ONLINE 4948b73db0514f47bf94ee53b98fdb51 (/dev/exadata_quorum/QD_ DATAC1_DB02) [DATAC1]
    Located 5 voting disk(s).
    

15. Move the Oracle ASM password file and the Oracle ASM SPFILE to the high redundancy disk group.
    1. Move the Oracle ASM password file:

       i) Get the source Oracle ASM password file location.

       $ asmcmd pwget --asm
       

       ii) Move the Oracle ASM password file to the high redundancy disk group.

       $ asmcmd pwmove --asm full_path_of_source_file full_path_of_destination_file
       

       Example:

       asmcmd pwmove --asm +recoc1/ASM/PASSWORD/pwdasm.256.898960531 +datac1/asmpwdfile
       

    2. Move the Oracle ASM SPFILE.

       i) Get the Oracle ASM SPFILE in use:

       $ asmcmd spget
       

       ii) Copy the Oracle ASM SPFILE to the high redundancy disk group.

       $ asmcmd spcopy full_path_of_source_file full_path_of_destination_file
       

       iii) Modify the Oracle Grid Infrastructure configuration to use the relocated SPFILE upon next restart.

       $ asmcmd spset full_path_of_destination_file
       

       The above commands should run from any one Oracle VM cluster node for Oracle RAC.

       At this point if a downtime can be afforded, restart Oracle Grid Infrastructure using the commands below.

       # Grid_home/bin/crsctl stop crs
       # Grid_home/bin/crsctl start crs
       

       If a downtime is not permitted, repeat step 15.b every time an initialization parameter modification to the Oracle ASM SPFILE is required until Oracle Grid Infrastructure is restarted.

16. Relocate the MGMTDB to the high redundancy disk group.

    Move the MGMTDB (if running) to the high redundancy disk group using How to Move/Recreate GI Management Repository to Different Shared Storage (Diskgroup, CFS or NFS etc) (Doc ID 1589394.1).

    Configure the MGMTDB to not use hugepages using the steps below:

    export ORACLE_SID=-MGMTDB
    export ORACLE_HOME=$GRID_HOME
    sqlplus ”sys as sysdba”
    SQL> alter system set use_large_pages=false scope=spfile  sid='*';
    

17. These steps are optional.
    1. Restart Oracle Grid Infrastructure.

       # Grid_home/bin/crsctl stop crs
       # Grid_home/bin/crsctl start crs
       

    2. Convert the normal redundancy disk group to a high redundancy disk group.

       Refer to How To Move The Database To Different Diskgroup (Change Diskgroup Redundancy) (Doc ID 438580.1) for details on how to change disk group redundancy.

2.19.4 Recreating Quorum Disks

In certain circumstances, you might need to recreate a quorum disk.

Some examples of when you might need to recreate a quorum disk are:

• When recreating a guest domU

• If you deleted the quorum disks without first dropping the quorum disks from the Oracle ASM disk group

1. Force drop the lost quorum disk.

   ALTER DISKGROUP dg_name DROP QUORUM DISK disk_name FORCE;
   

2. Follow the instructions in "Adding Quorum Disks to Database Nodes" to add a new quorum disk.

2.19.5 Use Cases

• New Deployments on Oracle Exadata 12.1.2.3.0 or Later

• Upgrading to Oracle Exadata Release 12.1.2.3.0 or Later

• Downgrading to a Pre-12.1.2.3.0 Oracle Exadata Release

• Changing Elastic Configurations

2.19.5.1 New Deployments on Oracle Exadata 12.1.2.3.0 or Later

For new deployments on Oracle Exadata release 12.1.2.3.0 and above, OEDA implements this feature by default when all of the following requirements are satisfied:

• The system has at least two database nodes and fewer than five storage servers.

• You are running OEDA release February 2016 or later.

• You meet the software requirements listed in Software Requirements for Quorum Disk Manager.

• Oracle Database is 11.2.0.4 and above.

• The system has at least one high redundancy disk group.

If the system has three storage servers in place, then two quorum disks will be created on the first two database nodes of the cluster picked by OEDA.

If the system has four storage servers in place, then one quorum disk will be created on the first database node picked by OEDA.

2.19.5.2 Upgrading to Oracle Exadata Release 12.1.2.3.0 or Later

If the target Exadata system has fewer than five storage servers, at least one high redundancy disk group, and two or more database nodes, you can implement this feature manually using quorumdiskmgr.

2.19.5.3 Downgrading to a Pre-12.1.2.3.0 Oracle Exadata Release

Rolling back to a pre-12.1.2.3.0 Oracle Exadata release, which does not support quorum disks, from a release that supports quorum disks, which is any release 12.1.2.3.0 and later, requires quorum disk configuration to be removed if the environment has quorum disk implementation in place. You need to remove the quorum disk configuration before performing the Exadata software rollback.

To remove quorum disk configuration, perform these steps:

1. Ensure there is at least one normal redundancy disk group in place. If not, create one.

2. Relocate the voting files to a normal redundancy disk group:

   $GI_HOME/bin/crsctl replace votedisk +normal_redundancy_diskgroup
   

3. Drop the quorum disks from ASM. Run the following command for each quorum disk:

   SQL> alter diskgroup diskgroup_name drop quorum disk quorum_disk_name force;
   

   Wait for the rebalance operation to complete. You can tell it is complete when v$asm_operation returns no rows for the disk group.

4. Delete the quorum devices. Run the following command from each database node that has quorum disks in place:

   /opt/oracle.SupportTools/quorumdiskmgr --delete --device [--asm-disk-group asm_disk_group] [--host-name host_name]
   

5. Delete the targets. Run the following command from each database node that has quorum disks in place:

   /opt/oracle.SupportTools/quorumdiskmgr --delete --target [--asm-disk-group asm_disk_group]
   

6. Delete the configuration. Run the following command from each database node that has quorum disks in place:

   /opt/oracle.SupportTools/quorumdiskmgr --delete –config
   

2.19.5.4 Changing Elastic Configurations

• Adding a Database Node

• Removing a Database Node

• Adding an Oracle Exadata Storage Server and Expanding an Existing High Redundancy Disk Group

• Removing an Oracle Exadata Storage Server

2.19.5.4.1 Adding a Database Node

If the existing Oracle RAC cluster has fewer than two database nodes and fewer than five storage servers, and the voting files are not stored in a high redundancy disk group, then Oracle recommends adding quorum disks to the database node(s) and relocating the voting files to a high redundancy disk group.

Note:

The requirements listed in "Software Requirements for Quorum Disk Manager" must be met.

If the existing Oracle RAC cluster already has quorum disks in place, the quorum disks need to be made visible to the newly added node prior to adding the node to the Oracle RAC cluster using the addnode.sh procedure. Follow the steps below for making the quorum disks visible to the node being added:

1. Login to the 2 database nodes that contain the quorum devices and retrieve the quorum disk ISCSI target configuration.

   /opt/oracle.SupportTools/quorumdiskmgr --list --target
   

   The output of this command should be similar to the folllowing (where the hostname is db01 and the diskgroup_name is DATA):

   Name: iqn.2015-05.com.oracle:QD_DATA_DB01
   Host name: DB01
   ASM disk group name: DATA
   Size: 128 MB
   Visible to: IP_address1, IP_address2, IP_address3, IP_address4... IP_address2n
   Discovered by: IP_address1, IP_address2, IP_address3, IP_address4
   

   IP_address1, IP_address2, IP_address3, IP_address4…IP_address2n above refer to the IP addresses of the InfiniBand interfaces of all the existing cluster nodes. In the example above, the number of number of nodes in the cluster is n.

2. Login to the 2 database nodes that contain the quorum devices and modify the target in each of them to make it visible to the node being added.

   /opt/oracle.SupportTools/quorumdiskmgr --alter --target --asm-disk-group asm_diskgroupname --visible-to 'comma_delimited_list_of_IP_addresses_from_visibleToList_in_step_2_above, IP_addressX, IP_addressY'
   

   IP_addressX and IP_addressY in the previous command refer to the IP addresses of the 2 InfiniBand interfaces of the node being added.

3. Run /opt/oracle.SupportTools/quorumdiskmgr --list –target in the 2 database nodes that contain the quorum devices and make sure the 2 IP addresses of the node being added are seen in the Visible to list.
4. Log in as the root user on the node being added.
5. Run the quorumdiskmgr command with the --create --config option to create quorum disk configurations.

   # /opt/oracle.SupportTools/quorumdiskmgr --create --config --owner=grid --group=dba --network-iface-list="ib0, ib1"
   

6. Run the quorumdiskmgr command with the --list --config option to verify that the configurations have been successfully created on the node.

   # /opt/oracle.SupportTools/quorumdiskmgr --list --config
   

   Owner: grid
   Group: oinstall
   ifaces: exadata_ib1 exadata_ib0
   

7. Run the quorumdiskmgr command with the --create --device option to create the quorum devices on the node being added pointing to the targets for the existing quorum devices.

   # /opt/oracle.SupportTools/quorumdiskmgr --create --device --target-ip-list='comma_delimited_list_of_IP_addresses_from_step_3_above'
   

8. Run the quorumdiskmgr command with the --list --device options to verify the existing quorum devices have been successfully discovered and are visible on the node being added.

   # /opt/oracle.SupportTools/quorumdiskmgr --list –device
   

   On the newly added node the output should be similar to the following and it should be the same as in any of the existing cluster nodes :

   Device path: /dev/exadata_quorum/QD_DATAC1_DB01 
   Size: 128 MB 
   Host name: DB01
   ASM disk group name: DATA 
   Device path: /dev/exadata_quorum/QD_DATAC1_DB02 
   Size: 128 MB 
   Host name: DB02
   ASM disk group name: DATA
   

2.19.5.4.2 Removing a Database Node

If the database node being removed did not host a quorum disk, then no action is required.

If database node being removed hosted a quorum disk containing a voting file and there are fewer than five storage servers in the RAC cluster, then a quorum disk must be created on a different database node before the database node is removed. Follow these steps:

1. Create a quorum disk on a database node that does not currently host a quorum disk.

   1. Log into db01 and db02 as root.

   2. Run the quorumdiskmgr command with the --create --config options to create quorum disk configurations on both db01 and db02.

      # /opt/oracle.SupportTools/quorumdiskmgr --create --config --owner=grid
       --group=dba --network-iface-list="ib0, ib1"
      

   3. Run the quorumdiskmgr command with the --list --config options to verify that the configurations have been successfully created on both db01 and db02.

      # /opt/oracle.SupportTools/quorumdiskmgr --list --config
      

      The output should look like:

      Owner: grid
      Group: dba
      ifaces: exadata_ib1 exadata_ib0
      

   4. Run the quorumdiskmgr command with the --create --target options to create a target on both db01 and db02 for ASM disk group DATAC1 and make the target visible to both db01 and db02.

      # /opt/oracle.SupportTools/quorumdiskmgr --create --target
       --asm-disk-group=datac1
       --visible-to="192.168.10.45, 192.168.10.46, 192.168.10.47, 192.168.10.48"
      

   5. Run the quorumdiskmgr command with the --list --target options to verify the target has been successfully created on both db01 and db02.

      # /opt/oracle.SupportTools/quorumdiskmgr --list --target
      

      On db01, the output should look like:

      Name: iqn.2015-05.com.oracle:QD_DATAC1_DB01 
      Size: 128 MB 
      Host name: DB01
      ASM disk group name: DATAC1 
      Visible to: 192.168.10.45, 192.168.10.46, 192.168.10.47, 192.168.10.48
      Discovered by:
      

      On db02, the output should be:

      Name: iqn.2015-05.com.oracle:QD_DATAC1_DB02 
      Size: 128 MB 
      Host name: DB02
      ASM disk group name: DATAC1 
      Visible to: 192.168.10.45, 192.168.10.46, 192.168.10.47, 192.168.10.48
      Discovered by:
      

   6. Run the quorumdiskmgr command with the --create --device options to create devices on both db01 and db02 from targets on both db01 and db02.

      # /opt/oracle.SupportTools/quorumdiskmgr --create --device
       --target-ip-list="192.168.10.45, 192.168.10.46, 192.168.10.47, 192.168.10.48"
      

   7. Run the quorumdiskmgr command with the --list --device options to verify the devices have been successfully created on both db01 and db02.

      # /opt/oracle.SupportTools/quorumdiskmgr --list --device
      

      On both db01 and db02, the output should look like:

      Device path: /dev/exadata_quorum/QD_DATAC1_DB01 
      Size: 128 MB 
      Host name: DB01
      ASM disk group name: DATAC1 
      Device path: /dev/exadata_quorum/QD_DATAC1_DB02 
      Size: 128 MB 
      Host name: DB02
      ASM disk group name: DATAC1
      

   8. Add the two quorum disk devices to a high redundancy ASM disk group.

      If there is no high redundancy disk group, create a high redundancy disk group and include the two new quorum disks. For example:

      SQL> create diskgroup DATAC1 high redundancy quorum failgroup db01 disk '/dev/exadata_quorum/QD_ DATAC1_DB01' quorum failgroup db02 disk '/dev/exadata_quorum/QD_ DATAC1_DB02' ...
      

      If a high redundancy disk group already exists, add the two new quorum disks. For example:

      SQL> alter diskgroup datac1 add quorum failgroup db01 disk '/dev/exadata_quorum/QD_DATAC1_DB02' quorum failgroup db02 disk '/dev/exadata_quorum/QD_DATAC1_DB01';
      

2. Once the database node is removed, its voting file will get relocated automatically to the quorum disk added in step 1.

2.19.5.4.3 Adding an Oracle Exadata Storage Server and Expanding an Existing High Redundancy Disk Group

When you add a storage server that uses quorum disks, Oracle recommends relocating a voting file from a database node to the newly added storage server.

1. Add the Exadata storage server. See Adding a Cell Node for details.

   In the example below, the new storage server added is called "celadm04".

2. After the storage server is added, verify the new fail group from v$asm_disk.

   SQL> select distinct failgroup from v$asm_disk;
   FAILGROUP
   ------------------------------
   ADM01
   ADM02
   CELADM01
   CELADM02
   CELADM03
   CELADM04
   

3. Verify at least one database node has a quorum disk containing a voting file.

   $ crsctl query css votedisk
   ##  STATE    File Universal Id                File Name Disk group
   --  -----    -----------------                --------- ---------
    1. ONLINE   834ee5a8f5054f12bf47210c51ecb8f4 (o/192.168.12.125;192.168.12.126/DATAC5_CD_00_celadm01) [DATAC5]
    2. ONLINE   f4af2213d9964f0bbfa30b2ba711b475 (o/192.168.12.127;192.168.12.128/DATAC5_CD_00_celadm02) [DATAC5]
    3. ONLINE   ed61778df2964f37bf1d53ea03cd7173 (o/192.168.12.129;192.168.12.130/DATAC5_CD_00_celadm03) [DATAC5]
    4. ONLINE   bfe1c3aa91334f16bf78ee7d33ad77e0 (/dev/exadata_quorum/QD_DATAC5_ADM01) [DATAC5]
    5. ONLINE   a3a56e7145694f75bf21751520b226ef (/dev/exadata_quorum/QD_DATAC5_ADM02) [DATAC5]
   Located 5 voting disk(s).
   

   The example above shows there are two quorum disks with voting files on two database nodes.

4. Drop one of the quorum disks.

   SQL> alter diskgroup datac5 drop quorum disk QD_DATAC5_ADM01;
   

   The voting file on the dropped quorum disk will be relocated automatically to the newly added storage server by the Grid Infrastructure as part of the voting file refresh. You can verify this as follows:

   $ crsctl query css votedisk
   ##  STATE    File Universal Id                File Name Disk group
   --  -----    -----------------                --------- ---------
    1. ONLINE   834ee5a8f5054f12bf47210c51ecb8f4 (o/192.168.12.125;192.168.12.126/DATAC5_CD_00_celadm01) [DATAC5]
    2. ONLINE   f4af2213d9964f0bbfa30b2ba711b475 (o/192.168.12.127;192.168.12.128/DATAC5_CD_00_celadm02) [DATAC5]
    3. ONLINE   ed61778df2964f37bf1d53ea03cd7173 (o/192.168.12.129;192.168.12.130/DATAC5_CD_00_celadm03) [DATAC5]
    4. ONLINE   a3a56e7145694f75bf21751520b226ef (/dev/exadata_quorum/QD_DATAC5_ADM02) [DATAC5]
    5. ONLINE   ab5aefd60cf84fe9bff6541b16e33787 (o/192.168.12.131;192.168.12.132/DATAC5_CD_00_celadm04) [DATAC5]
   

2.19.5.4.4 Removing an Oracle Exadata Storage Server

If removing a storage server results in the number of storage servers being used by the Oracle RAC cluster to fewer than five, and the voting files reside in a high redundancy disk group, then Oracle recommends adding quorum disks to the database nodes, if not in place already.

Prior to removing the storage server, add the quorum disks so that five copies of the voting files are available immediately after removing the storage server.

2.19.6 quorumdiskmgr Reference

The quorum disk manager utility (quorumdiskmgr) runs on each database server to enable you to create and manage iSCSI quorum disks on database servers. You use quorumdiskmgr to create, list, alter, and delete iSCSI quorum disks on database servers. The utility is installed on database servers when they are shipped.

This reference section contains the following topics:

• Syntax for the Quorum Disk Manager Utility

• quorumdiskmgr Objects

• Creating a Quorum Disk Configuration (--create --config)

• Creating a Target (--create --target)

• Creating a Device (--create --device)

• Listing Quorum Disk Configurations (--list --config)

• Listing Targets (--list --target)

• Listing Devices (--list --device)

• Deleting Configurations (--delete --config)

• Deleting Targets (--delete --target)

• Deleting Devices (--delete --device)

• Changing Owner and Group Values (--alter --config)

• Changing the InfiniBand IP Addresses (--alter --target)

2.19.6.1 Syntax for the Quorum Disk Manager Utility

The quorum disk manager utility is a command-line tool. It has the following syntax:

quorumdiskmgr --verb --object [--options] 

verb is an action performed on an object. It is one of: alter, create, delete, list.

object is an object on which the command performs an action.

options extend the use of a command combination to include additional parameters for the command.

When using the quorumdiskmgr utility, the following rules apply:

• Verbs, objects, and options are case-sensitive except where explicitly stated.

• Use the double quote character around the value of an option that includes spaces or punctuation.

2.19.6.2 quorumdiskmgr Objects

Object	Description
config	The quorum disk configurations include the owner and group of the ASM instance to which the iSCSI quorum disks will be added, and the list of network interfaces through which local and remote iSCSI quorum disks will be discovered.
target	A target is an endpoint on each database server that waits for an iSCSI initiator to establish a session and provides required IO data transfer.
device	A device is an iSCSI device created by logging into a local or remote target.

2.19.6.3 Creating a Quorum Disk Configuration (--create --config)

The --create --config action creates a quorum disk configuration. The configuration must be created before any targets or devices can be created.

Syntax

quorumdiskmgr --create --config [--owner owner --group group] --network-iface-list network-iface-list

Parameters

The following table lists the parameters for the --create --config action:

Parameter	Description
owner	Specifies the owner of the ASM instance to which the iSCSI quorum disks will be added. This is an optional parameter. The default value is grid.
group	Specifies the group of the ASM instance to which the iSCSI quorum disks will be added. This is an optional parameter. The default value is dba.
network-iface-list	Specifies the list of network interface names through which the local and remote targets will be discovered.

Example

quorumdiskmgr --create --config --owner=oracle --group=dba --network-iface-list="ib0, ib1"

2.19.6.4 Creating a Target (--create --target)

The --create --target action creates a target that can be accessed by database servers with an InfiniBand IP address in the specified InfiniBand IP address list and will be used to create devices that will be added to the specified ASM disk group.

After a target is created, its asm-disk-group, host-name, and size attributes cannot be changed.

Syntax

quorumdiskmgr --create --target --asm-disk-group asm_disk_group --visible-to infiniband_ip_list [--host-name host_name] [--size size]

Parameters

Parameter	Description
asm-disk-group	Specifies the ASM disk group to which the device created from the target will be added. The value of asm-disk-group is not case-sensitive.
visible-to	Specifies a list of InfiniBand IP addresses. Database servers with an InfiniBand IP address in the list will have access to the target.
host-name	Specifies the host name of the database server on which quorumdiskmgr runs. The total length of asm-disk-group and host-name cannot exceed 26 characters. If the host name is too long, a shorter host name can be specified as long as a different host name is specified for each database server in the quarter rack. This is an optional parameter. The default value is the host name of the database server on which quorumdiskmgr runs. The value of host-name is not case-sensitive.
size	Specifies the size of the target. This is an optional parameter. The default value is 128 MB.

Example

quorumdiskmgr --create --target --size=128MB --asm-disk-group=datac1 --visible-to="192.168.10.45, 192.168.10.46" --host-name=db01

2.19.6.5 Creating a Device (--create --device)

The --create --device action creates devices by discovering and logging into targets on database servers with an InfiniBand IP address in the specified list of IP addresses.

The created devices will be automatically discovered by the ASM instance with the owner and group specified during configuration creation.

Syntax

quorumdiskmgr --create --device --target-ip-list target_ip_list

Parameters

Parameter	Description
target-ip-list	Specifies a list of InfiniBand IP addresses. quorumdiskmgr will discover targets on database servers with an InfiniBand IP address in the list and log into the targets to create devices.

Example

quorumdiskmgr --create --device --target-ip-list="192.168.10.45, 192.168.10.46"

2.19.6.6 Listing Quorum Disk Configurations (--list --config)

The --list --config action lists the quorum disk configurations.

Syntax

quorumdiskmgr --list --config

Sample Output

Owner: grid
Group: dba
ifaces: exadata_ib1 exadata_ib0

2.19.6.7 Listing Targets (--list --target)

The --list --target action lists the attributes of targets, including target name, size, host name, ASM disk group name, the list of IP addresses (a visible-to IP address list) indicating which database servers have access to the target, and the list of IP addresses (a discovered-by IP address list) indicating which database servers have logged into the target.

If an ASM disk group name is specified, the action lists all local targets created for the specified ASM disk group. Otherwise, the action lists all local targets created for quorum disks.

Syntax

quorumdiskmgr --list --target [--asm-disk-group asm_disk_group]

parameters

Parameter	Description
asm-disk-group	Specifies the ASM disk group. quorumdiskmgr displays all local targets for this ASM disk group. The value of asm-disk-group is not case-sensitive.

Sample Output

Name: iqn.2015-05.com.oracle:QD_DATAC1_DB01 
Size: 128 MB 
Host name: DB01 
ASM disk group name: DATAC1 
Visible to: 192.168.10.48, 192.168.10.49, 192.168.10.46, 192.168.10.47 
Discovered by: 192.168.10.47, 192.168.10.46

2.19.6.8 Listing Devices (--list --device)

The --list --device action lists the attributes of devices, including device path, size, host name and ASM disk group name.

If only an ASM disk group name is specified, the action lists all the devices that have been added to the ASM disk group.

If only a host name is specified, the action lists all the devices created from the targets on the host.

If both an ASM disk group name and a host name are specified, the action lists a single device created from the target on the host and that has been added to the ASM disk group.

If neither an ASM disk group name nor a host name is specified, the action lists all quorum disk devices.

Syntax

quorumdiskmgr --list --device [--asm-disk-group asm_disk_group] [--host-name host_name]

parameters

Parameter	Description
asm-disk-group	Specifies the ASM disk group to which devices have been added. The value of asm-disk-group is not case-sensitive.
host-name	Specifies the host name of the database server from whose targets devices are created. The value of host-name is not case-sensitive.

Sample Output

Device path: /dev/exadata_quorum/QD_DATAC1_DB01 
Size: 128 MB 
Host name: DB01 
ASM disk group name: DATAC1 

Device path: /dev/exadata_quorum/QD_DATAC1_DB02 
Size: 128 MB 
Host name: DB02
ASM disk group name: DATAC1

2.19.6.9 Deleting Configurations (--delete --config)

The --delete --config action deletes quorum disk configurations. The configurations can only be deleted when there are no targets or devices present.

Syntax

quorumdiskmgr --delete --config

2.19.6.10 Deleting Targets (--delete --target)

The --delete --target action deletes the targets created for quorum disks on database servers.

If an ASM disk group name is specified, the action deletes all local targets created for the specified ASM disk group. Otherwise, the action deletes all local targets created for quorum disks.

Syntax

quorumdiskmgr --delete --target [--asm-disk-group asm_disk_group]

Parameters

Parameter	Description
asm-disk-group	Specifies the ASM disk group. Local targets created for this disk group will be deleted. The value of asm-disk-group is not case-sensitive.

Example

quorumdiskmgr --delete --target --asm-disk-group=datac1

2.19.6.11 Deleting Devices (--delete --device)

The --delete --device action deletes quorum disk devices.

If only an ASM disk group name is specified, the action deletes all the devices that have been added to the ASM disk group.

If only a host name is specified, the action deletes all the devices created from the targets on the host.

If both an ASM disk group name and a host name are specified, the action deletes a single device created from the target on the host and that has been added to the ASM disk group.

If neither an ASM disk group name nor a host name is specified, the action deletes all quorum disk devices.

Syntax

quorumdiskmgr --delete --device [--asm-disk-group asm_disk_group] [--host-name host_name]

Parameters

Parameter	Description
asm-disk-group	Specifies the ASM disk group whose device you want to delete. The value of asm-disk-group is not case-sensitive.
host-name	Specifies the host name of the database server. Devices created from targets on this host will be deleted. The value of host-name is not case-sensitive.

Example

quorumdiskmgr --delete --device --host-name=db01

2.19.6.12 Changing Owner and Group Values (--alter --config)

The --alter --config action changes the owner and group configurations.

Syntax

quorumdiskmgr --alter --config --owner owner --group group

Parameters

Parameter	Description
owner	Specifies the new owner for the quorum disk configuration. This parameter is optional. If not specified, the owner is unchanged.
group	Specifies the new group for the quorum disk configuration. This parameter is optional. If not specified, the group is unchanged.

Example

quorumdiskmgr --alter --config --owner=grid --group=dba

2.19.6.13 Changing the InfiniBand IP Addresses (--alter --target)

The --alter --target action changes the InfiniBand IP addresses of the database servers that have access to the local target created for the specified ASM disk group.

Syntax

quorumdiskmgr --alter --target --asm-disk-group asm_disk_group --visible-to infiniband_ip_list

Parameters

Parameter	Description
asm-disk-group	Specifies the ASM disk group to which the device created from the target will be added. The value of asm-disk-group is not case-sensitive.
visible-to	Specifies a list of InfiniBand IP addresses. Database servers with an InfiniBand IP address in the list will have access to the target.

Example

quorumdiskmgr --alter --target --asm-disk-group=datac1 --visible-to="192.168.10.45, 192.168.10.47"

2.20 Using vmetrics

The vmetrics package enables you to display system statistics gathered by the vmetrics service. You can access the system statistics from dom0 or domU. The vmetrics service runs on dom0, collects the statistics, and pushes them to the xenstore. This allows the domU's to access the statistics.

System statistics collected by the vmetrics service are shown below, with sample values:

com.sap.host.host.VirtualizationVendor=Oracle Corporation;

com.sap.host.host.VirtProductInfo=Oracle VM 3;

com.sap.host.host.PagedInMemory=0;

com.sap.host.host.PagedOutMemory=0;

com.sap.host.host.PageRates=0;

com.sap.vm.vm.uuid=2b80522b-060d-47ee-8209-2ab65778eb7e;

com.sap.host.host.HostName=scac10adm01.us.oracle.com;

com.sap.host.host.HostSystemInfo=scac10adm01;

com.sap.host.host.NumberOfPhysicalCPUs=24;

com.sap.host.host.NumCPUs=4;

com.sap.host.host.TotalPhyMem=98295;

com.sap.host.host.UsedVirtualMemory=2577;

com.sap.host.host.MemoryAllocatedToVirtualServers=2577;

com.sap.host.host.FreeVirtualMemory=29788;

com.sap.host.host.FreePhysicalMemory=5212;

com.sap.host.host.TotalCPUTime=242507.220000;

com.sap.host.host.Time=1453150151;

com.sap.vm.vm.PhysicalMemoryAllocatedToVirtualSystem=8192;

com.sap.vm.vm.ResourceMemoryLimit=8192;

com.sap.vm.vm.TotalCPUTime=10160.1831404;

com.sap.vm.vm.ResourceProcessorLimit=4;

2.20.1 Installing and Starting the vmetrics Service

To install the vmetrics service, run the install.sh script as the root user on dom0:

[root@scac10adm01]# cd /opt/oracle.SupportTools/vmetrics
[root@scac10adm01]# ./install.sh

The install.sh script verifies that it is running on dom0, stops any vmetrics services currently running, copies the package files to /opt/oracle.vmetrics, and copies vmetrics.svc to /etc/init.d.

To start the vmetrics service on dom0, run the following command as the root user on dom0:

[root@scac10adm01 vmetrics]# service vmetrics.svc start

The commands to gather the statistics are run every 30 seconds.

2.20.2 Files in the vmetrics Package

The vmetrics package contains the following files:

File	Description
install.sh	This file installs the package.
vm-dump-metrics	This script reads the statistics from the xenstore and displays them in XML format.
vmetrics	This Python script runs the system commands and uploads them to the xenstore. The system commands are listed in the vmetrics.conf file.
vmetrics.conf	This XML file specifies the metrics that the dom0 should push to the xenstore, and the system commands to run for each metric.
vmetrics.svc	The init.d file that makes vmetrics a Linux service.

2.20.3 Displaying the Statistics

Once the statistics have been pushed to the xenstore, you can view the statistics on dom0 and domU by running either of the following commands:

Note:

On domU's, ensure that the xenstoreprovider and ovmd packages are installed.

xenstoreprovider is the library which communicates with the ovmapi kernel infrastructure.

ovmd is a daemon that handles configuration and reconfiguration events and provides a mechanism to send/receive messages between the VM and the Oracle VM Manager.

The following command installs the necessary packages on Oracle Linux 5 and 6 to support the Oracle VM API.

# yum install ovmd xenstoreprovider

• The /usr/sbin/ovmd -g vmhost command displays the statistics on one line. The sed command breaks up the line into multiple lines, one statistic per line. You need to run this command as the root user.

  root@scac10db01vm04 ~]# /usr/sbin/ovmd -g vmhost |sed 's/; */;\n/g;s/:"/:"\n/g'
  com.sap.host.host.VirtualizationVendor=Oracle Corporation;
  com.sap.host.host.VirtProductInfo=Oracle VM 3;
  com.sap.host.host.PagedInMemory=0;
  com.sap.host.host.PagedOutMemory=0;
  com.sap.host.host.PageRates=0;
  com.sap.vm.vm.uuid=2b80522b-060d-47ee-8209-2ab65778eb7e;
  com.sap.host.host.HostName=scac10adm01.us.oracle.com;
  com.sap.host.host.HostSystemInfo=scac10adm01;
  com.sap.host.host.NumberOfPhysicalCPUs=24;
  com.sap.host.host.NumCPUs=4;
  ...
  

• The vm-dump-metrics command displays the metrics in XML format.

  [root@scac10db01vm04 ~]# ./vm-dump-metrics
  <metrics>
  <metric type='real64' context='host'>
  <name>TotalCPUTime</name>
  <value>242773.600000</value>
  </metric>
  <metric type='uint64' context='host'>
  <name>PagedOutMemory</name>
  <value>0</value>
  </metric>
  ...
  

  Note that you have copy the vm-dump-metrics command to the domU's from which you want to run the command.

2.20.4 Adding Metrics to vmetrics

You can add your own metric to be collected by the vmetrics service.

1. In /opt/oracle.SupportTools/vmetrics/vmetrics.conf, add the new metric and the system commands to retrieve and parse that metric. For example:

   <metric type="uint32" context="host">
    <name>NumCPUs</name>
    <action>grep -c processor /proc/cpuinfo</action>
    <action2>xm list | grep '^Domain-0' |awk '{print $4}'</action2>
   </metric>
   

   In the <name> element, enter the name of the new metric.

   In the <action> and <action2> elements, specify the system command for the new metric. You only need to have <action2>, but you can use <action> as a fallback in case <action2> does not work on some systems.

   Note that any action that needs the name of the vm should be done with scas07client07vm01. When vmetrics runs, it swaps out this dummy name for the actual domU names that are running in the dom0.

2. In /opt/oracle.SupportTools/vmetrics/vmetrics, add the metric in the list gFieldsList. Prefix the metric name with "host" if the metric is about the host (dom0) or with "vm" if the metric is about the vm (domU). For example:

   Suppose the gFieldsList looks like this:

   gFieldsList = [ 'host.VirtualizationVendor',
       'host.VirtProductInfo',
       'host.PagedInMemory',
       'vm.ResourceProcessorLimit' ]
   

   If you are adding a new metric called "NumCPUs" (as shown in the example in step 1), and this metric is intended to tell the domU how many cpu's the dom0 has available, then gFieldsList would now look like:

    gFieldsList = [ 'host.VirtualizationVendor',
       'host.VirtProductInfo',
       'host.PagedInMemory',
       'vm.ResourceProcessorLimit',
       'host.NumCPUs']
   

3. (optional) In /opt/oracle.SupportTools/vmetrics/vm-dump-metrics, add the new metric if you want the new metric to be included in the XML output.

   If you skip this step, you can view the new metric using the ovmd -g vmhost command.