5.7 Finalizing Upgrades on Oracle VM Server for x86

This section describes different post-upgrade steps that may be required after an upgrade.

5.7.1 Refreshing Server Pools To Validate Oracle VM Storage Connect plug-in

After an upgrade, it is good practice to refresh your server pools to ensure that the information presented within Oracle VM Manager is up to date and synchronized with the actual state of any servers in a server pool. More importantly, if you are using any third party Oracle VM Storage Connect plug-in for your storage devices, you may be unable to refresh your storage arrays until these plug-ins have been validated. Refreshing server pools allows the servers that have been presented to the storage to validate the Oracle VM Storage Connect plug-in that they are using. Once this has been done, operations on storage arrays can resume normally.

To refresh your server pools, refer to Refresh All in the Oracle VM Manager User's Guide.

5.7.2 Editing Boot Parameters To Enable Hypervisor Features

If you have upgraded from an earlier release, it is possible that some of the hypervisor boot parameters for your server may need editing to ensure that newer features are enabled. To update these boot parameters, on each affected Oracle VM Server, open the grub configuration in an editor. If you are using UEFI boot, the grub configuration file is located at /boot/efi/EFI/redhat/grub.cfg, otherwise the grub configuration file is located at /boot/grub2/grub.cfg. Edit the line starting with multiboot2 /xen.mb.efi and append the required boot parameters.

For large servers with more than 20 physical CPUs

For Oracle VM Servers with more than 20 physical CPUs, performance can be improved by pinning virtual CPUs to dom0 and setting a maximum value for the number of virtual CPUs that can be used by dom0. On a new install where a system has more than 20 physical CPUs, this value is usually set to 20. To ensure that you get the best possible performance on an Oracle VM Server with more than 20 physical CPUs, set the Xen boot parameters dom0_vcpus_pin and dom0_max_vcpus=20. For example:

multiboot2 /xen.mb.efi dom0_mem=max:6144M  dom0_vcpus_pin \
  dom0_max_vcpus=20 placeholder ${xen_rm_opts}


Some versions of the Xen 4.4.4 kernel require that you set the dom0_vcpus_pin parameter to dom0_vcpus_pin=true for the configuration to take effect. Run the following command on Oracle VM Server to find the Xen version: rpm -qa | grep xen.

If you have previously updated the Xen boot parameters on an earlier Oracle VM 3.4.x release, and then upgraded to Oracle VM Release 3.4.5 or later with Xen kernel version 4.4.4-130 or higher, either syntax for the parameter is accepted.

The standard syntax as of Oracle VM Release 3.4.5 is dom0_vcpus_pin; there is no need to append =true.

For Oracle VM Servers with fewer than 20 physical CPUs, you should ensure that the number of virtual CPUs specified is reduced accordingly and does not exceed the number of physical CPUs per socket. Assigning too many virtual CPUs can cause Non-Uniform Memory Access (NUMA) latency issues if the virtual CPUs are spread across sockets. For example, if an instance of Oracle VM Server with hyper-threading enabled has 2 sockets with 18 cores per socket and 2 CPU threads per core, this results in a total of 36 physical CPUs. In this case, it is correct to set dom0_max_vcpus to 20. However, with hyper-threading disabled, only 18 physical CPUs are on each socket and as a result, you should set the dom0_max_vcpus parameter to 18.

When you have finished modifying the grub configuration file, for example, /etc/default/grub, you must rebuild the system GRUB2 configuration that is used at boot time. This is done by running:

# grub2-mkconfig -o /boot/grub2/grub.cfg

You must also reboot the server for these settings to take effect.

For information on performance optimization goals and techniques for Oracle VM Server for x86, see Optimizing Oracle VM Server for x86 Performance, on Oracle Technology Network at: http://www.oracle.com/technetwork/server-storage/vm/ovm-performance-2995164.pdf.

5.7.3 Confirm kdump Service Settings after Upgrading

If the kdump service was enabled previously, it is recommended that you check and confirm the settings are correct after the upgrade or enable the kdump service after the upgrade if it was not enabled previously.

For more information, see Manually Configuring kdump for Oracle VM Server in the Oracle VM Administrator's Guide.

5.7.4 Handling Missing Disks, File Systems and Repositories

On a small number of systems, local physical disks may be allocated an alternate device mapper ID during the upgrade process. This only affects a few different ATA type disks. The change in device mapper ID can result in some local physical disks, their file systems and repositories as being marked as missing within Oracle VM Manager. Although the physical disk is discovered after the upgrade, it is detected as a new disk. The old entry within Oracle VM Manager, for the same disk, is retained but is marked as missing.

If you are affected by this, you may need to take some steps to reconfigure your environment for the change. This includes removing the missing disks and then refreshing the newly discovered disks, their file systems and any hosted repositories. The repository must additionally be presented to the affected Oracle VM Server again, so that it can be used.

If you have virtual machines that use local physical disks for storage, you may need to reconfigure these virtual machines to remove the missing disk and replace it with the newly discovered disk.

This section describes how to identify whether you are affected and the steps that you may need to take to return to a fully functioning environment.

Checking For Missing Disks

To see whether you are affected by this issue at all, you can use the Oracle VM Manager Web Interface to view the Physical Disks perspective for each Oracle VM Server in your environment. If a disk has been affected by the upgrade, it is displayed with a WARNING status. The event message shows "Physical disk is Offline".

In an environment with many Oracle VM Servers, it may not be practical to use the Oracle VM Manager Web Interface to check the status of the physical disks for every server. You can, however, use the Oracle VM Manager Command Line Interface in combination with the grep utility to quickly view all of the events that contain this event message. An example, where the command is run on the Oracle VM Manager host, follows:

$ ssh -l admin localhost -p 10000 "getEvents severity=WARNING" | grep -i "physical \
  disk is offline"

id:3190     createTime:timestamp     modTime:timestamp
type:storage.device.offline.     severity:Warning     summary:Physical disk is
Offline     acked:No description:OVMEVT_007005D_001 Rescan storage layer on
server [ovs237] did not return physical disk [3600605b0057feba01a0e216512d1b7aa]
for storage array [Generic Local Storage Array @ ovs237].

Note that for these events, you must check that the event occurs for a Generic Local Storage Array. The server that is affected is displayed in the output, as is the id for the missing disk.

If this event message exists for any of the local physical disks attached to any of your Oracle VM Servers, then you are most likely affected by this issue and should continue with the rest of the steps described here.

Identifying Newly Discovered Disks and Matching Them To Missing Disks

By default, the simple names that Oracle VM Manager assigns to physical disks are based on the disk Page83 ID. If you have not renamed your physical disks, identifying newly discovered disks that map onto missing disks is simple, since the disk names can be compared to find matches. This is illustrated in the following output from the Oracle VM Manager Command Line Interface:

OVM> list physicalDisk
Command: list physicalDisk
Status: Success
Time: timestamp

In this output, the first physical disk listed is the original disk item, while the second disk is the newly discovered version of the same disk. The first physical disk is marked as missing in Oracle VM Manager. Since the second disk has the same name, with the exclusion of the final underscore character, it is easy to identify that these entries actually point to the same disk.

If you have changed the names for your physical disks, this process is fractionally more complicated since you must compare the Page83 ID of each disk to find the match. Obtaining the Page83 ID of a disk using the Oracle VM Manager Command Line Interface is illustrated below:

OVM> show physicaldisk id=0004fb00001800009eed86fbc36b41bc
Command: show physicaldisk id=0004fb00001800009eed86fbc36b41bc
Status: Success
Time: timestamp
  Page83 ID = 350014ee20137ee44
  Server Reserved = No
  Shareable = No
  Size (GiB) = 465.76
  State = UNKNOWN
  Thin Provision = No
  Type = LUN
  Vendor = ATA
  File System 1 = 0004fb00000500000809e28f4fab56b1  [fs on 350014ee20137ee44]
  Volume Group = 0004fb00003200004dddc710a12aa1b7  [Local Storage Volume Group]
  Id = 0004fb00001800009eed86fbc36b41bc  [Server5 ATA Disk]
  Name = Server5 ATA Disk
  Description = WDC WD5001ABYS-0
  Locked = false

Note that the Page83 ID is shown as a property of the disk. You must obtain the Page83 ID for each missing disk and then find any physical disks that have their name or Page83 ID set to the same value. These disks are the newly discovered versions of the same physical disk.

Useful Steps Before Reconfiguring Your Environment

The names of any child objects of the missing disk, such as file system names, repository names and the names of any items within the repository are all associated with the missing disk. This information is lost when you delete the missing disk from your environment. If you have named any of these objects within Oracle VM Manager, you should take note of the names of these objects before continuing. Although this is not a necessary step, since the names are only helpful within Oracle VM Manager and do not affect the running of your environment, to return your environment to the same state after you delete any missing disks you may wish to rename these objects once they are associated with the newly discovered version of the same physical disk.

Using the Oracle VM Manager Command Line Interface you can run the following commands to obtain a dump of the names of all of these objects:

OVM> list physicaldisk
OVM> list filesystem
OVM> list repository
OVM> show repository id=0004fb0000030000e0f09683cd2ac72a

Substitute 0004fb0000030000e0f09683cd2ac72a with the id for each repository in your environment, and run the same command for each repository.

The goal of this exercise is to obtain a listing of the names of any of the objects that may be affected when you remove the physical disk that they are associated with. Store the information returned by these commands in a place that you can refer to when you need to rename objects.

Steps To Deal With Affected Repositories

If you have made a record of the object names of all affected objects, it is safe to delete the missing disks, as illustrated in the following output from the Oracle VM Manager Command Line Interface:

OVM> delete physicalDisk id=0004fb00001800007b2cdfbecb973c4d
Command: delete physicalDisk id=0004fb00001800007b2cdfbecb973c4d
Status: Success
Time: timestamp
JobId: 1399409030129

These actions can also be performed in the Oracle VM Manager Web Interface, if preferred.

Once you have deleted the missing disk entry, you must perform the following actions:

At this point, your environment is usable, although the names of various objects have been lost. For instance, any file system names, repository names, or the names of objects such as virtual machines in a repository are likely to have been reset to default values. If you kept a record of the object names affected, as recommended in the previous step, then you can manually rename objects as required.

Steps To Deal With Affected Virtual Machines

If you have any virtual machines that were configured to use a local physical disk that is marked as missing, you must remove the original disk mapping and then reconfigure the virtual machine to attach a new physical disk. When you select the new physical disk, you should select the physical disk with the same Page83 ID as the missing disk. To reconfigure these disk mappings, you must edit the virtual machine as described in Edit Virtual Machine in the Oracle VM Manager User's Guide.