Upgrading an existing Oracle Private Cloud Appliance environment to Controller Software Release 2.3.1-2.3.3 and Oracle VM 3.4.x is a highly complicated procedure. A number of requirements must be met before the upgrade can be attempted, steps must be performed in a very particular order, and various validations take place at different levels and stages.
Major challenges in the upgrade procedure include compute nodes running different versions of Oracle VM Server, pre-existing errors in log files, YUM configuration changes, firmware compatibility issues, and so on. These can lead to a variety of problem scenarios, which the upgrade utilities help you to detect and work around. This section explains how to use the upgrade utilities to work through the different pre-upgrade stages, and how to recover and proceed when certain failures occur. Additional, non-automated checks are required after the upgrade utilities have complete all checks successfully.
If the upgrade utilities report errors, additional details can
be found in this log file:
/tmp/.
<date_time>/pre_upgrade_check.log
Executing the Pre-Upgrade Validation Script with the Upgrade Utilities
The upgrade utilities are provided with the Oracle Private Cloud Appliance Release 2.3.1, 2.3.2 or 2.3.3
*.isofile. Due to its total size, the*.isofile is zipped in multiple parts. Download the different parts of the*.zipfile and unzip them in a directory of your choice. The directory should contain the following files:ovca-2.3.
x-bxxxx_1 ovca-2.3.x-bxxxx_2 ovca-2.3.x-bxxxx_header README.html RUN_ME_FIRST.sh support.txtIn the directory where you unzipped the downloaded files, execute the
RUN_ME_FIRST.shscript.# cd
<download_directory># ./RUN_ME_FIRST.sh Before upgrading, review the following documentation: IMPORTANT: If you are at any version older than PCA 2.1.1, please refer to the matrix of Supported Upgrade Paths in the README file, to determine if a direct upgrade to 2.3.xis possible from your current PCA version. Once you are running a version that can support upgrade to 2.3.x, including all Compute Nodes running Oracle VM Server version 3.2.10 or later, you may proceed with the upgrade to 2.3.x. This release comes with a set of prerequisite checks which must be performed in order to ensure a successful upgrade to PCA 2.3.xfrom version PCA 2.1.1 or newer. These checks are listed in My Oracle Support Knowledge Article: ****[ PCA ] 2.3.X Upgrade Checklist and Prerequisites (Doc ID 2242177.1)****. It is highly recommended that a Service Request with Oracle Support be opened, pro-actively, prior to performing the upgrade. This ensures that the pre-upgrade checklist can be validated and enables Oracle Support to assist while the upgrade is performed. Proceed [Y/N]? (Press Y to continue)Carefully read and understand the information displayed in the terminal window. Enter "Y" to continue. The script assembles the zipped Release 2.3.x
*.isofile from the downloaded parts.Proceed [Y/N]? (Press Y to continue) y Generating the PCA 2.3.
xISO... This may take some time. Please wait... 2f783e679de1dc6cc3652a745d463ce0 ./ovca-2.3.x-bxxxx.iso.zipCopy and unzip the Release 2.3.x image file to the active management node and mount it as a loopback device.
# mkdir /mnt/pca_2.3.
x_iso # mount -o loop ovca-2.3.x-bxxxx.iso /mnt/pca_2.3.x_isoChange to the
scriptssubdirectory, and then execute the scriptpre_upgrade_check.sh.# cd /mnt/pca_2.3.
x_iso/scripts # ./pre_upgrade_check.sh Validating installed PCA version...If an invalid version of the Oracle PCA Controller Software is detected, the script fails with a fatal error.
Upgrading from PCA 2.0.5 to PCA 2.3.3 is not a valid upgrade path FATAL ERROR: DO NOT PROCEED WITH THE UPGRADE! Unsupported upgrade path violations will cause severe damage to the rack [Minimum Installed PCA Version Check] FAILED: PCA 2.3.3 only supports upgrading from PCA 2.1.1 or above.
Make sure your current version of the controller software is Release 2.1.1 or newer. Then restart this procedure.
If validation of the installed Oracle PCA Controller Software is successful, the script proceeds to the management node and compute node checks. Track progress of the script in the console output.
Upgrading from PCA 2.2.1 to PCA 2.3.3 is a supported upgrade path [Minimum Installed PCA Version Check] PASSED: PCA 2.3.3 only supports upgrading from PCA 2.1.1 or above. Running MN and CN pre-upgrade checks... NODE: 192.168.4.4 MASTER: True [05/22/2017 14:37:53 812981] INFO (upgrade_utils:822) Running PCA Pre-upgrade Checks.... [05/22/2017 14:37:53 812981] INFO (upgrade_utils:833) Checking for Oracle VM Manager Upgrade...
The script checks known log file locations for existing occurrences of
ObjectNotFoundException. If such exceptions are found, the script outputs error messages to the console, the summary report and the log."[...]odof.exception.ObjectNotFoundException: No such object(level 1), cluster is null: <1759>" is found in/u01/app/oracle/ovm-manager-3/machine1/base_adf_domain/servers/AdminServer/logs/AdminServer.log* [05/22/2017 14:37:54 812981] ERROR (upgrade_utils:680) [ONF Check]FAILED: The check failed on the following management nodes: ['ovcamn05r1','ovcamn06r1'].
If even a single
ObjectNotFoundExceptionis found, attempts to update the controller software will fail. First, make sure that allObjectNotFoundExceptionoccurrences are properly addressed. Contact Oracle for support. For additional information, see Section 8.11, “Eliminating ObjectNotFound Exceptions and Restoring the Oracle VM Manager Database”.When the issues have been addressed, make sure the exceptions are removed from the log directories of both management nodes. For additional information, refer to Software Update Fails Due to Error in
AdminServer.login the Oracle Private Cloud Appliance Release Notes. Then restart this procedure.When the script finds no
ObjectNotFoundExceptionon either management node, the [ONF Check] is marked asPASSED.The script checks the status of the compute nodes and management nodes, verifies that their configuration is valid, and confirms that they run a valid version of Oracle Linux and Oracle VM Server. Because the servers are checked one by one, this phase may take some time to complete. If a problem is found, the output contains failures and indicates on which nodes they occurred, as shown in the example below.
[OS Check] FAILED: The check failed on the following nodes: ['ovcacn29r1', 'ovcacn32r1'].
Make sure that all compute nodes are running Oracle VM Server Release 3.2.10 or 3.2.11. Then restart this procedure.
When all management nodes and compute nodes are running a valid version of their operating system, the console output indicates that the servers have passed the validation.
[OS Check] PASSED: The compute nodes are running the correct Oracle Linux version. [OS Check] PASSED: The management nodes are running the correct Oracle Linux version. [Oracle VM Minimum Version Check] PASSED: The compute nodes are running Oracle VM equal or above the upgrade minimum version 3.2.10. [Oracle VM Minimum Version Check] PASSED: The management nodes are running Oracle VM equal or above the upgrade minimum version 3.2.10.
In this phase of the pre-upgrade check, the script also verifies that requirements are met with regard to these items:
ZFS directory mount points
banner file (
/etc/motd)default Oracle VM database objects
If the MOTD check fails, empty the banner file. Contact Oracle Support if any of the other checks fail.
[ZFS Dir Mounted Check] PASSED: ZFS directory MGMT_ROOT, Yum, and Incoming are mounted on the management nodes. [MOTD Check] PASSED: Banner file /etc/motd is empty on the management nodes. [OVMM Model DB Check] PASSED: Default OVMM objects exist on the management nodes.
A number of key components are subjected to a password check. This ensures that the passwords stored in the Oracle PCA Wallet match the passwords configured on the respective components. These components are:
Oracle Switch ES1-24 (
OPUS)Oracle Fabric Interconnect F1-15 (
Xsigo)Oracle VM Manager
Oracle MySQL database
Oracle ZFS Storage Appliance and its ILOM (
ZFS)all compute nodes and their ILOMs
[05/22/2017 14:38:43 812981] DEBUG (upgrade_utils:680) DEBUG (upgrade_utils:680) [Mysql password Check] PASSED: Password has been verified on Mysql. [05/22/2017 14:38:45 812981] INFO (upgrade_utils:231) Successfully verified OPUS password on 192.168.4.200. [05/22/2017 14:38:47 812981] INFO (upgrade_utils:231) Successfully verified OPUS password on 192.168.4.201. [05/22/2017 14:38:47 812981] DEBUG (upgrade_utils:680) [Password Check] PASSED: Password has been verified on OPUS. [05/22/2017 14:38:51 812981] INFO (upgrade_utils:231) Successfully verified Xsigo password on 192.168.4.204. [05/22/2017 14:38:56 812981] INFO (upgrade_utils:231) Successfully verified Xsigo password on 192.168.4.205. [05/22/2017 14:38:56 812981] DEBUG (upgrade_utils:680) [Password Check] PASSED: Password has been verified on Xsigo. [05/22/2017 14:38:57 812981] INFO (upgrade_utils:231) Successfully verified Oracle VM Manager password on 192.168.4.216. [05/22/2017 14:38:57 812981] DEBUG (upgrade_utils:680) [Password Check] PASSED: Password has been verified on Oracle VM Manager. [05/22/2017 14:38:57 812981] INFO (upgrade_utils:231) Successfully verified ZFS password on 192.168.4.2. [05/22/2017 14:38:57 812981] INFO (upgrade_utils:231) Successfully verified ZFS password on 192.168.4.1. [05/22/2017 14:38:57 812981] DEBUG (upgrade_utils:680) [Password Check] PASSED: Password has been verified on ZFS. [05/22/2017 14:39:02 812981] INFO (upgrade_utils:231) Successfully verified Compute Node ILOM password on 192.168.4.107. [05/22/2017 14:39:09 812981] INFO (upgrade_utils:231) Successfully verified Compute Node ILOM password on 192.168.4.106. [05/22/2017 14:39:14 812981] INFO (upgrade_utils:231) Successfully verified Compute Node ILOM password on 192.168.4.105. [05/22/2017 14:39:14 812981] DEBUG (upgrade_utils:680) [Password Check] PASSED: Password has been verified on Compute Node ILOM. [05/22/2017 14:39:15 812981] INFO (upgrade_utils:231) Successfully verified Compute Node host password on 192.168.4.5. [05/22/2017 14:39:15 812981] INFO (upgrade_utils:231) Successfully verified Compute Node host password on 192.168.4.6. [05/22/2017 14:39:16 812981] INFO (upgrade_utils:231) Successfully verified Compute Node host password on 192.168.4.7. [05/22/2017 14:39:17 812981] DEBUG (upgrade_utils:680) [Password Check] PASSED: Password has been verified on Compute Node host. [05/22/2017 14:39:21 812981] INFO (upgrade_utils:231) Successfully verified ZFS ILOM password on 192.168.4.101. [05/22/2017 14:39:26 812981] INFO (upgrade_utils:231) Successfully verified ZFS ILOM password on 192.168.4.102. [05/22/2017 14:39:27 812981] DEBUG (upgrade_utils:680) [Password Check] PASSED: Password has been verified on ZFS ILOM.
If a component fails the password check, set the password correctly and restart this procedure.
As part of the Oracle VM Release 3.4.x upgrade, the compute node operating system is also upgraded from Oracle Linux 5 to Oracle Linux 6. Therefore, the pre-upgrade script compares the list of installed packages with the Oracle Linux 6 packages in the Oracle PCA
*.isoimage.For a successful software update, all natively installed packages on the compute nodes must have an Oracle Linux 6 equivalent. Non-native packages are ignored.
DEBUG (upgrade_utils:395) Compute node 192.168.4.7: Ignoring non-native installed package: oswatcher-profile DEBUG (upgrade_utils:395) Compute node 192.168.4.7: Ignoring non-native installed package: ovs-support-tools DEBUG (upgrade_utils:395) Compute node 192.168.4.7: Ignoring non-native installed package: python-tools [...] DEBUG (upgrade_utils:395) Compute node 192.168.4.5: Ignoring non-native installed package: xen-debugger DEBUG (upgrade_utils:395) Compute node 192.168.4.5: Ignoring non-native installed package: xen-pvhvm-devel DEBUG (upgrade_utils:680) [YUM Repo Check] PASSED: Every package installed on compute nodes has an OL6 version of the package in the 2.3.3 iso.
However, if additional customer-installed packages are detected, the pre-upgrade script fails. You must uninstall the packages that cause the failure. Then restart this procedure.
The script also performs a health check of the RPM database.
[05/22/2017 14:39:36 814597] DEBUG (upgrade_utils:680) [RPMDB Check] PASSED: No rpmdb corruption found on compute nodes.
For the Oracle VM Server Release 3.4.x upgrade to succeed, a minimum amount of disk space must be available in
/and/boot. If the pre-upgrade script detects that there is insufficient disk space, it outputs a failure and indicates on which compute nodes it occurs.[Available Space Check] FAILED: The check failed on the following nodes:['ovcacn13r1', 'ovcacn28r1'].
Make sure that all compute nodes have sufficient disk space available. Remove or move old log files or other files that were copied to the compute nodes but are not required for the system to function. Then restart this procedure.
Make sure that the summary report of the pre-upgrade script shows that all checks have passed.
********************************************************************* Prerequisite checks for upgrading to PCA 2.3.3 PASSED! Check log file /tmp/2018-02-16_11.36.05/pre_upgrade_check.log for more details. Summary Report [Minimum Installed PCA Version Check] PASSED: PCA 2.3.3 only supports upgrading from PCA 2.1.1 or above. [Shared Storage Mounted Check] PASSED: Shared storage MGMT_ROOT, Yum, and Incoming are mounted on the management nodes. [NTP server Check] PASSED: NTP server has been verified on Compute Node ILOM. [NTP server Check] PASSED: NTP server has been verified on ZFSSA ILOM. [Ethernet Cards Check] PASSED: Ethernet cards have been verified on xms. [OVMM Model DB Check] PASSED: Default OVMM objects exist on both management nodes. [Mysql password Check] PASSED: Password has been verified on Mysql. [ONF Check] PASSED: No ObjectNotFoundException found on the management nodes. [MOTD Check] PASSED: Banner file /etc/motd is empty on the management nodes. [Password Check] PASSED: Password has been verified on OPUS. [Password Check] PASSED: Password has been verified on Xsigo. [Password Check] PASSED: Password has been verified on Oracle VM Manager. [Password Check] PASSED: Password has been verified on ZFSSA. [Password Check] PASSED: Password has been verified on Compute Node ILOM. [Password Check] PASSED: Password has been verified on Compute Node host. [Password Check] PASSED: Password has been verified on ZFSSA ILOM. [OS Check] PASSED: The compute nodes are running the correct Oracle Linux version. [OS Check] PASSED: The management nodes are running the correct Oracle Linux version. [Xsigo Config Check] PASSED: The configurations for each Xsigo are in sync. [NTP xsigo Check] PASSED: NTP server has been verified on Xsigo. [Oracle VM Minimum Version Check] PASSED: The compute nodes are running Oracle VM equal or above the upgrade minimum version 3.2.10. [Oracle VM Minimum Version Check] PASSED: The management nodes are running Oracle VM equal or above the upgrade minimum version 3.2.10. [Server Name Check] PASSED: Host names for the compute nodes have not been modified. [Server Name Check] PASSED: Host names for the management nodes have not been modified. [ZFS Label Check] PASSED: IPMP interface labels are correct on the ZFSSA. [Shared Storage Mounted Check] PASSED: Shared storage MGMT_ROOT, Yum, and Incoming are mounted on the management nodes. [Available Space Check] PASSED: Adequate space exists on /boot and / of compute nodes, for the upgrade to Oracle VM 3.4.4. [YUM Repo Check] PASSED: Every package installed on compute nodes has an OL6 version of the package in the 2.3.3 iso. [RPMDB Check] PASSED: No rpmdb corruption found on compute nodes. *********************************************************************
When all components in the environment pass these checks, please proceed with the required non-automated checks. The procedure for those additional manual verification tasks is documented in the support note with Doc ID 2242177.1.
When you have successfully completed the manual checks, the system is ready for the Oracle PCA Controller Software Release 2.3.1, 2.3.2 or 2.3.3 update process. Refer to Section 3.4, “Updating the Oracle PCA Controller Software Through the Oracle PCA CLI” for detailed instructions.