Preparing the Upgrade Environment
Software versions and upgrades for Oracle Private Cloud Appliance are made available for download through My Oracle Support. The ISO file contains all the files and packages required to upgrade the appliance hardware and software components to a given release. All the items within the ISO file have been tested to work with each other and qualified for installation on your rack system.
To be able to use an ISO file to upgrade your appliance, you need to download the file to a location from where a web server can make it available to the Private Cloud Appliance management nodes. If you have set up a bastion host connected to the internal administration network of the appliance, it is convenient to store the ISO file on that machine and run a web server to make the ISO file accessible over http.
Before performing any upgrade operations, you unpack the contents of the ISO file to populate the source directories in the shared storage that is mounted on all three management nodes. This ensures that the new version is installed when an upgrade command is executed.
- Prerequisite Version
-
The latest Upgrader code automatically enforces prerequisite software versions on your Private Cloud Appliance. In the early stages of upgrade or patch preparation, the Upgrader service validates the currently installed appliance software version against the new target version. The preparation process (
upgradePreConfig) documented in this section will only proceed if validation is successful.If the appliance is not running at least the minimum required version, the Upgrader exits the process and rolls back the environment to its previous state. View the details of the failed upgrade job:
PCA-ADMIN> getupgradejob upgradeJobId=1700153626051-prepare-40046 Data: Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_prepare_environment_2023_11_16-16.53.46.log Arguments = [...] Status = Failed [...] Tasks 23 - Name = Check Prerequisite Build Version Tasks 23 - Description = Check current build version not lower than prerequisite version Tasks 24 - Name = Check Prerequisite Build Version Tasks 24 - Message = (("Caught exception while checking prerequisite build number Exception: Command: ['/usr/bin/python3', '/var/lib/pca-upgrader/prerequisite_build_validator.py', 'rack=PCA', 'upgrade=ISO'] failed (1): stderr: b'' stdout: b'PCA version is lower than prerequisite build, must upgrade to prerequisite build 3.0.2-b799577 to proceed further upgrade\\n'",), {}) Tasks 24 - Status = FailedYou must first install the prerequisite version, which is indicated by the error message in the upgrade job output.
Using the Service Web UI
-
In the navigation menu, click Upgrade & Patching.
-
In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.
The Create Request window appears. Choose Upgrade as the Request Type.
-
Select the appropriate upgrade request type: Upgrade PreConfig.
-
Fill out the upgrade request parameters:
-
Option: Enter ISO.
-
Location: Enter the path to the location where the ISO image is stored.
-
ISO Checksum: Enter the checksum required to verify the integrity of the ISO image. The checksum is provided alongside the ISO image in a file named
<iso_image>.sha256sum. -
Log Level: Optionally, select a specific log level for the upgrade log file. The default log level is "Information". For maximum detail, select "Debug".
-
Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.
-
Alternative ULN Channel: This parameter applies to patching and can be ignored.
-
-
Click Create Request.
The new upgrade request appears in the Upgrade Jobs table.
The ISO image is unpacked on the shared storage of the management node cluster, from where its contents can be used to perform the required upgrade operations.
Using the Service CLI
-
Gather the information that you need to run the command:
-
the location of the ISO image to upgrade from
Enter the path to where the ISO image is stored. Its contents will be unpacked on the shared storage accessible from the management nodes.
-
the checksum used to verify the ISO image
The checksum is provided alongside the ISO image. Its file name is the ISO image name with
.sha256sumappended. The system uses the checksum to verify that the data in the ISO image is intact and valid for this upgrade.
-
-
Enter the upgrade pre-configuration command.
Syntax (entered on a single line):
upgradePreConfig option=ISO location=<path-to-iso> isoChecksum=<iso-file-checksum>
Example:
PCA-ADMIN> upgradePreConfig option=ISO \ location="http://host.example.com/pca-<version>-<build>.iso" \ isoChecksum=90e4505b098031afb02068080db2603dc6f580cd7cf52aa51ecd0c3b81668027 Command: upgradePreConfig option=ISO location="http://host.example.com/pca-<version>-<build>.iso" isoChecksum=90e4505b098031afb02068080db2603dc6f580cd7cf52aa51ecd0c3b81668027 Status: Success Time: 2022-11-06 06:35:38,884 UTC Data: Service request has been submitted. Upgrade Job Id = 1668417666968-prepare-28142 Upgrade Request Id = UWS-c94ba56a-1b91-49d8-8e51-afeae7f62186
-
Use the request ID and the job ID to check the status of the upgrade process.
PCA-ADMIN> getUpgradeJobs id upgradeRequestId commandName result -- ---------------- ----------- ------ 1630938939109-compute-7545 UWS-61736806-7e5a-4648-9259-07c54c39cacb compute Passed 1632849609034-kubernetes-35545 UWS-edfa3b32-c32a-4b67-8df5-2357096052bf kubernetes Passed 1668417666968-prepare-28142 UWS-c94ba56a-1b91-49d8-8e51-afeae7f62186 prepare Passed PCA-ADMIN> getUpgradeJob upgradeJobId=1668417666968-prepare-28142 Command: getUpgradeJob upgradeJobId=1668417666968-prepare-28142 Status: Success Time: 2022-11-06 07:24:00,793 UTC Data: Upgrade Request Id = UWS-c94ba56a-1b91-49d8-8e51-afeae7f62186 Name = prepare Start Time = 2022-06-14T06:35:56 End Time = 2022-11-06T06:35:58 Pid = 28142 Host = pcamn02 Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_prepare_environment_2022_11_16-06.35.56.log Arguments = {"component_names":null,"diagnostics":false,"display_task_plan":false,"dry_run_tasks":false,"expected_iso_checksum":null,"fail_halt":false,"fail_upgrade":null,"image_location":"http://host.example.com/pca-<version>-<build>.iso","online_upgrade":null,"precheck_status":false,"repo_config_override":null,"result_override":null,"task_time":0,"test_run":false,"upgrade":false,"upgrade_to":null,"user_uln_base_url":null,"verify_only":false,"host_ip":null,"log_level":null,"switch_type":null,"epld_image_location":null,"checksum":"90e4505b098031afb02068080db2603dc6f580cd7cf52aa51ecd0c3b81668027","composition_id":null,"request_id":"UWS-c94ba56a-1b91-49d8-8e51-afeae7f62186","uln":null,"patch":null} Status = Passed Execution Time(sec) = 616 Tasks 1 - Name = Validate Image Location Tasks 1 - Description = Verify that the image exists at the specified location and is correctly named [...] -
Proceed to the next upgrade preparation phase.
Backup Before Appliance Update
For system-critical components and services, Oracle Private Cloud Appliance runs a scheduled backup service that allows the appliance to be restored to its last known healthy state in case of a catastrophic failure. It is recommended that you implement a backup strategy for the users' cloud resources in the Compute Enclave as well.
Before upgrading or patching any component of the Private Cloud Appliance, you should create a backup of the latest state of the MySQL database, the ZFS Storage Appliance, and the Secret Service (Vault). The backup commands leverage the existing backup service but create an additional restore point that includes the most recent changes from right before you start an upgrade or patch workflow.
Using the Service CLI
-
Start the three required backup tasks.
PCA-ADMIN> backup target=vault Status: Success Data: Type = BackupJob Job Id = ocid1.brs-job.PCA3X62D9C1.mypca.joopwuv9403uzbfrh4x9mprmoduh3ljais6ex233v1b21ccqywu4a3vqykgm Display Name = brs-job-1668419778-backup Profile Id = ocid1.backup_profile.PCA3X62D9C1.mypca.wrxfwtxwxw6ydp2mwnypcaaxxzmwpuhsc33gcm3dyte7kgr4etuhb29qbs8q Time Created = 2022-11-06T09:56:18Z Lifecycle State = CREATING Retention = 14 PCA-ADMIN> backup target=zfs Status: Success Data: Type = BackupJob Job Id = ocid1.brs-job.PCA3X62D9C1.mypca.9oaeaa2kw5crqfcjkh8kyhbxcv8bwh0f4ud6n3lucf802oj15ss3k39874bc Display Name = brs-job-1668419842-backup Profile Id = ocid1.backup_profile.PCA3X62D9C1.mypca.p7w0tgbvhtjqsgc8rllca2cvotkpgrtf4huiph7466mjio0dgskij9f0bp06 Time Created = 2022-11-06T09:57:22Z Lifecycle State = CREATING Retention = 14 PCA-ADMIN> backup target=mysql Status: Success Data: Type = BackupJob Job Id = ocid1.brs-job.PCA3X62D9C1.mypca.iew5tphpgr3h6mhliw2fai2ywvv386a0xc7isfo8kisj0wrcx114irnit6ot Display Name = brs-job-1668419850-backup Profile Id = ocid1.backup_profile.PCA3X62D9C1.mypca.henfqzdbafs4z3mxeuslb1c6f4t049w0pxvwf1gi3eb8wm1y11v7m932tn4g Time Created = 2022-11-06T09:57:30Z Lifecycle State = CREATING Retention = 14
-
Use the backup job ID to check the status of the backups.
PCA-ADMIN> getBackupJobs Data: id displayName components -- ----------- ---------- ocid1.brs-job.PCA3X62D9C1.mypca.iew5tphpgr3h6mhliw2fai2ywvv386a0xc7isfo8kisj0wrcx114irnit6ot brs-job-1668419850-backup mysql ocid1.brs-job.PCA3X62D9C1.mypca.9oaeaa2kw5crqfcjkh8kyhbxcv8bwh0f4ud6n3lucf802oj15ss3k39874bc brs-job-1668419842-backup zfs ocid1.brs-job.PCA3X62D9C1.mypca.joopwuv9403uzbfrh4x9mprmoduh3ljais6ex233v1b21ccqywu4a3vqykgm brs-job-1668419778-backup vault PCA-ADMIN> getBackupJob backupJobId=ocid1.brs-job.PCA3X62D9C1.mypca.iew5tphpgr3h6mhliw2fai2ywvv386a0xc7isfo8kisj0wrcx114irnit6ot Status: Success Data: Type = BackupJob Job Id = ocid1.brs-job.PCA3X62D9C1.mypca.iew5tphpgr3h6mhliw2fai2ywvv386a0xc7isfo8kisj0wrcx114irnit6ot Display Name = brs-job-1668419850-backup Time Created = 2022-11-06T09:57:30Z Status = success Components = mysql
-
Confirm that all three backup operations have completed successfully. Then, proceed to the next upgrade preparation phase.
Upgrade the Upgrader
The code of the upgrader is changed regularly, as is the case with any other system service. Ensure that the upgrader is up-to-date first, so that all upgrade commands are run with the latest version.
Upgrade operations will not be allowed to run if the system detects that the Upgrader is not the latest available version.
In software version 3.0.2-b892153 and later, when the Upgrader is upgraded to the latest version, an upgrade plan is also generated. The upgrade plan is based on a comparison between the current installation and the packages downloaded during the preparation of the upgrade environment, and it determines which upgrades need to be performed in the next phase. For more information, see Checking Upgrade Plan Status and Progress.
Caution:
Ensure that the Upgrade Preconfig task has been completed first. See Preparing the Upgrade Environment.
To prevent inconsistencies while executing the upgrade plan later on, it is critical that both parts of the preparation process, Upgrade PreConfig and preUpgrade, are completed together in the specified order. If at any time you need to rerun the preUpgrade command, you must rerun the preceding command first.
Using the Service Web UI
-
In the navigation menu, click Upgrade & Patching.
-
In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.
The Create Request window appears. Choose Upgrade as the Request Type.
-
Select the appropriate upgrade request type: preUpgrade.
-
Fill out the upgrade request parameters:
-
Action: Enter start to retrieve the new version of the upgrader. (To check the status of the previous preUpgrade job, enter status instead.)
-
Type: Enter ISO. The upgrader packages are picked from the unpacked ISO image.
-
-
Click Create Request.
The new upgrade request appears in the Upgrade Jobs table. When the job has completed successfully, the upgrader is up-to-date and ready for use.
Using the Service CLI
-
Start the process to install the latest upgrader version on the management nodes.
PCA-ADMIN> preUpgrade action=start type=ISO Data: Successfully triggered the pre-upgrade task. Pre upgrade status = success
As part of the process, these operations are performed:
-
Save the existing yum configuration.
-
Configure the yum repository for the new upgrader files.
-
Install the new upgrader version on the management nodes, then restart the upgrader systemd service for the changes to take effect.
-
Restore the existing yum configuration that was saved in the first step.
-
-
Check the status of the upgrade process at any time using this command:
PCA-ADMIN> preUpgrade action=status Data: A pre-upgrade task is running! Pre upgrade status = IN-PROGRESS PCA-ADMIN> preUpgrade action=status Data: The previous pre-upgrade task succeeded! Pre upgrade status = SUCCESS
-
Confirm that the latest version of the upgrader has been installed successfully. Ensure that the system is in ready state. Then, proceed with the component upgrades.
-
Optionally, check which components need to be upgraded by displaying the upgrade plan.
Note:
This function is available when the appliance is running software version 3.0.2-b892153 or later.
PCA-ADMIN> getUpgradePlan Data: id componentType currentBuild targetBuild currentVersion targetVersion requireReboot timeEstimation requireUpgrade impactedInfra -- ------------- ------------ ----------- -------------- ------------- ------------- -------------- -------------- ------------- generic zfssa 3.0.2-b892153 3.0.2-b1052646 2013.06.05.8.57.1-2.57.5392.2 2013.06.05.8.57.1-2.57.5501.4 false 45 true host,compute 100.96.2.64 compute 3.0.2-b892153 3.0.2-b1052646 3.0.2-640 3.0.2-691 true 20 true compute 100.96.2.65 compute 3.0.2-b892153 3.0.2-b1052646 3.0.2-640 3.0.2-691 true 20 true compute 100.96.2.66 compute 3.0.2-b892153 3.0.2-b1052646 3.0.2-640 3.0.2-691 true 20 true compute 100.96.2.67 compute 3.0.2-b892153 3.0.2-b1052646 3.0.2-640 3.0.2-691 true 20 true compute 100.96.2.68 compute 3.0.2-b892153 3.0.2-b1052646 3.0.2-640 3.0.2-691 true 20 true compute 100.96.2.33 host 3.0.2-b892153 3.0.2-b1052646 oraclelinux-release-7.9-1.0.9 oraclelinux-release-7.9-1.0.9 true 35 true host 100.96.2.34 host 3.0.2-b892153 3.0.2-b1052646 oraclelinux-release-7.9-1.0.9 oraclelinux-release-7.9-1.0.9 true 35 true host 100.96.2.35 host 3.0.2-b892153 3.0.2-b1052646 oraclelinux-release-7.9-1.0.9 oraclelinux-release-7.9-1.0.9 true 35 true host generic mysql 3.0.2-b892153 3.0.2-b1052646 8.0.28-1.1 8.0.33-1.1 false 15 true host generic etcd 3.0.2-b892153 3.0.2-b1052646 3.3.10 3.3.10 false 5 false host generic vault 3.0.2-b892153 3.0.2-b1052646 v1.7.1-3 v1.7.1-3 false 5 false host generic kubernetes 3.0.2-b892153 3.0.2-b1052646 1.20.6-1 1.25.7-1 false 350 true host,compute generic platform 3.0.2-b892153 3.0.2-b1052646 None None false 50 true host,compute Oracle-Linux-7.9 ociImages 3.0.2-b892153 3.0.2-b1052646 2022.08.29_0 2023.09.26_0 false 5 true host Oracle-Linux-8 ociImages 3.0.2-b892153 3.0.2-b1052646 2022.08.29_0 2023.09.26_0 false 5 true host Oracle-Linux-9 ociImages 3.0.2-b892153 3.0.2-b1052646 None 2023.09.26_0 false 5 true host Oracle-Linux8-OKE-1.26.6 ociImages 3.0.2-b892153 3.0.2-b1052646 None 20240210 false 5 true host Oracle-Linux8-OKE-1.27.7 ociImages 3.0.2-b892153 3.0.2-b1052646 None 20240209 false 5 true host Oracle-Linux8-OKE-1.28.3 ociImages 3.0.2-b892153 3.0.2-b1052646 None 20240210 false 5 true host Oracle-Solaris-11 ociImages 3.0.2-b892153 3.0.2-b1052646 2023.04.18_0 2023.10.16_0 false 5 true host 100.96.0.33 ilom 3.0.2-b892153 3.0.2-b1052646 5.0.2.23 5.1.1.21 true 10 true host,compute 100.96.0.34 ilom 3.0.2-b892153 3.0.2-b1052646 5.0.2.23 5.1.1.21 true 10 true host,compute 100.96.0.35 ilom 3.0.2-b892153 3.0.2-b1052646 5.0.2.23 5.1.1.21 true 10 true host,compute 100.96.0.64 ilom 3.0.2-b892153 3.0.2-b1052646 5.0.2.23 5.1.2.20.a true 10 true host,compute 100.96.0.65 ilom 3.0.2-b892153 3.0.2-b1052646 5.0.2.23 5.1.1.21 true 10 true host,compute 100.96.0.66 ilom 3.0.2-b892153 3.0.2-b1052646 5.0.2.23 5.1.2.21 true 10 true host,compute 100.96.0.67 ilom 3.0.2-b892153 3.0.2-b1052646 5.0.2.23 5.1.2.20.a true 10 true host,compute 100.96.0.68 ilom 3.0.2-b892153 3.0.2-b1052646 5.0.2.23 5.1.2.21 true 10 true host,compute leaf switch 3.0.2-b892153 3.0.2-b1052646 10.2.5 10.3.4a false 60 true host,compute mgmt switch 3.0.2-b892153 3.0.2-b1052646 10.2.5 10.3.4a false 60 true host,compute spine switch 3.0.2-b892153 3.0.2-b1052646 10.2.5 10.3.4a false 60 true host,compute