Active Versions 3.0.2-b892153 to 3.0.2-b1185392
If your Private Cloud Appliance is running a software version with build number 3.0.2-b892153 to 3.0.2-b1185392 (both included), follow the instructions in this section to upgrade to appliance software version 3.0.2-b1261765.
Prepare for Appliance Upgrade
Caution:
When upgrading or patching Private Cloud Appliance, ensure that no other Service Enclave operations are ongoing or started. The upgrade or patching process might be adversely affected by appliance administration activities, including but not limited to: compute node operations (such as provisioning or reboot), operations on the management node cluster, configuration changes in any hardware or software component, rack network connectivity changes, password changes, and so on.
Appliance administrators must always use the supported interfaces: the Service CLI and Service Web UI. Performing operations not provided through the API, from the Linux command line and/or with elevated access privileges, is not supported, unless such explicit instructions are provided by Oracle.
Caution:
The granular appliance architecture with built-in redundancy allows administrators to upgrade or patch components without downtime. However, resource capacity and performance might be reduced while an upgrade or patch workflow is in progress.
We recommend that administrators responsible for upgrade or patching notify all Compute Enclave users in advance about such planned maintenance operations.
This is particularly important for users of Oracle Private Cloud Appliance Kubernetes Engine (OKE), because new cluster deployments are not allowed during the maintenance window, and some types of application clusters might experience service interruptions.
Software versions and upgrades for Oracle Private Cloud Appliance are made available for download through My Oracle Support. The ISO file contains all the files and packages required to upgrade the appliance hardware and software components to a given release. All the items within the ISO file have been tested to work with each other and qualified for installation on your rack system.
To be able to use an ISO file to upgrade your appliance, you need to download the file to a location from where a web server can make it available to the Private Cloud Appliance management nodes. If you have set up a bastion host connected to the internal administration network of the appliance, it is convenient to store the ISO file on that machine and run a web server to make the ISO file accessible over HTTP.
Before you start an upgrade procedure, ensure that you have performed these steps:
-
Download the new software ISO image to a suitable location.
-
Verify that you have the necessary permissions to perform an upgrade.
-
Back up the current system configuration.
-
Preconfigure the upgrade environment.
-
Set up the new appliance software sources to run component upgrade procedures.
Note:
In software version 3.0.2-b1185392 the preparation phase of the upgrade and patching workflows has been redesigned to bring the Upgrader functionality of the latest release into the appliance at the earliest time possible. As a result, some operations are different depending on the active software version of the appliance at the start of the upgrade or patch process. The instructions in this guide cover those differences.
Tools are provided to verify the status of the appliance before, during, and after an upgrade. The upgrade workflows follow the upgrade plan, which you can consult at any time to track progress.
Verify Permissions
To be able to run any upgrade or patch workflows, you must have an administrator account to log in to the Service Enclave. You must be a member of one of these authorization groups: SuperAdmin, Admin, or DR Admin. More information can be found in the chapter Administrator Account Management of the Oracle Private Cloud Appliance Administrator Guide.
When you log in to the Service CLI, you can verify
that the upgrade and patch commands are available to you by displaying all custom commands, or
custom commands in these object categories: UpgradeRequest,
PatchRequest, UpgradeJob, UpgradeJobList.
The list of commands is filtered based on your access profile. If the upgrade and patch
commands are listed, it means you have permission to run them.
PCA-ADMIN> showallcustomcmds
Operation Name: <Related Object(s)>
-----------------------------------
getUpgradeHistory: UpgradeRequest
getUpgradeHistoryDetails: UpgradeRequest
getUpgradeJob: UpgradeJob
getUpgradeJobs: UpgradeJobList
getUpgradePlan: UpgradeRequest
getUpgradeRequests: UpgradeRequest
getUpstreamUlnChannel: PatchRequest
getUpstreamUlnChannels: PatchRequest
[...]
PCA-ADMIN> showcustomcmds PatchRequest
patchCN
patchIlom
patchOCIImages
patchSwitch
[...]
setUpstreamUlnMirror
syncUpstreamUlnMirror
getUpstreamUlnChannels
getUpstreamUlnChannel
addUpstreamUlnChannel
removeUpstreamUlnChannel
PCA-ADMIN> showcustomcmds UpgradeJob
getUpgradeJob
killUpgradeJob
PCA-ADMIN> showcustomcmds UpgradeJobList
getUpgradeJobs
PCA-ADMIN> showcustomcmds UpgradeRequest
getUpgradeRequests
upgradeFullMN
upgradeCN
upgradeIlom
upgradeSwitch
[...]Backup Before Appliance Update
For system-critical components and services, Oracle Private Cloud Appliance runs a scheduled backup service that allows the appliance to be restored to its last known healthy state in case of a catastrophic failure. It is recommended that you implement a backup strategy for the users' cloud resources in the Compute Enclave as well.
Before upgrading or patching any component of the Private Cloud Appliance, you should create a backup of the latest state of the MySQL database, the ZFS Storage Appliance, and the Secret Service (Vault). The backup commands leverage the existing backup service but create an additional restore point that includes the most recent changes from right before you start an upgrade or patch workflow.
Caution:
In appliance software version 3.0.2-b1081557 and newer, the monitoring data from Prometheus is not included in automated backups. To preserve your Prometheus data, create a backup and restore it manually. For more information, refer to the note with Doc ID 3021643.1.
Using the Service CLI
-
Start the three required backup tasks.
PCA-ADMIN> backup target=vault Status: Success Data: Type = BackupJob Job Id = ocid1.brs-job.PCA3X62D9C1.mypca.joopwuv9403uzbfrh4x9mprmoduh3ljais6ex233v1b21ccqywu4a3vqykgm Display Name = brs-job-1668419778-backup Profile Id = ocid1.backup_profile.PCA3X62D9C1.mypca.wrxfwtxwxw6ydp2mwnypcaaxxzmwpuhsc33gcm3dyte7kgr4etuhb29qbs8q Time Created = 2022-11-06T09:56:18Z Lifecycle State = CREATING Retention = 14 PCA-ADMIN> backup target=zfs Status: Success Data: Type = BackupJob Job Id = ocid1.brs-job.PCA3X62D9C1.mypca.9oaeaa2kw5crqfcjkh8kyhbxcv8bwh0f4ud6n3lucf802oj15ss3k39874bc Display Name = brs-job-1668419842-backup Profile Id = ocid1.backup_profile.PCA3X62D9C1.mypca.p7w0tgbvhtjqsgc8rllca2cvotkpgrtf4huiph7466mjio0dgskij9f0bp06 Time Created = 2022-11-06T09:57:22Z Lifecycle State = CREATING Retention = 14 PCA-ADMIN> backup target=mysql Status: Success Data: Type = BackupJob Job Id = ocid1.brs-job.PCA3X62D9C1.mypca.iew5tphpgr3h6mhliw2fai2ywvv386a0xc7isfo8kisj0wrcx114irnit6ot Display Name = brs-job-1668419850-backup Profile Id = ocid1.backup_profile.PCA3X62D9C1.mypca.henfqzdbafs4z3mxeuslb1c6f4t049w0pxvwf1gi3eb8wm1y11v7m932tn4g Time Created = 2022-11-06T09:57:30Z Lifecycle State = CREATING Retention = 14
-
Use the backup job ID to check the status of the backups.
PCA-ADMIN> getBackupJobs Data: id displayName components -- ----------- ---------- ocid1.brs-job.PCA3X62D9C1.mypca.iew5tphpgr3h6mhliw2fai2ywvv386a0xc7isfo8kisj0wrcx114irnit6ot brs-job-1668419850-backup mysql ocid1.brs-job.PCA3X62D9C1.mypca.9oaeaa2kw5crqfcjkh8kyhbxcv8bwh0f4ud6n3lucf802oj15ss3k39874bc brs-job-1668419842-backup zfs ocid1.brs-job.PCA3X62D9C1.mypca.joopwuv9403uzbfrh4x9mprmoduh3ljais6ex233v1b21ccqywu4a3vqykgm brs-job-1668419778-backup vault PCA-ADMIN> getBackupJob backupJobId=ocid1.brs-job.PCA3X62D9C1.mypca.iew5tphpgr3h6mhliw2fai2ywvv386a0xc7isfo8kisj0wrcx114irnit6ot Status: Success Data: Type = BackupJob Job Id = ocid1.brs-job.PCA3X62D9C1.mypca.iew5tphpgr3h6mhliw2fai2ywvv386a0xc7isfo8kisj0wrcx114irnit6ot Display Name = brs-job-1668419850-backup Time Created = 2022-11-06T09:57:30Z Status = success Components = mysql
-
Confirm that all three backup operations have completed successfully. Then, proceed to the next upgrade preparation phase.
Preconfigure the Upgrade Environment
When the ISO file has been downloaded from My Oracle Support, and made available to the appliance over HTTP, a series of preconfiguration tasks must be performed. Although these operations make no changes to the system, they ensure that the environment meets all requirements for a correct upgrade workflow setup in the next phase of upgrade preparation.
The preconfiguration workflow differs depending on the active appliance software version before upgrade.
-
On systems running a version older than 3.0.2-b1185392, the
upgradePreConfigandpreUpgradecommands ensure that new software packages, latest Upgrader code, and upgrade plan are in place to perform system and component upgrades. -
If the system runs version 3.0.2-b1185392, a new script is added to the workflow. After download to internal storage, the new appliance software ISO is mounted, so the
pca-prerequisitescript can be installed and run. When all script operations are done, the ISO is unmounted again.Operations performed by the
pca-prerequisitescript include:-
validating current and new software versions
-
installing the latest
preUpgradefunctionality -
copying the latest Upgrader code to the appliance
-
writing environment parameters to a yaml file for use in the next preparation phase
-
generating a new upgrade history file
-
This entire preconfiguration process is launched by an appliance administrator with a single command.
Caution:
The Upgrader allows up to 10 hours to download the ISO image before it times out. This corresponds with a 50GB image file downloaded over a connection with at least 20Mbps of bandwidth. Ensure that the download connection meets this minimum requirement. For optimal performance, a higher bandwidth is recommended.
- Unprovisioned Compute Nodes
-
If the Private Cloud Appliance contains unprovisioned compute nodes, they can cause errors during upgrade or patching, and might experience provisioning issues later. Therefore, when a record exists in the node database, the compute node must be provisioned and added to a fault domain, otherwise the upgrade or patching process cannot proceed.
If you need to upgrade or patch the appliance software on a system with compute nodes that cannot be provisioned, these database records must be removed. Contact Oracle for assistance.
For more information about provisioning, refer to "Performing Compute Node Operations" in the chapter Hardware Administration of the Oracle Private Cloud Appliance Administrator Guide.
- Prerequisite Version
-
The latest Upgrader code automatically enforces prerequisite software versions on your Private Cloud Appliance. In the early stages of upgrade or patch preparation, the Upgrader service validates the currently installed appliance software version against the new target version. The preparation process (
upgradePreConfig) documented in this section will only proceed if validation is successful.If the appliance is not running at least the minimum required version, the Upgrader exits the process and rolls back the environment to its previous state. View the details of the failed upgrade job:
PCA-ADMIN> getupgradejob upgradeJobId=1700153626051-prepare-40046 Data: Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_prepare_environment_2023_11_16-16.53.46.log Arguments = [...] Status = Failed [...] Tasks 23 - Name = Check Prerequisite Build Version Tasks 23 - Description = Check current build version not lower than prerequisite version Tasks 24 - Name = Check Prerequisite Build Version Tasks 24 - Message = (("Caught exception while checking prerequisite build number Exception: Command: ['/usr/bin/python3', '/var/lib/pca-upgrader/prerequisite_build_validator.py', 'rack=PCA', 'upgrade=ISO'] failed (1): stderr: b'' stdout: b'PCA version is lower than prerequisite build, must upgrade to prerequisite build 3.0.2-b892153 to proceed further upgrade\\n'",), {}) Tasks 24 - Status = FailedYou must first install the prerequisite version, which is indicated by the error message in the upgrade job output.
Using the Service CLI
-
Gather the information that you need to run the command:
-
the location of the ISO image to upgrade from
Enter the path to where the ISO image is stored. Its contents will be unpacked on the shared storage accessible from the management nodes.
-
the checksum used to verify the ISO image
The checksum is provided alongside the ISO image. Its file name is the ISO image name with
.sha256sumappended. The system uses the checksum to verify that the data in the ISO image is intact and valid for this upgrade.
-
-
Enter the upgrade preconfiguration command.
Syntax (entered on a single line):
upgradePreConfig option=ISO location=<path-to-iso> isoChecksum=<iso-file-checksum>
Example:
PCA-ADMIN> upgradePreConfig option=ISO \ location=http://host.example.com/pca-<version>-<build>.iso \ isoChecksum=90e4505b098031afb02068080db2603dc6f580cd7cf52aa51ecd0c3b81668027 Status: Success Data: Service request has been submitted. Upgrade Job Id = 1668417666968-prepare-28142 Upgrade Request Id = UWS-c94ba56a-1b91-49d8-8e51-afeae7f62186
-
Use the request ID and the job ID to check the status of the upgrade process.
PCA-ADMIN> getUpgradeJobs id upgradeRequestId commandName result -- ---------------- ----------- ------ 1668417666968-prepare-28142 UWS-c94ba56a-1b91-49d8-8e51-afeae7f62186 prepare Passed PCA-ADMIN> getUpgradeJob upgradeJobId=1668417666968-prepare-28142
-
Proceed to the next upgrade preparation phase.
Using the Service Web UI
-
In the navigation menu, go to the Maintenance section and click Upgrade Plan. This provides an overview of current and target component versions.
-
Click Upgrade & Patching to display the Upgrade Jobs page.
-
In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.
The Create Request window appears. Choose Upgrade as the Request Type.
-
Select the appropriate upgrade request type: Upgrade PreConfig.
-
Fill out the upgrade request parameters:
-
Option: Enter ISO.
-
Location: Enter the path to the location where the ISO image is stored.
-
ISO Checksum: Enter the checksum required to verify the integrity of the ISO image. The checksum is provided alongside the ISO image in a file named
<iso_image>.sha256sum. -
Log Level: Optionally, select a specific log level for the upgrade log file. The default log level is "Information". For maximum detail, select "Debug".
-
Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.
-
Alternative ULN Channel: This parameter applies to patching and can be ignored.
-
-
Click Create Request.
The new upgrade request appears in the Upgrade Jobs table. The preconfiguration steps are performed as described at the start of this section.
Set Up New Software Sources for Upgrade
The setup phase of the upgrade preparation ensures that the Upgrader is up-to-date first, so that all upgrade commands are run with the latest version. Upgrade operations will not be allowed to run if the system detects that the Upgrader is not the latest available version.
Software sources are handled differently depending on the active appliance software version before upgrade.
-
On systems running version 3.0.2-b1185392, the latest version of the Upgrader is installed first. Next, the sources of the new appliance software are set up for use in the system upgrade procedures. The current software image is backed up, and the new software ISO is unpacked on the management cluster shared storage in the appropriate directory structure: appliance software RPM packages, new images, new firmware, and so on.
-
On systems running a version older than 3.0.2-b1185392, a different package setup was already completed during preconfiguration. This phase ensures that the latest Upgrader code is in place.
At the end of the setup phase, an upgrade plan is generated. The upgrade plan is based on a comparison between the current installation and the new version that was unpacked. It determines which upgrade procedures need to be performed in the next phase. For more information, see Check Upgrade Plan Status and Progress.
This entire setup process is launched by an appliance administrator with a single command.
Caution:
Ensure that the preconfiguration process has been completed first.
To prevent inconsistencies while executing the upgrade plan later on, it is critical that
both commands in the preparation process, upgradePreConfig
(preconfiguration) and preUpgrade (setup), are completed together in the
specified order. If at any time you need to rerun the preUpgrade command,
you must rerun the preceding command first.
Using the Service CLI
-
Start the installation of the latest Upgrader version, and setup of the new appliance software sources.
-
Active version older than 3.0.2-b1185392:
PCA-ADMIN> preUpgrade action=start type=ISO Data: Successfully triggered the pre-upgrade task.
-
Active version 3.0.2-b1185392 and newer:
PCA-ADMIN> preUpgrade type=ISO Status: Running JobId: 1714591883904-setup-25325 Data: in progress
Upgrader operations performed with this command:
-
Save the existing yum configuration.
-
Configure the yum repository for the new Upgrader files.
-
Install the new Upgrader version on the management nodes, then restart the Upgrader systemd service for the changes to take effect.
-
Restore the existing yum configuration that was saved in the first step.
Setup operations performed with this command:
-
Back up the current software image.
-
Unpack the ISO, store new software RPMs and installation files (images, firmware) in correct directories.
-
Remove obsolete backups and images from the appliance internal shared storage.
-
Generate the upgrade plan.
Note:
If the active software version is older than 3.0.2-b1185392, a different package setup was already completed during preconfiguration. The upgrade plan is always generated with version 3.0.2-b892153 and newer.
-
-
Check the status of the preUpgrade process at any time using this command:
-
Active version older than 3.0.2-b1185392:
PCA-ADMIN> preUpgrade action=status Data: The previous pre-upgrade task succeeded! Pre upgrade status = SUCCESS
-
Active version 3.0.2-b1185392 and newer:
PCA-ADMIN> getPreUpgradeStatus Data: status = IN-PROGRESS message = A pre-upgrade task is running! PCA-ADMIN> getPreUpgradeStatus Data: status = SUCCESS message = The previous pre-upgrade task succeeded!
-
-
Confirm that the latest version of the Upgrader has been installed successfully.
Active version 3.0.2-b1185392 and newer: in case the preUpgrade process fails, check the upgrade job for troubleshooting information.
PCA-ADMIN> getUpgradeJob upgradeJobId=1714591883904-setup-25325 Data: Upgrade Request Id = PREUPGRADE-7b5d6235-fdfa-4830-8824-349a5e1b9d47 Name = setup Pid = 25325 Host = pcamn02 Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_setup_upgrade_environment_2024_05_01-19.31.23.log Arguments = {"component_names":null,"diagnostics":false,"display_task_plan":false,"dry_run_tasks":false,"expected_iso_checksum":null,"fail_halt":false,"fail_upgrade":null,"image_location":null,"online_upgrade":null,"precheck_status":false,"repo_config_override":null,"result_override":null,"task_time":0,"test_run":false,"upgrade":false,"upgrade_to":null,"user_uln_base_url":null,"verify_only":false,"host_ip":null,"log_level":null,"switch_type":null,"epld_image_location":null,"checksum":null,"composition_id":null,"request_id":"PREUPGRADE-7b5d6235-fdfa-4830-8824-349a5e1b9d47","uln":null,"patch":null} Status = Failed Execution Time(sec) = 1069 Tasks 1 - Name = Configure MN Yum Repo Tasks 2 - Name = Validate Image Location Tasks 3 - Name = Validate Upgrader Version Tasks 4 - Name = Validate Upgrade Order [...] Tasks 22 - Name = Delete Old backups Tasks 22 - Message = [Errno 2] No such file or directory: '/nfs/shared_storage/yum/pca_upgrader' Tasks 22 - Status = Failed Tasks 27 - Name = UpdateHistoryTask Tasks 27 - Message = Failed to update the history of upgrade Tasks 27 - Status = Failed -
Ensure that the system is in ready state. Then, proceed with the component upgrades.
-
Optionally, check which components need to be upgraded by displaying the upgrade plan.
PCA-ADMIN> getUpgradePlan Data: id componentType currentBuild targetBuild currentVersion targetVersion requireReboot timeEstimation requireUpgrade impactedInfra -- ------------- ------------ ----------- -------------- ------------- ------------- -------------- -------------- ------------- generic zfssa 3.0.2-b892153 3.0.2-b1261765 2013.06.05.8.57.1-2.57.5392.2 2013.06.05.8.73.1-2.73.5701.1 false 45 true host,compute 100.96.2.64 compute 3.0.2-b892153 3.0.2-b1261765 3.0.2-640 3.0.2-859 true 20 true compute 100.96.2.65 compute 3.0.2-b892153 3.0.2-b1261765 3.0.2-640 3.0.2-859 true 20 true compute 100.96.2.66 compute 3.0.2-b892153 3.0.2-b1261765 3.0.2-640 3.0.2-859 true 20 true compute 100.96.2.67 compute 3.0.2-b892153 3.0.2-b1261765 3.0.2-640 3.0.2-859 true 20 true compute 100.96.2.68 compute 3.0.2-b892153 3.0.2-b1261765 3.0.2-640 3.0.2-859 true 20 true compute 100.96.2.69 compute 3.0.2-b892153 3.0.2-b1261765 3.0.2-640 3.0.2-859 true 20 true compute 100.96.2.33 host 3.0.2-b892153 3.0.2-b1261765 oraclelinux-release-7.9-1.0.9 oraclelinux-release-7.9-1.0.13 true 35 true host 100.96.2.34 host 3.0.2-b892153 3.0.2-b1261765 oraclelinux-release-7.9-1.0.9 oraclelinux-release-7.9-1.0.13 true 35 true host 100.96.2.35 host 3.0.2-b892153 3.0.2-b1261765 oraclelinux-release-7.9-1.0.9 oraclelinux-release-7.9-1.0.13 true 35 true host generic mysql 3.0.2-b892153 3.0.2-b1261765 8.0.28-1.1 8.0.36-1.1 false 15 true host generic etcd 3.0.2-b892153 3.0.2-b1261765 3.3.10 3.5.6 false 5 true host generic vault 3.0.2-b892153 3.0.2-b1261765 v1.7.1-3 v1.7.1-3 false 5 true host generic kubernetes 3.0.2-b892153 3.0.2-b1261765 1.20.6-1 1.25.16-2 false 350 true host,compute generic platform 3.0.2-b892153 3.0.2-b1261765 None None false 50 true host,compute Oracle-Linux-7.9 ociImages 3.0.2-b892153 3.0.2-b1261765 2022.08.29_0 2024.07.31_0 false 5 true host Oracle-Linux-8 ociImages 3.0.2-b892153 3.0.2-b1261765 2022.08.29_0 2024.07.31_0 false 5 true host Oracle-Linux-9 ociImages 3.0.2-b892153 3.0.2-b1261765 None 2024.07.31_0 false 5 true host Oracle-Linux8-OKE-1.26.15 ociImages 3.0.2-b892153 3.0.2-b1261765 None 20240908 false 5 true host Oracle-Linux8-OKE-1.27.12 ociImages 3.0.2-b892153 3.0.2-b1261765 None 20240908 false 5 true host Oracle-Linux8-OKE-1.28.8 ociImages 3.0.2-b892153 3.0.2-b1261765 None 20240908 false 5 true host Oracle-Solaris-11 ociImages 3.0.2-b892153 3.0.2-b1261765 2023.04.18_0 2024.08.26_0 false 5 true host 100.96.0.33 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute 100.96.0.34 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute 100.96.0.35 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute 100.96.0.64 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute 100.96.0.65 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute 100.96.0.66 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute 100.96.0.67 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute 100.96.0.68 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute 100.96.0.68 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute leaf switch 3.0.2-b892153 3.0.2-b1261765 10.2.5 10.3.4a false 60 true host,compute mgmt switch 3.0.2-b892153 3.0.2-b1261765 10.2.5 10.3.4a false 60 true host,compute spine switch 3.0.2-b892153 3.0.2-b1261765 10.2.5 10.3.4a false 60 true host,compute
Using the Service Web UI
-
In the navigation menu, go to the Maintenance section and click Upgrade Plan. This provides an overview of current and target component versions.
-
Click Upgrade & Patching to display the Upgrade Jobs page.
-
In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.
The Create Request window appears. Choose Upgrade as the Request Type.
-
Select the appropriate upgrade request type: preUpgrade.
-
Fill out the upgrade request parameters:
-
Action: This parameter is no longer available in version 3.0.2-b1185392 and newer. Enter start to retrieve the new version of the upgrader. (To check the status of the previous preUpgrade job, enter status instead.)
-
Type: Enter ISO. The upgrader packages are picked from the unpacked ISO image.
-
-
Click Create Request.
The new upgrade request appears in the Upgrade Jobs table. When the job has completed successfully, the Upgrader is up-to-date and the new software version sources are ready for use.
Check Upgrade Readiness and Status
It is generally recommended to run the most recent version of the Private Cloud Appliance operating software on your system, in order to take advantage of the latest functionality as well as security and stability features. The process to bring the appliance software to the latest version, whether through upgrading or patching, is designed for minimal disruption: the system might experience a temporary reduction in performance or availability, but workloads continue to run without downtime.
To help the appliance administrators prepare for upgrade or patching, scope the work required to take the appliance from its current software version to the latest, and monitor progress of an ongoing upgrade or patch workflow, the system provides a set of commands and tools that are described in this section. To make the process as fluent as possible, with minimal risk of delays or interruptions, we recommend running commands in verification-only mode first. The administrator's most important tool is the upgrade plan, which lays out all the required steps in the correct order and provides progress information from start to completion.
Check Current Version of Components
To evaluate upgrade requirements and the impact of an upgrade you can view the current state of the top-level rack components. The system lists the most important build and version numbers of all the components for which an upgrade procedure is documented in this guide.
The component version list can be viewed in two ways:
-
In the Service Web UI, go to Maintenance in the navigation menu and select Component Version.
-
In the Service CLI, enter the following command:
PCA-ADMIN> getComponentVersions Data: id component iso version -- --------- --- ------- 100.96.2.64 compute 3.0.2-b1046481 3.0.2-687 100.96.2.65 compute 3.0.2-b1046481 3.0.2-687 100.96.2.66 compute 3.0.2-b1046481 3.0.2-687 100.96.2.67 compute 3.0.2-b1046481 3.0.2-687 100.96.2.68 compute 3.0.2-b1046481 3.0.2-687 100.96.2.69 compute 3.0.2-b1046481 3.0.2-687 generic etcd 3.0.2-b1049367 3.3.10 100.96.2.33 host 3.0.2-b1049367 oraclelinux-release-7.9-1.0.9 100.96.2.34 host 3.0.2-b1049367 oraclelinux-release-7.9-1.0.9 100.96.2.35 host 3.0.2-b1049367 oraclelinux-release-7.9-1.0.9 100.96.0.33 ilom 3.0.2-b1049367 5.1.1.21 100.96.0.34 ilom 3.0.2-b1049367 5.1.1.21 100.96.0.35 ilom 3.0.2-b1049367 5.1.1.21 100.96.0.64 ilom 3.0.2-b1049367 5.1.2.20.a 100.96.0.65 ilom 3.0.2-b1049367 5.1.1.21 100.96.0.66 ilom 3.0.2-b1049367 5.1.1.21 100.96.0.67 ilom 3.0.2-b1049367 5.1.2.20.a 100.96.0.68 ilom 3.0.2-b1049367 5.1.1.21 100.96.0.69 ilom 3.0.2-b1049367 5.1.1.21 generic kubernetes 3.0.2-b1049367 1.25.15-1 generic mysql 3.0.2-b1049367 8.0.33-1.1 Oracle-Linux-7.9 ociImages 3.0.2-b1049367 2023.09.26_0 Oracle-Linux-8 ociImages 3.0.2-b1049367 2023.09.26_0 Oracle-Linux-9 ociImages 3.0.2-b1049367 2023.09.26_0 Oracle-Linux8-OKE-1.26.6 ociImages 3.0.2-b1049367 20240210 Oracle-Linux8-OKE-1.27.7 ociImages 3.0.2-b1049367 20240209 Oracle-Linux8-OKE-1.28.3 ociImages 3.0.2-b1049367 20240210 Oracle-Solaris-11 ociImages 3.0.2-b1049367 2023.10.16_0 generic platform 3.0.2-b1046481 None leaf switch 3.0.2-b1049367 10.3.4a mgmt switch 3.0.2-b1049367 10.3.4a spine switch 3.0.2-b1049367 10.3.4a generic vault 3.0.2-b1049367 v1.7.1-3 generic zfssa 3.0.2-b1046481 2013.06.05.8.57.1-2.57.5501.1
Check System Upgrade History
Note:
This function is available when the appliance is running software version 3.0.2-b1081557 or later.
Information about all component upgrades and patches is stored in upgrade jobs, which can be consulted in the Service Web UI and Service CLI. Over time, as the system goes through multiple upgrades, the large number of entries might make the list difficult to interpret. The upgrade history provides a clear way to drill down into the details of the upgrade and patching activity on your appliance.
The upgrade history presents the information from all upgrade and patch jobs in a categorized way so you can see which version upgrades have been performed, which jobs have been run for each of those upgrades, and from which source (ISO upgrade or ULN patch). Details include build versions, component versions before and after, job completion, success or failure, time stamps, and duration.
Appliance software builds are installed onto the appliance either through upgrade from an ISO image or patching from ULN. Display the build history by running the following command.
PCA-ADMIN> GetUpgradeHistory Data: id From Build To Build Type Status Start Time End Time Actual Upgrade Time(min) Total Upgrade Time(min) -- ----- ------- ----- -------- ---------- -------- -------------- ------------ pca-upgrade-history-3.0.2-b951413 3.0.2-b868711 3.0.2-b951413 ISO Incomplete 2023-06-14T10:32:14 2023-06-14T15:32:14 300 600 pca-upgrade-history-3.0.2-b868711 3.0.2-b854356 3.0.2-b868711 ULN Completed 2023-01-01T15:10:07 2023-01-01T19:10:07 250 500 pca-upgrade-history-3.0.2-b854356 3.0.2-b790137 3.0.2-b854356 ISO Completed 2022-12-14T06:01:00 2022-12-14T12:01:00 350 700
If necessary, you can filter results by type.
PCA-ADMIN> GetUpgradeHistory type=uln Data: id From Build To Build Type Status Start Time End Time Actual Upgrade Time(min) Total Upgrade Time(min) -- ----- ------- ----- -------- ---------- -------- -------------- ------------ pca-upgrade-history-3.0.2-b868711 3.0.2-b854356 3.0.2-b868711 ULN Completed 2023-01-01T15:10:07 2023-01-01T19:10:07 250 500
To display the upgrade job list related to a particular build, copy the build ID and run this command:
PCA-ADMIN> GetUpgradeHistoryDetails id=pca-upgrade-history-3.0.2-b951413 Data: id component Timestamp From Version To Version Status Job ID Time Taken(min) -- --------- --------- ------------ ---------- ------- ------- ---------- spine cisco 2023-06-14T10:32:14 9.3.2 10.2.3 Failed 1686766376945-cisco-1509 50 mgmt cisco 2023-06-14T10:22:14 9.3.2 10.2.3 Passed 1686762287214-cisco-31252 60 leaf cisco 2023-06-14T10:12:14 9.3.2 10.2.3 Passed 1686758831077-cisco-35671 45 generic zfssa 2023-06-14T09:52:14 2013.06.05.8.40.1-2.40.4958.31 2013.06.05.8.48.1-2.48.5222.1 Passed 1686768921264-zfssa-19292 30 100.96.0.66 ilom 2023-06-14T09:22:14 5.0.2.23 5.1.1.21 Passed 1686768921264-ilom-19293 55 100.96.0.65 ilom 2023-06-14T09:22:14 5.0.2.23 5.1.1.21 Passed 1686768921264-ilom-19292 50 100.96.0.64 ilom 2023-06-14T09:22:14 5.0.2.23 5.1.1.21 Passed 1686768921264-ilom-19294 60 100.96.0.35 ilom 2023-06-14T09:22:14 5.0.2.23 5.1.1.21 Passed 1686768921264-ilom-19295 35 100.96.0.34 ilom 2023-06-14T09:22:14 5.0.2.23 5.1.1.21 Passed 1686768921264-ilom-19296 30 100.96.0.33 ilom 2023-06-14T09:22:14 5.0.2.23 5.1.1.21 Passed 1686768921264-ilom-19297 65 generic platform 2023-06-14T09:12:14 None None Passed 1686552138506-platform-48794 100 generic kubernetes 2023-06-14T09:12:14 1.20.6-1 1.25.7-1 Passed 1685379294807-kubernetes-72808 40 generic vault 2023-06-14T09:12:14 v1.7.1-3 v1.7.1-3 Passed 1685614045193-vault-47652 63 generic etcd 2023-06-14T09:02:14 3.3.10 3.3.10 Passed 1685613743232-etcd-83924 50 generic mysql 2023-06-14T08:52:14 8.0.28-1.1 8.0.30-1.1 Passed 1685378389035-mysql-90009 45 100.96.2.35 host 2023-06-14T08:42:14 oraclelinux-release-7.9-1.0.9 oraclelinux-release-7.9-1.0.9 Passed 1686726229799-host-68919 20 100.96.2.34 host 2023-06-14T08:32:14 oraclelinux-release-7.9-1.0.9 oraclelinux-release-7.9-1.0.9 Passed 1686726229799-host-68919 30 100.96.2.33 host 2023-06-14T08:22:14 oraclelinux-release-7.9-1.0.9 oraclelinux-release-7.9-1.0.9 Passed 1686726229799-host-68919 25 100.96.2.66 compute 2023-06-14T07:52:14 3.0.2-502 3.0.2-630 Passed 1685607717395-compute-53331 50 100.96.2.65 compute 2023-06-14T07:42:14 3.0.2-502 3.0.2-630 Passed 1685606912300-compute-91673 66 100.96.2.64 compute 2023-06-14T07:32:14 3.0.2-502 3.0.2-630 Passed 1685372050358-compute-50568 30 100.96.2.64 compute 2023-06-14T07:22:14 3.0.2-502 3.0.2-630 Failed 1685372050358-compute-50560 45 generic preupgrade 2023-06-14T06:32:14 3.0.2-b868711 3.0.2-b951413 N/A N/A N/A
Note:
Alternatively, you can view this information in the Service Web UI. In the navigation menu, go to the Maintenance section, and click Upgrade History.
Ensure the System Is In Ready State
Upgrades can be performed with limited impact on the system. No downtime is required, and user workloads continue to run while the underlying infrastructure is being upgraded in stages. However, it is considered good practice to ensure that backups are created of the system and the resources in your environment.
Fault Log
Note:
The upgrade fault category is available when the appliance is running software version 3.0.2-b1185392 or later.
When preparing for an upgrade, check the system fault log. Issues with potential impact on upgrade and patching are flagged as upgrade faults, and will prevent any upgrade or patch command from running. The Service CLI provides filtering options for these faults, as shown in these examples:
PCA-ADMIN> list fault fields upgradeFault,Status,Severity Data: id Upgrade Fault Status Severity -- ------------- ------ -------- 37f6cefe-f7d5-49a8-adff-76c1a020bcc8 False Cleared Critical 2be57600-4dbd-40f0-a0f8-e4ffbb2c8468 True Active Critical 297c770e-16e1-11ef-9e78-a8698c107234 True Cleared Major d27e0895-eb87-4e66-bd4b-f6500153cf64 True Cleared Critical 77752a35-4cf7-49ed-88df-06158d846358 True Active Critical 9ff4fe22-8ed6-463f-95b5-458d6a76d185 False Active Critical 0a09204c-953e-4df3-912b-a2a175ce8c1a True Cleared Critical [...] PCA-ADMIN> list fault where upgradeFault EQ True Data: id Name Status Severity -- ---- ------ -------- 2be57600-4dbd-40f0-a0f8-e4ffbb2c8468 pcamn01--PCA-8000-44--asrclient Active Critical 297c770e-16e1-11ef-9e78-a8698c107234 ilom-pcacn001--PCA-8000-EA--ilom-pcacn001 Cleared Major 0a09204c-953e-4df3-912b-a2a175ce8c1a pcamn01--PCA-8000-AH--mysql_cluster Cleared Critical f7da7c82-03bc-4383-95cf-542cefbb5d39 pcamn01--PCA-8000-CD--mysql_cluster Cleared Critical 74fd23ac-3006-4ab6-9c50-544869ba78f9 pcamn02--PCA-8000-6C--registry Cleared Critical dc77b088-939d-4725-b45f-1072024b02ba pcamn02--PCA-8000-0E--etcd Cleared Critical d27e0895-eb87-4e66-bd4b-f6500153cf64 pcamn01--PCA-8000-22--vault Cleared Critical 77752a35-4cf7-49ed-88df-06158d846358 pcamn01--PCA-8000-93--mysql_cluster Active Critical
Pre-Checks
Every upgrade operation is preceded by a set of pre-checks. These are built into the upgrade code and will report an error if the system is not in the required state for the upgrade. The upgrade will only begin if all pre-checks are passed.
You can use the pre-checks to test in advance for any system health issues that would prevent a successful upgrade. After preparing the upgrade environment, run any or all of the upgrade commands with the "verify only" option.
Caution:
Oracle strongly recommends testing that the Private Cloud Appliance is ready for upgrading, by executing the full management node upgrade command in verify-only mode. The output provides a readiness report you can use to plan any corrective actions as well as the upgrade.
This verification can take a long time to complete. Run it long enough in advance so the actual upgrade can be completed within the scheduled window.
In the Service Web UI the verify-only option is activated with a check box when you create the upgrade request; in the Service CLI you use the optional upgrade command parameter shown in this example:
PCA-ADMIN> upgradeKubernetes verifyOnly=True [...] PCA-ADMIN> getUpgradeJobs id upgradeRequestId commandName result -- ---------------- ----------- ------ 1632849609034-kubernetes_verify-35545 UWS-8995e5b7-a237-4717-bb5c-01f1cf85daf0 kubernetes_verify Passed
If issues are detected, either from the fault log or the pre-checks, you can resolve them before the planned upgrade window, and keep the actual system upgrade as fluent and short as possible.
It is important to note that concurrent upgrade operations are not supported. An upgrade job must be completed before a new one can be started.
Check Upgrade Plan Status and Progress
In software versions 3.0.2-b892153 and later, the Upgrader uses an upgrade plan as a kind of checklist to perform all upgrade operations, which implies full management cluster upgrades as well as individual component upgrades. The Oracle Private Cloud Appliance Concepts Guide describes this approach in more detail as part of the "Upgrade" section in the chapter Appliance Administration Overview.
The upgrade plan is generated when the Upgrader itself is upgraded to the latest version. The plan is based on a comparison of the currently installed components on the rack, and the target component versions and latest packages downloaded to shared storage during the preparation of the upgrade environment. The resulting upgrade plan shows for which components an upgrade procedure needs to be executed in the next phase.
All components must be upgraded in a prescribed order. The upgrade plan will prevent a component upgrade procedure from starting if the preceding upgrades have not been completed. An error message informs the administrator which components need to be upgraded first.
Caution:
The ZFS Storage Appliance firmware must be upgraded before all other components. Other firmware may be upgraded whenever new versions are made available for your system. Those firmware upgrades can be applied in no particular order and independently of other components.
This is the order of operations enforced through the upgrade plan:
-
Prepare upgrade environment (Upgrade PreConfig)
-
Upgrade the Upgrader (PreUpgrade)
-
ZFS Storage Appliance firmware (version 3.0.2-b1081557 or later)
-
Compute nodes
-
Host operating system of management nodes
-
MySQL cluster database
-
Secret service (including Etcd and Vault)
-
Kubernetes container orchestration packages (platform layer)
-
Containerized microservices
-
Oracle Cloud Infrastructure images
Once the upgrade environment has been prepared, all upgrade operations required to bring the system up-to-date are listed in the upgrade plan. Whenever an upgrade procedure has been completed successfully, the upgrade plan is updated with the latest status: for upgraded components the source and target versions are identical and the "upgrade required" flag is disabled.
At any point in time you can check how far the system has progressed through the upgrade plan. It indicates which components are already up-to-date and which still require upgrading.
The upgrade plan can be viewed in two ways:
-
In the Service Web UI, go to Maintenance in the navigation menu and select Upgrade Plan.
-
In the Service CLI, enter the following command:
PCA-ADMIN> getUpgradePlan Data: id componentType currentBuild targetBuild currentVersion targetVersion requireReboot timeEstimation requireUpgrade impactedInfra -- ------------- ------------ ----------- -------------- ------------- ------------- -------------- -------------- ------------- generic zfssa 3.0.2-b892153 3.0.2-b1261765 2013.06.05.8.57.1-2.57.5392.2 2013.06.05.8.73.1-2.73.5701.1 false 45 true host,compute 100.96.2.64 compute 3.0.2-b892153 3.0.2-b1261765 3.0.2-640 3.0.2-859 true 20 true compute 100.96.2.65 compute 3.0.2-b892153 3.0.2-b1261765 3.0.2-640 3.0.2-859 true 20 true compute 100.96.2.66 compute 3.0.2-b892153 3.0.2-b1261765 3.0.2-640 3.0.2-859 true 20 true compute 100.96.2.67 compute 3.0.2-b892153 3.0.2-b1261765 3.0.2-640 3.0.2-859 true 20 true compute 100.96.2.68 compute 3.0.2-b892153 3.0.2-b1261765 3.0.2-640 3.0.2-859 true 20 true compute 100.96.2.69 compute 3.0.2-b892153 3.0.2-b1261765 3.0.2-640 3.0.2-859 true 20 true compute 100.96.2.33 host 3.0.2-b892153 3.0.2-b1261765 oraclelinux-release-7.9-1.0.9 oraclelinux-release-7.9-1.0.13 true 35 true host 100.96.2.34 host 3.0.2-b892153 3.0.2-b1261765 oraclelinux-release-7.9-1.0.9 oraclelinux-release-7.9-1.0.13 true 35 true host 100.96.2.35 host 3.0.2-b892153 3.0.2-b1261765 oraclelinux-release-7.9-1.0.9 oraclelinux-release-7.9-1.0.13 true 35 true host generic mysql 3.0.2-b892153 3.0.2-b1261765 8.0.28-1.1 8.0.36-1.1 false 15 true host generic etcd 3.0.2-b892153 3.0.2-b1261765 3.3.10 3.5.6 false 5 true host generic vault 3.0.2-b892153 3.0.2-b1261765 v1.7.1-3 v1.7.1-3 false 5 true host generic kubernetes 3.0.2-b892153 3.0.2-b1261765 1.20.6-1 1.25.16-2 false 350 true host,compute generic platform 3.0.2-b892153 3.0.2-b1261765 None None false 50 true host,compute Oracle-Linux-7.9 ociImages 3.0.2-b892153 3.0.2-b1261765 2022.08.29_0 2024.07.31_0 false 5 true host Oracle-Linux-8 ociImages 3.0.2-b892153 3.0.2-b1261765 2022.08.29_0 2024.07.31_0 false 5 true host Oracle-Linux-9 ociImages 3.0.2-b892153 3.0.2-b1261765 None 2024.07.31_0 false 5 true host Oracle-Linux8-OKE-1.26.15 ociImages 3.0.2-b892153 3.0.2-b1261765 None 20240908 false 5 true host Oracle-Linux8-OKE-1.27.12 ociImages 3.0.2-b892153 3.0.2-b1261765 None 20240908 false 5 true host Oracle-Linux8-OKE-1.28.8 ociImages 3.0.2-b892153 3.0.2-b1261765 None 20240908 false 5 true host Oracle-Solaris-11 ociImages 3.0.2-b892153 3.0.2-b1261765 2023.04.18_0 2024.08.26_0 false 5 true host 100.96.0.33 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute 100.96.0.34 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute 100.96.0.35 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute 100.96.0.64 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute 100.96.0.65 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute 100.96.0.66 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute 100.96.0.67 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute 100.96.0.68 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute 100.96.0.68 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute leaf switch 3.0.2-b892153 3.0.2-b1261765 10.2.5 10.3.4a false 60 true host,compute mgmt switch 3.0.2-b892153 3.0.2-b1261765 10.2.5 10.3.4a false 60 true host,compute spine switch 3.0.2-b892153 3.0.2-b1261765 10.2.5 10.3.4a false 60 true host,compute
Upgrade the Compute Nodes
Caution:
Ensure that all preparation steps for system upgrade have been completed.
The ZFS Storage Appliance firmware must be upgraded before all other components. For more information, see Check Upgrade Plan Status and Progress.
The compute node upgrade ensures that the latest Oracle Linux kernel and user space packages are installed,
as well as the ovm-agent package with appliance-specific optimizations.
Compute nodes must be locked and upgraded one at a time; concurrent upgrades are not
supported. After successful upgrade, when a compute node has rebooted, the administrator must
manually remove the locks to allow the node to return to normal operation.
Note:
In case the ILOM also needs to be upgraded, you can integrate it into this procedure by executing the optional steps. The combined procedure eliminates the need to evacuate and reboot the same node twice.
Note:
In software version 3.0.2-b892153 and later the Upgrader service uses the upgrade plan, generated during the pre-upgrade process, to determine which specific components need to be upgraded, and in what order. If a component is already at the required version, the upgrade command does start an upgrade job, but it is completed immediately because the upgrade plan indicates there is nothing to do.
Practically speaking, when a component is already at the required version, the upgrade
procedure is skipped. However, a same-version upgrade can be forced using the Service Web UI or Service CLI command option, if necessary. For example:
upgradeCN hostIp=100.96.2.64 force=True.
Obtaining a Host IP Address
From the Service CLI, compute nodes are upgraded one at a time, using each one's internal IP address as a command parameter. However, the locking commands use the compute node ID instead. To run all commands for a compute node upgrade you need both identifiers.
To obtain the host IP address and ID, as well as other information relevant to the upgrade procedure, use the Service CLI command provided in the following example. You can run the command as often as needed to check and confirm status as you proceed through the upgrade of all compute nodes.
PCA-ADMIN> list computeNode fields hostname,ipAddress,ilomIp,state,firmwareVersion,provisioningLocked,maintenanceLocked orderby hostname ASCENDING Data: id Hostname Ip Address ILOM Ip Address State Firmware Version Provisioning Locked Maintenance Locked -- -------- ---------- --------------- ----- ---------------- ------------------- ------------------ cf488903-fef8-4a51-8a41-c6990e4755c5 pcacn001 100.96.2.64 100.96.0.64 On PCA Hypervisor:3.0.2-681 false false 42a7594d-1173-4dbd-4755-07810cc2d527 pcacn002 100.96.2.65 100.96.0.65 On PCA Hypervisor:3.0.2-681 false false bc0f37d5-ba77-423e-bc11-017704b47e59 pcacn003 100.96.2.66 100.96.0.66 On PCA Hypervisor:3.0.2-681 false false 2e5ac527-01f5-4230-ae41-0522fcb57c9a pcacn004 100.96.2.67 100.96.0.67 On PCA Hypervisor:3.0.2-681 false false 5a6b61cf-7e99-4df2-87e4-b37c5fb0bfb8 pcacn005 100.96.2.68 100.96.0.68 On PCA Hypervisor:3.0.2-681 false false 885f2aa4-f017-41e8-b2bc-e588cc0c6162 pcacn006 100.96.2.69 100.96.0.69 On PCA Hypervisor:3.0.2-681 false false
Monitoring Displaced Instances
During compute node upgrade or patching, no active compute instances can be present, so the node must be evacuated and locked for maintenance. To evacuate a compute node, the Compute Service live-migrates instances to another compute node in the same fault domain. If the fault domain does not have sufficient capacity, high-availability configuration settings might cause instances to be live-migrated to another fault domain, and migrated back to their selected fault domain when the required capacity is available again.
Compute instances that have been migrated away from their assigned fault domain, are called displaced instances. Their migrations can interfere with compute node upgrade or patching. When the locks on a given compute node are released, its displaced instances start migrating back, during which time it might be impossible to lock the next compute node for maintenance.
Before upgrading or patching a compute node, monitor the status of displaced instances. Do not proceed with the next compute node until the list is empty.
-
In the Service CLI, use the command
getDisplacedInstances. In the following example, two instances have been migrated away from fault domain 1.PCA-ADMIN> getDisplacedInstances Data: id displayName compartmentId faultDomain faultDomainSelected -- ----------- ------------- ----------- ------------------- ocid1.instance.unique_ID inst-name ocid1.compartment.unique_ID FAULT-DOMAIN-3 FAULT-DOMAIN-1 ocid1.instance.unique_ID inst-name ocid1.compartment.unique_ID FAULT-DOMAIN-2 FAULT-DOMAIN-1
-
In the Service Web UI, click the navigation menu, click FD Instances, and then click Displaced Instances.
For more information, refer to the following sections in the Hardware Administration chapter of the Oracle Private Cloud Appliance Administrator Guide:
- Migrating instances and locking a compute node: see "Performing Compute Node Operations".
-
Compute service HA configuration: see "Configuring the Compute Service for High Availability".
Using the Service CLI
-
From the output you obtained with the compute node list command earlier, get the ID and the IP address of the compute node you intend to upgrade.
-
Set the provisioning and maintenance locks for the compute node you are about to upgrade.
Caution:
Depending on the high-availability configuration of the Compute service, automatic instance migrations can prevent you from successfully locking a compute node. For more information, refer to the following sections in the Hardware Administration chapter of the Oracle Private Cloud Appliance Administrator Guide:
- Migrating instances and locking a compute node: see "Performing Compute Node Operations".
-
Compute service HA configuration: see "Configuring the Compute Service for High Availability".
-
Disable provisioning for the compute node.
PCA-ADMIN> provisioningLock id=cf488903-fef8-4a51-8a41-c6990e4755c5 Status: Success JobId: 6ee78c8a-e227-4d31-a770-9b9c96085f3f
-
Evacuate the compute node. Wait for the migration job to finish before proceeding to the next step.
Note:
In case physical resources are limited, compute instances will be migrated to other fault domains during compute node evacuation. However, the strict fault domain enforcement (Strict FD) function must be disabled.
PCA-ADMIN> migrateVm id=cf488903-fef8-4a51-8a41-c6990e4755c5 Status: Running JobId: 6f1e94bc-7d5b-4002-ada9-7d4b504a2599 PCA-ADMIN> show Job id=6f1e94bc-7d5b-4002-ada9-7d4b504a2599 Run State = Succeeded
-
Lock the compute node for maintenance.
PCA-ADMIN> maintenanceLock id=cf488903-fef8-4a51-8a41-c6990e4755c5 Status: Success JobId: e46f6603-2af2-4df4-a0db-b15156491f88
-
Optionally, rerun the compute node list command to confirm lock status. For example:
PCA-ADMIN> list computeNode fields hostname,ipAddress,ilomIp,state,firmwareVersion,provisioningLocked,maintenanceLocked orderby hostname ASCENDING Data: id Hostname Ip Address ILOM Ip Address State Firmware Version Provisioning Locked Maintenance Locked -- -------- ---------- --------------- ----- ---------------- ------------------- ------------------ cf488903-fef8-4a51-8a41-c6990e4755c5 pcacn001 100.96.2.64 100.96.0.64 On PCA Hypervisor:3.0.2-681 true true 42a7594d-1173-4dbd-4755-07810cc2d527 pcacn002 100.96.2.65 100.96.0.65 On PCA Hypervisor:3.0.2-681 false false bc0f37d5-ba77-423e-bc11-017704b47e59 pcacn003 100.96.2.66 100.96.0.66 On PCA Hypervisor:3.0.2-681 false false 2e5ac527-01f5-4230-ae41-0522fcb57c9a pcacn004 100.96.2.67 100.96.0.67 On PCA Hypervisor:3.0.2-681 false false 5a6b61cf-7e99-4df2-87e4-b37c5fb0bfb8 pcacn005 100.96.2.68 100.96.0.68 On PCA Hypervisor:3.0.2-681 false false 885f2aa4-f017-41e8-b2bc-e588cc0c6162 pcacn006 100.96.2.69 100.96.0.69 On PCA Hypervisor:3.0.2-681 false false
-
Optionally, upgrade the server ILOM first.
- Enter the ILOM upgrade command.
Syntax (entered on a single line):
upgradeIlom hostIp=<ilom-ip>Example:
PCA-ADMIN> upgradeIlom hostIp=100.96.0.64 Data: Service request has been submitted. Upgrade Job Id = 1620921089806-ilom-21480 Upgrade Request Id = UWS-732d6fce-9f06-4329-b972-d093bee40010 PCA-ADMIN> getUpgradeJob upgradeJobId=1620921089806-ilom-21480
-
Wait 5 minutes to allow the ILOM upgrade job to complete. Then proceed to the host upgrade.
- Enter the ILOM upgrade command.
-
Enter the compute node upgrade command.
Syntax (entered on a single line):
upgradeCN hostIp=<compute-node-ip> [optional] imageLocation=<path-to-iso> [optional] isoChecksum=<iso-file-checksum>
The parameters marked optional are deprecated in software version 3.0.2-b892153 and later. For earlier versions, include the ISO image parameters with the command.
Example:
PCA-ADMIN> upgradeCN hostIp=100.96.2.64 \ imageLocation=http://host.example.com/pca-<version>-<build>.iso \ isoChecksum=240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7 Status: Success Data: Service request has been submitted. Upgrade Job Id = 1630938939109-compute-7545 Upgrade Request Id = UWS-61736806-7e5a-4648-9259-07c54c39cacb
-
Use the request ID and the job ID to check the status of the upgrade process.
PCA-ADMIN> getUpgradeJobs id upgradeRequestId commandName result -- ---------------- ----------- ------ 1630938939109-compute-7545 UWS-61736806-7e5a-4648-9259-07c54c39cacb compute Passed 1632850650836-platform-68465 UWS-26dba234-9b52-426d-836c-ac11f37e717f platform Passed 1632849609034-kubernetes-35545 UWS-edfa3b32-c32a-4b67-8df5-2357096052bf kubernetes Passed PCA-ADMIN> getupgradejob upgradeJobId=1630938939109-compute-7545 Data: Upgrade Request Id = UWS-61736806-7e5a-4648-9259-07c54c39cacb Name = compute Start Time = 2021-09-26T06:35:39 End Time = 2021-09-26T06:45:55 Pid = 7545 Host = pcamn02 Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_compute_2021_09_26-06.35.39.log Arguments = {"verify_only":false,"upgrade":false,"diagnostics":false,"host_ip":"100.96.2.64","result_override":null,"log_level":null,"switch_type":null,"precheck_status":false,"task_time":0,"fail_halt":false,"fail_upgrade":null,"component_names":null,"upgrade_to":null,"image_location":null,"epld_image_location":null,"expected_iso_checksum":null,"checksum":null,"composition_id":null,"request_id":"UWS-61736806-7e5a-4648-9259-07c54c39cacb","display_task_plan":false,"dry_run_tasks":false} Status = Passed Execution Time(sec) = 616 Tasks 1 - Name = Copy Scripts Tasks 1 - Description = Copy scripts to shared storage Tasks 1 - Time = 2021-09-26T06:35:39 [...] -
When the compute node upgrade has completed successfully and the node has rebooted, release the locks.
For more information, refer to the section "Performing Compute Node Operations". It can be found in the chapter Hardware Administration of the Oracle Private Cloud Appliance Administrator Guide.
-
Release the maintenance lock.
PCA-ADMIN> maintenanceUnlock id=cf488903-fef8-4a51-8a41-c6990e4755c5 Status: Success JobId: 625af20e-4b49-4201-879f-41d4405314c7
-
Release the provisioning lock.
PCA-ADMIN> provisioningUnlock id=cf488903-fef8-4a51-8a41-c6990e4755c5 Status: Success JobId: 523892e8-c2d4-403c-9620-2f3e94015b46
-
-
Proceed to the next compute node and repeat this procedure.
The output from the compute node list command indicates the current status. For example:
PCA-ADMIN> list computeNode fields hostname,ipAddress,ilomIp,state,firmwareVersion,provisioningLocked,maintenanceLocked orderby hostname ASCENDING Data: id Hostname Ip Address ILOM Ip Address State Firmware Version Provisioning Locked Maintenance Locked -- -------- ---------- --------------- ----- ---------------- ------------------- ------------------ cf488903-fef8-4a51-8a41-c6990e4755c5 pcacn001 100.96.2.64 100.96.0.64 On PCA Hypervisor:3.0.2-696 false false 42a7594d-1173-4dbd-4755-07810cc2d527 pcacn002 100.96.2.65 100.96.0.65 On PCA Hypervisor:3.0.2-696 false false bc0f37d5-ba77-423e-bc11-017704b47e59 pcacn003 100.96.2.66 100.96.0.66 On PCA Hypervisor:3.0.2-696 false false 2e5ac527-01f5-4230-ae41-0522fcb57c9a pcacn004 100.96.2.67 100.96.0.67 On PCA Hypervisor:3.0.2-696 false false 5a6b61cf-7e99-4df2-87e4-b37c5fb0bfb8 pcacn005 100.96.2.68 100.96.0.68 On PCA Hypervisor:3.0.2-681 false false 885f2aa4-f017-41e8-b2bc-e588cc0c6162 pcacn006 100.96.2.69 100.96.0.69 On PCA Hypervisor:3.0.2-681 false false
Using the Service Web UI
-
Set the provisioning and maintenance locks for the compute node you are about to upgrade. Ensure that no active compute instances are present on the node.
Caution:
Depending on the high-availability configuration of the Compute service, automatic instance migrations can prevent you from successfully locking a compute node. See Monitoring Displaced Instances.
-
In the navigation menu, click Rack Units. In the Rack Units table, click the name of the compute node you want to upgrade to display its detail page.
-
In the top-right corner of the compute node detail page, click Controls and select the Provisioning Lock command.
-
When the provisioning lock has been set, click Controls again and select the Migrate All Vms command. The Compute service evacuates the compute node, meaning it migrates the running instances to other compute nodes.
Note:
In case physical resources are limited, compute instances will be migrated to other fault domains during compute node evacuation. However, the strict fault domain enforcement (Strict FD) function must be disabled.
-
When compute node evacuation is complete, click Controls again and select the Maintenance Lock command. This command might fail if instance migrations are in progress. Wait a few minutes and retry.
-
-
In the navigation menu, go to the Maintenance section and click Upgrade Plan. This provides an overview of current and target component versions.
-
Click Upgrade & Patching to display the Upgrade Jobs page.
-
Optionally, upgrade the server ILOM first.
-
In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch. The Create Request window appears.
-
Choose Upgrade as the Request Type. Select the appropriate upgrade request type: Upgrade ILOM.
Fill out the server's assigned IP address in the ILOM network. This is an IP address in the internal 100.96.0.0/23 range.
-
Click Create Request. The new upgrade request appears in the Upgrade Jobs table.
-
Wait 5 minutes to allow the ILOM upgrade job to complete. Then proceed to the host upgrade.
-
-
In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.
The Create Request window appears. Choose Upgrade as the Request Type.
-
Select the appropriate upgrade request type: Upgrade CN.
-
Fill out the upgrade request parameters:
-
Host IP: Enter the compute node's assigned IP address in the internal administration network. This is an IP address in the internal 100.96.2.0/23 range.
-
Log Level: Optionally, select a specific log level for the upgrade log file. The default log level is "Information". For maximum detail, select "Debug".
-
Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.
-
Alternative ULN Channel: This parameter applies to patching and can be ignored.
-
Verify Only: Enable this option to run the operation in verification only mode.
-
Force: Enable this option to force the operation. Use only when instructed by Oracle.
-
-
Click Create Request.
The new upgrade request appears in the Upgrade Jobs table.
-
When the compute node has been upgraded successfully, release the provisioning and maintenance locks.
For more information, refer to the section "Performing Compute Node Operations". It can be found in the chapter Hardware Administration of the Oracle Private Cloud Appliance Administrator Guide.
-
Open the compute node detail page.
-
In the top-right corner of the compute node detail page, click Controls and select the Maintenance Unlock command.
-
When the maintenance lock has been released, click Controls again and select the Provisioning Unlock command.
-
Upgrade the Management Node Cluster
Caution:
Ensure that all preparation steps for system upgrade have been completed.
Ensure that the ZFS Storage Appliance firmware and all compute nodes have been upgraded.
A full management node cluster upgrade is a convenient way to upgrade all the required components on all three management nodes using just a single command. As part of this process, the following components are upgraded, in this specific order:
-
host operating system
-
Clustered MySQL database
-
secret service (including Etcd and Vault)
-
Kubernetes container orchestration packages
-
containerized microservices
-
Oracle Cloud Infrastructure images
Caution:
When the new images have been imported, a background job is launched to ensure that running OKE clusters receive the latest available CVE fixes delivered with the new images. For more information, see Upgrade the Oracle Cloud Infrastructure Images.
Note:
In software version 3.0.2-b892153 and later the Upgrader service uses the upgrade plan, generated during the pre-upgrade process, to determine which specific components need to be upgraded, and in what order. If a component is already at the required version, the upgrade command does start an upgrade job, but it is completed immediately because the upgrade plan indicates there is nothing to do.
Practically speaking, when a component is already at the required version, the upgrade
procedure is skipped. However, a same-version upgrade can be forced using the Service Web UI or Service CLI command option, if necessary. For example:
upgradeKubernetes force=True.
Using the Service CLI
-
Run the upgrade command in verify-only mode.
Caution:
Oracle strongly recommends using the verify-only option (
verifyOnly=True) on the first run. This starts a verification job for each component. Before you start the actual upgrade, all verification jobs must be completed, which could take around 45 minutes in total.-
Start the upgrade request in verify-only mode.
PCA-ADMIN> upgradeFullMN verifyOnly=True Data: Service request has been submitted. Upgrade Request Id = UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 -
Find the associated upgrade jobs and note the IDs.
PCA-ADMIN> getUpgradeJobs requestId=UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 Data: id upgradeRequestId commandName result -- ---------------- ----------- ------ 1698788901590-oci_verify-38799 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 oci_verify Passed 1698788685720-platform_verify-34138 UWS-20f69d05-9c43-4617-80a4-8cf8db5b8446 platform_verify Passed 1698788568097-kubernetes_verify-32571 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 kubernetes_verify Passed 1698788265717-vault_verify-27153 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 vault_verify Passed 1698787959640-etcd_verify-26413 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 etcd_verify Passed 1698787657287-mysql_verify-19656 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 mysql_verify Passed 1698787353293-host_verify-18867 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 host_verify Passed 1698787049308-host_verify-12573 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 host_verify Passed 1698786925154-host_verify-36612 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 host_verify Passed
-
Check the status in the upgrade job details.
PCA-ADMIN> getUpgradeJob upgradeJobId=1698787353293-host_verify-18867 Data: Upgrade Request Id = UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 Name = host_verify Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_host_os_pcamn03_verify_2023_06_16-13.07.11.log Status = Passed
-
-
When the upgrade verification has completed successfully, start the actual full management node upgrade.
PCA-ADMIN> upgradeFullMN Data: Service request has been submitted. Upgrade Request Id = UWS-39329657-1051-4267-8c5a-9314f8e63a64
-
Use the request ID to check the status of the upgrade process.
As the full management node upgrade is a multi-component upgrade process, there are multiple upgrade jobs associated with the upgrade request. You can filter for those jobs based on the request ID. Using the job ID, you can drill down into the details of each upgrade job.
PCA-ADMIN> getUpgradeJobs requestId=UWS-39329657-1051-4267-8c5a-9314f8e63a64 Data: id upgradeRequestId commandName result -- ---------------- ----------- ------ 1634579161548-oci-47486 UWS-39329657-1051-4267-8c5a-9314f8e63a64 oci Passed 1634578760906-platform-66082 UWS-39329657-1051-4267-8c5a-9314f8e63a64 platform Passed 1634578263434-kubernetes-63574 UWS-39329657-1051-4267-8c5a-9314f8e63a64 kubernetes Passed 1634578012353-vault-51696 UWS-39329657-1051-4267-8c5a-9314f8e63a64 vault Passed 1634577380954-etcd-46337 UWS-39329657-1051-4267-8c5a-9314f8e63a64 etcd Passed 1634577341291-mysql-40127 UWS-39329657-1051-4267-8c5a-9314f8e63a64 mysql Passed 1634576985926-host-36556 UWS-39329657-1051-4267-8c5a-9314f8e63a64 host Passed 1634576652071-host-27088 UWS-39329657-1051-4267-8c5a-9314f8e63a64 host Passed 1634576191050-host-24909 UWS-39329657-1051-4267-8c5a-9314f8e63a64 host Passed PCA-ADMIN> getUpgradeJob upgradeJobId=1634576652071-host-27088 Data: Upgrade Request Id = UWS-39329657-1051-4267-8c5a-9314f8e63a64 Composition Id = 1 Name = host Pid = 27088 Host = pcamn02 Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_host_os_2023_05_24-07.04.12.log Tasks 1 - Name = Validate Image Location Tasks 1 - Description = Verify that the image exists at the specified location and is correctly named [...]
The output of the
getUpgradeJobcommand provides detailed information about the tasks performed during the upgrade procedure. It displays descriptions, time stamps, duration, and success or failure. Whenever an upgrade operation fails, the command output indicates which task has failed. For in-depth troubleshooting you can search the log file at the location provided near the start of the command output.Note:
After upgrade, if the upgrade plan specifies that the management nodes must be rebooted for the changes to take effect, a reboot is performed as part of the upgrade process. No administrator action is required.
Using the Service Web UI
-
In the navigation menu, go to the Maintenance section and click Upgrade Plan. This provides an overview of current and target component versions.
-
Click Upgrade & Patching to display the Upgrade Jobs page.
-
In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.
The Create Request window appears. Choose Upgrade as the Request Type.
-
Select the appropriate upgrade request type.
For a full management node upgrade, select Upgrade MN.
Caution:
Oracle strongly recommends using the verify-only option on the first run. This starts a verification job for each component. Before you start the actual upgrade, all verification jobs must be completed, which could take around 45 minutes in total.
-
Fill out the upgrade request parameters:
-
Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.
-
Alternative ULN Channel: This parameter applies to patching and can be ignored.
-
Verify Only: Enable this option to run the operation in verification only mode.
-
Force: Enable this option to force the operation. Use only when instructed by Oracle.
-
-
Click Create Request.
The new upgrade request appears in the Upgrade Jobs table.
Note:
After upgrade, if the upgrade plan specifies that the management nodes must be rebooted for the changes to take effect, a reboot is performed as part of the upgrade process. No administrator action is required.
Upgrade Individual Components
A full management node cluster upgrade is a convenient way to upgrade all the required components on all three management nodes using just a single command. However, the process might be interrupted or an error might occur along the way. Instead of repeating the full management cluster upgrade, it might be more efficient to perform individual component upgrades in this situation.
Check the upgrade command output and the logs for any warnings or error messages. Determine at which stage the management node cluster upgrade failed, and resolve any issue that appears to have caused the failure. Proceed with individual component upgrades, starting with the component that was not upgraded successfully. Upgrade operations in the sections that follow, are listed in the correct order, exactly as they are performed during a management node cluster upgrade.
Note:
In software version 3.0.2-b892153 and later the Upgrader service uses the upgrade plan, generated during the pre-upgrade process, to determine which specific components need to be upgraded, and in what order. If a component is already at the required version, the upgrade command does start an upgrade job, but it is completed immediately because the upgrade plan indicates there is nothing to do.
Practically speaking, when a component is already at the required version, the upgrade
procedure is skipped. However, a same-version upgrade can be forced using the Service Web UI or Service CLI command option, if necessary. For example:
upgradeKubernetes force=True.
Upgrade the Management Node Operating System
The Oracle Linux host operating system of the management nodes must be upgraded one node at a time; a rolling upgrade of all management nodes is not possible. This upgrade process, which involves updating the kernel and system packages, detects which of the nodes in the three-management-node cluster owns the virtual IP and the primary role. When the current primary node is upgraded, the primary role is first transferred to another node in the cluster. This configuration change occurs in the background and requires no separate or additional intervention from an administrator.
Note:
In case the ILOM also needs to be upgraded, you can integrate it into this procedure by executing the optional steps. The combined procedure eliminates the need to evacuate and reboot the same node twice.
You must upgrade management nodes one at a time, using each one's internal IP address as a command parameter. The following example uses a Service CLI command that retrieves the management node IP addresses, as well as other information relevant to the upgrade procedure.
PCA-ADMIN> list managementNode fields hostname,ipAddress,ilomIp,state,firmwareVersion orderby hostname ASCENDING Data: id Hostname Ip Address ILOM Ip Address State Firmware Version -- -------- ---------- --------------- ----- ---------------- 22ae47d8-a57a-433c-988d-df62fd3548e1 pcamn01 100.96.2.33 100.96.0.33 On 5.0.2.23 042693b6-3ddb-4fb0-914d-f3deea838c8f pcamn02 100.96.2.34 100.96.0.34 On 5.0.2.23 bd2563b8-e310-4fca-ba1b-0e19b8040fc6 pcamn03 100.96.2.35 100.96.0.35 On 5.0.2.23
-
Get the IP address of the management node for which you intend to upgrade the host operating system.
-
Optionally, upgrade the server ILOM first.
- Enter the ILOM upgrade command.
Syntax (entered on a single line):
upgradeIlom hostIp=<ilom-ip>Example:
PCA-ADMIN> upgradeIlom hostIp=100.96.0.35 Data: Service request has been submitted. Upgrade Job Id = 1632990827394-ilom-23871 Upgrade Request Id = UWS-1a97a8d9-9f06-478d-a0c0-d093bee4672 PCA-ADMIN> getUpgradeJob upgradeJobId=1632990827394-ilom-23871
-
Wait 5 minutes to allow the ILOM upgrade job to complete. Then proceed to the host upgrade.
- Enter the ILOM upgrade command.
-
Enter the management node host upgrade command.
Syntax (entered on a single line):
upgradeHost hostIp=<management-node-ip>Example:
PCA-ADMIN> upgradeHost hostIp=100.96.2.35 Command: upgradeHost hostIp=100.96.2.35 Status: Success Time: 2021-09-25 05:47:02,735 UTC Data: Service request has been submitted. Upgrade Job Id = 1632990827394-host-56156 Upgrade Request Id = UWS-1a97a8d9-54ef-478d-a0c0-348a17ba6755
-
Use the request ID and the job ID to check the status of the upgrade process.
PCA-ADMIN> getUpgradeJobs id upgradeRequestId commandName result -- ---------------- ----------- ------ 1632990827394-host-56156 UWS-1a97a8d9-54ef-478d-a0c0-348a17ba6755 host Passed PCA-ADMIN> getUpgradeJob upgradeJobId=1632990827394-host-56156 Data: Upgrade Request Id = UWS-1a97a8d9-54ef-478d-a0c0-348a17ba6755 Composition Id = 1 Name = host Pid = 56156 Host = pcamn02 Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_host_os_2021_09_25-05.47.02.log [...]
-
When the first management node host operating system upgrade has completed successfully, execute the same command for the second management node. When that upgrade has completed successfully, execute the same command for the third management node.
PCA-ADMIN> upgradeHost hostIp=100.96.2.33 PCA-ADMIN> upgradeHost hostIp=100.96.2.34
When all three operating system upgrades have completed successfully, the management node cluster is up-to-date.
Note:
After upgrade, if the upgrade plan specifies that the management nodes must be rebooted for the changes to take effect, a reboot is performed as part of the upgrade process. No administrator action is required.
Upgrade the MySQL Cluster Database
The MySQL Cluster database is upgraded independently of the management node host operating system; the MySQL packages are deliberately kept separate from the Oracle Linux upgrade.
The MySQL Cluster database upgrade is a rolling upgrade: with one command the upgrade is executed on each of the three management nodes.
-
Enter the upgrade command.
PCA-ADMIN> upgradeMySQL Data: Service request has been submitted. Upgrade Job Id = 1632995409822-mysql-83013 Upgrade Request Id = UWS-77bc0c30-7ff5-4c50-ad09-6f96907e22e1
-
Use the request ID and the job ID to check the status of the upgrade process.
PCA-ADMIN> getUpgradeJobs id upgradeRequestId commandName result -- ---------------- ----------- ------ 1632995409822-mysql-83013 UWS-77bc0c30-7ff5-4c50-ad09-6f96907e22e1 mysql Passed PCA-ADMIN> getUpgradeJob upgradeJobId=1632995409822-mysql-83013 Data: Upgrade Request Id = UWS-77bc0c30-7ff5-4c50-ad09-6f96907e22e1 Name = mysql Pid = 83013 Host = pcamn01 Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_mysql_cluster_2021_09_25-09.21.16.log [...]
Upgrade the Secret Service
The secret service contains two components that need to be upgraded separately in this particular order: first Etcd, then Vault.
The Etcd and Vault upgrades are rolling upgrades: each upgrade is executed on all three management nodes with one command.
-
Enter the two upgrade commands. Wait until the Etcd upgrade is finished before starting the Vault upgrade.
PCA-ADMIN> upgradeEtcd Data: Service request has been submitted. Upgrade Job Id = 1632826770954-etcd-26973 Upgrade Request Id = UWS-fec15d32-fc2b-48bd-9ae0-62f49587a284 PCA-ADMIN> upgradeVault Data: Service request has been submitted. Upgrade Job Id = 1632850933353-vault-16966 Upgrade Request Id = UWS-352df3d1-c21f-441b-8f6e-9381ac075906
-
Use the request ID and the job ID to check the status of the upgrade process.
PCA-ADMIN> getUpgradeJobs id upgradeRequestId commandName result -- ---------------- ----------- ------ 1632995409822-mysql-83013 UWS-77bc0c30-7ff5-4c50-ad09-6f96907e22e1 mysql Passed 1632850933353-vault-16966 UWS-352df3d1-c21f-441b-8f6e-9381ac075906 vault Passed 1632826770954-etcd-26973 UWS-fec15d32-fc2b-48bd-9ae0-62f49587a284 etcd Passed PCA-ADMIN> getUpgradeJob upgradeJobId=1632850933353-vault-16966 Data: Upgrade Request Id = UWS-352df3d1-c21f-441b-8f6e-9381ac075906 Name = vault Pid = 16966 Host = pcamn02 Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_vault_2021_09_25-10.38.25.log [...]
Upgrade the Kubernetes Cluster
The Kubernetes container orchestration environment upgrade is also kept separate from the operating system. With a single command, all Kubernetes packages, such as kubeadm, kubectl and kubelet, are upgraded on the three management nodes and all the compute nodes. Note that this upgrade does not include the microservices running within the Kubernetes cluster.
- About the Kubernetes Upgrade Process
-
To ensure compatibility and continuation of service, Kubernetes must be upgraded one version at a time. Skipping versions – major or minor – is not supported. The Private Cloud Appliance Upgrader manages this process by upgrading or patching all parts of the Kubernetes cluster to the next available version, repeating the same sequence of operations until the entire environment runs the latest Kubernetes version available from the appliance software repositories.
Upgrading or patching the Kubernetes cluster is a time-consuming process that involves the Private Cloud Appliancemanagement nodes and compute nodes. Each additional compute node extends the process by appoximately 10 minutes for each incremental version of Kubernetes.
With appliance software version 3.0.2-b925538 or later, the container orchestration environment is upgraded or patched from Kubernetes version 1.20.x to version 1.25.y, meaning the entire process must run 5 times. After each successful run, the repository is synchronized to retrieve the next required version. However, with this version of the appliance software the repository is reconfigured to allow multiple versions of the Kubernetes packages, so the resync will no longer be required.
Each individual Kubernetes node upgrade is expected to take around 10 minutes. Testing indicates that upgrading or patching the Private Cloud Appliance Kubernetes cluster from version 1.20 to version 1.25 takes approximately 4-5 hours for a base rack configuration with 3 management nodes and 3 compute nodes. On a full rack with 20 compute nodes the entire process requires at least 9 hours and may take up to 18 hours to complete. The estimated time for the rack's specific configuration is reported in the upgrade plan.
To monitor the upgrade or patching progress, periodically check the job status or the logs.
-
Check job status through the Service CLI:
getUpgradeJob upgradeJobId=<id> -
View Upgrader logs on a management node:
tail -f /nfs/shared_storage/pca_upgrader/log/pca-upgrader_kubernetes_cluster_<time_stamp>.log.
During Kubernetes upgrade or patching, certain services could be temporarily unavailable.
-
The Compute Web UI, Service Web UI, OCI CLI, and Service CLI can all become temporarily unavailable. Users should wait a few minutes before attempting their operations again. Administrative operations in the Service Enclave (UI or CLI) must be avoided during upgrade or patching.
-
When the Kubernetes upgrade is initiated, the Kubernetes Workload Monitoring Operator (Sauron service) is taken down. As a result, the Grafana, Prometheus, and other Sauron ingress endpoints cannot be accessed. They become available again after both the Kubernetes cluster and the containerized microservices (platform layer) upgrade or patching processes have been completed.
-
-
Enter the upgrade command.
PCA-ADMIN> upgradeKubernetes Data: Service request has been submitted. Upgrade Job Id = 1632849609034-kubernetes-35545 Upgrade Request Id = UWS-edfa3b32-c32a-4b67-8df5-2357096052bf
-
Use the request ID and the job ID to check the status of the upgrade process.
PCA-ADMIN> getUpgradeJobs id upgradeRequestId commandName result -- ---------------- ----------- ------ 1632849609034-kubernetes-35545 UWS-edfa3b32-c32a-4b67-8df5-2357096052bf kubernetes Passed PCA-ADMIN> getUpgradeJob upgradeJobId=1632849609034-kubernetes-35545 Data: Upgrade Request Id = UWS-edfa3b32-c32a-4b67-8df5-2357096052bf Name = kubernetes Pid = 35545 Host = pcamn02 Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_kubernetes_cluster_2021_09_26-17.20.09.log [...]
Upgrade the Microservices
The microservices upgrade covers both the internal services of the platform layer, and the administrative and user-level services exposed through the infrastructure services layer.
Note:
In specific circumstances it is possible to upgrade certain platform services individually, by adding an optional JSON string to the command. This option should not be used unless Oracle provides explicit instructions to do so.
The containerized microservices have their own separate upgrade mechanism. A service is upgraded if a new Helm deployment chart and container image are found in the ISO image. When a new deployment chart is detected during the upgrade process, the pods running the services are restarted with the new container image.
-
Enter the upgrade command.
PCA-ADMIN> upgradePlatform Data: Service request has been submitted. Upgrade Job Id = 1632850650836-platform-68465 Upgrade Request Id = UWS-26dba234-9b52-426d-836c-ac11f37e717f
-
Use the request ID and the job ID to check the status of the upgrade process.
PCA-ADMIN> getUpgradeJobs id upgradeRequestId commandName result -- ---------------- ----------- ------ 1632850650836-platform-68465 UWS-26dba234-9b52-426d-836c-ac11f37e717f platform Passed PCA-ADMIN> getUpgradeJob upgradeJobId=1632850650836-platform-68465 Data: Upgrade Request Id = UWS-26dba234-9b52-426d-836c-ac11f37e717f Name = kubernetes Pid = 68465 Host = pcamn02 Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_platform_services_2021_09_26-20.48.41.log [...]
Upgrade the Oracle Cloud Infrastructure Images
When new Oracle Cloud Infrastructure Images become available and
supported for Oracle Private Cloud Appliance, you can make them
available for use in all existing tenancies with a single upgrade command. The images are
stored in the /nfs/shared_storage/oci_compute_images directory on the ZFS Storage Appliance.
The image versions are tracked through the upgrade plan. Review the upgrade plan to verify
if new images are included with the target software version. An upgrade adds new Oracle Cloud Infrastructure Images to your environment, but it never
removes any existing images. If you no longer need an image, you have the option to delete
it using the deletePlatformImage command.
-
Enter the upgrade command.
PCA-ADMIN> upgradeOCIImages Data: Service request has been submitted. Upgrade Job Id = 1680260388058-oci-38288 Upgrade Request Id = UWS-e601b085-d59d-4ae4-82a3-a09b446686ff
-
Use the request ID and the job ID to check the status of the upgrade process.
PCA-ADMIN> getUpgradeJobs Data: id upgradeRequestId commandName result -- ---------------- ----------- ------ 1680260388058-oci-38288 UWS-e601b085-d59d-4ae4-82a3-a09b446686ff oci Passed PCA-ADMIN> getUpgradeJob upgradeJobId=1680260388058-oci-38288 Data: Upgrade Request Id = UWS-e601b085-d59d-4ae4-82a3-a09b446686ff Name = oci [...]
Upgrade Component Firmware
Caution:
Ensure that all preparation steps for system upgrade have been completed.
The ZFS Storage Appliance firmware must be upgraded before all other components. Other firmware may be upgraded whenever new versions are made available for your system. Those firmware upgrades can be applied in no particular order and independently of other components.
Firmware is included in the ISO image for all component ILOMs, for the ZFS Storage Appliance, and for the switches. Select the instructions below for the component type you want to upgrade.
During the full management node cluster upgrade, an upgrade plan has been generated. You can use it to check the progress of the appliance upgrade and determine which firmware upgrades still need to be performed.
PCA-ADMIN> getUpgradePlan Data: id componentType currentBuild targetBuild currentVersion targetVersion requireReboot timeEstimation requireUpgrade impactedInfra -- ------------- ------------ ----------- -------------- ------------- ------------- -------------- -------------- ------------- generic zfssa 3.0.2-b1261765 3.0.2-b1261765 2013.06.05.8.73.1-2.73.5701.1 2013.06.05.8.73.1-2.73.5701.1 false 45 false host,compute 100.96.2.64 compute 3.0.2-b1261765 3.0.2-b1261765 3.0.2-859 3.0.2-859 false 20 false compute 100.96.2.65 compute 3.0.2-b1261765 3.0.2-b1261765 3.0.2-859 3.0.2-859 false 20 false compute 100.96.2.66 compute 3.0.2-b1261765 3.0.2-b1261765 3.0.2-859 3.0.2-859 false 20 false compute 100.96.2.67 compute 3.0.2-b1261765 3.0.2-b1261765 3.0.2-859 3.0.2-859 false 20 false compute 100.96.2.68 compute 3.0.2-b1261765 3.0.2-b1261765 3.0.2-859 3.0.2-859 false 20 false compute 100.96.2.69 compute 3.0.2-b1261765 3.0.2-b1261765 3.0.2-859 3.0.2-859 false 20 false compute 100.96.2.33 host 3.0.2-b1261765 3.0.2-b1261765 oraclelinux-release-7.9-1.0.13 oraclelinux-release-7.9-1.0.13 false 35 false host 100.96.2.34 host 3.0.2-b1261765 3.0.2-b1261765 oraclelinux-release-7.9-1.0.13 oraclelinux-release-7.9-1.0.13 false 35 false host 100.96.2.35 host 3.0.2-b1261765 3.0.2-b1261765 oraclelinux-release-7.9-1.0.13 oraclelinux-release-7.9-1.0.13 false 35 false host generic mysql 3.0.2-b1261765 3.0.2-b1261765 8.0.36-1.1 8.0.36-1.1 false 15 false host generic etcd 3.0.2-b1261765 3.0.2-b1261765 3.5.6 3.5.6 false 5 false host generic vault 3.0.2-b1261765 3.0.2-b1261765 v1.7.1-3 v1.7.1-3 false 5 false host generic kubernetes 3.0.2-b1261765 3.0.2-b1261765 1.25.16-2 1.25.16-2 false 350 false host,compute generic platform 3.0.2-b1261765 3.0.2-b1261765 None None false 50 false host,compute Oracle-Linux-7.9 ociImages 3.0.2-b1261765 3.0.2-b1261765 2024.07.31_0 2024.07.31_0 false 5 false host Oracle-Linux-8 ociImages 3.0.2-b1261765 3.0.2-b1261765 2024.07.31_0 2024.07.31_0 false 5 false host Oracle-Linux-9 ociImages 3.0.2-b1261765 3.0.2-b1261765 2024.07.31_0 2024.07.31_0 false 5 false host Oracle-Linux8-OKE-1.26.15 ociImages 3.0.2-b1261765 3.0.2-b1261765 20240908 20240908 false 5 false host Oracle-Linux8-OKE-1.27.12 ociImages 3.0.2-b1261765 3.0.2-b1261765 20240908 20240908 false 5 false host Oracle-Linux8-OKE-1.28.8 ociImages 3.0.2-b1261765 3.0.2-b1261765 20240908 20240908 false 5 false host Oracle-Solaris-11 ociImages 3.0.2-b1261765 3.0.2-b1261765 2024.08.26_0 2024.08.26_0 false 5 false host 100.96.0.33 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute 100.96.0.34 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute 100.96.0.35 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute 100.96.0.64 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute 100.96.0.65 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute 100.96.0.66 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute 100.96.0.67 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute 100.96.0.68 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute 100.96.0.68 ilom 3.0.2-b892153 3.0.2-b1261765 5.0.2.23 5.1.4.25 true 10 true host,compute leaf switch 3.0.2-b892153 3.0.2-b1261765 10.2.5 10.3.4a false 60 true host,compute mgmt switch 3.0.2-b892153 3.0.2-b1261765 10.2.5 10.3.4a false 60 true host,compute spine switch 3.0.2-b892153 3.0.2-b1261765 10.2.5 10.3.4a false 60 true host,compute
Obtaining an ILOM IP Address
Using the Service CLI
Note:
If the appliance is running software version 3.0.2-b892153 or earlier, use the third option from this list.
-
To list the ILOM IP addresses of all management nodes or compute nodes, use these commands:
PCA-ADMIN> getCNIloms Data: status = success data 1 = 100.96.0.66 data 2 = 100.96.0.64 data 3 = 100.96.0.65 PCA-ADMIN> getMNIloms Data: status = success data 1 = 100.96.0.33 data 2 = 100.96.0.34 data 3 = 100.96.0.35
-
To obtain the ILOM IP address that corresponds with a specific node host name, use the following command:
PCA-ADMIN> getServerILOMIP hostName=<node_name>For example:
PCA-ADMIN> getServerILOMIP hostName=pcacn002 Data: status = success data = 100.96.0.65 PCA-ADMIN> getServerILOMIP hostName=pcamn03 Data: status = success data = 100.96.0.35
-
To obtain the ID, IP addresses, and other key information about compute nodes or management nodes, use the following commands:
PCA-ADMIN> list managementNode fields hostname,ipAddress,ilomIp,state,firmwareVersion orderby hostname ASCENDING Data: id Hostname Ip Address ILOM Ip Address State Firmware Version -- -------- ---------- --------------- ----- ---------------- 22ae47d8-a57a-433c-988d-df62fd3548e1 pcamn01 100.96.2.33 100.96.0.33 On 5.0.2.23 042693b6-3ddb-4fb0-914d-f3deea838c8f pcamn02 100.96.2.34 100.96.0.34 On 5.0.2.23 bd2563b8-e310-4fca-ba1b-0e19b8040fc6 pcamn03 100.96.2.35 100.96.0.35 On 5.0.2.23 PCA-ADMIN> list computeNode fields hostname,ipAddress,ilomIp,state,firmwareVersion,provisioningLocked,maintenanceLocked orderby hostname ASCENDING Data: id Hostname Ip Address ILOM Ip Address State Firmware Version Provisioning Locked Maintenance Locked -- -------- ---------- --------------- ----- ---------------- ------------------- ------------------ cf488903-fef8-4a51-8a41-c6990e4755c5 pcacn001 100.96.2.64 100.96.0.64 On PCA Hypervisor:3.0.2-681 false false 42a7594d-1173-4dbd-4755-07810cc2d527 pcacn002 100.96.2.65 100.96.0.65 On PCA Hypervisor:3.0.2-681 false false bc0f37d5-ba77-423e-bc11-017704b47e59 pcacn003 100.96.2.66 100.96.0.66 On PCA Hypervisor:3.0.2-681 false false 2e5ac527-01f5-4230-ae41-0522fcb57c9a pcacn004 100.96.2.67 100.96.0.67 On PCA Hypervisor:3.0.2-681 false false 5a6b61cf-7e99-4df2-87e4-b37c5fb0bfb8 pcacn005 100.96.2.68 100.96.0.68 On PCA Hypervisor:3.0.2-681 false false 885f2aa4-f017-41e8-b2bc-e588cc0c6162 pcacn006 100.96.2.69 100.96.0.69 On PCA Hypervisor:3.0.2-681 false false
Using the Service Web UI
-
In the navigation menu, click Rack Units.
-
Click on the name of the component you are patching.
-
Select the Rack Unit Information tab.
-
Record the IP Address listed under ILOM IPs.
Upgrade ZFS Storage Appliance Operating Software
To upgrade the operating software of the appliance's ZFS Storage Appliance, you enter the upgrade command with no additional parameters. The IP addresses of the storage controllers and the location of the firmware package are known, and a single upgrade command initiates a rolling upgrade of both controllers. If a new ILOM firmware version is included for the two controllers, it will be installed as part of the ZFS Storage Appliance upgrade process.
Caution:
Ensure that no users are logged in to the ZFS Storage Appliance or the storage controller ILOMs during the upgrade process.
Do not make storage configuration changes while an upgrade is in progress. While controllers are running different software versions, configuration changes made to one controller are not propagated to its peer controller.
During firmware upgrade the storage controllers are placed in active/passive mode. They automatically return to active/active after the upgrade is completed.
Before You Begin
- From a management node, set the provisioning lock by issuing this command:
pca-admin locks set system provisioning
- Perform the ZFS Storage Appliance upgrade using either the Service Web UI or the Service CLI procedure below.
- Release the provisioning
lock.
pca-admin locks unset system provisioning
- Confirm the lock
state.
pca-admin locks show system
Using the Service CLI
-
Enter the upgrade command.
PCA-ADMIN> upgradeZfssa Data: Service request has been submitted. Upgrade Job Id = 1632914107346-zfssa-83002 Upgrade Request Id = UWS-881af57f-5dfb-4c75-8026-9f00cf3eb7c9
-
Use the request ID and the job ID to check the status of the upgrade process.
PCA-ADMIN> getUpgradeJobs id upgradeRequestId commandName result -- ---------------- ----------- ------ 1632914107346-zfssa-83002 UWS-881af57f-5dfb-4c75-8026-9f00cf3eb7c9 zfssa Passed PCA-ADMIN> getUpgradeJob upgradeJobId=1632914107346-zfssa-83002 Data: Upgrade Request Id = UWS-881af57f-5dfb-4c75-8026-9f00cf3eb7c9 Name = zfssa Pid = 83002 Host = pcamn02 Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_zfssa_ak_2021_09_29-11.15.07.log [...]
Using the Service Web UI
-
In the navigation menu, go to the Maintenance section and click Upgrade Plan. This provides an overview of current and target component versions.
-
Click Upgrade & Patching to display the Upgrade Jobs page.
-
In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.
The Create Request window appears. Choose Upgrade as the Request Type.
-
Select the appropriate upgrade request type: Upgrade Zfssa.
-
Fill out the upgrade request parameters:
-
Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.
-
Image Location: This parameter is deprecated.
-
Log Level: Optionally, select a specific log level for the upgrade log file. The default log level is "Information". For maximum detail, select "Debug".
-
Alternative ULN Channel: This parameter applies to patching and can be ignored.
-
Verify Only: Enable this option to run the operation in verification only mode.
-
Force: Enable this option to force the operation. Use only when instructed by Oracle.
-
-
Click Create Request.
The new upgrade request appears in the Upgrade Jobs table.
Upgrade ILOMs
Note:
In case a server node needs both a host and ILOM upgrade, you can avoid having to reboot the same node twice by combining the two upgrades. Instructions are provided in the following sections:
ILOM upgrades can be applied to management nodes and compute nodes; the firmware packages might be different per component type. You must upgrade ILOMs one at a time, using each one's internal IP address as a command parameter.
Caution:
You must NOT upgrade the ILOM of the management node that holds the management virtual IP address, and thus the primary role in the cluster. To determine which management node has the primary role in the cluster, and make another node the primary, use the following Service CLI commands:
PCA-ADMIN> getPrimaryMgmtNode status = success data = pcamn01 PCA-ADMIN> updatePrimaryNode node=pcamn02 Data: status = success message = Successfully issued update primary node command PCA-ADMIN> getPrimaryMgmtNode status = success data = pcamn02
Using the Service CLI
-
Get the IP address of the ILOM for which you intend to upgrade the firmware.
-
Enter the upgrade command.
Syntax (entered on a single line):
upgradeIlom hostIp=<ilom-ip>Example:
PCA-ADMIN> upgradeIlom hostIp=100.96.0.66 Status: Success Data: Service request has been submitted. Upgrade Job Id = 1620921089806-ilom-21480 Upgrade Request Id = UWS-732d6fce-9f06-4329-b972-d093bee40010
-
Use the request ID and the job ID to check the status of the upgrade process.
PCA-ADMIN> getUpgradeJobs id upgradeRequestId commandName result -- ---------------- ----------- ------ 1620921089806-ilom-21480 UWS-732d6fce-9f06-4329-b972-d093bee40010 ilom Passed PCA-ADMIN> getUpgradeJob upgradeJobId=1620921089806-ilom-21480 Data: Upgrade Request Id = UWS-732d6fce-9f06-4329-b972-d093bee40010 Name = ilom Pid = 21480 Host = pcamn02 Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_ilom_firmware_2021_09_24-11.18.31.log [...]
Using the Service Web UI
-
In the navigation menu, go to the Maintenance section and click Upgrade Plan. This provides an overview of current and target component versions.
-
Click Upgrade & Patching to display the Upgrade Jobs page.
-
In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.
The Create Request window appears. Choose Upgrade as the Request Type.
-
Select the appropriate upgrade request type: Upgrade ILOM.
-
Fill out the upgrade request parameters:
-
Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.
-
Host IP: Enter the component's assigned IP address in the ILOM network. This is an IP address in the internal 100.96.0.0/23 range.
-
Log Level: Optionally, select a specific log level for the upgrade log file. The default log level is "Information". For maximum detail, select "Debug".
-
Alternative ULN Channel: This parameter applies to patching and can be ignored.
-
Verify Only: Enable this option to run the operation in verification only mode.
-
Force: Enable this option to force the operation. Use only when instructed by Oracle.
-
-
Click Create Request.
The new upgrade request appears in the Upgrade Jobs table.
At the end of the upgrade, the ILOM itself is rebooted automatically. However, the server component also needs to be rebooted for all changes to take effect. Wait 5 minutes to allow the ILOM workflow to complete first.
For minimum operational impact, schedule the compute node and management node reboot operations after all ILOMs have been patched. Take into account that rebooting the compute nodes requires migrating the compute instances. For more information, refer to "Performing Compute Node Operations" in the Hardware Administration chapter of the Oracle Private Cloud Appliance Administrator Guide.
Caution:
Always verify the cluster state before rebooting a management node. Consult the Oracle Private Cloud Appliance Release Notes for more information: refer to the known issue "Rebooting a Management Node while the Cluster State is Unhealthy Causes Platform Integrity Issues".
Upgrade Switch Software
The appliance rack contains three categories of Cisco Nexus switches: a management switch, two leaf switches, and two spine switches. They all run the same Cisco NX-OS network operating software. You must perform the upgrades in this order: leaf switches first, then spine switches, and finally the management switch. Only one command per switch category is required, meaning that the leaf switches and the spine switches are upgraded in pairs.
Some versions of the network operating software consist of two files: a binary file and an additional EPLD (electronic programmable logic device) image. Both are automatically retrieved from their designated location during the upgrade process, and applied in the correct order.
Caution:
Verify that all required switch ports are operational. If ports are down, an outage will occur during switch upgrade/patching. For more information, see known issue Switch Upgrade or Patch Procedure Not Blocked When Ports Are Down in the "Oracle Private Cloud Appliance Release Notes".
Using the Service CLI
-
Determine the type of switch to upgrade (spine, leaf, management).
-
Enter the upgrade command.
Syntax (entered on a single line):
upgradeSwitch switchType=[MGMT | SPINE | LEAF]
Example:
PCA-ADMIN> upgradeSwitch switchType=LEAF Data: Service request has been submitted. Upgrade Job Id = 1630511206512-cisco-20299 Upgrade Request Id = UWS-44688fe5-b4f8-407f-a1b5-8cd1b685c2c3
-
Use the request ID and the job ID to check the status of the upgrade process.
PCA-ADMIN> getUpgradeJobs id upgradeRequestId commandName result -- ---------------- ----------- ------ 1630511206512-cisco-20299 UWS-44688fe5-b4f8-407f-a1b5-8cd1b685c2c3 cisco Passed PCA-ADMIN> getupgradeJob upgradeJobId=1630511206512-cisco-20299 Data: Upgrade Request Id = UWS-44688fe5-b4f8-407f-a1b5-8cd1b685c2c3 Name = cisco Start Time = 2021-09-24T14:46:46 End Time = 2021-09-24T14:59:44 Pid = 20299 Host = pcamn02 Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_cisco_firmware_2021_09_24-14.46.46.log [...]
Using the Service Web UI
-
In the navigation menu, go to the Maintenance section and click Upgrade Plan. This provides an overview of current and target component versions.
-
Click Upgrade & Patching to display the Upgrade Jobs page.
-
In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.
The Create Request window appears. Choose Upgrade as the Request Type.
-
Select the appropriate upgrade request type: Upgrade Switch.
-
Fill out the upgrade request parameters:
-
Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.
-
Log Level: Optionally, select a specific log level for the upgrade log file. The default log level is "Information". For maximum detail, select "Debug".
-
Switch Type: Select the switch type you intend to upgrade. The preferred upgrade order is as follows: leaf switches first, then spine switches, and finally the management switch.
-
Alternative ULN Channel: This parameter applies to patching and can be ignored.
-
Verify Only: Enable this option to run the operation in verification only mode.
-
Force: Enable this option to force the operation. Use only when instructed by Oracle.
-
-
Click Create Request.
The new upgrade request appears in the Upgrade Jobs table.
-
When the upgrade has completed successfully, but other switches in the system still need to be upgraded, repeat this procedure for any other type of switch that requires upgrading.