4 Performing a Full Rack Upgrade
Caution:
Ensure that all preparation steps for system upgrade have been completed. For instructions, see Prepare for Appliance Upgrade.
A full appliance rack upgrade is the recommended way to upgrade all components of the Private Cloud Appliance. The tools required for this approach are installed along with the latest Upgrader during the upgrade preparation stages. A single administrator command initiates a fully orchestrated workflow. The Upgrader Workflow Service (UWS) autonomously performs all tasks defined in the upgrade plan, which is generated during the pre-upgrade process. It determines which specific components need to be upgraded, and in what order.
The rack upgrade workflow includes these components:
-
ZFS Storage Appliance firmware (including ILOMs of storage controllers)
-
Compute nodes
-
Management nodes
-
Host operating system of the 3 nodes
-
MySQL cluster database
-
Secret service (including Etcd and Vault)
-
Kubernetes container orchestration packages (platform layer)
-
Containerized microservices
-
-
Oracle Cloud Infrastructure images
-
ILOM firmware (all nodes, includes internal hardware components)
-
Switch firmware (all switches)
The order shown applies to software version 3.0.2-b1392231. Components could be handled in a different order depending on the specific release. Consult the upgrade plan for the correct order.
Components are skipped if they are already at the required version, but a same-version upgrade can be forced using the Service Web UI or Service CLI command option, if necessary. In case a component must be upgraded separately from the full rack upgrade workflow, see Upgrading Individual Components.
Although the full rack upgrade workflow is launched with one convenient command, the internal orchestration of many small commands requires intricate logic and precise timing. To become familiar with the key operations in the workflow, and their impact on system operation, we recommend that administrators read and understand the following sections:
Using the Service Web UI
-
In the navigation menu, go to the Maintenance section and click Upgrade Plan. This provides an overview of current and target component versions.
-
Perform a rack-wide health check to ensure the appliance is ready to accept a new software version. See Rack-Wide Health Check.
Caution:
Oracle strongly recommends verifying in advance that the appliance is ready for upgrade. The preferred way is to perform a rack-wide health check. The alternate option is to run the upgrade command in verify-only mode. It is allowed to do both.
-
Click Upgrade & Patching to display the Upgrade Jobs page.
-
In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.
The Create Request window appears. Choose Upgrade as the Request Type.
-
Select the appropriate upgrade request type.
For a full rack upgrade, select Upgrade Rack.
-
Fill out the upgrade request parameters:
-
Action: To perform the upgrade or patch operation, select Apply.
-
Type: For upgrade, select ISO. The ULN option applies to patching.
-
Component: For full rack upgrade, select All.
-
Qualifier: This parameter is not required when upgrading all components.
-
Verify Only: Enable this option to run the operation in verification only mode.
-
Force: Enable this option to force the operation. Use only when instructed by Oracle.
-
-
Click Create Request.
All jobs related to the new upgrade request appear in the Upgrade Jobs table. Click the upgrade request ID to display a filtered job list. Click a job name to see the details of a job.
Note:
After upgrade, if the upgrade plan specifies that a component must be rebooted for the changes to take effect, a reboot is performed as part of the upgrade process. No administrator action is required.
-
Confirm that the upgrade workflow is completed without errors.
Consult the Upgrade Jobs table. Click the upgrade request ID to display a filtered job list. Click a job name to see the details of a job. For more information, see Browse the Job Framework.
After successful completion, all compute nodes, the entire management node cluster, and all platform and infrastructure services are on Oracle Linux 8. Any required firmware updates have been applied. The system is fully operational again.
Using the Service CLI
-
Perform a rack-wide health check to ensure the appliance is ready to accept a new software version. See Rack-Wide Health Check.
PCA-ADMIN> upgradeRack action=HEALTHCHECK JobId: de9eca17-e357-42c4-8aab-c73c311f787f Data: Service request has been submitted. Upgrade Request Id = UWS-ab2e72d9-d937-4afb-b6eb-112c87988f56
Caution:
Oracle strongly recommends verifying in advance that the appliance is ready for upgrade. The preferred way is to perform a rack-wide health check. The alternate option is to run the upgrade command in verify-only mode. It is allowed to do both.
-
Optionally, as an alternative to the health check, run the upgrade command in verify-only mode. This applies no changes but indicates whether the command will succeed based on the current status of the appliance. Before you start the actual upgrade, all component verification jobs must be completed, which could take around 45 minutes in total.
-
Start the upgrade request in verify-only mode.
PCA-ADMIN> upgradeRack type=ISO action=APPLY verifyOnly=True JobId: 4e99c667-e3df-42c4-4a37-c73c311f49f1 Data: Service request has been submitted. Upgrade Request Id = UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 -
Find the associated upgrade jobs and note the IDs.
PCA-ADMIN> getUpgradeJobs requestId=UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 Data: id Upgrade Request Id Command Name Result -- ------------------ ------------ ------ 1721306505338-cisco_verify-1482721 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 cisco_verify Passed 1721306428955-cisco_verify-1460275 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 cisco_verify Passed 1721306388669-cisco_verify-1447173 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 cisco_verify Passed 1721305430640-ilom_verify-1425717 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 ilom_verify Passed 1721304295877-ilom_verify-1413464 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 ilom_verify Passed 1721303749853-ilom_verify-1264181 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 ilom_verify Passed 1721303241144-ilom_verify-1197515 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 ilom_verify Passed 1721302164633-ilom_verify-1133965 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 ilom_verify Passed 1721301487584-ilom_verify-1019683 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 ilom_verify Passed 1721300618710-oci_verify-971867 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 oci_verify Passed 1721297014027-platform_verify-325471 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 platform_verify Passed 1721296952862-kubernetes_verify-387991 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 kubernetes_verify Passed 1721295894573-host_verify-125736 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 host_verify Passed 1721295631562-host_verify-366123 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 host_verify Passed 1721295558048-host_verify-81361 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 host_verify Passed 1721294395027-compute_verify-76178 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 compute_verify Passed 1721294332372-compute_verify-55394 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 compute_verify Passed 1721290297566-compute_verify-4534 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 compute_verify Passed 1721290264046-zfssa_verify-1752 UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 zfssa_verify Passed
-
Check the status in the upgrade job details.
PCA-ADMIN> getUpgradeJob upgradeJobId=1721295631562-host_verify-366123 Data: Upgrade Request Id = UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056 Name = host_verify Pid = 366123 Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_host_os_pcamn03_verify_2025_01_21-19.38.47.log Status = Passed
-
-
When the upgrade verification has completed successfully, start the actual full rack upgrade.
PCA-ADMIN> upgradeRack type=ISO action=APPLY JobId: de9eca17-e357-42c4-8aab-c73c311f787f Data: Service request has been submitted. Upgrade Request Id = UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718
Note the IDs of the upgrade request and upgrade job.
-
Use the request ID to check the status of the upgrade workflow.
PCA-ADMIN> getUpgradeStatus requestId=UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 Data: Request id = UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 Status = Running Composition ID = fullrack Type = ISO Previous Build = 3.0.2-b1300130 Target Build = 3.0.2-b1300385 Jobs 1 = 1737489464069-zfssa-1898 Jobs 2 = 1737489497566-compute-8759 Jobs 3 = 1737492632323-compute-49553 Jobs 4 = 1737492695074-compute-61798 Jobs 5 = 1737492758087-host-71365 Completed Components 1 = zfssa(generic) Completed Components 2 = compute(100.96.2.64) Completed Components 3 = compute(100.96.2.65) Completed Components 4 = compute(100.96.2.66) Completed Components 5 = kubernetes(generic) Completed Components 6 = switch(leaf) Completed Components 7 = switch(spine) Completed Components 8 = switch(mgmt) Pending Components 1 = host(100.96.2.33) Pending Components 2 = host(100.96.2.34) Pending Components 3 = host(100.96.2.35) Pending Components 4 = platform(generic) Pending Components 5 = ociImages(Oracle-Linux-7.9) Pending Components 6 = ociImages(Oracle-Linux8-OKE-1.28.8) Pending Components 7 = ociImages(Oracle-Solaris-11) Pending Components 8 = ociImages(Oracle-Linux8-OKE-1.27.12) Pending Components 9 = ociImages(Oracle-Linux8-OKE-1.26.15) Pending Components 10 = ociImages(Oracle-Linux-9) Pending Components 11 = ociImages(Oracle-Linux-8) Pending Components 12 = ilom(100.96.0.33) Pending Components 13 = ilom(100.96.0.34) Pending Components 14 = ilom(100.96.0.35) Pending Components 15 = ilom(100.96.0.64) Pending Components 16 = ilom(100.96.0.65) Pending Components 17 = ilom(100.96.0.66)
-
Optionally, check the status of an individual upgrade job.
Because the full rack upgrade is a multi-component upgrade process, there are multiple upgrade jobs associated with the upgrade request. You can filter for those jobs based on the request ID. Using the job ID, you can drill down into the details of each upgrade job.
PCA-ADMIN> getUpgradeJobs requestId=UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 Data: id Upgrade Request Id Command Name Result -- ------------------ ------------ ------ 1737496294531-host-396629 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 host Passed 1737494531521-host-162838 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 host Passed 1737492758087-host-71365 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 host Passed 1737492695074-compute-61798 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 compute Passed 1737492632323-compute-49553 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 compute Passed 1737489497566-compute-8759 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 compute Passed 1737489464069-zfssa-1898 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 zfssa Passed PCA-ADMIN> getUpgradeJob upgradeJobId=1737494531521-host-162838 Data: Upgrade Request Id = UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 Composition Id = fullrack Name = host Pid = 162838 Host = pcamn01 Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_host_os_pcamn02_2025_01_21-21.22.11.log Arguments = {"component_names":null,"diagnostics":false,"display_task_plan":false,"dry_run_tasks":false,"expected_iso_checksum":null,"fail_halt":false,"fail_upgrade":null,"image_location":null,"online_upgrade":null,"precheck_status":false,"repo_config_override":null,"result_override":null,"task_time":0,"test_run":false,"upgrade":false,"upgrade_to":null,"user_uln_base_url":null,"verify_only":false,"host_ip":"100.96.2.34","log_level":null,"switch_type":null,"epld_image_location":null,"checksum":null,"composition_id":"fullrack","request_id":"UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718","uln":null,"patch":"false"} Status = Passed Execution Time(sec) = 1736 Tasks 1 - Name = Validate Image Location Tasks 2 - Name = Check Registry Service Running Status Tasks 3 - Name = Validate Upgrader Version Tasks 4 - Name = Validate Upgrade Order [...]The output of the
getUpgradeJobcommand provides detailed information about the tasks performed during the upgrade procedure. It displays descriptions, time stamps, duration, and success or failure. Whenever an upgrade operation fails, the command output indicates which task has failed. For in-depth troubleshooting you can search the log file at the location provided near the start of the command output.Note:
After upgrade, if the upgrade plan specifies that a component must be rebooted for the changes to take effect, a reboot is performed as part of the upgrade process. No administrator action is required.
-
Confirm that the upgrade workflow is completed without errors.
For more information, see Browse the Job Framework.
Note:
In case you forgot the ID of the upgrade request or upgrade job, you can search using the commands
getUpgradeRequestsandgetUpgradeJobs.Check the job associated with the
upgradeRackcommand.PCA-ADMIN> show job id=de9eca17-e357-42c4-8aab-c73c311f787f Data: Id = de9eca17-e357-42c4-8aab-c73c311f787f Type = Job Associated Work Request Id = UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 Done = true Name = OPERATION-upgradeRack Progress Message = Done Run State = Succeeded Transcript = Created job OPERATION Username = admin WorkItemIds 1 = id:4fd5d71b-81b3-4a37-a0df-c4149e9c6609 type:WorkItem name:
Use the request ID to check the status of the upgrade workflow.
PCA-ADMIN> getUpgradeStatus requestId=UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 Data: Request id = UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 Status = Passed Composition ID = fullrack Type = ISO Previous Build = 3.0.2-b1300385 Target Build = 3.0.2-b1301270 Jobs 1 = 1737489464069-zfssa-1898 Jobs 2 = 1737489497566-compute-8759 Jobs 3 = 1737492632323-compute-49553 Jobs 4 = 1737492695074-compute-61798 Jobs 5 = 1737492758087-host-71365 Jobs 6 = 1737494531521-host-162838 Jobs 7 = 1737496294531-host-396629 Jobs 8 = 1737497752826-kubernetes-637866 Jobs 9 = 1737497814071-platform-646623 Jobs 10 = 1737500431705-oci-979395 Jobs 11 = 1737501529548-ilom-1060489 Jobs 12 = 1737502338638-ilom-1133349 Jobs 13 = 1737503158143-ilom-1197515 Jobs 14 = 1737503974834-ilom-1266068 Jobs 15 = 1737504796873-ilom-1361643 Jobs 16 = 1737505608602-ilom-1435802 Jobs 17 = 1737506421692-cisco-1507991 Jobs 18 = 1737506482956-cisco-1524202 Jobs 19 = 1737506544387-cisco-1532342 Completed Components 1 = zfssa(generic) Completed Components 2 = compute(100.96.2.64) Completed Components 3 = kubernetes(generic) Completed Components 4 = ociImages(Oracle-Linux-8) Completed Components 5 = ociImages(Oracle-Linux-7.9) Completed Components 6 = ociImages(Oracle-Linux8-OKE-1.27.12) Completed Components 7 = ociImages(Oracle-Linux8-OKE-1.28.8) Completed Components 8 = ociImages(Oracle-Linux-9) Completed Components 9 = ociImages(Oracle-Linux8-OKE-1.26.15) Completed Components 10 = ociImages(Oracle-Solaris-11) Completed Components 11 = ilom(100.96.0.33) Completed Components 12 = ilom(100.96.0.34) Completed Components 13 = ilom(100.96.0.35) Completed Components 14 = ilom(100.96.0.64) Completed Components 15 = ilom(100.96.0.65) Completed Components 16 = ilom(100.96.0.66) Completed Components 17 = switch(leaf) Completed Components 18 = switch(spine) Completed Components 19 = switch(mgmt) Start Time = 2025-01-21T19:57:44 End Time = 2025-01-22T00:42:25
Check the upgrade jobs associated with the upgrade request.
PCA-ADMIN> getUpgradeJobs requestId=UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 Data: id Upgrade Request Id Command Name Result -- ------------------ ------------ ------ 1737506544387-cisco-1532342 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 cisco Passed 1737506482956-cisco-1524202 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 cisco Passed 1737506421692-cisco-1507991 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 cisco Passed 1737505608602-ilom-1435802 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 ilom Passed 1737504796873-ilom-1361643 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 ilom Passed 1737503974834-ilom-1266068 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 ilom Passed 1737503158143-ilom-1197515 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 ilom Passed 1737502338638-ilom-1133349 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 ilom Passed 1737501529548-ilom-1060489 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 ilom Passed 1737500431705-oci-979395 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 oci Passed 1737497814071-platform-646623 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 platform Passed 1737497752826-kubernetes-637866 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 kubernetes Passed 1737496294531-host-396629 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 host Passed 1737494531521-host-162838 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 host Passed 1737492758087-host-71365 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 host Passed 1737492695074-compute-61798 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 compute Passed 1737492632323-compute-49553 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 compute Passed 1737489497566-compute-8759 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 compute Passed 1737489464069-zfssa-1898 UWS-8cbb37e3-1d9b-43bc-a793-2a84633fa718 zfssa Passed
After successful completion, all compute nodes, the entire management node cluster, and all platform and infrastructure services are on Oracle Linux 8. Any required firmware updates have been applied. The system is fully operational again.