Performing a Full Management Node Upgrade

6 Performing a Full Management Node Upgrade

Caution:

Ensure that all preparation steps for system upgrade have been completed. For instructions, see Preparing the Upgrade Environment.

A full management node upgrade is a convenient way to upgrade all the required components on all three management nodes using just a single command. As part of this process, the following components are upgraded, in this specific order:

host operating system
Clustered MySQL database
secret service (including Etcd and Vault)
Kubernetes container orchestration packages
containerized microservices
Oracle Cloud Infrastructure images

Note:

To obtain the latest images on systems with software versions 3.0.2-b852928 and earlier, you must manually perform the steps described in Upgrading Oracle Cloud Infrastructure Images.

Note:

In software version 3.0.2-b892153 and later the Upgrader service uses the upgrade plan, generated during the pre-upgrade process, to determine which specific components need to be upgraded. If a component is already at the required version, it is skipped. A same-version upgrade can be forced using the Service Web UI or Service CLI command option, if necessary. For example: upgradeKubernetes force=True.

Using the Service Web UI

In the navigation menu, click Upgrade & Patching.
In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.

The Create Request window appears. Choose Upgrade as the Request Type.
Select the appropriate upgrade request type.

For a full management node upgrade, select Upgrade MN.

Caution:

Oracle strongly recommends using the verify-only option on the first run. This starts a verification job for each component. Before you start the actual upgrade, all verification jobs must be completed, which could take around 45 minutes in total.

Note:

This function is available when the appliance is running software version 3.0.2-b892153 or later.
Fill out the upgrade request parameters:
- Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.
- Image Location: This parameter is deprecated.
- ISO Checksum: This parameter is deprecated.
Click Create Request.

The new upgrade request appears in the Upgrade Jobs table.

Note:

After upgrade, if the upgrade plan specifies that the management nodes must be rebooted for the changes to take effect, a reboot is performed as part of the upgrade process. No administrator action is required.

If the appliance is running software version 3.0.2-b892153 or earlier, all management nodes are rebooted as part of the upgrade process.

Using the Service CLI

Run the upgrade command in verify-only mode.

Caution:

Oracle strongly recommends using the verify-only option (verifyOnly=True) on the first run. This starts a verification job for each component. Before you start the actual upgrade, all verification jobs must be completed, which could take around 45 minutes in total.

Note:

This function is available when the appliance is running software version 3.0.2-b892153 or later.

Start the upgrade request in verify-only mode.

PCA-ADMIN> upgradeFullMN verifyOnly=True
Data: Service request has been submitted. Upgrade Request Id = UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056

Find the associated upgrade jobs and note the IDs.

PCA-ADMIN> getUpgradeJobs requestId=UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056
Data:
  id                                      upgradeRequestId                           commandName         result
  --                                      ----------------                           -----------         ------
  1698788901590-oci_verify-38799          UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056   oci_verify          Passed
  1698788685720-platform_verify-34138     UWS-20f69d05-9c43-4617-80a4-8cf8db5b8446   platform_verify     Passed
  1698788568097-kubernetes_verify-32571   UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056   kubernetes_verify   Passed
  1698788265717-vault_verify-27153        UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056   vault_verify        Passed
  1698787959640-etcd_verify-26413         UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056   etcd_verify         Passed
  1698787657287-mysql_verify-19656        UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056   mysql_verify        Passed
  1698787353293-host_verify-18867         UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056   host_verify         Passed
  1698787049308-host_verify-12573         UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056   host_verify         Passed
  1698786925154-host_verify-36612         UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056   host_verify         Passed

Check the status in the upgrade job details.

PCA-ADMIN> getUpgradeJob upgradeJobId=1698787353293-host_verify-18867
Data:
  Upgrade Request Id = UWS-9776b0fe-7f5f-4e46-9e3f-ceb4b1702056
  Name = host_verify
  Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_host_os_pcamn03_verify_2023_06_16-13.07.11.log
  Status = Passed

When the upgrade verification has completed successfully, start the actual full management node upgrade.

PCA-ADMIN> upgradeFullMN
Data:
  Service request has been submitted. Upgrade Request Id = UWS-39329657-1051-4267-8c5a-9314f8e63a64

Use the request ID to check the status of the upgrade process.

As the full management node upgrade is a multi-component upgrade process, there are multiple upgrade jobs associated with the upgrade request. You can filter for those jobs based on the request ID. Using the job ID, you can drill down into the details of each upgrade job.

PCA-ADMIN> getUpgradeJobs requestId=UWS-39329657-1051-4267-8c5a-9314f8e63a64
Data:
  id                               upgradeRequestId                           commandName   result
  --                               ----------------                           -----------   ------
  1634579161548-oci-47486          UWS-39329657-1051-4267-8c5a-9314f8e63a64   oci           Passed
  1634578760906-platform-66082     UWS-39329657-1051-4267-8c5a-9314f8e63a64   platform      Passed
  1634578263434-kubernetes-63574   UWS-39329657-1051-4267-8c5a-9314f8e63a64   kubernetes    Passed
  1634578012353-vault-51696        UWS-39329657-1051-4267-8c5a-9314f8e63a64   vault         Passed
  1634577380954-etcd-46337         UWS-39329657-1051-4267-8c5a-9314f8e63a64   etcd          Passed
  1634577341291-mysql-40127        UWS-39329657-1051-4267-8c5a-9314f8e63a64   mysql         Passed
  1634576985926-host-36556         UWS-39329657-1051-4267-8c5a-9314f8e63a64   host          Passed
  1634576652071-host-27088         UWS-39329657-1051-4267-8c5a-9314f8e63a64   host          Passed
  1634576191050-host-24909         UWS-39329657-1051-4267-8c5a-9314f8e63a64   host          Passed

PCA-ADMIN> getUpgradeJob upgradeJobId=1634576652071-host-27088
Data:
  Upgrade Request Id = UWS-39329657-1051-4267-8c5a-9314f8e63a64
  Composition Id = 1
  Name = host
  Start Time = 2023-05-24T07:04:12
  End Time = 2023-05-24T07:05:22
  Pid = 27088
  Host = pcamn02
  Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_host_os_2023_05_24-07.04.12.log
  Arguments = {"verify_only":false,"upgrade":false,"diagnostics":false,"host_ip":"100.96.2.35","result_override":null,"log_level":null,"switch_type":null,"precheck_status":false,"task_time":0,"fail_halt":false,"fail_upgrade":null,"component_names":null,"upgrade_to":null,"image_location":"file:///nfs/shared_storage/pca-3.0.1-b544818.iso","epld_image_location":null,"expected_iso_checksum":null,"checksum":"240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7","composition_id":"1","request_id":"UWS-39329657-1051-4267-8c5a-9314f8e63a64","display_task_plan":false,"dry_run_tasks":false}
  Status = Passed
  Execution Time(sec) = 139
  Tasks 1 - Name = Validate Image Location
  Tasks 1 - Description = Verify that the image exists at the specified location and is correctly named
  Tasks 1 - Time = 2023-05-24T17:04:16
[...]

The output of the getUpgradeJob command provides detailed information about the tasks performed during the upgrade procedure. It displays descriptions, time stamps, duration, and success or failure. Whenever an upgrade operation fails, the command output indicates which task has failed. For in-depth troubleshooting you can search the log file at the location provided near the start of the command output.

Note:

After upgrade, if the upgrade plan specifies that the management nodes must be rebooted for the changes to take effect, a reboot is performed as part of the upgrade process. No administrator action is required.

If the appliance is running software version 3.0.2-b892153 or earlier, all management nodes are rebooted as part of the upgrade process.