1. Administrator Guide for Release 3.0.1
  2. System Upgrade

7 System Upgrade

This chapter explains how an administrator upgrades the Oracle Private Cloud Appliance or one of its components.

Do not install or upgrade individual packages on the appliance components. Only upgrades as described in this chapter are supported. Security and other updates are provided through patches. Patching is separate from the upgrade functionality and uses a ULN mirror to download supported packages to the management nodes' shared storage.

Implementation details and technical background information for the upgrade and patching functionality can be found in the Oracle Private Cloud Appliance Concepts Guide. Refer to the sections "Upgrade" and "Patching" in the chapter Appliance Administration Overview.

Patching instructions are provided in a separate document. Refer to the Oracle Private Cloud Appliance Patching Guide.

Upgrade Requirements

Before you start an upgrade procedure, make sure that you have the required permissions and have downloaded the ISO image to a suitable location.

Verifying Permissions

To be able to execute an upgrade, you must have an administrator account to log in to the Service Enclave. You must be a member of one of these authorization groups: SuperAdmin, Admin, or DR Admin. For more information, see Administrator Account Management.

When you log in to the Service CLI, you can verify that the upgrade commands are available to you by displaying all custom commands. The list of commands is filtered based on your access profile. If the upgrade commands are listed, it means you have permission to execute them. 

PCA-ADMIN> showallcustomcmds
    Operation Name: <Related Object(s)>
    -----------------------------------
    [...]
    getUpgradeJob:  UpgradeJob
    getUpgradeJobs:  UpgradeJobList
    getUpgradeRequests:  UpgradeRequest
    killUpgradeJob:  UpgradeJob
    [...]
    upgradeCN:  UpgradeRequest
    upgradeEtcd:  UpgradeRequest
    upgradeFullMN:  UpgradeRequest
    upgradeHost:  UpgradeRequest
    upgradeIlom:  UpgradeRequest
    upgradeKubernetes:  UpgradeRequest
    upgradeMySQL:  UpgradeRequest
    upgradePlatform:  UpgradeRequest
    upgradeSwitch:  UpgradeRequest
    upgradeVault:  UpgradeRequest
    upgradeZfssa:  UpgradeRequest

Preparing the ISO Image

Software versions and upgrades for Oracle Private Cloud Appliance are made available for download through My Oracle Support. The ISO file contains all the files and packages required to upgrade the appliance hardware and software components to a given release. All the items within the ISO file have been tested to work with each other and qualified for installation on your rack system.

To be able to use an ISO file to upgrade your appliance, you only need to download the file to a location from where a web server can make it available to the Private Cloud Appliance management nodes. If you have set up a bastion host connected to the internal administration network of the appliance, it is convenient to store the ISO file on that machine and run a web server to make the ISO file accessible over http.

When you execute an upgrade command on the appliance, you provide the path to the ISO file as a parameter. At that point, the ISO file is copied to the shared storage mounted on all three management nodes, and unpacked into a well-defined directory structure. You do not need to perform these steps manually in advance.

Ensuring the System Is In Ready State

Upgrades can be performed with limited impact on the system. No downtime is required, and user workloads continue to run while the underlying infrastructure is being upgraded in stages. However, it is considered good practice to ensure that backups are created of the system and the resources in your environment.

Every upgrade operation is preceded by a set of pre-checks. The upgrade will only begin if all pre-checks are passed. You are not required to execute the pre-checks manually; they are built into the upgrade code and will report an error if the system is not in the required state for the upgrade.

It is important to note that concurrent upgrade operations are not supported. An upgrade job must be completed before a new one can be started.

Upgrading a Compute Node

The compute node upgrade is similar to the management node host operating system upgrade: it ensures that the latest Oracle Linux kernel and user space packages are installed, as well as the ovm-agent package with appliance-specific optimizations. Compute nodes must be locked and upgraded one at a time; concurrent upgrades are not supported. After successful upgrade, when a compute node has rebooted, the administrator must manually remove the locks to allow the node to return to normal operation.

To obtain the host IP addresses of a compute node, use the Service CLI command show ComputeNode name=<node_name> and look for the Ip Address in the output.

Using the Service Web UI

  1. Set the provisioning and maintenance locks for the compute node you are about to upgrade.

    For more information, refer to Performing Compute Node Operations.

    1. In the navigation menu, click Rack Units. In the Rack Units table, click the name of the compute node you want to upgrade to display its detail page.

    2. In the top-right corner of the compute node detail page, click Controls and select the Provisioning Lock command.

    3. When the provisioning lock is set, click Controls again and select the Maintenance Lock command.

  2. In the navigation menu, click Upgrade & Patching.

  3. In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.

    The Create Request window appears. Choose Upgrade as the Request Type.

  4. Select the appropriate upgrade request type: Upgrade CN.

  5. Fill out the upgrade request parameters:

    • Host IP: Enter the compute node's assigned IP address in the internal administration network. This is an IP address in the internal 100.96.2.0/23 range.

    • ISO Checksum: If you have not executed a previous upgrade command specifying this parameter, enter the checksum that allows the system to verify that the ISO image is valid for this upgrade. The checksum is provided alongside the ISO image; its file name is the ISO image name with .sha512sum appended.

    • Image Location: If you have not executed a previous upgrade command specifying this parameter, enter the path to the location where the ISO image is stored.

    • Log Level: Optionally, select a specific log level for the upgrade log file. The default log level is "Information". For maximum detail, select "Debug".

    • Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.

  6. Click Create Request.

    The new upgrade request appears in the Upgrade Jobs table.

  7. When the compute node has been upgraded successfully, release the provisioning and maintenance locks.

    For more information, refer to Performing Compute Node Operations.

    1. Open the compute node detail page.

    2. In the top-right corner of the compute node detail page, click Controls and select the Maintenance Unlock command.

    3. When the maintenance lock has been released, click Controls again and select the Provisioning Unlock command.

Using the Service CLI

  1. Gather the information that you need to run the command:

    • the location of the ISO image to upgrade from

    • the checksum used to verify that the ISO image is valid

    • the IP address of the compute node you intend to upgrade

  2. Set the provisioning and maintenance locks for the compute node you are about to upgrade.

    For more information, refer to Performing Compute Node Operations. 

    PCA-ADMIN> list ComputeNode
Data:
  id                                     name       provisioningState   provisioningType
  --                                     ----       -----------------   ----------------
  363a26f4-fa34-4e4c-8e17-a1671a0b77d1   pcacn001   Provisioned         KVM
  9e8745c7-52e3-4aae-984c-e198869ee2cc   pcacn002   Provisioned         KVM
  56a9ecda-2402-427f-92d1-7f9be57dba36   pcacn003   Provisioned         KVM

PCA-ADMIN> provisioningLock id=363a26f4-fa34-4e4c-8e17-a1671a0b77d1
PCA-ADMIN> maintenanceLock id=363a26f4-fa34-4e4c-8e17-a1671a0b77d1

  3. Enter the upgrade command.

    Syntax (entered on a single line):

    upgradeCN 
hostIp=<compute-node-ip>
imageLocation=<path-to-iso>
isoChecksum=<iso-file-checksum>

    Example:

    PCA-ADMIN> upgradeCN hostIp=100.96.2.64 \
imageLocation="http://host.example.com/pca-<version>-<build>.iso" \
isoChecksum=240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7
Command: upgradeCN hostIp=100.96.2.64 imageLocation="http://host.example.com/pca-<version>-<build>.iso" isoChecksum=240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7
Status: Success
Time: 2021-09-26 06:35:38,884 UTC
Data:
  Service request has been submitted. Upgrade Job Id = 1630938939109-compute-7545 Upgrade Request Id = UWS-61736806-7e5a-4648-9259-07c54c39cacb

  4. Use the request ID and the job ID to check the status of the upgrade process.

    PCA-ADMIN> getUpgradeJobs
  id                               upgradeRequestId                           commandName   result
  --                               ----------------                           -----------   ------
  1630938939109-compute-7545       UWS-61736806-7e5a-4648-9259-07c54c39cacb   compute       Passed
  1632850650836-platform-68465     UWS-26dba234-9b52-426d-836c-ac11f37e717f   platform      Passed
  1632849609034-kubernetes-35545   UWS-edfa3b32-c32a-4b67-8df5-2357096052bf   kubernetes    Passed

PCA-ADMIN> getupgradejob upgradeJobId=1630938939109-compute-7545
Command: getupgradejob upgradeJobId=1630938939109-compute-7545
Status: Success
Time: 2021-09-26 08:15:03,208 UTC
Data:
  Upgrade Request Id = UWS-61736806-7e5a-4648-9259-07c54c39cacb
  Name = compute
  Start Time = 2021-09-26T06:35:39
  End Time = 2021-09-26T06:45:55
  Pid = 7545
  Host = pcamn02
  Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_compute_2021_09_26-06.35.39.log
  Arguments = {"verify_only":false,"upgrade":false,"diagnostics":false,"host_ip":"100.96.2.64","result_override":null,"log_level":null,"switch_type":null,"precheck_status":false,"task_time":0,"fail_halt":false,"fail_upgrade":null,"component_names":null,"upgrade_to":null,"image_location":null,"epld_image_location":null,"expected_iso_checksum":null,"checksum":null,"composition_id":null,"request_id":"UWS-61736806-7e5a-4648-9259-07c54c39cacb","display_task_plan":false,"dry_run_tasks":false}
  Status = Passed
  Execution Time(sec) = 616
  Tasks 1 - Name = Copy Scripts
  Tasks 1 - Description = Copy scripts to shared storage
  Tasks 1 - Time = 2021-09-26T06:35:39
[...]

  5. When the compute node upgrade has completed successfully and the node has rebooted, release the locks.

    For more information, refer to Performing Compute Node Operations. 

    PCA-ADMIN> maintenanceUnlock id=363a26f4-fa34-4e4c-8e17-a1671a0b77d1
PCA-ADMIN> provisioningUnlock id=363a26f4-fa34-4e4c-8e17-a1671a0b77d1

  6. Proceed to the next compute node and repeat this procedure.

Performing a Full Management Node Upgrade

A full management node upgrade is a convenient way to upgrade all the required components on all three management nodes using just a single command. As part of this process, the following components are upgraded, in this specific order:

  1. the host operating system

  2. the MySQL cluster database

  3. the secret service (including Etcd and Vault)

  4. the Kubernetes container orchestration packages

  5. the containerized microservices

Using the Service Web UI

  1. In the navigation menu, click Upgrade & Patching.

  2. In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.

    The Create Request window appears. Choose Upgrade as the Request Type.

  3. Select the appropriate upgrade request type.

    For a full management node upgrade, select Upgrade MN.

  4. Fill out the upgrade request parameters:

    • Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.

    • ISO Checksum: Enter the checksum that allows the system to verify that the ISO image is valid for this upgrade. The checksum is provided alongside the ISO image; its file name is the ISO image name with .sha512sum appended.

    • Image Location: Enter the path to the location where the ISO image is stored.

  5. Click Create Request.

    The new upgrade request appears in the Upgrade Jobs table.

    Caution:

    When the upgrade job has completed successfully, the management nodes must be rebooted for the changes to take effect. This cannot be done from the Service Web UI.

  6. Reboot all three management nodes either from the Oracle Linux command line or through the ILOM. Refer to the final step of the management cluster upgrade instructions using the Service CLI.

Using the Service CLI

  1. Gather the information that you need to run the command:

    • the location of the ISO image to upgrade from

    • the checksum used to verify that the ISO image is valid

  2. Enter the upgrade command.

    Syntax (entered on a single line):

    upgradeFullMN 
imageLocation=<path-to-iso>
isoChecksum=<iso-file-checksum>

    Example:

    PCA-ADMIN> upgradeFullMN \
imageLocation="http://host.example.com/pca-<version>-<build>.iso" \
isoChecksum=240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7
Command: upgradeFullMN imageLocation="http://host.example.com/pca-<version>-<build>.iso" isoChecksum=240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7
Status: Success
Time: 2021-09-24 06:56:31,871 UTC
Data:
  Service request has been submitted. Upgrade Request Id = UWS-39329657-1051-4267-8c5a-9314f8e63a64

  3. Use the request ID to check the status of the upgrade process.

    As the full management node upgrade is a multi-component upgrade process, there are multiple upgrade jobs associated with the upgrade request. You can filter for those jobs based on the request ID. Using the job ID, you can drill down into the details of each upgrade job.

    PCA-ADMIN> getUpgradeJobs requestId=UWS-39329657-1051-4267-8c5a-9314f8e63a64
Command: getUpgradeJobs requestId=UWS-39329657-1051-4267-8c5a-9314f8e63a64
Status: Success
Time: 2021-09-24 17:32:31,595 UTC
Data:
  id                               upgradeRequestId                           commandName   result
  --                               ----------------                           -----------   ------
  1634578760906-platform-66082     UWS-39329657-1051-4267-8c5a-9314f8e63a64   platform      Passed
  1634578263434-kubernetes-63574   UWS-39329657-1051-4267-8c5a-9314f8e63a64   kubernetes    Passed
  1634578012353-vault-51696        UWS-39329657-1051-4267-8c5a-9314f8e63a64   vault         Passed
  1634577380954-etcd-46337         UWS-39329657-1051-4267-8c5a-9314f8e63a64   etcd          Passed
  1634577341291-mysql-40127        UWS-39329657-1051-4267-8c5a-9314f8e63a64   mysql         Passed
  1634576985926-host-36556         UWS-39329657-1051-4267-8c5a-9314f8e63a64   host          Passed
  1634576652071-host-27088         UWS-39329657-1051-4267-8c5a-9314f8e63a64   host          Passed
  1634576191050-host-24909         UWS-39329657-1051-4267-8c5a-9314f8e63a64   host          Passed

PCA-ADMIN> getUpgradeJob upgradeJobId=1634576652071-host-27088
Command: getUpgradeJob upgradeJobId=1634576652071-host-27088
Status: Success
Time: 2021-09-24 17:35:59,946 UTC
Data:
  Upgrade Request Id = UWS-39329657-1051-4267-8c5a-9314f8e63a64
  Composition Id = 1
  Name = host
  Start Time = 2021-09-24T07:04:12
  End Time = 2021-09-24T07:05:22
  Pid = 27088
  Host = pcamn02
  Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_host_os_2021_09_24-07.04.12.log
  Arguments = {"verify_only":false,"upgrade":false,"diagnostics":false,"host_ip":"100.96.2.35","result_override":null,"log_level":null,"switch_type":null,"precheck_status":false,"task_time":0,"fail_halt":false,"fail_upgrade":null,"component_names":null,"upgrade_to":null,"image_location":"file:///nfs/shared_storage/pca-3.0.1-b544818.iso","epld_image_location":null,"expected_iso_checksum":null,"checksum":"240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7","composition_id":"1","request_id":"UWS-39329657-1051-4267-8c5a-9314f8e63a64","display_task_plan":false,"dry_run_tasks":false}
  Status = Passed
  Execution Time(sec) = 139
  Tasks 1 - Name = Validate Image Location
  Tasks 1 - Description = Verify that the image exists at the specified location and is correctly named
  Tasks 1 - Time = 2021-10-18T17:04:16
[...]

    The output of the getUpgradeJob command provides detailed information about the tasks performed during the upgrade procedure. It displays descriptions, time stamps, duration, and success or failure. Whenever an upgrade operation fails, the command output indicates which task has failed. For in-depth troubleshooting you can search the log file at the location provided near the start of the command output.

    Caution:

    When the upgrade job has completed successfully, the management nodes must be rebooted for the changes to take effect. This cannot be done from the Service CLI.

  4. Reboot all three management nodes either from the Oracle Linux command line or through the ILOM.

    1. Verify which management node owns the cluster virtual IP. Run this command from the command line of one of the management nodes:

      # pcs status
[...]
 Resource Group: mgmt-rg
     vip-mgmt-int     (ocf::heartbeat:IPaddr2):      Started    pcamn02
     vip-mgmt-host    (ocf::heartbeat:IPaddr2):      Started    pcamn02
     vip-mgmt-ilom    (ocf::heartbeat:IPaddr2):      Started    pcamn02
     vip-mgmt-lb      (ocf::heartbeat:IPaddr2):      Started    pcamn02
     vip-mgmt-ext     (ocf::heartbeat:IPaddr2):      Started    pcamn02
[...]

    2. Reboot the other two management nodes in the cluster. (In this example: pcamn01 and pcamn03.)

    3. Move the virtual IP to one of the rebooted management nodes. (In this example: pcamn01.) 

      # pcs resource move mgmt-rg pcamn01
# pcs status
Cluster name: mncluster
Stack: corosync
[...]
 scsi_fencing   (stonith:fence_scsi):   Stopped (disabled)
 Resource Group: mgmt-rg
     vip-mgmt-int       (ocf::heartbeat:IPaddr2):       Started pcamn01
     vip-mgmt-host      (ocf::heartbeat:IPaddr2):       Started pcamn01
[...]

    4. Reboot the last of the three upgraded management nodes. (In this example: pcamn02.)

Upgrading Individual Components

The granular upgrade mechanism allows you to perform upgrade procedures for individual hardware and software components. Besides the components included in the management node upgrade, you can also upgrade different categories of firmware, and the operating system and appliance-specific software on the compute nodes.

Upgrading the Management Node Operating System

The Oracle Linux host operating system of the management nodes must be upgraded one node at a time; a rolling upgrade of all management nodes is not possible. This upgrade process, which involves updating the kernel and system packages, must always be initiated from the management node that holds the cluster virtual IP. Thus, in a three-management-node cluster, when you have upgraded two management nodes, you must reassign the cluster virtual IP to one of the upgraded management nodes and execute the final upgrade command from that node.

You must upgrade management nodes one at a time, using each one's internal IP address as a command parameter. To obtain the host IP addresses, use the Service CLI command show ManagementNode name=<node_name> and look for the Ip Address in the output.

Using the Service Web UI

  1. In the navigation menu, click Upgrade & Patching.

  2. In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.

    The Create Request window appears. Choose Upgrade as the Request Type.

  3. Select the appropriate upgrade request type: Upgrade Host.

  4. Fill out the upgrade request parameters:

    • Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.

    • Host IP: Enter the management node's assigned IP address in the internal administration network. This is an IP address in the internal 100.96.2.0/23 range.

    • ISO Checksum: Enter the checksum that allows the system to verify that the ISO image is valid for this upgrade. The checksum is provided alongside the ISO image; its file name is the ISO image name with .sha512sum appended.

    • Image Location: Enter the path to the location where the ISO image is stored.

    • Log Level: Optionally, select a specific log level for the upgrade log file. The default log level is "Information". For maximum detail, select "Debug".

  5. Click Create Request.

    The new upgrade request appears in the Upgrade Jobs table.

    Caution:

    When the upgrade jobs for all three management nodes have completed successfully, the management nodes must be rebooted for the changes to take effect. This cannot be done from the Service Web UI.

  6. Reboot all three management nodes either from the Oracle Linux command line or through the ILOM. Refer to the final step of the management node host OS upgrade instructions using the Service CLI.

Using the Service CLI

  1. Gather the information that you need to run the command:

    • the location of the ISO image to upgrade from

    • the checksum used to verify that the ISO image is valid

    • the IP address of the management node for which you intend to upgrade the host operating system

  2. Run the Service CLI from the management node that holds the management cluster virtual IP.

    1. Log on to one of the management nodes and check the status of the cluster.

      # ssh root@pcamn01
# pcs status
Cluster name: mncluster
Stack: corosync
Current DC: pcamn02 (version 1.1.23-1.0.1.el7-9acf116022) - partition with quorum

Online: [ pcamn01 pcamn02 pcamn03 ]

Full list of resources:

 scsi_fencing         (stonith:fence_scsi):          Stopped (disabled)
 Resource Group: mgmt-rg
     vip-mgmt-int     (ocf::heartbeat:IPaddr2):      Started    pcamn02
     vip-mgmt-host    (ocf::heartbeat:IPaddr2):      Started    pcamn02
     vip-mgmt-ilom    (ocf::heartbeat:IPaddr2):      Started    pcamn02
     vip-mgmt-lb      (ocf::heartbeat:IPaddr2):      Started    pcamn02
     vip-mgmt-ext     (ocf::heartbeat:IPaddr2):      Started    pcamn02
     l1api            (systemd:l1api):               Started    pcamn02
     haproxy          (ocf::heartbeat:haproxy):      Started    pcamn02
     pca-node-state   (systemd:pca_node_state):      Started    pcamn02
     dhcp             (ocf::heartbeat:dhcpd):        Started    pcamn02
     hw-monitor       (systemd:hw_monitor):          Started    pcamn02

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

      In this example, the command output indicates that the node with host name pcamn02 currently holds the cluster virtual IP.

    2. Log in to the management node with the virtual IP and launch the Service CLI. 

      # ssh pcamn02
# ssh admin@localhost -p 30006
PCA-ADMIN>

  3. Enter the upgrade command.

    Syntax (entered on a single line):

    upgradeHost 
imageLocation=<path-to-iso>
isoChecksum=<iso-file-checksum>
hostIp=<management-node-ip>

    Example:

    PCA-ADMIN> upgradeHost hostIp=100.96.2.35 \
imageLocation="http://host.example.com/pca-<version>-<build>.iso" \
isoChecksum=240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7
Command: upgradeHost hostIp=100.96.2.35 imageLocation="http://host.example.com/pca-<version>-<build>.iso" isoChecksum=240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7
Status: Success
Time: 2021-09-25 05:47:02,735 UTC
Data:
  Service request has been submitted. Upgrade Job Id = 1632990827394-host-56156 Upgrade Request Id = UWS-1a97a8d9-54ef-478d-a0c0-348a17ba6755

  4. Use the request ID and the job ID to check the status of the upgrade process.

    PCA-ADMIN> getUpgradeJobs
  id                               upgradeRequestId                           commandName   result
  --                               ----------------                           -----------   ------
  1632990827394-host-56156         UWS-1a97a8d9-54ef-478d-a0c0-348a17ba6755   host          Passed

PCA-ADMIN> getUpgradeJob upgradeJobId=1632990827394-host-56156
Command: getUpgradeJob upgradeJobId=1632990827394-host-56156
Status: Success
Time: 2021-09-25 05:54:28,054 UTC
Data:
  Upgrade Request Id = UWS-1a97a8d9-54ef-478d-a0c0-348a17ba6755
  Composition Id = 1
  Name = host
  Start Time = 2021-09-25T05:47:02
  End Time = 2021-09-25T05:48:38
  Pid = 56156
  Host = pcamn02
  Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_host_os_2021_09_25-05.47.02.log
  Arguments = {"verify_only":false,"upgrade":false,"diagnostics":false,"host_ip":"100.96.2.35","result_override":null,"log_level":null,"switch_type":null,"precheck_status":false,"task_time":0,"fail_halt":false,"fail_upgrade":null,"component_names":null,"upgrade_to":null,"image_location":"http://host.example.com/pca-3.0.1-b535176.iso","epld_image_location":null,"expected_iso_checksum":null,"checksum":"240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7","composition_id":"1","request_id":"UWS-1a97a8d9-54ef-478d-a0c0-348a17ba6755","display_task_plan":false,"dry_run_tasks":false}
  Status = Passed
  Execution Time(sec) = 96
  Tasks 1 - Name = Validate Image Location
  Tasks 1 - Description = Verify that the image exists at the specified location and is correctly named
  Tasks 1 - Time = 2021-09-25T05:47:02
  Tasks 2 - Name = Validate Image Location
[...]

  5. When the first management node host operating system upgrade has completed successfully, execute the same command for the next management node.

    PCA-ADMIN> upgradeHost hostIp=100.96.2.33 \
imageLocation="http://host.example.com/pca-<version>-<build>.iso" \
isoChecksum=240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7

  6. When the second management node host operating system upgrade has completed successfully, exit the Service CLI and move the cluster virtual IP to one of the upgraded nodes. 

    PCA-ADMIN> exit
Connection to localhost closed.
# pcs resource move mgmt-rg pcamn01
# pcs status
Cluster name: mncluster
Stack: corosync
[...]
 scsi_fencing   (stonith:fence_scsi):   Stopped (disabled)
 Resource Group: mgmt-rg
     vip-mgmt-int       (ocf::heartbeat:IPaddr2):       Started pcamn01
     vip-mgmt-host      (ocf::heartbeat:IPaddr2):       Started pcamn01
[...]

    Moving the cluster virtual IP to another management node should only take a number of seconds.

  7. Log in to the management node with the virtual IP and launch the Service CLI to execute the host operating system upgrade for the final management node. 

    # ssh pcamn01
# ssh admin@localhost -p 30006
PCA-ADMIN> upgradeHost hostIp=100.96.2.34 \
imageLocation="http://host.example.com/pca-<version>-<build>.iso" \
isoChecksum=240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7

    When this upgrade has completed successfully, the operating system on all management nodes is up-to-date.

    Caution:

    After upgrade, the management nodes must be rebooted for the changes to take effect. This cannot be done from the Service CLI.

  8. Reboot all three management nodes either from the Oracle Linux command line or through the ILOM.

    1. Verify which management node owns the cluster virtual IP. Run this command from the command line of one of the management nodes:

      # pcs status
[...]
 Resource Group: mgmt-rg
     vip-mgmt-int     (ocf::heartbeat:IPaddr2):      Started    pcamn01
     vip-mgmt-host    (ocf::heartbeat:IPaddr2):      Started    pcamn01
     vip-mgmt-ilom    (ocf::heartbeat:IPaddr2):      Started    pcamn01
     vip-mgmt-lb      (ocf::heartbeat:IPaddr2):      Started    pcamn01
     vip-mgmt-ext     (ocf::heartbeat:IPaddr2):      Started    pcamn01
[...]

    2. Reboot the other two management nodes in the cluster. (In this example: pcamn02 and pcamn03.)

    3. Move the virtual IP to one of the rebooted management nodes. (In this example: pcamn02.) 

      # pcs resource move mgmt-rg pcamn02
# pcs status
Cluster name: mncluster
Stack: corosync
[...]
 scsi_fencing   (stonith:fence_scsi):   Stopped (disabled)
 Resource Group: mgmt-rg
     vip-mgmt-int       (ocf::heartbeat:IPaddr2):       Started pcamn02
     vip-mgmt-host      (ocf::heartbeat:IPaddr2):       Started pcamn02
[...]

    4. Reboot the last of the three upgraded management nodes. (In this example: pcamn01.)

Upgrading the MySQL Cluster Database

The MySQL Cluster database is upgraded independently of the management node host operating system; the MySQL packages are deliberately kept separate from the Oracle Linux upgrade.

It is assumed that the database upgrade is performed after the management node host operating system upgrade. As the ISO image has already been unpacked on shared storage during the operating system upgrade, the ISO path and checksum are not considered mandatory parameters for the database upgrade command. However, if the database upgrade is not preceded by a host operating system upgrade, you must include the ISO path and checksum parameters to be able to launch the database upgrade.

The MySQL Cluster database upgrade is a rolling upgrade: with one command the upgrade is executed on each of the three management nodes.

Using the Service Web UI

  1. In the navigation menu, click Upgrade & Patching.

  2. In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.

    The Create Request window appears. Choose Upgrade as the Request Type.

  3. Select the appropriate upgrade request type: Upgrade MySQL.

  4. If required, fill out the upgrade request parameters:

    • Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.

    • ISO Checksum: If you have not executed a previous upgrade command specifying this parameter, enter the checksum that allows the system to verify that the ISO image is valid for this upgrade. The checksum is provided alongside the ISO image; its file name is the ISO image name with .sha512sum appended.

    • Image Location: If you have not executed a previous upgrade command specifying this parameter, enter the path to the location where the ISO image is stored.

    • Log Level: Optionally, select a specific log level for the upgrade log file. The default log level is "Information". For maximum detail, select "Debug".

  5. Click Create Request.

    The new upgrade request appears in the Upgrade Jobs table.

Using the Service CLI

  1. If you have not previously upgraded the management node host operating system, gather the information that you need to run the command:

    • the location of the ISO image to upgrade from

    • the checksum used to verify that the ISO image is valid

  2. Enter the upgrade command.

    Syntax (entered on a single line):

    upgradeMySQL 
imageLocation=<path-to-iso>
isoChecksum=<iso-file-checksum>

    Example:

    PCA-ADMIN> upgradeMySQL \
imageLocation="http://host.example.com/pca-<version>-<build>.iso" \
isoChecksum=240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7
Command: upgradeMySQL imageLocation="http://host.example.com/pca-<version>-<build>.iso" isoChecksum=240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7
Status: Success
Time: 2021-09-25 09:21:16,264 UTC
Data:
  Service request has been submitted. Upgrade Job Id = 1632995409822-mysql-83013 Upgrade Request Id = UWS-77bc0c30-7ff5-4c50-ad09-6f96907e22e1

    If the upgrade ISO image has already been unpacked on shared storage, simply enter the command without parameters: upgradeMySQL.

  3. Use the request ID and the job ID to check the status of the upgrade process.

    PCA-ADMIN> getUpgradeJobs
  id                               upgradeRequestId                           commandName   result
  --                               ----------------                           -----------   ------
  1632995409822-mysql-83013        UWS-77bc0c30-7ff5-4c50-ad09-6f96907e22e1   mysql         Passed
  1632926926773-host-32993         UWS-fef3b663-45b7-4177-a041-26f73e68848d   host          Passed
  1632990827394-host-56156         UWS-1a97a8d9-54ef-478d-a0c0-348a17ba6755   host          Passed
  1632990493570-host-6646          UWS-4c78f3ef-ac42-4f32-9483-bb43a309faa3   host          Passed

PCA-ADMIN> getUpgradeJob upgradeJobId=1632995409822-mysql-83013
Command: getUpgradeJob upgradeJobId=1632995409822-mysql-83013
Status: Success
Time: 2021-09-25 09:24:27,874 UTC
Data:
  Upgrade Request Id = UWS-77bc0c30-7ff5-4c50-ad09-6f96907e22e1
  Name = mysql
  Start Time = 2021-09-25T09:21:16
  End Time = 2021-09-25T09:22:04
  Pid = 83013
  Host = pcamn01
  Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_mysql_cluster_2021_09_25-09.21.16.log
  Arguments = {"verify_only":false,"upgrade":false,"diagnostics":false,"host_ip":null,"result_override":null,"log_level":null,"switch_type":null,"precheck_status":false,"task_time":0,"fail_halt":false,"fail_upgrade":null,"component_names":null,"upgrade_to":null,"image_location":"http://host.example.com/pca-3.0.1-b535176.iso","epld_image_location":null,"expected_iso_checksum":null,"checksum":"240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7","composition_id":null,"request_id":"UWS-77bc0c30-7ff5-4c50-ad09-6f96907e22e1","display_task_plan":false,"dry_run_tasks":false}
  Status = Passed
  Execution Time(sec) = 48
  Tasks 1 - Name = Validate Image Location
  Tasks 1 - Description = Verify that the image exists at the specified location and is correctly named
  Tasks 1 - Time = 2021-09-25T09:21:16
[...]

Upgrading the Secret Service

The secret service contains two components that need to be upgraded separately: Etcd and Vault. The order in which you upgrade them is not relevant.

It is assumed that the secret service upgrade is performed after the management node host operating system upgrade. As the ISO image has already been unpacked on shared storage during the operating system upgrade, the ISO path and checksum are not considered mandatory parameters for the secret service upgrade commands. However, if the host operating system upgrade has not yet been executed, you must include the ISO path and checksum parameters.

The Etcd and Vault upgrades are rolling upgrades: each upgrade is executed on all three management nodes with one command.

Using the Service Web UI

  1. In the navigation menu, click Upgrade & Patching.

  2. In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.

    The Create Request window appears. Choose Upgrade as the Request Type.

  3. Select the appropriate upgrade request type: Upgrade Etcd.

  4. If required, fill out the upgrade request parameters:

    • Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.

    • ISO Checksum: If you have not executed a previous upgrade command specifying this parameter, enter the checksum that allows the system to verify that the ISO image is valid for this upgrade. The checksum is provided alongside the ISO image; its file name is the ISO image name with .sha512sum appended.

    • Image Location: If you have not executed a previous upgrade command specifying this parameter, enter the path to the location where the ISO image is stored.

    • Log Level: Optionally, select a specific log level for the upgrade log file. The default log level is "Information". For maximum detail, select "Debug".

  5. Click Create Request.

    The new upgrade request appears in the Upgrade Jobs table.

  6. When the Etcd upgrade has completed successfully, repeat this procedure to create an upgrade request for Vault.

Using the Service CLI

  1. If you have not previously upgraded the management node host operating system, gather the information that you need to run the command:

    • the location of the ISO image to upgrade from

    • the checksum used to verify that the ISO image is valid

  2. Enter the two upgrade commands. Wait until one upgrade is finished before entering the second command.

    Syntax (entered on a single line):

    upgradeEtcd 
imageLocation=<path-to-iso>
isoChecksum=<iso-file-checksum>
    upgradeVault 
imageLocation=<path-to-iso>
isoChecksum=<iso-file-checksum>

    Example:

    PCA-ADMIN> upgradeEtcd \
imageLocation="http://host.example.com/pca-<version>-<build>.iso" \
isoChecksum=240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7
Command: upgradeEtcd imageLocation="http://host.example.com/pca-<version>-<build>.iso" isoChecksum=240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7
Status: Success
Time: 2021-09-25 10:24:52,177 UTC
Data:
  Service request has been submitted. Upgrade Job Id = 1632826770954-etcd-26973 Upgrade Request Id = UWS-fec15d32-fc2b-48bd-9ae0-62f49587a284

PCA-ADMIN> upgradeVault \
imageLocation="http://host.example.com/pca-<version>-<build>.iso" \
isoChecksum=240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7
Command: upgradeVault imageLocation="http://host.example.com/pca-<version>-<build>.iso" isoChecksum=240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7
Status: Success
Time: 2021-09-25 10:38:25,417 UTC
Data:
  Service request has been submitted. Upgrade Job Id = 1632850933353-vault-16966 Upgrade Request Id = UWS-352df3d1-c21f-441b-8f6e-9381ac075906

    If the upgrade ISO image has already been unpacked on shared storage, simply enter the commands without parameters: upgradeEtcd and upgradeVault.

  3. Use the request ID and the job ID to check the status of the upgrade process.

    PCA-ADMIN> getUpgradeJobs
  id                               upgradeRequestId                           commandName   result
  --                               ----------------                           -----------   ------
  1632995409822-mysql-83013        UWS-77bc0c30-7ff5-4c50-ad09-6f96907e22e1   mysql         Passed
  1632850933353-vault-16966        UWS-352df3d1-c21f-441b-8f6e-9381ac075906   vault         Passed
  1632826770954-etcd-26973         UWS-fec15d32-fc2b-48bd-9ae0-62f49587a284   etcd          Passed
  1632926926773-host-32993         UWS-fef3b663-45b7-4177-a041-26f73e68848d   host          Passed
  1632990827394-host-56156         UWS-1a97a8d9-54ef-478d-a0c0-348a17ba6755   host          Passed
  1632990493570-host-6646          UWS-4c78f3ef-ac42-4f32-9483-bb43a309faa3   host          Passed

PCA-ADMIN> getUpgradeJob upgradeJobId=1632850933353-vault-16966
Command: getUpgradeJob upgradeJobId=1632850933353-vault-16966
Status: Success
Time: 2021-09-25 10:39:31,308 UTC
Data:
  Upgrade Request Id = UWS-352df3d1-c21f-441b-8f6e-9381ac075906
  Name = vault
  Start Time = 2021-09-25T10:38:25
  End Time = 2021-09-25T10:39:07
  Pid = 16966
  Host = pcamn02
  Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_vault_2021_09_25-10.38.25.log
  Arguments = {"verify_only":false,"upgrade":false,"diagnostics":false,"host_ip":null,"result_override":null,"log_level":null,"switch_type":null,"precheck_status":false,"task_time":0,"fail_halt":false,"fail_upgrade":null,"component_names":null,"upgrade_to":null,"image_location":"http://host.example.com/pca-3.0.1-b535176.iso","epld_image_location":null,"expected_iso_checksum":null,"checksum":"240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7","composition_id":null,"request_id":"UWS-352df3d1-c21f-441b-8f6e-9381ac075906","display_task_plan":false,"dry_run_tasks":false}
  Status = Passed
  Execution Time(sec) = 42
  Tasks 1 - Name = Check Vault Running Status
  Tasks 1 - Description = Check vault service running status is healthy
  Tasks 1 - Time = 2021-09-25T10:38:25
[...]

Upgrading Firmware

Firmware is included in the ISO image for all component ILOMs, for the ZFS Storage Appliance, and for the switches. Select the instructions below for the component type you want to upgrade.

Upgrading ILOMs

ILOM upgrades can be applied to management nodes and compute nodes. Firmware packages may be different per component type, so make sure you select the correct one from the firmware directory. You must upgrade ILOMs one at a time, using each one's internal IP address as a command parameter.

To obtain the ILOM IP addresses, use the Service CLI command show ComputeNode name=<node_name> or show ManagementNode name=<node_name> and look for the ILOM Ip Address in the output.

Caution:

You must NOT upgrade the ILOM of the management node that holds the management virtual IP address, and thus the primary role in the cluster. To upgrade its ILOM, first reboot the management node in question so that another node in the cluster takes over the primary role. Once the node has rebooted completely, you can proceed with the ILOM upgrade.

To determine which management node has the primary role in the cluster, log in to any management node and run the command pcs status.

Using the Service Web UI

  1. In the navigation menu, click Upgrade & Patching.

  2. In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.

    The Create Request window appears. Choose Upgrade as the Request Type.

  3. Select the appropriate upgrade request type: Upgrade ILOM.

  4. Fill out the upgrade request parameters:

    • Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.

    • Host IP: Enter the component's assigned IP address in the ILOM network. This is an IP address in the internal 100.96.0.0/23 range.

    • Image Location: Enter the path to the location where the firmware package is stored. ILOM firmware is stored as a *.pkg file in the /pca_firmware/<component>/ subdirectory of the unpacked ISO image.

    • Log Level: Optionally, select a specific log level for the upgrade log file. The default log level is "Information". For maximum detail, select "Debug".

  5. Click Create Request.

    The new upgrade request appears in the Upgrade Jobs table.

Using the Service CLI

  1. Gather the information that you need to run the command:

    • the IP address of the ILOM for which you intend to upgrade the firmware

    • the path to the firmware package file in the unpacked ISO image

  2. Enter the upgrade command.

    Syntax (entered on a single line):

    upgradeIlom
imageLocation=<path-to-firmware>
hostIp=<ilom-ip>

    Example:

    PCA-ADMIN> upgradeIlom \
imageLocation="file:///nfs/shared_storage/pca_firmware/X9-2/.../ILOM-<version>-ORACLE_SERVER_X9-2-rom.pkg" \
hostIp=100.96.0.66
Command: upgradeIlom imageLocation="file:///nfs/shared_storage/pca_firmware/X9-2/.../ILOM-<version>-ORACLE_SERVER_X9-2-rom.pkg" hostIp=100.96.0.66
Status: Success
Time: 2021-09-24 11:18:31,044 UTC
Data:
  Service request has been submitted. Upgrade Job Id = 1620921089806-ilom-21480 Upgrade Request Id = UWS-732d6fce-9f06-4329-b972-d093bee40010

  3. Use the request ID and the job ID to check the status of the upgrade process.

    PCA-ADMIN> getUpgradeJobs
  id                               upgradeRequestId                           commandName   result
  --                               ----------------                           -----------   ------
  1620921089806-ilom-21480         UWS-732d6fce-9f06-4329-b972-d093bee40010   ilom          Passed
  1632926926773-host-32993         UWS-fef3b663-45b7-4177-a041-26f73e68848d   host          Passed
  1632990827394-host-56156         UWS-1a97a8d9-54ef-478d-a0c0-348a17ba6755   host          Passed
  1632990493570-host-6646          UWS-4c78f3ef-ac42-4f32-9483-bb43a309faa3   host          Passed

PCA-ADMIN> getUpgradeJob upgradeJobId=1620921089806-ilom-21480
Command: getUpgradeJob 1620921089806-ilom-21480
Status: Success
Time: 2021-09-24 11:24:49,243 UTC
Data:
  Upgrade Request Id = UWS-732d6fce-9f06-4329-b972-d093bee40010
  Name = ilom
  Start Time = 2021-09-24 11:18:32
  End Time = 2021-09-24 11:21:18
  Pid = 21480
  Host = pcamn02
  Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_ilom_firmware_2021_09_24-11.18.31.log
  Arguments = {"verify_only":false,"upgrade":false,"diagnostics":false,"host_ip":"100.96.0.66","result_override":null,"log_level":null,"switch_type":null,"precheck_status":false,"task_time":0,"fail_halt":false,"fail_upgrade":null,"component_names":null,"upgrade_to":null,"image_location":"file:///nfs/shared_storage/pca_firmware/X9-2/.../ILOM-5_0_2_21_r140740-ORACLE_SERVER_X9-2-rom.pkg","epld_image_location":null,"expected_iso_checksum":null,"checksum":null,"composition_id":null,"request_id":"UWS-732d6fce-9f06-4329-b972-d093bee40010","display_task_plan":false,"dry_run_tasks":false}
  Status = Passed
  Execution Time(sec) = 166
  Tasks 1 - Name = Validate Image Location
  Tasks 1 - Description = Verify that the image exists at the specified location and is correctly named
  Tasks 1 - Time = 2021-09-24T11:18:32
[...]

At the end of the upgrade, the ILOM itself is rebooted automatically. However, the server component also needs to be rebooted for all changes to take effect. It is the administrator's responsibility to manually reboot the management node or compute node after a successful ILOM upgrade.

Upgrading the ZFS Storage Appliance Operating Software

To upgrade the operating software of the appliance's ZFS Storage Appliance, you only need to provide the path to the firmware package in the unpacked ISO image. The IP addresses of the storage controllers are known, and a single upgrade command initiates a rolling upgrade of both controllers. If a new ILOM firmware version is included for the two controllers, it will be installed as part of the ZFS Storage Appliance upgrade process.

Caution:

Do not make storage configuration changes while an upgrade is in progress. While controllers are running different software versions, configuration changes made to one controller are not propagated to its peer controller.

Before You Begin

Before you initiate a ZFS Storage Appliance upgrade, you must disable the node state service to prevent errors in node states after the upgrade.
  1. From a management node, set the provisioning lock by issuing this command: 
    pca-admin locks set system provisioning
  2. Perform the ZFS Storage Appliance upgrade using either the Service Web UI or the Service CLI procedure below.
  3. Release the provisioning lock.
    pca-admin locks unset system provisioning
  4. Confirm the lock state.
    pca-admin locks show system

Using the Service Web UI

  1. In the navigation menu, click Upgrade & Patching.

  2. In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.

    The Create Request window appears. Choose Upgrade as the Request Type.

  3. Select the appropriate upgrade request type: Upgrade Zfssa.

  4. Fill out the upgrade request parameters:

    • Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.

    • Image Location: Enter the path to the location where the firmware package is stored. ZFS Storage Appliance operating software is stored as a *.pkg file in the /pca_firmware/zfs/ subdirectory of the unpacked ISO image.

    • Log Level: Optionally, select a specific log level for the upgrade log file. The default log level is "Information". For maximum detail, select "Debug".

  5. Click Create Request.

    The new upgrade request appears in the Upgrade Jobs table.

Using the Service CLI

  1. Gather the information that you need to run the command: the path to the AK-NAS firmware package in the unpacked ISO image.

  2. Enter the upgrade command.

    Syntax:

    upgradeZfssa imageLocation=<path-to-firmware>

    Example:

    PCA-ADMIN> upgradeZfssa imageLocation="file:///nfs/shared_storage/pca_firmware/zfs/ak-nas-<version>.pkg"
Command: upgradeZfssa imageLocation="file:///nfs/shared_storage/pca_firmware/zfs/ak-nas-<version>.pkg"
Status: Success
Time: 2021-09-27 11:15:07,453 UTC
Data:
  Service request has been submitted. Upgrade Job Id = 1632914107346-zfssa-83002 Upgrade Request Id = UWS-881af57f-5dfb-4c75-8026-9f00cf3eb7c9

  3. Use the request ID and the job ID to check the status of the upgrade process.

    PCA-ADMIN> getUpgradeJobs
  id                               upgradeRequestId                           commandName   result
  --                               ----------------                           -----------   ------
  1632914107346-zfssa-83002        UWS-881af57f-5dfb-4c75-8026-9f00cf3eb7c9   zfssa         Passed
  1632926926773-host-32993         UWS-fef3b663-45b7-4177-a041-26f73e68848d   host          Passed
  1632990827394-host-56156         UWS-1a97a8d9-54ef-478d-a0c0-348a17ba6755   host          Passed
  1632990493570-host-6646          UWS-4c78f3ef-ac42-4f32-9483-bb43a309faa3   host          Passed

PCA-ADMIN> getUpgradeJob upgradeJobId=1632914107346-zfssa-83002
Command: getUpgradeJob upgradeJobId=1632914107346-zfssa-83002
Status: Success
Time: 2021-09-27 11:42:10,729 UTC
Data:
  Upgrade Request Id = UWS-881af57f-5dfb-4c75-8026-9f00cf3eb7c9
  Name = zfssa
  Start Time = 2021-09-29T11:15:07
  End Time = 2021-09-29T11:26:42
  Pid = 83002
  Host = pcamn02
  Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_zfssa_ak_2021_09_29-11.15.07.log
  Arguments = {"verify_only":false,"upgrade":false,"diagnostics":false,"host_ip":null,"result_override":null,"log_level":null,"switch_type":null,"precheck_status":false,"task_time":0,"fail_halt":false,"fail_upgrade":null,"component_names":null,"upgrade_to":null,"image_location":"file:///nfs/shared_storage/pca_firmware/zfs/ak-nas-2021.08.27-1.0x-nondebug.pkg","epld_image_location":null,"expected_iso_checksum":null,"checksum":null,"composition_id":null,"request_id":"UWS-881af57f-5dfb-4c75-8026-9f00cf3eb7c9","display_task_plan":false,"dry_run_tasks":false}
  Status = Passed
  Execution Time(sec) = 697
  Tasks 1 - Name = Validate Image Location
  Tasks 1 - Description = Verify that the image exists at the specified location and is correctly named
  Tasks 1 - Time = 2021-09-29T11:15:08
[...]
Upgrading the Switch Software

The appliance rack contains three categories of Cisco Nexus switches: a management switch, two leaf switches, and two spine switches. They all run the same Cisco NX-OS network operating software. You must perform the upgrades in this order: leaf switches first, then spine switches, and finally the management switch.

When upgrading their firmware, use the same binary file with each upgrade command. Only one command per switch category is required, meaning that the leaf switches and the spine switches are upgraded in pairs.

Some versions of the network operating software consist of two files: a binary file and an additional EPLD (electronic programmable logic device) image. If the new firmware includes an EPLD file, make sure it is also added to the upgrade request.

Using the Service Web UI

  1. In the navigation menu, click Upgrade & Patching.

  2. In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.

    The Create Request window appears. Choose Upgrade as the Request Type.

  3. Select the appropriate upgrade request type: Upgrade Switch.

  4. Fill out the upgrade request parameters:

    • Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.

    • EPLD: If required for this firmware version, enter the path to the location where the EPLD image file is stored. If present, an EPLD file is an *.img file stored alongside the NX-OS operating software in the /pca_firmware/network/cisco/ subdirectory of the unpacked ISO image.

    • Image Location: Enter the path to the location where the firmware package is stored. Cisco NX-OS network operating software is stored as a *.bin file in the /pca_firmware/network/cisco/ subdirectory of the unpacked ISO image.

    • Log Level: Optionally, select a specific log level for the upgrade log file. The default log level is "Information". For maximum detail, select "Debug".

    • Switch Type: Select the switch type you intend to upgrade. The preferred upgrade order is as follows: leaf switches first, then spine switches, and finally the management switch.

  5. Click Create Request.

    The new upgrade request appears in the Upgrade Jobs table.

  6. When the upgrade has completed successfully, but other switches in the system still need to be upgraded, repeat this procedure for any other type of switch that requires upgrading.

Using the Service CLI

  1. Gather the information that you need to run the command:

    • the type of witch to upgrade (spine, leaf, management)

    • the path to the firmware binary file in the unpacked ISO image

    • if present with the new firmware version, the path to the EPLD upgrade file in the unpacked ISO image

  2. Enter the upgrade command.

    Syntax (entered on a single line):

    upgradeSwitch 
switchType=[MGMT | SPINE | LEAF] 
imageLocation=<path-to-firmware>
(epld=<path-to-epld-file>)

    Example:

    PCA-ADMIN> upgradeSwitch switchType=LEAF \
imageLocation="file:///nfs/shared_storage/pca_firmware/network/cisco/nxos.<version>.bin" \
epld="file:///nfs/shared_storage/pca_firmware/network/cisco/n9000-epld.<version>.img"
Command: upgradeSwitch switchType=LEAF imageLocation="file:///nfs/shared_storage/pca_firmware/network/cisco/nxos.<version>.bin" epld="file:///nfs/shared_storage/pca_firmware/network/cisco/n9000-epld.<version>.img"
Status: Success
Time: 2021-09-24 14:16:54,704 UTC
Data:
  Service request has been submitted. Upgrade Job Id = 1630511206512-cisco-20299 Upgrade Request Id = UWS-44688fe5-b4f8-407f-a1b5-8cd1b685c2c3

  3. Use the request ID and the job ID to check the status of the upgrade process.

    PCA-ADMIN> getUpgradeJobs
  id                               upgradeRequestId                           commandName   result
  --                               ----------------                           -----------   ------
  1632914107346-zfssa-83002        UWS-881af57f-5dfb-4c75-8026-9f00cf3eb7c9   zfssa         Passed
  1630511206512-cisco-20299        UWS-44688fe5-b4f8-407f-a1b5-8cd1b685c2c3   cisco         Passed
  1620921089806-ilom-21480         UWS-732d6fce-9f06-4329-b972-d093bee40010   ilom          Passed

PCA-ADMIN> getupgradeJob upgradeJobId=1630511206512-cisco-20299
Command: getupgradeJob upgradeJobId=1630511206512-cisco-20299
Status: Success
Time: 2021-09-24 15:48:08,455 UTC
Data:
  Upgrade Request Id = UWS-44688fe5-b4f8-407f-a1b5-8cd1b685c2c3
  Name = cisco
  Start Time = 2021-09-24T14:46:46
  End Time = 2021-09-24T14:59:44
  Pid = 20299
  Host = pcamn02
  Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_cisco_firmware_2021_09_24-14.46.46.log
  Arguments = {"verify_only":false,"upgrade":false,"diagnostics":false,"host_ip":null,"result_override":null,"log_level":null,"switch_type":"LEAF","precheck_status":false,"task_time":0,"fail_halt":false,"fail_upgrade":null,"component_names":null,"upgrade_to":null,"image_location":"file:///nfs/shared_storage/pca_firmware/network/cisco/nxos.9.3.2.bin","epld_image_location":null,"expected_iso_checksum":null,"checksum":null,"composition_id":null,"request_id":"UWS-44688fe5-b4f8-407f-a1b5-8cd1b685c2c3","display_task_plan":false,"dry_run_tasks":false}
  Status = Passed
  Execution Time(sec) = 777
  Tasks 1 - Name = Validate Image Location
  Tasks 1 - Description = Verify that the image exists at the specified location and is correctly named
  Tasks 1 - Time = 2021-09-24T14:46:47
[...]

Upgrading the Kubernetes Cluster

The Kubernetes container orchestration environment upgrade is also kept separate from the operating system. With a single command, all Kubernetes packages, such as kubeadm, kubectl and kubelet, are upgraded on the three management nodes and all the compute nodes. Note that this upgrade does not include the microservices running in Kubernetes containers.

For dependency reasons, Kubernetes must be upgraded after the management node host operating system. Thus, the upgrade ISO image has already been unpacked on shared storage; the Kubernetes upgrade command has no mandatory parameters.

Using the Service Web UI

  1. In the navigation menu, click Upgrade & Patching.

  2. In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.

    The Create Request window appears. Choose Upgrade as the Request Type.

  3. Select the appropriate upgrade request type: Upgrade Kubernetes.

  4. If required, fill out the upgrade request parameters:

    • Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.

    • ISO Checksum: If you have not executed a previous upgrade command specifying this parameter, enter the checksum that allows the system to verify that the ISO image is valid for this upgrade. The checksum is provided alongside the ISO image; its file name is the ISO image name with .sha512sum appended.

    • Image Location: If you have not executed a previous upgrade command specifying this parameter, enter the path to the location where the ISO image is stored.

    • Log Level: Optionally, select a specific log level for the upgrade log file. The default log level is "Information". For maximum detail, select "Debug".

  5. Click Create Request.

    The new upgrade request appears in the Upgrade Jobs table.

Using the Service CLI

  1. If you have not previously upgraded the management node host operating system, gather the information that you need to run the command:

    • the location of the ISO image to upgrade from

    • the checksum used to verify that the ISO image is valid

  2. Enter the upgrade command.

    Syntax (entered on a single line):

    upgradeKubernetes 
imageLocation=<path-to-iso>
isoChecksum=<iso-file-checksum>

    Example:

    PCA-ADMIN> upgradeKubernetes \
imageLocation="http://host.example.com/pca-<version>-<build>.iso" \
isoChecksum=240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7
Command: upgradeKubernetes imageLocation="http://host.example.com/pca-<version>-<build>.iso" isoChecksum=240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7
Status: Success
Time: 2021-09-26 17:20:09,423 UTC
Data:
  Service request has been submitted. Upgrade Job Id = 1632849609034-kubernetes-35545 Upgrade Request Id = UWS-edfa3b32-c32a-4b67-8df5-2357096052bf

    If the upgrade ISO image has already been unpacked on shared storage, simply enter the command without parameters: upgradeKubernetes.

  3. Use the request ID and the job ID to check the status of the upgrade process.

    PCA-ADMIN> getUpgradeJobs
  id                               upgradeRequestId                           commandName   result
  --                               ----------------                           -----------   ------
  1632849609034-kubernetes-35545   UWS-edfa3b32-c32a-4b67-8df5-2357096052bf   kubernetes    Passed
  1632826770954-etcd-26973         UWS-fec15d32-fc2b-48bd-9ae0-62f49587a284   etcd          Passed
  1632850933353-vault-16966        UWS-352df3d1-c21f-441b-8f6e-9381ac075906   vault         Passed

PCA-ADMIN> getUpgradeJob upgradeJobId=1632849609034-kubernetes-35545
Command: getUpgradeJob upgradeJobId=1632849609034-kubernetes-35545
Status: Success
Time: 2021-09-26 17:43:38,443 UTC
Data:
  Upgrade Request Id = UWS-edfa3b32-c32a-4b67-8df5-2357096052bf
  Name = kubernetes
  Start Time = 2021-09-26T17:20:09
  End Time = 2021-09-26T17:21:52
  Pid = 35545
  Host = pcamn02
  Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_kubernetes_cluster_2021_09_26-17.20.09.log
  Arguments = {"verify_only":false,"upgrade":false,"diagnostics":false,"host_ip":null,"result_override":null,"log_level":null,"switch_type":null,"precheck_status":false,"task_time":0,"fail_halt":false,"fail_upgrade":null,"component_names":null,"upgrade_to":null,"image_location":"http://host.example.com/pca-3.0.1-b535176.iso","epld_image_location":null,"expected_iso_checksum":null,"checksum":"240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7","composition_id":null,"request_id":"UWS-edfa3b32-c32a-4b67-8df5-2357096052bf","display_task_plan":false,"dry_run_tasks":false}
  Status = Passed
  Execution Time(sec) = 249
  Tasks 1 - Name = Retrieving Cluster Status
  Tasks 1 - Description = Retrieving cluster status and upgrade data from the kubernetes nodes
  Tasks 1 - Time = 2021-09-26T17:20:10
[...]

Upgrading the Microservices

The microservices upgrade covers both the internal services of the platform layer, and the administrative and user-level services exposed through the infrastructure services layer.

The containerized microservices have their own separate upgrade mechanism. A service is upgraded if a new Helm deployment chart and container image are found in the ISO image. When a new deployment chart is detected during the upgrade process, the pods running the services are restarted with the new container image.

It is assumed that the microservices upgrade is performed after the management node host operating system upgrade. As the ISO image has already been unpacked on shared storage during the operating system upgrade, the ISO path and checksum are not considered mandatory parameters for the microservices upgrade command. However, if the host operating system upgrade has not yet been executed, you must include the ISO path and checksum parameters.

Using the Service Web UI

  1. In the navigation menu, click Upgrade & Patching.

  2. In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.

    The Create Request window appears. Choose Upgrade as the Request Type.

  3. Select the appropriate upgrade request type: Upgrade Platform.

  4. If required, fill out the upgrade request parameters:

    • Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.

    • ISO Checksum: If you have not executed a previous upgrade command specifying this parameter, enter the checksum that allows the system to verify that the ISO image is valid for this upgrade. The checksum is provided alongside the ISO image; its file name is the ISO image name with .sha512sum appended.

    • Image Location: If you have not executed a previous upgrade command specifying this parameter, enter the path to the location where the ISO image is stored.

    • Log Level: Optionally, select a specific log level for the upgrade log file. The default log level is "Information". For maximum detail, select "Debug".

  5. Click Create Request.

    The new upgrade request appears in the Upgrade Jobs table.

Using the Service CLI

  1. If you have not previously upgraded the management node host operating system, gather the information that you need to run the command:

    • the location of the ISO image to upgrade from

    • the checksum used to verify that the ISO image is valid

  2. Enter the upgrade command.

    Syntax (entered on a single line):

    upgradePlatform 
imageLocation=<path-to-iso>
isoChecksum=<iso-file-checksum>

    Example:

    PCA-ADMIN> upgradePlatform \
imageLocation="http://host.example.com/pca-<version>-<build>.iso" \
isoChecksum=240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7
Command: upgradePlatform imageLocation="http://host.example.com/pca-<version>-<build>.iso" isoChecksum=240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7
Status: Success
Time: 2021-09-26 20:48:41,452 UTC
Data:
  Service request has been submitted. Upgrade Job Id = 1632850650836-platform-68465 Upgrade Request Id = UWS-26dba234-9b52-426d-836c-ac11f37e717f

    If the upgrade ISO image has already been unpacked on shared storage, simply enter the command without parameters: upgradePlatform.

  3. Use the request ID and the job ID to check the status of the upgrade process.

    PCA-ADMIN> getUpgradeJobs
  id                               upgradeRequestId                           commandName   result
  --                               ----------------                           -----------   ------
  1632850650836-platform-68465     UWS-26dba234-9b52-426d-836c-ac11f37e717f   platform      Passed
  1632849609034-kubernetes-35545   UWS-edfa3b32-c32a-4b67-8df5-2357096052bf   kubernetes    Passed
  1632826770954-etcd-26973         UWS-fec15d32-fc2b-48bd-9ae0-62f49587a284   etcd          Passed
  1632850933353-vault-16966        UWS-352df3d1-c21f-441b-8f6e-9381ac075906   vault         Passed

PCA-ADMIN> getUpgradeJob upgradeJobId=1632850650836-platform-68465
Command: getUpgradeJob 1632850650836-platform-68465
Status: Success
Time: 2021-09-26 21:03:16,264 UTC
Data:
  Upgrade Request Id = UWS-26dba234-9b52-426d-836c-ac11f37e717f
  Name = kubernetes
  Start Time = 2021-09-26T20:48:41
  End Time = 2021-09-26T20:59:34
  Pid = 68465
  Host = pcamn02
  Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_platform_services_2021_09_26-20.48.41.log
  Arguments = {"verify_only":false,"upgrade":false,"diagnostics":false,"host_ip":null,"result_override":null,"log_level":null,"switch_type":null,"precheck_status":false,"task_time":0,"fail_halt":false,"fail_upgrade":null,"component_names":null,"upgrade_to":null,"image_location":"http://host.example.com/pca-3.0.1-b535176.iso","epld_image_location":null,"expected_iso_checksum":null,"checksum":"240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7","composition_id":null,"request_id":"UWS-26dba234-9b52-426d-836c-ac11f37e717f","display_task_plan":false,"dry_run_tasks":false}
  Status = Passed
  Execution Time(sec) = 653
  Tasks 1 - Name = Check All Ingress Endpoints
  Tasks 1 - Description = Check whether all ingress endpoints are up and running
  Tasks 1 - Time = 2021-09-26T20:48:42
[...]