Active Versions 3.0.2-b776803 to 3.0.2-b852928

If your Private Cloud Appliance is running a software version with build number 3.0.2-b776803 to 3.0.2-b852928 (both included), follow the instructions in this section to upgrade to appliance software version 3.0.2-b892153.

Prepare the Upgrade Environment

Software versions and upgrades for Oracle Private Cloud Appliance are made available for download through My Oracle Support. The ISO file contains all the files and packages required to upgrade the appliance hardware and software components to a given release. All the items within the ISO file have been tested to work with each other and qualified for installation on your rack system. Do not install or upgrade individual packages on the appliance components.

Preparation of the upgrade environment consists of these operations:

  1. Preconfiguration: downloading and unpacking the ISO image

  2. Backing up system-critical services

  3. Upgrading the Upgrader

When environment preparation is complete, the system is ready for the appliance administrator to start the upgrade procedures.

Every upgrade operation is preceded by a set of pre-checks. The upgrade will only begin if all pre-checks are passed. Concurrent upgrade operations are not supported. An upgrade job must be completed before a new one can be started.

Download and Unpack the ISO Image

To be able to use an ISO file to upgrade your appliance, you need to download the file to a location from where a web server can make it available to the Private Cloud Appliance management nodes. If you have set up a bastion host connected to the internal administration network of the appliance, it is convenient to store the ISO file on that machine and run a web server to make the ISO file accessible over http.

Before performing any upgrade operations, you unpack the contents of the ISO file to populate the source directories in the shared storage that is mounted on all three management nodes. This ensures that the new version is installed when an upgrade command is run.

Using the Service CLI

  1. Gather the information that you need to run the command:

    • the location of the ISO image to upgrade from

      Enter the path to where the ISO image is stored. Its contents will be unpacked on the shared storage accessible from the management nodes.

    • the checksum used to verify the ISO image

      The checksum is provided alongside the ISO image. Its file name is the ISO image name with .sha256sum appended. The system uses the checksum to verify that the data in the ISO image is intact and valid for this upgrade.

  2. Enter the upgrade preconfiguration command.

    Syntax (entered on a single line):

    upgradePreConfig 
    option=ISO
    location=<path-to-iso>
    isoChecksum=<iso-file-checksum>

    Example:

    PCA-ADMIN> upgradePreConfig  option=ISO \
    location=http://host.example.com/pca-<version>-<build>.iso \
    isoChecksum=90e4505b098031afb02068080db2603dc6f580cd7cf52aa51ecd0c3b81668027
    Data:
      Service request has been submitted. Upgrade Job Id = 1668417666968-prepare-28142 Upgrade Request Id = UWS-c94ba56a-1b91-49d8-8e51-afeae7f62186
  3. Use the request ID and the job ID to check the status of the upgrade job. Ensure that it completes successfully.

    PCA-ADMIN> getUpgradeJobs
      id                               upgradeRequestId                           commandName   result
      --                               ----------------                           -----------   ------
      1668417666968-prepare-28142      UWS-c94ba56a-1b91-49d8-8e51-afeae7f62186   prepare       Passed
    
    PCA-ADMIN> getUpgradeJob upgradeJobId=1668417666968-prepare-28142
  4. Proceed to the next upgrade preparation phase.

Using the Service Web UI

Note:

The user interface offers this functionality as of version 3.0.2-b819070. If the active version is older, follow the Service CLI instructions.

  1. In the navigation menu, click Upgrade & Patching.

  2. In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.

    The Create Request window appears. Choose Upgrade as the Request Type.

  3. Select the appropriate upgrade request type: Upgrade PreConfig.

  4. Fill out the upgrade request parameters:

    • Option: Enter ISO.

    • Location: Enter the path to the location where the ISO image is stored.

    • ISO Checksum: Enter the checksum required to verify the integrity of the ISO image. The checksum is provided alongside the ISO image in a file named <iso_image>.sha256sum.

    • Log Level: Optionally, select a specific log level for the upgrade log file. The default log level is "Information". For maximum detail, select "Debug".

    • Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.

    • Alternative ULN Channel: This parameter applies to patching and can be ignored.

  5. Click Create Request.

    The new upgrade request appears in the Upgrade Jobs table. Ensure that it completes successfully.

    The ISO image is unpacked on the shared storage of the management node cluster, from where its contents can be used to perform the required upgrade operations.

Back Up Critical Services

For system-critical components and services, Oracle Private Cloud Appliance runs a scheduled backup service that allows the appliance to be restored to its last known healthy state in case of a catastrophic failure. It is recommended that you implement a backup strategy for the users' cloud resources in the Compute Enclave as well.

Before upgrading or patching any component of the Private Cloud Appliance, you should create a backup of the latest state of the MySQL database, the ZFS Storage Appliance, and the Secret Service (Vault). The backup commands leverage the existing backup service but create an additional restore point that includes the most recent changes from right before you start an upgrade or patch workflow.

Using the Service CLI

  1. Start the three required backup tasks.

    PCA-ADMIN> backup target=vault
    Status: Success
    Data:
      Type = BackupJob
      Job Id = ocid1.brs-job.PCA3X62D9C1.mypca.joopwuv9403uzbfrh4x9mprmoduh3ljais6ex233v1b21ccqywu4a3vqykgm
      Display Name = brs-job-1668419778-backup
      Profile Id = ocid1.backup_profile.PCA3X62D9C1.mypca.wrxfwtxwxw6ydp2mwnypcaaxxzmwpuhsc33gcm3dyte7kgr4etuhb29qbs8q
      Time Created = 2022-11-06T09:56:18Z
      Lifecycle State = CREATING
      Retention = 14
    
    PCA-ADMIN> backup target=zfs
    Status: Success
    Data:
      Type = BackupJob
      Job Id = ocid1.brs-job.PCA3X62D9C1.mypca.9oaeaa2kw5crqfcjkh8kyhbxcv8bwh0f4ud6n3lucf802oj15ss3k39874bc
      Display Name = brs-job-1668419842-backup
      Profile Id = ocid1.backup_profile.PCA3X62D9C1.mypca.p7w0tgbvhtjqsgc8rllca2cvotkpgrtf4huiph7466mjio0dgskij9f0bp06
      Time Created = 2022-11-06T09:57:22Z
      Lifecycle State = CREATING
      Retention = 14
    
    PCA-ADMIN> backup target=mysql
    Status: Success
    Data:
      Type = BackupJob
      Job Id = ocid1.brs-job.PCA3X62D9C1.mypca.iew5tphpgr3h6mhliw2fai2ywvv386a0xc7isfo8kisj0wrcx114irnit6ot
      Display Name = brs-job-1668419850-backup
      Profile Id = ocid1.backup_profile.PCA3X62D9C1.mypca.henfqzdbafs4z3mxeuslb1c6f4t049w0pxvwf1gi3eb8wm1y11v7m932tn4g
      Time Created = 2022-11-06T09:57:30Z
      Lifecycle State = CREATING
      Retention = 14
  2. Use the backup job ID to check the status of the backups.

    PCA-ADMIN> getBackupJobs
    Data:
      id                                                                                              displayName                 components
      --                                                                                              -----------                 ----------
      ocid1.brs-job.PCA3X62D9C1.mypca.iew5tphpgr3h6mhliw2fai2ywvv386a0xc7isfo8kisj0wrcx114irnit6ot   brs-job-1668419850-backup   mysql
      ocid1.brs-job.PCA3X62D9C1.mypca.9oaeaa2kw5crqfcjkh8kyhbxcv8bwh0f4ud6n3lucf802oj15ss3k39874bc   brs-job-1668419842-backup   zfs
      ocid1.brs-job.PCA3X62D9C1.mypca.joopwuv9403uzbfrh4x9mprmoduh3ljais6ex233v1b21ccqywu4a3vqykgm   brs-job-1668419778-backup   vault
    
    PCA-ADMIN> getBackupJob backupJobId=ocid1.brs-job.PCA3X62D9C1.mypca.iew5tphpgr3h6mhliw2fai2ywvv386a0xc7isfo8kisj0wrcx114irnit6ot
    Status: Success
    Data:
      Type = BackupJob
      Job Id = ocid1.brs-job.PCA3X62D9C1.mypca.iew5tphpgr3h6mhliw2fai2ywvv386a0xc7isfo8kisj0wrcx114irnit6ot
      Display Name = brs-job-1668419850-backup
      Time Created = 2022-11-06T09:57:30Z
      Status = success
      Components = mysql
  3. Confirm that all three backup operations have completed successfully. Then, proceed to the next upgrade preparation phase.

Upgrade the Upgrader

Ensure that the Upgrader is up-to-date first, so that all upgrade commands are run with the latest version. Upgrade operations will not be allowed to run if the system detects that the Upgrader is not the latest available version.

Caution:

Ensure that the upgrade preconfiguration task has been completed first.

It is critical that both core commands of the preparation process, upgradePreConfig and preUpgrade, are completed together in the specified order. If at any time you need to rerun the preUpgrade command, you must rerun the preceding command first.

Using the Service CLI

  1. Start the process to install the latest upgrader version on the management nodes.

    PCA-ADMIN> preUpgrade action=start type=ISO
    Data: 
      Successfully triggered the pre-upgrade task.
       Pre upgrade status = success

    As part of the process, these operations are performed:

    1. Save the existing yum configuration.

    2. Configure the yum repository for the new upgrader files.

    3. Install the new upgrader version on the management nodes, then restart the upgrader systemd service for the changes to take effect.

    4. Restore the existing yum configuration that was saved in the first step.

  2. Check the status of the upgrade process at any time using this command:

    PCA-ADMIN> preUpgrade action=status
    Data: 
      A pre-upgrade task is running!
      Pre upgrade status = IN-PROGRESS
     
    PCA-ADMIN> preUpgrade action=status
    Data: 
       The previous pre-upgrade task succeeded!
       Pre upgrade status = SUCCESS
  3. Confirm that the latest version of the upgrader has been installed successfully. Ensure that the system is in ready state. Then, proceed with the component upgrades.

Using the Service Web UI

Note:

The user interface offers this functionality as of version 3.0.2-b819070. If the active version is older, follow the Service CLI instructions.

  1. In the navigation menu, click Upgrade & Patching.

  2. In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.

    The Create Request window appears. Choose Upgrade as the Request Type.

  3. Select the appropriate upgrade request type: preUpgrade.

  4. Fill out the upgrade request parameters:

    • Action: Enter start to retrieve the new version of the upgrader. (To check the status of the previous preUpgrade job, enter status instead.)

    • Type: Enter ISO. The upgrader packages are picked from the unpacked ISO image.

  5. Click Create Request.

    The new upgrade request appears in the Upgrade Jobs table. When the job has completed successfully, the upgrader is up-to-date and ready for use.

Upgrade the Compute Nodes

Caution:

Ensure that all steps to prepare the upgrade environment have been completed.

Note:

In case the ILOM also needs to be upgraded, you can integrate it into this procedure by executing the optional steps. The combined procedure eliminates the need to evacuate and reboot the same node twice.

The compute node upgrade ensures that the latest Oracle Linux kernel and user space packages are installed, as well as the ovm-agent package with appliance-specific optimizations. Compute nodes must be locked and upgraded one at a time; concurrent upgrades are not supported. After successful upgrade, when a compute node has rebooted, the administrator must manually remove the locks to allow the node to return to normal operation.

Obtaining a Host IP Address

From the Service CLI, compute nodes are upgraded one at a time, using each one's internal IP address as a command parameter. However, the locking commands use the compute node ID instead. To run all commands for a compute node upgrade you need both identifiers.

To obtain the host IP address and ID, as well as other information relevant to the upgrade procedure, use the Service CLI command provided in the following example. You can run the command as often as needed to check and confirm status as you proceed through the upgrade of all compute nodes.

PCA-ADMIN> list computeNode fields hostname,ipAddress,ilomIp,state,firmwareVersion,provisioningLocked,maintenanceLocked orderby hostname ASCENDING
Data:
  id                                     Hostname   Ip Address    ILOM Ip Address   State   Firmware Version           Provisioning Locked   Maintenance Locked
  --                                     --------   ----------    ---------------   -----   ----------------           -------------------   ------------------
  cf488903-fef8-4a51-8a41-c6990e4755c5   pcacn001   100.96.2.64   100.96.0.64       On      PCA Hypervisor:3.0.2-502   false                 false             
  42a7594d-1173-4dbd-4755-07810cc2d527   pcacn002   100.96.2.65   100.96.0.65       On      PCA Hypervisor:3.0.2-502   false                 false             
  bc0f37d5-ba77-423e-bc11-017704b47e59   pcacn003   100.96.2.66   100.96.0.66       On      PCA Hypervisor:3.0.2-502   false                 false             
  2e5ac527-01f5-4230-ae41-0522fcb57c9a   pcacn004   100.96.2.67   100.96.0.67       On      PCA Hypervisor:3.0.2-502   false                 false             
  5a6b61cf-7e99-4df2-87e4-b37c5fb0bfb8   pcacn005   100.96.2.68   100.96.0.68       On      PCA Hypervisor:3.0.2-502   false                 false             
  885f2aa4-f017-41e8-b2bc-e588cc0c6162   pcacn006   100.96.2.69   100.96.0.69       On      PCA Hypervisor:3.0.2-502   false                 false             

Using the Service CLI

  1. From the output you obtained with the compute node list command earlier, get the ID and the IP address of the compute node you intend to upgrade.

  2. Set the provisioning and maintenance locks for the compute node you are about to upgrade.

    Caution:

    As of version 3.0.2-b819070, the high-availability configuration of the Compute service might trigger automatic instance migrations, which can prevent you from successfully locking a compute node. For more information, refer to the following sections in the Hardware Administration chapter of the Oracle Private Cloud Appliance Administrator Guide:

    1. Disable provisioning for the compute node.

      PCA-ADMIN> provisioningLock id=cf488903-fef8-4a51-8a41-c6990e4755c5
      Status: Success
      JobId: 6ee78c8a-e227-4d31-a770-9b9c96085f3f
    2. Evacuate the compute node. Wait for the migration job to finish before proceeding to the next step.

      Note:

      In case physical resources are limited, compute instances will be migrated to other fault domains during compute node evacuation. However, the strict fault domain enforcement (Strict FD) function must be disabled.

      PCA-ADMIN> migrateVm id=cf488903-fef8-4a51-8a41-c6990e4755c5 force=true
      Status: Running
      JobId: 6f1e94bc-7d5b-4002-ada9-7d4b504a2599
      
      PCA-ADMIN> show Job id=6f1e94bc-7d5b-4002-ada9-7d4b504a2599
        Run State = Succeeded
    3. Lock the compute node for maintenance.

      PCA-ADMIN> maintenanceLock id=cf488903-fef8-4a51-8a41-c6990e4755c5
      Status: Success
      JobId: e46f6603-2af2-4df4-a0db-b15156491f88
    4. Optionally, rerun the compute node list command to confirm lock status. For example:

      PCA-ADMIN> list computeNode fields hostname,ipAddress,ilomIp,state,firmwareVersion,provisioningLocked,maintenanceLocked orderby hostname ASCENDING
      Data:
        id                                     Hostname   Ip Address    ILOM Ip Address   State   Firmware Version           Provisioning Locked   Maintenance Locked
        --                                     --------   ----------    ---------------   -----   ----------------           -------------------   ------------------
        cf488903-fef8-4a51-8a41-c6990e4755c5   pcacn001   100.96.2.64   100.96.0.64       On      PCA Hypervisor:3.0.2-502   true                  false             
        42a7594d-1173-4dbd-4755-07810cc2d527   pcacn002   100.96.2.65   100.96.0.65       On      PCA Hypervisor:3.0.2-502   false                 false             
        bc0f37d5-ba77-423e-bc11-017704b47e59   pcacn003   100.96.2.66   100.96.0.66       On      PCA Hypervisor:3.0.2-502   false                 false             
        2e5ac527-01f5-4230-ae41-0522fcb57c9a   pcacn004   100.96.2.67   100.96.0.67       On      PCA Hypervisor:3.0.2-502   false                 false             
        5a6b61cf-7e99-4df2-87e4-b37c5fb0bfb8   pcacn005   100.96.2.68   100.96.0.68       On      PCA Hypervisor:3.0.2-502   false                 false             
        885f2aa4-f017-41e8-b2bc-e588cc0c6162   pcacn006   100.96.2.69   100.96.0.69       On      PCA Hypervisor:3.0.2-502   false                 false             
  3. Optionally, upgrade the server ILOM first.

    1. Enter the ILOM upgrade command.

      Syntax (entered on a single line):

      upgradeIlom
      hostIp=<ilom-ip>

      Example:

      PCA-ADMIN> upgradeIlom hostIp=100.96.0.64
      Data:
        Service request has been submitted. Upgrade Job Id = 1620921089806-ilom-21480 Upgrade Request Id = UWS-732d6fce-9f06-4329-b972-d093bee40010
      
      PCA-ADMIN> getUpgradeJob upgradeJobId=1620921089806-ilom-21480
    2. Wait 5 minutes to allow the ILOM upgrade job to complete. Then proceed to the host upgrade.

  4. Enter the compute node upgrade command.

    Syntax (entered on a single line):

    upgradeCN 
    hostIp=<compute-node-ip>
    [optional] imageLocation=<path-to-iso>
    [optional] isoChecksum=<iso-file-checksum>

    Example:

    PCA-ADMIN> upgradeCN hostIp=100.96.2.64 \
    imageLocation=http://host.example.com/pca-<version>-<build>.iso \
    isoChecksum=240420cfb9478f6fd026f0a5fa0e998e086275fc45e207fb5631e2e99732e192e8e9d1b4c7f29026f0a5f58dadc4d792d0cfb0279962838e95a0f0a5fa31dca7
    Status: Success
    Data:
      Service request has been submitted. Upgrade Job Id = 1630938939109-compute-7545 Upgrade Request Id = UWS-61736806-7e5a-4648-9259-07c54c39cacb
  5. Use the request ID and the job ID to check the status of the upgrade process.

    PCA-ADMIN> getUpgradeJobs
      id                               upgradeRequestId                           commandName   result
      --                               ----------------                           -----------   ------
      1630938939109-compute-7545       UWS-61736806-7e5a-4648-9259-07c54c39cacb   compute       Passed
    
    PCA-ADMIN> getupgradejob upgradeJobId=1630938939109-compute-7545
    Data:
      Upgrade Request Id = UWS-61736806-7e5a-4648-9259-07c54c39cacb
      Name = compute
      Pid = 7545
      Host = pcamn02
      Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_compute_2021_09_26-06.35.39.log
    [...]
  6. When the compute node upgrade has completed successfully and the node has rebooted, release the locks.

    For more information, refer to the section "Performing Compute Node Operations". It can be found in the chapter Hardware Administration of the Oracle Private Cloud Appliance Administrator Guide.

    1. Release the maintenance lock.

      PCA-ADMIN> maintenanceUnlock id=cf488903-fef8-4a51-8a41-c6990e4755c5
      Status: Success
      JobId: 625af20e-4b49-4201-879f-41d4405314c7
    2. Release the provisioning lock.

      PCA-ADMIN> provisioningUnlock id=cf488903-fef8-4a51-8a41-c6990e4755c5
      Status: Success
      JobId: 523892e8-c2d4-403c-9620-2f3e94015b46
  7. Proceed to the next compute node and repeat this procedure.

    The output from the compute node list command indicates the current status. For example:

    PCA-ADMIN> list computeNode fields hostname,ipAddress,ilomIp,state,firmwareVersion,provisioningLocked,maintenanceLocked orderby hostname ASCENDING
    Data:
      id                                     Hostname   Ip Address    ILOM Ip Address   State   Firmware Version           Provisioning Locked   Maintenance Locked
      --                                     --------   ----------    ---------------   -----   ----------------           -------------------   ------------------
      cf488903-fef8-4a51-8a41-c6990e4755c5   pcacn001   100.96.2.64   100.96.0.64       On      PCA Hypervisor:3.0.2-640   false                 false             
      42a7594d-1173-4dbd-4755-07810cc2d527   pcacn002   100.96.2.65   100.96.0.65       On      PCA Hypervisor:3.0.2-640   false                 false             
      bc0f37d5-ba77-423e-bc11-017704b47e59   pcacn003   100.96.2.66   100.96.0.66       On      PCA Hypervisor:3.0.2-640   false                 false             
      2e5ac527-01f5-4230-ae41-0522fcb57c9a   pcacn004   100.96.2.67   100.96.0.67       On      PCA Hypervisor:3.0.2-640   false                 false             
      5a6b61cf-7e99-4df2-87e4-b37c5fb0bfb8   pcacn005   100.96.2.68   100.96.0.68       On      PCA Hypervisor:3.0.2-502   false                 false             
      885f2aa4-f017-41e8-b2bc-e588cc0c6162   pcacn006   100.96.2.69   100.96.0.69       On      PCA Hypervisor:3.0.2-502   false                 false             

Using the Service Web UI

  1. Set the provisioning and maintenance locks for the compute node you are about to upgrade. Ensure that no active compute instances are present on the node.

    Caution:

    As of version 3.0.2-b819070, the high-availability configuration of the Compute service might trigger automatic instance migrations, which can prevent you from successfully locking a compute node. For more information, refer to the following sections in the Hardware Administration chapter of the Oracle Private Cloud Appliance Administrator Guide:

    1. In the navigation menu, click Rack Units. In the Rack Units table, click the name of the compute node you want to upgrade to display its detail page.

    2. In the top-right corner of the compute node detail page, click Controls and select the Provisioning Lock command.

    3. When the provisioning lock has been set, click Controls again and select the Migrate All Vms command. The Compute service evacuates the compute node, meaning it migrates the running instances to other compute nodes.

      Note:

      In case physical resources are limited, compute instances will be migrated to other fault domains during compute node evacuation. However, the strict fault domain enforcement (Strict FD) function must be disabled.

    4. When compute node evacuation is complete, click Controls again and select the Maintenance Lock command. This command might fail if instance migrations are in progress. Wait a few minutes and retry.

  2. In the navigation menu, click Upgrade & Patching.

  3. Optionally, upgrade the server ILOM first.

    1. In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch. The Create Request window appears.

    2. Choose Upgrade as the Request Type. Select the appropriate upgrade request type: Upgrade ILOM.

      Fill out the server's assigned IP address in the ILOM network. This is an IP address in the internal 100.96.0.0/23 range.

    3. Click Create Request. The new upgrade request appears in the Upgrade Jobs table.

    4. Wait 5 minutes to allow the ILOM upgrade job to complete. Then proceed to the host upgrade.

  4. In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.

    The Create Request window appears. Choose Upgrade as the Request Type.

  5. Select the appropriate upgrade request type: Upgrade CN.

  6. Fill out the upgrade request parameters:

    • Host IP: Enter the compute node's assigned IP address in the internal administration network. This is an IP address in the internal 100.96.2.0/23 range.

    • Image Location: Enter the path to the location where the ISO image is stored.

    • ISO Checksum: Enter the checksum to verify the ISO image. It is stored alongside the ISO file.

    • Log Level: Optionally, select a specific log level for the upgrade log file. The default log level is "Information". For maximum detail, select "Debug".

    • Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.

  7. Click Create Request.

    The new upgrade request appears in the Upgrade Jobs table.

  8. When the compute node has been upgraded successfully, release the provisioning and maintenance locks.

    For more information, refer to the section "Performing Compute Node Operations". It can be found in the chapter Hardware Administration of the Oracle Private Cloud Appliance Administrator Guide.

    1. Open the compute node detail page.

    2. In the top-right corner of the compute node detail page, click Controls and select the Maintenance Unlock command.

    3. When the maintenance lock has been released, click Controls again and select the Provisioning Unlock command.

Upgrade the Management Node Cluster

Caution:

Ensure that all steps to prepare the upgrade environment have been completed.

Ensure that all compute nodes have been upgraded.

A full management node cluster upgrade is a convenient way to upgrade all the required components on all three management nodes using just a single command. As part of this process, the following components are upgraded, in this specific order:

  1. Host operating system

  2. Clustered MySQL database

  3. Secret service (including Etcd and Vault)

  4. Kubernetes container orchestration packages

  5. Containerized microservices

  6. Oracle Cloud Infrastructure images

Using the Service CLI

  1. Enter the command to start the full management node cluster upgrade.

    PCA-ADMIN> upgradeFullMN
    Data:
      Service request has been submitted. Upgrade Request Id = UWS-39329657-1051-4267-8c5a-9314f8e63a64
  2. Use the request ID to check the status of the upgrade process.

    As the full management node upgrade is a multi-component upgrade process, there are multiple upgrade jobs associated with the upgrade request. You can filter for those jobs based on the request ID. Using the job ID, you can drill down into the details of each upgrade job.

    PCA-ADMIN> getUpgradeJobs requestId=UWS-39329657-1051-4267-8c5a-9314f8e63a64
    Data:
      id                               upgradeRequestId                           commandName   result
      --                               ----------------                           -----------   ------
      1634579161548-oci-47486          UWS-39329657-1051-4267-8c5a-9314f8e63a64   oci           Passed
      1634578760906-platform-66082     UWS-39329657-1051-4267-8c5a-9314f8e63a64   platform      Passed
      1634578263434-kubernetes-63574   UWS-39329657-1051-4267-8c5a-9314f8e63a64   kubernetes    Passed
      1634578012353-vault-51696        UWS-39329657-1051-4267-8c5a-9314f8e63a64   vault         Passed
      1634577380954-etcd-46337         UWS-39329657-1051-4267-8c5a-9314f8e63a64   etcd          Passed
      1634577341291-mysql-40127        UWS-39329657-1051-4267-8c5a-9314f8e63a64   mysql         Passed
      1634576985926-host-36556         UWS-39329657-1051-4267-8c5a-9314f8e63a64   host          Passed
      1634576652071-host-27088         UWS-39329657-1051-4267-8c5a-9314f8e63a64   host          Passed
      1634576191050-host-24909         UWS-39329657-1051-4267-8c5a-9314f8e63a64   host          Passed
    
    PCA-ADMIN> getUpgradeJob upgradeJobId=1634576652071-host-27088
    Data:
      Upgrade Request Id = UWS-39329657-1051-4267-8c5a-9314f8e63a64
      Composition Id = 1
      Name = host
      Pid = 27088
      Host = pcamn02
      Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_host_os_2023_05_24-07.04.12.log
      Tasks 1 - Name = Validate Image Location
      Tasks 1 - Description = Verify that the image exists at the specified location and is correctly named
    [...]

    The output of the getUpgradeJob command provides detailed information about the tasks performed during the upgrade procedure. It displays descriptions, time stamps, duration, and success or failure. Whenever an upgrade operation fails, the command output indicates which task has failed. For in-depth troubleshooting you can search the log file at the location provided near the start of the command output.

    Note:

    After upgrade, the management nodes must all be rebooted for the changes to take effect. This step is part of the upgrade process, not a separate administrator operation.

    However, if the ILOMs are also upgraded, it happens after the host upgrade. In this case, the administrator must manually reboot each host for the ILOM changes to take effect.

Using the Service Web UI

  1. In the navigation menu, click Upgrade & Patching.

  2. In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.

    The Create Request window appears. Choose Upgrade as the Request Type.

  3. Select the appropriate upgrade request type.

    For a full management node upgrade, select Upgrade MN.

  4. Fill out the upgrade request parameters, if necessary:

    • Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.

    • Image Location: This parameter is deprecated.

    • ISO Checksum: This parameter is deprecated.

  5. Click Create Request.

    The new upgrade request appears in the Upgrade Jobs table.

    Note:

    After upgrade, the management nodes must all be rebooted for the changes to take effect. However, this step is part of the upgrade process, so no administrator action is required.

Upgrade Individual Components

A full management node cluster upgrade is a convenient way to upgrade all the required components on all three management nodes using just a single command. However, the process might be interrupted or an error might occur along the way. Instead of repeating the full management cluster upgrade, it might be more efficient to perform individual component upgrades in this situation.

Check the upgrade command output and the logs for any warnings or error messages. Determine at which stage the management node cluster upgrade failed, and resolve any issue that appears to have caused the failure. Proceed with individual component upgrades, starting with the component that was not upgraded successfully. Upgrade operations in the sections that follow, are listed in the correct order, exactly as they are performed during a management node cluster upgrade.

Upgrade the Management Node Operating System

The Oracle Linux host operating system of the management nodes must be upgraded one node at a time; a rolling upgrade of all management nodes is not possible. This upgrade process, which involves updating the kernel and system packages, must always be initiated from the management node that holds the cluster virtual IP. Thus, in a three-management-node cluster, when you have upgraded two management nodes, you must reassign the cluster virtual IP to one of the upgraded management nodes and run the final upgrade command from that node.

You must upgrade management nodes one at a time, using each one's internal IP address as a command parameter. To obtain the host IP addresses, use the Service CLI command show ManagementNode name=<node_name> and look for the Ip Address in the output.

It is assumed that the ISO image has already been unpacked on shared storage during the upgrade preparation stages. Therefore, the ISO path and checksum are not considered mandatory parameters for the upgrade command.

  1. Get the IP address of the management node for which you intend to upgrade the host operating system.

  2. Run the Service CLI from the management node that holds the management cluster virtual IP.

    1. Log on to one of the management nodes and check the status of the cluster.

      # ssh root@pcamn01
      # pcs status
      Cluster name: mncluster
      Stack: corosync
      Current DC: pcamn02 (version 1.1.23-1.0.1.el7-9acf116022) - partition with quorum
      
      Online: [ pcamn01 pcamn02 pcamn03 ]
      
      Full list of resources:
      
       scsi_fencing         (stonith:fence_scsi):          Stopped (disabled)
       Resource Group: mgmt-rg
           vip-mgmt-int     (ocf::heartbeat:IPaddr2):      Started    pcamn02
           vip-mgmt-host    (ocf::heartbeat:IPaddr2):      Started    pcamn02
           vip-mgmt-ilom    (ocf::heartbeat:IPaddr2):      Started    pcamn02
           vip-mgmt-lb      (ocf::heartbeat:IPaddr2):      Started    pcamn02
           vip-mgmt-ext     (ocf::heartbeat:IPaddr2):      Started    pcamn02
           l1api            (systemd:l1api):               Started    pcamn02
           haproxy          (ocf::heartbeat:haproxy):      Started    pcamn02
           pca-node-state   (systemd:pca_node_state):      Started    pcamn02
           dhcp             (ocf::heartbeat:dhcpd):        Started    pcamn02
           hw-monitor       (systemd:hw_monitor):          Started    pcamn02
      
      Daemon Status:
        corosync: active/enabled
        pacemaker: active/enabled
        pcsd: active/enabled

      In this example, the command output indicates that the node with host name pcamn02 currently holds the cluster virtual IP.

    2. Log in to the management node with the virtual IP and launch the Service CLI.

      # ssh pcamn02
      # ssh admin@localhost -p 30006
      PCA-ADMIN>
  3. Enter the upgrade command.

    Syntax (entered on a single line):

    upgradeHost 
    hostIp=<management-node-ip>

    Example:

    PCA-ADMIN> upgradeHost hostIp=100.96.2.35
    Status: Success
    Data:
      Service request has been submitted. Upgrade Job Id = 1632990827394-host-56156 Upgrade Request Id = UWS-1a97a8d9-54ef-478d-a0c0-348a17ba6755
  4. Use the request ID and the job ID to check the status of the upgrade process.

    PCA-ADMIN> getUpgradeJobs
      id                               upgradeRequestId                           commandName   result
      --                               ----------------                           -----------   ------
      1632990827394-host-56156         UWS-1a97a8d9-54ef-478d-a0c0-348a17ba6755   host          Passed
    
    PCA-ADMIN> getUpgradeJob upgradeJobId=1632990827394-host-56156
    Status: Success
    Data:
      Upgrade Request Id = UWS-1a97a8d9-54ef-478d-a0c0-348a17ba6755
      Composition Id = 1
      Name = host
      Pid = 56156
      Host = pcamn02
      Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_host_os_2021_09_25-05.47.02.log
    [...]
  5. When the first management node host operating system upgrade has completed successfully, execute the same command for the next management node.

    PCA-ADMIN> upgradeHost hostIp=100.96.2.33
  6. When the second management node host operating system upgrade has completed successfully, exit the Service CLI and move the cluster virtual IP to one of the upgraded nodes.

    PCA-ADMIN> exit
    Connection to localhost closed.
    # pcs resource move mgmt-rg pcamn01
    # pcs status
    Cluster name: mncluster
    Stack: corosync
    [...]
     scsi_fencing   (stonith:fence_scsi):   Stopped (disabled)
     Resource Group: mgmt-rg
         vip-mgmt-int       (ocf::heartbeat:IPaddr2):       Started pcamn01
         vip-mgmt-host      (ocf::heartbeat:IPaddr2):       Started pcamn01
    [...]

    Moving the cluster virtual IP to another management node should only take a number of seconds.

  7. Log in to the management node with the virtual IP and launch the Service CLI to execute the host operating system upgrade for the final management node.

    # ssh pcamn01
    # ssh admin@localhost -p 30006
    PCA-ADMIN> upgradeHost hostIp=100.96.2.34

    When this upgrade has completed successfully, the operating system on all management nodes is up-to-date.

    Note:

    After upgrade, the management nodes must all be rebooted for the changes to take effect. However, this step is part of the upgrade process, so no administrator action is required.

Upgrade the MySQL Cluster Database

The MySQL Cluster database upgrade is a rolling upgrade: with one command the upgrade is executed on each of the three management nodes.

It is assumed that the ISO image has already been unpacked on shared storage during the upgrade preparation stages. Therefore, the ISO path and checksum are not considered mandatory parameters for the upgrade command.

  1. Enter the upgrade command.

    PCA-ADMIN> upgradeMySQL
    Status: Success
    Data:
      Service request has been submitted. Upgrade Job Id = 1632995409822-mysql-83013 Upgrade Request Id = UWS-77bc0c30-7ff5-4c50-ad09-6f96907e22e1
  2. Use the request ID and the job ID to check the status of the upgrade process.

    PCA-ADMIN> getUpgradeJobs
      id                               upgradeRequestId                           commandName   result
      --                               ----------------                           -----------   ------
      1632995409822-mysql-83013        UWS-77bc0c30-7ff5-4c50-ad09-6f96907e22e1   mysql         Passed
    
    PCA-ADMIN> getUpgradeJob upgradeJobId=1632995409822-mysql-83013
    Status: Success
    Data:
      Upgrade Request Id = UWS-77bc0c30-7ff5-4c50-ad09-6f96907e22e1
      Name = mysql
      Pid = 83013
      Host = pcamn01
      Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_mysql_cluster_2021_09_25-09.21.16.log
    [...]

Upgrade the Secret Service

The secret service contains two components that need to be upgraded separately in this particular order: first Etcd, then Vault. The Etcd and Vault upgrades are rolling upgrades: each upgrade is executed on all three management nodes with one command.

It is assumed that the ISO image has already been unpacked on shared storage during the upgrade preparation stages. Therefore, the ISO path and checksum are not considered mandatory parameters for the upgrade command.

  1. Enter the two upgrade commands. Wait until one upgrade is finished before entering the second command.

    PCA-ADMIN> upgradeEtcd
    Status: Success
    Data:
      Service request has been submitted. Upgrade Job Id = 1632826770954-etcd-26973 Upgrade Request Id = UWS-fec15d32-fc2b-48bd-9ae0-62f49587a284
    
    PCA-ADMIN> upgradeVault
    Status: Success
    Data:
      Service request has been submitted. Upgrade Job Id = 1632850933353-vault-16966 Upgrade Request Id = UWS-352df3d1-c21f-441b-8f6e-9381ac075906
  2. Use the request ID and the job ID to check the status of the upgrade process.

    PCA-ADMIN> getUpgradeJobs
      id                               upgradeRequestId                           commandName   result
      --                               ----------------                           -----------   ------
      1632850933353-vault-16966        UWS-352df3d1-c21f-441b-8f6e-9381ac075906   vault         Passed
      1632826770954-etcd-26973         UWS-fec15d32-fc2b-48bd-9ae0-62f49587a284   etcd          Passed
    
    PCA-ADMIN> getUpgradeJob upgradeJobId=1632850933353-vault-16966
    Status: Success
    Data:
      Upgrade Request Id = UWS-352df3d1-c21f-441b-8f6e-9381ac075906
      Name = vault
      Pid = 16966
      Host = pcamn02
      Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_vault_2021_09_25-10.38.25.log
    [...]

Upgrade the Kubernetes Cluster

The target release of this upgrade path does not contain new versions of the Kubernetes container orchestration packages. You may skip this component.

Upgrade the Microservices

The microservices upgrade covers both the internal services of the platform layer, and the administrative and user-level services exposed through the infrastructure services layer.

The containerized microservices have their own separate upgrade mechanism. A service is upgraded if a new Helm deployment chart and container image are found in the ISO image. When a new deployment chart is detected during the upgrade process, the pods running the services are restarted with the new container image.

It is assumed that the ISO image has already been unpacked on shared storage during the upgrade preparation stages. Therefore, the ISO path and checksum are not considered mandatory parameters for the upgrade command.

  1. Enter the upgrade command.

    PCA-ADMIN> upgradePlatform
    Status: Success
    Data:
      Service request has been submitted. Upgrade Job Id = 1632850650836-platform-68465 Upgrade Request Id = UWS-26dba234-9b52-426d-836c-ac11f37e717f
  2. Use the request ID and the job ID to check the status of the upgrade process.

    PCA-ADMIN> getUpgradeJobs
      id                               upgradeRequestId                           commandName   result
      --                               ----------------                           -----------   ------
      1632850650836-platform-68465     UWS-26dba234-9b52-426d-836c-ac11f37e717f   platform      Passed
    
    PCA-ADMIN> getUpgradeJob upgradeJobId=1632850650836-platform-68465
    Status: Success
    Data:
      Upgrade Request Id = UWS-26dba234-9b52-426d-836c-ac11f37e717f
      Name = kubernetes
      Pid = 68465
      Host = pcamn02
      Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_platform_services_2021_09_26-20.48.41.log
    [...]

Upgrade the Oracle Cloud Infrastructure Images

It is assumed that the compute images upgrade is performed after the microservices or platform upgrade. At this stage, the upgradeOCIImages command is available. The images are stored in the /nfs/shared_storage/oci_compute_images directory on the ZFS Storage Appliance. You can make them available for use in all existing tenancies with a single upgrade command.

An upgrade adds new Oracle Cloud Infrastructure Images to your environment, but it never removes any existing images. If you no longer need an image, you have the option to delete it using the deletePlatformImage command.

  1. Enter the upgrade command.

    PCA-ADMIN> upgradeOCIImages
    Status: Success
    Data:
      Service request has been submitted. Upgrade Job Id = 1680260388058-oci-38288 Upgrade Request Id = UWS-e601b085-d59d-4ae4-82a3-a09b446686ff
  2. Use the request ID and the job ID to check the status of the upgrade process.

    PCA-ADMIN> getUpgradeJobs
    Status: Success
    Data:
      id                              upgradeRequestId                           commandName   result
      --                              ----------------                           -----------   ------
      1680260388058-oci-38288         UWS-e601b085-d59d-4ae4-82a3-a09b446686ff   oci           Passed
      
    PCA-ADMIN> getUpgradeJob upgradeJobId=1680260388058-oci-38288
    Status: Success
    Data:
      Upgrade Request Id = UWS-e601b085-d59d-4ae4-82a3-a09b446686ff
      Name = oci
    [...]

Upgrade Component Firmware

Caution:

Ensure that all steps to prepare the upgrade environment have been completed.

Ensure that all compute nodes and the management node cluster have been upgraded.

Firmware is included in the ISO image for all component ILOMs, for the ZFS Storage Appliance, and for the switches. Select the instructions below for the component type you want to upgrade.

During the full management node cluster upgrade, an upgrade plan has been generated. You can use it to check the progress of the appliance upgrade and determine which firmware upgrades still need to be performed.

PCA-ADMIN> getUpgradePlan
Data:
  id            componentType   currentBuild    targetBuild     currentVersion                  targetVersion                   requireReboot   timeEstimation   requireUpgrade   impactedInfra
  --            -------------   ------------    -----------     --------------                  -------------                   -------------   --------------   --------------   -------------
  100.96.2.64   compute         3.0.2-b892153   3.0.2-b892153   3.0.2-640                       3.0.2-640                       false           20               false            compute
  100.96.2.65   compute         3.0.2-b892153   3.0.2-b892153   3.0.2-640                       3.0.2-640                       false           20               false            compute
  100.96.2.66   compute         3.0.2-b892153   3.0.2-b892153   3.0.2-640                       3.0.2-640                       false           20               false            compute
  100.96.2.33   host            3.0.2-b892153   3.0.2-b892153   oraclelinux-release-7.9-1.0.9   oraclelinux-release-7.9-1.0.9   false           35               false            host
  100.96.2.34   host            3.0.2-b892153   3.0.2-b892153   oraclelinux-release-7.9-1.0.9   oraclelinux-release-7.9-1.0.9   false           35               false            host
  100.96.2.35   host            3.0.2-b892153   3.0.2-b892153   oraclelinux-release-7.9-1.0.9   oraclelinux-release-7.9-1.0.9   false           35               false            host
  generic       mysql           3.0.2-b892153   3.0.2-b892153   8.0.28-1.1                      8.0.28-1.1                      false           15               false            host
  generic       vault           3.0.2-b892153   3.0.2-b892153   v1.7.1-3                        v1.7.1-3                        false           5                false            host
  generic       etcd            3.0.2-b892153   3.0.2-b892153   3.3.10                          3.3.10                          false           5                false            host
  generic       kubernetes      3.0.2-b892153   3.0.2-b892153   1.25.7-1                        1.25.7-1                        false           5                false            host,compute
  generic       platform        3.0.2-b892153   3.0.2-b892153   None                            None                            false           50               false            host,compute
  100.96.0.33   ilom            3.0.2-b799577   3.0.2-b892153   5.0.2.23                        5.0.2.23                        false           10               false            host,compute
  100.96.0.34   ilom            3.0.2-b799577   3.0.2-b892153   5.0.2.23                        5.0.2.23                        false           10               false            host,compute
  100.96.0.35   ilom            3.0.2-b799577   3.0.2-b892153   5.0.2.23                        5.0.2.23                        false           10               false            host,compute
  100.96.0.64   ilom            3.0.2-b799577   3.0.2-b892153   5.0.2.23                        5.0.2.23                        false           10               false            host,compute
  100.96.0.65   ilom            3.0.2-b799577   3.0.2-b892153   5.0.2.23                        5.0.2.23                        false           10               false            host,compute
  100.96.0.66   ilom            3.0.2-b799577   3.0.2-b892153   5.0.2.23                        5.0.2.23                        false           10               false            host,compute
  generic       zfssa           3.0.2-b799577   3.0.2-b892153   2013.06.05.8.48.1-2.48.5222.1   2013.06.05.8.57.1-2.57.5392.2   false           45               true             host,compute
  leaf          switch          3.0.2-b799577   3.0.2-b892153   10.2.3                          10.2.5                          false           60               true             host,compute
  mgmt          switch          3.0.2-b799577   3.0.2-b892153   10.2.3                          10.2.5                          false           60               true             host,compute
  spine         switch          3.0.2-b799577   3.0.2-b892153   10.2.3                          10.2.5                          false           60               true             host,compute

Upgrade ZFS Storage Appliance Operating Software

To upgrade the operating software of the appliance's ZFS Storage Appliance, you enter the upgrade command with no additional parameters. The IP addresses of the storage controllers and the location of the firmware package are known, and a single upgrade command initiates a rolling upgrade of both controllers. If a new ILOM firmware version is included for the two controllers, it will be installed as part of the ZFS Storage Appliance upgrade process.

Caution:

Ensure that no users are logged in to the ZFS Storage Appliance or the storage controller ILOMs during the upgrade process.

Do not make storage configuration changes while an upgrade is in progress. While controllers are running different software versions, configuration changes made to one controller are not propagated to its peer controller.

During firmware upgrade the storage controllers are placed in active/passive mode. They automatically return to active/active after the upgrade is completed.

Before You Begin

Before you initiate a ZFS Storage Appliance upgrade, you must disable the node state service to prevent errors in node states after the upgrade.
  1. From a management node, set the provisioning lock by issuing this command:
    pca-admin locks set system provisioning
  2. Perform the ZFS Storage Appliance upgrade using either the Service Web UI or the Service CLI procedure below.
  3. Release the provisioning lock.
    pca-admin locks unset system provisioning
  4. Confirm the lock state.
    pca-admin locks show system

Using the Service CLI

  1. Enter the upgrade command.

    PCA-ADMIN> upgradeZfssa
    Data:
      Service request has been submitted. Upgrade Job Id = 1632914107346-zfssa-83002 Upgrade Request Id = UWS-881af57f-5dfb-4c75-8026-9f00cf3eb7c9
  2. Use the request ID and the job ID to check the status of the upgrade process.

    PCA-ADMIN> getUpgradeJobs
      id                               upgradeRequestId                           commandName   result
      --                               ----------------                           -----------   ------
      1632914107346-zfssa-83002        UWS-881af57f-5dfb-4c75-8026-9f00cf3eb7c9   zfssa         Passed
    
    PCA-ADMIN> getUpgradeJob upgradeJobId=1632914107346-zfssa-83002
    Data:
      Upgrade Request Id = UWS-881af57f-5dfb-4c75-8026-9f00cf3eb7c9
      Name = zfssa
      Pid = 83002
      Host = pcamn02
      Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_zfssa_ak_2021_09_29-11.15.07.log
    [...]

Using the Service Web UI

  1. In the navigation menu, click Upgrade & Patching.

  2. In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.

    The Create Request window appears. Choose Upgrade as the Request Type.

  3. Select the appropriate upgrade request type: Upgrade Zfssa.

  4. Fill out the upgrade request parameters:

    • Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.

    • Image Location: This parameter is deprecated.

    • Log Level: Optionally, select a specific log level for the upgrade log file. The default log level is "Information". For maximum detail, select "Debug".

  5. Click Create Request.

    The new upgrade request appears in the Upgrade Jobs table.

Upgrade ILOMs

ILOM upgrades can be applied to management nodes and compute nodes; the firmware packages might be different per component type. You must upgrade ILOMs one at a time, using each one's internal IP address as a command parameter.

Caution:

You must NOT upgrade the ILOM of the management node that holds the management virtual IP address, and thus the primary role in the cluster. To determine which management node has the primary role in the cluster, and make another node the primary, use the following Service CLI commands:

PCA-ADMIN> getPrimaryMgmtNode
  status = success
  data = pcamn01

PCA-ADMIN> updatePrimaryNode node=pcamn02
Data:
  status = success
  message = Successfully issued update primary node command

PCA-ADMIN> getPrimaryMgmtNode
  status = success
  data = pcamn02

Obtaining an ILOM IP Address

You can find the IP address of a server ILOM using either the Service CLI or the Service Web UI.

  • In the Service CLI you obtain the ID, IP addresses, and other key information about compute nodes or management nodes with the following commands:

    PCA-ADMIN> list managementNode fields hostname,ipAddress,ilomIp,state,firmwareVersion orderby hostname ASCENDING
    Data:
      id                                     Hostname   Ip Address    ILOM Ip Address   State   Firmware Version
      --                                     --------   ----------    ---------------   -----   ----------------
      22ae47d8-a57a-433c-988d-df62fd3548e1   pcamn01    100.96.2.33   100.96.0.33       On      5.0.2.23
      042693b6-3ddb-4fb0-914d-f3deea838c8f   pcamn02    100.96.2.34   100.96.0.34       On      5.0.2.23
      bd2563b8-e310-4fca-ba1b-0e19b8040fc6   pcamn03    100.96.2.35   100.96.0.35       On      5.0.2.23
    
    PCA-ADMIN> list computeNode fields hostname,ipAddress,ilomIp,state,firmwareVersion,provisioningLocked,maintenanceLocked orderby hostname ASCENDING
    Data:
      id                                     Hostname   Ip Address    ILOM Ip Address   State   Firmware Version           Provisioning Locked   Maintenance Locked
      --                                     --------   ----------    ---------------   -----   ----------------           -------------------   ------------------
      cf488903-fef8-4a51-8a41-c6990e4755c5   pcacn001   100.96.2.64   100.96.0.64       On      PCA Hypervisor:3.0.2-640   false                 false             
      42a7594d-1173-4dbd-4755-07810cc2d527   pcacn002   100.96.2.65   100.96.0.65       On      PCA Hypervisor:3.0.2-640   false                 false             
      bc0f37d5-ba77-423e-bc11-017704b47e59   pcacn003   100.96.2.66   100.96.0.66       On      PCA Hypervisor:3.0.2-640   false                 false             
      2e5ac527-01f5-4230-ae41-0522fcb57c9a   pcacn004   100.96.2.67   100.96.0.67       On      PCA Hypervisor:3.0.2-640   false                 false             
      5a6b61cf-7e99-4df2-87e4-b37c5fb0bfb8   pcacn005   100.96.2.68   100.96.0.68       On      PCA Hypervisor:3.0.2-640   false                 false             
      885f2aa4-f017-41e8-b2bc-e588cc0c6162   pcacn006   100.96.2.69   100.96.0.69       On      PCA Hypervisor:3.0.2-640   false                 false             
  • When using the Service Web UI:

    1. In the navigation menu, click Rack Units.

    2. Click on the name of the component you are patching.

    3. Select the Rack Unit Information tab.

    4. Record the IP Address listed under ILOM IPs.

Using the Service CLI

  1. Get the IP address of the ILOM for which you intend to upgrade the firmware.

  2. Enter the upgrade command.

    Syntax (entered on a single line):

    upgradeIlom
    hostIp=<ilom-ip>

    Example:

    PCA-ADMIN> upgradeIlom hostIp=100.96.0.66
    Status: Success
    Data:
      Service request has been submitted. Upgrade Job Id = 1620921089806-ilom-21480 Upgrade Request Id = UWS-732d6fce-9f06-4329-b972-d093bee40010
  3. Use the request ID and the job ID to check the status of the upgrade process.

    PCA-ADMIN> getUpgradeJobs
      id                               upgradeRequestId                           commandName   result
      --                               ----------------                           -----------   ------
      1620921089806-ilom-21480         UWS-732d6fce-9f06-4329-b972-d093bee40010   ilom          Passed
    
    PCA-ADMIN> getUpgradeJob upgradeJobId=1620921089806-ilom-21480
    Data:
      Upgrade Request Id = UWS-732d6fce-9f06-4329-b972-d093bee40010
      Name = ilom
      Pid = 21480
      Host = pcamn02
      Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_ilom_firmware_2021_09_24-11.18.31.log
    [...]

Using the Service Web UI

  1. In the navigation menu, click Upgrade & Patching.

  2. In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.

    The Create Request window appears. Choose Upgrade as the Request Type.

  3. Select the appropriate upgrade request type: Upgrade ILOM.

  4. Fill out the upgrade request parameters:

    • Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.

    • Host IP: Enter the component's assigned IP address in the ILOM network. This is an IP address in the internal 100.96.0.0/23 range.

    • Image Location: This parameter is deprecated.

    • Log Level: Optionally, select a specific log level for the upgrade log file. The default log level is "Information". For maximum detail, select "Debug".

  5. Click Create Request.

    The new upgrade request appears in the Upgrade Jobs table.

Upgrade Switch Software

The appliance rack contains three categories of Cisco Nexus switches: a management switch, two leaf switches, and two spine switches. They all run the same Cisco NX-OS network operating software. You must perform the upgrades in this order: leaf switches first, then spine switches, and finally the management switch. Only one command per switch category is required, meaning that the leaf switches and the spine switches are upgraded in pairs.

Some versions of the network operating software consist of two files: a binary file and an additional EPLD (electronic programmable logic device) image. Both are automatically retrieved from their designated location during the upgrade process, and applied in the correct order.

Using the Service CLI

  1. Determine the type of switch to upgrade (spine, leaf, management).

  2. Enter the upgrade command.

    Syntax (entered on a single line):

    upgradeSwitch 
    switchType=[MGMT | SPINE | LEAF]

    Example:

    PCA-ADMIN> upgradeSwitch switchType=LEAF
    Data:
      Service request has been submitted. Upgrade Job Id = 1630511206512-cisco-20299 Upgrade Request Id = UWS-44688fe5-b4f8-407f-a1b5-8cd1b685c2c3
  3. Use the request ID and the job ID to check the status of the upgrade process.

    PCA-ADMIN> getUpgradeJobs
      id                               upgradeRequestId                           commandName   result
      --                               ----------------                           -----------   ------
      1630511206512-cisco-20299        UWS-44688fe5-b4f8-407f-a1b5-8cd1b685c2c3   cisco         Passed
    
    PCA-ADMIN> getupgradeJob upgradeJobId=1630511206512-cisco-20299
    Data:
      Upgrade Request Id = UWS-44688fe5-b4f8-407f-a1b5-8cd1b685c2c3
      Name = cisco
      Pid = 20299
      Host = pcamn02
      Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_cisco_firmware_2021_09_24-14.46.46.log
    [...]

Using the Service Web UI

  1. In the navigation menu, click Upgrade & Patching.

  2. In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.

    The Create Request window appears. Choose Upgrade as the Request Type.

  3. Select the appropriate upgrade request type: Upgrade Switch.

  4. Fill out the upgrade request parameters:

    • Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.

    • EPLD: This parameter is deprecated.

    • Image Location: This parameter is deprecated.

    • Log Level: Optionally, select a specific log level for the upgrade log file. The default log level is "Information". For maximum detail, select "Debug".

    • Switch Type: Select the switch type you intend to upgrade. The upgrade order is as follows: leaf switches first, then spine switches, and finally the management switch.

  5. Click Create Request.

    The new upgrade request appears in the Upgrade Jobs table.

  6. When the upgrade has completed successfully, but other switches in the system still need to be upgraded, repeat this procedure for any other type of switch that requires upgrading.