1. Upgrade Guide
  2. Upgrading Individual Components
  3. Upgrading Firmware

Upgrading Firmware

Caution:

Ensure that all preparation steps for system upgrade have been completed. For instructions, see Preparing the Upgrade Environment.

Firmware is included in the ISO image for all component ILOMs, for the ZFS Storage Appliance, and for the switches. Select the instructions below for the component type you want to upgrade.

If you have completed the full management node upgrade to software version 3.0.2-b892153 or later, an upgrade plan has been generated. You can use it to check the progress of the appliance upgrade and determine which firmware upgrades still need to be performed.

Obtaining an ILOM IP Address

Using the Service Web UI

  1. In the navigation menu, click Rack Units.

  2. Click on the name of the component you are patching.

  3. Select the Rack Unit Information tab.

  4. Record the IP Address listed under ILOM IPs.

Using the Service CLI

Note:

If the appliance is running software version 3.0.2-b892153 or earlier, use the third option from this list.

  • To list the ILOM IP addresses of all management nodes or compute nodes, use these commands: 

    PCA-ADMIN> getCNIloms
Data:
  status = success
  data 1 = 100.96.0.66
  data 2 = 100.96.0.64
  data 3 = 100.96.0.65

PCA-ADMIN> getMNIloms
Data:
  status = success
  data 1 = 100.96.0.33
  data 2 = 100.96.0.34
  data 3 = 100.96.0.35

  • To obtain the ILOM IP address that corresponds with a specific node host name, use the following command: 

    PCA-ADMIN> getServerILOMIP hostName=<node_name>

    For example:

    PCA-ADMIN> getServerILOMIP hostName=pcacn002
Data:
  status = success
  data = 100.96.0.65

PCA-ADMIN> getServerILOMIP hostName=pcamn03
Data:
  status = success
  data = 100.96.0.35

  • To obtain the ID, IP addresses, and other key information about compute nodes or management nodes, use the following commands:

    PCA-ADMIN> list managementNode fields hostname,ipAddress,ilomIp,state,firmwareVersion orderby hostname ASCENDING
Data:
  id                                     Hostname   Ip Address    ILOM Ip Address   State   Firmware Version
  --                                     --------   ----------    ---------------   -----   ----------------
  22ae47d8-a57a-433c-988d-df62fd3548e1   pcamn01    100.96.2.33   100.96.0.33       On      5.0.2.23
  042693b6-3ddb-4fb0-914d-f3deea838c8f   pcamn02    100.96.2.34   100.96.0.34       On      5.0.2.23
  bd2563b8-e310-4fca-ba1b-0e19b8040fc6   pcamn03    100.96.2.35   100.96.0.35       On      5.0.2.23

PCA-ADMIN> list computeNode fields hostname,ipAddress,ilomIp,state,firmwareVersion,provisioningLocked,maintenanceLocked orderby hostname ASCENDING
Data:
  id                                     Hostname   Ip Address    ILOM Ip Address   State   Firmware Version           Provisioning Locked   Maintenance Locked
  --                                     --------   ----------    ---------------   -----   ----------------           -------------------   ------------------
  cf488903-fef8-4a51-8a41-c6990e4755c5   pcacn001   100.96.2.64   100.96.0.64       On      PCA Hypervisor:3.0.2-681   false                 false             
  42a7594d-1173-4dbd-4755-07810cc2d527   pcacn002   100.96.2.65   100.96.0.65       On      PCA Hypervisor:3.0.2-681   false                 false             
  bc0f37d5-ba77-423e-bc11-017704b47e59   pcacn003   100.96.2.66   100.96.0.66       On      PCA Hypervisor:3.0.2-681   false                 false             
  2e5ac527-01f5-4230-ae41-0522fcb57c9a   pcacn004   100.96.2.67   100.96.0.67       On      PCA Hypervisor:3.0.2-681   false                 false             
  5a6b61cf-7e99-4df2-87e4-b37c5fb0bfb8   pcacn005   100.96.2.68   100.96.0.68       On      PCA Hypervisor:3.0.2-681   false                 false             
  885f2aa4-f017-41e8-b2bc-e588cc0c6162   pcacn006   100.96.2.69   100.96.0.69       On      PCA Hypervisor:3.0.2-681   false                 false

Upgrading ILOMs

Note:

In case a server node needs both a host and ILOM upgrade, you can avoid having to reboot the same node twice by combining the two upgrades. Instructions are provided in the following sections:

ILOM upgrades can be applied to management nodes and compute nodes; the firmware packages might be different per component type. You must upgrade ILOMs one at a time, using each one's internal IP address as a command parameter.

Caution:

You must NOT upgrade the ILOM of the management node that holds the management virtual IP address, and thus the primary role in the cluster. To determine which management node has the primary role in the cluster, and make another node the primary, use the following Service CLI commands: 

PCA-ADMIN> getPrimaryMgmtNode
  status = success
  data = pcamn01

PCA-ADMIN> updatePrimaryNode node=pcamn02
Data:
  status = success
  message = Successfully issued update primary node command

PCA-ADMIN> getPrimaryMgmtNode
  status = success
  data = pcamn02

Using the Service Web UI

  1. In the navigation menu, click Upgrade & Patching.

  2. In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.

    The Create Request window appears. Choose Upgrade as the Request Type.

  3. Select the appropriate upgrade request type: Upgrade ILOM.

  4. Fill out the upgrade request parameters:

    • Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.

    • Host IP: Enter the component's assigned IP address in the ILOM network. This is an IP address in the internal 100.96.0.0/23 range.

    • Image Location: This parameter is deprecated.

    • Log Level: Optionally, select a specific log level for the upgrade log file. The default log level is "Information". For maximum detail, select "Debug".

  5. Click Create Request.

    The new upgrade request appears in the Upgrade Jobs table.

Using the Service CLI

  1. Get the IP address of the ILOM for which you intend to upgrade the firmware.

  2. Enter the upgrade command.

    Syntax (entered on a single line):

    upgradeIlom
hostIp=<ilom-ip>

    Example:

    PCA-ADMIN> upgradeIlom hostIp=100.96.0.66
Status: Success
Data:
  Service request has been submitted. Upgrade Job Id = 1620921089806-ilom-21480 Upgrade Request Id = UWS-732d6fce-9f06-4329-b972-d093bee40010

  3. Use the request ID and the job ID to check the status of the upgrade process.

    PCA-ADMIN> getUpgradeJobs
  id                               upgradeRequestId                           commandName   result
  --                               ----------------                           -----------   ------
  1620921089806-ilom-21480         UWS-732d6fce-9f06-4329-b972-d093bee40010   ilom          Passed
  1632926926773-host-32993         UWS-fef3b663-45b7-4177-a041-26f73e68848d   host          Passed
  1632990827394-host-56156         UWS-1a97a8d9-54ef-478d-a0c0-348a17ba6755   host          Passed
  1632990493570-host-6646          UWS-4c78f3ef-ac42-4f32-9483-bb43a309faa3   host          Passed

PCA-ADMIN> getUpgradeJob upgradeJobId=1620921089806-ilom-21480
Status: Success
Data:
  Upgrade Request Id = UWS-732d6fce-9f06-4329-b972-d093bee40010
  Name = ilom
  Start Time = 2021-09-24 11:18:32
  End Time = 2021-09-24 11:21:18
  Pid = 21480
  Host = pcamn02
  Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_ilom_firmware_2021_09_24-11.18.31.log
  Arguments = {"verify_only":false,"upgrade":false,"diagnostics":false,"host_ip":"100.96.0.66","result_override":null,"log_level":null,"switch_type":null,"precheck_status":false,"task_time":0,"fail_halt":false,"fail_upgrade":null,"component_names":null,"upgrade_to":null,"image_location":"file:///nfs/shared_storage/pca_firmware/X9-2/.../ILOM-5_0_2_21_r140740-ORACLE_SERVER_X9-2-rom.pkg","epld_image_location":null,"expected_iso_checksum":null,"checksum":null,"composition_id":null,"request_id":"UWS-732d6fce-9f06-4329-b972-d093bee40010","display_task_plan":false,"dry_run_tasks":false}
  Status = Passed
  Execution Time(sec) = 166
  Tasks 1 - Name = Validate Image Location
  Tasks 1 - Description = Verify that the image exists at the specified location and is correctly named
  Tasks 1 - Time = 2021-09-24T11:18:32
[...]

At the end of the upgrade, the ILOM itself is rebooted automatically. However, the server component also needs to be rebooted for all changes to take effect. Wait 5 minutes to allow the ILOM workflow to complete first.

For minimum operational impact, schedule the compute node and management node reboot operations after all ILOMs have been patched. Take into account that rebooting the compute nodes requires migrating the compute instances. For more information, refer to "Performing Compute Node Operations" in the Hardware Administration chapter of the Oracle Private Cloud Appliance Administrator Guide.

Caution:

Always verify the cluster state before rebooting a management node. Consult the Oracle Private Cloud Appliance Release Notes for more information: refer to the known issue "Rebooting a Management Node while the Cluster State is Unhealthy Causes Platform Integrity Issues".

Upgrading the ZFS Storage Appliance Operating Software

To upgrade the operating software of the appliance's ZFS Storage Appliance, you enter the upgrade command with no additional parameters. The IP addresses of the storage controllers and the location of the firmware package are known, and a single upgrade command initiates a rolling upgrade of both controllers. If a new ILOM firmware version is included for the two controllers, it will be installed as part of the ZFS Storage Appliance upgrade process.

Caution:

Ensure that no users are not logged in to the ZFS Storage Appliance or the storage controller ILOMs during the upgrade process.

Do not make storage configuration changes while an upgrade is in progress. While controllers are running different software versions, configuration changes made to one controller are not propagated to its peer controller.

During firmware upgrade the storage controllers are placed in active/passive mode. They automatically return to active/active after the upgrade is completed.

Before You Begin

Before you initiate a ZFS Storage Appliance upgrade, you must disable the node state service to prevent errors in node states after the upgrade.
  1. From a management node, set the provisioning lock by issuing this command: 
    pca-admin locks set system provisioning
  2. Perform the ZFS Storage Appliance upgrade using either the Service Web UI or the Service CLI procedure below.
  3. Release the provisioning lock.
    pca-admin locks unset system provisioning
  4. Confirm the lock state.
    pca-admin locks show system

Using the Service Web UI

  1. In the navigation menu, click Upgrade & Patching.

  2. In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.

    The Create Request window appears. Choose Upgrade as the Request Type.

  3. Select the appropriate upgrade request type: Upgrade Zfssa.

  4. Fill out the upgrade request parameters:

    • Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.

    • Image Location: This parameter is deprecated.

    • Log Level: Optionally, select a specific log level for the upgrade log file. The default log level is "Information". For maximum detail, select "Debug".

  5. Click Create Request.

    The new upgrade request appears in the Upgrade Jobs table.

Using the Service CLI

  1. Enter the upgrade command.

    PCA-ADMIN> upgradeZfssa
Command: upgradeZfssa
Status: Success
Time: 2021-09-27 11:15:07,453 UTC
Data:
  Service request has been submitted. Upgrade Job Id = 1632914107346-zfssa-83002 Upgrade Request Id = UWS-881af57f-5dfb-4c75-8026-9f00cf3eb7c9

  2. Use the request ID and the job ID to check the status of the upgrade process.

    PCA-ADMIN> getUpgradeJobs
  id                               upgradeRequestId                           commandName   result
  --                               ----------------                           -----------   ------
  1632914107346-zfssa-83002        UWS-881af57f-5dfb-4c75-8026-9f00cf3eb7c9   zfssa         Passed
  1632926926773-host-32993         UWS-fef3b663-45b7-4177-a041-26f73e68848d   host          Passed
  1632990827394-host-56156         UWS-1a97a8d9-54ef-478d-a0c0-348a17ba6755   host          Passed
  1632990493570-host-6646          UWS-4c78f3ef-ac42-4f32-9483-bb43a309faa3   host          Passed

PCA-ADMIN> getUpgradeJob upgradeJobId=1632914107346-zfssa-83002
Command: getUpgradeJob upgradeJobId=1632914107346-zfssa-83002
Status: Success
Time: 2021-09-27 11:42:10,729 UTC
Data:
  Upgrade Request Id = UWS-881af57f-5dfb-4c75-8026-9f00cf3eb7c9
  Name = zfssa
  Start Time = 2021-09-29T11:15:07
  End Time = 2021-09-29T11:26:42
  Pid = 83002
  Host = pcamn02
  Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_zfssa_ak_2021_09_29-11.15.07.log
  Arguments = {"verify_only":false,"upgrade":false,"diagnostics":false,"host_ip":null,"result_override":null,"log_level":null,"switch_type":null,"precheck_status":false,"task_time":0,"fail_halt":false,"fail_upgrade":null,"component_names":null,"upgrade_to":null,"image_location":"file:///nfs/shared_storage/pca_firmware/zfs/ak-nas-2021.08.27-1.0x-nondebug.pkg","epld_image_location":null,"expected_iso_checksum":null,"checksum":null,"composition_id":null,"request_id":"UWS-881af57f-5dfb-4c75-8026-9f00cf3eb7c9","display_task_plan":false,"dry_run_tasks":false}
  Status = Passed
  Execution Time(sec) = 697
  Tasks 1 - Name = Validate Image Location
  Tasks 1 - Description = Verify that the image exists at the specified location and is correctly named
  Tasks 1 - Time = 2021-09-29T11:15:08
[...]

Upgrading the Switch Software

The appliance rack contains three categories of Cisco Nexus switches: a management switch, two leaf switches, and two spine switches. They all run the same Cisco NX-OS network operating software. You must perform the upgrades in this order: leaf switches first, then spine switches, and finally the management switch. Only one command per switch category is required, meaning that the leaf switches and the spine switches are upgraded in pairs.

Some versions of the network operating software consist of two files: a binary file and an additional EPLD (electronic programmable logic device) image. Both are automatically retrieved from their designated location during the upgrade process, and applied in the correct order.

Using the Service Web UI

  1. In the navigation menu, click Upgrade & Patching.

  2. In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.

    The Create Request window appears. Choose Upgrade as the Request Type.

  3. Select the appropriate upgrade request type: Upgrade Switch.

  4. Fill out the upgrade request parameters:

    • Advanced Options JSON: Optionally, add a JSON string to provide additional command parameters.

    • EPLD: This parameter is deprecated.

    • Image Location: This parameter is deprecated.

    • Log Level: Optionally, select a specific log level for the upgrade log file. The default log level is "Information". For maximum detail, select "Debug".

    • Switch Type: Select the switch type you intend to upgrade. The preferred upgrade order is as follows: leaf switches first, then spine switches, and finally the management switch.

  5. Click Create Request.

    The new upgrade request appears in the Upgrade Jobs table.

  6. When the upgrade has completed successfully, but other switches in the system still need to be upgraded, repeat this procedure for any other type of switch that requires upgrading.

Using the Service CLI

  1. Determine the type of switch to upgrade (spine, leaf, management).

  2. Enter the upgrade command.

    Syntax (entered on a single line):

    upgradeSwitch 
switchType=[MGMT | SPINE | LEAF]

    Example:

    PCA-ADMIN> upgradeSwitch switchType=LEAF
Command: upgradeSwitch switchType=LEAF
Status: Success
Time: 2021-09-24 14:16:54,704 UTC
Data:
  Service request has been submitted. Upgrade Job Id = 1630511206512-cisco-20299 Upgrade Request Id = UWS-44688fe5-b4f8-407f-a1b5-8cd1b685c2c3

  3. Use the request ID and the job ID to check the status of the upgrade process.

    PCA-ADMIN> getUpgradeJobs
  id                               upgradeRequestId                           commandName   result
  --                               ----------------                           -----------   ------
  1632914107346-zfssa-83002        UWS-881af57f-5dfb-4c75-8026-9f00cf3eb7c9   zfssa         Passed
  1630511206512-cisco-20299        UWS-44688fe5-b4f8-407f-a1b5-8cd1b685c2c3   cisco         Passed
  1620921089806-ilom-21480         UWS-732d6fce-9f06-4329-b972-d093bee40010   ilom          Passed

PCA-ADMIN> getupgradeJob upgradeJobId=1630511206512-cisco-20299
Command: getupgradeJob upgradeJobId=1630511206512-cisco-20299
Status: Success
Time: 2021-09-24 15:48:08,455 UTC
Data:
  Upgrade Request Id = UWS-44688fe5-b4f8-407f-a1b5-8cd1b685c2c3
  Name = cisco
  Start Time = 2021-09-24T14:46:46
  End Time = 2021-09-24T14:59:44
  Pid = 20299
  Host = pcamn02
  Log File = /nfs/shared_storage/pca_upgrader/log/pca-upgrader_cisco_firmware_2021_09_24-14.46.46.log
  Arguments = {"verify_only":false,"upgrade":false,"diagnostics":false,"host_ip":null,"result_override":null,"log_level":null,"switch_type":"LEAF","precheck_status":false,"task_time":0,"fail_halt":false,"fail_upgrade":null,"component_names":null,"upgrade_to":null,"image_location":"file:///nfs/shared_storage/pca_firmware/network/cisco/nxos.9.3.2.bin","epld_image_location":null,"expected_iso_checksum":null,"checksum":null,"composition_id":null,"request_id":"UWS-44688fe5-b4f8-407f-a1b5-8cd1b685c2c3","display_task_plan":false,"dry_run_tasks":false}
  Status = Passed
  Execution Time(sec) = 777
  Tasks 1 - Name = Validate Image Location
  Tasks 1 - Description = Verify that the image exists at the specified location and is correctly named
  Tasks 1 - Time = 2021-09-24T14:46:47
[...]