Patching a Compute Node

Caution:

Ensure that all preparation steps for system patching have been completed. For instructions, see Prepare for Appliance Patching.

When patching to appliance software version 3.0.2-b1081557 or later, the ZFS Storage Appliance firmware must be patched before all other components. For more information, see Check Upgrade Plan Status and Progress.

The compute node patching ensures that the latest Oracle Linux kernel and user space packages are installed, as well as the ovm-agent package with appliance-specific optimizations. Compute nodes must be provisioned and locked, then patched one at a time, concurrent patches are not supported. After a successful patch, when a compute node has rebooted, the administrator must manually remove the locks to allow the node to return to normal operation.

Ensure synchronization of the mirror on the shared storage is complete prior to compute node patching by issuing the syncUpstreamUlnMirror command. For more information, see Update and Synchronize with ULN Mirror.

Note:

In case the ILOM also needs to be patched, you can integrate it into this procedure by executing the optional steps. The combined procedure eliminates the need to evacuate and reboot the same node twice.

Note:

In software versions 3.0.2-b892153 and later all patch operations are based on the upgrade plan, which is generated when the pre-upgrade command is executed. For more information, see Set Up New Software Sources for Patching. When a component is already at the required version, the patch operation is skipped. However, patching with the same version can be forced using the Service Web UI or Service CLI command option (force=True), if necessary.

Obtaining a Host IP Address

From the Service CLI, compute nodes are patched one at a time, using each one's internal IP address as a command parameter. However, the locking commands use the compute node ID instead. To run all commands for a compute node patch procedure you need both identifiers.

To obtain the host IP address and ID, as well as other information relevant to the patch procedure, use the Service CLI command provided in the following example. You can run the command as often as needed to check and confirm status as you proceed through the upgrade of all compute nodes.

PCA-ADMIN> list computeNode fields hostname,ipAddress,ilomIp,state,firmwareVersion,provisioningLocked,maintenanceLocked orderby hostname ASCENDING
Data:
  id                                     Hostname   Ip Address    ILOM Ip Address   State   Firmware Version           Provisioning Locked   Maintenance Locked
  --                                     --------   ----------    ---------------   -----   ----------------           -------------------   ------------------
  cf488903-fef8-4a51-8a41-c6990e4755c5   pcacn001   100.96.2.64   100.96.0.64       On      PCA Hypervisor:3.0.2-681   false                 false             
  42a7594d-1173-4dbd-4755-07810cc2d527   pcacn002   100.96.2.65   100.96.0.65       On      PCA Hypervisor:3.0.2-681   false                 false             
  bc0f37d5-ba77-423e-bc11-017704b47e59   pcacn003   100.96.2.66   100.96.0.66       On      PCA Hypervisor:3.0.2-681   false                 false             
  2e5ac527-01f5-4230-ae41-0522fcb57c9a   pcacn004   100.96.2.67   100.96.0.67       On      PCA Hypervisor:3.0.2-681   false                 false             
  5a6b61cf-7e99-4df2-87e4-b37c5fb0bfb8   pcacn005   100.96.2.68   100.96.0.68       On      PCA Hypervisor:3.0.2-681   false                 false             
  885f2aa4-f017-41e8-b2bc-e588cc0c6162   pcacn006   100.96.2.69   100.96.0.69       On      PCA Hypervisor:3.0.2-681   false                 false

Monitoring Displaced Instances

During compute node upgrade or patching, no active compute instances can be present, so the node must be evacuated and locked for maintenance. To evacuate a compute node, the Compute Service live-migrates instances to another compute node in the same fault domain. If the fault domain does not have sufficient capacity, high-availability configuration settings might cause instances to be live-migrated to another fault domain, and migrated back to their selected fault domain when the required capacity is available again.

Compute instances that have been migrated away from their assigned fault domain, are called displaced instances. Their migrations can interfere with compute node upgrade or patching. When the locks on a given compute node are released, its displaced instances start migrating back, during which time it might be impossible to lock the next compute node for maintenance.

Before upgrading or patching a compute node, monitor the status of displaced instances. Do not proceed with the next compute node until the list is empty.

In the Service CLI, use the command getDisplacedInstances. In the following example, two instances have been migrated away from fault domain 1.

PCA-ADMIN> getDisplacedInstances
Data:
 id                        displayName  compartmentId                faultDomain     faultDomainSelected
 --                        -----------  -------------                -----------     -------------------
 ocid1.instance.unique_ID  inst-name    ocid1.compartment.unique_ID  FAULT-DOMAIN-3  FAULT-DOMAIN-1
 ocid1.instance.unique_ID  inst-name    ocid1.compartment.unique_ID  FAULT-DOMAIN-2  FAULT-DOMAIN-1

In the Service Web UI, click the navigation menu, click FD Instances, and then click Displaced Instances.

For more information, refer to the following sections in the Hardware Administration chapter of the Oracle Private Cloud Appliance Administrator Guide:

Migrating instances and locking a compute node: see "Performing Compute Node Operations".
Compute service HA configuration: see "Configuring the Compute Service for High Availability".

Using the Service Web UI

Set the provisioning and maintenance locks for the compute node you are about to patch. Ensure that no active compute instances are present on the node.

Caution:

Depending on the high-availability configuration of the Compute service, automatic instance migrations can prevent you from successfully locking a compute node. See Monitoring Displaced Instances.
1. In the navigation menu, click Rack Units. In the Rack Units table, click the name of the compute node you want to patch to display its detail page.
2. In the top-right corner of the compute node detail page, click Controls and select the Provisioning Lock command.
3. When the provisioning lock has been set, click Controls again and select the Migrate All Vms command. The Compute service evacuates the compute node, meaning it migrates the running instances to other compute nodes.
  
  Note:
  
  In case physical resources are limited, compute instances will be migrated to other fault domains during compute node evacuation. However, the strict fault domain enforcement (Strict FD) function must be disabled.
4. When compute node evacuation is complete, click Controls again and select the Maintenance Lock command. This command might fail if instance migrations are in progress. Wait a few minutes and retry.
In the navigation menu, go to the Maintenance section and click Upgrade Plan. This provides an overview of current and target component versions.
Click Upgrade & Patching to display the Upgrade Jobs page.
Optionally, patch the server ILOM first.
1. In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch. The Create Request window appears.
2. Choose Patch as the Request Type. Select the appropriate patch request type: Patch ILOM.
  
  Fill out the server's assigned IP address in the ILOM network. This is an IP address in the internal 100.96.0.0/23 range.
3. Click Create Request. The new patch request appears in the Upgrade Jobs table.
4. Wait 5 minutes to allow the ILOM patch job to complete. Then proceed to patching the host.
In the top-right corner of the Upgrade Jobs page, click Create Upgrade or Patch.

The Create Request window appears. Choose Patch as the Request Type.
Select the appropriate patch request type: Patch CN.
If required, fill out the request parameters:
- Host IP: Enter the compute node's assigned IP address in the internal administration network. This is an IP address in the internal 100.96.2.0/23 range.
- Log Level: Optionally, select a specific log level for the upgrade log file. The default log level is "Information". For maximum detail, select "Debug".
- Advanced Options JSON: Not available.
- Alternative ULN Channel: This parameter forces the request to use a non-standard ULN channel. Do not use this option unless Oracle explicitly instructs you to do so.
- Verify Only: Enable this option to run the operation in verification only mode.
- Force: Enable this option to force the operation. Use only when instructed by Oracle.
Click Create Request.

The new patch request appears in the Upgrade Jobs table.
When the compute node has been patched successfully, release the provisioning and maintenance locks.

For more information, refer to the section "Performing Compute Node Operations". It can be found in the chapter Hardware Administration of the Oracle Private Cloud Appliance Administrator Guide.
1. Open the compute node detail page.
2. In the top-right corner of the compute node detail page, click Controls and select the Maintenance Unlock command.
3. When the maintenance lock has been released, click Controls again and select the Provisioning Unlock command.

Using the Service CLI

From the output you obtained with the compute node list command earlier, get the ID and the IP address of the compute node you intend to patch.

Set the provisioning and maintenance locks for the compute node you are about to patch.

Caution:

Depending on the high-availability configuration of the Compute service, automatic instance migrations can prevent you from successfully locking a compute node. For more information, refer to the following sections in the Hardware Administration chapter of the Oracle Private Cloud Appliance Administrator Guide:

Migrating instances and locking a compute node: see "Performing Compute Node Operations".
Compute service HA configuration: see "Configuring the Compute Service for High Availability".

Disable provisioning for the compute node.

PCA-ADMIN> provisioningLock id=cf488903-fef8-4a51-8a41-c6990e4755c5
Status: Success
JobId: 6ee78c8a-e227-4d31-a770-9b9c96085f3f

Evacuate the compute node. Wait for the migration job to finish before proceeding to the next step.

Note:

In case physical resources are limited, compute instances will be migrated to other fault domains during compute node evacuation. However, the strict fault domain enforcement (Strict FD) function must be disabled.
```
PCA-ADMIN> migrateVm id=cf488903-fef8-4a51-8a41-c6990e4755c5
Status: Running
JobId: 6f1e94bc-7d5b-4002-ada9-7d4b504a2599

PCA-ADMIN> show Job id=6f1e94bc-7d5b-4002-ada9-7d4b504a2599
  Run State = Succeeded
```

Lock the compute node for maintenance.

PCA-ADMIN> maintenanceLock id=cf488903-fef8-4a51-8a41-c6990e4755c5
Status: Success
JobId: e46f6603-2af2-4df4-a0db-b15156491f88

Optionally, rerun the compute node list command to confirm lock status. For example:

PCA-ADMIN> list computeNode fields hostname,ipAddress,ilomIp,state,firmwareVersion,provisioningLocked,maintenanceLocked orderby hostname ASCENDING
Data:
  id                                     Hostname   Ip Address    ILOM Ip Address   State   Firmware Version           Provisioning Locked   Maintenance Locked
  --                                     --------   ----------    ---------------   -----   ----------------           -------------------   ------------------
  cf488903-fef8-4a51-8a41-c6990e4755c5   pcacn001   100.96.2.64   100.96.0.64       On      PCA Hypervisor:3.0.2-681   true                  true              
  42a7594d-1173-4dbd-4755-07810cc2d527   pcacn002   100.96.2.65   100.96.0.65       On      PCA Hypervisor:3.0.2-681   false                 false             
  bc0f37d5-ba77-423e-bc11-017704b47e59   pcacn003   100.96.2.66   100.96.0.66       On      PCA Hypervisor:3.0.2-681   false                 false             
  2e5ac527-01f5-4230-ae41-0522fcb57c9a   pcacn004   100.96.2.67   100.96.0.67       On      PCA Hypervisor:3.0.2-681   false                 false             
  5a6b61cf-7e99-4df2-87e4-b37c5fb0bfb8   pcacn005   100.96.2.68   100.96.0.68       On      PCA Hypervisor:3.0.2-681   false                 false             
  885f2aa4-f017-41e8-b2bc-e588cc0c6162   pcacn006   100.96.2.69   100.96.0.69       On      PCA Hypervisor:3.0.2-681   false                 false

Optionally, patch the server ILOM first.

Enter the ILOM patch command.

Syntax (entered on a single line):

patchIlom
hostIp=<ilom-ip>

Example:

PCA-ADMIN> patchIlom hostIp=100.96.0.64
Data:
  Service request has been submitted. Upgrade Job Id = 1620921089806-ilom-21480 Upgrade Request Id = UWS-732d6fce-9f06-4329-b972-d093bee40010

PCA-ADMIN> getUpgradeJob upgradeJobId=1620921089806-ilom-21480

Wait 5 minutes to allow the ILOM patch job to complete. Then proceed to patching the host.

Enter the compute node patch command.

Syntax (entered on a single line):

patchCN 
hostIp=<compute-node-ip>
[optional] uln=<http|https>://<hostname.domainname>/<sub-directories>

The parameter marked optional is deprecated in software version 3.0.2-b892153 and later. For earlier versions, include the fully qualified domain name of the ULN mirror with the command.

Example:

PCA-ADMIN> patchCN hostIp=100.96.2.64 ULN=http://host.example.com/yum
Status: Success
Data:
  Service request has been submitted.  Upgrade Job ID = 1685372050358-compute-50568 Upgrade Request ID = UWS-f226d7d2-549d-4902-8614-e1f40bdc9ff6

Use the request ID and the job ID to check the status of the patching process.

PCA-ADMIN> getUpgradeJobs
Command: getUpgradeJobs
Status: Success
Time: 2023-01-01 21:09:34.745 UTC
Data:
  id                              upgradeRequestId                           commandName   result
  --                              ----------------                           -----------   ------
  1685372050358-compute-50568     UWS-f226d7d2-549d-4902-8614-e1f40bdc9ff6   compute       Passed

PCA-ADMIN> getUpgradeJob upgradeJobId=1685372050358-compute-50568
Command: getUpgradeJob upgradeJobId=1685372050358-compute-50568
Status: Success
Time: 2023-01-01 21:10:13,804 UTC
Data:
  Upgrade Request Id = UWS-f226d7d2-549d-4902-8614-e1f40bdc9ff6
  Name = compute
[...]

When the compute node patch has completed successfully and the node has rebooted, release the locks.

For more information, refer to "Performing Compute Node Operations" in the Hardware Administration section of the Oracle Private Cloud Appliance Administrator Guide.
1. Release the maintenance lock.
```
PCA-ADMIN> maintenanceUnlock id=cf488903-fef8-4a51-8a41-c6990e4755c5
Status: Success
JobId: 625af20e-4b49-4201-879f-41d4405314c7
```
2. Release the provisioning lock.
```
PCA-ADMIN> provisioningUnlock id=cf488903-fef8-4a51-8a41-c6990e4755c5
Status: Success
JobId: 523892e8-c2d4-403c-9620-2f3e94015b46
```

Proceed to the next compute node and repeat this procedure.

The output from the compute node list command indicates the current status. For example:

PCA-ADMIN> list computeNode fields hostname,ipAddress,ilomIp,state,firmwareVersion,provisioningLocked,maintenanceLocked orderby hostname ASCENDING
Data:
  id                                     Hostname   Ip Address    ILOM Ip Address   State   Firmware Version           Provisioning Locked   Maintenance Locked
  --                                     --------   ----------    ---------------   -----   ----------------           -------------------   ------------------
  cf488903-fef8-4a51-8a41-c6990e4755c5   pcacn001   100.96.2.64   100.96.0.64       On      PCA Hypervisor:3.0.2-696   false                 false             
  42a7594d-1173-4dbd-4755-07810cc2d527   pcacn002   100.96.2.65   100.96.0.65       On      PCA Hypervisor:3.0.2-696   false                 false             
  bc0f37d5-ba77-423e-bc11-017704b47e59   pcacn003   100.96.2.66   100.96.0.66       On      PCA Hypervisor:3.0.2-696   false                 false             
  2e5ac527-01f5-4230-ae41-0522fcb57c9a   pcacn004   100.96.2.67   100.96.0.67       On      PCA Hypervisor:3.0.2-696   false                 false             
  5a6b61cf-7e99-4df2-87e4-b37c5fb0bfb8   pcacn005   100.96.2.68   100.96.0.68       On      PCA Hypervisor:3.0.2-681   false                 false             
  885f2aa4-f017-41e8-b2bc-e588cc0c6162   pcacn006   100.96.2.69   100.96.0.69       On      PCA Hypervisor:3.0.2-681   false                 false