Repairing GPU Hosts

Before configuring user-initiated repair, identify an eligible managed GPU node pool and a controlled test node. Ensure that workloads on the selected node can be interrupted.

User-initiated repair requires access control in both OCI IAM and Kubernetes RBAC. In OCI IAM, the OKE cluster resource principal requires permission to apply the defined tag used to report the host condition to Compute. In Kubernetes, only a trusted operator should be able to initiate the protected repair request. The trusted operator requires standard Kubernetes RBAC permissions to update the target Node resource and the OKE-specific authorization delegated through a ClusterRoleBinding.

When you request user-initiated repair, OKE coordinates the following actions:

1. Ensures only one repair request is processed at a time.
2. Cordons the node so that Kubernetes does not schedule new pods on it.
3. Attempts to drain workloads from the node.
4. Reports the condition to Compute and terminates the affected instance.
5. Restores the managed node pool’s configured capacity.

To configure and test user-initiated repair:

1. Verify that the cluster is an enhanced cluster, and is running Kubernetes version 1.32 or later. Verify that the node is an eligible managed GPU worker node. Verify that the OKE-managed Node Repair Controller and NodeHealthReport CRDs, the oke-customer-node-repair-trigger ClusterRole, and the trigger-protection admission policy and binding are installed on the cluster. If any OKE-managed resource is missing, contact Oracle Support. Do not recreate OKE-managed resources.

2. Before using live repair, open a Support request to enable user-reported hardware-fault handling for your tenancy. Include the tenancy, region, cluster OCID, and GPU node pool in the request.

3. Create or reuse the defined tag that the Node Repair Controller uses to report the host condition to Compute.

   Create or reuse a tag namespace named ComputeInstanceHostActions. In that namespace, create or reuse a tag key definition named CustomerReportedHostStatus. Do not create a free-form tag with the same text. For more information, see Creating a Tag Namespace and Creating a Tag Key Definition.

   Before terminating the original instance, the Node Repair Controller writes the value unhealthy to the ComputeInstanceHostActions.CustomerReportedHostStatus defined tag on the instance.

4. Create an OCI IAM policy that grants the OKE cluster resource principal the narrowly scoped permission to apply the defined tag:

   Allow any-user to use tag-namespace in tenancy where all {request.principal.type = 'cluster', request.principal.id = '<cluster-ocid>', target.tag-namespace.name = 'ComputeInstanceHostActions'}

   IAM changes can take time to propagate. Wait for propagation before performing destructive validation.

5. Delegate the protected user-initiated repair trigger to trusted operators.

   Create a ClusterRoleBinding that binds a trusted OCI IAM group OCID to the OKE-owned oke-customer-node-repair-trigger ClusterRole. Operators also require standard Kubernetes RBAC permissions to update the target Node resource.

   Create a file named cir-trigger-operators.yaml with the following content:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: oke-node-repair-trigger-operators
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: oke-customer-node-repair-trigger
   subjects:
   - apiGroup: rbac.authorization.k8s.io
     kind: Group
     name: <trusted-oci-iam-group-ocid>

   Replace <trusted-oci-iam-group-ocid> with the OCID of the OCI IAM group that contains the trusted operators.

   Apply and verify the binding by entering:

   kubectl apply -f cir-trigger-operators.yaml

   kubectl get clusterrolebinding \
     oke-node-repair-trigger-operators \
     -o yaml

   This binding does not grant general permission to update Node resources. Use your existing Kubernetes authorization model to grant only the standard Kubernetes RBAC permissions required by the trusted operators.

6. Create a non-overlapping, cluster-scoped NodeRepairConfig custom resource in dry-run mode.

   Select nodes by using a durable node-pool label that you manage and that is present on replacement nodes. The selector label is user-defined. Configure the label on the intended managed node pool so that replacement nodes retain it.

   A node must match exactly one NodeRepairConfig resource. Requests with no matching configuration or overlapping configurations are rejected without repair side effects. You can use the same or separate configurations for user-initiated repair and GPU instance recovery. If you use separate configurations, do not include the same nodes in multiple configurations; otherwise, the repair request will be rejected.

   The following example starts in dry-run mode, allows one repair at a time, sets the unhealthy-node threshold to 10%, uses a 60-minute drain grace period, and disables force-after-grace behavior:

   apiVersion: oci.oraclecloud.com/v1beta1
   kind: NodeRepairConfig
   metadata:
     name: gpu-node-repair
   spec:
     nodeSelector:
       matchLabels:
         example.com/gpu-repair: "enabled"
     policy:
       dryRun: true
       evictionGracePeriodMinutes: 60
       forceActionAfterGracePeriod: false
       maxUnhealthyNodeThreshold: "10%"
       maximumParallelNodeRepairs: 1

   In the policy section, dryRun and forceActionAfterGracePeriod are Boolean values. evictionGracePeriodMinutes is a non-negative integer. maximumParallelNodeRepairs is a positive integer. maxUnhealthyNodeThreshold accepts a positive node count or a quoted percentage, such as "10%".

   Start with dryRun: true. After validating the configuration, update only dryRun to false for live operation.

7. Request a dry-run repair by applying the protected label oci.oraclecloud.com/customer-report-host-status=unhealthy to the selected node. Inspect the NodeHealthReport and related node state. A dry-run request creates a completed NodeHealthReport record, but does not cordon, drain, apply the Compute defined tag, terminate, reboot, or replace the node.

8. Trigger a live repair by applying the protected trigger label together with the one-shot live override oci.oraclecloud.com/node-repair-dry-run=false. The live override is valid only with an active trigger, and is sampled only when a new trigger is created.

   To retry a user-initiated repair request, remove and reapply the oci.oraclecloud.com/customer-report-host-status=unhealthy label. Reapplying the same label value does not start another request. Removing the label does not cancel an active repair.

   Important
   
   A live user-initiated repair can drain workloads and terminates the original Compute instance. Draining uses the Kubernetes Eviction API and honors pod disruption budgets. DaemonSet pods and static pods are skipped. If force-after-grace is enabled, the Node Repair Controller does not force-delete remaining pods, but the subsequent host action can still disrupt them.

9. Verify the progress of the repair action, the Compute defined tag, termination of the original instance, managed-node-pool capacity restoration, and creation of a replacement node with a new UID and instance OCID. The NodeHealthReport for terminated instances is retained temporarily after the workflow completes.

   A Deferred result means that the workflow is temporarily blocked. A Failed result means that the attempt stopped. A Superseded result means that a higher-priority OKE workflow took over. A Completed result means that OKE finished the requested termination workflow. It does not mean that physical repair is complete or that replacement capacity is Ready. After the workflow completes, verify that the managed node pool has the expected capacity and that the replacement node reports current healthy status.

For user-initiated repair, creation of a replacement worker node depends on managed-node-pool reconciliation and available GPU capacity. A completed workflow does not confirm that the underlying GPU fault was corrected.

Before configuring GPU instance recovery, identify an eligible managed GPU node pool and confirm that its workloads can be interrupted. GPU instance recovery can cordon and drain an affected node before rebooting its instance.

Before configuring GPU instance recovery, create the IAM policies required by the GPU Health Monitoring add-on to monitor GPU hosts:

Allow dynamic-group <dg-name> to manage instance-family in compartment id <compartment-ocid>

Allow dynamic-group <dg-name> to read instance-agent-plugins in compartment id <compartment-ocid>

where:

• <dg-name> is the name of the dynamic group that includes the Compute instances for the GPU worker nodes. For example, using a rule such as ALL {instance.compartment.id = '<compartment-ocid>'}. By applying tags to node pools, you can create more restrictive rules such as:

  ALL {
    instance.compartment.id = 'ocid1.compartment.oc1..examplecompartment',
    tag.Operations.Workload.value = 'gpu-monitoring',
    tag.Operations.Environment.value = 'prod'
  }

  For more information, see Applying Tags to Node Pools.

• <compartment-ocid> is the OCID of the compartment that contains the GPU worker nodes.

GPU instance recovery is supported only on eligible managed GPU worker nodes using the following GPU shapes:

• BM.GPU.A10.4
• BM.GPU.A100-v2.8
• BM.GPU.B200.8
• BM.GPU.B300.8
• BM.GPU.B300.HS.8
• BM.GPU.B4.8
• BM.GPU.GB200-v2.4
• BM.GPU.GB200-v3.4
• BM.GPU.GB200.4
• BM.GPU.GB300.4
• BM.GPU.H100.8
• BM.GPU.H100T.8
• BM.GPU.H200-NC.8
• BM.GPU.H200.8
• BM.GPU.L40S-NC.4
• BM.GPU.L40S.4
• BM.GPU.MI300X.8
• BM.GPU.MI355X-v1.8
• BM.GPU.MI355X.8

GPU shape availability varies by region. Confirm that the selected GPU shape is available in the region where the cluster runs.

Oracle Cloud Agent version 1.64.0 or later is required by the Compute RDMA GPU Monitoring plug-in and must be installed on the selected nodes.

When OKE receives an eligible GPU health signal, it coordinates the following actions:

1. Ensures only one recovery request is processed at a time.
2. Cordons the node so that Kubernetes does not schedule new pods on it.
3. Attempts to drain workloads from the node.
4. Reboots the affected Compute instance.

To configure GPU instance recovery:

1. Verify that the cluster is running Kubernetes version 1.32 or later. Verify that the node is an eligible managed GPU worker node. Verify that the OKE-managed Node Repair Controller and NodeHealthReport CRDs are installed on the cluster, and verify that the GPU Health Monitoring add-on is available for the cluster.

2. Verify that the GPU Health Monitoring add-on has an ACTIVE option for the Kubernetes version running on the cluster by entering:

   oci ce addon-option list \
     --kubernetes-version <version> \
     --addon-name GpuHealthMonitoring

   When you install the add-on, install the latest ACTIVE option rather than pinning a version. The installed version must be v1.1.0 or later.

3. Install the OKE-managed GPU Health Monitoring add-on by entering:

   oci ce cluster install-addon \
     --addon-name GpuHealthMonitoring \
     --cluster-id $CLUSTER_ID \
     --region $REGION

   The command installs the latest available version of the add-on for the cluster.

   The supported add-on configuration keys are nodeSelectors, tolerations, and rollingUpdate. The rollingUpdate value controls rolling update behavior by maxSurge and maxUnavailable, using JSON format in plain text.

4. Create a GpuHealthMonitoringConfig resource to enable GPU health monitoring for the selected GPU worker nodes. The configuration requires spec.enabled and a non-empty node selector.

   apiVersion: oci.oraclecloud.com/v1beta1
   kind: GpuHealthMonitoringConfig
   metadata:
     name: gpu-health-monitoring
   spec:
     enabled: true
     nodeSelector:
       matchLabels:
         nvidia.com/gpu.present: "true"

   Save the configuration as gpu-health-monitoring.yaml, and apply it by entering:

   kubectl apply -f gpu-health-monitoring.yaml

   The spec.enabled field enables or disables monitoring for the selected nodes. The spec.nodeSelector field selects the GPU nodes on which the Compute RDMA GPU Monitoring plug-in and GPU Fault Monitor run.

   Avoid creating multiple enabled GpuHealthMonitoringConfig resources whose selectors match the same node. Overlapping policies are rejected with a Ready condition that has status set to False and reason set to MultipleMatchingPolicies.

5. Verify GPU health monitoring configuration on the selected nodes.

   The controller publishes the oci.oraclecloud.com/GpuHealthMonitoringConfigured Node condition. A value of True indicates that the Compute RDMA GPU Monitoring plug-in was configured as enabled. A value of False indicates that it was configured as disabled. A value of Unknown indicates that the configured state cannot currently be trusted.

   The GPU Fault Monitor publishes the oci.oraclecloud.com/GpuHealthMonitoringUnavailable Node condition. A value of True indicates that one or more required GPU health-result files are unavailable. Possible causes include an outdated Oracle Cloud Agent version, unsupported shape, disabled or unhealthy plug-in, or misconfiguration. A value of False indicates that the required result files are present.

6. Create a non-overlapping, cluster-scoped NodeRepairConfig custom resource in dry-run mode.

   Select nodes by using a durable node-pool label that you manage. The selector label is user-defined.

   A node must match exactly one NodeRepairConfig resource. Requests with no matching configuration or overlapping configurations are rejected without repair side effects. You can use the same or separate configurations for user-initiated repair and GPU instance recovery. If you use separate configurations, do not include the same nodes in multiple configurations; otherwise, the repair request will be rejected.

   The following example starts in dry-run mode, allows one repair at a time, sets the unhealthy-node threshold to 10%, uses a 60-minute drain grace period, and disables force-after-grace behavior:

   apiVersion: oci.oraclecloud.com/v1beta1
   kind: NodeRepairConfig
   metadata:
     name: gpu-node-repair
   spec:
     nodeSelector:
       matchLabels:
         example.com/gpu-repair: "enabled"
     policy:
       dryRun: true
       evictionGracePeriodMinutes: 60
       forceActionAfterGracePeriod: false
       maxUnhealthyNodeThreshold: "10%"
       maximumParallelNodeRepairs: 1
       pauseBeforeRepair: true

   In the policy section, dryRun, forceActionAfterGracePeriod, and pauseBeforeRepair are Boolean values. evictionGracePeriodMinutes is a non-negative integer. maximumParallelNodeRepairs is a positive integer. maxUnhealthyNodeThreshold accepts a positive node count or a quoted percentage, such as "10%".

   Start with dryRun: true and pauseBeforeRepair: true. Setting pauseBeforeRepair to true enables you to inspect the selected repair before OKE starts disruptive actions. Pause also applies during dry-run. When pauseBeforeRepair is set to true, you must continue the paused simulation before it can complete.

   After validating the configuration, update only dryRun to false for live operation.

   Note that GPU instance recovery is health-signal-driven. Do not apply the user-initiated repair label oci.oraclecloud.com/customer-report-host-status=unhealthy for GPU instance recovery.

7. Verify that an eligible GPU health signal is reported for the node.

   A node is eligible for automated reboot only when all of the following conditions are met:

   • The Node condition oci.oraclecloud.com/GpuHealthMonitoringFaultDetected is True.
   • The NodeHealthReport contains an entry under status.signals.clusterHealthCheck[].
   • The signal has a non-empty id.
   • The signal has action=REBOOT. The value is case-insensitive.
   • The signal has a non-null firstObservedAt value.
   • The signal has a non-null lastObservedAt value, indicating that the signal is active.
   • The signal has remained active for at least 10 minutes.
   • Node Repair Controller retry and cooldown rules permit another attempt. The repeat reboot cooldown is 60 minutes.
   • The effective dry-run value is false for an actual reboot.
   • Subsequent admission and disruption limits pass.

   Signals with action=CUSTOMER_ACTION_REQUIRED do not trigger an automated reboot. If you see one of these signals, review the NodeHealthReport and node conditions, investigate the reported GPU health issue, and contact Support if you need help resolving it.

8. Enable live repair in the NodeRepairConfig resource.

   GPU instance recovery starts only after the eligible signal has remained active for at least 10 minutes, and the repeat reboot cooldown is 60 minutes. Drain behavior follows the repair policy specified in the matching NodeRepairConfig resource. Draining uses the Kubernetes Eviction API and honors pod disruption budgets. The time required to complete GPU instance recovery can vary, depending on workload disruption settings, node drain behavior, and Compute instance state.

9. Verify that GPU instance recovery completed successfully.

   Confirm that the NodeHealthReport shows a Reboot action in live mode and has a terminal status of Completed. Confirm that the Compute instance is in the Running state, the Kubernetes node is Ready, and the current GPU health result no longer reports the fault.

10. Review the GPU health result after the workflow completes.

    If the GPU health result still reports the same fault after the reboot, or reports the fault again, the fault has not been resolved. A completed recovery workflow does not mean that the underlying GPU fault was corrected. GPU instance recovery uses cooldown rules to prevent immediate reboot loops. After the GPU health result stops reporting the fault, a later GPU health result that reports a new fault can make the node eligible for another repair.

Before configuring maintenance event repair, identify eligible bare metal GPU worker nodes and confirm that workloads on the selected nodes can be interrupted. Maintenance event repair can cordon and drain an affected node before OKE performs the applicable Compute action.

Maintenance event repair is supported for eligible bare metal GPU worker nodes, including managed nodes and self-managed nodes. The cluster must be an enhanced cluster and must be running Kubernetes version 1.32 or later. Maintenance event repair does not apply to virtual nodes or non-bare metal worker nodes.

Maintenance event repair uses GPU Health Monitoring to identify node-local GPU or RDMA faults on GPU shapes. OKE correlates the local fault signal with an OCI Compute maintenance event for the same Compute instance. If the node is selected by exactly one matching NodeRepairConfig resource and the repair policy allows repair, OKE coordinates the repair workflow.

Before configuring maintenance event repair, create the IAM policies required by the GPU Health Monitoring add-on to monitor GPU hosts:

Allow dynamic-group <dg-name> to manage instance-family in compartment id <compartment-ocid>

Allow dynamic-group <dg-name> to read instance-agent-plugins in compartment id <compartment-ocid>

where:

• <dg-name> is the name of the dynamic group that includes the Compute instances for the GPU worker nodes. For example, using a rule such as ALL {instance.compartment.id = '<compartment-ocid>'}. By applying tags to node pools, you can create more restrictive rules such as:

  ALL {
                instance.compartment.id = 'ocid1.compartment.oc1..examplecompartment',
                tag.Operations.Workload.value = 'gpu-monitoring',
                tag.Operations.Environment.value = 'prod'
                }

  For more information, see Applying Tags to Node Pools.

• <compartment-ocid> is the OCID of the compartment that contains the GPU worker nodes.

Also create a policy that allows the OKE cluster resource principal to use instance maintenance events in the compartment that contains the GPU worker node instances and their maintenance events:

Allow any-user to use instance-maintenance-events in compartment id <worker-compartment-ocid> where request.principal.type = 'cluster'

To restrict access to a specific cluster, use a policy such as:

Allow any-user to use instance-maintenance-events in compartment id <worker-compartment-ocid> where all {request.principal.type = 'cluster', request.principal.id = '<cluster-ocid>'}

where:

• <worker-compartment-ocid> is the OCID of the compartment that contains the GPU worker node instances and their maintenance events.
• <cluster-ocid> is the OCID of the cluster.

Maintenance event repair supports the following Compute maintenance event actions:

• STOP: OKE coordinates a non-terminating maintenance event. The same Compute instance and Kubernetes node are expected to return to service after the maintenance event is processed.
• TERMINATE: OKE coordinates termination of the affected Compute instance. For managed node pools, replacement depends on managed-node-pool capacity management and available GPU capacity. For self-managed nodes, OKE does not automatically replace the node after termination. Provision and join a replacement node using the node-management process that you normally use for self-managed nodes (see Working with Self-Managed Nodes).

Oracle Cloud Agent version 1.64.0 or later is required by the Compute RDMA GPU Monitoring plug-in and must be installed on the selected nodes.

When OKE admits a maintenance event repair request, it coordinates the following actions:

1. Ensures only one repair request is processed at a time.
2. Correlates the local GPU or RDMA fault signal with a Compute maintenance event for the same instance.
3. Records repair information in a NodeHealthReport.
4. Cordons the node so that Kubernetes does not schedule new pods on it.
5. Attempts to drain workloads from the node.
6. Performs the applicable STOP or TERMINATE maintenance event workflow.

To configure maintenance event repair:

1. Verify that the cluster is an enhanced cluster and is running Kubernetes version 1.32 or later. Verify that the node is an eligible bare metal GPU worker node, either a managed node or a self-managed node.

   Verify that the OKE-managed Node Repair Controller and NodeHealthReport CRDs are installed on the cluster, and verify that the GPU Health Monitoring add-on is available for the cluster. If any OKE-managed resource is missing, contact Oracle Support. Do not recreate OKE-managed resources.

2. Install and configure the GPU Health Monitoring add-on for the selected GPU worker nodes.

   Maintenance event repair requires GPU Health Monitoring to be enabled on the selected GPU worker nodes so that OKE can correlate a local GPU or RDMA fault signal with a Compute maintenance event.

   Verify that the GPU Health Monitoring add-on has an ACTIVE option for the Kubernetes version running on the cluster by entering:

   oci ce addon-option list \
     --kubernetes-version <version> \
     --addon-name GpuHealthMonitoring

   Install the OKE-managed GPU Health Monitoring add-on by entering:

   oci ce cluster install-addon \
     --addon-name GpuHealthMonitoring \
     --cluster-id $CLUSTER_ID \
     --region $REGION

   Create a GpuHealthMonitoringConfig resource to enable GPU health monitoring for the selected GPU worker nodes. The configuration requires spec.enabled and a non-empty node selector.

   apiVersion: oci.oraclecloud.com/v1beta1
   kind: GpuHealthMonitoringConfig
   metadata:
     name: gpu-health-monitoring
   spec:
     enabled: true
     nodeSelector:
       matchLabels:
         nvidia.com/gpu.present: "true"

   Save the configuration as gpu-health-monitoring.yaml, and apply it by entering:

   kubectl apply -f gpu-health-monitoring.yaml

   The spec.enabled field enables or disables monitoring for the selected nodes. The spec.nodeSelector field selects the GPU nodes on which the Compute RDMA GPU Monitoring plug-in and GPU Fault Monitor run.

   Avoid creating multiple enabled GpuHealthMonitoringConfig resources whose selectors match the same node. Overlapping policies are rejected with a Ready condition that has status set to False and reason set to MultipleMatchingPolicies.

   Verify GPU health monitoring configuration on the selected nodes. The controller publishes the oci.oraclecloud.com/GpuHealthMonitoringConfigured Node condition. The GPU Fault Monitor publishes the oci.oraclecloud.com/GpuHealthMonitoringUnavailable Node condition.

3. Create a non-overlapping, cluster-scoped NodeRepairConfig custom resource in dry-run mode.

   Select nodes by using a durable label that you manage and that is present on the intended GPU worker nodes. For managed node pools, configure the label on the node pool so that replacement nodes retain it. For self-managed nodes, ensure that the label is applied consistently to the intended nodes.

   A node must match exactly one NodeRepairConfig resource. Requests with no matching configuration or overlapping configurations are rejected without repair side effects. Do not create overlapping configurations for user-initiated repair, GPU instance recovery, and maintenance event repair.

   The following example starts in dry-run mode, allows one repair at a time, sets the unhealthy-node threshold to 10%, uses a 60-minute drain grace period, and disables force-after-grace behavior:

   apiVersion: oci.oraclecloud.com/v1beta1
   kind: NodeRepairConfig
   metadata:
     name: gpu-node-repair
   spec:
     nodeSelector:
       matchLabels:
         example.com/gpu-repair: "enabled"
     policy:
       dryRun: true
       evictionGracePeriodMinutes: 60
       forceActionAfterGracePeriod: false
       maxUnhealthyNodeThreshold: "10%"
       maximumParallelNodeRepairs: 1
       pauseBeforeRepair: true

   In the policy section, dryRun, forceActionAfterGracePeriod, and pauseBeforeRepair are Boolean values. evictionGracePeriodMinutes is a non-negative integer. maximumParallelNodeRepairs is a positive integer. maxUnhealthyNodeThreshold accepts a positive node count or a quoted percentage, such as "10%".

   Start with dryRun: true and pauseBeforeRepair: true. Setting pauseBeforeRepair to true enables you to inspect the selected repair before OKE starts disruptive actions. Pause also applies during dry-run. When pauseBeforeRepair is set to true, you must continue the paused simulation before it can complete.

4. Verify dry-run behavior.

   In dry-run mode, OKE evaluates the selected node, matching repair policy, correlated maintenance event, and proposed maintenance event action. Dry-run mode must not cordon, drain, stop, terminate, or replace the node.

   Inspect the NodeHealthReport for the correlated maintenance event and workflow result. The maintenance event OCID is shown in status.signals.maintenanceEvent[].maintenanceEventId. The correlated action is shown in status.signals.maintenanceEvent[].action. The Compute event lifecycle state is shown in status.signals.maintenanceEvent[].maintenanceLifecycleState.

   To verify the selected repair attempt, review status.repair.selectedSignalIds, status.repair.repairAction, status.repair.dryRun, status.repair.terminalState, and status.lastRepairAttempt. A successful dry-run simulation records Completed with dryRun: true. Dry-run completion does not prove that a real drain or Compute action will succeed.

5. Enable controlled live repair by updating the matching NodeRepairConfig resource.

   For live operation, update only the intended policy fields, such as changing dryRun from true to false. Changing dryRun does not replay an already admitted repair attempt. Changing the NodeRepairConfig resource does not re-evaluate an existing maintenance-event signal or alter an already admitted repair attempt. The updated policy applies to the next distinct maintenance-event signal.

   Do not edit NodeHealthReport resources or toggle internal Node conditions to force a retry.

   Important
   
   A live maintenance event repair can drain workloads and stop or terminate the affected Compute instance. Draining uses the Kubernetes Eviction API and honors pod disruption budgets. DaemonSet pods and static pods are skipped. If force-after-grace is enabled, the Node Repair Controller does not force-delete remaining pods, but the subsequent Compute action can still disrupt them.

6. If the repair pauses before the Compute action, inspect the NodeHealthReport and node conditions. A paused repair is indicated by the oci.oraclecloud.com/NodeAutoRepairPaused=True node condition.

   To continue the paused repair, enter:

   kubectl label node <node-name> oci.oraclecloud.com/node-repair-paused-response=continue --overwrite

   To cancel the paused repair, enter:

   kubectl label node <node-name> oci.oraclecloud.com/node-repair-paused-response=cancel --overwrite

   The Node Repair Controller consumes the response label and clears the pause condition. When you continue a paused repair, OKE checks the repair policy and disruption limits again before continuing.

   Canceling a paused repair stops only the OKE repair attempt. It does not cancel, reschedule, or remove the underlying Compute maintenance event. OKE records Cancelled for the repair attempt and clears the pause condition and response label. OKE continues tracking the Compute maintenance event, and the MaintenanceEventDetected condition might remain present. Canceling the OKE repair attempt does not prevent Compute from processing maintenance under its own schedule.

7. For a STOP maintenance event repair, verify that the maintenance event workflow completes and the same node returns to service.

   In a successful non-terminating STOP repair, the Kubernetes Node name, Kubernetes Node UID, and Compute instance OCID remain the same.

   Confirm that the NodeHealthReport records repairAction: STOP and is not a dry-run. Confirm that status.lastRepairAttempt records result: Completed and a completedAt timestamp. Confirm that the original Node is Ready and schedulable, with unchanged identities.

8. For a TERMINATE maintenance event repair, verify that the live repair matches the intended maintenance event and Compute instance, that the original instance is terminated, and that the original Kubernetes Node is removed.

   For managed node pools, confirm that replacement capacity becomes available when GPU capacity, service limits, and compartment quotas allow replacement. For self-managed nodes, provision and join a replacement node using the node-management process that you normally use for self-managed nodes (see Working with Self-Managed Nodes).

   A Completed result indicates that OKE completed the termination workflow. It does not confirm that physical host repair is complete, replacement capacity is ready, or workloads have recovered.

A completed maintenance event repair workflow does not confirm that the underlying GPU or RDMA fault was corrected, that physical host repair is complete, that replacement capacity is ready, or that workloads have recovered. After the workflow completes, verify the node's current health signals, node readiness, expected capacity, and application health.

Use Kubernetes resources and standard Kubernetes monitoring tools to inspect the NodeHealthReport, Node conditions, affected workloads, and repair workflow.

To monitor a GPU host repair workflow:

1. Inspect the NodeHealthReport and the affected worker node before, during, and after repair.

   Review the NodeHealthReport for the repair request, repair progress, action, dry-run or live mode, and workflow result. Review the affected Kubernetes Node resource for node labels and conditions. Review Kubernetes Events for recent node and repair activity. For GPU instance recovery, check the GpuHealthMonitoringConfigured, GpuHealthMonitoringUnavailable, and GpuHealthMonitoringFaultDetected conditions on the affected Kubernetes Node resource. For maintenance event repair, check the GPU or RDMA health signal, the correlated Compute maintenance event, and the maintenance event repair status recorded in the NodeHealthReport.

   Treat controller-owned resources as read-only. Do not create or edit generated actions or controller-owned resources.

2. Review the supported repair status and node state.

   Use the NodeHealthReport to review the repair policy, effective dry-run value, admission result, terminal result, and last repair attempt. Use the affected Kubernetes Node resource to review node conditions. Where applicable, verify the Compute instance lifecycle state, the Compute defined tag used for user-initiated repair, maintenance event details for maintenance event repair, and whether replacement capacity is ready.

   Do not rely on internal leases, request IDs, file paths, polling intervals, or intermediate workflow phases as status indicators.

3. For user-initiated repair, investigate rejected requests, deferred repair, failed repair, and superseded repair.

   A request can be rejected when no NodeRepairConfig resource matches the node or when more than one configuration applies.

   Use the NodeHealthReport result to decide what to investigate:

   • If the NodeHealthReport shows AdmissionRejected, verify the NodeRepairConfig resource, trigger authorization, and label configuration.
   • If the NodeHealthReport shows Deferred, wait for the blocking condition to clear, or adjust capacity if replacement capacity is unavailable.
   • If the NodeHealthReport shows Failed, inspect the NodeHealthReport, Kubernetes Events, pod disruption budgets, IAM policy, Compute defined tag, and Compute instance state.
   • If the NodeHealthReport shows Superseded, follow the OKE workflow that took over the repair.
   • If the NodeHealthReport shows Completed, verify that the original instance was terminated and that replacement capacity is ready.

4. For GPU instance recovery, investigate GPU health monitoring configuration, unavailable health results, reboot signals, signals that require action, incomplete drains, completed reboots with an ongoing fault, and continuing signals that do not trigger repeated reboots.

   Use node conditions, GPU health signals, and the NodeHealthReport result to decide what to investigate:

   • If the GpuHealthMonitoringConfigured node condition is False or Unknown, verify the Oracle Cloud Agent configuration and the GpuHealthMonitoring add-on configuration.
   • If the GpuHealthMonitoringUnavailable node condition is True, verify that the GPU shape is supported, Oracle Cloud Agent version 1.64.0 or later is installed, the Compute RDMA GPU Monitoring plug-in is enabled, and the GPU Health Monitoring add-on is installed and configured.
   • If an active GPU health signal has action=REBOOT, monitor the NodeHealthReport for the GPU instance recovery workflow.
   • If a GPU health signal has action=CUSTOMER_ACTION_REQUIRED, investigate the reported GPU health issue. This signal does not trigger an automated reboot.
   • If the NodeHealthReport shows an incomplete drain or failed repair, inspect Kubernetes Events, pod disruption budgets, workload state, and node conditions.
   • If the reboot completed but the GPU health result still reports the fault, investigate the ongoing fault.
   • If the same signal continues after reboot, GPU instance recovery does not immediately reboot the node again. Cooldown rules prevent immediate reboot loops.

5. For maintenance event repair, investigate GPU Health Monitoring configuration, local GPU or RDMA fault signals, correlated Compute maintenance events, matching NodeRepairConfig resources, paused repairs, incomplete drains, and the result of the STOP or TERMINATE workflow.

   For maintenance event repair, review the oci.oraclecloud.com/MaintenanceEventDetected and oci.oraclecloud.com/NodeAutoRepairPaused node conditions. In the NodeHealthReport, review the maintenance event OCID, action, fault codes, maintenance lifecycle state, repair ID, selected signal IDs, repair action, dry-run value, admission state, rejection details, terminal state, and last repair attempt.

   Use node conditions, maintenance event information, and the NodeHealthReport result to decide what to investigate:

   • If a local GPU or RDMA fault does not start maintenance event repair, verify GPU Health Monitoring, Oracle Cloud Agent, the GPU worker node shape, and the health signals recorded in the NodeHealthReport. Confirm that a matching Compute maintenance event exists for the same instance.
   • If maintenance event repair is not admitted, confirm that exactly one NodeRepairConfig selects the node and that repair policy and disruption limits allow repair. Inspect the NodeHealthReport admission result.
   • If repair is paused, inspect the NodeHealthReport and node conditions, and then use the supported response mechanism to continue or cancel the repair.
   • If a STOP repair completes but the node is not schedulable, check node readiness, cordon state, Compute maintenance event lifecycle state, current health signals, and Kubernetes Events.
   • If a TERMINATE repair completes but expected capacity is not available, check GPU shape capacity, service limits, compartment quotas, and the node pool state. For self-managed nodes, provision and join a replacement node using the node-management process that you normally use for self-managed nodes (see Working with Self-Managed Nodes).

   Relevant OKE repair outcomes include Completed, Failed, Cancelled, Superseded, and AdmissionRejected. A Deferred state represents temporary blocking, not a terminal outcome. Compute maintenance lifecycle values are separate from OKE repair outcomes.

6. Before opening a service request, collect the information needed to investigate the repair workflow.

   Include the tenancy OCID, cluster OCID, region, Kubernetes version, GPU Health Monitoring add-on version, Oracle Cloud Agent version, affected worker node name, Compute instance OCID, relevant NodeHealthReport and NodeRepairConfig resources, node labels and conditions, Kubernetes Events, pod disruption budget and workload state, timestamps, and current GPU health results. For maintenance event repair, also include the Compute maintenance event OCID, maintenance event action, fault codes, maintenance event lifecycle state, and any available maintenance event details from Compute.

   Do not modify OKE-owned CRDs, controller resources, admission objects, generated actions, or coordination resources.