Repairing GPU Worker Nodes with OKE
Cluster Health Checks (CHC) provides evidence about node health. Oracle Kubernetes Engine (OKE) GPU host repair uses eligible health signals and your configured repair policy to coordinate two recovery flows on eligible GPU worker nodes:
- GPU instance recovery responds to an eligible GPU health signal by rebooting the affected instance.
- Maintenance event repair responds when OKE matches a GPU or RDMA network fault with a Compute maintenance event for the same instance. Depending on the event, OKE coordinates non-terminating maintenance or instance termination.
In both flows, OKE stops scheduling new workloads on the node and attempts to drain running workloads before taking the recovery action. You select which nodes OKE can repair, limit simultaneous repairs, and configure how running workloads are handled before repair.
These flows require OKE GPU Health Monitoring and repair configuration. Installing CHC active tests alone does not enable recovery.
Use dry-run mode to inspect proposed repairs before enabling live actions. After recovery, verify node and workload health. A completed workflow does not necessarily mean the underlying fault is resolved.
For an overview of GPU host repair, see Repairing GPU Hosts. For prerequisites and configuration, see Configuring GPU Instance Recovery for Managed GPU Worker Nodes and Configuring Maintenance Event Repair for GPU Worker Nodes.