Periodic Health Checks

A latent fault is one that does not show itself until some other action occurs. For example, a hardware failure occurring in a device that is a cold stand-by could remain undetected until a fault occurs on the master device. At this point, it will be discovered that the system now contains two defective devices and might be unable to continue operation.

Generally, latent faults that are allowed to remain undetected will eventually cause system failure. Without latent fault checking, the overall availability of a redundant system is jeopardized. To avoid this, a device driver must detect latent faults and report them in the same way as other faults.

The driver should ensure that it has a mechanism for making periodic health checks on the device. In a fault-tolerant situation in which the device can be the secondary or failover device, early detection of a failed secondary device is essential to ensure that it can be repaired or replaced before any failure in the primary device occurs.

Periodic health checks can:

• Run a quick access check on the board (write, read), then check the device with the ddi_check_acc_handle(9F) routine.

• Check a register or memory location on the device that has a value the driver expects to have been deterministically altered since the last poll.

  Features of a device that typically exhibit deterministic behavior include heartbeat semaphores, device timers (for example, local lbolt that is used by download), and event counters. Reading an updated predictable value from the device gives a degree of confidence that things are proceeding satisfactorily.

• Time-stamp outgoing requests (transmit blocks or commands) when issued by the driver.

  The periodic health check can look for any overaged requests that have not completed.

• Initiate an action on the device that should be completed before the next scheduled check.

  If this action is an interrupt, this is an ideal way of ensuring that the device's circuitry is still capable of delivering an interrupt.