Fault Injection

In general, a device driver is connected to its parent bus driver using the bus driver interface (bus DDI, either common or bus architecture specific) as shown by the dashed line in Figure 14-1.

Typically, a device driver uses its bus DDI to:

• Map and access device I/O registers.

• Map device memory.

• Allocate and manage DMA buffers.

• Notify device interrupts.

• Notify bus exceptions.

• Notify bus and system events.

Figure 14-1 Driver Framework: Fault Injection Bus Drivers

The Driver Framework defines the DDIs and mechanisms to implement fault injection bus drivers which can be transparently inserted between the effective bus driver and the hardened device driver being tested, as shown in Figure 14-1 by continuous lines.

Basically, the fault injection (fi) bus driver is like a filter between the bus and device drivers. Note that the existence of such a filter is totally transparent to both bus and device drivers. From a bus driver perspective, the fi driver looks like a normal device driver instance started on a child node. From the device driver perspective, the fi driver looks like a normal parent bus driver providing the requested bus DDI (for example, PCI DDI).

The bus fi driver is responsible for:

• Binding itself in place of the effective device driver being tested (and retaining the initial binding).

• Starting the device driver instance.

• Providing its own bus DDI operations to the tested driver instance.

• Optionally providing an additional bus FI DDI which defines the fault injection operations allowed on that bus.

This bus FI DDI may be used by a bus fi driver's client to test, or validate, the hardened device driver. The bus DDI operations implemented in the bus fi driver are basically wrapper functions which simply call appropriate methods on the parent bus DDI (in an upstream direction) and call appropriate handlers in the device driver (in a downstream direction). However, internally or through its client, the bus fi driver's behavior can be changed by injecting faults to simulate a device malfunction. For example:

• load() may return a corrupted value for a given register.

• store() may ignore the write, or corrupt value to be written, to a given register.

• A device driver interrupt handler may be called to simulate spurious, or stuck, interrupts.

• An interrupt, delivered by the bus driver, may be ignored to simulate lost interrupts.

• A device driver exception handler may be called to simulate bus exceptions.

In addition, because the fi driver is able to snoop all requests issued by the device driver, it is able to corrupt memory regions and DMA buffers mapped, or allocated, by the device driver.

By watching the device driver requests issued to the bus, the fi driver is able to perform multiple checks on the validity of those requests and their arguments. This allows you to remove all the debugging checks from the real bus driver's code, and to detect many other problems. By monitoring driver requests, a fi driver is able to:

• Check if a bus resource is allocated before being used. For example, a region being mapped (I/O, memory) must be already allocated to a bus, as a resource.

• Check if a bus resource is not used twice. For example, a region should not be mapped twice.

• Check that arguments are in the correct range. For example, the offset of an I/O load and store operation must be within the mapped I/O region.

• Check that all bus resources are freed when the tested driver closes its connection with the bus.

There is one instance of the fi bus driver running for each hardened driver tested.

Each instance may have one or no client at all.

A fi bus driver may be either:

• Device oriented

• Bus oriented (generic)

A fi bus driver may be specifically developed for a given device (connected to a given bus architecture). This type of driver would typically incorporate many, embedded fault scenarios which are device hardware-specific. A driver of this kind could be a self-contained validation test for a given hardened device driver.

On the other hand, you could develop a generic fi bus driver for a given bus architecture. This type of driver would typically provide an FI DDI interface allowing you to dynamically specify a fault to be injected. Then, using such an interface, the fi driver may be driven by a client application. A client of this kind could be a fault scenario interpreter.

In either case, the Driver Framework should define:

• A mechanism to allow the bus fi driver to start transparently.

• A new bus FI DDI for each existing bus DDI (each class of bus, for example BUS, PCI, ISA and so on).