ChorusOS 5.0 Board Support Package Developer's Guide

Chapter 14 Testing Hardened Drivers

Testing the resilience of a hardened driver is crucial to its success. This chapter describes hardened driver testing and validation through fault injection.

"Fault Injection" explains how fault injection is used to test hardened drivers.
"Initializing a Fault Injection Bus Driver" illustrates fault injection testing in a bus driver.

Fault Injection

In general, a device driver is connected to its parent bus driver using the bus driver interface (bus DDI, either common or bus architecture specific) as shown by the dashed line in Figure 14-1.

Typically, a device driver uses its bus DDI to:

Map and access device I/O registers.
Map device memory.
Allocate and manage DMA buffers.
Notify device interrupts.
Notify bus exceptions.
Notify bus and system events.

Figure 14-1 Driver Framework: Fault Injection Bus Drivers

The Driver Framework defines the DDIs and mechanisms to implement fault injection bus drivers which can be transparently inserted between the effective bus driver and the hardened device driver being tested, as shown in Figure 14-1 by continuous lines.

Basically, the fault injection (fi) bus driver is like a filter between the bus and device drivers. Note that the existence of such a filter is totally transparent to both bus and device drivers. From a bus driver perspective, the fi driver looks like a normal device driver instance started on a child node. From the device driver perspective, the fi driver looks like a normal parent bus driver providing the requested bus DDI (for example, PCI DDI).

The bus fi driver is responsible for:

Binding itself in place of the effective device driver being tested (and retaining the initial binding).
Starting the device driver instance.
Providing its own bus DDI operations to the tested driver instance.
Optionally providing an additional bus FI DDI which defines the fault injection operations allowed on that bus.

This bus FI DDI may be used by a bus fi driver's client to test, or validate, the hardened device driver. The bus DDI operations implemented in the bus fi driver are basically wrapper functions which simply call appropriate methods on the parent bus DDI (in an upstream direction) and call appropriate handlers in the device driver (in a downstream direction). However, internally or through its client, the bus fi driver's behavior can be changed by injecting faults to simulate a device malfunction. For example:

load() may return a corrupted value for a given register.
store() may ignore the write, or corrupt value to be written, to a given register.
A device driver interrupt handler may be called to simulate spurious, or stuck, interrupts.
An interrupt, delivered by the bus driver, may be ignored to simulate lost interrupts.
A device driver exception handler may be called to simulate bus exceptions.

In addition, because the fi driver is able to snoop all requests issued by the device driver, it is able to corrupt memory regions and DMA buffers mapped, or allocated, by the device driver.

By watching the device driver requests issued to the bus, the fi driver is able to perform multiple checks on the validity of those requests and their arguments. This allows you to remove all the debugging checks from the real bus driver's code, and to detect many other problems. By monitoring driver requests, a fi driver is able to:

Check if a bus resource is allocated before being used. For example, a region being mapped (I/O, memory) must be already allocated to a bus, as a resource.
Check if a bus resource is not used twice. For example, a region should not be mapped twice.
Check that arguments are in the correct range. For example, the offset of an I/O load and store operation must be within the mapped I/O region.
Check that all bus resources are freed when the tested driver closes its connection with the bus.

There is one instance of the fi bus driver running for each hardened driver tested.

Each instance may have one or no client at all.

A fi bus driver may be either:

Device oriented
Bus oriented (generic)

A fi bus driver may be specifically developed for a given device (connected to a given bus architecture). This type of driver would typically incorporate many, embedded fault scenarios which are device hardware-specific. A driver of this kind could be a self-contained validation test for a given hardened device driver.

On the other hand, you could develop a generic fi bus driver for a given bus architecture. This type of driver would typically provide an FI DDI interface allowing you to dynamically specify a fault to be injected. Then, using such an interface, the fi driver may be driven by a client application. A client of this kind could be a fault scenario interpreter.

In either case, the Driver Framework should define:

A mechanism to allow the bus fi driver to start transparently.
A new bus FI DDI for each existing bus DDI (each class of bus, for example BUS, PCI, ISA and so on).

Initializing a Fault Injection Bus Driver

This section describes one way of starting a fi bus driver, transparently, to test particular devices.

Devices for which fault simulation is required are dynamically identified by the fi bus driver at binding time.

The following property is defined:

PROP_FI_DRIVER: serves as a secondary binding property for a device node. It is used to copy the initial binding while starting the fi bus driver instead of the driver initially bound. Its value type is a null terminated string, as for the PROP_DRIVER property.

A fi bus driver registers itself in the driver registry, specifying the bus class it emulates as its required parent class. For example, a fault injection bus driver emulating the PCI DDI bus class requires a PCI class as its parent bus class, like any other PCI device driver.

Probing

A fi bus driver has an empty (NULL) drv_probe() method. As it is a pseudo driver, having no associated physical device, and because it binds itself to tested device nodes (instead of effective device drivers), it does not need to create any device tree node.

Binding

The drv_bind() routine of a fi bus driver is called for each node that is a child node of the emulated bus class. In this routine, the driver gets the opportunity to detect a device requiring fault injection and binds itself to this device, instead of the initially bound device driver.

Code Example 14-1 illustrates how a generic PCI Fault Injection bus driver (pciFi(9DRV)) may bind itself to all PCI device nodes. By default, this driver is transparent and only becomes active for a particular device when requested by a client.

Example 14-1 pciFi `drv_bind()` routine

    /*
     * Driver bind method
     */
    static void
drv_bind (DevNode node)
{
    DevProperty propDriver;
    DevProperty propNewDriver;
    DevProperty propFiDriver;

    propDriver = dtreePropFind(node, PROP_DRIVER);
        /*
         * If the device node is not bound to any driver
         * try to bind it
         */
    if (! propDriver) {
        DrvRegId     drv_curr;
        DrvRegId     drv_prev;
        DrvRegEntry* entry;
        
        drv_curr = svDriverLookupFirst();
        while (drv_curr) {
            entry = svDriverEntry(drv_curr);
            if (entry != &pciFiDrv) {
                if (entry->drv_bind &&
                    !strcmp(pciFiDrv.bus_class, entry->bus_class) &&
                    (pciFiDrv.bus_version >= entry->bus_version)) {
                        /*
                         * Try to bind the node
                         */
                    entry->drv_bind(node);
                }
            }
            drv_prev = drv_curr;
            drv_curr = svDriverLookupNext(drv_curr);
            svDriverRelease(drv_prev);
        }
        propDriver = dtreePropFind(node, PROP_DRIVER);
    }
            /*
             * If node is bound to a driver, copy its PROP_DRIVER property
             * into a PROP_FI_DRIVER property. Then bind our own driver to
             * the node, in order to be started in place of the original
             * driver.
             */
    if (propDriver && !dtreePropFind(node, PROP_FI_DRIVER)) {
        propFiDriver = dtreePropAdd(node, PROP_FI_DRIVER,
                                    dtreePropValue(propDriver),
                                    dtreePropLength(propDriver));
        if (!propFiDriver) {
            return;
        }
                /*
                 * Replace old PROP_DRIVER with my own driver name
                 */
        propNewDriver = dtreePropAdd(node, PROP_DRIVER, pciFiDrv.drv_name,
                                     strlen(pciFiDrv.drv_name) + 1);
        if (!propNewDriver) {
            dtreePropDetach(propFiDriver);
            dtreePropFree(propFiDriver);
            return;
        }
        dtreePropDetach(propDriver);
        dtreePropFree(propDriver);
    }
}

At the end of the binding phase, all device nodes (children of the emulated bus class node) are bound to the fi bus driver. The standard binding mechanism is not disturbed and the original bindings are duplicated in a PROP_FI_DRIVER property in each node.

Initializing

At initialization, the fi bus driver's drv_init() method is called for each device node tested. The fi bus driver will then get the opportunity to:

Launch an instance of itself, on that node.
Launch an instance of the tested device driver referenced in the PROP_FI_DRIVER property, giving its own bus operations instead of the original bus driver operations.
Optionally register itself in the device registry, to export a bus FI DDI, if it is intended to have a client.

Code Example 14-2 illustrates the initialization of a generic PCI Fault Injection bus driver (pciFi(9DRV)). In this example, the PciFiDev structure contains the fault injection driver instance specific data. This data is allocated and initialized in drv_init(). Fields of interest for the examples are:

node, which contains the bus FI driver's device node.
dev.node, which contains the tested child driver's device node.
entry, which contains data to be registered in the device registry.
devRegId, which contains the identifier of allocated device registry entry.

Example 14-2 pciFi `drv_init()` routine

    /*
     * Try to start (initialize) tested child driver
     */
    static KnError
childInit(PciFiDev* pciFi)
{
    char*        drv_name;
    DrvRegId     drv_curr;
    DrvRegId     drv_prev;
    DrvRegEntry* entry;
    DevProperty  prop;
    DevNode      node;
        /*
         * Check if not already started
         */
    if (pciFi->dev.node) {
        return K_EBUSY;
    }
    node = pciFi->node;
        /* 
         * Check for PROP_FI_DRIVER property
         */
    prop = dtreePropFind(node, PROP_FI_DRIVER);
    if (prop == NULL) {
        DKI_ERR(("%s: error -- %s required property not found\n",
                 pciFi->path, PROP_FI_DRIVER));
        return K_EFAIL;
    }
        /* 
         * Try to start PROP_FI_DRIVER driver
         */
    drv_name = (char*)dtreePropValue(prop);
    drv_curr = svDriverLookupFirst();
    while (drv_curr) {
        entry = svDriverEntry(drv_curr);
        if (entry->drv_init &&
            !strcmp(pciFiDrv.bus_class, entry->bus_class) &&
            (pciFiDrv.bus_version >= entry->bus_version) &&
            !strcmp(drv_name, entry->drv_name)) {
            entry->drv_init(node, &pciFiPciBusOps, pciFi);
            if (pciFi->dev.node) {
                svDriverRelease(drv_curr);
                break;
            }
        }
        drv_prev = drv_curr;
        drv_curr = svDriverLookupNext(drv_curr);
        svDriverRelease(drv_prev);
    }

    return (pciFi->dev.node ? K_OK : K_EFAIL);
}

    /*
     * Driver initialization method
     */
    static void
drv_init (DevNode node, void* busOps, void* busId)
{
    PciFiDev*      pciFi;
    int            pathSz;
    char*          path;
    KnError        res;
    DevProperty    prop;
    PciPropBusNum  bus;
    PciPropDevNum  dev;
    PciPropFuncNum func;
        /*
         * Get my path name in the device tree (for errors)
         */
    pathSz = dtreePathLeng(node);
    path   = (char*)svMemAlloc(pathSz);
    if (!path) {
        DKI_ERR(("%s: error -- not enough memory\n", pciFiDrv.drv_name));
        return;
    }
    dtreePathGet(node, path);
        /*
         * Get mandatory properties
         */
    [...]
        /*
         * Allocate driver instance data
         */
    pciFi = (PciFiDev*)svMemAlloc(sizeof(PciFiDev));
    if (! pciFi) {
        DKI_ERR(("%s: error -- not enough memory\n", path));
        svMemFree(path, pathSz);
        return;
    }
        /*
         * Initialize driver instance data
         */
    [...]
        /*
         * Allocate objects associated to allocated resources
         */
    [...]
        /*
         * Open parent PCI bus connection
         */
    res = pciFi->pciOps->open(busId,
                              node,
                              eventHandler, /* my event handler   */
                              loadHandler,  /* my load handler    */
                              pciFi,        /* my handlers cookie */
                              &pciFi->pciDevId);
    if (res != K_OK) {
        DKI_ERR(("%s: error -- open() failed (%d)\n", path, res));
        svMemFree(path, pathSz);
        svMemFree(pciFi, sizeof(PciFiDev));
        return;
    }
        /*
         * Allocate PCI_FI instance driver descriptor in the device registry
         */
    pciFi->entry.dev_class = PCIFI_CLASS;
    pciFi->entry.dev_id    = pciFi;
    pciFi->entry.dev_node  = node;
    pciFi->entry.dev_ops   = &pciFiOps;
    pciFi->devRegId        = svDeviceAlloc(&pciFi->entry,
                                           PCIFI_VERSION_INITIAL,
                                           FALSE, /* pciFi cannot be 
                                                   * shared */
                                           relHandler);
    if (! pciFi->devRegId) {
        DKI_ERR(("%s: error -- not enough memory\n", path));
        pciFi->pciOps->close(pciFi->pciDevId);
        svMemFree(pciFi, sizeof(PciFiDev));
        svMemFree(path, pathSz);
        return;
    }
        /*
         * Chain driver instance in list
         */
    pciFi->next = pciFiDevs;
    pciFiDevs   = pciFi;
        /*
         * Finally, register the new device driver instance
         * in the device registry. In case a shut down event
         * has been signaled during the initialization, the device entry
         * remains invalid and the relHandler() handler is invoked
         * to shut down the device driver instance. Otherwise, the device
         * entry becames valid and therefore visible for driver clients.
         */
    svDeviceRegister(pciFi->devRegId);

    DKI_MSG(("%s: %s driver started\n", path, pciFiDrv.drv_name));

        /*
         * Try to start a tested driver. If this fails, a tested driver
         * may be started later by the loadHandler() handler.
         */
    (void)childInit(pciFi);
}