Using Scripts to Automate the Test Process

You can create fault-injection test scripts by using the logging access type of the th_define(1M) utility:

# th_define -n name -i instance -a log [-e fixup_script]

The th_define command takes the instance offline and brings it back online. Then th_define runs the workload that is described by the fixup_script and logs I/O accesses that are made by the driver instance.

The fixup_script is called twice with the set of optional arguments. The script is called once just before the instance is taken offline, and it is called again after the instance has been brought online.

The following variables are passed into the environment of the called executable:

DRIVER_PATH

Device path of the instance

DRIVER_INSTANCE

Instance number of the driver

DRIVER_UNCONFIGURE

Set to 1 when the instance is about to be taken offline

DRIVER_CONFIGURE

Set to 1 when the instance has just been brought online

Typically, the fixup_script ensures that the device under test is in a suitable state to be taken offline (unconfigured) or in a suitable state for error injection (for example, configured, error free, and servicing a workload). The following script is a minimal script for a network driver:

#!/bin/ksh
driver=xyznetdrv
ifnum=$driver$DRIVER_INSTANCE
 
if [[ $DRIVER_CONFIGURE = 1 ]]; then
   ifconfig $ifnum plumb	
   ifconfig $ifnum ...	
   ifworkload start $ifnum
elif [[ $DRIVER_UNCONFIGURE = 1 ]]; then	
   ifworkload stop $ifnum	
   ifconfig $ifnum down	
   ifconfig $ifnum unplumb
fi
exit $?

The ifworkload command should initiate the workload as a background task. The fault injection occurs after the fixup_script configures the driver under test and brings it online (DRIVER_CONFIGURE is set to 1).

If the -e fixup_script option is present, it must be the last option on the command line. If the -e option is not present, a default script is used. The default script repeatedly attempts to bring the device under test offline and online. Thus the workload consists of the driver's attach() and detach() paths.

The resulting log is converted into a set of executable scripts that are suitable for running unassisted fault-injection tests. These scripts are created in a subdirectory of the current directory with the name driver.test.id. The scripts inject faults, one at a time, into the driver while running the workload that is described by the fixup_script.

The driver tester has substantial control over the errdefs that are produced by the test automation process. See the th_define(1M) man page.

If the tester chooses a suitable range of workloads for the test scripts, the harness gives good coverage of the hardening aspects of the driver. However, to achieve full coverage, the tester might need to create additional test cases manually. Add these cases to the test scripts. To ensure that testing completes in a timely manner, you might need to manually delete duplicate test cases.

Automated Test Process

The following process describes automated testing:

1. Identify the aspects of the driver to be tested.

   Test all aspects of the driver that interact with the hardware:

   • Attach and detach

   • Plumb and unplumb under a stack

   • Normal data transfer

   • Documented debug modes

   A separate workload script (fixup_script) must be generated for each mode of use.

2. For each mode of use, prepare an executable program (fixup_script) that configures and unconfigures the device, and creates and terminates a workload.

3. Run the th_define(1M) command with the errdefs, together with an access type of -a log.

4. Wait for the logs to fill.

   The logs contain a dump of the bofi driver's internal buffers. This data is included at the front of the script.

   Because it can take from a few seconds to several minutes to create the logs, use the th_manage broadcast command to check the progress.

5. Change to the created test directory and run the master test script.

   The master script runs each generated test script in sequence. Separate test scripts are generated per register set.

6. Store the results for analysis.

   Successful test results, such as success (corruption reported) and success (corruption undetected), show that the driver under test is behaving properly. The results are reported as failure (no service impact reported) if the harness detects that the driver has failed to report the service impact after reporting a fault, or if the driver fails to detect that an access or DMA handle has been marked as faulted.

   It is fine for a few test not triggered failures to appear in the output. However, several such failures indicate that the test is not working properly. These failures can appear when the driver does not access the same registers as when the test scripts were generated.

7. Run the test on multiple instances of the driver concurrently to test the multithreading of error paths.

   For example, each th_define command creates a separate directory that contains test scripts and a master script:

   # th_define -n xyznetdrv -i 0 -a log -e script # th_define -n xyznetdrv -i 1 -a log -e script

   Once created, run the master scripts in parallel.

   ---

   Note –

   The generated scripts produce only simulated fault injections that are based on what was logged during the time the logging errdef was active. When you define a workload, ensure that the required results are logged. Also analyze the resulting logs and fault-injection specifications. Verify that the hardware access coverage that the resulting test scripts created is what is required.

   ---