3 Overview

Test Suite Components

The most fundamental components of a test suite are the tests themselves. Tests are typically Java programs that exercise aspects of an API or compiler. To work well with the JavaTest harness, these files are organized in the file system hierarchically. The JavaTest harness finds the tests and displays them in the JavaTest GUI test tree based on this hierarchy.

Before the JavaTest harness can execute a test, it must know some fundamental things about the test — for example, where to find the class file that implements the test and what arguments the test takes. This information is contained in a test description. The test description is a group of name/value pairs that can be embodied in different ways — for example, as tag test descriptions and HTML test descriptions. Tag test descriptions are inserted directly into the test source files using Javadoc style tags. HTML test descriptions are HTML tables contained in HTML files separate from the test source and class files. The examples included with the JavaTest Architect's release demonstrate both types of test descriptions.

The JavaTest harness uses a specialized class called a test finder to locate tests, read test descriptions, and pass test description values to the JavaTest harness. As the architect, you specify a test finder that knows how to read the test descriptions you have designed for your test suite. The JavaTest Architect's release includes test finders that know how to read tag and HTML test descriptions; you can use the included test finders as-is, modify them, or create your own.

Once the test finder locates the test and reads the test description, it is up to the test script to actually run the test. The test script is a Java class whose job is to interpret the test description values, run the tests, and report the results back to the JavaTest harness. As the test suite architect, you are responsible for providing the test script that JavaTest uses. Test scripts can be very simple or complex, depending on the requirements of your test suite. A number of test script examples are included with the JavaTest Architect's release that you can use as is, extend, or use as a template for your test script.

In addition to the fundamental information about each test, the test script might also require platform-specific information about each test to execute it. This information is provided by the person running the tests, usually by completing a wizard-like configuration interview designed by you. Platform-specific information includes information such as the location of the JVM to be used when running the tests, the names of remote computers, and other resources required to run the tests.

When test execution is complete, a test normally creates a Status object and passes it back to the test script; the test script then stores the test results in the TestResult object. Test status includes an integer that represents the status (pass, fail, error, not run) and a short message that describes what happened — for example, an error message. Test results include more detailed information about the results of the test's execution — for example, any additional messages produced by the test.

When the JavaTest harness loads a test suite, the first thing it does is read a file named testsuite.jtt located in the top-level directory of the test suite. The testsuite.jtt file is a registry of information about the test suite that includes the paths to the components described above and other static information about the test suite. The JavaTest harness internalizes this information in a TestSuite object that acts as a portal to all information about the test suite. Whenever the JavaTest harness requires information about the test suite, it queries the TestSuite object. As test suite architect, you create and maintain the testsuite.jtt file.

Figure 3-1 shows a graphical representation of the different test suite components:

The following table summarizes the sequence of steps the JavaTest harness uses to run test suites and your responsibility for each step. These steps are described in more detail in the following chapters.

Remote Execution

It is often convenient or necessary to run tests on a system other than the one running the JavaTest harness. In this case, an agent must be used to run the tests on the test platform and communicate with JavaTest harness. The JavaTest harness provides a general purpose agent (JavaTest Agent), but test architects can also create custom agents.

The JavaTest Agent is a lightweight program compatible with JDK 1.1 (does not require the Java SE platform, or Swing). The JavaTest Agent uses a bidirectional connection to communicate between the test platform and JavaTest—it supports both the TCP/IP and RS-232 protocols. Other types of connections can be added through the JavaTest API, for example, infrared, parallel, USB, firewire connections can be added and modelled on the existing system. If a test platform meets the following requirements the JavaTest Agent will probably work well:

• The device supports a communication layer that can last the duration of a test (couple of minutes)

• The agent code can be loaded into the device

If the test platform does not meet these requirements, the JavaTest API can be used to create a custom agent. Agents have been created to run tests on devices such as cell phones, PDAs, and pagers.