5 Writing Tests

The Test Execution Model

The design and invocation of a test is a reflection of the test execution model that you design for your test suite. The test execution model describes the steps involved in executing the tests in your test suite and is implemented by the test script.

As you design your test suite you should think about how your tests are going to be executed. Some typical questions you might ask:

• Is each test invoked by executing a single class?

• Do the tests require multiple steps, implemented by multiple class invocations?

• Must test classes be started on different machines and in a specific order?

Most TCK test suites test specific aspects of an API. These types of tests lend themselves to an execution model in which tests are run by invoking a single class that exercises a method or group of methods. The JavaTest Architect's release includes the Standard Test Script (com.sun.javatest.lib.StdTestScript) that implements this test execution model. The Standard Test Script is discussed in more detail in Chapter 10.

If your test suite requires a more complex test execution model, you have to create a test script to implement it. See Chapter 10 for information about creating a custom test script.

The Test Interface

If you plan to run your tests using the execution model embodied by the Standard Test Script, the tests must implement the run method of the interface com.sun.javatest.Test. The Test interface provides a very flexible mechanism that is well suited for creating most tests. If the Test interface does not suite your needs, you can write your own interface. You can find information about creating your own interface in Chapter 10.

The Test interface run method takes an array of strings and two output streams and returns a Status object. The array of strings is taken from the executeArgs entry in the test description. The output streams are provided by the JavaTest harness; any output written to the output streams is saved in the TestResult object and is displayed in the Test Run Messages tab in the JavaTest GUI. The end result of the test is a Status object — a combination of an integer code and a message string (see Test Status).

The following code example shows a template for tests written to work with the Standard Test Script; the areas you change are in bold font:

import java.io.PrintWriter;
import com.sun.javatest.Status;
import com.sun.javatest.Test;
 
/** @test
  * @executeClass MyTest
  * @sources MyTest.java
 **/
 
public class MyTest implements Test{
    public static void main(String[] args) {
        PrintWriter out = new PrintWriter(System.err, true);
        Test t = new MyTest();
        Status s = t.run(args, out, null);
        s.exit();
    }
 
    public Status run(String[] args, PrintWriter out1, PrintWriter out2){
        Status result;
        // your test code here ...
        return result;
    }
}

Note that the section delimited with the /** **/ characters is the test description portion of the test which is described in more detail later in this chapter in Test Description Entries. The Status object is described in Test Status.

Test Status

The Status object is an integer/string pair that encodes the exit status of the test. The JavaTest harness supports the following exit status values:

The integer portion of the Status object represents the exit status; the string portion is a message that summarizes the outcome (for example, an error message). Only the short integer portion is used by the JavaTest harness to determine the test status. The message string provides information to the user running the test. The message is passed to the test script which writes it into the test result file.

Note that the object is immutable once it is created — if the test script modifies the message string it must take the Status object created by the test and recreate the Status object including the new message string.

The JavaTest harness uses the information in the Status object in its GUI status displays and reports.

There are two important methods in the Status API that your tests can use: passed() and failed(). Both methods take a string as an argument and return a Status object. The JavaTest harness displays these strings in the Test Run Message tab in the JavaTest GUI and they can be an important source of information to users running the tests. The following example shows how these methods are used:

public Status run(String[] args, PrintWriter out1, PrintWriterout2) {
    Status result;
        if (1 + 1 == 2)
            result = Status.passed("OK");
        else
            result = Status.failed("Simple addition performed incorrectly");
    return result;
    }
}

The test entries in the reports generated by the JavaTest harness are grouped based on the string arguments you supply to Status.passed and Status.failed. It's generally a good idea to keep all of the Status.passed messages short and consistent so that similar tests are grouped together in reports.

Status.failed messages should generally be longer and more descriptive to help the user determine why the test failed. Complete details should be written to the output stream.

By default the Report function sorts passed and failed tests results alphabetically by test location and name (plain view) , or by the final status (grouped view).

See the API documentation (doc\javatest\api) for the Status class.

Test Description Entries

All tests must have an associated test description that contains entries that identify it as a test and provide the information required to run it. Test descriptions are located and read by a test finder; the two standard test finders included with the JavaTest harness read two styles of test description: tag test descriptions and HTML test descriptions. It is your decision as test suite architect which style to use (you can even create a custom style). Test finders are discussed in detail in Chapter 9. For simplicity, only the tag style is shown in this chapter.

Test finders read all entries listed in the test description and add them to the TestDescription object. The Standard Test Script looks for and uses the values specified in the executeClass, executeArgs, and sources entries; the script disregards any other entries. You can create your own custom script that recognizes additional test description entries and validate those entries. See Chapter 10 for more information.

The following table describes the entries understood by the Standard Test Script:

The following code snippet shows how a tag test description appears in a test source file:

/** @test
  * @executeClass MyTest
  * @sources MyTest.java
  * @executeArgs arg1 arg2
  * @keywords keyword1 keyword2
 **/

Multiple Tests in a Single Test File

If you find that you are writing lots of very small tests to test similar aspects of your API, you can group these similar tests together as test cases in a single test file. Tests that contain test cases should use the com.sun.javatest.lib.MultiTest class rather than the com.sun.javatest.Test interface. MultiTest implements com.sun.javatest.Test to add this functionality. One of the major benefits of using MultiTest to implement test cases, is the test cases can be addressed individually in the test suite's exclude list. Another advantage to using MultiTest is that the test cases are run in the same JVM which is generally faster than starting a new JVM for each test. The downside to using MultiTest is that tests are more susceptible to errors introduced by memory leaks.

MultiTest is included with the JavaTest release as a standard library class. MultiTest is a class that implements the com.sun.javatest.Test interface and allows you to write individual test cases as methods with a specific signature. These methods cannot take any parameters and must return a com.sun.javatest.Status object as a result. Argument decoding must be done once by a test for its test case methods. MultiTest uses reflection to determine the complete set of methods that match the specific signature. MultiTest calls test case methods individually, omitting any tests cases that are excluded. The individual Status results from those methods are combined by MultiTest into an aggregate Status object. The test result is presented as a summary of all the test cases in the test.

The following example shows a very simple test that uses MultiTest to implement test cases:

import java.io.PrintWriter;
import com.sun.javatest.Status;
import com.sun.javatest.Test;
import com.sun.javatest.lib.MultiTest;
 
/** @test
  * @executeClass MyTest
  * @sources MyTest.java
 **/
 
public class MyTest extends MultiTest{
    public static void main(String[] args) {
        PrintWriter err = new PrintWriter(System.err, true);
        Test t = new MyTest();
        Status s = t.run(args, null, err);
        // Run calls the individual testXXX methods and
        // returns an aggregate result.
        s.exit();
    }
    public Status testCase1() {
        if (1 + 1 == 2)
            return Status.passed("OK");
        else
            return Status.failed("1 + 1 did not make 2");
    }
    public Status testCase2() {
        if (2 + 2 == 4)
            return Status.passed("OK");
        else
            return Status.failed("2 + 2 did not make 4");
    }
    public Status testCase3() {
        if (3 + 3 == 6)
            return Status.passed("OK");
        else
            return Status.failed("3 + 3 did not make 6");
    }
}

For more information about com.sun.javatest.lib.MultiTest, please refer to the API documentation.

Organizing Tests Within Your Test Suite

This section describes some guidelines about how to organize your test source and class files.

Error Messages

It is important that your tests provide error messages that test users can readily use to debug problems in their test runs. One useful method is for your error messages to compare expected behavior to the actual behavior. For example:

Addition test failed: expected a result of "2"; got "3"

Longer detailed messages should go to the test and/or test script diagnostic streams. Use the Status object for shorter summary messages.