JavaTest Harness Architect's Guide, JavaTest Harness 4.5 for the Java Platform E20663-03 |
|
Previous |
Next |
This chapter describes how to write tests that work well with the JavaTest harness. Special attention is paid to the test execution model implemented by the Standard Test Script which is designed to work well with test suites that test the compatibility of Java APIs and should work well with most Java SE technology-based TCK test suites.
Note that this discussion focuses on the mechanics of writing tests that work with the JavaTest harness. For information about the "art" of writing compatibility tests, see the Test Suite Developers Guide.
The example test suites included with the JavaTest Architect's release contain numerous test examples. See the following directories:
jt_install\examples\javatest\simpleTags\tests
jt_install\examples\javatest\simpleHTML\tests
You might find it helpful to refer to those tests as you read this chapter.
The design and invocation of a test is a reflection of the test execution model that you design for your test suite. The test execution model describes the steps involved in executing the tests in your test suite and is implemented by the test script.
As you design your test suite you should think about how your tests are going to be executed. Some typical questions you might ask:
Is each test invoked by executing a single class?
Do the tests require multiple steps, implemented by multiple class invocations?
Must test classes be started on different machines and in a specific order?
Most TCK test suites test specific aspects of an API. These types of tests lend themselves to an execution model in which tests are run by invoking a single class that exercises a method or group of methods. The JavaTest Architect's release includes the Standard Test Script (com.sun.javatest.lib.StdTestScript
) that implements this test execution model. The Standard Test Script is discussed in more detail in Chapter 10.
If your test suite requires a more complex test execution model, you have to create a test script to implement it. See Chapter 10 for information about creating a custom test script.
Note: The test execution model implemented by the Standard Test Script includes an optional compilation step. The Standard Test Script can be used to:
See Chapter 7 for more information about compiling tests with the JavaTest harness. |
Test
InterfaceIf you plan to run your tests using the execution model embodied by the Standard Test Script, the tests must implement the run
method of the interface com.sun.javatest.Test
. The Test
interface provides a very flexible mechanism that is well suited for creating most tests. If the Test
interface does not suite your needs, you can write your own interface. You can find information about creating your own interface in Chapter 10.
The Test
interface run
method takes an array of strings and two output streams and returns a Status
object. The array of strings is taken from the executeArgs
entry in the test description. The output streams are provided by the JavaTest harness; any output written to the output streams is saved in the TestResult
object and is displayed in the Test Run Messages tab in the JavaTest GUI. The end result of the test is a Status
object — a combination of an integer code and a message string (see Test Status).
The following code example shows a template for tests written to work with the Standard Test Script; the areas you change are in bold font:
import java.io.PrintWriter;
import com.sun.javatest.Status;
import com.sun.javatest.Test;
/** @test
* @executeClass MyTest * @sources MyTest.java **/public class
MyTestimplements Test{
public static void main(String[] args) {
PrintWriter out = new PrintWriter(System.err, true);
Test t = new
MyTest();
Status s = t.run(args, out, null);
s.exit();
}
public Status run(String[] args, PrintWriter out1, PrintWriter out2){
Status result;
// your test code here ...return result;
}
}
Note that the section delimited with the /**
**/
characters is the test description portion of the test which is described in more detail later in this chapter in Test Description Entries. The Status
object is described in Test Status.
The com.sun.javatest.Test
interface is delivered in javatest.jar
; however, you should extract it into your test suite's classes
directory so that it is easily available to all of your test classes.
Note: To improve test performance, never add |
The Status
object is an integer/string pair that encodes the exit status of the test. The JavaTest harness supports the following exit status values:
Table 5-1 Exit Status Values
Status | Meaning |
---|---|
A test passes when the functionality being tested behaves as expected. |
|
A test fails when the functionality being tested does not behave as expected. |
|
A test is considered to be in error when something (usually a configuration problem) prevents the test from executing as expected. Errors often indicate a systemic problem — a single configuration problem can cause many tests to fail. For example, if the path to the Java runtime is configured incorrectly, no tests can run and all are in error. |
|
NOT_RUN |
A specific test has not been run. |
Note: NOT_RUN is a special case and is reserved for internal JavaTest harness use only. |
The integer portion of the Status
object represents the exit status; the string portion is a message that summarizes the outcome (for example, an error message). Only the short integer portion is used by the JavaTest harness to determine the test status. The message string provides information to the user running the test. The message is passed to the test script which writes it into the test result file.
Note that the object is immutable once it is created — if the test script modifies the message string it must take the Status
object created by the test and recreate the Status
object including the new message string.
The JavaTest harness uses the information in the Status
object in its GUI status displays and reports.
There are two important methods in the Status
API that your tests can use: passed()
and failed()
. Both methods take a string as an argument and return a Status
object. The JavaTest harness displays these strings in the Test Run Message tab in the JavaTest GUI and they can be an important source of information to users running the tests. The following example shows how these methods are used:
public Status run(String[] args, PrintWriter out1, PrintWriterout2) {
Status result;
if (1 + 1 == 2) result = Status.passed("OK"); else result = Status.failed("Simple addition performed incorrectly");return result;
}
}
The test entries in the reports generated by the JavaTest harness are grouped based on the string arguments you supply to Status.passed
and Status.failed
. It's generally a good idea to keep all of the Status.passed
messages short and consistent so that similar tests are grouped together in reports.
Status.failed
messages should generally be longer and more descriptive to help the user determine why the test failed. Complete details should be written to the output stream.
By default the Report function sorts passed and failed tests results alphabetically by test location and name (plain view) , or by the final status (grouped view).
See the API documentation (doc\javatest\api
) for the Status
class.
All tests must have an associated test description that contains entries that identify it as a test and provide the information required to run it. Test descriptions are located and read by a test finder; the two standard test finders included with the JavaTest harness read two styles of test description: tag test descriptions and HTML test descriptions. It is your decision as test suite architect which style to use (you can even create a custom style). Test finders are discussed in detail in Chapter 9. For simplicity, only the tag style is shown in this chapter.
Test finders read all entries listed in the test description and add them to the TestDescription
object. The Standard Test Script looks for and uses the values specified in the executeClass
, executeArgs
, and sources
entries; the script disregards any other entries. You can create your own custom script that recognizes additional test description entries and validate those entries. See Chapter 10 for more information.
The following table describes the entries understood by the Standard Test Script:
Table 5-2 Default Test Description Entries
Test Description Entry | Description |
---|---|
test |
Identifies the comment block as a test description. This entry is required. There is no "test" entry in the |
Specifies the name of the test's executable class file (assumed to be located in the |
|
Specifies the arguments (if any) that the test accepts. This entry is a list of strings separated by white space. This entry is optional. |
|
Specifies the names of the source files required to compile the test. This entry is required if you use the JavaTest harness to compile your tests. See Chapter 7 for more information. This tag is also used by the JavaTest harness to display a test's sources in the Files tab of the Test pane. This entry is optional. |
|
keywords |
Specifies keywords that the user can specify to direct the JavaTest harness to include or exclude tests from a test run. Keyword values consists of a list of words (letters and numbers only) separated by white space. This entry is optional. |
The following code snippet shows how a tag test description appears in a test source file:
/** @test * @executeClass MyTest * @sources MyTest.java * @executeArgs arg1 arg2 * @keywords keyword1 keyword2 **/
You can add keywords to test descriptions that provide a convenient means by which users can choose to execute or exclude pre-selected groups of tests. The person who runs the test suite can specify keyword expressions in the configuration editor. When the test suite is run, the JavaTest harness evaluates the keyword expressions and determines which tests to run based on the keywords specified in the test description. See the JavaTest harness online help for information about specifying keyword expressions.
If you find that you are writing lots of very small tests to test similar aspects of your API, you can group these similar tests together as test cases in a single test file. Tests that contain test cases should use the com.sun.javatest.lib.MultiTest
class rather than the com.sun.javatest.Test
interface. MultiTest
implements com.sun.javatest.Test
to add this functionality. One of the major benefits of using MultiTest
to implement test cases, is the test cases can be addressed individually in the test suite's exclude list. Another advantage to using MultiTest is that the test cases are run in the same JVM which is generally faster than starting a new JVM for each test. The downside to using MultiTest is that tests are more susceptible to errors introduced by memory leaks.
MultiTest
is included with the JavaTest release as a standard library class. MultiTest
is a class that implements the com.sun.javatest.Test
interface and allows you to write individual test cases as methods with a specific signature. These methods cannot take any parameters and must return a com.sun.javatest.Status
object as a result. Argument decoding must be done once by a test for its test case methods. MultiTest
uses reflection to determine the complete set of methods that match the specific signature. MultiTest
calls test case methods individually, omitting any tests cases that are excluded. The individual Status
results from those methods are combined by MultiTest
into an aggregate Status
object. The test result is presented as a summary of all the test cases in the test.
The following example shows a very simple test that uses MultiTest to implement test cases:
import java.io.PrintWriter;
import com.sun.javatest.Status;
import com.sun.javatest.Test;import com.sun.javatest.lib.MultiTest;
/** @test
* @executeClass MyTest * @sources MyTest.java **/public class
MyTestextends MultiTest{
public static void main(String[] args) {
PrintWriter err = new PrintWriter(System.err, true);
Test t = new
MyTest();
Status s = t.run(args, null, err);
// Run calls the individual testXXX methods and // returns an aggregate result.s.exit();
}
public Status testCase1() {
if (1 + 1 == 2)
return Status.passed("OK");
else
return Status.failed("1 + 1 did not make 2");
}
public Status testCase2() {
if (2 + 2 == 4)
return Status.passed("OK");
else
return Status.failed("2 + 2 did not make 4");
}
public Status testCase3() {
if (3 + 3 == 6)
return Status.passed("OK");
else
return Status.failed("3 + 3 did not make 6");
}
}
For more information about com.sun.javatest.lib.MultiTest
, please refer to the API documentation.
If you create a number of tests that are similar you can create a super class to implement functionality they have in common. You can also create this class as a subtype of the MultiTest
class rather than the Test
interface so that you can take advantage of the test case functionality it provides. Such subtypes are typically used to perform common argument decoding and validation, or common set-up and tear-down before each test or test case.
This section describes some guidelines about how to organize your test source and class files.
It is very important to ship the source files for tests in your test suite. Test users must be able to look at the sources to help debug their test runs.
Test sources should be located with the files that contain their test descriptions. If you use tag test descriptions, the test description is included as part of the source file; however, if you use HTML test descriptions, they are contained in separate HTML files that should be included in the same directories as their test source files.
The JavaTest harness assumes that tests are organized hierarchically in a tree structure under the ts_dir/tests
directory. The test hierarchy contained in the tests
directory is reflected in the test tree panel in the JavaTest GUI (technically, it is a tree of the test descriptions). When you organize your tests
directory, think about how it will look in the test tree panel. In test suites that test APIs, the upper part of the tree generally reflects the package structure of the product you are testing. Farther down the tree, you can organize the tests based on the sub-packages and classes being tested. The leaves of the tree might contain one test per method of that class. In some cases it might make sense to organize the tree hierarchy based on behavior; for example, you could group all event handling tests in one directory.
Experience has shown that it is a good idea to place all of your test class files in the ts_dir\classes
directory rather than locating them with the source files in the ts_dir\tests
directory. Placing class files in the classes
directory has the following benefits:
It simplifies the specification of the test execution class path, especially on smaller devices that can only specify a single class path for all the tests.
The standard configuration interview automatically places ts_dir\classes
on the test class path
It permits easier code sharing among tests
Note: In some cases the test platform may dictate where you can put your classes. For example, if your test platform requires the use of an application manager, it may require that your classes be placed in a specific location. |
It is important that your tests provide error messages that test users can readily use to debug problems in their test runs. One useful method is for your error messages to compare expected behavior to the actual behavior. For example:
Addition test failed: expected a result of "2"; got "3"
Longer detailed messages should go to the test and/or test script diagnostic streams. Use the Status object for shorter summary messages.