Java Device Test Suite Developer’s Guide

You must precede each test class and each test case with a descriptive comment block. These blocks identify the classes and methods that represent test classes and cases. They also give the build system and the harness information required to build and bundle tests and to display configurable properties to testers. The comment blocks are similar to Javadoc tool “doc comments” described at http://java.sun.com/j2se/javadoc/writingdoccomments/

This chapter has the following sections:

• Comment Block Overview
• Tag Details

---

Comment Block Overview

When you build a test pack, the build system generates the files shown in FIGURE 4-1 from comment blocks you code for each test class and test case.

FIGURE 4-1 Files Generated from Comment Blocks

The default build generates the following files:

• Documentation files - HTML files created by the Javadoc tool. When a tester clicks the Documentation tab, the graphical harness displays these files. There are additional test pack and package documentation files which you create separately as described in Chapter 5.
• Card file - Test class and case information used by the harness, such as property definitions and a list of related files that test cases require to run. The word “card” has no significance.

You can alternatively create a card file manually, as described in Chapter 25. You can turn off the autogeneration of card files with a build option.

You can verify the contents of a card file using the tool described in Chapter 12.

• Evaluation File - For interactive tests only, an HTML file that contains tester instructions for running and evaluating the test. You can alternatively create an evaluation file manually, as described in Chapter 27.

CODE EXAMPLE 4-1 shows a typical test case comment block.

/**
   * Interactive test that asks user to install more than one test MIDlet
   * suite associated with this test case. In order to do that the developer needs
   * to define the <code>JADPath&lt;N&gt;</code> and 
   * <code>JARPath&lt;N&gt;</code> properties, where
   * <code>N</code> is decimal integer starting from 1 to infinity (without gaps). 
   * For instance, if you need a testcase to install 3 MIDlet suites, use N=1,2, and 3.
   * Each property contains the pointer to the appropriate test MIDlet.
   * 
   * @testcase
   * 
   * @precondition none
   * 
   * @userInteraction
   * <ul>
   * <li>Install <code>syspropviewer.jad</code> using the
   *    provided URL</li>
   * <li>Select installed MIDlet and launch it</li>
   * <li>Install <code>choicegroup.jad</code> using the
   *    provided URL</li>
   * <li>Select installed MIDlet and launch it.</li>
   * </ul>
   * @postcondition The 2 MIDlet suites are installed
   * @passCriteria User is able to install both of the MIDlet suites and
   *    launch them. The "SysPropertiesViewer" MIDlet should show a list of 
   *    system properties describing your device. "ChoiceGroupDemo" application 
   *    should display different types of choice group UI.
   * 
   * @card.property JADPath1=syspropviewer/syspropviewer.jad
   * @card.property JARPath1=syspropviewer/syspropviewer.jar
   * @card.property JADPath2=choicegroup/choicegroup.jad
   * @card.property JARPath2=choicegroup/choicegroup.jar
   */
void t02InstallMoreThanOneMIDletSuite();

FIGURE 4-2 and FIGURE 4-3 show the (rendered) test case documentation and evaluation file generated from the block comment shown in CODE EXAMPLE 4-1.

FIGURE 4-2 Typical Generated Test Case Documentation

FIGURE 4-3 Typical Generated Interactive Test Evaluation Instructions

Although there is some flexibility in formatting a comment block, observe the following rules:

• Each block begins with a line containing only /** and ends with a line containing only */. The class or test case declaration associated with the comment block follows the */.
• Subsections of a comment block are introduced by tags, which have the form @tagName. The supported tags are defined in TABLE 4-1 and TABLE 4-2. You can also include standard Javadoc tool tags, such as @since, @throws. You can use {@inheritDoc} in JavaDoc tool tags. Case is significant in tag names. Write @userInteraction rather than @userinteraction.
• The untagged lines preceding the first tag introduce the class or case. In Javadoc tool terms, these lines are the “main description”
• Each block must have an @testcase or an @testclass tag.
• If the same tag appears in both the test class comment block and a test case comment block, the precedence rules described in Chapter 16 apply.
• All tags except @card.*, @testcase, and @testclass can be followed by lines of simple (version 4.0.1) HTML to create formatted output such as bullet lists. The lines following @userInteraction in CODE EXAMPLE 4-1 give an example.
• @card.* tags have these rules:
• They cannot contain HTML.
• Lines terminated by a \ (backslash) character continue on the next line. On the continuation line, the * (asterisk) character is ignored. For example:

* @card.property x=one \
*two

The value of x is one two.

• You can use Java programming language properties file escape sequences. For example: \\ (backslash), \uHHHH (Unicode character with hexadecimal value HHHH). However, do not use \n (line break) in the value of a property. The line break character might cause unexpected behavior. For property file details, see http://java.sun.com/j2se/1.5.0/docs/api/java/util/
  Properties.html#load(java.io.InputStream)

---

Test Class Comment Block Tags

TABLE 4-1 describes the Java Device Test Suite tags that can or must appear in a test class comment block. The Instances column specifies how many instances of a tag are permitted.

---

Test Case Comment Block Tags

TABLE 4-2 describes the Java Device Test Suite tags that can or must appear in a test case comment block. The Instances column specifies how many instances of a tag are permitted.

For tags that can be used with both test classes and test cases, use the class tag to specify values that apply to all cases in a class. Use the case tag to specify values that apply to one case only. For example, suppose you specify the following:

* @testclass
* @keyword interactive
* ...
* @testcase
* @keyword onePartnerMIDlet 

This means that all test cases in the class can be filtered with the interactive keyword but only the specified test case can also be filtered with onePartnerMIDlet.

---

Tag Details

Tags with more complex syntax or semantics are described in this section.

@card.property

If a test class or case needs a user-visible property, name the property and specify its default value with the following syntax:

* @card.property PropertyName=DefaultValue

For example:

* @card.property MaxDistance=12

Properties that apply to multiple classes can be defined globally at the test pack level. Such properties are defined in the testsuite.info file at the top of the test pack directory in the file system. A property defined at the test pack level can be overridden by defining a property of the same name within the test class or case comment blocks. See Properties and Parameter Expansion for details.

The following test pack property attributes are also supported for test class and case properties.

• scope: To hide a property from users, specify the value hidden. The advanced value is not supported for classes and cases.
• readonly: To prevent user modification of the property’s default value, specify the value true.
• doc: Give a short description of the property, for example:

* @card.property PixelHeightNoLessThan.doc=minimum number of pixels allowed for object height

FIGURE 4-4 shows an example of doc values that appear when a tester right-clicks a test case in the harness test tree and chooses Configure Test. Read-only properties are displayed in gray (for example, JADPath1 in FIGURE 4-4).

FIGURE 4-4 doc Properties in Configure Test Window

For the exact syntax of property attributes, refer to Scope, Read-only Properties, and Online Documentation for a Property

@card.specialproperty

Use a special property definition to direct the harness to add a line to the JAD file that it creates for a test bundle containing this test. In effect, a special property definition is a way to pass a parameter to an application management system (AMS), sometimes called a Java application manager. The static parameter applies only to the test bundle associated with the JAD file.

Use the following syntax to specify a line to be added to the JAD file:

* @card.specialproperty <jad>.n=LineToAdd

n is a number that distinguishes multiple @card.specialproperty entries in the same source file. The harness interprets LineToAdd in this line as a name:value pair taking the first colon symbol as the separator between the name and the value. The harness copies the name:value pair to the JAD file. The test device AMS interprets the name:value pair when it downloads the JAD file. It is the test developer’s responsibility to ensure that name:value pair is meaningful to the AMS and follows the JAD file syntax. The JAD file is defined in the MIDP specification.

For example, to add a line to a JAD file associated with a test bundle containing MIDlet2:AlarmMidlet, include a line like this in the test class block comment:

* @card.specialproperty <jad>.1=MIDlet2:AlarmMidlet

Parameter expansion, described in Properties and Parameter Expansion, also applies to special property definitions. The samples in
devKitRoot/tests/runtime/src/client/com/sun/samples/network/client/push/ illustrate its use.

@card.requires

This tag specifies files that must be in the test bundle for one or more test cases in a class to execute. Examples of required files include media files to play on the test device and helper classes. CODE EXAMPLE 4-2 gives examples for a class and a case:

* @testclass
* ...
* @card.requires Pic1.jpg
* @card.requires com/some/apackage/img/Pic2.jpg
* @card.requires *
* @card.requires com/some/apackage/Aclass.class *
 
* @testcase
* ...
* @card.requires Pic3.jpg

Observe the following when writing path names for this tag:

• The path name separator must be a forward slash (/) character.
• If the path name contains a separator (for example, com/some/Foo.class or img/Pic.png), the path is relative to the directory specified in the test pack property TestClassesDir. For example, for runtime test packs, this directory is typically bin/client/verified (relative to the test pack root).
• If the path name does not contain a separator (as in the Pic1.jpg line in CODE EXAMPLE 4-2), the file is in the same directory as the class of the file containing the @card.requires tag.
• The special file name of * directs the build to automatically find all class files required by the class containing the card.requires comment.
• Following the name of a required class, the * character means “include this file and find the files required by the required file”.
• You can specify class name explicitly as when a class is loaded using Class#forName(String), for example.
• If a test class refers to a constant field defined in another class, the class that defines the constant may not be found by the card file generator. The reason is “inlining” constant values by javac compiler. The test case testCase1 in devKitHome/tests/runtime/src/com/sun/samples/Automated/SampleAutomatedTest.java demonstrates this effect. See the comments in the classes IInlinedConstants.java and IReferencedConstants.java for more detail.
• Automatic searching applies to Java programming language .class files only. Do not specify an asterisk (*) with non-class resources, such as images. For example, the following is an error:

* @card.requires Pic1.png * 

The path in an @card.requires entry can specify parameter expansion notationas described in Properties and Parameter Expansion. CODE EXAMPLE 4-3 shows an example.

* @card.property req=a
* ...
* @card.requires com/some/samples/automated/${req}.html

In this example, changing the value of req to b will include the file com/some/samples/automated/b.html in test bundles.

@card.attribute

Each test class and case can have a failure severity indicator, which is computed from two factors, functionality and impact. TABLE 4-3 shows the three functionality and impact codes and the failure severity values that the Java Device Test Suite computes from them. The Java Device Test Suite Tester’s Guide gives more information on failure severity.

Use the following guidelines to select a functionality value:

• 1 - Primary: The associated Test Objectives are a mandatory requirement of the implementation function tested by the test.
• 2 - Secondary: Indicates a recommended practice of the implementation function in accordance with Test Objectives. There may be valid reasons in particular circumstances to ignore this recommendation of the implementation function.
• 3 - Nonessential: Indicates that the Test Objectives are an optional requirement of the implementation function tested by the test.

Use the following guidelines to select an impact value:

• 1 - Critical: The tested device is effectively unusable as a result of the test failure. The test failure causes critical impact on the operation of the device/implementation. Critical impact should cover the case when implementation does not work. The test failure renders the implementation ineffective.
• 2 - Significant: Device failure causes significant impact. A test failure identifies a serious but predictable and manageable device failure.
• 3 - Limited: Device failure causes only limited or insignificant impact on the behavior and performance of the device/implementation.

You express functionality and impact in @card.attribute tags of the following form:

* @card.attribute functionality=value
* @card.attribute impact=value

In both cases, value must be 1, 2, or 3. CODE EXAMPLE 4-4 shows an example. TABLE 4-3 shows that this case’s computed failure severity is 4 - Very Low.

* @testcase
* ...
* @card.attribute functionality=2
* @card.attribute impact=3