| Java Device Test Suite Administration Guide |    
|
| C H A P T E R 1 |
|
Overview |
This chapter describes the main Java Device Test Suite concepts and components from an administrator’s point of view. Additional overview topics can be found in the Java Device Test Suite Tester’s Guide and the Java Device Test Suite Developer’s Guide.
The chapter covers the following subjects:
The Java Device Test Suite helps Java technology wireless device manufacturers and network operators maximize product quality and minimize time to market. The Java Device Test Suite includes an extensible set of test packs, a shared management facility, and a distributed test execution harness that can be used to assess the quality of any device that implements a compatible combination of the following Java Platform, Micro Edition (Java ME platform) technologies:
For a detailed list of supported specifications and versions, see Appendix D.
The Java Device Test Suite also includes tests that do not relate to a particular specification:
The product’s total of about 11,000 tests can be extended with new tests written by Sun or by others, including Java Device Test Suite users.
When considering a device that implements Java technology, it is useful to distinguish between compatibility testing and quality testing. Both are important. Compatibility testing exercises individual application programming interfaces (APIs) defined by a specification, such as the MIDP 2.0 specification. If, when passed valid arguments, a device method call returns a valid result, the invoking test reports that the method implementation conforms to the specification. Compatibility testing is an important first step in testing a product. Technology Compatibility Kits (TCKs) perform compatibility testing, and, ideally, should be used in conjunction with the Java Device Test Suite.
A quality product requires additional tests before it can be confidently released to run real applications in real-world conditions. The Java Device Test Suite complements TCKs by performing quality testing, which can be divided into three main areas:
When you install the Java Device Test Suite software, you create the components shown in FIGURE 1-1, except for the tester harnesses, which test runners (testers) create.
FIGURE 1-1 Administrator’s View of Java Device Test Suite Architecture
A test pack is a collection of tests that are functionally related and have common setup requirements. “Functionally related” means that the tests exercise a cluster of test device functions, for example, the MIDP 2.0 runtime APIs, or assess a device attribute such as performance. “Common setup requirements” means that all the tests in the pack either require the same setup (such as starting a partner device that cooperates with the test device) or the same user interaction (such as inspecting test device behavior), or both. Developers create test packs as described in the Java Device Test Suite Developer’s Guide.
A test pack has the hierarchical structure shown in FIGURE 1-2.
FIGURE 1-2 Test Pack Hierarchy
A test pack contains at least one test package. A test package can contain a nested test package or a test class. A test class contains at least one test case.
FIGURE 1-3 shows how the harness displays an example test pack in which the test cases in one test class are expanded (exposed to view). The harness does not visually distinguish packs, packages, classes, and cases, except by their position in the hierarchy. Test packs have no ancestors. Test cases have no descendents. The parents of test cases are test classes. Everything else is a test package.
FIGURE 1-3 Sample Test Pack Hierarchy
Test packages subdivide a test pack’s name space. Test packages can be nested, and they often are. For example, there six packages are visibly nested in the com.sun.satsa package in FIGURE 1-3.
A test class is the unit of code that test developers create. A test class contains one or more test cases.
A test case, informally called a test, is the code in a test class that exercises one or more test device functions and passes, fails, or returns benchmark metrics. The Java Device Test Suite Developer’s Guide describes the internals of test classes and test cases in detail.
FIGURE 1-4 shows some of the same SATSA tests shown in FIGURE 1-3. Instead of a test pack hierarchy, the tests are arranged in a device feature hierarchy. This hierarchy emphasizes capabilities that might be supported by a given test device. For some users, particularly those who work for mobile network operators, this hierarchy is easier to use because it maps better to the description of a device provided by a manufacturer. Java Device Test Suite testers and administrators can use either hierarchy to select tests to run and to view results in a report. You specify test selection by package or feature hierarchy in a template or configuration’s (Configurations and Templates) How to Specify Tests question.
FIGURE 1-4 Sample Feature Hierarchy
The main harness test tree (shown on the left in FIGURE 1-6) always displays the package hierarchy. In either view, the test cases that run are the same. The two hierarchies organize the tests differently.
In addition to optionally selecting tests by device feature, after a test run, a tester can create a report that is organized by device feature. FIGURE 1-5 is an example. The online help describes how to create a feature-based report.
FIGURE 1-5 Feature Report Example
A configuration is a file that controls the execution of tests, usually for a particular device or variant. Two devices that have functionally identical software but differ in one or more physical details, such as screen size or memory capacity, are variants. The entries in a configuration specify the tests to run, how tests are to be transferred to the device, and many other general and test-specific properties.
A template is a pattern for one or more configurations. Users create configurations from templates. Administrators create templates. You can simplify the work of testers, and make testing more consistent, by specifying configuration information in templates. A tester only has to supply information that differs from a configuration’s parent template. If you update a template, the changes you make automatically flow to configurations that are based on the template.
The tester and administrator harnesses have extensive online help. The Central Installation holds the following additional documentation:
The Central Installation holds the Java Device Test Suite executables, which the administrator and tester launch scripts share. The Relay executable is normally installed on a host located in the demilitarized zone between the organization’s inner and outer firewalls.
The developer’s kit consists of sample test packs and test API documentation. The Java Device Test Suite Developer’s Guide describes it.
The administrator harness (see FIGURE 1-6) is a superset of the tester harness. If you are both tester and administrator, you can perform all functions with the administrator harness.
FIGURE 1-6 Administrator Harness
In addition to tester harness functions, the administrator harness has commands for performing the following functions:
You use the harness’s template editor (see FIGURE 1-7) to create and edit templates. You can delete a template with an operating system command. A template you create begins as a copy of a built-in template supplied by Sun. The built-in templates specify different combinations of test packs, for example, all test packs versus only those related to the MIDP 1.0 specification.
The template editor presents a template as a set of questions that is sometimes called an interview. The left pane lists the questions. Selecting a question in the left pane shows the question and, for many questions, default and possible answers in the center pane. The right pane gives guidance for choosing an answer. You can close the right pane to save display space.
Template updates propagate to configurations that are based on the template. The propagation occurs when a tester next uses such a configuration. The tester harness displays a notification that shows which template items have been updated. For more information on the template editor, see the administrator harness’s online help.
Judicious use of templates can improve the performance of tester harnesses. Including only the subset of tests that a tester needs in a template minimizes network accesses when the tester harness launches.
The MMAPI (JSR 135) test pack has tests for many media types, such as PNG and MPEG-1. But the specifications define an open-ended set of media types for MMAPI and AMMS (JSR 234). With the harness, you can create customized MMAPI and AMMS tests for audio, video, MIDI, and image media types. You can also create customized sensor tests for the MSAPI (Mobile Sensor API, JSR 256) test pack. The administrator harness online help describes the commands.
A benchmark test passes or fails based on a comparison of the test device’s performance to that of a reference device running the same test. The reference device is one you have judged to have acceptable performance. Its performance is represented in a threshold file. A tester creates the raw data for a threshold file by running the test several times on the reference device and selecting the run whose performance is minimally acceptable. You create the threshold file from the data with the administrator harness’s Configure > Thresholds command. You can use the same command to raise or lower threshold file values if necessary.
Test developers can use the developer’s kit to create new test packs. From time to time, Sun distributes updated test packs. You install both with the administrator harness’s Configure > Test Packs command. The administrator online help describes how to use this command.
Keystores hold digital certificates and private keys, which some security tests require and which can make runtime and benchmark tests more convenient to run. You can create a keystore and add certificates to it with the Java keytool utility or other software. You can import a copy of a keystore to the Central Installation with the administrator harness’s Configure > Keystores command. Importing a keystore makes its certificates and private keys accessible to templates and configurations. That accessibility enables you or a tester to specify, for example, a certificate for secure socket layer (SSL) tests or a private key to sign test MIDlets.