Java Device Test Suite Tester’s Guide

The failure of some tests is more important than others. For example, a test that probes an obscure feature is less important than one that tests a feature used by almost every application. Beginning in Java Device Test Suite version 2.2, all tests written by Sun are tagged with a severity value ranging from 1 (very high severity) through 5 (very low severity). Non-Sun test developers can tag their tests in the same way, as described in the Java Device Test Suite Developer’s Guide.

You can select tests by severity (to save time by running the most important tests), you can view severity values in the harness, and you can organize reports by severity to focus on the most important failures. You can also override default severity values. This chapter describes test failure severity features in the following sections:

• Viewing Test Failure Severity
• Selecting Tests by Severity
• Organizing a Report by Severity
• How Severity is Calculated

---

Viewing Test Failure Severity

To see a test’s failure severity, select the test in the harness test tree and click the Test Severity tab. FIGURE 8-1 shows an example for a test that has not been run (current severity equals default severity). The harness computes severity from the values of functionality and impact. How Severity is Calculated describes the calculation.

FIGURE 8-1 Default Test Severity Tab

A test can have a different severity after it has been run, as described in How Severity is Calculated. For an illustrative Test Severity tab, refer to FIGURE 5-30.

---

Selecting Tests by Severity

In a configuration or template (if you are an administrator), you can select tests to run by their pre-run failure severity (see Pre-run Severity). You can, for example, minimize the number of tests you run by filtering out low-severity tests. In the Test Selection section of the interview, answer these questions as follows:

• Specify Test Severity: Yes
• Severity: Check the boxes representing the severity values you want to run, for example, 1, 2, and 3. By default, all severity values are selected.

Selecting Tests by Device Feature and Severity has an exercise for selecting tests by Severity.

Selection by severity is a filter, like selection by keyword. A test runs if it passes all filters. Therefore, you can use severity in combination with keywords, status, and so on. For example, you can select failed network tests that have a severity of 1 or 2.

---

Organizing a Report by Severity

When you select JDTS HTML Report in the Reports > Create Report dialog, you can select Consider Tests Severities. If you check (tick) this box, the report is organized by test severity, so you can easily focus on the failures that are most severe. FIGURE 5-35 shows a report with severity data. Creating Feature and Severity Reports has an exercise for creating a report with severity data.

XML reports automatically include severity data. The online help describes report generation options.

---

How Severity is Calculated

The harness calculates a tests’s severity from two factors, called functionality and impact. Each factor has a numeric value of 1-3, with 1 indicating the highest importance. TABLE 8-1 shows how the harness derives severity from each combination of functionality and impact.

To see the developer guidelines for assigning functionality and impact values, refer to the Java Device Test Suite Developer’s Guide.

There are multiple sources of these factors, a default and several optional overrides (see FIGURE 8-2). The harness displays the final value, current severity, and the contributing sources, in the Test Severity tab.

FIGURE 8-2 Sources of Severity Factors

Pre-run Severity

Each test has a default severity, calculated from functionality and impact values specified by the test developer. The default values can optionally be overridden by an entry in a severity override list. Pre-run impact and functionality values persist across test runs.

Default Severity

Test developers assign functionality and impact values to each of their tests. These values determine a test’s default severity.

Severity Override List

You can override a test’s default severity in an override list file and then specifying the file in a template (if you are an administrator) or a configuration. Specify the fully qualified name of the override list file in the Test Selection > Severity List interview question.

A severity override list can contain blank lines, comment lines (# in first position), and test lines. A test line has the following format:

testName Impact Functionality 

The line components are:

• testName is a test pack name plus optional package names, optional class name, and optional case name. To obtain fully qualified test case names, do one of the following:
• Select the test names in the graphical interface’s Passed, Failed, Error, or Not Run tabs, right-click, and choose Copy > Names as multiple lines. Then paste into a text editor.
• Alternatively, after running the tests, create a plain text report and copy the fully qualified names from it.

• Impact is the impact of a failure, represented as a value in the set [1,2,3]. TABLE 8-1 gives the meaning of these values.
• Functionality is the importance of the function tested, represented as a value in the the set [1,2,3]. TABLE 8-1 gives the meaning of these values.

Here are some examples with explanatory comment lines:

# Override one test case
Bluetooth_(JSR_82)/com/sun/jsr082/bluetooth/functional/push/service/UpdateTests#testServiceRegistrationException 1 2
 
#Override all cases in a class
MMAPI_and_ABB_(JSR_135)/com/sun/mmapi_10/functional/video/playback/thirdgp/ThirdGenPlayerHTTP 2 3
 
# Override all tests in a package
MMAPI_and_ABB_(JSR_135)/com/sun/mmapi_10/functional/video 3 1
 
# Override all tests in a test pack
OpenGL_ES_(JSR_239) 3 2

Post-run Severity

A test’s pre-run severity can be overridden during or after the test run in three ways.

• Interactive tests: Interactive test evaluation windows include drop-down controls that you can use to set a test’s impact and functionality.
• Virtual machine exit: A test that returns a status of Error - VM_Exit automatically receives a severity value of 1. VM_Exit means that the Pass/Fail status of the test is indeterminate, as might happen if the tester restarts the test device.
• Test Severity tab. After a test has run, you can set its impact and function in the Test Severity tab. The values you specify override all others.

Post-run changes do not persist across test runs. The next time the test runs, it begins with its pre-run severity. It ends with that severity unless the test VM_Exits or you manually override impact or functionality in the test evaluation window or the Test Severity tab.