Subjectivity and Quality Testing

A P P E N D I X B

Subjectivity is unavoidable in quality testing. Unlike compatibility testing, there is usually no formal quality specification against which test device quality can be measured. Each testing organization needs to establish its own quality guidelines to minimize subjectivity, bearing in mind customer satisfaction, testing and support costs, competitive devices, and other factors. Although the development of such guidelines is beyond the scope of this guide, usability testing is one tool that deserves serious consideration.

The remainder of this appendix gives examples of subjectivity in different kinds of tests:

Interactive Tests

Benchmark Tests

Robustness Tests

Interactive Tests

Some interactive test results require subjective judgment to designate them as passed or failed. Two major categories are differences in device display and differences in device responses to anomalous conditions.

Display Differences

Devices with different display capabilities can produce different results for the same test. For example, TABLE B-1 shows the reference image for com.sun.m3g.functional.image2d.Conversions.alphaAndRGBa and the displays of five devices that are identical except for the number of colors or grays they display. Notice the use of the word “probably” in the assessments of device performance on this test.

TABLE B-1 Interpreting One Test on Different Devices
Test Device Display	Description
	Reference image. “Running Runtime Tests” in the online help describes test evaluation windows and reference images
	Test device has 4096 colors. This display is essentially identical to the reference image. It is obvious that this device passes the test.
	Test device has 4096 grays. This test could be considered to have passed result. Users would probably understand an image in an application that uses the tested feature.
	Test device has 16 grays. This test could be considered to have passed result. Users would probably understand an image in an application that uses the tested feature.
	Test device has eight grays. This test could be considered to have a failed result because it differs markedly from the reference image.
	Test device has two grays. This test could be considered to have a failed result. Users would most likely find a similar image displayed by an application as incomprehensible.

Interpreting One Test on Different Devices

Anomalous Conditions

Test evaluation windows cannot describe how test devices behave in anomalous conditions because there are no specifications for such behavior. Nor can all tests indicate that they have started and finished. Displaying such information might interfere with the test. Therefore, when an interactive test does not behave as described in the test description window, you must use your knowledge of the device and your role as a potential device user to decide if the test result is a pass or fail.

For example, a test device might decide that a test is not authorized to use a protected API. Accordingly, it could refuse to launch the test or test MIDlet. Alternatively, it could launch the test or MIDlet but terminate it when it tries to use the API. It could inform you of the problem in several ways or it could not inform you at all.

Benchmark Tests

Subjectivity is an element in other tests as well. Benchmark tests automatically have pass or fail results, but they do so based on a comparison to a reference device’s performance. That reference device is probably chosen subjectively. The only criterion for choosing it, as far as the Java Device Test Suite is concerned, is that the reference device consistently exhibits “acceptable” performance.

Robustness Tests

Robustness tests are another example of inherent quality test subjectivity. A test developer might decide that a robust implementation should perform 100 repetitions of a code sequence without error. The developer’s decision is an informed one, but it is also ultimately a subjective evaluation.