test

SunVTS 3.0 User's Guide

Chapter 5 SunVTS Test Environment

This chapter describes the commonly used features of SunVTS in a typical testing environment:

System Mapping

System mapping provides a view of the system configuration, depending on your requirements. The System Map only displays the devices for which tests are present. Devices that are not testable, or for which there is currently no SunVTS test, are not displayed.

You can view your System Map in one of two ways:

Switch your mapping view by selecting Physical view or Logical view from the SunVTS user interface.

Note -

Physical mapping is available only on systems that have the configd program installed. For more information on package installation and administration, refer to the System Administration Guides and to Chapter 2, Installing and Removing SunVTS.

Test Modes

Two modes of testing are available: Connection Test mode and Functional Test mode. These modes differ in their assumptions about the state of the system you are testing and your objectives.

In addition, Functional mode runs in a restricted fashion if it is invoked through the Solstice SyMON application.

Only one test mode can be selected per test session.

Connection Test Mode

In Connection Test mode, the tests determine if the devices are connected to the system you are testing and verifies that they are accessible.

Device functionality is not verified, however, you can safely run this mode when the system is online.

The limited nature of the tests in this mode makes it possible to run periodic checks for configuration verification on the system.

Functional Test Mode

In Functional Test mode, the tests fully exercise all aspects of the selected devices and their associated device drivers.

In Functional Test mode you can run a single test to verify the functionality of a single device, or run multiple tests on all system devices for full system verification.

For thorough testing, the tests use a significant portion of the system resources. For this reason, do not run critical applications on the system or use the system for production purposes while testing in Functional Test mode.

In Functional Test mode, tests expect all system resources that are associated with the device to be available for testing. If the test cannot access a device, it registers a failure. The tests do not economize on runtime, but focus on achieving complete coverage and thoroughly exercising the device.

Note -

SunVTS does not verify that the system is in a safe offline state for Functional mode testing. You must be sure that your applications and the SunVTS diagnostic will not compete for system resources. For more information regarding runtime considerations refer to "SunVTS Runtime Considerations".

Functional Test Mode via Solstice SyMON

Solstice SyMON identifies a range of hardware and system status states quickly. For example, it can monitor a major condition such as a CPU failure or a minor condition such as low swap space. You can also monitor hardware performance to detect incipient hardware failures, such as soft read errors on a disk.

Solstice SyMON has an online diagnostic interface so that you can access SunVTS when running SyMON. In this case, Functional Test mode does operational testing to find and isolate faults, while minimizing the impact on other applications and users.

When Functional Test mode is accessed from SyMON, the system may be running critical production software. The tests are sensitive to this fact and usually try to achieve as much coverage as possible within the constraints imposed. In this mode, some of the test parameters, execution options, and some system level options are fixed (have preassigned values) that cannot be changed. This ensures that the system state is not violated by selecting an option or combination of options which could trigger unsafe actions.

Table 5-1 shows the default values of the test execution options in different modes.

Table 5-1 Test Execution Options

 

Option 

Connection Test Mode 

Functional Test Mode 

Functional Test Mode via SyMON 

Stress 

Disabled (fixed) 

Disabled 

Disabled (fixed) 

Verbose 

Disabled (fixed) 

Disabled 

Disabled (fixed) 

Core File 

Disabled (fixed) 

Disabled 

Disabled (fixed) 

Run On Error 

Disabled (fixed) 

Disabled 

Disabled (fixed) 

Max Passes 

1 (fixed) 

0 

1 (fixed) 

Max Time 

0 (fixed) 

0 

0 (fixed) 

Number of Instances 

1 (fixed) 

Dependent on the number of processors 

1 (fixed) 

Test Selection

There is a SunVTS test for most hardware devices on Sun systems.

When you start SunVTS, the SunVTS kernel probes the system devices. The results of this probe are displayed on the SunVTS interface where you are presented with list of devices to select for testing. Tests that are not pertinent to the configuration of your system are not listed.

Selecting Test Groups and Individual Tests

You can select entire test groups or individual tests. The test mode and tests that you choose to run depend on your desired test strategy.

For example, if you are using SunVTS to troubleshoot intermittent SCSI bus errors you might choose to test the entire SCSI group in Functional Test mode to thoroughly test the whole subsystem. Then you might select specific SCSI devices to see which device reproduces the error.

In another example, you might want to validate a newly installed system before you put it in production. In this case you would choose to test all the groups in Functional mode for a specified period of time.

If you want to verify the connectivity of a particular device such as an option card, peripheral, or communication device, you might choose the Connection Test mode for that device.

The method used to select a test or group of tests varies based on the SunVTS user interface that you are using. For more information refer to one of the following sections:

Intervention

Certain tests require that you intervene before such tests can be selected. These tests require you to load test media or install loopback connectors, and enable the intervention mode before the tests can be selected.

You must install media in these devices if you plan to test them in Functional mode:

Note -

Install such requirements before the SunVTS kernel probes for devices (when SunVTS is started, or the reprobe function is run) to allow the SunVTS kernel to properly identify each device.

Intervention mode serves as a reminder that you may need to intervene before running certain tests; it does not verify that the appropriate test media or loopback connectors have been installed.

The location of the Intervention mode button varies based on the SunVTS interface that you are using. For specific details refer to:

Test Execution Options

You can customize your SunVTS environment by setting Test Execution options (refer to the table below).

Table 5-2 Test Execution Options
 Option Description

Stress 

Performs stress testing 

Core File

Creates a core file. If the <SunVTS bin> directory is writable then core.<testname>.xxxxxx is the core file name, where <testname> is the test that dumped core, and where xxxxxx is a character string generated by the system to make the file name unique. When core_file is disabled, a message indicating the signal that caused the failure is displayed and logged. See "The Log Files Button" in Chapter 4, SunVTS User Interfaces.

Max Errors

States the maximum allowable number of errors before stopping (0 makes the SunVTS kernel continue testing regardless of errors) 

Max Passes

Specifies the maximum number of passes a test can run 

Max Time

States, in minutes, the time limit a test can run (0 = no limit) 

Number of Instances 

Indicates the number of instances (processes) of scalable tests that can run on the same processor 

Processor Affinity 

Tests can be bound to a specific processor with this option 

Run On Error

Continues testing until the max_errs number is reached

Verbose 

Displays verbose messages in the SunVTS console window 

Note -

Some options have preassigned values when SunVTS is invoked through the Solstice SyMON diagnostic. These values that cannot be changed. This prevents changing or setting values for these options that may not be safe when the system is online.

The Test Execution Options can be applied at the following three levels:

When you assign a value to an option at the top level, that value applies to all the groups and tests below it. Similarly, setting an option at the group level causes the value to trickle down to all the sub-groups and tests below it. This is a powerful way to customize the SunVTS testing environment. For example, you can disable the Verbose option at the system level and then enable it only for a particular group (or only a particular test) so that just the verbose messages from that particular group (or test) are displayed.

Test Execution Locks and Overrides

You can control the effects of the Test Execution options by using locks and overrides. These features are accessible through the Advanced menu in the user interface.

Locks

Enabling the group lock prevents an option set at a higher level from affecting a specific group or test and the sub-groups and tests below it. Similarly, you can set the test lock to protect the option setting of a test.

Overrides

You can use the overrides to void the lock protection. For example, setting system override nullifies all the locks, and setting group override for a particular group nullifies all the locks below that group.

Setting Test Execution Options, Locks, and Overrides

The steps you perform to set options differs based on the SunVTS user interface that you are using. For a description of these steps refer to the following:

Test-Specific Options

Test-specific options are the set of options that accompany each test and pertain to that test only. For example, the amount of media to test in a disk test, the amount of memory to test in memory test, the remote host address in a net test, and so on are test-specific options. These options differ from one device to another. These option settings are covered in detail in the SunVTS Test Reference Manual for each test.

Scalability of Tests

Use the scalability feature to scale the number of test instances or processes to stress single or multiprocessor systems. You can modify the scalable test options so that each test instance runs simultaneously with a different option. Combined with the processor affinity mask, SunVTS provides a flexible testing environment. You can determine the amount of stress by the number of test instances you select using the options described below.

Saving a Test Configuration

Use the Option Files menu to save the current set of selected tests and SunVTS global, group and test-specific options to a file for later use. This is a convenient feature when you plan to use the same test parameters over and over again.

To Create an Option File

  1. Configure the SunVTS main display with your desired tests and options.

  2. Bring up the Option File menu based on the user interface that you are using:

    • CDE interface--click Options in the Menu bar and select Option Files (refer to the figure below)

      Figure 5-1 CDE Options Menu

      Graphic

    • OPEN LOOK interface--Click on the Option Files button.

    • TTY interface--Highlight Option_files from in the Control panel and press Return.

  3. Type a unique name in the Option file field as shown in the following figure:

    Figure 5-2 Option File Menu

    Graphic

  4. Click on the Store button to save the current configuration to the option file.

    Storing an option file saves the current settings from the global test options and specific test settings to a file in the /var/opt/SUNWvts/options directory.

    Note -

    Although you can manually edit the option file, you must adhere to strict format requirements. Unnecessary or spurious characters in the option files may cause unexpected actions with the SunVTS kernel or user interface.

To Load an Option File

  1. Bring up the Option File menu (as described in Step 2 in the previous procedure)

  2. Click on the name of the option file from the option file list.

  3. Click on Load.

  4. Click on the Close button.

All the tests and test options are set up for use by the SunVTS kernel, no time is spent on test parameter configuration.

To Load an Option File From the Command Line

(The following example assumes that an option file called Test.set2 was created as described "To Create an Option File")

  1. Use the -o option with the sunvts command (refer to the following example):

# ./sunvts -o Test.set2

All the tests and test options are set up for use by the SunVTS kernel, no time has to be spent on test parameter configuration.

To Remove an Option File

  1. Bring up the Option File menu (as described in Step 2 Step 2).

  2. Select the Option file from the option file list that you want to delete.

  3. Click the Remove button.

  4. Click on the Close button.

Testing Frame Buffers

If you are running SunVTS on a frame buffer that is running a graphics test, a false error may occur. This is not a problem when you are running windows (OPEN LOOK or CDE).

No other graphic application can be run in the same window once you start the graphic test.

To Activate the Frame Buffer Test

(See the test command line arguments in the SunVTS 3.0 Test Reference Manual.) 

# ./fbtest -o dev=cgthree0,lock=Enable

Caution - Caution -

Do not run TTY mode and frame buffer tests concurrently on the console monitor; the frame buffer tests may fail.

Note -

If window locking is disabled (unlocked) on frame buffers that are running vtsui, or you move the mouse, you will receive spurious error messages. Even slight mouse movements can cause a test to fail.

Testing Multiple Frame Buffers

These rules apply when testing multiple frame buffers (displays) simultaneously:

If you start SunVTS from a screen other than the console monitor, Frame buffer locking is not available. You must enable the frame buffer lock for the console monitor, as shown in the example above. The SunVTS user interface cannot display on a monitor if frame buffer locking is disabled.

Monitoring Test Status

While SunVTS is testing the system, you can view the progress of the test by monitoring the main SunVTS display and through log files.

Testing Progress in the Panels

There are four main panels to view while monitoring the testing progress; the System Status panel, System Map, Performance Meter, and the Console window.

System Status Panel

The System Status panel displays the current activity of the SunVTS kernel. For example, the status can be idle (when not testing), testing, or stopped (when the maximum number of errors is reached, or when you stop the testing).

This panel also displays the total number of successful system passes (a system pass is when all tests have been run once), and the total number of errors from all tests. The total elapsed time is also displayed on this panel.

System Map

This panel displays the status of all selected tests. The status of each selected test is listed as follows:

Console Panel

The Console panel displays the messages sent from the tests and SunVTS kernel. If you enabled verbose messages or system call trace messages on one or more tests, the messages appear in this window. Use the scrollbar to view these messages.

Performance Meter

This panel displays the performance statistics of the system you are testing. See "Meter Button" and the"OPEN LOOK Performance Meter" in for details.

Suspending and Resuming the Test Session

During the testing session, you may need to suspend or pause all tests. For example, you may want to look at messages on the Console panel that have scrolled out of view, or you may want to view and print a log file.

Select Suspend from the Commands in the menu bar to pause all of the SunVTS tests. Select Resume from the Commands in the menu bar to resume the test session.

Stopping the Test Session

Once you have adequately tested your system, you can stop the testing session and review the results by selecting the Stop button.

SunVTS ends the test execution automatically for a number of reasons. For example, if you have enabled the Single Pass option on the Schedule Option menu, all tests run once and stop, but the elapsed time continues to increment.

Testing stops if an error occurs and the run at error was not set or, if run at error was set but the error count reached the maximum number of errors.

Stop Button

Click the Stop button to stop the testing session. The System Status panel temporarily displays "stopping" while the tests wind down. The amount of time it takes to stop depends on which tests you are running. For example, the tapetest requires time to rewind the tape or remove the mounted partitions and temporary files. A pop-up menu displays the message, "testing completed: x pass(es), x error(s)" when testing is complete.

After you stop the testing session, you can view the System Map to see how many passes and errors occurred per test, and you can display or print the log files.

If a test hangs, SunVTS remains in Stop mode, and waits for the test to terminate. (This is probably due to a hardware problem.) Deselect the test, which signals SunVTS to ignore the stopped test and return to Idle mode.

Resetting the Test Environment

If you need to start the testing session over again, without quitting SunVTS, click the Reset button.

Reviewing Log Files

This section provides information about displaying, printing, and deleting log files. The Log file error message formats are also discussed in this section.

The status of testing is stored in three log files:

The sunvts.err file contains SunVTS test error messages and start and stop times. The information log file, sunvts.info, contains informative messages that are generated when you start and stop SunVTS. The messages file is a log of all the general UNIX messages.

Log Files Menu

You can access these files from the Log Files menu, which is displayed when you click the Log Files button (see Figure 5-3 and Figure 5-4).

Figure 5-3 CDE Log Files Menu

Graphic

Figure 5-4 OPEN LOOK Log Files Menu

Graphic

Displaying Log Files

You can display any of the three log files by selecting the name of the log file. You can print or remove them by selecting the print or remove button on the menu.

Note -

These log files can be very long. Make sure you want the entire file before you print it.

SunVTS Test Message Syntax

All SunVTS test messages follow this format: 

SUNWvts.testname[.subtest_name].message_number date time testname device_name [FRU_path]ERROR|FATAL|INFO|WARNING|VERBOSE message

Table 5-3 lists the SunVTS test message arguments and gives a brief description.

Table 5-3 SunVTS Test Message Syntax

Argument 

Description 

SUNWvts

SunVTS package name 

testname

SunVTS test name 

subtest_name

The subtest module name (optional) 

message_number

The message identifier, which is a unique number for the test. The number is usually within the following ranges: VERBOSE: 1 - 1999 INFO: 2000 - 3999 WARNING: 4000 - 5999 ERROR/FATAL: 6000 - 7999 FATAL: 8000 - 9998 (The number 9999 is reserved for any possible old message types in previous SunVTS releases for compatibility reasons.) 

date time

Tells when the error occurred 

testname

The name of the test reporting the error  

device_name

The device being tested when the error occurred 

FRU_path

A full Solaris device path of the failed FRU; this argument varies, depending on the type of test running when the error occurred; see "Interpreting Failed FRU Information" for details

message

Contains test messages, in addition to probable cause and recommended action 

Interpreting Failed FRU Information

When a test fails, SunVTS incorporates information about the failed FRU into the error message. The error message indicates the location of the failed FRU on the system. The location is indicated in the form of a path that starts with the board number to which the FRU is connected.

For example, the FRU path of cpu-unit0 on board number 0 is: 

cpu-unit0(board0)

The FRU path of a disk, c0t0d0, controlled by esp0 on board 0 is: 

c0t0d0(board0/sbi0/esp0)

The FRU path of a disk, c1t0d0, connected to a soc on board 1 is: 

c1t0d0(board1/sbi1/soc1/pln1)

Note -

The FRU path feature is available only on the server systems: SS1000, SS2000, Ultra Enterprise 6000, Ultra Enterprise 4000, and Ultra Enterprise 3000.

Debug Features

Use the following options to debug your system:

For a description on how to set these options refer to:

Testing With Record and Replay Options

Use the Start with Record and Replay option to record a sequence of significant events during a testing session, and to replay or view this information at a later time. Once a testing session is recorded, you can use this information to drive the SunVTS kernel so it reproduces the recorded sequence of events. These record and replay features are an effective way to reproduce testing conditions, a helpful debugging tool.

The Record with Replay option can closely reproduce the ordering of the testing events, but it cannot reproduce the time periods of these events because the execution times may vary from one run to another. When scheduling a replay using this option, give consideration only to the ordering and relative scheduling time differences between events during the original run.

These options are limited in the way that they replay a test session because you can only control the Replay option when a command starts, not after the command completes.

The Record with Replay option records the following test events:

The events are recorded in a file called vts_replay_file, which is saved in a log directory that you specify. If no log directory is specified, the /var/opt/SUNWvts/logs directory is used by default.

The first part of the vts_replay_file consists of the SunVTS kernel and test options settings you used when you started recording. This information is used during the replay to restore the original option settings.

The events are recorded after the option settings are saved. Information is recorded in the following format for each event: 

event_type rel_time target_object [instance_num] [event_specific]

The following table describes each event string.

Table 5-4 The vts_replay_file Event String Descriptions

Event String 

Description 

event_type

The type of event can be one of the following: START, STOP, SELECT, DESELECT, or INTERVENTION. 

rel_time

The time (in seconds) since the preceding event. 

target_object

The object involved in the event, for example, fpu (fputest). This field is ignored if the value is a dash (-) as in the case of an INTERVENTION event.

instance_num

The instance number of the test. A valid number is required for the START and STOP events. This field has a value of -1 for INTERVENTION, SELECT, and DESELECT events.

event_specific

Any additional event-specific information required to replay the event; this field contains the test command line arguments for the START event. For the INTERVENTION event, the value describes the state--either enabled or disabled. This field is blank in the case of STOP, SELECT, and DESELECT. 

Trace a Test

Use the Trace test option to create a log of every system call made when a test is running. This feature logs the system calls using the standard UNIX command truss. The trace messages logged by this feature give you a powerful debugging tool when isolating the specific cause of an error.

From a TTY command line, you can use the UNIX truss(1) command on any test while running SunVTS. (See the truss(1) man page for more information.)

To Enable Tracing

  1. Click Commands on the Menu bar and pull down to Trace test.

    The Trace Test window is displayed as shown in the figure below.

    Figure 5-5 Trace Test Window (CDE Interface)

    Graphic

  2. Choose one or more tests to be traced from the scrollable list of test names and then select Apply, or double-click a test name to enable tracing and close the window.

    When you select a test, system call tracing is enabled immediately. If the test is already running when you select it, tracing begins immediately and the trace test messages appear in the SunVTS console window.

  3. Select File and click Apply to write the trace test messages to a file.

To Disable Tracing

  1. Bring up the Trace Test window again, select the highlighted test name, and click Apply.

    This action removes the highlighting and disables tracing.

Security

The SunVTS user interface running on one host can control the SunVTS kernel running on a system under test, often referred to as SUT. The user interface has to connect to the SunVTS kernel before it can control the SunVTS kernel (see "Host (Connect to) Button" for more information).

The SunVTS kernel authenticates connect requests from the SunVTS interfaces based on one of the following attributes:

The list of hosts, groups, and users are specified in the security file, .sunvts_sec, which is installed in the <SunVTS3.0 install directory> /bin. A plus (+) entry in one of these lists means all hosts, groups, or users, respectively. A template of the security file (.sunvts_sec) is located in the <SunVTS3.0 install directory> /bin.

Note -

To enable security checking, remove the plus (+) sign in the HOST section of .sunvts_sec, the default.

The following table shows a security file template.

Security File Template

#This file should be <SunVTS3.0
install directory>/bin/.sunvts_sec
#
#Any line beginning with a # is a comment line
#
# Trusted Hosts entry
# One hostname per line.
# A "+" entry on a line indicates that ALL hosts are Trusted
Hosts.
# No password authentication is done.
# The line with the label HOSTS: is required to have the
list of hosts
#
#HOSTS:
+
#host1
#host2
#
# Trusted Groups entry
# One groupname per line.
# A "+" entry on a line indicates that ALL groups are Trusted Groups.
# User password authentication is done.
# The line with the label GROUPS: is required to have the
list of groups
#
#GROUPS:
#group1
#
# Trusted Users entry
# One username per line.
# A "+" entry on a line indicates that ALL users are Trusted
Users.
# User password authentication is done.
# The line with the label USERS: is required to have the
list of users.
#USERS:
#user1
#user2

The SunVTS kernel authenticates the requests based on the entries in the security file. All access except root is denied on the local machine if the security file entry is invalid or if there is no entry in the file. You can correct an entry in this file even when the SunVTS kernel is running. When you specify the -e option with the SunVTS kernel, then the SunVTS kernel accepts Connect to requests from any host, regardless of the user identification of the SunVTS User Interface process that is initiating it.

Note -

The user password needed for authentication purposes by vtsk is the same password used to log in to the system you are testing.

Installing a Custom Test

Developers can add their own custom tests to the SunVTS environment. This book does not describe test development, but does list the tasks necessary to integrate a custom test into the SunVTS environment.

To Prepare the SunVTS Environment for the Custom Test

  1. To install a custom test, modify the text file called .customtest.

    Modify the 32-bit or 64-bit file according to the Solaris environment that you are using:

    • 32-bit: /opt/SUNWvts/bin/.custometest

    • 64-bit: /opt/SUNWvts/bin/sparcv9/.custometest

    The options you set in the .customtest file become the default options for each test. You can change these options using the pop-up option menus on the SunVTS window interface. However, the Reset button returns the options to the default settings that you specify in the .customtest file.

  2. To display a custom test in the SunVTS interface, enter the test label, test name, and options specifications in the .customtest file. Refer to the file format below.

When invoked, SunVTS displays the custom test.

To Copy the Test Binary to the sunvts Installation Directory

  1. Edit the .customtest file according to the format given below.

  2. Restart sunvts or reprobe the system configuration.

    Note -

    If .customtest is renamed as .customtest-<group>, all its user tests will +appear under the specified <group>.

The .customtest File Format

The.customtest file is located in in two places. Modify the 32-bit or 64-bit .custometest file according to the Solaris environment that you are using:

Each line in this file is made up of two or more fields that are separated by a semicolon where:

A user test definition requires a minimum of two fields, separated by a semicolon, as shown in the following .customtest file format examples: 

% your_label_name;your_test_name

  1. To add the scalability option, append the keyword SCA. 

    % your_label_name;your_test_name;SCA

  1. To custom build an option menu, add an option specification: 

    % Option_Name<Option_Type|Value|Default_Value|Command_Line_Option>

  1. To specify more than one option, separate each option by a comma: 

    % label_name;test_name;Numeric<NUMERIC|0,100|50|numeric>, Exc_Choice<EXC_CHOICE|Top,Middle,Bottom|Middle|exc_choice>, Inc_Choice<INC_CHOICE|Left,Center,Right|Left+Center+Right|inc_choice>,Toggle<TOGGLE|This,That|This|toggle>, Text <TEXT|20|Type_Here|text>, Slidebar<SLIDEBAR|0,10|5|slidebar>, Errors<CYCLE|Yes,No|No|errors>, Cycle<CYCLE|First,Second,Third|First|cycle>;SCA

SunVTS invokes the above test as follows: 

% ./test_name -s[vq..] [-i n] -o dev=user[0,1..],Command_Line_Option=Value...

You cannot use the customtest facility with a test probe attached. You must ensure that the binaries are compatible with the version of the Solaris kernel on which SunVTS is currently running.