C H A P T E R  8

Diagnostics and Troubleshooting

This chapter describes diagnostic tests and troubleshooting for the Sun Crypto Accelerator 4000 software. This chapter includes the following sections:


Diagnostic Software

The Sun Crypto Accelerator 4000 software provides three interactive utilities for running diagnostics on the board. The first of these utilities, SunVTS, focuses on the system level network and cryptographic functionality of the Sun Crypto Accelerator 4000 subsystem (driver, firmware, and hardware). The other two utilities, vcaadm and vcadiag, perform low level diagnostics on individual hardware components of the Sun Crypto Accelerator 4000 board.

Performing SunVTS Diagnostics

SunVTS is Sun's Validation Test Suite software. The core SunVTS wrapper provides test control and a user interface to a suite of system level tests. These tests are delivered with packages SUNWvts and SUNWvtsts to make up a bundle that is contained on the Solaris 10 Software DVDs, and also available for download at http://www.sun.com/oem/vts.

The Sun Crypto Accelerator 4000 board can be tested by three SunVTS tests that are bundled with the core SunVTS software beginning with SunVTS 6.0 Patch Set 1 (PS1) released with Solaris 10. The first two SunVTS tests, nettest and netlbtest, operate on the Ethernet circuitry of the board. The third SunVTS test, cryptotest, provides diagnostics of the cryptographic circuitry of the board.

For the nettest and netlbtest, refer to the SunVTS 6.0 Test Reference Manual, User's Guide, and Quick Reference Card for instructions on how to perform and monitor these diagnostics tests. These documents are available on the Solaris 10 Documentation DVD and in the Solaris on Sun Hardware Documentation Set at http://docs.sun.com. For the cryptotest, also refer to the SunVTS 6.0 Patch Set 1 Documentation Supplement available at: http://www.sun.com/products-n-solutions/hardware/docs/Software/Diagnostics/index.html

Performing vcaadm Diagnostics

The vcaadm utility is used by a security officer to test an initialized card and is the recommended interactive diagnostic application. Both vcaadm and vcadiag invoke the same diagnostics routines on the card, but the vcaadm utility provides more information regarding any failures encountered. Details on how to run the vcaadm utility are provided in Chapter 4 of this document, and an example of how to run diagnostics using vcaadm is provided in Using the vcaadm diagnostics Command.

Performing vcadiag Diagnostics

The vcadiag interface allows the security administrator to perform diagnostics on both an initialized and uninitialized board. The vcadiag interface provides less information regarding diagnostic failures then the vcaadm interface and is primarily intended to provide a general pass/fail status to someone other than a board security officer. To run vcadiag diagnostics, the user invokes the vcadiag command with the -D parameter. Details on how to run the vcadiag utility are provided in Chapter 4, and an example of how to run diagnostics using vcadiag is provided in Using the vcadiag Utility.


Using kstat to Determine Cryptographic Activity

The Sun Crypto Accelerator 4000 board does not contain lights or other indicators to reflect cryptographic activity on the board. To determine whether cryptographic work requests are being performed on the board, use the kstat(1M) command to display the device usage. The following excerpt shows the various kstats that can be used to determine cryptographic activity.


# kstat vca:0
--------------
# kstat vca:0
module: vca                             instance: 0
name:   vca0                            class:    net
        3desbytes                       0
        3desjobs                        0
        aesbytes                        0
        aesjobs                         0
        dsasign                         0
        dsaverify                       0
        keygenbytes                     0
        keygenjobs                      0
        md5bytes                        0
        md5jobs                         0
        rngbytes                        280
        rngjobs                         14
        rsaprivate                      0
        rsapublic                       0
        sha1bytes                       0
        sha1jobs                        0
---------------



Note - In the previous example, 0 is the instance number of the vca device. This number should reflect the instance number of the board for which you are performing the kstat command.



Displaying the kstat information indicates whether cryptographic requests or "jobs" are being sent to the Sun Crypto Accelerator 4000 board. A change in the jobs values over time indicates that the board is accelerating cryptographic work requests sent to the Sun Crypto Accelerator 4000 board. If cryptographic work requests are not being sent to the board, verify your web server configuration per the web server specific configuration.


Using the OpenBoot PROM FCode Self-Test

The following tests are available to help identify problems with the adapter if the system does not boot.

You can invoke the FCode self-test diagnostics by using the test or test-all commands from the OpenBoot PROM ok prompt. If you encounter an error while performing diagnostics, appropriate messages will be displayed. Refer to the OpenBoot Command Reference Manual for more information on the test and test-all commands.

The FCode self-test exercises most functionality subsection by subsection and ensures the following:


procedure icon  Performing the Ethernet FCode Self-Test Diagnostic

To perform the Ethernet diagnostics, you must first bring the system to a stop at the OpenBoot PROM ok prompt after issuing a reset. If you do not reset the system, the diagnostic tests might cause the system to hang.

For more information about the OpenBoot commands in this section, refer to the OpenBoot Command Reference Manual.

1. Shut down the system.

Use the standard shutdown procedures described in the Solaris Handbook for Sun Peripherals.

2. At the OpenBoot PROM ok prompt, set the auto-boot? configuration variable to false.


ok setenv auto-boot? false

3. Reset the system.


ok reset-all

4. Type show-nets to display the list of devices and enter a selection:

You see a list of devices, similar to the example below, specific to the adapter:


ok show-netsa) /pci@8,600000/network@1
b) /pci@8,700000/network@5,1
q) NO SELECTION
Enter Selection, q to quit: a
/pci@8,600000/network@1 has been selected.
Type ^Y ( Control-Y ) to insert it in the command line.
e.g. ok nvalias mydev ^Y for creating devalias mydev for
/pci@8,600000/network@1



Note - To perform the following self-test with the test command, the Ethernet port must be connected to a network.



5. Perform the self-test using the test command:

The following tests are performed when the test command is executed:



Note - The Sun Crypto Accelerator 4000 UTP adapter self-test for a 1000 Mbps connection is not supported for use with an external loopback cable because the link-clock cannot be reconciled. For this test, the local and remote ports must reconcile as clock master and clock slave. If an external loopback cable is used, both the local and remote ports are identical. So, the single port cannot be both a clock master and a clock slave, because this causes the PHY link-up to fail. For a Sun Crypto Accelerator 4000 UTP adapter self-test for a 1000 Mbps connection to work, a remote 1000BASE-T port must be connected.



Type the following:


ok test device-path

If the test passes, you see the following messages:


ok test /pci@8,600000/network@1
Testing /pci@8,600000/network@1
Register tests: passed
Internal loopback test: passed
/pci@8,600000/network@1: 100 Mbps half duplex link up

If the board is not connected to a network, you see the following messages:


ok test /pci@8,600000/network@1
Testing /pci@8,600000/network@1
Register tests: passed
Internal loopback test: passed
/pci@8,600000/network@1: link down

6. After testing the adapter, type the following to return the OpenBoot PROM ok prompt interface to standard operating mode:


ok setenv diag-switch? false

7. Set the auto-boot? configuration parameter to true.


ok setenv auto-boot? true

8. Reset and reboot the system.


Sun's Predictive Self-Healing

Solaris 10 introduces a new architecture for building and deploying systems and services capable of Predictive Self-Healing. The vca driver delivers an error telemetry for diagnosis of hardware and software problems by the Solaris Fault Manager, fmd(1M).

When problems are detected by the vca driver or Sun Crypto Accelerator 4000 firmware, error reports are sent to the fault manager daemon for diagnosis and logging. The fmdump(1M) utility can be used to view the list of problems diagnosed by the fault manager, along with their Universal Unique Identifiers (UUIDs) and knowledge article message identifiers. The fmadm(1M) utility can be used to view the resources on the system believed to be faulty. The fmstat(1M) utility can be used to report statistics kept by the fault manager. The fault manager is started automatically when Solaris boots, so it is not necessary to use the fmd command directly. Refer to the man pages for more details regarding the use of these tools.

The fault manager also sends a message to the syslogd(1M) service to notify an administrator that a problem has been detected. The message directs administrators to a knowledge article at http://www.sun.com/msg/, which explains more about the problem impact and appropriate responses. A brief description of the problem and the action required by the administrator is also provided in the message.


Troubleshooting the Sun Crypto Accelerator 4000 Board

This section describes the commands available at the OpenBoot PROM level for troubleshooting the board. Refer to the OpenBoot Command Reference Manual for more information on the commands described in the following subsections.

show-devs

To determine whether the Sun Crypto Accelerator 4000 device is listed in the system: from the OpenBoot PROM ok prompt, type show-devs to display the list of devices. You see lines in the list of devices, similar to the examples below, specific to the board:


ok show-devs
.
.
/chosen
/packages
/upa@8,480000/SUNW,ffb@0,0
/pci@8,600000/network@1
/pci@8,600000/SUNW,qlc@4
/pci@8,600000/SUNW,qlc@4/fp@0,0
.
.

In the preceding example, the /pci@8,600000/network@1 entry identifies the device path to the board. There will be one such line for each board in the system.

.properties

To determine whether the Sun Crypto Accelerator 4000 device properties are listed correctly: from the ok prompt, type .properties to display the list of properties.


ok .properties
assigned-addresses       82000810 00000000 00102000 00000000 00002000
                          81000814 00000000 00000400 00000000 00000100
                          82000818 00000000 00200000 00000000 00200000
                          82000830 00000000 00400000 00000000 00100000
d-fru-len                00 00 00 00
d-fru-off                00 00 e8 00
d-fru-dev                eeprom
s-fru-len                00 00 08 00
s-fru-off                00 00 e0 00
s-fru-dev                eeprom
compatible               70 63 69 38 30 38 36 2c 62 35 35 35 2e 31 30 38
reg                      00000800 00000000 00000000 00000000 00000000
                          02000810 00000000 00000000 00000000 00002000
                          02000814 00000000 00000000 00000000 00000100
                          02000818 00000000 00000000 00000000 00200000
                          02000830 00000000 00000000 00000000 00100000
address-bits             00 00 00 30
max-frame-size           00 00 40 00
network-interface-type   ethernet
device-type              network
name                     network
local-mac-address        00 03 ba 0e 99 ca
version                  Sun PCI Crypto Accelerator 4000 1000Base-T 
FCode 2.11.13 03/03/04
phy-type                 mif
board-model              501-6039
model                    SUNW,pci-vca
fcode-rom-offset         00000000
66mhz-capable
fast-back-to-back
devsel-speed             00000001
class-code               00100000
interrupts               00000001
max-latency              00000040
min-grant                00000040
subsystem-vendor-id      0000108e
subsystem-id             00003de8
revision-id              00000002
device-id                0000b555
vendor-id                00008086

watch-net

To monitor a network connection: from the ok prompt, type the apply watch-net command with the device path:


ok apply watch-net /pci@8,600000/network@1
/pci@8,600000/network@1: 1000 Mbps full duplex link up
Watch ethernet packets
'.' is a good packet and 'X' is a bad packet
Press any key to stop
.....X...X......X.....

The system monitors network traffic, displaying "." each time it receives an error-free packet and "X" each time it receives a packet with an error that can be detected by the network hardware interface.