Chapter 11 Troubleshooting

This chapter contains information for troubleshooting possible problems with Solstice X.25. Problems are generally related to one of the following areas:

• Cabling

• Modem problems

• Logical Channel Ranges

• Other Configuration problems

"Resolving Common Problems" offers suggestions for dealing with these problems. Check this section first to see if the procedures solve the problem.

Otherwise, follow the complete troubleshooting procedure. In general it is best to take a "bottom up" approach to troubleshooting. The tests in this chapter are described in that order.

In summary, check:

• The Physical layer:

  • Hardware

  • Line status

• The Datalink layer

  • Packet and link-level traces

  • Protocol statistics

  • Trace information

  • Streams error messages

• The Network Layer:

  • Protocol status

  • Connectivity

  • Network addresses

  • Local routing tables

  • Permissions

  • Remote operations

  • NIS operations (if applicable)

Resolving Common Problems

This section provides information for resolving common problems. These problems are most likely to appear when you bring up Solstice X.25 for the first time. If you cannot resolve the problem using the procedures given here, carry out the full set of tests described in the remainder of this chapter.

Cabling Problems

The following may be symptoms of cabling problems:

• Network does not come up

• X.25 link layer does not come up, or does not stay up

• The message xmit hung appears in the console

• No data is being sent or received

Check the following:

1. Make sure you are using the correct type of cable. Refer to Appendix A, Cabling for cabling diagrams.

2. Make sure all cables are properly seated.

3. If you are operating over an LAPB link, make sure that you are using a cable that is designed for synchronous use. If you have a spare cable, replace your existing cable and retry the connection.

Modem Problems

The following may be symptoms of modem problems:

• Network does not come up

• X.25 link layer does not come up, or does not stay up

• The message xmit hung appears in the console

• No data is being sent and or received

Check the following:

1. Make sure your modem is correctly configured. Check your modem's documentation.

2. If you still have a connection, check the Transmit and Receive Data lights on your modem: with LAPB transmission, these should always be on.

3. If you have a connection, use a breakout box at the Sun (not the modem) end of the connection to check that RS-232-C pins 2, 3, 4, 5, 6, 7, 8, 15, 17, and 20 are functioning. Pay particular attention to the Transmit Clocking and Receive Clocking pins, 15 and 17. If the lights corresponding to these pins are not lit, it indicates that the local and/or remote modems are not supplying clocking. Notify the remote end and use your modem's documentation to perform troubleshooting.

4. If you are using a null modem connection, one side should be a DTE, the other a DCE and one or both sides must supply clocking. See "Building the Null Modem Cable" for instructions on setting up a null modem connection.

Logical Channel Ranges

Logical channel number mismatches can be difficult to diagnose, especially if you are viewing output from x25trace. You see a Call Request packet going out, but no response at the X.25 Packet Layer. In a facilities mismatch, for example, you receive a Clear Indication from the remote end. The Clear Indication contains an error code that is occasionally helpful in pinpointing the problem. The following symptoms may mean that the logical channel ranges are incorrectly configured:

• Inability to establish a packet level connection--for example PAD calls fail

• Traffic can be sent in one direction only

• PAD calls are cleared with the error Call Cleared - Out of Order (0900)

• Calls are cleared with an error code of 0x00 24

Check that the logical channel ranges you configured match those used on the X.25 network. If this is a PSDN, contact your service provider to find out what these should be. Refer to "Configuring a Single WAN Link" for information on how to use x25tool to set the logical channel ranges.

Other Configuration Problems

The following symptoms may indicate that there may be a mistake in the configuration:

• Network does not come up

• X.25 link layer does not come up, or does not stay up

• Inability to establish a packet level connection--for example PAD calls fail

• Link is up but not responding correctly

• Repeated Resets are being received

Check the following:

1. Make sure you specified the correct serial port when you ran x25tool or edited the configuration file. Refer to "Configuring a Single WAN Link" for how to configure this using x25tool.

2. For LAPB links, check that the link is configured as a DTE if communicating over a PSDN.

3. Facilities and other parameter mismatches, between your machine and the PSDN, can prevent the establishment of virtual circuits. Check the Negotiate Toward Defaults item in the Link Editor, Throughput window. For the majority of PSDNs this should be set to No.

4. Check where calls are being cleared from. If you receive an error code of the form 00 nn, the leading zeroes indicate the call-clearing request originated with the remote host. If the first two digits are not 00,the call-clearing request originated in the PSDN.

5. If you receive an error code of 0x00 43, check that your X.121 address is specified correctly. If it is, check to see whether there is a process listening for an incoming call.

6. If you receive error code 0x00 42, it is likely that your maximum I-frame size, an LAPB-layer parameter, is set incorrectly for your link. The maximum I-frame size is a subscription option to the PSDN. When LAPB is in extended mode, set this parameter eight bytes larger than the default packet size parameter at the packet level.When LAPB is in normal mode, set it five bytes larger. If Solstice X.25 receives a Call Request with a packet size that is within eight bytes (or five bytes) of the maximum I-frame size parameter, the X.25 software clears the call with the 0x00 42 diagnostic. Modify your LAPB parameters as necessary.

7. If you receive error code 0x00 70, check your console to see if you have a licensing problem.

8. If you can find out about the remote host, check that it is still up.

9. If you are running IP over X.25, run ifconfig commands to check on your ixe device (for example, zsh0). Use of ifconfig is described in "Checking the Protocol Status ". If the response from ifconfig is UP, POINTOPOINT, and RUNNING, check with your PSDN to see if there is a problem at the network end.

10. Run x25trace to trace the exchange of X.25 packets across the link. People who are familiar with the X.25 Packet Layer Protocol can interpret the output from x25trace to determine where errors are occurring. The x25trace command is described in "Obtaining Packet and Link-Level Traces".

11. If you receive a message from PAD such as, i/o error, open /dev/x25 failed, it might indicate that the X.25 network daemon is down. Check the status of the daemon in x25tool. This message may also indicate that there are two many simultaneous PAD calls for the software to cope with. Wait a moment, then try again.

12. If you experience intermittent disconnections or resets, check for configuration errors in your link configuration file. At the packet level, make sure that the window and packet size parameters agree, between your machine and the PSDN.

13. If you receive repeated RESETS with error code 0x0092, there may be a window size mismatch.

14. Perform a loopback test, to check your local configuration down to the link layer.

15. Perform a back-to-back test, to test your configuration against that of another machine.

For error code descriptions, refer to Appendix C, Error Messages and Error Codes.

Checking the Physical Layer

First of all make sure that all of your modem and power cables are in good working order, and that they are all plugged-in, switched on, and tightly seated. Then carry out loopback tests. Finally, check the line status.

If you are using the onboard serial port of a SPARCstation, the most thorough hardware test you can do is to use syncloop. If you are using the SUN HSI card, the most thorough hardware test you can do is to use hsi_loop. If you are using another manufacturer's card, refer to their documentation.

To check the line status, use the syncstat or hsi_stat command to observe the line over periods of ten seconds.

Checking a High Speed Interface

If you are using a high speed interface, use the hsi_loop command to perform a loopback test to check the following components of your communications link:

• Software configuration

• CPU-to-card communication

• Correct operation of the serial port

• EIA-449 or EIA-232 ports and cables

• Local and remote modems

• Transmission line

If you see errors in the output, contact your local technical support for help in interpreting them.

Checking the Serial Port Line Status

The syncstat command monitors traffic that uses the onboard serial port or the serial port on the SCiiExpress-X card.

If you see output packets but no input packets, then either the remote system is not initialized or the line is not properly connected to the remote system. If you see input packets with CRC errors, the transmission medium is causing errors. If you see neither input nor output packets, then the Solstice X.25 protocol module was not successfully initialized. Try restarting the device.

Checking High Speed Interface Line Status

The hsi_stat command monitors traffic over a high speed interface. You must log in as root, or become superuser, to run hsi_stat.

If you see output packets but no input packets, then either the remote system is not initialized or the line is not properly connected to the remote system. If you see input packets with CRC errors, the transmission medium is causing errors. If you see neither input nor output packets, then the Solstice X.25 protocol module was not successfully initialized. Try restarting the device.

Checking the Datalink Layer

If the problem is not at the physical layer, the next thing to check is the Solstice X.25 software and configuration. The most important thing is to try and work out in which layer--X.25, LAPB, LLC2, or WAN--the problem originates. In general, you should check the following:

• Check the link and packet layers are up--use x25stat.

• Observe connection attempts and obtain clearing causes and diagnostic codes--use x25trace.

• Log protocol activity--use the strace command and the strerr daemon.

• Obtain an ASCII record of the values of the parameters for all of the layers in your X.25 configuration-- use the x25info utility, in /opt/SUNWconn/bin. This is particularly useful when you need to send a description of your configuration to your support provider.

Using one or more of these diagnostic tools, you can usually obtain sufficient information to diagnose and correct your communications problem.

Obtaining Packet and Link-Level Traces

Use /opt/SUNWconn/bin/x25trace command to capture information about each packet and/or frame sent and received by Solstice X.25.

You can specify the layer you want to trace, the interface you want to trace, and the destination you want to trace. This lets you narrow down the information you receive.

The x25trace command takes the form shown below:

# /opt/SUNWconn/bin/x25trace [options] [-i interface] filter_expression 

You must run it as root, in the foreground. If you want to capture its output in a file, use standard Unix redirection to do so. (This is useful if you intend to contact your local technical support representative for help.) x25trace runs until you enter a Ctrl-C.

You can use specific MAC addresses as filters. For example, you can enter a command that has the effect of saying, "Trace all packets that travel over interface A between address 1 and address 2." Such commands can extend beyond the width of a command line. In this case, use the backslash (\) continuation character to go beyond a single line.

Table 11-1 lists the devices x25trace supports:

The following options are available. They are supported by all of the devices.

-a

By default, x25trace displays the user data in hexadecimal for the highest protocol specified. This option tells x25trace to only display the number of bytes of user data and not display the data in hexadecimal. For example, if you enter:

hostname# x25trace -a -i /dev/llc2 x25

x25trace displays only the number of bytes in X.25 packets sent and received over the llc0 interface.

-l number

Specifies the link on which x25trace is to trace packets. By default, x25trace traces on all links and prints the link number in the traced information. (This option is useful only in the situation in which you have multiple links.)

-u

This option causes x25trace to buffer display output line-by-line, instead of the default operation of packet-by-packet.

-x

This option causes x25trace to display entire packets in hexadecimal, in addition to its default operation of decoding the packet. This option is useful in troubleshooting malformed packets. With such packets, you often see an error message starting with two asterisks (**).

You can use the following filter expressions in an x25trace command line:

betweenmac

Trace only the packets or frames passing between the 802.x MAC addresses you specify.

dstmac

Trace only the packets or frames that have a destination address that is the MAC address you specify.

lapb (or hdlc)

Trace only LAPB frames.

mac

Trace packets and frames to/from the MAC address you specify. Use only when tracing on a LAN interface.

multicast

Trace only LLC2 packets that have multicast addresses.

pdu_in

Trace only incoming Protocol Data Units (PDUs).

pdu_out

Trace only outgoing Protocol Data Units (PDUs).

srcmac

Trace only the packets or frames that have a source address that is the MAC address you specify.

x25

Trace only X.25 Packet Layer Protocol packets.

x25lcn [+|num>]

Used with a plus sign (+), this expression means trace only the packets that travel on the next logical channel set up. Used with a number, it means trace only the packets that travel on the logical channel identified by num.

x25trace Examples

Below are some examples of using x25trace for tracing incoming and outgoing packets.

Tracing LAPB and X.25 on /dev/lapb:

To trace LAPB frames and X.25 packets as they are sent or received by LAPB, enter:

hostname# x25trace -i /dev/lapb 

To trace X.25 packets as they are seen by the X25 driver, type one of the two following commands:

hostname# x25trace -i /dev/x25 x25
hostname# x25trace

Tracing LLC2 and X.25 on /dev/llc2:

hostname# x25trace -i /dev/llc2 

The command above captures all LLC frames, including Unnumbered Information frames used to carry CLNP and ES-IS PDUs. When tracing at the X.25 level on an LLC2 link, MAC addresses are displayed as 0:0:0:0:0:0. Tracing at the LLC2 level on the same link returns the correct MAC addresses.

Tracing a single X.25 connection (logical channel number):

hostname# x25trace -i /dev/lapb x25lcn  lcn_num x25 

Tracing only the next X.25 connection being set up:

hostname# x25trace -i /dev/lapb x25lcn + x25

Tracing LLC2 frames on /dev/llc2:

hostname# x25trace -i /dev/llc2 llc

Tracing X.25 packets to/from specific MAC addresses on /dev/llc2:

hostname# x25trace -i /dev/llc2 srcmac 8:20:0:1:2:3 x25
hostname# x25trace -i /dev/llc2 dstmac 8:20:0:1:2:3 x25
hostname# x25trace -i /dev/llc2 betweenmac 8:20:0:1:2:3 9:0:2b:18:21:5 x25 

Trace outgoing LLC2 frames that are not multicast:

hostname# x25trace -i /dev/llc2 pdu_out not multicast

Redirecting trace of X.25 packets over an LAPB link to a file:

hostname# x25trace -a -i /dev/lapb x25 > /var/adm/x25/packets.record 

Displaying Protocol Statistics

Displaying protocol statistics lets you track what is going on a link or links. You can display statistics on a per-protocol basis, or display statistics for all protocols.

To display statistics, start x25tool, then pull down the Network menu. Choose the Statistics X.25 menu item. The Network Statistics window appears. Choose the protocol you would like statistics for by clicking on the box to the left of the protocol name. You can select more than one protocol; for example, you can select X.25 and MLP.

Once you have select a protocol or protocols, specify a Link Number or All. Specify the Interval (in seconds) that Statistics should be collected for, and the level of detail you require. For example, you may want to isolate the statistics for a particular VC, if you suspect that is where the problem is. Click on Display to display the statistics--Clear resets them to zero. Once you are happy with your settings, click on Apply.

To collect and display statistics, click on the Display button in the Network Statistics window. Statistics appear in the window and are updated every time the Interval elapses. If you do not have x25tool running on your system, you can access the same information by running the command x25stat.

Logging Trace Information

The SunOS 5.x streams strace (1M) command lets you log trace information. See the strace man page for details on the command's use.

The strace command must be followed by three arguments:

module id

Table 11-2 lists the possible values:

link number

The number of the link over which the driver you are tracing is running, or all to specify all links.

level

The tracing level that allows you to receive more or fewer packets or frames. Table 11-3 lists the available strace tracing levels for the X.25, LAPB, and LLC2 drivers.

Specify all to trace all available levels. For example, to trace X.25 PLP packets on all links at all tracing levels, type:

# /usr/sbin/strace 200 all all

Note that strace is owned by root and is executable only by root.

The tracing of an incoming event does not mean that the packet or frame has been accepted by the driver at the layer you are tracing. This is because, for a given layer, the tracing of incoming events is triggered on receiving data from the layer below. At this point, the packet or frame is not yet verified. If the packet or frame is subsequently found to be in error, it might be discarded or cause some further protocol action.

The successful completion of a trace of an outgoing event at the X.25 layer does not necessarily mean that the packet has been sent to the link layer. Following tracing, various consistency checks are performed on the link-level database. If these checks fail, the packet will be discarded. At the LAPB and LLC2 layers, successful tracing does mean that the frame was sent to the WAN or LAN driver. However, it does not mean that the frame will be transmitted on the line.

Capturing Streams Error Messages

In Solstice X.25, the X.25, LAPB, and LLC2 drivers can generate streams error messages. The SunOS 5.x streams strerr (1M) daemon captures streams error messages. This daemon receives error messages and appends them to log files in /var/adm/streams. Log file names take the form error.mm-dd, where mm and dd indicate the month and day the messages were written to the file.

If you are experiencing problems with Solstice X.25 and are uncertain of the source, run strerr. If you receive a streams error message, contact your local Sun customer service representative.

You must be root to run the strerr daemon. Start it like this:

hostname# /usr/sbin/strerr &

The daemon runs until you kill it.

See the strerr man page for a description of the syntax of the messages in the streams error message file.

Checking the Network Layer

Once you are sure that the physical and datalink layers are both working correctly, check to see whether the problem is at the network layer. Check the following:

• the state of the line, using ifconfig

• connectivity, using ping

• the network address, using netstat -i

• the routing tables, using netstat -r

• permissions in the /etc/hosts.equiv file

• remote operation, using rlogin

• NIS operation, using ypmatch

Checking the Protocol Status

When Solstice X.25 is running, you can use ifconfig to monitor the current state of the line. Give the X.25 interface name as an argument. For example,

hostname# ifconfig ixe
ixe: 192.9.200.2 flags=51<UP,RUNNING,PRIVATE>

If the words UP and RUNNING are displayed, then the connection is potentially intact.

If ifconfig does not display UP and RUNNING, then either you did not configure the X.25 module correctly or the remote system cannot be contacted.

The PRIVATE flag, shown in the display above, is associated with Solstice X.25 and SunLink X.25 devices. The flag means that the routing daemon does not broadcast the host route for the X.25 device to the local network.

Wait 30 seconds after bringing the X.25 link up to check these statistics, as these flags can be up to 30 seconds out of date.

Checking Connectivity

Use ping to check that the connection is up:

hostname# ping -r gateway_a
gateway_a is alive

The -roption tells ping that the remote host is on a directly-connected interface, such as a X.25 link. If the remote host does not respond, a routing problem exists at some point between the local and remote hosts.

Checking the Network Addresses

Use the netstat -icommand to check that the correct local and remote addresses are assigned to the Solstice X.25 interface:

hostname# netstat -i

Make sure that the IP addresses for the local and remote X.25 interfaces are the same in the /etc/hosts files or NIS host maps for the machines on both ends of the X.25 link.

Checking the Local Routing Tables

Use the netstat -r command to display the local routing tables:

hostname# netstat -r 

The routing table looks like this:

Routing tables
 Destination     Gateway     Flags     Refcnt     Use      Interface
 host_a          sun-bb      UGH       0          0        ie1
 host_b          sun-bb      UGH       0          0        ie1
 gateway_b       gateway_a   UGH       1          12897    ixe0
 route7          route7      UGH       0          0        ie0
 eastgate        route71     UGH       0          158      ie0
 backbone        alpha-bb    U         1          16087    ie1
 dresdenpc       route1      UG        0          0        ie1
 loopback        localhost   U         2          113436   lo0
 beta-bb         alpha       U         4063       146044   ie0
 dallas2         route7      UG        0          0        ie0
 trainingpc      route62     UG        0          0        ie1 

Make sure there is a routing table entry for each possible destination network. In particular, Solstice X.25 devices, listed under Interface, should be matched with the appropriate host names listed under Gateway. The Gateway entry should, in turn, be matched with the correct entry under Destination.

If there is not a routing table entry for each possible destination network, and you are using static routing, add the appropriate static routes. If you are using dynamic routing with in.routed, do the following:

1. Check that in.routed is running by typing:

   # ps -ef | grep route root process_id 1 80 Feb 22 1:55 /usr/sbin/in.routed -q

   If the routing tables still are not correct, become superuser, and continue with the next steps.

2. Kill in.routed and flush the routing tables:

   #kill -9 process_id #/usr/sbin route -f

   where process_ID is the process ID displayed by the ps -ef command.

3. Restart in.routed as follows:

   #/usr/sbin/in.routed

Checking Permissions

If you attempt to use rsh or rlogin and receive the message Permission denied, it is because the remote system's /etc/hosts.equiv or /.rhosts does not contain the sending system's host name or does not contain the line `+'. For example, if gateway_a is to have permissions for gateway_b, then gateway_a should appear in the /etc/hosts.equiv file of gateway_b.

Checking Remote Operations

Check that remote operations are working correctly by using rlogin or rsh to reach the remote host over the X.25 link. If this fails, it probably indicates that the machines on each end of the link have different MTU sizes.

Checking NIS Operations

If your network or the network at the opposite side of the X.25 link run NIS, you should ensure that NIS operation is working correctly. Type ypwhich at the command line. You should receive the name of an NIS server on the internetwork as a response. If you don't, this indicates problems with NIS over the X.25 link.

On devices where the interface running Solstice X.25 is the only network interface, you need to perform a few extra steps if you want to get NIS service from across the link. You can do either of the following:

1. Make the remote machine the NIS server.

2. Assign an NIS server for the local machine.

   If you choose the second option, do the following:

1. Become superuser.

2. Run ypbind as follows:

   # ypbind -ypsetme

3. Assign a known NIS server to the local X.25 routing gateway, as follows:

   # /usr/sbin/ypset server_addr

   where server_addr is the IP address of an NIS server on the local network.

Licensing Problems

Solstice X.25 only runs when a license is available. If you are using Solstice X.25, and its license ceases to be available, the following message is printed to the console:

Solstice X.25, waiting to get license

If, after three attempts, the license is still unavailable, Solstice X.25 prints the following message to the console and closes down:

Unable to regain license, X.25 closing down

The most likely cause is a network problem between the device running Solstice X.25 and the license server.