Chapter 6 Troubleshooting TCP/IP
This chapter describes general methods for troubleshooting TCP/IP networks and some of the tools available for doing so. These tools include ping, ifconfig, netstat, and route.
General Troubleshooting Methods
One of the first signs of trouble on the network is a loss of communications by one or more hosts. If a host refuses to come up at all the first time it is added to the network, the problem might lie in one of the configuration files, or in the network interface. If a single host suddenly develops a problem, the network interface might be the cause. If the hosts on a network can communicate with each other but not with other networks, the problem could lie with the router, or it could lie in another network.
You can use the ifconfig program to obtain information on network interfaces and netstat to display routing tables and protocol statistics. Third-party network diagnostic programs provide a number of troubleshooting utilities. Refer to third-party documentation for information.
Less obvious are the causes of problems that degrade performance on the network. For example, you can use tools like ping to quantify problems like the loss of packets by a host.
Running Software Checks
If there is trouble on the network, some actions that you can take to diagnose and fix software-related problems include:
-
Checking the hosts database to make sure that the entries are correct and up to date.
-
If you are running RARP, checking the Ethernet addresses in the ethers database to make sure that the entries are correct and up to date.
-
Trying to connect by telnet to the local host.
-
Ensuring that the network daemon inetd is running. To do this, log in as superuser and type:
# ps -ef | grep inetd
Here is an example of output displayed if the inetd daemon is running:
root 57 1 0 Apr 04 ? 3:19 /usr/sbin/inetd -s root 4218 4198 0 17:57:23 pts/3 0:00 grep inetd |
ping Command
Use the ping command to find out whether there is IP connectivity to a particular host. The basic syntax is:
/usr/sbin/ping host [timeout]
where host is the host name of the machine in question. The optional timeout argument indicates the time in seconds for ping to keep trying to reach the machine--20 seconds by default. The ping(1M) man page describes additional syntaxes and options.
When you run ping, the ICMP protocol sends a datagram to the host you specify, asking for a response. (ICMP is the protocol responsible for error handling on a TCP/IP network. See "ICMP Protocol" for details.)
Suppose you type:
$ ping elvis |
If host elvis is up, this message is displayed:
elvis is alive |
indicating that elvis responded to the ICMP request. However, if elvis is down or cannot receive the ICMP packets, you receive the following response from ping:
no answer from elvis |
If you suspect that a machine might be losing packets even though it is up, you can use the s option of ping to try to detect the problem. For example, type:
$ ping -s elvis |
ping continually sends packets to elvis until you send an interrupt character or a timeout occurs. The responses on your screen will resemble:
PING elvis: 56 data bytes 64 bytes from 129.144.50.21: icmp_seq=0. time=80. ms 64 bytes from 129.144.50.21: icmp_seq=1. time=0. ms 64 bytes from 129.144.50.21: icmp_seq=2. time=0. ms 64 bytes from 129.144.50.21: icmp_seq=3. time=0. ms . . . ----elvis PING Statistics---- 4 packets transmitted, 4 packets received, 0% packet loss round-trip (ms) min/avg/max = 0/20/80 |
The packet-loss statistic indicates whether the host has dropped packets.
If ping fails, check the status of the network reported by ifconfig and netstat, as described in "ifconfig Command" and "netstat Command".
ifconfig Command
The ifconfig command displays information about the configuration of an interface that you specify. (Refer to the ifconfig(1M) man page for complete details.) The syntax of ifconfig is:
ifconfig interface-name [protocol_family]
If you want information about a specific interface, for example le0, type:
$ ifconfig le0 |
For an le0 interface, your output resembles the following:
le0: flags=863<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 129.144.44.140 netmask ffffff00 broadcast 129.144.44.255 ether 8:0:20:8:el:fd |
The flags section just given shows that the interface is configured "up," capable of broadcasting, and not using "trailer" link level encapsulation. The mtu field tells you that this interface has a maximum transfer rate of 1500. Information on the second line includes the IP address of the host you are using, the netmask being currently used, and the IP broadcast address of the interface. The third line gives the machine address (Ethernet, in this case) of the host.
A useful ifconfig option is -a, which provides information on all interfaces on your network. For example, typing ifconfig -aproduces:
le0: flags=49<UP,LOOPBACK,RUNNING> mtu 8232
inet 127.144.44.140 netmask ff000000
le0:flags=863<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 129.144.44.140 netmask ffffff00 broadcast 129.144.44.255
ether 8:0:20:8:el:fd
|
Output that indicates an interface is not running might mean a problem with that interface. In this case, see the ifconfig(1M) man page.
netstat Command
The netstat command generates displays that show network status and protocol statistics. You can display the status of TCP and UDP endpoints in table format, routing table information, and interface information.
netstat displays various types of network data depending on the command line option selected. These displays are the most useful for system administration. The syntax for this form is:
netstat [-m] [-n] [-s] [-i | -r] [-f address_family]
The most frequently used options for determining network status are: s, r, and i. See the netstat(1M) man page for a description of the options.
Displaying Per Protocol Statistics
The netstat -s option displays per protocol statistics for the UDP, TCP, ICMP, and IP protocols. The result resembles the display shown in the example below. (Parts of the output have been truncated.) The information can indicate areas where a protocol is having problems. For example, statistical information from ICMP can indicate where this protocol has found errors.
UDP
udpInDatagrams = 39228 udpOutDatagrams = 2455
udpInErrors = 0
TCP
tcpRtoAlgorithm = 4 tcpMaxConn = -1
tcpRtoMax = 60000 tcpPassiveOpens = 2
tcpActiveOpens = 4 tcpEstabResets = 1
tcpAttemptFails = 3 tcpOutSegs = 315
tcpCurrEstab = 1 tcpOutDataBytes = 10547
tcpOutDataSegs = 288 tcpRetransBytes = 8376
tcpRetransSegs = 29 tcpOutAckDelayed = 23
tcpOutAck = 27 tcpOutWinUpdate = 2
tcpOutUrg = 2 tcpOutControl = 8
tcpOutWinProbe = 0 tcpOutFastRetrans = 1
tcpOutRsts = 0
tcpInSegs = 563 tcpInAckBytes = 10549
tcpInAckSegs = 289 tcpInAckUnsent = 0
tcpInDupAck = 27 tcpInInorderBytes = 673
tcpInInorderSegs = 254 tcpInInorderBytes = 673
tcpInUnorderSegs = 0 tcpInUnorderBytes = 0
tcpInDupSegs = 0 tcpInDupBytes = 0
tcpInPartDupSegs = 0 tcpInPartDupBytes = 0
tcpInPastWinSegs = 0 tcpInPastWinBytes = 0
tcpInWinProbe = 0 tcpInWinUpdate = 237
tcpInClosed = 0 tcpRttNoUpdate = 21
tcpRttUpdate = 266 tcpTimRetrans = 26
tcpTimRetransDrop = 0 tcpTimKeepalive = 0
tcpTimKeepaliveProbe= 0 tcpTimKeepaliveDrop = 0
IP
ipForwarding = 2 ipDefaultTTL = 255
ipInReceives = 4518 ipInHdrErrors = 0
ipInAddrErrors = 0 ipInCksumErrs = 0
ipForwDatagrams = 0 ipForwProhibits = 0
ipInUnknownProtos = 0 ipInDiscards = 0
ipInDelivers = 4486 ipOutRequests = 2805
ipOutDiscards = 5 ipOutNoRoutes = 0
ipReasmTimeout = 60 ipReasmReqds = 2
ipReasmOKs = 2 ipReasmReqds = 2
ipReasmDuplicates = 0 ipReasmFails = 0
ipFragOKs = 20 ipReasmPartDups = 0
ipFragCreates = 116 ipFragFails = 0
tcpInErrs = 0 ipRoutingDiscards = 0
udpInCksumErrs = 0 udpNoPorts = 33
rawipInOverflows = 0 udpInOverflows = 6
ICMP
icmpInMsgs = 0 icmpInErrors = 0
icmpInCksumErrs = 0 icmpInUnknowns = 0
icmpInDestUnreachs = 0 icmpInTimeExcds = 0
icmpInParmProbs = 0 icmpInSrcQuenchs = 0
icmpInRedirects = 0 icmpInBadRedirects = 0
icmpInEchos = 0 icmpInEchoReps = 0
icmpInTimestamps = 0 icmpInTimestampReps = 0
icmpInAddrMasks = 0 icmpInAddrMaskReps = 0
icmpInFragNeeded = 0 icmpOutMsgs = 7
icmpOutDestUnreachs = 1 icmpOutErrors = 0
icmpOutDrops = 5 icmpOutTimeExcds = 0
icmpOutParmProbs = 0 icmpOutSrcQuenchs = 6
icmpOutRedirects = 0 icmpOutEchos = 0
icmpOutEchoReps = 0 icmpOutTimestamps = 0
icmpOutTimestampReps= 0 icmpOutAddrMasks = 0
icmpOutAddrMaskReps = 0 icmpOutFragNeeded = 0
icmpInOverflows = 0
IGMP:
0 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 membership queries received
0 membership queries received with invalid field(s)
0 membership reports received
0 membership reports received with invalid field(s)
0 membership reports received for groups to which we belong
0 membership reports sent
|
Displaying Network Interface Status
The i option of netstat shows the state of the network interfaces that are configured with the machine where you ran the command. Here is a sample display produced by netstat -i.
Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis Queue le0 1500 b5-spd-2f-cm tatra 14093893 8492 10174659 1119 2314178 0 lo0 8232 loopback localhost 92997622 5442 12451748 0 775125 0 |
Using this display, you can find out how many packets a machine thinks it has transmitted and received on each network. For example, the input packet count (Ipkts) displayed for a server can increase each time a client tries to boot, while the output packet count (Opkts) remains steady. This suggests that the server is seeing the boot request packets from the client, but does not realize it is supposed to respond to them. This might be caused by an incorrect address in the hosts or ethers database.
On the other hand, if the input packet count is steady over time, it means that the machine does not see the packets at all. This suggests a different type of failure, possibly a hardware problem.
Displaying Routing Table Status
The -r option of netstat displays the IP routing table. Here is a sample display produced by netstat -r run on machine tenere.
Routing tables Destination Gateway Flags Refcnt Use Interface temp8milptp elvis UGH 0 0 irmcpeb1-ptp0 elvis UGH 0 0 route93-ptp0 speed UGH 0 0 mtvb9-ptp0 speed UGH 0 0 . mtnside speed UG 1 567 ray-net speed UG 0 0 mtnside-eng speed UG 0 36 mtnside-eng speed UG 0 558 mtnside-eng tenere U 33 190248 le0 |
The first column shows the destination network, the second the router through which packets are forwarded. The U flag indicates that the route is up; the G flag indicates that the route is to a gateway. The H flag indicates that the destination is a fully qualified host address, rather than a network.
The Refcnt column shows the number of active uses per route, and the Use column shows the number of packets sent per route. Finally, the Interface column shows the network interface that the route uses.
Logging Network Problems
If you suspect a routing daemon malfunction, you can log its actions, including all packet transfers. To create a log file of routing daemon actions, supply a file name when you start up the routed daemon. For example:
# /usr/sbin/in.routed /var/routerlog |
Caution - On a busy network, this can generate almost continuous output.
Displaying Packet Contents
You can use snoop to capture network packets and display their contents. Packets can be displayed as soon as they are received, or saved to a file. When snoop writes to an intermediate file, packet loss under busy trace conditions is unlikely. snoop itself is then used to interpret the file. For information about using the snoop command, refer to the snoop(1M) man page.
The snoop command must be run by root (#) to capture packets to and from the default interface in promiscuous mode. In summary form, only the data pertaining to the highest-level protocol is displayed. For example, an NFS packet will have only NFS information displayed. The underlying RPC, UDP, IP, and Ethernet frame information is suppressed but can be displayed if either of the verbose options is chosen.
The snoop capture file format is described in RFC 1761. To access, use your favorite web browser with the URL: http://ds.internic.net/rfc/rfc1761.txt.
snoop server client rpc rstatd collects all RPC traffic between a client and server, and filters it for rstatd.
How to check all packets from your system
-
Type netstat -i to find the interfaces attached to the system.
Snoop normally uses the first non-loopback device (le0).
-
Become root and type snoop
Use Ctl C to halt the process.
# snoop Using device /dev/le (promiscuous mode) maupiti -> atlantic-82 NFS C GETATTR FH=0343 atlantic-82 -> maupiti NFS R GETATTR OK maupiti -> atlantic-82 NFS C GETATTR FH=D360 atlantic-82 -> maupiti NFS R GETATTR OK maupiti -> atlantic-82 NFS C GETATTR FH=1A18 atlantic-82 -> maupiti NFS R GETATTR OK maupiti -> (broadcast) ARP C Who is 129.146.82.36, npmpk17a-82 ? -
Interpret results
In the example, client maupiti transmits to server atlantic-82 using NFS file handle 0343. atlantic-82 acknowledges with OK. The conversation continues until maupiti broadcasts an ARP request asking who is 129.146.82.36?
This example demonstrates the format of snoop. The next step is to filter snoop to capture packets to a file.
Interpret the capture file using details described in RFC 1761. To access, use your favorite web browser with the URL: http://ds.internic.net/rfc/rfc1761.txt
How to capture snoop results to a file
-
As root, type snoop -o filename. Example:
# snoop -o /tmp/cap Using device /dev/le (promiscuous mode) 30 snoop: 30 packets captured
This has captured 30 packets in a file /tmp/cap. The file can be anywhere there is enough disk space. The number of packets captured is displayed on the command line, enabling you to press Ctl-C to abort at any time.
snoop creates a noticeable networking load on the host machine, which can skew the results. To see reality at work, run snoop from a third system, (see the next section).
-
Type snoop -i filename to inspect the file:
# snoop -i /tmp/cap 1 0.00000 frmpk17b-082 -> 224.0.0.2 IP D=224.0.0.2 S=129.146.82.1 LEN=32, ID=0 2 0.56104 scout -> (broadcast) ARP C Who is 129.146.82.63, grail ? 3 0.16742 atlantic-82 -> (broadcast) ARP C Who is 129.146.82.76, honeybea ? 4 0.77247 scout -> (broadcast) ARP C Who is 129.146.82.63, grail ? 5 0.80532 frmpk17b-082 -> (broadcast) ARP C Who is 129.146.82.92, holmes ? 6 0.13462 scout -> (broadcast) ARP C Who is 129.146.82.63, grail ? 7 0.94003 scout -> (broadcast) ARP C Who is 129.146.82.63, grail ? 8 0.93992 scout -> (broadcast) ARP C Who is 129.146.82.63, grail ? 9 0.60887 towel -> (broadcast) ARP C Who is 129.146.82.35, udmpk17b-82 ? 10 0.86691 nimpk17a-82 -> 129.146.82.255 RIP R (1 destinations)
Refer to specific protocol documentation for detailed analysis and recommended parameters for ARP, IP, RIP and so forth. Searching the web is a good place to look at RFCs.
How to check packets between server and client
-
Establish a snoop system off a hub connected to either the client or server.
The third system (the snoop system) sees all the intervening traffic, so the snoop trace reflects reality on the wire.
-
As root, type snoop with options and save to a file.
-
Inspect and interpret results.
Look at RFC 1761 for details of the snoop capture file. To access, use your favorite web browser with the URL: http://ds.internic.net/rfc/rfc1761.txt
Use snoop frequently and consistently to get a feel for normal system behavior. For assistance in analyzing packets, look for recent white papers and RFCs, and seek the advice of an expert in a particular area, such as NFS or YP. For complete details on using snoop and its options, refer to the snoop(1M) man page.
- © 2010, Oracle Corporation and/or its affiliates