Go to main content

Administering TCP/IP Networks, IPMP, and IP Tunnels in Oracle® Solaris 11.4

Exit Print View

Updated: November 2020
 
 

Monitoring and Analyzing the Network

This section describes selected tools you can use to monitor your network's components and its hosted traffic to gauges performance and detect potential problem areas.

Using tshark and Wireshark

tshark is a command-line network traffic analyzer that can capture packet data from a live network. A decoded form of the data is either printed to standard output or written to a file. In addition, tshark is capable of detecting, reading, and writing the same capture files as those that are supported by Wireshark.

Used without any options, tshark works similarly to the tcpdump command and also uses the same live capture file format, libpcap.

Wireshark is a third-party graphical user interface (GUI) network protocol analyzer that is used to interactively dump and analyze network traffic. Similar to the snoop command, Wireshark displays packet data on a live network or from a previously saved capture file. Like the tcpdump utility and other tools, Wireshark uses the libpcap format for file captures. However, Wireshark is also capable of reading and importing several other file formats.

    Both tshark and Wireshark provide several unique features, such as:

  • Capability to assemble all of the packets in a TCP conversation and displaying the data in that conversation in ASCII, EBCDIC or hex format

  • More fields that can be filtered than in other network protocol analyzers

  • Richer syntax than other network protocol analyzers for creating filters

To use these tools, make sure that the tshark and wireshark software packages are installed.

For reference, see the Wireshark documentation at http://www.wireshark.org/ and the tshark(1) and wireshark(1) man pages.

Using the netstat Command

The netstat command displays network status and protocol statistics of TCP, SCTP, and UDP endpoints as well as routing table and interface information.

By itself, the command displays the status of connected sockets on the system. To display the status of both connected and unconnected sockets, use the netstat -a syntax.

The command has other options which can be used alone or in combination to tailor the output to your preference. The examples that follow show how to use different netstat options.

For reference, see the netstat(8) man page.

Displaying the Status of Sockets

By default, the netstat command displays both IPv4 and IPv6 information. To limit the information to a specific IP version, do one of the following:

  • Set the DEFAULT_IP variable in the /etc/default/inet_type file:

    DEFAULT_IP=ip-version

    ip-version can be IP_VERSION4 or IP_VERSION6.

    For more information, see the inet_type(5) man page.

  • Use the –f version option with the command. version can be one of the following:

    • inet for IPv4
    • inet6 for IPv6
    • sdp for Socket Description Protocol
    • unix for UNIX domain sockets used for internal communications
Example 1  Displaying Connected Sockets

This example shows how to limit the output to the status of connected IPv4 sockets only.

$ netstat -f inet

TCP: IPv4
   Local Address        Remote Address    Swind Send-Q Rwind Recv-Q    State
-------------------- -------------------- ----- ------ ----- ------ -----------
system-1.ssh          remote.38474     128872      0 128872      0 ESTABLISHED
system2.40721         remote.ldap      49232       0 128872      0 ESTABLISHED
Example 2  Displaying State Only of Sockets That Use the SO_REUSEPORT Mechanism

This example shows how to use the –L option to display information about sockets that are using the SO_REUSEPORT socket option. This option is currently only supported for TCP, UDP, and SCTP.

The –u option provides additional information about users, process IDs, and programs that either created the network endpoint or currently controls the network endpoint.

$ netstat -Lu
TCP: IPv4
Local    Remote
Address  Address  User     Pid     Command     Swind  Send-Q  Rwind  Recv-Q  State
-------  -------  -------  ------  ----------  -----  ------  ------ ------  ------
*.8001   *.*      userfoo  102185  web_server  0      0       128000 0       LISTEN
*.8001   *.*      userfoo  102188  web_server  0      0       128000 0       LISTEN
*.1234   *.*      userfoo  102138  foo_server  0      0       128000 0       LISTEN
*.1234   *.*      userfoo  101945  foo_server  0      0       128000 0       LISTEN

The output shows two TCP listeners on port 8001 and two TCP listeners on port 1234. Both ports are using the SO_REUSEPORT load balancing feature.

Displaying Statistics by Protocol

The –P argument option filters the output of netstat by protocol. argument can be the following:

  • icmp
  • icmpv6
  • igmp
  • ipv6tcp
  • rawip
  • sctp
  • tcp
  • udp

For example, to display UDP output only for connected and unconnected sockets, you would type:

$ netstat -aP udp
UDP: IPv4

   Local                 Remote            Send                 Recv
   Address               Address  State    Buf    TxOverflows   Buf    RxOverflows
-----------------------  -------  -------  -----  -----------   -----  -----------
      *.*                         Unbound  57344            0   57344            0
      *.*                         Unbound  57344            0   57344            0
      *.*                         Unbound  57344            0   57344            0
      *.*                         Unbound  57344            0   57344            0
...
      *.bootpc                    Idle     57344            0   57344            0
      *.dhcpv6-client             Idle     57344            0   57344            0
ip-10-134-63-206.bootpc           Idle     57344            0   57344            0
      *.sunrpc                    Idle     57344            0   57344            0
      *.*                         Unbound  57344            0   57344            0
      *.59730                     Idle     57344            0   57344            0
      *.sunrpc                    Idle     57344            0   57344            0
      *.*                         Unbound  57344            0   57344            0
      *.47158                     Idle     57344            0   57344            0
      *.*                         Unbound  57344            0   57344            0
      *.631                       Idle     57344            0   57344            0
      *.ntp                       Idle     57344            0   57344            0
      *.ntp                       Idle     57344            0   57344            0
localhost.ntp                     Idle     57344            0   57344            0
ip-10-134-63-206.ntp              Idle     57344            0   57344            0

UDP: IPv6

   Local               Remote                Send                Recv
   Address             Address  State    If  Buf    TxOverflows  Buf    RxOverflows
---------------------  -------  -------  --  -----  -----------  -----  -----------
      *.*                       Unbound      57344            0  57344            0
      *.*                       Unbound      57344            0  57344            0
      *.*                       Unbound      57344            0  57344            0
      *.*                       Unbound      57344            0  57344            0
      *.dhcpv6-client           Idle         57344            0  57344            0
...
localhost.ntp                   Idle         57344            0  57344            0

The output also includes statistics for send and receive buffers as well as information about transmit and receive overflows.

The counter for transmit overflows increases whenever IP cannot send the outgoing packet to the MAC layer due to unavailable space.

The counter for receive overflows increases whenever IP cannot send the incoming packet to the socket due to unavailable space. In such cases, the incoming packet is dropped.

Displaying Network Interface Status

The –i option displays the state of the network interfaces on the system and the number of packets passing through the interfaces.

This example displays the output pertaining only to the IPv4 traffic of net0:

$ netstat -i -I net0 -f inet
Name  Mtu  Net/Dest        Address        Ipkts  Ierrs Opkts  Oerrs Collis Queue 
net0  1500 abc.oracle.com  abc.oracle.com 231001 0     55856  0     0      0     

The input packet count (Ipkts) can increase each time a client tries to boot. A count increase of input packets but a steady count of output packets (Opkts) indicate that the system does not know how to respond to request packets. This situation might be due to an incorrect address in the hosts or ethers database.

If the input packet count is steady over time, then the system does not see the packets at all and might indicate some type of failure, including hardware.

Displaying Information About UDP Sockets That Use Network Data Path Bypass

The –k option displays information about specific UDP sockets that use the network data path bypass capability. Without this option, these sockets are displayed like any other sockets.

For this feature's description, see Using the Network Data Path Bypass Capability for UDP.

$ netstat -aukP

UDP: IPv4
   Local Address  Remote Address  State  If  Ipkts   Opkts   Dpkts  User  Pid  Command
---------------   ------------   ------ ----- -----  ------  -----  ----- ---- --------
   *.64768                        Idle  net4   0     12556800  0    root 111530 udp_cli

Displaying the Status of Known Routes

The –r option displays the routing table, which lists all network routes that are known to a system.

$ netstat -r
Routing Table: IPv4
Destination           Gateway           Flags  Ref   Use   Interface
-------------------- -------------------- ----- ----- ------ ---------
host15               myhost               U         1  31059  net0
203.0.113.14         myhost               U         1      0  net0
default              distantrouter        UG        1      2  net0
localhost            localhost            UH        42019361  lo0

Routing Table: IPv6
Destination/Mask      Gateway                            Flags Ref  Use   If
--------------------- ---------------------------          ----- --- ------ -----
2002:0a00:3010:2::/64 2002:0a00:3010:2:1b2b:3c4c:5e6e:abcd U     1   0      net0:1
fe80::/10             fe80::1a2b:3c4d:5e6f:12a2            U     1   23     net0
ff00::/8              fe80::1a2b:3c4d:5e6f:12a2            U     1   0      net0
default               fe80::1a2b:3c4d:5e6f:12a2            UG    1   0      net0
localhost             localhost                            UH    9   21832  lo0

The following table describes the information that is displayed by the netstat –r command.

Parameter
Description
Destination
Destination/Mask
Specifies the system that is the destination endpoint of the route. Note that the IPv6 routing table shows the prefix for a 6to4 tunnel endpoint (2002:0a00:3010:2::/64) as the route destination endpoint.
Gateway
Specifies the gateway to use for forwarding packets.
Flags
Indicates the current status of the route. The U flag indicates that the route is up. The G flag indicates that the route is to a gateway.
Use
Shows the number of packets sent.
Interface
Indicates the particular interface on the local host that is the source endpoint of the transmission.

Using the ping Command

The ping command helps to determine whether your system can exchange IP packets with a remote host. Through the command, the ICMP protocol sends a datagram to the target host and waits for a response. A response indicates available connectivity with that host.

When you issue the command, attempts to probe the remote host continue automatically for 20 seconds before the operation times out. You can specify a longer timeout period, which must be in seconds.

$ /usr/sbin/ping host [timeout]

Sending an interrupt character immediately stops the operation.

For reference, see the ping(8) man page.

Command Modifications for IPv6 Support

The ping command can use both IPv4 and IPv6 protocols to probe target systems. Protocol selection depends on the addresses that are returned by the name server for the specific target system. If the name server returns an IPv6 address for the target system, the command uses the IPv6 protocol. If the server returns only an IPv4 address, the command uses the IPv4 protocol.

To use a specific protocol, use the –A option and specify either inet or inet6 for IPv4 and IPv6 protocols, respectively.

$ ping host -A inet|inet6

Investigating Dropped Packets

Packet loss can degrade network performance because additional time is spent retransmitting dropped data. The –s option reports packet loss between hosts.

$ ping -s host1.domain8
PING host1.domain8 : 56 data bytes
64 bytes from host1.example.COM (198.51.100.64): icmp_seq=0. time=1.67 ms
64 bytes from host1.example.COM (198.51.100.64): icmp_seq=1. time=1.02 ms
64 bytes from host1.example.COM (198.51.100.64): icmp_seq=2. time=0.986 ms
64 bytes from host1.example.COM (198.51.100.64): icmp_seq=3. time=0.921 ms
64 bytes from host1.example.COM (198.51.100.64): icmp_seq=4. time=1.16 ms
64 bytes from host1.example.COM (198.51.100.64): icmp_seq=5. time=1.00 ms
64 bytes from host1.example.COM (198.51.100.64): icmp_seq=5. time=1.980 ms

^C

----host1.domain8  PING Statistics----
7 packets transmitted, 7 packets received, 0% packet loss
round-trip (ms)  min/avg/max/stddev = 0.921/1.11/1.67/0.26

By adding the –W option, you can specify a period in seconds in which the command waits for a response from the target host. By default, the wait period is 10 seconds.

Alternatively, you can use the –w option instead, where you specify a deadline, in seconds, before the ping operation ends, regardless of the number of packets sent or received.


Note -  The –W or –w options must be used with either the –s or –I option.

If you use both –W or –w together in a single command, the –W option is ignored.


In the following example, the system waits 5 additional seconds for a response from the target after receiving no reply at the end of 10 ping requests:

$ ping -W 5 -s target 10

In the following example, the ping operation ends if one of the following conditions is met:

  • The 10 second deadline elapses.

  • The system receives 5 ping responses from the target

  • System receives an error notification from the network

$ ping -w 10 -s target 5

Using the traceroute Command

The traceroute command traces the route that an IP packet follows to a remote system. Use this command to uncover any routing misconfiguration and routing path failures that make a remote host unreachable.

The command also displays the round trip time for each gateway along the path to the target system. This information can be useful for analyzing where network traffic is slow between the two systems.

For reference, see the traceroute(8) man page.

Command Modifications for IPv6 Support

You can use the traceroute command to trace both IPv4 and IPv6 routes to a specific system. Just like with the ping command, protocol selection depends on the addresses that are returned by the name server for the target system. If the name server returns an IPv6 address for the target system, the command uses the IPv6 protocol. If the server returns only an IPv4 address, the command uses the IPv4 protocol.

To use a specific protocol, use the –A option and specify either inet or inet6 for IPv4 and IPv6 protocols, respectively.

$ traceroute destination-hostname -A inet|inet6

Discovering the Route to a Remote Host

The following sample output shows the seven-hop path that a packet follows from the local system to a remote system called farhost.

$ traceroute farhost
traceroute to farhost (198.51.100.39/27), 30 hops max, 40 byte packets
1  frbldg7c-86 (198.51.100.1/27)  1.516 ms  1.283 ms  1.362 ms
2  bldg1a-001 (198.51.100.2/27)  2.277 ms  1.773 ms  2.186 ms
3  bldg4-bldg1 (198.51.100.66/27)  1.978 ms  1.986 ms  13.996 ms
4  bldg6-bldg4 (198.51.100.132/27)  2.655 ms  3.042 ms  2.344 ms
5  farhost (198.51.100.39/27)  3.430 ms  3.312 ms  3.451 ms

The output also shows the time that it takes for a packet to traverse each hop.

Tracing All Routes

To trace all routes, use the –a option. The following example displays all of the possible routes to a dual-stack host called v6host:

$ traceroute -a v6host
traceroute: Warning: Multiple interfaces found; using 2001:db8:4a3a:1:56:a0:a8 @ net0:2 
traceroute to v6rout86 (2001:db8:4a3b:5:102:a00:fe79:19b0),30 hops max, 60 byte packets 
1 v6-rout86 (2001:db8:4a3b:1:56:a00:fe1f:59a1) 35.534 ms 56.998 ms *
2 2001:db8::255:0:c0a8:717 32.659 ms 39.444 ms *
3  farhost (2001:db8:4a3b:2:103:a00:fe9a:ce7b)  401.518 ms  7.143 ms *
4  distant (2001:db8:4a3b:3:100:a00:fe7c:cf35)  113.034 ms  7.949 ms *
5  v6host (2001:db8:4a3b:5:102:a00:fe79:19b0)  66.111 ms *  36.965 ms *

traceroute to v6host (192..0.2.75),30 hops max,40 byte packets
1  v6-rout86 (198.51.100.1/27)  4.360 ms  3.452 ms  3.479 ms
2  flrmpj17u (198.51.100.131/27)  4.062 ms  3.848 ms  3.505 ms
3  farhost (203.0.113.23)  4.773 ms *  4.294 ms
4  distant (192..0.2.104)  5.128 ms  5.362 ms *
5  v6host  (192..0.2.85)  7.298 ms  5.444 ms *

Using the My Traceroute Utility

The My Traceroute (mtr) utility combines the functionality of the ping and traceroute commands into a single networking diagnostics tool. The utility sends exploratory packets to a specified system at regular intervals and also tracks network hops between the current system and a target system. On the screen, the utility displays timing information which is updated constantly as new packets are sent out and responses are returned.

To use the mtr utility on your Oracle Solaris system, you must first install the network/mtr IPS package. Note that the utility uses the same security model that the traceroute and ping commands use.

For reference, see the mtr(8) man page.

Using the snoop Command

The snoop command enables you to monitor network traffic by capturing network packets which are displayed or saved to a file.

In summary form, the displayed data pertains only to the highest-level protocol. For example, an NFS packet only displays NFS information. The underlying remote procedure call (RPC), UDP, IP, and Ethernet frame information is suppressed unless the verbose options are used.

When you issue the command, output is continuously generated until you send an interrupt character.

For reference, see the snoop(8) man page.

Command Modifications for IPv6 Support

By default, the snoop command displays both IPv4 and IPv6 packets. However, you can filter the displayed information to just IPv4 or IPv6 packets by specifying ip or ip6, respectively, with the command.

$ snoop ip6
fe80::a00:20ff:fecd:4374 -> ff02::1:ffe9:2d27 ICMPv6 Neighbor solicitation
fe80::a00:20ff:fee9:2d27 -> fe80::a00:20ff:fecd:4375 ICMPv6 Neighbor
solicitation
fe80::a00:20ff:fee9:2d27 -> fe80::a00:20ff:fecd:4375 ICMPv6 Neighbor
solicitation
fe80::a00:20ff:febb:e09 -> ff02::9      RIPng R (11 destinations)
fe80::a00:20ff:fee9:2d27 -> ff02::1:ffcd:4375 ICMPv6 Neighbor solicitation

Displaying Packets From All Interfaces

The following example shows the basic output for a dual-stack host.

$ snoop
Using device /dev/net (promiscuous mode)
router5.local.com -> router5.local.com ARP R 203.0.113.13, router5.local.com is
0:10:7b:31:37:80
router5.local.com -> BROADCAST     TFTP Read "network-confg" (octet)
myhost -> DNSserver.local.com      DNS C 192..0.2.10.in-addr.arpa. Internet PTR ?
DNSserver.local.com  foohost       DNS R 192..0.2.10.in-addr.arpa. Internet PTR
niserve2.
.
.
fe80::a00:20ff:febb:e09 -> ff02::9 RIPng R (5 destinations)

In the output, the captured packets show a DNS query and response, Address Resolution Protocol (ARP) packets from the local router, and advertisements of the IPv6 link-local address to the in.ripngd daemon.

Monitoring Packets on IP Layer Devices

Use the –I option to check network traffic on IP layer devices. These devices provide access to all of the packets with addresses that are associated with the network interface. The addresses include both IPv4 and IPv6 addresses. The addresses can be local or hosted on non-loopback interfaces or logical interfaces. The traffic can be loopback IP traffic, packets from remote machines, packets that are being sent from the system, or all forwarded traffic. In summary, you can monitor all traffic that is destined for the system. Type the following:

$ snoop -I interface [-V|-v]

The –V and –v options generate verbose output.

interface has a wider scope than just physical IP interfaces. For example, an IPMP group is configured on an IPMP interface. Thus, this option enables you to monitor traffic from an IPMP group. You no longer need to monitor traffic separately on each underlying interface. Instead, the output is consolidated into a single output stream from the IPMP interface. For example:

$ snoop -I ipmp0 [-V|-v]

The following examples show different ways of monitoring network traffic on the IP layer.

Example 3  Observing Traffic on the Loopback Interface

To obtain general and summary loopback traffic information, type the following:

$ snoop -I lo0
Using device ipnet/lo0 (promiscuous mode)
localhost -> localhost    ICMP Echo request (ID: 5550 Sequence number: 0)
localhost -> localhost    ICMP Echo reply (ID: 5550 Sequence number: 0)

To generate verbose output, use the –v option:

$ snoop -v -I lo0
Using device ipnet/lo0 (promiscuous mode)
IPNET:  ----- IPNET Header -----
IPNET:
IPNET:  Packet 1 arrived at 10:40:33.68506
IPNET:  Packet size = 108 bytes
IPNET:  dli_version = 1
IPNET:  dli_type = 4
IPNET:  dli_srczone = 0
IPNET:  dli_dstzone = 0
IPNET:
IP:   ----- IP Header -----
IP:
IP:   Version = 4
IP:   Header length = 20 bytes
.
elided output

The output for IP layer information includes the ipnet header, which precedes the packets that are being observed. This header informs you of the source and destination of the traffic. In the sample output, the 0 ID of both dli_srczone and dli_dstzone indicates that the traffic is being generated from the global zone.

Example 4  Observing Packet Flow for Interfaces in Local Zones

An administrator for the global zone can monitor traffic between zones, as well as within a zone. An administrator of a non-global zone can observe traffic that is sent and received by that zone.

This example shows zone traffic in the system. Both simple and verbose command syntax are used to display in different formats all packets that are associated with net0.

The example assumes the existence of two non-global zones: sandbox and toybox.

$ snoop -I net0
Using device ipnet/net0 (promiscuous mode)
toybox -> sandbox TCP D=22 S=62117 Syn Seq=195630514 Len=0 Win=49152 Options=<mss
sandbox -> toybox TCP D=62117 S=22 Syn Ack=195630515 Seq=195794440 Len=0 Win=49152
toybox -> sandbox TCP D=22 S=62117 Ack=195794441 Seq=195630515 Len=0 Win=49152
sandbox -> toybox TCP D=62117 S=22 Push Ack=195630515 Seq=195794441 Len=20 Win=491
$ snoop -I net0 -v port 22
IPNET:  ----- IPNET Header -----
IPNET:
IPNET:  Packet 5 arrived at 15:16:50.85262
IPNET:  Packet size = 64 bytes
IPNET:  dli_version = 1
IPNET:  dli_type = 0
IPNET:  dli_srczone = 0
IPNET:  dli_dstzone = 1
IPNET:
IP:   ----- IP Header -----
IP:
IP:   Version = 4
IP:   Header length = 20 bytes
IP:   Type of service = 0x00
IP:         xxx. .... = 0 (precedence)
IP:         ...0 .... = normal delay
IP:         .... 0... = normal throughput
IP:         .... .0.. = normal reliability
IP:         .... ..0. = not ECN capable transport
IP:         .... ...0 = no ECN congestion experienced
IP:   Total length = 40 bytes
IP:   Identification = 22629
IP:   Flags = 0x4
IP:         .1.. .... = do not fragment
IP:         ..0. .... = last fragment
IP:   Fragment offset = 0 bytes
IP:   Time to live = 64 seconds/hops
IP:   Protocol = 6 (TCP)
IP:   Header checksum = 0000
IP:   Source address = 198.51.100.1, 198.51.100.1
IP:   Destination address = 198.51.100.3, 198.51.100.3
IP:   No options
IP:
TCP:  ----- TCP Header -----
TCP:
TCP:  Source port = 46919
TCP:  Destination port = 22
TCP:  Sequence number = 3295338550
TCP:  Acknowledgement number = 3295417957
TCP:  Data offset = 20 bytes
TCP:  Flags = 0x10
TCP:        0... .... = No ECN congestion window reduced
TCP:        .0.. .... = No ECN echo
TCP:        ..0. .... = No urgent pointer
TCP:        ...1 .... = Acknowledgement
TCP         .... 0... = No push
TCP         .... .0.. = No reset
TCP:        .... ..0. = No Syn
TCP:        .... ...0 = No Fin
TCP:  Window = 49152
TCP:  Checksum = 0x0014
TCP:  Urgent pointer = 0
TCP:  No options
TCP:

The ipnet header in the output indicates that the packet is coming from the global zone (ID 0 of dli_srczone) to sandbox (ID 1 of dli_dstzone).


Note -  Using the snoop command with zone names is not supported.

Capturing snoop Output to a File

To direct snoop output to a file and analyze the file's contents, you use two command options separately. First you generate output to a file (snoop –o). Then you read the file's data (snoop –i).

$ snoop -o /tmp/cap
Using device /dev/eri (promiscuous mode)
30 snoop: 30 packets captured
.
.
^C

Note -  The snoop command creates a noticeable network load on the host system, which can distort the results. To see the actual results, run the snoop command from a third system.

Then you view the contents of the generated file.

$ snoop -i /tmp/cap
1   0.00000 fe80::a00:20ff:fee9:2d27 -> fe80::a00:20ff:fecd:4375
ICMPv6 Neighbor advertisement
...
10  0.91493    203.0.113.40 -> (broadcast)  ARP C Who is 203.0.113.40, 203.0.113.40 ?
34  0.43690 nearserver.example.com  -> 224.0.1.1  IP  D=224.0.1.1 S=203.0.113.40 LEN=28,
ID=47453, TO =0x0, TTL=1
35  0.00034  203.0.113.40 -> 224.0.1.1    IP  D=224.0.1.1 S=203.0.113.40 LEN=28, ID=57376,
TOS=0x0, TTL=47

Using the ipstat and tcpstat Commands

The ipstat and tcpstat commands monitor, gather, and report statistics about IP traffic. Each command provides different types of information.

  • ipstat gathers and reports statistics about IP traffic, IP packet drops, and IP-related Management Information Base (MIB) events on a server. The reported information is based on the selected output mode and the sort order that you specify in the command syntax. The command enables you to monitor network traffic and events at the IP layer. The data is aggregated on source, destination, higher-layer protocol, and interface. Use the ipstat command when you want to monitor the amount of traffic between servers.

  • tcpstat gathers and reports statistics about TCP and UDP traffic, error events, and MIB events on a server based. The reported information is based on the selected output mode and the sort order that you specify in the command syntax. The command enables you to monitor network traffic and events at the transport layer, specifically for TCP and UDP. In addition to source and destination IP addresses, you can observe the source and destination TCP or UDP ports, the PID of the process that is sending or receiving the traffic, and the name of the zone in which that process is running.

    The two commands help you in the following ways:

  • Identify the largest sources of TCP and UDP traffic on a server.

  • Examine the traffic that is being generated by a particular process.

  • Examine the traffic that is being generated from a particular zone.

  • Determine which process is bound to a local port.

  • Monitor packet drops and IP-related MIB events.


Note -  The previous list is not exhaustive. You can use these commands in several other ways. For reference, see the ipstat(8) and tcpstat(8) man pages.

Examples in the Use of ipstat

The following examples show different ways to display IP statistics.

Example 5  Displaying Statistics for All IP Communication

This example shows output of the ipstat –c command syntax. The option prints newer reports without overwriting previous reports. The number 3 indicates the interval in seconds for displaying data.

$ ipstat -c 3
SOURCE                     DEST                       PROTO    INT        BYTES
zucchini                   antares                    TCP      net0       72.0
zucchini                   antares                    SCTP     net0       64.0
antares                    zucchini                   SCTP     net0       56.0
amadeus.foo.example.com    203.0.113.255              UDP      net0       40.0
antares                    zucchini                   TCP      net0       40.0
zucchini                   antares                    UDP      net0       16.0
antares                    zucchini                   UDP      net0       16.0
Total: bytes in: 192.0  bytes out: 112.0 
Example 6  Observing the Five Most Active IP Traffic Flows

The –l n-lines option specifies the top n most active IP traffic.

$ ipstat -l 5
SOURCE                     DEST                       PROTO    IFNAME     RATE
charybdis.foo.example.com  achilles.exampl            UDP      net0        6.6K
eratosthenes.example.com   aeneas.example.c           TCP      tun0        6.1K
achilles.exampl            charybdis.foo.example.com  UDP      net0      964.0
aeneas.example.c           eratosthenes.example.com   TCP      tun0      563.0
odysseus.example.          255.255.255.255            UDP      net0       66.0
Total: bytes in: 12.6K bytes out:  2.2K
Example 7  Displaying a Time Stamp

This example reports the top IP traffic with a time stamp in standard date format (–d d).

To print the timestamp in seconds or UNIX time, use –d u. The interval count is set to 10 seconds.

$ ipstat -d d -c 10
Monday, March 26, 2012 08:34:07 PM EDT
SOURCE                     DEST                       PROTO    IFNAME     RATE
charybdis.foo.example.com  achilles.exampl            UDP      net0       15.1K
eratosthenes.example.com   aeneas.example.c           TCP      tun0       13.9K
achilles.exampl            charybdis.foo.example.com  UDP      net0        2.4K
aeneas.example.c           eratosthenes.example.com   TCP      tun0        1.5K
odysseus.example.          255.255.255.255            UDP      net0       66.0
cassiopeia.foo.example.com aeneas.example.c           TCP      tun0       29.0
aeneas.example.c           cassiopeia.foo.example.com TCP      tun0       20.0
Total: bytes in: 29.1K bytes out:  3.8K
Example 8  Reporting IP Drops in Packets

This example shows how to use the ipstat command to display the number of packets that are dropped for each source, destination, and protocol combination.

$ ipstat -e
SOURCE                     DEST                   PROTO IFNAME    RATE   EVENT
--------------------------------------------------------------------------------
fe80::214:4fff:fe40:d0c8   ff02::202              IP    net0        6.0  drop-in
systl                      ntp.mcast.net          IGMP  net0        1.0  drop-in
tes-01-11.company.us.com   all-systems.mcast.net  IGMP  net0        1.0  drop-in
Total: bytes in:  9.0  bytes out:  0.0

Examples in the Use of tcpstat

The following examples show different ways to display TCP and UDP statistics.

Example 9  Displaying Statistics for All TCP and UDP Communication

Compare the information here with the output in Example 5, Displaying Statistics for All IP Communication that also used the –c option.

$ tcpstat -c 3
ZONE         PID PROTO  SADDR             SPORT DADDR             DPORT   RATE
global    100680 UDP    antares           62763 agamemnon          1023   76.0
global    100680 UDP    antares             775 agamemnon          1023   38.0
global    100680 UDP    antares             776 agamemnon          1023   37.0
global    100680 UDP    agamemnon          1023 antares           62763   26.0
global    104289 UDP    zucchini          48655 antares            6767   16.0
global    104289 UDP    clytemnestra      51823 antares            6767   16.0
global    104289 UDP    antares            6767 zucchini          48655   16.0
global    104289 UDP    antares            6767 clytemnestra      51823   16.0
global    100680 UDP    agamemnon          1023 antares             776   13.0
global    100680 UDP    agamemnon          1023 antares             775   13.0
global    104288 TCP    zucchini          33547 antares            6868    8.0
global    104288 TCP    clytemnestra      49601 antares            6868    8.0
global    104288 TCP    antares            6868 zucchini          33547    8.0
global    104288 TCP    antares            6868 clytemnestra      49601    8.0
Total: bytes in: 101.0  bytes out: 200.0 
Example 10  Observing the Five Most Active TCP Traffic Flows

This example reports the five most active TCP traffic flows for a server.

$ tcpstat -l 5
ZONE            PID PROTO  SADDR            SPORT DADDR            DPORT   BYTES
global        28919 TCP    achilles.exampl  65398 aristotle.exampl   443   33.0
zone1          6940 TCP    ajax.example.com  6868 achilles.exampl  61318    8.0
zone1          6940 TCP    achilles.exampl  61318 ajax.example.com  6868    8.0
global         8350 TCP    ajax.example.com  6868 achilles.exampl  61318    8.0
global         8350 TCP    achilles.exampl  61318 ajax.example.com  6868    8.0
Total: bytes in: 16.0  bytes out: 49.0
Example 11  Displaying Timestamp Information

In this example, the tcpstat command displays the timestamp information for TCP network traffic on a server in standard date format.

$ tcpstat -d d -c 10
Saturday, March 31, 2012 07:48:05 AM EDT
ZONE            PID PROTO  SADDR            SPORT DADDR            DPORT   RATE
global         2372 TCP    penelope.example 58094 polyphemus.examp    80   37.0
zone1          6940 TCP    ajax.example.com  6868 achilles.exampl  61318    8.0
zone1          6940 TCP    achilles.exampl  61318 ajax.example.com  6868    8.0
global         8350 TCP    ajax.example.com  6868 achilles.exampl  61318    8.0
global         8350 TCP    achilles.exampl  61318 ajax.example.com  6868    8.0
Total: bytes in: 16.0  bytes out: 53.0 
Example 12  Reporting TCP and UDP Statistics by Packet Count

The –k option displays the number of packets instead of number of bytes that are exchanged.

$ tcpstat -k
ZONE         PID PROTO  SADDR             SPORT DADDR             DPORT   RATE
-------------------------------------------------------------------------------
global    100574 TCP    syst1                22 dhcp-system1     59198       2
global    100574 TCP    dhcp-system1      59198 systl               22       2
global    100531 UDP    10.5.238.52       46066 10.255.255.255     111       1
Total: packets in: 3 out: 2
Example 13  Reporting Traffic for TCP-Related Events Grouped by Flow

This example shows how to display the rate at which transport layer events occur, grouped by flow. The flows are displayed with numeric source and destination IP addresses instead of their respective host names. To display only a subset of the events, instead of specifying all, provide a comma separated list of event names after the –E option. For a complete list of possible event names, use the tcpstat –L command.

$ tcpstat -E all -T tcp -n -g
ZONE       PID PROTO SADDR          SPORT  DADDR          DPORT RATE            EVENT
-------------------------------------------------------------------------------------
global  100519 TCP   10.132.148.89  39443  10.134.71.92      22    5 tcpInInorderSegs
global  100519 TCP   10.132.148.89  39443  10.134.71.92      22    1     tcpInAckSegs
global  100519 TCP   10.132.148.89  39443  10.134.71.92      22    1     tcpRttUpdate
global  100519 TCP   10.134.71.92      22  10.132.148.8   39443    4        tcpOutAck
global  100519 TCP   10.134.71.92      22  10.132.148.89  39443    1   tcpOutDataSegs

Total packets: 12

Tracing and Logging IP Operations

This section describes additional methods for checking network operations.

Logging Actions of the IPv4 Routing Daemon

If you suspect a malfunction of the IPv4 routing daemon, routed, you can start a log that traces the daemon's activity. The log includes all of the packet transfers when you start the in.routed daemon.

Create a log file that traces the routing daemon's actions as follows:

$ /usr/sbin/in.routed /var/log-file-name

Caution  -  On a busy network, this command can generate almost continuous output.


The following example shows the beginning of the log that is created when you perform the Logging Actions of the IPv4 Routing Daemon procedure.

-- 2003/11/18 16:47:00.000000 --
Tracing actions started
RCVBUF=61440
Add interface lo0  #1   127.0.0.1      -->127.0.0.1/32
<UP|LOOPBACK|RUNNING|MULTICAST|IPv4> <PASSIVE>
Add interface net0 #2   203.0.113.112    -->203.0.113.0/25
<UP|BROADCAST|RUNNING|MULTICAST|IPv4>
turn on RIP
Add    203.0.113.0        -->203.0.113.112      metric=0  net0  <NET_SYN>
Add    203.0.113.85/25  -->203.0.113.112      metric=0  net0  <IF|NOPROP>

Tracing the Activities of the IPv6 Neighbor Discovery Daemon

If you suspect a malfunction of the IPv6 in.ndpd daemon, you can start a log that traces the daemon's activity. This trace is displayed on the standard output until it is terminated. This trace includes all packet transfers when you start the in.ndpd daemon.

Before you start the trace, disable the NDP service first. Then, after running the trace for a determined period, restart the service. Stop the trace anytime by sending an interrupt character. For example:

$ svcadm disable ndp
$ /usr/inet/in.ndpd -t
Nov 18 17:27:28 Sending solicitation to  ff02::2 (16 bytes) on net0
Nov 18 17:27:28         Source LLA: len 6 <08:00:20:b9:4c:54>
Nov 18 17:27:28 Received valid advert from fe80::a00:20ff:fee9:2d27 (88 bytes) on net0
Nov 18 17:27:28         Max hop limit: 0
Nov 18 17:27:28         Managed address configuration: Not set
Nov 18 17:27:28         Other configuration flag: Not set
Nov 18 17:27:28         Router lifetime: 1800
Nov 18 17:27:28         Reachable timer: 0
Nov 18 17:27:28         Reachable retrans timer: 0
Nov 18 17:27:28         Source LLA: len 6 <08:00:20:e9:2d:27>
Nov 18 17:27:28         Prefix: 2001:08db:3c4d:1::/64
Nov 18 17:27:28                 On link flag:Set
Nov 18 17:27:28                 Auto addrconf flag:Set
Nov 18 17:27:28                 Valid time: 2592000
Nov 18 17:27:28                 Preferred time: 604800
Nov 18 17:27:28         Prefix: 2002:0a00:3010:2::/64
Nov 18 17:27:28                 On link flag:Set
Nov 18 17:27:28                 Auto addrconf flag:Set
Nov 18 17:27:28                 Valid time: 2592000
Nov 18 17:27:28                 Preferred time: 604800

^C

$ svcadm enable ndp