Troubleshooting Network Administration Issues in Oracle^® Solaris 11.3

Observing Network Configuration and Traffic Usage

The following information demonstrates how to use the observability tools that are described in Figure 2, Table 2, Networking Features Grouped by Layer the Oracle Solaris Network Protocol Stack..

The following figure illustrates a common hypothetical network configuration consisting of some of the networking features that are configured at the various layers of the Oracle Solaris network protocol stack. The figure, which is provided as an example only, is followed by a set of example commands that show how you can use Oracle Solaris tools in practice to observe and gather statistics about the various components that are involved.

Figure 1  Network Configuration Within the Oracle Solaris Network Protocol Stack

This figure depicts the following configuration, which illustrates how you can combine some Oracle Solaris features at the various layers of the network protocol stack:

• At the physical layer of the network stack, three physical NICs, nxgbe0, ixgbe0, and ixgbe1, are present in the system and appear as physical datalink instances, net2, net1, and net0, respectively.

• These physical NICs are then grouped together into a link aggregation called aggr0.

• The link aggregation datalink is then configured directly with an IP address (aggr0/v4) and is also used simultaneously as the uplink of a virtual switch, called tenant/hr, which is configured as an elastic virtual switch. The virtual switch has two virtual ports, vport0 and vport1.

• The Oracle Solaris zone (zone-A) has a VNIC called zone-A/net0, which is connected to one of the virtual ports. Within the zone itself, the VNIC appears as a datalink (net0), which is configured with an IP address (net0/v4).

• A flow for HTTP traffic is also created on top of the aggregation (aggr0).

The following examples further describe how you can obtain configuration information about these components and how to observe network traffic usage for various networking features by using the tools that are provided in Oracle Solaris.

Tools for Observing Network Configuration and Traffic Usage

You configure and administer datalinks by using the dladm command. You use the dlstat command to obtain statistics on network traffic usage for datalinks. For example, you would display inbound and outbound traffic statistics per link by using one of the following commands:

# dlstat link

# dlstat show-link link

Display inbound and outbound traffic statistics per physical network device as follows:

# dlstat show-phys link

Display inbound and outbound traffic statistics per port and per link aggregation as follows:

# dlstat show-aggr link

For more information, see the dlstat(1M) man page.

You configure and administer flows by using the flowadm command. You use the flowstat command to obtain statistics on network traffic usage for flows. As shown in Network Configuration Within the Oracle Solaris Network Protocol Stack, depending on the attributes that you specify, you can use flows to observe network traffic usage at different layers of the network stack.

# flowstat

For more information, see the flowstat(1M) man page.

The following examples show how to display network configuration information and observe network traffic statistics, per feature and per layer of the Oracle Solaris network protocol stack. For more information about monitoring network traffic usage, see Chapter 8, Monitoring Network Traffic and Resource Usage in Managing Network Virtualization and Network Resources in Oracle Solaris 11.3.

Observing Network Configuration and Traffic Usage at the Hardware Layer

Troubleshooting network configuration and performance issues at the hardware layer of the network protocol stack might involve observing the following:

• How many rings there are per physical NIC and how many packets are being transmitted through those rings.

• How many packet drops are occurring:

  • Per physical NIC

  • Per physical ring

• NIC-specific counters that might be useful.

• Number of rings and number of descriptors that are configured per physical NIC.

For physical devices, you can use the dladm show-phys and dlstat show-phys commands to observe network traffic usage. These two commands display different output, depending on the type of information that you want to obtain.

For example, use the dladm show-phys command without any options to display the physical device and the attributes of all of the physical links on a system:

# dladm show-phys
LINK MEDIA    STATE   SPEED DUPLEX   DEVICE
net1 Ethernet unknown 0     unknown  bge0
net0 Ethernet up      1000  full     nge0

For more details and examples, see Chapter 2, Administering Datalink Configuration in Oracle Solaris in Configuring and Managing Network Components in Oracle Solaris 11.3 and the dladm(1M) man page.

The dlstat show-phys command displays statistics about the packets and bytes that are transmitted and received per physical device. This subcommand operates on the hardware rings, which are at the device layer of the network stack.

The following example displays statistics for all of the physical links on a system. The output displays both incoming and outgoing traffic statistics for each link on a system. The number of packets and byte sizes per packet are also displayed:

# dlstat show-phys
LINK    IPKTS   RBYTES    OPKTS   OBYTES
net1        0        0        0        0
net0    1.95M  137.83M   37.95K    3.39M

You can use the –r option to display receive-side statistics on each of the hardware rings for a datalink device. The output of this command Includes the bytes and packets that are received and the hardware and software drops, etc., for the datalink device. This example shows that net4 has eight rings, which are identified under the INDEX field in the following output:

# dlstat show-phys -r net4
LINK TYPE INDEX IPKTS RBYTES
net4 rx   0     701   42.06K
net4 rx   1     0     0
net4 rx   2     0  0
net4 rx   3     0  0
net4 rx   4     0  0
net4 rx   5     0  0
net4 rx   6     0  0
net4 rx   7     0  0

To obtain similar information for transmitted traffic, use the –t option.

The following example displays the number of inbound dropped packets per physical link:

dlstat show-phys -o idrops
IDROPS   
0
871.14K

The –o field[,...] option is used to specify a case-insensitive, comma-separated list of output fields to display.

In the following example, the number of inbound and outbound dropped packets and bytes per physical link are displayed:

# dlstat show-phys -o idrops,idropbytes,odrops,odropbytes
IDROPS    IDROPBYTES ODROPS ODROPBYTES
0         0          0      0
871.14K   0          0      0

It is recommended that you specify both the idrops and idropbytes options with the dlstat show-phys –o command. Note that one of these values can be zero, depending on the system's hardware capabilities, as shown in the previous output where the IDROPS field is non-zero, while the IDROBYTES field is zero.

For driver configuration, you can manage property values for specific drivers through the driver.conf file and also through dladm properties. Driver configuration files enable you to provide device property values that override the default values that are provided by the devices themselves. For more information about managing information through the driver.conf file, see the driver.conf(4) man page.

Observing Network Configuration and Traffic Usage at the Datalink Layer

Several networking features are configured at the datalink layer (L2) of the network protocol stack. These features include both physical and virtual datalinks. Certain commands that you use to observe network traffic usage at this layer of the stack are generic and can be used for any type of configured datalink. Other subcommands are specific to the feature itself and therefore can be used to display additional information about that feature's configuration..

The commands that you use at this layer of the stack also depend on the type of information that you want to observe. For example, at the datalink layer of the stack, you might want to display fan-out statistics or per-link statistics. You would use different commands to obtain each type of information.

For basic information about datalinks, use the dladm show-link. This command displays link configuration information, either for all of the datalinks on the system or for a specified datalink, as shown in the following output:

# dladm show-link
LINK CLASS MTU STATE   OVER
net1 phys 1500 unknown --
net0 phys 1500 up      --

The previous output shows that this system has two datalinks, which are directly associated with their corresponding physical NICs. No special datalinks exist on the system, for example, an aggregation or a VNIC. These types of L2 entities are configured over the physical datalinks under the phys class.

You use the dlstat show-link command to observe network traffic usage at the datalink layer. The show-link subcommand operates at the datalink layer of the network protocol stack and provides statistics that refer to the lanes that are configured over the physical link.

The following output shows inbound and outbound traffic statistics per link:

# dlstat show-link
LINK   IPKTS  RBYTES  OPKTS  OBYTES
net1   0      0         0         0
net0   1.96M  137.97M   38.40K    3.29M

In the following example, receive-side traffic statistics for the net4 device are reported. The statistics for INTRS and POLLS counters are also displayed. These statistics report how many packets were received in the interrupt context versus polling mode. The IDROPS counter indicates how many packets were dropped at the datalink layer of the network stack.

# dlstat show-link -r net4
LINK TYPE   ID    INDEX    IPKTS  RBYTES    INTRS    POLLS   IDROPS
net4 rx    local  --        0      0        0        0       0
net4 rx    other  --        0      0        0        0       0
net4 rx    hw     0     7.46M  1.06G    5.62M    1.84M       0
net4 rx    hw     1        0      0         0        0       0
net4 rx    hw     2        0      0         0        0       0
net4 rx    hw     3        0      0         0        0       0
net4 rx    hw     4        2    196         2        0       0
net4 rx    hw     5        0      0         0        0       0
net4 rx    hw     6        0      0         0        0       0
net4 rx    hw     7        0      0         0        0       0

In the previous output, only statistics for the named link, physical device (for the show-phys subcommand), or aggregation (for the show-aggr subcommand) are displayed. If link is not specified in the command, then statistics for all of the links, devices, and aggregations are displayed in the output.

In this example, the information that is displayed under the ID field is interpreted as follows:

• local –. Denotes corresponding loopback traffic on layer 2 (L2) of the network stack.

• other – Includes broadcast and multicast traffic.

  Over the lifetime of a datalink, the hardware resources that are associated with a datalink might vary, depending on resource utilization, link configuration, or assignment of physical NICs to link aggregations. The rx entries that are listed in the output of the show-link –r command correspond to the hardware resource that is currently assigned to the link. The output for the other row includes traffic for hardware resources that are no longer assigned to the datalink.

• hw – Denotes a hardware lane.

• sw – Denotes a software lane (as shown in the following example).

The distinction between hardware and software lanes is based on a NIC's ability to support ring allocation. On hardware lanes, rings are dedicated to the packets that use those lanes. In contrast, rings on software lanes are shared among datalinks.

The following output reports statistics for outbound packets on the rings that are used by net1:

# dlstat show-link -t net1
LINK  TYPE      ID  INDEX    OPKTS   OBYTES   ODROPS
net4    tx   local     --        0        0        0  
net4    tx   other     --        0        0        0  
net4    tx      hw      0      372   15.67K        0  
net4    tx      hw      1        1       98        0
net4    tx      hw      2        0        0        0 
net4    tx      hw      3        0        0        0
net4    tx      hw      4        0        0        0
net4    tx      hw      5        0        0        0
net4    tx      hw      6        1        98       0
net4    tx      hw      7        0        0        0

Observing Network Configuration and Traffic Usage for Aggregations

Aggregations are also configured at the datalink layer (L2) of the network protocol stack. Depending on the type of information that you want to obtain, for example, the overall distribution of traffic between the physical NIC or aggregation statistics, use the following commands:

dladm show-aggr

Displays aggregation configuration (the default), LACP information or DLMP probe-based failure and recovery detection status, either for all of the aggregations or for a specified aggregation.

Specify the –x option to display detailed per-aggregation information for an aggregation.

dlstat show-aggr

Displays per-port statistics about the packets and bytes that are transmitted and received for an aggregation..

dlstat show-link

Displays statistics about the packets and bytes that are transmitted and received for an aggregation (the aggregation being a datalink).

One difference between the dlstat show-aggr and dlstat show-link commands is that the dlstat show-aggr command displays per-port statistics, while the dlstat show-link command provides per-link statistics. Another important difference between these two commands is that the dlstat show-aggr command displays the overall statistics for the entire aggregation. Whereas, the dlstat show-link command displays only the statistics for the primary client of the aggregation, for example IP.

Thus, if you create VNICs on top of an aggregation, the dlstat show-aggr command would report the total number of packets across all of the VNICs, plus the primary client (IP). This output is similar to the show-phys subcommand compared to the show-link subcommand, where show-phys displays the total traffic usage, while show-link displays only the traffic usage for the primary datalink.

The following examples demonstrate how you can observe network traffic usage for aggregations. For more information about administering aggregations, see Chapter 2, Administering Datalink Configuration in Oracle Solaris in Configuring and Managing Network Components in Oracle Solaris 11.3.

Example 1  Displaying Aggregation Configuration Information

The following example output reports the status of the existing aggregations that are configured on a system.

# dladm show-aggr -x
LINK  PORT SPEED    DUPLEX  STATE  ADDRESS          PORTSTATE
aggr1 --   1000Mb   full    up     0:14:4f:29:d1:9d --
net1       1000Mb   full    up     0:14:4f:29:d1:9d attached
net3       0Mb      unknown down   0:14:4f:29:d1:9f standby

Example 2  Displaying Per-Port Statistics for Aggregations

The following example output for the dlstat show-aggr command reports per-port statistics for an aggregation. Both the packets and bytes that are transmitted and received for the aggregation are displayed.

# dlstat show-aggr
LINK            PORT    IPKTS      RBYTES    OPKTS   OBYTES
aggr1             --      99 1      2.18K     23       966
aggr1             net4    25        1.50K      8       336
aggr1             net5    74        10.68K    15       630

Example 3  Displaying Per-Link Statistics for Aggregations

The following example output for the dlstat show-link command reports per-link statistics for an aggregation. Both the packets and bytes that are transmitted and received for the aggregation are displayed. The difference between this example and the previous example is that the show-aggr subcommand reports per-port statistics, while the show-link subcommand reports per-link statistics.

# dlstat show-link
LINK    IPKTS     RBYTES    OPKTS   OBYTES
net5        0          0        0        0
net2        0          0    5.60K    1.49M
net4        0          0        0        0
net6    4.43K      1.32M    6.39K    1.56M
net1    4.43K      1.32M    6.39K    1.56M
net0  387.10K     99.42M   59.43K    7.67M
net3        0          0    5.61K    1.50M
aggr1     150     18.65K        30   1.26K

Example 4  Displaying Receive-Side Traffic Statistics for an Aggregation's Hardware Rings

The following example reports receive-side statistics for each of an aggregation's (aggr1) hardware rings.

# dlstat show-phys -r aggr1
LINK TYPE INDEX   IPKTS  RBYTES
aggr1   rx     0     723  43.38K
aggr1   rx     1       0       0
aggr1   rx     2       0       0
aggr1   rx     3       0       0
aggr1   rx     4       0       0
aggr1   rx     5       0       0
aggr1   rx     6       0       0
aggr1   rx     7       0       0
aggr1   rx     8  22.20K   1.33M
aggr1   rx     9     692  63.66K
aggr1   rx    10       0       0
aggr1   rx    11       0       0
aggr1   rx    12       0       0
aggr1   rx    13  12.99K   4.44M
aggr1   rx    14       0       0
aggr1   rx    15  10.39K 623.34K

Example 5  Displaying Receive-Side Traffic Statistics for an Aggregation's Hardware Lanes

The following example reports receive-side traffic statistics for each of an aggregation's (aggr1) hardware lanes. Note that the statistics reported pertain to traffic for the primary client of the aggregation only, for example, the IP traffic on top of the aggregation. The output does not include traffic statistics for other clients that are configured on top of the aggregation, for example VNICs. To display traffic usage for all of the clients of an aggregation, you would use the dlstat show-aggr and dlstat show-phys, as shown in Example 1, Displaying Aggregation Configuration Information and Example 4, Displaying Receive-Side Traffic Statistics for an Aggregation's Hardware Rings.

# dlstat show-link -r aggr1
LINK   TYPE    ID    INDEX    IPKTS   RBYTES    INTRS   POLLS  IDROPS
aggr1   rx   local     --       0       0        0        0      0
aggr1   rx   other     --       0       0        0        0      0
aggr1   rx      hw      0     721   43.26K     721        0      0
aggr1   rx      hw      1       0       0        0        0      0
aggr1   rx      hw      2       0       0        0        0      0
aggr1   rx      hw      3       0       0        0        0      0
aggr1   rx      hw      4       0       0        0        0      0
aggr1   rx      hw      5       0       0        0        0      0
aggr1   rx      hw      6       0       0        0        0      0
aggr1   rx      hw      7       0       0        0        0      0
aggr1   rx      hw      8  22.23K   1.33M   22.23K        0      1
aggr1   rx      hw      9     693  63.76K      693        0      0
aggr1   rx      hw     10       0       0        0        0      0
aggr1   rx      hw     11       0       0        0        0      0
aggr1   rx      hw     12       0       0        0        0      0
aggr1   rx      hw     13  13.00K   4.45M    13.00K       0      2
aggr1   rx      hw     14       0       0        0        0      0
aggr1   rx      hw     15  10.40K 624.06K   10.40K        0      0

Observing Network Configuration and Traffic Usage for an EVS Switch

You configure and manage the elastic virtual switch (EVS) feature of Oracle Solaris at the datalink layer of the network protocol stack. Depending on the type of information that you want to display, use the following commands:

evsadm

Creates and manages EVS switches and their resources: IP networks (IPnets) and virtual ports (VPorts).

Use the show-evs subcommand to display information for all of the EVS switches that are managed by an EVS controller or for a specified EVS switch.

evsstat

Displays network traffic statistics for all of the VPorts for a large deployment or for all of the VPorts of a specified elastic virtual switch. The command also reports the statistics for any VNICs associated with the VPorts.

dladm show-vnic -c

Displays information about the VNICs that are connected to an EVS switch. The VPort and the EVS switch to which the VNIC is connected is determined by the EVS switch.

The following examples demonstrate how you can display configuration information for an elastic virtual switch and observe network traffic usage and other statistics for an EVS configuration. For more information about administering the EVS feature, see Chapter 6, Administering Elastic Virtual Switches in Managing Network Virtualization and Network Resources in Oracle Solaris 11.3.

Example 6  Displaying Information About an EVS Configuration

In the following example, the evsadm command is used to display basic information about an EVS configuration.

# evsadm
NAME          TENANT        STATUS  VNIC    IP          HOST
evs0          sys-global    busy    --      ipnet0      sysabc-02
sys-vport0    --            used    vnic0   203.0.113.2/24 sysabc-02

Example 7  Displaying Inbound and Outbound Traffic Usage for VPorts That Are Connected to an EVS Switch

In the following example the evsstat command is used to display incoming and outgoing network traffic statistics for the only VPort that is connected to an EVS switch.

# evsstat
VPORT       EVS   TENANT      IPKTS    RBYTES   OPKTS  OBYTES
sys-vport0  evs0  sys-tenant  101.88K  32.86M  40.16K  4.37M
sys-vport1  evs0  sys-tenant    4.50M   6.78G   1.38M  90.90M

Example 8  Displaying Information About VNICs That Are Connected to an EVS Switch

There are two types of VNICs: those that you create on top of an underlying link and those that are connected to an EVS switch. To obtain information about a VNIC that is connected to an EVS switch, use the dladm show-vnic command with the –c option, as shown in the following example.

# dladm show-vnic -c
LINK       TENANT        EVS     VPORT       OVER   MACADDRESS        IDS
vnic0      sys-global    evs0    sys-vport0  net    12:8:20:f2:46:22  VID:200

Observing Network Configuration and Traffic Usage for VNICs

VNICs are configured at the datalink layer (L2) of the network protocol stack. Use the following commands to display configuration information and observe network traffic usage for these L2 entities:

dladm show-vnic

Displays VNIC configuration information for all of the VNICs on a system, all of the VNICs on a link, or for a specified vnic-link.

dlstat

Displays statistics about the packets and bytes that are transmitted and received per VNIC.

The following examples demonstrate how you can observe network traffic usage for VNICs. For more information about administering VNICs, see Managing VNICs in Managing Network Virtualization and Network Resources in Oracle Solaris 11.3.

Example 9  Displaying VNIC Configuration Information

The following example displays VNIC configuration information for the one existing VNIC on a system (vnic0).

# dladm show-vnic
LINK        OVER        SPEED  MACADDRESS       MACADDRTYPE IDS
vnic0       net1        1000   2:8:20:f2:46:22  fixed       VID:200

Example 10  Displaying Network Traffic Statistics for VNICs

In the following example, the dlstat command is used to display statistics about the packets and bytes that are transmitted and received by a specific VNIC (vnic0).

# dlstat vnic0            
LINK    IPKTS   RBYTES    OPKTS    OBYTES           
vnic0   1.53M   158.18M   154.22K  32.84M

Observing Network Configuration and Traffic Usage at the IP Layer

You observe network traffic usage at the IP layer (L3) of the network protocol stack by using several different commands. Depending on the type of information that you want to display, use the following commands:

flowstat

Displays statistics for flows that you create for various IP addresses or subnets that are configured at the IP layer of the network protocol stack.

ipadm

Displays general configuration information for IP interfaces and addresses.

ipadm show-addr

Displays IP address information, either for the given address object (addrobj) or for all of the address objects that are configured on a specified interface, including those that are only in the persistent configuration.

ipadm show-if

Displays network interface configuration information, either for all of the network interfaces that are configured on the system, including those that are only in the persistent configuration, or for a specified interface.

ipstat

Displays statistics on IP traffic based on the selected output mode and sort order.

netstat

Displays the contents of certain network-related data structures in various formats.

The following examples demonstrate how to observe network traffic usage and gather statistics for networking features that are configured at the IP layer of the network protocol stack. For more information about administering IP configuration, see Monitoring IP Interfaces and Addresses in Configuring and Managing Network Components in Oracle Solaris 11.3. See also the ipstat(1M) and netstat(1M) man pages.

Example 11  Displaying General Information About IP Configuration

You use the ipadm command to display general information about IP configuration. The following example shows how to display information about all of the IP interfaces and addresses that are configured on a system.

# ipadm
NAME             CLASS/TYPE STATE        UNDER      ADDR
lo0              loopback   ok           --         --
   lo0/v4        static     ok           --         127.0.0.1/8
   lo0/v6        static     ok           --         ::1/128
net0             ip         ok           --         -- 
   net0/v4       static     ok           --         203.0.113.140/24

Example 12  Displaying Information About Configured IP Interfaces

You use the ipadm command with the show-if subcommand to display information about the configured IP interfaces on a system, as shown in the following example:

# ipadm show-if
IFNAME   CLASS      STATE   ACTIVE   OVER
lo0      loopback   ok       yes     ---
net0     ip         ok       yes     ---

Example 13  Displaying Information About Configured IP Address Objects

The following example shows how to use the ipadm command with the show-addr subcommand to display information about configured IP address objects on a system.

# ipadm show-addr
ADDROBJ      TYPE       STATE   ADDR
lo0/v4       static     ok      127.0.0.1/8

Example 14  Displaying Statistics About IP Traffic by Using the ipstat Command

The following example shows how to use the ipstat command to display statistics about IP traffic. The command provides options for reporting statistics about the IP traffic that matches a specified source or destination address, interface, and higher layer protocols.

# ipstat
SOURCE               DEST                  PROTO   INT   RATE
abc11example-02      dhcp-sys.example      TCP     net0  145.6
dns1.example.com     abc11example-02       UDP     net0  66.0
abc11example-02      dns1.example.com      UDP     net0  10.4
dhcp-sys.example     abc11example-02       TCP     net0  4.0
foo1.example.com     all-sys.mcast.net     ICMP    net0  3.2

For more information, see Observing Network Traffic With the ipstat and tcpstat Commands in Administering TCP/IP Networks, IPMP, and IP Tunnels in Oracle Solaris 11.3.

Example 15  Displaying Connected Sockets by Using the netstat Command

The netstat command displays network status and protocol statistics. You can display the status of TCP, SCTP, and UDP endpoints in table format. You can also display routing table and interface information by using this command. The various types of network data that is reported depends on the command-line option that you specify.

In the following example, the netstat command is used without any options to display a list of active sockets for each protocol.

# netstat
 TCP: IPv4
   Local Address        Remote Address          Swind  Send-Q  Rwind   Recv-Q  State
  ------------------     -----------------    -------  ------  ------- ------  --------
   localsys.local.port1  remotesys1             65535   0      128592   0      ESTABLISHED
   localsys.local.port2  localsys.local.port5   130880  0      139264   0      ESTABLISHED
   localsys.local.port3  localsys.local.port6   139060  0      130880   0      ESTABLISHED
   localsys.local.port4  remotesys2.remote.port2 65572  63     128480   0      ESTABLISHED

Use the netstat –P protocol command as follows to limit the display of statistics or state of just those sockets that are applicable to a specific protocol, as shown in the following example.

# netstat -P tcp
TCP: IPv4
Local Address Remote Address               Swind  Send-Q  Rwind  Recv-Q  State
------------ --------------------          ------ ------ ------- ------  -------- 
sys3.48962   foo.com.ldaps                 49232   0     128872  0       ESTABLISHED
sys3.ssh     dhcp1-203-0-113-210.foo.com   64292  63     128480  0       ESTABLISHED

You can specify protocol as any of the following: ip, ipv6, icmp, icmpv6, igmp, udp, tcp, rawip.

The following abbreviated example shows how to use the netstat –s command to display per-protocol statistics:

# netstat -s
RAWIP   rawipInDatagrams    =     2    rawipInErrors   =  0
        rawipInCksumErrs    =     0
        rawipOutDatagrams   =     2
        rawipOutErrors      =     0
    
UDP     udpInDatagrams      =   1023   udpInErrors    =   0
        udpOutDatagrams     =   1023
        udpOutErrors        =      0
    
TCP     tcpRtoAlgorithm     =      4    tcpRtoMin     =   200
        tcpRtoMax           =  60000
        tcpMaxConn          =     -1
        tcpActiveOpens      =    382
        tcpPassiveOpens     =     83
        tcpAttemptFails     =     81
        tcpEstabResets      =      1
        tcpCurrEstab        =      2
        tcpOutSegs          =   6598
        tcpOutDataSegs      =   5653
        tcpOutDataBytes     = 836393
        tcpRetransSegs      =     16 
. . .

Example 16  Displaying Statistics About Flows That Are Configured at the IP Layer of the Network Stack

As shown in the following example, you can create flows for various IP addresses or subnets of interest that are configured at the IP layer of the network protocol stack. You can then use the flowstat command to display statistics about these flows.

# flowadm add-flow -l net0 -a transport=tcp tcpflow1
# flowadm add-flow -l net4 -a transport=tcp tcpflow2

# flowstat
FLOW      IPKTS   RBYTES   IDROPS    OPKTS   OBYTES   ODROPS
tcpflow2      0       0        0        0        0        0
tcpflow1     53   5.62K        0       45    5.52K        0

Specify the –l option to display flow information for a specific datalink device:

# flowstat -l net0
FLOW      IPKTS   RBYTES  IDROPS   OPKTS  OBYTES ODROPS
tcpflow1    108   11.19K       0     86   10.45K      0

Example 17  Creating and Observing Flows for Specific IP Addresses

You can create flows for specific IP addresses by using the local_ip and remote_ip attributes with the flowadm add-flow command. You can then use the flowstat command display statistics for these flows, as shown in the following example.

# flowadm add-flow -l net0 -a local_ip=203.0.113.45 flow1
# flowadm add-flow -l net4 -a remote_ip=192.0.2.0/24 flow2
# flowstat 
FLOW      IPKTS     RBYTES   IDROPS    OPKTS  OBYTES   ODROPS
flow2   528.54K    787.39M        0  179.39K  11.85M        0
flow1   742.81K      1.10G        0        0       0        0

Observing Network Configuration and Traffic Usage at the Transport Layer

You observe network traffic for features that are configured and administered at the transport layer (L4) of the network protocol stack by using the following commands:

flowstat

Reports runtime statistics about user-defined flows. Use the flowadm show-flow command to determine the flow name to specify with the flowstat command.

You can use flows as an observability tool rather than just for bandwidth control, for example, to measure the amount of traffic that a specific service consumes.

netstat

Displays the contents of certain network-related data structures in various formats. With no arguments, the netstat command displays the connected sockets for PF_INET, PF_INET6, and PF_UNIX, unless modified by using the –f option.

tcpstat

Reports statistics on TCP and UDP traffic on a server based on the selected output mode and sort order that is specified in the command syntax.

The following examples demonstrate how you can observe network traffic usage and gather statistics for features that are configured at the transport layer of the network protocol stack.

For more information about administering TCP/IP networks with the netstat and tcpstat commands, see.Chapter 1, Administering TCP/IP Networks in Administering TCP/IP Networks, IPMP, and IP Tunnels in Oracle Solaris 11.3.

For more information about the flowstat command, see Displaying Network Traffic Statistics of Flows in Managing Network Virtualization and Network Resources in Oracle Solaris 11.3 and the flowstat(1M) man page.

Example 18  Displaying Runtime Statistics for Flows by Using the flowstat Command

The following example output shows a static display of traffic information for all of the configured flows that are on a system. The flowadm command is used to determine the name of the flow.

# flowadm
FLOW        LINK     PROTO LADDR   LPORT RADDR  RPORT DIR
tcpflow1    net1     tcp   --      --    --     --    bi
tcpflow0    net0     tcp   --      --    --     --    bi
udpflow0    net0     udp   --      --    --     --    bi

# flowstat
FLOW       IPKTS   RBYTES  IDROPS    OPKTS   OBYTES   ODROPS
tcpflow1      0        0       0        0        0        0
tcpflow0  1.39K  117.86K       0     2.16K  260.77K        0
udpflow0      5    1.43K       0         0        0        0

You can also use the flowstat command with the –l option to display statistics for all of the flows for a specified link or statistics for a specified flow, as shown in output of the following two examples.

# flowstat -l net0
FLOW      IPKTS   RBYTES   IDROPS    OPKTS    OBYTES   ODROPS
tcpflow0  1.51K  126.85K        0    2.43K   292.85K        0
udpflow0      9    2.80K        0        0         0        0

# flowstat -l net0 tcpflow0
FLOW      IPKTS    RBYTES   IDROPS     OPKTS   OBYTES  ODROPS
tcpflow0  1.66K   137.11K        0     2.69K  324.42K       0

Example 19  Displaying Information About Transport Layer Data Structures by Using the netstat Command

You can use the netstat command to display information about data structures at the transport layer (L4) of the network protocol stack, for example TCP or UDP. In the following example, the netstat –P transport-protocol command is used to display information about TCP.

# netstat -p tcp
TCP: IPv4 
  Local Address     Remote Address   Swind   Send-Q  Rwind  Recv-Q  State
  ----------------  ---------------  ------- ------ ------- ------ -----------
  localsys.ssh     remotesys1.port4   65380     63     128480   0    ESTABLISHED
  localsys.port1   remotesys2.ldaps   65535      0     128592   0    ESTABLISHED
  localsys.port2   localsys.port5    130880     0      139264   0    ESTABLISHED
  localsys.port3   localsys.port6    139060     0      130880   0    ESTABLISHED

Example 20  Displaying Statistics for TCP and UDP Traffic by Using the tcpstat Command

You use the tcpstat command to observe network traffic at the transport layer of the network protocol stack, specifically for TCP and UDP. In addition to the source and destination IP addresses, you can observe the source and destination TCP or UDP ports, the PID of the process that is sending or receiving the traffic, and the name of the global zone in which that process is running.

The following example shows the type of information that is reported when you use the tcpstat command with the –c option. The –c option specifies to print newer reports after previous reports, without overwriting previous reports.

# tcpstat -c 3
ZONE         PID PROTO  SADDR             SPORT DADDR             DPORT   BYTES
global    100680 UDP    antares           62763 agamemnon          1023   76.0
global    100680 UDP    antares             775 agamemnon          1023   38.0
global    100680 UDP    antares             776 agamemnon          1023   37.0
global    100680 UDP    agamemnon          1023 antares           62763   26.0
global    104289 UDP    zucchini          48655 antares            6767   16.0
global    104289 UDP    clytemnestra      51823 antares            6767   16.0
global    104289 UDP    antares            6767 zucchini          48655   16.0
global    104289 UDP    antares            6767 clytemnestra      51823   16.0
global    100680 UDP    agamemnon          1023 antares             776   13.0
global    100680 UDP    agamemnon          1023 antares             775   13.0
global    104288 TCP    zucchini          33547 antares            6868    8.0
global    104288 TCP    clytemnestra      49601 antares            6868    8.0
global    104288 TCP    antares            6868 zucchini          33547    8.0
global    104288 TCP    antares            6868 clytemnestra      49601    8.0
Total: bytes in: 101.0  bytes out: 200.0 

In the following output, the tcpstat command reports the five most active TCP traffic flows for a server:

# tcpstat -l 5
ZONE            PID PROTO  SADDR            SPORT DADDR            DPORT   BYTES
global        28919 TCP    achilles.exampl  65398 aristotle.exampl   443   33.0
zone1          6940 TCP    ajax.example.com  6868 achilles.exampl  61318    8.0
zone1          6940 TCP    achilles.exampl  61318 ajax.example.com  6868    8.0
global         8350 TCP    ajax.example.com  6868 achilles.exampl  61318    8.0
global         8350 TCP    achilles.exampl  61318 ajax.example.com  6868    8.0
Total: bytes in: 16.0  bytes out: 49.0

For more information, see Observing Network Traffic With the ipstat and tcpstat Commands in Administering TCP/IP Networks, IPMP, and IP Tunnels in Oracle Solaris 11.3 and the tcpstat(1M) man page.