Go to main content

Troubleshooting Network Administration Issues in Oracle® Solaris 11.4

Exit Print View

Updated: November 2020
 
 

Observing Network Configuration and Traffic Usage

The following information further describes how to use the observability tools that are described in the tables in About Network Troubleshooting and Observability.

The following figure shows a common, hypothetical network configuration to illustrate how various networking features are configured at each layer of the Oracle Solaris network protocol stack.


Note -  The figure does not include every possible datalink type that you can configure. For a more detailed description of all of the features that you can configure on each layer of the network protocol stack, see Chapter 1, Summary of Oracle Solaris Network Administration in Strategies for Network Administration in Oracle Solaris 11.4.

Figure 1  Network Configuration Within the Oracle Solaris Network Protocol Stack

image:Figure describes tools for observing network traffic usage for networking             features by layer of network stack the feature is configured.

    The figure illustrates the following configuration:

  • At the physical layer of the network stack, three physical NICs, nxgbe0, ixgbe0, and ixgbe1, are present in the system and appear as physical datalink instances, net2, net1, and net0, respectively.

  • These physical NICs are then grouped together into an aggregation called aggr0.

  • The aggregated datalink is then configured directly with an IP address (aggr0/v4) and is also used simultaneously as the uplink of a virtual switch, called tenant/hr, which is configured as an elastic virtual switch. The virtual switch has two virtual ports, vport0 and vport1.

  • An Oracle Solaris zone (zone-A) has a VNIC called zone-A/net0, which is connected to one of the virtual ports. Within the zone itself, the VNIC appears as a datalink (net0), which is configured with an IP address (net0/v4).

  • A flow for the HTTP traffic is also created on top of the aggregation (aggr0).

Tools for Observing Network Configuration and Traffic Usage

You configure and administer datalinks by using the dladm command. You use the dlstat command to obtain statistics on network traffic usage for datalinks.

To display inbound and outbound traffic statistics per link, use one of the following commands:

# dlstat link
# dlstat show-link link

To display inbound and outbound traffic statistics per physical network device, use the following command:

# dlstat show-phys link

To display inbound and outbound traffic statistics per port and per link aggregation, use the following command:

# dlstat show-aggr link

For more information, see the dlstat(8) man page.

You configure and administer flows by using the flowadm command. You use the flowstat command to obtain statistics on network traffic usage for flows. As shown in Network Configuration Within the Oracle Solaris Network Protocol Stack, depending on the attributes that you specify, you can use flows to observe network traffic usage at different layers of the network stack. For more information, see the flowadm(8) and flowstat(8) man pages.

For more information about monitoring network traffic usage, see Chapter 8, Monitoring Network Traffic and Resource Usage in Managing Network Virtualization and Network Resources in Oracle Solaris 11.4.

Observing Network Configuration and Traffic Usage at the Hardware Layer

    Troubleshooting network configuration and performance issues at the hardware layer of the network protocol stack might involve observing the following:

  • Number of rings per physical NIC and number of packets being transmitted through those rings.

  • Number of packet drops per physical NIC and per physical ring.

  • NIC-specific counters that might be useful.

  • Number of rings and number of descriptors that are configured per physical NIC.

For physical devices, you can use the dladm show-phys and dlstat show-phys commands to observe network traffic usage. These two commands display different output, depending on the type of information that you want to obtain.

For example, use the dladm show-phys command without any options to display the physical device and the attributes for all of the physical links on a system:

# dladm show-phys
LINK              MEDIA                STATE      SPEED    DUPLEX    DEVICE
net1              Ethernet             unknown    0        unknown   bge0
net0              Ethernet             up         1000     full      nge0

For more information, see Chapter 2, Administering Datalink Configuration in Oracle Solaris in Configuring and Managing Network Components in Oracle Solaris 11.4 and the dladm(8) man page.

The dlstat show-phys command displays statistics about the packets and bytes that are transmitted and received per physical device. This subcommand operates on the hardware rings, which are at the hardware layer of the network stack.

The following example displays statistics for all of the physical links on a system. The output displays both incoming and outgoing traffic statistics for each link on a system. Information about the number of packets and byte sizes per packet is also displayed.

# dlstat show-phys
LINK    IPKTS   RBYTES    OPKTS   OBYTES
net1        0        0        0        0
net0    1.95M  137.83M   37.95K    3.39M

You can use the –r option to display receive-side statistics on each of the hardware rings for a device. The output of this command includes the bytes and packets that are received and the hardware and software drops, and so on, for the device. The following example shows that the net4 datalink has eight rings, which are identified under the INDEX field.

# dlstat show-phys -r net4
LINK  TYPE  INDEX    IPKTS   RBYTES
net4    rx      0      701   42.06K
net4    rx      1        0        0
net4    rx      2        0        0
net4    rx      3        0        0
net4    rx      4        0        0
net4    rx      5        0        0
net4    rx      6        0        0
net4    rx      7        0        0

To obtain similar information for transmitted traffic, use the –t option.

The following example displays the number of inbound dropped packets per physical link.

# dlstat show-phys -o idrops
IDROPS   
     0         
871.14K

The –o field[,...] option is used to specify a case-insensitive, comma-separated list of output fields to display.

In the following example, the number of inbound and outbound dropped packets and bytes per physical link are displayed.

# dlstat show-phys -o idrops,idropbytes,odrops,odropbytes
IDROPS    IDROPBYTES   ODROPS ODROPBYTES
0                  0        0          0
871.14K            0        0          0

You must specify both the idrops and idropbytes options with the dlstat show-phys –o command. Note that one of these values can be zero, depending on the system's hardware capabilities, as shown in the previous output where the IDROPS field is non-zero, while the IDROBYTES field is zero.

For driver configuration, you can manage property values for specific drivers through the driver.conf file and through datalink properties. Driver configuration files enable you to provide device property values that override the default values that are provided by the driver devices themselves. For more information, see the driver.conf(5) man page.

Observing Network Configuration and Traffic Usage at the Datalink Layer

Several networking features are configured at the datalink layer (L2) of the network protocol stack. These features include both physical and virtual datalinks. Certain commands that you use to observe network traffic usage at this layer of the stack are generic and can be used for any type of configured datalink. Other subcommands are specific to the feature itself and therefore can be used to display additional information about the configuration of that feature.

The commands that you use at this layer of the stack also depend on the type of information that you want to observe. For example, at the datalink layer of the stack, you might want to display fan-out statistics or per-link statistics. You can use different commands to obtain each type of information.

For basic information about datalinks, use the dladm show-link command. This command displays link configuration information, either for all of the datalinks on the system or for a specified datalink.

# dladm show-link
LINK             CLASS     MTU    STATE    OVER
net1             phys      1500   unknown  --
net0             phys      1500   up       --

The output shows that this system has two datalinks, which are directly associated with their corresponding physical NICs. No special datalinks exist on the system, for example, an aggregation or a VNIC. These types of L2 entities are configured over the physical datalinks under the phys class.

You use the dlstat show-link command to observe network traffic usage at the datalink layer. The show-link subcommand operates at the datalink layer of the network protocol stack and provides statistics that refer to the lanes that are configured over the physical link.

The following output shows inbound and outbound traffic statistics per link:

# dlstat show-link
LINK      IPKTS    RBYTES     OPKTS    OBYTES
net1          0        0         0         0
net0      1.96M  137.97M     38.40K    3.29M

In the following example, receive-side traffic statistics for the net4 device are reported. The statistics for INTRS and POLLS counters are also displayed, which indicates how many packets were received in the interrupt context versus the polling mode. The IDROPS counter indicates how many packets were dropped at the datalink layer of the network stack.

# dlstat show-link -r net4
LINK  TYPE      ID    INDEX    IPKTS  RBYTES    INTRS    POLLS   IDROPS
net4    rx     local     --        0      0        0        0        0
net4    rx     other     --        0      0        0        0        0
net4    rx        hw      0    7.46M  1.06G    5.62M    1.84M        0
net4    rx        hw      1        0      0        0        0        0
net4    rx        hw      2        0      0        0        0        0
net4    rx        hw      3        0      0        0        0        0
net4    rx        hw      4        2    196        2        0        0
net4    rx        hw      5        0      0        0        0        0
net4    rx        hw      6        0      0        0        0        0
net4    rx        hw      7        0      0        0        0        0

In the example output, only statistics for the named link, physical device (for the show-phys subcommand), or aggregation (for the show-aggr subcommand) is displayed. If link is not specified in the command, then statistics for all of the links, devices, and aggregations are displayed in the output.

    In this example, the information that is displayed under the ID field is interpreted as follows:

  • local – Denotes corresponding loopback traffic on layer 2 (L2) of the network stack.

  • other – Includes broadcast and multicast traffic.

    Over the lifetime of a datalink, the hardware resources that are associated with a datalink might vary, depending on resource utilization, link configuration, or the assignment of physical NICs to link aggregations. The rx entries that are listed in the output of the show-link –r command correspond to the hardware resource that is currently assigned to the link. The output for the other row includes traffic for hardware resources that are no longer assigned to the datalink.

  • hw – Denotes a hardware lane.

  • sw – Denotes a software lane (as shown in the following example).

The distinction between hardware and software lanes is based on the ability of a NIC to support ring allocation. On hardware lanes, rings are dedicated to the packets that use those lanes. In contrast, rings on software lanes are shared amongst datalinks.

The following output displays statistics for outbound packets on the rings that are used by net4:

# dlstat show-link -t net4
LINK  TYPE      ID  INDEX    OPKTS   OBYTES   ODROPS
net4    tx   local     --        0        0        0            
net4    tx   other     --        0        0        0            
net4    tx      hw      0      372   15.67K        0            
net4    tx      hw      1        1       98        0
net4    tx      hw      2        0        0        0            
net4    tx      hw      3        0        0        0            
net4    tx      hw      4        0        0        0             
net4    tx      hw      5        0        0        0            
net4    tx      hw      6        1       98        0            
net4    tx      hw      7        0        0        0 

You can use the –f option to display received and dropped fanout packets, as in the following example that shows how to display fanout statistics:

# dlstat show-link -f
LINK  INDEX    IPKTS   IDROPS
net0      0    1.38K        0
net0      1       36        0
net0      2       25        0
net0      3      147        0
net0      4       15        0
net0      5       45        0
net0      6       18        0
net0      7       39        0

Observing Network Configuration and Traffic Usage for Aggregations

Aggregations are also configured at the datalink layer (L2) of the network protocol stack. Depending on the type of information that you want to obtain, for example, the overall distribution of traffic between the physical NIC or aggregation statistics, use the following commands:

dladm show-aggr

Displays aggregation configuration (the default), Link Aggregation Control Protocol (LACP) information or DLMP probe-based failure and recovery detection status, either for all of the aggregations or for a specified aggregation.

Specify the –x option to display detailed per-aggregation information for an aggregation.

dlstat show-aggr

Displays statistics about the packets and bytes that are transmitted and received for an aggregation.

dlstat show-link

Displays statistics about the packets and bytes that are transmitted and received for an aggregation (the aggregation being a datalink).

One difference between the dlstat show-aggr and dlstat show-link commands is that the dlstat show-aggr command displays per-port statistics, while the dlstat show-link command provides per-link statistics. Another important difference between these two commands is that the dlstat show-aggr command displays the overall statistics for the entire aggregation. Whereas, the dlstat show-link command displays only the statistics for the master client of the aggregation, for example IP.

Thus, if you create VNICs on top of an aggregation, the dlstat show-aggr command would report the total number of packets across all of the VNICs, plus the primary client (IP). This output is similar to the output of the show-phys subcommand compared to the output of the show-link subcommand, where show-phys displays the total traffic usage, while show-link displays only the traffic usage for the primary datalink.

For more information about administering aggregations, see Chapter 2, Configuring High Availability by Using Link Aggregations in Managing Network Datalinks in Oracle Solaris 11.4.

Example 6  Displaying Aggregation Configuration Information

The output of the dladm show-aggr -x displays the status of the existing aggregations that are configured on a system.

# dladm show-aggr -x
LINK       PORT        SPEED    DUPLEX      STATE     ADDRESS             PORTSTATE
aggr1      --         1000Mb      full         up    0:14:4f:29:d1:9d          --
net1                  1000Mb      full         up    0:14:4f:29:d1:9d     attached
net3                     0Mb   unknown       down    0:14:4f:29:d1:9f     standby
Example 7  Displaying Per-Port Statistics for Aggregations

The output for the dlstat show-aggr command displays per-port statistics for an aggregation. Both the packets and bytes that are transmitted and received for the aggregation are displayed.

# dlstat show-aggr
LINK            PORT    IPKTS      RBYTES    OPKTS   OBYTES
aggr1            --      99        2.18K     23       966
aggr1            net4    25        1.50K      8       336
aggr1            net5    74        10.68K    15       630
Example 8  Displaying Per-Link Statistics for Aggregations

The output of the dlstat show-link command displays per-link statistics for an aggregation. Both the packets and bytes that are transmitted and received for the aggregation are displayed. The difference between this example and the previous example is that the show-aggr subcommand displays per-port statistics, while the show-link subcommand displays per-link statistics.

# dlstat show-link
LINK    IPKTS     RBYTES    OPKTS   OBYTES
net5        0          0        0        0
net2        0          0    5.60K    1.49M
net4        0          0        0        0
net6    4.43K      1.32M    6.39K    1.56M
net1    4.43K      1.32M    6.39K    1.56M
net0  387.10K     99.42M   59.43K    7.67M
net3        0          0    5.61K    1.50M
aggr1     150     18.65K        30   1.26K
Example 9  Displaying Receive-Side Traffic Statistics for Hardware Rings of an Aggregation

This example displays receive-side statistics for each of an aggregation's (aggr1) hardware rings.

# dlstat show-aggr -r
LINK      PORT  INDEX    IPKTS   RBYTES
aggr1     --       --    7.93M    4.37G
aggr1     net4      0  541.37K  298.21M
aggr1     net4      1        0        0
aggr1     net4      2        0        0
aggr1     net4      3    1.79M  991.63M
aggr1     net4      4  780.47K  433.30M
aggr1     net4      5        1       98
aggr1     net4      6  530.05K  292.34M
aggr1     net4      7  548.36K  301.74M
aggr1     net5      0  524.10K  288.56M
aggr1     net5      1    1.34M  739.92M
aggr1     net5      2  818.08K  448.12M
aggr1     net5      3  235.62K  129.96M
aggr1     net5      4        0        0
aggr1     net5      5        0        0
aggr1     net5      6  554.44K  305.24M
aggr1     net5      7  260.61K  143.71M
Example 10  Displaying Receive-Side Traffic Statistics for an Aggregation's Hardware Lanes

This example displays receive-side traffic statistics for each hardware lane of an aggregation (aggr1). Note that the statistics reported pertain to the traffic for the primary client of the aggregation only, for example, the IP traffic on top of the aggregation. The output does not include traffic statistics for other clients that are configured on top of the aggregation, for example VNICs. To display traffic usage for all of the clients of an aggregation, you must use the dlstat show-aggr command, as shown in Example 6, Displaying Aggregation Configuration Information and Example 9, Displaying Receive-Side Traffic Statistics for Hardware Rings of an Aggregation.

# dlstat show-link -r aggr1
LINK   TYPE    ID    INDEX    IPKTS   RBYTES    INTRS   POLLS  IDROPS
aggr1   rx   local     --       0       0        0        0      0
aggr1   rx   other     --       0       0        0        0      0
aggr1   rx      hw      0     721   43.26K     721        0      0
aggr1   rx      hw      1       0       0        0        0      0
aggr1   rx      hw      2       0       0        0        0      0
aggr1   rx      hw      3       0       0        0        0      0
aggr1   rx      hw      4       0       0        0        0      0
aggr1   rx      hw      5       0       0        0        0      0
aggr1   rx      hw      6       0       0        0        0      0
aggr1   rx      hw      7       0       0        0        0      0
aggr1   rx      hw      8  22.23K   1.33M   22.23K        0      1
aggr1   rx      hw      9     693  63.76K      693        0      0
aggr1   rx      hw     10       0       0        0        0      0
aggr1   rx      hw     11       0       0        0        0      0
aggr1   rx      hw     12       0       0        0        0      0
aggr1   rx      hw     13  13.00K   4.45M    13.00K       0      2
aggr1   rx      hw     14       0       0        0        0      0
aggr1   rx      hw     15  10.40K 624.06K   10.40K        0      0

Observing Network Configuration and Traffic Usage for an EVS Switch

You configure and manage the elastic virtual switch (EVS) feature of Oracle Solaris at the datalink layer (L2) of the network protocol stack. Depending on the type of information that you want to display, use the following commands:

evsadm

Creates and manages EVS switches and their resources: IP networks (IPnets) and virtual ports (VPorts).

Use the show-evs subcommand to display information for all of the EVS switches that are managed by an EVS controller or for a specified EVS switch.

evsstat

Displays network traffic statistics for all of the VPorts in a data center or for all of the VPorts of a specified elastic virtual switch. The command also displays the statistics for any VNICs associated with the VPorts.

dladm show-vnic –c

Displays information about the VNICs that are connected to an EVS switch. The VPort and the EVS switch to which the VNIC is connected is determined by the EVS switch.

For more information about administering the EVS feature, see Chapter 6, Administering Elastic Virtual Switches in Managing Network Virtualization and Network Resources in Oracle Solaris 11.4.

Example 11  Displaying Information About an EVS Configuration

In this example, the evsadm command is used to display basic information about an EVS configuration.

# evsadm
NAME          TENANT        STATUS  VNIC    IP          HOST
evs0          sys-global    busy    --      ipnet0      sysabc-02
sys-vport0    --            used    vnic0   203.0.113.2/24 sysabc-02
Example 12  Displaying Inbound and Outbound Traffic Usage for VPorts That Are Connected to an EVS Switch

In this example the evsstat command is used to display incoming and outgoing network traffic statistics for the only VPort that is connected to an EVS switch.

# evsstat
VPORT        EVS     TENANT    IPKTS   RBYTES   OPKTS  OBYTES
sys-vport0  evs0 sys-tenant  101.88K   32.86M  40.16K  4.37M
sys-vport1  evs0 sys-tenant    4.50M    6.78G   1.38M 90.90M
Example 13  Displaying Information About VNICs That Are Connected to an EVS Switch

You can create VNICs on top of an underlying link and you can also connect VNICs to an EVS switch. To obtain information about a VNIC that is connected to an EVS switch, use the dladm show-vnic command with the –c option, as shown in the following example.

# dladm show-vnic -c
LINK       TENANT        EVS     VPORT       OVER   MACADDRESS        IDS
vnic0      sys-global    evs0    sys-vport0  net    12:8:20:f2:46:22  VID:200

Observing Network Configuration and Traffic Usage for VNICs

VNICs are configured at the datalink layer (L2) of the network protocol stack. Use the following commands to display configuration information and observe network traffic usage for these L2 entities:

dladm show-vnic

Displays VNIC configuration information for all of the VNICs on a system, all of the VNICs on a link, or for a specified vnic-link.

dlstat

Displays statistics about the packets and bytes that are transmitted and received per VNIC.

For more information about administering VNICs, see Managing VNICs in Managing Network Virtualization and Network Resources in Oracle Solaris 11.4.

Example 14  Displaying VNIC Configuration Information

The following example displays VNIC configuration information for the existing VNIC on a system (vnic0).

# dladm show-vnic
LINK        OVER        SPEED  MACADDRESS       MACADDRTYPE IDS
vnic0       net1        1000   2:8:20:f2:46:22  fixed       VID:200
Example 15  Displaying Network Traffic Statistics for VNICs

In the following example, the dlstat command is used to display statistics of the packets and bytes that are transmitted and received by a specific VNIC (vnic0).

# dlstat vnic0
LINK    IPKTS   RBYTES    OPKTS    OBYTES
vnic0   1.53M   158.18M   154.22K  32.84M

Observing Network Configuration and Traffic Usage at the IP Layer

You observe network traffic usage at the IP layer (L3) of the network protocol stack by using a few different commands. Depending on the type of information that you want to display, use the following commands:

ipadm

Displays general configuration information for IP interfaces and addresses.

ipadm show-addr

Displays IP address information, either for the given address object (addrobj) or for all of the address objects that are configured on a specified interface, including those that are only in the persistent configuration.

ipadm show-if

Displays network interface configuration information, either for all of the network interfaces that are configured on the system, including those that are only in the persistent configuration, or for a specified interface.

ipstat

Displays statistics on IP traffic based on the selected output mode and sort order.

netstat

Displays the contents of certain network-related data structures in various formats.

For more information about administering IP configuration, see Monitoring IP Interfaces and Addresses in Configuring and Managing Network Components in Oracle Solaris 11.4. See also the ipstat(8) and netstat(8) man pages.

Example 16  Displaying General Information About IP Configuration

You use the ipadm command to display general information about IP configuration. This example shows how to display information about all of the IP interfaces and addresses that are configured on a system.

# ipadm
NAME              CLASS/TYPE STATE        UNDER      ADDR
lo0               loopback   ok           --         --
   lo0/v4         static     ok           --         127.0.0.1/8
   lo0/v6         static     ok           --         ::1/128
net0              ip         ok           --         -- 
   net0/v4        static     ok           --         203.0.113.140/24
Example 17  Displaying Information About Configured IP Interfaces

You use the ipadm command with the show-if subcommand to display information about the configured IP interfaces on a system, as shown in the following example.

# ipadm show-if
IFNAME   CLASS      STATE   ACTIVE   OVER
lo0      loopback   ok       yes     ---
net0     ip         ok       yes     ---
Example 18  Displaying Information About Configured IP Address Objects

The following example shows how to use the ipadm command with the show-addr subcommand to display information about configured IP address objects on a system.

# ipadm show-addr
ADDROBJ      TYPE       STATE   ADDR
lo0/v4       static     ok      127.0.0.1/8
Example 19  Displaying Statistics About IP Traffic by Using the ipstat Command

The following example shows how to use the ipstat command to display statistics about IP traffic. The command provides options for reporting statistics about the IP traffic that matches a specified source or destination address, interface, and higher layer protocols.

# ipstat
SOURCE                    DEST                  PROTO      INT      RATE
abc11example-02           dhcp-sys.example       TCP      net0     145.6
dns1.example.com          abc11example-02        UDP      net0      66.0
abc11example-02           dns1.example.com       UDP      net0      10.4
dhcp-sys.example          abc11example-02        TCP      net0       4.0
foo1.example.com          all-sys.mcast.net     ICMP      net0       3.2

For more information, see Using the ipstat and tcpstat Commands in Administering TCP/IP Networks, IPMP, and IP Tunnels in Oracle Solaris 11.4.

Example 20  Displaying Connected Sockets by Using the netstat Command

The netstat command displays network status and protocol statistics. You can display the status of TCP, SCTP, and UDP endpoints in a table format. You can also display routing table and interface information by using this command. The various types of network data that is reported depends upon the command option that you specify.

For example, you can use the netstat command without any options to display a list of active sockets for each protocol:

# netstat
 TCP: IPv4
   Local Address         Remote Address        Swind   Send-Q  Rwind   Recv-Q  State
  ------------------     -----------------    -------  ------  ------- ------  --------
   localsys.local.port1  remotesys1              65535  0      128592   0      ESTABLISHED
   localsys.local.port2  localsys.local.port5   130880  0     139264   0      ESTABLISHED
   localsys.local.port3  localsys.local.port6   139060  0     130880   0      ESTABLISHED
   localsys.local.port4  remotesys2.remote.port2 65572 63     128480   0      ESTABLISHED

Use the netstat –P protocol command to limit the display of statistics or state of just those sockets that are applicable to a specific protocol, as shown in this example:

# netstat -P tcp
TCP: IPv4
Local Address Remote Address               Swind  Send-Q  Rwind  Recv-Q  State
------------ --------------------          ------ ------ ------- ------  -------- 
sys3.48962   foo.com.ldaps                 49232   0     128872  0       ESTABLISHED
sys3.ssh     dhcp1-203-0-113-210.foo.com   64292  63     128480  0       ESTABLISHED

You can specify protocol as any of the following: ip, ipv6, icmp, icmpv6, igmp, udp, tcp, rawip.

The following example shows how to use the netstat –s command to display per-protocol statistics:

# netstat -s
RAWIP   rawipInDatagrams    =      2    rawipInErrors   =  0
        rawipInCksumErrs    =      0
        rawipOutDatagrams   =      2
        rawipOutErrors      =      0
    
UDP     udpInDatagrams      =   1023    udpInErrors    =   0
        udpOutDatagrams     =   1023
        udpOutErrors        =      0
    
TCP     tcpRtoAlgorithm     =      4    tcpRtoMin     =   200
        tcpRtoMax           =  60000
        tcpMaxConn          =     -1
        tcpActiveOpens      =    382
        tcpPassiveOpens     =     83
        tcpAttemptFails     =     81
        tcpEstabResets      =      1
        tcpCurrEstab        =      2
        tcpOutSegs          =   6598
        tcpOutDataSegs      =   5653
        tcpOutDataBytes     = 836393
        tcpRetransSegs      =     16        
. . .
Example 21  Displaying Statistics About Flows That Are Configured at the IP Layer of the Network Stack

You can create flows for the various IP addresses or subnets that are configured at the IP layer of the network protocol stack. You can then use the flowstat command to display statistics about these flows as shown in the following example.

# flowadm add-flow -l net0 -a transport=tcp tcpflow1
# flowadm add-flow -l net4 -a transport=tcp tcpflow2

# flowstat
FLOW      IPKTS   RBYTES   IDROPS    OPKTS   OBYTES   ODROPS
tcpflow2      0       0        0        0        0        0
tcpflow1     53   5.62K        0       45    5.52K        0

To display flow information for a particular device, specify the –l option as follows:

# flowstat -l net0
FLOW      IPKTS   RBYTES  IDROPS   OPKTS  OBYTES ODROPS
tcpflow1    108   11.19K       0     86   10.45K      0
Example 22  Creating and Observing Flows for Specific IP Addresses

You can create flows for specific IP addresses by using the local_ip and remote_ip attributes with the flowadm add-flow command. You can then use the flowstat command display statistics for these flows, as shown in the following example.

# flowadm add-flow -l net0 -a local_ip=203.0.113.45 flow1
# flowadm add-flow -l net4 -a remote_ip=192.0.2.0/24 flow2
# flowstat 
FLOW      IPKTS     RBYTES   IDROPS    OPKTS  OBYTES   ODROPS
flow2   528.54K    787.39M        0  179.39K  11.85M        0
flow1   742.81K      1.10G        0        0       0        0

Observing Network Configuration and Traffic Usage at the Transport Layer

You observe network traffic for features that are configured and administered at the transport layer (L4) of the network protocol stack by using the following commands:

flowstat

Displays runtime statistics about user-defined flows. Use the flowadm show-flow command to determine the flow name to specify with the flowstat command.

You can use flows as an observability tool rather than just for bandwidth control, for example, to measure the amount of traffic that a specific service consumes.

netstat

Displays the contents of certain network-related data structures in various formats. With no arguments, the netstat command displays the connected sockets for PF_INET, PF_INET6, and PF_UNIX, unless modified by using the –f option.

tcpstat

Displayss statistics on TCP and UDP traffic on a server based on the selected output mode and sort order that is specified in the command syntax.

For more information about administering TCP/IP networks with the netstat and tcpstat commands, see Chapter 1, Administering TCP/IP Networks in Administering TCP/IP Networks, IPMP, and IP Tunnels in Oracle Solaris 11.4.

For more information about the flowstat command, see Displaying Network Traffic Statistics of Flows in Managing Network Virtualization and Network Resources in Oracle Solaris 11.4. See also the flowstat(8) man page.

Example 23  Displaying Runtime Statistics for Flows by Using the flowstat Command

The following example output displays a static display of traffic information for all of the configured flows that are on a system. The flowadm command is used to determine the name of the flow.

# flowadm
FLOW        LINK     PROTO LADDR             LPORT RADDR  RPORT DIR
tcpflow1    net1     tcp   --                --    -- --    bi
tcpflow0    net0     tcp   --                --    -- --    bi
udpflow0    net0     udp   --                --    -- --    bi

# flowstat
FLOW       IPKTS   RBYTES  IDROPS    OPKTS   OBYTES   ODROPS       
tcpflow1      0        0       0        0        0        0       
tcpflow0  1.39K  117.86K       0    2.16K  260.77K        0       
udpflow0      5    1.43K       0        0        0        0

You can also use the flowstat command with the –l option to display statistics for all of the flows for a specified link or statistics for a specified flow, as shown in output of the following two examples:

# flowstat -l net0
FLOW      IPKTS   RBYTES   IDROPS    OPKTS    OBYTES   ODROPS
tcpflow0  1.51K  126.85K        0    2.43K   292.85K        0
udpflow0      9    2.80K        0        0         0        0
# flowstat -l net0 tcpflow0
FLOW      IPKTS    RBYTES   IDROPS     OPKTS   OBYTES  ODROPS
tcpflow0  1.66K   137.11K        0     2.69K  324.42K       0
Example 24  Displaying Information About Transport Layer Data Structures by Using the netstat Command

You can use the netstat command to display information about data structures at the transport layer (L4) of the network protocol stack, for example TCP or UDP. In the following example, the netstat –P transport-protocol command is used to display information about TCP.

# netstat -p tcp
TCP: IPv4 
  Local Address     Remote Address   Swind   Send-Q  Rwind  Recv-Q  State
  ----------------  ---------------  ------- ------ ------- ------ -----------
  localsys.ssh     remotesys1.port4   65380    63     128480   0    ESTABLISHED
  localsys.port1   remotesys2.ldaps   65535     0     128592   0    ESTABLISHED
  localsys.port2   localsys.port5    130880     0     139264   0    ESTABLISHED
  localsys.port3   localsys.port6    139060     0     130880   0    ESTABLISHED
Example 25  Displaying Statistics for TCP and UDP Traffic by Using the tcpstat Command

You use the tcpstat command to observe network traffic at the transport layer of the network protocol stack, specifically for TCP and UDP. In addition to the source and destination IP addresses, you can observe the source and destination TCP or UDP ports, the PID of the process that is sending or receiving the traffic, and the name of the global zone in which that process is running.

The following example shows the type of information that is reported when you use the tcpstat command with the –c option. The –c option specifies to print newer reports after previous reports, without overwriting previous reports:

# tcpstat -c 3
ZONE         PID PROTO  SADDR             SPORT DADDR             DPORT   BYTES
global    100680 UDP    antares           62763 agamemnon          1023   76.0
global    100680 UDP    antares             775 agamemnon          1023   38.0
global    100680 UDP    antares             776 agamemnon          1023   37.0
global    100680 UDP    agamemnon          1023 antares           62763   26.0
global    104289 UDP    zucchini          48655 antares            6767   16.0
global    104289 UDP    clytemnestra      51823 antares            6767   16.0
global    104289 UDP    antares            6767 zucchini          48655   16.0
global    104289 UDP    antares            6767 clytemnestra      51823   16.0
global    100680 UDP    agamemnon          1023 antares             776   13.0
global    100680 UDP    agamemnon          1023 antares             775   13.0
global    104288 TCP    zucchini          33547 antares            6868    8.0
global    104288 TCP    clytemnestra      49601 antares            6868    8.0
global    104288 TCP    antares            6868 zucchini          33547    8.0
global    104288 TCP    antares            6868 clytemnestra      49601    8.0
Total: bytes in: 101.0  bytes out: 200.0 

In the following output, the tcpstat command reports the five most active TCP traffic flows for a server:

# tcpstat -l 5
ZONE            PID PROTO  SADDR            SPORT DADDR            DPORT   BYTES
global        28919 TCP    achilles.exampl  65398 aristotle.exampl   443   33.0
zone1          6940 TCP    ajax.example.com  6868 achilles.exampl  61318    8.0
zone1          6940 TCP    achilles.exampl  61318 ajax.example.com  6868    8.0
global         8350 TCP    ajax.example.com  6868 achilles.exampl  61318    8.0
global         8350 TCP    achilles.exampl  61318 ajax.example.com  6868    8.0
Total: bytes in: 16.0  bytes out: 49.0

See Using the ipstat and tcpstat Commands in Administering TCP/IP Networks, IPMP, and IP Tunnels in Oracle Solaris 11.4 and the tcpstat(8) man page.