Chapter 12 Administering Virtual Networks and Resource Controls (Tasks)

This chapter contains monitoring and statistics-gathering tasks for a system that has configured interface-based flow control, or a virtual network, or both. The connectivity monitoring tasks use standard network tools to observe and evaluate activity on the virtual network. The tasks for gathering usage statistics on packet flows and, if applicable, on VNICs, use the dladm, flowadm, and acctadm commands. These last tasks are especially useful for provisioning, consolidation, and billing purposes. The following subjects are discussed:

• Monitoring and Statistics–Gathering Task Map

• Verifying Virtual Network Connectivity

• Observing Traffic on Virtual Networks

• Gathering Usage Statistics for VNICs and Flows

Monitoring and Statistics–Gathering Task Map

Verifying Virtual Network Connectivity

You can use standard network tools to verify your virtual network's connectivity. This section contains simple tasks to help you verify that the VNICs of your virtual network are correctly configured and have the expected network connectivity. Following is a list of the tools used in the tasks, along with links to their man pages.

• dladm(1M)

• ifconfig(1M)

• netstat(1M)

• ping(1M)

How to Verify the VNIC Configuration for the Global Zone

Before You Begin

The following task assumes that you have created a VNIC for the global zone of your system.

1. On the system with the virtual network, become superuser or assume the equivalent root role.

   To create and assign the root role, see How to Make root User Into a Role in System Administration Guide: Security Services.

2. Verify the state of the data links on the system.

   # dladm show-link

   Your output should resemble either of the following:

   • For a system that has a publicly accessible virtual network, such as the network that is configured in How to Create a Virtual Network Interface:

     # dladm show-link LINK CLASS MTU STATE OVER bge0 phys 1500 up bge0 vnic0 vnic 9000 up bge10

     In this output, both the physical network interface bge0 and the VNIC pseudo-interface vnic0 are configured as data links.

   • For a system with a private virtual network that cannot be accessed by external users, such as the network that is configured in How to Create Etherstubs and VNICs for the Private Virtual Network:

     # dladm show-link LINK CLASS MTU STATE OVER e1000g2 phys 1500 unknown -- e1000g0 phys 1500 up -- vnic0 vnic 9000 up etherstub0 vnic1 vnic 9000 up etherstub0

     The network interface e1000g0 is configured as a data link. The presence of etherstub0 indicates this is a private network. Two VNICs, vnic0 and vnic1, are successfully configured over the etherstub.

3. Verify that the VNIC is plumbed and running on the IP level of the TCP/IP protocol stack:

   # ifconfig -a

   You should receive output similar to the following:

   lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 bge0: flags=1000843 <UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3 inet 192.168.8.50 netmask ffffff00 broadcast 192.168.8.255 ether 8:0:20:c8:f4:1d vnic0: flags=201000842<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 2 inet 192.168.8.10 netmask ffffff00 broadcast 192.168.8.255 ether 2:8:20:54:f4:74

   Both the network interface bge0 and the VNIC vnic0 are plumbed and up.

How to Verify Configuration of a Virtual Network of Exclusive IP Zones

Before You Begin

The procedure assumes that you have created at least two VNICs and corresponding exclusive IP zones to form a virtual network. You also have configured and plumbed these VNICs while logged into their respective zones. The next task verifies the configuration of the virtual network created in Basic Virtual Network on a Single System.

1. On the system where you create the virtual network, become superuser or assume the equivalent root role in the global zone.

   To create and assign the root role, see How to Make root User Into a Role in System Administration Guide: Security Services.

2. Ensure that the VNICs are configured as data links in the global zone.

   # dladm show-vnic

   You should receive output similar to the following:

   LINK OVER SPEED MACADDRESS MACADDRTYPE vnic1 e1000g0 1000 Mbps 2:8:20:5f:84:ff random vnic2 e1000g0 1000 Mbps 2:8:20:54:f4:74 random

   In this example, both VNICs of the virtual network are configured as data links over network interface e1000g0.

3. Verify that any interfaces known to the global zone are plumbed and up.

   # ifconfig -a lo0: flags=2001000849<UP,UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 e1000g0: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 2 inet 192.168.3.70 netmask ffffff00 broadcast 192.168.83.255 ether 0:14:4f:94:d0:40

   Only the network interface e1000g0 is plumbed for the global zone. This interface has the IP address 192.168.3.70 and connects the system to the external 192.168.3.0/24 network. For the virtual network configuration, ifconfig -a in the global zone should not report any VNICs.

4. Check the state of the configured zones.

   # zoneadm list -v ID NAME STATUS PATH BRAND IP 0 global running / native shared 5 zone2 running /export/home/zone2 native excl 7 zone1 running /export/home/zone1 native excl

   The STATUS column indicates that the zones are up and running. If the status of the zones indicates a condition other than “running,” you need to reboot the zone. For instructions, refer to Chapter 20, Installing, Booting, Halting, Uninstalling, and Cloning Non-Global Zones (Tasks), in System Administration Guide: Virtualization Using the Solaris Operating System.

5. Check the global zone's known routes.

   # netstat -rn

   You should receive output similar to the following:

   Routing Table: IPv4 Destination Gateway Flags Ref Use Interface -------------------- -------------------- ----- ----- ---------- --------- default 192.168.3.1 UG 1 8 e1000g0 192.168.3.0 192.168.3.70 U 1 143 e1000g0 127.0.0.1 127.0.0.1 UH 1 13 lo0 Routing Table: IPv6 Destination/Mask Gateway Flags Ref Use If --------------------------- --------------------------- ----- --- ------- ----- ::1 ::1 UH 1 22 lo0

   The global zone's default route to external networks is through the gateway 192.168.3.1. This is the IP address of the default router for network 192.168.3.0/24. The global zone also reports that the route to the gateway is through 192.168.3.70, the IP address of the system's e1000g0 interface.

6. Log in to one of the zones of the virtual network, for example, zone1, and ensure that the zone's VNIC is plumbed and up.

   # zlogin zone1 # ifconfig -a vnic1 vnic1: flags=201000842<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 2 inet 192.168.3.20 netmask ffffff00 broadcast 192.168.3.255 ether 2:8:20:54:f4:74

7. Check the known routes between the local zone and the external network.

   # netstat -rn

   You should receive output similar to the following:

   Routing Table: IPv4 Destination Gateway Flags Ref Use Interface -------------------- -------------------- ----- ----- ---------- --------- default 192.168.3.1 UG 1 0 vnic1 192.168.3.0 192.168.3.20 U 1 2 vnic1 127.0.0.1 127.0.0.1 UH 1 23 lo0

   The output verifies that the default route for zone1 is to the default router, 192.168.3.1. zone1 also knows to route packets through vnic1, 192.168.3.20. This traffic is then passed to the global zone, where the packets travel through the network interface e1000g0.

8. Verify the VNICs' connectivity.

   Perform these steps while logged into a local zone. The following steps assume that you are logged into zone1.

   1. Check the connectivity between the local zone's VNIC and the system's network interface.

      # ping network-interface-address

      For example, check that vnic1 can pass traffic to network interface e1000g0, IP address 192.168.3.70.

      # ping 192.168.3.70 192.168.3.70 is alive

   2. Check that the VNIC can pass traffic through the default router, IP address 192.168.3.1.

      # ping 192.168.3.1 192.168.3.1 is alive

   3. Check that the VNIC can pass traffic to another VNIC in the virtual network.

      # ping vnic-IP-address

      For example, to check that vnic1 can pass traffic to vnic2 (IP address192.168.3.22), run the following command.

      # ping 192.168.3.22 192.168.3.22 is alive

Next Steps

• If Steps 5–7 complete successfully in one exclusive IP zone, then repeat them for each exclusive IP zone in the virtual network.

• To observe packet flows and take statistics, go on to the procedure Observing Traffic on Virtual Networks.

Observing Traffic on Virtual Networks

Use the standard snoop command to observe and analyze the status of the virtual network. snoop gathers packets and displays their output, enabling you to observe and analyze their content. You can use snoop output to verify the connectivity by observing the “conversation” among the VNICs on a virtual network. For full details on snoop usage, refer to the snoop(1M) man page.

How to Verify Virtual Network Connectivity by Using the snoop Command

The following task observes traffic on the private network configured in Example 11–7. However, you can use snoop to observe traffic over a publicly-accessible virtual network, as well.

1. On the system where you create the private virtual network, become superuser or assume the equivalent root role in the global zone.

   To create and assign the root role, see How to Make root User Into a Role in System Administration Guide: Security Services.

2. Gather information about network traffic on the private virtual network.

   # snoop -d etherstub0

   By “snooping” on the private network's etherstub, you can obtain information about activities on all the VNICs configured over that etherstub. You can also snoop the individual VNICs.

3. Check the snoop output to verify connectivity among the VNICs of the etherstub.

   Using device etherstub0 (promiscuous mode) 192.168.0.250 -> 192.168.0.200 RIP R (10 destinations) 192.168.0.250 -> 192.168.0.200 RIP R (10 destinations) 192.168.0.250 -> 224.0.0.1 ICMP Router advertisement (Lifetime 1800s [1]: {192.168.0.250 0}) 192.168.0.250 -> 192.168.0.200 RIP R (10 destinations) 192.168.0.250 -> 192.168.0.200 RIP R (10 destinations) 192.168.0.220 -> (broadcast) ARP C Who is 192.168.0.250, 192.168.0.250 ? 192.168.0.200-> (broadcast) ARP C Who is 192.168.0.220, 192.168.0.220 ? 192.168.0.250 -> (broadcast) ARP C Who is 192.168.0.200, 192.168.0.200 ? 192.168.0.200 -> 192.168.0.220 ARP R 192.168.0.200, 192.168.0.200 is 2:8:20:45:8f:c9 192.168.0.250 -> 192.168.0.200 ICMP Echo request (ID: 20291 Sequence number: 0) 192.168.0.200 -> (broadcast) ARP C Who is 192.168.0.250, 192.168.0.250 ? 192.168.0.250 -> 192.168.0.200 ARP R 192.168.0.250, 192.168.0.250 is 2:8:20:c2:39:38 192.168.0.200 -> 192.168.0.250 ICMP Echo reply (ID: 20291 Sequence number: 0) 192.168.0.250 -> 192.168.0.250 RIP R (10 destinations)

   This output shows the contents of packets that are exchanged among the three VNICs in Figure 10–2 as they contact each other.

   • etherstub0, over vnic0 (IP address 192.168.0.250) sends out RIP routing protocol packets to vnic1 (192.168.0.200).

   • vnic2 (192.168.0.220) sends out an ARP broadcast message (“Who is”), to vnic0 (192.168.0.250). Then, vnic0 sends out an ARP request to vnic1:

     192.168.0.220 -> (broadcast) ARP C Who is 192.168.0.250, 192.168.0.250 ? 192.168.0.250 -> (broadcast) ARP C Who is 192.168.0.200, 192.168.0.200 ?

   • vnic1 (192.168.0.200) sends out an ARP broadcast message (“Who is”) to vnic2 (192.168.0.220).

   • Eventually vnic1 responds to the vnic2's “Who is” broadcast by sending its MAC address, as follows:

     192.168.0.200 -> 192.168.0.220 ARP R 192.168.0.200, 192.168.0.200 is 2:8:20:45:8f:c9

     This response proves that the two VNICs of the virtual network, vnic1 and vnic2, can send packet traffic to each other.

   • Then vnic0 responds to vnic1's ARP request with its MAC address:

     192.168.0.250 -> 192.168.0.200 ARP R 192.168.0.250, 192.168.0.250 is 2:8:20:c2:39:38

     This response proves that etherstub0 on vnic0 and vnic1 on the virtual network can send traffic to each other.

   The previous output is but a sample of the many ways to use snoop for diagnostic purposes. For more information, refer to the snoop(1M) man page.

Gathering Usage Statistics for VNICs and Flows

Network virtualization and resource control introduces tools for observing traffic on a per-interface or per-flow basis. The next tasks show how to use options of the dladm and flowadm commands to obtain statistics on packet traffic. For full details on each command, refer to the following man pages:

• dladm(1M)

• flowadm(1M)

How to Obtain Statistics on VNICs

VNIC statistics are useful for provisioning purposes, such as determining whether a system needs additional VNICs or possibly additional interfaces. You can also use VNIC traffic statistics to determine how well network consolidation onto a virtual network is working.

Before You Begin

The task assumes that you have a working virtual network, such as the network configured in Configuring a Basic Virtual Network.

1. On the system where you create the virtual network, become superuser or assume the equivalent root role in the global zone.

   To create and assign the root role, see How to Make root User Into a Role in System Administration Guide: Security Services.

2. Observe traffic flow over network interfaces

   Before checking the flow usage on individual VNICs, you might want to view overall usage of the VNIC's underlying interface.

   where link-name is the name of a currently plumbed interface, for example internal0.

   # dladm show-link -s -i 5 internal0 LINK IPACKETS RBYTES IERRORS OPACKETS OBYTES OERRORS internal0 5315127 400738164 0 169526 29260024 0 LINK IPACKETS RBYTES IERRORS OPACKETS OBYTES OERRORS internal0 1 60 0 2 340 0 LINK IPACKETS RBYTES IERRORS OPACKETS OBYTES OERRORS internal0 17 1020 0 4 456 0 ^C

   To halt the display, press Ctrl-C.

3. Observe traffic flow over configured VNICs.

   Use the following syntax for each VNIC in the virtual network. You must run this command in the global zone for all VNICs configured on the system.

   # dladm show-link -s -i 5 vnic-link-name

   where vnic-name is the name of the VNIC whose traffic you want to observe.

   You should receive output similar to the following:

   # dladm show-link -s -i 5 vnic0 ipackets rbytes ierrors opackets obytes oerrors vnic0 537001 48104701 0 5 210 0 ipackets rbytes ierrors opackets obytes oerrors vnic0 3 270 0 0 0 0 ^C

   The output indicates that vnic0 has had both incoming packet (ipackets) and outgoing packet (opackets) traffic.

   To halt the display, press Ctrl-C.

How to Obtain Statistics on Flows

Flow statistics help you evaluate packet traffic on your network before assigning bandwidth and priorities to already configured flows. Like VNIC statistical information, flow statistics are also useful for provisioning and, possibly, for billing purposes.

Before You Begin

This procedure assumes that you have configured flows, as described in Chapter 13, Configuring Resource Management on an Interface.

1. On the system where you configure flow control, become superuser or assume the equivalent root role in the global zone.

   To create and assign the root role, see How to Make root User Into a Role in System Administration Guide: Security Services.

2. Observe packet flow statistics for a system that is configured with interface-based flow control.

   The following example shows output for a system with two network interfaces that uses traditional bandwidth control. The system does not have a virtual network configured.

   # flowadm show-flow -s FLOW IPACKETS RBYTES IERRORS OPACKETS OBYTES OERRORS net20 0 0 0 1723 72366 0 httpsflow 0 0 0 32932 3225851 0 httpflow 29 3982 0 42 20799 0

   where

   net20

   is a flow for all UDP traffic across interface 10.10.3.20/24 on the internal network.

   httpsflow

   is a flow for all secure HTTP traffic over interface 192.168.3.25 , which is connected to the external network

   httpflow

   is a flow for all HTTP traffic over interface 192.168.3.25.

3. To halt the display, press Ctrl-C.

Setting Up Accounting for Packet Flows

You can use the extended accounting feature to capture these statistics into a file, which can then be printed for usage reports.

How to Configure Flow Accounting

Use the flowadm command and extended accounting feature of the Solaris OS to collect statistical data on flow usage. In this manner, you can maintain records of flow traffic for tracking, provisioning, or billing purposes. The following task shows how to gather statistics for a system on a traditional network, which has implemented flow control on the system's interfaces. For an example, refer to Figure 10–3. For the tasks that implement flow control on this network, see How to Set Up and Configure Flow Control for a System on a Traditional Network.

Before You Begin

You must have configured flows on the system's interfaces, as explained in Interface-Based Flow Control for Traditional Networks.

1. Assume the Primary Administrator role or become superuser on the system with the interfaces whose usage you want to track.

   The Primary Administrator role includes the Primary Administrator profile. To create the role and assign the role to a user, see Chapter 2, Working With the Solaris Management Console (Tasks), in System Administration Guide: Basic Administration.

2. On the system with the interfaces whose usage you want to track, become superuser or assume the equivalent root role.

   To create and assign the root role, see How to Make root User Into a Role in System Administration Guide: Security Services.

3. View the system's currently configured flows.

   # flowadm show-flow NAME LINK ATTR VALUE app-flow internal0 ip_version 4 local_ip 10.10.12.45/ httpsflow nge0 transport tcp local_port 443 httpflow nge0 transport tcp local_port 80

   The output shows three flows. app-flow is a flow configured on interface internal0. This interface isolates traffic coming from the application server on the network, by using the IP address 10.10.12.45/ of the server as the key. The flows httpflow and httpsflow are configured on interface nge0. These flows isolate HTTP and secure HTTP packets by using their port numbers, 80 and 443, respectively.

4. View usage statistics for the system's three flows.

   # flowadm show-flow -s FLOW IPACKETS RBYTES IERRORS OPACKETS OBYTES OERRORS app-flow 65 0 0 1723 72366 0 httpsflow 0 0 0 32932 3225851 0 httpflow 29 3982 0 42 20799 0

   The -soption of flowadm gathers usage statistics at the point when the command is issued.

5. View the accounting options that are available from extended accounting.

   # acctadm Task accounting: inactive Task accounting file: none Tracked task resources: none Untracked task resources: extended Process accounting: inactive Process accounting file: none Tracked process resources: none Untracked process resources: extended,host Flow accounting: inactive Flow accounting file: none Tracked flow resources: none Untracked flow resources: extended Network accounting: inactive Network accounting file: none Tracked Network resources: none Untracked Network resources: extended

   The last four options implement extended accounting for flows. The previous output verifies that network accounting is not turned on and an accounting file is unknown.

6. Confirm that an extended accounting file does not already exist.

   # cd /var/adm/exacct ; ls

   If no accounting file is listed, this confirms that you can now create the file.

7. Create an extended accounting file for the flows on the system.

   Use the syntax:

   # acctadm -e extended -f /var/adm/exacct/file-name net

   The net argument turns on extended accounting for network flows.

   Suppose you wanted to track usage of the flows you have defined for a system. You would type the following:

   # acctadm -e extended -f /var/adm/exacct/httpflow net

8. Verify that extended accounting is now turned on.

   # acctadm Task accounting: inactive . . Network accounting: active Network accounting file: /var/adm/exacct/httpflow Tracked Network resources: extended Untracked Network resources: none

9. Show usage of all the system's flows.

   # flowadm show-usage -f /var/adm/exacct/httpflow Bytes Packets Errors Duration Bandwidth Link/Flow ___________________________________________________________________________ 68912 76 0 140 3.94 Kbps httpflow 3449 32 0 140 197.09 bps httpsflow 0 0 0 140 0.00 bps app-flow

10. Show the contents of the extended accounting file that have to do with a particular flow.

    Use the following syntax:

    # flowadm show-usage -f /var/adm/exacct/filename flow

    For example, use the following command to obtain data on HTTP usage on the system.

    # flowadm show-usage -f /var/adm/exacct/httpflow httpflow 14:35:51 14:36:11 0 0 0.00 bps httpflow 14:36:11 14:36:31 0 0 0.00 bps httpflow 14:36:31 14:36:51 3789 64895 27.47 Kbps httpflow 14:36:51 14:37:11 120 108 91.20 bps httpflow 14:37:11 14:37:31 0 0 0.00 bps httpflow

    This output gathers statistics on httpflow since the command was issued.

11. Obtain a count of all packet flows on the system.

    Gathering statistics on interface usage can indicate over time whether you should configure a virtual network on the interface or possibly add another interface.

    # dladm show-usage -f /var/adm/exacct/httpflow Bytes Packets Errors Duration Bandwidth Link/Flow ___________________________________________________________________________ 159080 1188 0 400 3.18 Kbps nge0 1600 17 0 400 32.00 bps internal0

    The output shows packet count for the nge0 interface, which has flows for HTTP and HTTPS traffic, as well as unfiltered traffic. The internal0 interface has a flow for traffic received from IP address 10.10.12.45, the address of the application server.

12. Obtain a count of all packet traffic for a specific interface.

    # dladm show-usage -f /var/adm/exacct/httpflow nge0 Start Time End Time In Bytes Out Bytes Bandwidth Device ___________________________________________________________________________ 14:34:51 14:35:11 2512 0 1.00 Kbps nge0 14:35:11 14:35:31 986 0 394.40 bps nge0 14:35:31 14:35:51 2108 0 843.20 bps nge0 14:35:51 14:36:11 2632 0 1.05 Kbps nge0 14:36:11 14:36:31 1653 91 697.60 bps nge0 14:36:31 14:36:51 5237 64937 28.07 Kbps nge0 14:36:51 14:37:11 8902 3424 4.93 Kbps nge0 14:37:11 14:37:31 9648 4958 5.84 Kbps nge0 14:37:31 14:37:51 1766 0 706.40 bps nge0 14:37:51 14:38:11 3334 0 1.33 Kbps nge0 14:38:11 14:38:31 1834 578 964.80 bps nge0