flowadm - administer bandwidth resource control and priority for a connection, protocols, services, containers, and virtual machines
flowadm
flowadm show-flow [-P] [[
-p] -o field[,...]] [-v] [{
-l link | flow}]
flowadm match-flow [-P] [[-p] -o field[,...]] [-v] [-l link]
-a attr=value[,...]
flowadm add-flow [-t] [-R
root-dir] -l
link -a attr=
value[,...]
[-p prop=value[,...]]
flow
flowadm remove-flow [-t] [-R
root-dir] {-l link |
flow}
flowadm show-filter [-P] [[-p] -o <field>,...] [{-l <link> | <flow>}]
flowadm match-filter [-P] [[-p] -o <field>,...] [-l <link>]
-a <attr>=<value>[,...]
flowadm add-filter -t -a <attr>=<value>[,...] <flow>
flowadm remove-filter [-t] {<filter> | -a <attr>=<value>[,...] <flow>}
flowadm set-flowprop [-t] [
-R root-dir] -p
prop=value[,...] flow
flowadm reset-flowprop [-t] [-R
root-dir] [-p prop[,...]]
flow
flowadm show-flowprop [-P] [[-c]
-o field[,...]] [-l
link]
[-p prop[,...]] [
flow]
flowadm help [subcommand-name]
The flowadm command is used to create, modify, remove, and show networking bandwidth, priority, and associated resources for a type of traffic on a particular link.
The flowadm command allows users to manage networking bandwidth resources for a transport, service, or a subnet. The service can be specified as a combination of all or some of the attributes: transport, local port, remote port, local IP address, and remote IP address. The subnet is specified by its IP address and subnet mask. The command can be used on any type of data link, including physical links, virtual NICs, and link aggregations. The flowadm command can not be used on data links that are managed by Open vSwitch. For more information, see dladm(8) man page.
A flow is defined as a set of attributes based on Layer 3 and Layer 4 headers, which can be used to identify a protocol, service, a single connection, or a virtual machine.
A flow filter is defined as a tuple of L3/L4 attributes. A flow can consist of multiple flow filters, stats, and optional resource control properties. All the tuples must have the same signature (combination of attributes).
There are restrictions on valid flow names. The flow names cannot be longer than 95 characters, and they must consist of only alphanumeric (a-z, A-Z, 0-9), underscore ('_'), period ('.'), and hyphen ('-') and must begin with alphabetic characters.
The flowadm command can be used to identify a flow without imposing any bandwidth resource control. This would result in better observability for the flow when used along with flowstat(8).
Flows can be created, modified, and removed in all of global, non-global, and kernel zones. A zone administrator can create a flow only in their zone, global or non-global. However, a flow created in the global zone can migrate to a non-global zone, as described in the following paragraph. An administrator can modify or remove a flow only from within the zone, global or non-global, in which the flow was created. From the global zone, one can view all flows on a system, within the global and any non-global zones. From a non-global zone, one can view only those flows in that zone.
After an administrator creates a flow in the global zone, the data link associated with that flow can be assigned to a non-global zone. In such a case, the associated flow is also assigned to the same non-global zone. When this non-global zone is halted, the data link and its associated flow return to the global zone.
Different zone names distinguish flows of the same name. For example, one can have three flows named fastpak, if each fastpak is in a different zone. For example, zone1/ fastpak, zone2/fastpak, and zone3/fastpak are all valid zone names.
flowadm is implemented as a set of subcommands with corresponding options. Options are described in the context of each subcommand. If flowadm is invoked with no subcommand, then all of the flows configured on the system will be displayed. See EXAMPLES below for more information.
The following subcommands are supported:
Show flow configuration information either for all flows, all flows on a link, or for the specified flow. Output is in ranking order, that is the first flow in the output is searched first for a given packet and so on.
A case-insensitive, comma-separated list of output fields to display. The field name must be one of the fields listed below, or a special value all, to display all fields. For each flow found, the following fields can be displayed:
The name of the flow.
The name of the link the flow is on.
The name of the transport layer protocol to be used.
Local IP address of the flow. If not specified, it will be shown as '--'.
Local port of service for flow.
Remote IP address of the flow. If not specified, it will be shown as '--'.
Remote port of service for flow.
Flow direction. Values are in for inbound only, out for outbound only, or bi for bidirectional.
Differentiated services value for flow and mask used with DSFLD value to state the bits of interest in the differentiated services field of the IP header. This field is not shown in the default flowadm output, but can be displayed with:
flowadm show-flow -o all or flowadm show-flow -o flow,dsfld
IP address of the flow. This can be either local or remote depending on how the flow was defined. This field is deprecated and exists only for backward compatibility. Therefore it will not be shown by default, unless specified with –o option. The users are encouraged to use laddr and raddr instead.
Specifies the PID of the process that created this flow. This field is meaningful only for the system generated flows and will show '--' for user generated flows.
A system generated flow has the prefix "<id>.sys.sock" and is a temporary flow that gets created by applications that call setsockopt() with the SO_FLOW_SLA option.
Display using a stable machine-parseable format.
Display persistent flow property information.
Display information for all flows on the named link or information for the named flow.
Display output for the flow and all its flow filters. Not compatible with the –p option.
Find a flow(s) that is a potential match for the specified list of flow attributes among the existing flows on the system. In case multiple flows are returned, they are shown in the order they are looked up for flow classification of a packet.
Display using a stable machine-parseable format.
Display persistent flow property information.
See –o field description for show-flow subcommand above.
Limit the match to flows on the specified link. If no link is specified, flows on all the links are used.
A comma-separated list of attributes that is used as the key for the lookup for a matching flow(s).
Displays output for the matching flow and all its matching flow filters. Not compatible with the –p option.
Adds a flow to the system. The flow is identified by its flow attributes and properties.
As part of identifying a particular flow, its bandwidth resource and priority can be limited.
The changes are temporary and will not persist across reboots. Persistence is the default.
Specifies an alternate root directory where flowadm should apply persistent creation.
Specify the link to which the flow will be added.
A comma-separated list of attributes to be set to the specified values.
A comma-separated list of properties to be set to the specified values.
Remove an existing flow identified by its link or name.
The changes are temporary and will not persist across reboots. Persistence is the default.
Specifies an alternate root directory where flowadm should apply persistent removal.
If a link is specified, remove all flows from that link. If a single flow is specified, remove only that flow.
Show flow filter information either for all flows, all flows on a link, or for the specified flow. The output is in ranking order, that is, the first flow filter in the output is searched first for a given packet and so on.
See the section for show-flow.
Display using a stable machine-parseable format.
Display information for all flow filters on the named link or the named flow.
Find a filter(s) that is a potential match for the specified list of flow attributes among the existing flows on the system. In case multiple flow filters are returned, they are shown in the order they are looked up for flow classification of a packet.
Display using a stable machine-parseable format.
Display persistent flow property information.
See the –o field description for the show-flow subcommand above.
Limit the match to flows on the specified link. If no link is specified, flows on all the links are used.
A comma-separated list of attributes that is used as the key for the lookup for a matching filter(s).
Add a filter to the specified flow. All the filters for a flow must have the same signature. Currently, only temporary flow filters can be added, and the –t option is required.
A comma-separated list of attributes to be set to the specified values.
Remove the filter from the specified flow. A filter can be removed by specifying either the filter name or its attributes and the flow name.
Note that the name of a filter is generated by the system, and its format is a private interface. The only guarantee is that the filter name from show-filter output will work with remove-filter.
A comma-separated list of attributes to be set to the specified values.
Set values of one or more properties on the flow specified by name. The complete list of properties can be retrieved using the show-flow subcommand.
The changes are temporary and will not persist across reboots. Persistence is the default.
Specifies an alternate root directory where flowadm should apply persistent setting of properties.
A comma-separated list of properties to be set to the specified values.
Resets one or more properties to their default values on the specified flow. If no properties are specified, all properties are reset. See the show-flowprop subcommand for a description of properties, which includes their default values.
Specifies that the resets are temporary. Temporary resets last until the next reboot.
Specifies an alternate root directory where flowadm should apply persistent setting of properties.
A comma-separated list of properties to be reset.
Show the current or persistent values of one or more properties, either for all flows, flows on a specified link, or for the specified flow.
By default, current values are shown. If no properties are specified, all available flow properties are displayed. For each property, the following fields are displayed:
The name of the flow.
The name of the property.
The permission of the property. 'r-' for read-only property and 'rw' for the property that can be both read and written.
The current (or persistent) property value. The value is shown as -- (double hyphen), if it is not set, and ? (question mark), if the value is unknown. Persistent values that are not set or have been reset will be shown as -- and will use the system DEFAULT value (if any).
Effective value of property set by the system. The value is shown as -- (double hyphen), if it is not set, and ? (question mark), if the value is unknown.
The default value of the property. If the property has no default value, -- (double hyphen), is shown.
A comma-separated list of the values the property can have. If the values span a numeric range, the minimum and maximum values might be shown as shorthand. If the possible values are unknown or unbounded, -- (double hyphen), is shown.
Flow properties are documented in the “Flow Properties” section, below.
Display using a stable machine-parseable format.
Display persistent flow property information.
A comma-separated list of properties to show.
Displays all the supported flowadm subcommands or usage for the given subcommand. If you display help for a specific subcommand, the command syntax is displayed, along with an example. Using flowadm help without any argument displays all the subcommands.
The flow operand that identifies a flow in a flowadm command is a comma-separated list of one or more keyword, value pairs from the list below.
Identifies a network flow by the local IP address. value must be a IPv4 address in dotted-decimal notation or an IPv6 address in colon-separated notation. prefix_len is optional.
If prefix_len is specified, it describes the netmask for a subnet address, following the same notation convention of ifconfig(8) and route(8) addresses. If unspecified, the given IP address will be considered as a host address for which the default prefix length for a IPv4 address is /32 and for IPv6 is /128.
Identifies a network flow by the remote IP address. The syntax is the same as local_ip attributes
Identifies a layer 4 protocol to be used. It is typically used in combination with local_port or remote_port to identify the local or remote service that needs special attention.
Identifies a service specified by the local port.
Identifies a service specified by the remote port.
Identifies the flow direction as inbound only, or outbound only, or bidirectional. A flow is treated as bidirectional if the attributes are not specified.
Identifies the 8-bit differentiated services field (as defined in RFC 2474).
The optional dsfield_mask is used to state the bits of interest in the differentiated services field when comparing with the dsfield value. A 0 in a bit position indicates that the bit value needs to be ignored and a 1 indicates otherwise. The mask can range from 0x01 to 0xff. If dsfield_mask is not specified, the default mask 0xff is used. Both the dsfield value and mask must be in hexadecimal.
All combinations of attributes are allowed. This removes the limitation in earlier releases, where only some combinations were allowed.
Flows with different combinations of attributes can be created on a link. This removes the limitation in earlier releases, where all the flows on a given link must have the same combination of flow attributes. Note that when flows with different combinations of attributes are created, they can be either mutually exclusive or they can be overlapping (non exclusive).
In the latter case, the flows are ranked using the number of attributes specified when creating the flow. Flows with the same number of attributes are ranked based on a system default policy. The flowadm show-flow command's output, lists the set of flows on a given link in the ranking order, that is, the first flow in the output is searched first for a given packet and so on.
Note that overlapping of the flows does not imply any nested flows. Each flow is independent and the flow properties like maxbw and the flow statistics are always limited in scope to one flow. For example, packets classified to a TCP socket level flow with 5 attributes are not further classified to a TCP transport flow with 1 attribute.
There are individual flow restrictions and flow restrictions per zone.
Individual Flow RestrictionsAn attribute can be listed only once for each flow. For example, the following command is not valid:
# flowadm add-flow -l vnic1 -a local_port=80,local_port=8080 httpflow
transport and local_port or transport and remote_port:
TCP, UDP, or SCTP flows can be specified with a local address, local_port, remote_address and/or remote_port. An ICMP or ICMPv6 flow that specifies a port is not allowed.
The following commands are valid:
# flowadm add-flow -l e1000g0 -a transport=udp udpflow # flowadm add-flow -l e1000g0 -a transport=tcp,local_port=80 \ udp80flow
The following commands is not valid:
# flowadm add-flow -l e1000g0 -a transport=icmpv6,remote_port=16 \ flow16Flow Restrictions Per Zone
Within a zone, no two flows can have the same name. After adding a flow with the link specified, the link will not be required for display, modification, or deletion of the flow.
The following flow properties are supported. Note that the ability to set a given property to a given value depends on the driver and hardware.
Sets the full duplex bandwidth for the flow. The bandwidth is specified as an integer with one of the scale suffixes(K, M, or G for Kbps, Mbps, and Gbps). If no units are specified, the input value will be read as Mbps. The default is no bandwidth limit.
Sets the priority of the flow. Priority values can take one of 'high', 'medium' and 'low'. The default value of priority is 'medium'.
Setting the token to 'high' on a flow has the effect that packets classified to that flow are processed ahead of packets from normal flows on the same link. Also, the flow is offloaded to the NIC if the NIC has the flow offload capability. A high priority flow may offer a better latency depending on the availability of system resources.
Bandwidth share for a flow is the minimum share of the bandwidth the flow will get when there is competition from other flows on the same data link. Note that the bandwidth is allocated among all the active flows. The amount of allocation is proportional to their share. For example,
# flowadm set-flowprop -p bw-share=40 flow1 # flowadm set-flowprop -p bw-share=10 flow2
Assuming a 1Gbps link, and assuming these two flows, flow1 and flow2 are the only flows, flow1 can have up to 800 Mbps (1Gbps * 40/(40+10)) and flow2 can have up to 200 Mbps (1Gbps * 10/(40+10)).
The above example assumes both the flows have traffic to consume their share of the bandwidth. However, if flow1 consumes only 100Mbps, then flow2 can go up to 900 Mbps. The goal with bandwidth shares is no wasted bandwidth when there is a flow that can use the bandwidth while assuring the allocated share when there is competition from other flows.
This property is currently supported only on certain NICs. dladm show-linkprop -H -p bw-share command can be used to determine if bw-share property is supported on a given link. The value can range from 1 to 100. The value is a relative share value and does not indicate a percentage of the bandwidth. The effective value is printed as a percentage of the physical link bandwidth. This is the minimum percentage of the bandwidth assured to the flow when there is competition. The effective value can keep changing depending on the other flows on the link.
The other flows can be ring-group VNICs, VF-VNIC which has dedicated ring-group hardware resources. For example, datalink has exclusive ring-group vnic1, hardware flows tcpflow1 and udpflow1.
#dladm show-linkprop -pbw-share vnic1 LINK PROPERTY PERM VALUE EFFECTIVE DEFAULT POSSIBLE vnic1 bw-share rw 10 33.33% -- 1-100 #flowadm show-flowprop -pbw-share FLOW PROPERTY PERM VALUE EFFECTIVE DEFAULT POSSIBLE tcpflow1 bw-share rw 10 33.33% -- 1-100 udpflow1 bw-share rw 10 33.33% -- 1-100
Sets the specified DSCP value on all outgoing IP packets for the flow. The valid values are 0 to 63. The value must be specified in decimal. For IPV4, set the six DSCP bits in the DSCP field and for IPv6 set the 6 DSCP bits in the traffic class field. The two ECN bits are untouched. This conforms to RFC 2474.
Note that any IP_TOS value set by an application using setsockopt(3C) is overwritten if the 'dscp' property is set on the flow and the packet is classified to that flow.
Show/specify whether the flow is offloaded to the underlying NIC. The valid values are 'auto', 'on', and 'off' with default being 'auto'. The possible value field shows 'on' if the NIC is capable of offloading, and the value 'auto' means the system decides. The effective value field shows the current value for the NIC offload. The value 'on' means the flow will either be offloaded or flowadm will fail and the value 'off' means the flow will not be offloaded.
The hw-flow property can be specified at any time including the flow creation time. It is highly recommended to leave the hw-flow value to 'auto' (the default) instead of setting it to 'on'. Setting the hw-flow property to 'on' will cause failures in some cases like VNIC migration or VNIC fail over on an DLMP aggregation if the destination port does not support flow offload. Note that setting to hw-flow=off is fine in all cases.
For underlying link which support exclusive ring-group resources, hw-flow means request underlying link allocate dedicated hardware ring-group resources for the flow.
With dedicated ring-group, flow classification can be done by the underlying hardware. This will improve the data path performance and save CPU utilization for flows. In addition to improve resource isolation, flows with dedicated ring-group will benefit from hardware SLA enforcement.
priority and rank are applicable only to shared ring-group flows, so these two properties will not be shown for hardware ring-group flows.
Examples:
#dladm show-linkprop -pring-group net5 LINK PROPERTY PERM VALUE EFFECTIVE DEFAULT POSSIBLE net5 ring-group r- exclusive exclusive -- --
Certain underlying link can only support specific flow attributes to get hardware network flow. See underlying link driver man page for more details.
For example, i40e driver only support the following flow attributes.
# flowadm add-flow -lnet3 -a transport=udp,local_ip=19.0.0.2,local_port=8888 -phw-flow=on udpflow1 # flowadm add-flow -lnet3 -a transport=tcp,local_ip=19.0.0.2,local_port=5000, remote_ip=19.0.0.1,remote_port=5001 -phw-flow=on tcpflow1
Specify the rank for a flow. Valid values for a rank are 1 to 65535.
There is no requirement to set the rank property on all the flows. A flow with the rank property specified is always ahead in lookup order than a flow with no rank specified.
A flow with a low rank value is ahead in lookup order than a flow with a high rank value. It is legal to have two flows with the same rank. In this case, any tie is broken by following the default system policy.
The following command invokes flowadm with no arguments, thereby displaying all flows in the system.
# flowadm FLOW LINK PROTO LADDR LPORT RADDR RPORT DIR tcpflow net0 tcp -- -- -- -- bi udpflow net0 udp -- -- -- -- biExample 2 Creating a Policy Around a Mission-Critical Port
The command below creates a policy around inbound HTTPS traffic on an HTTPS server so that HTTPS obtains dedicated NIC hardware and kernel TCP/IP resources. The name specified, https-1, can be used later to modify or delete the policy.
# flowadm add-flow -l net0 -a transport=TCP,local_port=443 https-1 # flowadm show-flow -l net0 FLOW LINK PROTO LADDR LPORT RADDR RPORT DIR https-1 net0 tcp -- 443 -- -- biExample 3 Modifying an Existing Policy to Add Bandwidth Resource Control
The following command modifies the https-1 policy from the preceding example. The command adds bandwidth control.
# flowadm set-flowprop -p maxbw=500M https-1 # flowadm show-flow https-1 FLOW LINK PROTO LADDR LPORT RADDR RPORT DIR https-1 net0 tcp -- 443 -- -- bi
# flowadm show-flowprop https-1 FLOW PROPERTY PERM VALUE DEFAULT POSSIBLE https-1 maxbw rw 500 -- -- https-1 priority rw medium medium low,medium,high https-1 dscp rw -- -- 0-63 https-1 rank rw -- -- 1-65535 https-1 hw-flow r- off -- on,offExample 4 Limiting the UDP Bandwidth Usage
The following command creates a policy for UDP protocol so that it cannot consume more than 100Mbps of available bandwidth. The flow is named limit-udp-1.
# flowadm add-flow -l net0 -a transport=UDP -p maxbw=100M \
limit-udp-1
Example 5 Setting
Policy to a Flow Defined by Local Address/Port
The following command creates a policy for a TCP flow whose local ip port is 192.168.200.102:443. That is, we want to give special treatment to HTTPS packets such that it is delivered with high priority and maximum bandwidth of 800 Mbps.
# flowadm add-flow -l net0 -a transport=tcp,\
local_ip=192.168.200.102,local_port=443 \
-p priority=high,maxbw=800M my-https
# flowadm show-flow FLOW LINK PROTO LADDR LPORT RADDR RPORT DIR my-https net0 tcp 192.168.200.102 443 -- -- bi
# flowadm show-flowprop FLOW PROPERTY PERM VALUE DEFAULT POSSIBLE my-https maxbw rw 800 -- -- my-https priority rw high medium low,medium,high my-https dscp rw -- -- 0-63 my-https rank rw -- -- 1-65535 my-https hw-flow r- off -- on,offExample 6 Setting Policy to a Flow Defined by Local/Remote Address/Port
The following command creates a policy for a TCP flow whose local ip port is 192.168.200.102:443 and remote ip port is 192.168.200.104:12785. That is, we want to give special treatment to HTTPS packets that are communicating with specific remote ip port. Any packets that belong to this flow will be delivered with high priority. At the same time this flow cannot consume more that 800 Mbps of available bandwidth.
# flowadm add-flow -l net0 -a transport=tcp,\
local_ip=192.168.200.102,local_port=443,\
remote_ip=192.168.200.104,remote_port=12785 \
-p priority=high,maxbw=800M my-flow
# flowadm show-flow FLOW LINK PROTO LADDR LPORT RADDR RPORT DIR my-flow net0 tcp 192.168.200.102 443 192.168.200.104 12785 bi
# flowadm show-flowprop FLOW PROPERTY PERM VALUE DEFAULT POSSIBLE my-flow maxbw rw 800 -- -- my-flow priority rw high medium low,medium,high my-flow dscp rw -- -- 0-63 my-flow rank rw -- -- 1-65535 my-flow hw-flow r- off -- on,offExample 7 Setting Policy, Making Use of dsfield Attribute
The following command sets a policy for EF PHB (DSCP value of 101110 from RFC 2598) with a bandwidth of 500 Mbps. The dsfield value for this flow will be 0x2e (101110) with the dsfield_mask being 0xfc (because we want to ignore the 2 least significant bits).
# flowadm add-flow -l net0 -a dsfield=0x2e:0xfc -p maxbw=500M efphb-flowExample 8 Viewing Flows in Multiple Zones
The following command shows two flows of the same name. The first flow is in the global zone, the second is in the zone zone1. The command is invoked from the global zone, enabling you to view all flows on the system.
# flowadm FLOW LINK PROTO LADDR LPORT RADDR RPORT DIR tcpflow net0 tcp -- -- -- -- bi zone1/tcpflow zone1/net0 tcp -- -- -- -- biExample 9 Getting the Processes That Created a System Flow
The following command shows how to get the pid and the process name, given a system flow.
# flowadm FLOW LINK PROTO LADDR LPORT RADDR RPORT DIR 1.sys.sock net5 tcp 10.1.5.100 51204 10.1.5.101 22 bi
# ps `flowadm show-flow -p -o pid 5.sys.sock`
PID TT S TIME COMMAND
1581 pts/1 T 0:00 ssh 10.1.5.101
The following command shows how to find any flows created by a pid.
# flowadm show-flow -p -o pid,flow | grep 1581
1581:1.sys.sock
Example 10 Setting bandwidth limit on an outbound only flow
The following command shows how to create an outbound flow and set some bandwidth limit on it.
# flowadm add-flow -l net4 -a remote_ip=10.1.5.101,direction=out backup # flowadm set-flowprop -p maxbw=500M backup
All actions were performed successfully.
An error occurred.
See attributes(7) for descriptions of the following attributes:
|
ifconfig(8), attributes(7), acctadm(8), dladm(8), flowstat(8), ifconfig(8), prstat(8), route(8)
The show-usage subcommand, present in previous releases of flowadm, has been replaced by the flowstat(8) –h command.