Overview of Data Center Bridging
Data center bridging is used to manage the bandwidth, relative priority, and flow control of multiple traffic types when sharing the same network link, for example, when sharing a datalink between networking and storage protocols. Fibre channel can be dedicated to host this type of traffic. However, using dedicated links to service only fibre channel traffic can be costly. Therefore, fibre channel over Ethernet (FCoE) is more commonly used. DCB addresses the sensitivity of fibre channels to packet loss while traversing an Ethernet network.
DCB distinguishes traffic based on priorities, which are also called class of service (CoS) priorities. The host and the next hop use the DCB exchange protocol (DCBX) to negotiate a network configuration, such as no traffic loss and minimum bandwidth share, based on priorities. This process enables packets from different applications on the host and in the network to be treated according to their priorities and the corresponding configuration to be negotiated by using DCBX.
Oracle Solaris supports the IEEE 802.1qaz DCBX specification and also the pre-standard Converged Enhanced Ethernet (CEE) DCBX specification v1.01 to enable interoperation with a diverse set of switches when using DCB. For more information about how to choose different versions for negotiations, see Setting the Mode of Operation for DCB.
Each packet in a DCB network has a VLAN header that contains a DCB 3-bit priority value, which is a DCB priority. This IEEE 802.1p priority value differentiates each Ethernet packet in the network from the other packets. Depending on the priority values of the packets, you can configure DCB to allocate specific bandwidth to the packets. For example, all packets with a priority 1 must have PFC enabled and all packets with a priority 2 must have PFC disabled and a bandwidth share of 10%.
You can configure DCB features such as priority-based flow control (PFC) and enhanced transmission selection (ETS) based on priorities. For more information about PFC and ETS, see Priority-Based Flow Control and Enhanced Transmission Selection.
The DCB cos datalink property enables you to specify the CoS or priority of the datalink. The cos value that is set on a primary datalink does not apply to the VNICs that are created over this physical link. For information about customizing PFC based on the cos property, see Customizing Priority-Based Flow Control for DCB. For information about customizing ETS based on the cos property, see Customizing Enhanced Transmission Selection for DCB.
-
PFC prevents packet loss by implementing a mechanism that pauses traffic flow for packets with a defined class of service (CoS). For more information about CoS, see the description of the cos link property in the dladm(1M) man page.
-
ETS enables bandwidth sharing among packets based on the defined CoS. See Enhanced Transmission Selection.
In Oracle Solaris, LLDP is used to exchange DCBX type-length-value (TLV) units. For more information about LLDP, see Exchanging Network Connectivity Information With Link Layer Discovery Protocol. Provided that the underlying network interface card (NIC) supports DCB features such as priority-based flow control and enhanced transmission selection, configuration information for these features can be shared with peers on the network, as follows:
Considerations When Using DCB
-
DCB is only supported on Intel Niantic physical NICs.
To verify whether the NIC supports DCB, issue the following command:
# dladm show-linkprop -p ntcs agent
A property value that is greater than zero (0) indicates that the NIC supports DCB.
-
DCB ports that are configured in DCB mode cannot be aggregated (trunk or DLMP mode).
-
Because DCB supports only the IEEE and CEE versions of DCBX (not the CIN version), external bridges must support the IEEE or CEE version to interoperate with Oracle Solaris DCB.
-
DCB supports ETS configuration and recommendation TLVs.
-
DCB is only supported in the eight traffic class configuration.
-
DCB does not support congestion notification (CN).
Note the following considerations for using DCB:
Priority-Based Flow Control
Priority-based flow control (PFC) extends the standard IEEE 802.3x PAUSE frame to include IEEE 802.1p CoS values. With PFC, instead of halting all traffic on the link when a PAUSE frame is sent, traffic is paused only for those CoS values that are enabled in the PFC frame. A PFC frame is sent for the enabled cos property value for which traffic needs to be paused. The sending host stops traffic for that cos property value while traffic for other disabled cos property values are unaffected. After a time interval that is specified in the PFC frame, transmission resumes for the paused packets.
Pausing based on CoS values ensures that packets are not dropped for that cos property value. For packets without any defined CoS value or with CoS values that do not have PFC enabled, no PAUSE frames are sent. Therefore, traffic continues to flow, and packets might be dropped during traffic congestion. Handling loss of packets depends on the protocol stack, for example, TCP.
Two types of DCB information exist on the host: local DCB information and remote DCB information. For the PFC features to be effective, the local and remote types of DCB information for PFC on the host must be symmetric. The local host must be able to match the DCB information that it receives from the peer. If you enable DCB on your system, DCB synchronizes the DCB information with the peer.
In most cases, the default configuration for PFC is sufficient. This configuration is automatically set up when you enable LLDP. However, you can adjust different options when configuring PFC. For more information, see Customizing Priority-Based Flow Control for DCB and Displaying PFC Configuration Information.
Enhanced Transmission Selection
ETS is a DCB feature that enables you to allocate bandwidth on a NIC to applications based on their DCB priority.
The following figure shows the ETS feature of DCB in a network.
Figure 16 Enhanced Transmission Selection in DCB
The host in the figure has different types of traffic, such as FCoE and iSCSI, that share the link bandwidth. In the figure, the priority and bandwidth as shown in the following table are assigned for different types of traffic.
|
The DCBX ETS TLV with the corresponding ETS bandwidth allocation is as follows:
|
To use the ETS feature, the NIC must support the feature and run in the DCB mode. When you enable LLDP, the default configuration for the ETS feature is automatically set up if the underlying link supports DCB. However, you can modify the default configuration. For more information, see Customizing Enhanced Transmission Selection for DCB, Recommending ETS Configuration to the Peer, and Displaying ETS Configuration Information.