Go to main content

Managing Network Datalinks in Oracle® Solaris 11.4

Exit Print View

Updated: November 2020

Overview of Data Center Bridging

Data center bridging is used to manage the bandwidth, relative priority, and flow control of multiple traffic types when sharing the same network link, for example, when sharing a datalink between networking and storage protocols. Fibre channel can be dedicated to host this type of traffic. However, using dedicated links to service only fibre channel traffic can be costly. Therefore, fibre channel over Ethernet (FCoE) is more commonly used. DCB addresses the sensitivity of fibre channels to packet loss while traversing an Ethernet network.

DCB enables information exchange with the peers about the features that support converged network by using LLDP. The information is related to the configurations affecting the integrity of network packets especially in heavy traffic environments, such as data centers. DCB enables efficient network infrastructure by consolidating storage area network (SAN) and local area network (LAN) and thereby reducing operational and management costs in large deployments.

DCB distinguishes traffic based on priorities, which are also called class of service (CoS) priorities. The host and the next hop use the DCB exchange protocol (DCBX) to negotiate a network configuration, such as no traffic loss and minimum bandwidth share, based on priorities. This process enables packets from different applications on the host and in the network to be treated according to their priorities and the corresponding configuration to be negotiated by using DCBX.

Oracle Solaris supports the IEEE 802.1qaz DCBX specification and also the pre-standard Converged Enhanced Ethernet (CEE) DCBX specification v1.01 to enable interoperation with a diverse set of switches when using DCB. For more information about how to choose different versions for negotiations, see Setting the Mode of Operation for DCB.

Each packet in a DCB network has a VLAN header that contains a DCB 3-bit priority value, which is a DCB priority. This IEEE 802.1p priority value differentiates each Ethernet packet in the network from the other packets. Depending on the priority values of the packets, you can configure DCB to allocate specific bandwidth to the packets. For example, all packets with a priority 1 must have PFC enabled and all packets with a priority 2 must have PFC disabled and a bandwidth share of 10%.

You can configure DCB features such as priority-based flow control (PFC) and enhanced transmission selection (ETS) based on priorities. For more information about PFC and ETS, see Priority-Based Flow Control and Enhanced Transmission Selection.

The DCB cos datalink property enables you to specify the CoS or priority of the datalink. The cos value that is set on a primary datalink does not apply to the VNICs that are created over this physical link. For information about customizing PFC based on the cos property, see Customizing Priority-Based Flow Control for DCB. For information about customizing ETS based on the cos property, see Customizing Enhanced Transmission Selection for DCB.

    In Oracle Solaris, LLDP is used to exchange DCBX type-length-value (TLV) units. For more information about LLDP, see Exchanging Network Connectivity Information With Link Layer Discovery Protocol. Provided that the underlying network interface card (NIC) supports DCB features such as priority-based flow control and enhanced transmission selection, configuration information for these features can be shared with peer hosts on the network, as follows:

  • PFC prevents packet loss by implementing a mechanism that pauses traffic flow for packets with a defined class of service (CoS). For more information about CoS, see the description of the cos link property in the dladm(8) man page.

  • ETS enables bandwidth sharing among packets based on the defined CoS. See Enhanced Transmission Selection.

Considerations When Using DCB

    Note the following considerations for using DCB:

  • DCB is only supported on Intel Niantic physical NICs.

    To verify whether the NIC supports DCB, issue the following command:

    $ dladm show-linkprop -p ntcs agent

    A property value that is greater than zero (0) indicates that the NIC supports DCB.

  • DCB ports that are configured in DCB mode cannot be aggregated (trunk or DLMP mode).

  • Because DCB supports only the IEEE and CEE versions of DCBX (not the CIN version), external bridges must support the IEEE or CEE version to interoperate with Oracle Solaris DCB.

  • DCB supports ETS configuration and recommendation TLVs.

  • DCB is only supported in the eight traffic class configuration.

  • DCB does not support congestion notification (CN).

Priority-Based Flow Control

Priority-based flow control (PFC) extends the standard IEEE 802.3x PAUSE frame to include IEEE 802.1p CoS values. With PFC, instead of halting all traffic on the link when a PAUSE frame is sent, traffic is paused only for those CoS values that are enabled in the PFC frame. A PFC frame is sent for the enabled cos property value for which traffic needs to be paused. The sending host stops traffic for that cos property value while traffic for other disabled cos property values are unaffected. After a time interval that is specified in the PFC frame, transmission resumes for the paused packets.

Pausing based on CoS values ensures that packets are not dropped for that cos property value. For packets without any defined CoS value or with CoS values that do not have PFC enabled, no PAUSE frames are sent. Therefore, traffic continues to flow, and packets might be dropped during traffic congestion. Handling loss of packets depends on the protocol stack, for example, TCP.

Two types of DCB information exist on the host: local DCB information and remote DCB information. For the PFC features to be effective, the local and remote types of DCB information for PFC on the host must be symmetric. The local host must be able to match the DCB information that it receives from the peer. If you enable DCB on your system, DCB synchronizes the DCB information with the peer.

In most cases, the default configuration for PFC is sufficient. This configuration is automatically set up when you enable LLDP. However, you can adjust different options when configuring PFC. For more information, see Customizing Priority-Based Flow Control for DCB and Displaying PFC Configuration Information.

Enhanced Transmission Selection

ETS is a DCB feature that enables you to allocate bandwidth on a NIC to applications based on their DCB priority.

The following figure shows the ETS feature of DCB in a network.

Figure 16  Enhanced Transmission Selection in DCB

image:Graphic shows the enhanced transmission selection feature of DCB

The host in the figure has different types of traffic, such as FCoE and iSCSI, that share the link bandwidth. In the figure, the priority and bandwidth as shown in the following table are assigned for different types of traffic.

Ethernet (non iSCSI) traffic

The DCBX ETS TLV with the corresponding ETS bandwidth allocation is as follows:

Bandwidth Allocation Percentage

To use the ETS feature, the NIC must support the feature and run in the DCB mode. When you enable LLDP, the default configuration for the ETS feature is automatically set up if the underlying link supports DCB. However, you can modify the default configuration. For more information, see Customizing Enhanced Transmission Selection for DCB, Recommending ETS Configuration to the Peer, and Displaying ETS Configuration Information.