Overview of Link Aggregations
The Solaris OS supports the organization of network interfaces into link aggregations. A link aggregation consists of several interfaces on a system that are configured together as a single, logical unit. Link aggregation, also referred to as trunking, is defined in the IEEE 802.3ad Link Aggregation Standard.
The IEEE 802.3ad Link Aggregation Standard provides a method to combine the capacity of multiple full-duplex Ethernet links into a single logical link. This link aggregation group is then treated as though it were, in fact, a single link.
The following are features of link aggregations:
-
Increased bandwidth – The capacity of multiple links is combined into one logical link.
-
Automatic failover/failback – Traffic from a failed link is failed over to working links in the aggregation.
-
Load balancing – Both inbound and outbound traffic is distributed according to user selected load-balancing policies, such as source and destination MAC or IP addresses.
-
Support for redundancy – Two systems can be configured with parallel aggregations.
-
Improved administration – All interfaces are administered as a single unit.
-
Less drain on the network address pool – The entire aggregation can be assigned one IP address.
Link Aggregation Basics
The basic link aggregation topology involves a single aggregation that contains a set of physical interfaces. You might use the basic link aggregation in the following situations:
-
For systems that run an application with distributed heavy traffic, you can dedicate an aggregation to that application's traffic.
-
For sites with limited IP address space that nevertheless require large amounts of bandwidth, you need only one IP address for a large aggregation of interfaces.
-
For sites that need to hide the existence of internal interfaces, the IP address of the aggregation hides its interfaces from external applications.
Figure 6–1 shows an aggregation for a server that hosts a popular web site. The site requires increased bandwidth for query traffic between Internet customers and the site's database server. For security purposes, the existence of the individual interfaces on the server must be hidden from external applications. The solution is the aggregation aggr1 with the IP address 192.168.50.32. This aggregation consists of three interfaces,bge0 through bge2. These interfaces are dedicated to sending out traffic in response to customer queries. The outgoing address on packet traffic from all the interfaces is the IP address of aggr1, 192.168.50.32.
Figure 6–1 Basic Link Aggregation Topology
Figure 6–2 depicts a local network with two systems, and each system has an aggregation configured. The two systems are connected by a switch. If you need to run an aggregation through a switch, that switch must support aggregation technology. This type of configuration is particularly useful for high availability and redundant systems.
In the figure, System A has an aggregation that consists of two interfaces, bge0 and bge1. These interfaces are connected to the switch through aggregated ports. System B has an aggregation of four interfaces, e1000g0 through e1000g3. These interfaces are also connected to aggregated ports on the switch.
Figure 6–2 Link Aggregation Topology With a Switch
Back-to-Back Link Aggregations
The back-to-back link aggregation topology involves two separate systems that are cabled directly to each other, as shown in the following figure. The systems run parallel aggregations.
Figure 6–3 Basic Back-to-Back Aggregation Topology
In this figure, device bge0 on System A is directly linked to bge0 on System B, and so on. In this way, Systems A and B can support redundancy and high availability, as well as high-speed communications between both systems. Each system also has interface ce0 configured for traffic flow within the local network.
The most common application for back-to-back link aggregations is mirrored database servers. Both servers need to be updated together and therefore require significant bandwidth, high-speed traffic flow, and reliability. The most common use of back-to-back link aggregations is in data centers.
Policies and Load Balancing
If you plan to use a link aggregation, consider defining a policy for outgoing traffic. This policy can specify how you want packets to be distributed across the available links of an aggregation, thus establishing load balancing. The following are the possible layer specifiers and their significance for the aggregation policy:
-
L2 – Determines the outgoing link by hashing the MAC (L2) header of each packet
-
L3 – Determines the outgoing link by hashing the IP (L3) header of each packet
-
L4 – Determines the outgoing link by hashing the TCP, UDP, or other ULP (L4) header of each packet
Any combination of these policies is also valid. The default policy is L4. For more information, refer to the dladm(1M) man page.
Aggregation Mode and Switches
If your aggregation topology involves connection through a switch, you must note whether the switch supports the link aggregation control protocol (LACP). If the switch supports LACP, you must configure LACP for the switch and the aggregation. However, you can define one of the following modes in which LACP is to operate:
-
Off mode – The default mode for aggregations. LACP packets, which are called LACPDUs are not generated.
-
Active mode – The system generates LACPDUs at regular intervals, which you can specify.
-
Passive mode – The system generates an LACPDU only when it receives an LACPDU from the switch. When both the aggregation and the switch are configured in passive mode, they cannot exchange LACPDUs.
See the dladm(1M) man page and the switch manufacturer's documentation for syntax information.
Requirements for Link Aggregations
Your link aggregation configuration is bound by the following requirements:
-
You must use the dladm command to configure aggregations.
-
An interface that has been plumbed cannot become a member of an aggregation.
-
All interfaces in the aggregation must run at the same speed and in full-duplex mode.
-
You must set the value for MAC addresses to “true” in the EEPROM parameter local-mac-address? For instructions, refer to How to Ensure That the MAC Address of an Interface Is Unique.
Certain devices do not fulfill the requirement of the IEEE 802.3ad Link Aggregation Standard to support link state notification. This support must exist in order for a port to attach to an aggregation or to detach from an aggregation. Devices that do not support link state notification can be aggregated only by using the -f option of the dladm create-aggr command. For such devices, the link state is always reported as UP. For information about the use of the -f option, see How to Create a Link Aggregation.
Flexible Names for Link Aggregations
Flexible names can be assigned to link aggregations. Any meaningful name can be assigned to a link aggregation. For more information about flexible or customized names, see Assigning Names to Data Links. Previous Solaris releases identify a link aggregation by the value of a key that you assign to the aggregation. For an explanation of this method, see Overview of Link Aggregations. Although that method continues to be valid, preferably, you should use customized names to identify link aggregations.
Similar to all other data-link configurations, link aggregations are administered with the dladm command.
How to Create a Link Aggregation
Before You Begin
Note –
Link aggregation only works on full-duplex, point-to-point links that operate at identical speeds. Make sure that the interfaces in your aggregation conform to this requirement.
If you are using a switch in your aggregation topology, make sure that you have done the following on the switch:
-
Configured the ports to be used as an aggregation
-
If the switch supports LACP, configured LACP in either active mode or passive mode
-
Assume the Primary Administrator role, or become superuser.
The Primary Administrator role includes the Primary Administrator profile. To create the role and assign the role to a user, see Chapter 2, Working With the Solaris Management Console (Tasks), in System Administration Guide: Basic Administration.
-
Display the network data-link information.
# dladm show-link
-
Make sure that the link over which you are creating the aggregation is not opened by any application.
For example, if the IP interface over the link is plumbed, then unplumb the interface.
# ifconfig interface unplumb
where interface refers to the IP interface that is plumbed and using the link.
-
Create a link aggregation.
# dladm create-aggr [-f] -l link1 -l link2 [...] aggr
- -f
-
Forces the creation of the aggregation. Use this option when you are attempting to aggregate devices that do not support link state notification.
- linkn
-
Specifies the data links that you want to aggregate.
- aggr
-
Specifies the name that you want to assign to the aggregation.
-
Plumb and configure an IP interface over the newly created aggregation.
# ifconfig interface plumb IP-address up
where interface takes the name of the aggregation.
-
Check the status of the aggregation you just created.
The aggregation's state should be UP.
# dladm show-aggr
-
(Optional) Make the IP configuration of the link aggregation persist across reboots.
-
Create the /etc/hostname file for the aggregation's interface.
If the aggregation contains IPv4 addresses, the corresponding hostname file is /etc/hostname.aggr. For IPv6–based link aggregations, the corresponding hostname file is /etc/hostname6.aggr.
-
Type the IPv4 or IPv6 address of the link aggregation into the file.
-
Perform a reconfiguration boot.
# reboot -- -r
-
Example 6–1 Creating a Link Aggregation
This example shows the commands that are used to create a link aggregation with two data links, subvideo0 and subvideo1. The configuration is persistent across system reboots.
# dladm show-link LINK CLASS MTU STATE OVER subvideo0 phys 1500 up ---- subvideo1 phys 1500 up ---- # dladm create-aggr -l subvideo0 -l subvideo1 video0 # ifconfig video0 plumb 10.8.57.50/24 up # dladm show-aggr LINK POLICY ADDRPOLICY LACPACTIVITY LACPTIMER FLAGS video0 L4 auto off short ----- # echo 10.8.57.50/24 > /etc/hostname.video0 # reboot -- -r |
When you display link information, the link aggregation is included in the list.
# dladm show-link LINK CLASS MTU STATE OVER subvideo0 phys 1500 up ---- subvideo1 phys 1500 up ---- video0 aggr 1500 up subvideo0, subvideo1 |
How to Modify an Aggregation
This procedure shows how to make the following changes to an aggregation definition:
-
Modifying the policy for the aggregation
-
Changing the mode for the aggregation
-
Assume the System Administrator role.
The System Administrator role includes the Network Management profile. To create the role and assign the role to a user, see Chapter 9, Using Role-Based Access Control (Tasks), in System Administration Guide: Security Services.
-
Modify the policy of the aggregation.
# dladm modify-aggr -P policy-key aggr
- policy-key
-
Represents one or more of the policies L2, L3, and L4, as explained in Policies and Load Balancing.
- aggr
-
Specifies the aggregation whose policy you want to modify.
-
Modify the LACP mode of the aggregation.
# dladm modify-aggr -L LACP-mode -T timer-value aggr
- -L LACP-mode
-
Indicates the LACP mode in which the aggregation is to run. The values are active, passive, and off. If the switch runs LACP in passive mode, be sure to configure active mode for your aggregation.
- -T timer-value
-
Indicates the LACP timer value, either short or long.
Example 6–2 Modifying a Link Aggregation
This example shows how to modify the policy of aggregation video0 to L2 and then turn on active LACP mode.
# dladm modify-aggr -P L2 video0 # dladm modify-aggr -L active -T short video0 # dladm show-aggr LINK POLICY ADDRPOLICY LACPACTIVITY LACPTIMER FLAGS video0 L2 auto active short ----- |
How to Add a Link to an Aggregation
-
Assume the System Administrator role or become superuser.
The System Administrator role includes the Network Management profile. To create the role and assign the role to a user, see Chapter 9, Using Role-Based Access Control (Tasks), in System Administration Guide: Security Services.
-
Ensure that the link you want to add has no IP interface that is plumbed over the link.
# ifconfig interface unplumb
-
Add the link to the aggregation.
# dladm add-aggr -l link [-l link] [...] aggr
where link represents a data link that you are adding to the aggregation.
-
Perform other tasks to modify the entire link aggregation configuration after more data links are added.
For example, in the case of a configuration that is illustrated in Figure 6–3, you might need to add or modify cable connections and reconfigure switches to accommodate the additional data links. Refer to the switch documentation to perform any reconfiguration tasks on the switch.
Example 6–3 Adding a Link to an Aggregation
This example shows how to add a link to the aggregation video0.
# dladm show-link LINK CLASS MTU STATE OVER subvideo0 phys 1500 up ---- subvideo1 phys 1500 up ---- video0 aggr 1500 up subvideo0, subvideo1 net3 phys 1500 unknown ---- # dladm add-aggr -l net3 video0 # dladm show-link LINK CLASS MTU STATE OVER subvideo0 phys 1500 up ---- subvideo1 phys 1500 up ---- video0 aggr 1500 up subvideo0, subvideo1, net3 net3 phys 1500 up ---- |
How to Remove a Link From an Aggregation
-
Assume the System Administrator role.
The System Administrator role includes the Network Management profile. To create the role and assign the role to a user, see Chapter 9, Using Role-Based Access Control (Tasks), in System Administration Guide: Security Services.
-
Remove a link from the aggregation.
# dladm remove-aggr -l link aggr-link
Example 6–4 Removing a Link From an Aggregation
This example shows how to remove a link from the aggregation video0.
dladm show-link LINK CLASS MTU STATE OVER subvideo0 phys 1500 up ---- subvideo1 phys 1500 up ---- video0 aggr 1500 up subvideo0, subvideo1, net3 net3 phys 1500 up ---- # dladm remove-aggr -l net3 video0 # dladm show-link LINK CLASS MTU STATE OVER subvideo0 phys 1500 up ---- subvideo1 phys 1500 up ---- video0 aggr 1500 up subvideo0, subvideo1 net3 phys 1500 unknown ---- |
How to Delete an Aggregation
-
Assume the System Administrator role.
The System Administrator role includes the Network Management profile. To create the role and assign the role to a user, see Chapter 9, Using Role-Based Access Control (Tasks), in System Administration Guide: Security Services.
-
Unplumb the aggregation.
# ifconfig aggr unplumb
-
Delete the aggregation.
# dladm delete-aggr aggr
-
To make the deletion persistent, remove the IP configuration for the link aggregation in /etc/hostname.interface file.
# rm /etc/hostname.interface
Example 6–5 Deleting an Aggregation
This example deletes the aggregation video0. The deletion is persistent.
# ifconfig video0 unplumb # dladm delete-aggr video0 # rm /etc/hostname.video0 |
How to Configure VLANs Over
a Link Aggregation
In the same manner as configuring VLANs over an interface, you can also create VLANs on a link aggregation. VLANs are described in Chapter 5, Administering VLANs. This section combines configuring VLANs and link aggregations.
Before You Begin
Create the link aggregation first and configure it with a valid IP address. To create link aggregations, refer to How to Create a Link Aggregation.
-
List the aggregations that are configured in the system.
# dladm show-link
-
Create a VLAN over the link aggregation.
# dladm create-vlan -l link -v VID vlan-link
where
- link
-
Specifies the link on which the VLAN interface is being created. In this specific case, the link refers to the link aggregation.
- VID
-
Indicates the VLAN ID number
- vlan-link
-
Specifies the name of the VLAN, which can also be an administratively-chosen name.
-
Repeat Step 2 to create other VLANs over the aggregation.
-
Configure IP interfaces over the VLANs with valid IP addresses.
-
To create persistent VLAN configurations, add the IP address information to the corresponding /etc/hostname.interface configuration files.
The interface takes the name of the VLAN that you assigned.
Example 6–6 Configuring Multiple VLANs Over a Link Aggregation
In this example, two VLANs are configured on a link aggregation. The VLANs are assigned VIDs 193 and 194, respectively.
# dladm show-link LINK CLASS MTU STATE OVER subvideo0 phys 1500 up ---- subvideo1 phys 1500 up ---- video0 aggr 1500 up subvideo0, subvideo1 # dladm create-vlan -l video0 -v 193 salesregion1 # dladm create-vlan -l video0 -v 194 salesregion2 # ifconfig salesregion1 192.168.10.5/24 plumb up # ifconfig salesregion2 192.168.10.25/24 plumb up # vi /etc/hostname.salesregion1 192.168.10.5/24 # vi /etc/hostname.salesregion2 192.168.10.25/24 |
Combining Network Configuration Tasks While Using Customized Names
This section provides an example that combines all the procedures in the previous chapters about configuring links, VLANs, and link aggregations while using customized names. For a description of other networking scenarios that use customized names, see the article in http://www.sun.com/bigadmin/sundocs/articles/vnamingsol.jsp.
Example 6–7 Configuring Links, Aggregations, and VLANs
In this example, a system that consists of 4 NICs needs to be configured to be a router for 8 separate subnets. To attain this objective, 8 links will be configured, one for each subnet. First, a link aggregation is created on all 4 NICs. This untagged link becomes the default untagged subnet for the network to which the default route points.
Then VLAN interfaces are configured over the link aggregation for the other subnets. The subnets are named by basing on a color-coded scheme. Accordingly, the VLAN names are likewise named to correspond to their respective subnets. The final configuration consists of 8 links for the eight subnets: 1 untagged link, and 7 tagged VLAN links.
To make the configurations persist across reboots, the same procedures apply as in previous Solaris releases. For example, IP addresses need to be added to configuration files like /etc/inet/ndpd.conf or /etc/hostname.interface. Or, filter rules for the interfaces need to be included in a rules file. These final steps are not included in the example. For these steps, refer to the appropriate chapters in System Administration Guide: IP Services, particularly TCP/IP Administration and DHCP.
# dladm show-link LINK CLASS MTU STATE OVER nge0 phys 1500 up -- nge1 phys 1500 up -- e1000g0 phys 1500 up -- e1000g1 phys 1500 up -- # dladm show-phys LINK MEDIA STATE SPEED DUPLEX DEVICE nge0 Ethernet up 1000Mb full nge0 nge1 Ethernet up 1000Mb full nge1 e1000g0 Ethernet up 1000Mb full e1000g0 e1000g1 Ethernet up 1000Mb full e1000g1 # ifconfig nge0 unplumb # ifconfig nge1 unplumb # ifconfig e1000g0 unplumb # ifconfig e1000g1 unplumb # dladm rename-link nge0 net0 # dladm rename-link nge1 net1 # dladm rename-link e1000g0 net2 # dladm rename-link e1000g1 net3 # dladm show-link LINK CLASS MTU STATE OVER net0 phys 1500 up -- net1 phys 1500 up -- net2 phys 1500 up -- net3 phys 1500 up -- # dladm show-phys LINK MEDIA STATE SPEED DUPLEX DEVICE net0 Ethernet up 1000Mb full nge0 net1 Ethernet up 1000Mb full nge1 net2 Ethernet up 1000Mb full e1000g0 net3 Ethernet up 1000Mb full e1000g1 # dladm create-aggr -P L2,L3 -l net0 -l net1 -l net2 -l net3 default0 # dladm show-link LINK CLASS MTU STATE OVER net0 phys 1500 up -- net1 phys 1500 up -- net2 phys 1500 up -- net3 phys 1500 up -- default0 aggr 1500 up net0 net1 net2 net3 # dladm create-vlan -v 2 -l default0 orange0 # dladm create-vlan -v 3 -l default0 green0 # dladm create-vlan -v 4 -l default0 blue0 # dladm create-vlan -v 5 -l default0 white0 # dladm create-vlan -v 6 -l default0 yellow0 # dladm create-vlan -v 7 -l default0 red0 # dladm create-vlan -v 8 -l default0 cyan0 # dladm show-link LINK CLASS MTU STATE OVER net0 phys 1500 up -- net1 phys 1500 up -- net2 phys 1500 up -- net3 phys 1500 up -- default0 aggr 1500 up net0 net1 net2 net3 orange0 vlan 1500 up default0 green0 vlan 1500 up default0 blue0 vlan 1500 up default0 white0 vlan 1500 up default0 yellow0 vlan 1500 up default0 red0 vlan 1500 up default0 cyan0 vlan 1500 up default0 # dladm show-vlan LINK VID OVER FLAGS orange0 2 default0 ----- green0 3 default0 ----- blue0 4 default0 ----- white0 5 default0 ----- yellow0 6 default0 ----- red0 7 default0 ----- cyan0 8 default0 ----- # ifconfig orange0 plumb ... # ifconfig green0 plumb ... # ifconfig blue0 plumb ... # ifconfig white0 plumb ... # ifconfig yellow0 plumb ... # ifconfig red0 plumb ... # ifconfig cyan0 plumb ... |
- © 2010, Oracle Corporation and/or its affiliates