| Sun Trunking 1.3 Installation and User's Guide |    
|
| Sun Trunking 1.3 Installation and User's Guide |
|
| C H A P T E R 3 |
|
Configuring the Sun Trunking 1.3 Software |
This chapter describes how to edit and create the Sun Trunking 1.3 configuration files. This chapter includes the following sections:
Before rebooting your system, you can edit and create configuration files to define how you will link the Ethernet network interfaces. To configure these files, you need to know the following for each Ethernet adapter:
Always check to be sure the link mode is set to run full-duplex.
Use the kstats (See the kstats manual page: kstats(1M)) or the ndd command to verify link mode. If the information returned indicates that your trunking device is not running at full-duplex, refer to your networking device product documentation for information on setting the link mode.
Once you have installed the trunking software and connected the Ethernet interfaces to a switch that is configured for trunking, you cannot use that interface to boot from the network.
Each network interface on an Ethernet adapter has a corresponding instance number. You must know the device instance numbers for each network interface before you can configure the Sun Trunking software files.
You can find the instance numbers by searching the /etc/path_to_inst file.
The file contains the physical name, the instance number, and driver name for each device on the system. By searching this file, you can find the instance numbers that will be used with the Sun Trunking 1.3 software.
Using the grep command, search the path_to_inst file for all of the devices on your system:
In the previous example, instances 0 and 1 are GigaSwift Ethernet adapters. Your information will be similar if you are using a different adapter. TABLE 3-1 lists the network interface number, physical name, and instance number for each GigaSwift Ethernet instance on this example system.
You will need to know the instance numbers of the GigaSwift Ethernet network interfaces in order to configure the Sun Trunking software.
|
Note - You cannot trunk an interface that is already plumbed. To display a list of all plumbed interfaces, execute ifconfig -a command. |
1. Before trunking ensure that the links are available:
The output from the ifconfig -a command in the previous example shows that ce0 is already plumbed, and therefore, cannot be aggregated.
2. Unplumb the interface as follows:
# ifconfig ce0 unplumb # ifconfig -a lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 |
3. Setup the trunk and plumb the interface:
# nettr -setup 10 device=ce members=0,1,2,3 policy=2 Aggregation (key) : 10 ce0 ce1 ce2 ce3 # ifconfig ce0 plumb 199.99.10.10 up |
Trunk members can be accessed only through the trunk head. Since trunking appears to the system as a single "fat pipe," the trunk head is the single access point. Other links in an aggregation are not plumbed beneath TCP/IP and are not visible. Networking commands such as ifconfig(1M) and netstat(1M) work only on the head instance.
You cannot do a DLPI attach on a non-head member. The following error message was returned for a snoop command for a non-head member:
In this example, ge1 is a member of a trunk. The trunk head for that trunk is ge0. To use the snoop command, for example, in such a case, the following would be the correct usage:
TABLE 3-2 shows a matrix of supported drivers and the maximum number of supported links for each:
The two network interfaces of the GigaSwift Ethernet (ce) adapter can be linked as follows.
FIGURE 3-1 GigaSwift Ethernet Trunking Network Configuration Example
The two network ports of the GigabitEthernet (ge) adapter can be linked as follows.
FIGURE 3-2 GigabitEthernet Trunking Network Configuration Example
The four network ports of the Quad FastEthernet (qfe) adapter can be linked in a number of ways, depending on the needs of your network. All four of the network ports can be linked into one trunk, or two network ports can be linked into one trunk. FIGURE 3-3 shows four different common linking possibilities.
FIGURE 3-3 Quad FastEthernet Trunking Network Configuration Examples
The configuration of the Sun Trunking 1.3 software files depends on how you want to organize the Ethernet network. You must also configure the Ethernet switch software to be symmetrical with the way you configured the Sun Trunking 1.3 software. Refer to your Ethernet switch documentation for the switch configuration instructions.
Each Ethernet adapter has been assigned a unique Media Access Control (MAC) address, which represents the 48-bit Ethernet address for that network interface. Each host system also provides a single global MAC address. For historical reasons, the default MAC address for each interface is that of the single global host MAC address rather than the MAC address of each individual Ethernet adapter. This must be modified for trunking. You can set the OpenBoot firmware local-mac-address property.
The mac-address property of the network device specifies the network address (system-wide or local-mac-address?) used for booting the system. To start using the MAC addresses assigned to the network interface of the Ethernet adapter, set the NVRAM configuration variable local-mac-address? to true.
Client machines with multiple interfaces that communicate with the trunked machine should also set local-mac-address? to true.
After locating the instance numbers of the Ethernet network interfaces, and deciding how you want to organize your network, you can begin to configure the Sun Trunking 1.3 software.
|
This section describes the minimum set of instructions to configure trunking. For more details, see Configuring Trunking with the nettr Utility.
The main configuration file of the Sun Trunking 1.3 software is the
/etc/opt/SUNWconn/bin/nettr.sh file. You will define each trunk by adding information to this file. Before you edit the nettr.sh file, you must know how many Ethernet network interfaces you want to link into a trunk, the trunking policy and the lacp mode of operation. You can also add new members to an existing trunk or delete an existing member from the trunk.
Using a text editor, add commands to the nettr.sh file to define your trunking network.
In the nettr.sh file, add the following nettr command to configure a ce trunk. Use the following format to set up a trunk:
./nettr -setup <key> device=<dev1> members=<instance,...> [ { device=<dev2> members=<instnace,...> }... ] [ policy=<policy-no> ] [ mac=<a:b:c:d:e:f> ] [ lacp=<A|a|P|p|O|o> [ ptimer=<0|1> ] ] |
In the nettr.sh file, add the following nettr command to configure a ge or qfe trunk. Use the following format to set up a trunk:
In the previous command, the key is the aggregation identifier for (ce devices) head-inst is the aggregation identifier for qfe or ge devices. The device represents the adapter type (qfe for a Quad FastEthernet adapter or ce for GigaSwift Ethernet adapter, or ge for Gigabit Ethernet). The member option is used to select members of a trunk. The policy option can either be the default value of 1 for MAC or 2 for round robin, 3 for IP Destination, or 4 for IP Source/IP Destination. For the lacp option, specify lacp mode of operation. It can be Active (A|a), Passive (P| p), or Off (O| o). For the ptimer option, specify a short(0) or long(1) lacp timeout value.
|
Note - The following examples use the instance numbers found in TABLE 3-1. The instance numbers on your system will be different. See Determining Instance Numbers for more information. |
It will be similar to the following:
|
Note - The command syntax of Sun Trunking 1.3 is fully compatible with Trunking 1.2.1. Existing nettr.sh files do not need any modifications. |
# nettr -run Configuring Sun Trunking devices Aggregation (key) : 10 ce0 ce1 Aggregation (key) : 20 ce2 ce3 |
|
Before you can use these trunked Ethernet network interfaces, you must create an
/etc/hostname.driver# file.
Where driver = ce for GigaSwift Ethernet, ge for Gigabit Ethernet, or qfe for Quad FastEthernet
Then add an entry in the /etc/hosts file for each trunk.
1. For each trunk, create a /etc/hostname.ce# file, where # corresponds to the instance number used as the trunk head.
For example, if you had two trunks using two network interfaces each, you would have to create two files containing the host name of the trunk as seen by network. The extensions of these files would have to correspond with the trunk head interface numbers of the trunks.
|
GigaSwift Ethernet Adapter Network Interfaces Used in the Trunk1 |
|
|---|---|
|
1 See FIGURE 3-1 for more information. |
|
Using the example in TABLE 3-3, the following example shows the three /etc/hostname.device# files required for a system called proboscis, which has an onboard bge device (proboscis), and two trunks (proboscis-11, proboscis-12).
# cat /etc/hostname.bge0 proboscis # cat /etc/hostname.ce0 proboscis-11 # cat /etc/hostname.ce2 proboscis-11 |
2. Create an appropriate entry in the /etc/hosts file for each trunk.
Using the example from Step 1, you would have:
# cat /etc/hosts # # Internet host table # 127.0.0.1 localhost 129.144.10.57 proboscis loghost 129.144.11.83 proboscis-11 129.144.12.41 proboscis-12 |
|
Note - Make sure you configure the switch software to be symmetrical with the Sun Trunking 1.3 software. Refer to your Ethernet switch documentation for instructions. |
|
Setup the ce adapter using the ifconfig command, where ip_address corresponds to the system IP address.
Create an entry in the /etc/hosts file for each active ce interface.
The following example shows the /etc/hosts file.
# cat /etc/hosts # # Internet host table # 127.0.0.1 localhost 129.144.10.57 proboscis loghost 129.144.11.83 proboscis-11 |
|
Note - Make sure you configure the switch software to be symmetrical with the Sun Trunking 1.3 software. Refer to your Ethernet switch documentation for instructions. |
|
If you do not use ifconfig, you will need to reboot your system to make these trunked networks active. If you did use the previous ifconfig instructions, you can ignore this section.
2. Set local-mac-address? to true.
3. Perform a reconfiguration boot.
You can verify trunking networks using the ifconfig and the nettr commands.
Use the ifconfig -a command to print out the addressing information for each interface on the system.
Even though a trunk may be composed of two linked GigaSwift Ethernet network interfaces, the ifconfig -a command will only print out the addressing information for the plumbed interface (ce0 in the example above).
For a more complete listing of the ce devices, use the nettr -conf command. This command prints out a list of the trunked ce instances on the system.
The nettr -conf command will also show the policy of each trunk on the system, as well as listing the original MAC address of each interface.
For qfe or ge, use the nettr -stat trunkhead device=device command to monitor the network statistics of each interface on the system.
trunkhead is the trunk head interface number of the trunk you want to monitor
This command will list each interface and the network performance statistics, of the specified trunk. See Configuring Trunking with the nettr Utility for more information.
You can also monitor the network statistics of a trunk at regular intervals. The full usage of the command is nettr -stats key interval=sec, with sec being the number of seconds between monitoring the trunk's interfaces. In the example below, the trunk is monitored once every second.
You can also use the netstat(1M)or kstat(1M) command to monitor the network statistics. Refer to the netstat(1M) and kstat(1M) man pages for more information.
To permanently disable the trunking interface, comment out the nettr command in the /etc/opt/SUNWconn/bin/nettr.sh file. You will need to reconfigure the
/etc/hostname.ce# and /etc/hosts files if you want to use the Ethernet network interfaces separately. Refer to the product documentation for more information.
You can use the nettr(1M) utility to configure trunking. Use nettr(1M) to:
Following is the command syntax for nettr.
Use the following format to enable the trunking interface for ce devices:
nettr -setup <key> device=<dev1> members=<instance,...> [ { device=<dev2> members=<instnace,...> }... ] [ policy=<policy-no> ] [ mac=<a:b:c:d:e:f> ] [ lacp=<A|a|P|p|O|o> [ ptimer=<0|1> ] ] |
The key is the aggregation identifier and can be any number you choose. It does not have to be tied to a member instance. The device represents the adapter type (ce). The member option is used to select members of a trunk. The policy option can either be the default value of 1 for MAC or 2 for round robin, 3 for IP Destination, or 4 for IP Source/IP Destination. For the lacp option specify lacp mode of operation. It can be Active(A|a), Passive(P| p), or Off(O| o). For the ptimer option, specify a short(0) or long(1) lacp timeout value.
For ge and qfe devices the format of this command is:
In the previous command, the head instance is the instance number of the first linked network interface of the trunk. The device represents the adapter type (qfe for a Quad FastEthernet adapter or ge for GigabitEthernet adapter). The member option is used to select members of a trunk. The policy option can either be the default value of 1 for MAC or 2 for round robin, 3 for IP Destination, or 4 for IP Source/IP Destination. If you want to use the MAC trunking policy and two links for a trunk, execute the following line:
You can verify trunking networks using the ifconfig and the nettr commands.
Use the ifconfig -a command to print out the addressing information for each interface on the system.
Even though a trunk can be composed of two linked GigaSwift Ethernet network interfaces, the ifconfig -a command will only print out the addressing information for the key head interfaces (ce0 in the previous example).
Use the nettr -conf command to print out a list of trunked ce instances on the system.
In the previous example, the ce0 interface is the key for a two-network interface trunk, composed of the ce0, and ce1 interfaces. The nettr -conf command also shows the policy of each trunk on the system and lists the original MAC address of each interface.
Use the nettr -stat key device=device command to monitor the network statistics of each interface on the system.
key is the trunk head interface number of the trunk you want to monitor
# /etc/opt/SUNWconn/bin/nettr -stats 0 Mar 30 15:26:58 2003 Name Ipkts Ierrs Opkts Oerrs Collis Crc %Ipkts %Opkts ge0 66518099 0 67498218 0 0 0 50.00 50.00 ge1 66527490 0 67498218 0 0 0 50.00 50.00 |
The nettr -stats command will list each interface and the network performance statistics, summarized in TABLE 3-4, of the specified trunk.
You can also monitor the network statistics of a trunk at regular intervals. For a Sun GigaSwift Ethernet driver, the full usage of the command is nettr -stats key device=[ce] interval=sec, with sec being the number of seconds between monitoring the trunk's interfaces. In the example below, the trunk is monitored once every second.
To monitor the network statistics of a trunk for a Sun Gigabit Ethernet driver or a Sun Quad FastEthernet driver, the full usage of the command is nettr -stats trunkhead device=[ge|qfe] interval=sec, with sec being the number of seconds between monitoring the trunk's interfaces. In the example below, the trunk is monitored once every second.
You can also use the netstat(1M)or kstat(1M) command to monitor the network statistics. Refer to the netstat(1M) and kstat(1M) man pages for more information.
If you need to add a new member to the trunk, use -add command.
With the new member added to the trunk, the output from the -conf command would look similar to the following:
If you need to remove a member from the trunk, use -rem command.
The previous command removes ce1 from the trunk. The output from the -conf command would look similar to the following:
If you need to disable a trunk, use the nettr -release key command, with key being the trunk identifier.
nettr automatically unplumbs IPv4 and IPv6 stacks. If any other application (such as VLAN) are plumbed, remember to manually unplumb the application.
The four supported trunking policies used in the Sun Trunking 1.3 software are MAC, Round Robin, IP Destination, and IP Source+Destination. With these policies, if a link fails, the traffic failsover to the next available link. The policies are defined below.
|
Note - Do not use MAC-hashing for connecting two servers back to back. |
The primary use of the IP Source/IP Destination Address policy occurs where you use the IP virtual address feature to give multiple IP addresses to a single physical interface. For example, you might have a cluster of servers providing network services, in which each service is associated with a virtual IP address over a given interface. If a service associated with an interface fails, the virtual IP address migrates to a physical interface on a different machine in the cluster. In such an arrangement, the IP Source Address/IP Destination Address policy gives you a greater chance of using more, different links within the trunk than would the IP Destination Address policy.
The -hash option to the nettr (1M) command enables you to determine over which link a given packet will travel. The following section describes hashing in greater detail.
|
# /etc/opt/SUNWconn/bin/nettr -hash 0 device=ce ip=199.100.1.29/199.100.1.3 Name Key Policy Source Address Dest Address Link to Transmit ce0 ce0 4 199.100.1.29 199.100.1.3 0 |
The -hash option to the nettr (1M) command enables you to determine over which link a given packet will travel. For example, if you use the Destination IP Address policy, you can supply an IP address to a nettr -hash command to determine over which link a packet with the specified IP address will travel.
Sun Trunking 1.3 operates as a virtual network adapter that is visible to the system through the head or key instance. Applications will work transparently as though they are on a single link, yet experience the benefits of increased bandwidth and automatic failover. On applications that deal with a physical NIC, such as DR, you must simply unaggregate the trunk prior to performing those functions, then reaggregate the trunk when you are finished.
Sun Trunking 1.3 works in the static mode with 802.3ad switches. For a link to be part of an aggregation it must be running at full-duplex and at the same speed. The endpoints of the links must be terminated at the same switch or system.
Sun Trunking 1.3 also supports dynamic mode providing a fully compliant IEEE 802.3ad implementation. This includes the LACP protocol and associated Marker Responder. For more information on the LACP protocol, see Appendix B.
IP Multipathing (IPMP) works transparently with Sun Trunking 1.3. Configure two aggregations (two trunk-heads) and group these two aggregations into one IPMP group.
|
Note - IPMP failover may not happen if one of the links in an aggregation fails because Sun Trunking 1.3 software will failover to the next available link in the aggregation. |
|
Dynamic Reconfiguration (DR) works transparently with Sun Trunking 1.3. Here is how it works:
1. Unconfigure the aggregation:
2. Unconfigure the Ethernet adapter:
3. Change the Ethernet adapter.
4. Configure the Ethernet adapter:
5. Reconfigure the aggregation:
Sun Trunking 1.3 works transparently with 802.1q, but it is available only on Sun GigaSwift Ethernet adapters. For more information on configuring VLANs, refer to "Configuring VLANs," in the Sun GigaSwift Ethernet Adapter Installation and User's Guide, part number 816-1702-11.
When configuring VLANs on an aggregation (trunk), make sure that the link partner (or switch) has VLAN enabled on all the ports belonging this aggregation.
The SunVTS software executes multiple diagnostic hardware tests and is used to verify the configuration and functionality of most hardware controllers and devices.
Since trunking aggregates many network adapters behind a single head instance, these tests may not provide the information intended. Unaggregate all trunking interfaces to be tested before running SunVTS.
If you still want to use the SunVTS nettest option on an aggregated link, use the Round Robin policy option and then use the nettr -stats option to ensure that test traffic is flowing over all interfaces in the aggregation.
Refer to the Solaris operating environment user documentation for more information.
The default behavior for Sun Trunking 1.3 does not expose trunk members to upper layers like IP. However, for performance sake, you may want to plumb multiple IP streams on member instances with different subnets on the trunk members to derive multiple connections to IP. You can achieve this by setting the laggr_multistream tuning variable in ce driver using ndd or ce.conf file.
|
1. Release all trunks to the ce device.
# ./nettr -conf # ndd -set /dev/ce instance 0 # ndd -set /dev/ce laggr_multistream 1 # ndd -set /dev/ce instance 1 # ndd -set /dev/ce laggr_multistream 1 |
2. Repeat for all member instances within the trunk
You can configure the trunk now and plumb on one or all member instances.
|
1. Edit the /platform/sun4u/kernel/drv/ce.conf file and add the following line:
If you have problems with Sun Trunking 1.3, use the following commands to gather information that may help resolve the problems.
Use the nettr -stats command to check whether network packets are showing in each trunk instance. In the following example, 1 represents the trunk head instance being monitored.
# /etc/opt/SUNWconn/bin/nettr -stats 0 Mar 30 15:26:58 2003 Name Ipkts Ierrs Opkts Oerrs Collis Crc %Ipkts %Opkts ce0 66518099 0 67498218 0 0 0 50.00 50.00 ce1 66527490 0 67498218 0 0 0 50.00 50.00 |
If you use MAC policy, you may not see packets on certain interfaces, as shown in the example. This occurs because two or more clients may have MAC addresses with the last two bits identical. See Trunking Policies for more information.
Use the snoop -d command to capture and inspect network packets. This command enables you to troubleshoot network problems at the packet level, allowing examination of the contents of a packet. The following shows example output for ce0.
In this example, if there were packets meant for hostname hs4-net11, but none showed in the output from the snoop command, you would know a problem existed. In that case, you would check your host file, network connection, or your ethernet switch setup. Refer to the snoop(1M) man page for more information.
Because of the trunk head concept (as virtual link) in Sun Trunking 1.3, you might expect the snoop command on the trunk head instance to monitor the aggregation traffic. While Sun provides a feature in ce drivers that allows you to snoop the trunk member instance to monitor the running traffic on that instance, using the snoop command on trunk member instance can present confusing results.
The best way to look at the aggregation traffic is by using the trunk head instance (virtual link) and the nettr -stat key instead of using the snoop command on the trunk member instance. For example, if ce0 and ce1 are aggregated and ce0 is the trunk head and you run snoop on both ce0 and ce1, you will see some duplicated packets appear on the snoop output for both instances. That is because all the packets to this aggregation link will be seen by the snoop command to the trunk head instance (ce0), which appears as a "fat pipe" presenting this aggregation link, and at the same time, the packets physical using the trunk member port will also be seen by the snoop command to this trunk member instance (ce1).
Networking commands such as ifconfig (1M) and netstat (1M) work on any member instance.
Use the nettr -conf command to verify your trunking configuration.
# nettr -conf Name Key Policy DEV Type Original-Mac-Addr Speed Duplex Link ce0 ce0 2 pci 8:0:20:8f:be:24 1000 Full Up ce1 ce0 pci 8:0:20:8f:be:24 1000 Full Up |
Make sure the Ethernet switch software and the Sun Trunking 1.3 software are configured symmetrically. For example, both Sun Trunking and the switch should be configured with the same number of links per trunk.
Once you have configured Sun Trunking 1.3 software, be sure to re-configure your switch to match the trunking configuration.
See your console for error messages when configuring a Sun GigaSwift Ethernet adapter or a Sun Dual FastEthernet and Dual SCSI/P adapter for trunking. It is best to aggregate and unaggregate links on the console because error messages from the console and nettr are displayed inline.
|
Note - If you are configuring a Sun Gigabit Ethernet adapter or a Sun Quad FastEthernet adapter, error messages are returned by nettr, but may not appear in the console. |
A message similar to the following appears in the console stating that the aggregation already exists:
A message similar to the following appears in the console stating that the aggregation already exists:
A message similar to the following appears in the console stating that the aggregation already exists:
proboscis# nettr -setup 1 device=ce members=0,1,2,3 policy=7 Jul 10 17:40:04 trunk5-127 laggr: trunk(1) laggr_ioc: Policy 7 not within supported range(1 - 4), Using default Policy 1 |
If you cannot resolve the problems and you have a SunService contract, send the information gathered from these commands to your SunService representative. You must have the following information ready:
Use the showrev command to display your operating system release:
% showrev Hostname: proboscis-11 Hostid: 8081d6ca Release: 5.9 Kernel architecture: sun4u Application architecture: sparc Hardware provider: Sun_Microsystems |
| Sun Trunking 1.3 Installation and User's Guide | 817-3374-11 |
|
Copyright © 2007, Sun Microsystems, Inc. All Rights Reserved.
To Resolve the Problem of Plumbed Interfaces