test

System Administration Guide: IP Services

Chapter 27 IP Network Multipathing (Overview)

IP Network Multipathing provides both load spreading and failover when you have multiple network interface cards that are connected to the same IP link (for example, Ethernet).

This chapter contains the following information:

Introduction

IP Network Multipathing provides your system with the following capabilities:

If a failure occurs in the network adapter, and if you have an alternate adapter connected to the same IP link, the system switches all the network accesses automatically from the failed adapter to the alternate adapter. This process ensures uninterrupted access to the network. Also, when you have multiple network adapters that are connected to the same IP link, you achieve increased traffic throughput by spreading the traffic across multiple network adapters.

Note –

Other IP-related documents, such as RFC 2460, use the term link instead of IP link. This document uses the term IP link to avoid confusion with IEEE 802. In IEEE 802, link refers to a single wire from an Ethernet NIC to an Ethernet switch.

See IP link definition in the Glossary or refer to Table 27–1.

IP Network Multipathing Features

The Solaris implementation of IP Network Multipathing provides the following features:

Communication Failures

Communication failures can occur in the following ways:

  1. Transmit/receive path of the NIC can stop transmitting packets.

  2. Attachment of the NIC to the link is down.

  3. Port on the switch does not transmit/receive packets.

  4. Physical interface in a group not present at system boot.

  5. Host on the other end is not responding or the router that is forwarding the packets is not responding.

The Solaris implementation of IP Network Multipathing addresses the first four types of communication failures.

IP Network Multipathing Components

The following table identifies and describes the components that compose IP Network Multipathing.

Table 27–1 IP Network Multipathing Components

Component 

Description 

IP Link 

A communication facility or medium over which nodes can communicate at the link layer. The link layer is the layer immediately below IPv4/IPv6. Examples include Ethernets (simple or bridged) or ATM networks. One or more IPv4 subnet numbers/prefixes are assigned to an IP link. A subnet number/prefix cannot be assigned to more than one IP link. In ATM LANE, an IP link is a single emulated LAN. When using ARP, the scope of the ARP protocol is a single IP link. 

Network Interface Card (NIC) 

Network adapter that is either internal or a separate card that serves as an interface to a link. 

Physical interface 

A node's attachment to a link. This attachment is often implemented as a device driver plus a network adapter. Some network adapters can have multiple points of attachment, for example, qfe. The usage of Network adapter in this document refers to a "Single Point of Attachment."

Physical interface group 

The set of physical interfaces on a system that are connected to the same link. They are identified by assigning the same (non-null) character string name to all the physical interfaces in the group. 

Physical interface group name 

A name that is assigned to a physical interface that identifies the group. The name is local to a system. Multiple physical interfaces, sharing the same group name, form a physical interface group. 

Failure detection 

The process of detecting when a NIC or the path from the NIC to some layer 3 device no longer works. 

Repair detection 

The process of detecting when a NIC or the path from the NIC to some layer 3 device starts operating correctly after a failure. 

Failover 

The process of switching network access from a failed interface to a good physical interface. Network access includes IPv4 unicast, multicast, and broadcast traffic, as well as IPv6 unicast and multicast traffic. 

Failback 

The process of switching back network access to an interface that is detected to have been repaired. 

Standby Interface 

A physical interface that is not used to carry data traffic unless some other physical interface in the group has failed.  

Solaris Network Multipathing

The following components implement Solaris network multipathing:

The in.mpathd daemon detects failures and implements various policies for failover and failback. After in.mpathd detects a failure or repair, in.mpathd sends an ioctl to do the failover or failback. IP, which implements the ioctl, does the network access failover transparently and automatically.

Caution – Caution –

Do not use Alternate Pathing while using IP Network Multipathing on the same set of NICs. Likewise, you should not use IP Network Multipathing while you are using Alternate Pathing. You can use Alternate Pathing and IP Network Multipathing at the same time on different sets of NICs.

Detecting Physical Interface Failures

The in.mpathd daemon can detect interface failure and repair by two methods. In the first method, the daemon sends and receives ICMP echo probes through the interface. In the second method, the daemon monitors the RUNNING flag on the interface. The link state on some models of network interface cards is reflected by the RUNNING flag. Consequently, when the link fails, the failure is detected much sooner. An interface is considered to have failed if either of the previous two methods indicates failure. An interface is considered repaired only if both methods indicate that the interface is repaired.

The in.mpathd daemon sends ICMP echo probes to the targets that are connected to the link on all the interfaces that belong to a group to detect failures and repair. After you add an interface to a multipathing group and assign a test address, the daemon sends probes to detect failures on all the interfaces of the multipathing group. How to Configure a Multipathing Interface Group With Two Interfaces describes the steps you perform to configure test address and groups.

Because in.mpathd determines which targets to probe dynamically, you cannot configure the targets. Routers that are connected to the link are chosen as targets for probing. If no routers exist on the link, arbitrary hosts on the link are chosen. A multicast packet that is sent to the “all hosts” multicast address (224.0.0.1 in IPv4 and ff02::1 in IPv6) determines the arbitrary hosts. The first few hosts that respond to the echo packets are chosen as targets for probing. If in.mpathd cannot find routers or hosts that responded to ICMP echo packets, in.mpathd cannot detect failures.

To ensure that each NIC in the group functions properly, in.mpathd probes all the targets separately through all the interfaces in the multipathing group. If no replies are made to five consecutive probes, in.mpathd considers the interface to have failed. The probing rate depends on the failure detection time (FDT). The default value for failure detection time is 10 seconds. The in.mpathd(1M) man page describes how to change the failure detection time. For a failure detection time of 10 seconds, the probing rate is approximately one probe every two seconds.

The failure detection time only applies to the ICMP echo probe method of detecting failures. If link failure results in the clearing of the RUNNING flag for an interface, the in.mpathd daemon responds immediately to the change in the flag status.

After a failure is detected, failover of all network access occurs from the failed interface to another functional interface in the group. If you have configured a standby interface, in.mpathd chooses the standby interface for failover of IP addresses, broadcasts, and multicast memberships. If you have not configured a standby interface, in.mpathd chooses the interface with the least number of IP addresses.

Physical interfaces in the same group that are not present at system boot represent a special instance of failure detection. The startup script /etc/init.d/network detects these types of failure. This type of failure displays error messages similar to the following:


moving addresses from failed IPv4 interfaces: hme0 (moved to hme1)
moving addresses from failed IPv6 interfaces: hme0 (moved to hme1)
Note –

In this special instance of failure detection, only static IP addresses that are specified in host name files are moved to a different physical interface within the same multipathing group.

This type of failure can be automatically repaired by a failback. The RCM DR Post-attach feature for IP Network Multipathing automates the DR attachment of a NIC. When a NIC is DR attached, the interface is plumbed and configured. If the interface was removed prior to a reboot, the IP multipathing Reboot-safe feature recovers the IP address. The IP address is transferred to the replaced NIC. The replaced NIC is added to the original IP multipathing interface group. See How to Recover a Physical Interface That Was Not Present at System Boot.

Detecting Physical Interface Repairs

The in.mpathd daemon considers an interface repaired if the daemon receives responses to 10 consecutive probe packets, and the RUNNING flag is set on the interface.

When an interface fails, all addresses are moved to another functional interface in the group. Because in.mpathd needs an address for probing so that it can detect repairs, you must configure a test IP address that cannot move during the failover. Moreover, you should not allow a normal application to use this test address, because the failover of network access cannot occur for these addresses. How to Configure a Multipathing Interface Group With Two Interfaces describes the steps that you perform. If in.mpathd detects a repair, failback of all network access occurs to the repaired interface.

As noted in Detecting Physical Interface Failures, automatic failback is supported for physical interfaces that are not present at system boot. See How to Recover a Physical Interface That Was Not Present at System Boot.

Group Failures

A group failure is when all the network interface cards appear to fail at the same time. in.mpathd does not do any failovers for a group failure. Also, no failover occurs when all the targets fail at the same time. In this instance, in.mpathd flushes all of its current targets and discovers new targets (see Detecting Physical Interface Failures).

Note –

Group failures were previously known as link failures.

Administering Multipathing Groups With Multiple Physical Interfaces

This section describes how you enable IP Network Multipathing. To use the IP Network Multipathing feature, you should have more than one physical interface connected to the same IP link. For example, the same Ethernet switch or the same IP subnet, configured under the same multipathing group, works. If you have just one physical interface, refer to Administering Multipathing Groups With a Single Physical Interface.

Multipathing groups are identified by non-null names. For example, math-link, bio-link, and chem-link make good names. The names typically represent where these groups are connected. When failure is detected in one of the network adapters in the multipathing group, all network access is failed over from the failed adapter to the working adapter in the group. The failover of network access includes IPv4 unicast, broadcast, and multicast traffic, as well as IPv6 unicast and multicast traffic. For IP network multipathing to function properly, the following conditions must exist for the network adapters that are part of the same multipathing group:

  1. You must push and configure the same set of STREAMS modules on all network adapters in the multipathing group.

  2. If you have plumbed IPv4 on one network adapter, then you must plumb IPv4 on all network adapters in the multipathing group.

  3. If you have plumbed IPv6 on one network adapter, then you must plumb IPv6 on all network adapters in the multipathing group.

  4. All Ethernet network adapters in the system should have unique MAC address in the case of ethernets. This is achieved by setting the local-mac-address? to TRUE in the openboot PROM for SPARC platforms. Nothing needs to be done for IA (x86) platforms.

  5. All network adapters of the multipathing group must be connected to the same IP link.

  6. The multipathing group should not contain dissimilar interfaces. The interfaces that are grouped together should be of the same interface type that is defined in /usr/include/net/if_types.h. For example, you cannot combine Ethernet with Token ring, and you cannot combine a Token bus with asynchronous transfer mode (ATM).

  7. When you use IP network multipathing with ATMs, you must configure the ATM for LAN emulation (multipathing over classical IP instances is not currently supported).

Note –

The fourth condition concerns all interfaces in the system, not just those belonging to the multipathing group.

For the adapters that do not come with factory-set unique MAC addresses, you can manually configure a MAC address for each adapter as a workaround. For example, use the ifconfig ether command in a startup script file.

Note –

The MAC addresses configured manually cannot be maintained across system reboot. You are responsible for choosing unique MAC addresses. IP Network Multipathing might behave unpredictably if the MAC addresses of adapters are not unique.

Grouping Physical Interfaces

You use the ifconfig command to configure groups. This command uses a new group parameter that requires a group name and places both the IPv4 and IPv6 instances of the interface in that group. The group parameter has the following syntax:


ifconfig interface-name group group-name
Note –

Avoid using spaces in group names. The ifconfig status display does not show spaces. Consequently, if you have created two similar group names, but one of them contains a space, these group names look alike in the status display. However, they are different group names. This difference might be confusing.

Placing the IPv4 instance under a particular group automatically places the IPv6 instance under the same group. Also, you can place a second interface, connected to the same subnet, in the same group by using the same command. See How to Configure a Multipathing Interface Group With Two Interfaces.

You can remove an interface from a multipathing group by using a null string with the group sub-command. See How to Remove an Interface From a Group.

To place an interface in a new group when it is already part of some multipathing group, you do not need to remove it from any existing group. Placing the interface in a new group automatically removes it from any existing group. See How to Move an Interface From an Existing Group to a Different Group.

You can have any number of network adapters that you can configure in the same multipathing group. You cannot use the group parameter with logical interfaces. For example, you can use the parameter with hme0, but not with hme0:1.

You must connect all the interfaces in the multipathing group to the same IP link, because when an interface fails, the failover operation moves all the IP addresses from the failed interface to an interface in the group that is functional. For routers to continue routing packets to the addresses that have been switched to the functional interface, the functional interface must be connected to the same IP link.

Configuring Test Addresses

You must configure all physical interfaces of a multipathing group with a test address. You need test addresses to detect failures and repairs. If a test address is not configured, it is not chosen for failover. Only in.mpathd uses test addresses. Normal applications should not use this address. This address does not fail over when the interface fails. In IPv4, you should configure the test address in such a way that normal applications do not use the test address. See How to Configure a Multipathing Interface Group With Two Interfaces.

This section describes test address configuration concepts for the following Internet protocols:

IPv4 Test Addresses

The in.mpathd multipathing daemon requires a test IP address for detecting failures and repairs. You must use a routeable address for this IP address. That is, the subnet prefix of the address must be known to any routers present on the link. You use the ifconfig command's new -failover option to configure a test address. Use the following syntax to configure a test address:


# ifconfig interface-name addif ip-address <other-parameters> -failover up

For <other-parameters>, use the parameters that are required by your configuration. See the ifconfig(1M) man page for descriptions. How to Configure a Multipathing Interface Group With Two Interfaces shows the steps you perform for an IPv4 test address.

For example, to add a new logical interface with an address of 19.16.85.21, the netmask and broadcast address set to the default value, and also configure the interface with a test address, type the following:


# ifconfig hme0 addif 19.16.85.21 netmask + broadcast + -failover up
Note –

You must mark an IPv4 test address as deprecated to prevent applications from using the test address. See How to Configure a Multipathing Interface Group With Two Interfaces.

Use failover without the dash to turn on the failover attribute of the address.

Note –

All test IP addresses in a multipathing group must use the same network prefix. That is, the test IP addresses must belong to a single IP subnet.

IPv6 Test Addresses

To configure an IPv6 test address, you use the link-local address, because link-local addresses are tied to the physical interface. Thus, you do not need a separate IP address in the IPv6 situation. For IPv6, the -failover option has the following syntax:


# ifconfig interface-name inet6 -failover

How to Configure a Multipathing Interface Group With Two Interfaces shows the steps you perform for an IPv6 test address.

When a multipathing group has both IPv4 and IPv6 plumbed on all the group's interfaces, you might not need a separate IPv4 test address. The in.mpathd daemon can probe the interfaces by using an IPv6 link-local address. IPv6 link-local addresses are created when IPv6 is plumbed.

Use failover without the dash to turn on the failover attribute of the address.

Note –

The only valid IPv6 test address is the link-local address.

Preventing Applications From Using Test Addresses

After you have configured a test address, you need to ensure that this address is not used by normal applications. If you let applications use the test address, applications fail, because test addresses do not fail over during the failover operation. To ensure that IP does not pick the test address for normal applications, you mark the test address deprecated by using the ifconfig command. This parameter has the following syntax:


ifconfig interface-name deprecated

After you mark the address as deprecated, IP does not pick this address as a source address for any communication, unless the applications explicitly bind to the address. Only in.mpathd explicitly binds to such an address. See How to Configure a Multipathing Interface Group With Two Interfaces.

Because link-local addresses are not present in the name service (DNS, NIS, and NIS+), applications do not use link-local addresses for communication. Consequently, you do not need to mark IPv6 test addresses as deprecated.

Note –

You must not mark IPv6 link-local addresses as deprecated.

Use the -deprecated option to turn off the deprecated attribute of the address.

Note –

IPv4 test addresses should not be placed in the name service tables (DNS/NIS/NIS+). In IPv6, link-local addresses are used as test addresses and are not normally placed in the name service tables.

Autoconfigured IPv6 addresses are not preserved across system reboot. If you require that IP addresses be preserved across reboot, then applications should use static IP addresses.

Using the hostname File to Configure Groups and Test Addresses

You can use the /etc/hostname.interface files to configure multipathing groups and test addresses. To configure a multipathing group by using the /etc/hostname.interface file, you can add a line to the file by using the following syntax:


interface-address <parameters>  group group-name up \
addif logical-interface-address <parameters> up

For example, to create the group test with the following configuration:

You add the following line to the /etc/hostname.hme0 file:


19.16.85.19 netmask + broadcast + group test up \
addif 19.16.85.21 deprecated -failover netmask + broadcast + up

How to Configure a Multipathing Interface Group With Two Interfaces shows the steps you perform to configure the IPv4 hostname file.

For IPv6 setup, add a line to the /etc/hostname6.interface file by using the following syntax:


<parameter> group group-name up

For example, to create a test group for hme0 with an IPv6 test address, add the following line to the /etc/hostname6.hme0 file:


-failover group test up
addif 1080::56:a00:20ff:feb9:19fa up

How to Configure a Multipathing Interface Group With Two Interfaces shows the steps you perform to configure the IPv6 hostname6 file.

Configuring Standby Interfaces

You can configure multipathing groups with standby interfaces. As the name implies, the interface is considered as standby and is not used unless some other interface in the group fails. A standby interface has an IFF_INACTIVE flag when the interface is not hosting any failover IP address. Consequently, when an active interface fails, the standby interface is always chosen for failover. After the standby interface is chosen, the IFF_INACTIVE flag is cleared on the standby interface. From that instant, the active standby is treated the same as other active interfaces. That is, some failures might not choose a standby interface. Instead, some failures might choose an active interface that hosts fewer IP addresses than the standby interface.

The standby interface is not used to send normal data packets. Consequently, limited traffic flows on a standby interface. You must configure standby interfaces with a test address to ensure that probes are sent to determine if the interface is functional. If you do not configure standby interfaces with a test address, the interface is not chosen for failovers when another interface in the group fails. A standby interface might carry traffic under the following conditions:

Thus, the system does not normally select a standby interface (except for probes), unless it is explicitly chosen by an application. If some interface in the group fails, all network access is failed over to the standby interface. To configure a standby interface, you use the ifconfig command's new standby parameter by using the following syntax:


# ifconfig interface-name standby group group-name

How to Configure a Multipathing Group With One of the Interfaces a Standby Interface shows the steps you perform.

The in.mpathd daemon sends probes on the standby interface after a test address is configured on the standby interface. You should configure only test addresses on a standby interface. If any other address is added on the standby, the addition of this address fails. If standby is marked on an interface that already has addresses other than test addresses, automatic failover of these addresses occurs to a different interface in the group, leaving behind only the test address, if one exists. It is advisable not to configure non-test address on a standby interface.

You need to mark the address as a test address by using the ifconfig command's deprecated and -failover options before setting standby or setting up.

To configure a test address on a standby interface, use the following syntax:


# ifconfig interface-name plumb ip-address
   <other-parameters> deprecated -failover standby up

For <other-parameters>, use the parameters that are required by your configuration. See the ifconfig(1M) man page for descriptions.

Note –

Standby interfaces are not used for failover if no test address is configured on that interface.

For example, to create a test address with the following configuration:

You type the following command line:


# ifconfig hme2 plumb 19.16.85.22 netmask + broadcast + deprecated -failover standby up
Note –

The interface is marked as a standby interface only after the address is marked as a NOFAILOVER address.

How to Configure a Multipathing Group With One of the Interfaces a Standby Interface shows the steps you perform.

You can clear a standby interface by using the following syntax:


# ifconfig interface-name -standby

Administering Multipathing Groups With a Single Physical Interface

When you have only one network adapter in the multipathing group, you can configure the network adapter to detect failures on that NIC alone.

Although failovers cannot occur with only one NIC in the group, you still need a separate test address on each of the physical interfaces in the group. You can configure the test address as an IFF_NOFAILOVER address, which is sufficient for the daemon to send out probes on that interface. Unlike the multiple physical interface instance, you do not have to mark a single physical interface as deprecated.

Use the following syntax to configure the interface's IPv4 address as a NOFAILOVER:


# ifconfig interface-name -failover group group-name

For IPv6, use the following syntax:


# ifconfig interface-name inet6 -failover group group-name

When the daemon detects failures, the interface is marked and logged appropriately on the console.

Note –

You cannot verify whether the target that is being probed has failed or the NIC has failed, because the target can be probed through only one physical interface. If only one default router is on the subnet, turn off multipathing if a single physical interface is in the group. If a separate IPv4 and IPv6 default router exists (or multiple default routers exist), more than one target needs to be probed. Hence, you can safely turn on multipathing.

Removing Network Adapters From Multipathing Groups

When you execute the ifconfig command's group parameter with a null string, the interface is removed from the existing group. See How to Remove an Interface From a Group. Be careful when removing interfaces from a group. If some other interface in the multipathing group failed, a failover could have happened earlier. For example, if hme0 failed previously, all addresses are failed over to hme1 (if hme1 is part of the same group). The removal of hme1 from the group causes in.mpathd to return all the failover addresses to some other interface in the group. If no other interfaces are functioning in the group, failover might not restore all the network accesses.

Similarly, when an interface is part of the group and the interface needs to be unplumbed, you should remove the interface from the group first. Then ensure that the interface has all the original IP addresses configured on it. The in.mpathd daemon tries to restore the original configuration of an interface that is removed from the group. You need to ensure that the configuration is restored before unplumbing the interface. Refer to Multipathing Daemon to see how interfaces look before and after a failover.

Detached Network Adapters

Dynamic Reconfiguration (DR) uses IP Network Multipathing to decommission a specific network device without impacting existing IP users. Before a NIC is DR-detached (off lined), all failover IP addresses that are hosted on that NIC are automatically failed over to another NIC in the same IP Network Multipathing group. The test addresses are brought down and the NIC is unplumbed.

With the IP Network Multipathing reboot-safe feature, the static IP addresses in the /etc/hostname.* file that are associated with the missing card are hosted automatically on an alternate interface within the same IP Network Multipathing group. However, these addresses are returned to the original interface automatically if the original interface is inserted back into the system at a later time.

Multipathing Daemon

The in.mpathd multipathing daemon detects failures and repairs by sending out probes on all the interfaces that are part of a group. The in.mpathd multipathing daemon also detects failures and repairs by monitoring the RUNNING flag on each interface in the group. When an interface is part of a group and has a test address, the daemon starts sending out probes for determining failures on that interface. If the daemon does not receive any replies to five consecutive probes, or the RUNNING flag is not set, the interface is considered to have failed. The probing rate depends on the failure detection time. By default, failure detection time is 10 seconds. Thus, the probing rate is one probe every two seconds. To avoid synchronization in the network, probing is not periodic. If five consecutive probes fail, in.mpathd considers the interface as failed and performs a failover of the network access from the failed interface to another interface in the group that is functioning properly. If a standby interface is configured, it is chosen for failover of the IP addresses, and broadcasts and multicast memberships. If no standby interface exists, the interface with the least number of IP addresses is chosen. Refer to the man page in.mpathd(1M) for more information.

The following two examples show a typical configuration and how the configuration automatically changes when an interface fails. When the hme0 interface fails, notice that all addresses move from hme0 to hme1.

Example 27–1 Interface Configuration Before an Interface Failure


hme0: flags=9000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
     inet 19.16.85.19 netmask ffffff00 broadcast 19.16.85.255
     groupname test
hme0:1: flags=9000843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 
     index 2 inet 19.16.85.21 netmask ffffff00 broadcast 129.146.85.255
hme1: flags=9000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
     inet 19.16.85.20 netmask ffffff00 broadcast 19.16.85.255
     groupname test
hme1:1: flags=9000843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 
     index 2 inet 19.16.85.22 netmask ffffff00 broadcast 129.146.85.255
hme0: flags=a000841<UP,RUNNING,MULTICAST,IPv6,NOFAILOVER> mtu 1500 index 2
     inet6 fe80::a00:20ff:feb9:19fa/10
     groupname test
hme1: flags=a000841<UP,RUNNING,MULTICAST,IPv6,NOFAILOVER> mtu 1500 index 2
     inet6 fe80::a00:20ff:feb9:1bfc/10
     groupname test
Example 27–2 Interface Configuration After an Interface Failure


hme0: flags=19000842<BROADCAST,RUNNING,MULTICAST,IPv4,NOFAILOVER,FAILED> mtu 0 index 2
        inet 0.0.0.0 netmask 0 
        groupname test
hme0:1: flags=19040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,FAILED> 
        mtu 1500 index 2 inet 19.16.85.21 netmask ffffff00 broadcast 129.146.85.255
hme1: flags=9000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 19.16.85.20 netmask ffffff00 broadcast 19.16.85.255
        groupname test
hme1:1: flags=9000843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 
        index 2 inet 19.16.85.22 netmask ffffff00 broadcast 129.146.85.255
hme1:2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 6
        inet 19.16.85.19 netmask ffffff00 broadcast 19.16.18.255
hme0: flags=a000841<UP,RUNNING,MULTICAST,IPv6,NOFAILOVER,FAILED> mtu 1500 index 2
        inet6 fe80::a00:20ff:feb9:19fa/10 
        groupname test
hme1: flags=a000841<UP,RUNNING,MULTICAST,IPv6,NOFAILOVER> mtu 1500 index 2
        inet6 fe80::a00:20ff:feb9:1bfc/10 
        groupname test

You can see that the FAILED flag is set on hme0 to indicate that hme0 has failed. You can also see that hme1:2 is now created. hme1:2 was originally hme0. The address 19.16.85.19 then becomes accessible through hme1. Multicast memberships that are associated with 19.16.85.19 can still receive packets, but now through hme1. When the failover of address 19.16.85.19 from hme0 to hme1 occurred, a dummy address 0.0.0.0 was created on hme0. The dummy address is removed when a subsequent failback takes place. The dummy address is created so that hme0 can still be accessed. hme0:1 cannot exist without hme0.

Similarly, failover of the IPv6 address from hme0 to hme1 occurred. In IPv6, multicast memberships are associated with interface indexes. They also fail over from hme0 to hme1. All the addresses that in.ndpd configures also move, this action is not shown in the examples.

The in.mpathd daemon continues to probe through the failed NIC, hme0. After the daemon receives 10 consecutive replies for a default failure detection time of 10 seconds, the daemon considers the interface repaired and invokes the failback. After failback, the original configuration is reestablished.

See in.mpathd(1M) man page for a description of all error messages there are logged on the console during failures and repairs.

Multipathing Configuration File

The in.mpathd daemon uses the settings in the /etc/default/mpathd configuration file to invoke multipathing. Changes to this file are read by in.mpathd at startup and on SIGHUP. This file contains the following default settings and information:


#
# Time taken by mpathd to detect a NIC failure in ms. The minimum time
# that can be specified is 100 ms.
# 
FAILURE_DETECTION_TIME=10000
#

# Failback is enabled by default. To disable failback turn off this option
#
FAILBACK=yes
#

# By default only interfaces configured as part of multipathing groups 
# are tracked. Turn off this option to track all network interfaces 
# on the system
#
TRACK_INTERFACES_ONLY_WITH_GROUPS=yes

How to Configure the Multipathing Configuration File shows the steps you perform to configure the /etc/default/mpathd configuration file.

Failure Detection Time

You can set a lower value of failure detection time. Sometimes these values might not be achieved if the load on the network is too high. Then in.mpathd prints a message on the console, indicating that the time cannot be met. The daemon also prints the time that it can meet currently. If the response comes back correctly, in.mpathd meets the failure detection time that is provided in this file.

Failback

After a failover, failbacks occurs when the failed interface is repaired. However, in.mpathd does not fail back the interface if FAILBACK is set to no.

As noted in Detecting Physical Interface Failures, automatic failback is supported for physical interfaces that are not present at system boot. See How to Recover a Physical Interface That Was Not Present at System Boot.

Track Interfaces Only With Groups Option

By turning off this option, in.mpathd tracks all interfaces in the system. When a failure is detected, an appropriate message is logged on the console. For this option to function properly, Ethernet addresses on all the interfaces must be unique.