The Networking Configuration features permit you to create a variety of advanced networking setups out of your physical network ports, including link-aggregations, virtual LANs (VLANs), and multipathing groups. You can then define any number of IPv4 and IPv6 addresses for these abstractions, for use in connecting to the various data services on the system.
There are four components to a system's network configuration:
Devices - Physical network ports. These correspond to your physical network connections or IP on InfiniBand (IPoIB) partitions.
Datalinks - The basic construct for sending and receiving packets. Datalinks may correspond 1:1 with a device (that is, with a physical network port) or IB Partition, or you may define Aggregation and VLAN datalinks composed of other devices and datalinks.
Interface - The basic construct for IP configuration and addressing. Each IP interface is associated with a single datalink, or is defined to be an IP MultiPathing (IPMP) group comprised of other interfaces.
Routing - IP routing configuration. This controls how the system will direct IP packets.
In this model, network devices represent the available hardware - they have no configurable settings. Datalinks are a layer 2 entity, and must be created to apply settings such as LACP to these network devices. Interfaces are a layer 3 entity containing the IP settings, which they make available via a datalink. This model has separated network interface settings into two parts - datalinks for layer 2 settings, and interfaces for layer 3 settings.
An example of a single IP address on a single port (common configuration) is:
The following configuration is for a 3-way link aggregation:
The datalink entity (which we named "aggr1") groups the network devices in a configurable way (LACP aggregation policy). The interface entity (which we named "phobos") provides configurable IP address settings, which it makes available on the network via the datalink. The network devices (named "nge1", "nge2", ..., by the system) have no direct settings.
Datalinks are required to complete the network configuration, whether they apply specific settings to the network devices or not.
These are created by the system to represent the available network or InfiniBand ports. They have no configuration settings of their own.
These manage devices, and are used by interfaces. They support:
VLANs - Virtual LANs to improve local network security and isolation.
LACP - Link Aggregation Control Protocol, to bundle multiple network devices to behave as one. This improves performance (multiplies bandwidth) and reliability (can survive network port failure), however the appliance must be connected to a switch that supports LACP and has it enabled for those ports.
IB Partitions - InfiniBand partitions to connect to logically isolated IB fabric domains.
The following datalink settings are available:
These configure IP addresses via datalinks. They support:
IPv4 and IPv6 protocols.
IPMP - IP MultiPathing, to improve network reliability by allowing IP addresses to automatically migrate from failed to working datalinks.
The following interface settings are available:
IP MultiPathing groups are used to provide IP addresses that will remain available in the event of a IP interface failure (such a physical wire disconnection or a failure of the connection between a network device and its switch) or in the event of a path failure between the system and its network gateways. The system detects failures by monitoring the IP interface's underlying datalink for link-up and link-down notifications, and optionally by probing using test addresses that can be assigned to each IP interface in the group, described below. Any number of IP interfaces can be placed into an IPMP group so long as they are all on the same link (LAN, IB partition, or VLAN), and any number of highly-available addresses can be assigned to an IPMP group.
Each IP interface in an IPMP group is designated either <i>active</i> or <i>standby</i>:
Active: The IP interface will be used to send and receive data so long as IPMP has determined it is functioning correctly.
Standby: The IP interface will only be used to send and receive data if an active interface (or a previously-activated standby) stops functioning.
Multiple active and standby IP interfaces can be configured, but each IPMP group must be configured with at least one active IP interface. IPMP will strive to activate as many standbys as necessary to preserve the configured number of active interfaces. For example, if an IPMP group is configured with two active interfaces and two standby interfaces and all interfaces are functioning correctly, only the two active interfaces will be used to send and receive data. If an active interface fails, one of the standby interfaces will be activated. If the other active interface fails (or the activated standby fails), the second standby interface will be activated. If the active interfaces are subsequently repaired, the standby interfaces will again be deactivated.
If probe-based failure detection is enabled on an IP interface (i.e., a test address is configured), the system will determine which target systems to probe dynamically. First, the routing table will be scanned for gateways (routers) on the same subnet as the IP interface's test address and up to five will be selected. If no gateways on the same subnet were found, the system will send a multicast ICMP probe (to 224.0.01. for IPv4 or ff02::1 for IPv6) and select the first five systems on the same subnet that respond. Therefore, for network failure detection and repair using IPMP, you should be sure that at least one neighbor on each link or the default gateway responds to ICMP echo requests. IPMP works with both IPv4 and IPv6 address configurations. In the case of IPv6, the interface's link-local address is used as the test address.
The system will probe selected target systems in round-robin fashion. If five consecutive probes are unanswered, the IP interface will be considered failed. Conversely, if ten consecutive probes are answered, the system will consider a previously-failed IP interface as repaired. You can set the system's IPMP probe failure detection time from the IPMP screen. This time indirectly controls the probing rate and the repair interval -- for instance, a failure detection time of 10 seconds means that the system will send probes at roughly two second intervals and that the system will need 20 seconds to detect a probe-based interface repair. You cannot directly control the system's selected targeted systems, though it can be indirectly controlled through the routing table.
The system will monitor the routing table and automatically adjust its selected target systems as necessary. For instance, if the system using multicast-discovered targets but a route is subsequently added that has a gateway on the same subnet as the IP interface's test address, the system will automatically switch to probing the gateway. Similarly, if multicast-discovered targets are being probed, the system will periodically refresh its set of chosen targets (e.g., because some previously-selected targets have become unresponsive).
Step by step instructions for building IPMP groups can be found in the Tasks section below.
IPMP and link aggregation are different technologies available in the appliance to achieve improved network performance as well as maintain network availability. In general, you deploy link aggregation to obtain better network performance, while you use IPMP to ensure high availability.
In link aggregations, incoming traffic is spread over the multiple links that comprise the aggregation. Thus, networking performance is enhanced as more NICs are installed to add links to the aggregation. IPMP's traffic uses the IPMP interface's data addresses as they are bound to the available active interfaces. If, for example, all the data traffic is flowing between only two IP addresses but not necessarily over the same connection, then adding more NICs will not improve performance with IPMP because only two IP addresses remain usable.
The two technologies complement each other and can be deployed together to provide the combined benefits of network performance and availability.
The system provides a single IP routing table, consisting of a collection of routing table entries. When an IP packet needs to be sent to a given destination, the system selects the routing entry whose destination most closely matches the packet's destination address (subject to the system's multihoming policy -- see below). It then uses the information in the routing entry to determine what IP interface to send the packet on and -- if the destination is not directly reachable -- the next-hop gateway to use. If no routing entries match the destination, the packet will be dropped. If multiple routing entries tie for closest match (and are not otherwise prioritized by multihoming policy), the system will load-spread across those entries on a per-connection basis.
The system does not act as a router.
The routing table is comprised of routing entries, each of which has the following fields:
A routing entry with a "destination" field of 0.0.0.0/0 will match any packet (if no other route matches more precisely), and is thus known as a 'default' route. In the BUI, default routes are distinguished from non-default routes by an additional property:
As above, a given packet will be sent on the IP interface specified in the routing entry's "interface" field. If an IPMP interface is specified, then one of the active IP interfaces in the IPMP group will be chosen randomly on a per-connection basis and automatically refreshed if the chosen IP interface subsequently becomes unusable. Conversely, if a given IP interface is part of an IPMP group, it cannot be specified in the "interface" field because such a route would not be highly-available.
Routing entries come from a number of different origins, as identified by the "type" field. Although the origin of a routing entry has no bearing on how it is used by the system, its origin does control if and how it can be edited or deleted. The system supports the following types of routes:
One additional type identifies a static route that cannot currently be used:
If a system is configured with more than one IP interface, then there may be multiple equivalent routes to a given destination, forcing the system to choose which IP interface to send a packet on. Similarly, a packet may arrive on one IP interface, but be destined to an IP address that is hosted on another IP interface. The system's behavior in such situations is determined by the selected multihoming policy. Three policies are supported:
When selecting the multihoming policy, a key consideration is whether any of the appliance's IP interfaces will be dedicated to administration (for example, for dedicated BUI access) and thus accessed over a separate administration network. In particular, if a default route is created to provide remote access to the administration network, and a separate default route is created to provide remote access to storage protocols, then the default system policy of "loose" may cause the administrative default route to be used for storage traffic. By switching the policy to "adaptive" or "strict", the appliance will consider the IP address associated with the request as part of selecting the route for the reply. If no route can be found on the same IP interface, the "adaptive" policy will cause the system to use any available route, whereas the "strict" policy will cause the system to drop the packet.
When using the BUI to reconfigure networking, the system makes every effort to preserve the current networking connection to your browser. However, some network configuration changes such as deleting the specific address to which your browser is connected, will unavoidably cause the browser to lose its connection. For this reason it is recommended that you assign a particular IP address and network device for use by administrators and always leave the address configured. You can also perform particularly complex network reconfiguration tasks from the CLI over the serial console if necessary.
The following icons are used in the Configuration->Network section:
At top right is local navigation for Configuration, Addresses and Routing, which display alternate configuration views.
The Configuration page is shown by default, and lists Devices, Datalinks and Interfaces, along with buttons for administration. Mouse-over an entry to expose an additional icon, and click on any entry to highlight other components that are associated with it.
The Devices list shows links status on the right, as well as an icon to reflect the state of the network port. If ports appear disconnected, check that they are plugged into the network properly.
To configure an IP address on a network devices, first create a datalink, and then create an interface to use that datalink. The icon may be used to do both, which will display dialogs for the Datalink and Interface properties.
There is more than one way to configure a network interface. Try clicking on the icon for a device, then dragging it to the datalink table. Then drag the datalink over to the interfaces table. Other moves are possible. This can be helpful for complex configurations, where valid moves are highlighted.
This page shows a summary table of the current network configuration, with fields:
This page provides configuration of the IP routing table and associated properties, as discussed above. By default, all entries in the routing table are shown, but the table can be filtered by type by using the subnavigation bar.
Network configuration is under the configuration net, which has sub commands for devices, datalinks, interfaces, and routing. The show command can be used with each to show the current configuration:
caji:> configuration net caji:configuration net> devices show Devices:
DEVICE UP SPEED MAC nge0 true 1000 Mbit/s 0:14:4f:9a:b9:0 nge1 true 1000 Mbit/s 0:14:4f:9a:b9:1 nge2 true 1000 Mbit/s 0:14:4f:9a:b8:fe nge3 true 1000 Mbit/s 0:14:4f:9a:b8:ff caji:configuration net> datalinks show Datalinks: DATALINK CLASS LINKS LABEL nge0 device nge0 datalink1 caji:configuration net> interfaces show Interfaces: INTERFACE STATE CLASS LINKS ADDRS LABEL nge0 up ip nge0 192.168.2.80/22 caji caji:configuration net> routing show Properties: multihoming = loose Routes: ROUTE DESTINATION GATEWAY INTERFACE TYPE route-000 0.0.0.0/0 192.168.1.1 nge0 dhcp route-001 192.168.0.0/22 192.168.2.142 nge0 system
Type help in each section to see the relevant commands for creating and configuring datalinks, interfaces, and routes.
The following demonstrates creating a datalink using the device command, and interface using the ip command:
caji:configuration net> datalinks caji:configuration net datalinks> device caji:configuration net datalinks device (uncommitted)> set links=nge1 links = nge1 (uncommitted) caji:configuration net datalinks device (uncommitted)> set label=datalink2 label = datalink2 (uncommitted) caji:configuration net datalinks device (uncommitted)> set mtu=9000 mtu = 9000 (uncommitted) caji:configuration net datalinks device (uncommitted)> commit caji:configuration net datalinks> show Datalinks: DATALINK CLASS LINKS LABEL nge0 device nge0 datalink1 nge1 device nge1 datalink2 caji:configuration net datalinks> cd .. caji:configuration net> interfaces caji:configuration net interfaces> ip caji:configuration net interfaces ip (uncommitted)> set label="caji2" label = caji2 (uncommitted) caji:configuration net interfaces ip (uncommitted)> set links=nge1 links = nge1 (uncommitted) caji:configuration net interfaces ip (uncommitted)> set v4addrs=10.0.1.1/8 v4addrs = 10.0.1.1/8 (uncommitted) caji:configuration net interfaces ip (uncommitted)> commit caji:configuration net interfaces> show Interfaces: INTERFACE STATE CLASS LINKS ADDRS LABEL nge0 up ip nge0 192.168.2.80/22 caji nge1 up ip nge1 10.0.1.1/8 caji2
The following demonstrates creating a default route via 10.0.1.2 over the new nge1 IP interface:
caji:configuration net routing> create caji:configuration net route (uncommitted)> set family=IPv4 family = IPv4 (uncommitted) caji:configuration net route (uncommitted)> set destination=0.0.0.0 destination = 0.0.0.0 (uncommitted) caji:configuration net route (uncommitted)> set mask=0 mask = 0 (uncommitted) caji:configuration net route (uncommitted)> set interface=nge1 interface = nge1 (uncommitted) caji:configuration net route (uncommitted)> set gateway=10.0.1.2 gateway = 10.0.1.2 (uncommitted) caji:configuration net route (uncommitted)> commit
The administrative model for InfiniBand datalink partitions has changed. InfiniBand datalinks and interfaces will not be preserved across an upgrade to Q3.2010. You must tear down all InfiniBand-based interfaces and datalink (partitions) prior to beginning the upgrade to Q3.2010. Once the system has been upgraded, the InfiniBand datalink partitions and interfaces may be re-created. There are no actions required on the subnet manager or switch. Failure to tear down the interfaces and datalinks will result in faulted datalink and non-functioning interfaces.