Network

Configuring networking

Network Configuration

The Networking Configuration features permit you to create a variety of advanced networking setups out of your physical network ports, including link-aggregations, virtual LANs (VLANs), and multipathing groups. You can then define any number of IPv4 and IPv6 addresses for these abstractions, for use in connecting to the various data services on the system.

There are four components to system's network configuration:

• Devices - Physical network ports. These correspond to your physical network connections or IP on Infiniband (IPoIB) partitions.

• Datalinks - The basic construct for sending and receiving packets. Datalinks may correspond 1:1 with a device (that is, with a physical network port) or IB Partition, or you may define Aggregation and VLAN datalinks composed of other devices and datalinks.

• Interface - The basic construct for IP configuration and addressing. Each IP interface is associated with a single datalink, or is defined to be an IP MultiPathing (IPMP) group comprised of other interfaces.

• Routing - IP routing configuration. This controls how the system will direct IP packets.

In this model, network devices represent the available hardware - they have no configurable settings. Datalinks are a layer 2 entity, and must be created to apply settings such as LACP to these network devices. Interfaces are a layer 3 entity containing the IP settings, which they make available via a datalink. This model has separated network interface settings into two parts - datalinks for layer 2 settings, and interfaces for layer 3 settings.

To show this with an example, the following configuration is for a 4-way link aggregation:

The datalink entity (which we named "aggr1") groups the network devices in a configurable way (LACP aggregation policy). The interface entity (which we named "deimos") provides configurable IP address settings, which it makes available on the network via the datalink. The network devices (named "nge0", "nge1", ..., by the system) have no direct settings.

Datalinks are required to complete the network configuration, whether they apply specific settings to the network devices or not. An example of a single IP address on a single port (common configuration) is:

Devices

These are created by the system to represent the available network or Infiniband ports. They have no configuration settings of their own.

Datalinks

These manage devices, and are used by interfaces. They support:

• VLANs - Virtual LANs to improve local network security and isolation.

• LACP - Link Aggregation Control Protocol, to bundle multiple network devices to behave as one. This improves performance (multiplies bandwidth) and reliability (can survive network port failure), however the appliance must be connected to a switch that supports LACP and has it enabled for those ports.

• IB Partitions - Infiniband partitions to connect to logically isolated IB fabric domains.

The following datalink settings are available:

Interfaces

These configure IP addresses via datalinks. They support:

• IPv4 and IPv6 protocols.

• IPMP - IP MultiPathing, to improve network reliability by allowing IP addresses to automatically migrate from failed to working datalinks.

The following interface settings are available:

IP MultiPathing (IPMP)

IP MultiPathing groups are used to provide IP addresses that will remain available in the event of a IP interface failure (such a physical wire disconnection or a failure of the connection between a network device and its switch) or in the event of a path failure between the system and its network gateways. The system detects failures by monitoring the IP interface's underlying datalink for link-up and link-down notifications, and optionally by probing using test addresses that can be assigned to each IP interface in the group, described below. Any number of IP interfaces can be placed into an IPMP group so long as they are all on the same link (LAN, IB partition, or VLAN), and any number of highly-available addresses can be assigned to an IPMP group.

Each IP interface in an IPMP group is designated either <i>active</i> or <i>standby</i>:

• Active: The IP interface will be used to send and receive data so long as IPMP has determined it is functioning correctly.

• Standby: The IP interface will only be used to send and receive data if an active interface (or a previously-activated standby) stops functioning.

Multiple active and standby IP interfaces can be configured, but each IPMP group must be configured with at least one active IP interface. IPMP will strive to activate as many standbys as necessary to preserve the configured number of active interfaces. For example, if an IPMP group is configured with two active interfaces and two standby interfaces and all interfaces are functioning correctly, only the two active interfaces will be used to send and receive data. If an active interface fails, one of the standby interfaces will be activated. If the other active interface fails (or the activated standby fails), the second standby interface will be activated. If the active interfaces are subsequently repaired, the standby interfaces will again be deactivated.

If probe-based failure detection is enabled on an IP interface (i.e., a test address is configured), the system will determine which target systems to probe dynamically. First, the routing table will be scanned for gateways (routers) on the same subnet as the IP interface's test address and up to five will be selected. If no gateways on the same subnet were found, the system will send a multicast ICMP probe (to 224.0.01. for IPv4 or ff02::1 for IPv6) and select the first five systems on the same subnet that respond. Therefore, for network failure detection and repair using IPMP, you should be sure that at least one neighbor on each link or the default gateway responds to ICMP echo requests. IPMP works with both IPv4 and IPv6 address configurations. In the case of IPv6, the interface's link-local address is used as the test address.

The system will probe selected target systems in round-robin fashion. If five consecutive probes are unanswered, the IP interface will be considered failed. Conversely, if ten consecutive probes are answered, the system will consider a previously-failed IP interface as repaired. You can set the system's IPMP probe failure detection time from the IPMP screen. This time indirectly controls the probing rate and the repair interval -- for instance, a failure detection time of 10 seconds means that the system will send probes at roughly two second intervals and that the system will need 20 seconds to detect a probe-based interface repair. You cannot directly control the system's selected targeted systems, though it can be indirectly controlled through the routing table.

The system will monitor the routing table and automatically adjust its selected target systems as necessary. For instance, if the system using multicast-discovered targets but a route is subsequently added that has a gateway on the same subnet as the IP interface's test address, the system will automatically switch to probing the gateway. Similarly, if multicast-discovered targets are being probed, the system will periodically refresh its set of chosen targets (e.g., because some previously-selected targets have become unresponsive).

Step by step instructions for building IPMP groups can be found in the Tasks section below.

Performance and Availability

IPMP and link aggregation are different technologies available in the appliance to achieve improved network performance as well as maintain network availability. In general, you deploy link aggregation to obtain better network performance, while you use IPMP to ensure high availability.

In link aggregations, incoming traffic is spread over the multiple links that comprise the aggregation. Thus, networking performance is enhanced as more NICs are installed to add links to the aggregation. IPMP's traffic uses the IPMP interface's data addresses as they are bound to the available active interfaces. If, for example, all the data traffic is flowing between only two IP addresses but not necessarily over the same connection, then adding more NICs will not improve performance with IPMP because only two IP addresses remain usable.

The two technologies complement each other and can be deployed together to provide the combined benefits of network performance and availability.

Routing

The system provides a single IP routing table, consisting of a collection of routing table entries. When an IP packet needs to be sent to a given destination, the system selects the routing entry whose destination most closely matches the packet's destination address (subject to the system's multihoming policy -- see below). It then uses the information in the routing entry to determine what IP interface to send the packet on and -- if the destination is not directly reachable -- the next-hop gateway to use. If no routing entries match the destination, the packet will be dropped. If multiple routing entries tie for closest match (and are not otherwise prioritized by multihoming policy), the system will load-spread across those entries on a per-connection basis.

The system does not act as a router.

Routing Entries

The routing table is comprised of routing entries, each of which has the following fields:

A routing entry with a "destination" field of 0.0.0.0/0 will match any packet (if no other route matches more precisely), and is thus known as a 'default' route. In the BUI, default routes are distinguished from non-default routes by an additional property:

As above, a given packet will be sent on the IP interface specified in the routing entry's "interface" field. If an IPMP interface is specified, then one of the active IP interfaces in the IPMP group will be chosen randomly on a per-connection basis and automatically refreshed if the chosen IP interface subsequently becomes unusable. Conversely, if a given IP interface is part of an IPMP group, it cannot be specified in the "interface" field because such a route would not be highly-available.

Routing entries come from a number of different origins, as identified by the "type" field. Although the origin of a routing entry has no bearing on how it is used by the system, its origin does control if and how it can be edited or deleted. The system supports the following types of routes:

One additional type identifies a static route that cannot currently be used:

Routing Properties

If a system is configured with more than one IP interface, then there may be multiple equivalent routes to a given destination, forcing the system to choose which IP interface to send a packet on. Similarly, a packet may arrive on one IP interface, but be destined to an IP address that is hosted on another IP interface. The system's behavior in such situations is determined by the selected multihoming policy. Three policies are supported:

When selecting the multihoming policy, a key consideration is whether any of the appliance's IP interfaces will be dedicated to administration (for example, for dedicated BUI access) and thus accessed over a separate administration network. In particular, if a default route is created to provide remote access to the administration network, and a separate default route is created to provide remote access to storage protocols, then the default system policy of "loose" may cause the administrative default route to be used for storage traffic. By switching the policy to "adaptive" or "strict", the appliance will consider the IP address associated with the request as part of selecting the route for the reply. If no route can be found on the same IP interface, the "adaptive" policy will cause the system to use any available route, whereas the "strict" policy will cause the system to drop the packet.

BUI

When using the BUI to reconfigure networking, the system makes every effort to preserve the current networking connection to your browser. However, some network configuration changes such as deleting the specific address to which your browser is connected, will unavoidably cause the browser to lose its connection. For this reason it is recommended that you assign a particular IP address and network device for use by administrators and always leave the address configured. You can also perform particularly complex network reconfiguration tasks from the CLI over the serial console if necessary.

The following icons are used in the Configuration->Network section:

icon description Add new datalink/interface/route Edit datalink/interface/route settings Destroy datalink/interface/route Drag-and-drop icon connected network port connected network port with I/O activity disconnected network port (link down, cable problem?) active Infiniband port active Infiniband port with I/O activity inactive Infiniband port (down, init, or arm state) Infiniband partition device is up Infiniband partition device is down (subnet manager problem) network datalink network datalink VLAN network datalink aggregation network datalink aggregation VLAN network datalink IB partition interface is being used to send and receive packets (either up or degraded) interface has been disabled by the user interface is offline (owned by the cluster peer) interface has failed or has been configured with a duplicate IP address

At top right is local navigation for Configuration, Addresses and Routing, which display alternate configuration views.

Configuration

The Configuration page is shown by default, and lists Devices, Datalinks and Interfaces, along with buttons for administration. Mouse-over an entry to expose an additional icon, and click on any entry to highlight other components that are associated with it.

The Devices list shows links status on the right, as well as an icon to reflect the state of the network port. If ports appear disconnected, check that they are plugged into the network properly.

To configure an IP address on a network devices, first create a datalink, and then create an interface to use that datalink. The icon may be used to do both, which will display dialogs for the Datalink and Interface properties.

There is more than one way to configure a network interface. Try clicking on the icon for a device, then dragging it to the datalink table. Then drag the datalink over to the interfaces table. Other moves are possible. This can be helpful for complex configurations, where valid moves are highlighted.

Addresses

This page shows a summary table of the current network configuration, with fields:

Routing

This page provides configuration of the IP routing table and associated properties, as discussed above. By default, all entries in the routing table are shown, but the table can be filtered by type by using the subnavigation bar.

CLI

Network configuration is under the configuration net, which has sub commands for devices, datalinks, interfaces, and routing. The show command can be used with each to show the current configuration:

caji:> configuration net
caji:configuration net> devices show
Devices:

DEVICE      UP     SPEED         MAC                                            
nge0        true   1000 Mbit/s   0:14:4f:9a:b9:0                               
nge1        true   1000 Mbit/s   0:14:4f:9a:b9:1                               
nge2        true   1000 Mbit/s   0:14:4f:9a:b8:fe                              
nge3        true   1000 Mbit/s   0:14:4f:9a:b8:ff                              

caji:configuration net> datalinks show
Datalinks:

   DATALINK CLASS          LINKS       LABEL
       nge0 device         nge0        datalink1

caji:configuration net> interfaces show
Interfaces:

  INTERFACE STATE  CLASS LINKS       ADDRS                  LABEL
       nge0 up     ip    nge0        192.168.2.80/22        caji
 
caji:configuration net> routing show
Properties:
                  multihoming = loose
 
Routes:
 
ROUTE      DESTINATION                      GATEWAY         INTERFACE TYPE
route-000  0.0.0.0/0                        192.168.1.1     nge0      dhcp
route-001  192.168.0.0/22                   192.168.2.142   nge0      system

Type help in each section to see the relevant commands for creating and configuring datalinks, interfaces, and routes.

The following demonstrates creating a datalink using the device command, and interface using the ip command:

caji:configuration net> datalinks 
caji:configuration net datalinks> device
caji:configuration net datalinks device (uncommitted)> set links=nge1
                          links = nge1 (uncommitted)
caji:configuration net datalinks device (uncommitted)> set label=datalink2
                          label = datalink2 (uncommitted)
caji:configuration net datalinks device (uncommitted)> set jumbo=true
                          jumbo = true (uncommitted)
caji:configuration net datalinks device (uncommitted)> commit
caji:configuration net datalinks> show
Datalinks:

   DATALINK CLASS          LINKS       LABEL
       nge0 device         nge0        datalink1
       nge1 device         nge1        datalink2

caji:configuration net datalinks> cd ..
caji:configuration net> interfaces
caji:configuration net interfaces> ip
caji:configuration net interfaces ip (uncommitted)> set label="caji2"
                         label = caji2 (uncommitted)
caji:configuration net interfaces ip (uncommitted)> set links=nge1 
                         links = nge1 (uncommitted)
caji:configuration net interfaces ip (uncommitted)> set v4addrs=10.0.1.1/8
                       v4addrs = 10.0.1.1/8 (uncommitted)
caji:configuration net interfaces ip (uncommitted)> commit
caji:configuration net interfaces> show
Interfaces:

  INTERFACE STATE  CLASS LINKS       ADDRS                  LABEL
       nge0 up     ip    nge0        192.168.2.80/22        caji
       nge1 up     ip    nge1        10.0.1.1/8             caji2

The following demonstrates creating a default route via 10.0.1.2 over the new nge1 IP interface:

caji:configuration net routing> create
caji:configuration net route (uncommitted)> set family=IPv4
                   family = IPv4 (uncommitted)
caji:configuration net route (uncommitted)> set destination=0.0.0.0
                   destination = 0.0.0.0 (uncommitted)
caji:configuration net route (uncommitted)> set mask=0
                   mask = 0 (uncommitted)
caji:configuration net route (uncommitted)> set interface=nge1
                   interface = nge1 (uncommitted)
caji:configuration net route (uncommitted)> set gateway=10.0.1.2
                   gateway = 10.0.1.2 (uncommitted)
caji:configuration net route (uncommitted)> commit

Tasks

BUI

Creating a single port interface

1. Click the Datalink icon.
2. Optionally set name and jumbo frames.
3. Choose a device from the Devices list.
4. Click "APPLY". The datalink will appear in the Datalinks list.
5. Click the Interface icon.
6. Set desired properties, and choose the datalink previously created.
7. Click "APPLY". The interface will appear in the Interfaces list.
8. The running appliance network configuration has not yet changed. When you are finished configuring interfaces, click "APPLY" at the top to commit the configuration.

Modifying an interface

1. Click the edit icon on either the datalink or the interface.
2. Change settings to desired values.
3. Click "APPLY" on the dialog.
4. Click "APPLY" at the top of the page to commit the configuration.

Creating a single port interface, drag-and-drop

1. Mouse over a device and click the drag-and-drop icon ().
2. Drag it to the Datalink list and release.
3. Optionally set name and jumbo frames.
4. Click "APPLY".
5. Now Drag the datalink over to the Interfaces list.
6. Set desired properties, and click "APPLY".
7. Click "APPLY" at the top of the screen to commit the configuration.

Creating an LACP aggregated link interface

1. Click the Datalinks icon.
2. Optionally set the datalink name.
3. Select LACP Aggregation.
4. Select two or more devices from the Devices list, and click "APPLY".
5. Click the Interfaces icon.
6. Set desired properties, choose the aggregated link from the Datalinks list, and click "APPLY".
7. Click "APPLY" at the top to commit the configuration.

Create an IPMP group using probe-based and link-state failure detection

1. Create one or more "underlying" IP interfaces that will be used as components of the IPMP group. Each interface must have an IP address to be used as the probe source (see separate task to create a single-port interfaces above).
2. Click the Interface icon.
3. Optionally change the name of the interface.
4. Click the IP MultiPathing Group check box.
5. Click the Use IPv4 Protocol or/and the Use IPv6 Protocol and specify the IP addresses for the IPMP interface.
6. Choose the interfaces created in the fist step from the Interfaces list.
7. Set each chosen interface to be either "Active" or "Standby", as desired.
8. Click "APPLY".

Create an IPMP group using link-state only failure detection

1. Create one or more "underlying" IP interfaces with the IP address 0.0.0.0/8 to be used as the components of the IPMP group (see separate task to create a single-port interfaces above).
2. Click the Interface icon.
3. Optionally change the name of the interface.
4. Click the IP MultiPathing Group check box.
5. Click the Use IPv4 Protocol or/and the Use IPv6 Protocol and specify the IP addresses for the IPMP interface.
6. Choose the interfaces created in the first step from the Interfaces list.
7. Set each chosen interface to be either "Active" or "Standby", as desired.
8. Click "APPLY".

Extend an LACP aggregation

1. Mouse-over a device in the Devices list.
2. Click the icon, and drag the device onto an aggregation datalink, and release.
3. Click "APPLY" at the top of the page to commit this configuration.

Extend an IPMP group

1. Mouse-over an interface in the Interfaces list.
2. Click the icon, and drag the device onto an IPMP interface, and release.
3. Click "APPLY" at the top of the page to commit this configuration.

Create an Infiniband partition datalink and interface

1. Click the Datalink icon.
2. Optionally set name.
3. Click the IB Partition checkbox
4. Choose a device from the Partition Devices list.
5. Click "APPLY". The new partition datalink will appear in the Datalinks list.
6. Click the Interface icon.
7. Set desired properties, and choose the datalink previously created.
8. Click "APPLY". The interface will appear in the Interfaces list.
9. The running appliance network configuration has not yet changed. When you are finished configuring interfaces, click "APPLY" at the top to commit the configuration.

Adding a static route

1. Go to Configuration->Network->Routing
2. Click the add icon.
3. Fill in the properties as described earlier.
4. Click "ADD". The new route will appear in the table.

Deleting a static route

1. Go to Configuration->Network->Routing
2. Mouse-over the route entry, then click the trash icon on the right.

CLI

Adding a static route

1. Go to configuration network routing.
2. Enter create.
3. Type show to list required properties, and set each.
4. Enter commit.

Deleting a static route

1. Go to configuration network routing.
2. Type show to list routes, and route names (eg, route-002).
3. Enter destroy route name.

Infiniband Upgrade Procedures for Q3.2010

The administrative model for Infiniband datalink partitions has changed. Infiniband datalinks and interfaces will not be preserved across an upgrade to Q3.2010. You must teardown all Infiniband-based interfaces and datalink (partitions) prior to beginning the upgrade to Q3.2010. Once the system has been upgraded, the Infiniband datalink partitions and interfaces may be re-created. There are no actions required on the subnet manager or switch. Failure to tear down the interfaces and datalinks will result in sfaulted datalink and non-functioning interfaces.