C H A P T E R 6 |
This chapter contains information and tasks about using the Logical Domains software that are not described in the preceding chapters.
The following sections describe the restrictions on entering names in the Logical Domains Manager CLI.
The logical domain configuration name (config_name) that you assign to a configuration stored on the system controller must have no more than 64 characters.
This section shows the syntax usage for the ldm subcommands, defines some output terms, such as flags and utilization statistics, and provides examples of the output.
If you are creating scripts that use ldm list command output, always use the -p option to produce the machine-readable form of the output. See To Generate a Parseable, Machine-Readable List (-p) for more information.
The following flags can be shown in the output for a domain.
- | normal |
placeholder | s |
c | starting or stopping |
control domain | t |
d | transition |
delayed reconfiguration | v |
n | virtual I/O domain |
If you use the long (-l) option for the command, the flags are spelled out. If not, you see the letter abbreviation.
The list flag values are position dependent. Following are the values that can appear in each of the five columns from left to right.
Column 1 | Column 3 | Column 5 | n or t | c or - |
Column 2 | Column 4 | s or - | d or - | v or - |
The per virtual CPU utilization statistic (UTIL) is shown on the long (-l) option of the ldm list command. The statistic is the percentage of time, since the last statistics display, that the virtual CPU spent executing on behalf of the guest operating system. A virtual CPU is considered to be executing on behalf of the guest operating system except when it has been yielded to the hypervisor. If the guest operating system does not yield virtual CPUs to the hypervisor, the utilization of CPUs in the guest operating system will always show as 100%.
The utilization statistic reported for a logical domain is the average of the virtual CPU utilizations for the virtual CPUs in the domain.
To view the current software versions installed, do the following and you receive a listing similar to the following.
primary$ ldm list NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME primary active -t-cv 4 1G 0.5% 3d 21h 7m ldg1 active -t--- 5000 8 1G 23% 2m |
primary# ldm list-domain ldg1 NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME ldg1 active -t--- 5000 8 1G 0.3% 2m |
primary$ ldm list-variable boot-device ldg1 boot-device=/virtual-devices@100/channel-devices@200/disk@0:a |
To the Logical Domains Manager, constraints are one or more resources you want to have assigned to a particular domain. You either receive all the resources you ask to be added to a domain or you get none of them, depending upon the available resources. The list-constraints subcommand lists those resources you requested assigned to the domain.
An ldm stop-domain command can time out before the domain completes shutting down. When this happens, an error similar to the following is returned by the Logical Domains Manager.
LDom ldg8 stop notification failed |
However, the domain could still be processing the shutdown request. Use the ldm list-domain command to verify the status of the domain. For example:
# ldm list-domain ldg8 NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME ldg8 active s---- 5000 22 3328M 0.3% 1d 14h 31m |
The preceding list shows the domain as active, but the s flag indicates that the domain is in the process of stopping. This should be a transitory state.
The following example shows the domain has now stopped.
# ldm list-domain ldg8 NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME ldg8 bound ----- 5000 22 3328M |
There is no way to determine the Solaris OS network interface name on a guest, corresponding to a given virtual device, directly from the output provided by the ldm list-* commands. However, you can do this by using a combination of the output from ldm list -l command and the entries under /devices on the Solaris OS guest.
In this example, guest domain ldg1 contains two virtual network devices, net-a and net-c. To find the Solaris OS network interface name in ldg1 that corresponds to net-c, do the following.
Use the ldm command to find the virtual network device instance for net-c.
# ldm list -l ldg1 ... NETWORK NAME SERVICE DEVICE MAC net-a primary-vsw0@primary network@0 00:14:4f:f8:91:4f net-c primary-vsw0@primary network@2 00:14:4f:f8:dd:68 ... # |
To find the corresponding network interface on ldg1, log into ldg1 and find the entry for this instance under /devices.
# uname -n ldg1 # find /devices/virtual-devices@100 -type c -name network@2\* /devices/virtual-devices@100/channel-devices@200/network@2:vnet1 # |
The network interface name is the part of the entry after the colon; that is, vnet1.
Plumb vnet1 to see that it has the MAC address 00:14:4f:f8:dd:68 as shown in the ldm list -l output for net-c in Step 1.
# ifconfig vnet1 vnet1: flags=1000842<BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3 inet 0.0.0.0 netmask 0 ether 0:14:4f:f8:dd:68 # |
You must have enough media access control (MAC) addresses to assign to the number of logical domains, virtual switches, and virtual networks you are going to use. You can have the Logical Domains Manager automatically assign MAC addresses to a logical domain, a virtual network (vnet), and a virtual switch (vswitch), or you can manually assign MAC addresses from your own pool of assigned MAC addresses. The ldm subcommands that set MAC addresses are add-domain, add-vsw, set-vsw, add-vnet, and set-vnet. If you do not specify a MAC address in these subcommands, the Logical Domains Manager assigns one automatically.
The advantage to having the Logical Domains Manager assign the MAC addresses is that it utilizes the block of MAC addresses dedicated for use with logical domains. Also, the Logical Domains Manager detects and prevents MAC address collisions with other Logical Domains Manager instances on the same subnet. This frees you from having to manually manage your pool of MAC addresses.
MAC address assignment happens as soon as a logical domain is created or a network device is configured into a domain. In addition, the assignment is persistent until the device, or the logical domain itself, is removed.
The following topics are addressed in this section:
Logical domains have been assigned the following block of 512K MAC addresses:
00:14:4F:F8:00:00 ~ 00:14:4F:FF:FF:FF
The lower 256K addresses are used by the Logical Domains Manager for automatic MAC address allocation, and you cannot manually request an address in this range:
00:14:4F:F8:00:00 - 00:14:4F:FB:FF:FF
You can use the upper half of this range for manual MAC address allocation:
When you do not specify a MAC address in creating logical domain or a network device, the Logical Domains Manager automatically allocates and assigns a MAC address to that logical domain or network device. To obtain this MAC address, the Logical Domains Manager iteratively attempts to select an address and then checks for potential collisions.
Before selecting a potential address, the Logical Domains Manager first looks to see if it has a recently freed, automatically assigned address saved in a database for this purpose (see Freed MAC Addresses). If so, the Logical Domains Manager selects its candidate address from the database.
If no recently freed addresses are available, the MAC address is randomly selected from the 256K range of addresses set aside for this purpose. The MAC address is selected randomly to lessen the chance of a duplicate MAC address being selected as a candidate.
The address selected is then checked against other Logical Domains Managers on other systems to prevent duplicate MAC addresses from actually being assigned. The algorithm employed is described in Duplicate MAC Address Detection. If the address is already assigned, the Logical Domains Manager iterates, choosing another address, and again checking for collisions. This continues until a MAC address is found that is not already allocated, or a time limit of 30 seconds has elapsed. If the time limit is reached, then the creation of the device fails, and an error message similar to the following is shown.
Automatic MAC allocation failed. Please set the vnet MAC address manually. |
To prevent the same MAC address from being allocated to different devices, one Logical Domains Manager checks with other Logical Domains Managers on other systems by sending a multicast message over the control domain’s default network interface, including the address that the Logical Domain Manager wants to assign to the device. The Logical Domains Manger attempting to assign the MAC address waits for one second for a response back. If a different device on another LDoms-enabled system has already been assigned that MAC address, the Logical Domains Manager on that system sends back a response containing the MAC address in question. If the requesting Logical Domains Manager receives a response, it knows the chosen MAC address has already been allocated, chooses another, and iterates.
By default, these multicast messages are sent only to other managers on the same subnet; the default time-to-live (TTL) is 1. The TTL can be configured using the Service Management Facilities (SMF) property ldmd/hops.
Each Logical Domains Manager is responsible for:
If the Logical Domains Manager on a system is shut down for any reason, duplicate MAC addresses could occur while the Logical Domains Manager is down.
Automatic MAC allocation occurs at the time the logical domain or network device is created and persists until the device or the logical domain is removed.
When a logical domain or a device associated with an automatic MAC address is removed, that MAC address is saved in a database of recently freed MAC addresses for possible later use on that system. These MAC addresses are saved to prevent the exhaustion of Internet Protocol (IP) addresses from a Dynamic Host Configuration Protocol (DHCP) server. When DHCP servers allocate IP addresses, they do so for a period of time (the lease time). The lease duration is often configured to be quite long, generally hours or days. If network devices are created and removed at a high rate without the Logical Domains Manager reusing automatically allocated MAC addresses, the number of MAC addresses allocated could soon overwhelm a typically configured DHCP server.
When a Logical Domains Manager is requested to automatically obtain a MAC address for a logical domain or network device, it first looks to the freed MAC address database to see if there is a previously assigned MAC address it can reuse. If there is a MAC address available from this database, the duplicate MAC address detection algorithm is run. If the MAC address had not been assigned to someone else since it was previously freed, it will be reused and removed from the database. If a collision is detected, the address is simply removed from the database. The Logical Domains Manager then either tries the next address in the database or if none is available, randomly picks a new MAC address.
The Solaris Fault Management Architecture (FMA) reports CPU errors in terms of physical CPU numbers and memory errors in terms of physical memory addresses.
If you want to determine within which logical domain an error occurred and the corresponding virtual CPU number or real memory address within the domain, then you must perform a mapping.
The domain and the virtual CPU number within the domain, which correspond to a given physical CPU number, can be determined with the following procedures.
The domain and the real memory address within the domain, which correspond to a given physical memory address (PA), can be determined as follows.
Generate a long parseable list for all domains.
primary$ ldm ls -l -p |
Look for the line in the list’s MEMORY sections where the PA falls within the inclusive range pa to (pa + size - 1); that is, pa <= PA < (pa + size - 1).
Here pa and size refer to the values in the corresponding fields of the line.
Suppose you have a logical domain configuration as shown in EXAMPLE 6-16, and you want to determine the domain and the virtual CPU corresponding to physical CPU number 5, and the domain and the real address corresponding to physical address 0x7e816000.
Looking through the VCPU entries in the list for the one with the pid field equal to 5, you can find the following entry under logical domain ldg1.
|vid=1|pid=5|util=29|strand=100 |
Hence, the physical CPU number 5 is in domain ldg1 and within the domain it has virtual CPU number 1.
Looking through the MEMORY entries in the list, you can find the following entry under domain ldg2.
ra=0x8000000|pa=0x78000000|size=1073741824 |
Where 0x78000000 <= 0x7e816000 <= (0x78000000 + 1073741824 - 1); that is, pa <= PA <= (pa + size - 1).Hence, the PA is in domain ldg2 and the corresponding real address is 0x8000000 + (0x7e816000 - 0x78000000) = 0xe816000.
The PCI Express (PCI-E) bus on a Sun UltraSPARC T1-based server consists of two ports with various leaf devices attached to them. These are identified on a server with the names pci@780 (bus_a) and pci@7c0 (bus_b). In a multidomain environment, the PCI-E bus can be programmed to assign each leaf to a separate domain using the Logical Domains Manager. Thus, you can enable more than one domain with direct access to physical devices instead of using I/O virtualization.
When the Logical Domains system is powered on, the control (primary) domain uses all the physical device resources, so the primary domain owns both the PCI-E bus leaves.
The example shown here is for a Sun Fire T2000 server. This procedure also can be used on other Sun UltraSPARC T1-based servers, such a Sun Fire T1000 server and a Netra T2000 server. The instructions for different servers might vary slightly from these, but you can obtain the basic principles from the example. Mainly, you need to retain the leaf that has the boot disk and remove the other leaf from the primary domain and assign it to another domain.
Verify that the primary domain owns both leaves of the PCI Express bus.
primary# ldm list-bindings primary ... IO DEVICE PSEUDONYM OPTIONS pci@780 bus_a pci@7c0 bus_b ... |
Determine the device path of the boot disk, which needs to be retained.
primary# df / / (/dev/dsk/c1t0d0s0 ): 1309384 blocks 457028 files |
Determine the physical device to which the block device c1t0d0s0 is linked.
primary# ls -l /dev/dsk/c1t0d0s0 lrwxrwxrwx 1 root root 65 Feb 2 17:19 /dev/dsk/c1t0d0s0 -> ../ ../devices/pci@7c0/pci@0/pci@1/pci@0,2/LSILogic,sas@2/sd@0,0:a |
In this example, the physical device for the boot disk for domain primary is under the leaf pci@7c0, which corresponds to our earlier listing of bus_b. This means that we can assign bus_a (pci@780) of the PCI-Express bus to another domain.
Check /etc/path_to_inst to find the physical path of the onboard network ports.
primary# grep e1000g /etc/path_to_inst |
Remove the leaf that does not contain the boot disk (pci@780 in this example) from the primary domain.
primary# ldm remove-io pci@780 primary |
Add this split PCI configuration (split-cfg in this example) to the system controller.
primary# ldm add-config split-cfg |
This configuration (split-cfg) is also set as the next configuration to be used after the reboot.
Note - Currently, there is a limit of 8 configurations that can be saved on the SC, not including the factory-default configuration. |
Reboot the primary domain so that the change takes effect.
primary# shutdown -i6 -g0 -y |
Add the leaf (pci@780 in this example) to the domain (ldg1 in this example) that needs direct access.
primary# ldm add-io pci@780 ldg1 Notice: the LDom Manager is running in configuration mode. Any configuration changes made will only take effect after the machine configuration is downloaded to the system controller and the host is reset. |
If you have an Infiniband card, you might need to enable the bypass mode on the pci@780 bus. See Enabling the I/O MMU Bypass Mode on a PCI Bus for information on whether you need to enable the bypass mode.
Reboot domain ldg1 so that the change takes effect.
All domains must be inactive for this reboot. If you are configuring this domain for the first time, the domain will be inactive.
ldg1# shutdown -i6 -g0 -y |
Confirm that the correct leaf is still assigned to the primary domain and the correct leaf is assigned to domain ldg1.
primary# ldm list-bindings primary NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME primary active -n-cv SP 4 4G 0.4% 18h 25m ... IO DEVICE PSEUDONYM OPTIONS pci@7c0 bus_b ... ---------------------------------------------------------------- NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME ldg1 active -n--- 5000 4 2G 10% 35m ... IO DEVICE PSEUDONYM OPTIONS pci@780 bus_a ... |
This output confirms that the PCI-E leaf bus_b and the devices below it are assigned to domain primary, and bus_a and its devices are assigned to ldg1.
If you have an Infiniband Host Channel Adapter (HCA) card, you might need to turn the I/O memory management unit (MMU) bypass mode on. By default, Logical Domains software controls PCI-E transactions so that a given I/O device or PCI-E option can only access the physical memory assigned within the I/O domain. Any attempt to access memory of another guest domain is prevented by the I/O MMU. This provides a higher level of security between the I/O domain and all other domains. However, in the rare case where a PCI-E or PCI-X option card does not load or operate with the I/O MMU bypass mode off, this option allows you to turn the I/O MMU bypass mode on. However, if you turn the bypass mode on, there no longer is a hardware-enforced protection of memory accesses from the I/O domain.
The bypass=on option turns on the I/O MMU bypass mode. This bypass mode should be enabled only if the respective I/O domain and I/O devices within that I/O domain are trusted by all guest domains. This example turns on the bypass mode.
primary# ldm add-io bypass=on pci@780 ldg1 |
The virtual network terminal server daemon, vntsd(1M), enables you to provide access for multiple domain consoles using a single TCP port. At the time of domain creation, the Logical Domains Manager assigns a unique TCP port to each console by creating a new default group for that domain’s console. The TCP port is then assigned to the console group as opposed to the console itself. The console can be bound to an existing group using the set-vcons subcommand.
Bind the consoles for the domains into one group.
The following example shows binding the console for three different domains (ldg1, ldg2, and ldg3) to the same console group (group1).
primary# ldm set-vcons group=group1 service=primary-vcc0 ldg1 primary# ldm set-vcons group=group1 service=primary-vcc0 ldg2 primary# ldm set-vcons group=group1 service=primary-vcc0 ldg3 |
Connect to the associated TCP port (localhost at port 5000 in this example).
# telnet localhost 5000 primary-vnts-group1: h, l, c{id}, n{name}, q: |
List the domains within the group by selecting l (list).
primary-vnts-group1: h, l, c{id}, n{name}, q: l DOMAIN ID DOMAIN NAME DOMAIN STATE 0 ldg1 online 1 ldg2 online 2 ldg3 online |
You can move a logical domain, which is not running, from one server to another. Before you move the domain, if you set up the same domain on two servers, the domain would be easier to move. In fact, you do not have to move the domain itself; you only have to unbind and stop the domain on one server and bind and start the domain on the other server.
Create a domain with the same name on two servers; for example, create domainA1 on serverA and serverB.
Add a virtual disk server device and a virtual disk to both servers. The virtual disk server opens the underlying device for export as part of the bind.
Bind the domain only on one server; for example, serverA. Leave the domain inactive on the other server.
This section describes how to remove all guest domains and revert to a single OS instance that controls the whole server.
List all the logical domain configurations on the system controller.
primary# ldm ls-config |
Remove all configurations (config_name) previously saved to the system controller (SC). Use the following command for each such configuration.
primary# ldm rm-config config_name |
Once you remove all the configurations previously saved to the SC, the factory-default domain would be the next one to use when the control domain (primary) is rebooted.
Stop all guest domains using the -a option.
primary# ldm stop-domain -a |
List all domains to see all the resources attached to guest domains.
primary# ldm ls |
Release all the resources attached to guest domains. To do this, use the ldm unbind-domain command for each guest domain (ldom) configured in your system.
Note - You might not be able to unbind an I/O domain in a split-PCI configuration if it is providing services required by the control domain. In this situation, skip this step. |
primary# ldm unbind-domain ldom |
primary# shutdown -i1 -g0 -y |
Power-cycle the system controller so that the factory-default configuration is reloaded.
sc> poweroff sc> poweron |
This section describes the changes in behavior in using the Solaris OS that occur once a configuration created by the Logical Domains Manager is instantiated; that is, domaining is enabled.
Note - Any discussion about whether domaining is enabled pertains only to Sun UltraSPARC T1–based platforms. Otherwise, domaining is always enabled. |
Domaining is enabled once a logical domains configuration created by the Logical Domains Manager is instantiated. If domaining is enabled, the OpenBoot firmware is not available after the Solaris OS has started, because it is removed from memory.
To reach the ok prompt from the Solaris OS, you must halt the domain. You can use the Solaris OS halt command to halt the domain.
Whenever performing any maintenance on a system running LDoms software that requires power-cycling the server, you must save your current logical domain configurations to the SC first.
The OpenBoot power-off command does not power down a system. To power down a system while in OpenBoot firmware, use your system controller’s or system processor’s poweroff command. The OpenBoot power-off command displays the following message.
NOTICE: power-off command is not supported, use appropriate NOTICE: command on System Controller to turn power off. |
If domaining is not enabled, the Solaris OS normally goes to the OpenBoot prompt after a break is issued. The behavior described in this section is seen in two situations:
You press the L1-A key sequence when the input device is set to keyboard.
You enter the send break command when the virtual console is at the telnet prompt.
If domaining is enabled, you receive the following prompt after these types of breaks.
c)ontinue, s)ync, r)eboot, h)alt? |
Type the letter that represents what you want the system to do after these types of breaks.
The following table shows the expected behavior of halting or rebooting the control (primary) domain.
Note - The question in TABLE 6-1 regarding whether domaining is enabled pertains only to the Sun UltraSPARC T1 processors. Otherwise, domaining is always enabled. |
The section describes information to be aware of in using Advanced Lights Out Manager (ALOM) chip multithreading (CMT) with the Logical Domains Manager. For more information about using the ALOM CMT software, refer to the Advanced Lights Out Management (ALOM) CMT v1.3 Guide.
An additional option is available to the existing ALOM CMT command.
bootmode [normal|reset_nvram|bootscript=strong|config=”config-name”] |
The config=”config-name” option enables you to set the configuration on the next power on to another configuration, including the factory-default shipping configuration.
You can invoke the command whether the host is powered on or off. It takes effect on the next host reset or power on.
To Reset the Logical Domain Configuration to the Default or Another Configuration |
Reset the logical domain configuration on the next power on to the default shipping configuration by executing this command in ALOM CMT software.
sc> bootmode config=”factory-default” |
You also can select other configurations that have been created with the Logical Domains Manager using the ldm add-config command and stored on the system controller (SC). The name you specify in the Logical Domains Manager ldm add-config command can be used to select that configuration with the ALOM CMT bootmode command. For example, assume you stored the configuration with the name ldm-config1.
sc> bootmode config=”ldm-config1” |
Refer to the ldm(1M) man page or the Logical Domains (LDoms) Manager 1.0.3 Man Page Guide for more information about the ldm add-config command.
The Logical Domains Manager uses the Solaris OS Basic Security module (BSM) auditing capability. BSM auditing provides the means to examine the history of actions and events on your control domain to determine what happened. The history is kept in a log of what was done, when it was done, by whom, and what was affected.
If you want to use this auditing capability, this section describes how to enable, verify, disable, print output, and rotate audit logs. You can find further information about BSM auditing in the Solaris 10 System Administration Guide: Security Services.
You can enable BSM auditing in one of two ways. When you want to disable auditing, be sure you use the same method that you used in enabling. The two methods are:
Use the enable-bsm.fin finish script in the Solaris Security Toolkit.
The enable-bsm.fin script is not used by default by the ldm_control-secure.driver. You must enable the finish script in your chosen driver.
Here are the procedures for both methods.
Copy the ldm_control-secure.driver to my-ldm.driver, where my-ldm.driver is the name for your copy of the ldm_control-secure.driver.
Copy the ldm_control-config.driver to my-ldm-config.driver, where my-ldm-config.driver is the name for your copy of the ldm_control-config.driver.
Copy the ldm_control-hardening.driver to my-ldm-hardening.driver, where my-ldm-hardening.driver is the name for your copy of the ldm_control-hardening.driver.
Edit my-ldm.driver to refer to the new configuration and hardening drivers, my-ldm-control.driver and my-ldm-hardening.driver, respectively.
Edit my-ldm-hardening.driver, and remove the pound sign (#) from in front of the following line in the driver.
enable-bsm.fin |
# /opt/SUNWjass/bin/jass-execute -d my-ldm.driver |
You can disable auditing in one of two ways, depending on how you enabled it. See Enabling and Using BSM Auditing.
In a logical domains environment, the virtual switch service running in a service domain can directly interact with GLDv3-compliant network adapters. Though non-GLDv3 compliant network adapters can be used in these systems, the virtual switch cannot interface with them directly. See Configuring Virtual Switch and Service Domain for NAT and Routing for information about how to use non-GLDv3 compliant network adapters.
The virtual switch (vswitch) is a layer-2 switch, that also can be used as a network device in the service domain. The virtual switch can be configured to act only as a switch between the virtual network (vnet) devices in the various logical domains but with no connectivity to a network outside the box through a physical device. In this mode, plumbing the vswitch as a network device and enabling IP routing in the service domain enables virtual networks to communicate outside the box using the service domain as a router. This mode of operation is very essential to provide external connectivity to the domains when the physical network adapter is not GLDv3-compliant.
The advantages of this configuration are:
The virtual switch does not need to use a physical device directly and can provide external connectivity even when the underlying device is not GLDv3-compliant.
The configuration can take advantage of the IP routing and filtering capabilities of the Solaris OS.
To Set Up the Virtual Switch to Provide External Connectivity to Domains |
Create a virtual switch with no associated physical device.
If assigning an address, ensure that the virtual switch has an unique MAC address.
primary# ldm add-vsw [mac-addr=xx:xx:xx:xx:xx:xx] primary-vsw0 primary |
Plumb the virtual switch as a network device in addition to the physical network device being used by the domain.
See To Configure the Virtual Switch as the Primary Interface for more information about plumbing the virtual switch.
Configure the virtual switch device for DHCP, if needed.
See To Configure the Virtual Switch as the Primary Interface for more information about configuring the virtual switch device for DHCP.
Configure IP routing in the service domain, and set up required routing tables in all the domains.
For information about how to do this, refer to the section on “Packet Forwarding and Routing on IPv4 Networks” in Chapter 5 “Configuring TCP/IP Network Services and IPv4 Administration” in the System Administration Guide: IP Services in the Solaris Express System Administrator Collection.
Internet Protocol Network Multipathing (IPMP) provides fault-tolerance and load balancing across multiple network interface cards. By using IPMP, you can configure one or more interfaces into an IP multipathing group. After configuring IPMP, the system automatically monitors the interfaces in the IPMP group for failure. If an interface in the group fails or is removed for maintenance, IPMP automatically migrates, or fails over, the failed interface’s IP addresses. In a Logical Domains environment, either the physical or virtual network interfaces can be configured for failover using IPMP.
A logical domain can be configured for fault-tolerance by configuring its virtual network devices to an IPMP group. When setting up an IPMP group with virtual network devices, in a active-standby configuration, set up the group to use probe-based detection. Link-based detection and failover currently are not supported for virtual network devices in Logical Domains 1.0.3 software.
The following diagram shows two virtual networks (vnet1 and vnet2) connected to separate virtual switch instances (vsw0 and vsw1) in the service domain, which, in turn, use two different physical interfaces (e1000g0 and e1000g1). In the event of a physical interface failure, the IP layer in LDom_A detects failure and loss of connectivity on the corresponding vnet through probe-based detection, and automatically fails over to the secondary vnet device.
Further reliability can be achieved in the logical domain by connecting each virtual network device (vnet0 and vnet1) to virtual switch instances in different service domains (as shown in the following diagram). Two service domains (Service_1 and Service_2) with virtual switch instances (vsw1 and vsw2) can be set up using a split-PCI configuration. In this case, in addition to network hardware failure, LDom_A can detect virtual network failure and trigger a failover following a service domain crash or shutdown.
Refer to the Solaris 10 System Administration Guide: IP Services for more information about how to configure and use IPMP groups.
Network failure detection and recovery can also be set up in a Logical Domains environment by configuring the physical interfaces in the service domain into a IPMP group. To do this, configure the virtual switch in the service domain as a network device, and configure the service domain itself to act as an IP router. (Refer to the Solaris 10 System Administration Guide: IP Services for information on setting up IP routing).
Once configured, the virtual switch sends all packets originating from virtual networks (and destined for an external machine), to its IP layer, instead of sending the packets directly via the physical device. In the event of a physical interface failure, the IP layer detects failure and automatically re-routes packets through the secondary interface.
Since the physical interfaces are directly being configured into a IPMP group, the group can be set up for either link-based or probe-based detection. The following diagram shows two network interfaces (e1000g0 and e1000g1) configured as part of an IPMP group. The virtual switch instance (vsw0) has been plumbed as a network device to send packets to its IP layer.
Copyright © 2008, Sun Microsystems, Inc. All rights reserved.