C H A P T E R 9

Domain Services

Sun Fire high-end system hardware incorporates internal, private point-to-point Ethernet connections between the SC and each domain. This network, called the Management Network (MAN), is used to provide support services for each domain. This chapter describes those services.

This chapter includes the following sections:

Management Network Overview

Management Network Services

Management Network Overview

The Management Network (MAN) function maintains the private point-to-point network connections between the SC and each domain. No packets addressed to one domain can be routed along the network connection between the SC and another domain (FIGURE 9-1).

FIGURE 9-1 Management Network Overview

Figure depicting an overview of the Management Network.

I1 Network

The hardware built into the Sun Fire high-end system chassis to support MAN is complex. It includes 18 Network Interface Cards (NICs) on each SC that are connected in a point-to-point fashion to NICs located on each of the 18 expander I/O slots on the Sun Fire 15K system and on each of the 9 expander I/O slots on the Sun Fire 12K system. Using this design, the number of point-to-point Ethernet links between an SC and a given DSD varies based on the number of I/O boards configured in that DSD. Each NIC from the SC connects to a hub and NIC on the I/O board. The NIC is an internal part of the I/O board and not a separate adapter card. Likewise, the Ethernet hub is on the I/O board. The hub is intelligent and can collect statistics.

All of these point-to-point links are collectively called the I1 network. Since there can be multiple I/O boards in a given domain, multiple redundant network connections from the SC to a domain are possible. FIGURE 9-2 shows a network overview of the Sun Fire E25K/15K.

FIGURE 9-2 I1 Network Overview of the Sun Fire E25K/15K

Figure depicting an overview of the I1 network for the Sun Fire 15K/E25K.

Note - The I1 MAN network is a private network, not a general-purpose network. No external IP traffic should be routed across it. Access to MAN is restricted to the system controller and the domains.

On the SC, MAN software creates a meta-interface for the I1 network, presenting to the Solaris OS a single network interface, scman0. For more information, refer to the Solaris scman(1M) man page.

MAN software detects communication errors and automatically initiates a path switch, provided an alternate path is available. MAN software also enforces domain isolation of network traffic on the I1 network. Similar software operates on the domain side.

I2 Network

There is also an internal network between the two system controllers consisting of two NICs per system controller. This network is called the I2 network. It is a private SC-to-SC network and is entirely separate from the I1 network.

MAN software creates a meta-interface for the I2 network as well. This interface is presented to the Solaris software as scman1. As with the I1 network, I2 has a mechanism for detecting path failure and switching paths, providing an alternative is available.

FIGURE 9-3 I2 Network Overview

Figure depicting an overview of the I2 network.

The virtual network adapter on the SC presents itself as a standard network adapter. It can be managed and administered just like any other network adapter (for example, qfe, hme). The usual system administration tools such as ndd(1M), netstat(1M), and ifconfig(1M), can be used to manage the virtual network adapter. Certain operations of these tools (for example, changing the Ethernet address) should not be allowed for security reasons.

MAN operates and is managed as an IP network with special characteristics (for example, IP forwarding is disallowed by the MAN software). As such, the MAN operation is the same as any other IP network, with the previously noted exception. Domains can be connected to your network depending on your site configuration and security requirements. Connecting domains is not within the scope of this document-refer to the System Administration Guide: Resource Management and Network Services.

External Network Monitoring

External Network Monitoring for the Sun Fire high-end system provides highly available network connections from the SCs to customer networks called communities. This feature is built on top of the IP Network Multipathing (IPMP) framework provided in the Solaris 9 OS. For more information on IPMP, refer to the System Administration Guide: IP Services.

External networks can consist of communities. You can have zero, one, or two communities. Zero communities means external networks are not monitored. During installation, user communities are connected by physical cable to the RJ45 jacks on the SC connecting a node to the network.

For more information on connecting external networks, refer to the Sun Fire 15K/12K System Site Planning Guide. FIGURE 9-4 shows an external network overview.

FIGURE 9-4 External Network Overview

Figure depicting an external network overview.

The term community refers to an IP network at your site. For example, you might have an engineering community and an accounting community. A community name is used as the interface group name. An interface group is a group of network interfaces that attach to the same community.

Configuring External Network Monitoring requires allocating several additional IP addresses for each system controller.

The addresses can be categorized as follows:

Test addressees - These IP addresses are assigned to the external network interfaces on each system controller. Each IP test address is used to test the health of the particular network interface to which it is assigned. One IP test address is permanently assigned to each network interface. They are permanently associated with a particular network interface. If a network interface fails, the IP test address associated with that network interface becomes unreachable.

Failover addresses - There are two types of failover addresses:

SC path group specific addresses - These IP addresses are assigned to a particular interface group on each system controller. They are used to provide highly available IP connectivity to a particular system controller for a given community. The SC path group specific address is reachable as long as at least one of the network interfaces in the interface group is functioning.

Note - An SC path group-specific address is not needed if there is only one network interface in an interface group. Since there is no other network interface in the group to failover to, only the test addresses and the community failover addresses are required.

Community failover addresses - These IP addresses are assigned to a particular community on the MAIN SC (that is, Community C1). They are used to provide IP connectivity to the MAIN SC, either SC0 or SC1.

All external software should reference the community failover address when communicating with the SC. This address always connects to the main SC. That way, if a failover occurs, external clients do not need to alter their configuration to reach the SC. For more information on SC failover, see Chapter 12.

MAN Daemons and Drivers

For more information on the MAN daemon and device drivers, refer to the SMS mand(1M) and Solaris scman(1M) and dman(1M) man pages. See also Management Network Daemon.

Management Network Services

The primary network services that MAN provides between the SC and the domains are:

Domain consoles

Message logging

Dynamic reconfiguration (DR)

Network boot/Solaris installation

System controller (SC) heartbeats

Domain Console

The software running in a domain (OpenBoot PROM, kadb, and Solaris software) uses the system console for critical communications.

The domain console supports a login session and is secure, since the default configuration of the Solaris environment allows only the console to accept superuser logins. Domain console access is provided securely to remote administrators over a possibly public network.

The behavior of the console reflects the health of the software running in the domain. Character echo for user entries is nearly equivalent to that of a 9600-baud serial terminal attached to the domain. Output characters that are not echoes of user input are typically either the output from an executed command or from a command interpreter, or they might be unsolicited log messages from the Solaris software. Activity on other domains or SMS support activity for the domain do not noticeably alter the response latency of user entry echo.

You can run kadb on the domain's Solaris software from the domain console. Interactions with the OpenBoot PROM running on a domain use the domain console. The console can serve as the destination for log messages from the Solaris software; refer to syslog.conf(4). The console is available when software (Solaris, OpenBoot PROM, kadb) is running on the domain.

You can open multiple connections to view the domain console output. However, the default is an exclusive locked connection.

For more information, see SMS Console Window.

A domain administrator can forcibly break the domain console connection held by another domain.

You can forcibly break into the OpenBoot PROM or kadb from the domain console; however, it is not suggested. (This is a replacement for the physical L1-A or STOP-A key sequence available on a Sun SPARC® system with a physical console.) SMS captures console output history for subsequent analysis of domain crashes. A log of the console output for every domain is available in /var/opt/SUNWSMS/adm/domain-id/console.

The Sun Fire high-end system provides the hardware to either implement a shared-memory console or implement an alternate network data path for console. The hardware utilized for a shared-memory console imposes less direct latency upon console data transfers, but is also used for other monitoring and control purposes for all domains, so there is a risk of latency introduced by contention for the hardware resources.

MAN provides private network paths to securely transfer domain console traffic to the SC; see Management Network Services. The console has a dual-pathed nature so that at least one path provides acceptable console response latency when the Solaris software is running. The dual-pathed console is robust in the face of errors. It detects failures on one domain console path and fails over to the other domain console path automatically. It supports user-directed selection of the domain console path to use.

The smsconfig(1M) command is the SC configuration utility that initially configures or later modifies the hostname, IP address, and netmask settings used by management network daemon, mand(1M). See Management Network Daemon.

The mand daemon initializes and updates these respective fields in the platform configuration database (pcd).

The mand daemon is automatically started by ssd. The Management Network daemon runs on the main SC in main mode and on the spare SC in spare mode.

For more information, refer to the SMS console(1M), mand(1M), and smsconfig(1M) man pages as well as the Solaris dman(1M) and scman(1M) man pages.

Message Logging

When configured to do so, MAN transports copies of important syslog messages from the domains to disk storage on the SC. This facilitates failure analysis for crashed or unbootable domains. For more information, see Log File Maintenance.

Dynamic Reconfiguration

The MAN software layer is used to simplify the interface to the MAN hardware. MAN software handles the aspects of dynamic reconfiguration (DR) used by a DSD without requiring network configuration work by the domain or platform administrator.

Software in the domains using MAN need not be aware of which SC is currently the main SC. For more information on dynamic reconfiguration, refer to the System Management Services (SMS) 1.6 Dynamic Reconfiguration User Guide.

Network Boot and Solaris Software Installation

The SC provides network Solaris boot services to each domain.

Note - Diskless Sun Fire high-end system domains cannot be supported entirely by network services from the SC; the SC network boot service is intended primarily for recovery after a catastrophic disk failure on the domain.

When Solaris software is first installed on a domain, the network interface connecting it to the MAN is automatically created for subsequent system reboots. There are no additional tasks required by the domain administrator to configure or use MAN.

MAN is configured as a private network. A default address assignment for the Management Network is provided, using the IP address space reserved for private networks. You can override the default address assignment for MAN to handle the case where the Sun Fire high-end system is connected to a private customer network that already uses the selected MAN default IP address range.

The SC supports simultaneous network boots of domains running at least two different versions of Solaris software.

The SC provides software installation services to no more than one domain at a time.

SC Heartbeats

The I2 network supplies the intersystem controller communication. This is also called the heartbeat network. SMS failover mechanisms on the main SC use this network as one means of determining the health of the spare SC. For more information, see Chapter 12. For a description of the I2 network, see I2 Network.