This appendix introduces some of the basic concepts of campus clustering and provides some configuration and setup examples. It does not attempt to fully explain clustering, provide information on clustering administration, or to furnish details about hardware installation and configuration. For this information, see other Sun Cluster manuals, in particular the Sun Cluster 3.0 12/01 Concepts guide, the Sun Cluster 3.0 12/01 System Administration Guide, and the Sun Cluster 3.0 12/01 Hardware Guide, as well as manuals specifically referenced here.
The only significant difference between traditional clustering and campus clustering is that of distance. In campus clustering, the nodes of a cluster configuration can be up to several kilometers apart. This increases the likelihood that, in the case of a catastrophe such as a fire or earthquake, at least one server and its storage will survive.
Sun Cluster software currently supports exactly two nodes in a campus cluster configuration. However, both two- and three-room configurations are supported. A room can be thought of as a functionally independent hardware grouping (such as a node and its attendant storage, or a quorum device physically separated from any nodes) that has been separated from other rooms to increase the likelihood of failover and redundancy in case of accident or failure. The definition of a room therefore depends on the type of failures to be safeguarded against, as indicated by Table E-1.
Table E-1 Definitions of "Room"
Failure Scenario |
Sample Definitions of Separate Rooms |
---|---|
Power line failure |
Isolated and independent power supplies |
Minor accidents, furniture collapse, seepage, etc. |
Different parts of physical room |
Small fire. Sprinklers (fire) starting |
Different physical area s (for example, sprinkler zone) |
Structural failure (building-wide fire, for example) |
Different buildings |
Large-scale natural disaster (for example, earthquake or flood) |
Different corporate campuses up to several kilometers apart |
Since all campus clusters are two-node clusters, each campus cluster must have a quorum disk. In two-room configurations, the quorum disk occupies the same room as one node (see "Example Two-Room Configuration"). In the case of a three-room configuration, the third room is used for the quorum disk ("Example Three-Room Configuration").
In a two-room configuration, the quorum disk should be located in the room that is more likely to survive an accident or catastrophe in the event that all cluster transport and disk connectivity is lost between rooms. (If only cluster transport is lost, the node sharing a room with the quorum disk will not necessarily be the room that reserves the quorum disk first.) For more information about quorum and quorum devices, see the Sun Cluster 3.0 12/01 Concepts.
The advantage to a three-room cluster is that, if any one of the three rooms is lost, automatic failover should typically be possible; whereas with a two-room cluster, if an entire room is lost, automatic failover is possible only if the surviving room contains the quorum disk. Only a three-room configuration guarantees system availability in the event of complete loss of an entire room (in the absence of any other failures).
As with non-campus configurations, data integrity will be compromised if there are other unrecovered I/O failures present when a room is destroyed, and the most up-to-date submirror was in the destroyed room.
In a campus cluster configuration, each of the two rooms used for nodes should have an equal number of shared disks. (In a two-room configuration, one room can have a separate quorum disk, so the two rooms need not have the same number of total disks.) Mirroring of the shared disks must always be done between rooms, rather than within rooms. In other words, both submirrors of a two-way mirror should never be in the same room. Mirroring is required for all campus cluster configurations, since RAID-5 alone does not lend itself to providing data redundancy across rooms.
If you use Solstice DiskSuite as your volume manager for shared device groups, pay special consideration to the distribution of replicas. In two-room configurations, all disksets should be configured with an additional replica in the room that houses the cluster quorum disk. Further, all Solstice DiskSuite device groups should be configured to use the node in the quorum disk room as their default primary room. In three-room configurations, the third room should not only house the quorum disk, but also include at least one extra disk configured into each of the disksets. Each diskset should include a third-room disk with an extra Solstice DiskSuite replica per diskset. Quorum disks can be used as metadb replicas in a metaset. Sun Cluster software does not currently support using the third room for data storage.
Sun Cluster software supports campus cluster configurations with rooms up to 10km apart.
This section provides examples of two- and three-room campus clustering configurations:
Figure E-1 shows a two-room configuration using Sun StorEdge T3 partner groups.
Figure E-2 shows a three-room campus cluster using Sun StorEdge A5x00 disk arrays.
The examples in this chapter illustrate general configurations and are not intended to indicate required or recommended setups. The type of storage device shown, for example, is purely for illustrative purposes. For simplicity's sake, the diagrams and explanations concentrate only on features unique to understanding campus clustering, so that, for example, public-network Ethernet connections are not shown.
See "Additional Campus Cluster Configuration Examples" for other example setups.
A two-room configuration is defined as follows:
Two separate rooms
Both rooms with one node each and disk subsystems
Data mirrored across disk subsystems in these rooms
At least one disk subsystem, attached to both hosts, used as a quorum device, located in one of the rooms
In the event of loss of the room containing the quorum disk, the system will be unable to recover automatically. Recovery will require operator intervention.
Figure E-1 shows a sample two-room configuration using a partner-group of Sun StorEdge T3/T3+ disk arrays in each room.
Figure E-1 is similar to a standard non-campus configuration; the most obvious difference is that Fibre Channel switches have been added to switch from multimode to single-mode fibers.
Although not so depicted in this chapter, campus clustering allows for configurations using multiple storage arrays. Where large storage environments are required, additional SAN switches for Sun StorEdge T3/T3+ arrays might be required.
A three-room configuration is defined as follows:
Three separate rooms
Two rooms with one node each and an equal number of disk arrays, in this case, Sun StorEdge A5x00 disk subsystems; data is mirrored across disk-subsystems in these rooms
Third room with at least one disk subsystem attached to both hosts to be used as a quorum device
Figure E-2 shows this configuration using Sun StorEdge A5x00 disk arrays. Note that, unlike Figure E-1, which used Sun StorEdge T3 disk trays, this configuration does not use Fibre Channel switches to connect Sun StorEdge A5x00 disk arrays. (Switches are not needed, as long-wave GBICs are already present in the A5x00s and should also be present in the servers' host bus adapters.)
In this configuration, as long as at least two rooms are up and communicating, recovery will be automatic. This is the only configuration in which loss of any one room is guaranteed to be automatically handled. Loss of two rooms requires the replacement or rebuilding of one room and typically requires SunService intervention.
This section provides an overview of requirements for campus clustering. For specific information on supported and required software and hardware, see the Sun Cluster 3.0 12/01 Release Notes and consult your Sun sales representative.
Campus clusters require the following software:
Sun Cluster 3.0 Release 12/01 or later
Solaris 8
Volume management software (such as Solstice DiskSuite)
See the Sun Cluster 3.0 12/01 Release Notes or contact your sales representative for a list of Sun Cluster-certified servers.
Fibre Channel switches. Sun StorEdge T3/T3+ disk arrays do not connect directly to single-mode fiber, so a cluster using these devices requires Fibre Channel switches with long-wave GBICs installed in order to connect single-mode fibers to multimode fibers coming from disk arrays, hosts, and host bus adapters. (See Figure E-1.)
Storage fiber connections. For long-distance connections (up to 10km) between rooms, use 9/125-micron single-mode fiber.
When connecting Sun StorEdge T3/T3+ disk trays to Fibre Channel switches, use 50/125-micron multimode fibers at distances not exceeding 500m.
Host adapters. Connect Sun StorEdge A5x00 disk arrays using SOC+-based SBus host adapters.
With Sun StorEdge T3/T3+ disk trays, use a SAN-capable host adapter. Examples include the Sun StorEdge SBus Dual Fibre Channel Network Adapter, the Sun StorEdge PCI Single Fibre Channel Network Adapter, the Sun StorEdge PCI Dual Fibre Channel Network Adapter+, and the Sun StorEdge cPCI Dual Fibre Channel Network Adapter.
Sun Cluster 3.0 supports campus clustering configurations using various storage devices.These storage devices include Sun StorEdge A5x00 disk array, the Sun StorEdge 9910/9960 and the Sun StorEdge T3/T3+ disk array. The Sun StorEdge T3/T3+ arrays can be configured either singly ("single-controller," as shown in Figure E-5) or in pairs ("partner-group," as shown in Figure E-1). Consult your Sun sales representative for specific configurations.
Long-wave gigabit interface converters (LWGBICs) should be used for the StorEdge A5x00 and Sun StorEdge T3/T3+ for distances greater than 500 meters and up to 10Km. (Use shortwave GBICs for distances less than 500 meters.) Consult Sun Professional Services for implementations with long-wave GBICs.
A campus cluster configuration must include two (or more) SANs for storage devices, in which the storage forming one submirror is in one room and mirrored to its submirror in another, physically separate room. Additionally, the submirrors of a mirror must be on different host bus adapters. For more on configuring SANs, refer to the Sun StorEdge Network FC Switch-8 and Switch-16 Installation and Configuration Guide, Sun SAN 3.0.
A quorum disk can be either a separate device or one of the dedicated storage devices (either Sun StorEdge T3/T3+ or A5x00).
Because of limitations involving switch connectivity, it is not currently possible to use Sun StorEdge T3s in the third, quorum room.
The following components make up the campus cluster interconnect hardware configuration:
SunFastEthernet Adapter. The SunFastEthernet Adapter provides 10/100 Mbps Ethernet functionality with an RJ-45 connector.
Media converters. Use RJ-45 media converters to convert from copper to optical fiber connectors. Consult your Sun sales representative for additional information about media converters.
Two 9/125-micrometer single-mode fiber pairs.
Figure E-3 shows the setup for the cluster interconnect:
For the most part, using interconnect, storage, and fibre channel hardware does not differ markedly from non-campus cluster configurations.
The steps for installing Ethernet-based campus cluster interconnect hardware are very similar to those for non-campus clusters. Refer to the Sun Cluster 3.0 12/01 Hardware Guide for those steps.
When installing the media converters, consult the documentation that came with them, including requirements for fiber connections.
The steps for installing Sun StorEdge A5x00 and Sun StorEdge T3/T3+ arrays are very similar to those for non-campus clusters. Refer to the Sun Cluster 3.0 12/01 Hardware Guide for those steps.
However, when installing Sun StorEdge A5x00 arrays at distances greater than 500m, install the Sun Long Wave GBICs as indicated in the Sun StorEdge LW GBIC Service Manual. This manual also includes single-mode fiber specifications.
Campus clusters using Sun StorEdge T3/T3+ arrays require Sun Fibre Channel switches to mediate between multimode and single-mode fibers. The steps for configuring the settings on the Sun Fibre Channel switches are very similar to those for non-campus clusters.
Cascaded-switch SAN setups have requirements that are distinct from those of non-cascades setups, especially with regard to mirroring and zoning. Consult the Sun StorEdge Network FC Switch-8 and Switch-16 Installation and Configuration Guide, Sun SAN 3.0, as well as your service representative, for more information. Also see the Sun StorEdge Network FC Switch-8 and Switch-16 Configuration Supplement for SAN 3.1 Disaster Tolerance Configurations for more information about using the FC switches in a campus cluster configuration.
"Campus Cluster Configuration Examples" showed two possible configurations for campus clustering: a two-room configuration using Sun StorEdge T3 partner groups, and a three-room configuration using Sun StorEdge A5x00 arrays. While detailing all of the configurations possible in campus clustering is far beyond the scope of this document, the following illustrations depict some other variations on the setups previously shown.
The following are two-room examples:
Two-room campus cluster with a multipathing solution implemented(Figure E-4)
Two-room campus cluster without a multipathing solution implemented(Figure E-5)
Two-room cluster using Sun StorEdge A5x00 disk arrays (Figure E-6)
The following are three-room examples:
Three-room campus cluster with a multipathing solution implemented(Figure E-7)
Three-room campus cluster without a multipathing solution implemented(Figure E-8)
Three-room cluster using Sun StorEdge A5x00 disk arrays (Figure E-9)
Figure E-1 depicted a two-room campus cluster using Sun StorEdge T3 partner groups and four Fibre Channel switches. Figure E-4 shows a similar setup with a multipathing solution implemented. The four switches are added to the cluster for greater redundancy and potentially better I/O throughput. This configuration could be implemented using Sun StorEdge T3 partner groups or Sun StorEdge 9910/9960 arrays with Sun StorEdge Traffic Manager software installed.
Figure E-5 is similar to Figure E-1, except that it does not have a multipathing solution implemented. This configuration could be implemented using Sun StorEdge T3/T3+ arrays in single-controller configurations, rather than partner-groups.
Figure E-6 depicts a two-room campus cluster using Sun StorEdge A5x00s. Note the absence of switches.
The following examples all show three-room campus cluster configurations.
As mentioned in "Storage Devices", because of potential limitations with switch connectivity, only the Sun StorEdge A5x00 disk arrays can be used for the third, quorum room. Figure E-7 and Figure E-8 show Sun StorEdge A5x00 disk arrays in the quorum room.
Figure E-7 is the same as Figure E-1, except that the quorum disk is separate and in a third room.
Like Figure E-5, Figure E-8 depicts a campus cluster without a multipathing solution implemented. However, in Figure E-8 the quorum disk is separate, in a third room.
Because the configurations in Figure E-7 and Figure E-8 use heterogeneous storage setups (Sun StorEdge T3/T3+ or Sun StorEdge 9910/9960 and Sun StorEdge A5x00 arrays), the host bus adapter types will be mixed. The Sun StorEdge A5x00 connects to the server using a SOC+-based SBus host adapter, so the server must be SBus-based, meaning that the connection to the storage arrays must also be SBus-based. Therefore, the storage array connection must be via a Sun StorEdge SBus Dual Fibre Channel Network Adapter.
Figure E-9, like Figure E-6, depicts a campus cluster using Sun StorEdge A5x00 arrays; however, in Figure E-9, the quorum disk is now separate and in its own room.