This chapter provides planning information and guidelines for installing a Sun Cluster configuration.
The following overview information is in this chapter:
The following table shows where to find instructions for various installation tasks for Sun Cluster software installation and the order in which you should perform the tasks.
Table 1–1 Sun Cluster Software Installation Task Information
Task |
Instructions |
---|---|
Set up cluster hardware. |
Sun Cluster 3.1 - 3.2 Hardware Administration Manual for Solaris OS Documentation that shipped with your server and storage devices |
Plan global-cluster software installation. | |
Install software packages. Optionally, install and configure Sun StorageTekTM QFS software. |
Sun StorageTek QFS Installation and Upgrade Guide, Version 4, Update 6 |
Establish a new global cluster or a new global-cluster node. |
Establishing a New Global Cluster or New Global-Cluster Node |
Configure Solaris Volume Manager software. |
Configuring Solaris Volume Manager Software Solaris Volume Manager documentation |
Install and configure Veritas Volume Manager (VxVM) software. |
Installing and Configuring VxVM Software VxVM documentation |
Configure cluster file systems, if used. | |
(Optional) On the Solaris 10 OS, create non-global zones. | |
(Optional) On the Solaris 10 OS, create zone clusters. | |
(Optional) SPARC: Install and configure the Sun Cluster module to Sun Management Center. |
SPARC: Installing the Sun Cluster Module for Sun Management Center Sun Management Center documentation |
Plan, install, and configure resource groups and data services. Create highly available local file systems, if used. |
Sun Cluster Data Services Planning and Administration Guide for Solaris OS |
Develop custom data services. |
This section provides the following guidelines for planning Solaris software installation in a cluster configuration.
For more information about Solaris software, see your Solaris installation documentation.
You can install Solaris software from a local DVD-ROM or from a network installation server by using the JumpStartTM installation method. In addition, Sun Cluster software provides a custom method for installing both the Solaris OS and Sun Cluster software by using the JumpStart installation method. If you are installing several cluster nodes, consider a network installation.
See How to Install Solaris and Sun Cluster Software (JumpStart) for details about the scinstall JumpStart installation method. See your Solaris installation documentation for details about standard Solaris installation methods.
Consider the following points when you plan the use of the Solaris OS in a Sun Cluster configuration:
Solaris 10 Zones – Install Sun Cluster framework software only in the global zone.
To determine whether you can install a Sun Cluster data service directly in a non-global zone, see the documentation for that data service.
If you configure non-global zones on a global-cluster node, the loopback file system (LOFS) must be enabled. See the information for LOFS for additional considerations.
Loopback file system (LOFS) – During cluster creation with the Solaris 9 version of Sun Cluster software , LOFS capability is disabled by default. During cluster creation with the Solaris 10 version of Sun Cluster software, LOFS capability is not disabled by default.
If the cluster meets both of the following conditions, you must disable LOFS to avoid switchover problems or other failures:
Sun Cluster HA for NFS is configured on a highly available local file system.
The automountd daemon is running.
If the cluster meets only one of these conditions, you can safely enable LOFS.
If you require both LOFS and the automountd daemon to be enabled, exclude from the automounter map all files that are part of the highly available local file system that is exported by Sun Cluster HA for NFS.
Interface groups – Solaris interface groups are not supported in a Sun Cluster configuration. The Solaris interface groups feature is disabled by default during Solaris software installation. Do not re-enable Solaris interface groups. See the ifconfig(1M) man page for more information about Solaris interface groups.
Power-saving shutdown – Automatic power-saving shutdown is not supported in Sun Cluster configurations and should not be enabled. See the pmconfig(1M) and power.conf(4) man pages for more information.
IP Filter – Sun Cluster software does not support the Solaris IP Filter feature for scalable services, but does support Solaris IP Filter for failover services.
fssnap – Sun Cluster software does not support the fssnap command, which is a feature of UFS. However, you can use the fssnap command on local systems that are not controlled by Sun Cluster software. The following restrictions apply to fssnap support:
The fssnap command is supported on local files systems that are not managed by Sun Cluster software.
The fssnap command is not supported on cluster file systems.
The fssnap command is not supported on local file systems under the control of HAStoragePlus.
Sun Cluster 3.2 1/09 software requires at least the End User Solaris Software Group (SUNWCuser). However, other components of your cluster configuration might have their own Solaris software requirements as well. Consider the following information when you decide which Solaris software group you are installing.
Servers – Check your server documentation for any Solaris software requirements. For example , Sun EnterpriseTM 10000 servers require the Entire Solaris Software Group Plus OEM Support.
SCI-PCI adapters – To use SCI-PCI adapters, which are available for use in SPARC based clusters only, or the Remote Shared Memory Application Programming Interface (RSMAPI), ensure that you install the RSMAPI software packages, which are SUNWrsm and SUNWrsmo, and for the Solaris 9 OS on SPARC based platforms also SUNWrsmx and SUNWrsmox. The RSMAPI software packages are included only in some Solaris software groups. For example, the Developer Solaris Software Group includes the RSMAPI software packages but the End User Solaris Software Group does not.
If the software group that you install does not include the RSMAPI software packages, install the RSMAPI software packages manually before you install Sun Cluster software. Use the pkgadd(1M) command to manually install the software packages. See the Section (3RSM) man pages for information about using the RSMAPI.
Additional Solaris packages – You might need to install other Solaris software packages that are not part of the End User Solaris Software Group. The Apache HTTP server packages are one example. Third-party software, such as ORACLE®, might also require additional Solaris software packages. See your third-party documentation for any Solaris software requirements.
To avoid the need to manually install Solaris software packages, install the Entire Solaris Software Group Plus OEM Support.
Add this information to the appropriate Local File System Layout Worksheet.
When you install the Solaris OS, ensure that you create the required Sun Cluster partitions and that all partitions meet minimum space requirements.
swap – The combined amount of swap space that is allocated for Solaris and Sun Cluster software must be no less than 750 Mbytes. For best results, add at least 512 Mbytes for Sun Cluster software to the amount that is required by the Solaris OS. In addition, allocate any additional swap amount that is required by applications that are to run on the Solaris host.
If you create an additional swap file, do not create the swap file on a global device. Use only a local disk as a swap device for the host.
/globaldevices – Create a file system at least 512 Mbytes large that is to be used by the scinstall(1M) utility for global devices.
Volume manager – Create a 20-Mbyte partition on slice 7 for volume manager use. If your cluster uses Veritas Volume Manager (VxVM) and you intend to encapsulate the root disk, you need to have two unused slices available for use by VxVM.
To meet these requirements, you must customize the partitioning if you are performing interactive installation of the Solaris OS.
See the following guidelines for additional partition planning information:
As with any other system running the Solaris OS, you can configure the root (/), /var, /usr, and /opt directories as separate file systems. Or, you can include all the directories in the root (/) file system.
No file-system type other than UFS is valid for the root (/) file system. Do not attempt to change the file-system type after the root (/) file system is created.
The following describes the software contents of the root (/), /var, /usr, and /opt directories in a Sun Cluster configuration. Consider this information when you plan your partitioning scheme.
root (/) – The Sun Cluster software itself occupies less than 40 Mbytes of space in the root (/) file system. Solaris Volume Manager software requires less than 5 Mbytes, and VxVM software requires less than 15 Mbytes. To configure ample additional space and inode capacity, add at least 100 Mbytes to the amount of space you would normally allocate for your root ( /) file system. This space is used for the creation of both block special devices and character special devices used by the volume management software. You especially need to allocate this extra space if a large number of shared disks are in the cluster.
/var – The Sun Cluster software occupies a negligible amount of space in the /var file system at installation time. However, you need to set aside ample space for log files. Also, more messages might be logged on a clustered node than would be found on a typical stand-alone server. Therefore, allow at least 100 Mbytes for the /var file system.
/usr – Sun Cluster software occupies less than 25 Mbytes of space in the /usr file system. Solaris Volume Manager and VxVM software each require less than 15 Mbytes.
/opt – Sun Cluster framework software uses less than 2 Mbytes in the /opt file system. However, each Sun Cluster data service might use between 1 Mbyte and 5 Mbytes. Solaris Volume Manager software does not use any space in the /opt file system. VxVM software can use over 40 Mbytes if all of its packages and tools are installed.
In addition, most database and applications software is installed in the /opt file system.
SPARC: If you use Sun Management Center software to monitor the cluster, you need an additional 25 Mbytes of space on each Solaris host to support the Sun Management Center agent and Sun Cluster module packages.
Sun Cluster software requires you to set aside a dedicated file system on one of the local disks for use in managing global devices. This file system is usually located on your root disk. However, if you use different storage on which to locate the global-devices file system, such as a Logical Volume Manager volume, it must not be part of a Solaris Volume Manager shared disk set or part of a VxVM disk group other than a root disk group. This file system is later mounted as a UFS cluster file system. Name this file system /globaldevices, which is the default name that is recognized by the scinstall(1M) command.
No file-system type other than UFS is valid for the global-devices file system. Do not attempt to change the file-system type after the global-devices file system is created.
The scinstall command later renames the file system /global/.devices/node@nodeid, where nodeid represents the number that is assigned to a Solaris host when it becomes a global-cluster member. The original /globaldevices mount point is removed.
The /globaldevices file system must have ample space and ample inode capacity for creating both block special devices and character special devices. This guideline is especially important if a large number of disks are in the cluster. A file system size of 512 Mbytes should suffice for most cluster configurations.
If you use Solaris Volume Manager software, you must set aside a slice on the root disk for use in creating the state database replica. Specifically, set aside a slice for this purpose on each local disk. But, if you have only one local disk on a Solaris host, you might need to create three state database replicas in the same slice for Solaris Volume Manager software to function properly. See your Solaris Volume Manager documentation for more information.
If you use Veritas Volume Manager (VxVM) and you intend to encapsulate the root disk, you need to have two unused slices that are available for use by VxVM. Additionally, you need to have some additional unassigned free space at either the beginning or the end of the disk. See your VxVM documentation for more information about root disk encapsulation.
Table 1–2 shows a partitioning scheme for a Solaris host that has less than 750 Mbytes of physical memory. This scheme is to be installed with the End User Solaris Software Group, Sun Cluster software, and the Sun Cluster HA for NFS data service. The last slice on the disk, slice 7, is allocated with a small amount of space for volume-manager use.
This layout allows for the use of either Solaris Volume Manager software or VxVM software. If you use Solaris Volume Manager software, you use slice 7 for the state database replica. If you use VxVM, you later free slice 7 by assigning the slice a zero length. This layout provides the necessary two free slices, 4 and 7, as well as provides for unused space at the end of the disk.
Table 1–2 Example File-System Allocation
Slice |
Contents |
Size Allocation |
Description |
---|---|---|---|
0 |
/ |
6.75GB |
Remaining free space on the disk after allocating space to slices 1 through 7. Used for the Solaris OS, Sun Cluster software, data-services software, volume-manager software, Sun Management Center agent and Sun Cluster module agent packages, root file systems, and database and application software. |
1 |
swap |
1GB |
512 Mbytes for the Solaris OS. 512 Mbytes for Sun Cluster software. |
2 |
overlap |
8.43GB |
The entire disk. |
3 |
/globaldevices |
512MB |
The Sun Cluster software later assigns this slice a different mount point and mounts the slice as a cluster file system. |
4 |
unused |
- |
Available as a free slice for encapsulating the root disk under VxVM. |
5 |
unused |
- |
- |
6 |
unused |
- |
- |
7 |
volume manager |
20MB |
Used by Solaris Volume Manager software for the state database replica, or used by VxVM for installation after you free the slice. |
For information about the purpose and function of Solaris 10 zones in a cluster, see Support for Solaris Zones in Sun Cluster Concepts Guide for Solaris OS.
For guidelines about configuring a cluster of non-global zones, see Zone Clusters.
Consider the following points when you create a Solaris 10 non-global zone, simply referred to as a zone, on a global-cluster node.
Unique zone name – The zone name must be unique on the Solaris host.
Reusing a zone name on multiple nodes – To simplify cluster administration, you can use the same name for a zone on each node where resource groups are to be brought online in that zone.
Private IP addresses – Do not attempt to use more private IP addresses than are available in the cluster.
Mounts – Do not include global mounts in zone definitions. Include only loopback mounts.
Failover services – In multiple-host clusters, while Sun Cluster software permits you to specify different zones on the same Solaris host in a failover resource group's node list, doing so is useful only during testing. If a single host contains all zones in the node list, the node becomes a single point of failure for the resource group. For highest availability, zones in a failover resource group's node list should be on different hosts.
In single-host clusters, no functional risk is incurred if you specify multiple zones in a failover resource group's node list.
Scalable services – Do not create non-global zones for use in the same scalable service on the same Solaris host. Each instance of the scalable service must run on a different host.
Cluster file systems - Do not directly add a cluster file system from the global zone to a non-global zone. Instead, add a loopback mount of the cluster file system from the global zone to the non-global zone. This restriction does not apply to QFS shared file systems.
LOFS – Solaris Zones requires that the loopback file system (LOFS) be enabled. However, the Sun Cluster HA for NFS data service requires that LOFS be disabled, to avoid switchover problems or other failures. If you configure both non-global zones and Sun Cluster HA for NFS in your cluster, do one of the following to prevent possible problems in the data service:
Disable the automountd daemon.
Exclude from the automounter map all files that are part of the highly available local file system that is exported by Sun Cluster HA for NFS.
Exclusive-IP zones – The following guidelines apply specifically to exclusive-IP non-global zones:
Logical-hostname resource groups – In a resource group that contains a LogicalHostname resource, if the node list contains any non-global zone with the ip-type property set to exclusive, all zones in that node list must have this property set to exclusive. Note that a global zone always has the ip-type property set to shared, and therefore cannot coexist in a node list that contains zones of ip-type=exclusive. This restriction applies only to versions of the Solaris OS that use the Solaris zones ip-type property.
IPMP groups – For all public-network adapters that are used for data-service traffic in the non-global zone, you must manually configure IPMP groups in all /etc/hostname.adapter files on the zone. This information is not inherited from the global zone. For guidelines and instructions to configure IPMP groups, follow the procedures in Part VI, IPMP, in System Administration Guide: IP Services.
Private-hostname dependency - Exclusive-IP zones cannot depend on the private hostnames and private addresses of the cluster.
Shared-address resources – Shared-address resources cannot use exclusive-IP zones.
Consider the following points when you create a Sun Logical Domains (LDoms) I/O domain or guest domain on a physically clustered machine that is SPARC hypervisor capable:
SCSI LUN requirement – The virtual shared storage device, or virtual disk back end, of a Sun LDoms guest domain must be a full SCSI LUN in the I/O domain. You cannot use an arbitrary virtual device.
Fencing – Do not export a storage LUN to more than one guest domain on the same physical machine, unless you also disable fencing for that device. Otherwise, if two different guest domains on the same machine both are visible to a device, the device will be fenced whenever one of the guest domains dies. The fencing of the device will panic any other guest domain that subsequently tries to access the device.
Network isolation – Guest domains that are located on the same physical machine but are configured in different clusters must be network isolated from each other. Use one of the following methods:
Configure the clusters to use different network interfaces in the I/O domain for the private network.
Use different network addresses for each of the clusters.
Networking in guest domains – Network packets to and from guest domains must traverse service domains to reach the network drivers through virtual switches. Virtual switches use kernel threads that run at system priority. The virtual-switch threads must be able to acquire needed CPU resources to perform critical cluster operations, including heartbeats, membership, checkpoints, and so forth. Configuring virtual switches with the mode=sc setting enables expedited handling of cluster heartbeat packets. However, the reliability of other critical cluster operations can be enhanced by adding more CPU resources to the service domain under the following workloads:
High-interrupt load, for example, due to network or disk I/O. Under extreme load, virtual switches can preclude system threads from running for a long time, including virtual-switch threads.
Real-time threads that are overly aggressive in retaining CPU resources. Real-time threads run at a higher priority than virtual-switch threads, which can restrict CPU resources for virtual-switch threads for an extended time.
Exporting storage from I/O domains – If you configure a cluster that is composed of Sun Logical Domains I/O domains, do not export its storage devices to other guest domains that also run Sun Cluster software.
Solaris I/O multipathing – Do not run Solaris I/O multipathing software (MPxIO) from guest domains. Instead, run Solaris I/O multipathing software in the I/O domain and export it to the guest domains.
Private-interconnect IP address range – The private network is shared by all guest domains that are created on the same physical machine and it is visible to all these domains. Before you specify a private-network IP address range to the scinstall utility for use by a guest-domain cluster, ensure that the address range is not already in use by another guest domain on the same physical machine.
For more information about Sun Logical Domains, see the Logical Domains (LDoms) 1.0.3 Administration Guide.
This section provides guidelines for planning and preparing the following components for Sun Cluster software installation and configuration:
For detailed information about Sun Cluster components, see the Sun Cluster Overview for Solaris OS and the Sun Cluster Concepts Guide for Solaris OS.
Ensure that you have available all necessary license certificates before you begin software installation. Sun Cluster software does not require a license certificate, but each node installed with Sun Cluster software must be covered under your Sun Cluster software license agreement.
For licensing requirements for volume-manager software and applications software, see the installation documentation for those products.
After installing each software product, you must also install any required patches. For proper cluster operation, ensure that all cluster nodes maintain the same patch level.
For information about current required patches, see Patches and Required Firmware Levels in Sun Cluster Release Notes or consult your Sun service provider.
For general guidelines and procedures for applying patches, see Chapter 10, Patching Sun Cluster Software and Firmware, in Sun Cluster System Administration Guide for Solaris OS.
For information about the use of public networks by the cluster, see Public Network Adapters and IP Network Multipathing in Sun Cluster Concepts Guide for Solaris OS.
You must set up a number of public-network IP addresses for various Sun Cluster components, depending on your cluster configuration. Each Solaris host in the cluster configuration must have at least one public-network connection to the same set of public subnets.
The following table lists the components that need public-network IP addresses assigned. Add these IP addresses to the following locations:
Any naming services that are used
The local /etc/inet/hosts file on each global-cluster node, after you install Solaris software
For IPv6 IP addresses on the Solaris 9 OS, the local /etc/inet/ipnodes file on each global-cluster node, after you install Solaris software
The local /etc/inet/hosts file on any exclusive-IP non-global zone
Component |
Number of IP Addresses Needed |
---|---|
1 IP address per subnet. |
|
1 IP address per node, per subnet. |
|
1 IP address per node, per subnet. |
|
1 IP address per domain. |
|
(Optional) Non-global zones |
1 IP address per subnet. |
1 IP address. |
|
Logical addresses |
1 IP address per logical host resource, per subnet. |
Quorum server |
1 IP address. |
For more information about planning IP addresses, see Chapter 3, Planning Your TCP/IP Network (Task), in System Administration Guide: IP Services (Solaris 9) or Chapter 2, Planning Your TCP/IP Network (Tasks), in System Administration Guide: IP Services (Solaris 10).
You must have console access to all cluster nodes. If you install Cluster Control Panel software on an administrative console, you must provide the hostname and port number of the console-access device that is used to communicate with the cluster nodes.
A terminal concentrator is used to communicate between the administrative console and the global-cluster node consoles.
A Sun Enterprise 10000 server uses a System Service Processor (SSP) instead of a terminal concentrator.
A Sun Fire server uses a system controller instead of a terminal concentrator.
For more information about console access, see the Sun Cluster Concepts Guide for Solaris OS.
Alternatively, if you connect an administrative console directly to cluster nodes or through a management network, you instead provide the hostname of each global-cluster node and its serial port number that is used to connect to the administrative console or the management network.
Each data-service resource group that uses a logical address must have a hostname specified for each public network from which the logical address can be accessed.
For more information, see the Sun Cluster Data Services Planning and Administration Guide for Solaris OS. For additional information about data services and resources, also see the Sun Cluster Overview for Solaris OS and the Sun Cluster Concepts Guide for Solaris OS.
Public networks communicate outside the cluster. Consider the following points when you plan your public-network configuration:
Separation of public and private network – Public networks and the private network (cluster interconnect) must use separate adapters, or you must configure tagged VLAN on tagged-VLAN capable adapters and VLAN-capable switches to use the same adapter for both the private interconnect and the public network.
Minimum – All cluster nodes must be connected to at least one public network. Public-network connections can use different subnets for different nodes.
Maximum – You can have as many additional public-network connections as your hardware configuration allows.
Scalable services – All nodes that run a scalable service must either use the same subnet or set of subnets or use different subnets that are routable among themselves.
IPv4 – Sun Cluster software supports IPv4 addresses on the public network.
IPv6 – Sun Cluster software supports IPv6 addresses on the public network under the following conditions or restrictions:
Sun Cluster software does not support IPv6 addresses on the public network if the private interconnect uses SCI adapters.
Sun Cluster software supports IPv6 addresses for both failover and scalable data services.
IPMP groups – Each public-network adapter that is used for data-service traffic must belong to an IP network multipathing (IPMP) group. If a public-network adapter is not used for data-service traffic, you do not have to configure it in an IPMP group.
In the Sun Cluster 3.2 1/09 release, the scinstall utility no longer automatically configures a single-adapter IPMP group on each unconfigured public-network adapter during Sun Cluster creation. Instead, the scinstall utility automatically configures a multiple-adapter IPMP group for each set of public-network adapters in the cluster that uses the same subnet. On the Solaris 10 OS, these groups are probe based.
The scinstall utility ignores adapters that are already configured in an IPMP group. You can use probe-based IPMP groups or link-based IPMP groups in a cluster. But probe-based IPMP groups, which test the target IP address, provide the most protection by recognizing more conditions that might compromise availability.
If any adapter in an IPMP group that the scinstall utility configures will not be used for data-service traffic, you can remove that adapter from the group.
For guidelines and instructions to configure IPMP groups, follow the procedures in Part VI, IPMP, in System Administration Guide: IP Services. To modify IPMP groups after cluster installation, follow the guidelines in How to Administer IP Network Multipathing Groups in a Cluster in Sun Cluster System Administration Guide for Solaris OS and procedures in Chapter 28, Administering Network Multipathing (Task), in System Administration Guide: IP Services (Solaris 9) or Chapter 31, Administering IPMP (Tasks), in System Administration Guide: IP Services(Solaris 10).
Local MAC address support – All public-network adapters must use network interface cards (NICs) that support local MAC address assignment. Local MAC address assignment is a requirement of IPMP.
local-mac-address setting – The local-mac-address? variable must use the default value true for Ethernet adapters. Sun Cluster software does not support a local-mac-address? value of false for Ethernet adapters. This requirement is a change from Sun Cluster 3.0, which did require a local-mac-address? value of false.
For more information about public-network interfaces, see Sun Cluster Concepts Guide for Solaris OS.
You can use Sun Cluster Quorum Server software to configure a machine as a quorum server and then configure the quorum server as your cluster's quorum device. You can use a quorum server instead of or in addition to shared disks and NAS filers.
Consider the following points when you plan the use of a quorum server in a Sun Cluster configuration.
Network connection – The quorum-server computer connects to your cluster through the public network.
Supported hardware – The supported hardware platforms for a quorum server are the same as for a global-cluster node.
Operating system – Solaris software requirements for Sun Cluster software apply as well to Quorum Server software.
Service to multiple clusters – You can configure a quorum server as a quorum device to more than one cluster.
Mixed hardware and software – You do not have to configure a quorum server on the same hardware and software platform as the cluster or clusters that it provides quorum to. For example, a SPARC based machine that runs the Solaris 9 OS can be configured as a quorum server for an x86 based cluster that runs the Solaris 10 OS.
Using a cluster node as a quorum server – You can configure a quorum server on a cluster node to provide quorum for clusters other than the cluster that the node belongs to. However, a quorum server that is configured on a cluster node is not highly available.
Consider the following points when you plan the use of Network File System (NFS) in a Sun Cluster configuration.
NFS client – No Sun Cluster node can be an NFS client of a Sun Cluster HA for NFS-exported file system that is being mastered on a node in the same cluster. Such cross-mounting of Sun Cluster HA for NFS is prohibited. Use the cluster file system to share files among global-cluster nodes.
NFSv3 protocol – If you are mounting file systems on the cluster nodes from external NFS servers, such as NAS filers, and you are using the NFSv3 protocol, you cannot run NFS client mounts and the Sun Cluster HA for NFS data service on the same cluster node. If you do, certain Sun Cluster HA for NFS data-service activities might cause the NFS daemons to stop and restart, interrupting NFS services. However, you can safely run the Sun Cluster HA for NFS data service if you use the NFSv4 protocol to mount external NFS file systems on the cluster nodes.
Locking – Applications that run locally on the cluster must not lock files on a file system that is exported through NFS. Otherwise, local blocking (for example, flock(3UCB) or fcntl(2)) might interfere with the ability to restart the lock manager ( lockd(1M)). During restart, a blocked local process might be granted a lock which might be intended to be reclaimed by a remote client. This would cause unpredictable behavior.
NFS security features – Sun Cluster software does not support the following options of the share_nfs(1M) command:
secure
sec=dh
However, Sun Cluster software does support the following security features for NFS:
The use of secure ports for NFS. You enable secure ports for NFS by adding the entry set nfssrv:nfs_portmon=1 to the /etc/system file on cluster nodes.
The use of Kerberos with NFS. For more information, see Securing Sun Cluster HA for NFS With Kerberos V5 in Sun Cluster Data Service for NFS Guide for Solaris OS.
No fencing support for NAS devices in non-global zones – Sun Cluster software does not provide fencing support for NFS-exported file systems from a NAS device when such file systems are used in a non-global zone, including nodes of a zone cluster. Fencing support is provided only for NFS-exported file systems in the global zone.
Observe the following service restrictions for Sun Cluster configurations:
Routers – Do not configure cluster nodes as routers (gateways). If the system goes down, the clients cannot find an alternate router and cannot recover.
NIS+ servers – Do not configure cluster nodes as NIS or NIS+ servers. There is no data service available for NIS or NIS+. However, cluster nodes can be NIS or NIS+ clients.
Boot and install servers – Do not use a Sun Cluster configuration to provide a highly available boot or installation service on client systems.
RARP – Do not use a Sun Cluster configuration to provide an rarpd service.
RPC program numbers – If you install an RPC service on the cluster, the service must not use any of the following program numbers:
100141
100142
100248
These numbers are reserved for the Sun Cluster daemons rgmd_receptionist, fed, and pmfd, respectively.
If the RPC service that you install also uses one of these program numbers, you must change that RPC service to use a different program number.
Scheduling classes – Sun Cluster software does not support the running of high-priority process scheduling classes on cluster nodes. Do not run either of the following types of processes on cluster nodes:
Processes that run in the time-sharing scheduling class with a high priority
Processes that run in the real-time scheduling class
Sun Cluster software relies on kernel threads that do not run in the real-time scheduling class. Other time-sharing processes that run at higher-than-normal priority or real-time processes can prevent the Sun Cluster kernel threads from acquiring needed CPU cycles.
This section provides guidelines for the following Sun Cluster components that you configure:
Add this information to the appropriate configuration planning worksheet.
Specify a name for the global cluster during Sun Cluster configuration. The global cluster name should be unique throughout the enterprise.
For information about naming a zone cluster, see Zone Clusters.
The name of a voting node in a global cluster is the same name that you assign to the physical or virtual host when you install it with the Solaris OS. See the hosts(4) man page for information about naming requirements.
In single-host cluster installations, the default cluster name is the name of the voting node.
During Sun Cluster configuration, you specify the names of all voting nodes that you are installing in the global cluster.
For information about node names in a zone cluster, see Zone Clusters.
On the Solaris 10 OS in versions that support Solaris brands, a non-global zone of brand native is a valid potential node of a resource-group node list. Use the naming convention nodename:zonename to specify a non-global zone to a Sun Cluster command.
The nodename is the name of the Solaris host.
The zonename is the name that you assign to the non-global zone when you create the zone on the voting node. The zone name must be unique on the node. However, you can use the same zone name on different voting nodes. The different node name in nodename:zonename makes the complete non-global zone name unique in the cluster.
To specify the global zone, you need to specify only the voting-node name.
For information about a cluster of non-global zones, see Zone Clusters.
You do not need to configure a private network for a single-host global cluster. The scinstall utility automatically assigns the default private-network address and netmask, even though a private network is not used by the cluster.
Sun Cluster software uses the private network for internal communication among nodes and among non-global zones that are managed by Sun Cluster software. A Sun Cluster configuration requires at least two connections to the cluster interconnect on the private network. When you configure Sun Cluster software on the first node of the cluster, you specify the private-network address and netmask in one of the following ways:
Accept the default private-network address (172.16.0.0) and default netmask.
On the Solaris 10 OS, the default netmask is 255.255.240.0. This IP address range supports a combined maximum of 64 voting nodes and non-global zones, a maximum of 12 zone clusters, and a maximum of 10 private networks.
On the Solaris 9 OS, the default netmask is 255.255.248.0. This IP address range supports a combined maximum of 64 nodes and a maximum of 10 private networks.
The maximum number of voting nodes that an IP address range can support does not reflect the maximum number of voting nodes that the hardware or software configuration can currently support.
Specify a different allowable private-network address and accept the default netmask.
Accept the default private-network address and specify a different netmask.
Specify both a different private-network address and a different netmask.
If you choose to specify a different netmask, the scinstall utility prompts you for the number of nodes and the number of private networks that you want the IP address range to support. On the Solaris 10 OS, the utility also prompts you for the number of zone clusters that you want to support. The number of global-cluster nodes that you specify should also include the expected number of unclustered non-global zones that will use the private network.
The utility calculates the netmask for the minimum IP address range that will support the number of nodes, zone clusters, and private networks that you specified. The calculated netmask might support more than the supplied number of nodes, including non-global zones, zone clusters, and private networks. The scinstall utility also calculates a second netmask that would be the minimum to support twice the number of nodes, zone clusters, and private networks. This second netmask would enable the cluster to accommodate future growth without the need to reconfigure the IP address range.
The utility then asks you what netmask to choose. You can specify either of the calculated netmasks or provide a different one. The netmask that you specify must minimally support the number of nodes and private networks that you specified to the utility.
Changing the cluster private IP-address range might be necessary to support the addition of voting nodes, non-global zones, zone clusters, or private networks.
To change the private-network address and netmask after the cluster is established, see How to Change the Private Network Address or Address Range of an Existing Cluster in Sun Cluster System Administration Guide for Solaris OS. You must bring down the cluster to make these changes.
However, on the Solaris 10 OS the cluster can remain in cluster mode if you use the cluster set-netprops command to change only the netmask. For any zone cluster that is already configured in the cluster, the private IP subnets and the corresponding private IP addresses that are allocated for that zone cluster will also be updated.
If you specify a private-network address other than the default, the address must meet the following requirements:
Address and netmask sizes – The private network address cannot be smaller than the netmask. For example, you can use a private network address of 172.16.10.0 with a netmask of 255.255.255.0. But you cannot use a private network address of 172.16.10.0 with a netmask of 255.255.0.0.
Acceptable addresses – The address must be included in the block of addresses that RFC 1918 reserves for use in private networks. You can contact the InterNIC to obtain copies of RFCs or view RFCs online at http://www.rfcs.org.
Use in multiple clusters – You can use the same private-network address in more than one cluster, provided that the clusters are on different private networks. Private IP network addresses are not accessible from outside the physical cluster.
For Sun Logical Domains (LDoms) guest domains that are created on the same physical machine and that are connected to the same virtual switch, the private network is shared by such guest domains and is visible to all these domains. Proceed with caution before you specify a private-network IP address range to the scinstall utility for use by a cluster of guest domains. Ensure that the address range is not already in use by another guest domain that exists on the same physical machine and shares its virtual switch.
IPv6 – Sun Cluster software does not support IPv6 addresses for the private interconnect. The system does configure IPv6 addresses on the private-network adapters to support scalable services that use IPv6 addresses. But internode communication on the private network does not use these IPv6 addresses.
See Planning Your TCP/IP Network (Tasks), in System Administration Guide: IP Services (Solaris 9 or Solaris 10) for more information about private networks.
The private hostname is the name that is used for internode communication over the private-network interface. Private hostnames are automatically created during Sun Cluster configuration of a global cluster or a zone cluster. These private hostnames follow the naming convention clusternodenodeid -priv, where nodeid is the numeral of the internal node ID. During Sun Cluster configuration, the node ID number is automatically assigned to each voting node when the node becomes a cluster member. A voting node of the global cluster and a node of a zone cluster can both have the same private hostname, but each hostname resolves to a different private-network IP address.
After a global cluster is configured, you can rename its private hostnames by using the clsetup(1CL) utility. Currently, you cannot rename the private hostname of a zone-cluster node.
For the Solaris 10 OS, the creation of a private hostname for a non-global zone is optional. There is no required naming convention for the private hostname of a non-global zone.
The cluster interconnects provide the hardware pathways for private-network communication between cluster nodes. Each interconnect consists of a cable that is connected in one of the following ways:
Between two transport adapters
Between a transport adapter and a transport switch
For more information about the purpose and function of the cluster interconnect, see Cluster Interconnect in Sun Cluster Concepts Guide for Solaris OS.
You do not need to configure a cluster interconnect for a single-host cluster. However, if you anticipate eventually adding more voting nodes to a single-host cluster configuration, you might want to configure the cluster interconnect for future use.
During Sun Cluster configuration, you specify configuration information for one or two cluster interconnects.
If the number of available adapter ports is limited, you can use tagged VLANs to share the same adapter with both the private and public network. For more information, see the guidelines for tagged VLAN adapters in Transport Adapters.
You can set up from one to six cluster interconnects in a cluster. While a single cluster interconnect reduces the number of adapter ports that are used for the private interconnect, it provides no redundancy and less availability. If a single interconnect fails, the cluster is at a higher risk of having to perform automatic recovery. Whenever possible, install two or more cluster interconnects to provide redundancy and scalability, and therefore higher availability, by avoiding a single point of failure.
You can configure additional cluster interconnects, up to six interconnects total, after the cluster is established by using the clsetup(1CL) utility.
For guidelines about cluster interconnect hardware, see Interconnect Requirements and Restrictions in Sun Cluster 3.1 - 3.2 Hardware Administration Manual for Solaris OS. For general information about the cluster interconnect, see Cluster-Interconnect Components in Sun Cluster Overview for Solaris OS and Sun Cluster Concepts Guide for Solaris OS.
For the transport adapters, such as ports on network interfaces, specify the transport adapter names and transport type. If your configuration is a two-host cluster, you also specify whether your interconnect is a point-to-point connection (adapter to adapter) or uses a transport switch.
Consider the following guidelines and restrictions:
IPv6 – Sun Cluster software does not support IPv6 communications over the private interconnects.
Local MAC address assignment – All private network adapters must use network interface cards (NICs) that support local MAC address assignment. Link-local IPv6 addresses, which are required on private-network adapters to support IPv6 public-network addresses, are derived from the local MAC addresses.
Tagged VLAN adapters – Sun Cluster software supports tagged Virtual Local Area Networks (VLANs) to share an adapter between the private cluster interconnect and the public network. To configure a tagged VLAN adapter for the cluster interconnect, specify the adapter name and its VLAN ID (VID) in one of the following ways:
Specify the usual adapter name, which is the device name plus the instance number or physical point of attachment (PPA). For example, the name of instance 2 of a Cassini Gigabit Ethernet adapter would be ce2. If the scinstall utility asks whether the adapter is part of a shared virtual LAN, answer yes and specify the adapter's VID number.
Specify the adapter by its VLAN virtual device name. This name is composed of the adapter name plus the VLAN instance number. The VLAN instance number is derived from the formula (1000*V)+N, where V is the VID number and N is the PPA.
As an example, for VID 73 on adapter ce2, the VLAN instance number would be calculated as (1000*73)+2. You would therefore specify the adapter name as ce73002 to indicate that it is part of a shared virtual LAN.
For information about configuring VLAN in a cluster, see Configuring VLANs as Private Interconnect Networks in Sun Cluster 3.1 - 3.2 Hardware Administration Manual for Solaris OS. For general information about VLAN, see Administering Virtual Local Area Networks in System Administration Guide: IP Services.
SPARC: Sun LDoms guest domains – Specify adapter names by their virtual names, vnetN, such as vnet0 and vnet1. Virtual adapter names are recorded in the /etc/path_to_inst file.
SBus SCI adapters – The SBus Scalable Coherent Interface (SCI) is not supported as a cluster interconnect. However, the SCI-PCI interface is supported.
Logical network interfaces – Logical network interfaces are reserved for use by Sun Cluster software.
See the scconf_trans_adap_*(1M) family of man pages for information about a specific transport adapter.
If you use transport switches, such as a network switch, specify a transport switch name for each interconnect. You can use the default name switchN, where N is a number that is automatically assigned during configuration, or create another name.
Also specify the switch port name or accept the default name. The default port name is the same as the internal node ID number of the Solaris host that hosts the adapter end of the cable. However, you cannot use the default port name for certain adapter types, such as SCI-PCI.
Clusters with three or more voting nodes must use transport switches. Direct connection between voting cluster nodes is supported only for two-host clusters.
If your two-host cluster is direct connected, you can still specify a transport switch for the interconnect.
If you specify a transport switch, you can more easily add another voting node to the cluster in the future.
Fencing is a mechanism that is used by the cluster to protect the data integrity of a shared disk during split-brain situations. By default, the scinstall utility in Typical Mode leaves global fencing enabled, and each shared disk in the configuration uses the default global fencing setting of pathcount. With the pathcount setting, the fencing protocol for each shared disk is chosen based on the number of DID paths that are attached to the disk.
In Custom Mode, the scinstall utility prompts you whether to disable global fencing. For most situations, respond No to keep global fencing enabled. However, you can disable global fencing to support the following situations:
If you disable fencing under other situations than the following, your data might be vulnerable to corruption during application failover. Examine this data corruption possibility carefully when you consider turning off fencing.
The shared storage does not support SCSI reservations.
If you turn off fencing for a shared disk that you then configure as a quorum device, the device uses the software quorum protocol. This is true regardless of whether the disk supports SCSI-2 or SCSI-3 protocols. Software quorum is a protocol in Sun Cluster software that emulates a form of SCSI Persistent Group Reservations (PGR).
You want to enable systems that are outside the cluster to gain access to storage that is attached to the cluster.
If you disable global fencing during cluster configuration, fencing is turned off for all shared disks in the cluster. After the cluster is configured, you can change the global fencing protocol or override the fencing protocol of individual shared disks. However, to change the fencing protocol of a quorum device, you must first unconfigure the quorum device. Then set the new fencing protocol of the disk and reconfigure it as a quorum device.
For more information about fencing behavior, see Failfast Mechanism in Sun Cluster Concepts Guide for Solaris OS. For more information about setting the fencing protocol of individual shared disks, see the cldevice(1CL) man page. For more information about the global fencing setting, see the cluster(1CL) man page.
Sun Cluster configurations use quorum devices to maintain data and resource integrity. If the cluster temporarily loses connection to a voting node, the quorum device prevents amnesia or split-brain problems when the voting cluster node attempts to rejoin the cluster. For more information about the purpose and function of quorum devices, see Quorum and Quorum Devices in Sun Cluster Concepts Guide for Solaris OS.
During Sun Cluster installation of a two-host cluster, you can choose to let the scinstall utility automatically configure as a quorum device an available shared disk in the configuration. Shared disks include any Sun NAS device that is configured for use as a shared disk. The scinstall utility assumes that all available shared disks are supported as quorum devices.
If you want to use a quorum server or a Network Appliance NAS device as the quorum device, you configure it after scinstall processing is completed.
After installation, you can also configure additional quorum devices by using the clsetup(1CL) utility.
You do not need to configure quorum devices for a single-host cluster.
If your cluster configuration includes third-party shared storage devices that are not supported for use as quorum devices, you must use the clsetup utility to configure quorum manually.
Consider the following points when you plan quorum devices.
Minimum – A two-host cluster must have at least one quorum device, which can be a shared disk, a quorum server, or a NAS device. For other topologies, quorum devices are optional.
Odd-number rule – If more than one quorum device is configured in a two-host cluster, or in a pair of hosts directly connected to the quorum device, configure an odd number of quorum devices. This configuration ensures that the quorum devices have completely independent failure pathways.
Distribution of quorum votes – For highest availability of the cluster, ensure that the total number of votes that are contributed by quorum devices is less than the total number of votes that are contributed by voting nodes. Otherwise, the nodes cannot form a cluster if all quorum devices are unavailable, even if all nodes are functioning.
Connection – You must connect a quorum device to at least two voting nodes.
SCSI fencing protocol – When a SCSI shared-disk quorum device is configured, its fencing protocol is automatically set to SCSI-2 in a two-host cluster or SCSI-3 in a cluster with three or more voting nodes.
Changing the fencing protocol of quorum devices – For SCSI disks that are configured as a quorum device, you must unconfigure the quorum device before you can enable or disable its SCSI fencing protocol.
Software quorum protocol – You can configure supported shared disks that do not support SCSI protocol, such as SATA disks, as quorum devices. You must disable fencing for such disks. The disks would then use software quorum protocol, which emulates SCSI PGR.
The software quorum protocol would also be used by SCSI shared disks if fencing is disabled for such disks.
Replicated devices – Sun Cluster software does not support replicated devices as quorum devices.
ZFS storage pools – Do not add a configured quorum device to a ZFS storage pool. When a configured quorum device is added to a ZFS storage pool, the disk is relabeled as an EFI disk and quorum configuration information is lost. The disk can then no longer provide a quorum vote to the cluster.
After a disk is in a storage pool, you can configure that disk as a quorum device. Or, you can unconfigure the quorum device, add it to the storage pool, then reconfigure the disk as a quorum device.
For more information about quorum devices, see Quorum and Quorum Devices in Sun Cluster Concepts Guide for Solaris OS and Quorum Devices in Sun Cluster Overview for Solaris OS.
On the Solaris 10 OS, a zone cluster is a cluster of non-global zones. All nodes of a zone cluster are configured as non-global zones of the cluster brand. No other brand type is permitted in a zone cluster. You can run supported services on the zone cluster similar to a global cluster, with the isolation that is provided by Solaris zones.
Consider the following points when you plan the creation of a zone cluster.
Global cluster – The zone cluster must be configured on a global Sun Cluster configuration. A zone cluster cannot be configured without an underlying global cluster.
Minimum Solaris OS – The global cluster must run at least the Solaris 10 5/08 OS.
Cluster mode – The global-cluster voting node from which you create or modify a zone cluster must be in cluster mode. If any other voting nodes are in noncluster mode when you administer a zone cluster, the changes that you make are propagated to those nodes when they return to cluster mode.
Adequate private -IP addresses – The private IP-address range of the global cluster must have enough free IP-address subnets for use by the new zone cluster. If the number of available subnets is insufficient, the creation of the zone cluster fails.
Changes to the private IP-address range – The private IP subnets and the corresponding private IP-addresses that are available for zone clusters are automatically updated if the global cluster's private IP-address range is changed. If a zone cluster is deleted, the cluster infrastructure frees the private IP-addresses that were used by that zone cluster, making the addresses available for other use within the global cluster and by any other zone clusters that depend on the global cluster.
Supported devices – Devices that are supported with Solaris zones can be exported to a zone cluster. Such devices include the following:
Solaris disk devices (cNtXdYsZ)
DID devices (/dev/did/*dsk/dN)
Solaris Volume Manager and Solaris Volume Manager for Sun Cluster multi-owner disk sets (/dev/md/setname/*dsk/dN)
Distribution of nodes – You cannot host multiple nodes of the same zone cluster on the same node of the global cluster. A global-cluster node can host multiple zone-cluster nodes as long as each node is a member of a different zone cluster.
Node creation – You must create at least one zone-cluster node at the time that you create the zone cluster. The names of the nodes must be unique within the zone cluster. The infrastructure automatically creates an underlying non-global zone on each global-cluster node that hosts the zone cluster. Each non-global zone is given the same zone name, which is derived from, and identical to, the name that you assign to the zone cluster when you create the cluster. For example, if you create a zone cluster that is named zc1, the corresponding non-global zone name on each global-cluster node that hosts the zone cluster is also zc1.
Cluster name – The name of the zone cluster must be unique throughout the global cluster. The name cannot also be used by a non-global zone elsewhere in the global cluster, nor can the name be the same as that of a global-cluster node. You cannot use “all” or “global” as a zone-cluster name, because these are reserved names.
Public-network IP addresses – You assign a specific public-network IP address to each zone-cluster node.
Private hostnames – During creation of the zone cluster, a private hostname is automatically created for each node of the zone cluster, in the same way that hostnames are created in global clusters. Currently, you cannot rename the private hostname of a zone-cluster node. For more information about private hostnames, see Private Hostnames.
Solaris zones brand – All nodes of a zone cluster are configured as non-global zones of the cluster brand. No other brand type is permitted in a zone cluster.
Conversion to a zone-cluster node – You cannot add an existing non-global zone to a zone cluster.
File systems – You can use the clzonecluster command to add only the following types of file systems for use by a zone cluster:
Highly available local file systems
QFS shared file systems, which are supported for use by Oracle Real Application Clusters
Do not directly add a cluster file system from the global zone to a zone cluster node. Instead, add a loopback mount of the cluster file system from the global zone to the non-global zone.
To add a local file system to a zone cluster, you must instead use the zonecfg command as you normally would in a stand-alone system.
No fencing support for NAS devices in non-global zones – Sun Cluster software does not provide fencing support for NFS-exported file systems from a NAS device when such file systems are used in a non-global zone, including nodes of a zone cluster. Fencing support is provided only for NFS-exported file systems in the global zone.
This section provides the following guidelines for planning global devices and for planning cluster file systems:
For information about the purpose and function of global devices, see Global Devices, Local Devices, and Device Groups in Sun Cluster Overview for Solaris OS and Global Devices in Sun Cluster Concepts Guide for Solaris OS.
Sun Cluster software does not require any specific disk layout or file system size. Consider the following points when you plan your layout for global devices.
Mirroring – You must mirror all global devices for the global device to be considered highly available. You do not need to use software mirroring if the storage device provides hardware RAID as well as redundant paths to disks.
Disks – When you mirror, lay out file systems so that the file systems are mirrored across disk arrays.
Availability – You must physically connect a global device to more than one voting node in the cluster for the global device to be considered highly available. A global device with multiple physical connections can tolerate a single-node failure. A global device with only one physical connection is supported, but the global device becomes inaccessible from other voting nodes if the node with the connection is down.
Swap devices – Do not create a swap file on a global device.
Non-global zones – Global devices are not directly accessible from a non-global zone. Only cluster-file-system data is accessible from a non-global zone.
For information about the purpose and function of device groups, see Global Devices, Local Devices, and Device Groups in Sun Cluster Overview for Solaris OS and Device Groups in Sun Cluster Concepts Guide for Solaris OS.
Add this planning information to the Device Group Configurations Worksheet.
Consider the following points when you plan device groups.
Failover – You can configure multihost disks and properly configured volume-manager devices as failover devices. Proper configuration of a volume-manager device includes multihost disks and correct setup of the volume manager itself. This configuration ensures that multiple voting nodes can host the exported device. You cannot configure tape drives, CD-ROMs or DVD-ROMs, or single-ported devices as failover devices.
Mirroring – You must mirror the disks to protect the data from disk failure. See Mirroring Guidelines for additional guidelines. See Configuring Solaris Volume Manager Software or Installing and Configuring VxVM Software and your volume-manager documentation for instructions about mirroring.
Storage-based replication – Disks in a device group must be either all replicated or none replicated. A device group cannot use a mix of replicated and nonreplicated disks.
For information about the purpose and function of cluster file systems, see Cluster File Systems in Sun Cluster Overview for Solaris OS and Cluster File Systems in Sun Cluster Concepts Guide for Solaris OS.
You can alternatively configure highly available local file systems. This can provide better performance to support a data service with high I/O, or to permit use of certain file-system features that are not supported in a cluster file system. For more information, see Enabling Highly Available Local File Systems in Sun Cluster Data Services Planning and Administration Guide for Solaris OS.
Consider the following points when you plan cluster file systems.
Quotas – Quotas are not supported on cluster file systems. However, quotas are supported on highly available local file systems.
Non-global zones – If a cluster file system is to be accessed from a non-global zone, it must first be mounted in the global zone. The cluster file system is then mounted in the non-global zone by using a loopback mount. Therefore, the loopback file system (LOFS) must be enabled in a cluster that contains non-global zones.
Zone clusters – You cannot configure cluster file systems for use in a zone cluster. Use highly available local file systems instead. If the zone cluster is configured with Oracle Real Application Clusters (RAC), you can use shared QFS in that zone cluster to support Oracle RAC.
Loopback file system (LOFS) – During cluster creation with the Solaris 9 version of Sun Cluster software, LOFS is disabled by default. During cluster creation with the Solaris 10 version of Sun Cluster software, LOFS is enabled by default.
You must manually disable LOFS on each voting cluster node if the cluster meets both of the following conditions:
Sun Cluster HA for NFS is configured on a highly available local file system.
The automountd daemon is running.
If the cluster meets both of these conditions, you must disable LOFS to avoid switchover problems or other failures. If the cluster meets only one of these conditions, you can safely enable LOFS.
If you require both LOFS and the automountd daemon to be enabled, exclude from the automounter map all files that are part of the highly available local file system that is exported by Sun Cluster HA for NFS.
Process accounting log files – Do not locate process accounting log files on a cluster file system or on a highly available local file system. A switchover would be blocked by writes to the log file, which would cause the node to hang. Use only a local file system to contain process accounting log files.
Communication endpoints – The cluster file system does not support any of the file-system features of Solaris software by which one would put a communication endpoint in the file-system namespace.
Although you can create a UNIX domain socket whose name is a path name into the cluster file system, the socket would not survive a node failover.
Any FIFOs or named pipes that you create on a cluster file system would not be globally accessible.
Therefore, do not attempt to use the fattach command from any node other than the local node.
Device special files – Neither block special files nor character special files are supported in a cluster file system. To specify a path name to a device node in a cluster file system, create a symbolic link to the device name in the /dev directory. Do not use the mknod command for this purpose.
atime – Cluster file systems do not maintain atime.
ctime – When a file on a cluster file system is accessed, the update of the file's ctime might be delayed.
Installing applications - If you want the binaries of a highly available application to reside on a cluster file system, wait to install the application until after the cluster file system is configured. Also, if the application is installed by using the Sun Java System installer program and the application depends on any shared components, install those shared components on all nodes in the cluster that are not installed with the application.
This section describes requirements and restrictions for the following types of cluster file systems:
You can alternatively configure these and other types of file systems as highly available local file systems. For more information, see Enabling Highly Available Local File Systems in Sun Cluster Data Services Planning and Administration Guide for Solaris OS.
Follow these guidelines to determine what mount options to use when you create your cluster file systems.
See the mount_ufs(1M) man page for more information about UFS mount options.
Mount Option |
Usage |
Description |
---|---|---|
global |
Required |
This option makes the file system globally visible to all nodes in the cluster. |
log |
Required |
This option enables logging. |
See the VxFS mount_vxfs man page and Overview of Administering Cluster File Systems in Sun Cluster System Administration Guide for Solaris OS for more information about VxFS mount options.
Consider the following points when you plan mount points for cluster file systems.
Mount-point location – Create mount points for cluster file systems in the /global directory, unless you are prohibited by other software products. By using the /global directory, you can more easily distinguish cluster file systems, which are globally available, from local file systems.
SPARC: VxFS mount requirement – If you use Veritas File System (VxFS), globally mount and unmount a VxFS file system from the primary node. The primary node is the Solaris host that masters the disk on which the VxFS file system resides. This method ensures that the mount or unmount operation succeeds. A VxFS file-system mount or unmount operation that is performed from a secondary node might fail.
SPARC: VxFS feature restrictions –
The following VxFS features are not supported in a Sun Cluster 3.2 cluster file system. They are, however, supported in a local file system.
Quick I/O
Snapshots
Storage checkpoints
VxFS-specific mount options:
convosync (Convert O_SYNC)
mincache
qlog, delaylog, tmplog
Veritas cluster file system (requires VxVM cluster feature & Veritas Cluster Server)
Cache advisories can be used, but the effect is observed on the given node only.
All other VxFS features and options that are supported in a cluster file system are supported by Sun Cluster 3.2 software. See VxFS documentation for details about VxFS options that are supported in a cluster configuration.
Nesting mount points – Normally, you should not nest the mount points for cluster file systems. For example, do not set up one file system that is mounted on /global/a and another file system that is mounted on /global/a/b. To ignore this rule can cause availability and node boot-order problems. These problems would occur if the parent mount point is not present when the system attempts to mount a child of that file system. The only exception to this rule is if the devices for the two file systems have the same physical host connectivity. An example is different slices on the same disk.
forcedirectio – Sun Cluster software does not support the execution of binaries off cluster file systems that are mounted by using the forcedirectio mount option.
Add this planning information to the Device Group Configurations Worksheet and the Volume-Manager Configurations Worksheet. For Solaris Volume Manager, also add this planning information to the Volumes Worksheet (Solaris Volume Manager).
This section provides the following guidelines for planning volume management of your cluster configuration:
Sun Cluster software uses volume-manager software to group disks into device groups which can then be administered as one unit. Sun Cluster software supports Solaris Volume Manager software and Veritas Volume Manager (VxVM) software that you install or use in the following ways.
Table 1–4 Supported Use of Volume Managers With Sun Cluster Software
Volume-Manager Software |
Requirements |
---|---|
Solaris Volume Manager |
You must install Solaris Volume Manager software on all voting nodes of the cluster, regardless of whether you use VxVM on some nodes to manage disks. |
You must install and license VxVM with the cluster feature on all voting nodes of the cluster. |
|
VxVM without the cluster feature |
You are only required to install and license VxVM on those voting nodes that are attached to storage devices that VxVM manages. |
If you install both volume managers on the same voting node, you must use Solaris Volume Manager software to manage disks that are local to each node. Local disks include the root disk. Use VxVM to manage all shared disks. |
See your volume-manager documentation and Configuring Solaris Volume Manager Software or Installing and Configuring VxVM Software for instructions about how to install and configure the volume-manager software. For more information about the use of volume management in a cluster configuration, see Multihost Devices in Sun Cluster Concepts Guide for Solaris OS and Device Groups in Sun Cluster Concepts Guide for Solaris OS.
Consider the following general guidelines when you configure your disks with volume-manager software:
Software RAID – Sun Cluster software does not support software RAID 5.
Mirrored multihost disks – You must mirror all multihost disks across disk expansion units. See Guidelines for Mirroring Multihost Disks for guidelines on mirroring multihost disks. You do not need to use software mirroring if the storage device provides hardware RAID as well as redundant paths to devices.
Mirrored root – Mirroring the root disk ensures high availability, but such mirroring is not required. See Mirroring Guidelines for guidelines about deciding whether to mirror the root disk.
Unique naming – You might have local Solaris Volume Manager or VxVM volumes that are used as devices on which the /global/.devices/node@nodeid file systems are mounted. If so, the name of each local volume on which a /global/.devices/node@nodeid file system is to be mounted must be unique throughout the cluster.
Node lists – To ensure high availability of a device group, make its node lists of potential masters and its failback policy identical to any associated resource group. Or, if a scalable resource group uses more nodes than its associated device group, make the scalable resource group's node list a superset of the device group's node list. See the resource group planning information in the Sun Cluster Data Services Planning and Administration Guide for Solaris OS for information about node lists.
Multihost disks – You must connect, or port, all devices that are used to construct a device group to all of the nodes that are configured in the node list for that device group. Solaris Volume Manager software can automatically check for this connection at the time that devices are added to a disk set. However, configured VxVM disk groups do not have an association to any particular set of nodes.
Hot-spare disks – You can use hot-spare disks to increase availability, but hot spare disks are not required.
See your volume-manager documentation for disk layout recommendations and any additional restrictions.
Consider the following points when you plan Solaris Volume Manager configurations:
Local volume names – The name of each local Solaris Volume Manager volume on which a global-devices file system, /global/.devices/node@nodeid, is mounted must be unique throughout the cluster. Also, the name cannot be the same as any device-ID name.
Dual-string mediators – Each disk set configured with exactly two disk strings and mastered by exactly two Solaris hosts must have Solaris Volume Manager mediators configured for the disk set. A disk string consists of a disk enclosure, its physical disks, cables from the enclosure to the host or hosts, and the interface adapter cards. Observe the following rules to configure dual-string mediators:
You must configure each disk set with exactly two hosts that act as mediator hosts.
You must use the same two hosts for all disk sets that require mediators. Those two hosts must master those disk sets.
Mediators cannot be configured for disk sets that do not meet the two-string and two-host requirements.
See the mediator(7D) man page for details.
/kernel/drv/md.conf settings – SPARC: On the Solaris 9 OS, Solaris Volume Manager volumes that are used by each disk set are created in advance, at reconfiguration boot time. This reconfiguration is based on the configuration parameters that exist in the /kernel/drv/md.conf file.
With the Solaris 10 release, Solaris Volume Manager has been enhanced to configure volumes dynamically. You no longer need to edit the nmd and the md_nsets parameters in the /kernel/drv/md.conf file. New volumes are dynamically created, as needed.
You must modify the nmd and md_nsets fields as follows to support a Sun Cluster configuration on the Solaris 9 OS:
All voting cluster nodes must have identical /kernel/drv/md.conf files, regardless of the number of disk sets that are served by each node. Failure to follow this guideline can result in serious Solaris Volume Manager errors and possible loss of data.
md_nsets – The md_nsets field defines the total number of disk sets that can be created for a system to meet the needs of the entire cluster. Set the value of md_nsets to the expected number of disk sets in the cluster plus one additional disk set. Solaris Volume Manager software uses the additional disk set to manage the private disks on the local host.
The maximum number of disk sets that are allowed per cluster is 32. This number allows for 31 disk sets for general use plus one disk set for private disk management. The default value of md_nsets is 4.
nmd – The nmd field defines the highest predicted value of any volume name that will exist in the cluster. For example, if the highest value of the volume names that are used in the first 15 disk sets of a cluster is 10, but the highest value of the volume in the 16th disk set is 1000, set the value of nmd to at least 1000. Also, the value of nmd must be large enough to ensure that enough numbers exist for each device–ID name. The number must also be large enough to ensure that each local volume name can be unique throughout the cluster.
The highest allowed value of a volume name per disk set is 8192. The default value of nmd is 128.
Set these fields at installation time to allow for all predicted future expansion of the cluster. To increase the value of these fields after the cluster is in production is time consuming. The value change requires a reconfiguration reboot for each voting node. To raise these values later also increases the possibility of inadequate space allocation in the root (/) file system to create all of the requested devices.
At the same time, keep the value of the nmdfield and the md_nsets field as low as possible. Memory structures exist for all possible devices as determined by nmdand md_nsets, even if you have not created those devices. For optimal performance, keep the value of nmd and md_nsets only slightly higher than the number of volumes that you plan to use.
See System Files and Startup Files in Solaris Volume Manager Administration Guide (Solaris 9 or Solaris 10) for more information about the md.conf file.
Consider the following points when you plan Veritas Volume Manager (VxVM) configurations.
Accessibility to nodes – You must configure all volume-manager disk groups as either Sun Cluster device groups or as local-only disk groups. If you do not configure the disk group in one of these ways, the devices in the disk group will not be accessible to any node in the cluster.
A device group enables a secondary node to host multihost disks if the primary node fails.
A local-only disk group functions outside the control of Sun Cluster software and can be accessed from only one node at a time.
Enclosure-Based Naming – If you use Enclosure-Based Naming of devices, ensure that you use consistent device names on all cluster nodes that share the same storage. VxVM does not coordinate these names, so the administrator must ensure that VxVM assigns the same names to the same devices from different nodes. Failure to assign consistent names does not interfere with correct cluster behavior. However, inconsistent names greatly complicate cluster administration and greatly increase the possibility of configuration errors, potentially leading to loss of data.
Root disk group – The creation of a root disk group is optional.
A root disk group can be created on the following disks:
The root disk, which must be encapsulated
One or more local nonroot disks, which you can encapsulate or initialize
A combination of root and local nonroot disks
The root disk group must be local to the Solaris host.
Simple root disk groups – Simple root disk groups, which are created on a single slice of the root disk, are not supported as disk types with VxVM on Sun Cluster software. This is a general VxVM software restriction.
Encapsulation – Disks to be encapsulated must have two disk-slice table entries free.
Number of volumes – Estimate the maximum number of volumes any given device group can use at the time the device group is created.
If the number of volumes is less than 1000, you can use default minor numbering.
If the number of volumes is 1000 or greater, you must carefully plan the way in which minor numbers are assigned to device group volumes. No two device groups can have overlapping minor number assignments.
Dirty Region Logging – The use of Dirty Region Logging (DRL) decreases volume recovery time after a node failure. Using DRL might decrease I/O throughput.
Dynamic Multipathing (DMP) – The use of DMP alone to manage multiple I/O paths per Solaris host to the shared storage is not supported. The use of DMP is supported only in the following configurations:
A single I/O path per host is configured to the cluster's shared storage.
A supported multipathing solution is used, such as Solaris I/O multipathing software (MPxIO), EMC PowerPath, or Hitachi HDLM, that manages multiple I/O paths per host to the shared cluster storage.
See your VxVM installation documentation for additional information.
Logging is required for UFS and VxFS cluster file systems. Sun Cluster software supports the following choices of file-system logging:
Solaris UFS logging – See the mount_ufs(1M) man page for more information.
SPARC: Veritas File System (VxFS) logging – See the mount_vxfs man page provided with VxFS software for more information.
Both Solaris Volume Manager and Veritas Volume Manager support both types of file-system logging.
This section provides the following guidelines for planning the mirroring of your cluster configuration:
To mirror all multihost disks in a Sun Cluster configuration enables the configuration to tolerate single-device failures. Sun Cluster software requires that you mirror all multihost disks across expansion units. You do not need to use software mirroring if the storage device provides hardware RAID as well as redundant paths to devices.
Consider the following points when you mirror multihost disks:
Separate disk expansion units – Each submirror of a given mirror or plex should reside in a different multihost expansion unit.
Disk space – Mirroring doubles the amount of necessary disk space.
Three-way mirroring – Solaris Volume Manager software and Veritas Volume Manager (VxVM) software support three-way mirroring. However, Sun Cluster software requires only two-way mirroring.
Differing device sizes – If you mirror to a device of a different size, your mirror capacity is limited to the size of the smallest submirror or plex.
For more information about multihost disks, see Multihost Disk Storage in Sun Cluster Overview for Solaris OS and Sun Cluster Concepts Guide for Solaris OS.
Add this planning information to the Local File System Layout Worksheet.
For maximum availability, mirror root (/), /usr, /var, /opt, and swap on the local disks. Under VxVM, you encapsulate the root disk and mirror the generated subdisks. However, Sun Cluster software does not require that you mirror the root disk.
Before you decide whether to mirror the root disk, consider the risks, complexity, cost, and service time for the various alternatives that concern the root disk. No single mirroring strategy works for all configurations. You might want to consider your local Sun service representative's preferred solution when you decide whether to mirror root.
See your volume-manager documentation and Configuring Solaris Volume Manager Software or Installing and Configuring VxVM Software for instructions about how to mirror the root disk.
Consider the following points when you decide whether to mirror the root disk.
Boot disk – You can set up the mirror to be a bootable root disk. You can then boot from the mirror if the primary boot disk fails.
Complexity – To mirror the root disk adds complexity to system administration. To mirror the root disk also complicates booting in single-user mode.
Backups – Regardless of whether you mirror the root disk, you also should perform regular backups of root. Mirroring alone does not protect against administrative errors. Only a backup plan enables you to restore files that have been accidentally altered or deleted.
Quorum devices – Do not use a disk that was configured as a quorum device to mirror a root disk.
Quorum – Under Solaris Volume Manager software, in failure scenarios in which state database quorum is lost, you cannot reboot the system until maintenance is performed. See your Solaris Volume Manager documentation for information about the state database and state database replicas.
Separate controllers – Highest availability includes mirroring the root disk on a separate controller.
Secondary root disk – With a mirrored root disk, the primary root disk can fail but work can continue on the secondary (mirror) root disk. Later, the primary root disk might return to service, for example, after a power cycle or transient I/O errors. Subsequent boots are then performed by using the primary root disk that is specified for the eeprom(1M) boot-device parameter. In this situation, no manual repair task occurs, but the drive starts working well enough to boot. With Solaris Volume Manager software, a resync does occur. A resync requires a manual step when the drive is returned to service.
If changes were made to any files on the secondary (mirror) root disk, they would not be reflected on the primary root disk during boot time. This condition would cause a stale submirror. For example, changes to the /etc/system file would be lost. With Solaris Volume Manager software, some administrative commands might have changed the /etc/system file while the primary root disk was out of service.
The boot program does not check whether the system is booting from a mirror or from an underlying physical device. The mirroring becomes active partway through the boot process, after the volumes are loaded. Before this point, the system is therefore vulnerable to stale submirror problems.