Sun Cluster Software Installation Guide for Solaris OS

Configuring the Cluster

This section provides information and procedures to configure the software that you installed on the cluster or new cluster node. Before you start to perform these tasks, ensure that you completed the following tasks:

Installed software packages for the Solaris OS, Sun Cluster framework, and other products as described in Installing the Software
Established the new cluster or cluster node as described in Establishing the Cluster

The following table lists the tasks to perform to configure your cluster. Complete the procedures in the order that is indicated.

Note –

If you added a new node to a cluster that uses VxVM, you must perform steps in SPARC: How to Install VERITAS Volume Manager Software to do one of the following tasks:

Install VxVM on that node.
Modify that node's /etc/name_to_major file, to support coexistence with VxVM.

Table 2–5 Task Map: Configuring the Cluster


Task	Instructions
1. Install and configure volume management software:
Install and configure Solstice DiskSuite or Solaris Volume Manager software	Chapter 3, Installing and Configuring Solstice DiskSuite or Solaris Volume Manager Software Solstice DiskSuite or Solaris Volume Manager documentation
SPARC: Install and configure VERITAS Volume Manager software.	Chapter 4, SPARC: Installing and Configuring VERITAS Volume Manager VERITAS Volume Manager documentation
2. Create and mount cluster file systems.	How to Create Cluster File Systems
3. (Solaris 8 or SunPlex Installer installations) Create Internet Protocol (IP) Network Multipathing groups for each public-network adapter that is not already configured in an IP Network Multipathing group.	How to Configure Internet Protocol (IP) Network Multipathing Groups
4. (Optional) Change a node's private hostname.	How to Change Private Hostnames
5. Create or modify the NTP configuration file.	How to Configure Network Time Protocol (NTP)
6. (Optional) SPARC: Install the Sun Cluster module to Sun Management Center software.	SPARC: Installing the Sun Cluster Module for Sun Management Center Sun Management Center documentation
7. Install third-party applications and configure the applications, data services, and resource groups.	Sun Cluster Data Services Planning and Administration Guide for Solaris OS Third-party application documentation

How to Create Cluster File Systems

Perform this procedure for each cluster file system that you want to create. Unlike a local file system, a cluster file system is accessible from any node in the cluster. If you used SunPlex Installer to install data services, SunPlex Installer might have already created one or more cluster file systems.

Caution –

Any data on the disks is destroyed when you create a file system. Be sure that you specify the correct disk device name. If you specify the wrong device name, you might erase data that you did not intend to delete.

Before You Begin

Perform the following tasks:

Ensure that volume-manager software is installed and configured. For volume-manager installation procedures, see Installing and Configuring Solstice DiskSuite or Solaris Volume Manager Software or SPARC: Installing and Configuring VxVM Software.

Determine the mount options to use for each cluster file system that you want to create. Observe the Sun Cluster mount-option requirements and restrictions that are described in the following tables:

Mount Options for UFS Cluster File Systems

Mount Option	Description
`global`	Required. This option makes the file system globally visible to all nodes in the cluster.
`logging`	Required. This option enables logging.
`forcedirectio`	Required for cluster file systems that will host Oracle Real Application Clusters RDBMS data files, log files, and control files. Note – Oracle Real Application Clusters is supported for use only in SPARC based clusters.
`onerror=panic`	Required. You do not have to explicitly specify the `onerror=panic` mount option in the `/etc/vfstab` file. This mount option is already the default value if no other `onerror` mount option is specified. Note – Only the `onerror=panic` mount option is supported by Sun Cluster software. Do not use the `onerror=umount` or `onerror=lock` mount options. These mount options are not supported on cluster file systems for the following reasons: Use of the `onerror=umount` or `onerror=lock` mount option might cause the cluster file system to lock or become inaccessible. This condition might occur if the cluster file system experiences file corruption. The `onerror=umount` or `onerror=lock` mount option might cause the cluster file system to become unmountable. This condition might thereby cause applications that use the cluster file system to hang or prevent the applications from being killed. A node might require rebooting to recover from these states.
`syncdir`	Optional. If you specify `syncdir`, you are guaranteed POSIX-compliant file system behavior for the `write()` system call. If a `write()` succeeds, then this mount option ensures that sufficient space is on the disk. If you do not specify `syncdir`, the same behavior occurs that is seen with UFS file systems. When you do not specify `syncdir`, performance of writes that allocate disk blocks, such as when appending data to a file, can significantly improve. However, in some cases, without `syncdir` you would not discover an out-of-space condition (`ENOSPC`) until you close a file. You see `ENOSPC` on close only during a very short time after a failover. With `syncdir`, as with POSIX behavior, the out-of-space condition would be discovered before the close.

See the mount_ufs(1M) man page for more information about UFS mount options.

Mount Parameters for Sun StorEdge QFS Shared File Systems

Mount Parameter

Description

shared

Required. This option specifies that this is a shared file system, therefore globally visible to all nodes in the cluster.

Caution –
Ensure that settings in the /etc/vfstab file do not conflict with settings in the /etc/opt/SUNWsamfs/samfs.cmd file. Settings in the /etc/vfstab file override settings in the /etc/opt/SUNWsamfs/samfs.cmd file.

See the mount_samfs(1M) man page for more information about QFS mount parameters.

Certain data services such as Sun Cluster Support for Oracle Real Application Clusters have additional requirements and guidelines for QFS mount parameters. See your data service manual for any additional requirements.

Note –
Logging is not enabled by an /etc/vfstab mount parameter, nor does Sun Cluster software require logging for QFS shared file systems.

Mount Parameter	Description
`shared`	Required. This option specifies that this is a shared file system, therefore globally visible to all nodes in the cluster.

Mount Options for VxFS Cluster File Systems

Mount Option	Description
`global`	Required. This option makes the file system globally visible to all nodes in the cluster.
`log`	Required. This option enables logging.

See the VxFS mount_vxfs man page and Administering Cluster File Systems Overview in Sun Cluster System Administration Guide for Solaris OS for more information about VxFS mount options.

Steps

Become superuser on any node in the cluster.

Tip –
For faster file-system creation, become superuser on the current primary of the global device for which you create a file system.

Create a file system.

For a UFS file system, use the newfs(1M) command.

# newfs raw-disk-device

The following table shows examples of names for the raw-disk-device argument. Note that naming conventions differ for each volume manager.

Volume Manager	Sample Disk Device Name	Description
Solstice DiskSuite or Solaris Volume Manager	`/dev/md/nfs/rdsk/d1`	Raw disk device `d1` within the `nfs` disk set
SPARC: VERITAS Volume Manager	`/dev/vx/rdsk/oradg/vol01`	Raw disk device `vol01` within the `oradg` disk group
None	`/dev/global/rdsk/d1s3`	Raw disk device `d1s3`

For a Sun StorEdge QFS file system, follow the procedures for defining the configuration in the Sun StorEdge QFS and Sun StorEdge SAM-FS Software Installation and Configuration Guide.

SPARC: For a VERITAS File System (VxFS) file system, follow the procedures that are provided in your VxFS documentation.

On each node in the cluster, create a mount-point directory for the cluster file system.

A mount point is required on each node, even if the cluster file system is not accessed on that node.

Tip –
For ease of administration, create the mount point in the /global/device-group/ directory. This location enables you to easily distinguish cluster file systems, which are globally available, from local file systems.
# mkdir -p /global/device-group/mountpoint/
device-group

Name of the directory that corresponds to the name of the device group that contains the device

mountpoint

Name of the directory on which to mount the cluster file system

On each node in the cluster, add an entry to the /etc/vfstab file for the mount point.

See the vfstab(4) man page for details.
1. In each entry, specify the required mount options for the type of file system that you use.
  
  Note –
  Do not use the logging mount option for Solstice DiskSuite trans metadevices or Solaris Volume Manager transactional volumes. Trans metadevices and transactional volumes provide their own logging.
  
  In addition, Solaris Volume Manager transactional-volume logging (formerly Solstice DiskSuite trans-metadevice logging) is scheduled to be removed from the Solaris OS in an upcoming Solaris release. Solaris UFS logging provides the same capabilities but superior performance, as well as lower system administration requirements and overhead.
2. To automatically mount the cluster file system, set the mount at boot field to yes.
3. Ensure that, for each cluster file system, the information in its /etc/vfstab entry is identical on each node.
4. Ensure that the entries in each node's /etc/vfstab file list devices in the same order.
5. Check the boot order dependencies of the file systems.
  
  For example, consider the scenario where phys-schost-1 mounts disk device d0 on /global/oracle/, and phys-schost-2 mounts disk device d1 on /global/oracle/logs/. With this configuration, phys-schost-2 can boot and mount /global/oracle/logs/ only after phys-schost-1 boots and mounts /global/oracle/.

On any node in the cluster, run the sccheck(1M) utility.

The sccheck utility verifies that the mount points exist. The utility also verifies that /etc/vfstab file entries are correct on all nodes of the cluster.
# sccheck
If no errors occur, nothing is returned.

Mount the cluster file system.
# mount /global/device-group/mountpoint/
- For UFS and QFS, mount the cluster file system from any node in the cluster.
- SPARC: For VxFS, mount the cluster file system from the current master of device-group to ensure that the file system mounts successfully. In addition, unmount a VxFS file system from the current master of device-group to ensure that the file system unmounts successfully.
  
  Note –
  To manage a VxFS cluster file system in a Sun Cluster environment, run administrative commands only from the primary node on which the VxFS cluster file system is mounted.

On each node of the cluster, verify that the cluster file system is mounted.

You can use either the df(1M) or mount(1M) command to list mounted file systems.

Example 2–4 Creating a Cluster File System

The following example creates a UFS cluster file system on the Solstice DiskSuite metadevice /dev/md/oracle/rdsk/d1.

# newfs /dev/md/oracle/rdsk/d1
…
 
(on each node)
# mkdir -p /global/oracle/d1
# vi /etc/vfstab
#device           device        mount   FS      fsck    mount   mount
#to mount         to fsck       point   type   ; pass    at boot options
#                     
/dev/md/oracle/dsk/d1 /dev/md/oracle/rdsk/d1 /global/oracle/d1 ufs 2 yes global,logging
(save and exit)
 
(on one node)
# sccheck
# mount /global/oracle/d1
# mount
…
/global/oracle/d1 on /dev/md/oracle/dsk/d1 read/write/setuid/global/logging/largefiles
on Sun Oct 3 08:56:16 2000

Next Steps

If you installed Sun Cluster software on the Solaris 8 OS or you used SunPlex Installer to install the cluster, go to How to Configure Internet Protocol (IP) Network Multipathing Groups.

If you want to change any private hostnames, go to How to Change Private Hostnames.

If you did not install your own /etc/inet/ntp.conf file before you installed Sun Cluster software, install or create the NTP configuration file. Go to How to Configure Network Time Protocol (NTP).

SPARC: If you want to configure Sun Management Center to monitor the cluster, go to SPARC: Installing the Sun Cluster Module for Sun Management Center.

Otherwise, install third-party applications, register resource types, set up resource groups, and configure data services. Follow procedures in the Sun Cluster Data Services Planning and Administration Guide for Solaris OS and in the documentation that is supplied with your application software.

How to Configure Internet Protocol (IP) Network Multipathing Groups

Perform this task on each node of the cluster. If you used SunPlex Installer to install Sun Cluster HA for Apache or Sun Cluster HA for NFS, SunPlex Installer configured IP Network Multipathing groups for the public-network adapters those data services use. You must configure IP Network Multipathing groups for the remaining public-network adapters.

Note –

All public-network adapters must belong to an IP Network Multipathing group.

Before You Begin

Have available your completed Public Networks Worksheet.

Step

Configure IP Network Multipathing groups.

Perform procedures in Deploying Network Multipathing in IP Network Multipathing Administration Guide (Solaris 8), Configuring Multipathing Interface Groups in System Administration Guide: IP Services (Solaris 9), or Configuring IPMP Groups in System Administration Guide: IP Services (Solaris 10).
Follow these additional requirements to configure IP Network Multipathing groups in a Sun Cluster configuration:
- Each public network adapter must belong to a multipathing group.
- In the following kinds of multipathing groups, you must configure a test IP address for each adapter in the group:
  - On the Solaris 8 OS, all multipathing groups require a test IP address for each adapter.
  - On the Solaris 9 or Solaris 10 OS, multipathing groups that contain two or more adapters require test IP addresses. If a multipathing group contains only one adapter, you do not need to configure a test IP address.
- Test IP addresses for all adapters in the same multipathing group must belong to a single IP subnet.
- Test IP addresses must not be used by normal applications because the test IP addresses are not highly available.
- In the /etc/default/mpathd file, the value of TRACK_INTERFACES_ONLY_WITH_GROUPS must be yes.
- The name of a multipathing group has no requirements or restrictions.

Next Steps

If you want to change any private hostnames, go to How to Change Private Hostnames.

If you did not install your own /etc/inet/ntp.conf file before you installed Sun Cluster software, install or create the NTP configuration file. Go to How to Configure Network Time Protocol (NTP).

If you are using Sun Cluster on a SPARC based system and you want to use Sun Management Center to monitor the cluster, install the Sun Cluster module for Sun Management Center. Go to SPARC: Installing the Sun Cluster Module for Sun Management Center.

How to Change Private Hostnames

Perform this task if you do not want to use the default private hostnames, clusternodenodeid-priv, that are assigned during Sun Cluster software installation.

Note –

Do not perform this procedure after applications and data services have been configured and have been started. Otherwise, an application or data service might continue to use the old private hostname after the hostname is renamed, which would cause hostname conflicts. If any applications or data services are running, stop them before you perform this procedure.

Perform this procedure on one active node of the cluster.

Steps

Become superuser on a node in the cluster.

Start the scsetup(1M) utility.
# scsetup

From the Main Menu, choose the menu item, Private hostnames.

From the Private Hostname Menu, choose the menu item, Change a private hostname.

Follow the prompts to change the private hostname.

Repeat for each private hostname to change.

Verify the new private hostnames.

# scconf -pv | grep "private hostname"
(phys-schost-1) Node private hostname:      phys-schost-1-priv
(phys-schost-3) Node private hostname:      phys-schost-3-priv
(phys-schost-2) Node private hostname:      phys-schost-2-priv

Next Steps

If you did not install your own /etc/inet/ntp.conf file before you installed Sun Cluster software, install or create the NTP configuration file. Go to How to Configure Network Time Protocol (NTP).

SPARC: If you want to configure Sun Management Center to monitor the cluster, go to SPARC: Installing the Sun Cluster Module for Sun Management Center.

Otherwise, install third-party applications, register resource types, set up resource groups, and configure data services. See the documentation that is supplied with the application software and the Sun Cluster Data Services Planning and Administration Guide for Solaris OS.

How to Configure Network Time Protocol (NTP)

Note –

If you installed your own /etc/inet/ntp.conf file before you installed Sun Cluster software, you do not need to perform this procedure. Determine your next step:

SPARC: If you want to configure Sun Management Center to monitor the cluster, go to SPARC: Installing the Sun Cluster Module for Sun Management Center.
Otherwise, install third-party applications, register resource types, set up resource groups, and configure data services. See the documentation that is supplied with the application software and the Sun Cluster Data Services Planning and Administration Guide for Solaris OS.

Perform this task to create or modify the NTP configuration file after you perform any of the following tasks:

Install Sun Cluster software
Add a node to an existing cluster
Change the private hostname of a node in the cluster

If you added a node to a single-node cluster, you must ensure that the NTP configuration file that you use is copied to the original cluster node as well as to the new node.

The primary requirement when you configure NTP, or any time synchronization facility within the cluster, is that all cluster nodes must be synchronized to the same time. Consider accuracy of time on individual nodes to be of secondary importance to the synchronization of time among nodes. You are free to configure NTP as best meets your individual needs if this basic requirement for synchronization is met.

See the Sun Cluster Concepts Guide for Solaris OS for further information about cluster time. See the /etc/inet/ntp.cluster template file for additional guidelines on how to configure NTP for a Sun Cluster configuration.

Steps

Become superuser on a cluster node.

If you have your own file, copy your file to each node of the cluster.

If you do not have your own /etc/inet/ntp.conf file to install, use the /etc/inet/ntp.conf.cluster file as your NTP configuration file.

Note –
Do not rename the ntp.conf.cluster file as ntp.conf.

If the /etc/inet/ntp.conf.cluster file does not exist on the node, you might have an /etc/inet/ntp.conf file from an earlier installation of Sun Cluster software. Sun Cluster software creates the /etc/inet/ntp.conf.cluster file as the NTP configuration file if an /etc/inet/ntp.conf file is not already present on the node. If so, perform the following edits instead on that ntp.conf file.
1. Use your preferred text editor to open the /etc/inet/ntp.conf.cluster file on one node of the cluster for editing.
2. Ensure that an entry exists for the private hostname of each cluster node.
  
  If you changed any node's private hostname, ensure that the NTP configuration file contains the new private hostname.
3. If necessary, make other modifications to meet your NTP requirements.

Copy the NTP configuration file to all nodes in the cluster.

The contents of the NTP configuration file must be identical on all cluster nodes.

Stop the NTP daemon on each node.

Wait for the command to complete successfully on each node before you proceed to Step 6.
- For the Solaris 8 or Solaris 9 OS, use the following command:
  # /etc/init.d/xntpd stop
- For the Solaris 10 OS, use the following command:
  # svcadm disable ntp

Restart the NTP daemon on each node.
- If you use the ntp.conf.cluster file, run the following command:
  # /etc/init.d/xntpd.cluster start
  The xntpd.cluster startup script first looks for the /etc/inet/ntp.conf file.
  - If the ntp.conf file exists, the script exits immediately without starting the NTP daemon.
  - If the ntp.conf file does not exist but the ntp.conf.cluster file does exist, the script starts the NTP daemon. In this case, the script uses the ntp.conf.cluster file as the NTP configuration file.
- If you use the ntp.conf file, run one of the following commands:
  - For the Solaris 8 or Solaris 9 OS, use the following command:
    # /etc/init.d/xntpd start
  - For the Solaris 10 OS, use the following command:
    # svcadm enable ntp

Next Steps

SPARC: To configure Sun Management Center to monitor the cluster, go to SPARC: Installing the Sun Cluster Module for Sun Management Center.