Sun Cluster 3.0 System Administration Guide

Chapter 1 Introduction to Administering Sun Cluster

This chapter provides information on preparing to administer the cluster and the procedures for using Sun Cluster administration tools.

This is a list of the procedures in this chapter.

1.1 Administering Sun Cluster Overview

Sun Cluster's highly-available environment ensures that critical applications are available to end users. The system administrator's job is to make sure that Sun Cluster is stable and operational.

Before undertaking an administrative task, you should have familiarized yourself with the planning information in the Sun Cluster 3.0 Installation Guide and the glossary in the Sun Cluster 3.0 Concepts document. Sun Cluster administration is organized into tasks among the following manuals:

For the most part, you can perform Sun Cluster administration tasks while the cluster is operational, with the impact on cluster operationlimited to a single node. For those procedures that require that the entire cluster be shut down, schedule downtime for off hours, when there is minimal impact on the system. If you plan to take down the cluster or a cluster node, notify users ahead of time.

1.2 Graphical User Interface

The Sun Cluster-supplied module for Sun Management CenterTM (formerly Sun Enterprise SyMONTM) GUI Console enables you to graphically display cluster resources, resource types, and resource groups. It also enables you to monitor configuration changes and check the status of cluster components. However, the Sun Cluster-supplied module for Sun Management Center currently cannot perform all Sun Cluster administration tasks. You must use the command-line interface for other operations. See "1.3 Command-Line Interface" for more information.

For information on installing and starting the Sun Cluster module for Sun Management Center, and for viewing the cluster-specific online help supplied with the Sun Cluster module, see the Sun Cluster 3.0 Installation Guide.

Sun Cluster has created the SNMP MIB for the Sun Cluster module of Sun Management Center. This MIB can be used as the data definition by third-party SNMP based management stations.

The Sun Cluster MIB file is located in $BASEDIR/SUNWsymon/modules/cfg on any cluster node.

The Sun Cluster MIB file is an ASN.1 specification of the Sun Cluster data that is modeled. This is the same specification used by all Sun Management Center MIBs. To use the Sun Cluster MIB, refer to the instructions for using other Sun Management Center MIBs in Sun MC 2.1 Software User's Guide, Appendix B, "SNMP MIBs for Sun Management Center Modules."

1.3 Command-Line Interface

You can perform most Sun Cluster administration tasks interactively through the scsetup(1M) utility. Whenever possible, administration procedures in this guide are described using scsetup.

You can administer the following items through the scsetup utility:

Listed here are the other commands you use to administer Sun Cluster. See the man pages for more detailed information.

Table 1-1 Sun Cluster Command-Line Interface Commands

Command 

Description 

ccp(1M)

Starts remote console access to the cluster. 

pmfadm(1M)

Provides administrative access to the process monitor facility. 

pnmset(1M)

Configures Public Network Management (PNM). 

pnmstat(1M)

Reports the status of Network Adapter Failover (NAFO) groups monitored by PNM. 

sccheck(1M)

Checks and validates the global mount entries in the /etc/vfstab file.

scconf(1M)

Updates a Sun Cluster configuration. The -p option lists cluster configuration information.

scdidadm(1M)

Provides administrative access to the device ID configuration. 

scgdevs(1M)

Runs the global device namespace administration script. 

scinstall(1M)

Installs and configures Sun Cluster software; can be run interactively or non-interactively. The -p option displays release and package version information for the Sun Cluster software.

scrgadm(1M)

Manages the registration of resource types, the creation of resource groups, and the activation of resources within a resource group. The -p option displays information on installed resources, resource groups, and resource types.

scsetup(1M)

Runs the interactive cluster configuration utility. 

scshutdown(1M)

Shuts down the entire cluster. 

scstat(1M)

Provides a snapshot of the cluster status. 

scswitch(1M)

Performs changes affecting node mastery and states for resource groups and disk device groups. 

In addition, you use commands to administer the volume manager portion of Sun Cluster. These commands depend on the specific volume manager used in your cluster, either Solstice DiskSuiteTM or VERITAS Volume Manager.

1.4 Preparing to Administer the Cluster

This section describes what to do to prepare for administering your cluster.

1.4.1 Documenting a Sun Cluster Hardware Configuration

As your Sun Cluster configuration grows and changes, you need to document the hardware aspects that are unique to your site. Labeling cables and connections between the various cluster components also saves administration time when it becomes necessary to change or upgrade the cluster.

Keeping such a record of your cluster configuration will also come in handy when servicing the cluster. If you use a third-party service provider, such record keeping will make a service provider's job easier.

1.4.2 Using an Administrative Console

If you choose, you can use a dedicated SPARC workstation, known as the administrative console, to administer the active cluster. Typically, you install and run the Cluster Control Panel (CCP) and Sun Management Center server and console software on the administrative console. For more information on the CCP, see "1.5.1 How to Remotely Log In to Sun Cluster". For instructions on installing the Sun Management Center software, see the Sun Cluster 3.0 Installation Guide.

The administrative console is not a cluster node. The administrative console is used for remote access to the cluster nodes, either over the public network or through a network-based terminal concentrator. If your cluster consists of a Sun EnterpriseTM 10000 server, you must have the ability to log in from the administrative console to the System Service Processor (SSP) and connect by using the netcon command.

Sun Cluster does not require a dedicated administrative console, but using one provides these benefits:

1.4.3 Backing Up the Cluster

While not specifically discussed in this guide, it is important to back up your cluster on a regular basis. Even though Sun Cluster provides an HA environment, with mirrored copies of data on the storage devices, do not consider this a replacement for regular backups. Sun Cluster can survive multiple failures, but not a catastrophic failure in which all data is rendered unavailable. Therefore, you must have a backup procedure in place to protect against catastrophic data loss. Backups are also important to undo user or program errors.

The following information should be included as part of your backup.

1.5 Beginning to Administer the Cluster

Table 1-2 provides a starting point for administering your cluster.

Table 1-2 Sun Cluster 3.0 Administration Tools

If You Want To... 

Then... 

For More Information Go To... 

Remotely Log in to the Cluster 

Use the ccp command to launch the Cluster Control Panel (CCP). Then select one of the following icons: cconsole, crlogin, or ctelnet.

"1.5.1 How to Remotely Log In to Sun Cluster"

Interactively Configure the Cluster 

Launch the scsetup utility.

"1.5.2 How to Access the scsetup Utility"

Display Sun Cluster Release Number and Version Information 

Use the scinstall command with either the -p or -pv options.

"1.5.3 How to Display Sun Cluster Release and Version Information"

Display Installed Resources, Resource Groups, and Resource Types 

Use the scgradm -p command.

"1.5.4 How to Display Configured Resource Types, Resource Groups, and Resources"

Graphically Monitor Cluster Components 

Use the Sun Cluster module for Sun Management Center. 

Sun Cluster module for Sun Management Center online help

Check the Status of Cluster Components 

Use the scstat command.

"1.5.5 How to Check the Status of Cluster Components"

View the Cluster Configuration 

Use the scconf -p command.

"1.5.6 How to View the Cluster Configuration"

Check Global Mount Points 

Use the sccheck command.

"1.5.7 How to Check the Global Mount Points"

Look at Sun Cluster System Messages 

Examine the/var/adm/messages file.

Solaris system administration documentation 

Monitor the Status of Solstice DiskSuite 

Use the metastat or metatool commands.

Solstice DiskSuite documentation 

Monitor the Status of VERITAS Volume Manager 

Use the vxstat or vxva commands.

VERITAS Volume Manager documentation 

1.5.1 How to Remotely Log In to Sun Cluster

The Cluster Control Panel (CCP) provides a launch pad for cconsole, crlogin, and ctelnet tools. All three tools start a multiple window connection to a set of specified nodes. The multiple-window connection consists of a host window for each of the specified nodes and a common window. Input directed into the common window is sent to each of these host windows. See the ccp(1M) and cconsole(1M) man pages for more information.

  1. Verify that the following prerequisites are met. To start the Cluster Control Panel (CCP), you must:

    • Install the appropriate Sun Cluster software (SUNWccon package) on the administrative console.

    • Make sure the PATH variable on the administrative console includes the Sun Cluster tools directory, /opt/SUNWcluster/bin, and /usr/cluster/bin. You can specify an alternate location for the tools directory by setting the $CLUSTER_HOME environment variable.

    • Configure the clusters file, the serialports file, and the nsswitch.conf file if using a terminal concentrator. These can be either /etc files or NIS/NIS+ databases. See clusters(4) and serialports(4) for more information.

  2. Determine if you have a Sun Enterprise E10000 server platform.

    • If yes, log into the System Service Processor (SSP) and connect by using the netcon command. Once connected, enter Shift~@ to unlock the console and gain write access.

  3. Start the CCP launch pad.

    From the administrative console, enter the following command.


    # ccp clustername
    

    The CCP launch pad appears.

  4. To start a remote session with the cluster, click the appropriate icon (cconsole, crlogin, or ctelnet) in the CCP launch pad.

1.5.1.1 Example

The following example shows the Cluster Control Panel.

Figure 1-1 Cluster Control Panel

Graphic

1.5.1.2 Where to Go From Here

You can also start cconsole, crlogin, or ctelnet sessions from the command line. See cconsole(1M) for more information.

1.5.2 How to Access the scsetup Utility

The scsetup(1M) utility enables you to interactively configure quorum, cluster transport, private hostnames, device groups, and new node options for the cluster.

  1. Become superuser on a node in the cluster.

  2. Enter the scsetup utility.


    # scsetup
    

    The Main Menu appears.

  3. Make your selection from the menu and follow the onscreen instructions.

    See the scsetup online help for more information.

1.5.3 How to Display Sun Cluster Release and Version Information

You do not need to be logged in as superuser to perform these procedures.

    Display the Sun Cluster release number.


    % scinstall -p
    

    Display the Sun Cluster release number and version strings for all Sun Cluster packages.


    % scinstall -pv
    

1.5.3.1 Examples--Displaying the Sun Cluster Release Number

The following example displays the cluster's release number.


% scinstall -p
3.0

1.5.3.2 Examples--Displaying Sun Cluster Release and Version Information

The following example displays the cluster's release information and version information for all packages.


% scinstall -pv
SunCluster 3.0
SUNWscr:       3.0.0,REV=1999.10.20.15.01
SUNWscdev:     3.0.0,REV=1999.10.20.15.01
SUNWscu:       3.0.0,REV=1999.10.20.15.01
SUNWscman:     3.0.0,REV=1999.10.20.15.01
SUNWscsal:     3.0.0,REV=1999.10.20.15.01
SUNWscsam:     3.0.0,REV=1999.10.20.15.01
SUNWrsmop:     3.0.0,REV=1999.10.20.15.01
SUNWsci:       3.0,REV=1999.09.08.17.43
SUNWscid:      3.0,REV=1999.09.08.17.43
SUNWscidx:     3.0,REV=1999.09.08.17.43
SUNWscvm:      3.0.0,REV=1999.10.20.15.01

1.5.4 How to Display Configured Resource Types, Resource Groups, and Resources

You do not need to be logged in as superuser to perform this procedure.

    Display the cluster's configured resource types, resource groups, and resources.


    % scrgadm -p
    

1.5.4.1 Example--Displaying Configured Resource Types, Resource Groups, and Resources

The following example shows the resource types (RT Name), resource groups (RG Name), and resources (RS Name) configured for the cluster schost.


% scgradm -p
RT Name: SUNW.SharedAddress
  RT Description: HA Shared Address Resource Type 
RT Name: SUNW.LogicalHostname
  RT Description: Logical Hostname Resource Type 
RG Name: schost-sa-1 
  RG Description:  
    RS Name: schost-1
      RS Description: 
      RS Type: SUNW.SharedAddress
      RS Resource Group: schost-sa-1
RG Name: schost-lh-1 
  RG Description:  
    RS Name: schost-3
      RS Description: 
      RS Type: SUNW.LogicalHostname
      RS Resource Group: schost-lh-1

1.5.5 How to Check the Status of Cluster Components

You do not need to be logged in as superuser to perform this procedure.

    Check the status of cluster components.


    % scstat -p
    

1.5.5.1 Example--Checking the Status of Cluster Components

The following example provides a sample of status information for cluster components returned by scstat(1M).


% scstat -p
-- Cluster Nodes --
 
                    Node name           Status
                    ---------           ------
  Cluster node:     phys-schost-1      Online
  Cluster node:     phys-schost-2      Online
  Cluster node:     phys-schost-3      Online
  Cluster node:     phys-schost-4      Online
 
------------------------------------------------------------------
 
-- Cluster Transport Paths --
 
                    Endpoint            Endpoint            Status
                    --------            --------            ------
  Transport path:   phys-schost-1:qfe1 phys-schost-4:qfe1 Path online
  Transport path:   phys-schost-1:hme1 phys-schost-4:hme1 Path online
...
 
------------------------------------------------------------------
 
-- Quorum Summary --
 
  Quorum votes possible:      6
  Quorum votes needed:        4
  Quorum votes present:       6
 
-- Quorum Votes by Node --
 
                    Node Name           Present Possible Status
                    ---------           ------- -------- ------
  Node votes:       phys-schost-1      1        1       Online
  Node votes:       phys-schost-2      1        1       Online
...
 
-- Quorum Votes by Device --
 
                    Device Name         Present Possible Status  Owner
                    -----------         ------- -------- ------  -----
  Device votes:     /dev/did/rdsk/d2s2  1        1       Online  phys-schost-2
  Device votes:     /dev/did/rdsk/d8s2  1        1       Online  phys-schost-4
...
 
-- Device Group Servers --
 
                         Device Group        Primary             Secondary
                         ------------        -------             ---------
  Device group servers:  rmt/1               -                   -
  Device group servers:  rmt/2               -                   -
  Device group servers:  schost-1           phys-schost-2      phys-schost-1
  Device group servers:  schost-3           -                   -
 
-- Device Group Status --
 
                              Device Group        Status              
                              ------------        ------              
  Device group status:        rmt/1               Offline
  Device group status:        rmt/2               Offline
  Device group status:        schost-1           Online
  Device group status:        schost-3           Offline
 
------------------------------------------------------------------
 
-- Resource Groups and Resources --
 
            Group Name          Resources
            ----------          ---------
 Resources: test-rg             test_1
 Resources: real-property-rg    -
 Resources: failover-rg         -
 Resources: descript-rg-1       -
...
 
-- Resource Groups --
 
            Group Name          Node Name           State
            ----------          ---------           -----
     Group: test-rg             phys-schost-1      Offline
     Group: test-rg             phys-schost-2      Offline
...
 
-- Resources --
 
            Resource Name       Node Name           State     Status Message
            -------------       ---------           -----     --------------
  Resource: test_1              phys-schost-1      Offline   Offline
  Resource: test_1              phys-schost-2      Offline   Offline

1.5.6 How to View the Cluster Configuration

You do not need to be logged in as superuser to perform this procedure.

    View the cluster configuration.


    % scconf -p
    

1.5.6.1 Example--Viewing the Cluster Configuration

The following example lists the cluster configuration.


% scconf -p
Cluster name:                                      cluster-1
Cluster ID:                                        0x3908EE1C
Cluster install mode:                              disabled
Cluster private net:                               172.16.0.0
Cluster private netmask:                           255.255.0.0
Cluster new node authentication:                   unix
Cluster new node list:                             <NULL - Allow any node>
Cluster nodes:                                     phys-schost-1 phys-schost-2 phys-schost-3 phys-schost-4
 
Cluster node name:                                 phys-schost-1
  Node ID:                                         1
  Node enabled:                                    yes
  Node private hostname:                           clusternode1-priv
  Node quorum vote count:                          1
  Node reservation key:                            0x3908EE1C00000001
  Node transport adapters:                         hme1 qfe1 qfe2
 
Node transport adapter:                          hme1
    Adapter enabled:                               yes
    Adapter transport type:                        dlpi
    Adapter property:                              device_name=hme
   Adapter property:                              device_instance=1
    Adapter property:                              dlpi_heartbeat_timeout=10000
...
Cluster transport junctions:                       hub0 hub1 hub2
 
Cluster transport junction:                        hub0
  Junction enabled:                                yes
  Junction type:                                   switch
 Junction port names:                             1 2 3 4
...

Junction port:                                   1
    Port enabled:                                  yes
 
Junction port:                                   2
    Port enabled:                                  yes
...
Cluster transport cables
                    Endpoint            Endpoint        State
                    --------            --------        -----
  Transport cable:  phys-schost-1:hme1@0 hub0@1        Enabled
  Transport cable:  phys-schost-1:qfe1@0 hub1@1        Enabled
  Transport cable:  phys-schost-1:qfe2@0 hub2@1        Enabled
  Transport cable:  phys-schost-2:hme1@0 hub0@2        Enabled
...
Quorum devices:                                    d2 d8
 
Quorum device name:                                d2
  Quorum device votes:                             1
  Quorum device enabled:                           yes
  Quorum device name:                              /dev/did/rdsk/d2s2
  Quorum device hosts (enabled):                   phys-schost-1
 phys-schost-2
  Quorum device hosts (disabled): 
...
Device group name:                                 schost-3
  Device group type:                               SDS
  Device group failback enabled:                   no
  Device group node list:                          phys-schost-3, phys-schost-4
  Diskset name:                                    schost-3
...

1.5.7 How to Check the Global Mount Points

The sccheck(1M) command checks the /etc/vfstab file for configuration errors with the cluster file system and its global mount points. The sccheck command only returns errors. If no errors are found, sccheck merely returns to the shell prompt.


Note -

Run sccheck after making cluster configuration changes that have affected devices or volume management components.


  1. Become superuser on a node in the cluster.

  2. Verify the cluster configuration.


    # sccheck
    

1.5.7.1 Example--Verifying the Cluster Configuration

The following example shows that the node phys-schost-3 is missing the mount point /global/schost-1.


# sccheck
vfstab-check: WARNING - phys-schost-3 - Missing mount point /global/schost-1