Sun Cluster 2.2 System Administration Guide

Chapter 2 Sun Cluster Administration Tools

This chapter provides information about the following topics.

Administering the Sun Cluster software is facilitated by three Graphical User Interfaces (GUIs):

Cluster Control Panel - Launches the Cluster Console and other system administration tools.

Cluster Console - Executes commands on multiple nodes in the cluster simultaneously to simplify cluster administration.

Sun Cluster Manager - Monitors the current status of all nodes in the cluster via a HotJava browser.

Refer to the online help for complete documentation on these GUIs. You can also use utilities to monitor Sun Cluster software.

Monitoring Utilities

You can use the Sun Cluster hastat(1M)utility, in addition to the /var/adm/messages files, to monitor a Sun Cluster configuration. You can also use the Sun Cluster Manager graphical user interface, which shows the status of major cluster components and subcomponents. For more information about the Sun Cluster Manager, refer to "Using Sun Cluster Manager ". Sun Cluster also provides an SNMP agent that can be used to monitor up to 32 clusters at the same time. See Appendix C, Using Sun Cluster SNMP Management Solutions.

If you are running Solstice DiskSuite, you can also use the metastat(1M), metadb(1M), metatool(1M), medstat(1M), and mdlogd(1M) utilities to monitor the status of your disksets. The SNMP-based Solstice DiskSuite log daemon, mdlogd(1M), generates a generic SNMP trap when Solstice DiskSuite logs a message to the syslog file. You can configure mdlogd(1M) to send a trap only when certain messages are logged by specifying a regular expression in the mdlogd.cf(4) configuration file. The trap is sent to the administrative host specified in the configuration file. The administrative host must be running a network management application such as Solstice SunNet Manager(TM). You can use mdlogd(1M) if you don't want to run metastat(1M) periodically or scan the syslog output looking for Solstice DiskSuite errors or warnings. See the mdlogd(1M) man page for more information.

If you are running VxVM, you can use the vxprint, vxstat, vxtrace, vxnotify, and vxva utilities. Refer to your volume management software documentation for information about these utilities.


Note -

For information about troubleshooting and repairing defective components, refer to the appropriate hardware documentation.


Monitoring the Configuration With hastat(1M)

The hastat(1M) program displays the current state of the configuration. The program displays status information for the hosts, logical hosts, private networks, public networks, data services, local disks, and disksets, along with the most recent error messages. The hastat(1M) program extracts Sun Cluster-related error messages from the /var/adm/messages file and outputs the last few messages from each host if -m is specified. Because the recent error messages list is a filtered extract of the log messages, the context of some messages might be lost. Check the /var/adm/messages file for a complete list of the messages. The following pages show an example of output from hastat(1M):


# hastat -m 10

HIGH AVAILABILITY CONFIGURATION AND STATUS 
-------------------------------------------

LIST OF NODES CONFIGURED IN <ha-host1> CLUSTER
      phys-host1 phys-host2

CURRENT MEMBERS OF THE CLUSTER

     phys-host1 is a cluster member
     phys-host2 is a cluster member

CONFIGURATION STATE OF THE CLUSTER

     Configuration State on phys-host1: Stable
     Configuration State on phys-host2: Stable

UPTIME OF NODES IN THE CLUSTER

     uptime of phys-host1:         12:47pm  up 12 day(s), 21:11,  1 user, 
load average: 0.21, 0.15, 0.14
     uptime of phys-host2:         12:46pm  up 12 day(s),  3:15,  3 users, 
load average: 0.40, 0.20, 0.16


LOGICAL HOSTS MASTERED BY THE CLUSTER MEMBERS

Logical Hosts Mastered on phys-host1:
        ha-host-1
Loghost Hosts for which phys-host1 is Backup Node:
        ha-host2

Logical Hosts Mastered on phys-host2:
        ha-host2
Loghost Hosts for which phys-host2 is Backup Node:
        ha-host1

LOGICAL HOSTS IN MAINTENANCE STATE

     None

STATUS OF PRIVATE NETS IN THE CLUSTER

     Status of Interconnects on phys-host1:
        interconnect0: selected
        interconnect1: up
     Status of private nets on phys-host1:
        To phys-host1 - UP
        To phys-host2 - UP

     Status of Interconnects on phys-host2:
        interconnect0: selected
        interconnect1: up
     Status of private nets on phys-host2:
        To phys-host1 - UP
        To phys-host2 - UP

STATUS OF PUBLIC NETS IN THE CLUSTER

Status of Public Network On phys-host1:

bkggrp  r_adp   status  fo_time live_adp
nafo0   le0     OK      NEVER   le0

Status of Public Network On phys-host2:

bkggrp  r_adp   status  fo_time live_adp
nafo0   le0     OK      NEVER   le0

STATUS OF SERVICES RUNNING ON LOGICAL HOSTS IN THE CLUSTER

       Status Of Registered Data Services
       q:                       Off
       p:                       Off
       nfs:                     On
       oracle:                  On
       dns:                     On
       nshttp:                  Off
       nsldap:                  On

      Status Of Data Services Running On phys-host1
      Data Service HA-NFS: 
      On Logical Host ha-host1:      Ok
     
      Status Of Data Services Running On phys-host2
      Data Service HA-NFS: 
      On Logical Host ha-host2:      Ok
       
       Data Service "oracle":
       Database Status on phys-host2:
       SC22FILE - running; 

       No Status Method for Data Service "dns"

       RECENT  ERROR MESSAGES FROM THE CLUSTER

       Recent Error Messages on phys-host1
       ...
       Recent Error Messages on phys-host2
       ...

Checking Message Files

The Sun Cluster software writes messages to the /var/adm/messages file, in addition to reporting messages to the console. The following is an example of the messages reported when a disk error occurs.


...
Jun 1 16:15:26 host1 unix: WARNING: /io-unit@f,e1200000/sbi@0.0/SUNW,pln@a0000000,741022/ssd@3,4(ssd49):   
Jun 1 16:15:26 host1 unix: Error for command `write(I))' Err
Jun 1 16:15:27 host1 unix: or Level: Fatal
Jun 1 16:15:27 host1 unix: Requested Block 144004, Error Block: 715559
Jun 1 16:15:27 host1 unix: Sense Key: Media Error
Jun 1 16:15:27 host1 unix: Vendor `CONNER':
Jun 1 16:15:27 host1 unix: ASC=0x10(ID CRC or ECC error),ASCQ=0x0,FRU=0x15
...


Note -

Because Solaris and Sun Cluster error messages are written to the /var/adm/messages file, the /var directory might become full. Refer to "Maintaining the /var File System" for the procedure to correct this problem.


Highly Available Data Service Utilities

In addition, Sun Cluster provides utilities for configuring and administering the highly available data services. The utilities are described in their associated man pages. The utilities include:

Online Help System

Each Sun Cluster administration tool includes detailed online help. To access the online help, launch one of the administration tools from the administrative workstation and select Help from the menu bar.

Alternatively, double-click on the Help icon in the Cluster Control Panel.

Help topics cover the administration tools in detail, as well as some administration tasks. Also see Chapter 4, General Sun Cluster Administration, for additional detailed instructions on performing specific tasks.

Figure 2-1 shows a sample Help window for the Cluster Control Panel. The text covers a specific topic. When you first launch the Help window from a tool, it displays the top-level home topic. Afterwards, the Help window displays the last topic you viewed. Hypertext links to other topics are displayed as underlined, colored text.

Clicking once on a hypertext link displays the text for that topic. The online help system also includes an automatic history list feature that remembers the information you previously accessed. Display this list by choosing Topic History from the View menu.

The Help window has a scrollable text area, a menu bar, and several buttons. Each of these items is described in the following sections.

Figure 2-1 Sample Help Window Home Page for the Cluster Control Panel

Graphic

With Help, you can access each pull-down menu by:

You can customize the mnemonics and accelerators; refer to the online help for more information.

The tables in this section list each menu item, define the menu functions, and list the accelerators (keyboard combinations).

Help Window Menu Bar Items

The Help window menu contains the File, View, and Help menu items. You display the menus for these items by selecting them.

File Menu

The File menu contains these items:

Table 2-1 File Menu Items

Item 

Function 

Accelerator 

Print Topic 

Prints the topic currently displayed in the Help window scrolled text area. 

Alt + R 

Dismiss 

Dismisses the Help window. 

Alt + D 

View Menu

The View menu contains these items:

Table 2-2 View Menu Items

Item 

Function 

Accelerator 

Previous Topic 

Displays the previous help topic (if any). 

Alt + P 

Next Topic 

Displays the next help topic (if any). 

Alt + N 

Home Topic  

Displays the home (top level) topic. 

Alt + O 

Topic History... 

Displays the Help Topic History dialog box which allows you to navigate easily through the help topics you have already viewed. The uppermost topic in the scrolled list is the first topic you displayed. The bottommost topic is the last topic you have viewed in the current path. The highlighted topic is the current topic. 

Alt + I 

To display the dialog box, select View -> Topic History... (Figure 2-2).

Figure 2-2 Help Topic History of the Help Window

Graphic

Help Menu

The Help menu contains these items:

Table 2-3 Help Menu Items

Item 

Function 

Help on Help... 

Describes the Help window and explains how to use it.  

About... 

Displays the About Box, which contains information about the application, such as the version number. 

Help Window Buttons

The following tables lists the help window buttons and their functions.

Table 2-4 Help Menu Items

Button 

Function 

Home 

Displays the home page for the application. 

Dismiss 

Dismisses the Help window. 

Print Topic 

Prints the current topic on your default printer. 

Graphic

Steps back through the displayed Help topics to the previous topic. Clicking on the left arrow repeatedly steps the display back through the Help windows until the first topic you viewed is redisplayed. Topics are "remembered" by the automatic Help history list. 

Graphic

Steps forward through the displayed Help topics, one at a time, to the last topic in the list. 

Cluster Control Panel

The Cluster Control Panel (CCP) is a GUI that enables you to launch the Cluster Console, and other system administration tools. The CCP contains icons that represent these tools.

How to Launch the Cluster Control Panel

After you have installed the Sun Cluster client software on the administrative workstation, use this procedure to run an application from the CCP.

  1. As superuser, add the Sun Cluster tools directory /opt/SUNWcluster/bin to the path on the administrative workstation.


    Note -

    For E10000 Platforms, you must first log into the System Service Processor (SSP) and connect by using the netcon command. Once connected, enter Shift~@ to unlock the console and gain write access. Then proceed to Step 2.


  2. In a shell window on your workstation, bring up the CCP.

    Specify the name of the cluster to be monitored:


    # ccp clustername
    


    Note -

    If the Sun Cluster tools are not installed in the default location of /opt/SUNWcluster, the environment variable $CLUSTER_HOME must be set to the alternate location.


Cluster Control Panel Items

The CCP (shown in the following figure) has a menu bar and an icon pane that displays all of the tools currently in the control panel. From the menu bar you can add, delete, or modify the tools.

Figure 2-3 Sample Cluster Control Panel

Graphic

From the File or Properties menu you can:

For detailed information about the CCP, refer to the online help.

For information about the programs represented by these tools and their usage, see "Cluster Console". For information about using the HotJava browser to monitor cluster configurations, see "Using Sun Cluster Manager ".

Cluster Control Panel Configuration File Locations

The CCP stores properties and other information in configuration files within a configuration directory. By default, the configuration directory is located at /opt/SUNWcluster/etc/ccp.


Note -

You must be root (superuser) to write to this default location. Only root can add, delete, or change CCP items in this configuration directory.


You can, however, create your own configuration directory and define its location using the environment variable $CCP_CONFIG_DIR. The $CCP_CONFIG_DIR variable specifies the configuration directory in which the configuration files containing item properties are stored. If the path name is not set, it defaults to the standard location, /opt/SUNWcluster/etc/ccp. To create your configuration directory, create a new directory and set the environment variable $CCP_CONFIG_DIR to the full path name of the new directory.

These files do not need to be edited manually, because they are created, modified, or deleted by ccp whenever you create, modify, or delete an item.

Cluster Console

The Cluster Console (CC) GUI enables you to run commands on multiple nodes simultaneously, simplifying cluster administration. The Cluster Console displays one terminal window for each cluster node, plus a small Common window that you can use to control all windows simultaneously.

Different types of remote sessions enable you to connect to the console of the host, or remotely log in by using rlogin or telnet. Hosts can be specified on the command line and added or deleted from the Select Hosts dialog box after the program is running. The session type can be specified only on the command line. Once started, the session type cannot be changed.

You can issue commands to multiple hosts from the Common window, and you can issue commands to a single host from a terminal window. Terminal windows use VT100 terminal emulation.

Alternatively, you can turn off all hosts in the Hosts menu except the one you want to access, then issue commands from the Common window text field.

How to Launch the Cluster Console

You can launch the Cluster Console from the CCP (see "Cluster Control Panel") or from the command line in a shell window. If an optional parameter is specified, a terminal window is created for each host in the cluster or for each host specified.

  1. Type cconsole to initiate remote console access:


    % cconsole [clustername | hostname...]

  1. Type ctelnet to establish a telnet(1) connection from the console:


    % ctelnet [clustername | hostname...]

  1. Launch crlogin with your user name to establish an rlogin(1) connection from the console:


    % crlogin -l user name [clustername | hostname...]

All three of the preceding commands also take the standard X/Motif command-line arguments. Once the Cluster Console has started, the Console window is displayed.

For detailed information about the Cluster Console, refer to the online help.

The Common Window Menu Bar

The Common window (shown in the following figure) is the primary window used to send input to all nodes. The Common window is always displayed when you launch the Cluster Console.

Figure 2-4 Common Window Menu Bar of the Cluster Console

Graphic

This window has a menu bar with three menus and a text field for command entry. From the Hosts menu you can use the Select dialog box to:

From the Options menu, you can group or ungroup the Common Window and the terminal windows.

Configuration Files Used by the Cluster Console

Two configuration files are used by the Cluster Console: clusters and serialports. These can be /etc files or NIS/NIS+ databases. The advantage to using a NIS+ environment is that you can run the Cluster Console on multiple Administrative Workstations. Refer to your NIS/NIS+ system administration documentation for complete information about NIS/NIS+.

The clusters File

The clusters file maps a cluster name to the list of host names that comprise the cluster. Each line in the file specifies a cluster, as in this example:


planets      mercury venus earth mars
wine         zinfandel merlot chardonnay riesling

The clusters file is used by all three session types of the Cluster Console (cconsole, ctelnet, and crlogin) to map cluster names to host names on the command line and in the Select Hosts dialog box. For additional information, see "Modifying the clusters File".

The serialports File

The serialports file maps a host name to the Terminal Concentrator and Terminal Concentrator serial port to which the host is connected. Each line in this database specifies a serial port of the host.

Sample serialports file database entries for the Sun Enterprise 10000 are:


mercury    systemserviceprocessorname    23
venus      systemserviceprocessorname    23
earth      systemserviceprocessorname    23
mars       systemserviceprocessorname    23

Sample serialports file database entries for all other nodes are:


mercury        planets-tc   5002
venus          planets-tc   5003
earth          planets-tc   5004
mars           planets-tc   5005

The serialports file is used only by the cconsole variation of this program to determine which Terminal Concentrator and port to connect to for hosts or clusters that are specified in the command line or the Select Hosts dialog box.

In the preceding example, node mercury is connected to planets-tc Port 2, while node venus is connected to planets-tc Port 3. Port 1 is reserved for the administration of the Terminal Concentrator.

For additional information, see "Modifying the serialports File".

Using Sun Cluster Manager

Sun Cluster Manager (SCM) is the cluster management tool for Sun Cluster 2.2. SCM provides a single interface to many of Sun Cluster's command line monitoring features. SCM reports information about:

SCM consists of two parts--the SCM server software and the SCM Graphical User Interface (GUI). The SCM server must run on each node in the cluster. The SCM GUI runs as either an application or an applet. If run as an applet, SCM can be displayed through a browser compliant with the Java Development Kit (JDK(TM)), such as HotJava(TM) or Netscape. The browser can run on any machine, including the cluster nodes. For the most current information about which JDK versions are supported, see your service provider or the latest version of the Sun Cluster 2.2 Release Notes.

Use the information in the following sections to configure and run SCM.

Running SCM As an Application

To run SCM as an application, perform the following procedure.

How to Run SCM As an Application
  1. On the administrative workstation, install the SCM package (SUNWscmgr) from the Sun Cluster 2.2 product CD.

  2. Install the latest version of the SCM patch on all cluster nodes and on the administrative workstation.

    For the most current information about patches and patch numbers, see the Sun Cluster 2.2 Release Notes, your service provider, or the Sun patch web site, http://sunsolve.sun.com.

  3. Run the SCM application.

    Start the application by running the following command from any cluster node, where cluster_node is a current member of the cluster.


    # /opt/SUNWcluster/bin/scmgr cluster_node
    

Once the SCM application is running, you can access the online help for information about menu navigation, tasks, and reference. To display the Help window from the SCM application, select Help Contents from the Help menu. Alternatively, click on the Help icon in the tool bar above the folder icon.

Refer to the scmgr(1M) man page for more information about running SCM.

Running SCM As an Applet

To run SCM as an applet, you must perform these tasks, which are described in more detail in the procedures that follow.

  1. On all cluster nodes, install the latest version of the SCM patch from SunSolve. For the most current information about patches and patch numbers, see the Sun Cluster 2.2 Release Notes, your service provider, or the Sun patch web site, http://sunsolve.sun.com.

  2. Install a browser on the administrative workstation. Supported browsers for Sun Cluster 2.2 4/00 Release are HotJava (version 1.1.4 or greater) and Netscape (version 4.5 or greater).

  3. If you installed HotJava as your browser, install the JDK on the administrative workstation.

  4. Install and configure a web server on all cluster nodes.

  5. Start the SCM applet by typing the appropriate URL in the browser. Make sure the host you specify in the URL is a current member of the cluster.


    Note -

    If you use the HotJava browser shipped with your Solaris 2.6 or 2.7 operating system, you might experience problems when using the menus. For example, after you make a menu selection, the selection might remain visible on the browser. Refer to the Sun Cluster 2.2 Release Notes for more information about SCM open issues. Solaris 8 software does not support a HotJava browser. To run SCM with Solaris 8 software, you must use another browser, such as Netscape. See "How to Run the SCM Applet in a Netscape Browser From a Cluster Node".



    Note -

    If you use the HotJava browser with SCM, you should have at least 40 Megabytes of free swap space. If you find that swap space gets low, restarting the HotJava browser can help.


To run SCM, you must have the correct versions of HotJava and the JDK packages (SUNWjvrt and SUNWjvjit) installed on the system containing the HotJava browser. Check your version numbers against those listed in the following table. Also see the latest version of the Sun Cluster 2.2 Release Notes for possible updates to this information:

Table 2-5 SCM Requirements: JDK and HotJava

Operating Environment 

Java Developer Kit (JDK) version 

HotJava Version 

Netscape Version 

Solaris 2.6 

1.1.6 or later 

1.1.4 or later 

4.5 or later 

Solaris 7 

1.1.6 or later 

1.1.4 or later 

4.5 or later 

Solaris 8 

1.2 or later 

not supported 

4.5 or later 

You can choose to:

Use the following procedures as necessary to set up your preferred configuration.

How to Set Up the JDK
  1. Determine the current Java version by typing the following at the console prompt on a cluster server.


    # java -version
    

  2. If necessary, download a later version of the JDK.

    If the system displays a version of Java lower than 1.1.6, follow the instructions to download the JDK version 1.1.6 (or later) software from the following URL:

    http://www.sun.com/solaris/java

How to Download HotJava
  1. From the machine running the HotJava browser, select About HotJava from the Help menu.

    If the browser displays a version lower than 1.1.4, or if you do not have a HotJava browser, follow the instructions to download the HotJava software version 1.1.4 (or later) from the following URL:

    http://java.sun.com/products/hotjava/index.html

How to Run the SCM Applet in a HotJava Browser From a Cluster Node
  1. Run your HotJava browser on a cluster node.

    The HotJava browser is located in /usr/dt/bin.

  2. Remotely display the HotJava browser on an X windows workstation.

  3. In the HotJava browser, set the applet security preferences:

    1. Choose Applet Security from Preferences on the Edit menu.

    2. Click Medium Security as the Default setting for Unsigned applets.

  4. When you are ready to begin monitoring the cluster with SCM, type the appropriate URL. For example:


    file:/opt/SUNWcluster/scmgr/index.html
    

  5. Click OK on dialog boxes that ask for permission to access certain files, ports, and so forth.


    Note -

    HotJava takes some time to download and run the applet. No status information will appear during this time.


Refer to the online help for complete information about menu navigation, tasks, and reference.

How to Run the SCM Applet in a HotJava Browser From the Administrative Workstation
  1. Run your HotJava browser on a node in the cluster.

    The HotJava browser is located in /usr/dt/bin.

  2. Set up and start a web server on all cluser nodes.

    For details, see "How to Set Up a Web Server to Run With SCM".

  3. Set the applet security preferences in your HotJava browser:

    1. Choose Applet Security from Preferences on the Edit menu.

    2. Click Medium Security as the Default setting for Unsigned applets.

  4. When you are ready to begin monitoring the cluster with SCM, type the appropriate URL.


    http://cluster_node/link_to_scm/index.html
    

  5. Click OK on dialog boxes that ask for permission to access certain files, ports, and so forth from the remote display workstation to the cluster node on which the browser is started.


    Note -

    HotJava takes some time to download and run the applet. No status information will appear during this time.


Refer to the online help for complete information about menu navigation, tasks, and reference.

How to Run the SCM Applet in a Netscape Browser From a Cluster Node
  1. Install Netscape on the cluster nodes.

  2. Install SCM and the required SCM patch on the cluster nodes.

    To install SCM, use scinstall(1M). The scinstall(1M) command installs the SCM package (SUNWscmgr) as part of the server package set. To get the SCM patch, see your service representative or the SunSolve web site:

    http://sunsolve.sun.com/

  3. Add the following lines to the preferences.js file, if necessary.

    The file is located in the $HOME/.netscape directory. If the preferences are not included in the file already, add the following lines:


    user_pref("security.lower_java_network_security_by_trusting_proxies", true);
    user_pref("signed.applets.codebase_principal_support", true);

  4. On a cluster node, set your DISPLAY environment variable so that the Netscape browser is displayed remotely on your X Windows workstation, and then run the Netscape browser on that cluster node.

  5. When you are ready to begin monitoring the cluster with SCM, enter the appropriate URL.


    file:/opt/SUNWcluster/scmgr/index.html
    

  6. Click Grant on Java Security dialog boxes that ask for permission to access certain files, ports, and so forth from the remote display workstation.

    Refer to the online help for complete information about menu navigation, tasks, and reference.

How to Set Up a Web Server to Run With SCM

If you choose, you can install a web server on the cluster nodes to run with SCM.


Note -

If you are running the Sun Cluster HA for Netscape HTTP service and an HTTP server on SCM, you must configure the HTTP servers to listen on different ports, to prevent a port conflict between the HTTP servers.


  1. Install a web server on all nodes in the cluster.

  2. Follow the web server's configuration procedure to make sure that SCM's index.html file is accessible to the clients.

    The client applet for SCM is in the index.html file in the /opt/SUNWcluster/scmgr directory. For example, go to your HTTP server's document_root and create a link to the /opt/SUNWcluster/scmgr directory.

  3. Run the HotJava browser from your workstation.

  4. Set the applet security preferences in your HotJava browser:

    1. Choose Applet Security from Preferences on the Edit menu.

    2. Click Medium Security as the Default setting for Unsigned applets.

  5. When you are ready to begin monitoring the cluster with SCM, type the appropriate URL.

    For example, if you had created a link from the web server's document_root directory to the /opt/SUNWcluster/scmgr directory, you would type the following URL:


    http://cluster_node/scmgr/index.html
    

  6. Click OK on dialog boxes that ask for permission to access certain files, ports, and so forth on the cluster node on which the browser is started.


    Note -

    HotJava takes some time to download and run the applet. No status information will appear during this time.


Refer to the online help for complete information about menu navigation, tasks, and reference.

Accessing SCM Online Help

SCM provides online help information about menu navigation, tasks, and reference. This help is available whether you are running SCM as an application or an applet.

To display the Help window from SCM, select Help Contents from the Help menu. Alternatively, click on the Help icon (question mark) in the tool bar above the folder.

If necessary, you can run online help in a separate browser by typing the following URL:


file:/opt/SUNWcluster/scmgr/help/locale/en/main.howtotopics.html

For example, if you had created a link from the web server's document_root directory to the /opt/SUNWcluster/scmgr directory, you would type the following URL:


http://clusternode/scmgr/help/locale/en/main.howtotopics.html

When you finish viewing the online help, close its HotJava browswer. Selecting online help again brings up a new browser and loads the help.