test

Sun Cluster 2.2 System Administration Guide

Chapter 2 Sun Cluster Administration Tools

This chapter provides information on the following topics.

This chapter includes the following procedures:

Administering the Sun Cluster software is facilitated by three Graphical User Interfaces (GUIs):

Cluster Control Panel - Launches the Cluster Console and other system administration tools.

Cluster Console - Executes commands on multiple nodes in the cluster simultaneously to simplify cluster administration.

Sun Cluster Manager - Monitors the current status of all nodes in the cluster via a HotJava browser.

Refer to the online help for complete documentation on these GUIs. You can also use utilities to monitor Sun Cluster software.

2.1 Monitoring Utilities

You can use the Sun Cluster hastat(1M)utility, in addition to the /var/adm/messages files, to monitor a Sun Cluster configuration. You can also use the Sun Cluster Manager graphical user interface, which shows the status of major cluster components and subcomponents. For more information about the Sun Cluster Manager, refer to "2.5 Monitoring Sun Cluster Servers With Sun Cluster Manager ". Sun Cluster also provides an SNMP agent that can be used to monitor a maximum of 32 clusters at the same time. See Appendix D, Using Sun Cluster SNMP Management Solutions.

If you are running Solstice DiskSuite, you can also use the metastat(1M), metadb(1M), metatool(1M), medstat(1M), and mdlogd(1M) utilities to monitor the status of your disksets. The SNMP-based Solstice DiskSuite log daemon, mdlogd(1M), generates a generic SNMP trap when Solstice DiskSuite logs a message to the syslog file. You can configure mdlogd(1M) to send a trap only when certain messages are logged by specifying a regular expression in the mdlogd.cf(4) configuration file. The trap is sent to the administrative host specified in the configuration file. The administrative host must be running a network management application such as Solstice SunNet ManagerTM. You can use mdlogd(1M) if you don't want to run metastat(1M) periodically or scan the syslog output looking for Solstice DiskSuite errors or warnings. See the mdlogd(1M) man page for more information.

If you are running SSVM or CVM, you can use the vxprint, vxstat, vxtrace, vxnotify, and vxva utilities. Refer to your volume management software documentation for information on these utilities.

Note -

For information about troubleshooting and repairing defective components, refer to the appropriate hardware documentation.

2.1.1 Monitoring the Configuration With hastat(1M)

The hastat(1M) program displays the current state of the configuration. The program displays status information about the hosts, logical hosts, private networks, public networks, data services, local disks, and disksets, along with the most recent error messages. The hastat(1M) program extracts Sun Cluster-related error messages from the /var/adm/messages file and outputs the last few messages from each host if -m is specified. Because the recent error messages list is a filtered extract of the log messages, the context of some messages might be lost. Check the /var/adm/messages file for a complete list of the messages. The following pages show an example of output from hastat(1M): 

# hastat -m 10

 HIGH AVAILABILITY CONFIGURATION AND STATUS
 -------------------------------------------

 LIST OF NODES CONFIGURED IN <ha-host1> CLUSTER
       phys-host1 phys-host2

 CURRENT MEMBERS OF THE CLUSTER

      phys-host1 is a cluster member

      phys-host2 is a cluster member

 CONFIGURATION STATE OF THE CLUSTER

      Configuration State on phys-host1: Stable

      Configuration State on phys-host2: Stable

 UPTIME OF NODES IN THE CLUSTER

      uptime of phys-host1:         12:47pm  up 12 day(s), 21:11,  1 user,
 load average: 0.21, 0.15, 0.14

      uptime of phys-host2:         12:46pm  up 12 day(s),  3:15,  3 users,
 load average: 0.40, 0.20, 0.16

 LOGICAL HOSTS MASTERED BY THE CLUSTER MEMBERS

 Logical Hosts Mastered on phys-host1:
         ha-host-1
 Loghost Hosts for which phys-host1 is Backup Node:
         ha-host2

 Logical Hosts Mastered on phys-host2:
         ha-host2
 Loghost Hosts for which phys-host2 is Backup Node:
         ha-host1

 LOGICAL HOSTS IN MAINTENANCE STATE

      None

 STATUS OF PRIVATE NETS IN THE CLUSTER

      Status of Interconnects on phys-host1:
         interconnect0: selected
         interconnect1: up
      Status of private nets on phys-host1:
         To phys-host1 - UP
         To phys-host2 - UP

      Status of Interconnects on phys-host2:
         interconnect0: selected
         interconnect1: up
      Status of private nets on phys-host2:
         To phys-host1 - UP
         To phys-host2 - UP

 STATUS OF PUBLIC NETS IN THE CLUSTER

 Status of Public Network On phys-host1:

 bkggrp  r_adp   status  fo_time live_adp
 nafo0   le0     OK      NEVER   le0

 Status of Public Network On phys-host2:

 bkggrp  r_adp   status  fo_time live_adp
 nafo0   le0     OK      NEVER   le0

 STATUS OF SERVICES RUNNING ON LOGICAL HOSTS IN THE CLUSTER

        Status Of Registered Data Services
        q:                       Off
        p:                       Off
        nfs:                     On
        oracle:                  On
        dns:                     On
        nshttp:                  Off
        nsldap:                  On

       Status Of Data Services Running On phys-host1
       Data Service HA-NFS:
       On Logical Host ha-host1:      Ok
     
       Status Of Data Services Running On phys-host2
       Data Service HA-NFS:
       On Logical Host ha-host2:      Ok
       
        Data Service "oracle":
        Database Status on phys-host2:
        SC22FILE - running;

        No Status Method for Data Service "dns"

        RECENT  ERROR MESSAGES FROM THE CLUSTER

        Recent Error Messages on phys-host1
        ...
        Recent Error Messages on phys-host2
        ...

2.1.2 Checking Message Files

The Sun Cluster software writes messages to the /var/adm/messages file, in addition to reporting messages to the console. The following is an example of the messages reported when a disk error occurs. 

...
 Jun 1 16:15:26 host1 unix: WARNING: /io-unit@f,e1200000/sbi@0.0/SUNW,pln@a0000000,741022/ssd@3,4(ssd49):  
 Jun 1 16:15:26 host1 unix: Error for command `write(I))' Err
 Jun 1 16:15:27 host1 unix: or Level: Fatal
 Jun 1 16:15:27 host1 unix: Requested Block 144004, Error Block: 715559
 Jun 1 16:15:27 host1 unix: Sense Key: Media Error
 Jun 1 16:15:27 host1 unix: Vendor `CONNER':
 Jun 1 16:15:27 host1 unix: ASC=0x10(ID CRC or ECC error),ASCQ=0x0,FRU=0x15
 ...
Note -

Because Solaris and Sun Cluster error messages are written to the /var/adm/messages file, the /var directory might become full. Refer to "4.7 Maintaining the /var File System" for the procedure to correct this problem.

2.1.3 Highly Available Data Service Utilities

In addition, Sun Cluster provides utilities for configuring and administering the highly available data services. The utilities are described in Appendix B, Sun Cluster Man Page Quick Reference. The utilities include:

2.2 Online Help System

Each Sun Cluster administration tool includes detailed online help. To access the online help, launch one of the administration tools from the administrative workstation and select Help from the menu bar.

Alternatively, double-click on the Help icon in the Cluster Control Panel.

Help topics cover the administration tools in detail, as well as some administration tasks. Also see Chapter 4, General Sun Cluster Administration, for additional detailed instructions on performing specific tasks.

Figure 2-1 shows a sample Help window for the Cluster Control Panel. The text covers a specific topic. When you first launch the Help window from a tool, it displays the top-level home topic. Afterwards, the Help window displays the last topic you viewed. Hypertext links to other topics are displayed as underlined, colored text.

Clicking once on a hypertext link displays the text for that topic. The online help system also includes an automatic history list feature that remembers the information you previously accessed. Display this list by choosing Topic History from the View menu.

The Help window has a scrollable text area, a menu bar, and several buttons. Each of these items is described in the following sections.

Figure 2-1 Sample Help Window Home Page for the Cluster Control Panel

Graphic

With Help, you can access each pull-down menu by:

You can customize the mnemonics and accelerators; refer to the online help for more information.

The tables in this section list each menu item, define the menu functions, and list the accelerators (keyboard combinations).

2.2.1 Help Window Menu Bar Items

The Help window menu contains the File, View, and Help menu items. You display the menus for these items by selecting them.

2.2.1.1 File Menu

The File menu contains these items:

Table 2-1 File Menu Items

Item 

Function 

Accelerator 

Print Topic 

Prints the topic currently displayed in the Help window scrolled text area. 

Alt + R 

Dismiss 

Dismisses the Help window. 

Alt + D 

2.2.1.2 View Menu

The View menu contains these items:

Table 2-2 View Menu Items

Item 

Function 

Accelerator 

Previous Topic 

Displays the previous help topic (if any). 

Alt + P 

Next Topic 

Displays the next help topic (if any). 

Alt + N 

Home Topic 

Displays the home (top level) topic. 

Alt + O 

Topic History... 

Displays the Help Topic History dialog box which allows you to navigate easily through the help topics you have already viewed. The uppermost topic in the scrolled list is the first topic you displayed. The bottommost topic is the last topic you have viewed in the current path. The highlighted topic is the current topic. 

Alt + I 

To display the dialog box, select View Topic History... (Figure 2-2).

Figure 2-2 Help Topic History of the Help Window

Graphic

2.2.1.3 Help Menu

The Help menu contains these items:

Table 2-3 Help Menu Items

Item 

Function 

Accelerator 

Help on Help... 

Describes the Help window and explains how to use it.  

Alt + E 

About... 

 

Displays the About Box, which contains information on the application, such as the version number. 

Alt + B 

2.2.2 Help Window Buttons

The following table lists the Help window buttons and describes their functions.

Table 2-4 Help Window Buttons

Button 

Function 

Home  

Displays the home page for the application. 

Dismiss 

Dismisses the Help window. 

Print Topic 

Prints the current topic on your default printer. 

Previous

Steps back through the displayed Help topics to the previous topic. Clicking on the left arrow repeatedly steps the display back through the Help windows until the first topic you viewed is redisplayed. Topics are "remembered" by the automatic Help history list. 

Next

Steps forward through the displayed Help topics, one at a time, to the last topic in the list. 

2.3 About the Cluster Control Panel

The Cluster Control Panel (CCP) is a GUI that enables you to launch the Cluster Console, and other system administration tools. The CCP contains icons that represent these tools.

2.3.1 How to Launch the CCP

After you have installed the Sun Cluster client software on the administrative workstation, use this procedure to run an application from the CCP.

  1. As superuser, add the Sun Cluster tools directory /opt/SUNWcluster/bin to the path on the administrative workstation.

    Note -

    For E10000 Platforms, you must first log into the System Service Processor (SSP) and connect by using the netcon command. Once connected, enter Shift~@ to unlock the console and gain write access. Then proceed to Step 2.

  2. In a shell window on your workstation, bring up the CCP.

    Specify the name of the cluster to be monitored: 

    # ccp clustername
    Note -

    If the Sun Cluster tools are not installed in the default location of /opt/SUNWcluster, the environment variable $CLUSTER_HOME must be set to the alternate location.

2.3.2 CCP Items

The CCP (shown in the following figure) has a menu bar and an icon pane that displays all of the tools currently in the control panel. From the menu bar you can add, delete, or modify the tools.

Figure 2-3 Sample Cluster Control Panel

Graphic

From the File or Properties menu you can:

For detailed information about the CCP, refer to the online help.

For information about the programs represented by these tools and their usage, see "2.4 About the Cluster Console". For information about using the HotJava browser to monitor cluster configurations, see "2.5 Monitoring Sun Cluster Servers With Sun Cluster Manager ".

2.3.3 CCP Configuration File Locations

The CCP stores properties and other information in configuration files within a configuration directory. By default, the configuration directory is located at /opt/SUNWcluster/etc/ccp.

Note -

You must be root (superuser) to write to this default location. Only root can add, delete, or change CCP items in this configuration directory.

You can, however, create your own configuration directory and define its location using the environment variable $CCP_CONFIG_DIR. The $CCP_CONFIG_DIR variable specifies the configuration directory in which the configuration files containing item properties are stored. If the path name is not set, it defaults to the standard location, /opt/SUNWcluster/etc/ccp. To create your configuration directory, create a new directory and set the environment variable $CCP_CONFIG_DIR to the full path name of the new directory.

These files do not need to be edited manually, because they are created, modified, or deleted by ccp whenever you create, modify, or delete an item.

2.4 About the Cluster Console

The Cluster Console (CC) GUI enables you to run commands on multiple nodes simultaneously, simplifying cluster administration. The Cluster Console displays one terminal window for each cluster node, plus a small Common window that you can use to control all windows simultaneously.

Different types of remote sessions enable you to connect to the console of the host, or remotely log in by using rlogin or telnet. Hosts can be specified on the command line and added or deleted from the Select Hosts dialog box after the program is running. The session type can be specified only on the command line. Once started, the session type cannot be changed.

You can issue commands to multiple hosts from the Common window, and you can issue commands to a single host from a terminal window. Terminal windows use VT100 terminal emulation.

Alternatively, you can turn off all hosts in the Hosts menu except the one you want to access, then issue commands from the Common window text field.

2.4.1 How to Launch the Cluster Console

You can launch the Cluster Console from the CCP (see "2.3 About the Cluster Control Panel") or from the command line in a shell window. If an optional parameter is specified, a terminal window is created for each host in the cluster or for each host specified.

    Type cconsole to initiate remote console access: 

% cconsole [clustername | hostname...]

    Type ctelnet to establish a telnet(1) connection from the console: 

% ctelnet [clustername | hostname...]

    Launch crlogin with your user name to establish an rlogin(1) connection from the console: 

% crlogin -l user name [clustername | hostname...]

All three of the preceding commands also take the standard X/Motif command-line arguments. Once the Cluster Console has started, the Console window is displayed.

For detailed information on the Cluster Console, refer to the online help.

2.4.2 The Common Window Menu Bar

The Common window (shown in the following figure) is the primary window used to send input to all nodes. The Common window is always displayed when you launch the Cluster Console.

Figure 2-4 Common Window Menu Bar of the Cluster Console

Graphic

This window has a menu bar with three menus and a text field for command entry. From the Hosts menu you can use the Select dialog box to:

From the Options menu, you can group or ungroup the Common Window and the terminal windows.

2.4.3 Configuration Files Used by the Cluster Console

Two configuration files are used by the Cluster Console: clusters and serialports. These can be /etc files or NIS/NIS+ databases. The advantage to using a NIS+ environment is that you can run the Cluster Console on multiple Administrative Workstations. Refer to your NIS/NIS+ System Administration Guide for complete information on NIS/NIS+.

2.4.4 About the clusters File

The clusters file maps a cluster name to the list of host names that comprise the cluster. Each line in the file specifies a cluster, as in this example: 

planets      mercury venus earth mars
 wine         zinfandel merlot chardonnay riesling

The clusters file is used by all three session types of the Cluster Console (cconsole, ctelnet, and crlogin) to map cluster names to host names on the command line and in the Select Hosts dialog box. For additional information, see "3.10 Modifying the clusters File".

2.4.5 About the serialports File

The serialports file maps a host name to the Terminal Concentrator and Terminal Concentrator serial port to which the host is connected. Each line in this database specifies a serial port of the host.

Sample serialports file database entries for the Sun Enterprise 10000 are:

mercury    systemserviceprocessorname    23
 venus      systemserviceprocessorname    23
 earth      systemserviceprocessorname    23
 mars       systemserviceprocessorname    23

Sample serialports file database entries for all other nodes are:

mercury        planets-tc   5002
 venus          planets-tc   5003
 earth          planets-tc   5004
 mars           planets-tc   5005

The serialports file is used only by the cconsole variation of this program to determine which Terminal Concentrator and port to connect to for hosts or clusters that are specified in the command line or the Select Hosts dialog box.

In the preceding example, node mercury is connected to planets-tc Port 2, while node venus is connected to planets-tc Port 3. Port 1 is reserved for the administration of the Terminal Concentrator.

For additional information, see "3.11 Modifying the serialports File".

2.5 Monitoring Sun Cluster Servers With Sun Cluster Manager

Sun Cluster Manager (SCM) provides a single interface to many of Sun Cluster's command line monitoring features. SCM consists of two parts: SCM server software, and the SCM Graphical User Interface (GUI). The SCM server runs on each node in the cluster. The SCM GUI runs in a Java Development Kit (JDK) compliant browser such as HotJava. The HotJava browser can be running on any machine, including the cluster nodes. The SCM GUI reports information on:

2.6 Running the SCM GUI in a HotJava Browser

The following set of procedures outline what you need to do to run SCM in a HotJava browser.

You may need to determine if you have the correct version of the following:

Note -

If you choose to use the HotJava browser shipped with your Solaris 2.6 or 2.7 Operating System, there may be problem when using the menus, for example: after making your menu selection, the menu selection can remain visible on the browser. Refer to the Sun Cluster 2.2 Release Notes for complete information based on your software requirement needs.

Refer to the Sun Cluster 2.2 Release Notes for complete information on software requirements.

You will need to determine whether you want to:

Depending on what you decide, refer to the appropriate procedure.

2.6.1 How to Determine the JDK Version

    Type the following from the console prompt on the server in your cluster:

java -version

Refer to the Sun Cluster 2.2 Release Notes for complete information on software requirements.

2.6.2 How to Determine the HotJava Version

    From the machine running the HotJava browser, select About HotJava from the Help menu.

Refer to the Sun Cluster 2.2 Release Notes for complete information on software requirements.

2.6.3 How to Run the SCM Applet in a HotJava Browser From a Cluster Node

  1. Run your HotJava browser on a node in the cluster.

  2. Remotely display it on an X windows workstation.

  3. Set the applet security preferences in the HotJava browser:

  4. Choose Applet Security from Preferences on the Edit menu.

  5. Click Medium Security as the Default setting for Unsigned applets

  6. When you are ready to begin monitoring the cluster with SCM, type the appropriate URL.

    file:/opt/SUNWcluster/scmgr/index.html

  7. Click OK to dialog boxes that ask for permission to access certain files, ports, and so forth from the remote display workstation to the cluster node where the browser is started.

    Note -

    It may take HotJava some time to download and run the applet. No status information will appear during this time.

    Refer to the online help for complete information on menu navigation, tasks, and reference.

2.6.4 How to Set Up a Web Server to Run With SCM

If you choose, you can install a web server on the cluster nodes to run with SCM.

  1. Install a web server that supports HTML 2.0 or later on all nodes in the cluster.

    One option is Sun Web Server, which is available to download at http://www.sun.com/solaris/webserver.

    Note -

    If you are running the HA-HTTP service and an HTTP server on SCM, you need to configure the HTTP servers to listen on different ports. Otherwise there will be a port conflict between the two.

  2. Follow the web server's configuration procedure to make sure that SCM's index.html file is accessible to the clients.

    The client applet for SCM is in the index.html file in the /opt/SUNWcluster/scmgr directory.

    For example, go to your HTTP server's document root and create a link to the /opt/SUNWcluster/scmgr directory.

  3. Run your HotJava browser from your workstation.

  4. Set the applet security preferences in the HotJava browser:

    1. Choose Applet Security from Preferences on the Edit menu.

    2. Click Medium Security as the Default setting for Unsigned applets

  5. When you are ready to begin monitoring the cluster with SCM, type the appropriate URL.

    http://cluster node/scmgr/index.html

  6. Click OK to dialog boxes that ask for permission to access certain files, ports, and so forth to the cluster node where the browser is started.

    Note -

    It may take HotJava some time to download and run the applet. No status information will appear during this time

    Refer to the online help for complete information on menu navigation, tasks, and reference.

2.7 SCM Online Help System

2.7.1 How to Display SCM Online Help

    To display the Help window from SCM, select Help Help Contents

Alternatively, click on the Help Icon (question mark) in the tool bar on top of the folder.

If necessary, you can run online help in a separate browser by typing the following URL:

file:/opt/SUNWcluster/scmgr/help/locale/en/main.howtotopics.html