Chapter 1 -

Product Description

This chapter provides an overview of the Configuration and Service Tracker (CST) software for version CST 3.0. It includes the following sections:


Introduction

CST plays an important role in the overall strategy of proactive system management at a customer site. The software executing on each monitored host provides a macroscopic view of system configuration changes that are caused by failures or upgrades and service patterns over an extended period.

CST 3.0 includes a configuration for automatic emails to be sent to Sun Microsystems. The emails contain captured tracking data, which is used by Sun to track system availability and performance and to continually improve Sun products and services. Availability Management System (AMS) is a secure database where the captured availability data is stored in Sun Microsystems.

Some benefits are that CST:

CST 3.0 can be deployed either within Sun Management Center envinroment or in an independant framework


CST Features

The CST application tracks and record event and configuration data to be viewed and reported upon through the CST user interface.

Tracking and Recording Events

CST automatically detects the following events for the system on which it runs:

On Sun multi-domain systems, CST also detects the following events:

Tracking Configuration

When an event is detected, CST captures a snapshot of the current system configuration information and categorizes it into the following categories:

CST determines and records the changes in the configuration of the system by comparing the new snapshot with the previous snapshot.

Collecting Hardware FRU Information

In UltraSPARC® III platforms, CST collects more detailed hardware information about the Field Replaceable Units (FRUs) in the system. This hardware information is in addition to the data collected by the hardware probe common to all Sun systems. The additional FRU data is viewable as a sub-category in the Installed Hardware section of the Configurations folder. This is considered part of Hardware Information and changes in FRU do not generate separate events.

For the Sun Fire 3800, 4800, 4810, and 6800 systems, CST collects FRU information remotely from the CST middleware server host.

Viewing Events and Configuration Information

The CST user interface is also used interactively to enter service comments and cause codes for each event.


Benefits of CST for System Administration

CST provides a number of benefits for system administration, including:


CST Architecture

CST 3.0 has the following components:

CST server runs on the designated server system. The server receives data from all the reporting agents and stores it in the repository. The terms CST server and CST middleware mean the same thing, and they are used interchangeably in this document.
CST agent is the CST daemon running on the system tracked by the CST agent (aka host), and, when attached to a CST server, it reports all collected information to its CST server.
The information from the agents is stored in the CST repository. This is stored as a fixed format (ASCII) file in a file system on the CST server.
The CST console is a Java-based user interface that can run either as a Java applet in a browser or as a Java bean in the Sun Management Center console.
The functionality of both the user interfaces is similar. The user interface allows a user to view and manage the CST collected events.
The CST service agent daemon runs only if the environment is Sun Management Center based. The service agent transfers data between the CST server and the CST console in the Sun Management Center environment.

 

FIGURE 1-1 Architecture of CST 3.0

A graphical representation of the various CST components.

CST Server

The CST server is a multi-threaded daemon running on the designated server system. It manages agent data reported by the various 'attached' CST agents and communicates with the UI to display the collected data.

The server maintains all the agent data in a system. Data from the agents is maintained in unique folders within the file system for easy lookup and maintenance.

In the Sun Management Center environment, the CST server communicates to the CST console via the CST service agent.

In addition, the CST server has the agent functionality built-in, and it collects and tracks the configuration changes occuring on the system. Hence it is not required to install the CST agent on the CST server system

When registered, the CST server also sends the complete agent information back to Sun either via the Net Connect 3.0 secure pipe or via secure email if Net Connect 3.0 is unavailable.

 

FIGURE 1-2 CST Communication with SunMC

Graphical representation showing how CST works in the Sun Management Center Environment.

CST Agent

CST Agent is a daemon that runs on each of the systems being tracked. The daemon detects whenever a change occurs and creates an event according to the change.

CST Agent can be attached or unattached. An unattached agent is a self-monitoring unit that collects data but does not report the data to any CST server. The user interface is not available for unattached agents.

Once a CST utility, either cstattach or cstattagt, is used to make the connection, the CST agent starts reporting to a middleware, and the GUI is updated with the agent information. From that point forward, the agent data can be viewed using the GUI. See Chapter 6, Utilities , cstattach, and cstattagt, for details on the utilities.

When the CST agent is not attached, the following are available:

  • Event information and configuration changes in the History file:
/var/opt/SUNWcst/cst_history
  • Current configuration in:
/var/opt/SUNWcst/probe.current
  • Notification to the email aliases specified in the Preferences file

All information collected by the daemon is also sent to the CST server on the middleware system for automatic backup. If the server system or its software cannot be reached, the data is backed up and transmitted again when the connection is reestablished.

The CST "heartbeat" process ensures that a CST daemon is running and ready to collect data at all times.

CST Repository

The repository resides in the server system. It holds the data that CST collects from the tracked systems. The CST data is kept in a hierarchical directory structure, where each directory represents a specific system. The key data files in each directory are outlined in TABLE 1-1.

 

TABLE 1-1 CST Repository Data Files

File Name

File Type

Description

cst_history

ASCII

Holds the event history, configuration changes, and comments for each event.

probe.current

ASCII

Holds the current configuration information.

probe.inception

ASCII

Holds the configuration snapshot information at the time CST was installed for the first time.

user_data

ASCII

Holds user-entered custom information.

cst.pref

ASCII

Holds Preferences information.

cause_code

Binary encoded

Holds the event cause codes for each event.

selective-event

ASCII

Holds local notification settings.

fruconfig.current

ASCII

Holds the current FRU configuration information.

fruconfig.inception

ASCII

Holds the FRU configuration information from the inception of CST.

volconfig.current

ASCII

Holds the current Veritas Volume Manager configuration information.

volconfig.inception

ASCII

Holds the Veritas Volume Manager configuration information from the inception of CST.

sc30config.current

ASCII

Holds current Sun Cluster 3.0 configuration information.

sc3.1config.current

ASCII

Holds current Sun Cluster 3.1 configuration information.




caution icon

Caution - Do not edit these files manually. They are maintained by the CST software. Any edits will corrupt the file and disable software functionality.



The directories in the repository are created automatically by the server, based on the hierarchy information provided during the install of the CST agent, or at the time of "attach."

CST User Interface

The CST user interface is an applet when the CST installation is stand-alone based. In an environment where SunMC is installed, you can invoke the CST console via the SunMC soncole, Tool menu.

CST Applet

This is an applet that can be viewed using a browser, such as Netscape. It can be invoked from the URL:

http://<middleware-name>/cst.html

The hierarchy information is available in Enterprise View within the CST applet.

Both user interfaces provide the user with the collected information, history log, and configuration. Each includes printing reports, managing events, and retrieving statistics.

CST Interface to Sun Management Center

The CST service agent is a customized agent on the Sun Management Center server that provides two-way data transfer between the CST console and the CST server.

The CST service agent is bundled with the Sun Management Center server software, and activates automatically when a CST server (SUNWcstu) is also installed on that system.



Note - When you are running CST 3.0 with Sun Management Center, you open CST through the Tool menu. The Sun Management Center console is the user interface.



CST Console in Sun Management Center Environment

This is an application that runs from within the Sun Management Center console. The CST UI can be invoked from the tools menu in the Sun Management Center console. The CST console is a pop-up frame from the Sun Management Center console, and the hierarchy information is available on the Sun Management Center window.


Supported Systems

CST 3.0 supports all current Sun 4d- and 4u-based server platforms, Sun4u desktops, including the Sun Enterprisetrademark 10000, and Sun Firetrademark platforms, both mid-range and high-end models. CST 3.0 software supports Sun Cluster 3.0 and 3.1 environments.

CST 3.0 runs in Solaris 2.6, 7, and 8 and 9 operating environments.

CST 3.0 does not support Solaris versions 2.4 and 2.5.1. CST versions 1.5.1 and 2.1U1 - both AMS and non-AMS releases, run on Solaris 2.5.1 and can report into a CST 3.0 server.

To view CST data through an applet requires a browser, such as Netscape, that supports JDK 1.2 or later.


CST on Multi-Domain Systems

CST 3.0 can be installed on and can track events on all Solaris/Sparc platforms, including Sparc III versions. The following sections describe CST support for the multi-domain Sun platforms.

CST on Sun Enterprise 10000

Since Sun Enterprise 10000trademark servers can have multiple domains and a System Service Processor (SSP) to manage the platform, multiple copies of CST must be run on this platform. One instance is installed and run on each of the domains. Also, a separate CST agent must be installed on the SSP.



Note - It is important that the CST agent is installed on the SSP. It should be installed on the SSP before installing on the domains.



CST on the SSP tracks the following events:

  • System configuration changes
  • Board power-on
  • Domain changes
    • Domain(s) added
    • Domain(s) deleted
    • Hardware changes across domains

CST on the domain tracks the following events on that domain:

  • Domain up or down
  • Hardware/software changes

CST on Sun Fire Platforms

CST 3.0 is designed to support all Sun Firetrademark platforms:

  • High-end systems: Sun Firetrademark 15K, 12K
  • Midrange systems: Sun Firetrademark 3800, 4800, 4810, 6800

Sun Fire 15K and 12K

CST can track platform information from the system controllers on a Sun Enterprise 15K system, also known as a Sun Fire 15K and 12K.

The CST agent must be installed on the system controllers, and it must run on each of the domains. If the system has a spare system controller, the CST agent must also be installed on the spare system controller.

CST 3.0 supports creation of service events and annotation of event comments for system controllers and domains, but it does not support creation of service events or updates at the platform level.

The CST 3.0 application on the system controllers tracks the following events:

  • Domain changes
  • System configuration changes
  • Platform configuration changes
  • Other CST events

Sun Fire 3800, 4800, 4810, 6800

CST can track platform information for all Sun Fire systems.



Note - Since the system controllers on the SunFire midrange x8x0 systems are not Solaris Operating Environments, CST cannot be installed directly on these system controllers. Therefore, the monitoring agent for these system controllers have to be installed on the middleware, as part of the DST server, and manually configured to provide the platform tracking capability.



To facilitate manual configuration described in the above note, use the CST utility, addsunfire. This utility helps configure the remote monitoring agent. For details on this utility, see Chapter 6, Utilities, , addsunfire.

The CST agent is run on the Sun Fire midrange domains just as it is on other server systems.


CST on Clusters

When installed on Suntrademark Cluster 3.0 or Suntrademark Cluster 3.1 Cluster nodes, CST 3.0 tracks and provides a continuos record of the cluster membership and configuration changes. The data is stored in the repository as a formatted file and is available for viewing from the CST console. The information is viewable through a tabbed panel in the Configurations section of the CST user interface.

 

FIGURE 1-3 CST on Sun Clusters

Screen shot of tracked Sun Cluster events for the selected monitored system. The GUI selections are: Configurations/Current/Cluster/Summary.

The information is further sub-categorized into the following:

  • Summary Information
  • Membership Status
  • Resource Group Information
  • Resource List
  • Resource Information
  • Transport Path
  • Device Group Information
  • Quorum Votes

 

FIGURE 1-4 CST on Sun Cluster

History folder listing 7 events plus text in Event Cause Code and User Comments.

CST also detects various cluster failover events and records them in the CST history file. The events are annotated with additional relevant details such as cluster ID, affected resource group/node, failover status etc. These are available for viewing through the events table and the events comments section in the CST history folder.

The following cluster events are tracked.

 

TABLE 1-2 Sun Cluster Events Tracked

Event Tracked

Cluster Version

Cluster node joined

SC3.0/3.1

Cluster node left

SC3.0/3.1

Resource group online

SC3.0/3.1

Resource group offline

SC3.0/3.1

Resource group state change

SC3.1

Cluster membership change

SC3.1

Cluster failover

SC3.1

HA-Oracle server started

SC3.0/3.1

HA-Oracle server stopped

SC3.0/3.1

HA-Oracle listener started

SC3.0/3.1

HA-Oracle listener stopped

SC3.0/3.1

HA-Sybase adaptive server started

SC3.0/3.1

HA-Sybase adaptive server stopped

SC3.0/3.1

HA-Sybase backup sever started

SC3.0/3.1

HA-Sybase backup sever stopped

SC3.0/3.1




Note - The resource group, online and offline events, can be used as an indication of each failover. This depends on the resource group type which can be either Failover or Scalable/Load balancing resource group. In Sun Cluster 3.1, the event named "Cluster failover" is additional and is provided to give more detail on the reason of each failover.



 

FIGURE 1-5 CST on Sun Cluster

Configuration folder, Current radio button, Cluster with Summary selected. [ D ]

Cluster Platform Hierarchy

The "CLUSTER_PLATFORM" hierarchy is created on the server, and it is viewable from the CST user interface. With this feature, users can do the following:

  • See the list of all clusters in the enterprise
Each cluster has a hierarchy inside "CLUSTER_PLATFORM." The CST cluster ID hierarchy is created for each cluster, and the hierarchy name is in the following format:
CLUSTER_NAME,CLUSTER_ID,CLUSTER_VERSION
  • See the list of nodes inside each "CST Cluster ID" hierarchy
A soft link for each node is created under the hierarchy for easy viewing of the cluster member. Clicking the nodename displays generic cluster-node information.


Note - In the CST Enterprise View, each cluster node appears inside the CLUSTER_PLATFORM hierarchy and the user-specified hierarchy, which is specified during CST agent installation. Clicking the same node in either hierarchy yields the same effect.



  • The CST Cluster ID hierarchy is clickable and contains the following information:
    • Global view of events received from all cluster members where duplicated events across the nodes are filtered
    • "Last configuration snapshot" in the Configuration Cluster folder to indicate the most recent view of the cluster


Note - Events in the global CST history under "CST Cluster ID" hierarchy are logged and sorted based on the time when the CST server receives each event and not necessarily on the event timestamp.





Note - The "Summary" section of Cluster configuration for "CST Cluster ID" hierarchy reports the name of the last node with a configuration change.




CST on Volume Manager Systems

Volume Manager is viewable on a tabbed pane in the Configurations section of CST.

 

FIGURE 1-6 Volume Manager Configuration.

Configuration folder, Current radio button, Volume Manager with Disk Media selected.[ D ]

The information is further sub-categorized into the following:

  • Disk Media
  • Disk groups
  • Volumes
  • Plexes
  • Sub Disks


Note - Volume Manager events and differences are included on the History folder.



CST tracks the standard events as listed in Tracking and Recording Events.


Custom Event Application Programming Interface (API)

If there are applications on the system that are started or stopped through UNIX shell, awk, or perl scripts, the configuration changes, and start and stop activity on those applications can be monitored through CST. To manually register a custom event related to those applications, include the following line at the end of the application Start/Stop script:

/opt/SUNWcst/bin/app_event "event type" "comment"

The app_event only relays the event. It does not validate if the application successfully started or stopped.

For example, the following line adds the machine password change as a CST event to the CST History log:

/opt/SUNWcst/bin/app_event "password changed" "By machine owner"

Each time this line of code is included, CST registers the event in the CST History section of the CST Graphical User Interface (GUI) (See Chapter 4, Features and Functions,, FIGURE 4-10).

To track such password changes automatically, add the following script to your crontab file, and run it at your convenient time and frequency, for example, every hour or every day.

     #!/bin/sh
     # This script is used for creating a cst application event of
     # event type string "Password Change" by calling cst command 
     # app_event. You need to add this script to your crontab file.
     
     # Get the time info of the file: /etc/passwd
     if test -f /tmp/cst_app_passwd
     then
        ls -l /etc/passwd|awk '{print $6, $7, $8}' > /tmp/cst_app_passwd_new
    
     # Compare the time stamp of the passwd file, to see if there 
     # any difference.
        diff /tmp/cst_app_passwd /tmp/cst_app_passwd_new >/tmp/cst_result
        result=`ls -l /tmp/cst_result|awk '{print $5}'`
        if test $result = 0
        then
            echo "The password is not changed" >/tmp/cstpasswd
            rm /tmp/cst_app_passwd_new
            rm /tmp/cst_result
        else
            cp /tmp/cst_app_passwd_new /tmp/cst_app_passwd
            rm /tmp/cst_app_passwd_new
            # Call cst_app_event to register event
            /opt/SUNWcst/bin/app_event "Password Changed" "Changed by owner"
 
    fi
      fi

# End of the script


Data Feed to Sun

When the data feed to Sun is enabled, CST sends all tracked data to Sun in XML format through the secure Net Connect, Proxy if it is installed on the server where CST server exists. If the Proxy is absent, the data is sent as a secure email.

To send the data by Net Connect Proxy, Net Connect 3.0 must be installed on the CST server. If this is installed when the CST server is already running, the CST server must be stopped. Using the following command. Type:

# /opt/SUNWcstu/sbin/cstadm stop

Restart using the command:

#/opt/SUNWcstu/sbin/cstadm start

To send the data by secure email, the CST server must have "sendmail" installed.

The data is sent asynchronously, i.e. sent when a monitored host reports one or more of the following events:

  • a new event
  • a configuration change
  • a cause code assigned to an event
  • a comment provided fro an event
  • changes made to system-specific custom information

The data content is usually less than 1KB in size except when large configuration changes are involved.

Periodically CST sends summary information about the installed site to Sun. The summary data is used to checkpoint system status to provide accurate availability reporting.

If the continous data feed to Sun is not enabled, a skeleton summary of the install base is still sent to a secure site within Sun, once a week. This summary information is used to help generate statistics on CST deployment. For more details or to turn this ability onor off, contact your Sun representative.