An alarm is a notification that is triggered by an abnormal event, such as exceeding a certain percentage of memory usage. The Alarm Manager software monitors your hardware and software and triggers alarms when defined conditions occur.
This chapter describes the following topics:
The messages in the Alarms page of the Details window are always in English. These messages are not translated into other languages. However, the text in all dialog boxes and in suggested fixes is internationalized.
The Alarm Manager software displays alarm information for managed objects. You can view object alarm information in an administrative domain in the main console window and in the Details Alarm window.
The Sun Management Center agent is configured so that only one server receives alarm information from that agent.
The Sun Management Center 3.6.1 Alarm Manager enables you to do the following tasks:
View alarms in pages from a database
Manually run the currently registered action after an alarm has been triggered
Set and change the currently registered action from a list of all installed actions
Sort alarms
Read the factory default suggested fix for a rule
Create a new user-suggested fix for a rule
Keep a running record of user notes for an alarm instance
Acknowledge alarms when they occur
Delete closed alarms from the database
An alarm is a notification that is triggered by an abnormal event. Sun Management Center has two types of alarms:
Predefined alarm conditions included in the software modules, such as notification when the CPU usage exceeds a certain percentage. These alarms are triggered by conditions outside of a preset range or by Sun Management Center rules. Default alarm conditions and rules are included in the modules. For some predefined alarms, you can change the threshold at which the alarm is triggered. In addition, you can modify the action to take when the alarm occurs and add information to the suggested fixes. For a list of Sun Management Center rules, see Appendix D, Sun Management Center Software Rules.
User-defined alarm conditions. You define what causes an alarm to occur, what action to take, and a suggested fix, if desired.
The Alarm Manager software uses several different methods to alert you to an unacknowledged open alarm condition:
Colored icons in the Domain Status Summary on the main console
Colored icons in the hierarchy (tree) view
Colored icons in the topology (contents) view
Colored relevant row or relevant column in the property table (contents view)
The type and color of alarm icon identify the severity of the alarm. For example, a red alarm icon indicates that a critical condition has developed and that corrective action is required immediately. By contrast, a blue alarm icon indicates a potential or an impending service-affecting fault.
You can acknowledge, delete, and manage the object alarms by using the Alarms Details window. For more information, see Managing and Controlling Alarms.
Figure 12–1 shows an unacknowledged, open critical alarm in the Swap Statistics properties table Used KB row. The row is red, which indicates a critical alarm. The alarm information propagates up the hierarchy tree view, from the individual module up to the host. You would also see a red alarm icon on the following objects:
Swap Statistics properties table
Kernel Reader module
Operating System
Host
You also see a red alarm icon on the corresponding host, group (if any), or administrative domain in the main console window. The only exception would occur if an unacknowledged open black alarm of higher severity exists.
You can select Critical Alarms, Alert Alarms, and Caution Alarms simultaneously for a module table. A tick mark appears when these alarms are selected.
Unacknowledged alarms take precedence over acknowledged alarms. If the hierarchy has two or more types of alarms, the color of the more severe unacknowledged alarm is propagated up the tree. For example, if there is a yellow unacknowledged alarm in CPU usage and a red unacknowledged alarm in Disk Statistics, only the red alarm icon propagates. However, if there is a yellow unacknowledged alarm in CPU usage and a red acknowledged alarm in Disk Statistics, only the yellow alarm icon propagates.
The following alarm severities are supported:
A down alarm indicates
that a service-affecting condition has occurred that requires immediate corrective
action. An example of this condition is the following: A required resource that is
defined by a managed object has gone out of service. A specific example of this condition
is a module that has gone down.
A critical alarm indicates
that a service-affecting condition has developed that requires an urgent corrective
action. An example of this condition occurs when a severe degradation in the capability
of an object has occurred and you need to restore the object to full capability.
An alert alarm indicates that a non-service-affecting condition has developed.
Corrective action should be taken to prevent a more serious fault.
A caution alarm indicates the detection of a potential or impending service-affecting
fault before any significant effects have occurred. You should diagnose further if
necessary and correct the problem before it becomes a serious service-affecting fault.
A disabled alarm indicates
that a resource for a managed object is disabled. For example, a module is disabled.
Objects with black star icons are objects with indeterminate states, not to be confused with alarms. A black star or “splat” icon in the main console window means that a data acquisition failure occurred in that object. The failure is not the result of a rule infraction, so no alarm is associated with the failure.
When you view the data property table for an object, a pink row also indicates an indeterminate object state.
The Domain Status Summary section of the main console window provides a brief view of managed object status. The colored icons designate the severity of the alarms.
To see a definition for a status summary icon, place your cursor over the icon.
Numbers next to the alarm icons in the Domain Status Summary indicate the managed
objects for which the highest severity open, unacknowledged alarm is represented.
For example, a 1 next to the alert alarm icon (center) indicates that there is one managed
object for which the highest severity alarm is alert.
The Domain Status Summary displays the number of managed objects in the administrative domain that have at least one unacknowledged open alarm of a specific severity.
If multiple types of alarms exist in the host, the icon that indicates the more severe unacknowledged open alarm appears in the Domain Status Summary.
If the most severe alarm on one host is critical and
the most severe alarm on another host is alert
, a 1 appears on both alarm icons.
The alarms table contains a statistical summary of all alarm data for the managed object that you select.
If the object is a platform, see your platform supplement for more information.
You can filter and sort this table to display only the alarms that you want to see in a preferred sequence. You filter and sort alarms from the alarms table. For additional details, see To Filter the Alarms Table and To Sort the Alarms Table.
The maximum number of alarms that can be displayed on a page is 20. The current page number and total number of alarms for the selected object in the database are displayed at the top of the table.
When new alarms occur, the currently displayed alarms table does not change, even if these alarms affect the current page. Instead, the Refresh button displays a two-state icon. This icon indicates that new alarms have arrived, and you should update the table to include the new alarms soon.
When you delete alarms, the table is updated immediately. The deleted alarms no longer appear in the table. When another user deletes alarms, you might see blank rows in the alarms table. A refresh request recomputes the pages and updates the table to remove the deleted alarms. Only one page of alarms is displayed per request. You can see additional pages by using the page navigation buttons above the alarms table.
The Alarms Details window contains navigation buttons that enable you to move through the alarm pages. Informational messages inform you when the first and last pages of the table are displayed. A scrollbar enables you to scroll through each page of the table.
The alarms table presents different categories of detailed alarm information. The following information always appears in the alarms table.
Indicates the severity of the alarm. Black is most severe. Grey is least severe. A green check mark in this column indicates that the alarm was acknowledged.
Date and time that the alarm occurred
Indicates the state of the alarm: open, a “ringing” bell icon, or closed, a “silent”bell icon
Indicates the action taken by the user or the program in response to the alarm condition
Abbreviated message that describes the alarm
The following information appears on the bottom of the page when an alarm row is selected. This information is only displayed for closed or acknowledged alarms.
Date and time that the alarm condition was fixed
Date and time that the alarm was acknowledged, and the user ID of the person who acknowledged the alarm
Select an alarm row to display any available additional information that is associated with the alarm. The additional information includes the following data:
The alarm end time
The acknowledgment date and time
The user ID of the user who acknowledged the alarm
A bell icon in the State column of the alarms table indicates the state of each alarm. Each alarm has two states: open and closed.
An open alarm is an alarm for which the condition that caused the alarm still exists. A closed alarm means the condition no longer exists. Open alarm icons show a “ringing” bell. Closed alarm icons show a “silent” bell.
The Action column of the alarms table indicates the status of each alarm.
Each alarm can have one of three action conditions:
No action – An action has not been registered for that alarm.
Pending action – The action is manual. Click the Run button to execute the action.
Executed – The action is automatic. The Alarm Manager software has already performed the action.
The Alarm Manager software displays alarm information for managed objects. You can view object alarm information in an administrative domain in the main console and the Details Alarm windows.
When an alarm condition occurs, a colored alarm indicator appears in the console window. For more information about the alarm indicators, see Alarm Severity Levels.
To view more information about an alarm:
To see a summary of current alarms, click one of the Domain Status Summary buttons in the main console window.
A list of objects appears in the Domain Status Details window. This list includes those objects that have at least one open, unacknowledged alarm for which the highest severity matches the icon on the button. For more information about these icons, see Alarm Severity Levels.
From the Domain Status Details window, you can do the following tasks:
Change the sort order of the displayed alarms, as described in To Sort the Domain Status Alarms.
Access detailed information about a specific alarm, as described in Step 2.
Update the Domain Status Details window by clicking the Refresh Now button.
To view details about a specific alarm, use one of the following methods:
Double-click the desired row in the Domain Status alarm table.
Click the desired row to select the row, and then click the Details button.
The Details window for the applicable managed object appears with the Alarms tab selected and alarm information displayed. From this window, you can do the following tasks:
Acknowledge the alarm, as described in To Acknowledge New Alarms.
View and modify the suggested fix for the alarm, as explained in To View and Add Suggested Fixes.
View and add user notes about the alarm, as described in To Add an Alarm Note.
Modify a defined alarm action, as explained in To Modify Pending Alarm Actions.
Delete the alarm, as described in To Delete Alarms.
When an alarm icon appears next to a managed object icon in the main console window, you can view the alarms for that specific object.
Double-click the managed object icon.
The Details window for that managed object appears with the Module Browser tab selected.
Click the Alarms tab.
The Details window changes to show all open alarms for the current managed object.
Press mouse button 3 on a host icon in the main console window, and choose Details from the pop-up menu.
The Details window appears.
Click the Alarms tab.
The Alarms Details window appears.
A bold header identifies the column on which the alarms table is sorted. A down or up arrow indicates the order on which the column is sorted. For example, the alarms table shown in Figure 12–2 is sorted in descending order (newest to oldest alarm) based on the start date and time. This order is the default sort order for the table.
For information about changing the sort order, see To Sort the Alarms Table.
By default, alarms in the Domain Status Details window are sorted from newest to oldest.
In the Domain Status Details window, click the Sort button.
The Sort Options window appears.
Determine whether to sort the alarms alphabetically or by time received:
To sort alphabetically by status message, click Status Message (alphabetically).
To sort the alarms by time received, newest to oldest, click Time (newest to oldest).
Click the OK button.
The Domain Status Details window updates with any order changes as appropriate.
Double-click any column header in the alarms table.
If the alarms table is currently sorted in ascending order, the table is redisplayed immediately in descending order, based on the column that you select.
If the alarms table is currently sorted in descending order, the table is redisplayed immediately in ascending order, based on the column that you select.
If the table contains many alarms, changing the sort order might take several seconds.
The column headers have an down or up arrow to the right of the header. These arrows indicate the order in which the table is sorted, descending (down arrow) or ascending (up arrow). The arrow indicator and the selected column header are boldface to identify current sort order. The following list identifies the default sort order for each column heading in the alarm table.
Alarms are sorted from highest severity (down) to lowest severity (off or disabled).
Alarms are sorted from newest to oldest.
Alarms are sorted as follows:
Alarms that have completed and executed actions
Alarms that have pending actions
Alarms that have no actions
Alarms are sorted from open to closed.
Alarms are sorted alphabetically.
As new alarms are entered into the database, the currently displayed alarms table does not change. This behavior is true regardless of whether these alarms affect the currently displayed page. As existing alarms are deleted, the alarms table is updated automatically.
To update the alarms table, click the Refresh button.
The alarms table is updated to reflect the new alarms. The program recomputes the division of pages to add the new alarms. The date and time of refresh are displayed in the Last Refresh field.
With the Alarms tab selected in the Details window, choose View Alarms from the Options menu.
The View Specific Alarms dialog box appears.
To select the items to include in the alarms table, click the boxes next to the items.
A check mark is displayed in each box that you select. You can select the severity, state, and acknowledgement status of the alarms displayed in the table.
Accept your selections.
To accept your selections and close the View Specific Alarms dialog box, click the OK button.
To accept your selections and continue to display the View Specific Alarms dialog box, click the Apply button.
The alarms table is filtered. The table is redisplayed to reflect your selections.
The Sun Management Center software can show alarm activity in a log file. To view the log, follow these steps:
In the Details window for the Sun Management Center server, click the View Logs tab.
From the Choose Log File Type menu (the left-most icon at the top of the page), select Sun Management Center Log.
From the Sun Management Center Log list, select eventHistory.log.
The eventHistory.log file contents display in the window.
You can manage and control the alarms displayed in the alarms table.
If you acknowledge alarms when the alarms occur, you can more easily track new alarms. Unacknowledged alarms continue to be displayed in several locations:
Domain Status Summary
Hierarchy (tree) view
Contents view
Topology view
Relevant row or relevant column in the property table (contents view)
In the Details Alarms window, select the appropriate alarm row.
The selected row is highlighted.
To select multiple rows in the table, use the Shift key or drag the cursor. To select multiple, non-contiguous rows from the table, use the Control key.
Click the acknowledge button, which looks like a check mark.
The user ID, date, and time of acknowledgment are registered. A green check mark appears next to the alarm icon in the Severity column. Information appears in the Alarm Acknowledged On section of the Additional Alarm Information pane.
You should delete closed alarms to conserve space and to keep the alarms database current.
You can delete open alarms. If you delete open alarms, alarm information in the Alarms windows might differ from alarm information in the hierarchy and topology views. The alarms are removed from the view in the alarms table, but the alarm condition might still exist. You should delete open alarms only if those alarms are orphaned. An example of when this action might be required would be in the case of an unloaded module. If the module has triggered alarms that were not closed before the module was unloaded, the alarms are orphaned. If orphaned alarms remain in view, you need to be delete those alarms manually.
You should only delete open alarms when there is no other way to remove these alarms.
To delete all closed alarms, choose Delete All Closed Alarms from the Options menu.
To delete only specific alarms:
With the Alarms tab selected in the Details window, click the appropriate rows in the alarms table.
To select multiple rows in the table, use the Shift key or drag the cursor. To select multiple, non-contiguous rows from the table, use the Control key.
Choose Delete Alarms from the Options menu.
A confirmation alert appears.
You can track the history of an alarm by adding notes to an expanding list of notes for a specific alarm.
With the Alarms tab selected in the Details window, choose Edit Note from the Options menu.
The Edit Alarm Note window appears. The window shows any existing notes.
Click the Add button.
The Add window appears.
Type your note in the text field of the Add window.
Click the OK button to accept your changes and close the Add window.
The Edit Alarm Note window displays the following information:
The date and time that you added the note
The login identity that added the note
The text of the note
A default suggested fix might be displayed when a specific alarm is selected in the alarms table. You can also provide your own suggested fix to accompany the default suggestion.
With the Alarms tab selected in the Details window, select an alarm in the alarms table.
Choose Edit Suggested Fix from the Options menu.
The Edit Suggested Fix window appears. If a Sun Management Center suggested fix exists for this alarm, it is listed without a user identifier in the operator field.
To view the suggested fix, select the suggested fix in the Add window. Then, click the View Alarms button.
A read-only window appears that displays the text for the suggested fix. To close this window, click the Cancel button.
To add your own suggested fix for this alarm, click the Add button in the Edit Suggested Fix window.
The Add window appears.
Type your suggested fix in the text field of the Add window.
The length of the suggested fix text is limited to 4000 bytes.
Click the OK button to accept your changes and close the Add window.
By default, the Sun Management Center software checks whether a host or agent is not responding. However, no default action is defined in response to these conditions.
Select the managed object on which you want this action performed.
To apply this action to all children of a managed object, select the parent managed object. For example, to apply the action to all objects within a particular subnet, select the subnet.
Press mouse button 3 and select Alarm Action from the pop-up menu.
The Alarm Action window appears.
To define an action for host not responding, click the Actions button next to Host Not Responding.
The Action Selection window appears. To select or define an action, see To Select an Action.
The Actions buttons are at the right side of the window. If you have made your window smaller than expected, you might need to scroll to the right or enlarge the window to access the button.
To define an action for agent not responding, click the Actions button next to Agent Not Responding.
The Action Selection window appears. To select or define an action, see To Select an Action.
(Optional) If the current managed object is the child of an object for which an alarm action has been defined, choose whether to include group actions.
By default, host not responding and agent not responding actions apply to all children. You can choose that a particular child should not share the group action. Click the appropriate checkbox to disable the inherited action.
The Include Group Actions section of the Alarm Action window appears only for applicable child managed objects.
(Optional) To set the alarm action to apply only during certain times (for example, during business hours only), click the Scheduler button.
To run the alarm action starting on a day other than today, provide a date in the Start Date field.
You can edit the information in the Start Date field manually, or you can click on a date in the calendar to select the date.
Select an hour and minute from the pop-up menus next to Start Time.
Hours are based on a 24–hour clock. So, 16:00 is the same as 4:00 PM.
Select an hour and minute from the first pop-up menus next to End Time.
To allow this alarm action to occur more than once, choose how often it can occur from the Repeat Interval pop-up menu.
For example, to permit an alarm action to occur only on weekdays, select Mon-through-Fri.
(Optional) To limit how long the alarm action can be repeated, type a number in the Repeat Limit field.
For example, to run this alarm action for a month, type 4 in the Repeat Limit field when the word next to the field says “Weeks.” The word next to the Repeat Limit field depends on your Repeat Interval choice.
The alarm action is available indefinitely if you do both of the following:
You select a Repeat Interval other than One Time.
You do not provide a value in the Repeat Limit field.
Click OK to set this schedule, or click Cancel to exit from the Scheduler window.
To complete the alarm action definition and exit from the Alarm Action window, click the OK button.
Although alarms are defined, you need to tell Sun Management Center what to do when the alarm condition is met. For example, you might want to have the software send email notification to appropriate system administrators.
You need to perform this procedure on the managed object (host) that you are monitoring, not on system running the Sun Management Center server layer.
In the Attribute Editor, click the Actions tab.
The Attribute Editor Actions window is displayed.
Click the Actions button next to the action to be registered.
The Action Selection dialog box is displayed.
Select the type of alarm action that you want to register.
To send an email, follow these steps:
Select the Email button. Then, type the user name or alias to which to send the message.
To send to more than one user name, separate the user names with a space.
Type the message text.
To perform an action other than sending an email, follow these steps:
Define a script that specifies the action. For more information, see To Define an Alarm Action Script.
Select the Other button.
Choose a script to run from the list of available scripts.
Type any parameters that the script needs into the Arguments field.
To clear the alarm, select Clear.
To set the action as you defined and close the Action Selection window, click the OK button.
Click the OK button to accept the entries and close the Alarm Action dialog box.
The entry is displayed in the corresponding Action field of the Actions dialog box.
You can initiate some alarm actions manually after an alarm occurs.
Select one or more alarms with pending actions in the alarms table and click Run.
The currently registered alarm actions for the alarm are started. The action state in the alarms table changes from pending to executed. For additional details, see To Modify Pending Alarm Actions.
Several seconds can be needed for the action state in the alarm table to update.
For alarms with no registered actions, a “no action” icon is displayed in the Actions column of the alarms table. For alarms with a manual action, the “pending” icon is displayed until the user runs the action. Then, the “executed” icon is displayed. For alarms with an automatic action, the “executed” icon is always displayed.
With the Alarms tab selected in the Details window, select an alarm that has a pending action in the Alarms table.
Choose Modify Action from the Options menu.
The Action Selection dialog box is displayed. Only alarms that are pending manual action can be modified.
Select the action that you want to modify and make the necessary changes as described in To Select an Action.
When an alarm condition is met, you can choose to send an email, perform some other action, or clear the alarm.
To send an email, follow these steps:
To perform an action other than sending an email, follow these steps:
Select the Other button.
Select a script to run from the list of available scripts.
For information about creating a script to run, see To Define an Alarm Action Script.
Type any parameters that the script needs into the Arguments field.
To set the alarm to be cleared as the action, select the Clear button.
To set the action as you defined and close the Action Selection window, click the OK button.
The following procedure describes how to customize an alarm action to notify a user when an alarm condition occurs.
Create your custom alarm action script, using the following optional arguments:
The alarm severity, such as Warning, Critical, and so on.
The complete alarm string, including severity. For example: Critical: Machine A Kernel Reader Number of User Sessions > 10.
At the command-line level, become superuser.
# su - |
Install the script in the home Sun Management Center directory.
The default is the /var/opt/SUNWsymon/bin/ directory. For example:
# cp custom-alarm-script /var/opt/SUNWsymon/bin/ |
Click the Module Browser tab in the Details window.
Set the alarm threshold.
For more information, see Example: To Define and Respond to an Alarm.
Open the Attribute Editor for the property to which you want this script to apply.
Click the Actions tab.
The action rows are displayed.
Type custom-alarm-script %statusstringfmt in the Critical Action field of the Actions panel.
Ensure that the Automatic check box on the Critical Action line is selected.
By default, your script will execute automatically.
To apply this action script and close the Attribute Editor window, click the OK button.
There are many methods that you can use to define and manage alarms. Here are a few examples to better illustrate how you might perform these functions in your environment.
This example illustrates how to create and acknowledge a simple critical alarm when the memory usage on the specified system exceeds 50%.
Access the Details window for the managed object on which you want to define the alarm.
Click the Module Browser tab in the Details window.
Click the expansion icon next to the Operating System icon in the hierarchy tree view.
The Operating System modules are displayed.
Click the expansion icon next to the Kernel Reader icon.
The Kernel Reader properties are displayed.
Double-click the Memory Usage Statistics icon.
The Memory Usage Statistics properties table is displayed in the contents view.
Select the table cell for % Memory Used.
If the Attributes button is unavailable and is dimmed when you select the table cell, verify that you selected the data cell not the label cell.
Click the Attributes button.
The Attribute Editor window is displayed.
Click the Alarms tab.
The Alarms panel appears on which you can define threshold values for Critical, Caution, and Alert alarms.
Type 50 in the Critical Threshold (>) field.
A critical alarm will be generated when the memory usage exceeds 50% on the selected managed object.
This example uses a low threshold to create an alarm immediately. Normally, you might generate alarms according to the following criteria:
An alert alarm when the usage exceeds 50%
A caution alarm when the usage exceeds 65%
A critical alarm when the usage exceeds 80%
Click the OK button to apply your changes and close the Attribute Editor window.
Almost immediately, the % Memory Used data field in the table turns red. In addition, red alarm icons are displayed on the Operating System, Kernel Reader, and Memory Usage Statistics folders and icons. If the red icon does not appear, verify whether your system has an open, unacknowledged black alarm.
Click the Alarms tab in the Details window.
The alarm that you created is listed in the alarms table. See Chapter 12, Managing Alarms for more information.
Click the Acknowledge button to acknowledge this alarm.
The Acknowledge button looks like a check mark.
Create additional alarm thresholds, and familiarize yourself with their operation.
Once you have created these alarms, you can set up security permissions so that another Sun Management Center software user cannot change your alarm thresholds. For more information about security, see Chapter 18, Sun Management Center Security.
You do not need to fill in alarm information for all alarm thresholds. For example, you might choose to create only a critical alarm threshold.
This example illustrates creating a situation where an alarm is registered if a value exceeds the alarm limit. Table 10–1 lists other common alarm limits in the software.
This example sets up an email to be sent when the system load average exceeds the threshold.
When an alarm condition is met, sending an email message is a good solution in most situations. However, if a router or an uninterruptible power supply (UPS) is down, your email message would not be sent.
You might consider setting up a pager alarm to send an alarm message to a terminal and to a pager through a modem for such failures. For more information, investigate quick paging or instant paging solutions.
Click the Module Browser tab in the Details window.
Click the expansion icon next to the Operating System icon in the hierarchy tree view.
The Operating System modules are displayed.
Click the expansion icon next to the Kernel Reader icon.
The Kernel Reader properties are displayed.
Double-click the System Load Statistics icon.
The System Load Statistics properties table is displayed in the contents view.
Select the table cell for Load Averages Over the Last 5 Minutes.
Click the Attributes button.
The Attribute Editor window is displayed.
Click the Actions tab.
The Action Selection screen appears.
Click the Email radio button to activate the To and Message fields.
Type the usernames in the To field and the message in the Message field.
Click OK to accept the changes that you made and close this window.
The following email is sent to the usernames whenever the load averages alert alarm occurs.
Date: Wed, 30 Jun 2000 15:25:39 -0800 From: root@MachineB (0000-Admin(0000)) Subject: Sun Management Center - Alert Alarm Action Mime-Version: 1.0 Sun Management Center alarm action notification ... {Alert: machineB Kernel Reader Load Average Over The Last 5 Minutes > 0.01Jobs} |