11 Configuring Notifications and Logging

Oracle HSM file systems support automated, remote notification using the Simple Network Management Protocol (SNMP) and provide comprehensive, configurable logging facilities. This chapter outlines the following topics:

• Configuring Simple Network Management Protocol (SNMP)

• Enabling Oracle HSM Logging

• Configuring Device Logging

• Configuring Log Rotation

• Enabling Email Alerts.

Configuring Simple Network Management Protocol (SNMP)

Network management applications can monitor Oracle HSM file systems using the Simple Network Management Protocol (SNMP). You can configure the SNMP agent to automatically send traps that alert network management stations to faults and configuration changes.

The Oracle HSM Management Information Base (MIB) defines the types of information that SNMP traps provide. These include configuration errors, SCSI tapealert events, and various kinds of atypical system activity. For more information, see the Management Information Base (MIB) file, /var/snmp/mib/SUN-SAM-MIB.mib.

When an Oracle HSM trap event occurs, the Solaris kernel system-event notification daemon, syseventd, calls the script /etc/opt/SUNWsamfs/scripts/sendtrap. The script then sends the trap to either the local host or to a management station that you specify. The script supports version 2c of the SNMP standard, which is backward compatible with earlier versions of the standard. Note that version 2c exchanges authentication credentials—community strings—and management data in clear text. See the sendtrap man page for additional details.

To configure SNMP notification, carry out the following tasks:

• Make Sure that All SNMP Management Stations are Listed in the /etc/hosts file

• Enable Support for SNMP

• Designate Management Stations as Trap Recipients and Configure Authentication.

This section also includes instructions should you at any point need to Disable Support for SNMP.

Make Sure that All SNMP Management Stations are Listed in the /etc/hosts file

1. Log in to the Oracle HSM server as root.

   In the example, the Oracle HSM server host is samqfs1mds:

   [samqfs1mds]root@solaris:~# 
   

2. Open the /etc/hosts file in a text editor. Make sure that it contains an entry for each host that you intend to use as an SNMP management station.

   In the example, we use the vi editor. One of the intended management stations, management1, is listed. But the other, management2, is not:

   [samqfs1mds]root@solaris:~# vi /etc/hosts
   # Internet host table
   ::1 localhost
   127.0.0.1 localhost loghost
   10.0.0.10 server1
   10.0.0.20 management1
   

3. If the /etc/hosts file does not contain entries for some or all of your intended SNMP management station hosts, add the required entries and save the file.

   In the example, we use the vi editor to add the missing management station, management2:

   [samqfs1mds]root@solaris:~# vi /etc/hosts
   # Internet host table
   ::1 localhost
   127.0.0.1 localhost loghost
   10.0.0.10 server1
   10.0.0.20 management1
   10.0.0.30 management2
   

4. When the /etc/hosts file contains entries for all of your intended SNMP management station hosts, close the editor.

   [samqfs1mds]root@solaris:~# vi /etc/hosts
   ...
   10.0.0.20 management1
   10.0.0.30 management2
   :wq
   [samqfs1mds]root@solaris:~# 
   

5. Next, Enable Support for SNMP.

Enable Support for SNMP

By default, support for SNMP notifications is enabled, so no action is needed unless SNMP support has been disabled at some point. If you need to re-enable SNMP support, proceed as follows:

1. Log in to the Oracle HSM server as root.

   In the example, the Oracle HSM server host is samqfs1mds:

   [samqfs1mds]root@solaris:~# 
   

2. Open the file /etc/opt/SUNWsamfs/defaults.conf in a text editor. Locate the line alerts = off.

   The directive alerts = off disables SNMP support. In the example, we open the file in the vi editor and locate the line:

   [samqfs1mds]root@solaris:~# vi /etc/opt/SUNWsamfs/defaults.conf 
   # These are the defaults.  To change the default behavior, uncomment the
   # appropriate line (remove the '#' character from the beginning of the line)
   # and change the value.
   ...
   alerts = off
   

3. To enable support for SNMP notifications, change the value of the alerts directive to on. Then save the file and close the editor.

   [samqfs1mds]root@solaris:~# vi /etc/opt/SUNWsamfs/defaults.conf 
   # These are the defaults.  To change the default behavior, uncomment the
   # appropriate line (remove the '#' character from the beginning of the line)
   # and change the value.
   ...
   alerts = on
   :wq
   [samqfs1mds]root@solaris:~# 
   

4. Tell the Oracle HSM service to reread the defaults.conf file and reconfigure itself accordingly. Use the command samd config.

   [samqfs1mds]root@solaris:~# 1mds]samd config
   [samqfs1mds]root@solaris:~# 
   

5. Next, Designate Management Stations as Trap Recipients and Configure Authentication.

Designate Management Stations as Trap Recipients and Configure Authentication

1. Log in to the Oracle HSM server as root.

   In the example, the Oracle HSM server host is samqfs1mds:

   [samqfs1mds]root@solaris:~# 
   

2. Open the /etc/opt/SUNWsamfs/scripts/sendtrap file in a text editor, and locate the line starting TRAP_DESTINATION=.

   The sendtrap file is a configurable shell script. In the example, we open the file in the vi editor:

   [samqfs1mds]root@solaris:~# vi /etc/opt/SUNWsamfs/scripts/sendtrap 
   # /etc/opt/SUNWsamfs/scripts/sendtrap#!/usr/bin/sh
   # sendtrap:
   #  This script gets invoked by the sysevent configuration file.
   #  This is not expected to be run as a stand-alone program
   ...
   # CONFIGURATION PARAMETERS:
   TRAP_DESTINATION=`hostname`
   

3. In the line TRAP_DESTINATION=`, replace the text within the single quotation marks with a space-delimited list of one or more trap recipients, each of the form hostname:port, where hostname is the hostname for a management station, as listed in /etc/hosts, and port is the port on which the host listens for traps.

   By default, traps are sent to UDP port 161 of the localhost. In the example, we add hosts management1 and management2 to the default, localhost. The localhost and management1 use the default port, while management2 uses a custom port, 1161:

   ...
   # CONFIGURATION PARAMETERS:
   TRAP_DESTINATION=`localhost:161 management1:161 management1:1161`
   

4. Scroll down to the line that sets the community string, COMMUNITY="public".

   The community string is the shared, plain-text password that authenticates agents and management stations in SNMP version 2c. The default value is the SNMP standard, public.

   ...
   # CONFIGURATION PARAMETERS:
   TRAP_DESTINATION=`localhost:161 management1:161 management1:1161`
   ...
   COMMUNITY="public"
   

5. Set the COMMUNITY="" directive to the value used by your management stations. Then save the file and close the editor.

   Do not edit anything else in the file. COMMUNITY="" and TRAP_DESTINATION=`` are the only editable parameters.

   Note that the default SNMP community string, public, is insecure. So your network administrator may mandate a more secure choice. SNMP version 2c allows up to 32 alphanumeric characters. In the example, we set the community string to Iv0wQh2th74bVVt8of16t1m3s8it4wa9.

   ...
   # CONFIGURATION PARAMETERS:
   TRAP_DESTINATION=`localhost:161 management1:163 management1:1162`
   ...
   COMMUNITY="Iv0wQh2th74bVVt8of16t1m3s8it4wa9"
   :wq
   [samqfs1mds]root@solaris:~# 
   

6. Next, Enable Oracle HSM Application Logging.

Disable Support for SNMP

The remote notification facility is enabled by default. If you want to disable remote notification, proceed as follows:

1. Log in to the Oracle HSM server as root.

   In the example, the Oracle HSM server host is samqfs1mds:

   [samqfs1mds]root@solaris:~# 
   

2. Open the /etc/opt/SUNWsamfs/defaults.conf file in a text editor. Locate the line #alerts = on.

   In the example, we use the vi editor:

   [samqfs1mds]root@solaris:~# vi /etc/opt/SUNWsamfs/defaults.conf 
   # These are the defaults.  To change the default behavior, uncomment the
   # appropriate line (remove the '#' character from the beginning of the line)
   # and change the value.
   ...
   #alerts = on
   [samqfs1mds]root@solaris:~# 
   

3. To disable support for SNMP notifications, delete the hash character (#) to uncomment the line and change the value of alerts to off. Then save the file and close the editor.

   [samqfs1mds]root@solaris:~# vi /etc/opt/SUNWsamfs/defaults.conf 
   # These are the defaults.  To change the default behavior, uncomment the
   # appropriate line (remove the '#' character from the beginning of the line)
   # and change the value.
   ...
   alerts = off
   :wq
   [samqfs1mds]root@solaris:~# 
   

4. Tell the Oracle HSM service to reread the defaults.conf file and reconfigure itself accordingly. Use the command samd config.

   [samqfs1mds]root@solaris:~# samd config
   [samqfs1mds]root@solaris:~# 
   

5. Stop here. SNMP support is disabled.

Enabling Oracle HSM Logging

The /var/adm/sam-log file records status and error information for the Oracle HSM application and its component daemons and processes. To set up the logging process, proceed as follows:

Enable Oracle HSM Application Logging

1. Log in to the Oracle HSM server as root.

   In the example, the Oracle HSM server host is samqfs1mds:

   [samqfs1mds]root@solaris:~# 
   

2. Open the /etc/syslog.conf file in a text editor.

   In the example, we open the file in the vi editor:

   [samqfs1mds]root@solaris:~# vi /etc/syslog.conf
   # syslog configuration file ...
   *.err;kern.notice;auth.notice                   /dev/sysmsg
   *.err;kern.debug;daemon.notice;mail.crit        /var/adm/messages
   *.alert;kern.err;daemon.err                     operator
   *.alert                                         root
   ...
   

3. In the /etc/syslog.conf file, add a line consisting of the string local7.debug, one or more Tab characters, and the path string /var/adm/sam-log. Then save the file and close the editor.

   In the example, we also add a comment:

   [samqfs1mds]root@solaris:~# vi /etc/syslog.conf
   # syslog configuration file ...
   *.err;kern.notice;auth.notice                   /dev/sysmsg
   *.err;kern.debug;daemon.notice;mail.crit        /var/adm/messages
   *.alert;kern.err;daemon.err                     operator
   *.alert                                         root
   ...
   # Oracle HSM logging
   local7.debug    /var/adm/sam-log
   :wq
   [samqfs1mds]root@solaris:~# 
   

4. Create the log file /var/adm/sam-log. Use the command touch /var/adm/sam-log.

   [samqfs1mds]root@solaris:~# touch /var/adm/sam-log
   [samqfs1mds]root@solaris:~# 
   

5. Tell the Solaris syslogd daemon to re-read its configuration files and start Oracle HSM logging. Use the command pkill -HUP syslogd.

   Whenever it receives a HUP signal, the syslogd logging service re-reads the /etc/syslog.conf configuration file, closes all open log files, and then opens the log files that are listed in syslog.conf. When the command runs, Oracle HSM logging is enabled:

   [samqfs1mds]root@solaris:~# pkill -HUP syslogd
   [samqfs1mds]root@solaris:~# 
   

6. Go to Configuring Device Logging.

Configuring Device Logging

The device logging facility provides error information specific to individual hardware devices (it does not collect soft media errors). Each device has its own log file, named with the corresponding equipment ordinal number and stored in the /var/opt/SUNWsamfs/devlog/ directory.

Device logs can grow rapidly. So, by default, the system logs a limited set of event data: err, retry, syserr, and date. Later, if problems arise, you can use the samset command to log additional events on a per-device basis (see the devlog section of the samset man page for details).

Enable the Device Log in the defaults.conf File

To enable basic device logging, proceed as follows:

1. Log in to the Oracle HSM server as root.

   In the example, the Oracle HSM server host is samqfs1mds:

   [samqfs1mds]root@solaris:~# 
   

2. Open the /etc/opt/SUNWsamfs/defaults.conf in a text editor.

   In the example, we open the file in the vi editor:

   [samqfs1mds]root@solaris:~# vi /etc/opt/SUNWsamfs/defaults.conf 
   # These are the defaults.  To change the default behavior, uncomment the
   # appropriate line (remove the '#' character from the beginning of the line)
   # and change the value.
   ...
   

3. In the defaults.conf file, add a line that defines the default level of device logging that you require. Enter the directive devlog equipment-number loggable-events, where:

   • equipment-number is either the keyword all, for all equipment defined in the /etc/opt/SUNWsamfs/mcf file, or the equipment ordinal that identifies a specific piece of equipment defined in the mcf.

   • loggable-events is a space-delimited list of the defaults: err retry syserr date

     See the devlog section of the samset man page for a comprehensive list of event types. But, to minimize log sizes, use the default selections. For diagnostic purposes, the samset command can selectively enable additional events as needed.

   In the example, we enable device logging for all devices using the default logging level:

   [samfs-mds1]root@solaris:~# vi /etc/opt/SUNWsamfs/defaults.conf 
   # These are the defaults.  To change the default behavior, uncomment the
   # appropriate line (remove the '#' character from the beginning of the line)
   # and change the value.
   ...
   devlog all err retry syserr date
   

4. Save the defaults.conf file, and close the editor.

   [samfs-mds1]root@solaris:~# vi /etc/opt/SUNWsamfs/defaults.conf 
   # These are the defaults.  To change the default behavior, uncomment the
   # appropriate line (remove the '#' character from the beginning of the line)
   # and change the value.
   ...
   devlog all err retry syserr date
   :wq
   [samfs-mds1]root@solaris:~# 
   

5. Tell the Oracle HSM service to reread the defaults.conf file and reconfigure itself accordingly. Use the command samd config.

   [samfs-mds1]root@solaris:~# samd config
   [samfs-mds1]root@solaris:~# 
   

6. Next, Set Up Automatic Rotation for Oracle HSM Log Files.

Configuring Log Rotation

Log files can grow rapidly, consuming large amounts of space and making the logs hard to use. So you should configure automatic log rotation for Oracle HSM logs. The software includes a script for this purpose, log_rotate.sh, that you can run from the Solaris crontab file.

For each log that you intend to rotate, you create two crontab entries. The first one runs the log_rotate.sh script at the desired time. If the target log file has reached a specified minimum size (the default is 100000 bytes), the script renames it and deletes the oldest existing copy (seven are kept at any one time). The second crontab entry tells the Solaris logging daemon, syslogd, to restart logging with a new log file.

Set Up Automatic Rotation for Oracle HSM Log Files

Consider rotating the following logs:

• The Oracle HSM log file, sam-log, located as specified in the /etc/syslog.conf file.

• The device log files located in the /var/opt/SUNWsamfs/devlog/ directory.

• The stager log files specified in the /etc/opt/SUNWsamfs/stager.cmd file.

• The releaser log files specified in the /etc/opt/SUNWsamfs/releaser.cmd file.

• The recycler log files specified in the /etc/opt/SUNWsamfs/recycler.cmd file.

Archiver log files should not be rotated! The log information is valuable for analytics and file-system recovery. For proper handling of archiver logs, see "Configure File System Protection".

Once you have decided on the logs that should be rotated, proceed as follows for each log:

1. Log in to the Oracle HSM server as root.

   In the example, the Oracle HSM server host is samqfs1mds:

   [samqfs1mds]root@solaris:~# 
   

2. Copy the script file log_rotate.sh from /opt/SUNWsamfs/examples/ (the uninstalled location) to /etc/opt/SUNWsamfs/scripts/.

   Note that the command below is entered as a single line—the line break is escaped by the backslash:

   [samfs-mds1]root@solaris:~# cp /opt/SUNWsamfs/examples/log_rotate.sh \  /etc/opt/SUNWsamfs/scripts/
   

3. Open the root user's crontab file for editing. Use the command crontab -e.

   The crontab command opens an editable copy of the root user's crontab file in the text editor specified by the EDITOR environment variable (for full details, see the Solaris crontab man page). In the example, we use the vi editor:

   [samfs-mds1]root@solaris:~# crontab -e
   #ident   "%Z%%M%     %I%       %E% SMI"
   # Copyright 2007 Sun Microsystems, Inc.  All rights reserved.
   # Use is subject to license terms.
   # The root crontab should be used to perform accounting data collection.
   10 3 * * * /usr/sbin/logadm
   ...
   30 0,9,12,18,21 * * * /usr/lib/update-manager/update-refresh.sh
   

4. On a new line, specify the time of day when the log file will be rotated by entering minutes hour * * day-of-the-week, where:

   • minutes is an integer in the range [0-59] that species the minute when the job starts.

   • hour is an integer in the range [0-23] that species the hour when the job starts.

   • * (asterisk) specifies unused values.

     For a task that runs daily, the values for day of the month [1-31] and month [1-12] are unused.

   • day-of-the-week is an integer in the range [0-6], starting from Sunday (0).

   • Spaces separate the fields in the time specification.

   In the example, we schedule log rotation to begin at 3:10 AM every Sunday.

   [samfs-mds1]root@solaris:~# crontab -e
   #ident   "%Z%%M%     %I%       %E% SMI"
   # Copyright 2007 Sun Microsystems, Inc.  All rights reserved.
   # Use is subject to license terms.
   # The root crontab should be used to perform accounting data collection.
   10 3 * * * /usr/sbin/logadm
   ...
   30 0,9,12,18,21 * * * /usr/lib/update-manager/update-refresh.sh
   10 3 * * 0 
   

5. Continuing on the same line, enter the path and name of the shell script file that rotates Oracle HSM logs, /etc/opt/SUNWsamfs/scripts/log_rotate.sh, followed by a space.

   [samfs-mds1]root@solaris:~# crontab -e
   #ident   "%Z%%M%     %I%       %E% SMI"
   # Copyright 2007 Sun Microsystems, Inc.  All rights reserved.
   # Use is subject to license terms.
   # The root crontab should be used to perform accounting data collection.
   10 3 * * * /usr/sbin/logadm
   ...
   30 0,9,12,18,21 * * * /usr/lib/update-manager/update-refresh.sh
   10 3 * * 0 /etc/opt/SUNWsamfs/scripts/log_rotate.sh 
   

6. Continuing on the same line, enter the name of the log that you need to rotate and the minimum file size file to rotate. Enter the text samfslog [minimum-size] where samfslog is the path to an Oracle HSM log file and [minimum-size] is an optional integer that specifies the smallest file-size in bytes that the script rotates (the default is 100000).

   In the example, we need to rotate the /var/adm/sam-log. We accept the default minimum size:

   [samfs-mds1]root@solaris:~# crontab -e
   #ident   "%Z%%M%     %I%       %E% SMI"
   # Copyright 2007 Sun Microsystems, Inc.  All rights reserved.
   # Use is subject to license terms.
   # The root crontab should be used to perform accounting data collection.
   10 3 * * * /usr/sbin/logadm
   ...
   30 0,9,12,18,21 * * * /usr/lib/update-manager/update-refresh.sh
   10 3 * * 0 /etc/opt/SUNWsamfs/scripts/log_rotate.sh /var/adm/sam-log
   

7. Start a new line. Create a crontab entry that starts 10 minutes after the log_rotate.sh script. This entry tells the Solaris syslogd daemon to close its old log file and resume logging to a new file. Enter the line minutes hour * * day-of-the-week /bin/kill -HUP `/bin/cat /etc/syslog.pid`, where minutes hour * * day-of-the-week specifies a time 10 minutes later than the time specified in the previous step.

   In the example, the entry restarts Oracle HSM logging at 3:20 AM every Sunday:

   [samfs-mds1]root@solaris:~# crontab -e
   #ident   "%Z%%M%     %I%       %E% SMI"
   # Copyright 2007 Sun Microsystems, Inc.  All rights reserved.
   # Use is subject to license terms.
   # The root crontab should be used to perform accounting data collection.
   10 3 * * * /usr/sbin/logadm
   ...
   30 0,9,12,18,21 * * * /usr/lib/update-manager/update-refresh.sh
   10 3 * * 0 /etc/opt/SUNWsamfs/scripts/log_rotate.sh /var/adm/sam-log
   20 3 * * 0  /bin/kill -HUP `/bin/cat /etc/syslog.pid`
   

8. Save the file, and close the editor.

   [samfs-mds1]root@solaris:~# crontab -e
   #ident   "%Z%%M%     %I%       %E% SMI"
   # Copyright 2007 Sun Microsystems, Inc.  All rights reserved.
   # Use is subject to license terms.
   # The root crontab should be used to perform accounting data collection.
   10 3 * * * /usr/sbin/logadm
   ...
   30 0,9,12,18,21 * * * /usr/lib/update-manager/update-refresh.sh
   10 3 * * 0 /etc/opt/SUNWsamfs/scripts/log_rotate.sh /var/adm/sam-log
   20 3 * * 0  /bin/kill -HUP `/bin/cat /etc/syslog.pid`
   :wq
   [samfs-mds1]root@solaris:~# 
   

9. Repeat this procedure until you have configured log rotation for all required logs.

10. Next, see Enabling Email Alerts.

Enabling Email Alerts

Email alerts are best set up from the Oracle HSM Manager graphical user interface. See the online help for details.

If you must configure email alerts from the commandline interface, see the defaults.conf, archiver.sh, dev_down.sh, load_notify.sh, recycler.sh, archiver.cmd, recycler.cmd, and notify.cmd man pages.

Your Oracle HSM system is now configured. But before you begin to use it, protect your work. For instructions, see "Backing Up the Oracle HSM Configuration".