System Administration Guide: Advanced Administration

Chapter 10 System Accounting (Reference)

This chapter provides reference information about system accounting.

This is a list of the reference information in this chapter.

For more information about system accounting tasks, see Chapter 9, Managing System Accounting (Tasks).

`runacct` Script

The main daily accounting script, runacct, is normally invoked by the cron command outside of normal business hours. The runacct script processes connect, fee, disk, and process accounting files. This script also prepares daily and cumulative summary files for use by the prdaily and monacct scripts for billing purposes.

The runacct script takes care not to damage files if errors occur.

A series of protection mechanisms that are used to perform the following tasks:

Recognize an error
Provide intelligent diagnostics
Complete processing in such a way that the runacct script can be restarted with minimal intervention

This script records its progress by writing descriptive messages to the active file. Files used by the runacct script are assumed to be in the /var/adm/acct/nite directory, unless otherwise noted. All diagnostic output during the execution of the runacct script is written to the fd2log file.

When the runacct script is invoked, it creates the lock and lock1 files. These files are used to prevent simultaneous execution of the runacct script. The runacct program prints an error message if these files exist when it is invoked. The lastdate file contains the month and day the runacct script was last invoked, and is used to prevent more than one execution per day.

If the runacct script detects an error, the following occurs:

A message is written to the console
Email is sent to root and adm
Locks might be removed
Diagnostics are saved
Execution is ended

For instructions on how to restart the runacct script, see How to Restart the runacct Script.

To allow the runacct script to be restarted, processing is broken down into separate re-entrant states. The statefile file is used to track the last state completed. When each state is completed, the statefile file is updated to reflect the next state. After processing for the state is complete, the statefile file is read and the next state is processed. When the runacct script reaches the CLEANUP state, it removes the locks and ends. States are executed as shown in the following table.

Table 10–1 States of the runacct Script


State	Description
`SETUP`	The `turnacct` `switch` command is executed to create a new `pacct` file. The `/var/adm/pacctn` process accounting files (except for the `pacct` file) are moved to the `/var/adm/Spacctn.MMDD` files. The `/var/adm/wtmpx` file is moved to the `/var/adm/acct/nite/wtmp.MMDD` file (with the current time record added on the end) and a new `/var/adm/wtmp` file is created. The `closewtmp` and `utmp2wtmp` programs add records to the `wtmp`.`MMDD` file and the new `wtmpx` file to account for users who are currently logged in.
`WTMPFIX`	The `wtmpfix` program checks the `wtmp.MMDD` file in the `nite` directory for accuracy. Because some date changes cause the `acctcon` program to fail, the `wtmpfix` program attempts to adjust the time stamps in the `wtmpx` file if a record of a date change appears. This program also deletes any corrupted entries from the `wtmpx` file. The fixed version of the `wtmp.MMDD` file is written to the `tmpwtmp` file.
`CONNECT`	The `acctcon` program is used to record connect accounting records in the file `ctacct.MMDD`. These records are in `tacct.h` format. In addition, the `acctcon` program creates the `lineuse` and `reboots` files. The `reboots` file records all the boot records found in the `wtmpx` file.
`PROCESS`	The `acctprc` program is used to convert the `/var/adm/Spacctn.MMDD` process accounting files into complete accounting records in the `ptacctn.MMDD` files. The `Spacct` and `ptacct` files are correlated by number so that if the `runacct` script fails, the `Spacct` files are not processed.
`MERGE`	The `acctmerg` program merges the process accounting records with the connect accounting records to form the `daytacct` file.
`FEES`	The `acctmerg` program merges ASCII `tacct` records from the `fee` file into the `daytacct` file.
`DISK`	The `dodisk` script produces the `disktacct` file. If the `dodisk` script has been run, which produces the `disktacct` file, the `DISK` program merges the file into the `daytacct` file and moves the `disktacct` file to the `/tmp/disktacct.MMDD` file.
`MERGETACCT`	The `acctmerg` program merges the `daytacct` file with the `sum/tacct` file, the cumulative total accounting file. Each day, the `daytacct` file is saved in the `sum/tacct.MMDD` file so that the `sum/tacct` file can be re-created if it is corrupted or lost.
`CMS`	The `acctcms` program is run several times. This program is first run to generate the command summary by using the `Spacctn` files and write the data to the `sum/daycms` file. The `acctcms` program is then run to merge the `sum/daycms` file with the `sum/cms` cumulative command summary file. Finally, the `acctcms` program is run to produce `nite/daycms` and `nite/cms`, the ASCII command summary files from the `sum/daycms` and `sum/cms` files, respectively. The `lastlogin` program is used to create the `/var/adm/acct/sum/loginlog` log file. This file reports when each user last logged in. If the `runacct` script is run after midnight, the dates showing the time last logged in by some users will be incorrect by one day.
`USEREXIT`	Any installation-dependent (local) accounting program can be run at this point. The `runacct` script expects this program to be called the `/usr/lib/acct/runacct.local` program.
`CLEANUP`	This state cleans up temporary files, runs the `prdaily` script and saves its output in the `sum/rpt.MMDD` file, removes the locks, and then exits.

Caution –

When restarting the runacct script in the CLEANUP state, remove the last ptacct file because this file will not be complete.

Daily Accounting Reports

The runacct shell script generates five basic reports upon each invocation. The following table describes these reports.

Table 10–2 Daily Accounting Reports


Report Type	Description
Daily Report	Shows terminal line utilization by `tty` number.
Daily Usage Report	Indicates usage of system resources by users (listed in order of user ID).
Daily Command Summary	Indicates usage of system resources by commands, listed in descending order of memory use. In other words, the command that used the most memory is listed first. This same information is reported for the month in the monthly command summary.
Monthly Command Summary	A cumulative summary that reflects the data accumulated since the last invocation of the `monacct` program.
Last Login Report	Shows the last time each user logged in (listed in chronological order).

Daily Report

This report gives information about each terminal line used. The following is a sample Daily Report:

Jan 16 02:30 2004  DAILY REPORT FOR venus Page 1


from Mon Jan 15 02:30:02 2004
to   Tue Oan 16 02:30:01 2004
1       runacct
1       acctcon

TOTAL DURATION IS 1440 MINUTES
LINE         MINUTES  PERCENT  # SESS  # ON  # OFF
console      868      60       1       1     2
TOTALS       868      --       1       1     2

The from and to lines specify the time period reflected in the report. This time period covers the time the last Daily Report was generated to the time the current Daily Report was generated. Then, the report presents a log of system reboots, shutdowns, power failure recoveries, and any other record written to the /var/adm/wtmpx file by the acctwtmp program. For more information, see the acct(1M) man page.

The second part of the report is a breakdown of terminal line utilization. The TOTAL DURATION tells how long the system was in multiuser mode (accessible through the terminal lines). The following list describes the data provided by the Daily Report.

LINE: The terminal line or access port.
MINUTES: The number of minutes that the line was in use during the accounting period.
PERCENT: The TOTAL DURATION divided by the number of MINUTES.
# SESS: The number of times this line or port was accessed for a login session.
# ON: Same as SESS. (This column no longer has meaning. Previously, this column listed the number of times that a line or port was used to log in a user.)
# OFF: The number of times a user logs out and any interrupts that occur on that line. Generally, interrupts occur on a port when ttymon is first invoked after the system is brought to multiuser mode. If the # OFF exceeds the # SESS by a large factor, the multiplexer, modem, or cable is probably going bad. Or, a bad connection exists somewhere. The most common cause is an unconnected cable dangling from the multiplexer.

During real time, you should monitor the /var/adm/wtmpx file because it is the file from which the connect accounting is derived. If the wtmpx file grows rapidly, execute the following command to see which tty line is the noisiest.

# /usr/lib/acct/acctcon -l file < /var/adm/wtmpx

If interruption is occurring frequently, general system performance will be affected. Additionally, the wtmp file might become corrupted. To correct this problem, see How to Fix a Corrupted wtmpx File.

Daily Usage Report

The Daily Usage Report breaks down system resource utilization by user. A sample of this report follows:

Jan 16 02:30 2004  DAILY USAGE REPORT FOR skisun Page 1


     LOGIN  CPU  (MINS)  KCORE-   MINS    CONNECT  (MINS) DISK   # OF   # OF  # DISK  FEE
UID  NAME   PRIME NPRIME PRIME    NPRIME  PRIME    NPRIME BLOCKS PROCS  SESS  SAMPLES
0    TOTAL  72    148    11006173 51168   26230634 57792  539    330    0     2150    1
0    root   32    76     11006164 33664   26230616 22784  0      0      0     127     0
4    adm    0     0      22       51      0        0      0      420    0     0       0
101  rimmer 39    72     894385   1766020 539      330    0      1603   1     0       0

The following table describes the data provided by the Daily Usage Report.

Table 10–3 Daily Usage Report Data


Column	Description
`UID`	User ID number.
`LOGIN NAME`	Login (or user) name of the user. Identifies a user who has multiple login names.
`CPU (MINS)`	Amount of time, in minutes, that the user's process used the central processing unit. Divided into `PRIME` and `NPRIME` (nonprime) utilization. The accounting system's version of this data is located in the `/etc/acct/holidays` file.
`KCORE-MINS`	A cumulative measure of the amount of memory in Kbyte segments per minute that a process uses while running. Divided into `PRIME` and `NPRIME` utilization.
`CONNECT (MINS)`	Amount of time, in minutes, that the a user was logged in to the system, or “real time.” Divided into `PRIME` and `NPRIME` utilization. If these numbers are high while the `# OF PROCS` is low, you can conclude that the user logs in first thing in the morning and hardly touches the terminal the rest of the day.
`DISK BLOCKS`	Output from the `acctdusg` program, which runs the disk accounting programs and merges the accounting records (`daytacct`). For accounting purposes, a block is 512 bytes.
`# OF PROCS`	Number of processes invoked by the user. If large numbers appear, a user might have a shell procedure that has run out of control.
`# OF SESS`	Number of times that a user logged in to the system.
`# DISK SAMPLES`	Number of times that disk accounting was run to obtain the average number of `DISK BLOCKS`.
`FEE`	Often unused field that represents the total accumulation of units charged against the user by the `chargefee` script.

Daily Command Summary

The Daily Command Summary report shows the system resource utilization by command. With this report, you can identify the most heavily used commands. Based on how those commands use system resources, you can then gain insight on how best to tune the system.

These reports are sorted by TOTAL KCOREMIN, which is an arbitrary gauge but often useful for calculating drain on a system.

A sample Daily Command Summary follows:

								TOTAL COMMAND SUMMARY
COMMAND   NUMBER      TOTAL   TOTAL     TOTAL   MEAN    MEAN     HOG   CHARS   BLOCKS
NAME        CMDS    KCOREMIN CPU-MIN REAL-MIN  SIZE-K  CPU-MIN  FACTOR TRNSFD  READ

TOTALS      2150  1334999.75  219.59 724258.50 6079.48   0.10   0.00   397338982 419448

netscape      43  2456898.50   92.03  54503.12 26695.51  2.14   0.00   947774912 225568
adeptedi       7    88328.22    4.03    404.12 21914.95  0.58   0.01    93155160   8774
dtmail         1    54919.17    5.33  17716.57 10308.94  5.33   0.00   213843968  40192
acroread       8    31218.02    2.67  17744.57 11682.66  0.33   0.00   331454464  11260
dtwm           1    16252.93    2.53  17716.57 6416.05   2.53   0.00   158662656  12848
dtterm         5     4762.71    1.30  76300.29 3658.93   0.26   0.00    33828352  11604
dtaction      23     1389.72    0.33      0.60 4196.43   0.01   0.55    18653184    539
dtsessio       1     1174.87    0.24  17716.57 4932.97   0.24   0.00    23535616   5421
dtcm           1      866.30    0.18  17716.57 4826.21   0.18   0.00     3012096   6490

The following list describes the data provided by the Daily Command Summary.

COMMAND NAME: Name of the command. All shell procedures are lumped together under the name sh because only object modules are reported by the process accounting system. You should monitor the frequency of programs called a.out or core, or any other unexpected name. You can use the acctcom program to determine who executed an oddly named command and if superuser privileges were used.
NUMBER CMDS: Total number of times this command was run.
TOTAL KCOREMIN: Total cumulative measurement of the Kbyte segments of memory used by a process per minute of run time.
TOTAL CPU-MIN: Total processing time this program accumulated.
TOTAL REAL-MIN: Total real-time (wall-clock) minutes this program accumulated.
MEAN SIZE-K: Mean (average) of the TOTAL KCOREMIN over the number of invocations reflected by the NUMBER CMDS.
MEAN CPU-MIN: Mean (average) derived from the NUMBER CMDS and the TOTAL CPU-MIN.
HOG FACTOR: Total CPU time divided by elapsed time. Shows the ratio of system availability to system utilization, providing a relative measure of total available CPU time consumed by the process during its execution.
CHARS TRNSFD: Total number of characters transferred by the read and write system calls. Might be negative due to overflow.
BLOCKS READ: Total number of the physical block reads and writes that a process performed.

Monthly Command Summary

The format of the Daily Command Summary and the Monthly Command Summary reports are virtually the same. However, the daily summary reports only on the current accounting period while the monthly summary reports on the start of the fiscal period to the current date. In other words, the monthly report is a cumulative summary that reflects the data accumulated since the last invocation of the monacct program.

A sample Monthly Command Summary follows.

Jan 16 02:30 2004  MONTHLY TOTAL COMMAND SUMMARY Page 1


                                     TOTAL COMMAND SUMMARY
COMMAND   NUMBER      TOTAL   TOTAL     TOTAL   MEAN     MEAN    HOG      CHARS    BLOCKS
NAME        CMDS    KCOREMIN CPU-MIN  REAL-MIN  SIZE-K   CPU-MIN FACTOR  TRNSFD    READ

TOTALS     42718  4398793.50  361.92  956039.00 12154.09 0.01    0.00  16100942848 825171

netscape     789  3110437.25  121.03   79101.12 25699.58 0.15    0.00   3930527232 302486
adeptedi      84  1214419.00   50.20    4174.65 24193.62 0.60    0.01    890216640 107237
acroread     145   165297.78    7.01   18180.74 23566.84 0.05    0.00   1900504064  26053
dtmail         2    64208.90    6.35   20557.14 10112.43 3.17    0.00    250445824  43280
dtaction     800    47602.28   11.26      15.37  4226.93 0.01    0.73    640057536   8095
soffice.      13    35506.79    0.97       9.23 36510.84 0.07    0.11    134754320   5712
dtwm           2    20350.98    3.17   20557.14  6419.87 1.59    0.00    190636032  14049

For a description of the data provided by the Monthly Command Summary, see Daily Command Summary.

Last Login Report

This report gives the date when a particular login was last used. You can use this information to find unused logins and login directories that can be archived and deleted. A Last Login Report follows.

Jan 16 02:30 2004  LAST LOGIN Page 1


01-06-12  kryten         01-09-08  protoA      01-10-14  ripley
01-07-14  lister         01-09-08  protoB      01-10-15  scutter1
01-08-16  pmorph         01-10-12  rimmer      01-10-16  scutter2

Examining the `pacct` File With `acctcom`

At any time, you can examine the contents of the /var/adm/pacctn files, or any file with records in the acct.h format, by using the acctcom program. If you do not specify any files and do not provide any standard input when you run this command, the acctcom command reads the pacct file. Each record read by the acctcom command represents information about a terminated process. Active processes can be examined by running the ps command.

The default output of the acctcom command provides the following information:

# acctcom
COMMAND                           START    END          REAL     CPU    MEAN
NAME       USER     TTYNAME       TIME     TIME       (SECS)  (SECS) SIZE(K)
#accton    root      ?            02:30:01 02:30:01     0.03    0.01  304.00
turnacct   adm       ?            02:30:01 02:30:01     0.42    0.01  320.00
mv         adm       ?            02:30:01 02:30:01     0.07    0.01  504.00
utmp_upd   adm       ?            02:30:01 02:30:01     0.03    0.01  712.00
utmp_upd   adm       ?            02:30:01 02:30:01     0.01    0.01  824.00
utmp_upd   adm       ?            02:30:01 02:30:01     0.01    0.01  912.00
utmp_upd   adm       ?            02:30:01 02:30:01     0.01    0.01  920.00
utmp_upd   adm       ?            02:30:01 02:30:01     0.01    0.01 1136.00
utmp_upd   adm       ?            02:30:01 02:30:01     0.01    0.01  576.00
closewtm   adm       ?            02:30:01 02:30:01     0.10    0.01  664.00

The following list describes each field:

COMMAND NAME: Command name (pound (#) sign if the command was executed with superuser privileges)
USER: User name
TTYNAME: tty name (listed as ? if unknown)
START TIME: Command execution starting time
END TIME: Command execution ending time
REAL (SECS): Real time (in seconds)
CPU (SECS): CPU time (in seconds)
MEAN SIZE (K): Mean size (in Kbytes)

You can obtain the following information by using acctcom command options.

State of the fork/exec flag (1 for fork without exec)
System exit status
Hog factor
Total kcore minutes
CPU factor
Characters transferred
Blocks read

The following list describes the acctcom command options.

-a: Shows average statistics about the processes selected. The statistics are printed after the output is recorded.
-b: Reads the files backward, showing latest commands first. This option has no effect if reading standard input.
-f: Prints the fork/exec flag and system exit status columns. The output is an octal number.
-h: Instead of mean memory size, shows the hog factor, which is the fraction of total available CPU time consumed by the process during its execution. Hog factor = total-CPU-time/elapsed-time.
-i: Prints columns that contains the I/O counts in the output.
-k: Shows total kcore minutes instead of memory size.
-m: Shows mean core size. This size is the default.
-q: Prints average statistics, not output records.
-r: Shows CPU factor: user-time/(system-time + user-time).
-t: Shows separate system and user CPU times.
-v: Excludes column headings from the output.
-C sec: Shows only processes with total CPU time (system plus user) that exceeds sec seconds.
-e time: Shows processes existing at or before time, given in the format hr[:min[:sec]].
-E time: Shows processes starting at or before time, given in the format hr[:min[:sec]]. Using the same time for both -S and -E, shows processes that existed at the time.
-g group: Shows only processes that belong to group.
-H factor: Shows only processes that exceed factor, where factor is the “hog factor” (see the -h option).
-I chars: Shows only processes that transferred more characters than the cutoff number specified by chars.
-l line: Show only processes that belong to the terminal /dev/line.
-n pattern: Shows only commands that match pattern (a regular expression except that “+” means one or more occurrences).
-o ofile: Instead of printing the records, copies them in acct.h format to ofile.
-O sec: Shows only processes with CPU system time that exceeds sec seconds.
-s time: Show processes existing at or after time, given in the format hr[:min[:sec]].
-S time: Show processes starting at or after time, given in the format hr[:min[:sec]].
-u user: Shows only processes that belong to user.

System Accounting Files

The /var/adm directory contains the active data collection files. The following list describes the accounting files in this directory.

dtmp: Output from the acctdusg program
fee: Output from the chargefee program, which are the ASCII tacct records
pacct: Active process accounting file
pacct n: Process accounting files that are switched by running the turnacct script
Spacctn.MMDD: Process accounting files for MMDD during execution of the runacct script

The /var/adm/acct directory contains the nite, sum, and fiscal directories. These directories contain the actual data collection files. For example, the nite directory contains files that are reused daily by the runacct script. A brief summary of the files in the /var/adm/acct/nite directory follows.

Table 10–4 Files in the /var/adm/acct/nite Directory


File	Description
`active`	Used by the `runacct` script to record progress and print warning and error messages
`active`.`MMDD`	Same as the `active` file after the `runacct` script detects an error
`cms`	ASCII total command summary used by the `prdaily` script
`ctacct.MMDD`	Connect accounting records in `tacct.h` format
`ctmp`	Output of `acctcon1` program, which consists of connect session records in `ctmp.h` format (`acctcon1` and `acctcon2` are provided for compatibility purposes)
`daycms`	ASCII daily command summary used by the `prdaily` script
`daytacct`	Total accounting records for one day in `tacct.h` format
`disktacct`	Disk accounting records in `tacct.h` format, created by the `dodisk` script
`fd2log`	Diagnostic output during execution of the `runacct` script
`lastdate`	Last day the `runacct` script executed (in `date +%m%d` format)
`lineuse`	`tty` line usage report used by the `prdaily` script
`lock`	Used to control serial use of the `runacct` script
`log`	Diagnostic output from the `acctcon` program
`log.MMDD`	Same as the `log` file after the `runacct` script detects an error
`owtmpx`	Previous day's `wtmpx` file
`reboots`	Beginning and ending dates from the `wtmpx` file, and a listing of reboots
`statefile`	Used to record current state during execution of the `runacct` script
`tmpwtmp`	`wtmpx` file corrected by the `wtmpfix` program
`wtmperror`	Contains `wtmpfix` error messages
`wtmperror` `MMDD`	Same as the `wtmperror` file after the `runacct` script detects an error
`wtmpMMDD`	The `runacct` script's copy of the `wtmpx` file

The sum directory contains the cumulative summary files updated by the runacct script and used by the monacct script. The following table summarizes the files in the /var/adm/acct/sum directory.

Table 10–5 Files in the /var/adm/acct/sum Directory


File	Description
`cms`	Total command summary file for current fiscal period in binary format
`cmsprev`	Command summary file without latest update
`daycms`	Command summary file for the day's usage in internal summary format
`loginlog`	Record of last date each user logged in; created by the `lastlogin` script and used in the `prdaily` script
`rprt.MMDD`	Saved output of `prdaily` script
`tacct`	Cumulative total accounting file for current fiscal period
`tacctprev`	Same as the `tacct` file without latest update
`tacct.MMDD`	Total accounting file for `MMDD`

The fiscal directory contains periodic summary files that are created by the monacct script. The following table summarizes the files in the /var/adm/acct/fiscal directory.

Table 10–6 Files in the /var/adm/acct/fiscal Directory


File	Description
`cmsn`	Total command summary file for fiscal period `n` in internal summary format
`fiscrptn`	Report similar to `rprtn` for fiscal period `n`
`tacctn`	Total accounting file for fiscal period `n`

Files Produced by the `runacct` Script

The following table summarizes the most useful files produced by the runacct script. These files are found in the /var/adm/acct directory.

Table 10–7 Files Created by the runacct Script


File	Description
`nite/daytacct`	The total accounting file for the day in `tacct.h` format.
`nite/lineuse`	The `runacct` script calls the `acctcon` program to gather data on terminal line usage from the `/var/adm/acct/nite/tmpwtmp` file and writes the data to the `/var/adm/acct/nite/lineuse` file. The `prdaily` script uses this data to report line usage. This report is especially useful for detecting bad lines. If the ratio between the number of logouts to logins is greater than three to one, the line is very likely failing.
`sum/cms`	This file is the accumulation of each day's command summaries. The accumulation restarts when the `monacct` script is executed. The ASCII version is the `nite/cms` file.
`sum/daycms`	The `runacct` script calls the `acctcms` program to process the commands used during the day to create the Daily Command Summary report and stores the data in the `/var/adm/acct/sum/daycms` file. The ASCII version is the `/var/adm/acct/nite/daycms` file.
`sum/loginlog`	The `runacct` script calls the `lastlogin` script to update the last date logged in for the logins in the `/var/adm/acct/sum/loginlog` file. The `lastlogin` command also removes from this file any logins that are no longer valid.
`sum/rprt.MMDD`	Each execution of the `runacct` script saves a copy of the daily report that was printed by the `prdaily` script.
`sum/tacct`	Contains the accumulation of each day's `nite/daytacct` data and is used for billing purposes. The `monacct` script restarts accumulating this data each month or fiscal period.