System Administration Guide: Advanced Administration

Chapter 9 Managing System Accounting (Tasks)

This chapter describes how to set up and maintain system accounting.

This is a list of the overview information in this chapter.

For information on using extended accounting, see Chapter 4, Extended Accounting (Overview), in System Administration Guide: Oracle Solaris Containers-Resource Management and Oracle Solaris Zones.

For information on the step-by-step procedures that are associated with system accounting, see System Accounting (Task Map).

For reference information about the various system accounting reports, see Chapter 10, System Accounting (Reference).

What's New in System Accounting

This section describes new or changed features in system accounting in the Oracle Solaris release. For a complete listing of new features and a description of Oracle Solaris releases, see Oracle Solaris 10 9/10 What’s New.

Oracle Solaris Process Accounting and Statistics Improvements

Oracle Solaris 10: Changes have been made to the internals of the load averaging, cpu usr/sys/idle, and accounting functions. Microstate accounting has replaced the old accounting mechanism and is enabled by default all of the time. As a result, you might notice slightly different process usage and timing statistics.

Switching to microstate accounting provides substantially more accurate data about user processes and the amount of time they spend in various states. In addition, this information is used to generate more accurate load averages and statistics from the /proc file system. For more information, see the proc(4) man page.

What is System Accounting?

System accounting software in the Oracle Solaris OS is a set of programs that enables you to collect and record data about user connect time, CPU time charged to processes, and disk usage. Once you collect this data, you can generate reports and charge fees for system usage.

You can use system accounting on a daily or monthly basis. Or, you can tack disk usage per user.

You can use the accounting programs to perform these tasks:

After you set up the system accounting programs, they run mostly on their own.

How System Accounting Works

Automatic accounting is set up by first putting the accounting startup script into root's crontab file. The accounting startup script can then be started automatically by the cron command.

The following overview describes the system accounting process.

  1. Between system startup and shutdown, raw data about system use (such as user logins, running processes, and data storage) are collected in accounting files.

  2. Periodically (usually once a day), the /usr/lib/acct/runacct script processes the various accounting files and produces both cumulative summary files and daily accounting reports. Then, the /usr/lib/acct/prdaily script prints the daily reports.

    For more information about the runacct script, see runacct Script.

  3. Monthly, you can process and print the cumulative runacct summary files by executing the monacct script. The summary reports produced by the monacct script provide an efficient means for billing users on a monthly or other fiscal basis.

System Accounting Components

The system accounting software provides C language programs and shell scripts that organize data into summary files and reports. These programs reside in the /usr/lib/acct directory. The accounting reports reside in the /var/adm/acct directory.

Daily accounting can help you perform four types of auditing:

Connect Accounting

Connect accounting enables you to determine the following information:

To provide this information on connect sessions, the system stores the following data

These records are produced from the output of system programs such as date, init, login, ttymon, and acctwtmp. They are stored in the /var/adm/wtmpx file.

Entries in the wtmpx file can contain the following information:

Process Accounting

Process accounting enables you to keep track of the following data about each process that runs on your system:

Every time a process terminates, the exit program collects this information and writes it to the /var/adm/pacct file.

Disk Accounting

Disk accounting enables you to gather and format the following data about the files each user has on disks:

This data is collected by the /usr/lib/acct/dodisk shell script at intervals that are determined by the entry you add to the /var/spool/cron/crontabs/root file. In turn, the dodisk script invokes the acctdisk and acctdusg commands. These commands gather disk usage by login name.


Caution – Caution –

Information gathered by running the dodisk script is stored in the /var/adm/acct/nite/disktacct file. This information is overwritten the next time the dodisk script is run. Therefore, avoid running the dodisk script twice in the same day.


The acctdusg command might overcharge for files that are written randomly, which can create holes in the files. This problem occurs because the acctdusg command does not read the indirect blocks of a file when determining the file size. Rather, the acctdusg command determines the file size by checking the current file size value in the file's inode.

Fee Calculations

The chargefee utility stores charges for special services that are provided to a user in the /var/adm/fee file. A special service, for example, is file restoration. Each entry in the file consists of a user login name, user ID, and the fee. This file is checked by the runacct script every day, and new entries are merged into the accounting records. For instructions on running the chargefee script to bill users, see How to Bill Users.

How Daily Accounting Works

Here is a step-by-step summary of how daily accounting works:

  1. When the system is switched into multiuser mode, the /usr/lib/acct/startup program is executed. The startup program executes several other programs that invoke daily accounting.

  2. The acctwtmp program adds a “boot” record to the /var/adm/wtmpx file. In this record, the system name is shown as the user name in the wtmpx record. The following table summarizes how the raw accounting data is gathered and where it is stored.

    File in /var/adm

    Information Stored 

    Written By 

    Format 

    wtmpx

    Connect sessions  

    login, init

    Binary 

     

    Changes

    date

    Binary 

     

    Reboots

    acctwtmp

    Binary 

     

    Shutdowns

    shutacct

    Binary 

    pacctn

    Processes 

    Kernel (when the process ends)  

    Binary 

     

     

    turnacct switch (which creates a new file when the old file reaches 500 blocks)

    Binary 

    fee

    Special charges 

    chargefee

    ASCII 

    acct/nite/disktacct

    Disk space used 

    dodisk

    Binary 

  3. The turnacct script, invoked with the -on option, begins process accounting. Specifically, the turnacct script executes the accton program with the /var/adm/pacct argument.

  4. The remove shell script “cleans up” the saved pacct and wtmpx files that are left in the sum directory by the runacct script.

  5. The login and init programs record connect sessions by writing records into the /var/adm/wtmpx file. Date changes (using date with an argument) are also written to the /var/adm/wtmpx file. Reboots and shutdowns using the acctwtmp command are also recorded in the /var/adm/wtmpx file.

  6. When a process ends, the kernel writes one record per process, using the acct.h format, in the /var/adm/pacct file.

    Every hour, the cron command executes the ckpacct script to check the size of the /var/adm/pacct file. If the file grows beyond 500 blocks (default), the turnacct switch command is executed. (The program moves the pacct file to the pacctn file and creates a new file.) The advantage of having several smaller pacct files becomes apparent when you try to restart the runacct script if a failure occurs when processing these records.

  7. The runacct script is executed by the cron command each night. The runacct script processes the accounting files to produce command summaries and usage summaries by user name. These accounting files are processed: /var/adm/pacctn, /var/adm/wtmpx, /var/adm/fee, and /var/adm/acct/nite/disktacct.

  8. The /usr/lib/acct/prdaily script is executed on a daily basis by the runacct script to write the daily accounting information in the /var/adm/acct/sum/rprtMMDD files.

  9. The monacct script should be executed on a monthly basis (or at intervals you determine, such as at the end of every fiscal period). The monacct script creates a report that is based on data stored in the sum directory that has been updated daily by the runacct script. After creating the report, the monacct script “cleans up” the sum directory to prepare the directory's files for the new runacct data.

What Happens If the System Shuts Down

If the system is shut down by using the shutdown command, the shutacct script is executed automatically. The shutacct script writes a reason record into the /var/adm/wtmpx file and turns off process accounting.

System Accounting (Task Map)

Task 

Description 

For Instructions 

Set up system accounting. 

Set up system accounting by performing the following tasks:

  • Create the /etc/rc0.d/K22acct and /etc/rc2.d/S22acct files.

  • Modify the /var/spool/cron/crontabs/adm and /var/spool/cron/crontabs/root crontab files.

How to Set Up System Accounting

Bill users. 

Run the /usr/lib/acct/chargefee username amount command.

How to Bill Users

Fix a corrupted wtmpx file.

Convert the wtmpx file from binary to ASCII format.

How to Fix a Corrupted wtmpx File

Fix tacct errors.

Run the prtacct script to check the /var/adm/acct/sum/tacctprev file. Then, patch the latest/var/adm/acct/sum/tacctMMDD file. You will need to re-create the /var/adm/acct/sum/tacct file.

How to Fix tacct Errors

Restart the runacct script.

Remove the lastdate file and any lock files. Then, manually restart the runacct script.

How to Restart the runacct Script

Disable system accounting temporarily. 

Edit theadm crontab file to stop the ckpacct, runacct, and monacct programs from running.

How to Temporarily Stop System Accounting

Disable system accounting permanently. 

Delete the entries for the ckpacct, runacct, and monacct programs in the adm and crontab files.

How to Permanently Disable System Accounting

Setting Up System Accounting

You can set up system accounting to run while the system is in multiuser mode (Run Level 2). Generally, this task involves these steps:

  1. Creating the /etc/rc0.d/K22acct and /etc/rc2.d/S22acct startup scripts

  2. Modifying the /var/spool/cron/crontabs/adm and /var/spool/cron/crontabs/root crontab files

The following table describes the default accounting scripts.

Table 9–1 Default Accounting Scripts

Purpose 

Accounting Script 

Man Page 

Run Frequency 

Checks the size of the /usr/adm/pacct log file and makes sure that it does not get too large.

ckpacct

acctsh(1M)

Periodically 

Processes connect, disk, and fee accounting information. You can remove from this script the commands for the accounting features you do not want processed. 

runacct

runacct(1M)

Daily 

Generates fiscal accounting summary reports on a monthly basis. You can determine how often this script is run. You can remove from this script the commands for the accounting features you do not want to use. 

monacct

acctsh(1M)

On a fiscal basis 

You can choose which accounting scripts run by default. After these entries have been added to the crontab files, system accounting should run automatically.

ProcedureHow to Set Up System Accounting

  1. Become superuser or assume an equivalent role.

    Roles contain authorizations and privileged commands. For more information about roles, see Configuring RBAC (Task Map) in System Administration Guide: Security Services.

  2. If necessary, install the SUNWaccr and SUNWaccu packages on your system by using the pkgadd command.

  3. Install /etc/init.d/acct as the startup script for Run Level 2.


    # ln /etc/init.d/acct /etc/rc2.d/S22acct
    
  4. Install /etc/init.d/acct as the stop script for Run Level 0.


    # ln /etc/init.d/acct /etc/rc0.d/K22acct
    
  5. Add the following lines to the adm crontab file to start the ckpacct, runacct, and monacct scripts automatically.


    # EDITOR=vi; export EDITOR
    # crontab -e adm
    0 * * * * /usr/lib/acct/ckpacct
    30 2 * * * /usr/lib/acct/runacct 2> /var/adm/acct/nite/fd2log
    30 7 1 * * /usr/lib/acct/monacct
  6. Add the following line to the root crontab file to start the dodisk script automatically.


    # crontab -e
    30 22 * * 4 /usr/lib/acct/dodisk
  7. Edit /etc/acct/holidays to include national holidays and local holidays.

    For more information, see the holidays(4) man page and the example that follows.

  8. Reboot the system, or start system accounting manually by typing:


    # /etc/init.d/acct start
    

Example 9–1 Setting Up Accounting (adm crontab)

This modified adm crontab contains entries for the ckpacct, runacct, and monacct scripts.


#ident  "@(#)adm        1.5     92/07/14 SMI"    /* SVr4.0 1.2   */
#
# The adm crontab file should contain startup of performance 
# collection if the profiling and performance feature has been 
# installed.
0 * * * * /usr/lib/acct/ckpacct
30 2 * * * /usr/lib/acct/runacct 2> /var/adm/acct/nite/fd2log
30 7 1 * * /usr/lib/acct/monacct


Example 9–2 Setting Up Accounting (root crontab)

This modified root crontab contains entries for the dodisk program.


#ident  "@(#)root       1.19    98/07/06 SMI"   /* SVr4.0 1.1.3.1       */
#
# The root crontab should be used to perform accounting data collection.
#
#
10 3 * * * /usr/sbin/logadm
15 3 * * 0 /usr/lib/fs/nfs/nfsfind
30 3 * * * [ -x /usr/lib/gss/gsscred_clean ] && /usr/lib/gss/gsscred_clean
30 22 * * 4 /usr/lib/acct/dodisk


Example 9–3 Setting Up Accounting (/etc/acct/holidays)

The following example shows a sample /etc/acct/holidays file.


* @(#)holidays	January 1, 2004
*
* Prime/Nonprime Table for UNIX Accounting System
*
* Curr	Prime	Non-Prime
* Year	Start	Start
*
  2004	0800	1800
*
* only the first column (month/day) is significant.
*
* month/day	Company
* 		Holiday
*
1/1		New Years Day
7/4		Indep. Day
12/25		Christmas

Billing Users

If you provide special user services by request, you might want to bill users by running the chargefee utility. Special services include restoring files or remote printing. The chargefee utility records charges in the /var/adm/fee file. Each time the runacct utility is executed, new entries are merged into the total accounting records.

See the acctsh(1M) man page for more information.

ProcedureHow to Bill Users

  1. Become superuser or assume an equivalent role.

    Roles contain authorizations and privileged commands. For more information about roles, see Configuring RBAC (Task Map) in System Administration Guide: Security Services.

  2. Charge a user for special services.


    # /usr/lib/acct/chargefee username amount
    
    username

    Specifies the user account that you want to bill.

    amount

    Specifies the number of units to bill the user. This value is an arbitrary unit that you set to charge users based on some task such as printing or restoring a file. You would have to write a script that invokes the chargefee utility and charges a user for a specific task.


Example 9–4 Billing Users

In the following example, the user print_customer is charged 10 units.


# /usr/lib/acct/chargefee print_customer 10

Maintaining Accounting Information

This section describes how to fix corrupted system accounting files and how to restart the runacct script.

Fixing Corrupted Files and wtmpx Errors

Unfortunately, system accounting is not foolproof. Occasionally, a file becomes corrupted or lost. Some files can simply be ignored or restored from backup. However, certain files must be fixed to maintain the integrity of system accounting.

The wtmpx files seem to cause the most problems in the daily operation of system accounting. When the date is changed manually and the system is in multiuser mode, a set of date change records is written to the /var/adm/wtmpx file. The wtmpfix utility is designed to adjust the time stamps in the wtmp records when a date change is encountered. However, some combinations of date changes and reboots slip through the wtmpfix utility and cause the acctcon program to fail.

ProcedureHow to Fix a Corrupted wtmpx File

  1. Become superuser or assume an equivalent role.

    Roles contain authorizations and privileged commands. For more information about roles, see Configuring RBAC (Task Map) in System Administration Guide: Security Services.

  2. Change to the /var/adm directory.

  3. Convert the wtmpx file from binary format to ASCII format.


    # /usr/lib/acct/fwtmp < wtmpx > wtmpx.ascii 
    
  4. Edit the wtmpx.ascii file to delete the corrupted records.

  5. Convert the wtmpx.ascii file back to a binary file.


    # /usr/lib/acct/fwtmp -ic < wtmpx.ascii > wtmpx
    

    See the fwtmp(1M) man page for more information.

Fixing tacct Errors

The integrity of the /var/adm/acct/sum/tacct file is important if you are charging users for system resources. Occasionally, unusual tacct records appear with negative numbers, duplicate user IDs, or a user ID of 65535. First, check the /var/adm/acct/sum/tacctprev file by using the prtacct script to print the file. If the contents look all right, patch the latest /var/adm/acct/sum/tacctMMDD file. Then, re-create the /var/adm/acct/sum/tacct file. The following steps outline a simple patch procedure.

ProcedureHow to Fix tacct Errors

  1. Become superuser or assume an equivalent role.

    Roles contain authorizations and privileged commands. For more information about roles, see Configuring RBAC (Task Map) in System Administration Guide: Security Services.

  2. Change to the /var/adm/acct/sum directory.

  3. Convert the tacctMMDD file from binary format to ASCII format.


    # /usr/lib/acct/acctmerg -v < tacctMMDD > xtacct
    

    MMDD is pair of two-digit numbers that represent the month and day.

  4. Edit the xtacct file, removing corrupted records and writing duplicate records to another file.

  5. Convert the xtacct file from ASCII format to binary format.


    # /usr/lib/acct/acctmerg -i < xtacct > tacctMMDD
    
  6. Merge the files tacctprev and tacct.MMDD into the tacct file.


    # /usr/lib/acct/acctmerg < tacctprev tacctMMDD > tacct
    

Restarting the runacct Script

The runacct script can fail for several reasons.

The following are the most common reasons:

If the active.MMDD file exists, check it first for error messages. If the active and lock files exist, check the fd2log file for any relevant messages.

Run without arguments, the runacct script assumes that this invocation is the first invocation of the day. The argument MMDD is necessary if the runacct script is being restarted and specifies the month and day for which the runacct script reruns the accounting. The entry point for processing is based on the contents of the statefile file. To override the statefile file, include the desired state on the command line. For a description of the available states, see the runacct(1M) man page.


Caution – Caution –

When you run the runacct program manually, be sure to run it as user adm.


ProcedureHow to Restart the runacct Script

  1. Change directories to the /var/adm/acct/nite directory.


    $ cd /var/adm/acct/nite
    
  2. Remove the lastdate file and any lock* files, if any.


    $ rm lastdate lock*
    

    The lastdate file contains the date that the runacct program was last run. Restarting the runacct script in the next step re-creates this file.

  3. Restart the runacct script.


    $ /usr/lib/acct/runacct MMDD [state] 2> /var/adm/acct/nite/fd2log &
    
    MMDD

    Is the month and day specified by two-digit numbers.

    state

    Specifies a state, or starting point, where the runacct script processing should begin.

Stopping and Disabling System Accounting

You can temporarily stop system accounting or permanently disable it.

ProcedureHow to Temporarily Stop System Accounting

  1. Become superuser or assume an equivalent role.

    Roles contain authorizations and privileged commands. For more information about roles, see Configuring RBAC (Task Map) in System Administration Guide: Security Services.

  2. Edit the adm crontab file to stop the ckpacct, runacct, and monacct programs from running by commenting out the appropriate lines.


    # EDITOR=vi; export EDITOR
    # crontab -e adm
    #0 * * * * /usr/lib/acct/ckpacct
    #30 2 * * * /usr/lib/acct/runacct 2> /var/adm/acct/nite/fd2log
    #30 7 1 * * /usr/lib/acct/monacct
  3. Edit the root crontab file to stop the dodisk program from running by commenting out the appropriate line.


    # crontab -e
    #30 22 * * 4 /usr/lib/acct/dodisk
  4. Stop the system accounting program.


    # /etc/init.d/acct stop
    
  5. (Optional) Remove the newly added comment symbols from the crontab files.

  6. Restart the system accounting program to re-enable system accounting.


    # /etc/init.d/acct start
    

ProcedureHow to Permanently Disable System Accounting

  1. Become superuser or assume an equivalent role.

    Roles contain authorizations and privileged commands. For more information about roles, see Configuring RBAC (Task Map) in System Administration Guide: Security Services.

  2. Edit the adm crontab file and delete the entries for the ckpacct, runacct, and monacct programs.


    # EDITOR=vi; export EDITOR
    # crontab -e adm
    
  3. Edit the root crontab file and delete the entries for the dodisk program.


    # crontab -e
    
  4. Remove the startup script for Run Level 2.


    # unlink /etc/rc2.d/S22acct
    
  5. Remove the stop script for Run Level 0.


    # unlink /etc/rc0.d/K22acct
    
  6. Stop the system accounting program.


    # /etc/init.d/acct stop