System Administration Guide: Resource Management and Network Services

Chapter 7 Extended Accounting

By using the project and task facilities that are described in Chapter 6, Projects and Tasks to label and separate workloads, you can monitor resource consumption by each workload. You can use the extended accounting subsystem to capture a detailed set of resource consumption statistics on both running processes and tasks. The extended accounting subsystem labels the usage records with the project for which the work was done. You can also use extended accounting, in conjunction with the Internet Protocol Quality of Service (IPQoS) flow accounting module described in “Using Flow Accounting and Statistics Gathering (Tasks)” in IPQoS Administration Guide, to capture network flow information on a system.

To begin using extended accounting, see How to Activate Extended Accounting for Processes, Tasks, and Flows.

Overview

Before you can apply resource management mechanisms, you must first be able to characterize the resource consumption demands that various workloads place on a system. The extended accounting facility in the Solaris operating environment provides a flexible way to record system and network resource consumption on a task or process basis, or on the basis of selectors provided by IPQoS (see ipqos(7IPP)). Unlike online monitoring tools, which measure system usage in real time, extended accounting enables you to examine historical usage. You can then make assessments of capacity requirements for future workloads.

With extended accounting data available, you can develop or purchase software for resource chargeback, workload monitoring, or capacity planning.

How Extended Accounting Works

The extended accounting facility in the Solaris environment uses a versioned, extensible file format to contain accounting data. Files that use this data format can be accessed or be created by using the API provided in the included library, libexacct(3LIB). These files can then be analyzed on any platform with extended accounting enabled, and their data can be used for capacity planning and chargeback.

If extended accounting is active, statistics are gathered that can be examined by the libexacct API. libexacct allows examination of the exacct files either forward or backward. The API supports third-party files that are generated by libexacct as well as those files that are created by the kernel.There is a Practical Extraction and Report Language (Perl) interface to libexacct that enables you to develop customized reporting and extraction scripts. See Perl Interface to libexacct.

With extended accounting enabled, the task tracks the aggregate resource usage of its member processes. A task accounting record is written at task completion. Interim records can also be written. For more information on tasks, see Chapter 6, Projects and Tasks.

Figure 7–1 Task Tracking With Extended Accounting Activated

Flow diagram shows how aggregate resource usage of a task's processes is captured in the record that is written at task completion.

Extensible Format

The extended accounting format is substantially more extensible than the SunOSTM legacy system accounting software format (see “What is System Accounting?” in System Administration Guide: Advanced Administration). Extended accounting permits accounting metrics to be added and removed from the system between releases, and even during system operation.


Note –

Both extended accounting and legacy system accounting software can be active on your system at the same time.


exacct Records and Format

Routines that allow exacct records to be created serve two purposes.

The format permits different forms of accounting records to be captured without requiring that every change be an explicit version change. Well-written applications that consume accounting data must ignore records they do not understand.

The libexacct library converts and produces files in the exacct format. This library is the only supported interface to exacct format files.


Note –

The getacct, putacct, and wracct system calls do not apply to flows. The kernel creates flow records and writes them to the file when IPQoS flow accounting is configured.


Extended Accounting Configuration

The /etc/acctadm.conf file contains the current extended accounting configuration. The file is edited through the acctadm interface, not by the user.

The directory /var/adm/exacct is the standard location for placing extended accounting data. You can use the acctadm(1M) command to specify a different location for the process and task accounting-data files.

Commands Used With Extended Accounting

Command 

Description 

acctadm(1M)

Modifies various attributes of the extended accounting facility, stops and starts extended accounting, and is used to select accounting attributes to track for processes, tasks, and flows. 

wracct(1M)

Writes extended accounting records for active processes and active tasks. 

lastcomm(1)

Displays previously invoked commands. lastcomm can consume either standard accounting-process data or extended-accounting process data.

For information on commands that are associated with tasks and projects, see Commands Used to Administer Projects and Tasks. For information on IPQoS flow accounting, see ipqosconf(1M).

Perl Interface to libexacct

The Perl interface allows you to create Perl scripts that can read the accounting files produced by the exacct framework. You can also create Perl scripts that write exacct files.

The interface is functionally equivalent to the underlying C API. When possible, the data obtained from the underlying C API is presented as Perl data types. This feature makes accessing the data easier and it removes the need for buffer pack and unpack operations. Moreover, all memory management is performed by the Perl library.

The various project, task, and exacct-related functions are separated into groups. Each group of functions is located in a separate Perl module. Each module begins with the Sun standard Sun::Solaris:: Perl package prefix. All of the classes provided by the Perl exacct library are found under the Sun::Solaris::Exacct module.

The underlying libexacct(3LIB) library provides operations on exacct format files, catalog tags, and exacct objects. exacct objects are subdivided into two types:

The following table summarizes each of the modules.

Module 

Description 

For More Information 

Sun::Solaris::Project

This module provides functions to access the project manipulation functions getprojid(2), endprojent(3PROJECT) , fgetprojent(3PROJECT), getdefaultproj(3PROJECT), getprojbyid(3PROJECT), getprojbyname(3PROJECT), getprojent(3PROJECT), getprojidbyname(3PROJECT), inproj(3PROJECT), project_walk(3PROJECT), setproject(3PROJECT), and setprojent(3PROJECT).

Project(3PERL)

Sun::Solaris::Task

This module provides functions to access the task manipulation functions gettaskid(2) and settaskid(2).

Task(3PERL)

Sun::Solaris::Exacct

This module is the top-level exacct module. This module provides functions to access the exacct-related system calls getacct(2), putacct(2), and wracct(2). This module also provides functions to access the libexacct(3LIB) library function ea_error(3EXACCT). Constants for all of the exacct EO_*, EW_*, EXR_*, P_*, and TASK_* macros are also provided in this module.

Exacct(3PERL)

Sun::Solaris::Exacct:: Catalog

This module provides object-oriented methods to access the bitfields in an exacct catalog tag. This module also provides access to the constants for the EXC_*, EXD_*, and EXD_* macros.

Exacct::Catalog(3PERL)

Sun::Solaris::Exacct:: File

This module provides object-oriented methods to access the libexacct accounting file functions ea_open(3EXACCT), ea_close(3EXACCT), ea_get_creator(3EXACCT), ea_get_hostname(3EXACCT), ea_next_object(3EXACCT), ea_previous_object(3EXACCT), and ea_write_object(3EXACCT).

Exacct::File(3PERL)

Sun::Solaris::Exacct:: Object

This module provides object-oriented methods to access an individual exacct accounting file object. An exacct object is represented as an opaque reference blessed into the appropriate Sun::Solaris::Exacct::Object subclass. This module is further subdivided into the object types Item and Group. At this level, there are methods to access the ea_match_object_catalog(3EXACCT) and ea_attach_to_object(3EXACCT) functions.

Exacct::Object(3PERL)

Sun::Solaris::Exacct:: Object::Item

This module provides object-oriented methods to access an individual exacct accounting file Item. Objects of this type inherit from Sun::Solaris::Exacct::Object.

Exacct::Object::Item(3PERL)

Sun::Solaris::Exacct:: Object::Group

This module provides object-oriented methods to access an individual exacct accounting file Group. Objects of this type inherit from Sun::Solaris::Exacct::Object. These objects provide access to the ea_attach_to_group(3EXACCT) function. The Items contained within the Group are presented as a Perl array.

Exacct::Object::Group(3PERL)

For examples that show how to use the modules described in the previous table, see Using the Perl Interface to libexacct.

Using Extended Accounting Functionality

How to Activate Extended Accounting for Processes, Tasks, and Flows

To activate the extended accounting facility for tasks, processes, and flows, use the acctadm(1M) command. The optional final parameter to acctadm indicates whether the command should act on the process, system task, or flow accounting components of the extended accounting facility.

  1. Become superuser.

  2. Activate extended accounting for processes.


    # acctadm -e extended -f /var/adm/exacct/proc process 
    
  3. Activate extended accounting for tasks.


    # acctadm -e extended,mstate -f /var/adm/exacct/task task
    
  4. Activate extended accounting for flows.


    # acctadm -e extended -f /var/adm/exacct/flow flow
    

How to Activate Extended Accounting With a Startup Script

Activate extended accounting on an ongoing basis by linking the /etc/init.d/acctadm script into /etc/rc2.d.


# ln -s /etc/init.d/acctadm /etc/rc2.d/Snacctadm
# ln -s /etc/init.d/acctadm /etc/rc2.d/Knacctadm

The n variable is replaced by a number.

See Extended Accounting Configuration for information on accounting configuration.

How to Display Extended Accounting Status

Type acctadm without arguments to display the current status of the extended accounting facility.


# acctadm
                 Task accounting: active
            Task accounting file: /var/adm/exacct/task
          Tracked task resources: extended
        Untracked task resources: none
              Process accounting: active
         Process accounting file: /var/adm/exacct/proc
       Tracked process resources: extended
     Untracked process resources: host,mstate
                 Flow accounting: active
            Flow accounting file: /var/adm/exacct/flow
          Tracked flow resources: extended
        Untracked flow resources: none

In the previous example, system task accounting is active in extended mode and mstate mode. Process and flow accounting are active in extended mode.


Note –

In the context of extended accounting, microstate (mstate) refers to the extended data, associated with microstate process transitions, that is available in the process usage file (see proc(4)). This data provides much more detail about the activities of the process than basic or extended records.


How to View Available Accounting Resources

Available resources can vary from system to system, and from platform to platform. Use the -r option to view the available accounting resources on the system.


# acctadm -r
process:
extended pid,uid,gid,cpu,time,command,tty,projid,taskid,ancpid,wait-status,flag
basic    pid,uid,gid,cpu,time,command,tty,flag
task:
extended taskid,projid,cpu,time,host,mstate,anctaskid
basic    taskid,projid,cpu,timeprocess:
extended pid,uid,gid,cpu,time,command,tty,projid,taskid,ancpid,wait-status,flag
basic    pid,uid,gid,cpu,time,command,tty,flag
task:
extended taskid,projid,cpu,time,host,mstate,anctaskid
basic    taskid,projid,cpu,time
flow:
extended 
saddr,daddr,sport,dport,proto,dsfield,nbytes,npkts,action,ctime,lseen,projid,uid
basic    saddr,daddr,sport,dport,proto,nbytes,npkts,action

How to Deactivate Process, Task, and Flow Accounting

To deactivate process, task, and flow accounting, turn off each of them individually.

  1. Become superuser.

  2. Turn off process accounting.


    # acctadm -x process 
    
  3. Turn off task accounting.


    # acctadm -x task
    
  4. Turn off flow accounting.


    # acctadm -x flow
    
  5. Verify that task accounting, process accounting, and flow accounting have been turned off.


    	# acctadm
                Task accounting: inactive
           Task accounting file: none
         Tracked task resources: extended
       Untracked task resources: none
             Process accounting: inactive
        Process accounting file: none
      Tracked process resources: extended
    Untracked process resources: host,mstate
                Flow accounting: inactive
           Flow accounting file: none
         Tracked flow resources: extended
       Untracked flow resources: none

Using the Perl Interface to libexacct

How to Recursively Print the Contents of an exacct Object

Use the following code to recursively print the contents of an exacct object. Note that this capability is provided by the library as the Sun::Solaris::Exacct::Object::dump() function. This capability is also available through the ea_dump_object() convenience function.


sub dump_object
     {
             my ($obj, $indent) = @_;
             my $istr = '  ' x $indent;

             #
             # Retrieve the catalog tag.  Because we are 
             # doing this in an array context, the
             # catalog tag will be returned as a (type, catalog, id) 
             # triplet, where each member of the triplet will behave as 
             # an integer or a string, depending on context.
             # If instead this next line provided a scalar context, e.g.
             #    my $cat  = $obj->catalog()->value();
             # then $cat would be set to the integer value of the 
             # catalog tag.
             #
             my @cat = $obj->catalog()->value();

             #
             # If the object is a plain item
             #
             if ($obj->type() == &EO_ITEM) {
                     #
                     # Note: The '%s' formats provide s string context, so
                     # the components of the catalog tag will be displayed
                     # as the symbolic values. If we changed the '%s'
                     # formats to '%d', the numeric value of the components
                     # would be displayed.
                     #
                     printf("%sITEM\n%s  Catalog = %s|%s|%s\n", 
                        $istr, $istr, @cat);
                     $indent++;

                     #
                     # Retrieve the value of the item.  If the item contains
                     # in turn a nested exacct object (i.e., an item or
                     # group),then the value method will return a reference
                     # to the appropriate sort of perl object
                     # (Exacct::Object::Item or Exacct::Object::Group).
                     # We could of course figure out that the item contained
                     # a nested item orgroup by examining the catalog tag in
                     # @cat and looking for a type of EXT_EXACCT_OBJECT or
                     # EXT_GROUP.
                     #
                     my $val = $obj->value();
                     if (ref($val)) {
                             # If it is a nested object, recurse to dump it.
                             dump_object($val, $indent);
                     } else {
                             # Otherwise it is just a 'plain' value, so
                             # display it.
                             printf("%s  Value = %s\n", $istr, $val);
                     }

             #
             # Otherwise we know we are dealing with a group.  Groups
             # represent contents as a perl list or array (depending on
             # context), so we can process the contents of the group
             # with a 'foreach' loop, which provides a list context.
             # In a list context the value method returns the content
             # of the group as a perl list, which is the quickest
             # mechanism, but doesn't allow the group to be modified.
             # If we wanted to modify the contents of the group we could
             # do so like this:
             #    my $grp = $obj->value();   # Returns an array reference
             #    $grp->[0] = $newitem;
             # but accessing the group elements this way is much slower.
             #
             } else {
                     printf("%sGROUP\n%s  Catalog = %s|%s|%s\n",
                         $istr, $istr, @cat);
                     $indent++;
                     # 'foreach' provides a list context.
                     foreach my $val ($obj->value()) {
                             dump_object($val, $indent);
                     }
                     printf("%sENDGROUP\n", $istr);
             }
     }

How to Create a New Group Record and Write It to a File

Use this script to create a new group record and write it to a file named /tmp/exacct.


#!/usr/perl5/5.6.1/bin/perl

use strict;
use warnings;
use Sun::Solaris::Exacct qw(:EXACCT_ALL);
# Prototype list of catalog tags and values.
     my @items = (
             [ &EXT_STRING | &EXC_DEFAULT | &EXD_CREATOR      => "me"       ],
             [ &EXT_UINT32 | &EXC_DEFAULT | &EXD_PROC_PID     => $$         ],
             [ &EXT_UINT32 | &EXC_DEFAULT | &EXD_PROC_UID     => $<         ],
             [ &EXT_UINT32 | &EXC_DEFAULT | &EXD_PROC_GID     => $(         ],
             [ &EXT_STRING | &EXC_DEFAULT | &EXD_PROC_COMMAND => "/bin/rec" ],
     );

     # Create a new group catalog object.
     my $cat = ea_new_catalog(&EXT_GROUP | &EXC_DEFAULT | &EXD_NONE)

     # Create a new Group object and retrieve its data array.
     my $group = ea_new_group($cat);
     my $ary = $group->value();

     # Push the new Items onto the Group array.
     foreach my $v (@items) {
             push(@$ary, ea_new_item(ea_new_catalog($v->[0]), $v->[1]));
     }

     # Open the exacct file, write the record & close.
     my $f = ea_new_file('/tmp/exacct', &O_RDWR | &O_CREAT | &O_TRUNC)
        || die("create /tmp/exacct failed: ", ea_error_str(), "\n");
     $f->write($group);
     $f = undef;

How to Print the Contents of an exacct File

Use the following Perl script to print the contents of an exacct file.


#!/usr/perl5/5.6.1/bin/perl

     use strict;
     use warnings;
     use Sun::Solaris::Exacct qw(:EXACCT_ALL);

     die("Usage is dumpexacct <exacct file>\n") unless (@ARGV == 1);

     # Open the exact file and display the header information.
     my $ef = ea_new_file($ARGV[0], &O_RDONLY) || die(error_str());
     printf("Creator:  %s\n", $ef->creator());
     printf("Hostname: %s\n\n", $ef->hostname());

     # Dump the file contents
     while (my $obj = $ef->get()) {
             ea_dump_object($obj);
     }

     # Report any errors
     if (ea_error() != EXR_OK && ea_error() != EXR_EOF)  {
             printf("\nERROR: %s\n", ea_error_str());
             exit(1);
     }
     exit(0);

Example Output From Sun::Solaris::Exacct::Object->dump()

Here is example output produced by running Sun::Solaris::Exacct::Object->dump() on the file created in How to Create a New Group Record and Write It to a File.


Creator:  root
Hostname: localhost
GROUP
       Catalog = EXT_GROUP|EXC_DEFAULT|EXD_NONE
       ITEM
         Catalog = EXT_STRING|EXC_DEFAULT|EXD_CREATOR
         Value = me
       ITEM
         Catalog = EXT_UINT32|EXC_DEFAULT|EXD_PROC_PID
         Value = 845523
       ITEM
         Catalog = EXT_UINT32|EXC_DEFAULT|EXD_PROC_UID
         Value = 37845
       ITEM
         Catalog = EXT_UINT32|EXC_DEFAULT|EXD_PROC_GID
         Value = 10
       ITEM
         Catalog = EXT_STRING|EXC_DEFAULT|EXD_PROC_COMMAND
         Value = /bin/rec
ENDGROUP