C H A P T E R  6

The er_print Command Line Performance Analysis Tool

This chapter explains how to use the er_print utility for performance analysis. The er_print utility prints an ASCII version of the various displays supported by the Performance Analyzer. The information is written to standard output unless you redirect it to a file or printer. You must give the er_print utility the name of one or more experiments or experiment groups generated by the Collector as arguments. Using the er_print utility you can display metrics of performance for functions, callers and callees; source code and disassembly listings; sampling information; address-space data; and execution statistics.

This chapter covers the following topics.

For a description of the data collected by the Collector, see .

For instructions on how to use the Performance Analyzer to display information in a graphical format, see .


er_print Syntax

The command-line syntax for er_print is as follows.

er_print [ -script script | -command | - | -V ] experiment-list 

The options for er_print are listed in TABLE 6-1.

TABLE 6-1 Options for the er_print Command

Option

Description

-

Read er_print commands entered from the keyboard.

-script script

Read commands from the file script, which contains a list of er_print commands, one per line. If the -script option is not present, er_print reads commands from the terminal or from the command line.

-command [argument]

Process the given command.

-V

Display version information and exit.


Multiple options can appear on the er_print command line. They are processed in the order they appear. You can mix scripts, hyphens, and explicit commands in any order. The default action if you do not supply any commands or scripts is to enter interactive mode, in which commands are entered from the keyboard. To exit interactive mode type quit or Ctrl-D.

The commands accepted by er_print are listed in the following sections. You can abbreviate any command with a shorter string as long as the command is unambiguous.


Metric Lists

Many of the er_print commands use a list of metric keywords. The syntax of the list is as follows.

metric-keyword-1[:metric-keyword2...]

Except for the size, address, and name keywords, a metric keyword consists of three parts: a metric type string, a metric visibility string, and a metric name string. These are joined with no spaces, as follows.

<type><visibility><name>

The metric type and metric visibility strings are composed of type and visibility characters.

The allowed metric type characters are given in TABLE 6-2. A metric keyword that contains more than one type character is expanded into a list of metric keywords. For example, ie.user is expanded into i.user:e.user.

TABLE 6-2 Metric Type Characters

Character

Description

e

Show exclusive metric value

i

Show inclusive metric value

a

Show attributed metric value (only for callers-callees metrics)


The allowed metric visibility characters are given in TABLE 6-3. The order of the visibility characters in the visibility string does not matter: it does not affect the order in which the corresponding metrics are displayed. For example, both i%.user and i.%user are interpreted as i.user:i%user.

Metrics that differ only in the visibility are always displayed together in the standard order. If two metric keywords that differ only in the visibility are separated by some other keywords, the metrics appear in the standard order at the position of the first of the two metrics.

TABLE 6-3 Metric Visibility Characters

Character

Description

.

Show metric as a time. Applies to timing metrics and hardware counter metrics that measure cycle counts. Interpreted as "+" for other metrics.

%

Show metric as a percentage of the total program metric. For attributed metrics in the callers-callees list, show metric as a percentage of the inclusive metric for the selected function.

+

Show metric as an absolute value. For hardware counters, this value is the event count. Interpreted as a "." for timing metrics.

!

Do not show any metric value. Cannot be used in combination with other visibility characters.


When both type and visibility strings have more than one character, the type is expanded first. Thus ie.%user is expanded to i.%user:e.%user, which is then interpreted as i.user:i%user:e.user:e%user.

The visibility characters ".", "+" and "%" are equivalent for the purposes of defining the sort order. Thus sort i%user, sort i.user, and sort i+user all mean "sort by inclusive user CPU time if it is visible in any form", and sort i!user means "sort by inclusive user CPU time, whether or not it is visible".

TABLE 6-4 lists the available er_print metric name strings for timing metrics, synchronization delay metrics, memory allocation metrics, MPI tracing metrics, and the two common hardware counter metrics. For other hardware counter metrics, the metric name string is the same as the counter name. A list of counter names can be obtained by using the collect command with no arguments. See for more information on hardware counters.

TABLE 6-4 Metric Name Strings

Category

String

Description

Timing metrics

user

User CPU time

wall

Wall-clock time

total

Total LWP time

system

System CPU time

wait

CPU wait time

ulock

User lock time

text

Text-page fault time

data

Data-page fault time

owait

Other wait time

Synchronization delay metrics

sync

Synchronization wait time

syncn

Synchronization wait count

Memory allocation metrics

 

alloc

Number of allocations

balloc

Bytes allocated

leak

Number of leaks

bleak

Bytes leaked

MPI tracing metrics

mpitime

Time spent in MPI calls

mpisend

Number of MPI send operations

mpibytessent

Number of bytes sent in MPI send operations

mpireceive

Number of MPI receive operations

mpibytesrecv

Number of bytes received in MPI receive operations

mpiother

Number of calls to other MPI functions

Hardware counter overflow metrics

cycles

CPU cycles

insts

Instructions issued


In addition to the name strings listed in TABLE 6-4, there are two name strings that can only be used in default metrics lists. These are hwc, which matches any hardware counter name, and any, which matches any metric name string.


Function List Commands

The following commands control the display of function information.

functions

Write the function list with the currently selected metrics. The function list includes all functions in load objects that are selected for display of functions, and the load objects whose functions are hidden. See the command for more information.

The number of lines written can be limited by using the limit command (see Output Commands).

The default metrics printed are exclusive and inclusive user CPU time, in both seconds and percentage of total program metric. You can change the current metrics displayed with the metrics command. This must be done before you issue the functions command. You can also change the defaults with the dmetrics command.

fsummary

Write a summary metrics panel for each function in the function list. The number of panels written can be limited by using the limit command (see Output Commands).

The summary metrics panel includes the name, address and size of the function or load object, and for functions, the name of the source file, object file and load object, and all the recorded metrics for the selected function or load object, both exclusive and inclusive, as values and percentages.

fsingle function-name [N]

Write a summary metrics panel for the specified function. The optional parameter N is needed for those cases where there are several functions with the same name. The summary metrics panel is written for the Nth function with the given function name. When the command is given on the command line, N is required; if it is not needed it is ignored. When the command is given interactively without N but N is required, a list of functions with the corresponding N value is printed.

For a description of the summary metrics for a function, see the fsummary command description.

metrics metric-list

Specify a selection of function-list metrics. The string metric-list can either be the keyword default, which restores the default metric selection, or a list of metric keywords, separated by colons. The following example illustrates a metric list.

% metrics i.user:i%user:e.user:e%user 

This command instructs er_print to display the following metrics:

  • Inclusive user CPU time in seconds
  • Inclusive user CPU time percentage
  • Exclusive user CPU time in seconds
  • Exclusive user CPU time percentage

When the metrics command is finished, a message is printed showing the current metric selection. For the preceding example the message is as follows.

current: i.user:i%user:e.user:e%user:name

For information on the syntax of metric lists, see Metric Lists. To see a listing of the available metrics, use the metric_list command.

If a metrics command has an error in it, it is ignored with a warning, and the previous settings remain in effect.

objects

List the load objects with any error or warning messages that result from the use of the load object for performance analysis. The number of load objects listed can be limited by using the limit command (see Output Commands).

sort metric-keyword

Sort the function list on the specified metric. The string metric-keyword is one of the metric keywords described in Metric Lists, as shown in this example.

% sort i.user

This command tells er_print to sort the function list by inclusive user CPU time. If the metric is not in the experiments that have been loaded, a warning is printed and the command is ignored. When the command is finished, the sort metric is printed.


Callers-Callees List Commands

The following commands control the display of caller and callee information.

callers-callees

Print the callers-callees panel for each of the functions, in the order in which they are sorted. The number of panels written can be limited by using the limit command (see Output Commands). The selected (center) function is marked with an asterisk, as shown in this example.

Attr.     Excl.     Incl.      Name  
User CPU  User CPU  User CPU         
 sec.      sec.       sec.      
4.440     0.        42.910     commandline 
0.        0.         4.440    *gpf 
4.080     0.         4.080     gpf_b 
0.360     0.         0.360     gpf_a 

In this example, gpf is the selected function; it is called by commandline, and it calls gpf_a and gpf_b.

csingle function-name [N]

Write the callers-callees panel for the named function. The optional parameter N is needed for those cases where there are several functions with the same name. The callers-callees panel is written for the Nth function with the given function name. When the command is given on the command line, N is required; if it is not needed it is ignored. When the command is given interactively without N but N is required, a list of functions with the corresponding N value is printed.

cmetrics metric-list

Specify a selection of callers-callees metrics. metric-list is a list of metric keywords, separated by colons, as shown in this example.

% cmetrics i.user:i%user:a.user:a%user

This command instructs er_print to display the following metrics.

  • Inclusive user CPU time in seconds
  • Inclusive user CPU time percentage
  • Attributed user CPU time in seconds
  • Attributed user CPU time percentage

When the cmetrics command is finished, a message is printed showing the current metric selection. For the preceding example the message is as follows.

current: i.user:i%user:a.user:a%user:name

For information on the syntax of metric lists, see Metric Lists. To see a listing of the available metrics, use the cmetric_list command.

csort metric-keyword

Sort the callers-callees display by the specified metric. The string metric-keyword is one of the metric keywords described in Metric Lists, as shown in this example.

% csort a.user

This command tells er_print to sort the callers-callees display by attributed user CPU time. When the command finishes, the sort metric is printed.


Source and Disassembly Listing Commands

The following commands control the display of annotated source and disassembly code.

source | src { file | function } [N]

Write out annotated source code for either the specified file or the file containing the specified function. The file in either case must be in a directory in your path.

Use the optional parameter N (a positive integer) only in those cases where the file or function name is ambiguous; in this case, the Nth possible choice is used. If you give an ambiguous name without the numeric specifier, er_print prints a list of possible object-file names; if the name you gave was a function, the name of the function is appended to the object-file name, and the number that represents the value of N for that object file is also printed.

disasm { file | function } [N]

Write out annotated disassembly code for either the specified file, or the file containing the specified function. The file in either case must be in a directory in your path.

The optional parameter N is used in the same way as for the source command.

scc class-list

Specify the classes of compiler commentary that are shown in the annotated source listing. The class list is a colon-separated list of classes, containing zero or more of the following message classes.

  • b[asic]- Show the basic level messages.
  • v[ersion] - Show version messages, including source file name and last modified date, versions of the compiler components, compilation date and options.
  • pa[rallel] - Show messages about parallelization.
  • q[uery] - Show questions about the code that affect its optimization.
  • l[oop] - Show messages about loop optimizations and transformations.
  • pi[pe] - Show messages about pipelining of loops.
  • i[nline] - Show messages about inlining of functions.
  • m[emops] - Show messages about memory operations, such as load, store, prefetch.
  • f[e] - Show front-end messages.
  • all - Show all messages.
  • none - Do not show any messages.

The classes all and none cannot be used with other classes.

If no scc command is given, the default class shown is basic. If the scc command is given with an empty class-list, compiler commentary is turned off. The scc command is normally used only in a .er.rc file.

For compatibility, the highlighting threshold can also be specified using t[hreshold]=nn, where nn is the threshold percentage. See the sthresh section for more information.

sthresh value

Specify the threshold percentage for highlighting metrics in the annotated source code. If the value of any metric is equal to or greater than value % of the maximum value of that metric for any source line in the file, the line on which the metrics occur have ## inserted at the beginning of the line.

dcc class-list

Specify the classes of compiler commentary that are shown in the annotated disassembly listing. The class list is a colon-separated list of classes. The list of available classes is the same as the list of classes for annotated source code listing. The following options can be added to the class list.

  • h[ex]- Show the hexadecimal value of the instructions.
  • s[rc]- Interleave the source listing in the annotated disassembly listing.
  • as[rc]- interleave the annotated source code in the annotated disassembly listing.

For compatibility, the highlighting threshold can also be specified using t[hreshold]=nn, where nn is the threshold percentage. See the dthresh section for more information.

dthresh value

Specify the threshold percentage for highlighting metrics in the annotated disassembly code. If the value of any metric is equal to or greater than value % of the maximum value of that metric for any instruction line in the file, the line on which the metrics occur have ## inserted at the beginning of the line.


Memory Allocation List Commands

This section describes commands relating to memory allocations and deallocations.

allocs

Display a list of memory allocations, aggregated by common call stack. Each entry presents the number of allocations and the total bytes allocated for the given call stack. The list is sorted by the number of bytes allocated.

leaks

Display a list of memory leaks, aggregated by common call stack. Each entry presents the total number of leaks and the total bytes leaked for the given call stack. The list is sorted by the number of bytes leaked.


Filtering Commands

This section describes commands that are used to control selection of experiments, samples, threads, and LWPs for display, and to list the current selections.

Selection Lists

The syntax of a selection is shown in the following example. This syntax is used in the command descriptions.

[experiment-list:]selection-list[+[experiment-list:]selection-list ... ]

Each selection list can be preceded by an experiment list, separated from it by a colon and no spaces. You can make multiple selections by joining selection lists with a + sign.

The experiment list and the selection list have the same syntax, which is either the keyword all or a list of numbers or ranges of numbers (n-m) separated by commas but no spaces, as shown in this example.

2,4,9-11,23-32,38,40 

The experiment numbers can be determined by using the exp_list command.

Some examples of selections are as follows.

1:1-4+2:5,6
all:1,3-6

In the first example, objects 1 through 4 are selected from experiment 1 and objects 5 and 6 are selected from experiment 2. In the second example, objects 1 and 3 through 6 are selected from all experiments. The objects may be LWPs, threads, or samples.

Selection Commands

The commands to select LWPs, samples, and threads are not independent. If the experiment list for a command is different from that for the previous command, the experiment list from the latest command is applied to all three selection targets - LWPs, samples, and threads, in the following way.

  • Existing selections for experiments not in the latest experiment list are turned off.
  • Existing selections for experiments in the latest experiment list are kept.
  • Selections are set to "all" for targets for which no selection has been made.

lwp_select lwp-selection

Select the LWPs about which you want to display information. The list of LWPs you selected is displayed when the command finishes.

sample_select sample-selection

Select the samples for which you want to display information. The list of samples you selected is displayed when the command finishes.

thread_select thread-selection

Select the threads about which you want to display information. The list of threads you selected is displayed when the command finishes.

object_select object-list

Select the load objects for which you want to display information about the functions in the load object. object-list is a list of load objects, separated by commas but no spaces. For load objects that are not selected, information for the entire load object is displayed instead of information for the functions in the load object.

The names of the load objects should be either full path names or the basename. If an object name itself contains a comma, you must surround the name with double quotation marks.

Listing of Selections

The commands for listing what has been selected are given in this section, followed by some examples.

exp_list

Display the full list of experiments loaded with their ID number.

lwp_list

Display the list of LWPs currently selected for analysis.

object_list

Display the list of load objects.The name of each load object is preceded by a "+" if its functions are shown in the function list, and by a "-" if its functions are not shown in the function list.

sample_list

Display the list of samples currently selected for analysis.

thread_list

Display the list of threads currently selected for analysis.

The following example is an example of an experiment list.

(er_print) exp_list
ID Experiment 
== ========== 
1 test.1.er 
2 test.6.er 

The sample list, thread list and LWP list have the same format. The following example is an example of a sample list.

(er_print) sample_list 
Exp Sel     Total 
=== ======= ===== 
  1 1-6        31 
  2 7-10,15    31 

The following example is an example of a load object list.

(er_print) object_list 
Sel Load Object 
=== ================================================== 
yes /tmp/var/synprog/synprog 
yes /opt/SUNWspro/lib/libcollector.so 
yes /usr/lib/libdl.so.1 
yes /usr/lib/libc.so.1 


Metric List Commands

The following commands list the currently selected metrics and all available metric keywords.

metric_list

Display the currently selected metrics in the function list and a list of metric keywords that you can use in other commands (for example, metrics and sort) to reference various types of metrics in the function list.

cmetric_list

Display the currently selected metrics in the callers-callees list and a list of metric keywords that you can use in other commands (for example, cmetrics and csort) to reference various types of metrics in the callers-callees list.



Note - Attributed metrics can only be specified for display with the cmetrics command, not the metrics command, and displayed only with the callers-callees command, not the functions command.




Defaults Commands

The following commands can be used to set the defaults for er_print and for the Performance Analyzer. They can only be used for setting defaults: they cannot be used in input for er_print. They can be included in a defaults filed named .er.rc.

A defaults file can be included in your home directory, to set defaults for all experiments, or in any other directory, to set defaults locally. When er_print, er_src or the Performance Analyzer is started, the current directory and your home directory are scanned for defaults files, which are read if they are present, and the system defaults file is also read. Defaults from the .er.rc file in your home directory override the system defaults, and defaults from the .er.rc file in the current directory override both home and system defaults.



Note - To ensure that you read the defaults file from the directory where your experiment is stored, you must start the Performance Analyzer or the er_print utility from that directory.



The defaults file can also include the scc, sthresh, dcc, and dthresh commands. Multiple dmetrics and dsort commands can be given in a defaults file, and the commands within a file are concatenated.

dmetrics metric-list

Specify the default metrics to be displayed or printed in the function list. The syntax and use of the metric list is described in the section Metric Lists. The order of the metric keywords in the list determines the order in which the metrics are presented and the order in which they appear in the Metric chooser in the Performance Analyzer.

Default metrics for the Callers-Callees list are derived from the function list default metrics by adding the corresponding attributed metric before the first occurrence of each metric name in the list.

dsort metric-list

Specify the default metric by which the function list is sorted. The sort metric is the first metric in this list that matches a metric in any loaded experiment, subject to the following conditions:

  • If the entry in metric-list has a visibility string of "!", the first metric whose name matches is used, whether it is visible or not.
  • If the entry in metric-list has any other visibility string, the first visible metric whose name matches is used.

The syntax and use of the metric list is described in the section Metric Lists.

The default sort metric for the Callers-Callees list is the attributed metric corresponding to the default sort metric for the function list.

gdemangle library-name

Set the path to the shared object that supports an API to demangle C++ function names. The shared object must export the C function cplus_demangle(), conforming to the GNU standard libiberty.so interface.


Output Commands

The following commands control er_print display output.

limit n

Limit output to the first n entries of the report; n is an unsigned positive integer.

name { long | short }

Specify whether to use the long or the short form of function names (C++ only).

outfile { filename | - }

Close any open output file, then open filename for subsequent output. If you specify a dash (-) instead of filename, output is written to standard output.


Other Display Commands

header experiment-ID

Display descriptive information about the specified experiment. The experiment-ID can be obtained from the exp_list command. If the experiment-ID is all or is not given, the information is displayed for all experiments loaded.

Following each header, any errors or warnings are printed. Headers for each experiment are separated by a line of dashes.

experiment-ID is required on the command line, but not in a script or in interactive mode.

overview experiment-ID

Write out the sample data of each of the currently selected samples for the specified experiment. The experiment-ID can be obtained from the exp_list command. If the experiment-ID is all or is not given, the sample data is displayed for all experiments. experiment-ID is required on the command line, but not in a script or in interactive mode.

statistics experiment-ID

Write out execution statistics, aggregated over the current sample set for the specified experiment. For information on the definitions and meanings of the execution statistics that are presented, see the getrusage(3C) and proc(4) man pages. The execution statistics include statistics from system threads for which the Collector does not collect any data. The standard threads library in the Solaristrademark 7 and 8 operating environments creates system threads that are not profiled. These threads spend most of their time sleeping, and the time shows in the statistics display as Other Wait time.

The experiment-ID can be obtained from the exp_list command. If the experiment-ID is not given, the sum of data for all experiments is displayed, aggregated over the sample set for each experiment. If experiment-ID is all, the sum and the individual statistics for each experiment are displayed.

experiment-ID is required on the command line, but not in a script or in interactive mode.


Mapfile Generation Command

mapfile load-object { mapfilename | - }

Write a mapfile for the specified load object to the file mapfilename. If you specify a dash (-) instead of mapfilename, er_print writes the mapfile to standard output.


Control Commands

quit

Terminate processing of the current script, or exit interactive mode.

script script

Process additional commands from the script file script.


Information Commands

help

Print a list of er_print commands.

{ Version | version }

Print the current release number of er_print.


Obsolete Commands

address_space

Address-space data collection and display is no longer supported. This command is ignored with a warning.

osummary

The load objects list has been incorporated into the function list. To see metrics for load objects, use the object_select command and the fsummary command. This command is ignored with a warning.