C H A P T E R  6

The er_print Command Line Performance Analysis Tool

This chapter explains how to use the er_print utility for performance analysis. The er_print utility prints an ASCII version of the various displays supported by the Performance Analyzer. The information is written to standard output unless you redirect it to a file. You must give the er_print utility the name of one or more experiments or experiment groups generated by the Collector as arguments. You can use the er_print utility to display the performance metrics for functions, for callers and callees; the source code listing and disassembly listing; sampling information; data-space data; and execution statistics.

This chapter covers the following topics.

For a description of the data collected by the Collector, see Chapter 2.

For instructions on how to use the Performance Analyzer to display information in a graphical format, see Chapter 4 and the online help.


er_print Syntax

The command-line syntax for the er_print utility is.:


er_print [ -script script | -command | - | -V ] experiment-list 

The options for the er_print utility are:


-

Read er_print commands entered from the keyboard.

-script script

Read commands from the file script, which contains a list of er_print commands, one per line. If the -script option is not present, er_print reads commands from the terminal or from the command line.

-command [argument]

Process the given command.

-V

Display version information and exit.


Multiple options can appear on the er_print command line. They are processed in the order they appear. You can mix scripts, hyphens, and explicit commands in any order. The default action if you do not supply any commands or scripts is to enter interactive mode, in which commands are entered from the keyboard. To exit interactive mode type quit or Ctrl-D.

The commands accepted by the er_print utility are listed in the following sections. You can abbreviate any command with a shorter string as long as the command is unambiguous.


Metric Lists

Many of the er_print commands use a list of metric keywords. The syntax of the list is:


metric-keyword-1[:metric-keyword2...]

Except for the size, address, and name keywords, a metric keyword consists of three parts: a metric type string, a metric visibility string, and a metric name string. These are joined with no spaces, as follows.


<type><visibility><name>

The metric type and metric visibility strings are composed of type and visibility characters.

The allowed metric type characters are given in TABLE 6-1. A metric keyword that contains more than one type character is expanded into a list of metric keywords. For example, ie.user is expanded into i.user:e.user.


TABLE 6-1 Metric Type Characters

Character

Description

e

Show exclusive metric value

i

Show inclusive metric value

a

Show attributed metric value (only for callers-callees metrics)


The allowed metric visibility characters are given in TABLE 6-2. The order of the visibility characters in the visibility string does not matter: it does not affect the order in which the corresponding metrics are displayed. For example, both i%.user and i.%user are interpreted as i.user:i%user.

Metrics that differ only in the visibility are always displayed together in the standard order. If two metric keywords that differ only in the visibility are separated by some other keywords, the metrics appear in the standard order at the position of the first of the two metrics.


TABLE 6-2 Metric Visibility Characters

Character

Description

.

Show metric as a time. Applies to timing metrics and hardware counter metrics that measure cycle counts. Interpreted as "+" for other metrics.

%

Show metric as a percentage of the total program metric. For attributed metrics in the callers-callees list, show metric as a percentage of the inclusive metric for the selected function.

+

Show metric as an absolute value. For hardware counters, this value is the event count. Interpreted as a "." for timing metrics.

!

Do not show any metric value. Cannot be used in combination with other visibility characters.


When both type and visibility strings have more than one character, the type is expanded first. Thus ie.%user is expanded to i.%user:e.%user, which is then interpreted as i.user:i%user:e.user:e%user.

The visibility characters period (.), plus (+), and percent sign (%), are equivalent for the purposes of defining the sort order. Thus sort i%user, sort i.user, and sort i+user all mean that the Analyzer should sort by inclusive user CPU time if it is visible in any form, and sort i!user means the Analyzer should sort by inclusive user CPU time, whether or not it is visible.

TABLE 6-3 lists the available er_print metric name strings for timing metrics, synchronization delay metrics, memory allocation metrics, MPI tracing metrics, and the two common hardware counter metrics. For other hardware counter metrics, the metric name string is the same as the counter name. A list of counter names can be obtained by using the collect command with no arguments. See Hardware Counter Overflow Profiling Data for more information on hardware counters.


TABLE 6-3 Metric Name Strings

Category

String

Description

Timing metrics

user

User CPU time

wall

Wall-clock time

total

Total LWP time

system

System CPU time

wait

CPU wait time

ulock

User lock time

text

Text-page fault time

data

Data-page fault time

owait

Other wait time

Synchronization delay metrics

sync

Synchronization wait time

syncn

Synchronization wait count

MPI tracing metrics

mpitime

Time spent in MPI calls

mpisend

Number of MPI send operations

mpibytessent

Number of bytes sent in MPI send operations

mpireceive

Number of MPI receive operations

mpibytesrecv

Number of bytes received in MPI receive operations

mpiother

Number of calls to other MPI functions

Memory allocation metrics

 

alloc

Number of allocations

balloc

Bytes allocated

leak

Number of leaks

bleak

Bytes leaked

Hardware counter overflow metrics

cycles

CPU cycles

insts

Instructions issued


In addition to the name strings listed in TABLE 6-3, there are two name strings that can only be used in default metrics lists. These are hwc, which matches any hardware counter name, and any, which matches any metric name string. Also note that cycles and insts are common to SPARC® platforms and x86 platforms, but other flavors also exist that are architecture-specific. To list all available counters, use the collect command with no arguments.


Commands That Control the Function List

The following commands control how the function information is displayed.

functions

Write the function list with the currently selected metrics. The function list includes all functions in load objects that are selected for display of functions, and any load objects whose functions are hidden with the object_select command.

You can limit the number of lines written by using the limit command (see Commands That Control Output).

The default metrics printed are exclusive and inclusive user CPU time, in both seconds and percentage of total program metric. You can change the current metrics displayed with the metrics command, which you must issue before you issue the functions command. You can also change the defaults with the dmetrics command in an .er.rc file.

For applications written in the Java programming language, the displayed function information varies depending on whether Java mode is set to on, expert, or off.

metrics metric_spec

Specify a selection of function-list metrics. The string metric_spec can either be the keyword default, which restores the default metric selection, or a list of metric keywords, separated by colons. The following example illustrates a metric list.


% metrics i.user:i%user:e.user:e%user 

This command instructs the er_print utility to display the following metrics:

When the metrics command is processed, a message is printed showing the current metric selection. For the preceding example the message is as follows.


current: i.user:i%user:e.user:e%user:name

For information on the syntax of metric lists, see Metric Lists. To see a listing of the available metrics, use the metric_list command.

If a metrics command has an error in it, it is ignored with a warning, and the previous settings remain in effect.

sort metric_spec

Sort the function list on the specified metric. The string metric_spec is one of the metric keywords described in Metric Lists, as shown in this example.


% sort i.user

This command tells the er_print utility to sort the function list by inclusive user CPU time. If the metric is not in the experiments that have been loaded, a warning is printed and the command is ignored. When the command is finished, the sort metric is printed.

fsummary

Write a summary metrics panel for each function in the function list. You can limit the number of panels written by using the limit command (see Commands That Control Output).

The summary metrics panel includes the name, address and size of the function or load object, and for functions, the name of the source file, object file and load object, and all the recorded metrics for the selected function or load object, both exclusive and inclusive, as values and percentages.

fsingle function_name [N]

Write a summary metrics panel for the specified function. The optional parameter N is needed for those cases where there are several functions with the same name. The summary metrics panel is written for the Nth function with the given function name. When the command is given on the command line, N is required; if it is not needed it is ignored. When the command is given interactively without N but N is required, a list of functions with the corresponding N value is printed.

For a description of the summary metrics for a function, see the fsummary command description.


Commands That Control the Callers-Callees List

The following commands control how the caller and callee information is displayed.

callers-callees

Print the callers-callees panel for each of the functions, in the order in which they are sorted. You can limit the number of panels written by using the limit command (see Commands That Control Output). The selected (center) function is marked with an asterisk, as shown in this example.


Attr.     Excl.     Incl.      Name 
User CPU  User CPU  User CPU         
 sec.      sec.       sec.      
4.440     0.        42.910     commandline 
0.        0.         4.440    *gpf 
4.080     0.         4.080     gpf_b 
0.360     0.         0.360     gpf_a 

In this example, gpf is the selected function; it is called by commandline, and it calls gpf_a and gpf_b.

cmetrics metric_spec

Specify a selection of callers-callees metrics. metric_spec is a list of metric keywords, separated by colons, as shown in this example.


% cmetrics i.user:i%user:a.user:a%user

This command instructs er_print to display the following metrics.

When the cmetrics command is finished, a message is printed showing the current metric selection. For the preceding example the message is as follows.


current: i.user:i%user:a.user:a%user:name

For information on the syntax of metric lists, see Metric Lists. To see a listing of the available metrics, use the cmetric_list command.

csingle function_name [N]

Write the callers-callees panel for the named function. The optional parameter N is needed for those cases where there are several functions with the same name. The callers-callees panel is written for the Nth function with the given function name. When the command is given on the command line, N is required; if it is not needed it is ignored. When the command is given interactively without N but N is required, a list of functions with the corresponding N value is printed.

csort metric_spec

Sort the callers-callees display by the specified metric. The string metric_spec is one of the metric keywords described in Metric Lists, as shown in this example.


% csort a.user

This command tells the er_printutility to sort the callers-callees display by attributed user CPU time. When the command finishes, the sort metric is printed.


Commands That Control the Leak and Allocation Lists

This section describes the commands that relate to memory allocations and deallocations.

leaks

Display a list of memory leaks, aggregated by common call stack. Each entry presents the total number of leaks and the total bytes leaked for the given call stack. The list is sorted by the number of bytes leaked.

allocs

Display a list of memory allocations, aggregated by common call stack. Each entry presents the number of allocations and the total bytes allocated for the given call stack. The list is sorted by the number of bytes allocated.


Commands That Control the Source and Disassembly Listings

The following commands control how annotated source and disassembly code is displayed.

pcs

Write a list of program counters (PCs) and their metrics, ordered by the current sort metric. The list includes lines that show aggregated metrics for each load object whose functions are hidden with the object_select command.

psummary

Write the summary metrics panel for each PC in the PC list, in the order specified by the current sort metric.

lines

Write a list of source lines and their metrics, ordered by the current sort metric. The list includes lines that show aggregated metrics for each function that does not have line-number information, or whose source file is unknown, and lines that show aggregated metrics for each load object whose functions are hidden with the object_select command.

lsummary

Write the summary metrics panel for each line in the lines list, in the order specified by the current sort metric.

source { filename | function_name } [N]

Write out annotated source code for either the specified file or the file containing the specified function. The file in either case must be in a directory in your path.

Use the optional parameter N (a positive integer) only in those cases where the file or function name is ambiguous; in this case, the Nth possible choice is used. If you give an ambiguous name without the numeric specifier the er_print utility prints a list of possible object-file names; if the name you gave was a function, the name of the function is appended to the object-file name, and the number that represents the value of N for that object file is also printed.

The function name can also be specified as function`file`, where file is used to specify an alternate source context for the function. Immediately following the first instruction, an index line is added for the function. Index lines are displayed as text within angle brackets in the following form:

<Function: f_name>

The default source context for any function is defined as the source file to which the first instruction in that function is attributed. It is normally the source file compiled to produce the object module containing the function. Alternate source contexts consist of other files that contain instructions attributed to the function. Such contexts include instructions coming from include files and instructions from functions inlined into the named function. If there are any alternate source contexts, include a list of extended index lines at the beginning of the default source context to indicate where the alternate source contexts are located in the following form:

<Function: f, instructions from source file src.h>



Note - If you use the -source argument when invoking the er_print utiity on the command line, the backslash escape character must prepend the file quotes. In other words, the function name is of the form function\\`file\\`. The backslash is not required, and should not be used, when the er_print utiity is in interactive mode.



disasm { filename | function_name } [N]

Write out annotated disassembly code for either the specified file, or the file containing the specified function. The file must be in a directory in your path.

The optional parameter N is used in the same way as for the source command.

scc com_spec

Specify the classes of compiler commentary that are shown in the annotated source listing. The class list is a colon-separated list of classes, containing zero or more of the following message classes.


TABLE 6-4 Compiler Commentary Message Classes

Class

Meaning

b[asic]

Show the basic level messages.

v[ersion]

Show version messages, including source file name and last modified date, versions of the compiler components, compilation date and options.

pa[rallel]

Show messages about parallelization.

q[uery]

Show questions about the code that affect its optimization.

l[oop]

Show messages about loop optimizations and transformations.

pi[pe]

Show messages about pipelining of loops.

i[nline]

Show messages about inlining of functions.

m[emops]

Show messages about memory operations, such as load, store, prefetch.

f[e]

Show front-end messages.

all

Show all messages.

none

Do not show any messages.


The classes all and none cannot be used with other classes.

If no scc command is given, the default class shown is basic. If the scc command is given with an empty class-list, compiler commentary is turned off. The scc command is normally used only in an .er.rc file.

sthresh value

Specify the threshold percentage for highlighting metrics in the annotated source code. If the value of any metric is equal to or greater than value % of the maximum value of that metric for any source line in the file, the line on which the metrics occur have ## inserted at the beginning of the line.

dcc com_spec

Specify the classes of compiler commentary that are shown in the annotated disassembly listing. The class list is a colon-separated list of classes. The list of available classes is the same as the list of classes for annotated source code listing. You can add the following options to the class list.


TABLE 6-5 Additional Options for the dcc Command

Option

Meaning

h[ex]

Show the hexadecimal value of the instructions.

noh[ex]

Do not show the hexadecimal value of the instructions.

s[rc]

Interleave the source listing in the annotated disassembly listing.

nos[rc]

Do not interleave the source listing in the annotated disassembly listing.

as[rc]

Interleave the annotated source code in the annotated disassembly listing.


dthresh value

Specify the threshold percentage for highlighting metrics in the annotated disassembly code. If the value of any metric is equal to or greater than value % of the maximum value of that metric for any instruction line in the file, the line on which the metrics occur have ## inserted at the beginning of the line.

setpath path_list

Set the path used to find source, object, etc. files. path_list is a colon-separated list of directories. If any directory has a colon character in it, escape it with a backslash. The special directory name, $expts, refers to the set of current experiments, in the order in which they were loaded; youc can abbreviate it with a single $ character.

The default setting is: $expts:.. The compiled-in full pathname is used if a file is not found in searching the current path setting.

setpath with no argument prints the current path.

addpath path_list

Append path_list to the current setpath settings.


Commands That Control the Data Space List

data_objects

Write the list of data objects with their metrics. Applicable only to hardware counter overflow experiments where aggressive backtracking was specified, and for objects in files that were compiled with -xhwcprof. (Available on SPARC-based systemsfor C only). See the C User's Guide or the cc(1) man page for further information.

data_osingle name [N]

Write the summary metrics panel for the named data object. The optional parameter N is needed for those cases where the object name is ambiguous. When the directive is on the command-line, N is required; if it is not needed, it is ignored. Applicable only to hardware counter overflow experiments where aggressive backtracking was specified, and for objects in files that were compiled with -xhwcprof. (Available on SPARC-based systems for C only). See the C User's Guide or the cc(1) man page for further information.

data_olayout

Write the annotated data object layouts for all program data objects with data-derived metric data, in the order they are defined in the experiment's load objects. Each aggregate data object is shown with the total metrics attributed to it, followed by all of its elements in offset order, each with their own metrics and an indicator of its size and location relative to 32-byte blocks.

data_osort

Set the sort metric for data objects. No prefix, such as i., e., or a., is needed.


Commands That List Experiments, Samples, Threads, and LWPs

This section describes the commands that list experiments, samples, threads, and LWPs.

experiment_list

Display the full list of experiments loaded with their ID number. Each experiment is listed with an index, which is used when selecting samples, threads, or LWPs.

The following example is an example of an experiment list.


(er_print) experiment_list
ID Experiment 
== ========== 
1 test.1.er 
2 test.6.er 

sample_list

Display the list of samples currently selected for analysis.

The following example is an example of a sample list.


(er_print) sample_list 
Exp Sel     Total 
=== ======= ===== 
  1 1-6        31 
  2 7-10,15    31 

lwp_list

Display the list of LWPs currently selected for analysis.

thread_list

Display the list of threads currently selected for analysis.

cpu_list

Display the list of CPUs currently selected for analysis.


Commands That Control Selections

Selection Lists

The syntax of a selection is shown in the following example. This syntax is used in the command descriptions.


[experiment-list:]selection-list[+[experiment-list:]selection-list ... ]

Each selection list can be preceded by an experiment list, separated from it by a colon and no spaces. You can make multiple selections by joining selection lists with a + sign.

The experiment list and the selection list have the same syntax, which is either the keyword all or a list of numbers or ranges of numbers (n-m) separated by commas but no spaces, as shown in this example.


2,4,9-11,23-32,38,40 

The experiment numbers can be determined by using the exp_list command.

Some examples of selections are as follows.


1:1-4+2:5,6
all:1,3-6

In the first example, objects 1 through 4 are selected from experiment 1 and objects 5 and 6 are selected from experiment 2. In the second example, objects 1 and 3 through 6 are selected from all experiments. The objects may be LWPs, threads, or samples.

Selection Commands

The commands to select LWPs, samples, CPUs, and threads are not independent. If the experiment list for a command is different from that for the previous command, the experiment list from the latest command is applied to all three selection targets-- LWPs, samples, and threads, in the following way.

sample_select sample_spec

Select the samples for which you want to display information. The list of samples you selected is displayed when the command finishes.

lwp_select lwp_spec

Select the LWPs about which you want to display information. The list of LWPs you selected is displayed when the command finishes.

thread_select thread_spec

Select the threads about which you want to display information. The list of threads you selected is displayed when the command finishes.

cpu_select cpu_spec

Select the CPUs about which you want to display information. The list of CPUs you selected is displayed when the command finishes.


Commands That Control Load Object Selection

object_list

Display the list of load objects. The name of each load object is preceded either by a yes that indicates that the functions of that object are shown in the function list, or by a no that indicates that the functions of that object are not shown in the function list.

The following is an example of a load object list.


(er_print) object_list 
Sel Load Object 
=== ================================================== 
yes /tmp/var/synprog/synprog 
yes /opt/SUNWspro/lib/libcollector.so 
yes /usr/lib/libdl.so.1 
yes /usr/lib/libc.so.1 

object_select object1,object2,...

Select the load objects for which you want to display information about the functions in the load object. object-list is a list of load objects, separated by commas but no spaces. For load objects that are not selected, information for the entire load object is displayed instead of information for the functions in the load object.

The names of the load objects should be either full path names or the basename. If an object name itself contains a comma, you must surround the name with double quotation marks.


Commands That List Metrics

The following commands list the currently selected metrics and all available metric keywords.

metric_list

Display the currently selected metrics in the function list and a list of metric keywords that you can use in other commands (for example, metrics and sort) to reference various types of metrics in the function list.

cmetric_list

Display the currently selected metrics in the callers-callees list and a list of metric keywords that you can use in other commands (for example, cmetrics and csort) to reference various types of metrics in the callers-callees list.



Note - Attributed metrics can only be specified for display with the cmetrics command, not the metrics command, and displayed only with the callers-callees command, not the functions command.




Commands That Control Output

The following commands control er_print display output.

outfile { filename | - }

Close any open output file, then open filename for subsequent output. When opening filename, clear any pre-existing content. If you specify a dash (-) instead of filename, output is written to standard output.

appendfile filename

Close any open output file and open filename, preserving any pre-existing content, so that subsequent output is appended to the end of the file. If filename does not exist, the functionality of the appendfile command is the same as for the outfile command.

limit n

Limit output to the first n entries of the report; n is an unsigned positive integer.

name { long | short } [ :{shared_object_name | no_shared_object_name } ]"

Specify whether to use the long or the short form of function names (C++ and Java only). If shared_object_name is specified, append the shared-object name to the function name.

javamode { on | expert | off }

Set the mode for Java experiments to on (show the Java model), expert (show the Java model and additionally show internal JVM details), or off (show the machine model).


Commands That Print Other Displays

header exp_id

Display descriptive information about the specified experiment. The exp_id can be obtained from the exp_list command. If the exp_id is all or is not given, the information is displayed for all experiments loaded.

Following each header, any errors or warnings are printed. Headers for each experiment are separated by a line of dashes.

If the experiment directory contains a file named notes, the contents of that file are prepended to the header information. A notes file may be manually added or edited or specified with -C "comment" arguments to the collect command.

exp_id is required on the command line, but not in a script or in interactive mode.

objects

List the load objects with any error or warning messages that result from the use of the load object for performance analysis. The number of load objects listed can be limited by using the limit command (see Commands That Control Output).

overview exp_id

Write out the sample data of each of the currently selected samples for the specified experiment. The exp_id can be obtained from the exp_list command. If the exp_id is all or is not given, the sample data is displayed for all experiments. exp_id is required on the command line, but not in a script or in interactive mode.

statistics exp_id

Write out execution statistics, aggregated over the current sample set for the specified experiment. For information on the definitions and meanings of the execution statistics that are presented, see the getrusage(3C) and proc(4) man pages. The execution statistics include statistics from system threads for which the Collector does not collect any data. The standard threads library in the Solaris 7 OS and Solaris 8 OS creates system threads that are not profiled. These threads spend most of their time sleeping, and the time shows in the statistics display as Other Wait time.

The exp_id can be obtained from the experiment_list command. If the exp_id is not given, the sum of data for all experiments is displayed, aggregated over the sample set for each experiment. If exp_id is all, the sum and the individual statistics for each experiment are displayed.


Commands That Set Defaults

You can use the following commands to set the defaults for er_print and for the Performance Analyzer. You can only use these commands for setting defaults: they cannot be used in input for the er_print utility.They can be included in a defaults file named .er.rc. Some of the commands only apply to the Performance Analyzer.

You can include a defaults file in your home directory to set defaults for all experiments, or in any other directory to set defaults locally. When the er_print utility, the er_src utility, or the Performance Analyzer is started, the current directory and your home directory are scanned for defaults files, which are read if they are present, and the system defaults file is also read. Defaults from the .er.rc file in your home directory override the system defaults, and defaults from the .er.rc file in the current directory override both home and system defaults.



Note - To ensure that you read the defaults file from the directory where your experiment is stored, you must start the Performance Analyzer or the er_print utility from that directory.



The defaults file can also include the scc, sthresh, dcc, and dthresh commands. Multiple dmetrics and dsort commands can be given in a defaults file, and the commands within a file are concatenated.

dmetrics metric_spec

Specify the default metrics to be displayed or printed in the function list. The syntax and use of the metric list is described in the section Metric Lists. The order of the metric keywords in the list determines the order in which the metrics are presented and the order in which they appear in the Metric chooser in the Performance Analyzer.

Default metrics for the Callers-Callees list are derived from the function list default metrics by adding the corresponding attributed metric before the first occurrence of each metric name in the list.

dsort metric_spec

Specify the default metric by which the function list is sorted. The sort metric is the first metric in this list that matches a metric in any loaded experiment, subject to the following conditions:

The syntax and use of the metric list is described in the section Metric Lists.

The default sort metric for the Callers-Callees list is the attributed metric corresponding to the default sort metric for the function list.


Commands That Set Defaults Only For the Performance Analyzer

tlmode tl_mode

Set the display mode options for the Timeline tab of the Performance Analyzer. The list of options is a colon-separated list. The allowed options are described in the following table.


TABLE 6-6 Timeline Display Mode Options

Option

Meaning

lw[p]

Display events for LWPs

t[hread]

Display events for threads

c[pu]

Display events for CPUs

r[oot]

Align call stack at the root

le[af]

Align call stack at the leaf

d[epth] nn

Set the maximum depth of the call stack that can be displayed


The options lwp, thread, and cpu are mutually exclusive, as are root and leaf. If more than one of a set of mutually exclusive options is included in the list, the last one is the only one that is used.

tldata tl_data

Select the default data types shown in the Timeline tab of the Performance Analyzer. The types in the type list are separated by colons. The allowed types are listed in the following table.


TABLE 6-7 Timeline Display Data Types

Type

Meaning

sa[mple]

Display sample data

c[lock]

Display clock profiling data

hw[c]

Display hardware counter profiling data

sy[nctrace]

Display thread synchronization tracing data

mp[itrace]

Display MPI tracing data

he[aptrace]

Display heap tracing data


datamode { on| off }

Set the mode for showing dataspace-related screens to on (tabs are visible), or off (do not have them visible).


Miscellaneous Commands

mapfile load-object { mapfilename | - }

Write a mapfile for the specified load object to the file mapfilename. If you specify a dash (-) instead of mapfilename, er_print writes the mapfile to standard output.

script file

Process additional commands from the script file file.

version

Print the current release number of the er_print utility

quit

Terminate processing of the current script, or exit interactive mode.

help

Print a list of er_print commands.


Examples

er_print -outfile er_print.out -metrics e.user:e%user\

-sort e.user -limit 100 -functions -cmetrics a.user:a%user\

-csort a.user -callers-callees test.1.er

You can also simplify this example into the following independent commands. However, keep in mind that each call to er_print in a large experiment or application can be time intensive:

-limit 100 -functions test.1.er

-callers-callees test.1.er

er_print -functions test.*.er

er_print -callers-callees test.*.er

er_print -source myfunction 1 test.*.er

er_src -myfile.o

er_print -metrics ei.%wall -functions test.*.er

er_print -cmetrics aei.%wall -callers-callees test.*.er

er_print -functions test.*.er | grep PMPI_