cpustat - monitor system behavior using CPU performance counters
cpustat -c eventspec [-c eventspec]... [-p period] [-T u | d ] [-Dmnst] [-A cor|soc|bins] [-k keys] [-o limit] [-I statfile] [-O statfile] [interval [count]]
The cpustat utility allows CPU performance counters to be used to monitor the overall behavior of the CPUs in the system.
If interval is specified, cpustat samples activity every interval seconds, repeating forever. If a count is specified, the statistics are repeated count times. If neither are specified, an interval of five seconds is used, and there is no limit to the number of samples that are taken.
The following options are supported:
Aggregate output by core ID. Data rows having the same core ID are aggregated into one row. The columns are replaced with subtotals, by default. The –m option prints column averages, instead.
Aggregate output by socket ID. Data rows having the same socket ID are aggregated into one row. The columns are replaced with subtotals, by default. The –m option prints column averages, instead.
Aggregate the rows into a lesser number of bins within each sampling period, grouping them in the order in which they appear, and print the columnar subtotal over rows for each bin. The –m option may be used in order to compute the arithmetic mean instead of the subtotal. The –k sorting option may be used to change the row order prior to the binning step. The sze column prints the number of CPUs in each bin. The BIN column replaces the CPU column and prints the ordinal of each bin.
Specifies a set of events for the CPU performance counters to monitor. The syntax of these event specifications is:
You can use the –h option to obtain a list of available events and attributes. This causes generation of the usage message. You can omit an explicit counter assignment, in which case cpustat attempts to choose a capable counter automatically.
Attribute values can be expressed in hexadecimal, octal, or decimal notation, in a format suitable for strtoll(3C). An attribute present in the event specification without an explicit value receives a default value of 1. An attribute without a corresponding counter number is applied to all counters in the specification.
The semantics of these event specifications can be determined by reading the CPU manufacturer's documentation for the events.
Multiple –c options can be specified, in which case the command cycles between the different event settings on each sample.
Enables debug mode.
Prints an extensive help message on how to use the utility and how to program the processor-dependent counters.
Replay data previously saved in statfile. Create data files for replay by specifying –O. This option is especially useful for analyzing statistics on machines with large numbers of CPUs. The file may be reprocessed multiple times using different sorting and aggregation options.
The –I option is incompatible with an interval and count specification.
Read from the standard input if the file name is — (hyphen).
Sort rows within each sampling period from highest to lowest by key1, then key2, and so on. Each key is a comma-separated list of events. There may be multiple –k options specified.
When cpustat is run with multiple –c event-spec options it produces a report of alternating event-specs. Specify multiple –k options to sort each event-spec differently. For each event-spec, the first –k option whose keys contain a proper subset of the events in the event-spec is used.
Print the arithmetic mean value rather than the sum when the –b or –i is used to aggregate data over multiple CPUs.
Omits all header output (useful if cpustat is the beginning of a pipeline).
Print only the first num rows within each sampling period, after applying sorting and aggregation options.
Save all data to statfile. This data may be replayed at a later time using –I.
Write to the standard output if the file name is — (hyphen).
The purpose of –O is to capture all available data. It is incompatible with the data reduction options: –A, –k, –m and –o.
Causes cpustat to cycle through the list of eventspecs every period seconds. The tool sleeps after each cycle until period seconds have elapsed since the first eventspec was measured.
When this option is present, the optional count parameter specifies the number of total cycles to make (instead of the number of total samples to take). If period is less than the number of eventspecs times interval, the tool acts as it period is 0.
Creates an idle soaker thread to spin while system-only eventspecs are bound. One idle soaker thread is bound to each CPU in the current processor set. System-only eventspecs contain both the nouser and the sys tokens and measure events that occur while the CPU is operating in privileged mode. This option prevents the kernel's idle loop from running and triggering system-mode events.
Display a time stamp.
Prints an additional column of processor cycle counts, if available on the current architecture.
A closely related utility, cputrack(1), can be used to monitor the behavior of individual applications with little or no interference from other activities on the system.
The cpustat utility must be run by the super-user, as there is an intrinsic conflict between the use of the CPU performance counters system-wide by cpustat and the use of the CPU performance counters to monitor an individual process (for example, by cputrack.)
Once any instance of this utility has started, no further per-process or per-LWP use of the counters is allowed until the last instance of the utility terminates.
The times printed by the command correspond to the wallclock time when the hardware counters were actually sampled, instead of when the program told the kernel to sample them. The time is derived from the same timebase as gethrtime(3C).
The processor cycle counts enabled by the –t option always apply to both user and system modes, regardless of the settings applied to the performance counter registers.
On some hardware platforms running in system mode using the “sys” token, the counters are implemented using 32-bit registers. While the kernel attempts to catch all overflows to synthesize 64-bit counters, because of hardware implementation restrictions, overflows can be lost unless the sampling interval is kept short enough. The events most prone to wrap are those that count processor clock cycles. If such an event is of interest, sampling should occur frequently so that less than 4 billion clock cycles can occur between samples.
The output of cpustat is designed to be readily parseable by nawk(1) and perl(1), thereby allowing performance tools to be composed by embedding cpustat in scripts. Alternatively, tools can be constructed directly using the same APIs that cpustat is built upon using the facilities of libcpc(3LIB). See cpc(3CPC).
The cpustat utility only monitors the CPUs that are accessible to it in the current processor set. Thus, several instances of the utility can be running on the CPUs in different processor sets. See psrset(1M) for more information about processor sets.
Because cpustat uses LWPs bound to CPUs, the utility might have to be terminated before the configuration of the relevant processor can be changed.
The following example measures misses and references in the external cache. These occur while the processor is operating in user mode on an UltraSPARC machine.
example% cpustat -c EC_ref,EC_misses 1 3 time cpu event pic0 pic1 1.008 0 tick 69284 1647 1.008 1 tick 43284 1175 2.008 0 tick 179576 1834 2.008 1 tick 202022 12046 3.008 0 tick 93262 384 3.008 1 tick 63649 1118 3.008 2 total 651077 18204
The following example measures branch mispredictions and total branch instructions in user and system mode on a Pentium 4 machine.
example% cpustat -c \ pic12=branch_retired,emask12=0x4,pic14=branch_retired,\ emask14=0xf,sys 1 3 time cpu event pic12 pic14 1.010 1 tick 458 684 1.010 0 tick 305 511 2.010 0 tick 181 269 2.010 1 tick 469 684 3.010 0 tick 182 269 3.010 1 tick 468 684 3.010 2 total 2063 3101Example 3 Counting Memory Accesses on Opteron
The following example determines the number of memory accesses made through each memory controller on an Opteron, broken down by internal memory latency:
cpustat -c \ pic0=NB_mem_ctrlr_page_access,umask0=0x01, \ pic1=NB_mem_ctrlr_page_access,umask1=0x02, \ pic2=NB_mem_ctrlr_page_access,umask2=0x04,sys \ 1 time cpu event pic0 pic1 pic2 1.003 0 tick 41976 53519 7720 1.003 1 tick 5589 19402 731 2.003 1 tick 6011 17005 658 2.003 0 tick 43944 45473 7338 3.003 1 tick 7105 20177 762 3.003 0 tick 47045 48025 7119 4.003 0 tick 43224 46296 6694 4.003 1 tick 5366 19114 652Example 4 Displaying Multiple CPUs with a Filter
The following command displays the three CPUs with the highest DTLB_miss rate.
example% cpustat -c DTLB_miss -k DTLB_miss -n 3 1 1 time cpu event DTLB_miss 1.040 115 tick 107 1.006 18 tick 98 1.045 126 tick 31 1.046 96 total 236 event DTLB_miss total 236Example 5 Aggregating Multiple CPUs into Quartiles by a Filter
The following command aggregates 256 CPUs into quartiles by DTLB miss rate.
example% cpustat -c DTLB_miss -b 4 -k DTLB_miss -m 1 1 time bin event DTLB_miss sze 1.032 0 tick 46 24 1.021 1 tick 3 24 1.007 2 tick 2 24 1.022 3 tick 0 24 1.045 4 total 51 24 event DTLB_miss total 51Example 6 Sorting Multiple Events
The following sequence of commands sorts multiple events.
example% cpustat -O /tmp/OUT -c ITLB_miss,DTLB_miss -c PAPI_tot_ins 1 2 example% cpustat -I /tmp/OUT -b 4 -k ITLB_miss -k PAPI_tot_ins time bin event ITLB_miss DTLB_miss sze 1.020 0 tick 129 673 24 1.009 1 tick 0 61 24 1.005 2 tick 0 79 24 1.039 3 tick 0 64 24 1.082 4 total 129 877 24 time bin event PAPI_tot_ins sze 2.073 0 tick 51947 24 2.020 1 tick 14976 24 2.076 2 tick 14976 24 2.004 3 tick 14976 24 2.082 4 total 96875 24 event ITLB_miss DTLB_miss PAPI_tot_ins total 129 877 96875
By running the cpustat command, the super-user forcibly invalidates all existing performance counter context. This can in turn cause all invocations of the cputrack command, and other users of performance counter context, to exit prematurely with unspecified errors.
If cpustat is invoked on a system that has CPU performance counters which are not supported by Solaris, the following message appears:
cpustat: cannot access performance counters - Operation not applicable
This error message implies that cpc_open() has failed and is documented in cpc_open(3CPC). Review this documentation for more information about the problem and possible solutions.
If a short interval is requested, cpustat might not be able to keep up with the desired sample rate. In this case, some samples might be dropped.
See attributes(5) for descriptions of the following attributes:
When cpustat is run on a Pentium 4 with HyperThreading enabled, a CPC set is bound to only one logical CPU of each physical CPU. See cpc_bind_cpu(3CPC).