Go to main content

man pages section 8: System Administration Commands

Exit Print View

Updated: Thursday, June 13, 2019
 
 

etrace (8)

Name

etrace - End-to-end tracing utility

Synopsis

etrace [options] subcommand [subcmd-options] [operands]
etrace create-session -p provider-name
        [-a argument-name=argument-value,...] [-l span=span_name,
        base=baseval,lo_exp=lo_expval,hi_exp=hi_expval] [-t]
        session-name
        
etrace delete-session [-u user-name] session-name
etrace undelete-session [-u user-name] session-name
etrace show-session [-a] [-d] [[-p] -o field[,...]] [-P provider-name]
     [session-name]
etrace show-session [-a] [[-p] -o field[,...]] [-P provider-name]
     -s [interval [count]] [session-name]
etrace show-provider [[-p] -o field[,...]] [provider-name]
etrace show-provider -i provider-name
etrace help [subcommand-name]

Description

End-to-end tracing allows the tracing of operations across multiple nodes, OS layers, and application components. Traces are generated by new DTrace APIs, and information about collected traces can be obtained through the etracequery(3RAD) APIs and by using this tool. The End-to-End Tracing Guide provides more background and detailed information on this feature.

The etrace command allows the creation of an etrace session. A session consists of a collection of DTrace scripts enabled on a set of nodes. Sessions are automatically started on a set of nodes according to the specified session arguments.

Each session is associated with the user that created it. The session name space is separate for each user, and users do not see the sessions created by other users.

Because sessions consume system resources and may have a performance impact on the running system since they rely on DTrace, a user must have the solaris.system.etrace.session-user authorization to create or manage sessions.

The session management sub-commands are user-aware, so that a user sees only its own sessions and cannot delete sessions created by other users. There is one exception to this rule: a user with the new solaris.system.etrace.session-manage authorization is allowed to list all sessions or delete any sessions. Note that the solaris.system.etrace.session-manage authorization needs to be assigned in addition to, and does not include the solaris.system.etrace.session-user authorization.

Two new profiles which include these authorizations are being introduced as well ("End-to-End Tracing Session User" and "End-to-End Tracing Session Manage".)

Deleted sessions are remembered until the server host reboots or the server service restarts. Deleted sessions can be retrieved with the undelete-session subcommand. The maximum number of sessions which can remembered per-user can be configured by using the etrace:server SMF service config/max-user-deleted-sessions property.

Each session is also associated with a provider. A provider consists of a set of DTrace scripts and metadata corresponding to a specific subsystem. Providers allow users to leverage end-to-end tracing without detailed knowledge about the subsystem being observed.

Each provider takes a set of values as arguments. The arguments names supported by providers are described in the providers metadata, which be obtained by running the show-provider -i subcommand.

Some of the argument supported by a provider, define the roles of the provider. Such argument have their "role" field set to "true". A role argument specifies the IP address or host name of the node on which a provider role must be enabled.

Statistics

When a session is created, statistics will be created for each event span supported by the provider. These statistics are created as Statistics Store (sstore) statistics. For more information, see the sstore(1) man page.

End-to-end sstore statistics are created under the //:class.etrace class. Statistics associated with a session are created as resource type //:res.session and session names of the form <user>@<session>/span:<span> where "user" and "session" are the id of the user that created the session, and the session name specified when creating the session. One such resource will be created for every session span defined by the provider metadata, and "span" corresponds to the name of a span.

Sub Commands

The following subcommands are supported:

etrace create-session –p provider-name [–a argument-name=argument-value,...] [–l span=span_name,base=baseval,lo_exp=lo_expval, hi_exp=hi_expval] [-t] session-name

Creates a new session using the specified provider and provider arguments. The name of the new session is specified by the session-name argument. This subcommand requires the solaris.system.etrace.session-user authorization.

–p provider-name

Name of the provider to be associated with the session. A list of currently supported providers by the current clients can be obtained by using the show-provider subcommand.

–a argument-name=argument-value

Each option of that form specifies an argument to be passed to the provider. The list of supported arguments for a provider can be obtained from the provider information displayed by the show-provider -i subcommand. The argument value can be a single value, or a comma-separated list of values.

–l span=span-name,base=baseval,lo_exp=lo_expval,hi_exp=hi_expval

Enable llquantize operation for a specified span, with provided arguments. The order of argument input does not matter, but the option must be provided with 5 arguments in comma-separated values.

–t

Temporary sessions will not persist through a reboot, and their information will be lost. All sessions will persist through a restart of the svc:/system/etrace:server service.

etrace delete-session [–u user-name] session-name

Deletes the session of name specified by session-name of the user that invokes the command.

1000000000 ns               0             0.00%           100.00%
–u user-name

Name of the user for which the specified session must be deleted. Requires the solaris.system.etrace.session-manage authorization.

etrace undelete-session [–u user-name] session-name

Undeletes the session of name specified by session-name of the user that invokes the command.

–u user-name

Name of the user for which the specified session must be undeleted. Requires the solaris.system.etrace.session-manage authorization.

etrace show-session [–a] [–d] [[–p] –o field[,...]] [–P provider-name] [session-name]

Shows the sessions associated with the current user, or all users if the –a option is specified.

–o field[,...], –-output=field[,...]

A case-insensitive, comma-separated list of output fields to display. The field name must be one of the fields listed below, or the special value, all, to display all fields. By default (without the –o option), show-session displays the UID, SESSION, and PROVIDER fields.

UID

Name of the user that created the session.

SESSION

Name of the session.

PROVIDER

Name of the provider.

UUID

Unique session identifier.

STATE

State of the session, which can be one of the following:

ACTIVE

Session is currently active.

DELETED

Session was deleted.

–p

Displays the output in a stable and machine-parseable format. The –o option is required along with –p option.

–a

Shows the information about all sessions, and requires the solaris.system.etrace.session-manage authorization.

–d

When specified, lists all deleted sessions in addition to the active sessions. By default, only active sessions are listed. Using this option shows the STATE field by default, in addition to the default fields listed above.

etrace show-session [–a] [[–p] –o field[,...]] [–P provider-name] –s [interval [count]] [session-name]

Shows the stats of sessions associated with current user or all users if the –a option is specified.

–a

Shows the information about all sessions and requires the solaris.system.etrace.session-manage authorization.

–o field[,...], –-output=field[,...]

A case-insensitive, comma-separated list of output fields to display. The field name must be one of the fields listed below, or the special value, all, to display all fields. By default (without the –o option), show-session command displays the SESSION, and SPAN:AVG/STD-DEV field.

SESSION

Name of the session.

SPAN:AVG/STD-DEV

Stat of the session. The supported spans are specified by the provider metadata. For more information, see the `show-provider' subcommand.

<span>:avg (latency average for the specified span) and <span>:std-dev (Latency standard deviation for the specified span) are the supported spans.

–p

Displays the output in a stable and machine-parseable format. The –o option is required along with –p option.

–P provider-name

Shows all sessions associated with a provider.

–s

Shows stats associated with a particular session. By default, it shows the average latency of the spans defined by the provider metadata. This option can be combined with –o option to show specific spans, or to show the standard deviation for spans latency.

interval

If specified, it denotes the interval at which output rows are refreshed. All the rows show the current value of stats.

count

If specified, only prints the 'count' rows. If not specified, prints indefinitely.

etrace show-provider [[–p] –o field[,...]] [provider-name]

Shows the supported providers.

–o field[,...], –-output=field[,...]

A case-insensitive, comma-separated list of output fields to display. The field name must be one of the fields listed below, or the special value, all, to display all fields. By default (without the –o option), show-provider displays all fields.

PROVIDER

Name of the provider.

CLIENTS

The tracing clients which support the provider. If multiple clients support a provider, one row will be printed per client.

–p

Displays the output in a stable and machine-parseable format. The –o option is required along with –p option.

etrace show-provider –i provider-name

Displays the metadata associated with the specified provider in JSON format.

etrace help [subcommand-name]

Display usage information for the specified subcommand or all subcommands if none is specified.

Options

The following options are supported across all subcommands, and must be specified before the subcommand name.

–s server-uri

String value containing the rad URI to connect to the End-to-end tracing server service. If not specified, the command attempts to connect to the server running on the current node with the URI unix://.

The URI can be of the form ssh://[username@]servername or unix://.

Examples

Example 1 Creating a Session

The following command displays how to create a session.

user@host> etrace create-session -p prov -a client=10.10.10.1,10.10.10.2 \
-a server=10.10.10.3 example
Example 2 Showing the List of Sessions

The following example displays how to list the sessions.

user@host> etrace show-session
UID             SESSION         PROVIDER
user            example         prov
Example 3 Showing the List of End-to-end Tracing Statistics Available

The following example displays the available end-to-end tracing statistics.

user@host> sstore list //:class.etrace//:res.session/*
IDENTIFIER
//:class.etrace//:res.session/user@example/span:pre-work
//:class.etrace//:res.session/user@example/span:send-req

user@host> sstore list \
//:class.etrace//:res.session/user@example/span:*//:stat.*
IDENTIFIER
//:class.etrace//:res.session/user@example/span:pre-work//:stat.latency-average
//:class.etrace//:res.session/user@example/span:pre-work//:stat.latency-std-deviation
//:class.etrace//:res.session/user@example/span:send-req//:stat.latency-average
//:class.etrace//:res.session/user@example/span:send-req//:stat.latency-std-deviation
Example 4 Deleting a Session

The following example displays how to delete a session.

user@host> etrace delete-session example
Example 5 Undeleting a Session

The following example displays how to undelete a session.

user@host> etrace show-session -d
UID             SESSION         PROVIDER        STATE  
user            example         prov            DELETED

user@host> etrace undelete-session example

user@host> etrace show-session -d
UID             SESSION         PROVIDER        STATE  
user            example         prov            ACTIVE
Example 6 Showing the List of Providers and the Clients that Support Them

The following example lists of providers and the clients that support them.

user@host> etrace show-provider
PROVIDER        CLIENTS        
nfs3            10.10.10.1
nfs3            10.10.10.2
nfs3            10.10.10.3
nfs4            10.10.10.1
nfs4            10.10.10.2
nfs4            10.10.10.3
Example 7 Showing the Default stats for a Session

The following example reports the default stats for a session.

user@host:/usr/sbin/etrace show-session -s example
2016-05-09T17:18:44
session  client-preprocess:avg  network-request:avg  network-reply:avg  client-postprocess:avg  server-process:avg
example  53162                  6681596              7333041            6908                    133254
Example 8 Showing all the Default Session stats for nfs4 Provider

The following example reports all the default session stats for nfs4 provider.

uesr@host:/usr/sbin/etrace show-session -P nfs4
2016-05-09T17:18:44
session  client-preprocess:avg  network-request:avg  network-reply:avg  client-postprocess:avg  server-process:avg
example  53162                  6681596              7333041            6908                    133254
demo     89340                  7281217              7557193            7464                    103798
Example 9 Creating Session With llquantize Operation Enabled

The following example enables an llquantize operation for network-reply span in session1, with following arguments: base = 10, lo_exp = 7, hi_exp = 8, steps = 1.

user@host:/usr/sbin/etrace create-session -p nfs4 -a client=10.10.10.1\
-a server=10.10.10.3 -l base=10,lo_exp=7,hi_exp=8,steps=1,span=network-\
reply session1

user@host:/usr/sbin/etrace show-session -s -o network-reply:llquantize \
session1
2016-08-26T15:29:02
NETWORK-REPLY:LLQUANTIZE
  latency   range           count        prob.density  cum. distribution
< 10000000   ns               2             3.45%             3.45%
  10000000   ns               0             0.00%             3.45%
  100000000  ns              56            96.55%           100.00%
> 1000000000 ns               0             0.00%           100.00%
Example 10 Creating Temporary Sessions

The following example creates session1 as a temporary session.

    user@host:/usr/sbin/etrace create-session -p nfs4 -a client=10.10.10.1\
    -a server=10.10.10.3 -t session1
	        
    user@host:/usr/sbin/etrace show-session
    UID             SESSION         PROVIDER
    root            session1        nfs4
	      
    user@host:svcadm restart etrace:server
	      
    user@host:/usr/sbin/etrace show-session
    UID             SESSION         PROVIDER
    root            session1        nfs4
	      
    user@host: reboot
	      
    ...
	        
    user@host: show-session
UID             SESSION         PROVIDER

Attributes

See attributes(7) for descriptions of the following attributes:

ATTRIBUTE TYPE
ATTRIBUTE VALUE
Availability
system/etrace
Interface Stability
See below

Screen output is Uncommitted. The invocation is Committed.

See Also

etraced(8), etracestat(8)