etrace - End-to-end tracing utility
etrace [options] subcommand [subcmd-options] [operands]
etrace create-session -p provider-name
[-a argument-name=argument-value,...] [-l span=span_name,
base=baseval,lo_exp=lo_expval,hi_exp=hi_expval] [-t]
session-name
etrace delete-session [-u user-name] session-name
etrace undelete-session [-u user-name] session-name
etrace show-session [-a] [-d] [[-p] -o field[,...]] [-P provider-name]
[session-name]
etrace show-session [-a] [[-p] -o field[,...]] [-P provider-name]
-s [interval [count]] [session-name]
etrace show-provider [[-p] -o field[,...]] [provider-name]
etrace show-provider -i provider-name
etrace help [subcommand-name]
End-to-end tracing allows the tracing of operations across multiple nodes, OS layers, and application components. Traces are generated by new DTrace APIs, and information about collected traces can be obtained through the etracequery(3RAD) APIs and by using this tool. The End-to-End Tracing Guide provides more background and detailed information on this feature.
The etrace command allows the creation of an etrace session. A session consists of a collection of DTrace scripts enabled on a set of nodes. Sessions are automatically started on a set of nodes according to the specified session arguments.
Each session is associated with the user that created it. The session name space is separate for each user, and users do not see the sessions created by other users.
Because sessions consume system resources and may have a performance impact on the running system since they rely on DTrace, a user must have the solaris.system.etrace.session-user authorization to create or manage sessions.
The session management sub-commands are user-aware, so that a user sees only its own sessions and cannot delete sessions created by other users. There is one exception to this rule: a user with the new solaris.system.etrace.session-manage authorization is allowed to list all sessions or delete any sessions. Note that the solaris.system.etrace.session-manage authorization needs to be assigned in addition to, and does not include the solaris.system.etrace.session-user authorization.
Two new profiles which include these authorizations are being introduced as well ("End-to-End Tracing Session User" and "End-to-End Tracing Session Manage".)
Deleted sessions are remembered until the server host reboots or the server service restarts. Deleted sessions can be retrieved with the undelete-session subcommand. The maximum number of sessions which can remembered per-user can be configured by using the etrace:server SMF service config/max-user-deleted-sessions property.
Each session is also associated with a provider. A provider consists of a set of DTrace scripts and metadata corresponding to a specific subsystem. Providers allow users to leverage end-to-end tracing without detailed knowledge about the subsystem being observed.
Each provider takes a set of values as arguments. The arguments names supported by providers are described in the providers metadata, which be obtained by running the show-provider -i subcommand.
Some of the argument supported by a provider, define the roles of the provider. Such argument have their "role" field set to "true". A role argument specifies the IP address or host name of the node on which a provider role must be enabled.
When a session is created, statistics will be created for each event span supported by the provider. These statistics are created as Statistics Store (sstore) statistics. For more information, see the sstore(1) man page.
End-to-end sstore statistics are created under the //:class.etrace class. Statistics associated with a session are created as resource type //:res.session and session names of the form <user>@<session>/span:<span> where "user" and "session" are the id of the user that created the session, and the session name specified when creating the session. One such resource will be created for every session span defined by the provider metadata, and "span" corresponds to the name of a span.
The following subcommands are supported:
Creates a new session using the specified provider and provider arguments. The name of the new session is specified by the session-name argument. This subcommand requires the solaris.system.etrace.session-user authorization.
Name of the provider to be associated with the session. A list of currently supported providers by the current clients can be obtained by using the show-provider subcommand.
Each option of that form specifies an argument to be passed to the provider. The list of supported arguments for a provider can be obtained from the provider information displayed by the show-provider -i subcommand. The argument value can be a single value, or a comma-separated list of values.
Enable llquantize operation for a specified span, with provided arguments. The order of argument input does not matter, but the option must be provided with 5 arguments in comma-separated values.
Temporary sessions will not persist through a reboot, and their information will be lost. All sessions will persist through a restart of the svc:/system/etrace:server service.
Deletes the session of name specified by session-name of the user that invokes the command.
1000000000 ns 0 0.00% 100.00%
Name of the user for which the specified session must be deleted. Requires the solaris.system.etrace.session-manage authorization.
Undeletes the session of name specified by session-name of the user that invokes the command.
Name of the user for which the specified session must be undeleted. Requires the solaris.system.etrace.session-manage authorization.
Shows the sessions associated with the current user, or all users if the –a option is specified.
A case-insensitive, comma-separated list of output fields to display. The field name must be one of the fields listed below, or the special value, all, to display all fields. By default (without the –o option), show-session displays the UID, SESSION, and PROVIDER fields.
Name of the user that created the session.
Name of the session.
Name of the provider.
Unique session identifier.
State of the session, which can be one of the following:
Session is currently active.
Session was deleted.
Displays the output in a stable and machine-parseable format. The –o option is required along with –p option.
Shows the information about all sessions, and requires the solaris.system.etrace.session-manage authorization.
When specified, lists all deleted sessions in addition to the active sessions. By default, only active sessions are listed. Using this option shows the STATE field by default, in addition to the default fields listed above.
Shows the stats of sessions associated with current user or all users if the –a option is specified.
Shows the information about all sessions and requires the solaris.system.etrace.session-manage authorization.
A case-insensitive, comma-separated list of output fields to display. The field name must be one of the fields listed below, or the special value, all, to display all fields. By default (without the –o option), show-session command displays the SESSION, and SPAN:AVG/STD-DEV field.
Name of the session.
Stat of the session. The supported spans are specified by the provider metadata. For more information, see the show-provider subcommand.
<span>:avg (latency average for the specified span) and <span>:std-dev (Latency standard deviation for the specified span) are the supported spans.
Displays the output in a stable and machine-parseable format. The –o option is required along with –p option.
Shows all sessions associated with a provider.
Shows stats associated with a particular session. By default, it shows the average latency of the spans defined by the provider metadata. This option can be combined with –o option to show specific spans, or to show the standard deviation for spans latency.
If specified, it denotes the interval at which output rows are refreshed. All the rows show the current value of stats.
If specified, only prints the 'count' rows. If not specified, prints indefinitely.
Shows the supported providers.
A case-insensitive, comma-separated list of output fields to display. The field name must be one of the fields listed below, or the special value, all, to display all fields. By default (without the –o option), show-provider displays all fields.
Name of the provider.
The tracing clients which support the provider. If multiple clients support a provider, one row will be printed per client.
Displays the output in a stable and machine-parseable format. The –o option is required along with –p option.
Displays the metadata associated with the specified provider in JSON format.
Display usage information for the specified subcommand or all subcommands if none is specified.
The following options are supported across all subcommands, and must be specified before the subcommand name.
String value containing the rad URI to connect to the End-to-end tracing server service. If not specified, the command attempts to connect to the server running on the current node with the URI unix://.
The URI can be of the form ssh://[username@]servername or unix://.
The following command displays how to create a session.
user@host> etrace create-session -p prov -a client=10.10.10.1,10.10.10.2 \ -a server=10.10.10.3 exampleExample 2 Showing the List of Sessions
The following example displays how to list the sessions.
user@host> etrace show-session UID SESSION PROVIDER user example provExample 3 Showing the List of End-to-end Tracing Statistics Available
The following example displays the available end-to-end tracing statistics.
user@host> sstore list //:class.etrace//:res.session/* IDENTIFIER //:class.etrace//:res.session/user@example/span:pre-work //:class.etrace//:res.session/user@example/span:send-req user@host> sstore list \ //:class.etrace//:res.session/user@example/span:*//:stat.* IDENTIFIER //:class.etrace//:res.session/user@example/span:pre-work//:stat.latency-average //:class.etrace//:res.session/user@example/span:pre-work//:stat.latency-std-deviation //:class.etrace//:res.session/user@example/span:send-req//:stat.latency-average //:class.etrace//:res.session/user@example/span:send-req//:stat.latency-std-deviationExample 4 Deleting a Session
The following example displays how to delete a session.
user@host> etrace delete-session exampleExample 5 Undeleting a Session
The following example displays how to undelete a session.
user@host> etrace show-session -d UID SESSION PROVIDER STATE user example prov DELETED user@host> etrace undelete-session example user@host> etrace show-session -d UID SESSION PROVIDER STATE user example prov ACTIVEExample 6 Showing the List of Providers and the Clients that Support Them
The following example lists of providers and the clients that support them.
user@host> etrace show-provider PROVIDER CLIENTS nfs3 10.10.10.1 nfs3 10.10.10.2 nfs3 10.10.10.3 nfs4 10.10.10.1 nfs4 10.10.10.2 nfs4 10.10.10.3Example 7 Showing the Default stats for a Session
The following example reports the default stats for a session.
user@host:/usr/sbin/etrace show-session -s example 2016-05-09T17:18:44 session client-preprocess:avg network-request:avg network-reply:avg client-postprocess:avg server-process:avg example 53162 6681596 7333041 6908 133254Example 8 Showing all the Default Session stats for nfs4 Provider
The following example reports all the default session stats for nfs4 provider.
uesr@host:/usr/sbin/etrace show-session -P nfs4 2016-05-09T17:18:44 session client-preprocess:avg network-request:avg network-reply:avg client-postprocess:avg server-process:avg example 53162 6681596 7333041 6908 133254 demo 89340 7281217 7557193 7464 103798Example 9 Creating Session With llquantize Operation Enabled
The following example enables an llquantize operation for network-reply span in session1, with following arguments: base = 10, lo_exp = 7, hi_exp = 8, steps = 1.
user@host:/usr/sbin/etrace create-session -p nfs4 -a client=10.10.10.1\ -a server=10.10.10.3 -l base=10,lo_exp=7,hi_exp=8,steps=1,span=network-\ reply session1 user@host:/usr/sbin/etrace show-session -s -o network-reply:llquantize \ session1 2016-08-26T15:29:02 NETWORK-REPLY:LLQUANTIZE latency range count prob.density cum. distribution < 10000000 ns 2 3.45% 3.45% 10000000 ns 0 0.00% 3.45% 100000000 ns 56 96.55% 100.00% > 1000000000 ns 0 0.00% 100.00%Example 10 Creating Temporary Sessions
The following example creates session1 as a temporary session.
user@host:/usr/sbin/etrace create-session -p nfs4 -a client=10.10.10.1\
-a server=10.10.10.3 -t session1
user@host:/usr/sbin/etrace show-session
UID SESSION PROVIDER
root session1 nfs4
user@host:svcadm restart etrace:server
user@host:/usr/sbin/etrace show-session
UID SESSION PROVIDER
root session1 nfs4
user@host: reboot
...
user@host: show-session
UID SESSION PROVIDER
See attributes(7) for descriptions of the following attributes:
|
Screen output is Uncommitted. The invocation is Committed.