Debugging a Program With dbx

Chapter 6 Event Management

Event management refers to the capability of dbx to perform actions when events take place in the program being debugged.

This chapter is organized into the following sections:

Basic Concepts

Event management is based on the concept of a handler. The name comes from an analogy with hardware interrupt handlers. Each event management command typically creates a handler, which consists of an event specification and a series of side-effect actions.

An example of the association of a program event with a dbx action is setting a breakpoint on a particular line.

The most generic form of creating a handler is through the when command:


when event-specification
 {action;
... }

Although all event management can be performed through when, dbx has historically had many other commands, which are still retained, either for backward compatibility, or because they are simpler and easier to use.

In many places examples are given on how a command (like stop, step, or ignore) can be written in terms of when. These examples are meant to illustrate the flexibility of when and the underlying handler mechanism, but they are not always exact replacements.

Creating Event Handlers

The commands when, stop, and trace are used to create event handlers. An event-spec is a specification of an event as documented later in this chapter.

Every command returns a number known as a handler id (hid). This number can be accessed via the predefined variable $newhandlerid.

An attempt has been made to make the stop and when commands conform to the handler model. However, backward compatibility with previous dbx releases forces some deviations.

For example, the following samples from an earlier dbx release are equivalent.

Old 

New 

when cond body

when step -if cond body

when cond in func body

when next -if cond -in func body

These samples illustrate that cond is not a pure event; there is no internal handler for conditions.

when

When the event specified by the when command specified event occurs, the cmds are executed. Once the commands have all executed, the process is automatically continued.


when event-specification [ modifier 
] { cmds ... ; 
}

stop

When the event specified by the stop command occurs, the process is stopped.


stop event-specification
 [ modifier]

stop is shorthand for a common when idiom:


when event-specification { stop -update;
whereami; }

trace

When the event specified by the trace command oc


trace event-specification

curs, a trace message is printed:

Most of the trace commands can be hand-crafted by using the when command, ksh functionality, and event variables. This is especially useful if you want stylized tracing output.

Manipulating Event Handlers

The following list contains commands to manipulate event handlers. For more information on any of the commands, see "Command Reference".

Using Event Counters

Event handlers have trip counters. There is a count limit and the actual counter. Whenever the event occurs, the counter is incremented. The action associated with the handler executes only if the count reaches the limit, at which point the counter is automatically reset to 0. The default limit is 1. Whenever a process is rerun, all event counters are reset.

The count limit can be set using the -count modifier. Otherwise, use the handler command to individually manipulate event handlers:


handler [ -count | -reset ] hid new-count
 new-count-limit

Setting Event Specifications

Event specifiers are used by the stop, when and trace commands to denote event types and parameters. The format is that of a keyword to represent the event type and optional parameters.

Breakpoint Event Specifications

The following are event specifications, syntax and descriptions for breakpoint events.

in func

The function has been entered and the first line is about to be executed. If the -instr modifier is used, it is the first instruction of the function about to be executed. (Do not confuse in func with the -in func modifier.) The func specification can take a formal parameter signature to help with overloaded function names, or template instance specification. For example, you can say:


 stop in mumble(int, float, struct Node *)

at lineno

The designated line is about to be executed. If filename is specified, then the designated line in the specified file is about to be executed. The file name can be the name of a source file or an object file. Although quote marks are not required, they may be necessary if the file name contains odd characters.


at filename:lineno

If the designated line is in template code, a breakpoint is placed on all instances of that template.

infunction func

Equivalent to in func for all overloaded functions named func, or all template instantiations thereof.

inmember funcinmethod func

Equivalent to in func for the member function named func for every class.

inclass classname

Equivalent to in func for all member functions that are members of classname.

inobject obj-expr

A member function called on the specific object at the address denoted by obj-expr has been called.

Watchpoint Event Specifications

The following are event specifications, syntax, and descriptions for watchpoint events.

access mode addr-exp, [byte-size-exp]

The memory specified by addr-exp has been accessed.

addr-exp is any expression that can be evaluated to produce an address. If a symbolic expression is used, the size of the region to be watched is automatically deduced, or you can override that with the `,` syntax. You can also use nonsymbolic, typeless address expressions; in which case, the size is mandatory. For example:


stop modify 0x5678, sizeof(Complex)

mode specifies that the memory was accessed. It can be composed of one or all of the letters:

r

The memory has been read. 

w

The memory has been written to. 

x

The memory has been executed. 

mode can also contain one of:

a

Stops the process after the access (default). 

b

Stops the process before the access. 

In both cases the program counter will point at the offending instruction. The before and after refer to the side effect.

access is a replacement for modify. While both syntaxes work on Solaris 2.4, 2.5, 2.5.1, 2.6, and Solaris 7, on all of these operating environments except Solaris 2.6, access suffers the same limitations as modify and accepts only a mode of wa.

Limitations of access

No two matched regions may overlap.

change variable

The value of variable has changed.

cond cond-expr

The condition denoted by cond-expr evaluates to true. Any expression can be used for cond-expr, but it has to evaluate to an integral type.

modify addr-exp [ , byte-size ]

The specified address range has been modified. This is the older watchpoint facility.

addr-exp is any expression that can be evaluated to produce an address. If a symbolic expression is used, the size of the region to be watched is automatically deduced, or you can override that with the `,` syntax. You can also use nonsymbolic, typeless address expressions; in which case, the size is mandatory. For example:


stop modify 0x5678, sizeof(Complex)

Limitations of modify event-spec on Solaris 2.5.1

Addresses on the stack cannot be watched.

The event does not occur if the address being watched is modified by a system call.

Shared memory (MAP_SHARED) cannot be watched, because dbx cannot catch the other processes stores into shared memory. Also, dbx cannot properly deal with SPARC swap and ldstub instructions.

Addresses that do not exist at the time a handler for this event is created cannot be watched.


Note -

Multithreaded applications are prone to deadlock so mt watchpoints are nominally disallowed. They can be turned on by setting the dbx environment variable mt_watchpoints.


System Event Specifications

The following are event specifications, syntax, and descriptions for system events.

dlopen [ lib-path ] | dlclose [ lib-path ]

These events are fired after a dlopen() or a dlclose() call succeeds. A dlopen() or dlclose() call can cause more than one library to be loaded. The list of these libraries is always available in the predefined variable $dllist. The first word in $dllist is actually a "+" or a "-", indicating whether the list of libraries is being added or deleted.

lib-path is the name of a shared library you are interested in. If it is specified, the event only fires if the given library was loaded or unloaded. In that case $dlobj contains the name of the library. $dllist is still available.

If lib-path begins with a /, a full string match is performed. Otherwise, only the tails of the paths are compared.

If lib-path is not specified, then the events always occur whenever there is any dl-activity. In this case, $dlobj is empty but $dllist is valid.

fault fault

This event fires when the specified fault occurs. The faults are architecture dependent, but a set of them is known to dbx as defined by proc(4):

FLTILL

Illegal instruction 

FLTPRIV

Privileged instruction 

FLTBPT

Breakpoint instruction 

FLTTRACE*

Trace trap (single step) 

FLTACCESS*

Memory access (such as alignment) 

FLTBOUNDS*

Memory bounds (invalid address) 

FLTIOVF

Integer overflow 

FLTIZDIV

Integer zero divide 

FLTPE

Floating-point exception 

FLTSTACK

Irrecoverable stack fault 

FLTPAGE

Recoverable page fault 


Note -

Be aware that BPT, TRACE, and BOUNDS are used by dbx to implement breakpoints, single-stepping, and watchpoints. Handling them may interfere with the inner workings of dbx.


These faults are taken from /sys/fault.h. fault can be any of those listed above, in upper or lower case, with or without the FLT- prefix, or the actual numerical code.

lwp_exit

Fired when lwp has been exited. $lwp contains the id of the exited LWP.

sig sig

This event occurs when the signal is first delivered to the debugee. sig can either be a decimal number or the signal name in upper or lower case; the prefix is optional. This is completely independent of the catch/ignore commands, although the catch command can be implemented as follows:


function simple_catch {
	when sig $1 {
			stop;
			echo Stopped due to $sigstr $sig
			whereami
	}
}


Note -

When the sig event is received, the process has not seen it yet. Only if you cont the process with the given signal is the signal forwarded to it.


sig sig sub-code

When the specified signal with the specified sub-code is first delivered to the child, this event fires. Just as with signals, the sub-code can be entered as a decimal number, in capital or lower case; the prefix is optional.

sysin code | name

The specified system call has just been initiated and the process has entered kernel mode.

The concept of system call supported by dbx is that provided by procfs(4). These are traps into the kernel as enumerated in /usr/include/sys/syscall.h.

This is not the same as the ABI notion of system calls. Some ABI system calls are partially implemented in user mode and use non-ABI kernel traps. However, most of the generic system calls (the main exception being signal handling) are the same between syscall.h and the ABI.

sysout code | name

The specified system call is finished and the process is about to return to user mode.

sysin | sysout

Without arguments, all system calls are traced. Note that certain dbx features, for example modify event and RTC, cause the child to execute system calls for their own purposes and show up if traced.

Execution Progress Event Specifiers

The following are event specifications, syntax, and descriptions for events pertaining to execution progress.

next

Similar to step except that functions are not stepped into.

returns

This event is just a breakpoint at the return point of the current visited function. The visited function is used so that you can use the returns event spec after doing a number of up's. The plain returns event is always -temp and can only be created in the presence of a live process.

returns func

This event executes each time the given function returns to its call site. This is not a temporary event. The return value is not provided, but you can find integral return values by accessing:

Sparc 

$o0

Intel 

$eax

It is another way of saying:


when in func { stop returns; }

step

The step event occurs when the first instruction of a source line is executed. For example, you can get simple tracing with:


when step { echo $lineno: $line; }

When enabling a step event you instruct dbx to single-step automatically next time cont is used. The step command can be implemented as follows:


alias step="when step -temp { whereami; stop; }; cont"

Other Event Specifications

The following are event specificatons, syntax, and descriptions for other types of events.

attach

dbx has successfully attached to a process.

detach

The debugee has been detached from.

lastrites

The process being debugged is about to expire. There are only three reasons this can happen:

proc_gone

Fired when dbx is no longer associated with a debugged process. The predefined variable $reason will be signal, exit, kill, or detach.

prog_new

Fired when a new program has been loaded as a result of follow exec.


Note -

Handlers for this event are always permanent.


stop

The process has stopped. Whenever the process stops such that the user gets a prompt, particularly in response to a stop handler, this event occurs. For example, the following are equivalent:


display x
when stop {print x;}

sync

The process being debugged has just been executed with exec(). All memory specified in a.out is valid and present but pre-loaded shared libraries have not been loaded yet. For example, printf, although known to dbx, has not been mapped into memory yet.

A stop on this event is ineffective; however, you can use this event with the when command.

syncrtld

This event occurs after a sync (or attach if the process being debugged has not yet processed shared libraries). It executes after the dynamic linker startup code has executed and the symbol tables of all preloaded shared libraries have been loaded, but before any code in the .init section has run.

A stop on this event is ineffective; however, you can use this event with the when command.

throw

This event occurs whenever any exception that is not unhandled or unexpected is thrown by the application.

throw type

If an exception type is specified, only exceptions of that type cause the throw event to occur.

throw -unhandled

-unhandled is a special exception type signifying an exception that is thrown but for which there is no handler.

throw -unexpected

-unexpected is a special exception type signifying an exception that does not satisfy the exception specification of the function that throw it.

timer seconds

Occurs when the debugee has been running for seconds. The timer used with this is shared with collector command. The resolution is in milliseconds, so a floating point value for seconds is acceptable.

Event Specification Modifiers

An event specification modifier sets additional attributes of a handler, the most common kind being event filters. Modifiers have to appear after the keyword portion of an event spec. They all begin with a dash (-), preceded by blanks. Modifiers consist of the following:

-if cond

The condition is evaluated when the event specified by the event-spec occurs. The side effect of the handler is allowed only if the condition evaluates to nonzero.

If -if is used with an event that has an associated singular source location, such as in or at, cond is evaluated in the scope corresponding to that location, otherwise it should be properly qualified with the desired scope.

-in func

The handler is active only while within the given function, or any function called from func. The number of times the function is entered is reference counted so as to properly deal with recursion.

-disable

Create the handler in the disabled state.

-count n-count infinity

Have the handler count from 0. Each time the event occurs, the count is incremented until it reaches n. Once that happens, the handler fires and the counter is reset to zero.

Counts of all enabled handlers are reset when a program is run or rerun. More specifically, they are reset when the sync event occurs.

-temp

Create a temporary handler. Once the event is fired it is automatically deleted. By default, handlers are not temporary. If the handler is a counting handler, it is automatically deleted only when the count reaches 0 (zero).

Use the delete -temp command to delete all temporary handlers.

-instr

Makes the handler act at an instruction level. This replaces the traditional 'i' suffix of most commands. It usually modifies two aspects of the event handler.

-thread tid

The event is executed only if the thread that caused it matches tid.

-lwp lid

The event is executed only if the thread that caused it matches lid.

-hidden

Makes the handler not show up in a regular status command. Use status -h to see hidden handlers.

-perm

Normally all handlers get thrown away when a new program is loaded. Using this modifier causes the handler to be retained across debuggings. A plain delete command will not delete a permanent handler. Use delete -p to delete a permanent handler.

Parsing and Ambiguity

Syntax for event-specs and modifiers is:

Since expressions can have spaces embedded in them, this can cause ambiguous situations. For example, consider the following two commands:


when a -temp
when a-temp

In the first example, even though the application might have a variable named temp, the dbx parser resolves the event-spec in favor of -temp being a modifier. In the second example, a-temp is collectively passed to a language specific expression parser and there must be variables named a and temp or an error occurs. Use parentheses to force parsing.

Using Predefined Variables

Certain read-only ksh predefined variables are provided. The following variables are always valid:

Variable 

Definition 

$pc

Current program counter address (hexadecimal) 

$ins

Disassembly of the current instruction 

$lineno

Current line number in decimal 

$line

Contents of the current line  

$func

Name of the current function 

$vfunc

Name of the current "visiting" function 

$class

Name of the class to which $func belongs

$vclass

Name of the class to which $vfunc belongs

$file

Name of the current file 

$vfile

Name of the current file being visited 

$loadobj

Name of the current loadable object 

$vloadobj

Name of the current loadable object being visited 

$scope

Scope of the current PC in back-quote notation 

$vscope

Scope of the visited PC in back-quote notation 

$funcaddr

Address of $func in hex

$caller

Name of the function calling $func

$dllist

After dlopen or dlclose event, contains the list of load objects just dlopened or dlclosed. The first word of dllist is actually a "+" or a "-" depending on whether a dlopen or a dlclose has occurred.

$newhandlerid

ID of the most recently created handler 

$proc

Process id of the current process being debugged 

$lwp

Lwp id of the current LWP  

$thread

Thread id of the current thread  

$prog

Full pathname of the program being debugged 

$oprog

Old, or original value of $prog. This is very handy for getting back to what you were debugging following an exec().

$exitcode

Exit status from the last run of the program. The value is an empty string if the process hasn't actually exited. 

As an example, consider that whereami can be roughly implemented as:


function whereami {
  echo Stopped in $func at line $lineno in file $(basename
$file)
  echo "$lineno\t$line"
}

Event-Specific Variables

The following variables are only valid within the body of a when.

$handlerid

During the execution of the body, $handlerid is the id of the when command to which the body belongs. These commands are equivalent:


when X -temp { do_stuff; }
when X  { do_stuff; delete $handlerid; }

$booting

Is set to true if the event occurs during the boot process. Whenever a new program is debugged, it is first run without the user's knowledge so that the list and location of shared libraries can be ascertained. The process is then killed. This sequence is termed booting.

While booting is occurring, all events are still available. Use this variable to distinguish the sync and the syncrtld events occurring during a debug and the ones occurring during a normal run.

Variables Valid for the Given Event

For Event sig

$sig

Signal number that caused the event 

$sigstr

Name of $sig 

$sigcode

Subcode of $sig if applicable 

$sigcodestr

Name of $sigcode 

$sigsender

Process id of sender of the signal, if appropriate 

For Event exit

$exitcode

Value of the argument passed to _exit(2) or exit(3) or the return value of main 

For Events dlopen and dlclose

$dlobj

Pathname of the load object dlopened or dlclosed 

For Events sysin and sysout

$syscode

System call number 

$sysname

System call name 

For Event proc_gone

$reason

One of signal, exit, kill, or detach 

Examples

Use these examples for setting event handlers.

Set Watchpoint for Store to Array Member

To set a watchpoint on array[99]:


(dbx) stop access w &array[99]
(2) stop access w &array[99], 4
(dbx) run
Running: watch.x2
watchpoint array[99] (0x2ca88[4]) at line 22 in file "watch.c"	
   22	array[i] = i;

Simple Trace

To implement a simple trace:


(dbx) when step { echo at line $lineno; }

Enable Handler While Within the Given Function (in func)

For example:


trace
step -in foo

is equivalent to:


    # create handler in disabled state
    when step -disable { echo Stepped to $line; } 
    t=$newhandlerid    # remember handler id
    when in foo {
     # when entered foo enable the trace
     handler -enable "$t"
     # arrange so that upon returning from foo,
     # the trace is disabled.
     when returns { handler -disable "$t"; };
    }

Determine the Number of Lines Executed in a Program

To see how many lines were executed in a small program:


(dbx) stop step -count infinity	
 # step and stop when count=inf
(2) stop step -count 0/infinity
(dbx) run
...
(dbx) status
(2) stop step -count 133/infinity

The program never stops--the program terminates. 133 is the number of lines executed. This process is very slow though. This technique is more useful with breakpoints on functions that are called many times.

Determine the Number of Instructions Executed by a Source Line

To count how many instructions a line of code executes:


(dbx) ...                        # get
to the line in question
(dbx) stop step -instr -count infinity
(dbx) step ...
(dbx)status
(3) stop step -count 48/infinity # 48
instructions were executed

If the line you are stepping over makes a function call, you end up counting those as well. You can use the next event instead of step to count instructions, excluding called functions.

Enable Breakpoint after Event Occurs

Enable a breakpoint only after another event has fired. Suppose things go bad in function hash, but only after the 1300'th symbol lookup:


(dbx) when in lookup -count 1300 {
	stop in hash
	hash_bpt=$newhandlerid
	when proc_gone -temp { delete $hash_bpt; }
}


Note -

$newhandlerid is referring to the just executed stop in command.


Reset Application Files for replay

If your application processes files that need to be reset during a replay, you can write a handler to do that for you each time you run the program:


(dbx) when sync { sh regen ./database; }
(dbx) run < ./database...	# during which database gets clobbered
(dbx) save
...              # implies a RUN, which implies the SYNC event which
(dbx) restore	   # causes regen to run

Check Program Status

To see quickly where the program is while it's running:


(dbx) ignore sigint
(dbx) when sig sigint { where; cancel; }

Then type ^C to see a stack trace of the program without stopping it.

This is basically what the collector hand sample mode does (and more of course). Use SIGQUIT (^\) to interrupt the program because ^C is now used up.

Catch Floating Point Exceptions

To catch only specific floating-point exceptions, for example, IEEE underflow:


(dbx) ignore FPE 
             # turn off default handler 
(dbx) help signals | grep FPE
  # can't remember the subcode name
...
(dbx) stop sig fpe FPE_FLTUND
... 

Command Reference

when

To execute command(s) when the specified event occurs:


when event-specification
 { command(s); 
}

stop

To stop execution at a given event:


stop event-specification

To stop execution now and update displays. Whereas normally the process is continued after the body has executed, the stop command prevents that. This form is only valid within the body of a when:


stop -update

Same as above, but does not update displays:


stop -noupdate

step

The step command is equivalent to:


when step -temp { stop; }; cont

The step command can take a sig argument. A step by itself cancels the current signal just like cont does. To forward the signal, you must explicitly give the signal. You can use the variable $sig to step the program forwarding it the current signal:


step -sig $sig

cancel

Only valid within the body of when. The cancel command cancels any signal that might have been delivered, and lets the process continue. For example:


when sig SIGINT { echo signal info;  cancel; }

status

The status command lists handlers, those created by trace, when, and stop. status lists the given handler. If the handler is disabled, its hid is printed inside square brackets [ ] instead of parentheses ( ).


status [-s] -h hid

The output of status can be redirected to a file. Nominally, the format is unchanged during redirection. You can use the -s flag to produce output that allows a handler to be reinstated using the source command. If the -h option is used, hidden handlers are also listed.

The original technique of redirecting handlers using status and sourcing the file was a way to compensate for the lack of handler enabling and disabling functionality.

delete

The delete command deletes breakpoints and other handlers.


delete [-h] all -all -temp hid
 [hid ...
]

delete hid deletes the specified handler.

delete all, delete 0 (zero), or delete -all deletes all handlers including temporary handlers.

delete -temp deletes only all temporary handlers.

-h hid is required if hid is a hidden handler. -h all deletes all hidden handlers as well.

clear

clear with no arguments deletes all handlers based on breakpoints at the location where the process stopped. clear line deletes all handlers based on breakpoints on the given line.

clear line

handler

A handler is created for each event that needs to be managed in a debugging session. The commands trace, stop, and when create handlers. Each of these commands returns a number known as the handler ID (hid). The handler, status, and delete commands manipulate or provide information about handlers in a generic fashion.


handler [ -disable | -enable ] all hid
 [...]

handler -disable hid disables the specified event.

handler -disable all disables all handlers.

handler -enable hid enables the specified handler.

handler -enable all enables all handlers.


handler [ -count hid  [new-count-limit
] | -reset  hid ]

handler -count hid returns the count of the event in the form current-count/limit (same as printed by status). limit might be the keyword infinity. Use the ksh modifiers ${#} and ${##} to split the printed value.

handler -count hid new-count-limit assigns a new count limit to the given handler.

handler -reset hid resets the count of the handler to 0 (zero).