Debugging Aids

A debugging library and a debugging mdb(1) module are provided with the Oracle Solaris runtime linker. The debugging library enables you to trace the runtime linking process in more detail. The mdb(1) module enables interactive process debugging.

Debugging Facility

The runtime linker provides a debugging facility that allows you to trace the runtime linking of applications and their dependencies in detail. The type of information that is displayed by using this facility is expected to remain constant. However, the exact format of the information might change slightly from release to release.

Some of the debugging output might be unfamiliar to users who do not have an intimate knowledge of the runtime linker. However, many aspects might be of general interest to you.

Debugging is enabled by using the environment variable LD_DEBUG. All debugging output is prefixed with the process identifier. This environment variable must be augmented with one or more tokens to indicate the type of debugging that is required.

The tokens that are available with LD_DEBUG can be displayed by using LD_DEBUG=help.

$ LD_DEBUG=help prog

prog can be any dynamic executable. The process is terminated following the display of the help information, before control transfers to prog. The choice of executable is unimportant.

By default, all debug output is sent to stderr, the standard error output file. Debug output can be directed to a file instead, using the output token. For example, the help text can be captured in a file named rtld-debug.txt.

$ LD_DEBUG=help,output=rtld-debug.txt prog

Alternatively, debug output can be redirected by setting the environment variable LD_DEBUG_OUTPUT. When LD_DEBUG_OUTPUT is used, the process identifier is added as a suffix to the output filename.

If LD_DEBUG_OUTPUT and the output token are both specified, LD_DEBUG_OUTPUT takes precedence. If LD_DEBUG_OUTPUT and the output token are both specified, LD_DEBUG_OUTPUT takes precedence. Use of the output token with programs that call fork(2) result in each process writing debug output to the same file. The debug output will become jumbled and incomplete. LD_DEBUG_OUTPUT should be used in such cases to direct debug output for each process to a unique file.

The debugging of secure applications is not allowed.

One of the most useful debugging options is to display the symbol bindings that occur at runtime. The following example uses a very trivial dynamic executable that has a dependency on two local shared objects.

$ cat bar.c
int bar = 10;
$ cc -o bar.so.1 -K pic -G bar.c
 
$ cat foo.c
int foo(int data)
{
        return (data);
}
$ cc -o foo.so.1 -K pic -G foo.c
 
$ cat main.c
extern  int     foo();
extern  int     bar;
 
int main()
{
        return (foo(bar));
}
$ cc -o prog main.c -R/tmp:. foo.so.1 bar.so.1

The runtime symbol bindings can be displayed by setting LD_DEBUG=bindings.

$ LD_DEBUG=bindings prog
11753: .......
11753: binding file=prog to file=./bar.so.1: symbol bar
11753: .......
11753: transferring control: prog
11753: .......
11753: binding file=prog to file=./foo.so.1: symbol foo
11753: .......

The symbol bar, which is required by an immediate relocation, is bound before the application gains control. Whereas the symbol foo, which is required by a lazy relocation, is bound after the application gains control on the first call to the function. This relocation demonstrates the default mode of lazy binding. If the environment variable LD_BIND_NOW is set, all symbol bindings occur before the application gains control.

By setting LD_DEBUG=bindings,detail, additional information regarding the real and relative addresses of the actual binding locations is provided.

You can use LD_DEBUG to display the various search paths used. For example, the search path mechanism used to locate any dependencies can be displayed by setting LD_DEBUG=libs.

$ LD_DEBUG=libs prog
11775:
11775: find object=foo.so.1; searching
11775:  search path=/tmp:.  (RUNPATH/RPATH from file prog)
11775:  trying path=/tmp/foo.so.1
11775:  trying path=./foo.so.1
11775:
11775: find object=bar.so.1; searching
11775:  search path=/tmp:.  (RUNPATH/RPATH from file prog)
11775:  trying path=/tmp/bar.so.1
11775:  trying path=./bar.so.1
11775: .......

The runpath recorded in the application prog affects the search for the two dependencies foo.so.1 and bar.so.1.

In a similar manner, the search paths of each symbol lookup can be displayed by setting LD_DEBUG=symbols. A combination of symbols and bindings produces a complete picture of the symbol relocation process.

$ LD_DEBUG=bindings,symbols prog
11782: .......
11782: symbol=bar;  lookup in file=./foo.so.1  [ ELF ]
11782: symbol=bar;  lookup in file=./bar.so.1  [ ELF ]
11782: binding file=prog to file=./bar.so.1: symbol bar
11782: .......
11782: transferring control: prog
11782: .......
11782: symbol=foo;  lookup in file=prog  [ ELF ]
11782: symbol=foo;  lookup in file=./foo.so.1  [ ELF ]
11782: binding file=prog to file=./foo.so.1: symbol foo
11782: .......

In the previous example, the symbol bar is not searched for in the application prog. This omission of a data reference lookup is due to an optimization used when processing copy relocations. See Copy Relocations for more details of this relocation type.

Debugger Module

The debugger module provides a set of dcmds and walkers that can be loaded under mdb(1). This module can be used to inspect various internal data structures of the runtime linker. Much of the debugging information requires familiarity with the internals of the runtime linker. These internals can change from release to release. However, some elements of these data structures reveal the basic components of a dynamically linked process and can aid general debugging.

The following examples show some simple scenarios of using mdb(1) with the debugger module.

$ cat main.c
#include  <dlfnc.h>

int main()
{
        void *handle;
        void (*fptr)();

        if ((handle = dlopen("foo.so.1", RTLD_LAZY)) == NULL)
                return (1);

        if ((fptr = (void (*)())dlsym(handle, "foo")) == NULL)
                return (1);

        (*fptr)();
        return (0);
}
$ cc -o main main.c -R.

If mdb(1) has not automatically loaded the debugger module, ld.so, explicitly do so. The facilities of the debugger module can then be inspected.

$ mdb main
> ::load ld.so
> ::dmods -l ld.so

ld.so
-----------------------------------------------------------------
  dcmd Bind                 - Display a Binding descriptor
  dcmd Callers              - Display Rt_map CALLERS binding descriptors
  dcmd Depends              - Display Rt_map DEPENDS binding descriptors
  dcmd ElfDyn               - Display Elf_Dyn entry
  dcmd ElfEhdr              - Display Elf_Ehdr entry
  dcmd ElfPhdr              - Display Elf_Phdr entry
  dcmd Groups               - Display Rt_map GROUPS group handles
  dcmd GrpDesc              - Display a Group Descriptor
  dcmd GrpHdl               - Display a Group Handle
  dcmd Handles              - Display Rt_map HANDLES group descriptors
  ....
> ::bp main
> :r

Each dynamic object within a process is expressed as a link-map, Rt_map, which is maintained on a link-map list. All link-maps for the process can be displayed with Rt_maps.

> ::Rt_maps
Link-map lists (dynlm_list): 0xffbfe0d0
----------------------------------------------
  Lm_list: 0xff3f6f60  (LM_ID_BASE)
  ----------------------------------------------
    lmco        rtmap       ADDR()     NAME()
    ----------------------------------------------
    [0xc]       0xff3f0fdc 0x00010000 main
    [0xc]       0xff3f1394 0xff280000 /lib/libc.so.1
  ----------------------------------------------
  Lm_list: 0xff3f6f88  (LM_ID_LDSO)
  ----------------------------------------------
    [0xc]       0xff3f0c78 0xff3b0000 /lib/ld.so.1

An individual link-map can be displayed with Rt_map.

> 0xff3f9040::Rt_map
Rt_map located at: 0xff3f9040
     NAME: main
 PATHNAME: /export/home/user/main
     ADDR: 0x00010000         DYN: 0x000207bc
     NEXT: 0xff3f9460        PREV: 0x00000000
      FCT: 0xff3f6f18    TLSMODID:          0
     INIT: 0x00010710        FINI: 0x0001071c
   GROUPS: 0x00000000     HANDLES: 0x00000000
  DEPENDS: 0xff3f96e8     CALLERS: 0x00000000
    .....

The object's .dynamic section can be displayed with the ElfDyn dcmd. The following example shows the first 4 entries.

> 0x000207bc,4::ElfDyn
Elf_Dyn located at: 0x207bc
    0x207bc  NEEDED       0x0000010f
Elf_Dyn located at: 0x207c4
    0x207c4  NEEDED       0x00000124
Elf_Dyn located at: 0x207cc
    0x207cc  INIT         0x00010710
Elf_Dyn located at: 0x207d4
    0x207d4  FINI         0x0001071c

mdb(1) is also very useful for setting deferred break points. In this example, a break point on the function foo() might be useful. However, until the dlopen(3C) of foo.so.1 occurs, this symbol isn't known to the debugger. A deferred break point instructs the debugger to set a real breakpoint when the dynamic object is loaded.

> ::bp foo.so.1`foo
> :c
> mdb: You've got symbols!
> mdb: stop at foo.so.1`foo
mdb: target stopped at:
foo.so.1`foo:   save      %sp, -0x68, %sp

At this point, new objects have been loaded.

> *ld.so`lml_main::Rt_maps
lmco    rtmap       ADDR()     NAME()
----------------------------------------------
[0xc]   0xff3f0fdc 0x00010000 main
[0xc]   0xff3f1394 0xff280000 /lib/libc.so.1
[0xc]   0xff3f9ca4 0xff380000 ./foo.so.1
[0xc]   0xff37006c 0xff260000 ./bar.so.1

The link-map for foo.so.1 shows the handle returned by dlopen(3C). You can expand this structure using Handles.

> 0xff3f9ca4::Handles -v
HANDLES for ./foo.so.1
----------------------------------------------
  HANDLE: 0xff3f9f60 Alist[used 1: total 1]
    ----------------------------------------------
    Group Handle located at: 0xff3f9f28
    ----------------------------------------------
        owner:               ./foo.so.1
        flags: 0x00000000    [ 0 ]
       refcnt:          1    depends: 0xff3f9fa0 Alist[used 2: total 4]
        ----------------------------------------------
        Group Descriptor located at: 0xff3f9fac
           depend: 0xff3f9ca4    ./foo.so.1
            flags: 0x00000003    [ AVAIL-TO-DLSYM,ADD-DEPENDENCIES ]
        ----------------------------------------------
        Group Descriptor located at: 0xff3f9fd8
           depend: 0xff37006c    ./bar.so.1
            flags: 0x00000003    [ AVAIL-TO-DLSYM,ADD-DEPENDENCIES ]

The dependencies of a handle are a list of link-maps that represent the objects of the handle that can satisfy a dlsym(3C) request. In this case, the dependencies are foo.so.1 and bar.so.1.