Debugger

C H A P T E R 4

Debugger

This chapter describes the Netra DPS native debugger and GNU debugger (GDB). Topics include:

Debugger Introduction

Native Debugger

GNU Project Debugger (GDB)

Debugger Introduction

The Netra DPS native debugger is the default debugger and is useful for debugging during development. This debugger also identifies system hangs or crashes in the field deployment. To access the Netra DPS native debugger, press Ctrl-C.

To use the GNU Debugger (GDB), the user must have their own source code and the binary. It is necessary to turn on the flag for this application, for example,
USR_CFLAGS = -DTEJA_DEBUGGER_MODE=TEJA_DEBUGGER_GDB_MODE

See GNU Project Debugger (GDB) for detailed setup and example information.

Native Debugger

The native debugger runs on the target and enables users to do the following:

Set, clear, and display breakpoints

Set and display memory

Display registers

Display stack trace

Manage thread focus

Step to the next assembly instruction

The debugger is not symbolic. Symbol resolution is performed separately using a host-based tool called dbghelper.pl. See Resolving Symbols.

The native debugger is denoted by dbg. See Native Debugger Commands.

Debugging Configuration Code

As seen in tejacc Compiler Configuration, tejacc gets information about hardware architecture, software architecture, and mapping by executing the configuration code compiled into dynamic libraries.

The code is written in C and might contain errors causing tejacc to crash. Upon crashing, you are presented with a Java trademark Hotspot exception, as tejacc is internally implemented in Java. The information reported in the exception requires knowledgeable interpretation.

An alternative version of tejacc.sh, called tejacc_dbg.sh, is provided to assist debugging configuration code. This program runs tejacc inside the default host debugger (dbx for Solaris hosts), stopping the execution immediately after the configuration libraries have been loaded. The user can then continue execution to reach the instruction that causes the problem and verify its location. Alternatively, you can set breakpoints on the configuration functions, step through code, or use any other functionality provided by the host debugger.

To use tejacc_dbg.sh, replace the invocation of tejacc.sh in the makefile with tejacc_dbg.sh.

Entering the Debugger

The application program calls the native debugger when any of the following conditions occur:

At start time - If the application was compiled without the -O option, the application calls the debugger at start time. Applications compiled with the -O option start normally.

At a breakpoint - If the application was compiled without the -O option and while running encounters a breakpoint, the application calls the debugger. Applications compiled with the -O option cannot set breakpoints.

In a crash - If the application crashes, it calls the debugger. The debugger is called regardless of whether the application was compiled with or without the -O option.

Typing Ctrl-C - If the application calls the teja_debugger_check_ctrl_c() function and you type the Ctrl-C key sequence, the debugger is also called. The debugger is called regardless of whether the application was compiled with or without the -O option.

Note - A call to the debugger stops all threads.

Note - The teja_check_ctrl_c() function must be executed periodically by at least one of the threads in order for the Ctrl-C function to work. If the thread calling the teja_check_ctrl_c() function crashes or goes into a deadlock, the Ctrl-C key sequence stops.

Native Debugger Commands

Displaying Help

`help` command or
`h` command

Description

Displays help for a command. If the command variable is absent, a general help page is displayed.

Example

dbg>help 
        break <address> - set breakpoint 
                          not available for all instructions (see docs) 
        b <address>     - set breakpoint 
                          not available for all instructions (see docs) 
        bt n            - display stack trace 
        delete breakpoint <bpid>       - clear breakpoint 
        d breakpoint <bpid>    - clear breakpoint 
        info            - display info help 
        i               - display info help 
        help [cmd]      - display help 
        h [cmd]         - display help 
        ? [cmd]         - display help 
        cont            - resume execution 
        c               - resume execution 
        step            - step to next Assembly instruction 
                          not available for all instructions (see docs) 
        s               - step to next Assembly instruction 
                          not available for all instructions (see docs) 
        x/nfu <address> - display memory: 
                          n (count) 
                          u = {b|h|w|g} (unit) 
                          f = {x|d|u|o|t|a|f|s|i} (format) 
        thread <thdid> - switch thread focus 
        w/u addr value - set memory 
                          u = {b|h|w|g} (unit)

Managing Breakpoints

Setting breakpoints is only supported in nonoptimized mode and means that the application must be built without the -O option to tejacc.

`break` address Command or
`b` address Command

Description

Sets a breakpoint, where address is the hexadecimal address at which to break. The breakpoint is set only in regions of code that are characterized by sequential execution and not affected by control flow changes. The easiest way to set a proper breakpoint is to use the dbghelper script. See Resolving Symbols.

Example

dbg>break 50b188 
Breakpoint set at 0x50b188

`info break` Command or
`i break` Command

Description

Displays a list of active breakpoints.

Example

dbg>info break 
breakpoint [1] set at 0x50b188

In this example, only one breakpoint exists. The breakpoint has an ID of 1. When more than one breakpoint is set, each breakpoint receives a consecutive ID.

`delete breakpoint` ID Command or
`d breakpoint` ID Command

Description

Deletes a breakpoint, where ID is the ID of the breakpoint.

Example:

dbg>delete breakpoint [1]

Managing Program Execution

`cont` Command or
`c` Command

Description

Continues execution of the application.

Example

dbg>cont

`step` Command or
`s` Command

Description

Steps to the next assembly instruction within the application.

Example

dbg>step

Note - Only use the step command in regions of code that are characterized by sequential execution and not affected by control flow changes.

Displaying and Setting Memory

`x/`nfu address Command

Description

Displays memory contents where:

n - Number of memory units to display.

f - The display format. The only supported value is x, for hexadecimal format.

u - The size of the unit. Supported values are the following:

b - byte

h - 2-byte half-word

w - 4-byte word

g - 8-byte long word

address - The starting address in hexadecimal.

Example:

dbg>x/8xw 10000000 
count = 8; format = HEX; unitsize = 4 
[10000000] : 00000100 000000cd 00000001 00000114 00000100 000000ce 00000001 00518a44

`w/`u address value Command

Description

Sets memory where:

u - The size of the unit. Supported values are:

b - byte

h - 2-byte half-word

w - 4-byte word

g - 8-byte long word

address - The starting address in hexadecimal.

value - The value to write in hexadecimal.

Example

dbg>w/w 10000000 00518a44

Managing Threads

`info threads` Command or
`i threads` Command

Description

Displays a list of the active threads. The thread that has the focus is shown with an F symbol. Similarly, if a thread has crashed, it is shown with an F symbol.

Example

dbg>info threads 
  : generatorthread: Teja thread id 0, strand id 0 
F : classifierthread: Teja thread id 1, strand id 1

`thread` ID Command

Description

Changes the thread focus to the thread with the Teja thread ID of ID.

Example

dbg>thread 0 
Thread focus changed to 0

In the previous example, the focus (F) was on classifierthread, with Teja ID of 1. In this example, the focus has been moved to generatorthread.

Displaying Registers

`info reg` Command or `i reg` Command

Description

Displays the register contents for the thread in focus.

Example

dbg>info reg 
Registers of strand 0: 
G registers: 
g[0] :  0000000000000000 0000000000000000 0000000000500000 0000000000000000  
g[4] :  0000000000000000 0000000000615fa0 0000000000000000 0000000000000000 
I registers:  
i[0] :  000000000000006e ffffffffef1fe8d4 0000000000520c30 0000000010e01bc8  
i[4] :  0000000000000000 0000000000000000 0000000010e00d91 000000000051458c 
O registers:  
o[0] :  000000000000006e 0000000000520c30 0000000010e01bc8 0000000000000000  
o[4] :  0000000000600000 0000000000000061 0000000010e00cd1 0000000000514a18 
L registers:  
l[0] :  000000000000006e 0000000010e0172c 000000000051e8f0 ffffffffef1fe8d4  
l[4] :  0000000000520c30 0000000000000000 0000000000000000 0000000000000000 
gl      :  0000000000000001  
tl      :  0000000000000001  
tt      :  000000000000007c  
tpc     :  0000000000508c88  
tnpc    :  0000000000508c8c  
tstate  : 0000009914001600  
pstate  : 0000000000000014  
tick    :  000001884f873558  
tba     :  0000000000500000  
asi     :  0000000000000014

Displaying Stack Trace

`bt` frame_count Command

Description

Displays the stack trace for the thread in focus for frame_count number of frames.

Example

dbg>bt 4 
frame 1, sp 0x10e03580, call instruction at 0x50e888: 
l[0] :  0000000000000001 00000000111606a8 0000000011160600 0000000000000000  
l[4] :  00000000006170d8 0000000000001000 0000000000010000 0000000000000150  
i[0] :  0000000000000800 0000000010e036f0 0000000010e036e8 0000000010e036e4  
i[4] :  0000000000002000 0000000019ae8ec8 0000000010e02e31 000000000050e888 
frame 2, sp 0x10e03630, call instruction at 0x50fcc4:  
l[0] :  0000000000000001 00000000111606a8 0000000011160600 0000000000000000  
l[4] :  00000000006170d8 0000000000001000 0000000000010000 0000000000000150  
i[0] :  0000000000000800 0000000000000001 0000000019d8c148 0000000000000800  
i[4] :  0000000019d8c140 0000000019d8c000 0000000010e02f01 000000000050fcc4 
frame 3, sp 0x10e03700, call instruction at 0x50fbd8:  
l[0] :  0000000000000001 00000000111606a8 0000000011160600 0000000000000000  
l[4] :  00000000006170d8 0000000000001000 0000000000010000 0000000000000150  
i[0] :  00000000111000e0 0000000000000015 0000000000010000 0000000011160580  
i[4] :  0000000000002000 0000000019ae8ec8 0000000010e02fd1 000000000050fbd8 
frame 4, sp 0x10e037d0, call instruction at 0x50e104:  
l[0] :  ffffffffffffffd8 ffffffffffffffba 0000000000000003 0000000000000000  
l[4] :  00000000006170d8 0000000000617000 0000000000000617 0000000000000400  
i[0] :  00000000111000e0 000000000000792d 000000000000792d 0000000011100180  
i[4] :  000000000000792d 0000000000000000 0000000010e03081 000000000050e104

Resolving Symbols

The dbghelper.pl script is used to resolve symbols to set breakpoints in the correct places. The script is located in install_dir/tools/bin directory, where install_dir is the SUNWndps package installation directory. For example, /opt/SUNWndps/tools/bin/dbghelper.pl.

`-h` Option

Description

Displays help information.

`-f` function_name Option

Description

Prints a debugger command to set a breakpoint at the given function_name. This option does not work for static functions. To set a breakpoint inside of a static function, use the -l file_name:line_number option.

Example

$ dbghelper.pl -f classifier ./main 
b 50b17c

`-g` global_variable Option

Description

Prints a debugger command to display the contents of the given global_variable. The size of the memory displayed is fixed and does not consider the actual size of the global_variable. The user might need to increase the size of the memory.

Example

$ dbghelper.pl -g stats ./main 
x/1wx 13000640

`-l` file_name`:`line_number Option

Description

Prints a debugger command to set a breakpoint at the provided file_name:line_number. The file_name and line_number refer to your source code.

Example

$ dbghelper.pl -l src/classifier.c:57 ./main 
b 50b188

GNU Project Debugger (GDB)

GDB, the GNU Project debugger, enables the user to debug the program in C source code level. The following sections describe the reference Netra DPS application (gdb showcase application) that gives a showcase of the GDB support in Netra DPS over the Logical Domain Channel (LDC). In this release, only the IPFwd and GDB showcase applications have been prepared for gdb support. Other applications can easily be instrumented by following these examples.

Configuring LDoms for GDB Support

GDB requires Logical Domains 1.0.1 or later release. If not installed, download: http://www.sun.com/ldoms

To Configure LDoms Required to Run the Netra DPS Application with GDB Support

Refer to Chapter 9, Auto-Configuration, on how to create LDC channels. The GDB currently runs over LDC only, not over IPC. The GDB uses the vdpc service named ndps-cli and the corresponding client named solaris-cli. This service/client pair can be created either by using the ldm commands shown below or by using the respective commands in the auto configuration process (Chapter 9).

/opt/SUNWldm/bin/ldm add-vdpcs ndps-cli ndps-domain-name
/opt/SUNWldm/bin/ldm add-vdpcc solaris-cli ndps-cli solaris-domain-name

To Configure the Solaris Domain for GDB

Once the LDoms domains are configured and running, do the following steps to configure the gateway for GDB in the Solaris domain.

1. Ensure that the SUNWndpsd package is installed in the domain.

2. Load the driver as shown below:

rem_drv remldc
add_drv remldc

3. Run the following:

echo “remotegw 34980/tcp” >> /etc/services
svccfg import /var/svc/manifest/network/remotegw.xml
svcadm enable svc:/network/remotegw:remotegw

GDB Showcase Application

The Netra DPS package contains a simple test application to showcase the use of gdb. This application is used as an example below.

To Compile the GDB Showcase

From the SUNWndps package, compile the application under /src/apps/gdb. Type:

gmake clean
gmake CMT=N1(for UltraSPARC T1 based platforms)
gmake CMT=N2(for UltraSPARC T2 based platforms)
cd code/main
gmake

This action generates the binary file called main under src/apps/gdb/code/main. The required Solaris utility binaries are under src/apps/gdb/solaris-gw/ldc_so.

To Load the GDB Showcase Binary in the Netra DPS Domain

1. Verify that you have copied your GDB Netra DPS binary main into your install server under /tftpboot.

2. Type the following at the Netra DPS domain OpenBoot Prom ok prompt:

ok   boot /virtual-devices@100/channel-devices@200/network@0:,main

3. If compiled without the -o option, continue from initial break point.

To Run the GDB Command

Once your GDB showcase application is compiled, do the following under any host machine as long as you can access your code base src/apps/gdb/code/main:.

cd src/apps/gdb/code/main
/opt/SUNWndps/bin/gdb

main is the same binary code that you loaded into your Netra DPS domain.

The GDB debugger then displays the following:

GNU gdb 6.6
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type “show copying” to see the conditions.
There is absolutely no warranty for GDB.  Type “show warranty” for details.
This GDB was configured as “--host=sparc-sun-solaris2.10 --target=sparc64-elf”...
(gdb)

Acquire the host name or IP address for the Solaris domain by typing:

ifconfig -a

Assuming your IP address is 10.1.1.249

(gdb) target remote tcp:10.1.1.249:34980

This connects to your Netra DPS GDB application. The Netra DPS console then displays:

LDC Status = UP
calling set_debug_traps()....
Program started: initial breakpoint reached

This indicates that the GDB showcase application reached the initial breakpoint artificially created by the application. You can then use the following GDB commands to investigate your application.

GDB commands include the following:

target remote tcp:10.1.1.194:34980 - Connects to remote Netra DPS target

info thread - Display threads

thread # - Switch thread

info reg - Show the register files

info break - Show the breakpoint

b # - Set breakpoint

d # - Clear breakpoint

c - Continue

s - Step

x - Check memory location

p - Display variable

list - Display source code, for example, list debug_func

bt - Backtrace

detach - To end the remote communication, allow the Solaris gateway program to exit. See Step-By-Step Procedure to Run Netra DPS Application with GDB Support for more details.

For additional GDB information and instructions, see GDB: The GNU Project Debugger at http://sourceware.org/gdb/.

Step-By-Step Procedure to Run Netra DPS Application with GDB Support

As an example, to run the IPFwd application with GDB support, do the following.

1. Go to src/apps/ipfwd and compile with gdb as one of the arguments in the command line, as shown below:

./build cmt2 10g_niu ldoms gdb

2. Load the src/apps/ipfwd/code/ipfwd/ipfwd binary into your Netra DPS domain.

3. Configure the Solaris gateway in the Solaris domain (for example, 10.1.1.194).

4. In the Solaris domain, calculate the basepaddr and run the tnsmctl -P -v command, if the binary booted in the Netra DPS domain uses NIU.

See How Do I Calculate the Base PA Address for NIU/LDoms to Use with the tnsmctl Command?.

5. Run the commands shown in To Configure the Solaris Domain for GDB.

6. Go to /opt/SUNWndps/bin and run the gdb binary from the Solaris domain.

./gdb ipfwd

ipfwd is the same binary code that you loaded into your Netra DPS domain.

The following is the example output from the gdb showcase application which shows the usage of all of the GDB commands.

GNU gdb 6.6
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type “show copying” to see the conditions.
There is absolutely no warranty for GDB.  Type “show warranty” for details.
This GDB was configured as “--host=sparc-sun-solaris2.10 --target=sparc64-linux-elf”...
(gdb)  target remote tcp:10.1.1.249:34980
Remote debugging using tcp:10.1.1.249:34980
0x0053f34c in teja_breakpoint ()
Current language:  auto; currently minimal
(gdb) info thread
* 2 Thread 2 (stat_thd)  0x0053f34c in teja_breakpoint ()
 1 Thread 1 (main_thd00)  0x0050c5d8 in main_thread ()
   at src/apps/gdb/code/main/_src_app_remcon_impl.c:97
(gdb) thread 1
[Switching to thread 1 (Thread 1)]#0  0x0050c5d8 in main_thread ()
   at src/apps/gdb/code/main/_src_app_remcon_impl.c:97
97         while ( count  < (2))
(gdb) list debug_func
58         int i ;
59         int j ;
60      //      char *tmp; File: src/app/remcon_impl.c Line: 40
61         char * tmp ;
62      //      tmp = “0xdeadbeef”; File: src/app/remcon_impl.c Line: 41
63            tmp  = “0xdeadbeef”;
64      //      gdbptr = “0xbaddcafe”; File: src/app/remcon_impl.c Line: 42
65            gdbptr  = “0xbaddcafe”;
66      //      i = first_time++; File: src/app/remcon_impl.c Line: 43
67            i  =  first_time ++;
(gdb) b 67
Breakpoint 1 at 0x50c510: file src/apps/gdb/code/main/_src_app_remcon_impl.c, line 67.
(gdb) p first_time
$1 = 18255275
(gdb) c
Continuing.
Can’t send signals to this remote system. SIGSTOP not sent.

Program received signal SIGSTOP, Stopped (signal).
debug_func ()
   at src/apps/gdb/code/main/_src_app_remcon_impl.c:67
67            i  =  first_time ++;
(gdb) c
Continuing.
Can’t send signals to this remote system.  SIGSTOP not sent.
 
Program received signal SIGSTOP, Stopped (signal).
0x0050c514 in debug_func ()
   at src/apps/gdb/code/main/_src_app_remcon_impl.c:67
67            i  =  first_time ++;
(gdb) c
Continuing.
Can’t send signals to this remote system.  SIGSTOP not sent.
Program received signal SIGSTOP, Stopped (signal).
debug_func ()
   at src/apps/gdb/code/main/_src_app_remcon_impl.c:67
67            i  =  first_time ++;
(gdb) p first_time
$4 = 18255276
(gdb) info thread
 2 Thread 2 (stat_thd)  0x00508f04 in hv_ldc_rx_get_state ()
* 1 Thread 1 (main_thd00)  debug_func ()
   at src/apps/gdb/code/main/_src_app_remcon_impl.c:67
(gdb) s
Can’t send signals to this remote system.  SIGSTOP not sent.
 
Program received signal SIGSTOP, Stopped (signal).
0x0050c534 in debug_func ()
   at src/apps/gdb/code/main/_src_app_remcon_impl.c:67
67            i  =  first_time ++;
(gdb) s
Can’t send signals to this remote system.  SIGSTOP not sent.
 
Program received signal SIGSTOP, Stopped (signal).
debug_func ()
   at src/apps/gdb/code/main/_src_app_remcon_impl.c:69
69            j  =  i  + ( first_time );
(gdb) detach
Ending remote debugging.

7. Once debugging is completed, type detach.

(gdb) detach

Note - Once completed, always type detach in gdb. Otherwise, the remotegw process is left with an outdated state in the Solaris domain. If this happens, stop the remotegw process using the svcadm command (svcadm disable remotegw) before you start again.

8. (Optional) Reload your binary in the Netra DPS domain.

If your source changes, you need to quit the gdb and re-enter gdb.

Debugger Introduction

Native Debugger

Debugging Configuration Code

Entering the Debugger

Native Debugger Commands

Displaying Help

help command or h command

Description

Example

Managing Breakpoints

break address Command or b address Command

Description

info break Command or i break Command

Description

Example

delete breakpoint ID Command or d breakpoint ID Command

Description

Managing Program Execution

cont Command or c Command

Description

step Command or s Command

Description

Example

Displaying and Setting Memory

x/nfu address Command

Description

w/u address value Command

Description

Example

Managing Threads

info threads Command or i threads Command

Description

thread ID Command

Description

Example

Displaying Registers

info reg Command or i reg Command

Description

Example

Displaying Stack Trace

bt frame_count Command

Description

Example

Resolving Symbols

-h Option

Description

-f function_name Option

Description

-g global_variable Option

Description

Example

-l file_name:line_number Option

Description

Example

GNU Project Debugger (GDB)

Configuring LDoms for GDB Support

GDB Showcase Application

`help` command or
`h` command

`break` address Command or
`b` address Command

`info break` Command or
`i break` Command

`delete breakpoint` ID Command or
`d breakpoint` ID Command

`cont` Command or
`c` Command

`step` Command or
`s` Command

`x/`nfu address Command

`w/`u address value Command

`info threads` Command or
`i threads` Command

`thread` ID Command

`info reg` Command or `i reg` Command

`bt` frame_count Command

`-h` Option

`-f` function_name Option

`-g` global_variable Option

`-l` file_name`:`line_number Option