C H A P T E R  4

Debugger

This chapter describes the Sun Netra DPS native debugger and GNU debugger (GDB). Topics include:


Debugger Introduction

The Sun Netra DPS native debugger is the default debugger and is useful for debugging during development. This debugger also identifies system hangs or crashes in the field deployment. To access the Sun Netra DPS native debugger, press Ctrl-C.

To use the GNU Debugger (GDB), you must have their own source code and the binary. You must turn on the flag for this application, for example,
USR_CFLAGS = -DTEJA_DEBUGGER_MODE=TEJA_DEBUGGER_GDB_MODE

See GNU Project Debugger for detailed setup and example information.


Native Debugger

The native debugger runs on the target and enables you to do the following:

The debugger is not symbolic. Symbol resolution is performed separately using a host-based tool called dbghelper.pl. See Resolving Symbols Using Options.

The native debugger is denoted by dbg. See Native Debugger Commands.

Debugging Configuration Code

As seen in tejacc Compiler Configuration, tejacc gets information about hardware architecture, software architecture, and mapping by executing the configuration code compiled into dynamic libraries.

The code is written in C and might contain errors causing tejacc to crash. Upon crashing, you are presented with a Java Hotspot exception, as tejacc is internally implemented in Java software. The information reported in the exception requires knowledgeable interpretation.

An alternative version of tejacc.sh, called tejacc_dbg.sh, is provided to assist debugging configuration code. This program runs tejacc inside the default host debugger (dbx for Oracle Solaris hosts), stopping the execution immediately after the configuration libraries have been loaded. You can then continue execution to reach the instruction that causes the problem and verify its location. Alternatively, you can set breakpoints on the configuration functions, step through code, or use any other functionality provided by the host debugger.

To use tejacc_dbg.sh, replace the invocation of tejacc.sh in the makefile with tejacc_dbg.sh.

Entering the Debugger

The application program calls the native debugger when any of the following conditions occur:



Note - A call to the debugger stops all threads.




Note - The teja_check_ctrl_c() function must be executed periodically by at least one of the threads in order for the Ctrl-C function to work. If the thread calling the teja_check_ctrl_c() function crashes or goes into a deadlock, the Ctrl-C key sequence stops.


Native Debugger Commands

The following section contains descriptions of the native debugger commands.

Displaying Help

help or h

Displays help for a command. If the command variable is absent, a general help page is displayed.

Example:


dbg>help 
        break <address> - set breakpoint 
                          not available for all instructions (see docs) 
        b <address>     - set breakpoint 
                          not available for all instructions (see docs) 
        bt n            - display stack trace 
        delete breakpoint <bpid>       - clear breakpoint 
        d breakpoint <bpid>    - clear breakpoint 
        info            - display info help 
        i               - display info help 
        help [cmd]      - display help 
        h [cmd]         - display help 
        ? [cmd]         - display help 
        cont            - resume execution 
        c               - resume execution 
        step            - step to next Assembly instruction 
                          not available for all instructions (see docs) 
        s               - step to next Assembly instruction 
                          not available for all instructions (see docs) 
        x/nfu <address> - display memory: 
                          n (count) 
                          u = {b|h|w|g} (unit) 
                          f = {x|d|u|o|t|a|f|s|i} (format) 
        thread <thdid> - switch thread focus 
        w/u addr value - set memory 
                          u = {b|h|w|g} (unit)

Managing Breakpoints

Setting breakpoints is only supported in nonoptimized mode and means that the application must be built without the -O option to tejacc.

break address or b address

Sets a breakpoint, where address is the hexadecimal address at which to break. The breakpoint is set only in regions of code that are characterized by sequential execution and not affected by control flow changes. The easiest way to set a proper breakpoint is to use the dbghelper script. See Resolving Symbols Using Options.

Example:


dbg>break 50b188 
Breakpoint set at 0x50b188 

info break or i break

Displays a list of active breakpoints.

Example:


dbg>info break 
breakpoint [1] set at 0x50b188 

In this example, only one breakpoint exists. The breakpoint has an ID of 1. When more than one breakpoint is set, each breakpoint receives a consecutive ID.

delete breakpoint ID or d breakpoint ID

Deletes a breakpoint, where ID is the ID of the breakpoint.

Example:


dbg>delete breakpoint [1]

Managing Program Execution

cont or c

Continues execution of the application.

Example:


dbg>cont

step or s

Steps to the next assembly instruction within the application.

Example:


dbg>step



Note - Only use the step command in regions of code that are characterized by sequential execution and not affected by control flow changes.


Displaying and Setting Memory

x/nfu address

Displays memory contents where:

Example:


dbg>x/8xw 10000000 
count = 8; format = HEX; unitsize = 4 
[10000000] : 00000100 000000cd 00000001 00000114 00000100 000000ce 00000001 00518a44

w/u address value

Sets memory where:

Example:


dbg>w/w 10000000 00518a44

Managing Threads

info threads or i threads

Displays a list of the active threads. The thread that has the focus is shown with an F symbol. Similarly, if a thread has crashed, it is shown with an F symbol.

Example:


dbg>info threads 
  : generatorthread: Teja thread id 0, strand id 0 
F : classifierthread: Teja thread id 1, strand id 1 

thread ID

Changes the thread focus to the thread with the Teja thread ID of ID.

Example:


dbg>thread 0 
Thread focus changed to 0

In info threads or i threads example, the focus (F) was on classifierthread, with Teja ID of 1. In this example, the focus has been moved to generatorthread.

Displaying Registers

info reg or i reg

Displays the register contents for the thread in focus. Refer to the UltraSPARC T1 Supplement to the UltraSPARC Architecture and the UltraSPARC T2 Supplement to the UltraSPARC Architecture for detailed descriptions of these registers when using the native debugger on the UltraSPARC T1 and UltraSPARC T2 platform. In the following example, the tpc (program counter at trap point) is at 0x508c88. The tt (trap type) is 0x7c. The content of the code at address 0x508c88 can be located by an elf file dump utility, such as gobjdump (in SUNWbinutils package) or equivalent utility.


dbg>info reg 
Registers of strand 0: 
G registers: 
g[0] :  0000000000000000 0000000000000000 0000000000500000 0000000000000000  
g[4] :  0000000000000000 0000000000615fa0 0000000000000000 0000000000000000 
I registers:  
i[0] :  000000000000006e ffffffffef1fe8d4 0000000000520c30 0000000010e01bc8  
i[4] :  0000000000000000 0000000000000000 0000000010e00d91 000000000051458c 
O registers:  
o[0] :  000000000000006e 0000000000520c30 0000000010e01bc8 0000000000000000  
o[4] :  0000000000600000 0000000000000061 0000000010e00cd1 0000000000514a18 
L registers:  
l[0] :  000000000000006e 0000000010e0172c 000000000051e8f0 ffffffffef1fe8d4  
l[4] :  0000000000520c30 0000000000000000 0000000000000000 0000000000000000 
gl      :  0000000000000001  
tl      :  0000000000000001  
tt      :  000000000000007c  
tpc     :  0000000000508c88  
tnpc    :  0000000000508c8c  
tstate  : 0000009914001600  
pstate  : 0000000000000014  
tick    :  000001884f873558  
tba     :  0000000000500000  
asi     :  0000000000000014

Displaying Stack Trace

bt frame-count

Displays the stack trace for the thread in focus for frame_count number of frames.


dbg>bt 4 
frame 1, sp 0x10e03580, call instruction at 0x50e888: 
l[0] :  0000000000000001 00000000111606a8 0000000011160600 0000000000000000  
l[4] :  00000000006170d8 0000000000001000 0000000000010000 0000000000000150  
i[0] :  0000000000000800 0000000010e036f0 0000000010e036e8 0000000010e036e4  
i[4] :  0000000000002000 0000000019ae8ec8 0000000010e02e31 000000000050e888 
frame 2, sp 0x10e03630, call instruction at 0x50fcc4:  
l[0] :  0000000000000001 00000000111606a8 0000000011160600 0000000000000000  
l[4] :  00000000006170d8 0000000000001000 0000000000010000 0000000000000150  
i[0] :  0000000000000800 0000000000000001 0000000019d8c148 0000000000000800  
i[4] :  0000000019d8c140 0000000019d8c000 0000000010e02f01 000000000050fcc4 
frame 3, sp 0x10e03700, call instruction at 0x50fbd8:  
l[0] :  0000000000000001 00000000111606a8 0000000011160600 0000000000000000  
l[4] :  00000000006170d8 0000000000001000 0000000000010000 0000000000000150  
i[0] :  00000000111000e0 0000000000000015 0000000000010000 0000000011160580  
i[4] :  0000000000002000 0000000019ae8ec8 0000000010e02fd1 000000000050fbd8 
frame 4, sp 0x10e037d0, call instruction at 0x50e104:  
l[0] :  ffffffffffffffd8 ffffffffffffffba 0000000000000003 0000000000000000  
l[4] :  00000000006170d8 0000000000617000 0000000000000617 0000000000000400  
i[0] :  00000000111000e0 000000000000792d 000000000000792d 0000000011100180  
i[4] :  000000000000792d 0000000000000000 0000000010e03081 000000000050e104

Resolving Symbols Using Options

You can use the dbghelper.pl script to resolve symbols to set breakpoints in the correct places. The script is located in install-dir/tools/bin directory, where install-dir is the SUNWndps package installation directory (for example, /opt/SUNWndps/tools/bin/dbghelper.pl).

-h

Displays help information.

-f function-name

Prints a debugger command to set a breakpoint at the given function-name. This option does not work for static functions. To set a breakpoint inside of a static function, use the -l file-name:line-number option.


% dbghelper.pl -f classifier ./main 
b 50b17c 

-g global-variable

Prints a debugger command to display the contents of the given global-variable. The size of the memory displayed is fixed and does not consider the actual size of the global-variable. You might need to increase the size of the memory.


% dbghelper.pl -g stats ./main 
x/1wx 13000640 

-l file-name:line-number

Prints a debugger command to set a breakpoint at the provided file-name:line-number. The file-name and line-number refer to your source code.


% dbghelper.pl -l src/classifier.c:57 ./main 
b 50b188


GNU Project Debugger

GDB, the GNU Project debugger, enables you to debug the program in C source code level. The following sections describe the reference Sun Netra DPS application (gdb showcase application) that showcases the GDB support in Sun Netra DPS over the Logical Domain Channel (LDC). In this release, only the IPFwd and GDB showcase applications have been prepared for gdb support. Other applications can easily be instrumented by following these examples.

Configuring Oracle VM Server for SPARC Software for GDB Support

GDB requires the Oracle VM Server for SPARC software. If this product is not installed, download it at: http://www.sun.com/ldoms


procedure icon  To Configure the Oracle VM Server for SPARC Software Required to Run the Sun Netra DPS Application With GDB Support

The GDB currently runs over LDC only, not over IPC.

The GDB uses the vdpc service named ndps-cli and the corresponding client named solaris-cli. This service-client pair can be created either by using the ldm commands in this procedure or by using the respective commands in the auto configuration process.

single-step bullet  Execute the following commands:


# /opt/SUNWldm/bin/ldm add-vdpcs ndps-cli ndps-domain-name
# /opt/SUNWldm/bin/ldm add-vdpcc solaris-cli ndps-cli solaris-domain-name


procedure icon  To Configure the Oracle Solaris Domain for GDB

After the logical domains are configured and running, perform the following steps to configure the gateway for GDB in the Oracle Solaris domain.

1. Ensure that the SUNWndpsd package is installed in the domain.

2. Load the driver:


# rem_drv remldc
# add_drv remldc

3. Execute the following commands:


# echo “remotegw 34980/tcp” >> /etc/services
# svccfg import /var/svc/manifest/network/remotegw.xml
# svcadm enable svc:/network/remotegw:remotegw

GDB Showcase Application

The Sun Netra DPS package contains a simple test application to showcase the use of gdb. This application is used as shown in To Compile the GDB Showcase.


procedure icon  To Compile the GDB Showcase

single-step bullet  From the SUNWndps package, compile the application under /src/apps/gdb:


% gmake clean
% gmake CMT=N1(for UltraSPARC T1 based platforms)
% gmake CMT=N2(for UltraSPARC T2 based platforms)
% cd code/main
% gmake

This action generates the binary file called main under src/apps/gdb/code/main. The required Oracle Solaris utility binaries are under src/apps/gdb/solaris-gw/ldc_so.


procedure icon  To Load the GDB Showcase Binary in the Sun Netra DPS Domain

1. Verify that you have copied your GDB Sun Netra DPS binary main into your installation server under /tftpboot.

2. Execute the following at the Sun Netra DPS domain OpenBoot PROM ok prompt:


ok   boot /virtual-devices@100/channel-devices@200/network@0:,main

3. If the file was compiled without the -o option, continue from the initial break point.


procedure icon  To Run the GDB Command

1. Once your GDB showcase application is compiled, execute the following commands under any host machine, as long as you can access your code base src/apps/gdb/code/main:


% cd src/apps/gdb/code/main
% /opt/SUNWndps/bin/gdb main

Where main is the same binary code that you loaded into your Sun Netra DPS domain.

The GDB debugger then displays the following:


GNU gdb 6.6
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type “show copying” to see the conditions.
There is absolutely no warranty for GDB.  Type “show warranty” for details.
This GDB was configured as “--host=sparc-sun-solaris2.10 --target=sparc64-elf”...
(gdb)

2. Acquire the host name or IP address for the Oracle Solaris domain:


% ifconfig -a 

Assuming your IP address is: 10.1.1.249


(gdb) target remote tcp:10.1.1.249:34980

This connects to your Sun Netra DPS GDB application. The Sun Netra DPS console then displays:


LDC Status = UP
calling set_debug_traps()....
Program started: initial breakpoint reached

This indicates that the GDB showcase application reached the initial breakpoint artificially created by the application. You can then use the GDB commands in GDB Commands to investigate your application.

GDB Commands

GDB commands include the following:

For additional GDB information and instructions, see GDB: The GNU Project Debugger at http://sourceware.org/gdb/.


procedure icon  To Run Sun Netra DPS Application With GDB Support

As an example, to run the IPFwd application with GDB support, perform the following steps.

1. Go to src/apps/ipfwd and compile with gdb as one of the arguments in the command line.

For example:


% ./build cmt2 10g_niu ldoms gdb

2. Load the src/apps/ipfwd/code/ipfwd/ipfwd binary into your Sun Netra DPS domain.

3. Configure the Oracle Solaris gateway in the Oracle Solaris domain (for example, 10.1.1.194).

4. In the Oracle Solaris domain, calculate the basepaddr and run the tnsmctl -P -v command, if the binary booted in the Sun Netra DPS domain uses NIU.

See How Do I Calculate the Base PA Address for NIU or Logical Domains to Use with the tnsmctl Command?.

5. Run the commands shown in To Configure the Oracle Solaris Domain for GDB.

6. Go to /opt/SUNWndps/bin and run the gdb binary from the Oracle Solaris domain:


% ./gdb ipfwd

ipfwd is the same binary code that you loaded into your Sun Netra DPS domain.

The following example output from the gdb showcase application shows the usage of all of the GDB commands.

 


GNU gdb 6.6
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type “show copying” to see the conditions.
There is absolutely no warranty for GDB.  Type “show warranty” for details.
This GDB was configured as “--host=sparc-sun-solaris2.10 --target=sparc64-linux-elf”...
(gdb)  target remote tcp:10.1.1.249:34980
Remote debugging using tcp:10.1.1.249:34980
0x0053f34c in teja_breakpoint ()
Current language:  auto; currently minimal
(gdb) info thread
* 2 Thread 2 (stat_thd)  0x0053f34c in teja_breakpoint ()
 1 Thread 1 (main_thd00)  0x0050c5d8 in main_thread ()
   at src/apps/gdb/code/main/_src_app_remcon_impl.c:97
(gdb) thread 1
[Switching to thread 1 (Thread 1)]#0  0x0050c5d8 in main_thread ()
   at src/apps/gdb/code/main/_src_app_remcon_impl.c:97
97         while ( count  < (2))
(gdb) list debug_func
58         int i ;
59         int j ;
60      //      char *tmp; File: src/app/remcon_impl.c Line: 40
61         char * tmp ;
62      //      tmp = “0xdeadbeef”; File: src/app/remcon_impl.c Line: 41
63            tmp  = “0xdeadbeef”;
64      //      gdbptr = “0xbaddcafe”; File: src/app/remcon_impl.c Line: 42
65            gdbptr  = “0xbaddcafe”;
66      //      i = first_time++; File: src/app/remcon_impl.c Line: 43
67            i  =  first_time ++;
(gdb) b 67
Breakpoint 1 at 0x50c510: file src/apps/gdb/code/main/_src_app_remcon_impl.c, line 67.
(gdb) p first_time
$1 = 18255275
(gdb) c
Continuing.
Can’t send signals to this remote system. SIGSTOP not sent.

 


Program received signal SIGSTOP, Stopped (signal).
debug_func ()
   at src/apps/gdb/code/main/_src_app_remcon_impl.c:67
67            i  =  first_time ++;
(gdb) c
Continuing.
Can’t send signals to this remote system.  SIGSTOP not sent.
 
Program received signal SIGSTOP, Stopped (signal).
0x0050c514 in debug_func ()
   at src/apps/gdb/code/main/_src_app_remcon_impl.c:67
67            i  =  first_time ++;
(gdb) c
Continuing.
Can’t send signals to this remote system.  SIGSTOP not sent.
Program received signal SIGSTOP, Stopped (signal).
debug_func ()
   at src/apps/gdb/code/main/_src_app_remcon_impl.c:67
67            i  =  first_time ++;
(gdb) p first_time
$4 = 18255276
(gdb) info thread
 2 Thread 2 (stat_thd)  0x00508f04 in hv_ldc_rx_get_state ()
* 1 Thread 1 (main_thd00)  debug_func ()
   at src/apps/gdb/code/main/_src_app_remcon_impl.c:67
(gdb) s
Can’t send signals to this remote system.  SIGSTOP not sent.
 
Program received signal SIGSTOP, Stopped (signal).
0x0050c534 in debug_func ()
   at src/apps/gdb/code/main/_src_app_remcon_impl.c:67
67            i  =  first_time ++;
(gdb) s
Can’t send signals to this remote system.  SIGSTOP not sent.
 
Program received signal SIGSTOP, Stopped (signal).
debug_func ()
   at src/apps/gdb/code/main/_src_app_remcon_impl.c:69
69            j  =  i  + ( first_time );
(gdb) detach
Ending remote debugging.

7. After debugging is completed, type detach.


(gdb) detach



Note - After completed, always type detach in gdb. Otherwise, the remotegw process is left with an outdated state in the Oracle Solaris domain. If this happens, stop the remotegw process using the svcadm command (svcadm disable remotegw) before you start again.


8. (Optional) Reload your binary in the Sun Netra DPS domain.

If your source changes, you need to quit the gdb and re-enter gdb.