C H A P T E R  4

Debugger

This chapter describes the Netra DPS native debugger and GNU debugger (GDB). Topics include:


Debugger Introduction

The Netra DPS native debugger is the default debugger and is useful for debugging during development. This debugger also identifies system hangs or crashes in the field deployment. To access the Netra DPS native debugger, press Ctrl-C.

To use the GNU Debugger (GDB), you must have your own source code and the binary. You need to turn on the flag for this application, for example,
USR_CFLAGS = -DTEJA_DEBUGGER_MODE=TEJA_DEBUGGER_GDB_MODE

See GNU Project Debugger (GDB) Showcase Application for detailed setup and example information.


Native Debugger

The native debugger runs on the target and enables users to do the following:

The debugger is not symbolic. Symbol resolution is performed separately using a host-based tool called dbghelper.pl. See Resolving Symbols.

The native debugger is denoted by dbg. See Native Debugger Commands.

Debugging Configuration Code

As seen in tejacc Compiler Configuration, tejacc gets information about hardware architecture, software architecture, and mapping by executing the configuration code compiled into dynamic libraries.

The code is written in C and might contain errors causing tejacc to crash. Upon crashing, you are presented with a Javatrademark Hotspot exception, as tejacc is internally implemented in Java. The information reported in the exception requires knowledgeable interpretation.

An alternative version of tejacc.sh, called tejacc_dbg.sh, is provided to assist debugging configuration code. This program runs tejacc inside the default host debugger (dbx for Solaris hosts), stopping the execution immediately after the configuration libraries have been loaded. You can then continue execution to reach the instruction that causes the problem and verify its location. Alternatively, you can set breakpoints on the configuration functions, step through code, or use any other functionality provided by the host debugger.

To use tejacc_dbg.sh, replace the invocation of tejacc.sh in the makefile with tejacc_dbg.sh.

Entering the Debugger

The application program calls the native debugger when any of the following conditions occur:



Note - A call to the debugger stops all threads.




Note - The teja_check_ctrl_c() function must be executed periodically by at least one of the threads in order for the Ctrl-C function to work. If the thread calling the teja_check_ctrl_c() function crashes or goes into a deadlock, the Ctrl-C key sequence stops.


Native Debugger Commands

Displaying Help

help command or
h command

Description

Displays help for a command. If the command variable is absent, a general help page is displayed.

Example


dbg>help 
        break <address> - set breakpoint 
                          not available for all instructions (see docs) 
        b <address>     - set breakpoint 
                          not available for all instructions (see docs) 
        bt n            - display stack trace 
        delete breakpoint <bpid>       - clear breakpoint 
        d breakpoint <bpid>    - clear breakpoint 
        info            - display info help 
        i               - display info help 
        help [cmd]      - display help 
        h [cmd]         - display help 
        ? [cmd]         - display help 
        cont            - resume execution 
        c               - resume execution 
        step            - step to next Assembly instruction 
                          not available for all instructions (see docs) 
        s               - step to next Assembly instruction 
                          not available for all instructions (see docs) 
        x/nfu <address> - display memory: 
                          n (count) 
                          u = {b|h|w|g} (unit) 
                          f = {x|d|u|o|t|a|f|s|i} (format) 
        thread <thdid> - switch thread focus 
        w/u addr value - set memory 
                          u = {b|h|w|g} (unit)

Managing Breakpoints

Setting breakpoints is only supported in nonoptimized mode and means that the application must be built without the -O option to tejacc.

break address Command or
b address Command

Description

Sets a breakpoint, where address is the hexadecimal address at which to break. The breakpoint is set only in regions of code that are characterized by sequential execution and not affected by control flow changes. The easiest way to set a proper breakpoint is to use the dbghelper script. See Resolving Symbols.

Example


dbg>break 50b188 
Breakpoint set at 0x50b188 

info break Command or
i break Command

Description

Displays a list of active breakpoints.

Example


dbg>info break 
breakpoint [1] set at 0x50b188 

In this example, only one breakpoint exists. The breakpoint has an ID of 1. When more than one breakpoint is set, each breakpoint receives a consecutive ID.

delete breakpoint ID Command or
d breakpoint ID Command

Description

Deletes a breakpoint, where ID is the ID of the breakpoint.

Example:


dbg>delete breakpoint [1]

Managing Program Execution

cont Command or
c Command

Description

Continues execution of the application.

Example


dbg>cont

step Command or
s Command

Description

Steps to the next assembly instruction within the application.

Example


dbg>step



Note - You can only use the step command in regions of code that are characterized by sequential execution and not affected by control flow changes.


Displaying and Setting Memory

x/nfu address Command

Description

Displays memory contents where:

Example:


dbg>x/8xw 10000000 
count = 8; format = HEX; unitsize = 4 
[10000000] : 00000100 000000cd 00000001 00000114 00000100 000000ce 00000001 00518a44

w/u address value Command

Description

Sets memory where:

Example


dbg>w/w 10000000 00518a44

Managing Threads

info threads Command or
i threads Command

Description

Displays a list of the active threads. The thread that has the focus is shown with an F symbol. Similarly, if a thread has crashed, it is shown with an F symbol.

Example


dbg>info threads 
  : generatorthread: Teja thread id 0, strand id 0 
F : classifierthread: Teja thread id 1, strand id 1 

thread ID Command

Description

Changes the thread focus to the thread with the Teja thread ID of ID.

Example


dbg>thread 0 
Thread focus changed to 0

In the previous example, the focus (F) was on classifierthread, with Teja ID of 1. In this example, the focus has been moved to generatorthread.

Displaying Registers

info reg Command or i reg Command

Description

Displays the register contents for the thread in focus.

Example


dbg>info reg 
Registers of strand 0: 
G registers: 
g[0] :  0000000000000000 0000000000000000 0000000000500000 0000000000000000  
g[4] :  0000000000000000 0000000000615fa0 0000000000000000 0000000000000000 
I registers:  
i[0] :  000000000000006e ffffffffef1fe8d4 0000000000520c30 0000000010e01bc8  
i[4] :  0000000000000000 0000000000000000 0000000010e00d91 000000000051458c 
O registers:  
o[0] :  000000000000006e 0000000000520c30 0000000010e01bc8 0000000000000000  
o[4] :  0000000000600000 0000000000000061 0000000010e00cd1 0000000000514a18 
L registers:  
l[0] :  000000000000006e 0000000010e0172c 000000000051e8f0 ffffffffef1fe8d4  
l[4] :  0000000000520c30 0000000000000000 0000000000000000 0000000000000000 
gl      :  0000000000000001  
tl      :  0000000000000001  
tt      :  000000000000007c  
tpc     :  0000000000508c88  
tnpc    :  0000000000508c8c  
tstate  : 0000009914001600  
pstate  : 0000000000000014  
tick    :  000001884f873558  
tba     :  0000000000500000  
asi     :  0000000000000014

Displaying Stack Trace

bt frame_count Command

Description

Displays the stack trace for the thread in focus for frame_count number of frames.

Example


dbg>bt 4 
frame 1, sp 0x10e03580, call instruction at 0x50e888: 
l[0] :  0000000000000001 00000000111606a8 0000000011160600 0000000000000000  
l[4] :  00000000006170d8 0000000000001000 0000000000010000 0000000000000150  
i[0] :  0000000000000800 0000000010e036f0 0000000010e036e8 0000000010e036e4  
i[4] :  0000000000002000 0000000019ae8ec8 0000000010e02e31 000000000050e888 
frame 2, sp 0x10e03630, call instruction at 0x50fcc4:  
l[0] :  0000000000000001 00000000111606a8 0000000011160600 0000000000000000  
l[4] :  00000000006170d8 0000000000001000 0000000000010000 0000000000000150  
i[0] :  0000000000000800 0000000000000001 0000000019d8c148 0000000000000800  
i[4] :  0000000019d8c140 0000000019d8c000 0000000010e02f01 000000000050fcc4 
frame 3, sp 0x10e03700, call instruction at 0x50fbd8:  
l[0] :  0000000000000001 00000000111606a8 0000000011160600 0000000000000000  
l[4] :  00000000006170d8 0000000000001000 0000000000010000 0000000000000150  
i[0] :  00000000111000e0 0000000000000015 0000000000010000 0000000011160580  
i[4] :  0000000000002000 0000000019ae8ec8 0000000010e02fd1 000000000050fbd8 
frame 4, sp 0x10e037d0, call instruction at 0x50e104:  
l[0] :  ffffffffffffffd8 ffffffffffffffba 0000000000000003 0000000000000000  
l[4] :  00000000006170d8 0000000000617000 0000000000000617 0000000000000400  
i[0] :  00000000111000e0 000000000000792d 000000000000792d 0000000011100180  
i[4] :  000000000000792d 0000000000000000 0000000010e03081 000000000050e104

Resolving Symbols

The dbghelper.pl script is used to resolve symbols to set breakpoints in the correct places. The script is located in install_dir/tools/bin directory, where install_dir is the SUNWndps package installation directory. For example, /opt/SUNWndps/tools/bin/dbghelper.pl.

-h Option

Description

Displays help information.

-f function_name Option

Description

Prints a debugger command to set a breakpoint at the given function_name. This option does not work for static functions. To set a breakpoint inside of a static function, use the -l file_name:line_number option.

Example


$ dbghelper.pl -f classifier ./main 
b 50b17c 

-g global_variable Option

Description

Prints a debugger command to display the contents of the given global_variable. The size of the memory displayed is fixed and does not consider the actual size of the global_variable. You might need to increase the size of the memory.

Example


$ dbghelper.pl -g stats ./main 
x/1wx 13000640 

-l file_name:line_number Option

Description

Prints a debugger command to set a breakpoint at the provided file_name:line_number. The file_name and line_number refer to your source code.

Example


$ dbghelper.pl -l src/classifier.c:57 ./main 
b 50b188


GNU Project Debugger (GDB) Showcase Application

GDB, the GNU Project debugger, enables you to debug your program in C source code level. The following sections describe the reference Netra DPS application that gives a showcase of the GDB support in Netra DPS over the Logical Domain Channel (LDC).

Configuring LDoms for GDB Showcase

Configuring LDoms for GDB showcase requires the latest LDoms release. If not installed, download and follow the LDoms latest release instructions. Find the latest release of LDoms at: http://www.sun.com/download/index.jsp?tab=2#L)

Use Logical Domains 1.0.1 or later release from the Sun Download Center.


procedure icon  To Configure LDoms Required to Run the GDB Demo

1. Add the following on the primary domain:


/opt/SUNWldm/bin/ldm remove-mau 8 primary
/opt/SUNWldm/bin/ldm remove-vcpu 28 primary
/opt/SUNWldm/bin/ldm remove-mem 31G primary
 
/opt/SUNWldm/bin/ldm add-vdiskserver primary-vds0 primary
/opt/SUNWldm/bin/ldm add-vconscon port-range=5000-5100 primary-vcc0 primary
/opt/SUNWldm/bin/ldm add-vswitch net-dev=e1000g0 primary-vsw0 primary
/opt/SUNWldm/bin/ldm list-config
/opt/SUNWldm/bin/ldm add-config remotecli-config

2. Reboot the system for remotecli-config to take effect.

3. Configure LDoms running Netra DPS as follows:


/opt/SUNWldm/bin/ldm add-domain ldg1
/opt/SUNWldm/bin/ldm add-vcpu 16 ldg1
/opt/SUNWldm/bin/ldm add-mem 4G ldg1
/opt/SUNWldm/bin/ldm add-vnet vnet1 primary-vsw0 ldg1
/opt/SUNWldm/bin/ldm add-vdsdev /export/bootdisk1 vol1@primary-vds0
/opt/SUNWldm/bin/ldm add-vdisk vdisk1 vol1@primary-vds0 ldg1
/opt/SUNWldm/bin/ldm add-vdpcs ndps-cli ldg1
/opt/SUNWldm/bin/ldm bind ldg1
/usr/openwin/bin/xterm -sb -sl 5000 -T “Console: ldg1” -geometry 80x12+98+193 -bg black -fg white
-e /usr/bin/telnet localhost 5000 &
/opt/SUNWldm/bin/ldm start ldg1

4. Configure LDoms running Solaris as follows:


/opt/SUNWldm/bin/ldm add-domain ldg2
/opt/SUNWldm/bin/ldm add-vcpu 12 ldg2
/opt/SUNWldm/bin/ldm add-mem 8G ldg2
/opt/SUNWldm/bin/ldm add-vnet vnet2 primary-vsw0 ldg2
/opt/SUNWldm/bin/ldm add-vdsdev /export/bootdisk2 vol2@primary-vds0
/opt/SUNWldm/bin/ldm add-vdisk vdisk2 vol2@primary-vds0 ldg2
/opt/SUNWldm/bin/ldm add-vdpcc solaris-cli ndps-cli ldg2
/opt/SUNWldm/bin/ldm bind ldg2
/usr/openwin/bin/xterm -sb -sl 5000 -T “Console: ldg2” -geometry 80x12+98+193 -bg black -fg cyan -e /usr/bin/telnet localhost 5001 &
/opt/SUNWldm/bin/ldm start ldg2


procedure icon  To Compile the GDB Showcase

single-step bullet  From the SUNWndps package, compile the application under /src/apps/gdb. Type:


gmake clean
gmake CMT=N1(for UltraSPARC T1 based platforms)
gmake CMT=N2 (for UltraSPARC T2 based platforms)
cd code/main
gmake

This action generates the binary file called main under src/apps/gdb/code/main. The required Solaris utility binaries are under src/apps/gdb/solaris-gw/ldc_so.


procedure icon  To Configure the Solaris Domain for GDB

Once the LDoms domains are configured and running, do the following steps to configure the gateway for GDB in the Solaris domain.

1. Copy the following files to your Solaris LDoms domain under src/apps/gdb/solaris-gw/ldc_so/.

To do this, copy the drivers to the location as shown in this example:


cp src/apps/gdb/solaris-gw/ldc_so/remldc.conf /kernel/drv/
cp src/apps/gdb/solaris-gw/ldc_so/remldc /kernel/drv/sparcv9

2. Load the drivers as shown below:


rem_drv remldc
add_drv remldc

3. Copy the configuration utility file to the following locations as shown below:


cp -f src/apps/gdb/solaris-gw/ldc_so/remotegw /usr/bin
chmod ugo+xr /usr/bin/remotegw
echo “remotegw    34980/tcp” >> /etc/services
cp -f src/apps/gdb/solaris-gw/ldc_so/remotegw.xml /var/svc/manifest/network/remotegw.xml
svccfg import /var/svc/manifest/network/remotegw.xml
svcadm enable svc:/network/remotegw:remotegw


procedure icon  To Load the GDB Showcase Binary in the Netra DPS Domain

1. Verify that you have copied your GDB Netra DPS binary main into your install server under /tftpboot.

2. Type the following at the Netra DPS domain OpenBoot Prom ok prompt:


ok   boot /virtual-devices@100/channel-devices@200/network@0:,main


procedure icon  To Run the GDB Command

See To Get GDB for getting the GDB program.

single-step bullet  Once your GDB showcase application is compiled, do the following under any host machine as long as you can access your code base src/apps/gdb/code/main:.


cd src/apps/gdb/code/main
gdb main

main is the same binary code that you loaded into your Netra DPS domain. The GDB debugger then displays the following:


GNU gdb 6.6
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type “show copying” to see the conditions.
There is absolutely no warranty for GDB.  Type “show warranty” for details.
This GDB was configured as “--host=sparc-sun-solaris2.10 --target=sparc64-elf”...
(gdb)

You can acquire the host name or IP address for the Solaris domain by typing:


ifconfig -a 
do: (assuming your ip is 10.7.56.249)
 
(gdb) target remote tcp:10.7.56.249:34980

This connects to your Netra DPS GDB application. The Netra DPS console then displays:


LDC Status = UP
calling set_debug_traps()....
Program started: initial breakpoint reached

This indicates that the GDB showcase application reached the initial breakpoint artificially created by the application. You can then use the following GDB commands to investigate your application.

GDB commands include the following:

For additional GDB information and instructions, see GDB: The GNU Project Debugger at http://sourceware.org/gdb/.


procedure icon  To Get GDB

1. Download the GDB source code from: http://sourceware.org/gdb/download

GDB version 6.6 or above is required.

2. Under the gdb directory, type the following:


./configure --target=sparc64-elf

The gdb binary is gdb/gdb. Also, you can use gdb/gdbtui which is a text-based user interface that enables you to browse the source code while gdb is debugging the target in one single screen.


procedure icon  To Create a GDB Showcase

1. Go to src/apps/gdb/code/main.


cd src/apps/gdb/code/main

2. Load the src/apps/gdb/code/main/main into your Netra DPS domain.

3. Configure the Solaris gateway in the Solaris domain (for example, 10.7.56.194).

4. Type the following:


gdb main

In this example, the following output is displayed:


Remote debugging using tcp:10.7.56.194:34980
0x00540df4 in teja_breakpoint ()
Current language:  auto; currently minimal

 


GNU gdb 6.6
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type “show copying” to see the conditions.
There is absolutely no warranty for GDB.  Type “show warranty” for details.
This GDB was configured as “--host=sparc-sun-solaris2.10 --target=sparc64-linux-elf”...
(gdb)  target remote tcp:10.7.56.249:34980
Remote debugging using tcp:10.7.56.249:34980
0x0053f34c in teja_breakpoint ()
Current language:  auto; currently minimal
(gdb) info thread
* 2 Thread 2 (stat_thd)  0x0053f34c in teja_breakpoint ()
 1 Thread 1 (main_thd00)  0x0050c5d8 in main_thread ()
   at src/apps/gdb/code/main/_src_app_remcon_impl.c:97
(gdb) thread 1
[Switching to thread 1 (Thread 1)]#0  0x0050c5d8 in main_thread ()
   at src/apps/gdb/code/main/_src_app_remcon_impl.c:97
97         while ( count  < (2))
(gdb) list debug_func
58         int i ;
59         int j ;
60      //      char *tmp; File: src/app/remcon_impl.c Line: 40
61         char * tmp ;
62      //      tmp = “0xdeadbeef”; File: src/app/remcon_impl.c Line: 41
63            tmp  = “0xdeadbeef”;
64      //      gdbptr = “0xbaddcafe”; File: src/app/remcon_impl.c Line: 42
65            gdbptr  = “0xbaddcafe”;
66      //      i = first_time++; File: src/app/remcon_impl.c Line: 43
67            i  =  first_time ++;
(gdb) b 67
Breakpoint 1 at 0x50c510: file src/apps/gdb/code/main/_src_app_remcon_impl.c, line 67.
(gdb) p first_time
$1 = 18255275
(gdb) c
Continuing.
Can’t send signals to this remote system. SIGSTOP not sent.

 


Program received signal SIGSTOP, Stopped (signal).
debug_func ()
   at src/apps/gdb/code/main/_src_app_remcon_impl.c:67
67            i  =  first_time ++;
(gdb) c
Continuing.
Can’t send signals to this remote system.  SIGSTOP not sent.
 
Program received signal SIGSTOP, Stopped (signal).
0x0050c514 in debug_func ()
   at src/apps/gdb/code/main/_src_app_remcon_impl.c:67
67            i  =  first_time ++;
(gdb) c
Continuing.
Can’t send signals to this remote system.  SIGSTOP not sent.
Program received signal SIGSTOP, Stopped (signal).
debug_func ()
   at src/apps/gdb/code/main/_src_app_remcon_impl.c:67
67            i  =  first_time ++;
(gdb) p first_time
$4 = 18255276
(gdb) info thread
 2 Thread 2 (stat_thd)  0x00508f04 in hv_ldc_rx_get_state ()
* 1 Thread 1 (main_thd00)  debug_func ()
   at src/apps/gdb/code/main/_src_app_remcon_impl.c:67
(gdb) s
Can’t send signals to this remote system.  SIGSTOP not sent.
 
Program received signal SIGSTOP, Stopped (signal).
0x0050c534 in debug_func ()
   at src/apps/gdb/code/main/_src_app_remcon_impl.c:67
67            i  =  first_time ++;
(gdb) s
Can’t send signals to this remote system.  SIGSTOP not sent.
 
Program received signal SIGSTOP, Stopped (signal).
debug_func ()
   at src/apps/gdb/code/main/_src_app_remcon_impl.c:69
69            j  =  i  + ( first_time );
(gdb) detach
Ending remote debugging.

5. Once debugging is completed, type detach.


gdb detach



Note - Once completed, always type detach on gdb. Otherwise, the /usr/bin/remotegw process is left in the Solaris domain 10.7.56.249. Then use the kill command to kill the /usr/bin/remotegw process before you start again.


6. To start the process again, type the following:


./ldm stop the-ndps-domain
./ldm start the-ndps-domain

7. Reload your binary in the Netra DPS domain.

If your source changes, you need to quit the gdb and re-enter gdb.