This chapter presents the source-level debugging architecture in the ChorusOS 4.0 operating system. It explains how to configure the different servers and tools, and how to use them.
"Preparing the System for Symbolic Debugging" describes how to prepare your system for symbolic debugging.
"Application Debugging Architecture" gives an overview of the application debugging architecture and lists the steps involved in setting up a system debugging session.
"System Debugging Architecture" gives an overview of the system debugging architecture and lists the steps involved in setting up an application debugging session.
"Sample XRAY Start-up Script" presents a script which can be used to start the XRAY Debugger for the ChorusOS operating system.
In order to use all the debugging features in the XRAY Debugger, you need to generate symbolic debugging information when you compile components. This information is stored in the object files and describes the data type of each variable or function and the correspondence between source line numbers and addresses in the executable code.
How you enable debugging in components will depend on which release of ChorusOS 4.0 you have. The binary release of ChorusOS 4.0 includes the source code for the BSP, driver and example components. These you will compile in what is known as an imake build environment because the imake tool is used to create the Makefiles for these components.
The source release of ChorusOS 4.0 includes everything in the binary release plus source code for system components, such as the kernel and the operating system. These components are built in an mkmk build environment where the tool mkmk is used to build Makefiles. For more details see ChorusOS 4.0 Production Guide.
To build all your components with symbolic debugging information turned on:
Edit the Paths file located in the root of your build directory, created after you run the configure command, and add the following line to the end:
FREMOTEDEB=ON
Other ways can be used to selectively build your components with symbolic debugging information. These are presented below.
To enable symbolic debugging throughout the component directory and its sub-directories:
Edit the Project.tmpl file located in the root of the component source directory, and add the following line to the end:
DEBUG=$(DEBUG_ON)
Change directory to the root of your build directory and remove all the object files and executables:
% make clean |
Rebuild the local Makefile:
% make Makefile |
Rebuild the sub-directory Makefiles:
% make Makefiles |
Finally, rebuild the component:
% make |
To enable symbolic debugging in selected component directories:
Edit the Imakefile within each desired component source directory, and add the following line to the end:
DEBUG=$(DEBUG_ON)
Change directory to the root of your build directory and remove all the object files and executables:
% make clean |
Rebuild the local Makefile:
% make Makefile |
Finally, rebuild the component:
% make |
If you prefer not to modify the Imakefile or Project.tmpl files, there is an alternative way of enabling debugging. You can pass the debug option within the make command itself:
Change to your build directory and remove all the object files and executables:
% make clean |
Rebuild the component with symbolic debugging enabled:
% make MAKE="make DEBUG=-gdwarf-2" |
You can also create a DEBUG environment variable. If you use the C shell:
% setenv DEBUG -gdwarf-2 |
If you use the Bourne shell:
$ DEBUG=-gdwarf-2 $ export DEBUG |
Now call make with the -e option to import environment variables:
% make -e |
Once a component has been compiled in debug mode, rebuild and reboot the system image.
To enable symbolic debugging for system components:
Change to your build directory and remove all the object files and executables:
% make clean |
Create a mkmk build definition file:
% echo 'FREMOTEDEB=ON' > filename.df |
filename can be a name of your choice.
Rebuild the system component:
% make makemk |
The DebugAgent is activated by enabling the DEBUG_SYSTEM
feature with the configurator(1CC) command:
% configurator -set DEBUG_SYSTEM=true |
The DEBUG_SYSTEM
feature is set to true by default.
When the DebugAgent is activated, communications on the serial line are performed in binary mode.
The DebugAgent has eight tunable options that you can configure with ews or configurator. The following three tunables
control the behavior of the DebugAgent when it is enabled (DEBUG_SYSTEM
=true):
dbg.agent.startup
specifies the behavior
of the DebugAgent when the DebugServer attempts to connect to the target.
Possible values are stop or resume.
In stop mode the system will wait indefinitely until the
DebugServer connects to the DebugAgent. In resume mode
the system will wait for no more than one second. In both cases, the DebugAgent
will issue the prompt DebugAgent: trying to sync with DebugServer... over the serial line and sound a beep. If the DebugServer
does not connect, the system will continue booting and console operations
will be performed raw, without any processing by the DebugAgent, on the serial
line.
dbg.agent.exceptmode
specifies
the behavior of the DebugAgent when an exception is raised before the DebugServer
has connected. Possible values are catch or forward. In catch mode the DebugAgent catches all exceptions
and blocks the target until a DebugServer has connected and resumed the execution
of the target. In forward mode the DebugAgent forwards
all exceptions directly to the kernel-installed handlers.
dbg.agent.consolemode
specifies the operating mode of the system console. Possible values are sync or async. In sync mode
the target is blocked until each message has been transmitted to the host
for output and acknowledged. In async mode, console output
is buffered and transmitted to the host periodically, either when the buffer
is full or on each timer interrupt, whichever occurs first.
The following five tunables control the serial line used by the DebugAgent.
dbg.agent.device
specifies the serial
device used by the DebugAgent. Possible values are COM1, COM2, COM3, or COM4. COM1 refers to communication port 1 (COM 1) on a PC,
or the first serial line on other boards. COM2 refers to
communication port 2 (COM 2) on a PC, or the second serial
line on other boards, and so on.
dbg.agent.baud
specifies
the baud rate of the serial line. Possible values are: 115200, 57600, 38400,
19200, 9600, 4800, 2400, or 1200.
dbg.agent.parity
specifies
the parity of the serial line. Possible values are none, even, or odd.
dbg.agent.databits
specifies
the number of data bits. The only possible value is 8.
dbg.agent.stopbits
specifies
the number of stop bits. Possible values are 1 or 2.
When the DebugAgent is not active (DEBUG_SYSTEM
=false), the serial line is used by the system debugging console,
and the five tunables control the serial device and speed.
This section describes the components within the application debugging architecture.
The application debugging architecture has two components:
XRAY Debugger for ChorusOS
Remote debug server (RDBC) rdbc(1CC)
The XRAY Debugger for ChorusOS runs on the host. The remote debug server runs on the target and communicates with XRAY over the Ethernet. This is illustrated in Figure 10-1.
Application debugging is intended to be used for debugging user applications, dynamically loaded actors, as well as certain supervisor actors. It is not possible to debug the Actor Management (AM) or I/O Management components (IOM), the kernel, or the system drivers. Application debugging relies on the RDBC supervisor actor which uses the services of the AM, the IOM, the kernel, and system drivers such as the Ethernet driver. When an application is debugged, only that application is affected. Other applications in the operating system, as well as the operating system itself, will keep running.
To begin an application debugging session, follow these steps:
Ensure that your target is connected to your network.
Prepare the system for symbolic debugging. See "Preparing the System for Symbolic Debugging" for information on how to do this.
Configure and start rdbc, the ChorusOS remote debug server. See "RDBC Configuration and Usage" and rdbc(1CC).
Configure and start the XRAY Debugger for ChorusOS. See "Sample XRAY Start-up Script".
The RDBC server can be started automatically or manually:
For RDBC to be started automatically, the conf/sysadm.ini file, read during system initialization by C_INIT
, must contain a command which mounts the NFS root. If this mount command is present, edit
the conf/sysadm.ini file and add the following line after
it:
rdbc
It is extremely important that RDBC is started after the NFS root is mounted on the target.
RDBC can be started manually, as a normal application, as follows:
% rsh -n name arun rdbc |
Running RDBC manually gives you freedom to choose when you want to carry out application debugging, freeing up valuable resources.
To stop RDBC, use the akill command. First identify the actor process ID (aid):
% rsh name aps |
Then kill the RDBC process:
% rsh name akill aid |
Your XRAY application debug session will be lost if you stop RDBC.
Information about what targets are available to XRAY is held in the file chorusos.brd. There are four columns: the machine names where RDBC executes are specified in the first column, slot numbers are specified in the second column, and the last two columns are for comments. XRAY interprets integer values between 0 and 25 in the second column as slot numbers and larger values as TCP/IP port numbers, and will adapt its connection to the server accordingly. The default TCP/IP port number of RDBC is 2072.
Here is an example chorusos.brd file:
target-i386 2072 "i386" "Application debug of target-i386" target-ppc 2072 "ppc" "Application debug of target-ppc"
The entries specify the application debug of actors running on target-i386 and target-ppc respectively, and require that RDBC be running on both machines.
Two or more RDBC servers can be run on the same target to provide you with a separate console for each program being debugged.
See rdbc(1CC) for more information.
The name and port number specified in chorusos.brd have different meanings:
For application debug, the chorusos.brd file specifies the target name, and the port number corresponds to a UDP/IP port number, 2072 by default.
For system debug, the chorusos.brd file specifies the host name where the RDBS server is running. The port number corresponds to the RDBS slot number in the range 0..25.
This section describes the components within the system debugging architecture.
The system debugging architecture has the following components:
XRAY Debugger for ChorusOS
Remote debug server (RDBS) rdbs(1CC)
ChorusOS debug server (DebugServer) chserver(1CC)
Debug agent (DebugAgent)
The first three components run on the host. The fourth, the debug agent, runs on the target and communicates with the ChorusOS debug server through a serial connection. This is illustrated in Figure 10-2.
A more detailed description of the debugging architecture can be found in Chapter 2 and 3 of the ChorusOS Debug Architecture and API Specifications document (/opt/SUNWconn/SEW/4.0/chorus-doc/pdf/DebugApi.pdf).
System debugging is intended to be used for debugging different parts of the ChorusOS operating system. This includes the kernel, the system drivers, the BSP, and those supervisor actors that cannot be debugged with application debugging such as the AM, and the IOM. System debugging also allows you to debug interrupt and exception handlers. During system debugging, the whole operating system is affected.
To begin your first system debugging session, follow these steps:
Connect a serial cable between the host and target.
Prepare the system for symbolic debugging. See "Preparing the System for Symbolic Debugging" for information on how to do this.
Start and configure the ChorusOS DebugServer chserver. See "Setting up a Debugging Session" and chserver(1CC).
Register the target with chadmin, the ChorusOS DebugServer administration tool. See "Registering a Target".
Configure and start rdbs, the ChorusOS debug server for the XRAY Debugger. See "DebugServer Configuration File" and rdbs(1CC).
Start the ChorusOS debug console chconsole. See the chconsole(1CC) man page for further details.
Configure and start the XRAY Debugger for ChorusOS. See "Sample XRAY Start-up Script".
If you do not start chserver you will not be able to use chconsole. However, you can still view the system console using the tip or cu commands. See tip(1) and cu(1C) for more details.
For subsequent debugging sessions, you need only perform the following steps:
Start the ChorusOS DebugServer chserver, if it is not already running.
Start rdbs, the ChorusOS debug server for the XRAY debugger, if it is not already running.
Start the ChorusOS debug console chconsole.
Start the XRAY Debugger for ChorusOS.
The ChorusOS DebugServer chserver communicates with the target through a serial cable connection and must be run on the host to which the target is connected.
To identify the serial device, look in the /etc/remote file. This file contains references to remote systems that you can access through your local serial line. For more details, see remote(4). The device is usually named /dev/ttya or /dev/ttyb and will be the same device used by the tip or cu tools. The device must be readable and writable by all users.
The DebugServer is a Sun RPC server that is registered with the rpcbind server. When you require more than one debug server to run on the same host, assign a unique slot number (in the range 0..65535) to each of them so that individual debug servers can be identified. If only one debug server is started on a given host, it is not necessary to allocate a slot number as 0 will be used by default.
If you decide to assign a slot number to your DebugServer, use the DebugServer environment variable CHSERVER_HOST.
The DebugServer, as well as all the other tools based on the Debug Library, uses the optional environment variable CHSERVER_HOST. This environment variable indicates:
the host name where your DebugServer is running
optionally, the slot number for the DebugServer RPC service
The format of the environment string is host[:slot-id]. For example:
% setenv CHSERVER_HOST jeriko % setenv CHSERVER_HOST concerto:3 |
Configuration information about targets is held in a special file which the DebugServer reads every time you run it. For each target, the configuration file contains:
The name of the target
The serial device used to communicate with the target
Configuration parameters for the serial device (baud rate, parity)
The architecture type of the target (i386, PPC, SPARC)
The absolute path of the layout.xml image layout file generated by mkimage
When a new target is registered, see "Registering a Target" for details of how to do this, the configuration file is modified.
On the host to which your target or targets are connected, type the following command:
% chserver |
This will start the DebugServer as a background process. An empty configuration file called dbg_config.xml is copied into your home directory the first time you run the DebugServer.
If you have defined a slot number n and not set the environment variable, you can start the DebugServer as follows:
% chserver -slot n |
A complete description of the DebugServer is given in the chserver(1CC) man page.
If chserver is run on a different host to the one the system image was built on, particularly on a shared file system with a different view of the build directory, the tool will not be able to access the necessary source files during system debugging. This problem is NFS related, the symbolic link created on one host may not be valid for another, and is due to there being relative file references in the layout.xml file. There are two solutions to the problem:
Break the symbolic link with the file /build-NUCLEUS/conf/mkimage/layout_typedef.xml accessed by chserver.
Copy the conf and image directories to the host where chserver will run, then use chadmin to set the path of the layout.xml file.
Stop the DebugServer by using the chadmin tool.
% chadmin -shutdown |
Before registering a target you need to know:
The name of the target
The name of the serial device
The path of the layout.xml file generated by mkimage (the path of this file is printed by mkimage when a system image is built)
Now you can register the target by typing:
% chadmin -add-serial-target name/ -device device/ -layout-file layout_file |
name is the name of your target, device is the serial device that you have identified, and layout_file is the absolute path of the layout.xml file.
You only need to register a target once as configuration information is saved in your dbg_config.xml file.
Use chadmin to update the information that you gave during the registration of your target.
The following example sets the baud rate to 38400, the parity to none, and uses the device /dev/ttya for the target name:
% chadmin -baud 38400 -parity none -device /dev/ttya name |
If you wish to specify a new layout.xml file because the path has changed (see note), use the following command to inform the DebugServer of the new path:
% chadmin -layout-file path/layout.xml name |
When you change the mode of system image booting (using the BOOT_MODE
global variable) or select a different system image
configuration (using the SYSTEM
global variable),
the path to the layout.xml file will change.
For example, if you type:
% configurator -set SYSTEM=kts |
The path will change to build_dir/image/RAM/kts/layout.xml.
If you then type:
% configurator -set SYSTEM=chorus |
The path will change to build_dir/image/RAM/chorus/layout.xml.
Similarly, if you change the mode of system image booting:
% configurator -set BOOT_MODE=ROM |
The path will change to build_dir/image/ROM/chorus/layout.xml.
A target can be deactivated to disconnect the DebugServer from the DebugAgent and release the serial device used by the DebugServer. When a target is deactivated, the DebugAgent switches to a stand-alone mode. The chconsole must no longer be used as the DebugServer does not read the serial line any more. Instead, you must start the tip(1) or cu(1C) tools to gain access to the system debugging console.
A target may be temporarily removed (deactivated) with the following command:
% chadmin -deactivate name |
name is the name of your target.
When a target is deactivated, it is not removed from the DebugServer configuration file so that it is possible to reactivate it later.
Before the target can be reactivated, you must stop any tip or cu tools which may be using the serial line.
The target is reactivated with this command:
% chadmin -activate name |
name is the name of your target.
The DebugServer will synchronize with the DebugAgent and the DebugAgent will switch to a binary protocol mode. At this stage, you must use the chconsole to gain access to the system debugging console.
A target may be permanently removed by first deactivating it (see "Deactivating a Target") then unregistering it with this command:
% chadmin -remove-target name |
name is the name of your target.
Providing you have deactivated the target first, the target's configuration information will be deleted from the configuration file.
If RDBS is started without any parameters it will connect, by default, to the first target available on the DebugServer. However, you can specify a target name on the command-line provided the name you use is registered with the DebugServer.
This name is unrelated to the name under which the target might be known on the TCP/IP network (through another connection). It only identifies the serial line connecting the target with the DebugServer.
A complete set of command-line parameters are documented in rdbs(1CC).
Several RDBS servers may be run on one machine to debug several targets, provided you define a different slot for each server.
Information about what targets are available to XRAY is held in the file chorusos.brd. There are four columns: the machine names where RDBS executes are specified in the first column, slot numbers are specified in the second column, and the last two columns are for comments. XRAY interprets integer values between 0 and 25 in the second column as slot numbers and larger values as TCP/IP port numbers, and will adapt its connection to the server accordingly.
Here is an example chorusos.brd file:
rdbshost 0 "i386" "System debug of target-i386" rdbshost 1 "ppc" "System debug of target-ppc"
The entries specify that two copies of RDBS will run on the rdbshost machine (a Solaris workstation): one on slot 0, configured to debug the target-i386 target, and another on slot 1, configured to debug the target-ppc target.
The name and port number specified in chorusos.brd have different meanings:
For application debug, the chorusos.brd file specifies the target name, and the port number corresponds to a UDP/IP port number, 2072 by default.
For system debug, the chorusos.brd file specifies the host name where the RDBS server is running. The port number corresponds to the RDBS slot number in the range 0..25.
Combine the example chorusos.brd files given in "RDBC Configuration and Usage" and "RDBS Configuration and Usage":
rdbshost 0 "i386" "System debug of target-i386" rdbshost 1 "ppc" "System debug of target-ppc" target-i386 2072 "i386" "Application debug of target-i386" target-ppc 2072 "ppc" "Application debug of target-ppc"
The first two entries specify that two copies of RDBS will run on the rdbshost machine (a Solaris workstation): one on slot 0, configured to debug the target-i386 target, and another on slot 1, configured to debug the target-ppc target.
The last two entries specify the application debug of actors running on target-i386 and target-ppc respectively, and require that RDBC be running on both machines.
By attaching to the first and third targets, you can carry out application and system debugging on the same target concurrently. However, while the system is stopped it is not possible to carry out application debug because halting the system halts RDBC, as well as the application itself.
In this session the target is named target-i386, the workstation is named workstation1 and all host tools are available. Several actors and drivers have been added to the system, and they have been compiled for system debugging.
Make sure you have enabled the system debugging during system generation (see "Compiling for Debugging"), then run DebugServer (see "Starting the DebugServer") and connect a console to target-i386. Run RDBS in the following manner:
% rdbs target-i386 |
Run XRAY (by using the "Sample XRAY Start-up Script", for example), then go to the Managers window and select the Connect tab.
Select the Boards->Add or Copy board entry to register your target for system debug. XRAY opens the Add/Copy Board Entry pop-up dialog:
Enter the host name where RDBS is running in the Name of Board field. Enter the slot number used by RDBS (0 by default) in the Port as String field. Leave the other fields blank.
On Windows NT, XRAY uses native Windows pathnames and it not aware of the Cygwin UNIX emulation layer used by the ChorusOS host tools. As a result, pathname translations must be specified so that XRAY can translate the Unix-like pathnames returned by the DebugServer, or embedded in object modules, into native Windows NT pathnames. Typical pathname translations are /c/=C:\ and /d/=D:\. They must reflect the results of the Cygwin mount command.
After the dialog box is validated, the new board appears in the window. Connect to the RDBS server by double-clicking on it with the left mouse button.
This will connect you to RDBS, and through it to the DebugServer and the target. You can now view the system as it runs.
Enter the following in the command-line area of the Code window to see a list of actors running on the target:
Conn> stat actors |
Select the Process tab in the Managers window. The Available Processes list will show a single entry representing the system as process number 1. Double-clicking on it will stop the system and initiate a debugging session.
XRAY will present you with the list of actors for which symbols should be loaded. By default, all actors are selected and you can press the OK button. XRAY will find the executable files automatically. For some of them, it may not have the path and it will prompt for the pathname of the missing executable file. If the actor's binary file is statically linked, you must indicate the path where it is located. If the actor's binary file is relocatable, then your only option is to select Cancel, because system debugging does not support the debugging of actors loaded from relocatable binaries.
After all selected actors have been loaded, XRAY will show where the system has been stopped in the Code window. The name of the thread which was executing last, also known as the current thread, will be displayed in the title bar. Thread execution can now be controlled.
Think of a function you want to debug, myFunc() for example, and perform this command:
% scope myFunc |
XRAY displays the source code of the function in the Code window. You can place a breakpoint in it for the current thread by double-clicking with the left button on the selected line number. This will set a per-thread breakpoint, for the current thread.
If you do not know whether the current thread will execute this function, place a global breakpoint by opening a local menu with the right mouse button and selecting Set Break All Threads. Press the Go button to resume running the function.
Once the breakpoint has been reached, examine the values of variables by double-clicking on them with the right mouse button.
If the breakpoint is not reached, and the system continues to run, you can stop it asynchronously using the Stop button. The Code window will show the stop location.
Due to the way in which the stop operation has been implemented, this will always be the same location inside the clock interrupt handler, except if the system was blocked in a console input, or performing console output.
You can find the interrupted location by examining the stack with the Up button.
The chls tool is available from the XRAY command window with the dchls command. You can use this command to display values which are not directly visible in the XRAY windows. For example, to look at the processor specific registers, type the following command:
Stop> dchls -special-regs |
If the DebugServer process is terminated, RDBS will attempt to reconnect to a new DebugServer process automatically. If there was a debugging session open at the time, the single process representing the ChorusOS operating system will be killed, and the debugging context lost. You will need to re-grab the process after RDBS has reconnected to the new DebugServer. If this does not work, then kill and restart RDBS.
If the target is rebooted, the single process representing the ChorusOS operating system will appear in the XRAY output window first as killed, and shortly after as restarted. XRAY will then attempt to reinsert all previously set breakpoints and promote them from thread-specific to global. Any breakpoints that cannot be reinserted will be deleted.
If you stop the system while it is waiting for console input, it will not resume until you provide some keyboard input.
Currently, the DebugServer does not offer access to the target's ChorusOS IPC ports. RDBS will report this by printing a warning message on start-up.
If a given symbols is present in several actors, or in several modules in a single actor (a static symbol, for example), then you can use the ps /f symbol_name command to display all of the occurrences of the symbol, complete with a full pathname. The full pathname is of the @binary_file\\module\symbol_name form. Use this full pathname to reference symbols which not in the current scope.
Because XRAY asks for a thread list each time a debugged process stops, and generating the list takes a long time during system debugging, the thread list shown in the Threads Manager is simplified. It does not include fields names, such as actor names, and is only updated when the current thread changes. The full thread list is available from the Resource Viewer or through the dallthreads command. Per-actor threads can be displayed with the command dthreads=aid. You can force the full thread list to be displayed, both in the Resource Viewer and in the Threads Manager, by permanently leaving the Resource Viewer window open on the thread list.
A sample script for setting up and starting XRAY is provided below.
Create a sub-directory within your system image directory to hold the script. For example, if your system image is called chorus.RAM, the directory would be image/RAM/chorus/bin.
Remember to modify the line which initializes the XRAY_INSTALL_DIR environment variable to point to the directory where XRAY is installed. This directory also contains the bin, docs, docschxx, license, master, xraycore and xrayrdb sub-directories. The script assumes you have put the license.dat file into this directory.
This shell script works if you use either a time-limited licence for XRAY, or a license locked to your machine. Please refer to the XRAY documentation for details of the other options available.
#!/bin/sh set +x XRAY_INSTALL_DIR=<xray_install_dir> # Clean up possible crash /bin/rm -f core /tmp/.MasterMsg/.MasterSock.$DISPLAY* # Prepare environment variables XRAYMASTER=$XRAY_INSTALL_DIR/master export XRAYMASTER USR_MRI=$XRAY_INSTALL_DIR export USR_MRI LD_LIBRARY_PATH=$XRAYMASTER/lib export LD_LIBRARY_PATH # LM_LICENSE_FILE=$XRAY_INSTALL_DIR/license.dat LM_LICENSE_FILE=/Work/build/mir/mri/mri/license.dat export LM_LICENSE_FILE # If you use a license server, the following line starts it # ./mri/bin/lmgrd # Then we change the LM_LICENSE_FILE variable to point to the server # LM_LICENSE_FILE=port_number@machine_name # Run XRAY itself $XRAY_INSTALL_DIR/master/bin/xray -VABS=rdb $* # ./mri/master/bin/xray $*