Multithreaded Programming Guide

Chapter 8 Compiling and Debugging

This chapter describes how to compile and debug multithreaded programs. This chapter discusses the following topics:

Setting Up the Solaris Environment for Developing Multithreaded Applications

To build software on the Solaris OS, you must install the tools you need on your development machine. Whether you want to use the standard tools that are bundled in Solaris OS, or use the Sun Studio tools, you must first install the appropriate Solaris software for a developer environment. If you are using a Solaris Express Developer Edition release, the developer environment is installed by default.

For Solaris 10 releases, the Solaris OS includes a number of developer software packages, which are available when you install the appropriate Solaris software group for developers. For installation purposes, the Solaris OS is logically divided into software groups, which are collections of Solaris packages.

When you install the Solaris OS, you must select one of the following software groups, which contain the developer packages:

Developer
Entire
Entire Plus OEM

With these software groups, you get compilers such as the GNU C compiler (gcc) and the Java compiler (javac). Also installed are the GNU source-level debugger (gdb) and the Modular Debugger (mdb), a linker (ld), source control utilities (sccs), and build utilities such as make. The files are installed in /usr/css and /usr/sfw.

Note –

This manual describes building a multithreaded application using the Sun Studio C compiler. The Sun Studio software is included in the Solaris Express Developer Edition release. If you are using a Solaris 10 release, you can download the Sun Studio software from the http://developers.sun.com/sunstudio/ web site.

Compiling a Multithreaded Application

This section explains how to compile a multithreaded program using the Sun Studio C compiler. The Sun Studio C compiler is optimized for parallel programming and includes many features that are not available in other C compilers. See the Sun Studio 12: C User’s Guide for more information about the C compiler.

Preparing for Compilation

Your application must include <thread.h> for Solaris threads and <pthread.h> for POSIX threads. You should include the appropriate file for the API you are using, or both files if your application uses both thread APIs. See the pthread.h(3HEAD) man page for more information. The application must also include <errno.h>, <limits.h>, <signal.h> , <unistd.h> files.

Choosing Solaris or POSIX Threads

The Solaris implementation of Pthreads is completely compatible with Solaris threads. You can use both Solaris threads and Pthreads in the same application. See the pthreads(5) man page for a discussion of the differences between the thread implementations. See also Chapter 6, Programming With Solaris Threads in this manual for information about differences.

One difference between the thread types is the behavior of the fork functions.

In the Solaris 9 release, the behavior of the fork() function depended on whether or not the application was linked with the POSIX threads library. When linked with -lthread (Solaris Threads) but not linked with -lpthread (POSIX Threads), fork() would duplicate in the child thread all the threads from the parent process. When the application was linked with -lpthread, whether or not also linked with -lthread, fork() was the same as fork1() and only the calling thread is duplicated.

Starting in the Solaris 10 release, a call to the forkall() function replicates in the child process all of the threads in the parent process. A call to fork1() replicates only the calling thread in the child process. In the Solaris 10 release, a call to fork() is identical to a call to fork1(); only the calling thread is replicated in the child process. This is the POSIX-specified behavior for fork(). Applications that require replicate-all fork semantics must call forkall().

Including `<thread.h>` or `<pthread.h>`

The include file <thread.h> contains declarations for the Solaris threads functions. To call any Solaris thread functions, your program needs to include <thread.h>. This file enables you to produce compiled code that is compatible with earlier releases of the Solaris software.

The include file <pthread.h> contains declarations for the Pthreads functions and is required if your program uses Pthreads.

You can mix Solaris threads and POSIX threads in the same application by including both <thread.h> and <pthread.h> in the application. Then when linking and compiling you need to specify the -lpthread flag to link in the pthread APIs.

When using -mt, the Solaris threads APIs will be linked automatically. Always use the -mt option instead of listing -lthread explicitly. To use Pthreads, specify the -mt option and -lpthread option on the link command line. The libpthread library provides an interface to libthread, so you still need libthread when using Pthreads.

Compiling and Linking a Multithreaded Program

The Sun Studio C compiler (cc) provides the -mt option to compile and link multithreaded code. The -mt option assures that libraries are linked in appropriate order.

The -mt option must be used consistently. If you compile with -mt and link in a separate step, you must use the -mt option in the link step as well as the compile step. If you compile and link one translation unit with -mt, you must compile and link all units of the program with -mt.

Although the same commands can be used to compile and link in Solaris 10 and Solaris 9 releases, some differences should be considered:

In the Solaris 10 release, you do not need to explicitly link with the libthread or libpthread libraries because all the threading functions are in the libc library.
In the Solaris 9 release, the Solaris threads API and Pthreads API are contained in separate libraries, libthread and libpthread. An application compiled with the separate libraries on Solaris 9 should run on Solaris 10 because the libthread.so and libpthread.so shared objects are implemented as filters on libc.so.1. An application compiled on Solaris 10 can run on Solaris 9 if the application was explicitly linked with the libthread or libpthread libraries. Linking with -mt preserves the Solaris distinction between fork() and fork1() when the application runs on Solaris 9. Linking with the -lpthread option makes fork() behave the same way as the Solaris fork1() call in Solaris 9 and prior releases.
In the Solaris 9 release, all calls to libthread and libpthread are no-ops if the application does not link -lthread or -lpthread. The runtime library libc has many predefined libthread and libpthread stubs that are null procedures. True procedures are interposed by libthread or libpthread when the application links both libc and the thread library.
In Solaris 9 and subsequent releases, linking a non-threaded program with -mt, -lthread, or -lpthread makes no semantic difference to the program. No extra threads or extra LWPs are created. The main, and only, thread executes as a traditional single-threaded process. The only effect on the program is to make system library locks become real locks, as opposed to dummy function calls. You must pay the price of acquiring uncontended locks.

Compiling and Linking in the POSIX Threads Environment

If your application uses only Pthreads or uses both Solaris threads and Pthreads, use the following command to compile and link:

cc -mt [ flag ... ] file... [ library... ] -lpthread

The -mt option links in the libthread library, while the -lpthread option links in the libpthread library. Both flags are needed when using Pthreads because libpthread provides an interface to libthread.

The -mt option can appear anywhere in the command line. The -lpthread option should come after any user libraries. The relative positions of -mt and -lpthread do not matter.

For example, the following lines are equivalent:

cc -mt -o myprog f1.o f2.o   -lmylib -lpthread
cc     -o myprog f1.o f2.o -mt -lmylib -lpthread
cc     -o myprog f1.o f2.o -lmylib -mt -lpthread
cc     -o myprog f1.o f2.o -lmylib -lpthread -mt

See the Sun Studio cc(1) man page and Sun Studio 12: C User’s Guide for more information about the cc command options.

Compiling and Linking in the Solaris Threads Environment

In a Solaris threads environment, use the following options to compile and link your application:

If you application uses only Solaris threads, use the following command to compile and link:

cc -mt [ flag ... ] file... [ library... ]

The -mt option links in the libthread library.

See the Sun Studio cc(1) man page and Sun Studio 12: C User’s Guide for more information about the cc command options.

Compiling and Linking in a Mixed Threads Environment

If your application uses both Pthreads and Solaris threads functions, you can compile and link with the same command used for compiling for Pthreads only:

cc -mt [ flag ... ] file... [ library... ] -lpthread

In mixed usage, you need to include both thread.h and pthread.h.

Linking With `-lrt` for POSIX Semaphores

The Solaris semaphore routines, sema_*(3C), are contained in the standard C library. By contrast, you link with the -lrt library to get the standard sem_*(3RT) POSIX semaphore routines described in Synchronization With Semaphores.

Alternate Threads Library

The Solaris 8 release introduced an alternate threads library implementation that is located in the directories /usr/lib/lwp (32-bit) and /usr/lib/lwp/64 (64-bit). In the Solaris 9 release, this implementation became the standard threads implementation found in /usr/lib and /usr/lib/64. Effective with the Solaris 10 release, all threads functionality has been moved into libc and no separate threads library is required. The /usr/lib/lwp directories are maintained for compatibility of Solaris 8 applications.

Debugging a Multithreaded Program

The following discussion describes characteristics that can cause bugs in multithreaded programs. Utilities that you can use to help debug your program are also described.

Common Oversights in Multithreaded Programs

The following list points out some of the more frequent oversights that can cause bugs in multithreaded programs.

A pointer passed to the caller's stack as an argument to a new thread.
The shared changeable state of global memory accessed without the protection of a synchronization mechanism leading to a data race. A data race occurs when two or more threads in a single process access the same memory location concurrently, and at least one of the threads tries to write to the location. When the threads do not use exclusive locks to control their accesses to that memory, the order of accesses is non-deterministic, and the computation may give different results from run to run depending on that order. Some data races may be benign (for example, when the memory access is used for a busy-wait), but many data races are bugs in the program. The Thread Analyzer tool is useful for detecting data races. See Detecting Data Races and Deadlocks Using Thread Analyzer.
Deadlocks caused by two threads trying to acquire rights to the same pair of global resources in alternate order. One thread controls the first resource and the other controls the second resource. Neither thread can proceed until the other gives up. The Thread Analyzer tool is also useful for detecting deadlocks. See Detecting Data Races and Deadlocks Using Thread Analyzer.
Trying to reacquire a lock already held (recursive deadlock).
Creating a hidden gap in synchronization protection. This gap in protection occurs when a protected code segment contains a function that frees and reacquires the synchronization mechanism before returning to the caller. The result is misleading. To the caller, the appearance is that the global data has been protected when the data actually has not been protected.
When mixing UNIX signals with threads, and not using the sigwait(2) model for handling asynchronous signals.
Calling setjmp(3C) and longjmp(3C), and then long-jumping away without releasing the mutex locks.
Failing to re-evaluate the conditions after returning from a call to *_cond_wait() or *_cond_timedwait().
Forgetting that default threads are created PTHREAD_CREATE_JOINABLE and must be reclaimed with pthread_join(3C). Note that pthread_exit(3C) does not free up its storage space.
Making deeply nested, recursive calls and using large automatic arrays can cause problems because multithreaded programs have a more limited stack size than single-threaded programs.
Specifying an inadequate stack size, or using nondefault stacks.

Multithreaded programs, especially those containing bugs, often behave differently in two successive runs, even with identical inputs. This behavior is caused by differences in the order that threads are scheduled.

In general, multithreading bugs are statistical instead of deterministic. Tracing is usually a more effective method of finding the order of execution problems than is breakpoint-based debugging.

Tracing and Debugging with DTrace

DTrace is a comprehensive dynamic tracing facility that is built into the Solaris OS. The DTrace facility can be used to examine the behavior of your multithreaded program. DTrace inserts probes into running programs to collect data at points in the execution path that you specify. The collected data can be examined to determine problem areas. See the Solaris Dynamic Tracing Guide and the DTrace User Guide for more information about using DTrace.

The Sun Developers Network web site contains several articles about DTrace, including the DTrace Quick Reference Guide.

Profiling with Performance Analyzer

The Performance Analyzer tool, included in the Sun Studio software, can be used for extensive profiling of multithreaded and single threaded programs. The tool enables you to see in detail what a thread is doing at any given point. See the Sun Studio web page and Sun Studio Information Center for more information.

Detecting Data Races and Deadlocks Using Thread Analyzer

The Sun Studio software includes a tool called the Thread Analyzer. This tool enables you to analyze the execution of a multithreaded program. It can detect multithreaded programming errors such as data races or deadlocks in code that is written using the Pthread API, the Solaris thread API, OpenMP directives, Sun parallel directives, Cray® parallel directives, or a mix of these technologies.

See the Sun Studio 12: Thread Analyzer User’s Guide.

Using `dbx`

The dbx utility is a debugger included in the Sun Studio developer tools, available from http://developers.sun.com/sunstudio/. With the Sun Studio dbx command-line debugger, you can debug and execute source programs that are written in C, C++, and Fortran. You can use dbx by starting it in a terminal window and interactively debugging your program with dbx commands. If you prefer a graphical interface, you can use the same dbx functionality in the Debugging windows of the Sun Studio IDE (Integrated Development Environment). For a description of how to start dbx, see the dbx(1) man page. See the manual Sun Studio 12: Debugging a Program With dbx for an overview of dbx. The Debugging features in the Sun Studio IDE are described in the IDE online help.

See Chapter 11, Debugging Multithreaded Applications, in Sun Studio 12: Debugging a Program With dbx for detailed information about debugging multithreaded programs. The dbx debugger provides commands to manipulate event handlers for thread events, which are described in Appendix B, Event Management, in Sun Studio 12: Debugging a Program With dbx.

All the dbx options that are listed in Table 8–1 can support multithreaded applications.

Table 8–1 dbx Options for MT Programs


Option	Action
`cont at line [-sig signo id]`	Continues execution at `line` with signal `signo`. The `id`, if present, specifies which thread or LWP to continue. The default value is `all`.
`lwp [lwpid]`	Displays current LWP. Switches to given LWP [`lwpid`].
`lwps`	Lists all LWPs in the current process.
`next ... tid`	Steps the given thread. When a function call is skipped, all LWPs are implicitly resumed for the duration of that function call. Nonactive threads cannot be stepped.
`next ... lwpid`	Steps the given LWP. Does not implicitly resume all LWPs when skipping a function. The LWP on which the given thread is active. Does not implicitly resume all LWP when skipping a function.
`step... tid`	Steps the given thread. When a function call is skipped, all LWPs are implicitly resumed for the duration of that function call. Nonactive threads cannot be stepped.
`step... lwpid`	Steps the given LWP. Does not implicitly resume all LWPs when skipping a function.
`stepi... lwpid`	Steps machine instructions (stepping into calls) in the given LWP.
`stepi... tid`	Steps machine instructions in the LWP on which the given thread is active.
`thread [ tid ]`	Displays current thread, or switches to thread `tid`. In all the following variations, omitting the l `tid` implies the current thread.
`thread -info [ tid ]`	Prints everything known about the given thread.
`thread -blocks [ tid ]`	Prints all locks held by the given thread blocking other threads.
`thread -suspend [ tid ]`	Puts the given thread into suspended state, which prevents it from running. A suspended thread displays with an “S” in the `threads` listing.
`thread -resume [ tid ]`	Unsuspends the given thread so it resumes running.
`thread -hide [ tid ]`	Hides the given or current thread. The thread does not appear in the generic `threads` listing.
`thread -unhide [ tid ]`	Unhides the given or current thread.
`thread -unhide all`	Unhides all threads.
`threads`	Prints the list of all known threads.
`threads -all`	Prints threads that are not usually printed (zombies).
`threads -mode all\|filter`	Controls whether `threads` prints all threads or filters threads by default. When filtering is on, threads that have been hidden by the `thread -hide` command are not listed.
`threads -mode auto\|manual`	Enables automatic updating of the thread listing.
`threads -mode`	Echoes the current modes. Any of the previous forms can be followed by a thread or LWP ID to get the traceback for the specified entity.

Tracing and Debugging With the TNF Utilities

Although Dtrace, Performance Analyzer, Thread Analyzer, and dbx are more modern tools, you can also still use the older TNF utilities to trace, debug, and gather performance analysis information from your applications and libraries. The TNF utilities integrate trace information from the kernel as well as from multiple user processes and threads. The TNF utilities have long been included as part of the Solaris software. See the tracing(3TNF) man page for information about these utilities.

Using `truss`

See the truss(1)man page for information on tracing system calls, signals and user-level function calls.

Using `mdb`

For information about mdb, see the Solaris Modular Debugger Guide.

The following mdb commands can be used to access the LWPs of a multithreaded program.

$l: Prints the LWP ID of the representative thread if the target is a user process.
$L: Prints the LWP IDs of each LWP in the target if the target is a user process.
pid::attach: Attaches to process # pid.
::release: Releases the previously attached process or core file. The process can subsequently be continued by prun(1) or it can be resumed by applying MDB or another debugger.

These commands to set conditional breakpoints are often useful.

[ addr ] ::bp [+/-dDestT] [-c cmd] [-n count] sym ...: Set a breakpoint at the specified locations.
addr ::delete [ id | all]: Delete the event specifiers with the given ID number.