Sun HPC ClusterTools 6 Software User's Guide
|
   
|
| Sun HPC ClusterTools 6 Software User's Guide
|
|
| C H A P T E R 9 |
|
Using the DTrace Utility With Sun MPI |
This chapter discusses how to use the Solaris Dynamic Tracing utility (DTrace) with Sun MPI. DTrace is a comprehensive dynamic tracing utility that you can use to monitor the behavior of applications programs as well as the operating system itself. You can use DTrace on live production systems to understand those systems' behavior and to track down any problems that might be occurring.
The D language is the programming language used to create the source code for DTrace programs.
The material in this chapter assumes knowledge of the D language and how to use DTrace.
For more information about the D language and DTrace, refer to the Solaris Dynamic Tracing Guide (Part Number 817-6223). This guide is part of the Solaris 10 OS Software Developer Collection.
Solaris 10 OS documentation can be found on the web at the following location:
http://www.sun.com/documentation
Follow these links to the Solaris Dynamic Tracing Guide:
Solaris Operating Systems -> Solaris 10 -> Solaris 10 Software Developer Collection
|
Note - The sample program mpicommleak and other sample scripts are located at:
|
The following topics are covered in this chapter:
Before you run a program under DTrace, you need to make sure that you have the correct mprun privileges.
In order to run the script under mprun, make sure that you have dtrace_proc and dtrace_user privileges. Otherwise, DTrace will return the following error because it does not have sufficient privileges:
To determine whether you have the appropriate privileges on the entire cluster, perform the following steps:
1. Use your favorite text editor to create the following shell script, called mpppriv.sh:
#!/bin/sh # mpppriv.sh - run ppriv under a shell so you can get the privileges # of the process that mprun creates ppriv $$ |
2. Type the following command:
If the output of ppriv shows that the E privilege set has the dtrace privileges, then you will be able to run dtrace under mprun (see the two examples below) Otherwise, you will need to adjust your system to get dtrace access.
The following example shows the output from ppriv if the correct user privileges have not been set:
This example shows ppriv output when the privileges have been set:
% ppriv $$ 2075: tcsh flags = <none> E:basic,dtrace_proc,dtrace_user I:basic,dtrace_proc,dtrace_user P:basic,dtrace_proc,dtrace_user L: all |
|
Note - To update your privileges, ask your system administrator to add the dtrace_user and dtrace_proc privileges to your account in the /etc/user_attr file. |
After the privileges have been changed, you can use the ppriv command to execute the dtrace commands under mprun.
There are two ways to use Dynamic Tracing with MPI programs:
For illustration purposes, assume you have a program named mpiapp. To trace the program mpiapp using the mpitrace.d script, type the following command:
The advantage of tracing an MPI program in this way is that all the processes in the job will be traced from the beginning. This method is probably most useful in doing performance measurements, when you need to start at the beginning of an application and you need all the processes in a job to participate in collecting data.
This approach also has some disadvantages. One disadvantage of running a program like the one in the above example is that all the tracing output for all four processes is directed to standard output (stdout). One way around this problem is to create a script similar to the following:
#!/bin/sh # partrace.sh - a helper script to dtrace Sun MPI jobs from the # start of the job. dtrace -s $1 -c $2 -o $2.$MP_JOBID.$SUNHPC_PROC_RANK.trace |
To trace a parallel program and get separate trace files, type the following command to run the partrace.sh shell script:
This will run mpiapp under dtrace using the mpitrace.d script. The script saves the trace output for each process in a job under a separate file name, based on the job ID and rank of the process.
The second way to use dtrace with Sun MPI is to attach dtrace to a running process. Perform the following procedure:
1. Type the following command to get the process ID (PID) of the running process and the nodes on which it is running.
% mpps -p JOBNAME NPROC UID STATE AOUT cre.1 2 joeuser RUN mpiapp RANK PID STATE NODE 0 6390 RUN mynode 1 6391 RUN mynode |
2. Decide which rank you want to use to attach dtrace, and then log in to the node that contains the rank you want to use (in this example, rank 1 for the job cre.1). In the example, you would log in to the node mynode.
3. Type the following command to attach to the rank 1 process (identified by its process ID, which is 6391 in the example) and run the DTrace script mpitrace.d:
DTrace enables you to easily trace programs. When used in conjunction with MPI and the more than 200 functions defined in the MPI standard, DTrace provides an easy way to determine which functions might be in error during the debugging process, or those functions which might be of interest. After you determine the function showing the error, it is easy to locate the desired job, process, and rank on which to run your scripts. As demonstrated above, DTrace allows you to perform these determinations while the program is running
Although the MPI standard provides the MPI profiling interface, using DTrace does provide a number of advantages. The advantages of using DTrace include the following:
The following example shows a simple script that traces the entry and exit into all the MPI API calls.
mpitrace.d: pid$target:libmpi:MPI_*:entry { printf("Entered %s...", probefunc); } pid$target:libmpi:MPI_*:return { printf("exiting, return value = %d\n", arg1); } |
When you use this example script to attach DTrace to a job that performs send and recv operations, the output looks similar to the following:
You can easily modify the mpitrace.d script to include an argument list. The resulting output resembles truss output. For example:
The mpitruss.d script shows how you can specify wildcard names to match the functions. Both probes will match all send and receive type function calls in the MPI library. The first probe shows the usage of the built-in arg variables to print out the arglist of the function being traced.
Take care when wildcarding the entrypoint and the formatting argument output, because you could end up printing either too many arguments, or not enough arguments, for certain functions. For example, in the above case, the MPI_Irecv and MPI_Isend functions will not have their Request handle parameters printed out.
The following example shows a sample output of the mpitruss.d script:
One of the biggest issues with programming is the unintentional leaking of resources (such as memory). With MPI, tracking and repairing resource leaks can be somewhat more challenging because the objects being leaked are in the middleware, and thus are not easily detected by the use of memory checkers.
DTrace helps with debugging such problems using variables, the profile provider, and a callstack function. The mpicommcheck.d script (shown in the example below) probes for all the the MPI communicator calls that allocate and deallocate communicators, and keeps track of the stack each time the function is called. Every 10 seconds the script dumps out the current count of MPI communicator calls and the total calls for the allocation and deallocation of communicators. When the dtrace session ends (usually by typing Ctrl-C, if you attached to a running MPI program), the script will print out the totals and all the different stack traces, as well as the number of times those stack traces were reached.
In order to perform these tasks, the script uses DTrace features such as variables, associative arrays, built-in functions (count, ustack) and the predefined variable probefunc.
The following example shows the mpicommcheck.d script.
This script attaches dtrace to a suspect section of code in your program (that is, a section of code that might contain a resource leak). If, during the process of running the script, you see that the printed totals for allocations and deallocations are starting to steadily diverge, you might have a resource leak. Depending on how your program is designed, it might take some time and observation of the allocation/deallocation totals in order to definitively determine that the code contains a resource leak. Once you do determine that a resource leak is definitely occurring, you can type Ctrl-C to break out of the dtrace session. Next, using the stack traces dumped, you can try to determine where the issue might be occurring.
The following example shows code containing a resource leak, and the output that is displayed using the mpicommcheck.d script.
The sample MPI program containing the resource leak is called mpicommleak. This program performs three MPI_Comm_dup operations and two MPI_Comm_free operations. The program thus "leaks" one communicator operation with each iteration of a loop.
When you attach dtrace to mpicommleak using the mpicommcheck.d script above, you will see a 10-second periodic output. This output shows that the count of the allocated communicators is growing faster than the count of deallocations.
When you finally end the dtrace session by typing Ctrl-C, the session will have output a total of five stack traces, showing the distinct three MPI_Comm_dup and two MPI_Comm_free call stacks, as well as the number of times each call stack was encountered.
| Sun HPC ClusterTools 6 Software User's Guide
|
819-4131-10 |
|
Copyright © 2006, Sun Microsystems, Inc. All Rights Reserved.