Programming Utilities Guide

Controlling Kernel Tracing (prex)

Start prex on the kernel with the -k flag (make sure you are root). After prex successfully attaches to the kernel, it prompts you for commands.

# prex -k
prex>

Note -

Only one prex session can be attached to the kernel at any given time.


Buffer Allocation

The first step in taking a kernel trace is to allocate a kernel trace buffer. The trace buffer is circular, which means that newer data overwrites older data after the buffer fills up.

The default size for the buffer is 384 kilobytes, but you can override this when you allocate the buffer. When deciding on the buffer size, keep in mind that the buffer occupies both physical memory and kernel virtual memory, so it will impact the system you are tracing--the more RAM you have, the less the impact of measurement on your experiment, and the bigger the buffer you can allocate.

Use the buffer command in prex to allocate the buffer. For example, you could allocate a 512 kilobyte buffer as follows:

prex> buffer			      # do we have a buffer?
No trace buffer allocated 	
prex> buffer alloc 512k# allocate 512KB buffer
Buffer of size 524288 bytes allocated 	
prex> 

Note -

The minimum buffer size is 128 kilobytes; prex allocates a buffer of this size if you request anything smaller.


Selecting and Enabling Probes

Use the standard prex list command to list kernel probes and their attributes. For example, to list the name and keys attributes of the probe named pagein:

prex> list name=pagein keys
name=pagein keys=vm pageio io

To list the name and keys attributes of all probes in the io group:

prex> list name keys probes io 
name=biodone keys=io blockio
name=physio_start keys=io rawio
name=pagein keys=vm pageio io
name=pageout keys=vm pageio io
name=physio_end keys=io rawio
name=strategy keys=io blockio
prex>

The next step is to trace and enable the probes you need. For example, to select all probes whose keys specify thread, vm, and io, and prepare them for tracing:

prex> trace thread vm io    	# attach trace function to probes 	
prex> enable thread vm io	   # enable probes
prex> 

Note -

Unlike user-level probes, kernel probes are not automatically `traced' (in prex terminology) when prex attaches to the kernel. You must explicitly issue the trace and enable commands as shown above.


At this point the relevant probes are ready for tracing, but kernel tracing has not been globally enabled so no trace records are being written.

Process Filtering

If you want to trace all system activity (for example, on a busy server), you can proceed to "Enabling and Disabling Tracing". This is the most common use of kernel tracing, and usually the most informative.

However, you also have the option of restricting trace data generation to selected processes; this can reduce the amount of data you need to collect and analyze.

prex uses two abstractions to allow you to do process filtering.

Use the prex pfilter command to control process filtering.

prex> pfilter             			# are we filtering?
Process filtering is off 	
Process filter set is empty. 	
prex> pfilter add 408     		# add PID 408 to filter set 	
prex> pfilter 	
Process filtering is off 	
Process filter set is {408} 	
prex> pfilter on          		# enable process filter mode 	
prex>

Note -

System threads (such as interrupt threads) are treated as belonging to process 0.


Enabling and Disabling Tracing

The final step in starting up tracing is to globally enable kernel tracing. When you do this, the probes you have enabled write records into the kernel trace buffer as they are encountered.

prex> ktrace			        # check tracing status 	
Tracing is off 	
prex> ktrace on			     # enable kernel tracing 	
prex>

To trace the kernel behavior of your application (perhaps with user-level tracing enabled as well), start it running at this time. When your application finishes, or when you decide you have enough trace data to sample its behavior, globally disable kernel tracing.

prex> ktrace off		     # disable kernel tracing 	
prex> 

The trace buffer is still present in the kernel. Copy it out into a TNF file using tnfxtract. The procedure that you follow is explained in the subsection " Extracting Kernel Trace Data (tnfxtract)".

Resetting Kernel Tracing

After you have copied the trace buffer into a TNF file, reset kernel tracing by disabling all probes and deallocating the buffer. This is important because it brings the performance of the kernel back to where it was before you started tracing.

Use the standard prex commands to disable and untrace all probes.

prex> disable $all      # disable all probes 	
prex> untrace $all      # untrace all probes 	
prex>

Finally, use the prex buffer command to deallocate the trace buffer.


Caution - Caution -

Do not deallocate the trace buffer until you have copied it out into a trace file. Otherwise, you will lose the trace data you collected for your experiment.


prex> buffer dealloc	   # deallocate buffer 	
buffer deallocated
prex>

You can now quit prex and examine the trace data that you have collected.

prex> quit 	
#