Analyzing Memory Leaks Using the libumem Library

On Solaris systems, the libumem library is a memory agent library that tracks all of the address allocated into the process memory footprint. Usually it is not used in a production environment because it is much slower. However, it is helpful for analyzing the cause of a memory leak. For more information about the libumem library, see the technical article at the following location: http://access1.sun.com/techarticles/libumem.html

Restart the Directory Server using the following command:

# SUN_SUPPORT_SLAPD_NOSH=true LD_PRELOAD=libumem.so \
UMEM_DEBUG=contents,audit=40,guards UMEM_LOGGING=transaction ./start-slapd

The libumem library is now loaded before the Directory Server starts, instead of using SmartHeap.

Next, run the gcore command several times, once before the memory use started to grow and once after. The gcore command provides a list of addresses and pointers. Use these to read the libumem library.

# cd install-root/bin/slapd/server
gcore -o /tmp/directory-core process-id

Finally, use the mdb and splitrec tools to analyze the results. The splitrec tool compares the results to see the complete stack of the leak.

# cd install-root/bin/slapd/server
echo "::umausers -e" | mdb ./ns-slapd path_gcore1 > res.1
eacho "::umausers -e" | mdb ./ns-slapd path_gcore2 > res.2
splitrec -1 res.1 res.2

The splitrec tool is available through Sun Support. This tool provides a summary of the stacks that have been identified as responsible for leaking allocation stacks. Sun Support can use the contents of these stacks to identify known memory leaks in the SunSolve database. Sometimes the splitrec tool does not provide any output because by default it is configured to report leaks only for stacks that have been identified as leaking more than 100 times. Configure this limit to a lower value using the splitrec -l option.