It is important to do the following before collecting data about your memory leak:
Disable any custom plug-ins
Reduce the cache setting to very low values
Enable the audit log
Once you have done the above, run the prstat -L utility and check the VSZ column if it grows.
Collect the generic Directory Server data, as described in Collecting Generic Data. This data includes the version of Directory Server that you are running, logs from the test run, in particular the audit log, and the Directory Server configuration file.
With the data you collected, you can now contact the Sun Support Center for assistance with your problem.