|Sun ONE Directory Server 5.2 Installation and Tuning Guide|
Chapter 6 Tuning Cache Sizes
Directory Server caches directory information in memory and on disk in order to be able to respond more quickly to client requests. Properly tuned caching minimizes the need to access disk subsystems when handling client requests.
Unless caches are tuned and working properly, other tuning may have only limited impact on performance.
Types of Cache
Directory Server handles three types of cache as described in Table 6-1.
Table 6-1    Caches
Each Directory Server instance has one database cache that holds both indexes and entries in database format.
Each suffix has an entry cache that holds entries retrieved from the database during previous operations and formatted for quick delivery to client applications.
Each Directory Server instance has an import cache that is structurally similar to the database cache and is used during bulk loading.
Directory Server also benefits from file system cache, handled by the underlying operating system, and from I/O buffers in disk subsystems.
Figure 6-1 shows caches for an instance of Directory Server handling three suffixes, each with its own entry cache. The instance is configured to handle significant disk activity, placing the transaction log, database, and other files and logs on separate disk subsystems as suggested in Chapter 8 "Tuning Logging."
Figure 6-1    Entry and Database Caches in Context
Each Directory Server instance has one database cache. The database cache holds pages from the database containing indexes and entries. Each page is not an entry, but a slice of memory containing a portion of the database. You specify database cache size (nsslapd-dbcachesize). The change to database cache size takes effect after you restart the server, with database cache space allocated at server startup.
Directory Server moves pages between the database files and the database cache to maintain maximum database cache size. The actual amount of memory used by Directory Server for database cache may be up to 25 percent larger than the size you specify, due to additional memory needed to manage the database cache itself.
When using a very large database cache, verify through empirical testing and by monitoring memory use with tools such as pmap(1) on Solaris systems that the memory used by Directory Server does not exceed the size of available physical memory. Exceeding available physical memory causes the system to start paging repeatedly, resulting in severe performance degradation.
The ps(1) utility, present on the UNIX platforms Directory Server supports, can also be used with the -p pid and -o format options to view current memory used by a particular process such as Directory Server (ns-slapd). On Windows systems, the Task Manager Processes tab page lists memory usage per process (slapd.exe). Refer to the operating system documentation for details.
For 32-bit servers, database cache size must in practice be limited to 2 GB or less.
On Windows and AIX platforms, do not allocate more than 1 GB (1,073,741,824 bytes) for database cache.
Refer to the Sun ONE Directory Server Reference Manual for further details concerning the valid range of nsslapd-dbcachesize values.
The entry cache holds recently accessed entries, formatted for delivery to client applications. You specify entry cache size for a suffix (nsslapd-cachememsize) and the maximum number of entries (nsslapd-cachesize). Entry cache is allocated as needed.
Directory Server can return entries from an entry cache extremely efficiently, as entries stored in this cache are already formatted. Entries in the database are stored as a raw string of bytes, and must be formatted (and stored in the entry cache) before delivery to client applications.
When specifying entry cache size, know that nsslapd-cachememsize indicates how much memory Directory Server requests from the underlying memory allocation library. Depending on how the memory allocation library handles such requests, actual memory used may be much larger than the effective amount of memory ultimately available to Directory Server for the entry cache.
Actual memory used by the Directory Server process depends primarily on the memory allocation library used, and on the entries cached. Entries with many small attribute values usually require more overhead than entries with a few large attribute values.
For 32-bit servers, entry cache size must in practice be limited to 2 GB or less.
On AIX platforms, Directory Server is created with maxdata = 0x50000000, allowing you to allocate 1 GB each to both database cache and entry cache. Contact your Sun ONE support representative if you must change the value of maxdata.
Refer to the Sun ONE Directory Server Reference Manual for further details concerning the valid range of nsslapd-cachememsize and nsslapd-cachesize values.
The import cache is created and used during suffix initialization only, also known as bulk loading or importing. If the deployment involves offline suffix initialization only, import cache and database cache are not used together, so you need not add them together when aggregating cache size as described in "Total Aggregate Cache Size." You specify import cache size (nsslapd-import-cachesize). Changes to import cache size take effect the next time the suffix is reset and initialized, with import cache allocated for the initialization, then released after the initialization.
Directory Server handles import cache as it handles database cache. Ensure therefore that sufficient physical memory is available to prevent swapping.
For 32-bit servers, import cache size must in practice be limited to 2 GB or less. Refer to the Sun ONE Directory Server Reference Manual for further details concerning the valid range of nsslapd-import-cachesize values.
File System Cache
The operating system allocates available memory not used by Directory Server caches and other applications to the file system cache. This cache holds data recently read from the disk, making it possible for subsequent requests to obtain data copied from cache rather than to read it again from the disk. As memory access is many times faster than disk access, leaving some physical memory available to the file system cache can boost performance.
Refer to the operating system documentation for details on file system cache.
Total Aggregate Cache Size
The sum of all caches used simultaneously must remain smaller than the total size of available physical memory, less the memory intended for file system cache. For 32-bit servers, this means total aggregate cache size must in practice be limited to 2 GB or less. Total cache used may well be significantly larger than the size you specify. Refer to "Database Cache" for hints on how to check that the cache size and thus Directory Server process size does not exceed available physical memory.
On Windows platforms, the maximum address space available to an application is 2 GB. If the total aggregate cache size exceeds this limit, Directory Server exits with an error message.
If suffixes are initialized (bulk loaded) while Directory Server is online, the sum of database, entry, and import cache sizes should remain smaller than the total size of available physical memory.
Table 6-2    Suffix Initialization (Import) Operations and Cache Use
If all suffix initialization takes place offline with Directory Server stopped, you may be able to work around this limitation. In this case import cache does not coexist with database cache, so you may allocate the same memory to import cache for offline suffix initialization and to database cache for online use. If you opt to implement this special case, however, ensure that no one performs online bulk loads on the production system. The sum of the caches used simultaneously must still remain smaller than the total size of available physical memory.
How Searches Use Cache
Figure 6-2 illustrates how Directory Server handles both searches specifying a base DN and searches using filters. Individual lines represent threads accessing different levels of memory, with broken lines representing steps to minimize through effective tuning.
Figure 6-2    Searches and Cache
Base Search Process
As shown, base searches (those specifying a base DN) are the simplest type of searches for Directory Server to handle. To process such searches, Directory Server:
- Attempts to retrieve the entry having the specified base DN from the entry cache.
If the entry is found there, Directory Server checks whether the candidate entry matches the filter provided for the search.
If the entry matches, Directory Server then quickly returns the formatted, cached entry to the client application.
- Attempts to retrieve the entry from the database cache.
If the entry is found there, Directory Server copies the entry to the entry cache for the suffix, and then proceeds as if the entry had been found in the entry cache.
- Attempts to retrieve the entry from the database itself.
If the entry is found there, Directory Server copies the entry to the database cache, then proceeds as if the entry had been found in the database cache.
Subtree and One-Level Search Process
Also as shown in Figure 6-2, searches on a subtree or a level of a tree involve additional processing to handle sets of entries. To process such searches, Directory Server:
- Attempts to build a set of candidate entries that match the filter from indexes in the database cache.
If no appropriate index is present, the set of candidate entries must be generated from the relevant entries in the database itself.
- Handles each candidate entry by:
- Performing a base search to retrieve the entry.
- Checking whether the entry matches the filter provided for the search.
- Returning the entry to the client application if the entry matches the filter.
In this way, Directory Server avoids constructing the set in memory.
Ideally, you know what searches to expect before tuning Directory Server. In practice, verify assumptions through empirical testing.
How Updates Use Cache
Figure 6-3 illustrates how Directory Server handles updates. Individual lines represent threads accessing different levels of memory, with broken lines representing steps to minimize through effective tuning.
Figure 6-3    Updates and Cache
Updates involve more processing than searches. To process updates, Directory Server:
- Performs a base DN search to retrieve the entry to update or verify in the case of an add operation that it does not already exist.
- Changes the database cache, updating in particular any indexes affected by the update.
If the data affected by the update has not been loaded into the database cache, this step can result in disk activity while the relevant data are loaded into the cache.
- Writes information about the changes to the transaction log, waiting for the information to be flushed to disk.
Refer to "Transaction Logging" for details.
- Formats and copies the updated entry to the entry cache for the suffix.
- Returns an acknowledgement of successful update to the client application.
How Suffix Initialization Uses Cache
Figure 6-4 illustrates how Directory Server handles suffix initialization, also known as bulk load import. Individual lines represent threads accessing different levels of memory, with broken lines representing steps to minimize through effective tuning.
Figure 6-4    Suffix Initialization (Bulk Loading) and Cache
To initialize a suffix, Directory Server:
- Starts a thread to feed an entry cache, used as a buffer, from LDIF.
- Starts a thread for each index affected and a thread to create entries in the import cache. These threads consume entries fed into the entry cache.
- Reads from and writes to the database files when import cache runs out.
Directory Server may also write log messages during suffix initialization, but does not write to the transaction log.
Tools for suffix initialization such as ldif2db (/usr/sbin/directoryserver ldif2db) delivered with Directory Server provide feedback concerning cache hit rate and import throughput. Having both cache hit rate and import throughput drop together suggests that import cache may be too small. Consider increasing import cache size.
Optimizing For Searches
For top performance, cache as much directory data as possible in memory. In preventing the directory from reading information from disk, you limit the disk I/O bottleneck. There are a number of different possibilities for doing this, depending on the size of your directory tree, the amount of memory available and the hardware used. Depending on the deployment, you may choose to allocate more or less memory to entry and database caches to optimize search performance. You may alternatively choose to distribute searches across Directory Server consumers on different servers.
All Entries and Indexes in Memory
Imagine the optimum case. Database and entry caches fit into the physical memory available. The entry cache is large enough to hold all entries in the directory. The database cache is large enough to hold at minimum all indexes. In this case, searches find everything in cache. Directory Server never has to go to file system cache or to disk to retrieve entries.
In this case, ensure that database cache can contain all database indexes even after updates. When space runs out in the database cache for indexes, Directory Server must read indexes from disk for every search request, severely impacting throughput. Directory Server Console displays hit ratios and other useful information under the Status tab as shown in Figure 6-5.
Figure 6-5    Monitoring Cache Hit Rate Using Directory Server Console
Alternatively, paging and cache activity can be monitored by searching from the command line:
$ ldapsearch -D admin -w password \
-b cn=monitor,cn=database_name,cn=ldbm database,cn=plugins,cn=config
As a rough estimate of the amount of memory needed for all the database indexes held in .db3 files to fit in the database cache, use the following formula. This formula is approximately accurate for default index configuration used with typical entries having no large binary attributes such as photos.
nsslapd-dbcachesize = 1.2 x SUMall .db3 files(file size)
As a rough estimate of the number of entry cache slots and the amount of memory needed for all entries to fit in the entry cache, use the following formulas. Again these formulas are approximately accurate for default index configuration used with typical entries having no large binary attributes such as photos.
nsslapd-cachesize = 4.5 x (number of entries in LDIF)
nsslapd-cachememsize = 3.8 x (id2entry.db3 file size)
Verify and correct estimates through empirical testing. Entry caches in particular may use much more memory than you allocate to them.
Plenty of Memory, 32-Bit Directory Server
Imagine a system with sufficient memory to hold all data in entry and database caches, but no support for a 64-bit Directory Server process. If hardware constraints prevent you from deploying on a Solaris system, for example, the key is to size caches appropriately with respect to memory limitations for 32-bit processes, then to leave available memory to the file system cache.
File system cache is shared with other processes on the system, especially file based operations. It is thus considerably more difficult to control than other caches, particularly on systems not dedicated to Directory Server.
The system may reallocate file system cache to other processes.
Less Memory, Some File System Cache
Imagine a system with insufficient available memory to hold all data in entry and database caches, but yet with still significant available memory. The key in this case is to avoid causing combined entry and database cache sizes to exceed the available physical memory, resulting in heavy virtual memory paging that could bring the system to a virtual halt.
Consider leaving available memory to the file system cache, setting entry cache and database cache sizes to low figures such as 500 KB. By doing this, you may permit the system to retain enough data in the file system cache to end searches there, before Directory Server must repeatedly read entries and indexes from the disk.
Alternatively, if search patterns are less random, you may choose to set entry and database caches higher on the assumption that most of the searches in the particular deployment access the same small subset of all entries in the directory, and that the gains from having entries and indexes cached for such searches offset the cost of handling occasional unusual search requests. Verify and correct assumptions through empirical testing.
Low Memory, Low File System Cache
Imagine a system with insufficient available memory both for holding data in entry and database caches, but also for allowing the system to cache data in the file system cache. The key in this case is to make the most of the available memory you have.
Consider setting entry and database cache sizes as low as possible, leaving as much memory as possible to the file system cache. Leaving memory to the file system cache at least prevents entries from being expanded into database or entry by a factor of 3 to 4.5, theoretically limiting the disk I/O activity. Verify this assumption through empirical testing for the particular deployment.
Optimizing for Updates
For top update performance, first remove any transaction log bottlenecks observed. Refer to "Transaction Logging" for details.
Next, attempt to provide enough memory for the database cache to handle updates in memory and minimize disk activity. You can monitor the effectiveness of the database cache by reading the hit ratio in Directory Server Console. Directory Server Console displays hit ratios for suffixes under the Status tab as shown in Figure 6-5.
Attempt as well to leave significant memory available for the file system cache. After Directory Server has run for some time, the file system cache should contain enough entries and indexes that disk reads are no longer necessary. Updates should affect the database cache in memory, with data from the large database cache in memory being flushed only infrequently.
Flushing data to disk can itself be a bottleneck, so storing the database on a separate RAID system such as a Sun StorEdge disk array can help improve update performance. You may use utilities such as iostat(1M) on Solaris systems to isolate potential I/O bottlenecks. Refer to Windows help for details on handling I/O bottlenecks on Windows systems.
Cache Priming and Monitoring
Priming caches means filling them with data such that subsequent Directory Server behavior reflects normal operational performance, rather than ramp up. Prime caches before measuring and analyzing potential optimizations.
Prime the entry cache for the suffix using the ldapsearch utility. For example:
$ ldapsearch -D directoryManager -w password -b suffix objectclass=\* > /dev/null
Perform searches to prime the database cache and in particular to load indexes into the cache. You can prime presence indexes by performing searches with filters such as (mail=*). For other indexes, consider using the Sun ONE Directory Server Resource Kit searchrate utility applying filter formats to search for all possible values of each attribute to index. In other words, to check performance for equality searches against mail attributes, for example, generate a file with one mail address per line for each mail address, then use the searchrate utility to perform searches using the file. For example:
$ searchrate -b suffix -f "(mail=%s)" -i mail.file -K -t 10
Consider using -K and -t to save time. When used with the -K option, searchrate holds the connection open, binding only once, not binding for each search. The -t option lets you specify how many threads to use. Refer to the Sun ONE Directory Server Resource Kit documentation for details concerning the searchrate utility. The Sun ONE Directory Server Resource Kit can be obtained as described in "Downloading Directory Server Tools".
After other caches are primed, you can prime available file system cache. Although you cannot guarantee information in the file system cache is not flushed, priming file system cache nevertheless may improve ramp up time. To prime files system cache on UNIX systems you may use the dd(1M) command as super user. On Solaris systems with database files in the default location for example:
# for db in ServerRoot/slapd-serverID/db/*/*.db3
> dd if=`pwd`/$db of=/dev/null bs=512k
0+1 records in
0+1 records out
After caches are primed, you may run tests, and monitor whether cache tuning has produced the desired outcomes. Directory Server Console displays monitoring information for caches when you select the Suffixes node under the Status tab as shown in Figure 6-5. Alternatively, paging and cache activity can be monitored by searching from the command line:
$ ldapsearch -D admin -w password \
-b cn=monitor,cn=database_name,cn=ldbm\ database,cn=plugins,cn=config
If database cache size is large enough and the cache is primed, then the hit ratio (dbcachehitratio) should be high, and number of pages read in (dbcachepagein) and clean pages written out (dbcacheroevict) should be low. Here, "high" and "low" must be understood relative to the deployment constraints.
If entry cache for a suffix is large enough and the cache is primed, then the hit ratio (entrycachehitratio) should be high. The entry cache size (currententrycachesize) should be no more than 80 percent of the maximum size (maxentrycachesize). Finally, the size in entries (currententrycachecount) should be either equal or very close to the total number of entries in the suffix.
Tuning cache sizes represent only one approach to improving search, update or bulk load rates. As you tune the cache, performance bottlenecks from cache move to other parts of the system. Refer to the other chapters in this guide for more information.