Basic Directory Server Sizing Example: Disk and Memory Requirements (Sun Java System Directory Server Enterprise Edition 6.3 Deployment Planning Guide)

Sun Java System Directory Server Enterprise Edition 6.3 Deployment Planning Guide

Basic Directory Server Sizing Example: Disk and Memory Requirements

This section provides an example that shows initial steps in sizing Directory Server disk and memory requirements for deployment. The system used for this example was selected by chance and because it had sufficient processing power and memory to complete the sizing tasks quickly. It does not necessarily represent a recommended system for production use. You can it however to gain insight into how much memory and disk space might be required for production systems.

System Characteristics

The following system information was observed using the Solaris Management Console (smc).

2 AMD64 CPUs (2.2 gigahertz)
Solaris 10 Operating System
4 gigabytes physical memory
40 gigabytes swap
Physical memory in use before Directory Server installation: 700 megabytes
Physical memory free before Directory Server installation: 3400 megabytes
CPU usage: 1%
Local disk: one partition formatted UFS with logging

For this example, the system was dedicated to Directory Server. No other user was logged in, and only the default system processes were running.

Preparing a Directory Server Instance

After unpacking the zip distribution, install Directory Server software on local disk space.

$ ./dsee_deploy install -c DS -i /local

For convenience set environment variables as shown.

$ export PATH=/local/ds6/bin:/local/dsrk6/bin:/local/dsee6/bin:${PATH}
$ export DIRSERV_PORT=1389
$ export LDAP_ADMIN_PWF=~/.pwd

After installing the software and setting environment variables, create a Directory Server instance using default ports for LDAP and LDAPS, respectively.

$ dsadm create -p 1389 -P 1636 /local/ds
Choose the Directory Manager password:
Confirm the Directory Manager password:
$ du -hs /local/ds
610K   /local/ds

Until you create a suffix, the Directory Server instance uses less than one megabyte of disk space.

$ dsadm start /local/ds
Server started: pid=8046
$ dsconf create-suffix dc=example,dc=com
Certificate "CN=hostname, CN=1636, CN=Directory Server,
 O=Sun Microsystems" presented by the server is not trusted.
Type "Y" to accept, "y" to accept just once, "n" to refuse, "d" for more
 details: Y
$ du -hs /local/ds
53M   /local/ds

For this example, make no additional changes to the default Directory Server configuration except those shown explicitly.

Populating the Suffix With 10,000 Sample Directory Entries

Using the makeldif command with the example files provided as part of Directory Server Resource Kit, you can create sample LDIF files from one kilobyte to one megabyte in size. See To Install Directory Server Enterprise Edition 6.3 From Zip Distribution in Sun Java System Directory Server Enterprise Edition 6.3 Installation Guide for an example showing how to use the makeldif command.

The entries in these files are smaller than you would expect in a real deployment.

$ du -h /var/tmp/*
 57M   /var/tmp/100k.ldif
 5.7M   /var/tmp/10k.ldif
 573M   /var/tmp/1M.ldif

An example entry from these files is shown in the following LDIF.

dn: uid=Aartjan.Aalders,ou=People,dc=example,dc=com
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
givenName: Aartjan
sn: Aalders
cn: Aartjan Aalders
initials: AA
uid: Aartjan.Aalders
mail: Aartjan.Aalders@example.com
userPassword: trj49xeq
telephoneNumber: 935-748-6699
homePhone: 347-586-0252
pager: 906-399-8417
mobile: 452-898-9034
employeeNumber: 1000004
street: 64197 Broadway Street
l: Lawton
st: IN
postalCode: 57924
postalAddress: Aartjan Aalders$64197 Broadway Street$Lawton, IN  57924
description: This is the description for Aartjan Aalders.

Begin sizing by importing the content of 10k.ldif, which occupies 5.7 megabytes on disk.

$ dsadm stop /local/ds
Server stopped
$ dsadm import -i /local/ds /var/tmp/10k.ldif dc=example,dc=com
…

With default indexing the content of 10k.ldif increases the size of the instance files by 72 megabytes - 53 megabytes, or 19 megabytes.

$ du -hs /local/ds
 72M   /local/ds

If you index five more attributes, size increases by about seven megabytes.

$ dsconf create-index dc=example,dc=com employeeNumber street st \
 postalCode description
$ dsconf reindex dc=example,dc=com
…
## example: Finished indexing.

Task completed (slapd exit code: 0).
$ du -hs /local/ds
 79M   /local/ds

Observing memory size with the default cache settings, and nothing loaded from the suffix into entry cache yet, the server process occupies approximately 170 megabytes of memory with a heap size of about 56 megabytes.

$ dsadm start /local/ds
Server started: pid=8482
$ pmap -x 8482
…
         Address     Kbytes        RSS       Anon     Locked Mode   Mapped File
0000000000437000      61348      55632      55380          - rw---    [ heap ]
…
---------------- ---------- ---------- ---------- ----------
        total Kb     178444     172604      76532          -

As you then prime the cache and examine output from the pmap command again, the heap grows by about 10 megabytes, and so does the total size of the process.

$ ldapsearch -D cn=Directory\ Manager -w - -p 1389 -b dc=example,dc=com \
 objectclass=\* > /dev/null
Enter bind password:
$ pmap -x 8482
…
         Address     Kbytes        RSS       Anon     Locked Mode   Mapped File
…
0000000000437000      70564      65268      65024          - rw---    [ heap ]
…
---------------- ---------- ---------- ---------- ----------
        total Kb     187692     182272      86224          -

The numbers are comparable for default indexing.

$ dsconf delete-index dc=example,dc=com employeeNumber street st \
 postalCode description
$ dsconf reindex dc=example,dc=com
…
## example: Finished indexing.

Task completed (slapd exit code: 0).
$ dsadm stop /local/ds
 Server stopped
$ dsadm start /local/ds
 Server started: pid=8541
$ ldapsearch -D cn=Directory\ Manager -w - -p 1389 -b dc=example,dc=com \
 objectclass=\* > /dev/null
Enter bind password:
$ pmap -x 8541
…
         Address     Kbytes        RSS       Anon     Locked Mode   Mapped File
…
0000000000437000      70564      65248      65004          - rw---    [ heap ]
…
---------------- ---------- ---------- ---------- ----------
        total Kb     187680     182240      86192          -

For only 10,000 entries, do not change the default cache sizes.

$ dsconf get-server-prop | grep cache
db-cache-size                      :  32M
import-cache-size                  :  64M
$ dsconf get-suffix-prop dc=example,dc=com | grep entry-cache-size
entry-cache-size                   :  10M

The small default entry cache was no doubt filled completely after priming, even with only 10,000 entries. To see the size for a full entry cache, set a large entry cache size, import the data again, and prime the cache.

$ dsconf set-suffix-prop dc=example,dc=com entry-cache-size:2G
$ dsadm stop /local/ds
Server stopped
$ dsadm import -i /local/ds /var/tmp/10k.ldif dc=example,dc=com
…
$ dsadm start /local/ds
Server started: pid=8806
$ ldapsearch -D cn=Directory\ Manager -w - -p 1389 -b dc=example,dc=com \
 objectclass=\* > /dev/null
Enter bind password:
$ pmap -x 8806
…
         Address     Kbytes        RSS       Anon     Locked Mode   Mapped File
…
0000000000437000     116644     109996     109780          - rw---    [ heap ]

Here 10,000 entries occupy approximately 55 megabytes of entry cache memory (110 - 55).

Populating the Suffix With 100,000 Sample Directory Entries

As you move to 100,000 entries, you have more directory data to fit into database and entry caches. Initially, import 100,000 entries and examine the size required on disk for this volume of directory data.

$ dsadm import -i /local/ds /var/tmp/100k.ldif dc=example,dc=com
…
$ du -hs /local/ds
 196M   /local/ds

Directory data contained in the database for our example suffix, dc=example,dc=com, now occupy about 142 megabytes.

$ du -hs /local/ds/db/example/
 142M   /local/ds/db/example

You can increase the size of the database cache to hold this content. If you expect the volume of directory data to grow over time, you can set the database cache larger than currently necessary. You can also set the entry cache size larger than necessary. Entry cache grows as the server responds to client requests, unlike the database cache, which is allocated at startup.

$ dsconf set-server-prop db-cache-size:200M
$ dsconf set-suffix-prop dc=example,dc=com entry-cache-size:2G

$ dsadm stop /local/ds
 Server stopped
$ dsadm start /local/ds
 Server started: pid=8640
$ pmap -x 8640
…
         Address     Kbytes        RSS       Anon     Locked Mode   Mapped File
…
0000000000437000      61348      55404      55148          - rw---    [ heap ]
…
---------------- ---------- ---------- ---------- ----------
        total Kb     491984     485736     174620          -

This shows the server instance has a relatively small heap at startup, but that the database cache memory has been allocated. Process size is nearing half a gigabyte.

$ ldapsearch -D cn=Directory\ Manager -w - -p 1389 -b dc=example,dc=com \
 objectclass=\* > /dev/null
Enter bind password:
$ pmap -x 8640
…
         Address     Kbytes        RSS       Anon     Locked Mode   Mapped File
…
0000000000437000     610212     604064     603840          - rw---    [ heap ]
…
---------------- ---------- ---------- ---------- ----------
        total Kb    1040880    1034428     723360          -

Heap size now reflects the entry cache being filled. It has increased by roughly 550 megabytes for 100,000 small directory entries, whose LDIF occupied 57 megabytes on disk.

With five extra indexes, the process size is about the same. The database cache size has not changed.

$ dsconf create-index dc=example,dc=com employeeNumber street st \
 postalCode description
$ dsadm stop /local/ds
 Server stopped
$ dsadm import -i /local/ds /var/tmp/100k.ldif dc=example,dc=com
…
$ dsadm start /local/ds
 Server started: pid=8762
$ ldapsearch -D cn=Directory\ Manager -w - -p 1389 -b dc=example,dc=com \
 objectclass=\* > /dev/null
Enter bind password:
$ pmap -x 8762
…
         Address     Kbytes        RSS       Anon     Locked Mode   Mapped File
…
0000000000437000     610212     603832     603612          - rw---    [ heap ]
…
---------------- ---------- ---------- ---------- ----------
        total Kb    1040876    1034192     723128          -

The database is somewhat larger, however. The additional indexes increased the size of the database from 142 megabytes to 163 megabytes.

$ du -hs /local/ds/db/example/
 163M   /local/ds/db/example

Populating the Suffix With 1,000,000 Sample Directory Entries

As you move from 100,000 entries to 1,000,000 entries, you no longer have enough space on a system with 4 gigabytes of physical memory to include all entries in the entry cache. You can begin by importing the data and examining the size it occupies on disk.

$ dsadm import -i /local/ds /var/tmp/1M.ldif dc=example,dc=com
…
$ du -hs /local/ds/db/example/
 1.3G   /local/ds/db/example

Assuming you expect approximately 25% growth in directory data size during the lifetime of the instance, set the database cache size to 1700 megabytes.

$ dsadm start /local/ds
Server started: pid=9060
$ dsconf set-server-prop db-cache-size:1700M
$ dsadm stop /local/ds
Server stopped
$ dsadm start /local/ds
Server started: pid=9118
$ pmap -x 9118
…
         Address     Kbytes        RSS       Anon     Locked Mode   Mapped File
…
0000000000437000      65508      55700      55452          - rw---    [ heap ]
…
---------------- ---------- ---------- ---------- ----------
        total Kb    1882448    1034180      76616          -

Given a database cache this large and only 4 gigabytes of physical memory, you cannot fit more than a fraction of entries into the entry cache for the suffix. Here, set entry cache size to one gigabyte, and then prime the cache to see the change in the process heap size.

$ dsconf set-suffix-prop dc=example,dc=com entry-cache-size:1G
$ ldapsearch -D cn=Directory\ Manager -w - -p 1389 -b dc=example,dc=com \
 objectclass=\* > /dev/null
Enter bind password:
$ pmap -x 9118
…
         Address     Kbytes        RSS       Anon     Locked Mode   Mapped File
…
0000000000437000    1016868    1009852    1009612          - rw---    [ heap ]
…
---------------- ---------- ---------- ---------- ----------
        total Kb    2883268    2477064    1080076          -

Total process size is over 2.8 gigabytes.

$ prstat -p 9118
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
  9118 myuser   2816M 2374M sleep   59    0   0:03:26 0.5% ns-slapd/42

Extrapolating from earlier entry cache sizes, you can expect to use 5.5 or 6 gigabytes for entry cache alone if you had enough physical memory.

Examining the directory database size with five additional indexes, you find adding indexes has increased the size of the database by about 200 megabytes.

$ dsconf create-index dc=example,dc=com employeeNumber street st \
 postalCode description
$ dsadm stop /local/ds
Server stopped
$ dsadm import -i /local/ds /var/tmp/1M.ldif dc=example,dc=com
…
$ du -hs /local/ds/db/example
 1.5G   /local/ds/db/example

Summary of Observations

Table 6–3 records what was observed in this example. It includes neither server process size, nor default database cache file size.

Note –

Your observations made through empirical testing for your deployment are likely to differ significantly from those shown here.

Table 6–3 Sizing Summary


Number of Entries	LDIF File Size	Disk with Default Indexes	Disk with Five Additional Indexes	Database Cache	Entry Cache
0 [The suffix has been created, but is empty.]	n/a	n/a	n/a	n/a	n/a
10,000	5.7 megabytes	19 megabytes	26 megabytes	32 megabytes	55 megabytes
100,000	57 megabytes	142 megabytes	163 megabytes	200 megabytes	550 megabytes
1,000,000	573 megabytes	1300 megabytes	1500 megabytes	1700 megabytes (default indexing)	n/a

In an actual deployment, you may have significantly larger entries and more indexed attributes. Do your own empirical testing and tuning before ordering hardware.