Previous      Contents      Index      Next
Sun Java(TM) System Directory Server 5.2 2005Q1 Administration Guide

Chapter 10
Indexing Directory Data

Like a book index, Directory Server indexes speed up searches by associating search strings with references to the directory contents. Indexes are tables of attribute values that are stored in separate database files. Indexes are created and managed independently for each suffix in the directory. Once you create an index in the suffix configuration, the server maintains the index automatically.

For an introduction to indexing, its costs and benefits, an explanation of the nsslapd-allidsthreshold attribute, and how to improve Directory Server performance, see the Directory Server Performance Tuning Guide.

This chapter contains the following sections:

Overview of Indexing
Managing Indexes
Managing Browsing Indexes

---

Overview of Indexing

Indexes are stored for each suffix in files in the corresponding database directory. Each index file contains all of the indexes defined in the suffix for a given attribute. For example, all indexes maintained for the common name (cn) attribute are stored in the databaseName_cn.db3 file.

Index files are created when you initialize a suffix or use the commands described in this chapter. During client search operations and internal operations, the server accesses the indexes to find entries in the directory faster. During modify operations, the directory updates the directory contents and maintains the index by updating the index files.

Directory Server supports the following types of indexes:

Presence index (pres) - Contains a list of the entries that contain the particular attribute, regardless of its value.
Equality index (eq) - Enables you to search efficiently for entries containing a specific attribute value.
Approximate index (approx) - Provides efficient "sounds-like" searches using the ~= filter operator. For example, the approximate index is useful for searching partial names or misspelled names. Directory Server uses a variation of the metaphone phonetic algorithm to perform searches on an approximate index.

Note	The metaphone phonetic algorithm in Directory Server supports only US-ASCII letters. Therefore, use approximate indexing only with English values.

Substring index (sub) - Provides efficient searches of attribute value substrings, such as cn=*john or cn=john*. This is a costly index to maintain because of the many possible substrings for every value.

Directory Server indexes substrings such that searches for a minimum of two-character substrings may be found in the index. A search for (sn=*ab) can therefore be accelerated using an index, for example, but a search for (sn=*a) cannot. Directory Server offers further optimization allowing initial substring searches of only one character before the wildcard. Thus a search for (sn=a*), but not (sn=*a), can also be accelerated when a substring index is available. For more information on substring indexes, see the Directory Server Performance Tuning Guide.

Matching rule index - Speeds up searches in international directories by associating the OID of a localized matching rule (also called collation order) with the attributes to be indexed.
Browsing index - Improves the response time of searches performed with the virtual list view (VLV) control. You may create a browsing index on any branchpoint in the directory tree to improve display performance of a very populated subtree, for example ou=People,dc=example,dc=com.

System Indexes

System indexes are indexes that cannot be deleted or modified. They are required for Directory Server to function properly and efficiently. The following table lists the system indexes created automatically in every suffix:

Table 10-1 System Indexes in Every Suffix 

Attribute	Eq	Pres	Purpose
aci		X	Allows the directory server to quickly obtain the access control information maintained in the directory.
ancestorid	X		Contains a list of ancestors for every entry.
entrydn	X		Speeds up entry retrieval based on DN searches.
id2entry	X		Contains the actual database of directory entries. All other database files can be recreated from this one.
nsUniqueId	X		Used to search for specific entries.
nscpEntryDN	X		Used internally in Directory Server for replication.
nsds5ReplConflict	X	X	Used to help find replication conflicts.
numsubordinates		X	Used by Directory Server Console to enhance display performance on the Directory tab.
objectClass	X		Used to help accelerate subtree searches in the directory.
parentID	X		Enhances directory performance during one-level searches.

Default Indexes

When you create a new suffix in your directory, the server configures a set of default indexes in the corresponding database directory. The default indexes can be modified depending on your indexing needs, although you should ensure that no server plug-ins or other servers in your enterprise depend on an indexed attribute before you unconfigure its index.

To modify the set of default indexes that will be used when new suffixes are created, see Modifying the Set of Default Indexes.

The following table lists the pre-configured default indexes in Directory Server:

Table 10-2 Default Indexes in Every New Suffix 

Attribute	Eq	Pres	Sub	Purpose
cn	X	X	X	Improves the performance of the most common types of user directory searches.
givenName	X	X	X	Improves the performance of the most common types of user directory searches.
mail	X	X	X	Improves the performance of the most common types of user directory searches.
mailAlternateAddress	X			Used by Sun Java System Messaging Server.
mailHost	X			Used by Sun Java System Messaging Server.
member	X			Improves Sun Java System server performance. This index is also used by the referential integrity plug-in. See Maintaining Referential Integrity for more information.
nsCalXItemId	X	X	X	Used by Sun Java System Calendar Server.
nsLIProfileName	X			Used by roaming feature of Sun Java System Messaging Server.
nsRoleDN	X			Improves the performance of role-based operations.
nswcalCALID	X			Used by Sun Java System Calendar Server.
owner	X			Improves Sun Java System server performance. This index is also used by the referential integrity plug-in. See Maintaining Referential Integrity for more information.
pipstatus	X			Used by Sun Java System Servers.
pipuid		X		Used by Sun Java System Servers.
seeAlso	X			Improves Sun Java System server performance. This index is also used by the referential integrity plug-in. See Maintaining Referential Integrity for more information.
sn	X	X	X	Improves the performance of the most common types of user directory searches.
telephoneNumber	X	X	X	Improves the performance of the most common types of user directory searches.
uid	X			Improves Sun Java System server performance.
uniquemember	X			Improves Sun Java System server performance. This index is also used by the referential integrity plug-in. See Maintaining Referential Integrity for more information.

Attribute Name Quick Reference Table

The following table lists all attributes which have a primary or real name as well as an alias. When creating indexes be sure to use the primary name.

Table 10-3 Primary Attribute Name and Their Alias 

Primary Attribute Name	Attribute Alias
authorCn	documentAuthorCommonName
authorSn	documentAuthorSurname
c	countryName
cn	commonName
co	friendlyCountryName
dc	domainComponent
dn	distinguishedName
drink	favoriteDrink
facsimileTelephoneNumber	fax
l	localityName
labeledUri	labeledUrl
mail	rfc822mailbox
mobile	mobileTelephoneNumber
o	organizationName
ou	organizationalUnitName
pager	pagerTelephoneNumber
sn	surname
st	stateOrProvinceName
street	streetAddress
ttl	timeToLive
uid	userId

---

Managing Indexes

This section describes how to create and remove presence, equality, approximate, substring and international indexes for specific attributes using Directory Server Console and the command line. See Managing Browsing Indexes for the separate procedures required for virtual list view (VLV) operations.

Note	Because indexes are specific to each suffix, you need to remember to create your new indexes in every suffix configuration. When you create new suffixes using the console, you have the option of cloning the index configuration of an existing suffix.

Before you create new indexes, balance the benefits of maintaining indexes against the costs. Keep in mind that:

Approximate indexes should not be used for attributes commonly containing numbers, such as telephone numbers, because they are not efficient.
Substring indexes do not work for binary attributes.
Equality indexes should not be used for large values such as attributes intended to contain binary data, for example jpegPhoto.
Maintaining indexes requires more resources, therefore only attributes that are commonly searched should be indexed. Entry creation will require more CPU time because the server must examine all indexed attributes and generate new entries for each one contained in the new entry.
The size of each index file is proportional to directory contents.
Attributes that are not indexed can still be specified in search requests, although the search performance will not be comparable to that of indexed searches, depending on the type of search.

Managing Indexes Using the Console

If you plan to modify or add indexes on many attributes, you should first make the suffix read-only and then export its contents to LDIF. It will then be faster to reindex the suffix by reinitializing it from the LDIF file.

On the top-level Configuration tab of Directory Server Console, expand the Data node and select the suffix you want to index. Then select the Indexes tab in the right-hand panel.

The table of System Indexes cannot be modified. Add, modify or remove indexes on attributes in the table of Additional Indexes.

To add an index on an attribute that is not yet indexed, click on the Add Attribute button. In the dialog that is displayed, select one or more attributes to index, and click OK.

The new attributes appear in the table of Additional Indexes.

To modify the indexes of an attribute, select or deselect the checkboxes for each type of index you wish to maintain for that attribute in the table of Additional Indexes.
If you want to create an index for an attribute containing values in a language other than English, enter the OID of the collation order you want to use in the Matching Rules field.

You can index the attribute using multiple languages by listing multiple OIDs separated by commas (but no whitespace). For the list of supported locales and the OID of their associated collation order, see the Directory Server Administration Reference.

To remove all indexes for an attribute, select its row in the table and click the Delete attribute button.
Click Save to save the new index configuration.

If you removed all indexes of an attribute, the server will remove the index files for that attribute and the configuration is finished. If you modified the indexes of an attribute or added a new one, proceed with the following step.

A warning dialog informs you that you must update the database files to begin using the new index. You may either reindex the suffix or reinitialize it.
If you added or modified only one or two indexes, or if your suffix must not be unavailable, you should reindex the suffix. Click on the Reindex Suffix button to display the reindexing dialog. By default, the attributes that you modified or added to the index configuration are selected. Click OK to begin reindexing these attributes. Reindexing many attributes for directories with millions of entries may take several hours, but the suffix will always be online during the reindexing.
If you added or modified indexes on several attributes, and you have a recent LDIF file exported from this suffix, click on the Initialize suffix button. In the Initialize suffix dialog enter or browse the path and name of the LDIF file and click OK. The server will reinitialize the suffix from the LDIF file an create all indexes according to the new configuration. Depending on the size of the directory, reinitializing the suffix is usually faster than reindexing two or more attribute, but the suffix is unavailable during the initialization.
If you do not reindex or reinitialize the suffix, all data will still be available but the new index will not be created and will not improve directory access performance.

If you reindex or reinitialize the suffix, the new index is immediately active for any new data that you add and any existing data in your directory. You do not have to restart your server.

Managing Indexes From the Command Line

Creating or modifying indexes from the command line involves two steps:

Using the ldapmodify command-line utility to add or modify an index configuration entry. Indexes are configured separately in each suffix, and index configuration entries are stored with the corresponding database configuration.
Running the directoryserver db2index-task command to generate the new set of indexes to be maintained by the server.

Caution	You must not delete system indexes as deleting them can significantly affect Directory Server performance. System indexes are located in the following entries: cn=index,cn=databaseName,cn=ldbm database,cn=plugins,cn=config cn=default indexes,cn=config,cn=ldbm database,  cn=plugins,cn=config Take care when deleting default indexes, because they can also affect how Directory Server works.

Creating an Index Configuration Entry

To create an index for an attribute that is not already indexed, you must create a new entry for that attribute in the configuration of the corresponding database.

Index configuration entries have the following DN:

cn=attributeName,cn=index,cn=databaseName,cn=ldbm database,
 cn=plugins,cn=config

where databaseName is the name of the database corresponding to the suffix in which you wish to create the index. For example, the following command will create presence, equality, substring, and "sounds-like" indexes for values of the sn (surname) attribute in French:

ldapmodify -a -h host -p port -D "cn=Directory Manager" -w password
dn: cn=sn,cn=index,cn=databaseName,cn=ldbm database,
 cn=plugins,cn=config
objectClass: top
objectClass: nsIndex
cn: sn
nsSystemIndex: false
nsIndexType: pres
nsIndexType: eq
nsIndexType: sub
nsIndexType: approx
nsMatchingRule: 1.3.6.1.4.1.42.2.27.9.4.76.1
^D

Index configuration entries have the nsIndex object class, and the nsSystemIndex attribute must be present and its value must be false. You cannot create a new system index. Only the existing system indexes defined internally by Directory Server will be maintained.

The values of the nsIndexType attribute list the indexes that will be maintained for the given attribute. Use any of the values shown above to define the corresponding index.

You may also use the single value none to explicitly disable indexes for the attribute, for example in order to temporarily disable indexing of the attribute. If you do not include the nsIndexType attribute in your index configuration entry, all indexes will be maintained by default.

The optional nsMatchingRule attribute contains the OID for any matching rule known by the server. This includes the OID of a language collation order for internationalized indexes, but also other matching rules such as CaseExactMatch. For the list of supported locales and the OID of their associated collation order, see the Directory Server Administration Reference.

For more information about index configuration attributes, see the Directory Server Administration Reference.

Note	You should always use the attribute's primary name (not the attribute's alias) when creating indexes. The primary name of the attribute is the first name listed for the attribute in the schema, for example uid for the userid attribute. See Table 10-3 for a list of all primary and alias attribute names.

Modifying an Index Configuration Entry

To configure the indexes that have already been defined on an attribute, modify the corresponding index entry. For example, the following command on the previously defined sn index configuration will remove the "sounds-like" index and change the language to Canadian French:

ldapmodify -h host -p port -D "cn=Directory Manager" -w password
dn: cn=sn,cn=index,cn=databaseName,cn=ldbm database,
 cn=plugins,cn=config
changetype: modify
delete: nsIndexType
nsIndexType: approx
-
replace: nsMatchingRule
nsMatchingRule: 1.3.6.1.4.1.42.2.27.9.4.78.1
^D

Running db2index-task

Once you have created an indexing entry, added additional index types to an existing indexing entry, or modified its collation order, run the directoryserver db2index-task command to generate the new indexes. This command reads the contents of the suffix and reindexes the given attribute according to its configuration entry.

While this command is running, the contents of the suffix remain available through the server, but searches will not be indexed until the command has completed. Reindexing is a resource-intensive task that may impact the performance of other operations on the server. Depending on the size of the directory, reinitializing the suffix is usually faster than reindexing two or more attributes, but a suffix is unavailable during the initialization. For more information, see Reinitializing a Suffix.

The following example regenerates the sn index in the suffix corresponding to the databaseName.

# /usr/sbin/directoryserver db2index-task
  -D "cn=Directory Manager" -w password -n databaseName -t sn

For more information, see the Directory Server Man Page Reference.

Deleting all Indexes for an Attribute

If you wish to remove all indexes configured for an attribute, you may remove its configuration entry and database file. For example, the following command will unconfigure all indexes for the sn attribute in the database named databaseName.

ldapdelete -h host -p port -D "cn=Directory Manager" -w password \
"cn=sn,cn=index,cn=databaseName,cn=ldbm database,cn=plugins, \
 cn=config"

Once you have deleted this entry, the indexes for the sn attribute will no longer be maintained in the suffix corresponding to the databaseName database. To save disk space you may also delete the corresponding index file because it will no longer be used by the server. In this example, you could delete the following file:

ServerRoot/slapd-serverID/db/databaseName/databaseName_sn.db3

Reindexing a Suffix

If your index files become corrupt, you will need to re-index the suffix to recreate the index files in the corresponding database directory. There are two ways to re-index a suffix using Directory Server Console, either reindexing or reinitialization.

Reindexing a Suffix

When you reindex a suffix, the server examines all of the entries it contains and rebuilds the index files. During reindexing, the contents of the suffix are available for read and write operations. However, the server must scan the entire suffix for every attribute that is reindexed. This may take up to several hours for suffixes with millions of entries, depending on the indexes you configure. Also, during reindexing, indexes are not available and server performance is impacted.

To reindex a suffix using the console:

On the top-level Configuration tab of Directory Server Console, expand the Data node to display the suffix you want to re-index.
Right-click the suffix configuration node and select Reindex from the pop-up menu. Alternatively, left-click the node to select it and then choose Reindex from the Object menu.

The Reindex Suffix dialog is displayed with a list of all attributes that are indexed in the chosen suffix.

Select the checkbox beside each attribute you wish to re-index. Use the Check All and Check None buttons to help you make your selection. Because all indexes for a given attribute are stored in the same database file, you must re-index all of them together.
Click OK. The console displays a confirmation message about possible unexpected search results and performance impact during the reindexing.
Click Yes to begin reindexing.

The console displays a dialog with any messages about the reindexing. Close the dialog when done.

To re-index a suffix from the command line, follow the instructions in Running db2index-task and specify all attributes for which you wish to rebuild an index file.

Reinitializing a Suffix

When you reinitialize a suffix, its contents are replaced and new index files are created as the new contents are imported. Reinitializing a suffix is usually faster than reindexing more than one attribute because all attributes are indexed in one pass as the entries are loaded. However, the suffix is unavailable while it is being reinitialized.

All of the following steps may be performed using Directory Server Console or from the command line:

Set the suffix to read-only as described in Setting Access Permissions and Referrals. You must make the suffix unwritable first so that no modifications are made after you export the contents.
Export the entire suffix to an LDIF file as described in Exporting a Single Suffix to LDIF Using the Console.
Import the same LDIF file to reinitialize the suffix, as described in Initializing a Suffix.

During the initialization, the suffix will be unavailable. When the initialization is done, all configured indexes will be ready to use.

Make the suffix writable again, as described in Setting Access Permissions and Referrals.

Modifying the Set of Default Indexes

The set of default indexes that is used when creating a new suffix is defined under the following entry:

cn=default indexes,cn=config,cn=ldbm database,cn=plugins,cn=config

Whenever you create a suffix using the console or from the command line, the default index definition entries will be copied as-is to become the initial index configuration of the corresponding database.

The set of default indexes may only be configured using the command-line utilities. The default index entries have exactly the same syntax as index configuration entries described in Managing Indexes From the Command Line. For example, use the following ldapmodify command to add a default index configuration entry:

ldapmodify -a -h host -p port -D "cn=Directory Manager" -w password
dn: cn=drink,cn=default indexes,cn=config,cn=ldbm database,
 cn=plugins,cn=config
objectClass: top
objectClass: nsIndex
cn: drink
nsSystemIndex: false
nsIndexType: eq
nsIndexType: sub
nsMatchingRule: 1.3.6.1.4.1.42.2.27.9.4.76.1
^D

After this entry has been added, any new suffix will have values of the drink attribute indexed for equality and substring searches in French.

To modify or delete the default index entries, use the ldapmodify or ldapdelete commands to edit the set of indexes in cn=default indexes,cn=config,cn=ldbm database,cn=plugins,cn=config.

---

Managing Browsing Indexes

Browsing indexes are special indexes used only for search operations that request server-side sorting or virtual list view (VLV) results. Using browsing indexes can improve the performance of searches that request server-side sorting of a large number of results. Depending on your directory configuration, the server may refuse to perform searches that request sorting when no browsing index is defined. This prevents large sorting operations from overloading server resources.

Browsing indexes apply to an entry that is the base of your searches, and you must create a separate index for every search filter you use in sorted requests. For example, if your client applications often request a sorted list of all users, you would create a browsing index on ou=People for the filter string used by your clients.

As with other indexes, there is a performance cost during update operations needed to maintain a browsing index. You should carefully plan and test the deployment of browsing indexes.

Browsing Indexes for the Console

Directory Server Console often performs searches of the entire directory to refresh the contents of it panels. If you have configured the console to sort the entries in the directory tree as described in Directory Tree View Options, you should create browsing indexes for the console.

The browsing indexes for the console are specific to the searches performed by the console. They are also created using the console. To create a browsing index for the console:

On the top-level Directory tab of Directory Server Console, browse the directory tree to display the parent of a large subtree that needs to be sorted, for example ou=People,dc=example,dc=com containing thousands of user entries.
Right-click the parent entry and select Create Browsing Index from the pop-up menu. Alternatively, left-click the entry to select it, and choose Create Browsing Index from the Object menu.

The Create Browsing Index dialog box shows you the status of the index creation. The console creates the browsing index configuration entries shown below and then generates the contents of the index file.

Click Close to close the Create Browsing Index dialog box.

The new index is immediately active for any console refresh operations and it will be maintained for any new data that you add to your directory. You do not have to restart your server.

The browsing index configuration for the console consists of the following entries. The vlvSearch entry defines the base, scope and filter of the search that will be indexed. The vlvSort attribute of the vlvIndex entry shows the attributes supported for sorting in the order in which they are sorted:

dn: cn=MCC entryDN,cn=databaseName,cn=ldbm database,
 cn=plugins,cn=config
objectClass: top
objectClass: vlvSearch
cn: MCC entryDN
vlvBase: "entryDN"
vlvScope: 1
vlvFilter: (|(objectclass=*)(objectclass=ldapsubentry))

dn: cn=by MCC entryDN, cn=MCC entryDN,cn=databaseName,
 cn=ldbm database,cn=plugins,cn=config
objectClass: top
objectClass: vlvIndex
cn: by MCC entryDN
vlvSort: cn givenname o ou sn uid

To delete a browsing index for Directory Server Console:

On the top-level Directory tab of Directory Server Console, browse the directory tree to display an entry where you have created a browsing index.
Right-click the entry and select Delete Browsing Index from the pop-up menu. Alternatively, left-click the entry to select it, and choose Delete Browsing Index from the Object menu. This menu item will only be active if the chosen entry has a Browsing Index for the console.
A Delete Browsing Index dialog box appears asking you to confirm that you want to delete the index. Click Yes to delete the browsing index.

Browsing Indexes for Client Searches

Customized browsing indexes for sorting client search results must be defined manually. Creating a browsing index, or virtual list view (VLV) index from the command line involves two steps:

Add new browsing index entries or edit existing browsing index entries using the ldapmodify utility or the Directory tab of Directory Server Console.
Run the directoryserver vlvindex command to generate the new set of browsing indexes to be maintained by the server.

Specifying the Browsing Index Entries

A browsing index is specific to a given search on a given base entry and its subtree. The browsing index configuration is defined in the database configuration of the suffix containing the entry.

Note	You cannot create browsing indexes in chained suffixes, only in local suffixes and subsuffixes.

Two entries are used to configure a browsing index. The first uses the vlvSearch object class and specifies the base, scope, and filter of the search operation whose results will be indexed. The second is a child of the first and uses the vlvIndex object class to specify the attributes to sort, and order in which to sort them.

The following example uses the ldapmodify utility to create the two browsing index configuration entries:

ldapmodify -a -h host -p port -D "cn=Directory Manager" -w password
dn: cn=Browsing ou=People, cn=databaseName,
 cn=ldbm database,cn=plugins,cn=config
objectClass: top
objectClass: vlvSearch
cn: Browsing ou=People
vlvbase: ou=People,dc=example,dc=com
vlvscope: 1
vlvfilter: (objectclass=inetOrgPerson)

dn: cn=Sort rev employeenumber, cn=Browsing ou=People,
 cn=databaseName,cn=ldbm database,cn=plugins,cn=config
objectClass: top
objectClass: vlvIndex
cn: Sort rev employeenumber
vlvSort: -employeenumber
^D

The vlvscope is either 0 for the base entry alone, 1 for the immediate children of the base, or 2 for the entire subtree rooted at the base. The vlvfilter is the same LDAP filter that will be used in the client search operations. Because all browsing index entries are located in the same place, you should use descriptive cn values to name your browsing indexes.

Each vlvSearch entry must have at least one vlvIndex entry. The vlvSort attribute is a list of attribute names that defines the attribute to sort on and the sorting order. The dash (-) in front of an attribute name indicates reverse ordering. You may define more than one index for a search by defining several vlvIndex entries. With the previous example, you could add the following entry:

ldapmodify -a -h host -p port -D "cn=Directory Manager" -w password
dn: cn=Sort sn givenname uid, cn=Browsing ou=People,
 cn=databaseName,cn=ldbm database,cn=plugins,cn=config
objectClass: top
objectClass: vlvIndex
cn: Sort sn givenname uid
vlvSort: sn givenname uid
^D

To modify a browsing index configuration, edit the corresponding vlvSearch or vlvIndex entry. To remove a browsing index so it is no longer maintained by the server, remove the individual vlvIndex entries, or if there is only one, remove both the vlvSearch entry and the vlvIndex entry. When you remove a vlvIndex entry, you may also remove the corresponding database file, for example:

ServerRoot/slapd-serverID/db/dbName/dbName_vlv#Sortsngivennameuid.db3

Running the vlvindex Command

Once you have created the browsing index entries or modified existing ones, you must run the directoryserver vlvindex command to generate the new set of browsing indexes. This command will scan the directory contents and create a database file for the browsing index.

To generate browsing indexes, use the following command:

# /usr/sbin/directoryserver vlvindex

The following example generates the browsing indexes defined in the previous section:

  # /usr/sbin/directoryserver vlvindex -n databaseName -T "Browsing ou=People"

Table 10-4 Description of vlvindex Options Used in the Example

Option	Description
-n	Specifies the name of the database containing the entries to be indexed.
-T	Specifies the value of the naming attribute of the vlvSearch entry of the corresponding browsing index. All indexes corresponding to the vlvIndex entries of the given vlvSearch entry will be generated.

For more information on the commands, see the Directory Server Man Page Reference.

Previous      Contents      Index      Next

---

Part No: 817-7613-10.   Copyright 2005 Sun Microsystems, Inc. All rights reserved.