Managing Indexes

This section describes how to manage indexes for specific attributes. The section includes information about creating, modifying, and deleting indexes. See Managing Browsing Indexes for procedures specific to virtual list view (VLV) operations.

You can also check the attributes that need to be indexed by running the dsconf info command.

To List Indexes

You can use DSCC to perform this task. For information, see Directory Service Control Center Interface and the DSCC online help.

1. To list your existing indexes and their properties, use this command:

   $ dsconf list-indexes -h host -p port -v suffix-DN

To Create Indexes

You cannot create a new system index. Only the existing system indexes defined internally by Directory Server are maintained.

You can use DSCC to perform this task. For information, see Directory Service Control Center Interface and the DSCC online help.

1. Create the new index configuration.

   Use the dsconf create-index command-line utility to configure the new index information by specifying the attribute that you want to index.

   For example, to create an index entry for the preferredLanguage attribute, use this command:

   $ dsconf create-index -h host -p port dc=example,dc=com preferredLanguage

   ---

   Note –

   The command dsconf create-index sets the index configuration, but does not actually create the index files necessary for searches. Generating the index files can affect performance. To give you more control over the indexing procedure, generating the index files is done manually after the new index configuration has been created.

   Always use the attribute’s primary name when creating indexes. Do not use the attribute’s alias. The primary name of the attribute is the first name listed for the attribute in the schema, for example, uid for the userid attribute.

   ---

2. (Optional) Set the index properties by using the dsconf set-index-prop command.

   The dsconf create-index command creates an index with default properties. If you want to modify these properties, use the dsconf set-index-prop command. For more information about modifying index properties, see To Modify Indexes.

   ---

   Note –

   When the configuration of an index is modified, re-indexing is required for the changes to take effect and to use the index again.

   ---

3. Generate the index files.

   See To Generate Indexes.

4. Repeat the previous steps for all servers that you want to be indexed.

To Modify Indexes

You can use DSCC to perform this task. For information, see Directory Service Control Center Interface and the DSCC online help.

1. Modify the index properties.

   $ dsconf set-index-prop -h host -p port suffix-DN attr-name property:value

   For example, to enable the approximate index approx-enabled for the preferredLanguage index , use the command:

   $ dsconf set-index-prop -h host -p port dc=example,dc=com \ preferredLanguage approx-enabled:on

   You can modify the following properties for each index:

   • eq-enabled equality

   • pres-enabled presence

   • sub-enabled substring

   One of the properties that you might want to modify is the optional nsMatchingRule attribute. This attribute contains the OID for any matching rule known by the server. It enables the OID of a language collation order for internationalized indexes, and other matching rules such as CaseExactMatch. For a list of supported locales and the OID of their associated collation order, see Sun Directory Server Enterprise Edition 7.0 Reference.

   For more information about index configuration attributes, see Sun Directory Server Enterprise Edition 7.0 Reference.

   ---

   Note –

   When the configuration of an index is modified, re-indexing is required for the changes to take effect and to use the index again.

   Run dsconf info to display the attributes that need to be reindexed. For example, the following output shows the cn and uid attributes that need to be reindexed.

   $ dsconf info Instance path : /local/dsInst Global State : read-write Host Name : hostname Port : port Secure port : secure-port Total entries : 160 Suffixes : dc=example,dc=com No active tasks dc=example,dc=com ================= Attribute to reindex : cn uid

   ---

2. Regenerate your new indexes.

   See To Generate Indexes.

3. Repeat the previous steps for all servers that include the modified attribute index.

To Generate Indexes

This procedure generates index files so that new or modified indexes can be searchable. If you modify an index configuration for an attribute, all searches that include that attribute as a filter are not indexed. To ensure that searches including that attribute are successful, use this procedure command to regenerate existing indexes every time you create or modify an index configuration for an attribute.

You can use DSCC to perform this task. For information, see Directory Service Control Center Interface and the DSCC online help.

1. Generate your index files in one of the following ways:

   • Generate your new index files online.

     $ dsconf reindex -h host -p port [-t attr] suffix-DN

     where -t specifies that only the specified attribute or attributes are to be reindexed, not all attributes.

     For example, to regenerate the preferredLanguage index, type:

     $ dsconf reindex -h host -p port -t preferredLanguage dc=example,dc=com

     While the dsconf reindex command is running, the contents of the suffix remain available through the server. However, searches are not indexed until the command has completed. Reindexing is a resource-intensive task that can impact the performance of other operations on the server.

   • Generate your new index files offline.

     $ dsadm reindex -t attr instance-path suffix-DN

     For example, to regenerate the preferredLanguage index, type:

     $ dsadm reindex -t preferredLanguage /local/dsInst dc=example,dc=com

   • Regenerate all of your indexes quickly offline by reinitializing your suffix.

     When you reinitialize a suffix, all index files are automatically regenerated. Depending on the size of the directory, reinitializing the suffix is usually faster than reindexing two or more attributes. However, the suffix is unavailable during the initialization. For more information, see Reindexing a Suffix by Reinitialization.

Analyzing Indexes

Analyzing indexes requires gathering search filter usage patterns from user requests, especially for values as specified in the filters and subsequently looking the status of those values into the index files. When a search requests is processed, internal or user generated, a first phase uses indexes to find potential entries to be returned. Depending on the search filter, if the index of one of the specified attributes matches too many entries (ALLIDs), the search becomes non-indexed (notes=U in the access log). In any case, all entries thus gathered (or the entire DIT) are matched against the filter for actually returning the search result.

Directory Server tries to optimize the search so that not all index values matching ALLID will force the search to be not indexed.

To Analyze Index Filters

The dsconf analyze-index-filters command when enabled, displays the Directory Server search filters with their statistics.

1. Enable index filter analyzer using the following command:

   $ dsconf enable-index-filter-analyzer [--max-entries INT] SUFFIX_DN

   For example, to enable index filter analyzer on o=example.com, run the following command:

   $ dsconf enable-index-filter-analyzer -p port-number o=example.com $ dsconf get-suffix-prop -p port-number o=example.com \ index-filter-analyzer-enabled index-filter-analyzer-max-entries index-filter-analyzer-enabled : on index-filter-analyzer-max-entries : 2000

2. Analyze index filters to know the indexes that need to be reindexed.

   $ dsconf analyze-index-filters -p port-number SUFFIX_DN

   The output displays all the attributes that need to be reindexed, index filter usage statistics, number of hits, and number of allids hits. Based on the output, you can configure indexes differently by modifying the all-ids-threshold property or by creating indexes, to improve performance.

   $ dsconf analyze-index-filters -p port-number o=example.com Observations started at Nov 13, 2008 12:01:29 PM Total number of search requests : 8 Total number of Allids : 7 filter Type #allids/#hits Threshold Max matching Additional info entries ------------------- ---- ------------- --------- ------------ --------------- (departmentNumber eq 1/1 2 "departmentNumber" =9415) is not indexed (objectClass eq 1/1 *4000 10000 To investigate =inetOrgPerson) (objectClass=*) pres 1/1 10006 "pres" type is disabled for "objectClass" system index (roomNumber=*) pres 1/1 10000 "roomNumber" is not indexed (roomNumber=1*) eq 1/1 4071 "roomNumber" is not indexed (telephoneNumber=*) pres 2/2 *4000 10000 To investigate (telephoneNumber=1*) eq 0/1 4000 10000 ## * indicates thresholds which have been crossed. No attributes need to be reindexed Use "dsconf set-index-prop o=example.com ATTR_NAME..." to set the allids threshold value and to take benefit of indexes.

   The displayed filters are the basic filter elements along with the following information:

   • Type that is used during processing.

   • How many times the filter element is used and how many times the index value is Allids.

     ---

     Note –

     Allids can occur because of unsufficient privileges or if the index type configuration is changed. In the latter case, data may be skewed.

     Complex filters are broken down to these basic elements and will not appear in their entirety.

     ---

   The example output displays the following information:

   • Some index values are Allids

   • Some attributes are not indexed, that is, roomNumber, departmentNumber.

   • Total 4071 entries matched the roomNumber=1* filter

   • Some indexes are well configured for some filter use, that is, telephoneNumber=1*.

3. Create the new index for the attribute roomNumber.

   $ dsconf create-index -p port-number o=example.com roomNumber

4. Run the analyze-index-filters command again to check the status.

   $ dsconf analyze-index-filters -p port-number o=example.com Observations started at Nov 13, 2008 12:01:29 PM Total number of search requests : 9 Total number of Allids : 8 filter Type #allids/#hits Threshold Max matching Additional info entries -------------------- ---- ------------- --------- ------------ ----------------- (departmentNumber eq 1/1 2 "departmentNumber" =9415) is not indexed (objectClass eq 1/1 *4000 10000 To investigate =inetOrgPerson) (objectClass=*) pres 1/1 10006 "pres" type is disabled for "objectClass" system index (roomNumber=*) pres 1/1 *4000 10000 To investigate (roomNumber=1*) eq 1/1 *4000 4071 To investigate (telephoneNumber=*) pres 2/2 *4000 10000 To investigate (telephoneNumber=1*) eq 0/1 4000 10000 ## * indicates thresholds which have been crossed. Attributes to reindex : roomNumber Use "dsconf reindex --attr ATTR_NAME... o=example.com" to reindex. Use "dsconf set-index-prop o=example.com ATTR_NAME..." to set the allids threshold value and to take benefit of indexes.

   As required, follow the appropriate procedure as displayed at the end of the output.

5. Restart the index filter analyzer to consider the latest index updates.

   To restart the index filter analyzer, disable the analyzer using the disable-index-filter-analyzer subcommand and then start the analyzer again using the enable-index-filter-analyzer subcommand.

   ---

   Note –

   Monitoring affects performance. It also requires heavy memory resources, based on the configured maximum number of filters to monitor.

   Run the dsconf info command to know when the analyzer was enabled. If you do not want to monitor indexes and analyze-index-filters, it is not recommended to keep the analyzer running.

   Directory Server tries to optimize the search so that when complex filters are evaluated, not all elements could be processed. Do not expect a one-to-one relation with what appears in the access log, and a complex filter and its constituent elements.

   ---

See Also

For more information, see dsconf(1M).

To Analyze Attribute Indexes

Using the dsconf analyze-index-filters command, gather the most used filters and their behavior. On the other hand, to know the data as appears in the index files, use dsadm analyze-indexes to have a snapshot of index files.

1. Your server must be stopped before analyzing the attribute indexes.

   $ dsadm stop INSTANCE_PATH

2. Analyze attribute indexes using the following command:

   $ dsadm analyze-indexes [-bRi] [-o FILE] INSTANCE_PATH SUFFIX_DN

   For example, to analyze the attribute indexes of suffix o=example.com, run the following command:

   $ dsadm analyze-indexes /local/myinst o=example.com This operation may take a long time and generate important amounts of data Do you want to continue [y/n]? y Generating raw index data, please wait... Raw index data available in file '/local/myinst/logs/db_stat_example%2ecom' Index Type Total Keys ALLIDs 95% 90% 80% --------------- --------- ---------- ------ --- --- --- aci PRESENCE 1 0 0 0 0 ancestorid EQUALITY 6 6 0 0 0 cn EQUALITY 200000 0 0 0 0 cn SUBSTRING 14828 15 0 0 0 entrydn EQUALITY 100006 0 0 0 0 givenName EQUALITY 8605 0 0 0 0 givenName SUBSTRING 4762 4 0 0 0 givenName PRESENCE 1 1 0 0 0 mail EQUALITY 100000 0 0 0 0 mail SUBSTRING 14975 26 1 3 2 mail PRESENCE 1 1 0 0 0 nsuniqueid EQUALITY 100007 0 0 0 0 numsubordinates PRESENCE 1 0 0 0 0 objectclass EQUALITY 7 4 0 0 0 parentid EQUALITY 6 5 0 0 0 sn EQUALITY 100000 0 0 0 0 sn SUBSTRING 12993 0 0 0 0 telephoneNumber EQUALITY 99924 0 0 0 0 telephoneNumber SUBSTRING 1106 24 0 0 0 telephoneNumber PRESENCE 1 1 0 0 0 uid EQUALITY 200000 0 0 0 0 uid PRESENCE 1 1 0 0 0 aci PRESENCE ============ ALLIDs keys : 0 / 1 ancestorid EQUALITY =================== ALLIDs keys : 6 / 6 [1] [2] [3] [4] [5] [6] cn EQUALITY =========== ALLIDs keys : 0 / 200000 cn SUBSTRING ============ ALLIDs keys : 15 / 14828 [100] [101] [102] [103] [104] [105] [106] [107] [108] [109] [^us] [er1] [r10] [ser] [use] entrydn EQUALITY ================ ALLIDs keys : 0 / 100006 givenName EQUALITY ================== ALLIDs keys : 0 / 8605 givenName SUBSTRING =================== ALLIDs keys : 4 / 4762 [^ma] [ie$] [na$] [ne$] givenName PRESENCE ================== ALLIDs keys : 1 / 1 [pres] mail EQUALITY ============= ALLIDs keys : 0 / 100000 mail SUBSTRING ============== ALLIDs keys : 26 / 14975 [.co] [0@e] [1@e] [2@e] [3@e] [4@e] [5@e] [6@e] [7@e] [8@e] [9@e] [@ex] [^ma] [amp] [com] [ell] [exa] [e.c] [ie_] [le.] [mar] [mpl] [na_] [ne_] [om$] [ple] [xam] mail PRESENCE ============= ALLIDs keys : 1 / 1 [pres] nsuniqueid EQUALITY =================== ALLIDs keys : 0 / 100007 numsubordinates PRESENCE ======================== ALLIDs keys : 0 / 1 objectclass EQUALITY ==================== ALLIDs keys : 4 / 7 [inetorgperson] [organizationalperson] [person] [top] parentid EQUALITY ================= ALLIDs keys : 5 / 6 [2] [3] [4] [5] [6] sn EQUALITY =========== ALLIDs keys : 0 / 100000 sn SUBSTRING ============ ALLIDs keys : 0 / 12993 telephoneNumber EQUALITY ======================== ALLIDs keys : 0 / 99924 telephoneNumber SUBSTRING ========================= ALLIDs keys : 24 / 1106 [120] [121] [130] [140] [141] [151] [171] [180] [181] [206] [213] [303] [408] [415] [510] [714] [804] [818] [^12] [^13] [^14] [^15] [^17] [^18] telephoneNumber PRESENCE ======================== ALLIDs keys : 1 / 1 [pres] uid EQUALITY ============ ALLIDs keys : 0 / 200000 uid PRESENCE ============ ALLIDs keys : 1 / 1 [pres]

   Following what explained in Chapter 9, Directory Server Indexing, in Sun Directory Server Enterprise Edition 7.0 Reference, dsadm analyze-indexes displays the status of the value keys as used by Directory Server. If most of the keys are Allid or 95% of Allid, the number of entries matching the key is at least equal to all-ids-threshold. The index most likely has to be configured with a higher value for all-ids-threshold.

   ---

   Note –

   Too high an all-ids-threshold value can impact performance.

   ---

   The dsadm analyze-indexes displays which keys are ALLID or close to be, so it can be matched with the output of dsconf analyze-index-filters. If a search specifies a filter whose value is an allid key, the search might not be indexed, depending on the entire search filter as mentioned above.

To Delete Indexes

You can use DSCC to perform this task. For information, see Directory Service Control Center Interface and the DSCC online help.

1. Remove all indexes that are configured for an attribute.

   $ dsconf delete-index -h host -p port suffix-DN attr-name

   For example, the following command deletes all indexes for the preferredLanguage attribute:

   $ dsconf delete-index -h host -p port dc=example,dc=com preferredLanguage

   Take great care when deleting default indexes because it can affect Directory Server functioning.

Changing the Index List Threshold

Slow searches might be a result of your system index list size exceeding the index list threshold. The index list threshold is the maximum number of values for each index key. To determine whether the index list threshold size has been exceeded, examine the access log. The notes=U flag at the end of an access log RESULT message indicates that an unindexed search was performed. A previous SRCH message for the same connection and operation specifies the search filter that was used. The following two-line example traces an unindexed search for cn=Smith that returns 10,000 entries. Timestamps have been removed from the messages.

conn=2 op=1 SRCH base="o=example.com" scope=0 filter="(cn=Smith)"
conn=2 op=1 RESULT err=0 tag=101 nentries=10000 notes=U

If your system often exceeds the index list threshold, consider increasing the threshold to improve performance. The following procedure uses the dsconf set-server-prop command to modify the all-ids-threshold property.

To Change the Index List Threshold

You can use DSCC to perform this task. For information, see Directory Service Control Center Interface and the DSCC online help.

1. Adjust the index list threshold.

   You can adjust your index list threshold at any of the following levels:

   • At the instance level:

     dsconf set-server-prop -h host -p port all-ids-threshold:value

   • At the suffix level:

     dsconf set-suffix-prop -h host -p port suffix-DN all-ids-threshold:value

   • At the entry level:

     dsconf set-index-prop -h host -p port suffix-DN all-ids-threshold:value

   • At the index level, by search type:

     dsconf set-index-prop -h host -p port suffix-DN all-ids-threshold search-type:value

     where search-type is one of the following:

     • eq-enabled equality

     • pres-enabled presence

     • sub-enabled substring

     The all-ids-threshold property cannot be configured for the approximate index.

   You can use DSCC to set the threshold at the index level, by search type. For more information, see the Directory Server online help.

2. Regenerate the suffix indexes.

   See To Generate Indexes.

3. If the database cache size was tuned for the old all IDs threshold value and the server has adequate physical memory, consider increasing the database cache size.

   Increase the database cache size by 25 percent of the magnitude of the increase to the all IDs threshold.

   In other words, if you increase the all IDs threshold from 4000 to 6000, you can increase the database cache size by about 12 ½ percent to account for the increase in index list size.

   Database cache size is set using the attribute dbcachesize. Find the optimum size empirically before applying changes to production servers.

Reindexing a Suffix

If your index files become corrupt, or if you change the index for an attribute, you must reindex the suffix to recreate the index files in the corresponding database directory. You can reindex a suffix while the directory server is running or by reinitializing the suffix.

Reindexing a Suffix While the Directory Server Is Running

When you reindex a suffix, the server examines all of the entries the suffix contains and rebuilds the index files. During reindexing, the contents of the suffix are read-only. Because the server must scan the entire suffix for every attribute that is reindexed, this process might take up to several hours for suffixes with millions of entries. The length of time also depends on the indexes you configure. In addition, while the suffix is being reindexed, it is not available.

To Reindex All Indexes on A Suffix

You can use DSCC to perform this task. For information, see Directory Service Control Center Interface and the DSCC online help.

1. Reindex all indexes on a suffix.

   $ dsconf reindex -h host -p port suffix-DN

   For example, to initialize all indexes on the dc=example,dc=com suffix, use this command:

   $ dsconf reindex -h host -p port dc=example,dc=com

Reindexing a Suffix by Reinitialization

When you reinitialize a suffix, the new contents are imported, which means that the suffix contents are replaced and new index files are created.

To Reindex a Suffix Through Reinitialization

You can use DSCC to perform this task. For information, see Directory Service Control Center Interface and the DSCC online help.

1. Set the suffix to read-only, as described in Setting Referrals and Making a Suffix Read-Only.

2. Export the entire suffix to an LDIF file, as described in Backing Up to LDIF.

3. Import the same LDIF file to reinitialize the suffix, as described in Importing Data From an LDIF File.

   During the initialization, the suffix is unavailable. When the initialization is complete, all configured indexes are ready to be used.

4. Make the suffix writable again, as described in Setting Referrals and Making a Suffix Read-Only.