Analyzing Indexes

Analyzing indexes requires gathering search filter usage patterns from user requests, especially for values as specified in the filters and subsequently looking the status of those values into the index files. When a search requests is processed, internal or user generated, a first phase uses indexes to find potential entries to be returned. Depending on the search filter, if the index of one of the specified attributes matches too many entries (ALLIDs), the search becomes non-indexed (notes=U in the access log). In any case, all entries thus gathered (or the entire DIT) are matched against the filter for actually returning the search result.

Directory Server tries to optimize the search so that not all index values matching ALLID will force the search to be not indexed.

To Analyze Index Filters

The dsconf analyze-index-filters command when enabled, displays the Directory Server search filters with their statistics.

1. Enable index filter analyzer using the following command:

   $ dsconf enable-index-filter-analyzer [--max-entries INT] SUFFIX_DN

   For example, to enable index filter analyzer on o=example.com, run the following command:

   $ dsconf enable-index-filter-analyzer -p port-number o=example.com $ dsconf get-suffix-prop -p port-number o=example.com \ index-filter-analyzer-enabled index-filter-analyzer-max-entries index-filter-analyzer-enabled : on index-filter-analyzer-max-entries : 2000

2. Analyze index filters to know the indexes that need to be reindexed.

   $ dsconf analyze-index-filters -p port-number SUFFIX_DN

   The output displays all the attributes that need to be reindexed, index filter usage statistics, number of hits, and number of allids hits. Based on the output, you can configure indexes differently by modifying the all-ids-threshold property or by creating indexes, to improve performance.

   $ dsconf analyze-index-filters -p port-number o=example.com Observations started at Nov 13, 2008 12:01:29 PM Total number of search requests : 8 Total number of Allids : 7 filter Type #allids/#hits Threshold Max matching Additional info entries ------------------- ---- ------------- --------- ------------ --------------- (departmentNumber eq 1/1 2 "departmentNumber" =9415) is not indexed (objectClass eq 1/1 *4000 10000 To investigate =inetOrgPerson) (objectClass=*) pres 1/1 10006 "pres" type is disabled for "objectClass" system index (roomNumber=*) pres 1/1 10000 "roomNumber" is not indexed (roomNumber=1*) eq 1/1 4071 "roomNumber" is not indexed (telephoneNumber=*) pres 2/2 *4000 10000 To investigate (telephoneNumber=1*) eq 0/1 4000 10000 ## * indicates thresholds which have been crossed. No attributes need to be reindexed Use "dsconf set-index-prop o=example.com ATTR_NAME..." to set the allids threshold value and to take benefit of indexes.

   The displayed filters are the basic filter elements along with the following information:

   • Type that is used during processing.

   • How many times the filter element is used and how many times the index value is Allids.

     ---

     Note –

     Allids can occur because of unsufficient privileges or if the index type configuration is changed. In the latter case, data may be skewed.

     Complex filters are broken down to these basic elements and will not appear in their entirety.

     ---

   The example output displays the following information:

   • Some index values are Allids

   • Some attributes are not indexed, that is, roomNumber, departmentNumber.

   • Total 4071 entries matched the roomNumber=1* filter

   • Some indexes are well configured for some filter use, that is, telephoneNumber=1*.

3. Create the new index for the attribute roomNumber.

   $ dsconf create-index -p port-number o=example.com roomNumber

4. Run the analyze-index-filters command again to check the status.

   $ dsconf analyze-index-filters -p port-number o=example.com Observations started at Nov 13, 2008 12:01:29 PM Total number of search requests : 9 Total number of Allids : 8 filter Type #allids/#hits Threshold Max matching Additional info entries -------------------- ---- ------------- --------- ------------ ----------------- (departmentNumber eq 1/1 2 "departmentNumber" =9415) is not indexed (objectClass eq 1/1 *4000 10000 To investigate =inetOrgPerson) (objectClass=*) pres 1/1 10006 "pres" type is disabled for "objectClass" system index (roomNumber=*) pres 1/1 *4000 10000 To investigate (roomNumber=1*) eq 1/1 *4000 4071 To investigate (telephoneNumber=*) pres 2/2 *4000 10000 To investigate (telephoneNumber=1*) eq 0/1 4000 10000 ## * indicates thresholds which have been crossed. Attributes to reindex : roomNumber Use "dsconf reindex --attr ATTR_NAME... o=example.com" to reindex. Use "dsconf set-index-prop o=example.com ATTR_NAME..." to set the allids threshold value and to take benefit of indexes.

   As required, follow the appropriate procedure as displayed at the end of the output.

5. Restart the index filter analyzer to consider the latest index updates.

   To restart the index filter analyzer, disable the analyzer using the disable-index-filter-analyzer subcommand and then start the analyzer again using the enable-index-filter-analyzer subcommand.

   ---

   Note –

   Monitoring affects performance. It also requires heavy memory resources, based on the configured maximum number of filters to monitor.

   Run the dsconf info command to know when the analyzer was enabled. If you do not want to monitor indexes and analyze-index-filters, it is not recommended to keep the analyzer running.

   Directory Server tries to optimize the search so that when complex filters are evaluated, not all elements could be processed. Do not expect a one-to-one relation with what appears in the access log, and a complex filter and its constituent elements.

   ---

See Also

For more information, see dsconf(1M).

To Analyze Attribute Indexes

Using the dsconf analyze-index-filters command, gather the most used filters and their behavior. On the other hand, to know the data as appears in the index files, use dsadm analyze-indexes to have a snapshot of index files.

1. Your server must be stopped before analyzing the attribute indexes.

   $ dsadm stop INSTANCE_PATH

2. Analyze attribute indexes using the following command:

   $ dsadm analyze-indexes [-bRi] [-o FILE] INSTANCE_PATH SUFFIX_DN

   For example, to analyze the attribute indexes of suffix o=example.com, run the following command:

   $ dsadm analyze-indexes /local/myinst o=example.com This operation may take a long time and generate important amounts of data Do you want to continue [y/n]? y Generating raw index data, please wait... Raw index data available in file '/local/myinst/logs/db_stat_example%2ecom' Index Type Total Keys ALLIDs 95% 90% 80% --------------- --------- ---------- ------ --- --- --- aci PRESENCE 1 0 0 0 0 ancestorid EQUALITY 6 6 0 0 0 cn EQUALITY 200000 0 0 0 0 cn SUBSTRING 14828 15 0 0 0 entrydn EQUALITY 100006 0 0 0 0 givenName EQUALITY 8605 0 0 0 0 givenName SUBSTRING 4762 4 0 0 0 givenName PRESENCE 1 1 0 0 0 mail EQUALITY 100000 0 0 0 0 mail SUBSTRING 14975 26 1 3 2 mail PRESENCE 1 1 0 0 0 nsuniqueid EQUALITY 100007 0 0 0 0 numsubordinates PRESENCE 1 0 0 0 0 objectclass EQUALITY 7 4 0 0 0 parentid EQUALITY 6 5 0 0 0 sn EQUALITY 100000 0 0 0 0 sn SUBSTRING 12993 0 0 0 0 telephoneNumber EQUALITY 99924 0 0 0 0 telephoneNumber SUBSTRING 1106 24 0 0 0 telephoneNumber PRESENCE 1 1 0 0 0 uid EQUALITY 200000 0 0 0 0 uid PRESENCE 1 1 0 0 0 aci PRESENCE ============ ALLIDs keys : 0 / 1 ancestorid EQUALITY =================== ALLIDs keys : 6 / 6 [1] [2] [3] [4] [5] [6] cn EQUALITY =========== ALLIDs keys : 0 / 200000 cn SUBSTRING ============ ALLIDs keys : 15 / 14828 [100] [101] [102] [103] [104] [105] [106] [107] [108] [109] [^us] [er1] [r10] [ser] [use] entrydn EQUALITY ================ ALLIDs keys : 0 / 100006 givenName EQUALITY ================== ALLIDs keys : 0 / 8605 givenName SUBSTRING =================== ALLIDs keys : 4 / 4762 [^ma] [ie$] [na$] [ne$] givenName PRESENCE ================== ALLIDs keys : 1 / 1 [pres] mail EQUALITY ============= ALLIDs keys : 0 / 100000 mail SUBSTRING ============== ALLIDs keys : 26 / 14975 [.co] [0@e] [1@e] [2@e] [3@e] [4@e] [5@e] [6@e] [7@e] [8@e] [9@e] [@ex] [^ma] [amp] [com] [ell] [exa] [e.c] [ie_] [le.] [mar] [mpl] [na_] [ne_] [om$] [ple] [xam] mail PRESENCE ============= ALLIDs keys : 1 / 1 [pres] nsuniqueid EQUALITY =================== ALLIDs keys : 0 / 100007 numsubordinates PRESENCE ======================== ALLIDs keys : 0 / 1 objectclass EQUALITY ==================== ALLIDs keys : 4 / 7 [inetorgperson] [organizationalperson] [person] [top] parentid EQUALITY ================= ALLIDs keys : 5 / 6 [2] [3] [4] [5] [6] sn EQUALITY =========== ALLIDs keys : 0 / 100000 sn SUBSTRING ============ ALLIDs keys : 0 / 12993 telephoneNumber EQUALITY ======================== ALLIDs keys : 0 / 99924 telephoneNumber SUBSTRING ========================= ALLIDs keys : 24 / 1106 [120] [121] [130] [140] [141] [151] [171] [180] [181] [206] [213] [303] [408] [415] [510] [714] [804] [818] [^12] [^13] [^14] [^15] [^17] [^18] telephoneNumber PRESENCE ======================== ALLIDs keys : 1 / 1 [pres] uid EQUALITY ============ ALLIDs keys : 0 / 200000 uid PRESENCE ============ ALLIDs keys : 1 / 1 [pres]

   Following what explained in Chapter 9, Directory Server Indexing, in Oracle Fusion Middleware Reference for Oracle Directory Server Enterprise Edition, dsadm analyze-indexes displays the status of the value keys as used by Directory Server. If most of the keys are Allid or 95% of Allid, the number of entries matching the key is at least equal to all-ids-threshold. The index most likely has to be configured with a higher value for all-ids-threshold.

   ---

   Note –

   Too high an all-ids-threshold value can impact performance.

   ---

   The dsadm analyze-indexes displays which keys are ALLID or close to be, so it can be matched with the output of dsconf analyze-index-filters. If a search specifies a filter whose value is an allid key, the search might not be indexed, depending on the entire search filter as mentioned above.