Chapter 22 Directory Proxy Server Distribution

Directory Proxy Server enables distribution through the definition of data views. Data views are defined with a view base, which determines the base DN of the entries in that data view. Based on the distribution algorithms provided in Directory Proxy Server, you can specify how entries are divided among the different data views.

For an overview of Directory Proxy Server distribution and a description of example use cases, see Chapter 17, Directory Proxy Server Distribution, in Sun Java System Directory Server Enterprise Edition 6.3 Reference.

This chapter covers the following topics:

• Configuring Directory Proxy Server Distribution Algorithms

• Configuring Directory Proxy Server for Distribution of Suffix Data

• Creating and Configuring Data Views for Example Use Cases

Configuring Directory Proxy Server Distribution Algorithms

Directory Proxy Server provides the following distribution algorithms:

• Pattern matching

• Numeric

• Lexicographic

• Replication

• Custom

Configuring Pattern Matching Distribution Algorithm

Directory Proxy Server distributes the requests to data views based on the match between the parameters of the requests and one or more patterns. Set the following parameters to configure the Pattern matching distribution algorithm:

• pattern-matching-base-object-search-filter pattern-matching-base-object-search-filter(5dpconf)

• pattern-matching-dn-regular-expression pattern-matching-dn-regular-expression(5dpconf)

• pattern-matching-one-level-search-filter pattern-matching-one-level-search-filter(5dpconf)

• pattern-matching-subtree-search-filter pattern-matching-subtree-search-filter(5dpconf)

The configuration attributes that end with filter are LDAP filters, not regular expressions. These LDAP filters are evaluated against LDAP filters contained in the incoming search requests.

For example, use the following settings to configure the Pattern Matching distribution algorithm to send the requests for the users with even uid to even data view and the users with odd uid to odd data view.

$ dpconf set-ldap-data-view-prop even
pattern-matching-base-object-search-filter:'|(uid=\2a)(uid=*0)(uid=*2)\
(uid=*4)(uid=*6)(uid=*8))'\
pattern-matching-one-level-search-filter:'|(uid=\2a)(uid=*0)(uid=*2)\
(uid=*4)(uid=*6)(uid=*8))'\
pattern-matching-subtree-search-filter:'|(uid=\2a)(uid=*0)(uid=*2)\
(uid=*4)(uid=*6)(uid=*8))'\
pattern-matching-dn-regular-expression:'uid=[0-9]+[02468]'
distribution-algorithm: pattern-matching 

$ dpconf set-ldap-data-view-prop odd
pattern-matching-base-object-search-filter:'|(uid=\2a)(uid=*1)(uid=*3)\
(uid=*5)(uid=*7)(uid=*9))'\
pattern-matching-one-level-search-filter:'|(uid=\2a)(uid=*1)(uid=*3)\
(uid=*5)(uid=*7)(uid=*9))'\
pattern-matching-subtree-search-filter:'|(uid=\2a)(uid=*1)(uid=*3)\
(uid=*5)(uid=*7)(uid=*9))'\
pattern-matching-dn-regular-expression:'uid=[0-9]+[13579]'
distribution-algorithm: pattern-matching 

In the (uid=\2a) expression, the \2a is an ASCII representation of * where 2 and a are two hexadecimal digits. The (uid=\2a) expression makes sure that the data view accepts the requests for all uids.

The syntax supported by the pattern matching algorithm is specified by the Java Pattern class (documented at http://java.sun.com/j2se/1.4.2/docs/api/java/util/regex/Pattern.html). This syntax is not the same as the usual regex syntax.

Configuring Numeric Distribution Algorithm

Directory Proxy Server distributes the requests to data views according to the numeric value of the RDN in the request. The numeric value is taken from the value of the first RDN beneath the base DN of the data view. Set the following parameters define the Numeric bounds:

• numeric-attrs numeric-attrs(5dpconf)

• numeric-default-data-view numeric-default-data-view(5dpconf)

• numeric-lower-bound numeric-lower-bound(5dpconf)

• numeric-upper-bound numeric-upper-bound(5dpconf)

For example, to configure the numeric distribution algorithm to send the requests for uid between 0 to 99 to a specific data view. Use the same syntax for the rest of the users but with a different data view.

$ dpconf set-ldap-data-view-prop dataview distribution-algorithm:numeric \
 numeric-attrs:uid numeric-lower-bound:0 numeric-upper-bound:99

Configuring Lexicographic Distribution Algorithm

Directory Proxy Server distributes the requests to data views according to the lexicographic value of the RDN in the request. Lexicographic bounds are taken from the value of the first RDN beneath the base DN of the data view. Set the following parameters to define the Lexicographic bounds:

• lexicographic-attrs lexicographic-attrs(5dpconf)

• lexicographic-lower-bound lexicographic-lower-bound(5dpconf)

• lexicographic-upper-bound lexicographic-upper-bound(5dpconf)

For example, to configure the Lexicographic distribution algorithm to send the requests of the users whose name starts between A to M to one data view and the requests for the rest of the users to another data view.

$ dpconf set-ldap-data-view-prop dataview distribution-algorithm:lexicographic \
 lexicographic-attrs:cn lexicographic-lower-bound:A lexicographic-upper-bound:M

Configuring Replication Distribution Algorithm

Directory Proxy Server distributes the requests to data views according to the role of the data view in replication. The algorithm distributes write operations to all data sources in the data source pool and read operations to a single data source. The replication role is defined by the replication-role parameter. A data view can have a master role or a consumer role.

$ dpconf set-ldap-data-view-prop dataview distribution-algorithm:replication

Configuring Custom Distribution Algorithm

Custom distribution algorithm can be configured for all types of data views, that is, ldap-data-view, jdbc-data-view, ldif-data-view, and join-data-view. In the following procedure the algorithm is set only for ldap-data-view.

To Configure Custom Distribution Algorithm

1. Set the extension-jar-file-url property to contain the path of the Java Archive (JAR) file containing your distribution algorithm class.

   $ dpconf set-server-prop -h host -p port extension-jar-file-url:jar file path

   The jar file path can be replaced with a valid JAR file path such as file:/expt/dps/custom_plugin/myjar.jar.

2. Before you configure custom-distribution-algorithm, set distribution-algorithm to none.

   $ dpconf set-ldap-data-view-prop view name distribution-algorithm:none

3. Set the custom-distribution-algorithm property to your custom distribution algorithm class.

   $ dpconf set-ldap-data-view-prop view name custom-distribution-algorithm:PackageName.AlgoClassName

Configuring Directory Proxy Server for Distribution of Suffix Data

A simple case of data distribution consists of storing entries having UIDs beginning with A through M in one set of directories, and storing entries having UIDs beginning with N through Z in another set of directories. Directory Proxy Server receives all client operations. The server must determine which set of directories handles A through M, and which set handles N through Z.

The key stages in configuring Directory Proxy Server to handle this data distributions scenario are as follows.

1. Add directories as data sources for Directory Proxy Server.

2. Add the data sources to data source pools to handle the different data distributions.

3. Create data views designed to distribute client requests to the appropriate data pools.

4. Split the LDIF to be loaded into the appropriate data sources.

5. Import the split LDIF into the appropriate data sources.

6. Adjust the operation based weights for the data sources attached to the appropriate data pools.

The following example involves Directory Proxy Server, listening on port 9389. To keep the example simple, the proxy is configured here to distribute as described across only three Directory Server instances. For availability and read scalability, use replicated directory topologies to store LDAP data. One Directory Server instance, dsA-M:1389 handles the user entries having UIDs beginning with A through M. Another Directory Server instance, dsN-Z:2389, handles the user entries having UIDs beginning with N through Z. A final directory instance handles the base entries of the suffix, dsBase:3389.

The first step creates and enables the data sources. The base data source holds entries near the root of the suffix that do not have UIDs. In a typical deployment, these entries would be much fewer in number than distributed entries.

$ dpconf create-ldap-data-source -p 9389 dsA-M localhost:1389
$ dpconf set-ldap-data-source-prop -p 9389 dsA-M is-enabled:true

$ dpconf create-ldap-data-source -p 9389 dsN-Z localhost:2389
$ dpconf set-ldap-data-source-prop -p 9389 dsN-Z is-enabled:true

$ dpconf create-ldap-data-source -p 9389 dsBase localhost:3389
$ dpconf set-ldap-data-source-prop -p 9389 dsBase is-enabled:true

The second step adds the data sources to a data source pool.

$ dpconf create-ldap-data-source-pool -p 9389 "Base Pool"
$ dpconf attach-ldap-data-source -p 9389 "Base Pool" dsBase

$ dpconf create-ldap-data-source-pool -p 9389 "A-M Pool"
$ dpconf attach-ldap-data-source -p 9389 "A-M Pool" dsA-M

$ dpconf create-ldap-data-source-pool -p 9389 "N-Z Pool"
$ dpconf attach-ldap-data-source -p 9389 "N-Z Pool" dsN-Z

The third step creates data views designed to distribute client requests to the appropriate data pools. Notice how the base pool handles dc=example,dc=com, whereas the pools holding data distributed according to UID values handle ou=people,dc=example,dc=com. This step requires a server restart.

$ dpconf create-ldap-data-view -p 9389 "Base View" "Base Pool" \
dc=example,dc=com

$ dpconf create-ldap-data-view -p 9389 "A-M View" "A-M Pool" \
ou=people,dc=example,dc=com
$ dpconf set-ldap-data-view-prop -p 9389 "A-M View" \
distribution-algorithm:lexicographic lexicographic-attrs:uid \
lexicographic-lower-bound:a lexicographic-upper-bound:m
The proxy server will need to be restarted in order for the changes to take effect

$ dpconf create-ldap-data-view -p 9389 "N-Z View" "N-Z Pool" \
ou=people,dc=example,dc=com
$ dpconf set-ldap-data-view-prop -p 9389 "N-Z View" \
distribution-algorithm:lexicographic lexicographic-attrs:uid \
lexicographic-lower-bound:n lexicographic-upper-bound:z
The proxy server will need to be restarted in order for the changes to take effect
$ dpadm restart /local/dps

The fourth step splits the LDIF to be loaded into the appropriate data sources. This example uses both the dsadm split-ldif command to perform the initial split, and also some file editing to retain the top entry in all the data sources. This makes it possible both to retain the top entry that specifies access control instructions, and to use a single import command for each data source.

$ dpadm split-ldif /local/dps /local/ds6/ldif/Example.ldif /tmp/
[14/May/2007:21:14:13 +0200] - STARTUP    - INFO  - Java Version: 1.5.0_09
 (Java Home: /local/jre)
[14/May/2007:21:14:13 +0200] - STARTUP    - INFO  - Java Heap Space: Total Memory
 (-Xms) = 3MB,
 Max Memory (-Xmx) = 63MB
[14/May/2007:21:14:13 +0200] - STARTUP    - INFO  - Operating System: SunOS/sparc 5.10
[14/May/2007:21:14:15 +0200] - INTERNAL   - ERROR - Entry starting at line 0 does not
 start with a DN
[14/May/2007:21:14:15 +0200] - INTERNAL   - ERROR - Unable to parse line "# Kirsten is
 a Directory Administrator and therefore should not" of entry "uid=kvaughan, ou=People,
 dc=example,dc=com" starting at line 112 as an attribute/value pair -- no colon found.
[14/May/2007:21:14:15 +0200] - INTERNAL   - ERROR - Unable to parse line "# Robert is
 a Directory Administrator and therefore should not" of entry "uid=rdaugherty,
 ou=People, dc=example,dc=com" starting at line 298 as an attribute/value pair --
 no colon found.
[14/May/2007:21:14:16 +0200] - INTERNAL   - ERROR - Unable to parse line "# Harry is
 a Directory Administrator and therefore should not" of entry "uid=hmiller, ou=People,
 dc=example,dc=com" starting at line 556 as an attribute/value pair -- no colon found.
[14/May/2007:21:14:16 +0200] - INTERNAL   - INFO  - SplitLDIF processing complete.
  Processed 156 entries.
$ ls /tmp/*ldif
/tmp/a-m view.ldif    /tmp/base view.ldif   /tmp/n-z view.ldif

This step also requires a top entry that is added to the LDIF before import.

$ cp /local/ds6/ldif/Example.ldif /tmp/top.ldif
$ vi /tmp/top.ldif 
$ cat /tmp/top.ldif  
dn: dc=example,dc=com
objectclass: top
objectclass: domain
dc: example
aci: (target ="ldap:///dc=example,dc=com")(targetattr !=
 "userPassword")(version 3.0;acl "Anonymous read-search access";
 allow (read, search, compare)(userdn = "ldap:///anyone");)
aci: (target="ldap:///dc=example,dc=com") (targetattr =
  "*")(version 3.0; acl "allow all Admin group"; allow(all) groupdn =
  "ldap:///cn=Directory Administrators,ou=Groups,dc=example,dc=com";)

$ cat /tmp/top.ldif /tmp/base\ view.ldif > /tmp/top\ and\ base\ view.ldif
$ cat /tmp/top.ldif /tmp/a-m\ view.ldif > /tmp/top\ and\ a-m\ view.ldif
$ cat /tmp/top.ldif /tmp/n-z\ view.ldif > /tmp/top\ and\ n-z\ view.ldif

The fifth step imports the split LDIF into the appropriate data sources. Here, the directory handling the base entries is on port 3389. The directory handling A-M is listening on port 1389. The directory handling N-Z is listening on port 2389.

$ dsconf import -p 1389 /tmp/top\ and\ a-m\ view.ldif dc=example,dc=com
...
Task completed (slapd exit code: 0).

$ dsconf import -p 2389 /tmp/top\ and\ n-z\ view.ldif dc=example,dc=com
...
Task completed (slapd exit code: 0).
$ dsconf import -p 3389 /tmp/top\ and\ base\ view.ldif dc=example,dc=com
...
Task completed (slapd exit code: 0).

The sixth step adjusts the operation based weights for the data sources attached to the appropriate data pools. If client applications perform operations other than searches, then weights must be set for those operations as well.

$ dpconf set-attached-ldap-data-source-prop -p 9389 "Base Pool" dsBase search-weight:1
$ dpconf set-attached-ldap-data-source-prop -p 9389 "A-M Pool" dsA-M search-weight:1
$ dpconf set-attached-ldap-data-source-prop -p 9389 "N-Z Pool" dsN-Z search-weight:1

After the operations based weights are set, client applications can search through Directory Proxy Server as if the data were not physically distributed.

The following search looks for a user whose UID begins with R.

$ ldapsearch -p 9389 -b dc=example,dc=com uid=rfisher
version: 1
dn: uid=rfisher, ou=People, dc=example,dc=com
cn: Randy Fisher
sn: Fisher
givenName: Randy
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
ou: Human Resources
ou: People
l: Cupertino
uid: rfisher
mail: rfisher@example.com
telephoneNumber: +1 408 555 1506
facsimileTelephoneNumber: +1 408 555 1992
roomNumber: 1579

The next search looks for one of the base entries.

$ ldapsearch -p 9389 -b ou=groups,dc=example,dc=com cn=hr\ managers
version: 1
dn: cn=HR Managers,ou=groups,dc=example,dc=com
objectClass: top
objectClass: groupOfUniqueNames
cn: HR Managers
ou: groups
uniqueMember: uid=kvaughan, ou=People, dc=example,dc=com
uniqueMember: uid=cschmith, ou=People, dc=example,dc=com
description: People who can manage HR entries

Creating and Configuring Data Views for Example Use Cases

This section contains the following information about data views and how to create and configure them:

• Data Views That Provide a Single Point of Access When Different Parts of a Subtree Are Stored in Different Data Sources

• Data Views With Hierarchy and a Distribution Algorithm

The examples in this section assume that the connection handler allows all client connections to be processed by Directory Proxy Server.

Data Views That Provide a Single Point of Access When Different Parts of a Subtree Are Stored in Different Data Sources

This section describes how to configure a data view that provides a single point of access to different parts of a subtree. This example contains two data views with the same base DN. A numeric distribution algorithm is used to separate entries into different data views. A data source pool is configured for each set of data-equivalent data sources. The following figure shows the example deployment.

For information about this type of deployment, see Data Views to Route Requests When Different Parts of a Subtree Are Stored in Different Data Sources in Sun Java System Directory Server Enterprise Edition 6.3 Reference.

Figure 22–1 Sample Deployment That Provides a Single Point of Access When Different Parts of a Subtree Are Stored in Different Data Sources

To Configure Data Views That Provide a Single Point of Access When Different Parts of a Subtree Are Stored in Different Data Sources

You can use DSCC to perform this task. For information, see Directory Service Control Center Interface and the DSCC online help.

1. Create a data source for each LDAP server as described in Creating and Configuring LDAP Data Sources.

2. Create two data source pools as described in Creating and Configuring LDAP Data Source Pools.

3. Attach the data sources that contain one part of the subtree to data-source-pool-1, and the data sources that contain the other part of the subtree to data-source-pool-2, as described in Attaching LDAP Data Sources to a Data Source Pool.

4. (Optional) Configure load balancing.

   For information, see Configuring Load Balancing.

5. Create a data view with a distribution algorithm to select entries in ou=people,dc=example,dc=com with uid between 0 and 99, and configure the data view to direct requests to data-source-pool-1.

   $ dpconf set-ldap-data-view-prop -h host1 -p 1389 dataview-1 \ ldap-data-source-pool:data-source-pool-1 base-dn:ou=people,dc=example,dc=com \ distribution-algorithm :numeric numeric-attrs:uid numeric-lower-bound :0 \ numeric-upper-bound :99

6. Create another data view with a distribution algorithm to select entries in ou=people,dc=example,dc=com with uid between 100 and 199, and configure the data view to direct requests to data-source-pool-2.

   $ dpconf set-ldap-data-view-prop -h host1 -p 1389 dataview-2 \ ldap-data-source-pool:data-source-pool-2 base-dn:ou=people,dc=example,dc=com \ distribution-algorithm:numeric numeric-attrs:uid numeric-lower-bound:100 numeric-upper-bound :199

   The other properties of the data views are the same as the default data view in Default Data View.

7. If necessary, restart the instance of Directory Proxy Server for the changes to take effect.

   For information about restarting Directory Proxy Server, see To Restart Directory Proxy Server.

Data Views With Hierarchy and a Distribution Algorithm

This section describes how to configure a data view to combine hierarchy with distribution algorithms. For information about this type of deployment, see Data Views With Hierarchy and a Distribution Algorithm in Sun Java System Directory Server Enterprise Edition 6.3 Reference.

The example in this section contains four data views. The base DN of data view 1 is superior to the base DNs of the other data views. Data view 3 and data view 4 have the same base DN, but a numeric distribution algorithm separates the entries into different data views.

Directory Proxy Server automatically excludes a subordinate branch of a subtree from a data view when the subordinate branch is configured as the base DN of a separate data view. A numeric distribution algorithm separates entries from the same subtree into different data views. A data source pool is configured for each set of data-equivalent data sources.

The following figure shows the example deployment.

Figure 22–2 Sample Data View With Hierarchy and a Distribution Algorithm

To Configure Data Views With Hierarchy and a Distribution Algorithm

You cannot use DSCC to perform this task. Use the command line, as described in this procedure.

1. Create a data source for each LDAP server as described in Creating and Configuring LDAP Data Sources.

2. Create four data source pools as described in Creating and Configuring LDAP Data Source Pools.

3. Attach the data sources to the data source pools by following the instructions in Attaching LDAP Data Sources to a Data Source Pool.

   • Attach the data sources that contain dc=example,dc=com to data-source-pool-1.

   • Attach the data sources that contain ou=computer,dc=example,dc=com to data-source-pool-2.

   • Attach the data sources that contain entries in ou=people,dc=example,dc=com with uid between 0 and 99 to data-source-pool-3.

   • Attach the data sources that contain entries in ou=people,dc=example,dc=com with uid between 100 and 199 to data-source-pool-4.

4. (Optional) Configure load balancing.

   For information, see Configuring Load Balancing.

5. Create a data view with a base DN at dc=example,dc=com, that refers to data-source-pool-1.

   $ dpconf create-ldap-data-view -h host1 -p 1389 dataview-1 \ data-source-pool-1 dc=example,dc=com

6. Create a data view with a base DN at ou=computer,dc=example,dc=com that refers to data-source-pool-2.

   $ dpconf create-ldap-data-view -h host1 -p 1389 dataview-2 \ data-source-pool-2 ou=computer,dc=example,dc=com

7. Create a data view with a base DN at ou=people,dc=example,dc=com that refers to data-source-pool-3. Configure a distribution algorithm on the data view to select entries with uid between 0 and 99.

   $ dpconf create-ldap-data-view -h host1 -p 1389 dataview-3 \ data-source-pool-3 ou=people,dc=example,dc=com $ dpconf set-ldap-data-view-prop dataview-3 distribution-algorithm:numeric \ numeric-attrs:uid numeric-lower-bound:0 numeric-upper-bound:99

8. Create a data view with a base DN at ou=people,dc=example,dc=com that refers to data-source-pool-4, and configure a distribution algorithm on the data view to select entries with uid between 100 and 199.

   $ dpconf create-ldap-data-view -h host1 -p 1389 dataview-4 \ data-source-pool-4 ou=people,dc=example,dc=com $ dpconf set-ldap-data-view-prop dataview-4 distribution-algorithm:numeric \ numeric-attrs:uid numeric-lower-bound:100 numeric-upper-bound:199

9. Verify that the subtrees ou=computer,dc=example, dc=com and ou=people,dc=example, dc=com have been excluded from dataview-1 by looking at the excluded-subtrees parameter.

   $ dpconf get-ldap-data-view-prop -h host1 -p 1389 dataview-1 excluded-subtrees

   The list of excluded subtrees is returned.

10. Restart the instance of Directory Proxy Server for the changes to take effect.

    For information about restarting Directory Proxy Server, see To Restart Directory Proxy Server.