test

Sun Directory Server Enterprise Edition 7.0 Reference

Chapter 7 Directory Server Replication

This chapter includes the following sections:

Introduction to Replication

Replication is a topology wide feature that always involves more than one participant.

Replication works as follows:

  1. A master receives a change. Once the change has been applied to the entry in the database, then because the server is a master, it stores the change in the change log database.

  2. The master updates its Replica Update Vector (RUV).

  3. The master notifies the replication threads that a new change has been recorded in the change log.

  4. These replication threads contact replication partners to propagate the information.

For example, Master 1 receives a change, applies it to the entry and updates its change log. When master 1 contacts the consumer, the consumer shows that master replica its RUV. The master looks at the RUV and compares it with its own RUV to see if it contains more recent changes than the consumer. If, for example, it sees that the consumer contains a higher RUV, it does not send changes. If it contains a more recent change, it sends another request to the consumer asking for a lock on replica ID 1 so that it can make updates. If the lock is unavailable, the update will be made later. If the lock is available, then the master can proceed to make the change.

This introduction to replication addresses the following topics:

Suppliers and Consumers

A Directory Server that replicates to other servers is called a supplier. A Directory Server that is updated by other servers is called a consumer. The supplier replays all updates on the consumer through specially designed LDAP v3 extended operations. In terms of performance, a supplier is therefore likely to be a demanding client application for the consumer.

A server can be both a supplier and a consumer, as in the following situations:

A server that plays the role of a consumer only is called a dedicated consumer.

For a master replica, the server must do the following:

The server that contains the master replica is responsible for recording any changes made to the master replica and for replicating these changes to consumers.

For a hub replica, the server must do the following:

For a consumer replica, the server must do the following:

Unit of Replication

The smallest logical unit of replication is a suffix, also known as a naming context. The term suffix arises from the way the base DN for the naming context is a suffix for all DNs in that context. For example, the suffix dc=example,dc=com contains all directory entries in the Example.com naming context.

The replication mechanism requires one suffix to correspond to one database. The unit of replication applies to both suppliers and consumers. Therefore, two suffixes on a master replica cannot be replicated to one suffix on a consumer replica, and vice versa.

Replica Identity

Master replicas require a unique replica identifier that is a 16-bit integer between 1 and 65534. Consumer and hub replicas all have the replica ID of 65535. The replica ID identifies the replica on which changes are made.

If multiple suffixes are configured on one master, you can use the same replica ID for each suffix on the master. In this way, when a change is made on that replica ID, it is possible to identify the server on which change was made.

Types of Replica

A suffix that participates in replication is called a replica. There are three kinds of replica:

A single instance of Directory Server can be configured to manage several replicas.

A replica can act as a supplier of updates, or a consumer of updates, or both.

A replica can be promoted or demoted to change its behavior with respect to other replicas. Dedicated consumers can be promoted to hubs, and hubs can be promoted to masters. Masters can be demoted to hubs, and hubs can be demoted to dedicated consumers.

A server that contains a consumer replica only is called a dedicated consumer.

Replication Agreements

Replication agreements define the relationships between a supplier and a consumer. The replication agreement is configured on the supplier. A replication agreement contains the following replication parameters:

Replication Authentication

Before a master can update a consumer, the consumer authenticates the master by using a special entry called the Replication Manager entry. The master uses the Replication Manager entry to bind to the consumer.

The Replication Manager entry has a special user profile that bypasses all access control rules defined on the consumer server. The special user profile is only valid in the context of replication.

The Replication Manager entry has the following characteristics.

The Replication Manager entry is created by default when you configure replication through the browser-based interfaceDirectory Service Control Center. You can also create your own Replication Manager entry. For information about how to create a Replication Manager entry, see Using a Non-Default Replication Manager in Sun Directory Server Enterprise Edition 7.0 Administration Guide.

Authentication can be performed in the following ways for SSL with replication.

Replication Change Log

All modifications received by a master replica are recorded in a change log. A change log is maintained on all master replicas and hub replicas.

If your application needs to read the change log, use the retro change log plug-in for backward compatibility. For more information about the retro change log plug-in, see Replication and the Retro Change Log Plug-In.

Change Sequence Number

Each change to a master replica is identified by a change sequence number, CSN. The CSN is generated by the master server and is not visible to the client application. The CSN contains the timestamp, a sequence number, the replica ID, and a subsequence number. The change log is ordered by the CSN.

Replication is sequential, meaning that entries are replicated in an orderly way. Because replication is orderly, any change generated by a master is labeled by a change sequence number (CSN) that is unique for any change inside a multi-master topology. The CSN is a hexadecimal string that appears in the logs as follows:


41e6ee93000e00640000

The first 8 hexa-digits represent the time when the change was generated in the master. The time is represented in seconds since January 1, 1970.

The next four digits are the sequence number, or the order in the current second in which the change occurred. For example, multiple changes occur in second 41e6ee93. The sequence number tells us the progressive numbering of the change.

The next four digits specify the replica ID of the master that received the change in the first place.

The last four digits are reserved. Most of the time, they are 0000.

CSNs are generated only when local traffic introduces a new change to a replica. So only masters that receive updates generate CSNs. Consumers always refer to masters, because all the updates they receive are through replication.

Replica Update Vector

The replica update vector, RUV, identifies the state of each replica in a topology. Stored on the supplier and on the consumer, the RUV is used to establish which changes need to be replicated. The RUV stores the URL of the supplier, the ID of the supplier, the minimum CSN, and the maximum CSN.

Any replica in a replication topology stores its current replication state in a replica update vector (RUV). The RUV is stored in memory by a process that is running and provides the exact knowledge this replica has of itself and every other participant in the replication topology. The RUV entry on a given server contains a line for each master participating in a replication topology. Each line contains an identifier of one of the masters, the URL of the replica, and the CSN of the first and last changes made on the server. The CSN records only the first and last changes known by the server, not necessarily the most recent changes made by the master.

The RUV is mainly in memory and can be accessed using ldapsearch on the cn=replica,cn=suffix,cn=mapping tree,cn=config entry. For example, an ldapsearch for the ou=people suffix might yield the following results:


# ldapsearch -h host1 -p 1389 -D "cn=Directory Manager" -w secret \
-b "cn=replica,cn=ou=people,cn=mapping tree,cn=config" \
-s base objectclass=* nsds50ruv

nsds50ruv: {replicageneration} 45e8296c000000010000
nsds50ruv: {replica 1 ldap://server1:1389} 45ed8751000000010000 4600f252000000010000
nsds50ruv: {replica 2 ldap://server1:2389} 45eec0e1000000020000 45f03214000000020000

For clarity, we will simplify the RUV syntax to CSNchangenumber-replicaid. The change-number shows which change the RUV corresponds to in the successive changes that occurred on the master. For example, 45ed8751000000010000 can be written as CSN05-1. In the previous illustration, master 1 contains the following RUVs:


replica 1: CSN05-1 CSN43-1
replica 2: CSN05-2 CSN40-2

The first line provides information about the first change and the last change that this replica knows about from itself, master 1, as indicated by the replica ID 1. The second line provides information about the first change and the last change that it knows about from master 2. The information that is most interesting to us is the last change. In normal operations, master 1 should know more about the updates it received than master 2. We confirm this by looking at the RUV for master 2:


replica 2: CSN05-2 CSN50-2
replica 1: CSN01-1 CSN35-1

Looking at the last change, we see that master 2 knows more about the last change it received (CSN50-2) than master 1 (which shows the last change as having occurred at CSN40-2). By contrast, master 1 knows more about its last change (CSN43-1) than master 2 (CSN35-1).

When troubleshooting problems with replication, the CSNs can be useful in identifying the problem. Master 1 should always know at least as much about its own replica ID as any other participant in the replication topology because the change was first applied on master 1 and then replicated. So, CSN43-1 should be the highest value attributed to replica ID 1 in the topology.

A problem is identified if, for example, after 30 minutes the RUV on master 1 is still CSN40-2 but on master 2 the RUV has increased significantly to CSN67-2. This indicates that replication is not happening from master 2 to master 1.

If a failure occurs and you need to reinitialize the topology while saving as much data as possible, you can use the RUV picture to determine which machine contains the most recent changes. For example, in the replication topology described previously you have a hub that contains the following RUV:


2: CSN05-2 CSN50-2
1: CSN05-1 CSN43-1

In this case, server 1 seems like a good candidate for providing the most recent changes.

RUVs can be read through nsds50ruv(5dsconf) and ds6ruv(5dsconf) attributes.

Deleted Entries: Tombstones

Directory entries deleted on one replica are maintained by Directory Server until no longer needed for replication. Such deleted entries are called tombstones, as they have objectclass: nsTombstone. In rare cases, you might need to remove tombstones manually over LDAP.

Tombstones are visible only to Directory Manager. Furthermore, tombstones show up only in a search with filter (objectclass=nsTombstone). The following ldapsearch command returns tombstone entries under dc=example,dc=com.


$ ldapsearch -D "cn=Directory Manager" -b dc=example,dc=com "(objectclass=nsTombstone)"

Consumer Initialization and Incremental Updates

During consumer initialization, or total update, all data is physically copied from a master to a consumer. When you have created a replication agreement, the consumer defined by that agreement must be initialized. When a consumer has been initialized, the master can begin to replay, or replicate, update operations to the consumer. Under normal circumstances, the consumer should not require further initialization. However, if the data on a master is restored from a backup, it might be necessary to reinitialize the consumers that depend on that master.

In a multi-master replication topology, the default behavior of a read-write replica that has been re-initialized from a backup or from an LDIF file, is to refuse client update requests. By default, the replica remains in read-only mode until it is configured to accept updates again. You set the suffix property repl-accept-client-update-enabled to on using the dsconf set-suffix-prop command when the oldest updates are on the read-only replica.

When a consumer has been initialized, replication updates are sent to the consumer when the modifications are made on the supplier. These updates are called incremental updates. A consumer can be incrementally updated by several suppliers at once, provided that the updates originate from different replica IDs.

The binary copy feature can be used to clone master replicas or consumer replicas by using the binary backup files of one server to restore another server. For information about how to use binary copy for replication, see Initializing a Replicated Suffix by Using Binary Copy in Sun Directory Server Enterprise Edition 7.0 Administration Guide.

Referrals and Replication

When a consumer receives a request to modify data, it does not forward the request to the server that contains the master replica. Instead, it returns to the client a list of the URLs of the masters that can satisfy the request. These URLs are called referrals.

The replication mechanism automatically configures consumers to return referrals for all known masters in the replication topology. However, you can also add your own referrals and overwrite the referrals set automatically by the server. The ability to control referrals helps enables you to perform the following tasks:

Directory Proxy Server is able to follow referrals.

Replication and the Retro Change Log Plug-In

The retro change log is a plug-in used by LDAP clients for maintaining application compatibility with earlier versions of Directory Server. The retro change log is stored in a separate database from the Directory Server change log, under the suffix cn=changelog.

A retro change log can be enabled on a standalone server or on each server in a replication topology. When the retro change log is enabled on a server, updates to all suffixes on that server are logged by default.

The retro changelog receives updates from all master replicas in the topology. The updates from each master replica are combined in the retro changelog. The retro changelog provides a way for applications to track changes so that they can be synchronized. Directory Server enables you to access a coherent version of the retro changelog on any master in a multi-master topology. You can also update your application to manage its state according to change numbers. This makes it possible to fail over between retro changelogs on different servers.

The global retro changelog contains all of the changes. If two changes occur on the same entry in two different locations, the retro changelog provides an ordered change description. If you query the retro changelog from any server, it will contain similar information.

For information about how to use the retro change log, see Using the Retro Change Log in Sun Directory Server Enterprise Edition 7.0 Administration Guide.

Retro Change Log and Multi-Master Replication

The following figure illustrates the retro change log on two servers in a multi-master topology.

Figure 7–1 Retro Change Log and Multi-Master Replication

Figure shows the retro change log on two servers in a multi-master topology.

The retro change log uses the following attributes during replication:

changeNumber (cN)

Identifies the order in which an update is logged to the retro change log

replicationCSN (CSN)

Identifies the time when an update is made to a given replica

replicaIdentifier (RI)

Identifies the replica that is updating the retro change log

The diagram shows that the retro change logs, RCL1 and RCL2, contain the same list of updates, but that the updates do not have the same order. However, for a given replicaIdentifier, updates are logged in the same order on each retro change log. The order in which updates are logged to the retro change log is given by the changeNumber attribute.

Failover of the Retro Change Log

The following figure illustrates a simplified replication topology where a client reads a retro change log on a consumer server.

Figure 7–2 Simplified Topology for Replication of the Retro Change Log

Figure shows a simplified replication topology where a client reads a retro change log on a consumer server.

All of the updates made to each master replica in the topology are logged to each retro change log in the topology.

The client application reads the retro change log of Directory Server 3 and stores the last CSN for each replica identifier. The last CSN for each replica identifier is given by the replicationCSN attribute.

The following figure shows the client redirecting its reads to Directory Server 2 after the failure of Directory Server 3.

Figure 7–3 Failover of the Retro Change Log

Figure shows a client redirecting its reads to Directory Server 2 after the failure of Directory Server 3.

After failover, the client application must use the retro change log (RCL2) of Directory Server 2 to manage its updates. Because the order of the updates in RCL2 is not the same as the order in RCL3, the client must synchronize its updates with RCL2.

The client examines RCL2 to identify the cN that corresponds to its record of the last CSN for each replica identifier. In the example in Failover of the Retro Change Log, the client identifies the following correspondence between last CSN and cN:

The client identifies the update corresponding to the lowest cN in this list. In the example in Failover of the Retro Change Log, the lowest cN in the list is cN4. To ensure that the client processes all updates, it must process all updates logged to RCL2 after cN4. The client does not process updates logged to RCL2 before cN4 nor does it process the update corresponding to cN4.

Replication Conflicts and the Retro Change Log

When a replication conflict occurs, Directory Server performs operations to resolve the conflict. When the retro change log is running and the changeIsReplFixupOp attribute is set to true, the following information about the operations is logged in the changeHasReplFixupOp attribute:

For more information about these attributes, see the Sun Directory Server Enterprise Edition 7.0 Man Page Reference.

Restrictions on Using the Retro Change Log

In a replicated topology, the retro change logs on replicated servers must be up-to-date with each other. This allows switchover of the retro change log. Using the example in Failover of the Retro Change Log, the last CSN for each replica ID on RCL3 must be present on RCL2.