Multi-target Reversal

Basic Setup

A typical multi-target reversal setup consists of the following entities:

Source - A replication source responsible for sending out replication updates and performing snapshot management. There should be exactly one source.
Potential source - A potential source is a replication target in the multi-target reversal configuration that needs to take over the job of sending out incremental replication updates (along with other tasks such as snapshot management, and so on) to all the targets. There should at least one potential source.
Dedicated target - A dedicated target is a replication target in the multi-target reversal configuration that cannot perform a reverse operation. The dedicated target needs fewer snapshots than a potential source. An appliance running an older version of software can be configured as a dedicated target in a multi-target reversal setup. Dedicated targets may or may not be present in a multi-target reversal setup.
Multi-target reversal group - A group of replication actions for the same dataset that allows performing a reverse on one of the potential sources. Following the reverse, the new source can continue sending incremental updates to all originally configured targets.

In the following figure, this example of a multi-target reversal setup consists of a source (S), three potential sources (PS1, PS2, PS3), and a dedicated target (DT1). The source is responsible to send updates to the targets. When the source goes down, one of the potential sources, in this case PS1, is chosen to be the new source.

Image of a multi-target reversal group setup

When a reverse is performed on this new source, it continues to send incremental updates to the other targets, as shown in the following figure.

Image of multi-target reversal group with a reverse operation performed on the new source

Multi-target Reversal Group Management

A multi-target group is created by configuring at least one project-level action as a potential source. In the absence of a potential source, the multi-target reversal group is not created, and reversal on any target is permitted, but the reverse will only create a replication action to replicate back to the original source.

Note:

While using multi-target reversal, multiple replication snapshots may be maintained for every action in the multi-target reversal group.

Creating a Multi-target Reversal Group

Configure a potential source.

The potential source can be configured on a project-level replication action as shown in the following BUI figure. Configuration changes to a replication action are propagated as part of a replication update. There are no guarantees that such configuration changes will be delivered to all the members of the multi-target group at the same time.
Create replication targets on the potential sources as described in Creating a Replication Target (BUI) or Creating a Replication Target (CLI).

When a reverse is performed on a potential source, actions to all original targets will be created. If there are no replication targets configured on the potential source, the newly created actions will not have replication targets. Therefore, these new actions will be unbound. When appropriate replication targets are created at later time, the unbound actions will be automatically bound to correct targets.

Note:
Because the IP address of the target is used by the auto-bind process, targets to the same appliances and the same IP addresses as used on the original source need to be created.

Viewing Unbound Actions on the Source using the CLI
1. Select the appropriate project.
2. Go under the replication sub-node of that project.
3. Use the show command to display actions as shown in the following figure. Unbound actions have an <undefined> target.
Viewing Unbound Actions on the Source using the BUI
1. Select the appropriate project.
2. Click Replication.
3. Unbound actions have an undefined target (<None>).

Monitoring Potential Targets

Replication packages on potential sources show a list of potential targets. During the reverse actions to potential targets will be created in addition to the action to replicate back to the original source. The list includes the target name, or IP address, and the package ID on the corresponding target.

Potential targets can be monitored as shown in the following CLI figure.

Image of monitoring potential targets in the CLI

Potential targets can be monitored as shown in the following BUI figure.

Image of monitoring potential targets in the BUI

Conflict Detection and Resolution

In the following figure of a multi-target reversal group, A is the source of the group.

Image of a multi-target reversal group with A as the group source

At any point of time, only one source should exist in a multi-target reversal group. A conflict can arise when more than one source may exist, as in the following scenarios.

Due to an administration error when a reverse is performed on two or more potential sources receiving updates from the same replication source.
In the event of a network segmentation, the administrator may choose to create multiple multi-target reversal segments, each with its own source. After recovering from such a network segmentation, sources from each of the multi-target reversal segments would attempt to send updates to each other, and this results in a conflict.

As shown in the following figure, there are now two sources: A1 and B.

Detecting Conflict on the Source(s) and Target(s)

As shown in the following figure, conflict is detected on the targets when two or more sources attempt to send updates to the same target. In this scenario, updates from one source will succeed, and that source will be able to continue sending successful updates to the target. Updates from other sources will fail with a specific alert, and the target will show conflict notification prompting the administrator to resolve the conflict by selecting the correct source. After the conflict has been resolved, the target will receive the update from the source that was set by the administrator. However, if the incorrect source continues to send an update, a new alert will be raised at the incorrect source, and a conflict resolution notification will again be raised at the target.
Conflict is detected when the source (node B) of one group sends an update to a source in the other group (node A1) as shown in the following figure. The update on the sending node (node B) will fail with an alert, and the source that receives the update (node A1) will show a conflict notification, prompting the administrator to resolve the conflict by selecting the correct source. After the conflict has been resolved, the incorrect source will convert its project to a package on the next update and, therefore, becomes a target.

Note:
When targets receive updates from incorrect sources, data rollback may be performed during the first update from the correct source.

Example of a CLI alert raised on the source that caused the conflict:

Example of a BUI alert raised on the source that caused the conflict:

General Steps for Resolving a Conflict

Disable updates from all sources.
Select the correct source.
On the target(s), resolve the conflict by setting the correct source using the following CLI or BUI Target procedure.
On the incorrect source(s), resolve the conflict by setting the correct source using the following CLI or BUI Source procedure.
Enable updates from the correct source.

Note:
1) Updates from other sources will cause corresponding conflict notifications to show up. 2) The conflict resolution tool does not store information persistently. If the node restarts for any reason, conflict and resolution selections will be discarded, and any updates from sources that had caused conflict before the restart will show the conflict notification again.

Target: Resolving the Conflict using the CLI

Select the appropriate replication package, go under the project, and then select replication.
Check the value of property conflict_detected. When it is true, it indicates that conflict has been detected.
Go under sub-node conflict and resolve the conflict by setting the value of property source to the correct source, as shown in the following figure.

Target: Resolving the Conflict using the BUI

Select the replication package.
Click Replication.

Conflict notification should be displayed, as shown in the following figure.
Click on the conflict notification to open the replication conflict resolution dialog box, as shown in the following figure.
Select the correct source, and click APPLY.

The updates from that source will now succeed.

Source: Resolving the Conflict using the CLI

Select the incorrectly reversed project.
Go under sub-node replication of that project.
Check the value of property conflict_detected. When it is true, it indicates that conflict has been detected.
Go under sub-node conflict and resolve the conflict by setting the value of property source to the correct source, as shown in the following figure.

From this point, the update from the selected source will cause the project to be converted into a replication package.

Source: Resolving the Conflict using the BUI

On the incorrect source, select the project created by the unintended reverse.
Click Replication.

The conflict notification should be displayed, as shown in the following figure.
Click on the conflict notification to open the replication conflict resolution dialog box, as shown in the following figure.
Select the correct source, and click APPLY.

The very first update from that source will convert the project into a replication package, and the update from that source should succeed.

Compatibility Considerations

When configuring multi-target reversal, follow these compatibility rules:

Only targets that support multi-target reversal or are running OS8.7.0 and later firmware can be members of a multi-target reversal group.
Only targets that support multi-target reversal can be configured as potential sources.
Only targets running OS8.7.0 and later firmware can be configured as dedicated targets in a multi-target reversal group.

The following failures occur due to incompatibility. This list is not exhaustive.

An error occurs when configuring a potential source on a replication action to a target that does not support multi-target reversal.
An error occurs when configuring a potential source under a dataset with existing replication actions to targets running firmware older than OS8.7.0.
An error occurs when configuring a replication action to a target running firmware older than OS8.7.0 in an existing multi-target reversal group.
Targets that are part of a multi-target reversal group and are configured as dedicated targets, but are running firmware older than OS8.8.6, can perform a replication reverse. However, they will not be able to perform a replication update back to their source. It is not possible to recover replication relations in such a scenario.

Automatic Snapshot Retention Implications

When automatic snapshot retention is used for some targets but not others in a multi-target reversal setup, after a reverse, targets that did not specify an automatic snapshot retention value keep the same number of snapshots as the new source, which might be different from what the targets had before. To avoid this situation, configure the retention policy for each target in a multi-target group as "independent." For more information, see Configuring Automatic Snapshot Retention on a Target (BUI) or Configuring Automatic Snapshot Retention on a Target (CLI).