1. Oracle ZFS Storage Appliance Administration Guide, Release OS8.8.x
  2. Remote Replication
  3. Remote Replication Concepts
  4. Cascaded Replication

Cascaded Replication

Cascaded replication allows replication of replicated data by configuring replication actions on replication packages. Multiple replication actions can be created on the same package in order to replicate the package to multiple target appliances.

Basic Setup

A basic cascaded replication setup consist of a source, one or more intermediate targets, and a final target.

The following example of cascaded replication consists of source node A that has an action set up on one of its projects to replicate it to a package on node B. Node B in turn has an action created to replicate this package to another package on node C, and node C has an action set up on this package to replicate it to a package on node D. Thus, node B and node C cascade a replica from node A to node D, as shown in the following figure.


Image of cascaded replication example configuration

As shown in the following figure, source node A has an action set up on one of its projects to replicate it to a package on node B. Node B in turn has actions created to replicate this package to node C and node D. Thus, node B cascades a replica from node A to nodes C and D.


Image of cascaded replication with node B cascading a replica from node A to nodes C and D

Replication reverse in a cascaded configuration is only supported on the nodes receiving replication updates directly from the source (node B in the previous example) and the reverse is disabled on any other nodes (nodes C and D). Multi-target reversal should be considered when planning for disaster recovery in a cascaded replication setup. See Multi-target Reversal and "Using the distant_target Property for Managing Reversal" in the following sections.

Note:

Replication reversal is not supported to a filesystem with mandatory file retention. For information about the file retention feature, see File Retention Management.

Cascaded Replication Management

Managing cascaded replication consists of two tasks:

  • Creating an action under the replication package

  • Configuring cascaded replication schedules

Creating an Action under the Replication Package

Similar to creating a replication action under a project, a replication action can be created under a replication package. The following figure shows how to create an action under a replication package using the BUI.

Note that cascaded replication is not supported for share-level replication. Actions cannot be configured under shares in replication packages, and actions cannot be configured under packages created as result of share-level replication.


Image of creating an action under a replication package using the BUI

The following figure shows how to create an action under a replication package using the CLI.


Image of creating an action under a replication package using the CLI

Configuring Cascaded Replication Schedules

Every action has two schedules:

  • A schedule on an action that is configured as part of a project is called a source schedule.

  • A schedule on an action that is configured as part of a package is called a cascading schedule.

The following figure shows how to configure a new cascading schedule on a replication action using the CLI.


Image of configuring a new cascading schedule on a replication action using the CLI

The following figure shows how to configure a new cascading schedule on a replication action using the BUI.


Image of configuring a new cascading schedule on a replication action using the BUI

Both schedules can be configured at any time, but only one schedule is effective, depending where the action is configured. If there is a change in replication topology due to replication reverse or conversion, the effective schedule may change. For example, after performing a reverse on a replication package, actions configured in the reversed package are now configured in the project, and so their source schedules become active. Similarly, after performing a replication conversion on project actions that used to be configured under the project are now configured in the package, and so their cascading schedules are active.

The following figure shows how to view the effective schedule using the CLI.


Image of how to view the effective schedule using the CLI

The following figure shows how to view the effective schedule using the BUI. The leftmost schedule tab on an action is the effective schedule.


Image of how to view the effective schedule using the BUI

Both source and cascading schedules are preserved when new actions are created during replication reverse. See Reverse the Direction of Replication and Multi-target Reversal.

The after_update option is available for cascading schedules. When set, it will initiate the update only after an incoming update has completed.

The update_cascade_delay property can be used to specify the delay between the completion of an incoming replication update and the initiation of the outgoing cascading update. This property is effective only when the after_update option is selected.

The after_update option and the update_cascade_delay property can be configured as shown in the following figure.


Image of setting options in the CLI

Note that the continuous schedule option is not available for actions under packages, and cascading replication actions do not create new replication snapshots. When a new update is delivered from the source, cascading actions use the latest package snapshot to perform replication updates to its targets. Therefore, performing an update on a cascading action without first receiving an update from the source will have no effect; the successful update status will be set and the last_try property will be updated.

Using the distant_target Property for Managing Reversal

In cascaded replication, a distant target relationship between two nodes differentiates between local and distant targets. The distant_target property preserves this relationship after topology transformations as result of performing replication reverse or conversion.

The following configuration scenarios illustrate the distant_target property:

  1. When the distant_target property is not set and there is no multi-target reversal set up. In this case, when a reverse is performed on a target, not all actions will be preserved. For more information, see Multi-target Reversal. As shown in the following figure, neither Nodes A nor B have a distant target relationship, nor Node B is configured as a potential source (action AB: potential_source=false, distant_target=false).


    Image of distant_target property not set and no multi-target reversal set up

    When A fails and a reverse on B is performed, no actions to A{1..3} are created, as shown in the following figure.


    Image of A fails and a reverse on B is performed

  2. When the distant_target property is not set and there is multi-target reversal set up. In this case, when a reverse is performed on a potential source, it creates actions to all targets in the multi-target reversal group. There is no distant target relationship and, therefore, there is no differentiation between local and distant targets. Node A is the source and Node B is the potential source, as shown in the following figure. Nodes A and B have no distant target relationship (action AB: potential_source=true, distant_target=false).


    Image of distant_target property not set and there is a multi-target reversal set up

    However, when Node A fails and Node B is the new source after a reverse is performed, actions to A{1..3} are created, as shown in the following figure.


    Image of node A0 fails and node B0 is new source after a reverse

  3. When the distant_target property is set and there is a multi-target reversal set up. In this case, when a reverse is performed on a potential source, it does not create actions to the targets that are local to the distant source. Node A is the source and Node B is the potential source, as shown in the following figure. Nodes A and B have a distant target relationship (action AB: potential_source=true, distant_target=true).


    Image of distant_target property is set and there is a multi-target reversal set up

    When Node A fails and after a reverse on Node B, Node B becomes the new source. However, because of the distant target relationship between A and B, actions to A{1..3} are not created. After A recovers, it will cascade its package to A{1..3}.


    Image of node A fails after a reverse on node B

The following figure shows how to configure a distant target using the CLI.


Image of how to configure a distant target using the CLI

The following figure shows how to configure a distant target using the BUI.


Image of how to configure a distant target using the BUI

It is recommend to configure multi-target reversal by setting a potential source when using cascaded replication; otherwise, it may not be possible to return to the original cascaded configuration after multiple replication reversals. Also, the distant target can only be set when a potential source is set. For more information, see Multi-target Reversal.

Failure Scenarios

Failure of a single node in a cascaded replication can impact multiple appliances or even multiple data centers connected to each other by replication relations. Although there is no limitation on the cascaded replication topology, the following typical failure scenarios are presented:

  • Original source failure

  • Intermediate target failure

  • Source site failure

 

Original Source Failure

In this scenario, the source of the cascading chain fails. Recovery from this failure requires an initial multi-target reversal configuration on the source. The process is initiated on a target by performing a replication reverse. The reversed target becomes the source and continues to send incremental updates to all other original targets and to the original source. For further details, see Multi-target Reversal.

As shown in the following figure, A is the source and A1 is the potential source. A is replicated to A1 and B. B, in turn, replicates to C and D, and source A fails.


Image of source A failing

As shown in the following figure, after a reverse is performed on A1, it replicates to A (when A recovers) and to B which, in turn, continues to send updates to C and D.


Image of a reverse performed on A1

 

Intermediate Target Failure

This failure scenario pertains to the failure of an intermediate node in a cascaded chain. Recovery from this scenario can be achieved by following the retargeting procedure. As shown in the following figure, intermediate node B in the cascaded chain fails.


Image of intermediate target failure

As shown in the following figure, after the retarget/bypass procedure is applied, source A bypasses failed node B and continues sending updates to node C, which, in turn, sends an update to node D.


Image of source A bypassing failed node B and continuing to send updates to node C, then D

As shown in the following figure, when B recovers, action between A and B is restored, while action between B and C remains bypassed.


Image of action between A and B is restored; action between B and C remains bypassed

 

Bypassing a failed node using the CLI:

  1. Select the action to the failed node on the source.

  2. Enter retarget.

  3. Set the retarget_mode property to bypass.

  4. Set the target to be the bypass target, as shown in the following figure.


    Image of setting the target to be the bypass target in the CLI

  5. The bypassed_id property shown in the bypassing action can be used to select the original action to perform a retarget restore after the failed node has recovered. The bypassed_id generated can be viewed, as shown in the following figure.


    Image of viewing the bypassed_id property in the CLI

  6. At a later stage when B has recovered, use the retarget/restore procedure to get back to the original cascaded topology, as shown in the following figure.


    Image of the restore procedure in the CLI

 

Bypassing a failed node using the BUI:

  1. On the source node, select the action to the failed node.

  2. In the Edit Replication Action dialog box, click Retarget.

  3. In the Replication Retarget dialog box, set the mode to Bypass.

  4. Select the target from the drop-down list to be the bypass target, as shown in the following figure.


    Image of selecting the target from the drop-down list to be the bypass target in the BUI

  5. Click APPLY.

  6. At a later stage when B has recovered, use the restore procedure to get back to the original cascaded topology, as shown in the following figure.


    Image of the restore procedure in the BUI

 

Source Site Failure

In this scenario, the source and all its direct local targets are not available, and are replaced by a new source at one of the distant targets. Recovery from this scenario requires multi-target reversal configuration on the source. For further details, see Multi-target Reversal. It also requires some targets to be configured as distant targets. For details, refer to "Using the distant_target Property for Managing Reversal."

As shown in the following figure, A0, B0 and C0 form a multi-target reversal group, where B0 and C0 are potential sources and distant targets. A0,B0 and A0,C0 form a distant target relation (A0B0 and A0C0 should be configured as potential source and distant target), while A1,A2,A3 form a local relation with A0, and B1,B2,B3 form a local relation with B0.


Image of source and all its direct local targets are not available, and are replaced by a new source at one of the distant targets

In the case when a reverse is performed on B0, as shown in the following figure, it treats A1,A2,A3 as local targets of A0 and hence does not recreate actions to A1,A2,A3. Due to the properties of multi-target reversal, B0 sends updates to C0.


Image of reverse performed and source site failure

Compatibility Considerations

When configuring cascaded replication, follow these compatibility rules:

  • Nodes running firmware older than OS8.7.0 cannot be configured as part of cascaded replication.

  • Nodes running firmware later than OS8.7.0 can be configured as final nodes.

  • Final nodes running firmware that does not support cascaded replication may perform replication reverse. However, they will not be able to perform a replication update back to their source. It is not possible to recover replication relations in such a scenario.

  • The retarget cannot be performed to final nodes running firmware that does not support cascaded replication.

  • Replication compatibility rules for targets running different firmware in cascaded chains are the same as for any source and target. For example, when a target does not support a feature used by the source, the replication update from the corresponding source may fail.

  • System rollback to a firmware version that does not support cascaded replication will result in the destruction of all cascaded replication actions.

Related Topics