Example: Replication Configuration for Clustered Appliances

Configuration Guidelines

When configuring replication for clustered Oracle ZFS Storage Appliance systems, follow these guidelines:

Ensure that both replication source and target appliances are in the CLUSTERED state. For details, see table "Cluster States" in Clustered Controller States.
Select network interfaces and IP addresses to be used for replication traffic on the replication source and target appliances.
- Select a singleton network interface. Unlike a private network interface, a singleton network interface will be taken over by the surviving controller following the loss of one of the controllers in the cluster. Using a singleton interface ensures successful replication following a cluster takeover or failback transition. For more information about singleton interfaces, see table "Cluster Resource Types" in Cluster Resource Management.
- Ensure that the selected network interface on the source appliance and the pool from which the data will be replicated are both assigned to the same controller. This is always the case when the source cluster is in the CLUSTERED state.
- Similarly for the target cluster, the selected network interface on the target appliance and the pool into which the data will be replicated must both be assigned to the same controller. This association is guaranteed when the replication configuration is performed while the target cluster is in CLUSTERED state.
- The source and the target appliances must be able to successfully communicate using the selected network interfaces and IP addresses.
Create static /32 host-based routing between target and source appliances to ensure that following replication reversal, the selected interface is used for outbound replication traffic when reversal has transformed the current target into a replication source.
After the static route has been created, configure the replication target object on the source appliance using the selected IP address of the target.
When the target appliance is in the OWNER state, all shared resources including network interfaces and storage pools are taken over and owned by the one surviving controller, the controller that is now in the OWNER state. On the controller in the OWNER state, it is possible to select a network interface that is assigned to one controller and use it to deliver replication traffic to a pool that is assigned to a different controller. When the controllers are returned to the CLUSTERED state, the network interfaces and storage pools are returned to their assigned controllers. Therefore, replication updates might not be possible because the source appliance will use the network interface on the target controller that no longer owns the pool. This configuration error cannot arise when replication configuration is performed while the target appliance is in the CLUSTERED state.

Example: Configuring Replication for Clustered Appliances

The example procedure uses the following source and target network interfaces and IP addresses:

The source appliance cluster consists of source controllers S1 and S2. Storage pool sp1 is assigned to S1 and pool sp2 is assigned to S2. The cluster network interfaces consist of:

Private interface ixgbe0 on S1 with IP address 198.51.100.81/24
Private interface ixgbe0 on S2 with IP address 198.51.100.82/24
Singleton interface ixgbe1 with IP address 192.0.2.101/25 assigned to S1
Singleton interface ixgbe2 with IP address 192.0.2.102/25 assigned to S2
Singleton interface ixgbe3 with IP address 192.0.2.201/25 assigned to S1
Singleton interface ixgbe4 with IP address 192.0.2.202/25 assigned to S2

The appliance is Initially in the CLUSTERED state where:

S1 owns sp1, ixgbe1, and ixgbe3
S2 owns sp2, ixgbe2 and ixgbe4

The target appliance cluster consists of controllers T1 and T2. Storage pool tp1 is assigned to T1 and pool tp2 is assigned to T2. The cluster network interfaces consist of:

Private interface ixgbe0 on T1 with IP address 198.51.100.83/24
Private interface ixgbe0 on T2 with IP address 198.51.100.84/24
Singleton interface ixgbe1 with IP address 192.0.2.103/25 assigned to T1
Singleton interface ixgbe2 with IP address 192.0.2.104/25 assigned to T2
Singleton interface ixgbe3 with IP address 192.0.2.203/25 assigned to T1
Singleton interface ixgbe4 with IP address 192.0.2.204/25 assigned to T2

The appliance is initially in the CLUSTERED state where:

T1 owns tp1, ixgbe1, ixgbe3
T2 owns tp2, ixgbe2 and ixgbe4

The following steps describe how to configure replication using the CLI for projects Red, Blue, and Green.

Select network interfaces and IP addresses.
- Start by selecting network interfaces and IP addresses for replication of project Red.
  
  Because the source S is in the CLUSTERED state, it is sufficient to ensure that the selected network interfaces and IP addresses are not private. Thus, on S1 use either ixgbe1 or ixgbe3.
- The same applies to target T, therefore, use either ixgbe1 or ixgbe3 on appliance T1. Because ixgbe1 and ixgbe3 on both S1 and T1 belong to the same subnet, select either to perform replication of project Red. For this example, select interface ixgbe1 on S1 and on T1.

Set up a static route on S1.

The following example sets up the static route for replication of project Red on source controller S1:

S1:configuration net routing> create
S1:configuration net route (uncommitted)> set family=IPv4
                            family = IPv4 (uncommitted)
S1:configuration net route (uncommitted)> set destination=192.0.2.103
                       destination = 192.0.2.103 (uncommitted)
S1:configuration net route (uncommitted)> set mask=32
                              mask = 32 (uncommitted)
S1:configuration net route (uncommitted)> set interface=ixgbe1
                         interface = ixgbe1 (uncommitted)
S1:configuration net route (uncommitted)> set gateway=192.0.2.1
                           gateway = 192.0.2.1 (uncommitted)
S1:configuration net route (uncommitted)> commit
S1:configuration net routing> list
ROUTE      DESTINATION      GATEWAY    INTERFACE  TYPE    STATUS
...
route-003  192.0.2.103/32   192.0.2.1  ixgbe1     static  active

Set up a static route on T1.

The following example sets the static route for replicating project Red on target controller T1:

T1:configuration net routing> create
T1:configuration net route (uncommitted)> set family=IPv4
                            family = IPv4 (uncommitted)
T1:configuration net route (uncommitted)> set destination=192.0.2.101
                       destination = 192.0.2.101 (uncommitted)
T1:configuration net route (uncommitted)> set mask=32
                              mask = 32 (uncommitted)
T1:configuration net route (uncommitted)> set interface=ixgbe1
                         interface = ixgbe1 (uncommitted)
T1:configuration net route (uncommitted)> set gateway=192.0.2.1
                           gateway = 192.0.2.1 (uncommitted)
T1:configuration net route (uncommitted)> commit
T1:configuration net routing> list
ROUTE      DESTINATION      GATEWAY    INTERFACE  TYPE    STATUS
...
route-003  192.0.2.101/32   192.0.2.1  ixgbe1     static  active

Create a replication target on S1.

The following example creates the replication target object on S1 to be used to replicate project Red from sp1 to tp1:

S1:shares replication targets>target
S1:shares replication target (uncommitted)> set hostname=192.0.2.103
                          hostname = 192.0.2.103 (uncommitted)
S1:shares replication target (uncommitted)> set label=t1-1
                             label = t1-1 (uncommitted)
S1:shares replication target (uncommitted)> set root_password=(set)
                     root_password = (set) (uncommitted)
S1:shares replication target (uncommitted)> commit

Create a replication action for each project.

Replicate project Red from pool sp1 to tp1
Replicate project Blue from pool sp1 to pool tp2
Replicate project Green from pool sp2 to tp2

The following example creates the replication action for project Red:

S1:> shares select Red replication action
S1:shares Red action (uncommitted)> set target=t1-1
                            target=t1-1 (uncommitted)
S1:shares Red action (uncommitted)> set pool=tp1
                              pool=tp1 (uncommitted)
S1:shares Red action (uncommitted)> commit

Set up to replicate project Blue from pool sp1 to tp2.

Start with interface and address selection, and select interfaces S1/ixgbe3 and T2/ixgbe4, knowing that both S and T are in the CLUSTERED state and that the interface addresses are on the same subnet, 192.0.2.128/25. Next, define static routes on both appliances similar to the earlier examples. Then create replication target object t2-2 on S1, and create the replication action on S1 for project Blue using target object t2-2.
Set up to replicate project Green from pool sp2 to tp2.

Start with interface selection and select interfaces S2/ixgbe2 and T2/ixgbe2. Create static routes on S2 and T2 using the selected interfaces and their addresses, define replication target object t2-1 using address of T2/ixgbe2, and finally create the replication action for project Green using target object t2-1.
Initiate replication for all three actions.
1. Start with project Red:
```
S1:> shares select Red replication select action-000
S1:shares Red action-000> sendupdate
```
2. Initiate replication for the actions for projects Blue and Green by following the previous example.

Replication Data Path Illustrated Examples

The following figures illustrate the replication data paths during replication updates for the replication actions for projects Red, Blue, and Green:

Normal Replication Data Path

Diagram showing the replication data path among source and target controllers

Assume that controller T2 has been taken down for a maintenance. T1 performed a takeover and now owns all of the resources. If replication updates for projects Blue and Green are in progress during the takeover, they will be canceled. After T1 takes over, these replication updates can be resumed manually, or they will be resumed automatically if schedules are configured for the corresponding replication actions.

After controller T1 has completed the takeover, it owns interfaces ixgbe2 and ixgbe4 which are necessary to continue replication updates for projects Blue and Green. The following figure shows the replication data path after T1 completed the takeover.

Replication Data Path After T1 Takeover

Diagram showing the replication data path after a cluster takeover of the target

After T2 is back online, a failback is performed on the T1 controller and it takes over its resources. If replication updates of projects Blue and Green are in progress, they will be canceled and can be resumed following the completion of the failback.

Then controller S2 is taken down for maintenance, and the takeover performed on the S1 controller causes it to take ownership of all of the resources, including the interface required for continuing replication of project Green. If a replication update of project Green is in progress, it will be canceled and can be resumed following the completion of the takeover.

Data Path After Failback on T1 and Takeover on S1

Diagram showing the replication data path after a failback on the target