Overview of Data Replication Using Oracle Solaris ZFS Snapshot

Language:

ZFS has built-in features that perform the following actions:

Take a consistent snapshot of a ZFS dataset.
Create a stream representation of such a snapshot.
Create a snapshot based on the contents of such a stream.

Such features of ZFS can be used together to provide a method of replicating data on ZFS datasets from the primary cluster to the secondary cluster within a DR environment. This process ensures that the contents of the primary and secondary datasets are in sync. This is the basis of this replication feature supported by Oracle Solaris Cluster Geographic Edition (OSCGE) which in turn controls and monitors the direction of replication.

During data replication, data from a primary cluster is copied to a backup or secondary cluster. The secondary site can be located at a geographically separated location from the primary site.

Before you can replicate data with the Oracle Solaris ZFS snapshots feature, you must be familiar with Oracle Solaris ZFS administration and have the latest Oracle Solaris SRUs installed on your system. For information about Oracle Solaris ZFS, see Managing ZFS File Systems in Oracle Solaris 11.3. For information about replicating data using Oracle Solaris ZFS snapshot, see Chapter 8, Working With Oracle Solaris ZFS Snapshots and Clones in Managing ZFS File Systems in Oracle Solaris 11.3. For information about ZFS commands, see the zfs(1M) man page.

The following figure displays how Oracle Solaris ZFS snapshot is used to replicate data.

Figure 1 Oracle ZFS Snapshots Replication Model

image:This illustration shows the Oracle ZFS snapshots replication model

The basis of the Geographic Edition ZFS snapshot replication module is to periodically do this : create a ZFS snapshot on the source dataset, use the zfs send command to create a snapshot stream based on the snapshot, and replicate that snapshot on to the target dataset by feeding the stream to zfs receive command on the target host over SSH. Data replication is performed with the credentials of a replication user on each cluster.

The replication module consists of the following components:

HAStoragePlus resource for highly available ZFS pools in the Oracle Solaris Cluster environment.
Logical host name resource to manage host name for replication source or target.
Replication user with the credentials to perform snapshots replication.
Replication user's SSH setup with public/private keys and passphrase. You must be able to establish SSH connection between any node of the primary cluster and any node of the secondary cluster using replication user's SSH credentials.

The figure shows that the replication configuration exists on all nodes. However, functional parts such as the ZFS pool, HAStoragePlus resource, and logical hostname resource are active on one primary node and one secondary node at a time.

You must use the configuration file and the setup script provided with OSCGE ZFS snapshots replication package to set up a protection group with ZFS snapshot replication on the primary cluster in a partnership. For more information about the parameters used in the configuration file, see Overview of the Oracle Solaris ZFS Snapshot Remote Replication Configuration File.

About Oracle Solaris ZFS Snapshots Replication

The basis of the OSCGE ZFS snapshot replication module is to periodically performs the following actions:

Create a ZFS snapshot on the source dataset.
Create a snapshot stream using the zfs send command
Replicate the snapshot on to the target dataset by feeding the stream to the zfs receive command on the target host over SSH.

The replication module names the replication snapshots in the format oscge_num, where num is a monotonically increasing sequence number.

ZFS supports different types of snapshot streams and stream packages. OSCGE ZFS snapshot replication module supports full streams as well as incremental streams. If source and target datasets already have common snapshots thus enabling replication using incremental streams, then this replication module uses incremental streams. Otherwise, it uses full streams. The replication module always attempt to use incremental replication wherever possible. The replication module takes the decision to use the incremental replication and there is no option available to set this behavior explicitly.

This snapshot replication module supports data replication using ZFS replication stream package or ZFS recursive stream package. You can use these options to replicate a source dataset and its child datasets on to the target cluster. For example, suppose the source dataset is src-pool1/app1-ds1 and the target dataset is targ-pool1/app1-ds1-copy. The source dataset has child datasets src-pool1/app1-ds1/fs1 and src-pool1/app1-ds1/vol1. Then either of these stream package options can be used to replicate the dataset hierarchy starting at src-pool1/app1-ds1 on to the target. As a result of the replication, the data of source dataset src-pool1/app1-ds1 is replicated to the target dataset targ-pool1/app1-ds1-copy, the data of src-pool1/app1-ds1/fs1 is replicated to targ-pool1/app1-ds1-copy/fs1, and the data of src-pool1/app1-ds1/vol1 is replicated to targ-pool1/app1-ds1-copy/vol1.

For more information about ZFS snapshot streams, see Types of ZFS Snapshot Streams in Managing ZFS File Systems in Oracle Solaris 11.3. For more information about the zfs send and zfs receive commands and their options related to snapshot streams and stream packages, see the zfs(1M) man page.

Replication Stream Package

You can generate a replication stream package using the –R option with the zfs send command. You can choose this type of replication by using the REPLICATION_STREAM_PACKAGE parameter in the replication parameters configuration file. For more information about the parameters available in the configuration file, see Overview of the Oracle Solaris ZFS Snapshot Remote Replication Configuration File.

You can use replication stream package in one of the following situations:

Common snapshot does not exist between the source dataset and the target dataset, and none of the child datasets of the target dataset have snapshots replicated from the corresponding child dataset of the source dataset.

For example, suppose the source dataset is src-pool1/app1-ds1 and the target dataset is targ-pool1/app1-ds1-copy. There is no common snapshot between the following datasets:
- src-pool1/app1-ds1 and targ-pool1/app1-ds1-copy
- src-pool1/app1-ds1/fs1 and targ-pool1/app1-ds1-copy/fs1
- src-pool1/app1-ds1/vol1 and targ-pool1/app1-ds1-copy/vol1
In such situation, if you use the replication stream package option, the replication module uses full snapshot streams of all these datasets to create a replication stream package, and receives that stream package on the target dataset. This removes any existing data on the target dataset or its child datasets, and replicates data from the source dataset hierarchy.
Name of the latest replicated snapshot on the source and target datasets is the same as the name of the latest common snapshot on each pair of their corresponding child datasets. In such situation, the module will use incremental replication streams of all the datasets to create an incremental replication stream package.

For example, suppose the source dataset is src-pool1/app1-ds1 and the target dataset is targ-pool1/app1-ds1-copy, and the latest common replicated snapshot among each pair of datasets in the source and target dataset hierarchy is snap1. That is, the following common snapshot exists on the source and target dataset pairs in the hierarchy, and there is no other replicated snapshot on any of those dataset pairs that is newer than snap1:
- src-pool1/app1-ds1@snap1 and targ-pool1/app1-ds1-copy@snap1
- src-pool1/app1-ds1/fs1@snap1 and targ-pool1/app1-ds1-copy/fs1@snap1
- src-pool1/app1-ds1/vol1@snap1 and targ-pool1/app1-ds1-copy/vol1@snap1
In such situation, the replication module performs incremental replication by using snap1 as the base snapshot in the incremental replication stream package for this hierarchy.

If the latest replicated snapshots on every corresponding pair of datasets in the source and target dataset hierarchy do not have the same snapshot name, then you cannot use replication stream package to replicate the hierarchy together. Instead, you must use the recursive stream package.

Recursive Stream Package

Recursive stream package is created using –r option with the zfs send command. You can choose this type of replication by using the RECURSIVE parameter in the replication parameters configuration file. For more information about the parameters available in the configuration file, see Overview of the Oracle Solaris ZFS Snapshot Remote Replication Configuration File.

You can use recursive stream package in one of the following situations:

Common snapshot does not exist between the source dataset and the target dataset, and none of the child datasets of the target dataset have snapshots replicated from the corresponding child dataset of the source dataset.
Name of the latest replicated snapshot on the source and target datasets is the same as the name of the latest common snapshot on each pair of their corresponding child datasets. In such situation, the module will use incremental replication streams of all the datasets to create an incremental recursive stream package.
Name of the latest replicated snapshots on every corresponding pair of datasets in the source and target dataset hierarchy is not the same.

For example, suppose the latest replicated snapshot between src-pool1/app1-ds1 and targ-pool1/app1-ds1-copy is snap1. However, the latest replicated snapshot between src-pool1/app1-ds1/fs1 and targ-pool1/app1-ds1-copy/fs1 is snap2. In this situation, the module first individually replicates each dataset in the hierarchy on to the target so that the target dataset's hierarchy has a common latest replicated snapshot. Then, the replication module uses recursive stream package for subsequent replication attempts.

Note - If you want to use replication stream package, but you have different latest replicated snapshots on the target dataset hierarchy, then set up replication initially using recursive stream package. This will enable the replication to synchronize the target dataset hierarchy with the source dataset hierarchy, so that all datasets in the target dataset's hierarchy have the latest common replicated snapshot name. Then, modify the replication parameters to use replication stream package.

Incremental Replication and Intermediary Snapshots

In each replication attempt, the replication module will take a new snapshot on the source dataset. It is a recursive snapshot if the replication uses replication stream package or recursive stream package. If there is a base snapshot among the source and target datasets that can be used as the base for incremental replication, then the replication module will attempt incremental replication with the stream containing incremental changes between the base snapshot and the new snapshot. Incremental replication is performed by using the –i option with the zfs send command.

The source dataset may have snapshots that are more recent than the base snapshot but are older than the new snapshot created by the replication module. If you want such intermediary snapshots to be replicated, then set the INTERMEDIARY_SNAPSHOTS parameter to TRUE in the replication parameters configuration file and setup or modify the replication component. Then the replication module will use the –I option with the zfs send command to replicate such intermediary snapshots. Note that this option can be used when replicating a dataset individually or with replication stream package, but not with recursive stream package, as the zfs send command does not support –r and –I options to be used together.

Sending ZFS Properties

This replication feature supports the use of –p option with the zfs send command to send ZFS properties in a snapshot stream for replication. To use this feature, set the SEND_PROPERTIES parameter to TRUE in the replication parameters configuration file. For more information about the SEND_PROPERTIES parameter, see Overview of the Oracle Solaris ZFS Snapshot Remote Replication Configuration File.

Oracle Solaris ZFS Snapshot Replication Package

The Oracle Solaris ZFS snapshot replication feature in Geographic Edition software is delivered through the package ha-cluster/geo/replication/zfs-snapshot in the Oracle Solaris Cluster IPS repository. Install this package before attempting to create Oracle Solaris ZFS snapshot replication protection groups.

Type the following command to install the package:

# /usr/bin/pkg install ha-cluster/geo/replication/zfs-snapshot

Oracle Solaris ZFS Snapshot Infrastructure Resource Groups

The Geographic Edition creates an infrastructure resource group for each Application Resource Group whose ZFS datasets are added to a protection group for ZFS snapshot-based data replication. The name of the ZFS pool on the primary and secondary clusters can differ.

The name of an infrastructure resource group has the following format:

pg-AppRGName-infr-rg

For example, a protection group named pg1 that uses the application resource group app-rg1 has an infrastructure resource group named pg1-app-rg1-infr-rg.

Each infrastructure resource group contains the following two resources:

A logical hostname resource for the local logical host that is used for replication of ZFS datasets of the ZFS pool, if the local partner is a global zone. Each zpool in each cluster that is under replication using Oracle Solaris ZFS snapshot requires a logical hostname resource. One logical hostname resource is configured per infrastructure resource group. The name of the logical hostname resource has the following format:
```
pg-AppRGName-lh-rs
```
If local partner is a non-global zone, the logical hostname resource required for replication must reside in another resource group configured in the global zone of the local partner zone cluster instead of the infrastructure resource group. Such a resource group and resource must be created by the user. In this case, the logical hostname resource does not follow any naming format.
An HAStoragePlus resource for managing the ZFS pool within the infrastructure resource group. The name of this resource has the following format:
```
pg-zpool-stor-rs
```
An HAStoragePlus resource is configured in infrastructure resource group for each of the zpools managed by the application resource group. If multiple zpools are managed by a single application resource group, which is a failover resource group, then all such HAStoragePlus resource are combined in a single infrastructure Resource group.

For examples displaying resource groups and resources, see Use Cases for Oracle Solaris ZFS Snapshot Replication.

Note - The application resource group would already have a HAStoragePlus resource to manage the zpool used by the application. When creating replication setup, the OSCGE software creates another HAStoragePlus resource for the same zpool. The OSCGE software utilizes two properties of the SUNW.HAStoragePlus resource type, ZpoolsExportOnStop and ZpoolsImportOnly, to create such replication setup consisting of two HAStoragePlus resources for the same zpool. The OSCGE software also sets a strong positive affinity with failover delegation from the application resource group to the infrastructure resource group and an offline-restart resource dependency from the application's HAStoragePlus resource managing the zpool to the replication infrastructure HAStoragePlus resource for the same zpool. The configuration of multiple HAStoragePlus resources for the same zpool and use of such properties of the SUNW.HAStoragePlus resource type is not supported outside of the OSCGE ZFS snapshot replication setup. When such replication setup is created by the Geo software, a device group is also created for the same zpool. The zpool device group can be seen in the global zone of the cluster in the output of commands that display information about device groups, such as cldevicegroup list or cldevicegroup status. Note that such setup is solely for use by the OSCGE ZFS snapshot replication software.

Oracle Solaris ZFS Snapshot Replication Agent Resource Groups

When a replication component is added to a protection group, the Geographic Edition software creates a ZFS snapshot replication agent resource for that replication component in a ZFS snapshot replication agent resource group. The resource type of this resource is ORCL.ZFSSnapRepAgent, which is a new type shipped with the Oracle Solaris ZFS snapshot replication package. This resource performs the ZFS snapshot replication associated with the replication component.

One ZFS snapshot replication agent resource group with one ZFS snapshot replication agent resource is created for each replication component.

The name of the replication agent resource group has the following format:

repcomp-snap-rg

where repcomp is the name of the replication component.

The name of a replication agent resource has the following format:

repcomp-snap-rs

The Geographic Edition software sets a strong positive affinity from such a replication agent resource group to the associated Oracle Solaris ZFS snapshot infrastructure resource group, and an offline-restart resource dependency from such a replication agent resource to the infrastructure HAStoragePlus resource managing the associated ZFS pool. This is set so that the replication agent gets online on the same node as the infrastructure HAStoragePlus resource when activated and the ZFS pool is imported, and can perform ZFS commands on the associated ZFS pool as required for its replication work.

The replication agent resource group must have a strong positive affinity on the associated infrastructure resource group. This is managed by Geographic Edition software and you do not need to manually operate on it typically.

Note - The intent of using a strong affinity without failover delegation is that a failover of the replication agent resource group must not result in the failover of the infrastructure resource group. The Geographic Edition software sets a strong positive affinity with failover delegation from the application resource group to the infrastructure resource group. If the replication agent resource group has a strong affinity with failover delegation on to the infrastructure resource group, then a failover of replication agent resource group would induce a failover of the infrastructure resource group, thereby initiating a failover of the application resource group. Thus, a failure in replication will also attempt to move the application, which may not be desirable always. Hence, the default affinity is just a strong affinity. But if you still want to set a strong positive affinity with failover delegation to ensure failure in replication attempts to failover the application and replication both to a different node, you can set such affinity manually from the replication agent resource group to the infrastructure resource group after creating the replication component. The Geographic Edition software will not overwrite such affinity.

Oracle Solaris ZFS Snapshot Replication Status Resource Groups

When a replication component is added to a protection group, the Geographic Edition software creates a special replication status resource for that replication component in the replication status resource group of the protection group. The resource type of this resource is ORCL.ZFSSnapRepStatus, which is a new type shipped with the Oracle Solaris ZFS snapshot replication package. The Geographic Edition software uses the replication status resource groups to monitor the overall status of replication. One replication status resource group is created for a protection group, and one replication status resource in that replication status resource group is created for each replication component in the protection group. The replication status resource in the replication status resource group monitors the replication status of the replication component.

The names of the replication status resource groups and resources do not conform to any format. You must provide them in the configuration file when filling in replication parameters to set up replication.