Implement Peer-to-Peer rsync Replication

This implementation uses the rsync technology and follows the peer-to-peer model. In this model, the copy is done directly between the mid-tier peer hosts. Each node has SSH connectivity to its peer and uses rsync commands over SSH to replicate the primary mid-tier file artifacts.

The advantages of implementing rsync peer-to-peer replication are as follows:

• It is a general-purpose solution applicable to any mid-tier, so if you have multiple systems, you can use the same approach in all of them.
• It doesn’t depend on the underlying storage type; it is valid for replicating file artifacts that reside in block volumes, in NFS, and so on.
• It doesn’t need additional hardware, like a central host or storage.
• The storage can remain mounted in the secondary nodes. Hence, it doesn’t require additional steps to attach or mount the storage in the secondary in every switchover or failover operation.

The considerations for implementing the rsync peer-to-peer are as follows:

• It is the user’s responsibility to create the custom scripts for each environment and run them periodically.
• It is the user’s responsibility to implement a way to reverse the replica direction.
• It requires maintenance across many nodes since scripts are not centralized, so the solution is more complex in large clusters.

The peer-to-peer rsync scripts can use a pull or a push model. In the “pull” model, the script copies the files from the remote node to the local node. In the “push” model, the script copies the files from the local node to the remote node. When the rsync scripts are executed on the nodes with the standby role, they run a “pull” operation to retrieve the content from the primary. When the rsync scripts are executed on the nodes with the primary role, they run a push operation to copy the contents to the secondary nodes. Oracle recommends the pull model for peer-to-peer. This way, the rsync scripts use fewer resources of the primary system’s hosts, since all the operations of the copy (for example, the checksum comparison of the copy) run in the secondary nodes.

Set Up Replication for rsync Peer-to-Peer

The following is required to implement the rsync peer-to-peer model:

• Allow SSH connectivity between the hosts and their peer hosts.
• Create scripts that use rsync to copy the mid-tier file artifacts from primary to secondary hosts. The rsync scripts can skip certain folders from the copy (like lock files, logs, temporary files, and so on)
• Implement a way to manage the site-specific information, either by excluding that information from the copy or by updating it with the appropriate info after the replica.
• Schedule these scripts to run periodically.
• A mechanism to change the direction of the replica after a switchover or failover. This mechanism can be a dynamic check that identifies the role of the site, or a manual change after a switchover or failover (for example, disabling and enabling the appropriate scripts).

Example 1: Use the Oracle Fusion Middleware Disaster Recovery Guide rsync scripts for peer-to-peer replication

Note:

This example applies to any mid-tier system. It uses the scripts provided by Oracle Fusion Middleware Disaster Recovery Guide to perform the mid-tier replica for a WebLogic DR system: rsync_for_WLS.sh and rsync_copy_and_validate.sh. But these scripts are generally applicable and provide enough flexibility to synchronize any mid-tier file artifacts in OCI. Refer to Explore More for links to these and other resources.

In this example, each host in the secondary site establishes a connection with its peer primary node and performs a pull of the contents. To set up the mid-tier replication with these scripts, see Replicating the Primary File Systems to the Secondary Site in the Oracle Fusion Middleware Disaster Recovery Guide, and the Rsync Replication Approach section and Using a Peer-to-Peer steps in particular.

Description of the illustration dr-scripts-peer-peer.png

dr-scripts-peer-peer-oracle.zip

Validate Replication for rsync Peer-to-Peer

In a switchover or failover operation, the replicated information must be available and usable in the standby site before the processes are started. This is also required when you validate the secondary system (by opening the standby database in snapshot mode).

In this implementation, the storage is always available in the standby; you don’t need to attach or mount any volume. The only action you need is to ensure that it contains the latest version of the contents.

1. Run a replication.
   Run the scripts to replicate the last version of the contents.
2. Disable scheduled replications.
   After the last replica finishes, disable all replica scripts. Otherwise, it can interfere with the switchover, failover, or validation procedure. You will enable the scripts again after the operation, in the appropriate direction.
3. Replace the information that is site-specific in the secondary mid-tier hosts.
   If the file system artifacts you are replicating contain information that is site-specific, then ensure it is not overridden by the copy or it is updated post-replica.

   Tip:

   For example, the rsync scripts skip the file tnsnames.ora from the copy, so you don’t need to alter it after every replication.

   If the system uses Oracle Autonomous Database, then restore the wallet folder needed in the secondary to connect to the secondary region’s database. You can use the replacement script fmwadb_switch_db_conn.sh in all of the standby mid-tier hosts. It requires, as inputs, the path where the secondary original wallet is and the wallet password.

Perform Ongoing Replication for rsync Peer-to-Peer

Run the replication scripts periodically to keep the secondary domain in sync with the primary.

Follow these recommendations when using rsync from the mid-tier hosts:

• Use the OS crontab or another scheduling tool to run the replication scripts periodically. For example, when using the rsync scripts provided by the Oracle Fusion Middleware Disaster Recovery Guide, follow the steps described in the Scheduling Ongoing Replication With Rsync Scripts section. Refer to Explore More in this playbook for links to these and other resources. For the replication frequency, follow the guidelines described in Mid-Tier File Artifacts at the beginning of this playbook.
• Keep the mid-tier processes stopped in the standby site. If the servers are up in the standby site while the changes are replicated, the changes will take effect the next time they are started. Start them only when you are validating the standby site or during the switchover or failover procedures.
• Maintain the information that is specific to each site up-to-date. For example, if the file system contains a folder with the artifacts to connect to an Autonomous Database, maintain a backup copy of this folder. Ensure that you update the backup of the wallet folder when you perform an update in the wallet. This way, it will be correctly restored in subsequent switchover and failovers.
• After a switchover or failover, reverse the replica direction. This depends on the specific implementation. This can be done using a dynamic check that identifies who is the active site, or with a manual change after a switchover or failover, disabling and enabling the appropriate scripts. For example, with the rsync scripts provided by the Oracle Fusion Middleware Disaster Recovery Guide, make sure you create the equivalent scripts to perform the replica in the other direction. In the crontab or the scheduled tool, enable only the appropriate scripts for the actual role.