Implement rsync with a Central Staging Location

Implement `rsync` with a Central Staging Location

This implementation uses the rsync technology and follows the model based on a central staging location. In this model, there is a bastion host node that acts as a coordinator. It connects to each host that needs to be replicated and copies the contents to a common staging location.

The advantages of implementing rsync with a central staging location are:

It is a general-purpose solution applicable to any mid-tier, so if you have multiple systems, you can use the same approach in all of them.
It doesn’t depend on the underlying storage type; it is valid for replicating file artifacts that reside in block volumes, in NFS, and so on.
The storage can remain mounted in the secondary nodes. Hence, it doesn’t require additional steps to attach or mount the storage in the secondary in every switchover or failover operation.
Compared to peer-to-peer implementation, the maintenance is simpler, since there is a central node for running the scripts.

The considerations for implementing the rsync with a central staging location are:

It is the user’s responsibility to create the custom scripts for each environment and run them periodically.
It is the user’s responsibility to implement a way to reverse the replica direction.
This model requires an additional host and storage for the central staging location.

Similar to the peer-to-peer model, the rsync scripts can use a pull or a push model. In the “pull” model, the script copies the files from the remote node to the local node. In the “push” model, the script copies the file from the local node to the remote node. Oracle recommends using a pull model to retrieve the contents from primary hosts, because it offloads the primary nodes from the overhead of the copies.

Set Up Replication for `rsync` with Central Staging

The following is required to implement rsync with a central staging location:

A bastion host with SSH connectivity to all the hosts (both primary and secondary).
A staging folder in the bastion host, with enough space to store the mid-tier file system contents that are replicated.
Scripts that use rsync to copy the mid-tier file artifacts from and to this staging folder. The rsync scripts can skip certain folders from the copy (like lock files, logs, temporary files, and so on).
A way to manage the site-specific information, either excluding that info from the copy or updating it with the appropriate info after the replica.
Schedule these scripts to run periodically.
A mechanism to change the direction of the replica after a switchover or failover. This mechanism can be a dynamic check that identifies the role of the site, or a manual change after a switchover or failover (for example, disabling and enabling the appropriate scripts).

This document presents two different implementation examples of this model:

Example 1: Use the Oracle Fusion Middleware Disaster Recovery Guide scripts
Example 2: Use the WLS-HYDR framework

Example 1: Use the Oracle Fusion Middleware Disaster Recovery Guide scripts

Note:

This example applies to any mid-tier system. As a reference, it uses the scripts provided by the Oracle Fusion Middleware Disaster Recovery Guide to perform the mid-tier replica for an Oracle WebLogic DR system: rsync_for_WLS.sh and rsync_copy_and_validate.sh. But these scripts are generally applicable and provide enough flexibility to synchronize the mid-tier file system artifacts in OCI.

The Oracle Fusion Middleware Disaster Recovery Guide provides rsync scripts to perform remote copies in a mid-tier system. These scripts are valid for any rsync model. This particular example shows how to use them for the central staging model. This implementation uses pull operations in two steps:

A bastion host pulls the contents from all the primary hosts and stores them in the central staging.
Then, all the secondary nodes perform a pull operation to gather the contents from the central staging.

To set up the mid-tier replication with these scripts, see Replicating the Primary File Systems to the Secondary Site in the Oracle Fusion Middleware Disaster Recovery Guide, and the Rsync Replication Approach section and Using a Staging Location steps in particular.

Description of replica-rsync-scripts.png follows

Description of the illustration replica-rsync-scripts.png

replica-rsync-scripts-oracle.zip

Example 2: Use the WLS-HYDR framework

Note:

This example applies to an Oracle WebLogic Server system. It uses the replication module of the WLS-HYDR framework, but it applies to any Oracle WebLogic Server DR environment, regardless of whether it was created with the WLS-HYDR framework or not.

In this model, a central host node acts as a total coordinator, performing pull and push operations. It connects to each host that needs to be replicated and copies the contents to a common staging location. This node also coordinates the copy from the staging location to the destination hosts. This approach offloads the individual nodes from the overhead of the copies.

The WLS-HYDR framework uses this approach for the initial copy during the DR setup. Then, you can reuse the replication module of the framework to repeat the pull and push periodically. Refer to Explore More in this playbook for links to the WLS-HYDR framework and other resources..

The bastion node performs the replica in two steps:

A pull operation, where it connects to the primary hosts and copies the file system contents to a staging folder in the bastion host.
A push operation, where it copies the contents from the bastion’s staging folder to all the secondary hosts.

A central node performs all the operations, so scheduling, logs, maintenance, etc., are centralized on that node. When the system has many nodes, this is more efficient compared to the peer-to-peer model or the previous example.

Description of replica-wls-hydr-framework.png follows

Description of the illustration replica-wls-hydr-framework.png

replica-wls-hydr-framework-oracle.zip

If you used the WLS-HYDR framework for creating the secondary system, the bastion host is already prepared to perform the replica. Otherwise, you can configure it at this point. Follow these steps for setting up the replica:

Prepare the bastion.

If you used the WLS-HYDR framework for creating the secondary system, the bastion host is already prepared to perform the replica. If not, check the README in the WLS-HYDR framework GitHub repository to prepare a bastion host.
Review the WLS-HYDR configuration files.
Ensure that the WLS-HYDR configuration files (replication.properties, oci.env and prem.env) contain the correct information for your system.
Run the WLS-HYDR replication module.
You can use the replication module of the framework to sync all the items (the Oracle WebLogic Server domain and products), or you can sync these items independently. In all the cases, the complete sync consists of two operations: a pull operation, to retrieve the contents from the primary, and a push operation, to copy the contents to the secondary.

Note:
Always run the PULL operation before the PUSH. Otherwise, you will not push the latest version of the contents.
1. To sync all content:
  - To pull all content from the primary hosts to the bastion host’s staging:
```
WLS-HYDR_BASE/lib/DataReplication.py pull
```
  - To push all contents from the bastion host’s staging to the secondary hosts:
```
WLS-HYDR_BASE/lib/DataReplication.py push
```
2. To sync the products artifacts only:
  - To pull the products from the primary hosts to the bastion host’s staging:
```
WLS-HYDR_BASE/lib/DataReplication.py pull -d products
```
  - To push the products from the bastion host’s staging to the secondary hosts:
```
WLS-HYDR_BASE/lib/DataReplication.py push -d products
```
3. To sync the configuration (the WebLogic domain) only.
  - To retrieve the configuration from the primary hosts to the bastion host’s staging, run this pull operation:
```
WLS-HYDR_BASE/lib/DataReplication.py pull -d private_config
```
  - To copy the configuration from the bastion host’s staging to the secondary hosts, run this push operation:
```
WLS-HYDR_BASE/lib/DataReplication.py push -d private_config
```
4. For cases where there is a shared configuration folder (shared Oracle WebLogic domain in an OCI File Storage file system):
  - To retrieve the shared configuration from the primary hosts to the bastion host’s staging, run this pull operation:
```
WLS-HYDR_BASE/lib/DataReplication.py pull -d shared_config
```
  - To copy the shared configuration from the bastion host’s staging to the secondary hosts, run this push operation:
```
WLS-HYDR_BASE/lib/DataReplication.py push -d shared_config
```

Prepare for the database connection string replacements.

The regular WLS-HYDR pull and push operations that copy the WebLogic configuration skip the tnsnames.ora file from the copies, so you don’t need to update the tnsnames.ora in every subsequent replication.

However, the approach is different when the database is an Autonomous Database. To connect to an autonomous database, the TNS admin folder also contains a keystore and trust store files, which are different for the primary and standby databases.

The following table summarizes how you can manage the database connection information in each scenario:

Database Type Replacement Script and Download Steps Usage

Database Type	Replacement Script and Download Steps	Usage
Oracle Base Database Service or Oracle Exadata Database Service	The script to manage the `tnsnames.ora` is included in the WLS-HYDR framework replication module.	Ensure that the JDBC section in the `replication.properties` file contains the correct data. This operation runs in the bastion, and performs the pull, update, and push for the `tnsnames.ora` file. To perform the complete operation: `WLS-HYDR/lib/DataReplication.py tnsnames` No need to run unless you perform changes to the `tnsnames.ora` (for example, adding an alias).
Autonomous Database	`fmwadb_switch_db_conn.sh` Go to the Oracle MAA repository in GitHub `https://github.com/oracle-samples/maa` Download all the scripts in the `app_dr_common` directory. Download all the scripts in the `fmw-wls-with-adb-dr` directory. Copy to all the mid-tier hosts. The scripts make calls to each other. Place all the scripts of both directories in the same folder.	This script must run on all the standby mid-tier hosts. It must run before starting the WebLogic servers in the standby for a validation, switchover, or failover operation. It replaces the TNS admin folder used by WebLogic with the one given as input. It also updates the wallet password properties in the data sources. Syntax to run the script: `fmwadb_switch_db_conn.sh WALLET_DIR WALLET_PASSWORD` Where the WALLET_DIR is a folder that contains the `tnsnames.ora`, keystore, and truststore files to connect to the local database. Ensure that the WALLET_DIR folder is not overridden in the replica.

Oracle Base Database Service or Oracle Exadata Database Service

The script to manage the tnsnames.ora is included in the WLS-HYDR framework replication module.

Ensure that the JDBC section in the replication.properties file contains the correct data.

This operation runs in the bastion, and performs the pull, update, and push for the tnsnames.ora file. To perform the complete operation: WLS-HYDR/lib/DataReplication.py tnsnames

No need to run unless you perform changes to the tnsnames.ora (for example, adding an alias).

Autonomous Database

fmwadb_switch_db_conn.sh

Go to the Oracle MAA repository in GitHub https://github.com/oracle-samples/maa

Download all the scripts in the app_dr_common directory.

Download all the scripts in the fmw-wls-with-adb-dr directory.

Copy to all the mid-tier hosts. The scripts make calls to each other. Place all the scripts of both directories in the same folder.

This script must run on all the standby mid-tier hosts. It must run before starting the WebLogic servers in the standby for a validation, switchover, or failover operation.

It replaces the TNS admin folder used by WebLogic with the one given as input. It also updates the wallet password properties in the data sources.

Syntax to run the script:

fmwadb_switch_db_conn.sh WALLET_DIR WALLET_PASSWORD

Where the WALLET_DIR is a folder that contains the tnsnames.ora, keystore, and truststore files to connect to the local database. Ensure that the WALLET_DIR folder is not overridden in the replica.

Validate Replication for `rsync` with Central Staging

In a switchover or failover operation, the replicated information must be available and usable in the standby site before the processes are started. This is also required when you validate the secondary system (by opening the standby database in snapshot mode).

In this implementation, the storage is always available in the secondary site, you don’t need to attach or mount any volume. The only action you may need is to ensure that it contains the latest version of the contents is the following.

Run a replication.
Run the scripts to replicate the last version of the contents.
Disable scheduled replications.
After the last replica finishes, disable all replica scripts. Otherwise, it can interfere with the switchover, failover, or validation procedure. You will enable the scripts again after the operation, in the appropriate direction.
Replace the information that is site-specific in the secondary mid-tier hosts.
If the file system artifacts you are replicating contain information that is site-specific, then ensure it is not overridden by the copy or it is updated post-replica.

Tip:
For example, the rsync scripts of the Oracle Fusion Middleware Disaster Recovery Guide and the WLS-HYDR replication module skip the file tnsnames.ora from the copy, so you don’t need to alter it after every replication.
If the system uses Oracle Autonomous Database, then restore the wallet folder needed in the secondary to connect to the secondary region’s database. You can use the replacement script fmwadb_switch_db_conn.sh in all of the standby mid-tier hosts. It requires, as inputs, the path where the secondary original wallet is located and the wallet password.

Then you can perform additional steps required to validate the system.

Perform Ongoing Replication for `rsync` with Central Staging Location

Run the replication scripts periodically to keep the secondary domain in sync with the primary.

Follow these recommendations for the ongoing replication when using this implementation:

Use the OS crontab or another scheduling tool to run the replication scripts periodically. For example, when using the rsync scripts provided by the disaster recovery guide, follow the steps in the Scheduling Ongoing Replication With Rsync Scripts section of the Oracle Fusion Middleware Disaster Recovery Guide. Refer to Explore More in this playbook for links to these and other resources. For the replication frequency, follow the guidelines described in Mid-tier File Artifacts earlier in this playbook.
Keep the mid-tier processes stopped in the standby site. If the servers are up in the standby site while the changes are replicated, the changes will take effect the next time they are started. Start them only when you are validating the standby site or during the switchover or failover procedures.
Maintain the information that is specific to each site up to date. For example, if the file system contains a folder with the artifacts to connect to an Autonomous Database, maintain a backup copy of this folder. Ensure that you update the backup of the wallet folder when you perform an update in the wallet. This way, it will be correctly restored in subsequent switchover and failovers.
After a switchover or failover, reverse the replica direction. This depends on the specific implementation. This can be done using a dynamic check that identifies who is the active site, or with a manual change after a switchover or failover, disabling and enabling the appropriate scripts.
Tip:
- When using the rsync scripts provided by the DR Guide (example 1), make sure you create the equivalent scripts to perform the replica in the other direction. In the crontab or the scheduled tool, enable only the appropriate scripts for the actual role.
- When using the WLS-HYDR (example 2), change the role of the primary in the WLS-HYDR framework, so the next replications go in the other direction. For this, edit the WLS-HYRDR/lib/DataReplication.py and change from these:
```
if True:
       PRIMARY = PREM 
       STANDBY = OCI
      	 else:
	        PRIMARY = OCI
               STANDBY = PREM
```
  to the following:
```
if False:
	        PRIMARY = PREM
                STANDBY = OCI
       	else:
                PRIMARY = OCI
                STANDBY = PREM
```