Implement OCI File Storage Replication

This implementation uses the Oracle Cloud Infrastructure File Storage replication feature, which provides an automated cross-region replica for the OCI File Storage file systems.

The advantages of implementing the OCI File Storage replication are:

• There is no need to create and run scripts periodically, as in other replication cases. Once you set up the replication, it is performed automatically by Oracle Cloud Infrastructure.
• It is a general-purpose solution applicable to any OCI File Storage file system mounted by any mid-tier system. If you have multiple systems using OCI File Storage, then you can use the same approach in all of them.
• The information on the replicated file system is an exact copy of the primary; all the files in the file system are replicated.

The considerations of implementing OCI File Storage are:

• It requires steps to mount the replicated OCI File Storage in the secondary system. You can’t directly mount the target file systems; you first have to clone them, and then you can mount the cloned file system. However, you can overcome this complexity using the OCI Full Stack Disaster Recovery service to automate these steps in the switchover, failover, and validation operations.
• This technology may not be enough for many systems. If the system has more types of storage (for example, block volumes), then you will need to use a different replica technology for them.

Set Up Replication for OCI File Storage

To implement OCI File Storage replication, the following steps are required:

• Use the OCI Console to create the target OCI file systems in the secondary site.
• Enable the replica in the primary OCI file systems, pointing to the appropriate target OCI file system.
• Connect to secondary region’s mid-tier hosts and unmount the file system that are going to be replicated from primary.
• Using the OCI Console UI, detach, and discard the OCI file systems that will be replicated from the primary.
• Implement a way to manage the site-specific information by updating it with the appropriate information after the replica.

Example 1: Use OCI File Storage replication to replicate the mid-tier configuration and runtime

Note:

This example applies to any mid-tier system. As a reference, it uses an Oracle WebLogic Server system that follows the Oracle Fusion Middleware Enterprise Deployment Guide's best practices. This system has two OCI File Storage file systems: one for the shared configuration (the WebLogic Administration domain, keystores, and so on) and the other for the runtime data. But you can follow the same steps to replicate any OCI File Storage file system of a mid-tier.

Perform the following to set up the cross-region replica for the OCI File Storage file systems:

1. Back up the information that is specific to each site.
   The file system can contain files with information that is specific to each site, for example, connection strings to databases or to LDAP servers. When using OCI File Storage replica, the replicated file system is an exact copy of the primary; you can’t skip specific files or folders from the replica. Hence, you must manage these differences by adapting the information on each site. There are various approaches:

   • You can perform a string search and replacement in the files with site-specific information.
   • You can back up this information before the replica and restore it afterward.

   At this point, before enabling the replica, identify and back up any file with site-specific information in the block volumes that are replicated. Make the backup copy in a location that isn’t under the replicated block volume; otherwise, it will be overridden.

   Tip:

   Oracle WebLogic Example

   For example, when replicating a file system that contains a WebLogic domain, there are files with information to connect to the database. This information is in the TNS admin folder. Check the tns_admin property in the WebLogic data sources to identify the folder. This document provides scripts to manage this, following the appropriate approach depending on the scenario:

   • If the system connects to a Oracle Base Database Service or Oracle Exadata Database Service, then you can just update the database connection string in the tnsnames.ora file of the secondary mid-tier system during the switchover and failover operations. This document provides an example script for this.
   • If the system connects to an Autonomous Database, then the TNS admin folder contains more artifacts (a trust store and a keystore). They are different in primary and standby, and they can’t be updated with a simple string replacement. This document provides an script that restores the backup copy of the TNS folder.

   At this point, you only need to perform a backup of the TNS folder information.

2. Identify the OCI File Storage file systems’ information on the primary site.
   • For the OCI File Storage file systems that are going to be replicated, identify the names, mount targets, exports, and mount points of the primary mid-tier hosts.
   • Go to the OCI Console, select your primary region, then choose your compartment.
   • Navigate to Storage, File Storage, then File Systems and identify the file systems.
   • Save the name, the export, the mount target, and AD where they are located.

   Identify which host mounts the exports and the mount points by checking the /etc/fstab of the hosts.

   Tip:

   Oracle WebLogic Example

   For example, in an Oracle WebLogic Server system that follows the Enterprise Deployment Guide:

   OCI File System	Mount Target	Export Path	AD	Hosts and Mount Points
   configFS	mt1_region1	/exports/configFS	AD1	apphost1, /u01/oracle/config apphost2, /u01/oracle/config
   runtimeFS	mt1_region1	/exports/runtimeFS	AD1	apphost1, /u01/oracle/runtime apphost2,/u01/oracle/runtime

3. Identify the OCI File Storage file systems’ information on the secondary site.
   Repeat the steps described in the previous step to gather the same information on the secondary site.

   Tip:

   Oracle WebLogic Example

   For example, in a WebLogic system that follows the Enterprise Deployment Guide:

   OCI File System	Mount Target	Export Path	AD	Hosts and Mount Points
   configFS	mt1_region2	/exports/configFS	AD1	apphost1, /u01/oracle/config apphost2, /u01/oracle/config
   runtimeFS	mt1_region2	/exports/runtimeFS	AD1	apphost1, /u01/oracle/runtime apphost2, /u01/oracle/runtime

4. Unmount the original OCI File Storage file systems from the secondary mid-tier hosts.
   For each mid-tier host in the secondary, unmount the file systems that will be replicated from the primary. For example:

   [opc@host ~]$ sudo umount  /u01/oracle/config
   [opc@host ~]$ sudo umount  /u01/oracle/runtime

   Ensure that no oracle processes are running; otherwise, the unmount will fail. Repeat these steps in all the mid-tier nodes in the secondary.

   Do not remove the entries from the /etc/fstab file for these mounts. If you always use the same values for the mount target and export names for the replicated file system, the entries will be valid throughout the entire lifecycle.

5. Delete or rename the original OCI File Storage file systems in the secondary.
   Only a file system that has never been exported can be set as the target file system for the OCI File Storage replication. Hence, the original file systems mounted on the secondary mid-tier hosts can’t be used as the replication target. They will no longer be used; delete them now (or rename and delete later), by removing the export and terminating the file system.

   Note:

   Do NOT delete the mount targets. They will be used to export the replicated file systems.
6. Enable the replica in the primary file systems.
   In primary, enable the replica for each OCI File Storage file system that must be replicated.
   1. Go to OCI Console, select your primary region, and choose the compartment.
   2. Select Storage, File Storage, then File Systems.
   3. Click on the file system name, navigate to Replications, and click Create Replication.
      Provide a name for the replication.
   4. Select Create new target File System, and provide the following details:
      • Name: the name for the file system replica that will be created in the secondary region. Use a name that identifies it clearly as a replica, for example: configFS_replica.
      • Target region: the region of the secondary system.
      • Availability Domain: the availability domain for the target file system. It must be the same as the mount target that will export it.
      • Compartment: the compartment for the target file system.
      • Replication interval: Interval in minutes that determines the frequency of data replication.

   Note:

   Alternatively, you can create the target file systems in advance in secondary and provide the OCID here.
7. If required, prepare the scripts to replace the information specific to each site.

   This action applies only when the OCI File Storage file system contain information specific to each site. Otherwise, no action is required.

   Create scripts to replace the local site information, according to your specific requirements (for example, performing a search and replace, or restoring a backup copy of the site-specific data). Make sure you store these scripts in a folder that is NOT replicated.

   IMPORTANT! Do not run the scripts at this point. The script will be used next time that a validation, a switchover or a failover is performed.

   Tip:

   Oracle WebLogic Example

   For example, when you replicate a file system that contains an Oracle WebLogic domain. During a switchover or failover, you must perform a replacement on the replicated configuration to point to the local database. This document provides example scripts to automate this replacement.

   Database Type	Replacement Script and Download Steps	Prepare Steps
   Oracle Base Database Service or Oracle Exadata Database Service	replacement_script_BVmodel.sh Go to the Oracle MAA repository in GitHub at https://github.com/oracle-samples/maa Download all the scripts in the wls_mp_dr directory. The script is located in the folder wls_mp_dr/Block_Volume_Replica_Method Copy to all the mid-tier hosts.	This script replaces the database connection strings. It also cleans up the state files of the WebLogic servers (.lck and .state) for a clean startup. Edit and customize it in each host with the appropriate values, by providing the local and remote values for the database in each site. Note that the values are different depending on the site. When you customize it in the site1 hosts, the “LOCAL” values refer to the site1’s values, and the “REMOTE” values refer to the site2's values. When you customize the script in the site2 hosts, the “LOCAL” values refer to the site2 and the “REMOTE” values to the site1. Go to the Oracle MAA repository in GitHub https://github.com/oracle-samples/maa Download all the scripts in the app_dr_common directory. Download all the scripts in the fmw-wls-with-adb-dr directory. Copy to all the mid-tier hosts. The scripts make calls to each other. Place all the scripts of both directories in the same folder.
   Oracle Autonomous Database	fmwadb_switch_db_conn.sh Go to the Oracle MAA repository in GitHub https://github.com/oracle-samples/maa Download all the scripts in the app_dr_common directory. Download all the scripts in the fmw-wls-with-adb-dr directory. Copy to all the mid-tier hosts. The scripts make calls to each other. Place all the scripts of both directories in the same folder.	This script replaces the TNS admin folder used by Oracle WebLogic Server with the one given as input. It also updates the wallet password properties in the data sources. You don’t need to edit the script. The values of the folder and the password will be passed as inputs. To run the script: ./fmwadb_switch_db_conn.sh WALLET_DIR WALLET_PASSWORD Where WALLET_DIR is a folder that contains the tnsnames.ora, keystore, and truststore files to connect to the local database. Ensure that the WALLET_DIR folder is not overridden in the replica. Do not run the script at this point.

OCI file system replication is now ready.

Validate Replication for OCI File Storage

In a switchover or failover operation, the replicated information must be available and usable in the standby site before the processes are started. This is also required when you validate the secondary system (by opening the standby database in snapshot mode).

To make the replicated OCI File Storage file systems available and usable in the standby system, follow these actions for each file system.

Perform the following steps to use the replicated file systems in standby:

1. Create a clone of the target file system.
   A target file system can’t be mounted directly, you must clone it first.
   1. In secondary, navigate to Storage, File Storage, and then File Systems.
   2. Click the target file system name.
   3. In the Replication section of File System Information, click the Replication Target name link.
   4. Click the Last Snapshot name link.
   5. Click Clone to create a regular file system from this snapshot.
   6. Edit the details to provide a name for the clone.
      For consistency, use the same name as in primary, for example, configFS.
2. Create an export for the cloned file system.
   1. In the cloned file system, navigate to Exports.
   2. Select the mount target in the secondary.
   3. Select the export name.
      To facilitate switchover management, use the same name as the export in the primary. For example: /exports/configFS.
3. Mount the file system from the standby hosts.
   1. If you always use the same export name and the same mount target for the file system, then the entry in the /etc/fstab file for the mount doesn’t change during the lifecycle.
   2. If you don't use the same export name and mount target for the file system, then you must edit the /etc/fstab file and modify the entry in every switchover, failover, and validation.
      The following is an example of the of /etc/fstab entry:

      10.1.80.131:/exports/configFS    /u01/oracle/config  nfs  defaults,nofail,nosuid,resvport 0 0

   3. Once the /etc/fstab file contains the appropriate mount entry, mount the file system in the host.
      For example:

      [opc@host opc]# sudo mount -a

   4. Repeat in all the standby hosts that mount the file system.
4. Run the replacement script on all the standby mid-tier hosts to replace the site-specific information in the secondary mid-tier hosts.

   Tip:

   Oracle WebLogic Example

   For example, in a file system that contains the Oracle WebLogic domain: update the database connection information to point to the local database by running the replacement script on all the standby mid-tier hosts:

   • If the system uses Oracle Base Database Service or Oracle Exadata Database Service, then the script is replacement_script_BVmodel.sh. Make sure it uses the appropriate values.
   • If the system uses Oracle Autonomous Database, then the script is fmwadb_switch_db_conn.sh. It requires, as inputs, the path where the secondary original wallet is and the wallet password.
5. Clean up the servers’ lock files.

   The replicated file system may contain lock files of the mid-tier process, because the replica runs while the primary processes are up. Before starting the processes in secondary, you may need to clean up these files. Otherwise, they can prevent the mid-tier processes from starting.

   Tip:

   Oracle WebLogic Example

   For example, in a file system that contains an Oracle WebLogic domain: there may be .lck, .pid, or .state files in the ${DOMAIN_HOME}/servers/*/data/nodemanager folders carried from the primary. Make sure that these files are cleaned up before trying to start the node manager and the servers. For example:

   rm -f ${DOMAIN_HOME}/servers/*/data/nodemanager/*.lck
   rm -f ${DOMAIN_HOME}/servers/*/data/nodemanager/*.state
   rm -f ${DOMAIN_HOME}/servers/*/data/nodemanager/*.pid

   You can include this action in the replacement scripts or as a previous step in the Oracle WebLogic start-up.

6. When the switchover or failover operation has finished, you must unmount and delete the OCI File Storage file systems in the site with the standby role. Perform the following steps to revert back to the standby role.
   This is also required when you have completed a validation on the standby site (by opening the standby database in snapshot mode) and want to revert it to the standby role.
   1. Unmount the OCI File Storage file system in the standby site that are replicated from the primary.
      For example:

      [opc@host opc]# sudo umount /u01/oracle/config

   2. Delete the unmounted file systems.
      Terminate the unmounted file systems in the standby site. They are no longer used.

Perform Ongoing Replication for OCI File Storage

Follow these recommendations for the ongoing replication when using this implementation.

• OCI automatically performs OCI File Storage replication in the background. The only thing you need to do during the lifecycle is ensure that the OCI File Storage file systems of the primary have the replica enabled.
• Consider using OCI Full Stack Disaster Recovery to automate the switchover and failover tasks. It provides the ability to run a switchover or failover plan with just one click using the OCI Console. It is very useful to simplify the execution of the tasks related to OCI File Storage replica.
• The replication feature is complementary to the snapshot feature, not a replacement. Ensure that you attach a snapshot policy for the OCI File Storage file systems as well. This will provide data protection in addition to the cross-region replica, allowing you to restore a file system to a point in time.
• Maintain the information that is specific to each site and keep it current up-to-date. For example, if the file system contains a folder with the artifacts to connect to an Autonomous Database, then maintain a backup copy of this folder. Ensure that you update the backup of the folder when you perform an update in the wallet. This way, it will be correctly restored in subsequent switchover and failovers.
• After a switchover or a failover operation, cleanup the unused file systems and change the replica direction. These actions are required to reverse the replica direction:
  1. Disable the previous replication from the previous primary and clean up (delete) the unused target file systems in the new primary.
  2. Enable the replica in the OCI File Storage file systems of the new primary.
  3. Delete the unused file systems in the new standby.