Note:

This tutorial requires access to Oracle Cloud. To sign up for a free account, see Get started with Oracle Cloud Infrastructure Free Tier.
It uses example values for Oracle Cloud Infrastructure credentials, tenancy, and compartments. When completing your lab, substitute these values with ones specific to your cloud environment.

Automate Cold Disaster Recovery for Oracle HeatWave MySQL using OCI Full Stack Disaster Recovery

Introduction

Oracle Cloud Infrastructure Full Stack Disaster Recovery (OCI Full Stack DR) orchestrates the transition of compute, database, and applications between OCI regions from around the globe with a single click. Customers can automate the steps needed to recover one or more business systems without redesigning or re-architecting existing infrastructure, databases, or applications and without needing specialized management or conversion servers.

Oracle HeatWave MySQL is a fully managed database service deployed within Oracle Cloud Infrastructure (OCI) that supports operators and developers looking to rapidly deploy secure, cloud-native applications. Oracle HeatWave MySQL is the only MySQL offering that includes the ability to use HeatWave, an integrated, high performance, in-memory query accelerator. HeatWave enables customers to run sophisticated analytics directly against their operational MySQL database, removing the requirement for complex, time-consuming, and expensive data migration and integration with a separate analytics database. HeatWave accelerates MySQL performance tremendously for analytics and mixed workloads. HeatWave is fully optimized for OCI and Oracle HeatWave MySQL is 100% built, managed, and supported by the OCI and MySQL engineering teams.

In this tutorial, you will learn how to automate a cold disaster recovery for Oracle HeatWave MySQL in OCI. It outlines the procedures for utilizing the OCI Full Stack DR service to manage the switchover and failover processes.

Note: This type of Disaster Recovery (DR) strategy relies on a Backup and Restore mechanism, making it best suited for non-critical applications where the business requirements for Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO) are not overly demanding.

Architecture Description

The architecture presented in this tutorial showcases a WordPress application running on OCI virtual machines (VMs) seamlessly integrated with Oracle HeatWave MySQL. Additionally, the setup leverages OCI File Storage service as a Network File System (NFS) share to store WordPress content. This shared storage is mounted to each VM, ensuring immediate and synchronized access to all new content across the environment.

An OCI Load Balancer is deployed within a public subnet to efficiently manage external user connections, distributing incoming traffic across the WordPress VMs.

Disaster Recovery Architecture Description

The DR strategy for the WordPress application VMs involves a comprehensive approach, including the complete replication of each boot volume of VM with volume group replication and the utilization of File Storage replication to ensure the synchronization of WordPress content.

A load balancer is pre-provisioned in the remote region, ensuring it is ready to handle traffic seamlessly when the WordPress VMs are transitioned to the remote region during a switchover or failover scenario.

For the Oracle HeatWave MySQL, backups are regularly copied to the remote region to ensure data protection and disaster recovery readiness. This process can be initiated from one application VM using a custom script, which can be scheduled through crontab for automated and consistent execution.

The following diagram illustrates the workflow for copying the latest available MySQL backup from the primary region to the remote region.

fsdr_mds_copy_restore_bkp-Logical_Workflow_Copy_Backup_to_Remote.png

The following crontab setup has been configured under the opc user on the WordPress Application VM1 to execute the mds_copy_bkp.py script every 4 hours. This ensures automatic backup synchronization to the standby region, contributing to improved disaster recovery:

15 */4 * * * /home/opc/mds_copy_restore_bkp/mds_copy_bkp.py mds01

This job runs the script at minute 15 past every 4th hour.

All the scripts are available on GitHub and are thoroughly detailed in the following sections.

Note: Make sure to schedule this script (or a similar one) as per your business requirements to regularly copy the MySQL backup to the remote region. Without this step, the restore process may fail due to the backup not being available in the secondary region.

How the Recovery Works?

After a planned switchover is executed, the roles will reverse: the primary workload will run in Region 2, while the standby will operate in Region 1. The architecture will appear as follows:

fsdr_mds_copy_restore_bkp-Physical_Switchover_Architecture.png

In the current setup, we leverage the OCI private DNS service to manage a DNS record that directs traffic to the active Oracle HeatWave MySQL endpoint. During the recovery process, this DNS record is updated through a custom script to reflect the new Oracle HeatWave MySQL, ensuring seamless failover and continuity of service.

The following diagram illustrates the workflow for restoring the most recent MySQL backup in the standby region.

fsdr_mds_copy_restore_bkp-Logical_Workflow_Restore_Backup_to_Remote.png

The recovery solution for this deployment requires OCI Full Stack DR to run a series of custom Python scripts during a recovery operation such as a failover or switchover. The scripts referenced in this tutorial are provided by the EMEA Cloud Architecture Specialists team and available here: full-stack-disaster-recovery, which is specifically tailored for this DR solution.

This tutorial explains how to download the scripts and how to use them in a later task.

Note: The following scripts are provided for generic guidance. You can either use your own scripts or customize the scripts according to your corporate policy and security requirements.

Definitions and Assumptions throughout the Tutorial

Regions:
- Region 1 is Dubai: Dubai will initially serve as the primary region. However, this role will transition to standby during the switchover process, which will be performed in the later tasks as part of the disaster recovery plan.
- Region 2 is Abu Dhabi: Abu Dhabi will initially function as the standby region. This role will later transition to primary following the switchover process, which will be carried out in the subsequent tasks as part of the disaster recovery procedure.
Compartments: You are free to organize this deployment and OCI Full Stack DR into any compartment scheme that works within your standards for IT governance. We have chosen to organize all the OCI resources for this tutorial in one single compartment.

WordPress Application VMs and OCI File Storage

The architecture utilized in this tutorial is designed around OCI File Storage to ensure that all application VMs have shared access to the same WordPress content.

For more information about how to correctly mount the file system on the Linux instances, see Configuring a File System to Automatically Mount (Linux Instances).

The following is an example of the /etc/fstab file configuration for mounting the file system on the WordPress application VM.

xxx.xxx.xxx.xxx:/wordpress /var/www/html nfs nfsvers=3,noacl,nosuid,nofail 0 0

Replace xxx.xxx.xxx.xxx with the IP address of the mount target located in Region 1 to ensure proper connectivity.

Objectives

The following tasks will be covered in this tutorial:

Task 1: Prepare the environment for Disaster Recovery
Task 2: Create DR Protection Groups (DRPG) in both Regions
Task 3: Add Members to the DR Protection Group
Task 4: Create Basic DR Plans in Region 2
Task 5: Customize the Switchover Plan in Region 2
Task 6: Customize the Failover Plan in Region 2
Task 7: Run the Prechecks of the DR plans in Region 2
Task 8: Run the switchover plan in Region 2
Task 9: Create and Customize the DR plans in Region 1

Task 1: Prepare the Environment for Disaster Recovery

Task 1.1: Create Volume Groups and Enable Replication

Create a volume group for the WordPress application VMs in Region 1 and ensure it is replicated in Region 2. Ensure the boot volume for each application VM (WordPress) is a member of the volume group and the volume group is replicated to Region 2.

The following image illustrates displaying the created volume group, which includes the boot volumes of the two WordPress application VMs, with replication successfully enabled to Region 2. For more information, see Creating a Volume Group.

Task 1.2: Enable File Storage Replication for WordPress Content

In Region 2, create a file system specifically designated for replication. This file system will be used to seamlessly replicate data from Region 1 to Region 2.
In Region 2, create a mount target that will be used during the recovery process from Region 1 to Region 2.
Use the file system created in step 1 to enable and configure replication from the source OCI File Storage in Region 1.

The following image illustrates the File Storage replication details to Region 2.

For more information, see File System Replication.

Task 1.3: Prepare Application VMs to Run Custom Automation

Install and configure the Oracle Cloud Infrastructure Command Line Interface (OCI CLI). For this tutorial, Oracle Linux 8 is used for the WordPress application VM. For more information, see Installing the CLI.
Deploy the script from the GitHub repository (mds_colddr_scripts) in /home/opc folder.
Install the pandas and tabulate modules used by of the provided script.
```
pip install pandas
pip install tabulate
```
Update the config.csv file with the necessary details for the Oracle HeatWave MySQL in Region 1.
- Replace MYSQL_DB_SYSTEM_OCID with the OCID of the Oracle HeatWave MySQL. You can keep or customize the MySQL label to align with your specific requirements.
- Replace COMPARTMENT_OCID with the OCID of the compartment where the MySQL system is located.
- Replace PRIMARY_SUBNET_OCID and STANDBY_SUBNET_OCID with the subnet OCIDs in both regions.
- Replace PRIMARY_DNS_VIEW_OCID and STANDBY_DNS_VIEW_OCID with the OCIDs of the private DNS views associated with the private DNS zone managing the Oracle HeatWave MySQL endpoint record.
Note: Ensure all values are accurately updated.

As part of the tutorial, we will use this same VM for running user-defined scripts. Ensure the VM acting as the DR Control Node has been configured to run commands. For more information, see Invoke custom scripts using the run command with Oracle Cloud Infrastructure Full Stack Disaster Recovery.

Task 1.4: Create OCI IAM Policies for OCI Full Stack DR

Configure the required OCI IAM policies for OCI Full Stack DR as outlined in the following documents.

Task 1.5: Create OCI IAM Policies for other Services managed by OCI Full Stack DR

OCI Full Stack DR must have the ability to control and manage other key OCI services such as compute, networking, storage and other miscellaneous services. To configure the required OCI IAM policies for other services, see Policies for Other Services Managed by Full Stack Disaster Recovery and OCI IAM policies.

Task 2: Create DR Protection Groups (DRPG) in both Regions

Create DR protection groups in Region 1 and Region 2 if the protection groups for this application stack do not exist yet.

Task 2.1: Create a Protection Group in Region 1

Go to the OCI Console and navigate to DR Protection Groups as shown in Figure 2.1.
1. Ensure the OCI region context is set to Region 1 (Dubai).
2. Click Migration & Disaster Recovery.
3. Click DR Protection Groups.
Figure 2.1: Navigate to DR protection groups
Create a basic DR protection group (DRPG) in Region 1 as shown in Figure 2.2. The peer, role and members will be assigned in later steps.
1. Select the compartment where you want the DRPG to be created.
2. Click Create DR protection group to open the dialog.
3. Use a meaningful name for the DRPG.
4. Select OCI Object Storage bucket for OCI Full Stack DR logs.
5. Click Create.
Figure 2.2: Parameters needed to create DR protection group in region 1

Task 2.2: Create a Protection Group in Region 2

Go to the OCI Console, navigate to DR Protection Groups as shown in Figure 2.3.
1. Ensure the OCI region context is set to Region 2 (Abu Dhabi).
2. Click Migration & Disaster Recovery.
3. Click DR Protection Groups.
Figure 2.3: Navigate to DR protection groups
Create a basic DR protection group (DRPG) in Region 2 as shown in Figure 2.4. The peer, role and members will be assigned in later steps.
1. Select the compartment where you want the DRPG to be created.
2. Click Create DR protection group to open the dialog.
3. Use a meaningful name for the DRPG.
4. Select OCI Object Storage bucket for OCI Full Stack DR logs.
5. Click Create.
Figure 2.4: Parameters needed to create DR protection group in region 2

Task 2.3: Associate Protection Groups in Region 1 and Region 2

Associate the DRPGs in each region as peers of each other and assign the peer roles of primary and standby. The roles of primary and standby are automatically changed by OCI Full Stack DR as part of any DR operation/DR plan execution; there is no need to manage the roles manually at any time.

Go to the DR protection group details page.
1. Ensure OCI region context is set to Region 1 (Dubai).
2. Click Associate to begin the process.
Fig: Begin DRPG association
Enter the parameters as shown in the following image.
1. Role: Select Primary role. OCI Full Stack DR will assign the standby role to Region 2 automatically.
2. Peer region: Select Region 2 (Abu Dhabi), where the other DRPG was created.
3. Peer DR protection group: Select the peer DRPG that was created.
4. Click Associate.
Fig: Parameters needed to associate the DRPGs

OCI Full Stack DR will show something like as shown in the following image, once the association is completed.

The current primary peer DRPG is Dubai (region 1).
The current standby peer DRPG is Abu Dhabi (region 2).

Fig: Showing the peer relationship from the individual DRPG perspective

The same information can be found whenever the context/view is from a global perspective showing all DR protection groups as shown in the following image.

The current primary peer DRPG is Dubai (region 1).
The current standby peer DRPG is Abu Dhabi (region 2).

Fig: Showing the peer relationship from the global DRPG perspective

Task 3: Add Members to the DR Protection Group

In this task, we will add the following OCI resources to the primary DRPG in Region 1.

The two compute instances hosting the WordPress application will be added as moving VMs. Additionally, ensure that the File Systems section in the Advanced Options is properly configured.
The volume group that contains the boot volume of the WordPress application compute nodes.
The OCI File Storage containing the WordPress content.
The primary load balancer.

Task 3.1: Add Members to DRPG in Region 1

Select the DRPG in Region 1 as shown in the following image.
1. Ensure the OCI region context is Region 1 (Dubai).
2. Select the DRPG in Region 1.
3. Select Members.
4. Click Add Member to begin the process.
Fig: How to begin adding members to DR protection group in region 1
Add a compute instance for the WordPress application VMs.
1. Acknowledge warning about DR plans.
2. Enter Compute as a member Resource type.
3. Select the compute instance hosting the WordPress application.
4. Select Moving instance.
5. Click Add VNIC mapping to select which VCN and subnet to assign to the VNIC(s) at Region 2 during a recovery.
6. Click Show advanced options.
7. In Settings, select Retain fault domain.
8. In File Systems, complete the export mapping section according to your setup as shown in the following images.
Fig: Parameters needed to add WordPress Application VM

Fig: Parameters needed to map the VNIC in region 2

Fig: Advanced options, Retain fault domain

Fig: Advanced options, File Systems mapping

Fig: Compute Instance Added to the DRPG in Region 1

Note: Repeat the previous sub-steps for all the compute instances for the WordPress application VMs.

Fig: Two Compute Instances Added to the DRPG in Region 1
Add the block volume group containing boot volume of the WordPress application VMs.
1. Acknowledge warning about DR plans.
2. Select Volume group as member Resource type.
3. Ensure the correct compartment containing the volume group is selected and select the volume group.
Fig: Parameters needed to add the Volume Group for WordPress Compute

Fig: Volume Group for WordPress Compute Added to the DRPG in Region 1
Add the OCI File Storage that contains the WordPress content.
1. Acknowledge warning about DR plans.
2. Select File System as member Resource type.
3. Ensure the correct compartment containing the file system is selected and select the file system.
4. Select Destination availability domain to be used in Region 2.
5. Select Source export path for this file system. This export path is created in the destination Region 2 after restoring the file system.
6. Select Destination mount target in Region 2 used to create an export for the restored file system.
Fig: Parameters needed to add the File System for WordPress content

Fig: File System for WordPress content Added to the DRPG in Region 1
Add OCI Load Balancer.

In this example, we will add the load balancer as a member of the DRPG in Region 1.
1. Acknowledge warning about DR plans.
2. Select Load Balancer as member Resource type.
3. Ensure correct compartments for load balancer is selected and select the load balancer you want to add.
4. Select Destination load balancer to be used in Region 2.
5. Select Source Backend Set, this is the backend set used by the WordPress application VMs. An OCI Load Balancer can be shared among multiple applications and may have multiple backend sets configured. During a DR switchover, only the backend sets specified here will have their configuration moved to the standby region.
6. Select Destination Backend Set, this is the empty backend set created in Region 2.
Fig: Parameters needed to add Load Balancer

Fig: Load Balancer Added to the DRPG in Region 1

Task 3.2: Add Members to DRPG in Region 2

Select the DRPG in Region 2 as shown in the following image.
1. Ensure the OCI region context is Region 1 (Abu Dhabi).
2. Select the DRPG in Region 2.
3. Select Members.
4. Click Add Member to begin the process.
Fig: How to begin adding members to DR protection group in region 2
Add OCI Load Balancer.

In this example, we will add the load balancer as a member of the DRPG in Region 2.
1. Acknowledge warning about DR plans.
2. Select Load Balancer as member Resource type.
3. Ensure correct compartment for load balancer is selected and select the load balancer you want to add.
4. Select the Destination load balancer to be used in Region 1.
5. Select Source Backend Set, this is the backend set used by the WordPress application VMs. An OCI Load Balancer can be shared among multiple applications and may have multiple backend sets configured. During a DR switchover, only the backend sets specified here will have their configuration moved to the standby region.
6. Select Destination Backend Set, this backend set is created in Region 2.
Fig: Parameters needed to add Load Balancer

Fig: Load Balancer Added to the DRPG in Region 2

Task 4: Create Basic DR Plans in Region 2

In this task, we will create the initial switchover and failover plans associated with the standby DR protection group in Region 2 (Abu Dhabi).

The purpose of these plans is to seamlessly transition the workload from the primary region (Region 1) to the standby region (Region 2). As part of any DR operation, the roles of the DR protection groups in both regions are automatically reversed: the protection group in Region 1 becomes the standby, while the protection group in Region 2 assumes the primary role following a failover or switchover.

OCI Full Stack DR will pre-populate these plans with built-in steps derived from the member resources added during the previous tasks. These plans will later be customized to manage tasks specific to Oracle HeatWave MySQL during the recovery process.

Switchover plans are always created within the protection group holding the standby role. Since Region 2 (Abu Dhabi) is currently the standby protection group, we will begin creating the plans there.

Task 4.1: Create DR Plans

Create a basic plan by selecting the DRPG in Region 2 (Abu Dhabi)
1. Ensure the OCI region context is Region 2 (Abu Dhabi).
2. Select the standby DRPG in Region 2.
3. Select Plans.
4. Click Create Plan to begin the process.
Fig: How to begin creating basic DR plans in Region 2
Create a switchover plan.
1. Enter a simple but meaningful Name for the switchover plan. The name should be as short as possible but easy to understand at a glance to help reduce confusion and human error during a crisis.
2. Select Plan type as Switchover (planned).
Fig: The parameters needed to create DR switchover plan
Create a failover plan.

Follow the same process to create a basic failover plan as shown in the following image.
1. Enter Name of the failover plan simple but meaningful. The name should be as short as possible but easy to understand at a glance to help reduce confusion and human error during a crisis.
2. Select Plan type as Failover (unplanned).
Fig: The parameters needed to create DR failover plan

The standby DR protection group in Region 2 should now have the two DR plans as shown in the following image. These will handle transitioning workloads from Region 1 to Region 2. You will create similar plans at Region 1 to transition workloads from Region 2 back to Region 1 in a later task.

Fig: Showing the two basic DR plans that must exist in region 2 before proceeding any further

Task 5: Customize the Switchover Plan in Region 2

The basic DR plans created in Task 4 contain pre-populated steps for recovery tasks that are built into OCI Full Stack DR and do not contain anything to manage recovery tasks specific to Oracle HeatWave MySQL. This task explains how to add custom, user-defined DR plan groups and steps to manage the tasks that need to be accomplished during a switchover:

Create a database backup after gracefully shutting down the application VMs to ensure data consistency.
Transfer the latest database backup to the remote OCI Region 2 for redundancy and disaster recovery.
Restore the most recent database backup in Region 2 to prepare the environment for switchover.
Update the private database DNS record, enabling the application VMs to seamlessly connect to the database in Region 2.
Terminate the source MySQL database in Region 1.

Task 5.1: Select the Switchover Plan

Navigate to the switchover plan created in Task 4.

Fig: How to begin customizing the switchover plan in region 2

Task 5.2: (Optional) Enable DR Plan Groups that Terminate Artifacts

There are three plan groups that are disabled by default in the switchover plans, as shown in the following image. These plan groups are disabled to provide reassurance during testing, ensuring that no artifacts are deleted and that a viable backup copy remains intact in case of issues during the test phase.

However, these three plan groups are designed to terminate (delete) artifacts that will no longer be needed for any future DR operations. Without these plan groups enabled, unused artifacts will continue to accumulate over time as you perform switchovers between the two regions, which can lead to confusion about which compute instances, OCI File Storage and volume groups should be active.

Optionally, enabling these plan groups now will help avoid the need for manual clean up of unnecessary artifacts before going into production. This proactive step can streamline the transition to production and maintain a cleaner, more manageable environment.

Fig: Plan groups disabled by default

To enable the plan groups, select Enable all steps from the context menu to the right of the plan group name.

Fig: How to enable terminate File System

Fig: Click on Enable to validate.

Fig: How to enable terminate compute instances

Fig: Click on Enable to validate.

Fig: How to enable terminate Volume Group

Fig: Click on Enable to validate.

Task 5.3: Reorder DR Plan Groups

It is necessary to mount the file system on the new VMs moved to Region 2 before updating the load balancer backend sets. This ensures that the application has proper access to its required storage resources, facilitating a smooth transition during the switchover process.

Fig: Screenshot showing the initial Switchover plan order and how to start reordering

Fig: Start the reordering

Move File Systems - Mount on Compute Instances before Load Balancers - Update Destination Backend Sets.
Click Save changes.

Fig: Final Switchover plan order

Task 5.4: Create a Plan group to Run Custom Scripts

Start by adding custom, user-defined DR plan groups to tailor the DR workflow to the specific needs of the MySQL Backup/Restore DR process.

These plan groups will invoke the necessary scripts from the WordPress VM1 in Region 1 and should be positioned in the DR workflow prior to starting the WordPress application VMs. This ensures that the MySQL database is fully restored and operational before the application VMs are brought online.

The following custom plan should be added to the pre-populated switchover plan: Create MySQL Database Backup, Copy MySQL Database Backup, Restore MySQL Database Backup, Update MySQL Database DNS and Terminate MySQL Database Source.

Add a custom plan group to create the MySQL database backup.
1. Click Add group.
2. Enter a simple but descriptive plan group Name. In this example, we will use Create MySQL DB Backup.
3. Select a position where the plan group will be inserted into the DR plan. In this example, select Add after to insert our user-defined plan group after the built-in plan group Compute Instances - Launch.
4. Click Add Step to open the dialog where we will specify the script to create the MySQL database backup.
Fig: Parameters to create plan group Create MySQL DB Backup

In Add plan group step, enter the following information.
1. Enter a simple but descriptive Step name. In this example, we will use Create MySQL DB Backup. Same as the plan group name.
2. Select a region which contains the instance on which this step will run. In this example, the script will be executed on the WordPress application VM1 in Region 1.
3. Select Run local script.
4. Select Target instance which is the WordPress application VM1 in Region 1.
5. In Script parameters, enter the full path of the script with the parameters. For example: /home/opc/mds_copy_restore_bkp/mds_create_bkp.py mds01.
6. In Run as user, enter opc.
7. Click Add Step.
Fig: Parameters to create Add a Step Create MySQL DB Backup

Click Add.

Fig: Add a plan group Create MySQL DB Backup
Add a custom plan group to copy the MySQL database backup.
1. Click Add group.
2. Enter simple but descriptive plan group Name. In this example, we will use Copy MySQL DB Backup.
3. Select a position where the plan group will be inserted into the DR plan. In this example, select Add after to insert our user-defined plan group after the built-in plan group Create MySQL DB Backup.
4. Click Add Step to open the dialog where we will specify the script to copy the MySQL database Backup.
Fig: Parameters to create plan group Copy MySQL DB Backup

In Add plan group step, enter the following information.
1. Enter a simple but descriptive Step name. In this example, we will use Copy MySQL DB Backup. Same as the plan group name.
2. Select a region which contains the instance on which this step will run. In this example, the script will be executed on the WordPress application VM1 in Region 1.
3. Select Run local script.
4. Select the Target instance which is the WordPress application VM1 in Region 1.
5. In Script parameters, enter the full path of the script with the parameters. For example: /home/opc/mds_copy_restore_bkp/mds_copy_bkp.py mds01.
6. In Run as user, enter opc.
7. Click Add Step.
Fig: Parameters to Add a Step Copy MySQL DB Backup

Click Add.

Fig: Add a plan group Copy MySQL DB Backup
Add custom plan group to restore the MySQL database backup.
1. Click Add group.
2. Enter a simple but descriptive plan group Name. In this example, we will use Restore MySQL DB Backup.
3. Select a position where the plan group will be inserted into the DR plan. In this example, select Add after to insert our user-defined plan group after the built-in plan group Copy MySQL DB Backup.
4. Click Add Step to open the dialog where we will specify the script to restore the MySQL database backup.
Fig: Parameters to create plan group Restore MySQL DB Backup

In Add plan group step, enter the following information.
1. Enter a simple but descriptive Step name. In this example, we will use Restore MySQL DB Backup. Same as the plan group name.
2. Select a region which contains the instance on which this step will run. In this example, the script will be executed on the WordPress application VM1 in Region 1.
3. Select Run local script.
4. Select the Target instance which is the WordPress application VM1 in Region 1.
5. In Script parameters, enter the full path of the script with the parameters. For example: /home/opc/mds_copy_restore_bkp/mds_restore_bkp.py mds01 1 --config --switch.
6. In Run as user, enter opc.
7. Click Add Step.
Fig: Parameters to Add a Step Restore MySQL DB Backup

Click Add.

Fig: Add a plan group Restore MySQL DB Backup
Add custom plan group to update the MySQL database DNS.
1. Click Add group.
2. Enter simple but descriptive plan group Name. In this example, we will use Update MySQL DB DNS.
3. Select a position where the plan group will be inserted into the DR plan. In this example, select Add after to insert our user-defined plan group after the built-in plan group Restore MySQL DB Backup.
4. Click Add Step to open the dialog where we will specify the script to update the MySQL database DNS.
Fig: Parameters to create plan group Update MySQL DB DNS

In Add plan group step, enter the following information.
1. Enter a simple but descriptive Step name. In this example, we will use Update MySQL DB DNS. Same as the plan group name.
2. Select a region which contains the instance on which this step will run. In this example, the script will be executed on the WordPress application VM1 in Region 1.
3. Select Run local script.
4. Select the Target instance which is the WordPress application VM1 in Region 1.
5. In Script parameters, enter the full path of the script with the parameters. For example: /home/opc/mds_copy_restore_bkp/mds_update_dns.py mds01 demo.local mds01.demo.local --remote.
6. In Run as user, enter opc.
7. Click Add Step.
Fig: Parameters to Add a Step Update MySQL DB DNS

Click Add.

Fig: Add a plan group Update MySQL DB DNS
Add a custom plan group to terminate the MySQL source database.
1. Click Add group.
2. Enter a simple but descriptive plan group Name. In this example, we will use Terminate MySQL DB Source.
3. Select a position where the plan group will be inserted into the DR plan. In this example, select Add after to insert our user-defined plan group after the built-in plan group File Systems - Remove from DR Protection Group.
4. Click Add Step to open the dialog where we will specify the script to terminate the MySQL database source.
Fig: Parameters to create plan group Terminate MySQL Source DB

In Add plan group step, enter the following information.
1. Enter a simple but descriptive Step name. In this example, we will use Terminate MySQL Source DB. Same as the plan group name.
2. Select a region which contains the instance on which this step will run. In this example, the script will be executed on the WordPress application VM1 in Region 1.
3. Select Run local script.
4. Select the Target instance which is the WordPress application VM1 in Region 1.
5. In Script parameters, enter the full path of the script with the parameters. For example: /home/opc/mds_copy_restore_bkp/mds_terminate_db.py mds01.
6. In Run as user, enter opc.
7. Click Add Step.
Fig: Parameters to Add a Step Terminate MySQL Source DB

Click Add.

Fig: Add a plan group Terminate MySQL Source DB
(Optional) Add custom plan group to stop WordPress application.
1. Click Add group.
2. Enter a simple but descriptive plan group Name. In this example, we will use Application - Stop.
3. Select a position where the plan group will be inserted into the DR plan. In this example, select Add after to insert our user-defined plan group at the end after the built-in plan group Load Balancers - Update Source Backend Sets.
4. Click Add Step to open the dialog where we will specify the script to stop the application.
Fig: Parameters to create plan group Application - Stop

In Add plan group step, enter the following information.
1. Enter a simple but descriptive Step name. In this example, we will use Application - Stop - VM1.
2. Select a region which contains the instance on which this step will run. In this example, the script will be executed on the WordPress application VM1 in Region 1.
3. Select Run local script.
4. Select the Target instance which is the WordPress application VM1 in Region 1.
5. In Script parameters, enter the full path of the script with the parameters. For example: /home/opc/fsdrscripts/wdp_stop.sh.
6. In Run as user, enter opc.
7. Click Add Step.
Fig: Parameters to create a Step Application - Stop - VM1

Click Add Step to add a second step to stop the application on VM2.

Fig: Add a Step Application - Stop - VM2

In Add plan group step, enter the following information.
1. Enter a simple but descriptive Step name. In this example, we will use Application - Stop - VM2.
2. Select a region which contains the instance on which this step will run. In this example, the script will be executed on the WordPress application VM2 in Region 1.
3. Select Run local script.
4. Select the Target instance which is the WordPress application VM2 in Region 1.
5. In Script parameters, enter the full path of the script with the parameters. For example: /home/opc/fsdrscripts/wdp_stop.sh.
6. In Run as user, enter opc.
7. Click Add Step.
Fig: Parameters to create A a Step Application - Stop - VM2

Click Add.

Fig: Add a plan group Application - Stop
(Optional) Add custom plan group to start the WordPress application.
1. Click Add group.
2. Enter a simple but descriptive plan group Name. In this example, we will use Application - Start.
3. Select a position where the plan group will be inserted into the DR plan. In this case, select Add After to insert our user-defined plan group at the end, after the built-in plan group File Systems - Mount on Compute Instances.
4. Click Add Step to open the dialog where we will specify the script to start the application.
Fig: Parameters to create plan group Application - Start

In Add plan group step, enter the following information.
1. Enter a simple but descriptive Step name. In this example, we will use Application - Start - VM1.
2. Select a region which contains the instance on which this step will run. In this case the script will be executed on the WordPress Application VM1 in Region 1.
3. Select Run local script.
4. Select the Target instance which is the WordPress application VM1 in Region 1.
5. In Script parameters, enter the full path of the script with the parameters. For example: /home/opc/fsdrscripts/wdp_start.sh.
6. In Run as user, enter opc.
7. Click Add Step.
Fig: Parameters to create a Step Application - Start - VM1

Click Add Step to add a second step to start the application on VM2.

Fig: Add a Step Application - Start - VM2

In Add plan group step, enter the following information.
1. Enter a simple but descriptive Step name. In this example, we will use Application - Start - VM2.
2. Select a region which contains the instance on which this step will run. In this case the script will be executed on the WordPress Application VM2 in Region 1.
3. Select Run local script.
4. Select the Target instance which is the WordPress application VM2 in Region 1.
5. In Script parameters, enter the full path of the script with the parameters. For example: /home/opc/fsdrscripts/wdp_start.sh.
6. In Run as user, enter opc.
7. Click Add Step.
Fig: Parameters to create A a Step Application - Start - VM2

Click Add.

Fig: Add a plan group Application - Start

The customization of the switchover plan has been successfully completed.

Task 6: Customize the Failover Plan in Region 2

In this task, add custom, user-defined DR plan groups and steps to manage the tasks that need to be accomplished during a failover.

Restore the most recent database backup in Region 2 to prepare the environment for switchover.
Update the private database DNS record, enabling the application VMs to seamlessly connect to the database in Region 2.

Task 6.1: Select the Failover Plan

Navigate to the failover plan created in Task 4.

Fig: How to begin customizing the Failover plan in region 2

Task 6.2: Create a Plan group to Run Custom Scripts at Region 2

Start by adding custom, user-defined DR plan groups to tailor the DR workflow to the specific needs of the MySQL Backup/Restore DR process.

These plan groups will invoke the necessary scripts from the WordPress application VM and should be positioned in the DR workflow prior to restarting the WordPress application VMs. This ensures that the MySQL database is fully restored and operational before the application VMs are brought online.

The following custom plan should be added to the pre-populated failover plan: Restore MySQL Database Backup and Update MySQL Database DNS.

Add custom plan group to restore MySQL database backup.
1. Click Add group to begin.
2. Enter a simple but descriptive plan group Name. In this example, we will use Restore MySQL DB Backup.
3. Select a position where the plan group will be inserted into the DR plan. In this example, select Add after to insert our user-defined plan group after the built-in plan group Compute Instances - Launch.
4. Click Add Step to open the dialog where we will specify the script to restore the MySQL database backup.
Fig: Parameters to create plan group Restore MySQL DB Backup

In Add plan group step, enter the following information.
1. Enter a simple but descriptive Step name. In this example, we will use Restore MySQL DB Backup. Same as the plan group name.
2. Select a region which contains the instance on which this step will run. In this example, the script will be executed on the WordPress application VM1 in Region 1.
3. Select Run local script.
4. Select the Target instance which is the WordPress application VM1 in Region 1.
5. In Script parameters, enter the full path of the script with the parameters. For example: /home/opc/mds_copy_restore_bkp/mds_restore_bkp.py mds01 1 --config.
6. In Run as user, enter opc.
7. Click Add Step.
Fig: Parameters to Add a Step Restore MySQL DB Backup

Click Add

Fig: Add a plan group Restore MySQL DB Backup
Add custom plan group to update MySQL database DNS.
1. Click Add group to begin.
2. Enter a simple but descriptive plan group Name. In this example, we will use Update MySQL DB DNS.
3. Select a position where the plan group will be inserted into the DR plan. In this example, select Add after to insert our user-defined plan group after the built-in plan group Restore MySQL DB Backup.
4. Click Add Step to open the dialog where we will specify the script to update the MySQL database DNS.
Fig: Parameters to create plan group Update MySQL DB DNS

In Add plan group step, enter the following information.
1. Enter a simple but descriptive Step name. In this example, we will use Update MySQL DB DNS. Same as the plan group name.
2. Select a region which contains the instance on which this step will run. In this example, the script will be executed on The WordPress application VM1 in Region 1.
3. Select Run local script.
4. Select the Target instance which is the WordPress application VM1 in Region 1.
5. In Script parameters, enter the full path of the script with the parameters. For example: /home/opc/mds_copy_restore_bkp/mds_update_dns.py mds01 demo.local mds01.demo.local.
6. In Run as user, enter opc.
7. Click Add Step.
Fig: Parameters to create Add a Step Update MySQL DB DNS

Click Add.

Fig: Add a plan group Update MySQL DB DNS
(Optional) Add custom plan group to restart the WordPress application.
1. Click Add group.
2. Enter a simple but descriptive plan group Name. In this example, we will use Application - Restart.
3. Select a position where the plan group will be inserted into the DR plan. In this example, select Add after to insert our user-defined plan group after the built-in plan group File Systems - Mount on Compute Instances.
4. Click Add Step to open the dialog where we will specify the script to start the application.
Fig: Parameters to create plan group Application - Restart

In Add plan group step, enter the following information.
1. Enter a simple but descriptive Step name. In this example, we will use Application - Restart - VM1.
2. Select a region which contains the instance on which this step will run. In this example, the script will be executed on the WordPress application VM1 in Region 1.
3. Select Run local script.
4. Select the Target instance which is the WordPress application VM1 in Region 1.
5. In Script parameters, enter the full path of the script with the parameters. For example: /home/opc/fsdrscripts/wdp_restart.sh.
6. In Run as user, enter opc.
7. Click Add Step.
Fig: Parameters to create a Step Application - Restart - VM1

Click Add Step to add a second step to restart the application on VM2.

Fig: Add a Step Application - Restart - VM2

In Add plan group step, enter the following information.
1. Enter a simple but descriptive Step name. In this example, we will use Application - Start - VM2.
2. Select a region which contains the instance on which this step will run. In this example, the script will be executed on the WordPress application VM2 in Region 1.
3. Select Run local script.
4. Select the Target instance which is the WordPress application VM2 in Region 1.
5. In Script parameters, enter the full path of the script with the parameters. For example: /home/opc/fsdrscripts/wdp_restart.sh.
6. In Run as user. enter opc.
7. Click Add Step.
Fig: Parameters to create A a Step Application - Restart - VM2

Click Add.

Fig: Add a plan group Application - Restart

The customization of the failover plan has been successfully completed.

Task 7: Run Prechecks in Region 2

Both switchover and failover DR plans have been successfully created in the standby Region 2. These plans enable OCI Full Stack DR to transition workloads from Region 1 to Region 2. The subsequent task involves running the prechecks for both the switchover and failover plans to ensure readiness and validate the transition process.

Task 7.1: Begin Prechecks for the Switchover Plan

Run prechecks for the switchover DR plan.

Ensure the region context is set to standby Region 2.
Ensure the correct DR protection group in Region 2 is selected, it should be the standby role.
Click the switchover plan name.
Click Run prechecks.

Fig: Showing how to run prechecks of the switchover plan

Fig: Showing a Completed prechecks of the switchover plan

Task 7.2: Begin Prechecks for the Failover Plan

Run the prechecks for the failover DR plan.

Ensure the region context is set to standby region 2.
Ensure the correct DR protection group in Region 2 is selected, it should be the standby role.
Click the failover plan name.
Click Run prechecks.

Fig: Showing how to run prechecks of the failover plan

Fig: Showing a Completed prechecks of the failover plan

Task 8: Run the Switchover Plan in Region 2

Run the switchover DR plan to begin the process of transitioning the WordPress application with the Oracle HeatWave MySQL from Region 1 to Region 2.

Ensure the region context is set to standby Region 2.
Ensure the correct DR protection group in Region 2 is selected, it should be the standby role.
Click the switchover plan name.
Click Execute plan.

This task runs the switchover plan in Region 2.

Deselect Enable prechecks, since they were already executed in Task 7.
Click Execute DR plan to begin.

Fig: Showing how to Run the switchover plan

Fig: Showing the switchover plan in Progress

Monitor the switchover plan until the complete workload has been fully transitioned from Region 1 to Region 2.

The execution of the switchover plan was successfully completed in approximately 52 minutes.

Fig: Showing a Completed switchover plan execution.

Task 9: Create and Customize DR plans in Region 1

With the successful completion of the switchover by OCI Full Stack DR, Region 2 has now assumed the role of the primary region, while Region 1 has transitioned to serve as the standby region.

Follow the same approach detailed in Task 1 to 8, proceed to create and customize the switchover and failover plans within the DR protection group for Region 1, which now serves as the standby peer region.

Fig: Screenshot showing Created plans in Region 1

Fig: Screenshot showing Switchover plan in Region 1

Fig: Screenshot showing Failover plan in Region 1

Next Steps

For more information, see the OCI Full Stack DR and Oracle HeatWave MySQL documentation in the Related Links section.

Oracle Cloud Infrastructure (OCI) Full Stack Disaster Recovery
Oracle HeatWave MySQL
OCI Full Stack Disaster Recovery
HeatWave MySQL
Join #full-stack-dr slack channel

Acknowledgments

Author - Antoun Moubarak (Hyperscaler to OCI Specialist)
Contributor - Suraj Ramesh (Product Manager for OCI Full Stack DR)

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.

Title and Copyright Information

Automate Cold Disaster Recovery for Oracle HeatWave MySQL using OCI Full Stack Disaster Recovery

G22692-01

January 2025

Oracle and/or its affiliates.