Note:

• This tutorial requires access to Oracle Cloud. To sign up for a free account, see Get started with Oracle Cloud Infrastructure Free Tier.
• It uses example values for Oracle Cloud Infrastructure credentials, tenancy, and compartments. When completing your lab, substitute these values with ones specific to your cloud environment.

Automate Recovery for Oracle Integration using OCI Full Stack Disaster Recovery

Part 1: Introduction

Oracle Cloud Infrastructure Full Stack Disaster Recovery (OCI Full Stack DR) orchestrates the transition of compute, database, and applications between Oracle Cloud Infrastructure (OCI) regions from around the globe with a single-click. Customers can automate the steps needed to recover one or more business systems without redesigning or re-architecting existing infrastructure, databases, or applications.

Oracle Integration is a secure, unified platform that lets you connect cloud and on-premises applications, automate business processes, gain insight into your business through business metrics analysis, and develop web and mobile applications. Oracle Integration is available in an OCI region governed by service-level agreements (SLAs). This tutorial details the procedure to build a cross-region, customer-managed disaster recovery solution for Oracle Integration, specifically for the integrations feature in Oracle Integration.

Oracle Integration is a managed OCI Platform as a Service (PaaS) offering which is not something OCI Full Stack DR can manage natively since Oracle Integration itself does not expose compute, storage or database to OCI users. But, OCI Full Stack DR can automate recovery for PaaS offerings as long as the engineering team for a given service such as Oracle Integration has documented a way to provision, configure and recover their service for disaster recovery between OCI regions. The Oracle Integration engineering team has documented Configuring a Disaster Recovery Solution for Oracle Integration Generation 2 explaining how to manually provision, configure and recover Oracle Integration.

OCI Full Stack Disaster Recovery (DR) is not used to provision or configure Oracle Integration. Oracle Integration must be provisioned and configured for DR across regions by following the step-by-step instructions found in Configuring a Disaster Recovery Solution for Oracle Integration Generation 2 before attempting to use OCI Full Stack DR in any way.

The manual failover steps prescribed by Oracle Integration engineering in this documentation: Configuring a Disaster Recovery Solution for Oracle Integration Generation 2 must also be successfully tested for switchover and switchback before using OCI Full Stack DR.

Oracle Integration is normally part of a larger system

This tutorial assumes that Oracle Integration is the only application being added to the DR protection groups. This is not normal.

This tutorial is unusual in the fact that only Oracle Integration is shown and discussed throughout the document to keep things simple. Normally, Oracle Integration will simply be one small part of a much larger, more complex business system that includes many different services and applications in a single OCI Full Stack DR protection group and set of DR plans. It is highly likely you will be following similar Oracle Help Center (OHC) tutorials for other applications and services such as PeopleSoft, WebLogic Server, Oracle Analytics Cloud, and so on.

Note: The steps in this tutorial were tested using Oracle Integration Generation 2. This tutorial just shows how to implement DR for application integration capability of Oracle Integration Generation 2, because we do not want to overwhelm the reader by introducing too many moving parts and pieces.

Caution about implementing incrementally

Adding more members to a DR protection group after creating DR plans will delete all existing DR plans in the protection groups at both regions.

OCI Full Stack DR is designed with the assumption that the entire application stack for a given business system is already deployed across OCI regions and manual DR has already been proven to work. If your business system includes more than Oracle Integration, then add all members for all other applications or OCI services to the DR Protection Groups before creating any DR plans.

How the recovery works

The recovery solution for Oracle Integration requires OCI Full Stack DR to execute a series of custom bash scripts during a recovery operation such as a failover or switchover. The scripts referenced in this tutorial are provided by the Oracle Integration Specialists team and available in a GitHub repository specifically tailored for this Oracle Integration DR solution. The bash scripts are downloaded to a compute instance that is part of the application stack that OCI Full Stack DR will manage during a recovery operation.

The following scripts are provided for generic guidance. You can either use your own scripts or customize the scripts according to your corporate policy and security requirements. You need to install OCI CLI and configure your credentials to use the scripts.

Also, to make sure that your primary instance is updated with the latest scheduled parameters, ensure that the integrations.json file is updated with all the integration names along with the version details, along with the integration_parameters.json file, which needs to be kept updated with the latest scheduled parameter values for all the scheduled integrations. You can employ preferred CI/CD process to achieve this.

This tutorial explains how to download the scripts and how to use them in a later step. This tutorial uses Option 2 below for hosting the bash scripts only because the tutorial does not include anything other than Oracle Integration.

Option 1 for hosting scripts

Oracle Integration is most often part of a larger, more complex business system that includes an application like Oracle E-Business Suite, PeopleSoft or JD Edwards Enterprise one plus other databases, compute instances and home-grown applications. In this case, simply choose any one of the “movable” compute instances that are already part of the business system to host the scripts. The selected compute instance can be anything where Oracle Linux is installed and will most likely be an existing VM that serves another purpose like an application server or admin server of some sort.

This tutorial will refer to this particular compute instance as the Control Node or DR Node even though it really fulfills another purpose in the application stack.

Option 2 for hosting scripts

If this is an unusual circumstance where Oracle Integration is going to be the only application service that OCI Full Stack DR is going to manage during a recovery operation, then a compute instance will need to be created just to host the scripts.

Normally, OCI Full Stack DR does not require any specialized management servers to automate recovery operations. However, you will create a compute instance that will act as a specialized management server in this case since Oracle Integration is not something OCI Full Stack DR can manage natively. The specialized management server is seen throughout this document as Control Node or DR Node. The entire purpose of the Control Node is simply to act as a server where custom scripts can reside and be called by OCI Full Stack DR during a recovery operation. This tutorial will explain how to create custom, user-defined DR plan groups and steps as part of your DR plans to call the scripts installed on the Control Node.

Oracle Integration Deployment Architecture

As shown in the figure, Oracle Integration Generation 2 DR architecture includes two Oracle Integration instances, distinguished as primary and secondary, operating concurrently. However, traffic is directed to only one instance at a time. Initially, traffic flows to the primary instance. If the primary instance becomes unavailable, the DNS record is adjusted to redirect traffic to the secondary instance.

Fig 1: Oracle Integration DR reference architecture

Once the instances are provisioned, export the metadata from the primary instance to the standby instance. You can either do it by using REST apis or by leveraging the Oracle Integration UI to export and import the metadata. After this initial, one-time migration of the metadata is complete, you must keep the metadata synchronized between the instances using CICD. You can use Jenkins or a similar tool to implement CICD for your instances and have the metadata synchronized. You can also use an OCI Compute instance as the Jenkins CI server and CD hub.

Oracle Integration must first be provisioned and configured for disaster recovery (DR) across OCI regions before introducing OCI Full Stack DR. It is extremely important that the manual steps to switchover Oracle Integration as documented by Oracle Integration engineering are tested and work correctly before attempting to automate the switchover/failover process using OCI Full Stack DR.

Becoming familiar with the entire process

The Full Stack Disaster Recovery engineering team has created a series of companion videos for this tutorial to understand the entire process flow. These videos are part of a OCI Full Stack DR Oracle Integration playlist in YouTube that can be accessed using the following links:

• Video 1: Deploy Oracle Integration for Disaster Recovery.
• Video 2: Automate Disaster Recovery operations for Oracle Integration using OCI Full Stack DR.
• Video 3: Scripts used to automate recovery for Oracle Integration.

Part 2: Step-by-Step Instructions

This part begins the step-by-step instructions needed to add Oracle Integration to OCI Full Stack DR.

Objectives

The following steps will be covered in this tutorial explaining how to automate recovery for Oracle Integration using OCI Full Stack DR:

1. Task 1: Deploy Oracle Integration for DR across OCI regions
   1. Prepare Oracle Integration DR Control Node
   2. Download custom scripts to the DR Control Node
   3. Manually install and deploy Oracle Integration for DR across two OCI regions
   4. Manually test all recovery steps from desired region 1 to region 2
   5. Manually test all recovery steps from desired region 2 to region 1
2. Task 2: Prepare for OCI Full Stack DR
   1. Create IAM policies for OCI Full Stack DR
   2. Create IAM policies for other OCI services
   3. Create object storage buckets for logs
3. Task 3: Create DR Protection Groups (DRPG)
4. Task 4: Add members to region 1 DRPG
5. Task 5: Create DR plans in region 2 (Phoenix)
   1. Create switchover plan
   2. Create failover plan
6. Task 6: Customize the switchover plan in region 2 (Phoenix)
7. Task 7: Customize the failover plan in region 2 (Phoenix)
8. Task 8: Execute the switchover plan in region 2 (Phoenix)
9. Task 9: Create basic DR plans in region 1 (Ashburn)
   1. Create switchover plan
   2. Create failover plan
10. Task 10: Customize the switchover plan in region 1 (Ashburn)
11. Task 11: Customize the failover plan in region 1 (Ashburn)

Definitions and assumptions throughout the tutorial

Regions

• Region 1 is Ashburn
  • Ashburn will start out as the primary region.
  • This role will eventually change to standby after you are instructed to perform a switchover in later steps.
• Region 2 is Phoenix
  • Phoenix will start out as the standby region.
  • This role will eventually change to primary after you are instructed to perform a switchover in later steps.

Compartments

You are free to organize Oracle Integration and OCI Full Stack DR into any compartment scheme that works within your standards for IT governance. We have chosen to organize applications into their own individual compartments, then organize all DR Protection Groups into a single compartment where completely different business systems can all be seen at a glance.

• Organizing all DR protection groups into a single compartment apart from the applications makes it much easier for IT staff to locate and execute DR plans for many completely different business systems.
• Having a single compartment for all DR protection groups helps eliminate human error and increases the speed in which DR plans can be found and executed
  • Compartment for Oracle Integration: OIC-demo. The compartment for OIC itself, storage, storage buckets, compute and database related to OIC is OIC-demo in this tutorial.
  • Compartment for OCI Full Stack DR: OIC-demo The compartment for OCI Full Stack DR protection groups and plans is OIC-demo in this tutorial.

Oracle Integration DR Control Node

The DR Control Node is any compute instance that you designate to host custom bash scripts that perform specific tasks to recover Oracle Integration. The scripts are called by OCI Full Stack DR during a recovery operation. This tutorial explains how to add the scripts to OCI Full Stack DR in steps 6, 7, 10 & 11.

• For Oracle Integration as a standalone application: This will be a specialized compute instance you create to act as the host for the custom scripts.
• For Oracle Integration as part of an application stack: This will be any existing compute instance that is a member of a DR protection group (DRPG). For example, Oracle E-Business Suite or PeopleSoft will have application servers that are members of the same DRPGs that are managing recovery for Oracle Integration; any one of these can fulfill the role of DR Control Node in this tutorial.

Prerequisites

Oracle Integration should be deployed for disaster recovery across both regions before beginning work with OCI Full Stack DR. This is covered in Task 1 below.

Task 1: Deploy Oracle Integration for Disaster Recovery

OCI Full Stack DR is not involved in any part of this step.

Task 1.1: Prepare the DR Control Node to run custom automation

Designate a compute instance to act as a DR Control Node for OIC. This can be an existing compute instance, or it can be a compute instance created just for this purpose. See the options below for more detail. Ensure the compute instance(s) acting as the DR Control Node has been configured to run commands using the OCI Cloud Agent: Running Commands on an Instance.

Option 1: Oracle Integration as a standalone application

This tutorial assumes Oracle Integration is a standalone service, so you will create a compute instance with Oracle Linux in region 1. Use the lowest cost shape with Oracle Linux since it will only be used to host the custom bash scripts. The need for a specialized compute instance dedicated to fulfilling this role is unusual; option 2 is the most common scenario for a majority of organizations.

The specialized compute instance will be added as a member of the DR protection group in region 1 in a later step.

Option 2: Oracle Integration as part of an application stack

You can use any single existing movable compute that is part of any Oracle or non-Oracle application being managed by OCI Full Stack DR in region 1. This will fulfill the role of the DR Control Node any time this tutorial refers to the DR Control Node.

It is best to use a movable compute instance, but you can also designate a non-movable compute instance in region 1 and another one in region 2 if you don’t have any movable compute as part of your DR solution. You will need to maintain any changes you make to scripts or the guest OS in both regions if non-movable compute is used for this role.

Option 3: Oracle Integration as part of an application stack with multiple PaaS offerings

Perhaps your business system also has Oracle HTTP Server (OHS), Oracle Integration, and Oracle Data Integrator (ODI). In this case, you might consider creating a specialized compute instance as you would with Option 1 to host DR recovery scripts for all the various PaaS service.

Task 1.2: Ensure volume group is replicated to region 2

Ensure the boot volume for the DR Control node is a member of a block volume group and the block volume group is replicated to region 2.

Ensure that any other boot & block belonging to any other movable compute for this OCI Full Stack DR project also belongs to block volume groups replicated to region 2. If you need more details, see OCI Block Storage documentation

Task 1.3: Download bash scripts to DR Control Node

Download the custom bash scripts from github that were written specifically for this Oracle Integration DR solution. The scripts shown below should be copied to any subdirectory on the compute instance acting as the DR Control Node for Oracle Integration

The link above should resolve to the GitHub repository:

1. This shows the repository path where the bash scripts are located on GitHub.
2. This shows the repository containing the bash scripts.

Figure 2-4: Screenshot of github repository containing bash scripts for Oracle Integration

Task 1.4: Deploy Oracle Integration for DR

Deploy Oracle Integration for DR across OCI regions using the step-by-step instructions provided by Oracle Integration Engineering team.

Task 1.5: Manually Test Recovery of Oracle Integration

It is a best practice to ensure the manual recovery steps The manual steps to recover Oracle Integration documented in Disaster Recovery Configuration for Oracle Integration must be successful before working with OCI Full Stack DR.

Task 1.6: Next Steps

Return to this document to begin working with OCI Full Stack DR once the following requirements have been completed.

1. Manually deploy Oracle Integration for DR across two desired OCI regions.
2. Manually test all recovery steps from region 1 (Ashburn) to region 2 (Phoenix).
3. Manually test all recovery steps from region 2 (Phoenix) to region 1 (Ashburn).

Task 2: Prepare for OCI Full Stack DR

OCI Full Stack DR is not involved in any part of this step. The following steps prepare the tenancy, compartment, OCI services and Oracle Integration for automated recovery by OCI Full Stack DR.

Task 2.1: Create IAM policies for OCI Full Stack DR

Configure the required OCI IAM policies for Full Stack Disaster Recovery as outlined in the following documents.

• Policies for Full Stack Disaster Recovery.
• Configuring Identity and Access Management (IAM) policies needed for Full Stack Disaster Recovery.

Task 2.2: Create OCI IAM policies for other services managed by OCI Full Stack DR

OCI Full Stack DR must have the ability to control and manage other key OCI services such as compute, networking, storage, vaults, databases and other miscellaneous services. Configure the required OCI IAM policies for other services as explained in the following document.

• Policies for Other Services Managed by Full Stack Disaster Recovery.

Task 2.3: Create OCI Object Storage buckets for DRPG logs

Note: Skip Task 2.3 entirely if you are adding Oracle Integration to existing DR Protection Groups.

Create OCI Object Storage buckets in the primary and standby regions to store logs generated by OCI Full Stack DR during recovery operations: Object Storage.

Task 2.3.1: Navigate to OCI Object Storage

Begin by navigating to Object Storage & Archive Storage as shown in Figure 2-1 below

1. Ensure the browser context is set to region 1 (Ashburn).
2. Select Storage.
3. Select Buckets.

Figure 2-1: Navigate to object storage

Task 2.3.2: OCI storage bucket in region 1

Create an object storage bucket in region 1. The bucket will be assigned to the DR protection group in region 1 in a later step.

1. Select the compartment that contains OIC related resources.
2. Select Create Bucket.
3. Give the bucket a meaningful name that easily identifies which application and purpose it serves; there is no reason to include the region as part of the name. For example, this name indicates it is used for the OCI Full Stack DR logs related to DR operations for OIC.
4. Use the default value for tier and Encryption.
5. Select Create to create the bucket.

Figure 2-2: Create an object storage bucket in region 1

Task 2.3.3: OCI storage bucket in region 2

Follow the same process to create an object storage bucket in region 2 (Phoenix). The bucket will be assigned to the DR protection group in region 2 in a later step.

1. Change the context to region 2.
2. Select the compartment that contains OIC related resources in region 2.
3. Use the same exact name that was assigned to the bucket in region 1 - this will make it easy to identify in a later step.
4. Select Create to create the bucket.

Figure 2-3: Create an object storage bucket in region 2

Task 3: Create DR Protection Groups in both regions

Note: Skip Task 3 entirely if Oracle Integration is being added to existing DR Protection Groups.

Create DR protection groups in region 1 and region 2 if the protection groups for this application stack do not exist yet.

Task 3.1: Navigate to DR Protection Groups

Begin by navigating to DR Protection Groups (OCI Full Stack DR) as shown in Figure 3-1 below.

1. Ensure the OCI region context is set to region 1 (Ashburn).
2. Select Migration & Disaster Recovery.
3. Select DR Protection Groups.

Figure 3-1: Navigate to DR protection groups

Task 3.2: Create a protection group in region 1

Create a basic DR protection group (DRPG) in region 1 as shown in Figure 3-2 below. The peer, role and members will be assigned in later steps.

1. Select the compartment where you want the DRPG to be created. This can be the same compartment where Oracle Integration resources exist, or as in this case, a compartment acting as a repository containing DRPGs for many different business systems.
2. Select Create DR protection group to open the dialog.

Figure 3-2: Begin creating DR protection group in region 1

Add a name and object storage bucket for the logs as shown in Figure 3-3 below.

1. Use a meaningful, simple name for the DRPG; this example shows the name of the business system and the region.
2. Select the object storage bucket created in Task 2 for region 1.

Figure 3-3: Parameters needed to create DR protection group in region 1

Task 3.3: Create a protection group in region 2

Create a basic DR protection group (DRPG) in region 2 as shown in Figure 3-4 below. The peer, role and members will be assigned in later steps.

1. Change the OCI region context to region 2.
2. Select the compartment where you want the DRPG to be created. This can be the same compartment where Oracle Integration resources exist, or as in this case, a compartment acting as a repository containing DRPGs for many different business systems.
3. Select Create DR protection group to open the dialog

Figure 3-4: Begin creating DR protection group in region 2

Add a name and object storage bucket for the logs as shown in Figure 3-5 below.

1. Use a meaningful, simple name for the DRPG; this example shows the name of the business system and the region.
2. Select the object storage bucket created in Task 2 for region 2

Figure 3-5: Parameters needed to create DR protection group in region 2

Task 3.4: Associate protection groups in region 1 & region 2

Associate the DRPGs in each region as peers of each other and assign the peer roles of primary and standby. This is how OCI Full Stack DR will know which two regions work together for Oracle Integration recovery. The roles of primary and standby are automatically changed by OCI Full Stack DR as part of any DR operation/DR plan execution; there is no need to manage the roles manually at any time.

Task 3.4.1: Begin the association

1. Ensure OCI region context is set to region 1 (Ashburn).
2. Select Associate to begin the process.

Figure 3-6: Begin DRPG association

Task 3.4.2: Associate protection groups in region 1 & region 2

Provide the parameters as shown in Figure 3-7 below.

1. Select primary role. OCI Full Stack DR will assign the standby role to region 2 automatically.
2. Select region 2 (Phoenix) where the other DRPG was created.
3. Select the peer DRPG that was created in.

Figure 3-7: Parameters needed to associate the DRPGs

Task 3.4.3: What you should see after association is complete

OCI Full Stack DR will show something like Figure 3-8 below once the association is completed.

1. The current primary peer DRPG is Ashburn (region 1).
2. The current standby peer DRPG is Phoenix (region 2).

Figure 3-8: Showing the peer relationship from the individual DRPG perspective

The same information can be found whenever the context/view is from a global perspective showing all DR protection groups as shown in Figure 3-9 below.

1. The current primary peer DRPG is Ashburn (region 1).
2. The current standby peer DRPG is Phoenix (region 2).

Figure 3-9: Showing the peer relationship from the global DRPG perspective

Task 4: Add members to the DR Protection Group

Note: This step will delete any existing DR plans in both regions when adding members to existing DR Protection Groups. OCI Full Stack DR cannot save copies or make backups of DR protection groups at the time of this writing. Make sure you have recorded all the information about any DR plan groups and steps in a text file or spreadsheet to help recreate the custom, user-defined plan groups and steps. You can also create bash scripts that call OCI Full Stack DR CLI commands to recreate the custom, user-defined plan groups and steps (this is beyond the scope of this tutorial).

Add the DR Control Node as members to the Primary DR protection group. The DR Control Node is either a compute instance you created just to control Oracle Integration or it is a compute instance that is part of the application stack you want to manage with OCI Full Stack DR.

You will add the following resources to the primary DRPG in region 1:

1. The DR Control Node,
2. The volume group containing the DR Control Node boot volume.

Task 4.1: Begin adding members to DRPG in region 1

Begin by selecting the DRPG in region 1 as shown int Figure 4-1 below.

1. Ensure the OCI region context is region 1 (Ashburn).
2. Select the DRPG in region 1.
3. Select Members.
4. Click on Add Member to begin the process.

Figure 4-1: How to begin adding members to DR protection group in region 1

Task 4.1.1: Add compute instance for DR node

Add compute instance for DR Control Node as shown in figure 4-2 below.

1. Acknowledge warning about DR plans.
2. Select Compute as member resource type.
3. Select the compute instance you want to use the DR control node.
4. Select moving instance.
5. Tell OCI Full Stack DR which VCN & subnet to assign to the VNIC(s) at region 2 during a recovery. Figure 4-2 shows a single VNIC. OCI Full Stack DR does not care how many VNICs you have or how they are configured at either region; specify whatever you need that fits your requirements.

Figure 4-2: Parameters needed to add DR Control Node

Task 4.1.2: Add block volume group for DR node

Add the block volume group containing boot for the DR Control Node. The block volume group must already have cross-region replication configured between the two regions before adding it the DR protection group.

1. Select Volume group as member resource type.
2. Ensure correct compartment containing the volume group is selected, then select the volume group.

Figure 4-3: Parameters needed to add boot volume group for DR Control Node

Task 4.1.3: Verify member resources for region 1

The DRPG for region 1 should have two member resources at a minimum as shown in Figure 4-5 below. The names of your member resources will be different.

1. The movable compute instance and block volume group for the compute instance that we designated to act the OIC DR Control Node.

Figure 4-5: Showing members of DRPG in region 1

There is no need to add any members in the Standby DR protection group as we are using moving compute instance as DR node to host the Oracle Integration scripts

Task 5: Create basic DR plans in region 2 (Phoenix)

This step creates basic switchover and failover plans associated with the standby DR Protection Group in region 2 (Phoenix).

The purpose of each plan is to transition the workload from primary region 1 to standby region 2. The roles of the DR protection groups in both regions are automatically reversed as part of any DR operation, so the protection group in region 1 will become the standby and the protection group in region 2 will become primary after a failover or switchover.

OCI Full Stack DR will pre-populate both plans with built-in steps based on the member resources added in the previous step. The plans will be customized in later steps to handle all the tasks related to Oracle Integration during a recovery operation.

The switchover plans are always created in the protection group with the standby role; region 2 is currently the standby protection group, so we will begin in Phoenix.

Task 5.1: Begin creating DR plans

Create a basic plans by selecting the DRPG in region 2 as shown in Figure 5-1 below.

1. Ensure the OCI region context is region 2 (Phoenix).
2. Select the standby DRPG in region 2.
3. Select Plans.
4. Click on Create Plan to begin the process.

Figure 5-1: How to begin creating basic DR plans in region 2

Task 5.1.1: Create a switchover plan

Creating a DR plan is simple as shown in Figure 5-2 below.

1. Make the name of the switchover plan simple but meaningful. The name should be as short as possible but easy to understand at a glance to help reduce confusion and human error during a crisis.
2. Choose the plan type. There are only four plan types at the time of this writing.

Figure 5-2: The parameters needed to create DR switchover plan

Task 5.1.2: Create a failover plan

Follow the same process to create a basic failover plan as shown in Figure 5-3 below.

1. Make the name of the failover plan simple but meaningful. The name should be as short as possible but easy to understand at a glance to help reduce confusion and human error during a crisis.
2. Choose the plan type. There are only two plan types at the time of this writing.

Figure 5-3: The parameters needed to create DR failover plan

The standby DR Protection Group in region 2 should now have the two DR plans as shown below. These will handle transitioning workloads from region 1 to region 2. You will create similar plans at region 1 to transition workloads from region 2 back to region 1 in a later step.

Figure 5-4: Showing the two basic DR plans that must exist in region 2 before proceeding any further

Task 6: Customize the switchover plan in region 2 (Phoenix)

The basic DR plans created in Task 5 contain pre-populated steps for recovery tasks that are built into OCI Full Stack DR and do not contain anything to manage recovery tasks specific to Oracle Integration. This step explains how to add custom, user-defined DR Plan Groups and steps to manage those things that need to be accomplished during a switchover for Oracle Integration:

1. Sync scheduled parameters from region 1 to region 2
2. Activate relevant integrations at region 2
3. Start scheduled integrations at region 2
4. Update DNS record at region 2
5. Deactivate scheduled integrations at region 1

Task 6.1: Select the switchover plan

Begin by navigating to the switchover plan created in the previous step.

Figure 6-1: How to begin customizing the switchover plan in region 2

Task 6.2: Enable DR Plan Groups that terminate artifacts (optional)

There are two plan groups that are disabled by default in switchover plans as shown in the screenshot below. They are disabled to provide a level of comfort during testing that nothing is actually being deleted and you still have a viable copy of the artifacts as a backup in case something goes wrong during testing.

However, these two plan groups terminate (delete) artifacts that will never be used again as part of any DR operation in the future. The artifacts will simply continue to accumulate over time as you switch back and forth between the two regions causing confusion about which compute instances and volume groups are the ones that should actually be active.

These plan groups should be enabled once OCI Full Stack DR goes into production. Any artifacts that were left in place during testing switchovers and switchbacks while these two plan groups were disabled should be terminated and cleaned up before going into to production to reduce confusion and the likelihood of human error during normal operations.

Optionally, these plan groups can be enabled now to avoid having to manually clean up the superfluous artifacts before going into production.

Figure 6-2: Plan groups disabled by default

Here is what the disabled plan groups do when they are enabled:

1. This plan group terminates artifacts of compute instances that are left behind at region 1 after the replicated versions of the VMs have been launched at region 2 during the OCI block storage operation to reverse the replication from region 2 back to region 1 as part of the switchover. The leftover VMs are not used during a switchback because the operation to reverse block volume replication creates all new VMs in completely new block volume groups.

2. This plan group terminates artifacts of block volume groups (VGs) that are left behind at region 1 after the replicated versions of the VGs have been activated at region 2 and volume group replication has been reversed during the switchover. The leftover block volume groups are never used again, not even as part of a switchover from region 2 back to region 1.

Task 6.2.1: Enable terminate compute plan group

Enable the plan group.

1. Select Enable all steps from the context menu to the right of the plan group name

Figure 6-3: How to enable terminate compute instances

Task 6.2.2 Enable terminate volume groups plan group

Enable the plan group.

1. Select Enable all steps from the context menu to the right of the plan group name

Figure 6-4: How to enable terminate volume groups

Task 6.3: Create plan group to sync scheduled parameters from region 1 to region 2

Now begin adding custom, user-defined DR Plan Groups.

The first user-defined plan group will sync scheduled parameters from region 1 to region. This plan group will contain a single step that calls the oic-sync-schedule-parameters.sh bash script that was downloaded to the DR Control Node in Task 1.4.

Task 6.3.1: Select add plan group

Begin the process to add a plan group.

1. Click on Add group to begin.

Figure 6-5: Begin adding plan group to sync scheduled parameters

Task 6.3.2: Provide plan group name, order and add step

A DR Plan Group can contain many steps that are all executed in parallel. We are just adding a single step to execute a bash script to sync scheduled parameters.

1. Give the plan group a simple but descriptive name.
2. Select a position where the plan group will be inserted into the DR plan. In this case, we are going to insert our user-defined plan group before the built-in plan group that stops the VMs at region 1.
3. Select the built-in Stop Compute Instances (Primary) plan group.
4. Click on Add step to open the dialog where we will specify the script to sync scheduled parameters.

Figure 6-6: Parameters to create plan group and add step to sync scheduled parameters

Task 6.3.3: Provide step name and local script parameters

The Add plan group step dialog allows us to specify parameters about what this one step will perform and how it will behave during recovery. In this case, it will sync scheduled parameters from region 1 to region 2.

We will explain all fields in this dialog, but leave out this detail in all the remaining screenshots in subsequent steps since we are just performing the same process repeatedly.

1. A descriptive name explaining what task this step performs.
2. Always select the region where the DR Control Node is running right now, not where it will be running during a switchover. OCI Full Stack DR will keep track of where the VM is running, so you just need to specify where it is right now. In this case, the DR Control Node is running in region 1 (Ashburn).
3. Select Run local script to inform OCI Full Stack DR that the script will be found on a compute instance. The bash scripts were downloaded to the DR Control Node in Task 1.3.
4. Select the correct compartment that contains the DR Control Node - it can be any compartment. The select the compute instance that was designated as the DR Control Node (it may be an application server or VM that was created just for this project/tutorial).
5. Paste in the absolute path where you installed the oic-sync-schedule-parameters.sh script on the DR Control Node. Add PHX as the parameter. This is target region where we want to sync the scheduled parameters. You may need to provide correct region parameters if you are using different OCI regions and update scripts.
6. Specify opc as the user to execute the script.
7. The DR plan should stop if the script fails to sync scheduled parameters. This will allow anyone to see there is a problem and fix it. OCI Full Stack DR provides the opportunity to continue running the switchover plan after fixing the problem.
8. The default value before OCI Full Stack DR declares a failure is one hour. This value can be changed to 30 minutes or whatever is felt to be a more realistic timeout value.
9. Click Add step to add this step to the plan group.

Figure 6-7: Parameters to create the plan step for sync scheduled parameters

Task 6.3.4: Complete adding plan group and step

The step to stop sync scheduled parameters is now added to the DR plan group as shown in figure 6-8 below.

1. This shows the plan step that was just added. It is possible to add additional steps to a DR plan group, but this plan group will only include the step to sync scheduled parameters.
2. Click on Add to add the DR plan group and step to the DR plan.

Figure 6-8: Finalize adding plan group and step to sync scheduled parameters

Task 6.4: Create plan group to activate relevant integrations at standby

The second user-defined plan group will activate relevant integrations at standby after the DR Control Node is launched at the standby region 2. This plan group will contain a single step that calls the oic-integration-switch.sh bash script that was downloaded to the DR Control Node in Task 1.3.

Task 6.4.1: Select add plan group

As before, click Add group to begin.

Figure 6-9: Begin adding plan group to activate relevant integrations at standby

Task 6.4.2: Provide plan group name, order and add step

Create a DR plan group to start activate relevant integrations at standby.

1. Give the plan group a simple but descriptive name.
2. Select a position where the plan group will be inserted into the DR plan. In this case, we are going to insert our user-defined plan group after the built-in plan group that launches the replicated version of the DR control node at region 2
3. Select the built-in Launch Compute Instances (Standby) plan group
4. Click on Add step to open the dialog where we will specify the script to activate relevant integrations at standby.

Figure 6-10: Parameters to create plan group and add step to activate relevant integrations at standby

Task 6.4.3: Provide step name and local script parameters

The Add plan group step dialog allows us to specify parameters about what this one step will perform and how it will behave during recovery. In this case, it will activate relevant integrations at region 2.

Everything in this step is the same as Task 6.3.3 except for the items show in Figure 6-11 below.

1. A descriptive name explaining what task this step performs.
2. Paste in the absolute path where you installed the oic-integration-switch.sh script on the DR Control Node. Add activate as the first parameter and PHX as the second.This is target region where we want to activate relevant integrations. You may need to provide correct region parameters if you are using different OCI regions and update scripts.
3. Click Add step to add this step to the plan group.

Figure 6-11: Parameters to create the plan step for activating relevant integrations at standby

Task 6.4.4: Complete adding plan group and step

The step to activate relevant integrations at standby is now added to the DR plan group as shown in figure 6-12 below.

1. This shows the plan step that was just added.
2. Click on Add to add the DR plan group and step to the DR plan.

Figure 6-12: Finalize adding plan group and step to activate relevant integrations at standby

Task 6.5: Create plan group to start scheduled integrations at region 2

The third user-defined plan group will start scheduled integrations at standby after the DR Control Node is launched at the standby region 2. This plan group will contain a single step that calls the oic-integration-schedule.sh bash script that was downloaded to the DR Control Node in Task 1.3.

Task 6.5.1: Select add plan group

As before, click Add group to begin.

Figure 6-13: Begin adding plan group to start scheduled integrations at standby

Task 6.5.2: Provide plan group name, order and add step

Create a DR plan group to start scheduled integrations at the standby region 2.

1. Give the plan group a simple but descriptive name.
2. Select a position where the plan group will be inserted into the DR plan. In this case, insert the user-defined plan group after the user-defined plan group created in the previous task to activate relevant integrations.
3. Select the built-in Activate relevant integrations at standby plan group
4. Click on Add step to open the dialog where we will specify the script to start scheduled integrations at standby.

Figure 6-14: Parameters to create plan group and add step to start scheduled integrations at standby

Task 6.5.3: Provide step name and local script parameters

The Add plan group step dialog allows us to specify parameters about what this one step will perform and how it will behave during recovery. In this case, it will start scheduled integrations at standby region 2.

Everything in this task is the same as Task 6.3.3 except for the items show in Figure 6-15 below.

1. A descriptive name explaining what task this step performs.
2. Paste in the absolute path where you installed the oic-integration-schedule.sh script on the DR Control Node. Add start as the first parameter and PHX as the second.
3. Click Add step to add this step to the plan group.

Figure 6-15: Parameters to create the plan step to start scheduled integrations at standby

Task 6.5.4: Complete adding plan group and step

The step to start scheduled integrations at standby is now added to the DR plan group as shown in figure 6-16 below.

1. This shows the plan step that was just added.
2. Click on Add to add the DR plan group and step to the DR plan.

Figure 6-16: Finalize adding plan group and to start scheduled integrations at standby

Task 6.6: Create plan group to update DNS record at region 2

The fourth user-defined plan group will update DNS record at standby after the DR Control Node is launched at the standby region 2. This plan group will contain a single step that calls the dns_record_update.sh bash script that was downloaded to the DR Control Node in Task 1.3.

Task 6.6.1: Select add plan group

As before, click Add group to begin.

Figure 6-17: Begin adding plan group to update DNS record at standby

Task 6.6.2: Provide plan group name, order and add step

Create a DR plan group to update DNS record to region 2.

1. Give the plan group a simple but descriptive name.
2. Select a position where the plan group will be inserted into the DR plan. In this case, we are going to insert our user-defined plan group after the built-in plan group to start scheduled integrations at standby
3. Select the built-in Start scheduled integrations at standby plan group.
4. Click on Add step to open the dialog where we will specify the script to update DNS record at standby.

Figure 6-18: Parameters to create plan group and add step to update DNS record at standby

Task 6.6.3: Provide step name and local script parameters

The Add plan group step dialog allows us to specify parameters about what this one step will perform and how it will behave during recovery. In this case, it will update DNS record at standby.

Everything in this task is the same as Task 6.3.3 except for the items show in Figure 6-19 below.

1. A descriptive name explaining what task this step performs.
2. Paste in the absolute path where you installed the dns_record_update.sh script on the DR Control Node. Add the OCI region key for region 2 (PHX in this example) as a parameter.
3. Click Add step to add this step to the plan group.

Figure 6-19: Parameters to create the plan step to update DNS record at standby

Task 6.6.4: Complete adding plan group and step

The step to update DNS record at standby is now added to the DR plan group as shown in figure 6-20 below.

1. This shows the plan step that was just added.
2. Click on Add to add the DR plan group and step to the DR plan.

Figure 6-20: Finalize adding plan group and step to update DNS record at standby

Task 6.7: Create plan group to deactivate scheduled integrations at region 1

The final user-defined plan group will deactivate scheduled integrations at region 1 after the DR Control Node is launched at the standby region 2. This plan group will contain a single step that calls the oic-integration-switch.sh script that was downloaded to the DR Control Node in Task 1.3.

Task 6.7.1: Select add plan group

As before, click Add group to begin.

Figure 6-21: Begin adding plan group to deactivate scheduled integrations at region 1

Task 6.7.2: Provide plan group name, order and add step

Create a DR plan group to deactivate scheduled integrations at region 1

1. Give the plan group a simple but descriptive name.
2. Select a position where the plan group will be inserted into the DR plan. In this case, we are going to insert our user-defined plan group after the built-in plan group to update DNS record at standby
3. Select the built-in Update DNS record at standby plan group.
4. Click on Add step to open the dialog where we will specify the script to deactivate scheduled integrations at region 1

Figure 6-22: Parameters to create plan group and add step to deactivate scheduled integrations at region 1

Task 6.7.3: Provide step name and local script parameters

The Add plan group step dialog allows us to specify parameters about what this one step will perform and how it will behave during recovery. In this case, it will to deactivate scheduled integrations at region 1

Everything in this task is the same as Task 6.3.3 except for the items show in Figure 6-23 below.

1. A descriptive name explaining what task this step performs.
2. Paste in the absolute path where you installed the oic-integration-switch.sh script on the DR Control Node. Add the OCI region key for region 1 (IAD in this example) as a parameter.
3. Click Add step to add this step to the plan group.

Figure 6-23: Parameters to create the plan step to deactivate scheduled integrations at region 1

Task 6.7.4: Complete adding plan group and step

The step to deactivate scheduled integrations at region 1 is now added to the DR plan group as shown in figure 6-20 below.

1. This shows the plan step that was just added.
2. Click on Add to add the DR plan group and step to the DR plan.

Figure 6-24: Finalize adding plan group and step to deactivate scheduled integrations at region 1

The switchover plan should now include the five DR Plan Groups for Oracle Integration as shown in the screenshot below. You may have additional plan groups if your protection group includes other applications or other OCI services along with Oracle Integration

Figure 6-25: Showing the five user-defined plan groups added to the switchover plan

Task 7: Customize the failover plan in region 2 (Phoenix)

This task explains how to add custom, user-defined DR Plan Groups and steps to manage those things that need to be accomplished during a failover for Oracle Integration at region 2 during an actual outage or loss of access to region 1. These will be a subset of the same steps that were just added to the switchover plan in Task 6 above. However, only steps that are executed at standby region 2 will be added to the failover plan since it is assumed region 1 is completely inaccessible during a failover.

1. Activate relevant integrations at region 2
2. Update scheduled parameters at region 2
3. Start scheduled integrations at region 2
4. Update DNS record at region 2

Task 7.1: Select the failover plan

Begin by navigating to the failover plan created in Task 5.

1. Ensure standby region 2 is still the current region context in the console.
2. Select the failover plan.

Figure 7-1: How to create begin customizing the failover plan in region 2

Task 7.2: Select add plan group

The first user-defined plan group will activate relevant integrations in region 2. This plan group will contain a single step that calls the oic-integration-switch.sh bash script that was downloaded to the DR Control Node in Task 1.3.

1. Click on Add group to begin.

Figure 7-2: Begin adding plan group to activate relevant integrations

Task 7.2.1: Provide plan group name, order and add step

A DR Plan Group can contain many steps that are all executed in parallel. We are just adding a single step to execute a bash script to activate relevant integrations.

1. Give the plan group a simple but descriptive name.
2. Select a position where the plan group will be inserted into the DR plan. In this case, we are going to insert our user-defined plan group after the built-in plan group that launches the replicated VMs at region 2
3. Select the built-in Launch Compute Instances (Standby) plan group
4. Click on Add step to open the dialog where we will specify the script to activate relevant integrations

Figure 7-3: Parameters to create plan group and add step to activate relevant integrations at standby

Task 7.2.2: Provide step name and local script parameters

The Add plan group step dialog allows us to specify parameters about what this one step will perform and how it will behave during recovery. In this case, it will activate relevant integrations at region 2 as shown in Figure 7-4 below.

We will explain all fields in this dialog, but leave out this detail in all the remaining screenshots in subsequent steps since we are just performing the same process repeatedly.

1. A descriptive name explaining what task this step performs.
2. The DR plan should stop if the script fails to to activate relevant integrations. This will allow anyone to see there is a problem and fix it. OCI Full Stack DR provides the opportunity to continue running the switchover plan after fixing the problem.
3. The default value before OCI Full Stack DR declares a failure is one hour. This value can be changed to 30 minutes or whatever is felt to be a more realistic timeout value.
4. Always select the region where the DR Control Node is running right now, not where it will be running during a switchover. OCI Full Stack DR will keep track of where the VM is running, so you just need to specify where it is right now. In this case, the DR Control Node is running in region 1 (Ashburn).
5. Select Run local script to inform OCI Full Stack DR that the script will be found on a compute instance. The bash scripts were downloaded to the DR Control Node in Task 1.3.
6. Select the correct compartment that contains the DR Control Node - it can be any compartment. The select the compute instance that was designated as the DR Control Node (it may be an application server or VM that was created just for this project/tutorial).
7. Paste in the absolute path where you installed the oic-integration-switch.sh script on the DR Control Node. Add activate as the first parameter and the PHX as the second.
8. Specify opc as the user to execute the script.
9. Click Add step to add this step to the plan group.

Figure 7-4: Parameters to create the plan step for to activate relevant integrations at standby

Task 7.2.3: Complete adding plan group and step

The step to activate relevant integrations is now added to the DR plan group as shown in figure 7-5 below.

1. This shows the plan step that was just added.
2. Click on Add to add the DR plan group and step to the DR plan.

Figure 7-5: Finalize adding plan group and step to activate relevant integrations

Task 7.3: Create plan group to update scheduled parameters at region 2

The second user-defined plan group will update scheduled parameters at the standby region 2.This plan group will contain a single step that calls the oic-update-parameters.sh bash script that was downloaded to the DR Control Node in Task 1.3.

Task 7.3.1: Select add plan group

As before, click Add group to begin.

Figure 7-6: Begin adding plan group to update scheduled parameters at standby

Task 7.3.2: Provide plan group name, order and add step

Create a DR plan group to to update scheduled parameters at region 2.

1. Give the plan group a simple but descriptive name.
2. Select a position where the plan group will be inserted into the DR plan. In this case, insert the user-defined plan group after the user-defined plan group created in the previous task to activate relevant integrations.
3. Select Activate relevant integrations at standby plan group.
4. Click on Add step to open the dialog where we will specify the script to update scheduled parameters at standby.

Figure 7-7: Parameters to create plan group and add step to update scheduled parameters at standby

Task 7.3.3: Provide step name and local script parameters

The Add plan group step dialog allows us to specify parameters about what this one step will perform and how it will behave during recovery. In this case, it will recover the update scheduled parameters at region 2.

Everything in this task is the same as Task 7.3.2 except for the items show in Figure 7-8 below.

1. A descriptive name explaining what task this step performs.
2. Paste in the absolute path where you installed the oic-update-parameters.sh script on the DR Control Node. Add PHX as the only parameter (PHX in this example).
3. Click Add step to add this step to the plan group

Figure 7-8: Parameters to create the plan step for update scheduled parameters at standby

Task 7.3.4: Complete adding plan group and step

The step to update scheduled parameters at standby is now added to the DR plan group as shown in figure 7-9 below.

1. This shows the plan step that was just added.
2. Click on Add to add the DR plan group and step to the DR plan.

Figure 7-9: Finalize adding plan group and step update scheduled parameters at standby

Task 7.4: Create plan group to start scheduled integrations at region 2

The third user-defined plan group will start scheduled integrations at standby after the DR Control Node is launched at the standby region 2. This plan group will contain a single step that calls the oic-integration-schedule.sh bash script that was downloaded to the DR Control Node in Task 1.3.

Task 7.4.1: Select add plan group

As before, click Add group to begin.

Figure 7-10: Begin adding plan group to start scheduled integrations at standby

Task 7.4.2: Provide plan group name, order and add step

Create a DR plan group to start scheduled integrations at standby

1. Give the plan group a simple but descriptive name.
2. Select a position where the plan group will be inserted into the DR plan. In this case, we are going to insert our user-defined plan group after the Update scheduled parameters at standby plan groups
3. Select the built-in Update scheduled parameters at standby plan group.
4. Click on Add step to open the dialog where we will specify the script to start scheduled integrations at standby

Figure 7-11: Parameters to create plan group and add step to start scheduled integrations at standby

Task 7.4.3: Provide step name and local script parameters

The Add plan group step dialog allows us to specify parameters about what this one step will perform and how it will behave during recovery.

Everything in this task is the same as Task 7.2.2 except for the items show in Figure 6-19 below.

1. A descriptive name explaining what task this step performs.
2. Paste in the absolute path where you installed the oic-integration-schedule.sh script on the DR Control Node. Add start as the first parameter and PHX as the second.
3. Click Add step to add this step to the plan group.

Figure 7-12: Parameters to create the plan step to start scheduled integrations at standby

Task 7.4.4: Complete adding plan group and step

The step to start scheduled integrations at standby is now added to the DR plan group as shown in figure 7-13 below.

1. This shows the plan step that was just added.
2. Click on Add to add the DR plan group and step to the DR plan.

Figure 7-13: Finalize adding plan group and step to start scheduled integrations at standby

Task 7.5: Create plan group to update DNS record at region 2

The fourth user-defined plan group will update DNS record at standby after the DR Control Node is launched at the standby region 2. This plan group will contain a single step that calls the dns_record_update.sh bash script that was downloaded to the DR Control Node in Task 1.3.

Task 7.5.1: Select add plan group

As before, click Add group to begin.

Figure 7-14: Begin adding plan group to update DNS record at standby

Task 7.5.2: Provide plan group name, order and add step

Create a DR plan group to update DNS record to region 2.

1. Give the plan group a simple but descriptive name.
2. Select a position where the plan group will be inserted into the DR plan. In this case, we are going to insert our user-defined plan group after the built-in plan group to start scheduled integrations at standby
3. Select the built-in Start scheduled integrations at standby plan group.
4. Click on Add step to open the dialog where we will specify the script to update DNS record at standby.

Figure 7-14: Parameters to create plan group and add step to update DNS record at standby

Task 7.5.3: Provide step name and local script parameters

The Add plan group step dialog allows us to specify parameters about what this one step will perform and how it will behave during recovery. In this case, it will update DNS record at standby.

Everything in this task is the same as Task 6.3.3 except for the items show in Figure 7-15 below.

1. A descriptive name explaining what task this step performs.
2. Paste in the absolute path where you installed the dns_record_update.sh script on the DR Control Node. Add the OCI region key for region 2 (PHX in this example) as a parameter.
3. Click Add step to add this step to the plan group.

Figure 7-15: Parameters to create the plan step to update DNS record at standby

Task 7.5.4: Complete adding plan group and step

The step to update DNS record at standby is now added to the DR plan group as shown in figure 7-16 below.

1. This shows the plan step that was just added.
2. Click on Add to add the DR plan group and step to the DR plan.

Figure 7-16: Finalize adding plan group and step to update DNS record at standby

The failover plan should now include the four DR Plan Groups for Oracle Integration as shown in the screenshot below. You may have additional plan groups if your protection group includes other applications or OCI services along with Oracle Integration.

Figure 7-14: Showing the four user-defined plan groups added to the failover plan

Task 8: Execute the switchover plan in region 2 (Phoenix)

Both switchover and failover DR plans have been completed in the standby region 2 (Phoenix). The DR plans in region 2 allow OCI Full Stack DR to transition workloads from region 1 to region 2. The next task is to create switchover and failover plans in the protection group for region 1 (Ashburn) so OCI Full Stack DR can transition workloads from region 2 back to region 1.

However, DR plans can only be created and modified in the protection group with the standby role. The DR protection group in region 1 is currently the primary, which means DR plans cannot be created in region 1.

Therefore, we need to reverse the roles of the protection groups so region 1 is the standby and region 2 is the primary. Execute the switchover plan that was just created to transition the workload from region 1 (Ashburn) to region 2 (Phoenix).

Task 8.1: Begin plan execution

Execute DR plan to begin the process of transitioning the Oracle Integration workload from region 1 to region 2.

1. Ensure the region context is still set to standby region 2 (Phoenix).
2. Use the breadcrumbs at the top of the console to help ensure DR protection group details is the current plan context.
3. Ensure the correct DR protection group in region 2 is selected; it should be the Standby role.
4. Ensure both the failover and switchover plans exist before proceeding; if not, go back to the previous steps to create and customise both DR plans.
5. Click the Execute DR plan.

Figure 8-1: Showing how to execute a switchover to standby region

Task 8.2: Select switchover plan and execute

This task executes the switchover plan in region 2.

1. Select the switchover plan.
2. Ensure enable prechecks is selected.
3. Click the Execute DR plan to begin.

Figure 8-2: Choose and execute the switchover plan

Task 8.3: Next steps

Monitor the switchover plan until the Oracle Integration workload has been fully transitioned from region 1 to region 2. OCI Full Stack DR will take care of cleaning up artifacts and changing the roles of primary and standby between the regions.

Region 2 (Phoenix) will be the primary region and region 1 (Ashburn) will be the standby region once OCI Full Stack DR has completed the switchover.

Task 9: Create DR plans in region 1 (Ashburn)

Create the same basic switchover and failover plans in the DR Protection Group for region 1 (Ashburn) which is now the standby peer.

The purpose of each plan is to transition the workload from region 2 to region 1 whenever region 2 is the primary peer. The roles of the DR protection groups in both regions are automatically reversed as part of any DR operation, so the protection group in region 2 will become the standby and the protection group in region 1 will become primary after a failover or switchover.

OCI Full Stack DR will pre-populate both plans with built-in steps based on the member resources added in the previous step. The plans will be customized in later steps to handle all the tasks related to Oracle Integration during a recovery operation.

The DR plans are always created in the protection group with the standby role; region 1 is currently the standby protection group after executing the switchover plan in Task 8.

Task 9.1: Begin creating DR plans

Create a basic plans by selecting the DRPG in region 1 as shown in Figure 9-1 below.

1. Ensure the OCI region context is region 1 (Ashburn).
2. Select the standby DRPG in region 1.
3. Select Plans.
4. Click on Create Plan to begin the process.

Figure 9-1: How to begin creating basic DR plans in region 1

Task 9.1.1: Create a switchover plan

Creating a DR plan is simple as shown in Figure 9-2 below.

1. Make the name of the switchover plan simple but meaningful. The name should be as short as possible but easy to understand at a glance to help reduce confusion and human error during a crisis.

2. Choose the plan type. There are only two plan types at the time of this writing.

   
   Figure 9-2: The parameters needed to create DR switchover plan

3. Click on Create to create basic switchover plan prepopulated with basic built-in steps.

Task 9.2: Create a failover plan

Follow the same process to create a basic failover plan as shown in Figure 9-3 below.

1. Make the name of the failover plan simple but meaningful. The name should be as short as possible but easy to understand at a glance to help reduce confusion and human error during a crisis.

2. Choose the plan type. There are only two plan types at the time of this writing.

3. Click on Create to create basic failover plan prepopulated with basic built-in steps.

Figure 9-3: The parameters needed to create DR failover plan

The standby DR Protection Group in region 1 should now have the two DR plans as shown below. These will handle transitioning workloads from region 2 back to region 1.

Figure 9-4: Showing the two basic DR plans that must exist in region 2 before proceeding any further

Task 10: Customize the switchover plan in region 1 (Ashburn)

Everything about this task is almost exactly the same as what we did in Task 6 for region 2 except this is being done in region 1.

The basic DR plans created in Task 9 do not contain anything to manage recovery tasks specific to Oracle Integration. This task explains how to add custom, user-defined DR Plan Groups and steps to manage those things that need to be accomplished during a switchover for Oracle Integration:

1. Sync scheduled parameters from region 1 to region 2.
2. Activate relevant integrations at region 2.
3. Start scheduled integrations at region 2.
4. Update DNS record at region 2.
5. Deactivate scheduled integrations at region 1.

Task 10.1: Select the switchover plan

Begin by navigating to the switchover plan created in the previous step.

Figure 10-1: How to begin customizing the switchover plan in region 1

Task 10.2: Enable DR Plan Groups that terminate artifacts (optional)

These are the same steps performed for region 2 in an earlier step; the same process needs to be followed for region 1.

Two plan groups are disabled by default in switchover plans as shown in the screenshot below. They are disabled to provide a level of comfort during testing that nothing is actually being deleted, and you still have a viable copy of the artifacts as a backup in case something goes wrong during testing.

However, these two plan groups terminate (delete) artifacts that will never be used again as part of any DR operation in the future. The artifacts will simply continue to accumulate over time as you switch back and forth between the two regions causing confusion for humans about which compute instances and volume groups are the ones that should actually be active.

These plan groups should be enabled once OCI Full Stack DR goes into production. Any artifacts that were left in place during testing switchovers and switchbacks while these two plan groups were disabled should be terminated and cleaned up before going into to production to reduce confusion and the likelihood of human error during normal operations.

Optionally, these plan groups can be enabled now to avoid having to manually clean up the superfluous artifacts before going into production.

Figure 10-2: Plan groups disabled by default

Here is what the disabled plan groups do when they are enabled:

1. This plan group terminates artifacts of compute instances that are left behind at region 2 after the replicated versions of the VMs have been launched at region 1 during the OCI block storage operation to reverse the replication from region 1 back to region 2 as part of the switchover. The leftover VMs are not used during a switchback because the operation to reverse block volume replication creates all new VMs in completely new block volume groups.

2. This plan group terminates artifacts of block volume groups (VGs) that are left behind at region 2 after the replicated versions of the VGs have been activated at region 1 and volume group replication has been reversed during the switchover. The leftover block volume groups are never used again, not even as part of a switchover from region 1 back to region 2.

Task 10.2.1: Enable terminate compute plan group

Enable the plan group.

1. Select Enable all steps from the context menu to the right of the plan group name.

Figure 10-3: How to enable terminate compute instances

Task 10.2.2 Enable terminate volume groups plan group

Enable the plan group.

1. Select Enable all steps from the context menu to the right of the plan group name.

Figure 10-4: How to enable terminate volume groups

Task 10.3: Create various user-defined plans for the Switchover plan

We have already shown how to create the various user-defined plan for the switchover plan from Task 6.3 to 6.7. By using the similar process, create five custom user-defined plan groups. You must use the compute instance running in the Phoenix region for executing scripts.

1. Sync Scheduled Parameters from primary to standby /home/opc/oic-scripts/oic-sync-schedule-parameters.sh IAD.
2. Activate relevant integrations at standby /home/opc/oic-scripts/oic-integration-switch.sh activate IAD.
3. Start scheduled integrations at standby /home/opc/oic-scripts/oic-integration-schedule.sh start IAD.
4. Update DNS record at standby /home/opc/oic-scripts/dns-record-update.sh IAD.
5. Deactivate scheduled integrations at primary /home/opc/oic-scripts/oic-integration-switch.sh deactivate IAD.

Once those are created,the switchover plan should now include the five DR Plan Groups for Oracle integration as shown in the screenshot below. You may have additional plan groups if your protection group includes other applications or OCI services along with Oracle integration.

Figure 10-21: Showing the five user-defined plan groups added to the switchover plan

Task 11: Customize the failover plan in region 1 (Ashburn)

This task explains how to add custom, user-defined DR Plan Groups and steps to manage those things that need to be accomplished during a failover for Oracle Integration at region 1 during an actual outage or loss of access to region 2. These will be a subset of the same steps that were just added to the switchover plan in Task 10 above. However, only steps that are executed at standby region 1 will be added to the failover plan since it is assumed region 2 is completely inaccessible during a failover.

1. Activate relevant integrations at region 1.
2. Update scheduled parameters at region 1.
3. Start scheduled integrations at region 1.
4. Update DNS record at region 1.

Task 11.1: Select the switchover plan

Begin by navigating to the failover plan created in Task 9.

1. Ensure standby region 2 is still the current region context in the console.
2. Select the failover plan.

Figure 7-1: How to create begin customizing the failover plan in region 2

Task 11.2: Create multiple user-defined plan groups at region 1 (standby)

We have already shown how to create the various user-defined plan for the failover plan from Task 7.2 to 7.5. By using the similar process, create the below four custom user-defined plan groups. You must use the compute instance running in the Phoenix region for executing scripts.

1. Activate relevant integrations at standby /home/opc/oic-scripts/oic-integration-switch.sh activate IAD.

2. Update scheduled parameters at standby /home/opc/oic-scripts/oic-update-parameters.sh IAD.

3. Start scheduled integrations at standby /home/opc/oic-scripts/oic-integration-schedule.sh start IAD.

4. Update DNS record at standby /home/opc/oic-scripts/dns-record-update.sh IAD.

Once those are created, the failover plan should now include the four DR Plan Groups for Oracle integration as shown in the screenshot below. You may have additional plan groups if your protection group includes other applications or OCI services along with Oracle integration.

Figure 11-14: Showing the four user-defined plan groups added to the failover plan

Next Steps

OCI Full Stack DR for Oracle Integration should be fully implemented at this point. However, full functionality should be validated before using OCI Full Stack DR for production. All failover and switchover plans should be executed to validate that everything works as expected and the recovery team fully understands the entire process.

Testing switchover plans

Switchover plans are designed to clean up all artifacts and ensure all roles for built-in recovery steps such as load balancer, block storage, file systems, BaseDB, ExaCS and Autonomous Data Base are ready to recovered from the standby region without human intervention.

Testing failover plans

Failovers are different. Failovers by their very nature cannot clean up artifacts or ensure services and databases at the failed region are ready to transition workloads back to region 1. The recovery team needs to understand and perform tasks to ensure Data Guard is in the correct state, artifacts for storage and compute instances have been terminated, etc. Please read Resetting DR Configuration After a Failover in OCI Full Stack DR documentation to understand the process.

Validate all DR plans for final acceptance

The recovery team needs to perform a final validation to demonstrate the readiness of OCI Full Stack DR protection groups and plans for production workloads. Region 2 (Phoenix) should be the primary region at this point in the process. Begin final validate of all plans by completing the following steps:

• Test switchover from region 2 (primary) back to region 1 (standby).

• Test failover from region 1 (primary) to region 2 (standby).

• Prepare region 1 (primary) for failover from region 2.

• Test failover from region 2 (primary) to region 1 (standby).

• Prepare region 2 (primary) for either a failover or switchover to region 2.

• The DR protection groups and application stack should be in a normal operational state and ready for a failover or switchover at this point.

Note: To make sure that the same client application can be used to authenticate the rest APIs for both the Oracle Integration instances, you can add the scope of both the instances ( i.e. Primary and Secondary) to the OAuth client application. For the setup of OAuth client application, you can refer to the official documentation.

Related Links

• Technical Paper: Configuring a Disaster Recovery Solution for Oracle Integration Generation 2

• Sample scripts: Download sample scripts from GitHub

• OCI Full Stack DR Oracle Integration playlist in YouTube

  • Deploy Oracle Integration for Disaster Recovery (video 1)

  • Automate Disaster Recovery operations for Oracle Integration using OCI Full Stack DR (video 2)

  • Scripts used to automate recovery for Oracle Integration (video 3)

• OCI Full Stack Disaster Recovery

• OCI Full Stack Disaster Recovery Documentation

• OCI Full Stack Disaster Recovery - User defined group scripts

• Join #full-stack-dr public slack channel

Acknowledgments

• Author - Suraj Ramesh (Full Stack Disaster Recovery Product Manager)

• Contributors - Bonnie Rockey (Cloud Engineering,Oracle Integration Specialist), Akshay Saxena (Cloud Engineering,Oracle Integration Specialist)

Video 1: Deploy Oracle Integration for Disaster Recovery Video 2: Automate Disaster Recovery operations for Oracle Integration using OCI Full Stack DR Video 3: Scripts used to automate recovery for Oracle Integration

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.

---

Title and Copyright Information

Automate Recovery for Oracle Integration using OCI Full Stack Disaster Recovery

F98207-01

May 2024

Copyright ©2024,

Oracle and/or its affiliates.