Note:

This tutorial requires access to Oracle Cloud. To sign up for a free account, see Get started with Oracle Cloud Infrastructure Free Tier.
It uses example values for Oracle Cloud Infrastructure credentials, tenancy, and compartments. When completing your lab, substitute these values with ones specific to your cloud environment.

Automate Recovery for Multi-Node Oracle E-Business Suite Using OCI Full Stack Disaster Recovery

Introduction

OCI Full Stack Disaster Recovery relies on application engineering teams to design, write and maintain their own disaster recovery capabilities and features. If an application engineering team has built-in, OCI native disaster recovery capabilities that include OCI software development kit (SDK) APIs for disaster recovery, then that application can be added to Full Stack DR as a native member resource.

Oracle E-Business Suite does not currently have any OCI native disaster recovery capabilities or features and as such, is not a native member resource in Full Stack DR. However, the Oracle E-Business Suit engineering team has documented a process for manually deploying and recovering EBS across OCI regions. The manual process can be automated using Full Stack DR. This document explains how to automate the manual recovery process by adding custom, user-defined plan groups and steps to basic DR plans for failover,switchover and drills.

Oracle Cloud Infrastructure Full Stack Disaster Recovery (OCI Full Stack DR) orchestrates the transition of compute, database, and applications between Oracle Cloud Infrastructure (OCI) regions from around the globe with a single click. Customers can automate the steps needed to recover one or more business systems without redesigning or re-architecting existing infrastructure, databases, or applications.

Oracle E-Business Suite supports today’s evolving business models, drives productivity, and meets the demands of the modern mobile user. Building on a 30-year history of innovation, Oracle E-Business Suite continues to deliver new application functionality and expand the capabilities of existing features while helping you gain all the benefits of OCI.

Oracle E-Business Suite is Normally Part of a Larger System

Oracle E-Business Suite is normally the foundation of a more complex business system that includes a number of additional databases, OCI marketplace applications and OCI services that all need to be recovered as a single unit. To keep things simple and focused on Oracle E-Business Suite, this tutorial shows only those tasks related to Oracle E-Business Suite. But it is very unusual for Oracle E-Business Suite to be the only application that is part of DR Protection Group and DR plans.

Caution about Implementing Incrementally

Adding or deleting more members to a DR Protection Group after creating DR plans will refresh the existing DR plans in the protection groups at both regions. For more information, see Refresh a Disaster Recovery Plan.

OCI Full Stack DR is designed with the assumption the entire application stack for a given business system is already deployed across OCI regions and manual DR has already been proven to work. If your business system includes more than Oracle E-Business Suite, then add all other compute, storage, databases, applications or OCI services to the DR Protection Groups before creating any DR plans.

How the Recovery Works

The recovery solution for Oracle E-Business Suite requires OCI Full Stack DR to execute a series of custom bash scripts during a recovery operation such as a failover or switchover or drills. In this tutorial, we will use the Oracle E-Business Suite application nodes as the control node for hosting and running scripts.

The scripts referenced in this tutorial are provided by the Oracle EMEA Cloud Engineering Oracle Apps to OCI specialist team and available in Oracle E-Business Suite scripts for OCI Full Stack Disaster Recovery. The bash scripts may need to be modified slightly to fit the unique needs of your deployment.

Fig 1: Oracle E-Business Suite Scripts available in Oracle sample GitHub repository

This tutorial explains how to download the scripts and how to use them in a later step. This tutorial uses Option 1 for hosting the bash scripts only because the tutorial does not include anything other than Oracle E-Business Suite.

Note: The following scripts are provided for generic guidance. You can either use your own scripts or customize the scripts according to your corporate policy and security requirements.

Option 1 for Hosting Scripts

Install the bash scripts on the Oracle E-Business Suite application servers running at both regions. This tutorial will refer to the compute instance(s) where you install the scripts as the Control Node or DR Node even though it is one or more Oracle E-Business Suite application servers in your application stack.

Option 2 for Hosting Scripts

Create a specialized Control Node or DR Node to host all Oracle E-Business Suite scripts plus scripts needed for other applications or OCI services.

Oracle E-Business Suite is most often part of a larger, more complex business system that includes custom in-house applications, Oracle Cloud Marketplace applications, Oracle Business Analytics plus other databases, compute instances and home-grown applications. In this case, simply choose any one of the compute instances that are already part of the business system to host the scripts. The selected compute instance can be anything where Oracle Linux is installed and will most likely be an existing VM that serves another purpose like an application server or admin server of some sort.

Normally, OCI Full Stack DR does not require any specialized management servers to automate recovery operations. However, you will create a compute instance that will act as a specialized management server in this case. The specialized management server is seen throughout this document as Control Node or DR Node. The entire purpose of the control node is simply to act as a server where all custom scripts can reside and be called by OCI Full Stack DR during a recovery operation.

Note: In this tutorial, we will use Option 1, where we will store the custom Oracle E-Business Suite application related scripts in the Oracle E-Business Suite application servers.

Oracle E-Business Suite Deployment Architecture

Oracle E-Business Suite must first be deployed for disaster recovery (DR) across OCI regions before introducing OCI Full Stack DR. It is very important that the manual steps for performing a recovery are tested and work correctly before attempting to automate the recovery process using OCI Full Stack DR.

Either of the following two reference architectures shown below can be followed when deploying Oracle E-Business Suite for DR across OCI regions. Both reference architectures illustrate a multi-tier topology with redundant resources distributed across two OCI regions.

The multi-instance databases are built with Oracle Real Application Clusters (Oracle RAC) and use Oracle Data Guard to keep the database in sync across two OCI regions.

Reference Architecture for Oracle E-Business Suite using Oracle Base Database Service

Choose this deployment architecture if Oracle E-Business Suite is using Oracle Base Database Service. Oracle E-Business Suite must be deployed for DR across two OCI regions as explained in Business Continuity for Oracle E-Business Suite Release 12.2 with Oracle Database 19c on Oracle Base Database Service DB Systems (Doc ID 2875417.1). The illustration shown in Figure 1 was obtained from Doc ID 2875417.1.

Fig 1: Oracle E-Business Suite DR deployment architecture using OCI Base Database (refer to Doc ID 2875417.1 for more details)

Reference architecture for Oracle E-Business Suite using Oracle Exadata Database Service on Dedicated Infrastructure

Choose this deployment architecture if Oracle E-Business Suite is using Oracle Exadata Database Service on Dedicated Infrastructure. Oracle E-Business Suite must be deployed for DR across two OCI regions as explained in Business Continuity for Oracle E-Business Suite Release 12.2 with Oracle Database 19c on Oracle Exadata Database Service on Dedicated Infrastructure (Doc ID 2919723.1). The illustration shown in Figure 1 was obtained from Doc ID 2919723.1.

Fig 2: Oracle E-Business Suite DR deployment architecture with OCI Exadata Database (refer to Doc ID 2919723.1 for more details)

Difference Between Logical Hostnames and Physical Hostnames

Oracle E-Business Suite strongly suggests that logical hostnames be used for the application servers and physical hostnames be used for the database. Both Business Continuity for Oracle E-Business Suite Release 12.2 with Oracle Database 19c on Oracle Base Database Service DB Systems (Doc ID 2875417.1) and Business Continuity for Oracle E-Business Suite Release 12.2 with Oracle Database 19c on Oracle Base Database Service DB Systems (Doc ID 2919723.1) written by Oracle E-Business Suite engineering require that the application tier uses logical hostnames while the database tier uses physical hostnames.

OCI Full Stack DR has no requirements on whether you use logical or physical hostnames, but we recommend following Oracle E-Business Suite requirements to ensure support from Oracle E-Business Suite if problems arise. Some customers use logical hostnames for both application and database tiers. The scripts provided with this solution are designed to be used with logical hostnames for the application tier and physical hostnames database tier; see Task 1.2 for download instructions.

OCI Full Stack DR Deployment Architecture

The following illustrations show the compute resources added as members to each DR Protection Group (DRPG) for OCI Full Stack DR. These represent the various components that OCI Full Stack DR can manage outside the Oracle E-Business Suite application.

OCI Full Stack DR has built-in automation to handle OCI Compute, OCI Block Storage, OCI File Storage, Oracle databases, OCI Load Balancer, Oracle Cloud Infrastructure Kubernetes Engine (OKE) clusters and many other resources during a recovery, but it does not have built-in automation for Oracle E-Business Suite itself. Oracle E-Business Suite recovery is controlled by a series of bash scripts that can be downloaded from a GitHub repository dedicated to this tutorial. The bash scripts need to be installed on a compute instance of your choice by following any of the following options for placement and control of the scripts.

Option 1: Automate Recovery for Oracle E-Business Suite using Oracle Base Database Service

This deployment architecture is not typical and was devised for very rare situations where Oracle E-Business Suite is the only application being recovered by OCI Full Stack DR. In this case, we will host the custom scripts in the Oracle E-Business Suite application nodes.

Fig 3: Full Stack DR deployment architecture using OCI Base Database service

Option 2: Automate Recovery for Oracle E-Business Suite using Oracle Exadata Database Service on Dedicated Infrastructure

The simplistic deployment architecture shown in Figure 4 is an example of a more common deployment of Oracle E-Business Suite where it is simply one component of a larger, more complex application stack where many services and applications need to be recovered together. Most business systems are much more complex than the fictitious one shown in the following image and usually include additional databases, other Oracle and/or non-Oracle applications, along with other OCI services such as OIC, ODI, OHS, OCI IAM, and so on.

Fig 4: Full Stack DR deployment architecture using Oracle Exadata Database Service on Dedicated Infrastructure

Definitions and Assumptions throughout the Tutorial

Regions

The role of primary and standby are a function of the OCI Full Stack DR protection groups, not the regions themselves. A region can function as primary for one application stack while also function as standby for a completely different application stack. The role of primary and standby are fluid.

This tutorial begins with the Oracle E-Business Suite application stack running in Phoenix and all the standby resource running in Ashburn. These two regions are examples only; any two OCI regions that support your application stack can be used in practice.

Region 1 is Phoenix.
- Phoenix will start out as the primary region.
- Phoenix begins as primary region because:
  - The Oracle E-Business Suite application is running in Phoenix.
  - The Oracle E-Business Suite rsync cron job is copying data from /u01 on the Oracle E-Business Suite applications servers (VMs) in Phoenix to the application servers running in Ashburn.
  - The database in Phoenix has the Oracle Data Guard primary role.
- Phoenix will eventually become the standby after you are instructed to perform a switchover in later tasks.
Region 2 is Ashburn.
- Ashburn will start out as the standby region.
- Ashburn begins as standby region because:
  - The Oracle E-Business Suite standby application servers (VMs) are running in Ashburn.
  - The Oracle E-Business Suite application is not running in Ashburn.
  - Data in /u01 is being copied from Phoenix to the VMs in Ashburn.
  - The database in Ashburn has the Oracle Data Guard standby role.
- Ashburn will eventually become the primary after you are instructed to perform a switchover in later tasks.

Compartments

You are free to organize Oracle E-Business Suite and OCI Full Stack DR into any compartment scheme that works within your standards for IT governance. We have chosen to organize applications into their own individual compartments, then organize all DR Protection Groups into a single compartment where completely different business systems can all be seen at a glance.

Organizing all DR protection groups into a single compartment apart from the applications makes it much easier for IT staff to locate and execute DR plans for many completely different business systems.
Having a single compartment for all DR protection groups helps eliminate human error and increases the speed in which DR plans can be found and executed.
- Compartment for Oracle E-Business Suite (ebstesting). The compartment for Oracle E-Business Suite itself, storage, storage buckets, compute and database related to Oracle E-Business Suite is ebstesting in this tutorial.
- Compartment for OCI Full Stack DR myprojects. The compartment for OCI Full Stack DR protection groups and plans is myprojects in this tutorial.

Objectives

The following tasks will be covered in this tutorial explaining how to automate recovery for Oracle E-Business Suite using OCI Full Stack DR.

Note: We start with Region 1 (Phoenix) and Region 2 (Ashburn). Production Oracle E-Business Suite application runs in Region 1 and DR is configured in Region, this is really important as throughout the tutorial we refer as Region 1 and Region 2 rather than the actual region name.If your deployment uses different OCI regions, then please make sure to refer to those respective Production and DR regions.

Task 1: Deploy Oracle E-Business Suite for DR across OCI regions.
1. Manually install and deploy Oracle E-Business Suite for DR across two OCI regions.
2. Manually test all recovery steps from desired Region 1 to Region 2.
3. Manually test all recovery steps from desired Region 2 to Region 1.
Task 2: Prepare for OCI Full Stack DR.
1. Set up OCI IAM policies for Full Stack DR.
2. Set up OCI IAM policies for other OCI services.
3. Download and install custom Oracle E-Business Suite scripts on Oracle E-Business Suite application servers.
4. Ensure Oracle E-Business Suite application servers can execute scripts and commands.
5. Create vault secrets for the Oracle E-Business Suite database.
6. Create object storage buckets for logs.
Task 3: Create and associate DR Protection Groups.
Task 4: Add Oracle E-Business Suite database and compute members to region 1 and region 2 DR protection groups.
Task 5: Create the DR plans in region 2 (Ashburn).
1. Create switchover plan.
2. Create failover plan.
3. Create start drill plan.
Task 6: Customize the DR plans in region 2 (Ashburn).
Task 7: Execute the switchover plan in region 2 (Ashburn).
Task 8: Create basic DR plans in region 1 (Phoenix) and customize the DR plans in region 1 (Phoenix).
1. Create switchover plan and customize the switchover plan in region 1 (Phoenix).
2. Create failover plan and customize the failover plan in region 1 (Phoenix).
3. Create start drill plan and customize the start drill plan in region 1 (Phoenix).

Prerequisites

Oracle E-Business Suite should be deployed for disaster recovery across both regions before beginning work with OCI Full Stack DR. This is covered in Task 1.

Task 1: Deploy Oracle E-Business Suite for Disaster Recovery

OCI Full Stack DR is not involved in any part of this task.

Oracle E-Business Suite must be deployed for disaster recovery across OCI regions using one of two different My Oracle Support (MOS) knowledge articles (KM notes) before beginning any work with OCI Full Stack DR.

It is also very important that the manual steps described in the KM notes along with the custom scripts/automation needed to recover Oracle E-Business Suite have been fully tested before beginning any work with OCI Full Stack DR.

Oracle E-Business Suite currently supports Oracle Base Database Service and Oracle Exadata Database Service on Dedicated Infrastructure services. Oracle Autonomous Database service is not currently supported with Oracle E-Business Suite. Follow the instructions found in one or the other of the following KM notes:

Task 2: Prepare Tenancy for OCI Full Stack DR

OCI Full Stack DR is not involved in any part of this task. The following steps prepare the tenancy, compartment, OCI services and Oracle E-Business Suite for automated recovery by OCI Full Stack DR. Most tasks articulated in this section are discussed in Prerequisites for Full Stack Disaster Recovery.

Task 2.1: Set up OCI IAM Policies for OCI Full Stack DR

Configure the required OCI IAM policies for OCI Full Stack DR as outlined in the following documents.

Task 2.2: Set up OCI IAM Policies for Other Services Managed by OCI Full Stack DR

OCI Full Stack DR must have the ability to control and manage other key OCI services such as compute, networking, storage, vaults, databases and other miscellaneous services. Configure the required OCI IAM policies for other services as explained in the following document

Policies for Other Services Managed by Full Stack Disaster Recovery.

Task 2.3: Download and Install Custom Oracle E-Business SuiteScripts on Oracle E-Business Suite Application servers

OCI Full Stack DR has built-in intelligence to orchestrate recovery for OCI Infrastructure as a Service (IaaS) and Platform as a Service (PaaS) resources. OCI Full Stack DR does not have built-in intelligence to orchestrate recovery for Oracle E-Business Suite since Oracle E-Business Suite does not currently provide OCI native disaster recovery APIs to deploy or manage their own built-in disaster recovery.

However, OCI Full Stack DR can still orchestrate recovery for Oracle E-Business Suite by adding user-defined DR Plan Groups and steps to the base DR plans created later in this tutorial. As discussed in Part 1 of this tutorial, the user-defined steps will call a series of custom scripts that perform various tasks needed to recover Oracle E-Business Suite. The scripts should be downloaded and installed on all compute instances on the Oracle E-Business Suite application servers in both regions:

Download the Oracle E-Business Suite scripts from here: Oracle E-Business Suite scripts for OCI Full Stack Disaster Recovery.
Copy the scripts to any directory on each of the Oracle E-Business Suite application servers in region 1.
Copy the scripts to the same directory location on each of the Oracle E-Business Suite application servers in region 2.
Ensure files are owned by oracle user.
Ensure the files are executable.

Task 2.4: Ensure Oracle E-Business Suite Application Servers can Execute Scripts and Commands

OCI Full Stack DR will need to execute the Oracle E-Business Suite scripts that were downloaded to the Oracle E-Business Suite application servers. OCI Full Stack DR executes the scripts using the Oracle Cloud Agent on each compute instance. It is very important that all the compute instances used as Oracle E-Business Suite application servers in both regions are able to execute commands through the OCI Console using the Oracle Cloud Agent.

For more information, see Preparing Compute Instances for Full Stack Disaster Recovery. Pay particular attention to the instructions for Running Commands with Administrator Privileges.

Validate that Commands can be Executed from the OCI Console

Note: This task just ensures the Oracle Could Agent can run commands. It does not validate that appropriate policies are in place to allow Full Stack DR to execute commands through the Oracle Cloud Agent. This will become apparent in later steps when the tutorial explains how to validate the switchover DR Plan in region 2.

Use the Run command feature of compute instanced to validate the Oracle Cloud Agent is able to execute commands. The following images shows the Run command in the details page for compute instances in the OCI Console.

Select Run command.
Select Create command and type any valid Linux command such as date. Change the timeout to something reasonable like 3 minutes before executing the command.

Figure 2.4.1.1: Execute a command from the OCI console

Task 2.5: Create OCI Vault Secrets for the Oracle E-Business Suite Database

OCI Full Stack DR will need the database sys password so it can automatically trigger Oracle Data Guard during failovers and switchovers. Add the database sys password to a vault secret in both regions as explained here: Preparing Oracle Databases for Full Stack Disaster Recovery.

Task 2.6: Create OCI Object Storage Buckets for Logs

Note: Skip Task 2.3 entirely if you are adding Oracle E-Business Suite to existing DR Protection Groups.

Create OCI Object Storage Buckets in the primary and standby regions to store logs generated by OCI Full Stack DR during recovery operations as explained here: Preparing Log Location for Operation Logs.

Task 2.6.1: Navigate to OCI Object Storage

Begin by navigating to Object Storage & Archive Storage as shown in Figure 2-1.

Ensure the browser context is set to region 1 (Phoenix).
Click Storage.
Click Buckets.

Figure 2.6.1.1: Navigate to object storage

Task 2.6.2: Create an OCI Object Storage Bucket in Region 1 and 2

Create an OCI Object Storage Bucket in region 1. Then create an identical storage bucket in region 2. The buckets will be assigned to the DR protection groups in region 1 and 2 in a later task.

Select the compartment that contain Oracle E-Business Suite related resources. The compartment can be different in each region.
Click Create Bucket.
Give the bucket a meaningful name that easily identifies which application and purpose it serves; there is no reason to include the region as part of the name. For example, this name indicates it is used for the OCI Full Stack DR logs related to DR operations for Oracle E-Business Suite.
Use the default value for Tier and Encryption.
Click Create to create the bucket.

Figure 2.6.2.1: Create an object storage bucket in region 1 & 2

Task 3: Create and Associate DR Protection Groups

This will be the first task that involves Full Stack DR. Create DR protection groups in region 1 and region 2 if the protection groups for this application stack do not exist yet.

Note: Skip Task 3 entirely if Oracle E-Business Suite is being added to existing DR Protection Groups.

This task begins the first step of configuring OCI Full Stack DR for the application stack that includes Oracle E-Business Suite. DR Protection Groups inform OCI Full Stack DR which OCI IaaS and PaaS services are part of a single application stack and which two regions will act as primary and standby for the business system. DR protection groups form the foundation on which all else is built.

All OCI IaaS and PaaS resources that belong to the application stack in both regions will be added as part of Task 4.

Note: Although this tutorial only includes Oracle E-Business Suite, DR Protection Groups normally contain OCI IaaS and PaaS services for many different Oracle and non-Oracle applications and in addition to those needed by Oracle E-Business Suite.

Task 3.1: Create a Protection Group in Standby Region 2 (Ashburn)

Begin by creating an unassociated DR Protection Group in region 2. There is no requirement that the protection group be created in the standby region, the process just flows a little better by starting in region 2.

Task 3.1.1: Navigate to DR Protection Groups

Begin by navigating to DR Protection Groups (OCI Full Stack DR) as shown in Figure 3.1.1.

Ensure the OCI region context is set to region 2 (Ashburn).
Click Migration & Disaster Recovery.
Click DR Protection Groups.

Figure 3.1.1: Navigate to DR protection groups

Task 3.1.2: Create the Protection Group

Create a basic DR protection group (DRPG) in region 2 as shown in Figure 3-1-2. The peer, role and members will be assigned in later steps.

Select the compartment where you want the DRPG to be created. This can be the same compartment where Oracle E-Business Suite resources exist or any other compartment that is related to the project.
Select Create DR protection group to open the dialog box where you will enter the parameters to create the protection group.

Figure 3.1.2: Begin creating DR protection group in region 2

Task 3.1.3: Add Parameters Needed to Create the Protection Group

Add a name and OCI Object Storage bucket for the logs as shown in Figure 3.1.3.

Use a meaningful, simple name for the DRGP; this example shows the name of the business system and the region.
Select the compartment where you want the DRPG to be created in region 2.
Select the object storage bucket created in Task 2 for region 2. You may need to change the compartment if you created the bucket in a different compartment.

Do not enter any additional parameters. Click Create at the bottom of the dialog box (not shown here).

Figure 3.1.3: Parameters needed to create DR protection group in region 2

What you should see after creating the protection group

The first DR protection group will be created as shown in Figure 3-2.3. The roles and peer information will be assigned as part of Task 3.4.

Figure 3.1.4: Showing the newly created DR protection group in region 2

Task 3.2: Create a Protection group in Primary Region 1 (Phoenix)

Create a protection group in region 1. In a later task, you will add all the OCI resources that belong to the application stack in this region. This is how OCI Full Stack DR knows what assets are considered to be part of a business system in this region.

Task 3.2.1: Create the Protection Group

Create a basic DR protection group (DRPG) in region 1 as shown in Figure 3.2.1. The peer, role and members will be assigned in later steps.

Change the OCI region context to region 1.
Select the compartment where you want the DRPG to be created. People normally choose the same compartment that was used to create the DRPG in region 2.
Select Create DR protection group to open the dialog box where you will enter the parameters to create the protection group.

Figure 3.2.1: Begin creating DR protection group in region 1

Task 3.2.2: Add Parameters Needed to Create the Protection Group

Add a name and object storage bucket for the logs as shown in Figure 3-5.

Use a meaningful, simple name for the DR PRotection group; this example shows the name of the business system and the region.
Select the compartment where you want the DRPG to be created in region 1. People generally use the same compartment in both regions.
Select the object storage bucket created in Task 2 for region 1. You may need to change the compartment if you created the bucket in a different compartment.

Figure 3.2.2: Parameters needed to create DR protection group in region 1

What you should see after creating the protection group

The second DR protection group will be created as shown in Figure 3.2.3. The roles and peer information will be assigned as part of Task 3.3.

Figure 3.2.3: Showing the newly created DR protection group in region 2

Task 3.3: Associate Protection Groups in Region 1 and Region 2

Associate the DRPGs in each region as peers of each other and assign the roles of primary and standby. This is how OCI Full Stack DR will know which two regions work together for Oracle E-Business Suite recovery. The roles of primary and standby are automatically changed by OCI Full Stack DR as part of any DR operation/DR plan execution; there is no need to manage the roles manually at any time after the DRPGs are assigned their initial roles in this task.

Task 3.3.1: Begin the Association

Ensure OCI region context is set to region 1 (Phoenix).
Click Associate to begin the process.

Figure 3.3.1: Begin DRPG association

Task 3.3.2: Associate Protection Groups in Region 1 and Region 2

Provide the parameters as shown in Figure 3.3.2.

Select primary role. OCI Full Stack DR will assign the standby role to region 2 automatically.
Select region 2 (Phoenix) where the other DRPG was created.
Select the peer DRPG that was created in.

Figure 3.3.2: Parameters needed to associate the DRPGs

What you should see after association is complete:

OCI Full Stack DR will show something like Figure 3.3.3 once the association is completed.

The current primary peer DRPG is Phoenix (region 1).
The current standby peer DRPG is Ashburn (region 2).

Figure 3.3.3: Showing the peer relationship from the individual DRPG perspective

The same information can be found whenever the context/view is from a global perspective showing all DR protection groups as shown in Figure 3.4.3.2.

The current primary peer DRPG is Phoenix (region 1).
The current standby peer DRPG is Ashburn (region 2).

Figure 3.4.3.2: Showing the peer relationship from the global DRPG perspective

Task 4: Add Oracle E-Business Suite Members to Region 1 and Region 2 DR Protection Groups

Add the database and non-moving Oracle E-Business Suite application servers as members of the DR protection group (DRPG) in both regions. Non-moving compute means the Oracle E-Business Suite application servers that exist in region 1 are never launched or started in region 2. The boot volumes for non-moving compute are not replicated using OCI Storage replication to region 2 and are not added as members of either DR protection group.

This tutorial only shows steps related to Oracle E-Business Suite, but you should also take this opportunity to add any additional OCI IaaS and PaaS services that need to be recovered along with Oracle E-Business Suite. For example, there may be other moving compute, non-moving compute, databases, load balancers, file systems, block storage or object storage associated with other in-house or Oracle applications that are part of the Oracle E-Business Suite ecosystem.

Note: This task will refresh any existing DR plans in both regions when adding members to existing DR Protection Groups. For more information, see Refresh a Disaster Recovery Plan.

Task 4.1: Add Member Resources to DR Protection Group to Region 1 (Phoenix)

You will add the following resources as members of the primary DRPG in region 1.

Two non-moving Oracle E-Business Suite application servers (Oracle E-Business Suite installed and running).
The Oracle RAC database for Oracle E-Business Suite (Oracle Data Guard primary peer).

Task 4.1.1: Navigate to Primary DR Protection Group

Navigate to the DR Protection Group in region 1 as shown in Figure 4.1.1.

Ensure the OCI region context is region 1 (Phoenix).
Select the DR Protection Group in region 1.

Figure 4.1.1: Navigate to DR protection group in region 1

Task 4.1.2: Add Oracle E-Business Suite Application Server 1

Begin by opening the Add member dialog box to add the first application server.

Click Members.
Click Add member.

Figure 4.1.2.1: Open the Add members dialog box

Select application server 1 in region 1 as shown in Figure 4.1.2.2. You do not need to add any block volume groups or specify any network properties since this is a non-moving instance.

Select Compute as Resource Type.
Choose the compartment containing the Oracle E-Business Suite application servers and select the compute instance you have designated as application server 1.
Select Non-moving instance. This informs OCI Full Stack DR that it should not attempt to launch a replicated virtual machine in the standby region during a DR operation.
Click Add to add the compute instance to the DRPG (not shown).

Figure 4.1.2.2: Specify parameters for application server 1

Task 4.1.3: Add Oracle E-Business Suite Application Server 2

Click Add member.

Figure 4.1.3.1: Open the Add members dialog box

Select application server 2 in region 1 as shown in Figure 4.1.3.2. You do not need to add any block volume groups or specify any network properties since this is a non-moving instance.

Select Compute as Resource Type.
Choose the compartment containing the Oracle E-Business Suite application servers and select the compute instance you have designated as application server 2.
Select Non-moving instance. This informs OCI Full Stack DR that it should not attempt to launch a replicated virtual machine in the standby region during a DR operation.
Click Add to add the compute instance to the DRPG (not shown).

Figure 4.1.3.2: Specify parameters for application server 2

Task 4.1.4: Add Primary Clustered Oracle Database

Click Add member to add the database as a member.

Figure 4.1.4.1: Open the Add members dialog box

Select the RAC database for Oracle E-Business Suite in region 1 as shown in Figure 4.1.4.2.This also works exactly the same for single instance databases.

Choose Database as the Resource Type. The Database resource type allows the choice of either Oracle Base Database Service or Oracle Exadata Database Service on Dedicated Infrastructure which are both supported by Oracle E-Business Suite engineering as valid choices.
The figure image shows Oracle Base Database Service, but you can also use the popular Oracle Exadata Database Service on Dedicated Infrastructure.
Select the values for these fields that match your deployment. Ask your database administrator if you do not know the answers to these selections.
Select the compartment and vault secret containing the password for the database. You should have created a similar vault and secret in region 2 as part of Task 2.4. This parameter is in the dialog for compatibility with other DBaaS APIs, but the password is not actually used at all.
Click Add to add the database as a member (not shown).

Figure 4.1.4.2: Specify parameters for RAC database in region 1

The list of members in the DR Protection Group should look something like the screenshot shown in Figure 4.1.4.3. Your list may look different if you are adding more than Oracle E-Business Suite to this particular DR Protection Group or you are adding Oracle E-Business Suite member resources to an existing DR protection group.

Figure 4.1.4.3: Complete list of members needed for Oracle E-Business Suite in region 1

Task 4.2: Add Member Resources to DRPG in Region 2 (Ashburn)

You will add the resources shown in the list as members of the standby DRPG in region 2.

The two non-moving Oracle E-Business Suite application servers (Oracle E-Business Suite installed, but not running).
The RAC database for Oracle E-Business Suite (Oracle Data Guard standby peer).

Oracle E-Business Suite requires that application servers exist in both regions with the application installed. The application server(s) should be in a running state at all times in the standby region with the Oracle E-Business Suite application installed, but not running in the standby region.

This means Operating System and application upgrades, patches, and other routine maintenance must be performed independently in both regions.

OCI Full Stack DR refers to this model of as non-moving compute since the VMs are not replicated to another region and never move to any other regions during a DR operation. Boot devices for compute instances that are designated as non-moving compute in OCI Full Stack DR should not be replicated to the standby region. Therefore, no replicated block volume groups containing boot devices for non-moving compute are added to the DR Protection Groups (DRPG).

Note: As mentioned previously, it is unusual for Oracle E-Business Suite to be the only application or service associated with a pair of OCI Full Stack DR Protection Groups. You may also have other IaaS and PaaS resources as members of the DR Protection Group in the standby region.

Task 4.2.1: Navigate to Standby DR Protection Group

Navigate to the DR Protection Group in region 2 as shown int Figure 4.2.1.

Ensure the OCI region context is region 2 (Ashburn).
Select the DR Protection Group in region 2.

Figure 4.2.1: Navigate to DR protection group in region 2

Task 4.2.2: Add Oracle E-Business Suite Application Server 1

Begin by opening the Add member to add the first application server.

Select Members.
Click Add member.

Figure 4.2.2.1: Open the Add members dialog box

Select application server 1 in region 2 as shown in Figure 4.2.2.2. You do not need to add any block volume groups or specify any network properties since this is a non-moving instance.

Select Compute as Resource Type.
Choose the compartment containing the Oracle E-Business Suite application servers and select the compute instance you have designated as application server 1.
Select Non-moving instance. This informs OCI Full Stack DR that it should not attempt to launch a replicated virtual machine in the standby region during a DR operation.
Click Add to add the compute instance to the DRPG (not shown).

Figure 4.2.2.2: Specify parameters for application server 1

Task 4.2.3: Add Oracle E-Business Suite Application Server 2

Click Add member to add the 2nd application server.

Figure 4.2.3.1: Open the Add members dialog box

Select application server 2 in region 2 as shown in Figure 4.2.3.2. You do not need to add any block volume groups or specify any network properties since this is a non-moving instance.

Select Compute as Resource Type.
Choose the compartment containing the Oracle E-Business Suite application servers and select the compute instance you have designated as application server 2.
Select Non-moving instance. This informs OCI Full Stack DR that it should not attempt to launch a replicated virtual machine in the standby region during a DR operation.
Click Add to add the compute instance to the DRPG (not shown).

Figure 4.2.3.2: Specify parameters for application server 2

Task 4.2.4: Add Oracle Standby Clustered Database

Click Add member to add the database as a member.

Figure 4.2.4.1: Open the Add members dialog box

Select the RAC database for Oracle E-Business Suite in region 2 as shown in Figure 4.2.4.2. This also works exactly the same for single instance databases.

Choose Database as the Resource Type. The Database resource type allows the choice of either Oracle Base Database Service or Oracle Exadata Database Service on Dedicated Infrastructure which are both supported by Oracle E-Business Suite engineering as valid choices.
The figure below shows Oracle Base Database Service, but you can also use the popular Oracle Exadata Database Service on Dedicated Infrastructure.
Choose the values for these fields that match your deployment. Ask your database administrator if you do not know the answers to these selections.
Select the compartment and vault secret containing the password for the database. You should have created a similar vault and secret in region 2 as part of Task 2.4. This parameter is in the dialog for compatibility with other DBaaS APIs, but the password is not actually used at all .
Click Add to add the database as a member (not shown).

Figure 4.2.4.2: Specify parameters for RAC database in region 2

The list of members in the DR Protection Group should look something like the screenshot shown in Figure 4.2.4.3. Your list may look different if you are adding more than Oracle E-Business Suite to this particular DR Protection Group or you are adding Oracle E-Business Suite member resources to an existing DR protection group.

Figure 4.2.4.3: Complete list of members needed for Oracle E-Business Suite in region 2

Task 5: Create the DR Plans in Region 2

This task creates basic switchover, failover and start drill plans associated with the standby DR Protection Group in region 2 (Ashburn).

The purpose of each plan is to transition the workload from primary region 1 to standby region 2. The roles of the DR protection groups in both regions are automatically reversed as part of any DR operation, so the protection group in region 1 will become the standby and the protection group in region 2 will become primary after a failover or switchover.

OCI Full Stack DR will pre-populate both plans with built-in steps based on the member resources added in the previous tasks. The plans will be customized in later steps to handle all the tasks related to Oracle E-Business Suite System during a recovery operation.

The DR plans are always created in the protection group with the standby role; region 2 is currently the standby protection group, so we will begin in Ashburn.

Create a basic plans by selecting the DRPG in region 2 as shown in Figure 5-1.

Ensure the OCI region context is region 2 (Ashburn).
Select the standby DRPG in region 2.
Select Plans.
Click Create Plan to begin the process.

Figure 5-0: How to begin creating basic DR plans in region 2

Task 5.1: Create DR Plan for Switchover to Region 2

Creating a DR plan is simple as shown in Figure 5-2.

Make the name of the switchover plan simple but meaningful. The name should be as short as possible but easy to understand at a glance to help reduce confusion and human error during a crisis.
Select Plan type. There are only four plan types at the time of this writing.

Figure 5-1: The parameters needed to create DR switchover plan

Task 5.2: Create DR Plan for Failover to Region 2

Follow the same process to create a basic failover plan as shown in Figure 5-2.

Make the name of the failover plan simple but meaningful.
Select Plan type. There are only four plan types at the time of this writing.

Figure 5-2: The parameters needed to create DR failover plan

Task 5.3: Create DR Plan for Start Drill in Region 2

Follow the same process to create a start drill plan as shown in Figure 5-3.

Make the name of the start drill plan simple but meaningful.
Select Plan type. There are only four plan types at the time of this writing.

Figure 5-3: The parameters needed to create DR start drill plan

The standby DR Protection Group in region 2 should now have the three DR plans as shown in the following image. These will handle transitioning workloads from region 1 to region 2. You will create similar plans at region 1 to transition workloads from region 2 back to region 1 in a later task.

Figure 5-4: Showing the three basic DR plans that must exist in region 2 before proceeding any further

Note: OCI Full Stack DR will support creating a stop drill plan only after the start drill plan has been executed successfully. For the tutorial, it is out of scope to create a stop drill plan, but it is highly recommended that a stop drill be also created in both DR protection groups.

Task 6: Customize the DR Plans in Region 2 (Ashburn)

However, OCI Full Stack DR can still orchestrate recovery for Oracle E-Business Suite by adding user-defined DR Plan Groups and steps to the base DR plans created as part of Task 5. The user-defined steps call the Oracle E-Business Suite scripts that were downloaded and installed on the Oracle E-Business Suite application servers in both regions or dedicated DR control node as part of Task 2.3. The following are the user-defined plan groups at high level. The plan groups which needs to be added will vary depending on the DR plan types.

Shutdown Oracle E-Business Suite and disable rsync jobs on node2 at region 1.
Shutdown Oracle E-Business SuiteS and disable rsync jobs on node1 at region 1.
Clear node names in database fnd tables at region 2.
Setup app context in database on node1 at region 2.
Setup app context in database on node2 at region 2.
Execute autoconfig on node1 at region 2.
Execute autoconfig on node2 at region 2.
Startup Oracle E-Business Suite and enable rsync jobs on node1 at region 2.
Startup Oracle E-Business Suite and enable rsync jobs on node2 at region 2.

Task 6.1: Select the Switchover Plan

Begin by navigating to the switchover plan created in Task 5.

Figure 6.1: Select switchover plan in region 2

OCI Full Stack DR will generate the Prechecks-Built in and Databases-Switchover plan groups, this was generated based on the members we added to the DR protection groups in region 1 and region 2.

Figure 6.1.1: Default plan groups for switchover plan in region 2

Task 6.2: Create Plan Group to Shutdown Oracle E-Business Suite and Disable Rsync Jobs on Node 2 at Region 1

Now begin adding custom, user-defined DR Plan Groups.

The first user-defined plan group will shutdown Oracle E-Business Suite and disable rsync jobs on node2 at region 1. This plan group will contain two steps that calls the shutdownapps.sh and fsdr-rsync-ebs.sh bash scripts that was downloaded to the Oracle E-Business Suite app server nodes in Task 2.3.

Task 6.2.1: Select Add Plan Group

Begin the process to add a plan group.

Click Add group to begin.

Figure 6-2.1: Begin adding plan group to shutdown Oracle E-Business Suite and disable rsync jobs on node2

Task 6.2.2: Provide Plan Group Name, Order and Add Step

A DR Plan Group can contain many steps that are all executed in parallel. We are going to add two steps to execute bash scripts to stop Oracle E-Business Suite on node2 and disable rsync cron jobs on node2 at region 1. In our example name of node2 is ebshaapp02phx.

Give the plan group a simple but descriptive name.
Select a position where the plan group will be inserted into the DR plan. In this case, we are going to insert our user-defined plan group add before the Databases-Switchover plan group.
Select the built-in Databases - Switchover plan group.
We will be adding two steps. First step is to stop Oracle E-Business Suite on node 2 and second script is to disable rsync jobs on node 2.
Click Add Step to open the dialog where we will specify the script to stop Oracle E-Business Suite on node2.

Figure 6.2.2: Plan group to shutdown Oracle E-Business Suite and stop rsync jobs

Task 6.2.3: Provide Step Names and Local Script Parameters

The Add plan group step dialog allows us to specify parameters about what this one step will perform and how it will behave during recovery. We are going to add two steps in this plan group.

Step 1 is to stop Oracle E-Business Suite app on node2.
Step 2: is to disable rysnc jobs on node2.

For this set up, node 2 is ebshaapp02phx

We will explain all fields in this dialog, but leave out this detail in all the remaining screenshots in subsequent steps since we are just performing the same process repeatedly.

A descriptive name explaining what task this step performs.
Select the Region 1, so in our case it is Phoenix..
Select Run local script.
Select the correct compartment that contains the Oracle E-Business Suite app node2 and select node 2, in our case it is ebshaapp02phx.
Paste in the absolute path where you installed the shutdownapps.sh script on the Oracle E-Business Suite app node2 with the right parameters. You can refer the readme details from the GitHub link provided in Task 2.3 for the exact syntax.
Specify oracle as the user to execute the script.
The DR plan should stop if the script fails to stop Oracle E-Business Suite app on node2. This will allow anyone to see there is a problem and fix it. OCI Full Stack DR provides the opportunity to continue running the switchover plan after fixing the problem.
The default value before OCI Full Stack DR declares a failure is 60 minutes (3600 seconds). This value can be changed to 1800 seconds or whatever is felt to be a more realistic timeout value.
Click Add Step to add this step to the plan group.

Figure 6.2.3.1: Parameters to create the plan step for stop Oracle E-Business Suite on node 2
It will show as below once the step 1 is added.

Figure 6.2.3.2: Stop Oracle E-Business Suite on node 2 step is added
Click Add Step to add second step to this plan group.
A descriptive name explaining what task this step performs.
Select the Region 1, so in our case it is Phoenix.
Select Run local script.
Select the correct compartment that contains the Oracle E-Business Suite app node2 and select node 2, in our case it is ebshaapp02phx.
Paste in the absolute path where you installed the fsdr-rsync-ebs.sh script on the Oracle E-Business Suite app node2 with the right parameters. You can refer the readme details from the GitHub link provided in Task 2.3 for the exact syntax.
Specify oracle as the user to execute the script.
The DR plan should stop if the script fails to disable rsync jobs on node2. This will allow anyone to see there is a problem and fix it. OCI Full Stack DR provides the opportunity to continue running the switchover plan after fixing the problem.
The default value before OCI Full Stack DR declares a failure is 60 minutes (3600 seconds). This value can be changed to 300 seconds or whatever is felt to be a more realistic timeout value.
Click Add Step to add this step to the plan group.

Figure 6.2.3.1: Parameters to create the plan step to disable rysnc jobs on node 2
It will show as below once the step 2 is added.

Figure 6.2.3.2: Disable rysnc jobs on node 2 is added

Task 6.2.4: Complete Adding Plan Group and Steps

Both stop Oracle E-Business Suite app and disable rsync jobs on node2 are added to the DR plan group.

Click Add to add the DR plan group and steps to the DR plan.

Figure 6.2.4.1: Finalize adding plan group and steps to stop Oracle E-Business Suite and disable rsync job on node1
Verify the added plan group is available in the switchover plan.

Figure 6.2.4.1: Stop Oracle E-Business Suite and disable rsync job on node1 plan group added

Task 6.3: Create Plan Group to Shutdown Oracle E-Business Suite and Disable Rsync Jobs on Node 1 at Region 1

Now let us add the next user-defined plan group.

This user-defined plan group will Shutdown Oracle E-Business Suite and disable rsync jobs on node1 at region 1. This plan group will contain two steps that calls the shutdownapps.sh and fsdr-rsync-ebs.sh bash scripts that was downloaded to the Oracle E-Business Suite app server nodes in Task 2.3.

Task 6.3.1: Select Add Plan Group

Begin the process to add a plan group.

Click Add group to begin.

Figure 6-3.1: Begin adding plan group to shutdown Oracle E-Business Suite and disable rsync jobs on node1

Task 6.3.2: Provide Plan Group Name, Order and Add Step

A DR Plan Group can contain many steps that are all executed in parallel. We are going to add two steps to execute bash scripts to stop Oracle E-Business Suite on node1 and disable rsync cron jobs on node1 at region 1.In our example name of node1 is ebshaapp01phx.

Give the plan group a simple but descriptive name.
Select a position where the plan group will be inserted into the DR plan. In this case, we are going to insert our user-defined plan group add after the Shutdown EBS on node2 at PHX plan group.
Select the user-defined Shutdown EBS on node2 at PHX plan group.
We will be adding two steps. First step is to stop Oracle E-Business Suite on node 1 and second script is to disable rsync jobs on node 1.
Click Add Step to open the dialog where we will specify the script to stop Oracle E-Business Suite on node2.

Figure 6.3.2: Plan group to shutdown Oracle E-Business Suite and stop rsync jobs

Task 6.3.3: Provide Step Names and Local Script Parameters

The Add plan group step dialog allows us to specify parameters about what this one step will perform and how it will behave during recovery. We are going to add two steps in this plan group.

Step 1 is to stop Oracle E-Business Suite app on node1.
Step 2 is to disable rysnc jobs on node1.

For this set up, node 1 is ebshaapp01phx

We will explain all fields in this dialog, but leave out this detail in all the remaining screenshots in subsequent steps since we are just performing the same process repeatedly.

A descriptive name explaining what task this step performs.
Select user-defined type as Run local script.
Select the Region 1, so in our case it is Phoenix.
Select the correct compartment that contains the Oracle E-Business Suite app node1 and select node 1, in our case it is ebshaapp01phx.
Paste in the absolute path where you installed the shutdownapps.sh script on the Oracle E-Business Suite app node2 with the right parameters. You can refer the readme details from the GitHub link provided in Task 2.3 for the exact syntax.
Specify oracle as the user to execute the script.
The DR plan should stop if the script fails to stop Oracle E-Business Suite app on node1. This will allow anyone to see there is a problem and fix it. OCI Full Stack DR provides the opportunity to continue running the switchover plan after fixing the problem.
The default value before OCI Full Stack DR declares a failure is 60 minutes (3600 seconds). This value can be changed to 1800 seconds or whatever is felt to be a more realistic timeout value.
Click Add Step to add this step to the plan group.

Figure 6.3.3.1: Parameters to create the plan step for stop Oracle E-Business Suite on node 1
It will show as below once the step 1 is added.

Figure 6.3.3.2: Stop Oracle E-Business Suite on node 1 step is added
Click Add Step to add second step to this plan group.
A descriptive name explaining what task this step performs.
Select the Region 1, so in our case it is Phoenix.
Select Run local script.
Select the correct compartment that contains the Oracle E-Business Suite app node1 and select node 1, in our case it is ebshaapp02phx.
Paste in the absolute path where you installed the fsdr-rsync-ebs.sh script on the Oracle E-Business Suite app node2 with the right parameters. You can refer the readme details from the GitHub link provided in Task 2.3 for the exact syntax.
Specify oracle as the user to execute the script.
The DR plan should stop if the script fails to disable rsync jobs on node1. This will allow anyone to see there is a problem and fix it. OCI Full Stack DR provides the opportunity to continue running the switchover plan after fixing the problem.
The default value before OCI Full Stack DR declares a failure is 60 minutes (3600 seconds). This value can be changed to 300 seconds or whatever is felt to be a more realistic timeout value.
Click Add Step to add this step to the plan group.

Figure 6.3.3.1: Parameters to create the plan step to disable rysnc jobs on node 1
It will show as below once the step 2 is added.

Figure 6.3.3.2: Disable rysnc jobs on node 1 is added

Task 6.3.4: Complete Adding Plan Group and Steps

Both stop Oracle E-Business Suite app and disable rsync jobs on node2 are added to the DR plan group.

Click Add to add the DR plan group and steps to the DR plan.

Figure 6.3.4.1: Finalize adding plan group and steps to stop Oracle E-Business Suite and disable rsync job on node1
Verify the added plan group is available in the switchover plan.

Figure 6.3.4.1: Stop Oracle E-Business Suite and disable rsync job on node1 plan group added

Task 6.4: Create Plan Group to Clear Node Names in Database `fnd` Tables at Region 2

Now let us add the next user-defined plan group.

This user-defined plan group will clear node names in database fnd tables at region 2. This plan group will contain one step that calls the fndnodeclean.sh bash script that was downloaded to the Oracle E-Business Suite app server nodes in Task 2.3.

Task 6.4.1: Select add plan group

Begin the process to add a plan group.

Click Add group to begin.

Figure 6.4.1: Begin adding plan group to shutdown Oracle E-Business Suite and disable rsync jobs on node1

Task 6.4.2: Provide Plan Group Name, Order and Add Step

We are going to add one to execute bash script to clear node names in database fnd tables on node1 at region 1. In our example name of node1 is ebshaapp01iad.

Give the plan group a simple but descriptive name.
Select a position where the plan group will be inserted into the DR plan. In this case, we are going to insert our user-defined plan group add after the Databases - Switchover plan group.
Select the built-in Databases - Switchover plan group.
We will be adding one step to clear node names in database fnd tables.
Click Add Step to open the dialog where we will specify the script to clear node names in database fnd tables.

Figure 6.4.2: Plan group to clear node names in database fnd tables.

Task 6.4.3: Provide Step Names and Local Script Parameters

The Add plan group step dialog allows us to specify parameters about what this one step will perform and how it will behave during recovery. We are going to add a single in this plan group.

Step is to run fndnodeclean on node1.

For this set up, node 1 is ebshaapp01iad

We will explain all fields in this dialog, but leave out this detail in all the remaining screenshots in subsequent steps since we are just performing the same process repeatedly.

A descriptive name explaining what task this step performs.
Select user-defined type as Run local script.
Select the Region 2, so in our case it is Ashburn.
Select the correct compartment that contains the Oracle E-Business Suite app node1 and select node 1, in our case it is ebshaapp01iad.
Paste in the absolute path where you installed the fndnodeclean.sh script on the Oracle E-Business Suite app node1 with the right parameters. You can refer the readme details from the GitHub link provided in Task 2.3 for the exact syntax.
Specify oracle as the user to execute the script.
The DR plan should stop if the script fails to stop Oracle E-Business Suite app on node1. This will allow anyone to see there is a problem and fix it. OCI Full Stack DR provides the opportunity to continue running the switchover plan after fixing the problem.
The default value before OCI Full Stack DR declares a failure is 60 minutes (3600 seconds). This value can be changed to 900 seconds or whatever is felt to be a more realistic timeout value.
Click Add Step to add this step to the plan group.

Figure 6.4.3.1: Parameters to create the plan step to clear node names in database fnd tables
It will show as below once the step 1 is added.

Figure 6.4.3.2: Clear node names in database fnd tables

Task 6.4.4: Complete Adding Plan Group and Steps

Step to clear node names in database fnd tables is added to the DR plan group.

Click Add to add the DR plan group and steps to the DR plan.

Figure 6.4.4.1: Finalize adding plan group and step to clear node names in database fnd tables
Verify the added plan group is available in the switchover plan.

Figure 6.4.4.2: To clear node names in database fnd tables on node1 plan group added

Task 6.5: Create Plan Group to Setup App Context in Database on node1 at Region 2

Now let us add the next user-defined plan group.

This user-defined plan group will setup app context in database on node1 at region 2. This plan group will contain one step that calls the dbtxkconfig.sh bash script that was downloaded to the Oracle E-Business Suite app server nodes in Task 2.3.

Task 6.5.1: Select Add Plan Group

Begin the process to add a plan group.

Click Add group to begin.

Figure 6.5.1: Begin adding plan group to setup app context in database on node1

Task 6.5.2: Provide Plan Group Name, Order and Add Step

We are going to add one step to execute bash script to setup app context in database on node1 at region 2. In our example name of node1 is ebshaapp01iad.

Give the plan group a simple but descriptive name.
Select a position where the plan group will be inserted into the DR plan. In this case, we are going to insert our user-defined plan group add after the Clear node names in database fnd tables at IAD plan group.
Select the built-in Clear node names in database fnd tables at IAD plan group.
We will be adding one step to setup app context in database on node1.
Click Add Step to open the dialog where we will specify the script to setup app context in database on node1.

Figure 6.5.2: Plan group to setup app context in database on node1

Task 6.5.3: Provide Step Names and Local Script Parameters

The Add plan group step dialog allows us to specify parameters about what this one step will perform and how it will behave during recovery. We are going to add a single in this plan group.

Step is to run dbtxkconfig on node1 (change physical hostnames).

For this set up, node 1 is ebshaapp01iad

We will explain all fields in this dialog, but leave out this detail in all the remaining screenshots in subsequent steps since we are just performing the same process repeatedly.

A descriptive name explaining what task this step performs.
Select user-defined type as Run local script.
Select the Region 2, so in our case it is Ashburn.
Select the correct compartment that contains the Oracle E-Business Suite app node1 and select node 1, in our case it is ebshaapp01iad.
Paste in the absolute path where you installed the dbtxkconfig.sh script on the Oracle E-Business Suite app node1 with the right parameters. You can refer the readme details from the GitHub link provided in Task 2.3 for the exact syntax. Providing all the details with the right parameters is extremely important.
Specify oracle as the user to execute the script.
The DR plan should stop if the script fails to setup app context in database node1. This will allow anyone to see there is a problem and fix it. OCI Full Stack DR provides the opportunity to continue running the switchover plan after fixing the problem.
The default value before OCI Full Stack DR declares a failure is 60 minutes (3600 seconds). This value can be changed to 900 seconds or whatever is felt to be a more realistic timeout value.
Click Add Step to add this step to the plan group.

Figure 6.5.3.1: Parameters to create the plan step to setup app context in database node1
It will show as below once the step 1 is added.

Figure 6.5.3.2: To setup app context in database node1

Task 6.5.4: Complete Adding Plan Group and Steps

Step to setup app context in database node1 is added to the DR plan group.

Click Add to add the DR plan group and steps to the DR plan.

Figure 6.5.4.1: Finalize adding plan group and step to setup app context in database node1
Verify the added plan group is available in the switchover plan.

Figure 6.5.4.2: To setup app context in database node1 plan group added

Task 6.6: Create Plan Group to Setup App Context in Database on node2 at Region 2

Now let us add the next user-defined plan group.

This user-defined plan group will setup app context in database on node2 at region 2. This plan group will contain one step that calls the dbtxkconfig.sh bash script that was downloaded to the Oracle E-Business Suite app server nodes in Task 2.3.

Task 6.6.1: Select Add Plan Group

Begin the process to add a plan group.

Click Add group to begin.

Figure 6.6.1: Begin adding plan group to setup app context in database on node2

Task 6.6.2: Provide Plan Group Name, Order and Add Step

We are going to add one step to execute bash script to setup app context in database on node2 at region 2. In our example name of node2 is ebshaapp02iad.

Give the plan group a simple but descriptive name.
Select a position where the plan group will be inserted into the DR plan. In this case, we are going to insert our user-defined plan group add after the Setup app context in database on node1 at IAD plan group.
Select the built-in Setup app context in database on node1 at IAD plan group.
We will be adding one step to setup app context in database on node2.
Click Add Step to open the dialog where we will specify the script to setup app context in database on node1.

Figure 6.6.2: Plan group to setup app context in database on node2

Task 6.6.3: Provide Step Names and Local Script Parameters

The Add plan group step dialog allows us to specify parameters about what this one step will perform and how it will behave during recovery. We are going to add a single in this plan group.

Step is to run dbtxkconfig on node2 (change physical hostnames).

For this set up, node 2 is ebshaapp02iad

We will explain all fields in this dialog, but leave out this detail in all the remaining screenshots in subsequent steps since we are just performing the same process repeatedly.

A descriptive name explaining what task this step performs.
Select user-defined type as Run local script.
Select the Region 2, so in our case it is Ashburn.
Select the correct compartment that contains the Oracle E-Business Suite app node1 and select node 1, in our case it is ebshaapp02iad.
Paste in the absolute path where you installed the dbtxkconfig.sh script on the Oracle E-Business Suite app node1 with the right parameters. You can refer the readme details from the GitHub link provided in Task 2.3 for the exact syntax. Providing all the details with the right parameters is extremely important.
Specify oracle as the user to execute the script.
The DR plan should stop if the script fails to setup app context in database node2. This will allow anyone to see there is a problem and fix it. OCI Full Stack DR provides the opportunity to continue running the switchover plan after fixing the problem.
The default value before OCI Full Stack DR declares a failure is 60 minutes (3600 seconds). This value can be changed to 900 seconds or whatever is felt to be a more realistic timeout value.
Click Add Step to add this step to the plan group.

Figure 6.6.3.1: Parameters to create the plan step to setup app context in database node2
It will show as below once the step 1 is added.

Figure 6.6.3.2: To setup app context in database node2

Task 6.6.4: Complete Adding Plan Group and Steps

Step to setup app context in database node1 is added to the DR plan group.

Click Add to add the DR plan group and steps to the DR plan.

Figure 6.6.4.1: Finalize adding plan group and step to setup app context in database node2
Verify the added plan group is available in the switchover plan.

Figure 6.6.4.2: To setup app context in database node2 plan group added

Task 6.7: Create Plan Group to Execute `autoconfig` on App node1 at Region 2

Now let us add the next user-defined plan group.

This user-defined plan group will execute autoconfig on app node1 at region 2. This plan group will contain one step that calls the autoconfigapps.sh bash script that was downloaded to the Oracle E-Business Suite app server nodes in Task 2.3.

Task 6.7.1: Select Add Plan Group

Begin the process to add a plan group.

Click Add group to begin.

Figure 6.7.1: Begin adding plan group to execute autoconfig on app node1

Task 6.7.2: Provide Plan Group Name, Order and Add Step

We are going to add one step to execute bash script to execute autoconfig on app node1 at region 2. In our example name of node1 is ebshaapp01iad.

Give the plan group a simple but descriptive name.
Select a position where the plan group will be inserted into the DR plan. In this case, we are going to insert our user-defined plan group add after the Setup app context in database on node2 at IAD plan group.
Select the built-in Setup app context in database on node2 at IAD plan group.
We will be adding one step to setup app context in database on node2.
Click Add Step to open the dialog where we will specify the script to execute autoconfig on app node1.

Figure 6.7.2: Plan group to execute autoconfig on app node1

Task 6.7.3: Provide step names and local script parameters

The Add plan group step dialog allows us to specify parameters about what this one step will perform and how it will behave during recovery. We are going to add a single in this plan group.

Step is to run autoconfig on node1.

For this set up, node 1 is ebshaapp01iad

We will explain all fields in this dialog, but leave out this detail in all the remaining screenshots in subsequent steps since we are just performing the same process repeatedly.

A descriptive name explaining what task this step performs.
Select user-defined type as Run local script.
Select the Region 2, so in our case it is Ashburn.
Select the correct compartment that contains the Oracle E-Business Suite app node1 and select node 1, in our case it is ebshaapp01iad.
Paste in the absolute path where you installed the autoconfigapps.sh script on the Oracle E-Business Suite app node1 with the right parameters. You can refer the readme details from the GitHub link provided in Task 2.3 for the exact syntax. Providing all the details with the right parameters is extremely important.
Specify oracle as the user to execute the script.
The DR plan should stop if the script fails to to execute autoconfig on app node1. This will allow anyone to see there is a problem and fix it. OCI Full Stack DR provides the opportunity to continue running the switchover plan after fixing the problem.
The default value before OCI Full Stack DR declares a failure is 60 minutes (3600 seconds). This value can be changed to 900 seconds or whatever is felt to be a more realistic timeout value.
Click Add Step to add this step to the plan group.

Figure 6.7.3.1: Parameters to create the plan step to execute autoconfig on app node1
It will show as below once the step 1 is added.

Figure 6.7.3.2: To execute autoconfig on app node1

Task 6.7.4: Complete Adding Plan Group and Steps

Step to execute autoconfig on app node1 is added to the DR plan group.

Click Add to add the DR plan group and steps to the DR plan.

Figure 6.7.4.1: Finalize adding plan group and step to execute autoconfig on app node1
Verify the added plan group is available in the switchover plan.

Figure 6.7.4.2: To execute autoconfig on app node1 plan group added

Task 6.8: Create Plan Group to Execute `autoconfig` on App node2 at Region 2

Now let us add the next user-defined plan group.

This user-defined plan group will execute autoconfig on app node2 at region 2. This plan group will contain one step that calls the autoconfigapps.sh bash script that was downloaded to the Oracle E-Business Suite app server nodes in Task 2.3.

Task 6.8.1: Select Add Plan Group

Begin the process to add a plan group.

Click Add group to begin.

Figure 6.8.1: Begin adding plan group to execute autoconfig on app node2

Task 6.8.2: Provide Plan Group Name, Order and Add Step

We are going to add one step to execute bash script to execute autoconfig on app node2 at region 2. In our example name of node2 is ebshaapp02iad.

Give the plan group a simple but descriptive name.
Select a position where the plan group will be inserted into the DR plan. In this case, we are going to insert our user-defined plan group add after the Execute autoconfig on app node1 at IAD plan group.
Select the built-in Execute autoconfig on app node1 at IAD plan group.
We will be adding one step to setup app context in database on node2.
Click Add Step to open the dialog where we will specify the script to execute autoconfig on app node2.

Figure 6.8.2: Plan group to execute autoconfig on app node2

Task 6.8.3: Provide Step Names and Local Script Parameters

The Add plan group step dialog allows us to specify parameters about what this one step will perform and how it will behave during recovery. We are going to add a single in this plan group.

Step is to run autoconfig on node2.

For this set up, node 1 is ebshaapp02iad

We will explain all fields in this dialog, but leave out this detail in all the remaining screenshots in subsequent steps since we are just performing the same process repeatedly.

A descriptive name explaining what task this step performs.
Select user-defined type as Run local script.
Select the Region 2, so in our case it is Ashburn.
Select the correct compartment that contains the Oracle E-Business Suite app node2 and select node 2, in our case it is ebshaapp02iad.
Paste in the absolute path where you installed the autoconfigapps.sh script on the Oracle E-Business Suite app node1 with the right parameters. You can refer the readme details from the GitHub link provided in Task 2.3 for the exact syntax. Providing all the details with the right parameters is extremely important.
Specify oracle as the user to execute the script.
The DR plan should stop if the script fails to to execute autoconfig on app node2. This will allow anyone to see there is a problem and fix it. OCI Full Stack DR provides the opportunity to continue running the switchover plan after fixing the problem.
The default value before OCI Full Stack DR declares a failure is 60 minutes (3600 seconds). This value can be changed to 900 seconds or whatever is felt to be a more realistic timeout value.
Click Add Step to add this step to the plan group.

Figure 6.8.3.1: Parameters to create the plan step to execute autoconfig on app node2
It will show as below once the step 1 is added.

Figure 6.8.3.2: To execute autoconfig on app node2

Task 6.8.4: Complete Adding Plan Group and Steps

Step to execute autoconfig on app node2 is added to the DR plan group.

Click Add to add the DR plan group and steps to the DR plan.

Figure 6.8.4.1: Finalize adding plan group and step to execute autoconfig on app node2
Verify the added plan group is available in the switchover plan.

Figure 6.8.4.2: To execute autoconfig on app node2 plan group added

Task 6.9: Create Plan Group to Startup Oracle E-Business Suite and Enable Rsync Jobs on node 1 at Region 2

Now let us add the next user-defined plan group.

This user-defined plan group will startup Oracle E-Business Suite and enable rsync jobs on node1 at region 2. This plan group will contain two steps that calls the startapps.sh and fsdr-rsync-ebs.sh bash scripts that was downloaded to the Oracle E-Business Suite app server nodes in Task 2.3.

Task 6.9.1: Select Add Plan Group

Begin the process to add a plan group.

Click Add group to begin.

Figure 6-9.1: Begin adding plan group to startup Oracle E-Business Suite and enable rsync jobs on node1

Task 6.9.2: Provide Plan Group Name, Order and Add Step

A DR Plan Group can contain many steps that are all executed in parallel. We are going to add two steps to execute bash scripts to start Oracle E-Business Suite on node1 and enable rsync cron jobs on node1 at region 2. In our example name of node1 is ebshaapp01iad.

Give the plan group a simple but descriptive name.
Select a position where the plan group will be inserted into the DR plan. In this case, we are going to insert our user-defined plan group add after the Execute autoconfig on app node2 at IAD plan group.
Select the user-defined Execute autoconfig on app node2 at IAD plan group.
We will be adding two steps. First step is to start Oracle E-Business Suite on node 1 and second script is to enable rsync jobs on node 1.
Click Add Step to open the dialog where we will specify the script to start Oracle E-Business Suite on node2.

Figure 6.9.2: Plan group to startup ebs and enable rsync jobs on node1

Task 6.9.3: Provide Step Names and Local Script Parameters

The Add plan group step dialog allows us to specify parameters about what this one step will perform and how it will behave during recovery. We are going to add two steps in this plan group.

Step 1 is to start Oracle E-Business Suite app on node1.
Step 2 is to enable rysnc jobs on node1.

For this set up, node 1 is ebshaapp01iad.

We will explain all fields in this dialog, but leave out this detail in all the remaining screenshots in subsequent steps since we are just performing the same process repeatedly.

A descriptive name explaining what task this step performs.
Select user-defined type as Run local script.
Select the Region 2, so in our case it is Ashburn.
Select the correct compartment that contains the Oracle E-Business Suite app node1 and select node 1, in our case it is ebshaapp01iad.
Paste in the absolute path where you installed the startapps.sh script on the Oracle E-Business Suite app node1 with the right parameters. You can refer the readme details from the GitHub link provided in Task 2.3 for the exact syntax.
Specify oracle as the user to execute the script.
The DR plan should stop if the script fails to start Oracle E-Business Suite app on node1. This will allow anyone to see there is a problem and fix it. OCI Full Stack DR provides the opportunity to continue running the switchover plan after fixing the problem.
The default value before OCI Full Stack DR declares a failure is 60 minutes (3600 seconds). This value can be changed to 1800 seconds or whatever is felt to be a more realistic timeout value.
Click Add Step to add this step to the plan group.

Figure 6.9.3.1: Parameters to create the plan step for start Oracle E-Business Suite on node 1
It will show as below once the step 1 is added.

Figure 6.9.3.2: Start Oracle E-Business Suite on node 1 step is added
Click Add Step to add second step to this plan group.
A descriptive name explaining what task this step performs.
Select the Region 2, so in our case it is Ashburn.
Select Run local script.
Select the correct compartment that contains the Oracle E-Business Suite app node1 and select node 1, in our case it is ebshaapp0iad.
Paste in the absolute path where you installed the fsdr-rsync-ebs.sh script on the Oracle E-Business Suite app node1 with the right parameters. You can refer the readme details from the GitHub link provided in Task 2.3 for the exact syntax.
Specify oracle as the user to execute the script.
The DR plan should stop if the script fails to enable rsync jobs on node1. This will allow anyone to see there is a problem and fix it. OCI Full Stack DR provides the opportunity to continue running the switchover plan after fixing the problem.
The default value before OCI Full Stack DR declares a failure is 60 minutes (3600 seconds). This value can be changed to 300 seconds or whatever is felt to be a more realistic timeout value.
Click Add Step to add this step to the plan group.

Figure 6.9.3.1: Parameters to create the plan step to enable rysnc jobs on node 1
It will show as below once the step 2 is added.

Figure 6.9.3.2: Enable rysnc jobs on node 1 is added

Task 6.9.4: Complete Adding Plan Group and Steps

Both start Oracle E-Business Suite app and enable rsync jobs on node1 are added to the DR plan group.

Click Add to add the DR plan group and steps to the DR plan.

Figure 6.9.4.1: Finalize adding plan group and steps to start Oracle E-Business Suite and enable rsync job on node1
Verify the added plan group is available in the switchover plan.

Figure 6.9.4.1: Start Oracle E-Business Suite and enable rsync job on node1 plan group added

Task 6.10: Create Plan Group to Startup Oracle E-Business Suite and Enable Rsync Jobs on node 2 at Region 2

Now let us add the next user-defined plan group.

This user-defined plan group will startup Oracle E-Business Suite and enable rsync jobs on node2 at region 2. This plan group will contain two steps that calls the startapps.sh and fsdr-rsync-ebs.sh bash scripts that was downloaded to the Oracle E-Business Suite app server nodes in Task 2.3.

Task 6.10.1: Select Add Plan Group

Begin the process to add a plan group.

Click Add group to begin.

Figure 6-10.1: Begin adding plan group to startup Oracle E-Business Suite and enable rsync jobs on node2

Task 6.10.2: Provide Plan Group Name, Order and Add Step

Give the plan group a simple but descriptive name.
Select a position where the plan group will be inserted into the DR plan. In this case, we are going to insert our user-defined plan group add after the Startup EBS on node1 at IAD plan group.
Select the user-defined Startup EBS on node1 at IAD plan group.
We will be adding two steps. First step is to start Oracle E-Business Suite on node 2 and second script is to enable rsync jobs on node 2.
Click Add Step to open the dialog where we will specify the script to start Oracle E-Business Suite on node2.

Figure 6.10.2: Plan group to startup Oracle E-Business Suite and enable rsync jobs on node2

Task 6.10.3: Provide Step Names and Local Script Parameters

The Add plan group step dialog allows us to specify parameters about what this one step will perform and how it will behave during recovery. We are going to add two steps in this plan group.

Step 1 is to start Oracle E-Business Suite app on node2.
Step 2 is to enable rysnc jobs on node2.

For this set up, node 2 is ebshaapp02iad

We will explain all fields in this dialog, but leave out this detail in all the remaining screenshots in subsequent steps since we are just performing the same process repeatedly.

A descriptive name explaining what task this step performs.
Select user-defined type as Run local script.
Select the Region 2, so in our case it is Ashburn.
Select the correct compartment that contains the Oracle E-Business Suite app node1 and select node 2, in our case it is ebshaapp02iad.
Paste in the absolute path where you installed the startapps.sh script on the Oracle E-Business Suite app node1 with the right parameters. You can refer the readme details from the GitHub link provided in Task 2.3 for the exact syntax.
Specify oracle as the user to execute the script.
The DR plan should stop if the script fails to start Oracle E-Business Suite app on node2. This will allow anyone to see there is a problem and fix it. OCI Full Stack DR provides the opportunity to continue running the switchover plan after fixing the problem.
The default value before OCI Full Stack DR declares a failure is 60 minutes (3600 seconds). This value can be changed to 1800 seconds or whatever is felt to be a more realistic timeout value.
Click Add Step to add this step to the plan group.

Figure 6.10.3.1: Parameters to create the plan step for start Oracle E-Business Suite on node 2
It will show as below once the step 1 is added.

Figure 6.10.3.2: Start Oracle E-Business Suite on node 2 step is added
Click Add Step to add second step to this plan group.
A descriptive name explaining what task this step performs.
Select the Region 2, so in our case it is Ashburn.
Select Run local script.
Select the correct compartment that contains the Oracle E-Business Suite app node1 and select node 2, in our case it is ebshaapp0iad.
Paste in the absolute path where you installed the fsdr-rsync-ebs.sh script on the Oracle E-Business Suite app node1 with the right parameters. You can refer the readme details from the GitHub link provided in Task 2.3 for the exact syntax.
Specify oracle as the user to execute the script.
The DR plan should stop if the script fails to enable rsync jobs on node2. This will allow anyone to see there is a problem and fix it. OCI Full Stack DR provides the opportunity to continue running the switchover plan after fixing the problem.
The default value before OCI Full Stack DR declares a failure is 60 minutes (3600 seconds). This value can be changed to 300 seconds or whatever is felt to be a more realistic timeout value.
Click Add Step to add this step to the plan group.

Figure 6.10.3.1: Parameters to create the plan step to enable rysnc jobs on node 2
It will show as below once the step 2 is added.

Figure 6.10.3.2: Enable rysnc jobs on node 2 is added

Task 6.10.4: Complete Adding Plan Group and Steps

Both start Oracle E-Business Suite app and enable rsync jobs on node2 are added to the DR plan group.

Click Add to add the DR plan group and steps to the DR plan.

Figure 6.10.4.1: Finalize adding plan group and steps to start Oracle E-Business Suite and enable rsync job on node2
Verify the added plan group is available in the switchover plan.

Figure 6.10.4.1: Start Oracle E-Business Suite and enable rsync job on node2 plan group added

Note: With this, we have completed all the required user-defined plan groups for the Switchover Plan. Make sure to verify you got the plan groups in the right order, this is really important.

Once the switchover plan is customized, it should look like this:

Figure 6.10.5.1: All plan groups for switchover plan in region 2

Note: You can reorder the plan group in case if you have not ordered in the right order.

Task 6.11: Select the Failover plan

Begin by navigating to the failover plan created in the previous task.

Figure 6.11: Select failover plan in region 2

OCI Full Stack DR will generate the Prechecks-Built in and Databases-Failover plan groups, this was generated based on the members we added to the DR protection groups in region 1 and region 2.

Figure 6.11.1: Default plan groups for Failover plan in region 2

Similar to how we have customized the switchover plan, we will be adding the required user-defined plan groups for the failover plan.

You must add the following mentioned user-defined plan groups after the built-in plan group Databases-Failover.

Clear node names in database fnd tables at region 2.
Setup app context in database on node1 at region 2.
Setup app context in database on node2 at region 2.
Execute autoconfig on node1 at region 2.
Execute autoconfig on node2 at region 2.
Startup Oracle E-Business Suite and enable rsync jobs on node1 at region 2.
Startup Oracle E-Business Suite and enable rsync jobs on node2 at region 2.

Note: We are not going through the steps to add the user-defined plan groups again. Refer to Task 6.4 to 6.9 for the detailed instructions.

Once the failover plan is customized, it should look like this:

Figure 6.11.1: All plan groups for failover plan in region 2

Note:

You can reorder the plan groups in case if you have not ordered in the right order.

You can also customize the Start Drill plan.

Note that for the start drill plan, there will not be any built-in plan groups created for the Database.

You must create a user-defined plan group to convert the physical standby to a snapshot standby database. You can use standard Oracle Data Guard broker commands to perform those operation and put in a wrapper script.

In addition, you can add application-related user-defined plan groups, similar to what you do for a Failover plan.

OCI Full Stack DR will allow to create stop drill plan once the start drill plan in executed. For more information, see DR Drill plans. Stop Drill plan is reversing the operations which was done in start drill plan.

You can also customize the Stop Drill plan.

Note that for the stop drill plan, there will not be any built-in plan groups created for the Database.

You must create a user-defined plan group to convert the snapshot standby to a physical database. You can use standard Oracle Data Guard broker commands to perform those operation and put in a wrapper script.

In addition, you can add application-related user-defined plan groups, to stop all the Oracle E-Business Suite related applications.

As part of this tutorial, we will run a **Switchover plan in the subsequent tasks.**

Task 7: Run the Switchover Plan in Region 2 (Ashburn)

Both switchover and failover DR plans have been created in the standby region 2 (Ashburn). The DR plans in region 2 allow OCI Full Stack DR to transition workloads from region 1 to region 2. The next task is to create switchover, failover and start drill plans in the protection group for region 1 (Phoenix) so OCI Full Stack DR can transition workloads from region 2 back to region 1.

However, DR plans can only be created and modified in the protection group with the standby role. The DR protection group in region 1 is currently the primary, which means DR plans cannot be created in region 1.

Therefore, we need to reverse the roles of the protection groups so region 1 is the standby and region 2 is the primary. Execute the switchover plan that was just created to transition the workload from region 1 (Phoenix) to region 2 (Ashburn).

Task 7.1: Begin Plan Execution

Execute DR plan to begin the process of transitioning the Oracle Integration Cloud workload from region 1 to region 2.

Ensure the region context is still set to standby region 2 (Ashburn).
Use the breadcrumbs at the top of the console to help ensure DR protection group details is the current plan context.
Ensure the correct DR protection group in region 2 is selected; it should be the Standby role.
Ensure both the failover and switchover plans exist before proceeding; if not, go back to the previous steps to create and customise both DR plans.
Click Execute DR plan.

Figure 7-1: Showing how to execute a switchover to standby region

Task 7.2: Select Switchover Plan and Execute

This task executes the switchover plan in region 2.

Select the switchover plan.
Select Enable prechecks.
Click Execute DR plan to begin.

Figure 7-2: Choose and execute the switchover plan

Task 7.3: Monitor the DR Plan Execution

Monitor the switchover plan until the Oracle E-Business Suite workload has been fully transitioned from region 1 to region 2. Once the switchover plan is completed successfully, OCI Full Stack DR will take care of cleaning up changing the roles of primary and standby DR protection groups between the regions.

Region 2 (Phoenix) will be the primary region and region 1 (Ashburn) will be the standby region once OCI Full Stack DR has completed the switchover.

Verify the successful switchover plan execution.

Figure 7-2: Switchover plan completed successfully in Region 2
Verify the role change in the DR protection group, Region 2 (Phoenix) will have standby role now and Region 1 (Ashburn) will have primary role now.

Figure 7-2: DR protection group role change after switchover plan

Task 8: Create DR and Customize DR Plans in Region 1 (Phoenix)

Create the DR plans in the DR Protection Group in region 1 (Phoenix) which is now the standby peer.

The purpose of each plan is to transition the workload from region 2 to region 1 whenever region 2 is the primary peer. The roles of the DR protection groups in both regions are automatically reversed as part of any DR operation, so the protection group in region 2 will become the standby and the protection group in region 1 will become primary after a failover or switchover.

OCI Full Stack DR will pre-populate both plans with built-in steps based on the member resources added in the previous step. The plans will be customized in later steps to handle all the tasks related to Oracle Integration Cloud during a recovery operation.

The DR plans are always created in the protection group with the standby role; region 1 is currently the standby protection group after executing the switchover plan in Task 7.

Follow the same process as Task 5 to create the DR plans in region 1. Once the plans are created, verify the plans.

Figure 8-1: DR plans in region 1

Follow the same process as per Task 6 to customize the DR plans in region 1. Make sure by adding the user-defined plan groups. It is super important to select the right Oracle E-Business Suite application VM’s for production and DR regions for executing the scripts.

Shutdown Oracle E-Business Suite and disable rsync jobs on node2 at region 2.
Shutdown Oracle E-Business Suite and disable rsync jobs on node1 at region 2.
Clear node names in database fnd tables at region 1.
Setup app context in database on node1 at region 1.
Setup app context in database on node2 at region 1.
Execute autoconfig on node1 at region 1.
Execute autoconfig on node2 at region 1.
Startup Oracle E-Business Suite and enable rsync jobs on node1 at region 1.
Startup Oracle E-Business Suite and enable rsync jobs on node2 at region 1.

Once the switchover plan is customized, it should look like:

Figure 8.2: All plan groups for switchover plan in region 1

Once the failover plan is customized, it should look like:

Figure 8.3: All plan groups for failover plan in region 1

Note:

You can reorder the plan groups in case if you have not ordered in the right order.

You can also customize the Start Drill plan.

Note that for the start drill plan, there will not be any built-in plan groups created for the Database.

You must create a user-defined plan group to convert the physical standby to a snapshot standby database. You can use standard Oracle Data Guard broker commands to perform those operation and put in a wrapper script.

In addition, you can add application-related user-defined plan groups, similar to what you do for a Failover plan.

OCI Full Stack DR will allow to create stop drill plan once the start drill plan in executed. For more information, see DR Drill plans. Stop Drill plan is reversing the operations which was done in start drill plan.

You can also customize the Stop Drill plan.

Note that for the stop drill plan, there will not be any built-in plan groups created for the Database.

You must create a user-defined plan group to convert the snapshot standby to a physical database. You can use standard Oracle Data Guard broker commands to perform those operation and put in a wrapper script.

In addition, you can add application-related user-defined plan groups, to stop all the Oracle E-Business Suite related applications.

Next Steps

OCI Full Stack DR for Oracle E-Business Suite should be fully implemented at this point. However, full functionality should be validated before using OCI Full Stack DR for production. All failover and switchover plans should be executed to validate that everything works as expected and the recovery team fully understands the entire process.

Test Switchover Plans

Switchover plans are designed to clean up all artifacts and ensure all roles for built-in recovery steps such as OCI Load Balancer, OCI Block Storage, OCI File Systems, Oracle Base Database Service and Oracle Autonomous Database are ready to recovered from the standby region without human intervention.

Test Failover Plans

Failovers are different. Failovers by their very nature cannot clean up artifacts or ensure services and databases at the failed region are ready to transition workloads back to region 1. The recovery team needs to understand and perform tasks to ensure Oracle Data Guard is in the correct state, artifacts for storage and compute instances have been terminated, etc. For more information, see Resetting DR Configuration After a Failover.

Validate all DR Plans for Final Acceptance

The recovery team needs to perform a final validation to demonstrate the readiness of OCI Full Stack DR protection groups and plans for production workloads. Region 2 (Ashburn) should be the primary region at this point in the process. Begin final validate of all plans by completing the following steps:

Test switchover from region 2 (primary) back to region 1 (standby).
Test failover from region 1 (primary) to region 2 (standby).
Prepare region 1 (primary) for failover from region 2.
Test failover from region 2 (primary) to region 1 (standby).
Prepare region 2 (primary) for either a failover or switchover to region 2.
The DR protection groups and application stack should be in a normal operational state and ready for a failover or switchover at this point.
You could also run DR drill plans in either of the regions depending on the requirement.

Get Support for this Solution

OCI Full Stack DR engineering provides 1st level support for this solution.

However, the solution and any custom automation presented in this tutorial was devised and implemented by the Oracle Cloud EMEA specialists team independently of the Oracle E-Business Suite organization or OCI Full Stack DR engineering team.

Relationship between Oracle E-Business Suite and OCI Full Stack DR

OCI Full Stack DR is not part of the Oracle E-Business Suite installation or deployment process and is not mentioned in any documentation written and maintained by Oracle E-Business Suite. The method for installing, configuring and deploying Oracle E-Business Suite for disaster recovery between OCI regions was devised and written byOracle E-Business Suite engineering and is completely independent of this tutorial or OCI Full Stack DR.

The Oracle E-Business Suite organization is not familiar with OCI Full Stack DR and can only help solve problems with the Oracle E-Business Suite process as described in the documentation written by Oracle E-Business Suite for manually executed disaster recovery outside of OCI Full Stack DR.

Troubleshooting problems with the manual Oracle E-Business Suite process documented by Oracle E-Business Suite is the responsibility of Oracle E-Business Suite support and done independently of OCI Full Stack DR. OCI Full Stack DR does not prevent customers or Oracle E-Business Suite support from executing the documented Oracle E-Business Suite disaster recovery process manually. Therefore, Oracle E-Business Suite support will solve problems with their manual disaster recovery process without involving OCI Full Stack DR.

However, walking through the manual recovery process documented by Oracle E-Business Suite will likely leave the OCI Full Stack DR protection groups in an unusable state which will need to be reset before operations can resume using OCI Full Stack DR with Oracle E-Business Suite. OCI Full Stack DR support will help customers reset the DR protection groups after Oracle E-Business Suite support has completed any manual troubleshooting and resolution.

Support Begins with OCI Full Stack DR

OCI Full Stack DR support is the first point of contact for help with any problems encountered with the steps and tasks explained in this tutorial. OCI Full Stack DR support will isolate the problem and determine which support organization is best qualified to solve the problem.

OCI Full Stack DR support is responsible for:
1. Helping customers understand the meaning and cause of error messages generated during a DR operation.
2. Helping customers resolve issues with IAM policies preventing Full Stack DR from managing resources during a DR operation.
3. Problems creating or managing DR protection groups.
4. Problems with DR Plan creation, execution or failures to properly call the custom automation provided by Oracle E-Business Suite or the EMEA Cloud Solutions team.
5. Customers are responsible for troubleshooting and resolving any issues with custom scripts written by the customer that are not part of this tutorial.
OCI Full Stack DR support will open an SR with Oracle E-Business Suite support for any issues that are directly related to Oracle E-Business Suite itself. Customers will work directly with Oracle E-Business Suite support to resolve issues with Oracle E-Business Suite. Oracle E-Business Suite is responsible for:
1. Any problems with the My Oracle Support (MOS) documentation written and maintained by Oracle E-Business Suite.
2. Any problems with scripts provided to customers by Oracle E-Business Suite.
3. Any database issues related to Oracle E-Business Suite.
4. Any application issues related to Oracle E-Business Suite.
5. Any problems with the documented Oracle E-Business Suite process for recovering Oracle E-Business Suite manually outside of OCI Full Stack DR.
OCI Full Stack DR support will open an SR with the appropriate internal Oracle Cloud Solutions team for any issues that are directly related to the scripts or process described in this tutorial. Customers will work directly with the Oracle Cloud Solutions team to resolve issues with this tutorial. The Oracle Cloud Solutions team is responsible for:
1. Any problems with the custom scripts provided by the cloud solutions team.
2. Any problems with the overall process as described in this tutorial.

How to Open a Support Request

Use the OCI console to open a support request with OCI Full Stack DR support. Ensure the browser context is showing the DR protection group you created as part of this tutorial before starting this process.

Ensure the browser context is set to the appropriate DR Protection Group.
Select the life preserver in the OCI Console.
Select Create a Support Request.

Fig 5: Use the OCI console help tool to open a service request

Fig 6: Support requests must be opened using the request help button

Acknowledgments

Authors - Chandra Dharanikota (Oracle Apps to OCI Specialist, Oracle Technology Cloud Engineering, EMEA), Suraj Ramesh - (Full Stack DR, Technical reviewer)

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.

Title and Copyright Information

Automate Recovery for Multi-Node Oracle E-Business Suite Using OCI Full Stack Disaster Recovery

G35190-02

Automate Recovery for Multi-Node Oracle E-Business Suite Using OCI Full Stack Disaster Recovery

Introduction

Oracle E-Business Suite is Normally Part of a Larger System

Caution about Implementing Incrementally

How the Recovery Works

Option 1 for Hosting Scripts

Option 2 for Hosting Scripts

Oracle E-Business Suite Deployment Architecture

Reference Architecture for Oracle E-Business Suite using Oracle Base Database Service

Reference architecture for Oracle E-Business Suite using Oracle Exadata Database Service on Dedicated Infrastructure

Difference Between Logical Hostnames and Physical Hostnames

OCI Full Stack DR Deployment Architecture

Option 1: Automate Recovery for Oracle E-Business Suite using Oracle Base Database Service

Option 2: Automate Recovery for Oracle E-Business Suite using Oracle Exadata Database Service on Dedicated Infrastructure

Definitions and Assumptions throughout the Tutorial

Regions

Compartments

Objectives

Prerequisites

Task 1: Deploy Oracle E-Business Suite for Disaster Recovery

Task 2: Prepare Tenancy for OCI Full Stack DR

Task 2.1: Set up OCI IAM Policies for OCI Full Stack DR

Task 2.2: Set up OCI IAM Policies for Other Services Managed by OCI Full Stack DR

Task 2.3: Download and Install Custom Oracle E-Business SuiteScripts on Oracle E-Business Suite Application servers

Task 2.4: Ensure Oracle E-Business Suite Application Servers can Execute Scripts and Commands

Task 2.5: Create OCI Vault Secrets for the Oracle E-Business Suite Database

Task 2.6: Create OCI Object Storage Buckets for Logs

Task 2.6.1: Navigate to OCI Object Storage

Task 2.6.2: Create an OCI Object Storage Bucket in Region 1 and 2

Task 3: Create and Associate DR Protection Groups

Task 3.1: Create a Protection Group in Standby Region 2 (Ashburn)

Task 3.1.1: Navigate to DR Protection Groups

Task 3.1.2: Create the Protection Group

Task 3.1.3: Add Parameters Needed to Create the Protection Group

Task 3.2: Create a Protection group in Primary Region 1 (Phoenix)

Task 3.2.1: Create the Protection Group

Task 3.2.2: Add Parameters Needed to Create the Protection Group

Task 3.3: Associate Protection Groups in Region 1 and Region 2

Task 3.3.1: Begin the Association

Task 3.3.2: Associate Protection Groups in Region 1 and Region 2

Task 4: Add Oracle E-Business Suite Members to Region 1 and Region 2 DR Protection Groups

Task 4.1: Add Member Resources to DR Protection Group to Region 1 (Phoenix)

Task 4.1.1: Navigate to Primary DR Protection Group

Task 4.1.2: Add Oracle E-Business Suite Application Server 1

Task 4.1.3: Add Oracle E-Business Suite Application Server 2

Task 4.1.4: Add Primary Clustered Oracle Database

Task 4.2: Add Member Resources to DRPG in Region 2 (Ashburn)

Task 4.2.1: Navigate to Standby DR Protection Group

Task 4.2.2: Add Oracle E-Business Suite Application Server 1

Task 4.2.3: Add Oracle E-Business Suite Application Server 2

Task 4.2.4: Add Oracle Standby Clustered Database

Task 5: Create the DR Plans in Region 2

Task 5.1: Create DR Plan for Switchover to Region 2

Task 5.2: Create DR Plan for Failover to Region 2

Task 5.3: Create DR Plan for Start Drill in Region 2

Task 6: Customize the DR Plans in Region 2 (Ashburn)

Task 6.1: Select the Switchover Plan

Task 6.2: Create Plan Group to Shutdown Oracle E-Business Suite and Disable Rsync Jobs on Node 2 at Region 1

Task 6.2.1: Select Add Plan Group

Task 6.2.2: Provide Plan Group Name, Order and Add Step

Task 6.2.3: Provide Step Names and Local Script Parameters

Task 6.2.4: Complete Adding Plan Group and Steps

Task 6.3: Create Plan Group to Shutdown Oracle E-Business Suite and Disable Rsync Jobs on Node 1 at Region 1

Task 6.3.1: Select Add Plan Group

Task 6.3.2: Provide Plan Group Name, Order and Add Step

Task 6.3.3: Provide Step Names and Local Script Parameters

Task 6.3.4: Complete Adding Plan Group and Steps

Task 6.4: Create Plan Group to Clear Node Names in Database fnd Tables at Region 2

Task 6.4.1: Select add plan group

Task 6.4.2: Provide Plan Group Name, Order and Add Step

Task 6.4.3: Provide Step Names and Local Script Parameters

Task 6.4.4: Complete Adding Plan Group and Steps

Task 6.5: Create Plan Group to Setup App Context in Database on node1 at Region 2

Task 6.5.1: Select Add Plan Group

Task 6.5.2: Provide Plan Group Name, Order and Add Step

Task 6.5.3: Provide Step Names and Local Script Parameters

Task 6.5.4: Complete Adding Plan Group and Steps

Task 6.6: Create Plan Group to Setup App Context in Database on node2 at Region 2

Task 6.6.1: Select Add Plan Group

Task 6.6.2: Provide Plan Group Name, Order and Add Step

Task 6.4: Create Plan Group to Clear Node Names in Database `fnd` Tables at Region 2

Task 6.7: Create Plan Group to Execute `autoconfig` on App node1 at Region 2

Task 6.8: Create Plan Group to Execute `autoconfig` on App node2 at Region 2