Configure a DR Solution with an Oracle Autonomous Database

To ensure business continuity in the event of disasters, you'll want to implement a disaster recovery (DR) strategy for your Oracle WebLogic Suite applications. This solution provides data protection and enables you to use Oracle Autonomous Database to quickly switch to the standby system with minimal loss of data and productivity.

You can setup and manage a disaster recovery system that uses Oracle Autonomous Database as the database for Oracle WebLogic Server for Oracle Cloud Infrastructure, Oracle SOA Suite on Marketplace, or any other middle-tier Oracle Cloud Infrastructure (OCI) service that uses Oracle Fusion Middleware.

Oracle Autonomous Database Serverless and Oracle Exadata Database Service on Dedicated Infrastructure provide a Snapshot Standby feature. This allows you to open the standby database temporarily for read-write. When you convert your standby database to a snapshot standby, it is fully updateable. It continues receiving the redo data from its remote source database (so you are still DR-protected), but it does not apply the redo until it is converted back to a physical standby. All of the changes performed in a snapshot standby database are reverted when it is converted to physical standby again. Use this feature to provision the mid-tier system in the standby region. You can also use this feature during the lifecycle of your DR system to validate the standby system without performing a complete switchover.

In addition to the Snapshot Standby feature, Oracle Autonomous Database Serverless provides remote refreshable clones. This provides equivalent functionality to the traditional Oracle Data Guard snapshot standby database functionality, but using an additional database. The Remote Refreshable Clone is managed individually and separately from the Oracle Autonomous Data Guard Standby. When you use Oracle Autonomous Database Serverless, you can use a Remote Refreshable Clone instead of a Snapshot Standby database to provision the mid-tier system in the secondary region, or to perform tasks in standby like testing, opening for validations, patching, and so on. However, the standby and refreshable clone databases use different connection wallets and their connect strings must be managed properly to perform these tasks.

Before You Begin

There are multiple Oracle Maximum Availability Architecture (MAA) technical briefs that describe how to set up a disaster recovery (DR) system for Oracle WebLogic Server for Oracle Cloud Infrastructure and Oracle SOA Suite on Marketplace when these middleware systems use Oracle Base Database Service.

Oracle WebLogic Server for Oracle Cloud Infrastructure, Oracle SOA Suite on Marketplace, and Oracle Fusion Middleware DR topologies use an active-passive model. The primary system is an Oracle Cloud Infrastructure (OCI) data center, and a secondary system in a different and remote OCI data center.

Review the following for more details and topologies for each case:

The above papers provide in-depth details, setup and management steps, as well as many other considerations, for Oracle WebLogic Server for Oracle Cloud Infrastructure and Oracle SOA Suite on Marketplace.

In addition to the technical briefs, ensure that you're familiar with Oracle Cloud Infrastructure (OCI) concepts and administration, including networking, compute instances, load balancing, and Oracle Autonomous Database before proceeding with the analysis and steps described in this playbook.

The steps and examples in this playbook are verified with Oracle WebLogic Server for OCI and Oracle SOA Suite on Marketplace for the following scenarios:

Oracle Autonomous Data Guard on Oracle Exadata Database Service on Dedicated Infrastructure, using the Snapshot Standby feature for the disaster recovery setup.
Oracle Autonomous Data Guard on Oracle Autonomous Database Serverless, using the Snapshot Standby feature for the disaster recovery setup.
Oracle Autonomous Data Guard on Oracle Autonomous Database Serverless, using Remote Refreshable Clones for the disaster recovery setup.

Most of the steps are common for the three scenarios. Only some of the steps differ and are specific to each case.

It should not be difficult to adjust the steps to any Oracle WebLogic Server or Oracle Fusion Middleware system that uses Oracle Autonomous Database.

Architecture

This architecture diagram shows the disaster recovery (DR) system's topology for the approaches used in this playbook. All runtime, configuration, and metadata information residing in the primary database is replicated from Region 1 to Region 2 with Oracle Autonomous Data Guard. Oracle WebLogic Server for Oracle Cloud Infrastructure, Oracle SOA Suite on Marketplace, and Oracle Fusion Middleware DR topologies use an active-passive model. The primary system is an Oracle Cloud Infrastructure (OCI) data center, and a secondary system in a different and remote OCI data center.

The Oracle WebLogic domain configuration is replicated using an Oracle Cloud Infrastructure File Storage (OCI File Storage) stage directory in the primary region, which is replicated to an OCI File Storage stage directory in the secondary region. Then, the configuration is copied to the real domain directory in the secondary. Direct copy of the domain presents risks that are avoided using a stage directory. Since file copies are a non-transactional operation, a first copy to a stage directory is done. Files are first verified in this intermediate directory and then they are transferred to the real (final) Oracle WebLogic domain.

Description of the illustration wls-dr-adb.png

wls-dr-adb-oracle.zip

The topology is typical of what is used by Oracle WebLogic Server, Oracle SOA, or Oracle Fusion Middleware disaster recovery environments in OCI. For provisioning the standby midtier and for lifecycle operations like testing secondary, you convert the standby Oracle Autonomous Database into a Snapshot Standby Database or use a Remote Refreshable Clone.

When using a Remote Refreshable Clone, there is an "auxiliary database" (a refreshable clone in a remote region) for the initial provisioning and for the test and validation operations in the secondary. In this case, the secondary middle tier gets configured with Datasources that need to be changed back to the standby address on switchover and failover events.

This architecture supports the following components:

Region
An Oracle Cloud Infrastructure region is a localized geographic area that contains one or more data centers, called availability domains. Regions are independent of other regions, and vast distances can separate them (across countries or even continents).
Availability domain
Availability domains are standalone, independent data centers within a region. The physical resources in each availability domain are isolated from the resources in the other availability domains, which provides fault tolerance. Availability domains don’t share infrastructure such as power or cooling, or the internal availability domain network. So, a failure at one availability domain shouldn't affect the other availability domains in the region.
Fault domain
A fault domain is a grouping of hardware and infrastructure within an availability domain. Each availability domain has three fault domains with independent power and hardware. When you distribute resources across multiple fault domains, your applications can tolerate physical server failure, system maintenance, and power failures inside a fault domain.
Virtual cloud network (VCN) and subnet
A VCN is a customizable, software-defined network that you set up in an Oracle Cloud Infrastructure region. Like traditional data center networks, VCNs give you complete control over your network environment. A VCN can have multiple non-overlapping CIDR blocks that you can change after you create the VCN. You can segment a VCN into subnets, which can be scoped to a region or to an availability domain. Each subnet consists of a contiguous range of addresses that don't overlap with the other subnets in the VCN. You can change the size of a subnet after creation. A subnet can be public or private.
Load balancer
The Oracle Cloud Infrastructure Load Balancing service provides automated traffic distribution from a single entry point to multiple servers in the back end.
Oracle WebLogic domain
A domain is the basic administration unit for WebLogic Server instances. A domain consists of one or more WebLogic Server instances (and their associated resources) that you manage with a single Administration Server. The Oracle WebLogic domain is configured following Oracle Maximum Availability Architecture (MAA) best practices for High Availability.
Dynamic routing gateway (DRG)
The DRG is a virtual router that provides a path for private network traffic between VCNs in the same region, between a VCN and a network outside the region, such as a VCN in another Oracle Cloud Infrastructure region, an on-premises network, or a network in another cloud provider.
Security list
For each subnet, you can create security rules that specify the source, destination, and type of traffic that must be allowed in and out of the subnet.
Oracle Cloud Infrastructure File Storage
Oracle Cloud Infrastructure File Storage service provides a durable, scalable, secure, enterprise-grade network file system. You can connect to a File Storage service file system from any bare metal, virtual machine, or container instance in your Virtual Cloud Network (VCN). The File Storage service supports the Network File System version 3.0 (NFSv3) protocol. The service supports the Network Lock Manager (NLM) protocol for file locking functionality.

Oracle Cloud Infrastructure File Storage employs 5-way replicated storage, located in different fault domains, to provide redundancy for resilient data protection. Data is protected with erasure encoding. The service is designed to meet the needs of applications and users that need an enterprise file system across a wide range of use cases.
Autonomous Database
Oracle Autonomous Database is a fully managed, preconfigured database environments that you can use for transaction processing and data warehousing workloads. You do not need to configure or manage any hardware, or install any software. Oracle Cloud Infrastructure handles creating the database, as well as backing up, patching, upgrading, and tuning the database.
Oracle Autonomous Database on Dedicated Exadata Infrastructure
Oracle Autonomous Database on Dedicated Exadata Infrastructure is an Oracle Autonomous Database with a private database cloud in the public cloud. With your dedicated database, you get a completely dedicated compute, storage, network, and database service, providing the highest security, isolation, and governance levels.
Oracle Autonomous Database Serverless

Oracle Autonomous Database Serverless is an Oracle Autonomous Database. You have a fully elastic database where Oracle autonomously operates all aspects of the database lifecycle from database placement to backup and updates.
Data Guard
Oracle Data Guard provides a comprehensive set of services that create, maintain, manage, and monitor one or more standby databases to enable production Oracle databases to remain available without interruption. Oracle Data Guard maintains these standby databases as copies of the production database. Then, if the production database becomes unavailable because of a planned or an unplanned outage, Oracle Data Guard can switch any standby database to the production role, minimizing the downtime associated with the outage.
Oracle Autonomous Data Guard
Oracle Autonomous Data Guard enables a standby (peer) database to provide data protection and disaster recovery for your Autonomous Database instance. It provides a comprehensive set of services that create, maintain, manage, and monitor one or more standby databases to enable production Oracle databases to remain available without interruption. Oracle Data Guard maintains these standby databases as copies of the production database. Then, if the production database becomes unavailable because of a planned or an unplanned outage, you can switch any standby database to the production role, minimizing the downtime associated with the outage.
Snapshot Standby
A snapshot standby database is a fully updateable standby database created by converting a physical standby database into a snapshot standby database.

A snapshot standby database receives and archives, but does not apply, redo data from a primary database. The redo data received from the primary database is applied once a snapshot standby database is converted back into a physical standby database, after discarding all local updates to the snapshot standby database.
Refreshable Clone
Oracle Autonomous Database provides cloning where you can choose to create a full clone of the active instance, create a metadata clone, or create a refreshable clone. With a refreshable clone the system creates a clone that can be easily updated with changes from the source database.

Considerations for the Middle Tier

All of the considerations in Oracle WebLogic Server for Oracle Cloud Infrastructure Disaster Recovery and SOA Suite on Oracle Cloud Infrastructure Marketplace Disaster Recovery for the middle tier are applicable in the Oracle Autonomous Database scenarios described in this document.

Consider the following aspects for the middle tier:

Front-end address
Access from clients to the Oracle WebLogic Server system must be agnostic to the site that is being used as the primary site. To accomplish this, the front-end address name used to access the system must be unique and always point to the system that is primary at the moment. This name is usually referred to as the virtual front-end or vanity URL.

You can reuse the existing system’s front-end host name address as the virtual front-end for disaster protection. For example, if the original system has mywebapps.example.com as the vanity URL for the primary system, then you can remap the same virtual host name to the second site’s load balancer IP after a switchover or failover.

Use an appropriate domain name system (DNS) service for the front-end name to be mapped to either site. For example, (Oracle Cloud Infrastructure DNS service, other commercial DNS, local DNS, or local hosts resolution).
Instance Name Prefix
Provide an Instance Name Prefix when you provision Oracle WebLogic Server for OCI or Oracle SOA Suite on Marketplace. This property is used to construct the names of all of the resources, including: the WebLogic Server domain name, the cluster name, the WebLogic server names, the VM’s host names, and so on.

Set the Instance Name Prefix to the same value in primary and secondary Oracle WebLogic Server systems, so that both systems have the same name for the Oracle WebLogic resources. Using the same name guarantees consistency and is required for the recovery of JMS messages and TLogs. It also simplifies customizations and operations in both sites.

You can use the same Instance Name Prefix in multiple instances in the same Oracle Cloud tenancy, as long as you create them in different regions or compartments. Each instance is only shown in its specific region and compartment.

The Oracle SOA Suite on Marketplace provisioning process provides an optional feature that allows you to configure custom names for the domain, the cluster, the admin server, the managed server’s prefix, etc. In that case, the names are not derived from the Instance Name Prefix. They take the values provided instead. It is possible to use this feature in the disaster recovery (DR) topology described in this document, as long as the custom names provided are the same in the primary and standby systems.
Custom files
Most of the Oracle WebLogic Server for OCI domain configuration that WebLogic Cloud uses is synced initially across sites with the following considerations:

Each WebLogic system maintains the original JDBC URLs used to connect to their local database, even after the DR set up has completed. Only the schema prefix is altered so that both locations point to the same schemas (primary schemas).

The WebLogic Domain Infrastructure feature automatically distributes the configuration under the weblogic_domain_name/config directory to the other nodes on the same domain.

Custom application deployments (ear/war files, deployment plans, etc.) and everything residing under the Oracle WebLogic Administration Server domain directory (except temp data) is initially synced across sites using the procedures described in this document. If Oracle WebLogic Administration Server uses data that resides in other nodes or outside the domain directory, then you must manually copy it to the secondary location. Additional details about replicating the configuration are provided later.

Considerations for Snapshot Standby on Oracle Autonomous Database Serverless

When implementing this solution, consider the following when enabling Snapshot Standby on an Oracle Autonomous Database Serverless database.

Time limit for snapshot standby in a serverless infrastructure
When a snapshot standby in Oracle Autonomous Database Serverless is not reconnected within 48 hours, the snapshot standby automatically reconnects to the source database.
Converting back to disaster recovery peer
Oracle recommends that you convert a snapshot standby back to a disaster recovery peer as soon as you are done with the operations that required the standby to be open for read-write operations. When you convert back to a disaster recovery peer, the accumulated changes from the source database are applied on the peer. If you keep the disaster recovery peer open as a snapshot standby for a longer period, assuming there are ongoing changes on the primary during this time, then it takes longer to convert back to a disaster recovery peer.
Cost implications of the snapshot standby in Oracle Autonomous Database Serverless
Snapshot standby CPU usage is billed based on the base CPU count and any additional CPU usage if compute auto scaling is enabled. The number of base CPUs is specified by the number of ECPUs (OCPUs if your database uses OCPUs) as shown in the ECPU count or OCPU count field on the Oracle Cloud Infrastructure Console.

Snapshot standby storage usage is billed based on the storage of the snapshot standby plus 1 times the storage of the source primary database.

For more details, see About Disaster Recovery Snapshot Standby Databases.

Considerations for Snapshot Standby on Oracle Autonomous Database on Dedicated Exadata Infrastructure

When implementing this solution, consider the following when enabling Snapshot Standby on an Oracle Autonomous Database on Dedicated Exadata Infrastructure.

Time limit for snapshot standby in a dedicated infrastructure
When a snapshot standby in Oracle Autonomous Database on Dedicated Exadata Infrastructure is not converted to physical standby within 7 days, the snapshot standby is automatically converted to physical standby.
Converting back to a physical standby
Oracle recommends that you convert a snapshot standby back to a physical standby as soon as you are done with the operations that require the standby to be open for read-write operations. When you convert back physical standby, the accumulated changes from the source database are applied on the standby. If you keep the standby database open as a snapshot standby for a longer period, assuming there are ongoing changes on the primary during this time, then it takes longer to convert back to a physical standby.
Database services when converting to a snapshot standby
In Oracle Autonomous Database on Dedicated Exadata Infrastructure, the "Convert to snapshot standby" dialog displays two options:
- Use new Database services: Select this option to connect to a snapshot standby using new services that are active only in the snapshot standby mode.
- Use primary Database services: Select this option if you wish to connect to snapshot standby database using the same services as the primary database.
For the disaster recovery setup, use the option Use primary Database services when you convert the standby into physical standby. This way, the connect alias name configured by Oracle WebLogic Server in the secondary mid-tier is consistent with the primary.

For more details, see About Autonomous Data Guard.

Considerations for Remote Refreshable Clones on Oracle Autonomous Database Serverless

Consider the following when using an Oracle Autonomous Database Serverless Refreshable Clone to test and validate a secondary Oracle WebLogic Server for Oracle Cloud Infrastructure, Oracle SOA Suite on Marketplace, or Oracle Fusion Middleware system.

Refreshable Clones lifecycle
Contrary to a traditional Oracle Data Guard Standby, Remote Refreshable Clones are enabled separately and managed separately from a primary and a standby. It is a separate entity with its own lifecycle beyond the refresh operations that bring it in sync with its source database (primary).
CPU resource allocation
Remote Refreshable Clones are not created based on the primary or standby system's CPU resource allocation. This means that you must specify OCPU options for the Refreshable Clone separately. You must manually configure workload tests on the remote Refreshable Clone to match the primary system's capacity. Ideally, you should create the remote Refreshable Clone with a configuration matching the primary's database so that testing workloads is realistic on the secondary. However, remote Refreshable Clones "carry over" the storage configuration used in primary.
Patching
Patches are automatically applied weekly on Oracle Autonomous Database Serverless, per the weekly maintenance window, so there is a continuous and forceful sync of patches between primary, standby, and remote Refreshable Clone.
Service limits
Remote Refreshable Clones are a first class entity, they incur in additional cost per the storage, CPU, and license implications of a formal Autonomous Database and will count towards the region’s service limits for Oracle Autonomous Database Serverless.
Refreshable Clone at switchover
When a failover or "non-immediately reversible switchover" takes place, you must manually create a refreshable clone on primary to keep the system ready for test and maintenance operations in what is now the secondary system, with the appropriate service limits, management, and other considerations). Remote Refreshable Clones lack role-reversal control.

After a switchover to the secondary, the refreshable clone created won't be able to refresh anymore (since its source is now a standby) and is marked as disconnected. After 24 hours it becomes a non-refreshable and disconnected clone.
Refreshable Clones refresh window
Remote Refreshable Clones must be refreshed at least once a week. After that, being in sync with the primary's data requires creating a new remote Refreshable Clone and discarding the non-refreshed one.
Refreshable Clones write mode
Remote Refreshable Clones cannot stay in write mode (disconnect from primary) for more than 24 hours. After that period, the remote Refreshable Clones database cannot be connected again to its primary. After that, being in sync with the primary's data requires creating a new Remote Refreshable Clone and discarding the non-refreshed one.

Considerations for the `tns_admin` Configuration Folder Location

Consider the following for the tns_admin configuration folder.

Use consistent location for the tns_admin folder
The mid-tiers in a WebLogic Domain use a folder to store the Oracle Autonomous Database wallet and the tnsnames.ora file. The property oracle.net.tns_admin points to this folder in the datasources and jps configuration files. The path of this folder must be the same in primary and standby midtiers. If the folder path is different, change the folder either in primary or standby, so they use the same folder, before running the disaster recovery (DR) setup scripts.
Note:
The following might cause differences between primary and standby in this folder location:
- Oracle SOA Suite on Marketplace instances provisioned before the end of February 2023 (before release 23.1.1) use $DOMAIN_HOME/config/atp for the tns_admin folder. Starting with release 23.1.1, the location is $DOMAIN_HOME/config/tnsadmin.
The tns_admin folder under the domain config folder
Check the location of the Oracle Autonomous Database wallet in the WebLogic datasource configuration. If the wallet is not located under the DOMAIN_HOME/config directory, then changes to the contents of the wallet directory will NOT be replicated by the Oracle WebLogic Server infrastructure to other nodes automatically. In these cases, you must either change the wallet directory (and update the required datasources configurations), or manually copy it to other nodes when it gets updated.