3 Setting Up and Managing Disaster Recovery Sites
Learn about the Oracle SOA Suite enterprise deployment topology that illustrates how to set up production and standby sites.
Note:
You can automate disaster recovery operations such as switchover and failover by using Oracle Site Guard. See Oracle Site Guard Administrator's Guide.
This chapter includes the following sections:
- Setting Up a Site
Learn how to set up an Oracle Disaster Recovery site. - Creating a Production Site on an Oracle SOA Enterprise Topology
Learn how to create a production site on an Oracle SOA enterprise deployment topology. - Creating a Standby Site
Learn how to create a standby site. - Performing Site Operations and Administration
Learn how to operate and administer your Oracle Fusion Middleware Disaster Recovery topology. - Using Oracle Site Guard for Disaster Recovery
Oracle Site Guard is a disaster-recovery solution that enables administrators to automate complete site switchover or failover. It orchestrates the coordinated failover of Oracle Fusion Middleware, Oracle Fusion Applications, and Oracle Databases. - Patching an Oracle Fusion Middleware Disaster Recovery Site
Apply an Oracle Fusion Middleware patch set to upgrade the Oracle homes that participate in an Oracle Fusion Middleware Disaster Recovery site. - Accessing and Managing T3/T3s External Clients in DR Scenarios
This section discusses different approaches for accessing and managing external T3/T3s clients in the context of a Disaster Recovery scenario.
Setting Up a Site
Learn how to set up an Oracle Disaster Recovery site.
Before you start creating the production site, ensure that you:
-
Set up the host name aliases for the middle tier hosts, as described in Planning Host Names.
-
Create the required volumes on the shared storage on the production site, as described in Designing Directory Structure and Volumes.
-
Determine the Oracle Data Guard configuration to use based on the data loss requirements of the database and network considerations, such as the available bandwidth and latency when compared to the redo generation.
This section includes the following topics:
- Designing Directory Structure and Volumes
Learn about the recommended directory structure in your disaster recovery topology. - Setting Up Storage Replication
Learn how to set up storage replication for the Oracle Fusion Middleware Disaster Recovery topology using Storage level replication, Rsync and Database file system. - Configuring Oracle Data Guard for the FMW Database
Learn how to install and configure Oracle Data Guard for the FMW Database in an Oracle SOA Suite enterprise deployment.
Parent topic: Setting Up and Managing Disaster Recovery Sites
Designing Directory Structure and Volumes
Learn about the recommended directory structure in your disaster recovery topology.
You can choose a directory layout different from the one recommended in this document, but the model adopted enables maximum availability, provides the best isolation of components and symmetry in the configuration, and facilitates backup and disaster recovery.
The following list describes directories and directory environment variables:
-
ORACLE_BASE
: This environment variable and related directory path refers to the base directory below which Oracle products are installed. -
ORACLE_HOME
: This related directory path refers to the location where Oracle Fusion Middleware resides. -
WL_HOME
: This environment variable and related directory path contains installed files that are necessary to host an Oracle WebLogic Server. -
PROD_DIR
: This environment variable and related directory path refers to the location where a product suite, such as Oracle SOA Suite, Oracle WebCenter Portal, or Oracle Identity Management is installed. -
DOMAIN
directory: This directory path refers to the location where the Oracle WebLogic Domain information (configuration artifacts) is stored. Different WebLogic Servers can use different domain directories even when in the same node. -
ORACLE_INSTANCE
: An Oracle instance contains one or more system components. An Oracle instance directory contains updatable files, such as configuration files, log files, and temporary files.
See Recommended Directory Structure for Oracle SOA Suite.
This section includes the following topic:
- Recommended Directory Structure for Oracle SOA Suite
Learn about the recommended directory structures for Oracle SOA Suite. - Recommended Volume Design for Oracle SOA Suite
Learn about the recommended volume design for Oracle SOA Suite.
Parent topic: Setting Up a Site
Recommended Directory Structure for Oracle SOA Suite
Learn about the recommended directory structures for Oracle SOA Suite.
Oracle Fusion Middleware allows you to create multiple SOA Managed Servers from a single binary installation. This allows the installation of binary files in a single location on a shared storage and the reuse of this installation by the servers in different nodes. However, for maximum availability, Oracle recommends that you use redundant binary installations. In this model, two Oracle homes (each of which has a WL_HOME
and an ORACLE_HOME
for each product suite) are installed in a shared storage. Additional servers (when scaling out or up) of the same type can use either one of these two locations without requiring more installations. Ideally, users should use two different volumes for redundant binary location, this isolating the failures as much as possible in each volume. For additional protection, Oracle recommends using storage replication for these volumes. If multiple volumes are not available, Oracle recommends that you use mount points to simulate the same mount location in a different directory in the shared storage. Although this does not guarantee the protection that multiple volumes provide, it does allow protection from user deletions and individual file corruption.
Oracle also recommends separating the domain directory that is used by the Administration Server from the domain directory that is used by Managed Servers. This allows a symmetric configuration for the domain directories that is used by Managed Servers, and isolates the failover of the Administration Server. The domain directory for the Administration Server must reside in a shared storage to allow failover to another node with the same configuration. In addition, Oracle recommends that you place the Managed Server's domain directories on a shared storage, although having them on the local file system is also supported. This is especially important when you design a production site with the disaster recovery site in mind. Figure 3-1 shows the directory structure layout for Oracle SOA Suite.
Figure 3-1 Recommended Shared Storage Directory Structure for an Enterprise Deployment
![This image shows the recommended shared storage directory structure for an enterprise deployment. This image shows the recommended shared storage directory structure for an enterprise deployment.](img/edg-shared-storage-directory-structure.png)
Figure 3-2 Recommended Local Storage Directory Structure for an Enterprise Deployment
![This image shows the recommended local storage directory structure for an enterprise deployment. This image shows the recommended local storage directory structure for an enterprise deployment.](img/edg-app-tier-local-storage.png)
For information about setting up this directory structure, see Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.
Parent topic: Designing Directory Structure and Volumes
Recommended Volume Design for Oracle SOA Suite
Learn about the recommended volume design for Oracle SOA Suite.
Figure 3-3 and Figure 3-4 shows an Oracle SOA Suite topology diagram. The volume design described in this section is for this Oracle SOA Suite topology. Detailed instructions for installing and configuring this topology are provided in the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.
Figure 3-3 Oracle SOA Suite and Oracle Business Activity Monitoring Enterprise Topology Diagram
![This image shows the Oracle SOA, Business Process Management, and Oracle Service Bus enterprise deployment topology. This image shows the Oracle SOA, Business Process Management, and Oracle Service Bus enterprise deployment topology.](img/soa-bam-ent-topology.png)
Figure 3-4 Oracle SOA Suite and Oracle Service Bus Enterprise Deployment Reference Topology Diagram
![This image shows the SOA Topology with Oracle Access Manager and Business Process Management. This image shows the SOA Topology with Oracle Access Manager and Business Process Management.](img/soa-service-bus-ent-deploy.png)
For disaster recovery of this Oracle SOA Suite topology, Oracle recommends the following volume design:
-
Provision two volumes for two Oracle homes that contain the redundant product binary files (
VOLFMW1
andVOLFMW2
in Table 3-1). -
Provision one volume for the Administration Server domain directory (
VOLADMIN
in Table 3-1). -
Provision one volume on each node for the Managed Server domain directory (
VOLSOA1
andVOLSOA2
in Table 3-1). This directory is shared among all the Managed Servers on that node. -
Provision one volume on each node for the Oracle HTTP Server Oracle home (
VOLWEB1
andVOLWEB2
in Table 3-1). -
Provision one volume on each node for the Oracle HTTP Server Domain Directory (
VOLOHS1
andVOLOHS2
in Table 3-1).Note:
For WebTier hosts, local storage is usually recommended. You can replicate this configuration on a regular basis to one of the other app tier volumes to sync to standby or directly from the production webhost to the standby webhost.
Table 3-1 provides a summary of Oracle recommendations for volume design for the Oracle SOA Suite topology shown in Figure 3-3 and Figure 3-4.
Table 3-1 Volume Design Recommendations for Oracle SOA Suite
Tier | Volume Name | Mounted on Host | Mount Point | Comments |
---|---|---|---|---|
Web |
|
|
|
Volume for Oracle HTTP Server installation |
Web |
|
|
|
Volume for Oracle HTTP Server installation |
Web |
|
|
|
Volume for Oracle HTTP Server domain directory |
Web |
|
|
|
Volume for Oracle HTTP Server domain directory |
Web |
|
|
|
(Optional) Volume for static HTML content |
Web |
|
|
|
(Optional) Volume for static HTML content |
Application |
|
|
|
Volume for the WebLogic Server and Oracle SOA Suite binary files |
Application |
|
|
|
Volume for the WebLogic Server and Oracle SOA Suite binary files |
Application |
|
|
|
Volume for Administration Server domain directory and other shared configurations, such as Deployment Plans, applications, and keystores |
Application |
|
|
|
Volume for Managed Server domain directory |
Application |
|
|
|
Volume for Managed Server domain directory |
Application |
|
|
|
Volume for shared runtime content. For example, files used by file adapters, MFT transfers, and other runtime artifacts. Note: It is recommended to store JMS messages and TLOGS in the database, using JDBC persistent stores, instead of this folder. |
For consistency group recommendations, see:
- Recommended Consistency Groups for Oracle SOA Suite
Learn about the recommended consistency groups for Oracle SOA Suite.
Parent topic: Designing Directory Structure and Volumes
Recommended Consistency Groups for Oracle SOA Suite
Learn about the recommended consistency groups for Oracle SOA Suite.
Oracle recommends the following consistency groups for the Oracle SOA Suite topology:
-
Create one consistency group with the volumes that contains the domain directories for the Administration Server and Managed Servers as members (
DOMAINGROUP
in Table 3-2). -
Create one consistency group with the volume that contains the JMS file store and transaction log data as members (
RUNTIMEGROUP
in Table 3-2). -
Create one consistency group with the volume that contains the Oracle homes as members (
FMWHOMEGROUP
in Table 3-2).
Table 3-2 provides a summary of Oracle recommendations for consistency groups for the Oracle SOA Suite topology as shown in Figure 3-3.
Table 3-2 Consistency Groups for Oracle SOA Suite
Tier | Group Name | Members | Comments |
---|---|---|---|
Application |
|
|
Consistency group for the Administration Server, and the Managed Server domain directory |
Application |
|
|
Consistency group for the shared runtime content |
Application |
|
|
Consistency group for the Oracle homes |
Parent topic: Recommended Volume Design for Oracle SOA Suite
Setting Up Storage Replication
Learn how to set up storage replication for the Oracle Fusion Middleware Disaster Recovery topology using Storage level replication, Rsync and Database file system.
- Storage Level Replication
The Oracle Fusion Middleware Disaster Recovery solution uses storage replication technology for disaster protection of Oracle Fusion Middleware middle tier components. - Rsync
Alternatively, rsync can be used for replication. Rsync is a versatile copying tool, that can copy locally, or to/from another host over any remote shell. It offers a large number of options that control every aspect of its behavior, and permit very flexible specification of the set of files to be copied. - Oracle Database File System
Oracle Database File System (DBFS) is an additional method that can be used for replicating the configuration. Conceptually, a database file system is a file system interface placed on top of files and directories that are stored in database tables.
Parent topic: Setting Up a Site
Storage Level Replication
The Oracle Fusion Middleware Disaster Recovery solution uses storage replication technology for disaster protection of Oracle Fusion Middleware middle tier components.
To set up storage replication for the Oracle Fusion Middleware Disaster Recovery topology:
-
On the standby site, ensure that the alias host names that are created are the same as the alias host names that are used for the peer hosts at the production site.
-
On the shared storage at the standby site, create the same volumes as created on the shared storage at the production site.
-
On the standby site, create the same mount points and symbolic links that you created at the production site.
Note:
-
The symbolic links only need to be set up on the standby site if you set up symbolic links at the production site.
-
The symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes. For more information about symbolic links, see Setting Up Storage.
-
-
It is not necessary to install the same Oracle Fusion Middleware instances at the standby site as installed at the production site. When the production site storage is replicated to the standby site storage, the Oracle software installed on the production site volumes are replicated at the standby site volumes.
-
Create the baseline snapshot copy of the production site shared storage that sets up the replication between the production site and standby site shared storage. Create the initial baseline copy and subsequent snapshot copies by using asynchronous replication mode. After the baseline snapshot copy is performed, validate that all the directories inside the standby site volumes have the same contents as the directories inside the production site volumes.
-
Set up the frequency of subsequent copies of the production site shared storage, which is replicated at the standby site. When asynchronous replication mode is used, then at the requested frequency the changed data blocks at the production site shared storage (based on comparison to the previous snapshot copy) become the new snapshot copy, and the snapshot copy is transferred to the standby site shared storage.
-
Ensure that disaster protection for any database that is included in the Oracle Fusion Middleware Disaster Recovery production site is provided by Oracle Data Guard. Do not use storage replication technology to provide disaster protection for Oracle databases.
-
The standby site shared storage receives snapshots transferred periodically from the production site shared storage. After the snapshots are applied, the standby site shared storage includes all the data up to and including the data contained in the last snapshot transferred from the production site before the failover or switchover.
-
Oracle strongly recommends that you manually force a synchronization operation whenever a change is made to the middle tier at the production site (for example, when a new application is deployed at the production site). Follow the vendor-specific instructions for forcing a synchronization by using storage replication technology.
Parent topic: Setting Up Storage Replication
Rsync
Alternatively, rsync can be used for replication. Rsync is a versatile copying tool, that can copy locally, or to/from another host over any remote shell. It offers a large number of options that control every aspect of its behavior, and permit very flexible specification of the set of files to be copied.
Rsync implements delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Due to these advantages and easy to use ability, it is a widely used tool for backups and mirroring.
Although rsync is reliable and implements implicit retries, the network outages and other connectivity issues can still cause failures in the file synchronization. Hence, rsync can be used as an alternative to the storage level replication under these conditions:
- When the storage replication is not possible or feasible and/or does not meet the cost requirements.
- When primary and standby use a reliable and secure network connection for the copy.
- When checks are performed in the copy to make sure that it is valid.
- When the disaster recovery site is validated on a regular basis.
For more details about how to use rsync to replicate the file system artifacts, see Using Peer-To-Peer File Copy.
Parent topic: Setting Up Storage Replication
Oracle Database File System
Oracle Database File System (DBFS) is an additional method that can be used for replicating the configuration. Conceptually, a database file system is a file system interface placed on top of files and directories that are stored in database tables.
DBFS is similar to NFS in that it provides a shared network file system that looks like a local file system and has both a server component and a client component. The DBFS file system can be mounted from the mid-tier hosts and accessed as a regular shared file system. It could be used as an intermediate location to replicate content: any content that is copied to the DBFS mount, as it resides in the database, is automatically replicated to the standby site through the underlying Data Guard replication.
This method takes advantage of the robustness of the Data Guard replica. It has good availability through Oracle Driver’s retry logic and provides a resilient behavior. It can be used in scenarios with medium or high latencies between the data centers. However, using DBFS for configuration replication has also additional implications from the setup, database storage and lifecycle perspectives:
- It introduces some complexity due to the configuration and maintenance
required by the DBFS mount. It requires the DB client to be installed in the host
that is going to mount it, it requires an initial setup of some artifacts in the
database (tablespace, user, and so on) and in the client (the wallet, the
tnsnames.ora
, and so on ). See the scriptdbfs_dr_setup_root.sh
(in Using the Application DR common scripts) as an example script that installs the database client, creates the DBFS schema in the database, configures the client artifacts, and mounts the DBFS file system in a mid-tier host. - It requires additional capacity in the Database, because the content copied to the DBFS mount is stored in the database.
- Oracle does not recommend to store the domain configuration or the binaries directly in the DBFS mount. This would create a very strong dependency between the FMW files and the database. Instead, it is recommended to use the DBFS as an "assistance" file system: an intermediate staging file system to place the info that is going to be replicated to the standby site. Any replication to standby would have two steps: from the primary's origin folder to the intermediate DBFS mount, and then, in the standby site, from the DBFS mount to the standby's destination folder.
- The DBFS can be mounted only if the database is open. When the Data Guard is not an Active Data Guard (ADG), the standby database is in mount state. Hence, in order to access to the DBFS mount in the standby site in such cases, the database needs to be converted to snapshot standby. When ADG is used, however, the file system can be mounted for reads and there is no need to transition to snapshot.
Due to the above stated reasons, it is not recommended to use this approach as a general purpose solution to replicate all the artifacts to the standby. For example, using DBFS to replicate the binaries is an overkill. However, this approach is suitable to replicate some dynamic artifacts during the lifecycle, like the domain shared configuration, when other methods like the storage replication or rsync are not feasible. See the Oracle WebLogic Server for Oracle Cloud Infrastructure Disaster Recovery paper to see an example on how to use DBFS to replicate the domain configuration.
Parent topic: Setting Up Storage Replication
Configuring Oracle Data Guard for the FMW Database
Learn how to install and configure Oracle Data Guard for the FMW Database in an Oracle SOA Suite enterprise deployment.
For recommendations and considerations for setting up Oracle databases that are used in an Oracle Fusion Middleware Disaster Recovery topology, see Database Considerations.
Oracle Maximum Availability Architecture (MAA) is Oracle's comprehensive architecture to reduce downtime for scheduled outages, and to prevent, detect and recover, from unscheduled outages.
Real Application Clusters (RAC) and Data Guard provide the basis of the database MAA solution, where the primary site contains the RAC database, and the secondary site contains the RAC physical standby database.
Tip:
Alternatively, you can perform many of the tasks in this section by using Oracle Enterprise Manager Cloud Control.
Setting up and managing databases using Cloud Control helps in controlling downtime and simplifies disaster recovery operations.
For information about Installing Enterprise Manager Cloud Control 13c, see Cloud Control Basic Installation Guide .
For more information about Setting up Oracle Data Guard using Cloud Control, see Provisioning Oracle Standby Databases in Oracle Enterprise Manager Lifecycle Management Administrator's Guide.
- Prerequisites and Assumptions
- Oracle Data Guard Environment Description
- Procedure for Configuring the Data Guard
- Verifying the Data Guard Broker Configuration
- Testing Database Switchover and Switchback
You can perform a database switchover and switchback.
Parent topic: Setting Up a Site
Prerequisites and Assumptions
Ensure that the following prerequisites are met:
-
The Oracle RAC cluster and Automatic Storage Management (ASM) instances on the standby site have been created.
-
The Oracle RAC databases on the standby site and the production site are using a flash recovery area.
-
The Oracle RAC databases are running in
archivelog
mode. -
The database hosts on the standby site already have Oracle software installed.
-
In a shared
ORACLE_HOME
configuration, theTNS_ADMIN
directory must be a local, non-shared directory.
Parent topic: Configuring Oracle Data Guard for the FMW Database
Oracle Data Guard Environment Description
The examples given in this section contain environment variables as described in Table 3-3.
Table 3-3 Variables Used by Primary and Standby Databases
Variable | Primary Database | Standby Database |
---|---|---|
Database names |
|
|
SOA Database Host Names |
|
|
Database unique names |
|
|
Instance names |
|
|
Service names |
|
|
Parent topic: Configuring Oracle Data Guard for the FMW Database
Procedure for Configuring the Data Guard
The configuration of a Data Guard involves several steps: you need to prepare the primary database with the recommended parameters, prepare the TNS aliases in primary and standby environments, create the physical standby database as a duplication of the primary database, configure the Data Guard Broker, and so on.
Oracle provides a set of sample scripts that can be used to automate most of
these actions. These scripts are available here: https://github.com/oracle-samples/maa/raw/main/dg_setup_scripts/dg_setup_scripts.zip.
These scripts are designed to setup a standby database for an existing primary database,
using the restore from service feature and Data Guard Broker. They allow
customization (OS user names and Oracle Homes are configurable values). They support
databases with and without TDE encryption, and they work in environments with Read Only
Oracle Home (ROOH) or with regular Oracle Homes. The scripts are validated in 12c (12.2),
18c, 19c and 21c RDBMS versions. Download the file and follow the instructions included in
the README.md
to configure a standby database.
If the scripts are not feasible for your specific environment, you can manually configure the Data Guard by following one of these documents:
- Creating a Physical Standby database using RMAN restore database from service (Doc ID 2283978.1)
- Creating a Physical Standby using RMAN Duplicate (RAC or Non-RAC) (Doc ID 1617946.1)
Note:
You can find the above documents in My Oracle Support.Parent topic: Configuring Oracle Data Guard for the FMW Database
Verifying the Data Guard Broker Configuration
Complete the following steps to verify that the Data Guard Broker configuration was created successfully.
Example 3-1 Verifying the Data Guard Broker Configuration
[oracle@dbhost1 ~]$ dgmgrl sys/'<password>' DGMGRL for Linux: Release 19.0.0.0.0 - Production on Tue Feb 1 09:00:16 2022 Version 19.6.0.0.0 Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved. Welcome to DGMGRL, type "help" for information. Connected to "soa_pri" Connected as SYSDBA. DGMGRL> show configuration Configuration - dg_config Protection Mode: MaxPerformance Databases: soa_pri - Primary database soa_stby - Physical standby database Fast-Start Failover: DISABLED Configuration Status: SUCCESS
Parent topic: Configuring Oracle Data Guard for the FMW Database
Testing Database Switchover and Switchback
You can perform a database switchover and switchback.
Performing a Switchover Operation by Using Oracle Data Guard Broker
To perform a switchover operation by using Oracle Data Guard Broker, complete the following tasks:
-
Verify the Oracle Data Guard Broker configuration by running the following command:
DGMGRL> show configuration; Configuration - dg_config Protection Mode: MaxPerformance Databases: soa_pri - Primary database soa_stby - Physical standby database Fast-Start Failover: DISABLED Configuration Status: SUCCESS
-
Swap the roles of the primary and standby databases by running the
SWITCHOVER
command. Example 3-1 shows how Data Guard Broker automatically shuts down and restarts the old primary database as part of the switchover operation.DGMGRL> switchover to 'soa_stby' Performing switchover NOW, please wait... Operation requires a connection to database "soa_stby" Connecting ... Connected to "soa_stby" Connected as SYSDBA. New primary database "soa_stby" is opening... Oracle Clusterware is restarting database "soa_pri" ... Connected to "soa_pri" Connected to "soa_pri" Switchover succeeded, new primary is "soa_stby"
-
After the switchover is complete, use the
SHOW CONFIGURATION
command to verify that the switchover operation was successful:DGMGRL> show configuration; Configuration - dg_config Protection Mode: MaxPerformance Databases: soa_stby - Primary database soa_pri - Physical standby database Fast-Start Failover: DISABLED Configuration Status: SUCCESS
Note:
For information about switchover and failover operation of Oracle Data Guard Broker, see Switchover and Failover Operations in the Oracle Data Guard Broker.
Parent topic: Configuring Oracle Data Guard for the FMW Database
Creating a Production Site on an Oracle SOA Enterprise Topology
Learn how to create a production site on an Oracle SOA enterprise deployment topology.
Before you create your production site:
-
Set up the host name aliases for the middle tier hosts, as described in Planning Host Names.
-
Create the required volumes on the shared storage on the production site, as described in Designing Directory Structure and Volumes.
-
Determine the Oracle Data Guard configuration to use based on the data loss requirements of the database and network considerations, such as the available bandwidth and latency when compared to the redo generation.
This section includes the following topics:
- Creating a Production Site
This section provides information about how to create a production site for the Oracle SOA Suite topology. - Configuring Data Sources for Oracle Fusion Middleware Active-Passive Deployment
Configure the data sources that Oracle Fusion Middleware uses for active-passive disaster recovery.
Parent topic: Setting Up and Managing Disaster Recovery Sites
Creating a Production Site
This section provides information about how to create a production site for the Oracle SOA Suite topology.
If you plan to create a production site for a different topology, see the appropriate Oracle Fusion Middleware Enterprise Deployment Guide listed under the Install a Production Environment: Plan, Install & Configure an Enterprise Deployment category.
Install and configure your production site as described in the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.
The following sections describe how to complete the installation and configuration of your production site:
- Creating Volumes and Consistency Groups
Create volumes and consistency groups on the shared storage device. - Setting Up Physical Host Names and Alias Host Names
Set up physical host names on the production site and set up the physical host names and alias hostnames on the standby site. - Installing and Configuring Oracle SOA Suite
Install and configure Oracle SOA Suite.
Creating Volumes and Consistency Groups
Create volumes and consistency groups on the shared storage device.
To create volumes and consistency groups on the shared storage device, see Recommended Volume Design for Oracle SOA Suite.
Parent topic: Creating a Production Site
Setting Up Physical Host Names and Alias Host Names
Set up physical host names on the production site and set up the physical host names and alias hostnames on the standby site.
For information about planning host names for the production and standby sites, see Planning Host Names.
Parent topic: Creating a Production Site
Installing and Configuring Oracle SOA Suite
Install and configure Oracle SOA Suite.
To install and configure Oracle SOA Suite, see Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite and apply the following modifications:
- Install the Oracle SOA Suite components into the volumes created on the shared storage device.
- Set up the production and standby sites by using the aliases to the physical and virtual host names.
- Create SSL certificates by using the host name aliases on all of the Oracle Fusion Middleware hosts for proper Node Manager communication.
Parent topic: Creating a Production Site
Configuring Data Sources for Oracle Fusion Middleware Active-Passive Deployment
Configure the data sources that Oracle Fusion Middleware uses for active-passive disaster recovery.
As explained in the Setting Up DataSources in the Middle Tier, Oracle recommends that you use a TNS alias in the data sources
to access each site's local database. The TNS alias is the same name in production
and standby. Hence, the data sources use the same db connect string. The TNS alias
is resolved with a tnsnames.ora
file stored separately from the
WebLogic domain configuration and not replicated between sites. Therefore, you can
have different tnsnames.ora
content in each site. Each site
will resolve the TNS alias with the appropriate connection string in each site,
pointing to the local database only.
Another advantage of the active-passive deployment approach is that
changes (such as retries and timeouts) to the tnsnames.ora
file
can be done without requiring a WebLogic Server restart or a WebLogic Server data
source restart. At the same time, this approach reduces the repetition of addresses
and settings that are typically shared by many data sources, thereby reducing the
configuration creation and maintenance footprint.
If you are not already using this approach in the production system, you
can use the following steps to configure it. Configure a tns
alias
in all the data sources that are used in the domain, including the data sources used
by the JDBC persistence stores, leasing data sources, and custom data sources, and
in the JDBC URL of the OPSS security stores.
-
Create the
tns
folder in all the middle-tier hosts.This folder must be readable by the Oracle user and placed in a file system that is not replicated between sites.
For example:mkdir -p /home/oracle/tnsnames_dir
Given that the
tns
folder is part of the configuration, you can also create it under theconfig
folder that is shared by all the servers. But in that case, ensure that you exclude thetns
folder when you copy the domain configuration from primary to standby or updatetnsnames.ora
in the standby system, after a failover or switchover, to point to the secondary database.Note:
Create this folder in the middle-tier hosts in both production and standby. -
Create a
tnsnames.ora
file in thetns
folder, with thetns
alias that will be used in the data sources. In the production middle-tier hosts, the alias must point to the production database.For example:
SOAEDG= (DESCRIPTION= (ADDRESS_LIST= (LOAD_BALANCE=ON) (ADDRESS=(PROTOCOL=TCP)(HOST=prmy-scan)(PORT=1521)) ) (CONNECT_DATA=(SERVICE_NAME=soaedg.example.com)) )
In the standby middle-tier hosts, the alias is the same name, but it is pointing to the standby database.
For example:
SOAEDG= (DESCRIPTION= (ADDRESS_LIST= (LOAD_BALANCE=ON) (ADDRESS=(PROTOCOL=TCP)(HOST=stby-scan)(PORT=1521)) ) (CONNECT_DATA=(SERVICE_NAME=soaedg.example.com)) )
If you connect to additional databases or services, ensure that you add the appropriate
tns
alias to them too. -
Specify the
oracle.net.tns_admin
property pointing to the directory location of thetnsnames.ora
file. Use one of the following methods:Note:
Do not use a mix of these methods. It can cause unexpected behavior.Option 1: Set the property as a data source connection property. Oracle recommends this method.
Specify the
oracle.net.tns_admin=tns_directory
property in the data source configuration. To specify this property in the WebLogic Administration Console, go to Services, click Data Sources, select a data source from the list, click Connection Pool, and then add it to the Properties text box. Repeat this step for each data source.For example:
Add the following property to the Properties text box:oracle.net.tns_admin=/home/oracle/tnsnames_dir
You must also specify this property in the OPSS security stores files
jps-config-jse.xml
andjps-config.xm
available in the$ASERVER_HOME/config/fmwconfig
folder. To modify thesejps
files, edit them and add theoracle.net.tns_admin
property after thejdbc.url
property.For example:… <property name="jdbc.url" value="jdbc:oracle:thin:@soaedg"/> <property name="oracle.net.tns_admin" value="tns_directory"/> …
Note:
This property applies to the specific file (data source,jps
file) in which it is specified.Option 2: Set the property as a java system property.
Specify the-Doracle.net.tns_admin=tns_directory
system property wheretns_directory
is the directory location of thetnsnames.ora
file. To set it as a java property for the servers, edit the following files:$ASERVER_HOME/bin/setUserOverrides.sh
$MSERVER_HOME/bin/setUserOverrides.sh
(This file is not shared. Therefore, you should edit the file in all the SOA mid-tier hosts.)
Add the following content to these files:# For using tns alias in the datasources export EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES} -Doracle.net.tns_admin=/home/oracle/tnsnames_dir
Note:
- This property applies to all the data sources and
jps
files in WebLogic Server. - Before you run some of the WLST commands and the
Configuration Wizard, this approach requires that you set the
property in the environment. Before running WLST, you should set the
property in the WLST_PROPERTIES
environment variable. Before running the Configuration Wizard, you
should add the property to the JVM_ARGS environment variable of the
config_internal.sh
script.
Option 3: Set the property in the
jdbc
URL.Specify the location of thetnsnames.ora
file as part of the connection string in the data sources andjps
files:jdbc:oracle:thin:@alias?TNS_ADMIN=tns_directory
Note:
- This property applies to the specific file (data source,
jps
file) in which it is specified. - You can use this method with JDBC Driver 18.3 and later. This applies to Fusion Middleware 12.2.1.4 (which uses JDBC Driver 19.3) and later.
-
Modify the URL defined in the data sources by replacing the connection string with the alias. The following is a sample JDBC URL using the
tns
aliasjdbc:oracle:thin:@soaedg
.You can use the WebLogic Administration Console to perform this change. Alternatively, you can perform the modification directly in the files, as described here:- Sign in to
APPHOST1
. - Take a backup of
$ASERVER_HOME/config/jdbc
which contains the data source config files. - Run a command to replace the previous database
connection string by the new one that uses the
tns
alias.For example:cp -rf $ASERVER_HOME/config/jdbc $ASERVER_HOME/config/jdbc_bck export PREV_STRING='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=prmy-scan)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=soaedg.example.com)))' export NEW_STRING='soaedg' cd $ASERVER_HOME/config/jdbc find . -name '*.xml' | xargs sed -I 's|'${PREV_STRING}'|'${NEW_STRING}'|gI'
- Sign in to
- Update the JDBC URL of the OPSS security stores too by using the following
steps:
- Sign in to
APPHOST1
and go to the$ASERVER_HOME/config/fmwconfig
folder. - Take a backup of the
jps-config-jse.xml
andjps-config.xml
files. - Edit both files to update the value of the
property name="jdbc.url"
with the appropriate JDBC URL used in the data sources.For example:cp -rf $ASERVER_HOME/config/fmwconfig $ASERVER_HOME/config/fmwconfig_bck cd $ASERVER_HOME/config/fmwconfig export PREV_STRING='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=prmy-scan)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=soaedg.example.com)))' export NEW_STRING='soaedg' find . -name '*.xml' | xargs sed -i 's|'${PREV_STRING}'|'${NEW_STRING}'|gI'
- Sign in to
-
Restart all the WebLogic Servers in the domain for the changes to take effect.
- Stop all the WebLogic Servers in the domain (Administration Server and Managed Servers).
- Start the Administration Server in the domain.
- Start the Managed Servers.
-
Verify that the data source connections are established correctly with the database.
Verify that the JDBC URL is updated correctly in the OPSS store. To verify this:- Go to the Enterprise Manager Console.
- Navigate to WebLogic Domain, select Security, and then click Security Provider Configuration.
- Expand Security Stores and verify that the Database URL is updated.
Note:
Regarding the ONS host and port configuration in the data sources, these values are not required when you are using Oracle Database 12c and later because the ONS list is automatically obtained from the database by the client driver.
As per Oracle's consistent recommendation in this guide, use this feature instead of providing the scan address or the list of the RAC nodes in the ONS configuration of each data source.
Ensure that Fast Application Notification (FAN) is enabled and that the ONS nodes property is empty in each data source. To check this property in the WebLogic Administration Console, go to Services, click Data Sources, select Configuration, and then click ONS.
Creating a Standby Site
Learn how to create a standby site.
To create the standby site, the Oracle SOA enterprise deployment topology is used as an example.
This section includes the following topics:
- Preparing the Standby Site
Prepare your standby site for operation. - Validating the Standby Site Setup
Validate your standby site.
Parent topic: Setting Up and Managing Disaster Recovery Sites
Preparing the Standby Site
Prepare your standby site for operation.
To prepare it for operation, on your standby site:
-
Set up the correct alias host names and physical host names by following the instructions in Planning Host Names.
Ensure that each standby site host has an alias host name that is the same as the physical host name of its peer host at the production site.
-
Create, on the shared storage, the same volumes that were created on the shared storage at the production site.
-
Create the same mount points and symbolic links (if required) that you created at the production site. Note that symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes.
For more details about symbolic links, see Setting Up Storage.
- Setting Up Middle Tier Hosts
Middle tier hosts on a standby site do not require installation nor configuration of Oracle Fusion Middleware or Oracle WebLogic Server software. When the production site storage is replicated to the standby site storage, the software installed on the production site is replicated at the standby site. - About Self-Signed certificates and Keys on the Standby Site
Learn about certificates and keystores.
Parent topic: Creating a Standby Site
Setting Up Middle Tier Hosts
Middle tier hosts on a standby site do not require installation nor configuration of Oracle Fusion Middleware or Oracle WebLogic Server software. When the production site storage is replicated to the standby site storage, the software installed on the production site is replicated at the standby site.
To set up the middle tier hosts on the standby site:
- Create a baseline snapshot copy of the shared storage on the production site, which sets up the replication between the storage devices. Create the initial baseline copy and subsequent snapshot copies by using asynchronous replication mode.
- Synchronize the shared storage at the production site with the shared storage at the standby site. This transfers the initial baseline snapshot from the production site to the standby site.
- Set up the frequency of subsequent copies of the production site shared storage, which is replicated at the standby site. When asynchronous replication mode is used, then at the requested frequency the changed data blocks at the production site shared storage (based on comparison to the previous snapshot copy) become the new snapshot copy, and the snapshot copy is transferred to the standby site shared storage.
- After the baseline snapshot copy is performed, validate that all the directories inside the standby site volumes have the same content as the directories inside the production site volumes.
Parent topic: Preparing the Standby Site
About Self-Signed certificates and Keys on the Standby Site
Learn about certificates and keystores.
As recommended in Planning Host Names the FMW components use alias host names, that are resolvable to the IP addresses of the appropriate system in each site. If the primary self-signed certificates are created with these host names, the certificates are valid both for the production and for the standby systems.
For more information about creating primary self-signed certificates, see Common Configuration and Management Tasks for an Enterprise Deployment in Enterprise Deployment Guide for Oracle SOA Suite.
The certificates and keystores are replicated to standby along with the configuration, so there is no need to create specific self-signed certificates or keystores for the standby site.
Parent topic: Preparing the Standby Site
Validating the Standby Site Setup
Validate your standby site.
To validate a standby site:
- Shut down any processes still running on the production site. These include the database instances in the data tier, Oracle Fusion Middleware instances, and any other processes in the application tier and web tier.
- Stop the replication between the production site shared storage and the standby site shared storage.
- Perform a switchback operation. A switchback operation is a subsequent switchover operation to return the roles to their original state.
- On the standby site host, manually start all the processes. These include the database instances in the data tier, Oracle Fusion Middleware instances, and any other processes in the application tier and web tier.
- Use a browser client to perform post-failover testing to confirm that requests are being resolved and redirected to the standby site.
Parent topic: Creating a Standby Site
Performing Site Operations and Administration
Learn how to operate and administer your Oracle Fusion Middleware Disaster Recovery topology.
This section includes the following topics:
- Synchronizing the Production and Standby Sites
Learn how to force a synchronization of the production and standby sites when you introduce a change in the middle tier at the production site. - Performing a Switchover
A switchover operation sets the standby site as the production role. - Performing a Switchback
A switchback operation reverts the roles of the current production and standby sites. - Performing a Failover
A failover operation sets the standby site as the production role when the production site becomes unavailable. - Testing the Standby Site
Learn how to create a clone of the read-only standby site shared storage and use it for converting database secondary_db_unqname to snapshot standby. - Using Peer-To-Peer File Copy
Thersync
utility (which uses peer-to-peer file copy) can be used to replicate middle tier file system data from a production site host to a standby site peer host in an Oracle Fusion Middleware Disaster Recovery topology. The use of thersync
utility is explained in the context of symmetric topologies.
Parent topic: Setting Up and Managing Disaster Recovery Sites
Synchronizing the Production and Standby Sites
Learn how to force a synchronization of the production and standby sites when you introduce a change in the middle tier at the production site.
In normal operations, the standby site shared storage receives snapshots transferred periodically from the production site shared storage. After the snapshots are applied, the standby site shared storage includes all the data up to the data contained in the last snapshot transferred from the production site before the failover or switchover.
Be sure to force a synchronization when you introduce a change to the middle tier at the production site such as, for example, when you deploy a new application at the production site. Follow the vendor-specific instructions to force a synchronization by using the storage replication technology.
The databases synchronization in an Oracle Fusion Middleware Disaster Recovery topology is managed by Oracle Data Guard.
Parent topic: Performing Site Operations and Administration
Performing a Switchover
A switchover operation sets the standby site as the production role.
This operation is needed when you plan to take down the production site (for example, to perform maintenance) and to make the current standby site the production site.
To perform a switchover operation:
At this point, the former standby site becomes the new production site, and you can perform maintenance at the original production site. After you have carried out the maintenance of the original production site, you can use it as either the production site or the standby site.
Note:
This note is applicable for BI-specific systems only.
After a switchover operation, the creation of an Essbase cube with CDS may fail with an error similar to the following:
oracle.essbase.cds.util.CDSException: oracle.essbase.cds.util.CDSException: java.sql.SQLException: ORA-25153:
-
Identify the temporary tablespaces by using a select statement similar to the following (where
BIS17V1
is the Oracle Business Intelligence RCU prefix):select username,temporary_tablespace from dba_users where username like 'BIS17V1%'
Assume that the above command returns the following list of temporary tablespaces:
USERNAME.....TEMPORARY_TABLESPACE
BIS17V1_IAU_VIEWER.....BIS17V1_IAS_TEMP
BIS17V1_STB.....BIS17V1_IAS_TEMP
BIS17V1_IAU_APPEND.....BIS17V1_IAS_TEMP
BIS17V1_MDS.....BIS17V1_IAS_TEMP
BIS17V1_IAU.....BIS17V1_IAS_TEMP
BIS17V1_BIPLATFORM......BIS17V1_IAS_TEMP
BIS17V1_OPSS.....BIS17V1_IAS_TEMP
-
After the switchover, drop the tablespace
BIS17V1_IAS_TEMP
including contents and datafiles. -
Create the temporary tablespace
BIS17V1_IAS_TEMP
, as a tempfile, in the location (for example)/work/primy/oradata/stnby/BIS17V1_IAS_TEMP.dbf
, with size 250 m. -
Issue the following alter commands (here is where you use the list temporary tablespaces):
alter user BIS17V1_OPSS temporary tablespace BIS17V1_IAS_TEMP ;
alter user BIS17V1_BIPLATFORM temporary tablespace BIS17V1_IAS_TEMP ;
alter user BIS17V1_IAU temporary tablespace BIS17V1_IAS_TEMP ;
alter user BIS17V1_MDS temporary tablespace BIS17V1_IAS_TEMP ;
alter user BIS17V1_IAU_APPEND temporary tablespace BIS17V1_IAS_TEMP ;
alter user BIS17V1_STB temporary tablespace BIS17V1_IAS_TEMP ;
alter user BIS17V1_IAU_VIEWER temporary tablespace BIS17V1_IAS_TEMP ;
To use the original production site as the production site, perform a switchback as explained in Performing a Switchback.
Parent topic: Performing Site Operations and Administration
Performing a Switchback
A switchback operation reverts the roles of the current production and standby sites.
To perform a switchback operation:
Parent topic: Performing Site Operations and Administration
Performing a Failover
A failover operation sets the standby site as the production role when the production site becomes unavailable.
To perform a failover operation:
Note:
After a failover operation, the creation of an Essbase cube with CDS may fail with an error similar to the following:oracle.essbase.cds.util.CDSException: oracle.essbase.cds.util.CDSException: java.sql.SQLException: ORA-25153:
-
Identify the temporary tablespaces by using a select statement similar to the following, where
BIS17V1
is the Oracle Business Intelligence RCU prefix):select username,temporary_tablespace from dba_users where username like 'BIS17V1%'
Assume that the above command returns the following list of temporary tablespaces:
USERNAME.....TEMPORARY_TABLESPACE
BIS17V1_IAU_VIEWER.....BIS17V1_IAS_TEMP
BIS17V1_STB.....BIS17V1_IAS_TEMP
BIS17V1_IAU_APPEND.....BIS17V1_IAS_TEMP
BIS17V1_MDS.....BIS17V1_IAS_TEMP
BIS17V1_IAU.....BIS17V1_IAS_TEMP
BIS17V1_BIPLATFORM......BIS17V1_IAS_TEMP
BIS17V1_OPSS.....BIS17V1_IAS_TEMP
-
After the failover, drop the tablespace
BIS17V1_IAS_TEMP
including contents and datafiles. -
Create the temporary tablespace
BIS17V1_IAS_TEMP
, as a tempfile, in location. For example,/work/primy/oradata/stnby/BIS17V1_IAS_TEMP.dbf
with size 250 m. -
Issue the following alter commands (here is where you use the list temporary tablespaces):
alter user BIS17V1_OPSS temporary tablespace BIS17V1_IAS_TEMP ;
alter user BIS17V1_BIPLATFORM temporary tablespace BIS17V1_IAS_TEMP ;
alter user BIS17V1_IAU temporary tablespace BIS17V1_IAS_TEMP ;
alter user BIS17V1_MDS temporary tablespace BIS17V1_IAS_TEMP ;
alter user BIS17V1_IAU_APPEND temporary tablespace BIS17V1_IAS_TEMP ;
alter user BIS17V1_STB temporary tablespace BIS17V1_IAS_TEMP ;
alter user BIS17V1_IAU_VIEWER temporary tablespace BIS17V1_IAS_TEMP ;
To again use the original production site as the production site, perform a switchback as explained in Performing a Switchback.
Parent topic: Performing Site Operations and Administration
Testing the Standby Site
Learn how to create a clone of the read-only standby site shared storage and use it for converting database secondary_db_unqname to snapshot standby.
A typical Oracle Fusion Middleware Disaster Recovery configuration uses:
-
Storage replication to copy Oracle Fusion Middleware middle tier file systems and data from the production site shared storage to the standby site shared storage. During normal operation, the production site is active and the standby site is passive. When the production site is active, the standby site is passive and the standby site shared storage is in read-only mode; the only write operations made to the standby site shared storage are the storage replication operations from the production site shared storage to the standby site shared storage.
-
Oracle Data Guard to copy database data for the production site Oracle databases to the standby databases at standby site. By default, the production site databases are active and the standby databases at the standby site are passive. The standby databases at the standby site are in Managed Recovery mode while the standby site is in the standby role (is passive). When the production site is active, the only write operations made to the standby databases are the database synchronization operations performed by Oracle Data Guard.
-
The standby site as the production role when the production site becomes unavailable. If the current production site becomes unavailable unexpectedly, then a failover operation (described in Performing a Failover) is performed to enable the standby site to assume the production role. Or, if the current production site is taken down intentionally (for example, for planned maintenance), then a switchover operation (described in Performing a Switchover) is performed to enable the standby site to assume the production role.
The usual method of testing a standby site is to shut down the current production site and perform a switchover operation to enable the standby site to assume the production role.
However, some enterprises may want to perform periodic testing of their Disaster Recovery standby site without shutting down the current production site and a complete switchover operation. This is possible by converting the standby database to snapshot standby. This allows the standby servers to be started in the standby site and verify the secondary system. Any change performed in the standby site database while it is in snapshot standby mode will be discarded once it is converted to physical standby again, so primary data will not be affected by secondary site validations.
To use this testing method:
-
Use the cloning technology provided by the shared storage vendor to create a clone of the standby site's read-only volumes on the shared storage at the standby site. Ensure that the cloned standby site volumes are writable. If you want to test the standby site just once, then this can be a one-time clone operation. However, if you want to test the standby site regularly, you can set up periodic cloning of the standby site read-only volumes to the standby site's cloned read/write volumes.
-
Convert the standby database into snapshot standby by the following steps:
-
Use Data Guard broker in primary database host to convert the secondary database to snapshot standby.
Example:
[oracle@primarydbhost~]$dgmgrl sys/your_sys_password@primary_db_unqname DGMGRL> convert database “secondary_db_unqname” to snapshot standby
-
. Use
show configuration
command to verify that the conversion has been correctly performed.
-
-
On the standby site computers, modify the mount commands to point to the volumes on the standby site's cloned read/write shared storage by following these steps:
-
Unmount the read-only shared storage volumes.
-
Mount the cloned read/write volumes at the same mount point.
-
-
Before you test the standby site, modify the host name resolution method for the computers that are used to perform the testing to ensure that the host names point to the standby site computers and not the production site computers. For example, on a Linux computer, change the
/etc/hosts
file to point to the virtual IP of the load balancer for the standby site. -
Perform the standby site testing.
Note:
This operation must be done with caution in SOA environments: if there are pending messages or composites in the database when it is converted into snapshot, the standby site's SOA servers will process them when they start. Check that there are no pending actions in primary database when converting to snapshot standby, otherwise, remove records from runtime SOA tables in the standby database after it is converted to snapshot standby database and before starting the secondary site's SOA servers. See, Removing Records from the Runtime Tables Without Dropping the Tables.After you complete the standby site testing, follow these steps to begin using the original production site as the production site again:
-
Modify the mount commands on the standby site computers to point to the volumes on the standby site's read-only shared storage. In other words, reset the mount commands back to what they were before the testing was performed.
-
Unmount the cloned read/write shared storage volume.
-
Mount the read-only shared storage volumes.
At this point, the mount commands are reset to what they were before the standby site testing was performed.
-
-
Convert the standby database into snapshot standby by the following steps:
-
Use Data Guard broker in primary database host to convert the secondary database to physical standby again.
Example:
[oracle@primarydbhost~]$dgmgrl sys/your_sys_password@primary_db_unqname DGMGRL> convert database “secondary_db_unqname” to physical standby
-
Use
show configuration
command to verify that the conversion has been correctly performed.
-
-
Before you use the original production site again, modify the host name resolution method for the computers that are used to access the production site to ensure that the host names point to the production site computers and not the standby site computers. For example, on a Linux computer, change the
/etc/hosts
file to point to the virtual IP of the load balancer for the production site.
Parent topic: Performing Site Operations and Administration
Using Peer-To-Peer File Copy
The rsync
utility (which uses peer-to-peer file copy) can
be used to replicate middle tier file system data from a production site host to a
standby site peer host in an Oracle Fusion Middleware Disaster Recovery topology.
The use of the rsync
utility is explained in the context of
symmetric topologies.
For information about the conditions for using the rsync
in Disaster
Recovery environments, see the point rsync
in the
section Setting Up Storage Replication of this document.
Ensure that you are familiar with storage replication and Oracle Data Guard in an
Oracle Fusion Middleware Disaster Recovery topology, because there are many
similarities between using storage replication and using the
rsync
utility for disaster protection and disaster
recovery of your Oracle Fusion Middleware components.
Note the following important differences between using storage
replication technologies and using the rsync
utility to
replicate middle tier file systems:
-
When you use storage replication, you can roll changes back to the point in time when any previous snapshot was taken at the production site.
When you use the
rsync
utility, replicated production site data overwrites the standby site data, and you cannot roll back a replication. -
When you use storage replication, the volume that you set up for each host cluster in the shared storage systems ensures data consistency for that host cluster across the production site's shared storage system and the standby site's shared storage system.
When you use the
rsync
utility, data consistency is not guaranteed.
This section includes the following topic:
- Using rsync and Oracle Data Guard in Oracle Fusion Middleware Disaster Recovery Topologies
Learn how to use the UNIXrsync
utility and Oracle Data Guard in your Oracle Fusion Middleware Disaster Recovery topology.
Parent topic: Performing Site Operations and Administration
Using rsync and Oracle Data Guard in Oracle Fusion Middleware Disaster Recovery Topologies
Learn how to use the UNIX rsync
utility and Oracle Data Guard in your Oracle Fusion Middleware Disaster Recovery topology.
Note:
For information about how to set up Oracle Data Guard for Oracle database, see Database Considerations.
The following sections describe how to use the rsync
utility and Oracle Data Guard to protect and force synchronization between your production and standby sites in an Oracle Fusion Middleware Disaster Recovery topology:
- Using rsync for Oracle Fusion Middleware Middle Tier Components
Use the UNIXrsync
utility to protect and recover your Oracle Fusion Middleware middle tier components. - Performing Failover and Switchover Operations
Learn how to perform failover or switchover operations when you use thersync
utility.
Parent topic: Using Peer-To-Peer File Copy
Using rsync for Oracle Fusion Middleware Middle Tier Components
Use the UNIX rsync
utility to protect and recover your Oracle Fusion Middleware middle tier components.
To use the rsync
utility:
Performing Failover and Switchover Operations
Learn how to perform failover or switchover operations when you use the rsync
utility.
To perform a failover or switchover from the production site to the standby site when you use rsync
:
To use the original production site as the new production site, perform the preceding steps again but configure the rsync replications to go in the original direction (from the original production site to the original standby site).
Using Oracle Site Guard for Disaster Recovery
Oracle Site Guard is a disaster-recovery solution that enables administrators to automate complete site switchover or failover. It orchestrates the coordinated failover of Oracle Fusion Middleware, Oracle Fusion Applications, and Oracle Databases.
Oracle Site Guards offers the following benefits:
- Fully automate disaster recovery operations and launch them with a single click
- Minimizes disaster-recovery time
- Reduces human errors
- Flexible and customizable
- Eliminates the need for special skills
- Use a single pane of glass to manage disaster recovery
- Assure disaster recovery readiness using on-demand or scheduled disaster recovery drills
For more information about how to use Oracle Site Guard, see Oracle Site Guard Administrator's Guide.
Parent topic: Setting Up and Managing Disaster Recovery Sites
Patching an Oracle Fusion Middleware Disaster Recovery Site
Apply an Oracle Fusion Middleware patch set to upgrade the Oracle homes that participate in an Oracle Fusion Middleware Disaster Recovery site.
It is assumed that the Oracle Central Inventory for any Oracle Fusion Middleware instance that you are patching is located on the production site shared storage, so that the Oracle Central Inventory for the patched instance can be replicated to the standby site.
To apply an Oracle Fusion Middleware patch:
- Perform a backup of the production site to ensure that the starting state is secured.
- Apply the patch set to upgrade the production site instances.
- After you apply the patch set, manually force a synchronization of the production site shared storage and standby site shared storage. This replicates the production site's patched instance and Oracle Central Inventory in the standby site's shared storage.
- After you apply the patch set, use Oracle Data Guard to manually force a synchronization of the Oracle databases at the production site and standby sites. Because some Oracle Fusion Middleware patch sets make updates to repositories, this step ensures that any changes made to production site databases are synchronized to the standby site databases.
- The upgrade is now complete. Your Disaster Recovery topology is ready to resume processing.
Note:
Patches must be applied only at the production site for an Oracle Fusion Middleware 12c Disaster Recovery topology. If a patch is for an Oracle Fusion Middleware instance or for the Oracle Central Inventory, the patch is copied when the production site shared storage is replicated to the standby site shared storage. A synchronization operation should be performed when a patch is installed at the production site.When patching the database, check the specific patch’s documentation on how to apply the patch in a Data Guard topology.
Parent topic: Setting Up and Managing Disaster Recovery Sites
Accessing and Managing T3/T3s External Clients in DR Scenarios
This section discusses different approaches for accessing and managing external T3/T3s clients in the context of a Disaster Recovery scenario.
Note:
This section does not apply to the internal T3/T3s clients, which run in the same domain as the T3/T3s servers. When connecting to a cluster in the same domain, internal clients can utilize the T3/T3s cluster syntax, such ascluster:t3://cluster_name
. Only when the
clusters are in the same domain can you utilize this cluster syntax.
WebLogic uses the T3/T3s protocol for Remote Method Invocation (RMI). It is used by several WebLogic services, such as JMS, EJB, JTA, JNDI, JMX, WLST, and so on. The external T3/T3s clients, which are those that do not run in the same domain as the T3/T3s servers, can use different ways to connect to a WebLogic cluster. For more information, see Using WebLogic RMI with T3 Protocol.
External T3/T3s clients can connect directly to WebLogic servers' channels (default or custom) that listen to the T3/T3s requests. The connection URL configured in the client's provider property contains the list of all the servers and ports of the cluster.
For example:
t3://host1.example.com:9073,host2.example.com:9073
The external T3/T3s clients can access to T3/T3s services through TCP Load Balancer (LBR) too. In this scenario, the client’s provider URL points to the load balancer service and port, and the requests are load balanced to the WebLogic Server's T3/T3s ports. In the initial contact for the JNDI context retrieval, the external clients connect to a WebLogic Server through the load balancer and download the stubs. These stubs contain the connect information that is used for the subsequent requests.
In general, the WebLogic T3/T3s is TCP/IP-based, so it can support TCP load
balancing when services are homogeneous, such as JMS and EJB. For example, a JMS
front-end can be configured in a WebLogic cluster in which remote JMS clients can
connect to any cluster member. By contrast, a JTA subsystem cannot safely use TCP load
balancing in transactions that span across multiple WebLogic domains. The JTA
transaction coordinator must establish a direct RMI connection to the server instance
that acts as the sub coordinator of the transaction when that transaction is either
committed or rolled back. Due to this, the load balancer is normally used only for
the JNDI initialContext
creation. The WebLogic Server load
balancing system controls the future T3/T3s requests, which connect to the WebLogic
managed servers' addresses and ports (default or custom channels) indicated in the stubs
retrieved during the initial context retrieval.
This section also explains how to use the load balancer for all communications (both initial and subsequent requests), which is applicable if JTA is not used.
Note:
External clients can access to T3 services using T3 tunneling over HTTP. But, this approach is not discussed in this document. This approach creates an HTTP session for each RMI session and uses standard HTTP protocols to transfer the session ID back and forth between the client and the server. This introduces some overhead and is less frequently used.- Different Approaches to Access T3/T3s Services
A Disaster Recovery scenario presents specific aspects which affect the configuration of external T3/T3s clients. This topic explains different approaches that you can use when accessing T3/T3s services from external clients, as well as learn how to manage them in a disaster recovery scenario. - About T3/T3s Client’s DNS Cache
All the approaches to access T3/T3s services mostly requires an DNS update. Since DNS update is required, you must set the limit for DNS Cache TTL (Time To Live) in DNS server and client's specific cache.
Parent topic: Setting Up and Managing Disaster Recovery Sites
Different Approaches to Access T3/T3s Services
A Disaster Recovery scenario presents specific aspects which affect the configuration of external T3/T3s clients. This topic explains different approaches that you can use when accessing T3/T3s services from external clients, as well as learn how to manage them in a disaster recovery scenario.
Note:
These approaches apply to a DR scenario that complies with the MAA guidelines for Fusion Middleware and PaaS DR, so the secondary domain configuration is a mirror replica of the primary system.- Direct T3/T3s Using Default Channels
In this approach, the external T3/T3s client connects directly to the WebLogic managed servers’ default channels. These default channels listen in theListen Address
andListen Port
specified in the general configuration of each WebLogic managed server. - Direct T3/T3s Using Custom Channels
In this approach, the external T3/T3s client connects directly to custom channels defined in the WebLogic servers of the cluster. TheListen Address
, theExternal Listen Address
, and theExternal Port
are customizable values in the custom channels. These values can differ from the WebLogic server’s default listen values. - Using Load Balancer for Initial Lookup
In this scenario, the client’s provider URL point to the Load Balancer service. - Using Load Balancer for All Traffic
In this approach, not only the initial context lookup goes through the load balancer, but also the subsequent connections. There are no direct connections from the external T3/T3s client to the servers.
Direct T3/T3s Using Default Channels
In this approach, the external T3/T3s client connects directly to the
WebLogic managed servers’ default channels. These default channels listen in the
Listen Address
and Listen Port
specified in the
general configuration of each WebLogic managed server.
Figure 3-5 Direct T3/T3s Using Default Channels
![This image shows the external T3/T3s client connecting directly to default channels. This image shows the external T3/T3s client connecting directly to default channels.](img/t3dr_default_channels.png)
Configuration
-
The provider URL in the T3 client uses the list of the WebLogic servers’ default listen addresses and ports.
Example: In the following DR example, the listener addresses of the WebLogic servers are the primary hostnames:
t3://soahost1.example.com:8001,soahost2.example.com:8001
In the case of using the T3s protocol, the port must be set to the server’s
SSL Listen Port
.Example:t3s://soahost1.example.com:8002,soahost2.example.com:8002
-
The external client must resolve these hostnames with the primary host’s IPs. These IPs must be reachable from the client. It is possible to use Network Address Translating (NAT), as long as there is a NAT IP address for each server. In that case, the client will resolve each server name with the appropriate NAT IP for the server.
Example: Naming resolution at the external client side
This is achievable though
local /etc/hosts
or formal DNS server resolution.172.11.2.113 soahost1.example.com 172.11.2.114 soahost2.example.com
Switchover
In a switchover scenario, there is no need to update the client's provider
URL. Instead, you need to perform an update of the entries in the DNS (or
client's /etc/hosts)
of the client. After the switchover, the names used to
connect to the servers must resolve with the IPs of the secondary servers.
172.22.2.113 soahost1.example.com
172.22.2.114 soahost2.example.com
Advantages
-
You don't require additional configuration either at the server’s side or at the front-end tier. All you require is only opening the ports to the client.
Disadvantages
- When a switchover happens, you need to update the
DNS (or client's /etc/hosts)
to alter the resolution of all the hostnames that the external client uses to connect to the managed servers. For more information about implications of the client’s cache, see About T3/T3s Client’s DNS Cache. - Clients must be able to resolve and reach the hostnames set in
WebLogic's
Listen Address
. It is not possible to use alternate names, because the default channels use the WebLogic server’s defaultListen Address
. - The WebLogic default channels not only listen for T3(s) requests, but also for HTTP(s). You cannot disable this setting. If you open the default port for the clients, direct HTTP(s) to the server is also allowed, which can result in security concerns.
- You need to modify the client’s provider URL if you have to either add or remove the WebLogic server nodes from the WebLogic cluster. First contact only uses the list in the client’s provider URL. So, even without updating the list, the subsequent requests can connect to any server of the cluster as the recovered stubs list all of the members. But it is a good practice that the client’s provider URL matches the real list of the servers, for failover purposes in the first contact.
Parent topic: Different Approaches to Access T3/T3s Services
Direct T3/T3s Using Custom Channels
In this approach, the external T3/T3s client connects directly to custom
channels defined in the WebLogic servers of the cluster. The Listen
Address
, the External Listen Address
, and the External
Port
are customizable values in the custom channels. These values can differ
from the WebLogic server’s default listen values.
Figure 3-6 Direct T3/T3s Using Custom Channels
![This image shows the external T3/T3s client connecting directly to custom channels. This image shows the external T3/T3s client connecting directly to custom channels.](img/t3dr_default_channels.png)
Once the T3/T3s external client has connected to the server during the
initial context retrieval, the subsequent T3 calls connects directly to one of the
listen addresses and ports configured in the custom channels as External Listen
address
and External Port
. These requests will be load
balanced per the mechanism specified in the connection factory defined in the WebLogic
and used by the client.
This approach is similar to Direct T3/T3s Using Default Channels, with the exception that you can customize the addresses and ports used in T3/T3s calls when using WebLogic custom channels.
Configuration
-
Each WebLogic server has the appropriate custom channels configured and these custom channel uses a unique
External Listen address
that points to that server node. Table 3-4 and Table 3-5 provides example of Custom Channel in Server 1 and Server 2.Table 3-4 Custom Channel in Server 1
Name Protocol Enabled Listen Address Listen Port Public Address Public Port t3_external_channel
t3 true soahost1.example.com
8111 soahost1-t3ext.example.com
8111 Table 3-5 Custom Channel in Server 2
Name Protocol Enabled Listen Address Listen Port Public Address Public Port t3_external_channel
t3 true soahost2.example.com
8111 soahost2-t3ext.example.com
8111 -
The external T3 client’s provider URL contains the list of the external address and port of the custom channels.
Example:
t3://soahost1-t3ext.example.com:8111,soahost2-t3ext.example.com:8111
If you are using the T3s protocol, you must create the custom channels with T3s protocol. Then the clients will connect using T3s protocol and appropriate port.
Example:
t3s://soahost1-t3ext.example.com:8112,soahost2-t3ext.example.com:8112
In a T3s channel, you can add a specific SSL certificate for the name used as an
External Listen address
. -
The external T3/T3s client must resolve the custom channels’ external hostnames with the primary host’s IPs. These hostnames must be reachable from the client. It is possible to use NAT, as long as there is a NAT address for each server. In that case, the client will resolve each server name with the appropriate NAT IP for the server.
Example: Naming resolution at the external client side
172.11.2.113 soahost1-t3ext.example.com 172.11.2.114 soahost2-t3ext.example.com
With this approach, all the requests from the external client connect to
soahost1-t3ext.example.com
andsoahost2-t3ext.example.com
, both for the initial context retrieval and for the subsequent calls. You can control these requests by using the WebLogic Server load balancing mechanism.
Switchover
You don't have to update the client's provider URL if you have performed a
switchover. Instead, you must update the entries in the DNS (or in the
/etc/hosts)
of the client. After the switchover, the names used to connect
to the servers must resolve with the IPs of the secondary servers.
172.22.2.113 soahost1-t3ext.example.com
172.22.2.114 soahost2-t3ext.example.com
Advantages
-
This method allows using specific hostnames for the external T3/T3s communication, different from the server's default listen address. This is useful if you do not want to expose the default server's
Listen Address
to the external clients for security reason. -
This method is also useful if you are using NAT IPs and you do not want to resolve the servers' default listen addresses with different IPs internally and externally.
-
This method is also useful in case you want to use different names for external T3/T3s accesses just for organizational purposes. You can also use this method to isolate protocols in different interfaces or network routes.
-
The protocol in the custom channel can be limited to T3/T3s only. The HTTP(s) protocol can be disabled in the custom channels.
Disadvantages
- When a switchover happens, you need to update the
DNS (or client's /etc/hosts)
at the external client side for all the hostnames used to connect to the managed servers. For more information about the implications of the client’s cache, see About T3/T3s Client’s DNS Cache. - If you are using T3s, then you must create and configure specific SSL certificates for the external names in the custom channels to avoid SSL hostname verification errors in the client.
- You have to modify the client’s provider URL if you have added or removed the WebLogic server nodes from the WebLogic cluster. First contact only uses the list in the client’s provider URL. So, even without updating the list, the subsequent requests can connect to any server of the cluster as the recovered stubs list all of the members. But in any case, it is a good practice that the client’s provider URL matches the real list of the servers, for failover purposes in the first contact.
Parent topic: Different Approaches to Access T3/T3s Services
Using Load Balancer for Initial Lookup
In this scenario, the client’s provider URL point to the Load Balancer service.
Direct T3/T3s Using Default Channels and Direct T3/T3s Using Custom Channels approaches can also use a load balancer for the initial context lookup.
However, the subsequent T3/T3s calls connects to WebLogic servers directly. If you use the default channels then the requests goes to the default channel’s listen address and port. Similarly, if you use the custom channels then the subsequent request goes to the external listen address and port defined in the customs channels.
Figure 3-7 Using Load Balancer for Initial Lookup
![This image shows using load balancer for the initial lookup. This image shows using load balancer for the initial lookup.](img/t3dr_lbr_forinitial.png)
Configuration
-
You will need a TCP service in the load balancer to load balances the requests to the WebLogic servers' T3/T3s ports (either to the default channels or to the custom channels when used).
-
The external T3/T3s client’s provider uses the front-end name and port of the load balancer as the point of contact.
Example:
t3://t3lbr.example.com:8111
-
You can use default channels or custom channels as explained in Direct T3/T3s Using Default Channels and Direct T3/T3s Using Custom Channels scenarios. The external T3/T3s client must resolve the custom channels’ external hostnames (or the default server’s listeners hostnames if you are using default channels) with the primary host’s IPs. The client must be able to access them. It is possible to use NAT, as long as there is a NAT address for each server. In that case, the client will resolve each server name with the appropriate NAT IP for the server.
Example: Naming resolution at the external client side111.111.111.111 t3lbr.example.com 172.11.2.113 soahost1-t3ext.example.com 172.11.2.114 soahost2-t3ext.example.com
Switchover
In case of a complete switchover, you don't have to update the client's
provider URL. Instead, you must update the entries in the DNS (or in the
/etc/hosts)
used by the client so that they resolve with the IPs of the
secondary site. The name used to connect to the load balancer must resolve with the IP
of the secondary load balancer, and the server names used in subsequent requests must
point to the IPs of the secondary servers.
222.222.222.222 t3lbr.example.com
172.22.2.113 soahost1-t3ext.example.com
172.22.2.113 soahost2-t3ext.example.com
Advantages
You don't have to modify the client’s provider URL if you add or remove the WebLogic server nodes from the WebLogic cluster.
Disadvantages
-
Despite using a load balancer in front, the client still needs to reach the servers directly.
-
When a switchover happens, you need to update the
DNS (or /etc/hosts)
of servers’ addresses at the external client side. For more information about the implications of the client’s cache, see About T3/T3s Client’s DNS Cache. -
The complexity of this method is higher for the T3s cases. The client connects both through the front-end LBR and directly to the server using a secure protocol. In this case, you will need to either skip the hostname verification at the client side, or use an SSL certificate that is valid for front and back addresses (e.g. wildcard or SAN certificates).
Parent topic: Different Approaches to Access T3/T3s Services
Using Load Balancer for All Traffic
In this approach, not only the initial context lookup goes through the load balancer, but also the subsequent connections. There are no direct connections from the external T3/T3s client to the servers.
Oracle does not recommend using LBR for load balancing all types of T3/T3s communications. It is only recommended for initial context lookup. For more information, see, WebLogic RMI Integration with Load Balancers. However, there are T3/T3s use cases, where you can use Load Balancer for complete T3/T3s communication flow. WebLogic T3/T3s is TCP/IP-based protocols, so it can support TCP load balancing when services are homogeneous, such as JMS and EJB. For example, you can configure an LBR front-ended JMS subsystem in a WebLogic cluster in which remote JMS clients can connect to any cluster member.
This approach, however, will not work with external clients that use JTA connections. A JTA subsystem cannot safely use TCP load balancing in transactions that span across multiple WebLogic domains. When the transaction is either committed or rolled back, the JTA transaction coordinator must establish a direct RMI connection to the server instance that has been chosen as the transaction's sub coordinator.
This method is not suitable also for cases where you require direct connection to an specific server, like JMX or WLST when you want to connect to a particular server only.
Figure 3-8 Using Load Balancer for All Traffic
![This image shows using load balancer for all traffic. This image shows using load balancer for all traffic.](img/t3dr_lbr_forall.png)
Configuration
-
Load balancer requires a TCP service to load balance the requests to the WebLogic servers’ T3/T3s ports defined in the custom channels.
-
The external T3/T3s client’s provider URL uses the front-end name and port of the load balancer as the point of contact.
Example:t3://t3lbr.example.com:8111
-
The external client must resolve the load balancer address with the IP of the primary site’s load balancer.
Example:111.111.111.111 t3lbr.example.com
-
WebLogic Server requires custom channels. Configure the external listen address and port of these custom channels with the Load Balancer's address and port. Table 3-6 and Table 3-7 provides example of Custom Channel in Server 1 and Server 2.
Table 3-6 Custom Channel in Server 1
Name Protocol Enabled Listen Address Listen Port Public Address Public Port t3_external_channel
t3 true soahost1.example.com
8111 t3blr.example.com
8111 Table 3-7 Custom Channel in Server 2
Name Protocol Enabled Listen Address Listen Port Public Address Public Port t3_external_channel
t3 true soahost2.example.com
8111 t3lbr.example.com
8111
Switchover
In case of a switchover, you don't have to update the client's provider URL.
Instead, you must update the entries in the DNS (or in the /etc/hosts)
of the client. After the switchover, the names used to connect to the servers resolves
with the IP of the secondary LBR service.
222.222.222.222 t3lbr.example.com
Advantages
-
All the communication goes through the load balancer. The client only needs to know and reach the load balancer’s service.
-
You don't have to modify the client’s provider URL if you have to either add or remove the WebLogic server nodes from the WebLogic cluster.
-
In case of a switchover, you only have to update the load balancer’s front-end name in the
DNS (or in the /etc/hosts)
.Note:
Although only one DNS name is updated, you need to refresh the client’s DNS cache. For more information, see About T3/T3s Client’s DNS Cache. -
In T3s cases, you can use the same SSL certificate in all the custom channels, associated to the load balancer service’s front-end name.
Disadvantages
-
This approach is not suitable for all the T3/T3s cases. Scenarios with JTA cannot use this approach, because JTA needs to explicitly connect to the server instance that acts as the sub coordinator of the transaction.
Parent topic: Different Approaches to Access T3/T3s Services
About T3/T3s Client’s DNS Cache
All the approaches to access T3/T3s services mostly requires an DNS update. Since DNS update is required, you must set the limit for DNS Cache TTL (Time To Live) in DNS server and client's specific cache.
The TTL limit in the DNS service is a setting that tells the DNS resolver how long to cache a query before requesting a new one. Note that the TTL value of the DNS entries will affect the effective RTO of the switchover; if the TTL is high (for example, 20 min), the DNS change will take that time to be effective in the clients. Using lower TTL values will make this switchover faster, however, this can cause overhead because the clients check the DNS more frequently. A good approach is to set the TTL to a low value temporarily (for example, one min), before the change in the DNS. Then, perform the change, and once the switchover procedure is completed, set the TTL to the normal value again.
Besides the DNS server's TTL, networkaddress.cache.ttl
Java
property controls the Java clients' cache TTL. This Java property indicates the caching
policy for successful name lookups from the name service. Specify the value as an
integer to indicate the number of seconds to cache the successful lookup. Ensure you set
a limit to the networkaddress.cache.ttl
Java property so the client's
Java cache does not cache the DNS entries forever. Else, with each switchover, you might
have to restart the client.