4 Setting Up and Managing Disaster Recovery Sites

This chapter uses the Oracle SOA Suite enterprise deployment topology to illustrate the steps required to set up the production site and standby site.

This chapter includes the following sections:

Section 4.1, "Setting Up a Site"
Section 4.2, "Creating a Production Site"
Section 4.3, "Creating a Standby Site"
Section 4.4, "Creating an Asymmetric Standby Site"
Section 4.5, "Performing Site Operations and Administration"
Section 4.6, "Using Oracle Site Guard for Disaster Recovery"
Section 4.7, "Patching an Oracle Fusion Middleware Disaster Recovery Site"

Note:

You can automate disaster recovery operations like switchover and failover using Oracle Site Guard. For more information, see Oracle Site Guard Administrator's Guide.

4.1 Setting Up a Site

This section provides steps to set up a site.

It includes the following topics:

Directory Structure and Volume Design
Storage Replication
Database

Before you start creating the production site, ensure that you perform the following steps:

Set up the host name aliases for the middle tier hosts, as described in Section 3.1.1.
Create the required volumes on the shared storage on the production site as described in Section 4.1.1
Determine the Oracle Data Guard configuration to use based on the data loss requirements of the database and network considerations such as the available bandwidth and latency when compared to the redo generation.

See Oracle Data Guard Concepts and Administration as well as related Maximum Availability Architecture collateral at the following URL for more information:
```
http://www.oracle.com/technology/deploy/availability/htdocs/maa.htm
```

4.1.1 Directory Structure and Volume Design

The following section details the directory structure recommended by Oracle. The end user is free to choose other directory layouts, but the model adopted here enables maximum availability, providing the best isolation of components and symmetry in the configuration, and facilitating backup and disaster recovery.

This list describes directories and directory environment variables:

ORACLE_BASE: This environment variable and related directory path refers to the base directory below which Oracle products are installed.
Oracle_Home: This related directory path refers to the location where Oracle Fusion Middleware resides.
WL_HOME: This environment variable and related directory path contains installed files necessary to host an Oracle WebLogic Server.
ORACLE_HOME: This environment variable and related directory path refers to the location where a product suite (such as Oracle SOA Suite, Oracle WebCenter Portal, or Oracle Identity Management) is installed.
DOMAIN directory: This directory path refers to the location where the Oracle WebLogic Domain information (configuration artifacts) is stored. Different WebLogic Servers can use different domain directories even when in the same node.
ORACLE_INSTANCE: An Oracle instance contains one or more system components, such as Oracle Web Cache, Oracle HTTP Server, or Oracle Internet Directory. An Oracle instance directory contains updatable files, such as configuration files, log files, and temporary files.

For more information, see "Directory Structure Recommendations for Oracle SOA Suite".

4.1.1.1 Directory Structure Recommendations for Oracle SOA Suite

Oracle Fusion Middleware 12c allows the creation of multiple SOA Managed Servers from a single binary installation. This allows the installation of binary files in a single location on a shared storage and the reuse of this installation by the servers in different nodes. However, for maximum availability, Oracle recommends using redundant binary installations. In this model, two Oracle homes (each of which has a WL_HOME and an ORACLE_HOME for each product suite) are installed in a shared storage. Additional servers (when scaling out or up) of the same type can use either one of these two locations without requiring more installations. Ideally, users should use two different volumes for redundant binary location, this isolating the failures as much as possible in each volume. For additional protection, Oracle recommends using storage replication for these volumes. If multiple volumes are not available, Oracle recommends using mount points to simulate the same mount location in a different directory in the shared storage. Although this does not guarantee the protection that multiple volumes provide, it does allow protection from user deletions and individual file corruption.

Oracle also recommends separating the domain directory used by the Administration Server from the domain directory used by Managed Servers. This allows a symmetric configuration for the domain directories used by Managed Servers, and isolates the failover of the Administration Server. The domain directory for the Administration Server must reside in a shared storage to allow failover to another node with the same configuration. In addition, Oracle recommends placing the Managed Servers' domain directories on a shared storage, although having them on the local file system is also supported. This is especially important when designing a production site with the disaster recovery site in mind. Figure 4-1 represents the directory structure layout for Oracle SOA Suite.

Figure 4-1 Directory Structure for SOA

Description of "Figure 4-1 Directory Structure for SOA"

For information about setting up this directory structure, see Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.

4.1.1.1.1 Volume Design for Oracle SOA Suite

Figure 4-2 and Figure 4-3 shows an Oracle SOA Suite topology diagram. The volume design described in this section is for this Oracle SOA Suite topology. Detailed instructions for installing and configuring this topology are provided in the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.

Figure 4-2 Oracle SOA Suite and Oracle Business Activity Monitoring Enterprise Topology Diagram

Description of "Figure 4-2 Oracle SOA Suite and Oracle Business Activity Monitoring Enterprise Topology Diagram"

Figure 4-3 Oracle SOA Suite and Oracle Service Bus Enterprise Deployment Reference Topology Diagram

Description of "Figure 4-3 Oracle SOA Suite and Oracle Service Bus Enterprise Deployment Reference Topology Diagram"

For disaster recovery of this Oracle SOA Suite topology, Oracle recommends the following volume design:

Provision two volumes for two Oracle homes that contain redundant product binary files (VOLFMW1 and VOLFMW2 in Table 4-1).
Provision one volume for the Administration Server domain directory (VOLADMIN in Table 4-1).
Provision one volume on each node for the Managed Server domain directory (VOLSOA1 and VOLSOA2 in Table 4-1). This directory is shared among all the Managed Servers on that node.
Provision one volume for the JMS file store and JTA transaction logs (VOLDATA in Table 4-1). One volume for the entire domain is mounted on all the nodes in the domain.
Provision one volume on each node for the Oracle HTTP Server Oracle home (VOLWEB1 and VOLWEB2 in Table 4-1).
Provision one volume on each node for the Oracle HTTP Server Oracle instance (VOLWEBINST1 and VOLWEBINST2 in Table 4-1).

Table 4-1 provides a summary of Oracle recommendations for volume design for the Oracle SOA Suite topology shown in Figure 4-2 and Figure 4-3.

Table 4-1 Volume Design Recommendations for Oracle SOA Suite

Tier	Volume Name	Mounted on Host	Mount Point	Comments
Web	`VOLWEB1`	`WEBHOST1`	`/u01/oracle/products/fmwnnnn/web`	Volume for Oracle HTTP Server installation
Web	`VOLWEB2`	`WEBHOST2`	`/u01/oracle/products/fmwnnnn`/web	Volume for Oracle HTTP Server installation
Web	`VOLWEBINST1`	`WEBHOST1`	`/u01/oracle/admin/ohs_instance`	Volume for Oracle HTTP Server instance
Web	`VOLWEBINST2`	`WEBHOST2`	`/u01/oracle/admin/ohs_instance`	Volume for Oracle HTTP Server instance
Web	`VOLSTATIC1`^Foot 1	`WEBHOST1`	`/u01/oracle/admin/ohs_instance/config/static`	Volume for static HTML content
Web	`VOLSTATIC2`^Foot 2	`WEBHOST2`	`/u01/oracle/admin/ohs_instance/config/static`	Volume for static HTML content
Application	`VOLFMW1`	`SOAHOST1`	`/u01/oracle/products/fmwnnnn/`	Volume for the WebLogic Server and Oracle SOA Suite binary files
Application	`VOLFMW2`	`SOAHOST2`	`/u01/oracle/products/fmwnnnn/`	Volume for the WebLogic Server and Oracle SOA Suite binary files
Application	`VOLADMIN`	`SOAHOST1`	`/u01/oracle/admin/soaDomain/admin`	Volume for Administration Server domain directory
Application	`VOLSOA1`	`SOAHOST1`	`/u01/oracle/admin/soaDomain/mng1`	Volume for Managed Server domain directory
Application	`VOLSOA2`	`SOAHOST2`	`/u01/oracle/admin/soaDomain/mng2`	Volume for Managed Server domain directory
Application	`VOLDATA`	`SOAHOST1`, `SOAHOST2`	`/u01/oracle/config/soaDomain/soaCluster/jms` `/u01/oracle/config/soaDomain/soaCluster/tlogs`	Volume for transaction logs and JMS data

^Footnote 1This volume for static HTML data is optional. Oracle Fusion Middleware will operate normally without it.

^Footnote 2This volume for static HTML data is optional. Oracle Fusion Middleware will operate normally without it.

4.1.1.1.2 Consistency Group Recommendations for Oracle SOA Suite

Oracle recommends the following consistency groups for the Oracle SOA Suite topology:

Create one consistency group with the volumes containing the domain directories for the Administration Server and Managed Servers as members (DOMAINGROUP in Table 4-2).
Create one consistency group with the volume containing the JMS file store and transaction log data as members (DATAGROUP in Table 4-2).
Create one consistency group with the volume containing the Oracle homes as members (FMWHOMEGROUP in Table 4-2).
Create one consistency group with the volumes containing the Oracle HTTP Server Oracle homes as members (WEBHOMEGROUP in Table 4-2).
Create one consistency group with the volumes containing the Oracle HTTP Server Oracle instances as members (WEBINSTANCEGROUP in Table 4-2).

Table 4-2 provides a summary of Oracle recommendations for consistency groups for the Oracle SOA Suite topology shown in Figure 4-2.

Table 4-2 Consistency Groups for Oracle SOA Suite

Tier	Group Name	Members	Comments
Application	`DOMAINGROUP`	`VOLADMIN` `VOLSOA1` `VOLSOA2`	Consistency group for the Administration Server, Managed Server domain directory
Application	`DATAGROUP`	`VOLDATA`	Consistency group for the JMS file store and transaction log data
Application	`FMWHOMEGROUP`	`VOLFMW1` `VOLFMW2`	Consistency group for the Oracle homes
Web	`WEBHOMEGROUP`	`VOLWEB1` `VOLWEB2`	Consistency group for the Oracle HTTP Server Oracle homes
Web	`WEBINSTANCEGROUP`	`VOLWEBINST1` `VOLWEBINST2` `VOLSTATIC1`^Foot 1 `VOLSTATIC2`^Foot 2	Consistency group for the Oracle HTTP Server Oracle instances

^Footnote 1This volume for static HTML data is optional. Oracle Fusion Middleware will operate normally without it.

^Footnote 2This volume for static HTML data is optional. Oracle Fusion Middleware will operate normally without it.

4.1.2 Storage Replication

Follow these steps to set up storage replication for the Oracle Fusion Middleware Disaster Recovery topology:

On the standby site, ensure that alias host names that are created are the same as the physical host names used for the peer hosts at the production site.
On the shared storage at the standby site, create the same volumes as were created on the shared storage at the production site.
On the standby site, create the same mount points and symbolic links that you created at the production site (note that symbolic links only need to be set up on the standby site if you set up symbolic links at the production site). Note that symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes. For more information about symbolic links, see section Section 3.2.3.
It is not necessary to install the same Oracle Fusion Middleware instances at the standby site as were installed at the production site. When the production site storage is replicated to the standby site storage, the Oracle software installed on the production site volumes will be replicated at the standby site volumes.
Create the baseline snapshot copy of the production site shared storage that sets up the replication between the production site and standby site shared storage. Create the initial baseline copy and subsequent snapshot copies using asynchronous replication mode. After the baseline snapshot copy is performed, validate that all the directories inside the standby site volumes have the same contents as the directories inside the production site volumes.
Set up the frequency of subsequent copies of the production site shared storage, which will be replicated at the standby site. When asynchronous replication mode is used, then at the requested frequency the changed data blocks at the production site shared storage (based on comparison to the previous snapshot copy) become the new snapshot copy, and the snapshot copy is transferred to the standby site shared storage.
Ensure that disaster protection for any database that is included in the Oracle Fusion Middleware Disaster Recovery production site is provided by Oracle Data Guard. Do not use storage replication technology to provide disaster protection for Oracle databases.
The standby site shared storage receives snapshots transferred on periodically from the production site shared storage. After the snapshots are applied, the standby site shared storage includes all the data up to and including the data contained in the last snapshot transferred from the production site before the failover or switchover.
Oracle strongly recommends that you manually force a synchronization operation whenever a change is made to the middle tier at the production site (for example, when a new application is deployed at the production site). Follow the vendor-specific instructions for forcing a synchronization using storage replication technology.

4.1.3 Database

This section describes how to install and configure Oracle Database 11.2 or 12.1 MAA environments in an Oracle SOA Suite enterprise deployment.

For recommendations and considerations for setting up Oracle databases that are used in an Oracle Fusion Middleware Disaster Recovery topology, see Section 3.3.

4.1.3.1 Installing and Configuring Oracle Database 11.2 or 12.1 MAA Environments

Oracle Maximum Availability Architecture (MAA) is Oracle's comprehensive architecture for reducing downtime for scheduled outages, and preventing, detecting and recovering from unscheduled outages.

Real Application Clusters (RAC) and Data Guard provide the basis of the database MAA solution, where the primary site contains the RAC database, and the secondary site contains the RAC physical standby database.

This section contains the following topics:

Prerequisites and Assumptions
Oracle Data Guard Environment Description
Procedure for Duplicating the Primary Database
Procedure for Completing RAC Configuration for Standby Database
Creating a Data Guard Broker Configuration
Verifying the Data Guard Broker Configuration
Testing Database Switchover and Switchback

Tip:

Alternatively, you can perform many of the tasks in this section using Oracle Enterprise Manager Cloud Control (Cloud Control).

Setting up and managing databases using Cloud Control helps in controlling downtime and simplifies disaster recovery operations.

For information about installing Enterprise Manager Cloud Control 12c, see Oracle Enterprise Manager Cloud Control Basic Installation Guide.

For more information about setting up Oracle Data Guard using Cloud Control, see Set Up and Manage Oracle Data Guard using Oracle Enterprise Manager Cloud Control 12c.

4.1.3.1.1 Prerequisites and Assumptions

Ensure that the following prerequisites are met:

The Oracle RAC cluster and Automatic Storage Management (ASM) instances on the standby site have been created.
The Oracle RAC databases on the standby site and the production site are using a flash recovery area.
The Oracle RAC databases are running in archivelog mode.
The database hosts on the standby site already have Oracle software installed.
In a shared ORACLE_HOME configuration, the TNS_ADMIN directory must be a local, non-shared directory.

4.1.3.1.2 Oracle Data Guard Environment Description

The examples given in this section contain environment variables as described in Table 4-3.

Table 4-3 Variables Used by Primary and Standby Databases

Variable	Primary Database	Standby Database
Database names	`soa`	`soa`
SOA Database Host Names	`soadc1.dbhost1,` `soadc1.dbhost2`	`soadc2.dbhost1`, `soadc2.dbhost2`
Database unique names	`psoa`	`ssoa`
Instance names	`soa1`, `soa2`	`soa1`, `soa2`
Service names	`psoa`, `ssoa`	`psoa`, `ssoa`

4.1.3.1.3 Procedure for Duplicating the Primary Database

Follow these steps to prepare the primary database for setting up Oracle Data Guard:

Note:

For information about prerequisites for setting up Oracle Data Guard, see "Prerequisites" in Oracle Data Guard Broker.

Enable force logging on the primary database:

SQL> alter database force logging;
SQL> select name, log_mode, force_logging from v$database;
NAME LOG_MODE FOR
---------- ---------------- -------
PSOA ARCHIVELOG YES

In the standby node 1, create and start a listener (for example, LISTENER_DUPLICATE) that offers a static SID entry for the standby database, and has the same value of ORACLE_SID as the primary (soa1), as shown in Example 4-1.

Provide the following path to the listener.ora file:
```
GRID_HOME/network/admin/listener.ora
```
Example 4-1 Script for Creating and Starting a Listener with Static SID
```
LISTENER_duplicate =
 (DESCRIPTION_LIST =
  (DESCRIPTION =
   (ADDRESS = (PROTOCOL = TCP)
   (HOST = soadc2.dbhost1)
   (PORT = 1521)(IP = FIRST))))
 
SID_LIST_LISTENER_duplicate =
 (SID_LIST =
  (SID_DESC =
   (SID_NAME = soa1)
   (ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1)))
```

Run the following command to start and verify the listeners:

lsnrctl start listener_duplicate
lsnrctl status listener_duplicate

In the standby node 2, create and start a listener (for example, LISTENER_DUPLICATE) that offers a static SID entry for the standby database, and has the same value of ORACLE_SID as the primary (soa2), as shown in Example 4-2.

Provide the following path to the listener.ora file:
```
GRID_HOME/network/admin/listener.ora
```
Example 4-2 Script for Creating and Starting a Listener with Static SID
```
LISTENER_duplicate =
 (DESCRIPTION_LIST =
  (DESCRIPTION =
   (ADDRESS = (PROTOCOL = TCP)
   (HOST = soadc2.dbhost2)
   (PORT = 1521)(IP = FIRST))))

SID_LIST_LISTENER_duplicate =
 (SID_LIST =
  (SID_DESC =
   (SID_NAME = soa2)
   (ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1)))
```
Note:

Listeners are configured at the cluster level, and all nodes inherit the port and environment settings of the listener. Therefore, the TNS_ADMIN directory path have the same values on all nodes.

In a shared ORACLE_HOME configuration, the TNS_ADMIN directory must be a local, non-shared directory. These network files are included as IFILES.

Complete the following steps to set up TNS_ADMIN for a shared ORACLE_HOME in a two-node cluster, SOADC1.DBHOST1 and SOADC1.DBHOST2, with respective instances SOA1 and SOA2:
1. Create a local network directory on each node. For example, /local_network_dir/network_admin.
2. Create a local listener.ora file in the location: /local_network_dir/network_admin on each node.
3. In the local listener.ora file, add the values for the LISTENER_duplicate parameter.
4. In the common listener.ora file, in GRID_HOME/network/admin, add the IFILE parameter values, as follows:
```
IFILE=/local_network_dir/network_admin/listener.ora
```

Run the following command to start and verify the listeners:

lsnrctl start listener_duplicate
lsnrctl status listener_duplicate

In the database home of the primary node, create an Oracle Net alias to connect to the listener that you created in step 2. See Example 4-3.
Example 4-3 Script for Creating an Oracle Net Alias
```
dup =
(DESCRIPTION =
 (ADDRESS =
  (PROTOCOL = TCP)
   (HOST = soadc2.dbhost1)
   (PORT = 1521))
 (CONNECT_DATA =
  (SERVER = DEDICATED)
   (SID = soa1)))
```
Configure Oracle password file authentication for redo transport. Make sure it meets the following requirements:
- Set remote_login_passwordfile to EXCLUSIVE.
```
SQL> show parameter 
remote_login_passwordfile
NAME TYPE VALUE
--------------------------------- -------------- --------------------------------
remote_login_passwordfile string EXCLUSIVE
```
- Ensure that primary and standby databases have the byte-identical password files.
In the ORACLE_HOME/dbs directory of the standby host, create a pfile, initsoa1.ora, with the following parameters:
```
db_name=soa
db_unique_name=ssoa
sga_target=5g
```
Create the audit directory for the soa database on all standby hosts:
```
mkdir -p /u01/app/oracle/admin/soa/adump
```

Create an Oracle Net alias on all primary hosts to reach the ssoa database on the standby hosts.

Ensure that all hosts have Oracle Net alias for psoa and ssoa. All the aliases that you create need to reference the scan listener and not the node VIP. Also, if the local_listener variable is set to an alias on the primary host, then enter the details of the variable on the standby site, that point to the local listener on the primary host. See Example 4-4 and Example 4-5.

Example 4-4 Sample tnsnames.ora File on Primary Node 1 (SOADC1.DBHOST1)

psoa =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL= TCP)
(HOST=prmy-scan)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = psoa)
    )
  )
 
ssoa =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL = TCP)
(HOST=stby-scan)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = ssoa)
    )
  )
 
psoa_local_listener = 
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL = TCP)
(HOST=prmy1-vip)(PORT = 1521))
    )
  )

Note:

For primary node 2 (SOADC1.DBHOST2), update the HOST value in psoa_local_listener to point to the VIP of primary node 2, as follows:
```
psoa_local_listener = 
    (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL = TCP)
(HOST=prmy2-vip)(PORT = 1521))
    )
  )
```

If you are using a shared Oracle home, add the VIPs of the two nodes to the local_listener parameter.

For example:

psoa_local_listener =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL = TCP)
(HOST=prmy1-vip)(PORT = 1521))
    (ADDRESS=(PROTOCOL = TCP)
(HOST=prmy2-vip)(PORT = 1521))
    )
  )

Example 4-5 Sample tnsnames.ora File on Standby Node 1 (SOADC2.DBHOST1)

psoa =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL= TCP)(HOST=prmy-scan)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = psoa)
    )
  )
 
ssoa =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL = TCP)
(HOST=stby-scan)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = ssoa)
    )
  )
 
ssoa_local_listener = 
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL = TCP)
(HOST=stby1-vip)(PORT = 1521))
    )
  )

Note:

For standby node 2 (SOADC2.DBHOST2), update the HOST value in ssoa_local_listener to point to the VIP of standby node 2, as follows:
```
ssoa_local_listener = (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL = TCP)
(HOST=stby2-vip)(PORT = 1521))
    )
  )
```

If you are using a shared Oracle home, add the VIPs of the two nodes to the local_listener parameter.

For example:

ssoa_local_listener =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL = TCP)
(HOST=stby1-vip)(PORT = 1521))
    (ADDRESS=(PROTOCOL = TCP)
(HOST=stby2-vip)(PORT = 1521))
    )
  )

On the standby host, set the ORACLE_SID to the same value as that of the primary database (ORACLE_SID=soa1), and run the startup nomount command on the standby database with standby PFILE.

Disable the parameter cluster_interconnects on the primary host if it is set:

SQL> alter system reset cluster_interconnects scope=spfile sid='soa1';
SQL> alter system reset cluster_interconnects scope=spfile sid='soa2';

On the primary host, run the Recovery Manager (RMAN) script to duplicate the primary database using the command duplicate target database for standby from active database.

Note:

This command varies depending on your environment. For information about how to use the command, see "Duplicating a Database" in Oracle Database Backup and Recovery User's Guide.

See Example 4-6 to understand how to duplicate between two systems with different disk-group names.

Example 4-6 Duplicating Data Between Two Systems with Different Disk-Group Names

rman <<EOF
connect target sys/password;
connect auxiliary sys/password@dup;
run { 
allocate channel prmy1 type disk; 
allocate channel prmy2 type disk; 
allocate channel prmy3 type disk; 
allocate channel prmy4 type disk; 
allocate auxiliary channel stby type disk; 
duplicate target database for standby from active database 
spfile 
parameter_value_convert '+DATA_prmy','+DATA_stby','+RECO_prmy','+RECO_stby' 
set db_file_name_convert '+DATA_prmy','+DATA_stby' 
set db_unique_name='ssoa' 
set db_create_online_log_dest_1='+DATA_stby' 
set db_create_file_dest='+DATA_stby' 
set db_recovery_file_dest='+RECO_stby' 
set log_file_name_convert '+DATA_prmy','+DATA_stby','+RECO_prmy','+RECO_stby' 
set control_files='+DATA_stby/ssoa/standby.ctl' 
set local_listener='ssoa_local_listener'
set remote_listener='stby-scan:1521'; 
} EOF

See Example 4-7 to understand how to duplicate between two systems that have the same disk-group name +DATA.

Example 4-7 Duplicating Data Between Two Systems with Same Disk-Group Name +DATA

rman <<EOF
connect target sys/password;
connect auxiliary sys/password@dup;
run {
allocate channel prmy1 type disk;
allocate channel prmy2 type disk;
allocate channel prmy3 type disk;
allocate channel prmy4 type disk;
allocate auxiliary channel stby type disk;
 
duplicate target database for standby from active database
spfile
set db_unique_name='ssoa'
set control_files='+DATA/ssoa/standby.ctl'
set local_listener='ssoa_local_listener'
set remote_listener='stby-scan:1521';
}
EOF

If you have disabled the parameter cluster_interconnects on the primary host as described in step 12, then you must set it back to the original values in the spfile.
(Optional) Stop and remove the listener that you created in step 2.

4.1.3.1.4 Procedure for Completing RAC Configuration for Standby Database

To complete the RAC configuration on the standby database, complete the steps given in Section 4.1.3.1.3. Then, perform the following steps:

Create a temporary parameter file:

SQL> create pfile='/tmp/p.ora' from spfile;

Create an SPFILE in +DATA_stby for the standby database:

SQL> create spfile='+DATA_stby/ssoa/spfilessoa.ora' from pfile='/tmp/p.ora';

Create an initsoan.ora file on all the standby hosts. The file needs to point to the location of the SPFILE created in step 2.
On the standby system, restart the instances in mount state:
```
startup mount
```

srvctl add database -d ssoa –o /u01/app/oracle/product/11.2.0/db_1
srvctl add instance -d ssoa -i soa1 -n soadc2.dbhost1
srvctl add instance -d ssoa -i soa2 -n soadc2.dbhost2
srvctl modify database –d ssoa –r physical_standby

Create standby redo logs that are the same size as the online redo logs, on the primary database and standby database.

Oracle recommends having the same number plus one additional redo logs for each thread as shown in Example 4-8.
Example 4-8 Sample Redo Log
```
SQL> alter database add standby logfile thread 1
group 9 size 500M,
group 10 size 500M,
group 11 size 500M;
 
SQL> alter database add standby logfile thread 2
group 12 size 500M,
group 13 size 500M,
group 14 size 500M;
```

4.1.3.1.5 Creating a Data Guard Broker Configuration

This section describes the basic steps for creating a Data Guard configuration.

For complete information about Data Guard Broker, see the Oracle Data Guard Broker guide.

To create a Data Guard Broker configuration, complete the following steps:

Add the values of static SID to the local node file, listener.ora, that is located in the grid infrastructure home on all hosts in the configuration:

LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2))))
LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3))))
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))) 

LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON 
SID_LIST_LISTENER = 
(SID_LIST = 
(SID_DESC = 
(GLOBAL_DBNAME = psoa_DGMGRL) 
(SID_NAME = soa1)
(ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1)))
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON

Note:

Ensure that you enter the appropriate values for parameters GLOBAL_DBNAME and SID_NAME, for each host.
If IFILE is used for shared ORACLE_HOME, modify the corresponding local listener.ora file specified in IFILE.

Bounce the listeners on the primary site and standby site to apply the modifications made to the files:
```
srvctl stop listener
srvctl start listener
```

Configure the Data Guard Broker metadata files, and enable the broker on the primary site and standby site as follows:

On the primary site

SQL> alter system set dg_broker_config_file1='+DATA_prmy/psoa/dr1.dat' scope=both;
SQL> alter system set dg_broker_config_file2='+DATA_prmy/psoa/dr2.dat' scope=both;
SQL> alter system set dg_broker_start=true scope=both;

On the standby site

SQL> alter system set dg_broker_config_file1='+DATA_stby/ssoa/dr1.dat' scope=both;
SQL> alter system set dg_broker_config_file2='+DATA_stby/ssoa/dr2.dat' scope=both;
SQL> alter system set dg_broker_start=true scope=both;

Access dgmgrl on the primary host, and create the configuration. See Example 4-9.

Example 4-9 Accessing dgmgrl to Create the Data Guard Broker Configuration on Primary Host

dgmgrl sys/password
 
DGMGRL> create configuration 'dg_config' as 
primary database is 'psoa'
connect identifier is psoa;
 
Configuration "dg_config" created with primary database "psoa"
 
DGMGRL> add database 'ssoa' as
connect identifier is ssoa;
 
Database "ssoa" added
 
DGMGRL> enable configuration;
 
Enabled.

4.1.3.1.6 Verifying the Data Guard Broker Configuration

Complete the following steps to verify that the Data Guard Broker configuration was created successfully.

Verify the Oracle Data Guard configuration by querying the V$ARCHIVED_LOG view to identify existing files in the archived redo log. For example:

SQL> SELECT SEQUENCE#, FIRST_TIME, NEXT_TIME 
  2> FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;

SEQUENCE# FIRST_TIME NEXT_TIME
---------- ------------------ ------------------
         8 11-JUL-13 17:50:45 11-JUL-13 17:50:53
         9 11-JUL-1317:50:53 11-JUL-1317:50:58
        10 11-JUL-13 17:50:58 11-JUL-13 17:51:03
 
3 rows selected

On the primary database, issue the following SQL statement to force a log switch and archive the current online redo log file group:
```
SQL> alter system archive log current;
```

On the standby database, query the V$ARCHIVED_LOG view to verify that the redo data was received and archived on the standby database:

SQL> SELECT SEQUENCE#, FIRST_TIME, NEXT_TIME 
  2> FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;

SEQUENCE# FIRST_TIME NEXT_TIME
---------- ------------------ ------------------
         8 11-JUL-13 17:50:45 11-JUL-13 17:50:53
         9 11-JUL-1317:50:53 11-JUL-1317:50:58
         10 11-JUL-13 17:50:58 11-JUL-13 17:51:03
         11 11-JUL-13 17:51:03 11-JUL-1318:34:11

4 rows selected

Using the show configuration command, verify that the configuration was created successfully on the standby site. See Example 4-10.

Example 4-10 Verifying the Data Guard Broker Configuration

DGMGRL> show configuration;

Configuration - dg_config

Protection Mode: MaxPerformance
Databases:
psoa- Primary database
ssoa- Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

4.1.3.1.7 Testing Database Switchover and Switchback

This section contains the following topics:

Performing a Switchover Operation Using Oracle Data Guard Broker
Performing a Switchover Operation Using SQL*Plus

Performing a Switchover Operation Using Oracle Data Guard Broker

To perform a switchover operation using Oracle Data Guard Broker, complete the following tasks:

Verify the Oracle Data Guard Broker configuration that you created using the instructions provided in Section 4.1.3.1.5, "Creating a Data Guard Broker Configuration."

To verify the configuration, run the following command:

DGMGRL> show configuration;
 
Configuration - dg_config
 
Protection Mode: MaxPerformance
Databases:
psoa- Primary database
ssoa- Physical standby database
 
Fast-Start Failover: DISABLED
 
Configuration Status:
SUCCESS

Swap the roles of the primary and standby databases by running the SWITCHOVER command. Example 4-11 shows how Data Guard Broker automatically shuts down and restarts the old primary database as a part of the switchover operation.

Example 4-11

DGMGRL> switchover to 'ssoa';
Performing switchover NOW, please wait...
New primary database "ssoa" is opening...
Operation requires shutdown of instance "psoa1" on database "psoa"
Shutting down instance "psoa1"...
ORA-01109: database not open
 
Database dismounted.
ORACLE instance shut down.
Operation requires startup of instance "psoa1" on database "psoa"
Starting instance "psoa1"...
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is "ssoa"

After the switchover is complete, using the SHOW CONFIGURATION command verify that the switchover operation was successful:

DGMGRL> show configuration;
Configuration - dg_config
Protection Mode: MaxPerformance
Databases:
ssoa- Primary database
psoa- Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS

Performing a Switchover Operation Using SQL*Plus

Perform the following operations to switchover or switchback databases correctly between the newly created physical standby database and the primary Oracle RAC databases:

Shut down all but one instance of the Oracle RAC databases (PSOA) on the primary site. For example, run the following command on SOADC1.DBHOST1 of the production site:
```
srvctl stop instance -d psoa -i soa2
```
Initiate the role transition to the physical standby on the current primary database. For example, run the following command on SOADC1.DBHOST1 of the production site:
```
SQL > ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY WITH SESSION SHUTDOWN;
```
Shut down the primary instance and mount the primary instance. For example, run the following command on SOADC1.DBHOST1 of the production site:
```
SQL > shutdown immediate
SQL > startup mount
```
At this point, both the databases are in Physical Standby mode.

Log in to the instances to continue.
To verify that both the databases are in Physical Standby mode, run this SQL query on both the databases:
```
SQL> select database_role from v$database;
DATABASE_ROLE
----------------
PHYSICAL_STANDBY
```
Switch the physical standby database role to the primary role. For example, run the following command on SOADC2.DBHOST1 of the standby site:
```
SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WITH SESSION SHUTDOWN;
```
Now the physical standby database is the new primary database.
Shut down the new primary database and start up both the RAC nodes using srvctl. For example, run the following command on the new primary site SOADC2.DBHOST1:
```
srvctl start database -d ssoa
```
On the new physical standby database (the old primary) start the managed recovery of the database. For example, run the following command on SOADC1.DBHOST1 of the primary site:
```
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;
```
Start sending the redo data to the new physical standby database. For example, run the following command on the new primary site SOADC2.DBHOST1:
```
SQL> ALTER SYSTEM SWITCH LOGFILE;
```
Check the new physical standby database to see if it is receiving the archive log files by querying the V$ARCHIVED_LOG view.
Perform a switchback operation. A switchback operation is a subsequent switchover operation to return the roles to their original state.

Note:

For information about switchover and failover operation of Oracle Data Guard Broker, see "Switchover and Failover Operations" in the Oracle Data Guard Broker.

4.2 Creating a Production Site

This section provides the steps to create the production site on an Oracle SOA Suite enterprise deployment topology.

It contains the following topics:

Creating the Production Site for the Oracle SOA Suite Topology
Configuring Data Sources for Oracle Fusion Middleware SOA Active-Passive Deployment

Perform the following prerequisites before you start creating the production site:

Set up the host name aliases for the middle tier hosts as described in Section 3.1.1.
Create the required volumes on the shared storage on the production site, as described in Section 4.1.1.
Determine the Oracle Data Guard configuration to use based on the data loss requirements of the database and network considerations such as the available bandwidth and latency when compared to the redo generation.

See Oracle Data Guard Concepts and Administration as well as related Maximum Availability Architecture collateral at the following URL for more information:

http://www.oracle.com/technology/deploy/availability/htdocs/maa.htm

4.2.1 Creating the Production Site for the Oracle SOA Suite Topology

The production site should be installed and configured as described in the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite with the following variations. Complete the following tasks to install and configure the production site:

Task 1 Create volumes and consistency groups

Create volumes and consistency groups on the shared storage device, as described in Section 4.1.1.1.1, "Volume Design for Oracle SOA Suite."

Task 2 Set up physical host names and alias host names

Set up physical host names on the production site, and physical host names and alias host names for the standby site. See Section 3.1.1, "Planning Host Names" for information about planning host names for the production and standby sites.

Task 3 Install and configure Oracle SOA Suite

Install and configure Oracle SOA Suite as described in the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite with the following modifications:

Install the Oracle SOA Suite components into the volumes created on the shared storage device.
Set up the production and standby sites using the aliases to the physical and virtual host names.
Create a separate volume on each site for the JMS stores and transaction logs.
After the installation and configuration of the production site, turn off host name verification. See Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite for detailed instructions about turning off host name verification for an Administration Server and Managed Server.
If you do not plan on turning host name verification off, follow the steps in Section 4.3.2 to configure Node Manager communication.
Create SSL certificates using the host name aliases on all of the Oracle Fusion Middleware hosts for proper Node Manager communication.

4.2.2 Configuring Data Sources for Oracle Fusion Middleware SOA Active-Passive Deployment

The data sources used by Oracle Fusion Middleware SOA Suite should be configured to automate failover of connections in case there is a failover or switchover of the primary database. The following data sources need to be configured properly to automate this failover:

EDNDataSource
EDNLocalTxDataSource
mds-owsm
mds-soa
OraSDPMDataSource
SOADataSource
SOALocalTxDataSource

Additionally any persistence store using database and the leasing data source used for server migration should be configured to automate failover. The GridLink data sources must be modified:

Update the ONS configuration to include both production and standby site ONS.

The items in the list of ONS addresses must be separated by commas.

For example:
```
prmy-scan:6200,stby-scan:6200
```
On the Test ONS Client Configuration page, review the connection parameters, and click Test All ONS Nodes.

Following is an example of a successful connection notification:
```
Connect test for prmy-scan:6200 succeeded.
```

Update the JDBC URL to include the appropriate services in both sites.

The following is a sample JDBC URL for the SOA Data source in an Oracle Fusion Middleware SOA Active/Passive configuration where the database uses Data Guard.

jdbc:oracle:thin:@
(DESCRIPTION_LIST =
         (LOAD_BALANCE = off)
         (FAILOVER = on)
         (DESCRIPTION =
                  (CONNECT_TIMEOUT = 10)
                  (TRANSPORT_CONNECT_TIMEOUT = 3)
                  (RETRY_COUNT = 3)
                  (ADDRESS_LIST =
                       (LOAD_BALANCE = on)
                       (ADDRESS =
                           (PROTOCOL = TCP)(HOST = prmy-scan)(PORT = 1521)
                  )
         )
         (CONNECT_DATA =
              (SERVICE_NAME =soaedg.example.com)
                  )
         )
(DESCRIPTION =
         (CONNECT_TIMEOUT =10)
         (TRANSPORT_CONNECT_TIMEOUT = 3)
         (RETRY_COUNT = 3)
         (ADDRESS_LIST =
                       (LOAD_BALANCE = on)
                       (ADDRESS =
                           (PROTOCOL =TCP)(HOST = stby-scan)(PORT = 1521)
                  )
         )
         (CONNECT_DATA =
                       (SERVICE_NAME = soaedg.example.com))

                  )

)

In the Test GridLink Database Connection page, review the connection parameters, and click Test All Listeners.

Following is an example of a successful connection notification:

Connection test for jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=prmy-scan)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=soaedg.example.com))) succeeded.

4.3 Creating a Standby Site

This section provides the steps to create the standby site. The Oracle SOA enterprise deployment topology is used as an example.

It includes the following topics:

Creating the Standby Site
Updating Self-Signed Certificates and Keystore on Standby Site
Validating the Standby Site Setup

4.3.1 Creating the Standby Site

Perform the following prerequisites before you start creating the standby site:

On the standby site, set up the correct alias host names and physical host names by following the instructions in Section 3.1.1.

Ensure that each standby site host has an alias host name that is the same as the physical host name of its peer host at the production site.
On the shared storage on the standby site, create the same volumes that were created on the shared storage at the production site.
On the standby site, create the same mount points and symbolic links (if required) that you created at the production site. Note that symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes.

For more details about symbolic links, see Section 3.2.3.

4.3.1.1 Setting Up Middle Tier Hosts

The middle tier hosts on the standby site do not require the installation or configuration of any Oracle Fusion Middleware or Oracle WebLogic Server software. When the production site storage is replicated to the standby site storage, the software installed on the production site volumes is replicated at the standby site volumes.

Do the following to set up the middle tier hosts on the standby site:

Create a baseline snapshot copy of shared storage on the production site, which sets up the replication between the storage devices. Create the initial baseline copy and subsequent snapshot copies using asynchronous replication mode.
Synchronize the shared storage at the production site with the shared storage at the standby site. This will transfer the initial baseline snapshot from the production site to the standby site.
Set up the frequency of subsequent copies of the production site shared storage, which will be replicated at the standby site. When asynchronous replication mode is used, then at the requested frequency the changed data blocks at the production site shared storage (based on comparison to the previous snapshot copy) become the new snapshot copy, and the snapshot copy is transferred to the standby site shared storage.
After the baseline snapshot copy is performed, validate that all the directories inside the standby site volumes have the same contents as the directories inside the production site volumes.

4.3.2 Updating Self-Signed Certificates and Keystore on Standby Site

If you enable the hostname verification for the Administration Server, you must also update the appropriate trust stores and key stores with the certificates of the standby site. Certificates must be specifically created for the nodes in the standby site.

For more information about creating certificates for nodes, see "Enabling SSL Communication Between the Middle Tier and the Hardware Load Balancer" in Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.

This section includes the following topics:

Generate Self-Signed Certificates
Create an Identity KeyStore
Create a Trust KeyStore

The examples in these sections show how to perform tasks for the Oracle SOA Suite enterprise topology shown in Figure 4-2.

Note:

When you set up the Oracle SOA Suite enterprise topology shown in Figure 4-2 as the production site for a Disaster Recovery topology, you must use the physical host names shown in Table 3-1 for the production site hosts instead of the host names shown in Figure 4-2.
The steps in this section must be performed on the application tier hosts on which Oracle WebLogic Server is installed.

4.3.2.1 Generate Self-Signed Certificates

Follow these steps to generate self-signed certificates:

Set up your environment by running the WL_HOME/server/bin/setWLSEnv.sh script:

In the Bourne shell, run the following command:
```
. setWLSEnv.sh
```
Verify that the CLASSPATH environment variable is set:
```
echo $CLASSPATH
```
Create a user-defined directory for the certificates.
```
mkdir home/user_projects/domains/SOADomain/certs
```

Change directory to the user-defined directory.

cd home/user_projects/domains/SOADomain/certs

Run the utils.CertGen tool from the user-defined directory to create the certificates for both the physical hostnames and the virtual hostnames used by servers in the node:

java utils.CertGen key_passphrase cert_file_name key_file_name [export|domestic] [hostname]

For example:

java utils.CertGen password soadc1host1_cert soahost1_key domestic soahost1
java utils.CertGen password soadc1host2_cert soahost2_key domestic soahost2

4.3.2.2 Create an Identity KeyStore

Follow these steps to create an identity keystore using the utils.ImportPrivateKey utility:

Import the certificate and private key into the Identity Store using the following syntax. Make sure that you use a different alias for each of the certificate/key pair imported:

java utils.ImportPrivateKey keystore_file keystore_password certificate_alias_to_use private_key_passphrase certificate_file private_key_file keystore_type

Note:

Default value for keystore_type is jks.

For example:

java utils.ImportPrivateKey appIdentityKeyStore.jks password
            appIdentity1 password
            ASERVER_HOME/certs/SOAHOST1.example.com_cert.pem 
            ASERVER_HOME/certs/SOAHOST1.example.com_key.pem

Repeat step 1 for all the hosts on the primary and the standby sites.

4.3.2.3 Create a Trust KeyStore

Follow these steps to create a trust keystore:

Copy the standard java keystore to create the new trust keystore since it already contains most of the root CA certificates needed.

Oracle does not recommend modifying the standard Java trust key store directly.

Copy the standard Java keystore CA certificates located under the WL_HOME/server/lib directory to the same directory as the certificates.

For example:
```
cp $WL_HOME/server/lib/cacerts 
$home/user_projects/domains/SOADomain/certs/appTrustKeyStore.jks
```

The default password for the standard Java keystore is changeit.

Oracle recommends that you change the default password. Use the keytool utility to do this.

The syntax is:

keytool -storepasswd -new NewPassword -keystore TrustKeyStore -storepass 
OriginalPassword

For example, enter this command:

keytool -storepasswd -new password -keystore home//user_projects/domains/SOADomain/certs/appTrustKeyStore.jks -storepass 
changeit

The CA certificate CertGenCA.der is used to sign all certificates generated by the utils.CertGen tool and is located at WL_HOME/server/lib directory. This CA certificate must be imported into the appTrustKeyStore using the keytool utility.

The syntax is:

keytool -import -v -noprompt -trustcacerts -alias AliasName -file 
CA_file_location -keystore key_store_location -storepass key_store_password

For example, enter this command:

keytool -import -v -noprompt -trustcacerts -alias clientCACert -file 
$WL_HOME/server/lib/CertGenCA.der -keystore appTrust.jks -storepass password

4.3.3 Validating the Standby Site Setup

Validate the standby site by performing the following steps:

Shut down any processes still running on the production site. These include the database instances in the data tier, Oracle Fusion Middleware instances, and any other processes in the application tier and web tier.
Stop the replication between the production site shared storage and the standby site shared storage.
Perform a switchback operation. A switchback operation is a subsequent switchover operation to return the roles to their original state.
On the standby site host, manually start all the processes. These include the database instances in the data tier, Oracle Fusion Middleware instances, and any other processes in the application tier and web tier.
Use a browser client to perform post-failover testing to confirm that requests are being resolved and redirected to the standby site.

4.4 Creating an Asymmetric Standby Site

The steps in this section describe how to set up an asymmetric Oracle Fusion Middleware Disaster Recovery topology.

An asymmetric topology is a disaster recovery configuration that is different across tiers at the production site and standby site. In most asymmetric Oracle Fusion Middleware Disaster Recovery topologies, the standby site has fewer resources than the production site.

Before you read this section, ensure that you read and understand the concepts and information about setting up a symmetric topology presented earlier in this manual. Many of the concepts for setting up a symmetric topology are also valid for setting up an asymmetric topology.

The following sections describe the basic steps for creating an asymmetric topology:

Creating the Asymmetric Standby Site
Validating the Asymmetric Standby Site Setup

4.4.1 Creating the Asymmetric Standby Site

This section describes the high-level steps for creating any type of asymmetric Oracle Fusion Middleware Disaster Recovery topology. The production site is the Oracle SOA Suite enterprise deployment shown in Figure 4-2. The standby site will be different from the production site.

To create an asymmetric topology:

Design the production site and the standby site. Determine the resources that will be necessary at the standby site to ensure acceptable performance when the standby site assumes the production role.

Note:

The ports for the standby site instances must use the same port numbers as the peer instances at the production site. Therefore, ensure that all the port numbers that will be required at the standby site are available (not in use at the standby site).
Create the Oracle Fusion Middleware Disaster Recovery production site by performing these operations:
1. Create volumes on the production site's shared storage system for the Oracle Fusion Middleware instances that will be installed for the production site. For more information, see Section 4.1.1.
2. Create mount points and symbolic links on the production site hosts to the Oracle home directories for the Oracle Fusion Middleware instances on the production site's shared storage system volumes. Note that symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes;
  
  For more details about symbolic links, see Section 3.2.3.
  
  For more information about volume design, see Section 4.1.1.1.1.
3. Create mount points and symbolic links on the production site hosts to the Oracle Central Inventory directories for the Oracle Fusion Middleware instances on the production site's shared storage system volumes. Note that symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes;
  
  For more details about symbolic links, see Section 3.2.3.
  
  For more information about the Oracle Central Inventory directories, see Section 3.2.2.
4. Create mount points and symbolic links on the production site hosts to the static HTML pages directories for the Oracle HTTP Server instances on the production site's shared storage system volumes, if applicable. Note that symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes;
  
  For more details about symbolic links, see Section 3.2.3.
5. Install the Oracle Fusion Middleware instances for the production site on the volumes in the production site's shared storage system. For more information, see Section 4.2.1.
Create the same volumes with the same file and directory privileges on the standby site's shared storage system as you created for the Oracle Fusion Middleware instances on the production site's shared storage system. This step is critical because it enables you to use storage replication later to create the peer Oracle Fusion Middleware instance installations for the standby site instead of installing them using Oracle Universal Installer.

Note:

When you configure storage replication, ensure that all the volumes you set up on the production site's shared storage system are replicated to the same volumes on the standby site's shared storage system.

Even though some of the instances and hosts at the production site may not exist at the standby site, you must configure storage replication for all the volumes set up for the production site's Oracle Fusion Middleware instances.
Perform any other necessary configuration required by the shared storage vendor to enable storage replication between the production site's shared storage system and the standby site's shared storage system. Configure storage replication to asynchronously copy the volumes in the production site's shared storage system to the standby site's shared storage system.
Create the initial baseline snapshot copy of the production site shared storage system to set up the replication between the production site and standby site shared storage systems. Create the initial baseline snapshot and subsequent snapshot copies using asynchronous replication mode. After the baseline snapshot copy is performed, validate that all the directories for the standby site volumes have the same contents as the directories for the production site volumes. Refer to the documentation for your shared storage vendor for information about creating the initial snapshot and enabling storage replication between the production site and standby site shared storage systems.
After the baseline snapshot has been taken, perform these steps for the Oracle Fusion Middleware instances for the standby site hosts:
1. Set up a mount point directory on the standby site host to the Oracle home directory for the Oracle Fusion Middleware instance on the standby site's shared storage system. The mount point directory that you set up for the peer instance on the standby site host must be the same as the mount point directory that you set up for the instance on the production site host.
2. Set up a symbolic link on the standby site host to the Oracle home directory for the Oracle Fusion Middleware instance on the standby site's shared storage system. Note that symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes;
  
  For more details about symbolic links, see Section 3.2.3. The symbolic link that you set up for the peer instance on the standby site host must be the same as the symbolic link that you set up for the instance on the production site host.
3. Set up a mount point directory on the standby site host to the Oracle Central Inventory directory for the Oracle Fusion Middleware instance on the standby site's shared storage system. The mount point directory that you set up for the peer instance on the standby site host must be the same as the mount point directory that you set up for the instance on the production site host.
4. Set up a symbolic link on the standby site host to the Oracle Central Inventory directory for the Oracle Fusion Middleware instance on the standby site's shared storage system. Note that symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes;
  
  For more details about symbolic links, see Section 3.2.3. The symbolic link you set up for the peer instance on the standby site host must be the same as the symbolic link that you set up for the instance on the production site host.
5. Set up a mount point directory on the standby site host to the Oracle HTTP Server static HTML pages directory for the Oracle HTTP Server instance on the standby site's shared storage system. The mount point directory that you set up for the peer instance on the standby site host must be the same as the mount point directory that you set up for the instance on the production site host.
6. Set up a symbolic link on the standby site host to the Oracle HTTP Server static HTML pages directory for the Oracle HTTP Server instance on the standby site's shared storage system. Note that symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes; see Section 3.2.3 for more details about symbolic links. The symbolic link that you set up for the peer instance on the standby site host must be the same as the symbolic link that you set up for the instance on the production site host.

After you complete these steps, the Oracle Fusion Middleware instance installations for the production site have been replicated to the standby site. At the standby site, all of the following are true:

The Oracle Fusion Middleware instances are installed into the same Oracle home directories on the same volumes as at the production site, and the hosts use the same mount point directories and symbolic links for the Oracle home directories as at the production site. Note that symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes; see Section 3.2.3 for more details about symbolic links.
The Oracle Central Inventory directories are located in the same directories on the same volumes as at the production site, and the hosts use the same mount point directories and symbolic links for the Oracle Central Inventory directories as at the production site. Note that symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes; see Section 3.2.3 for more details about symbolic links.
The Oracle HTTP Server static HTML pages directories are located in the same directories on the same volumes as at the production site, and the hosts use the same mount point directories and symbolic links for the Oracle HTTP Server static HTML pages directories as at the production site. Note that symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes; see Section 3.2.3 for more details about symbolic links.
The same ports are used for the standby site Oracle Fusion Middleware instances as were used for the same instances at the production site.

4.4.1.1 Creating an Asymmetric Standby Site with Fewer Hosts and Instances

This section describes how to create an asymmetric standby site that has fewer hosts and Oracle Fusion Middleware instances than the production site.

The production site for this Oracle Fusion Middleware Disaster Recovery topology is the Oracle SOA Suite enterprise deployment shown in Figure 4-2. Section 4.1 through Section 4.1.1 describe how to set up this production site and the volumes for its shared storage system, and how to create the necessary mount points.

Figure 4-4 shows the asymmetric standby site for the production site shown in Figure 4-2.

Figure 4-4 An Asymmetric Standby Site with Fewer Hosts and Instances

Description of "Figure 4-4 An Asymmetric Standby Site with Fewer Hosts and Instances"

The Oracle SOA Suite asymmetric standby site shown in Figure 4-4 has fewer hosts and instances than the Oracle SOA Suite production site shown in Figure 4-2.

The hosts WEBHOST2 and SOAHOST2 and the instances on those hosts exist at the production site in Figure 4-2, but these hosts and their instances do not exist at the asymmetric standby site in Figure 4-4. The standby site therefore has fewer hosts and fewer instances than the production site.

It is important to ensure that this asymmetric standby site will have sufficient resources to provide adequate performance when it assumes the production role.

When you follow the steps in Section 4.4.1 to set up this asymmetric standby site, the standby site should be properly configured to assume the production role.

To set up the asymmetric standby site correctly, create the same volumes and consistency groups on the standby site shared storage as you did on the production site shared storage. For example, for the Oracle SOA Suite deployment, the volume design recommendations in Table 4-1 and the consistency group recommendations in Table 4-2 were used to set up the production site shared storage. Use these same volume design recommendations and consistency group recommendations that you used for the production site shared storage to set up the asymmetric standby site shared storage.

Note that at an asymmetric standby site, some hosts that exist at the production site do not exist at the standby site. For example, the asymmetric standby site for Oracle SOA Suite shown in Figure 4-4, WEBHOST2 and SOAHOST2 do not exist; therefore, it is not possible or necessary for you to create mount points on these hosts to the standby site shared storage volumes.

4.4.2 Validating the Asymmetric Standby Site Setup

Validate the standby site by performing the following the steps:

Shut down any processes still running on the production site. These include the database instances in the data tier, Oracle Fusion Middleware instances, and any other processes in the application tier and web tier.
Stop the replication between the production site shared storage and the standby site shared storage.
Use Oracle Data Guard to switchover the databases.
On the standby site host, manually start all the processes. These include Oracle Fusion Middleware instances and any other processes in the application tier and web tier.
Use a browser client to perform post-failover testing to confirm that requests are being resolved and redirected to the standby site.

4.5 Performing Site Operations and Administration

This section describes operations and administration to perform on your Oracle Fusion Middleware Disaster Recovery topology.

It contains the following sections:

Synchronizing the Sites
Performing a Switchover
Performing a Switchback
Performing a Failover
Performing Periodic Testing of the Standby Site
Using Peer-to-Peer File Copy for Testing

4.5.1 Synchronizing the Sites

The standby site shared storage receives snapshots transferred periodically from the production site shared storage. After the snapshots are applied, the standby site shared storage will include all the data up to and including the data contained in the last snapshot transferred from the production site before the failover or switchover.

You should manually force a synchronization operation whenever a change is made to the middle tier at the production site (for example, when a new application is deployed at the production site). Follow the vendor-specific instructions for forcing a synchronization using storage replication technology.

The synchronization of the databases in the Oracle Fusion Middleware Disaster Recovery topology is managed by Oracle Data Guard.

4.5.2 Performing a Switchover

When you plan to take down the production site (for example, to perform maintenance) and make the current standby site the new production site, you must perform a switchover operation so that the standby site takes over the production role.

Follow these steps to perform a switchover operation:

Shut down any processes still running on the production site. These include the database instances in the data tier, Oracle Fusion Middleware instances, and any other processes in the application tier and web tier.
Stop the replication between the production site shared storage and the standby site shared storage.
Unmount the shared storage volume with the middle tier artifacts, on the current production site, and mount the corresponding volumes on the current standby site, which will be the new production site.
Use Oracle Data Guard to switch over the databases.
On the standby site host, manually start all the processes. These include the database instances in the data tier, Oracle Fusion Middleware instances, and any other processes in the application tier and web tier.
Ensure that all user requests are routed to the standby site by performing a global DNS push or something similar, such as updating the global load balancer.
Use a browser client to perform post-switchover testing to confirm that requests are being resolved and redirected to the standby site.

At this point, the former standby site is the new production site and the former production site is the new standby site.
Reestablish the replication between the two sites, but configure the replication so that the snapshot copies go in the opposite direction (from the current production site to the current standby site). See the documentation for your shared storage to learn how to configure the replication so that snapshot copies are transferred in the opposite direction.

After these steps have been performed, the former standby site is the new production site. At this point, you can perform maintenance at the original production site. After performing the planned tasks on the original production site, you can use in the future, you can use it as either the production site or the standby site.

To use the original production site as the new production site, perform the switchback steps described in Section 4.5.3.

4.5.3 Performing a Switchback

After a switchover operation has been performed, a switchback operation can be performed to revert the current production site and the current standby site to the roles they had prior to the switchover operation.

Follow these steps to perform a switchback operation:

Shut down any processes running on the current production site. These include the database instances in the data tier, Oracle Fusion Middleware instances, and any other processes in the application tier and web tier.
Stop the replication between the current production site shared storage and standby site shared storage.
Unmount the shared storage volume with the middle tier artifacts, on the current production site, and mount the corresponding volumes on the current standby site, which will be the new production site.
Use Oracle Data Guard to switch back the databases.
On the new production site hosts, manually start all the processes. These include Oracle Fusion Middleware instances and any other processes in the application tier and web tier.
Ensure that all user requests are routed to the new production site by performing a global DNS push or something similar, such as updating the global load balancer.
Use a browser client to perform post-switchback testing to confirm that requests are being resolved and redirected to the new production site.

At this point, the former standby site is the new production site and the former production site is the new standby site.
Reestablish the replication between the two sites, but configure the replication so that the snapshot copies go in the opposite direction (from the new production site to the new standby site). See the documentation for your shared storage to learn how to configure the replication so that snapshot copies are transferred in the opposite direction.

4.5.4 Performing a Failover

When the production site becomes unavailable unexpectedly, you must perform a failover operation so that the standby site takes over the production role.

Follow these steps to perform a failover operation:

Stop the replication between the production site shared storage and the standby site shared storage.
Mount the shared storage volume with the middle-tier artifacts, on the current standby site, which will be the new production site.
From the standby site, use Oracle Data Guard to fail over the databases.
On the standby site hosts, manually start all the processes. These include the Oracle Fusion Middleware instances and any other processes in the application tier and web tier.
Ensure that all user requests are routed to the standby site by performing a global DNS push or something similar, such as updating the global load balancer.
Use a browser client to perform post-failover testing to confirm that requests are being resolved and redirected to the production site.

At this point, the standby site is the new production site. You can examine the issues that caused the former production site to become unavailable.
To use the original production site as the current standby site, you must reestablish the replication between the two sites, but configure the replication so that the snapshot copies go in the opposite direction (from the current production site to the current standby site). See the documentation for your shared storage system to learn how to configure the replication so that snapshot copies are transferred in the opposite direction.

To use the original production site as the new production site, perform the switchback steps in Section 4.5.3.

4.5.5 Performing Periodic Testing of the Standby Site

This manual describes how to set up Disaster Recovery for an Oracle Fusion Middleware production site and standby site. In a normal Oracle Fusion Middleware Disaster Recovery configuration, the following are true:

Storage replication is used to copy Oracle Fusion Middleware middle tier file systems and data from the production site shared storage to the standby site shared storage. During normal operation, the production site is active and the standby site is passive. When the production site is active, the standby site is passive and the standby site shared storage is in read-only mode; the only write operations made to the standby site shared storage are the storage replication operations from the production site shared storage to the standby site shared storage.
Oracle Data Guard is used to copy database data for the production site Oracle databases to the standby databases at standby site. By default, the production site databases are active and the standby databases at the standby site are passive. The standby databases at the standby site are in Managed Recovery mode while the standby site is in the standby role (is passive). When the production site is active, the only write operations made to the standby databases are the database synchronization operations performed by Oracle Data Guard.
When the production site becomes unavailable, the standby site is enabled to take over the production role. If the current production site becomes unavailable unexpectedly, then a failover operation (described in Section 4.5.4) is performed to enable the standby site to assume the production role. Or, if the current production site is taken down intentionally (for example, for planned maintenance), then a switchover operation (described in Section 4.5.2) is performed to enable the standby site to assume the production role.

The usual method of testing a standby site is to shut down the current production site and perform a switchover operation to enable the standby site to assume the production role. However, some enterprises may want to perform periodic testing of their Disaster Recovery standby site without shutting down the current production site and performing a switchover operation.

An alternate method of testing the standby site is to create a clone of the read-only standby site shared storage and then using the cloned standby site shared storage in testing. To use this alternate testing method, perform these steps:

Use the cloning technology provided by the shared storage vendor to create a clone of the standby site's read-only volumes on the shared storage at the standby site. Ensure that the cloned standby site volumes are writable. If you want to test the standby site just once, then this can be a one-time clone operation. However, if you want to test the standby site regularly, you can set up periodic cloning of the standby site read-only volumes to the standby site's cloned read/write volumes.
Perform a backup of the standby site databases, then modify the Oracle Data Guard replication between the production site and standby site databases.
- For 10.2 and later databases, follow these steps to establish a snapshot standby database:
  1. If you do not have a flash recovery area, set one up.
  2. Cancel Redo Apply:
```
SQL> ALTER DATABASE RECOVER MANAGED STANDBY
     DATABASE CANCEL;
```
  3. Create a guaranteed restore point:
```
SQL> CREATE RESTORE POINT standbytest
     GUARANTEE FLASHBACK DATABASE;
```
  4. Archive the current logs at the primary (production) site:
```
SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;
```
  5. Defer the standby site destination that you will activate:
```
SQL> ALTER SYSTEM SET
     LOG_ARCHIVE_DEST_STATE_2=DEFER;
```
  6. Activate the target standby database:
```
SQL> ALTER DATABASE ACTIVATE STANDBY DATABASE;
```
  7. Mount the database with the Force option if the database was opened as read-only:
```
SQL> STARTUP MOUNT FORCE;
```
  8. Lower the protection mode and open the database:
```
SQL> ALTER DATABASE SET STANDBY DATABASE TO 
     MAXIMIZE PERFORMANCE;
SQL> ALTER DATABASE OPEN;
```
Use Oracle Data Guard database recovery procedures to bring the standby databases online.
On the standby site computers, modify the mount commands to point to the volumes on the standby site's cloned read/write shared storage by following these steps:
1. Unmount the read-only shared storage volumes.
2. Mount the cloned read/write volumes at the same mount point.
Before testing the standby site, modify the host name resolution method for the computers that will be used to perform the testing to ensure that the host names point to the standby site computers and not the production site computers. For example, on a Linux computer, change the /etc/hosts file to point to the virtual IP of the load balancer for the standby site.
Perform the standby site testing.

After you complete the standby site testing, follow these steps to begin using the original production site as the production site again:

Modify the mount commands on the standby site computers to point to the volumes on the standby site's read-only shared storage. In other words, reset the mount commands back to what they were before the testing was performed.
1. Unmount the cloned read/write shared storage volume.
2. Mount the read-only shared storage volumes.
At this point, the mount commands are reset to what they were before the standby site testing was performed.
Configure Oracle Data Guard to perform replication between the production site databases and standby databases at the standby site. Performing this configuration puts the standby database into Managed Recovery mode again:
- For Oracle Database 10.2 and later, follow these steps:
  1. Revert the activated database back to a physical standby database:
```
SQL> STARTUP MOUNT FORCE;
SQL> FLASHBACK DATABASE TO POINT standbytest;
SQL> ALTER DATABASE CONVERT TO PHYSICAL STANDBY;
SQL> STARTUP MOUNT FORCE;
```
  2. Restart managed recovery:
```
SQL> ALTER DATABASE RECOVER MANAGED STANDBY
     DATABASE USING CURRENT LOGFILE DISCONNECT;
```
  3. Reenable the standby destination and switch logs:
```
SQL> ALTER SYSTEM SET
     LOG ARCHIVE DEST STATE 2=ENABLE;
```
- For Oracle Database 12c, set up the replication again by following the steps in the "Managing a Snapshot Standby Database" section in Oracle Data Guard Concepts and Administration.
Before using the original production site again, modify the host name resolution method for the computers that will be used to access the production site to ensure that the host names point to the production site computers and not the standby site computers. For example, on a Linux computer, change the /etc/hosts file to point to the virtual IP of the load balancer for the production site.

4.5.6 Using Peer-to-Peer File Copy for Testing

As an alternative to using storage replication technology for disaster protection and disaster recovery of Oracle Fusion Middleware middle tier components, you can use peer to peer file copy mechanisms in test environments to replicate middle tier file system data from a production site host to a standby site peer host in an Oracle Fusion Middleware Disaster Recovery topology. An example of a peer-to-peer file copy mechanism is rsync (an open source utility for UNIX systems).

This section describes how to use rsync instead of storage replication in your Oracle Fusion Middleware Disaster Recovery topology. It discusses rsync in the context of symmetric topologies. The information provided for rsync in this section also applies to other peer to peer file copy mechanisms.

Before you read the information in this section, read the rest of this manual to ensure that you are familiar with how to use storage replication and Oracle Data Guard in an Oracle Fusion Middleware Disaster Recovery topology. There are many similarities between using storage replication and rsync for disaster protection and disaster recovery of your Oracle Fusion Middleware components.

Note:

You can use rsync instead of storage replication technology to replicate middle tier file system data from the production site to the standby site. However, be aware that the following beneficial storage replication features are not available when you use rsync:

With storage replication, you can roll changes back to the point in time when any previous snapshot was taken at the production site.

With rsync, replicated production site data overwrites the standby site data, and you cannot roll back a replication.
With storage replication, the volume that you set up for each host cluster in the shared storage systems ensures data consistency for that host cluster across the production site's shared storage system and the standby site's shared storage system.

With rsync, data consistency is not guaranteed.

Because of these deficiencies in comparison to storage replication, rsync is not supported for disaster recovery use in actual production environments.

4.5.6.1 Using rsync and Oracle Data Guard for Oracle Fusion Middleware Disaster Recovery Topologies

The following sections describe the basic principles that apply when you use rsync and Oracle Data Guard to provide disaster protection and disaster recovery for your Oracle Fusion Middleware Disaster Recovery topology:

Using rsync for Oracle Fusion Middleware Middle Tier Components
Performing Failover and Switchover Operations

Note:

For information about how to set up Oracle Data Guard for Oracle database, see Section 3.3.

4.5.6.1.1 Using rsync for Oracle Fusion Middleware Middle Tier Components

Follow these steps to use rsync to provide disaster protection and disaster recovery for your Oracle Fusion Middleware middle tier components:

Set up rsync to enable replication of files from a production site host to its standby site peer host. See the rsync man page for instructions on installing and setting up rsync, and for syntax and usage information. Information about rsync is also available at http://rsync.samba.org.
For each production site host on which one or more Oracle Fusion Middleware components has been installed, set up rsync to copy the following directories and files to the same directories and files on the standby site peer host:
- The Oracle home directory and subdirectories, and all the files in them
- The Oracle Central Inventory directory and files for the host, which includes the Oracle Universal Installer entries for the Oracle Fusion Middleware installations
- If applicable, the Oracle Fusion Middleware static HTML pages directory for the Oracle HTTP Server installations on the host
- If applicable, the .fmb and .fmx deployment artifact files created by Oracle Forms on the host, and the .rdf deployment artifact files created by Oracle Reports on the host
  
  Note:
  
  Run rsync as root. If you want rsync to work without prompting users for a password, set up SSH keys between the production site host and standby site host, so that SSH does not prompt for a password.
Set up scheduled jobs, for example, cron jobs, for the production site hosts for which you set up rsync in the previous step. These scheduled jobs enable rsync to automatically perform replication of these files from the production site hosts to the standby site hosts on a regular interval. An interval of once a day is recommended for a production site where the Oracle Fusion Middleware configuration does not change very often.
Whenever a change is made to the configuration of an Oracle Fusion Middleware middle tier configuration on a production site host (for example, when a new application is deployed), you should perform a manual synchronization of that host with its standby site peer host using rsync.
Whenever you perform a manual rsync synchronization of an Oracle Fusion Middleware middle tier instance on a production site host to the peer standby site host, you should also manually force a synchronization of any associated database repository for the production site's Oracle Fusion Middleware instance to the standby site using Oracle Data Guard. See Section 3.3.2 for more information about manually forcing a synchronization of an Oracle database using Oracle Data Guard.

4.5.6.1.2 Performing Failover and Switchover Operations

Follow these steps to perform a failover or switchover from the production site to the standby site when you are using rsync:

Shut down any processes still running on the production site (if applicable).
Stop the rsync jobs between the production site hosts and their standby site peer hosts.
Use Oracle Data Guard to fail over the production site databases to the standby site.
On the standby site, manually start the processes for the Oracle Fusion Middleware Server instances.
Route all user requests to the standby site by performing a global DNS push or something similar, such as updating the global load balancer.
Use a browser client to perform post-failover or post-switchover testing to confirm that requests are being resolved at the standby site (current production site).

At this point, the standby site is the new production site and the production site is the new standby site.
Reestablish the rsync replications between the two sites, but configure the replications so that they go in the opposite direction (from the current production site to the current standby site).

To use the original production site as the new production site, perform the preceding steps again, but configure the rsync replications to go in the original direction (from the original production site to the original standby site).

4.6 Using Oracle Site Guard for Disaster Recovery

Oracle Site Guard primarily orchestrates switchover and failover between two disaster recovery sites. It offers the following features:

Ensures high availability, data protection, and disaster recovery for enterprise data.
Performs Oracle Site Guard operations like switchover and failover. If the primary site becomes unavailable due to a planned or an unplanned outage, a Switchover or Failover process needs to be initiated using Oracle Site Guard.

For more information about how to use Oracle Site Guard, see Oracle Site Guard Administrator's Guide.

4.7 Patching an Oracle Fusion Middleware Disaster Recovery Site

This section describes how to apply a 12c Oracle Fusion Middleware patch set to upgrade the Oracle homes that participate in an Oracle Fusion Middleware Disaster Recovery site.

The list in this section describes the steps for applying a patch set to upgrade the Oracle Fusion Middleware 12c homes in an Oracle Fusion Middleware Disaster Recovery production site.

The following steps assume that the Oracle Central Inventory for any Oracle Fusion Middleware instance that you are patching is located on the production site shared storage, so that the Oracle Central Inventory for the patched instance can be replicated to the standby site.

Use the following procedure to upgrade Oracle Fusion Middleware 12c patch versions:

Perform a backup of the production site to ensure that the starting state is secured.
Apply the patch set to upgrade the production site instances.
After applying the patch set, manually force a synchronization of the production site shared storage and standby site shared storage. This replicates the production site's patched instance and Oracle Central Inventory in the standby site's shared storage.
After applying the patch set, use Oracle Data Guard to manually force a synchronization of the Oracle databases at the production site and standby sites. Because some Oracle Fusion Middleware patch sets make updates to repositories, this step ensures that any changes made to production site databases are synchronized to the standby site databases.
The upgrade is now complete. Your Disaster Recovery topology is ready to resume processing.

Note:

Patches must be applied only at the production site for an Oracle Fusion Middleware 12c Disaster Recovery topology. If a patch is for an Oracle Fusion Middleware instance or for the Oracle Central Inventory, the patch will be copied when the production site shared storage is replicated to the standby site shared storage. A synchronization operation should be performed when a patch is installed at the production site.

Similarly, if a patch is installed for a production site database, Oracle Data Guard will copy the patch to the standby database at the standby site when a synchronization is performed.