Setting Up and Managing Disaster Recovery Sites

4.1 Setting Up a Site

Learn how to set up an Oracle Disaster Recovery site.

Before you start creating the production site, ensure that you:

Set up the host name aliases for the middle tier hosts, as described in Planning Host Names.
Create the required volumes on the shared storage on the production site, as described in Designing Directory Structure and Volumes.
Determine the Oracle Data Guard configuration to use based on the data loss requirements of the database and network considerations, such as the available bandwidth and latency when compared to the redo generation.

This section includes the following topics:

Designing Directory Structure and Volumes
Learn about the recommended directory structure in your disaster recovery topology.
Setting Up Storage Replication
Learn how to set up storage replication for the Oracle Fusion Middleware Disaster Recovery topology.
Installing and Configuring Databases
Learn how to install and configure Oracle Database 11.2 or Oracle Database 12.1 MAA databases in an Oracle SOA Suite enterprise deployment.

Parent topic: Setting Up and Managing Disaster Recovery Sites

4.1.1 Designing Directory Structure and Volumes

Learn about the recommended directory structure in your disaster recovery topology.

You can choose a directory layout different from the one recommended in this document, but the model adopted enables maximum availability, provides the best isolation of components and symmetry in the configuration, and facilitates backup and disaster recovery.

The following list describes directories and directory environment variables:

ORACLE_BASE: This environment variable and related directory path refers to the base directory below which Oracle products are installed.
ORACLE_HOME: This related directory path refers to the location where Oracle Fusion Middleware resides.
WL_HOME: This environment variable and related directory path contains installed files that are necessary to host an Oracle WebLogic Server.
PROD_DIR: This environment variable and related directory path refers to the location where a product suite, such as Oracle SOA Suite, Oracle WebCenter Portal, or Oracle Identity Management is installed.
DOMAIN directory: This directory path refers to the location where the Oracle WebLogic Domain information (configuration artifacts) is stored. Different WebLogic Servers can use different domain directories even when in the same node.
ORACLE_INSTANCE: An Oracle instance contains one or more system components. An Oracle instance directory contains updatable files, such as configuration files, log files, and temporary files.

This section includes the following topic:

Recommended Directory Structure for Oracle SOA Suite
Learn about the recommended directory structures for Oracle SOA Suite.
Recommended Volume Design for Oracle SOA Suite
Learn about the recommended volume design for Oracle SOA Suite.

Parent topic: Setting Up a Site

4.1.1.1 Recommended Directory Structure for Oracle SOA Suite

Learn about the recommended directory structures for Oracle SOA Suite.

Oracle Fusion Middleware allows you to create multiple SOA Managed Servers from a single binary installation. This allows the installation of binary files in a single location on a shared storage and the reuse of this installation by the servers in different nodes. However, for maximum availability, Oracle recommends that you use redundant binary installations. In this model, two Oracle homes (each of which has a WL_HOME and an ORACLE_HOME for each product suite) are installed in a shared storage. Additional servers (when scaling out or up) of the same type can use either one of these two locations without requiring more installations. Ideally, users should use two different volumes for redundant binary location, this isolating the failures as much as possible in each volume. For additional protection, Oracle recommends using storage replication for these volumes. If multiple volumes are not available, Oracle recommends that you use mount points to simulate the same mount location in a different directory in the shared storage. Although this does not guarantee the protection that multiple volumes provide, it does allow protection from user deletions and individual file corruption.

Oracle also recommends separating the domain directory that is used by the Administration Server from the domain directory that is used by Managed Servers. This allows a symmetric configuration for the domain directories that is used by Managed Servers, and isolates the failover of the Administration Server. The domain directory for the Administration Server must reside in a shared storage to allow failover to another node with the same configuration. In addition, Oracle recommends that you place the Managed Servers' domain directories on a shared storage, although having them on the local file system is also supported. This is especially important when you design a production site with the disaster recovery site in mind. Figure 4-1 represents the directory structure layout for Oracle SOA Suite.

Figure 4-1 Directory Structure for Oracle SOA Suite

Description of "Figure 4-1 Directory Structure for Oracle SOA Suite"

For information about setting up this directory structure, see Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.

For volume design, see the following section:

Parent topic: Designing Directory Structure and Volumes

4.1.1.2 Recommended Volume Design for Oracle SOA Suite

Learn about the recommended volume design for Oracle SOA Suite.

Figure 4-2 and Figure 4-3 shows an Oracle SOA Suite topology diagram. The volume design described in this section is for this Oracle SOA Suite topology. Detailed instructions for installing and configuring this topology are provided in the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.

Figure 4-2 Oracle SOA Suite and Oracle Business Activity Monitoring Enterprise Topology Diagram

Description of "Figure 4-2 Oracle SOA Suite and Oracle Business Activity Monitoring Enterprise Topology Diagram"

Figure 4-3 Oracle SOA Suite and Oracle Service Bus Enterprise Deployment Reference Topology Diagram

Description of "Figure 4-3 Oracle SOA Suite and Oracle Service Bus Enterprise Deployment Reference Topology Diagram"

For disaster recovery of this Oracle SOA Suite topology, Oracle recommends the following volume design:

Provision two volumes for two Oracle homes that contain the redundant product binary files (VOLFMW1 and VOLFMW2 in Table 4-1).
Provision one volume for the Administration Server domain directory (VOLADMIN in Table 4-1).
Provision one volume on each node for the Managed Server domain directory (VOLSOA1 and VOLSOA2 in Table 4-1). This directory is shared among all the Managed Servers on that node.
Provision one volume for the JMS file store and the JTA transaction logs (VOLDATA in Table 4-1). One volume for the entire domain is mounted on all the nodes in the domain.
Provision one volume on each node for the Oracle HTTP Server Oracle home (VOLWEB1 and VOLWEB2 in Table 4-1).
Provision one volume on each node for the Oracle HTTP Server Domain Directory (VOLOHS1 and VOLOHS2 in Table 4-1).

Note:
For WebTier hosts, local storage is usually recommended. You can replicate this configuration on a regular basis to one of the other app tier volumes to sync to standby or directly from the production webhost to the standby webhost.

Table 4-1 provides a summary of Oracle recommendations for volume design for the Oracle SOA Suite topology shown in Figure 4-2 and Figure 4-3.

Table 4-1 Volume Design Recommendations for Oracle SOA Suite

Tier	Volume Name	Mounted on Host	Mount Point	Comments
Web	`VOLWEB1`	`WEBHOST1`	`/u02/oracle/products/fmw`	Volume for Oracle HTTP Server installation
Web	`VOLWEB2`	`WEBHOST2`	`/u02/oracle/products/fmw`	Volume for Oracle HTTP Server installation
Web	`VOLOHS1`	`WEBHOST1`	`/u02/oracle/config/domains/ohs_domain`	Volume for Oracle HTTP Server domain directory
Web	`VOLOHS2`	`WEBHOST2`	`/u02/oracle/config/domains/ohs_domain`	Volume for Oracle HTTP Server domain directory
Web	`VOLSTATIC1`	`WEBHOST1`	`/u02/oracle/config/static`	(Optional) Volume for static HTML content
Web	`VOLSTATIC2`	`WEBHOST2`	`/u02/oracle/config/static`	(Optional) Volume for static HTML content
Application	`VOLFMW1`	`SOAHOST1`	`/u01/oracle/products/fmw`	Volume for the WebLogic Server and Oracle SOA Suite binary files
Application	`VOLFMW2`	`SOAHOST2`	`/u01/oracle/products/fmw`	Volume for the WebLogic Server and Oracle SOA Suite binary files
Application	`VOLADMIN`	`SOAHOST1`, `SOAHOST2`	`/u01/oracle/config`	Volume for Administration Server domain directory and other shared configurations, such as Deployment Plans, applications, and keystores
Application	`VOLSOA1`	`SOAHOST1`	`/u02/oracle/config`	Volume for Managed Server domain directory
Application	`VOLSOA2`	`SOAHOST2`	`/u02/oracle/config`	Volume for Managed Server domain directory
Application	`VOLRUNTIME`	`SOAHOST1`, `SOAHOST2`	`/u01/oracle/runtime`	Volume for transaction logs and JMS data

For consistency group recommendations, see:

Recommended Consistency Groups for Oracle SOA Suite
Learn about the recommended consistency groups for Oracle SOA Suite.

Parent topic: Designing Directory Structure and Volumes

4.1.1.2.1 Recommended Consistency Groups for Oracle SOA Suite

Learn about the recommended consistency groups for Oracle SOA Suite.

Oracle recommends the following consistency groups for the Oracle SOA Suite topology:

Create one consistency group with the volumes that contains the domain directories for the Administration Server and Managed Servers as members (DOMAINGROUP in Table 4-2).
Create one consistency group with the volume that contains the JMS file store and transaction log data as members (RUNTIMEGROUP in Table 4-2).
Create one consistency group with the volume that contains the Oracle homes as members (FMWHOMEGROUP in Table 4-2).

Table 4-2 provides a summary of Oracle recommendations for consistency groups for the Oracle SOA Suite topology as shown in Figure 4-2.

Table 4-2 Consistency Groups for Oracle SOA Suite

Tier Group Name Members Comments

Tier	Group Name	Members	Comments
Application	`DOMAINGROUP`	`VOLADMIN` `VOLSOA1` `VOLSOA2`	Consistency group for the Administration Server, and the Managed Server domain directory
Application	`RUNTIMEGROUP`	`VOLRUNTIME`	Consistency group for the JMS file store and transaction log data
Application	`FMWHOMEGROUP`	`VOLFMW1` `VOLFMW2`	Consistency group for the Oracle homes

Application

DOMAINGROUP

VOLADMIN

VOLSOA1

VOLSOA2

Consistency group for the Administration Server, and the Managed Server domain directory

Application

RUNTIMEGROUP

VOLRUNTIME

Consistency group for the JMS file store and transaction log data

Application

FMWHOMEGROUP

VOLFMW1

VOLFMW2

Consistency group for the Oracle homes

Parent topic: Recommended Volume Design for Oracle SOA Suite

4.1.2 Setting Up Storage Replication

Learn how to set up storage replication for the Oracle Fusion Middleware Disaster Recovery topology.

To set up storage replication for the Oracle Fusion Middleware Disaster Recovery topology:

On the standby site, ensure that the alias host names that are created are the same as the physical host names that are used for the peer hosts at the production site.
On the shared storage at the standby site, create the same volumes as were created on the shared storage at the production site.
On the standby site, create the same mount points and symbolic links that you created at the production site.
Note:
- The symbolic links only need to be set up on the standby site if you set up symbolic links at the production site.
- The symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes. For more information about symbolic links, see Storage Replication.
It is not necessary to install the same Oracle Fusion Middleware instances at the standby site as were installed at the production site. When the production site storage is replicated to the standby site storage, the Oracle software installed on the production site volumes are replicated at the standby site volumes.
Create the baseline snapshot copy of the production site shared storage that sets up the replication between the production site and standby site shared storage. Create the initial baseline copy and subsequent snapshot copies by using asynchronous replication mode. After the baseline snapshot copy is performed, validate that all the directories inside the standby site volumes have the same contents as the directories inside the production site volumes.
Set up the frequency of subsequent copies of the production site shared storage, which is replicated at the standby site. When asynchronous replication mode is used, then at the requested frequency the changed data blocks at the production site shared storage (based on comparison to the previous snapshot copy) become the new snapshot copy, and the snapshot copy is transferred to the standby site shared storage.
Ensure that disaster protection for any database that is included in the Oracle Fusion Middleware Disaster Recovery production site is provided by Oracle Data Guard. Do not use storage replication technology to provide disaster protection for Oracle databases.
The standby site shared storage receives snapshots transferred periodically from the production site shared storage. After the snapshots are applied, the standby site shared storage includes all the data up to and including the data contained in the last snapshot transferred from the production site before the failover or switchover.
Oracle strongly recommends that you manually force a synchronization operation whenever a change is made to the middle tier at the production site (for example, when a new application is deployed at the production site). Follow the vendor-specific instructions for forcing a synchronization by using storage replication technology.

Parent topic: Setting Up a Site

4.1.3 Installing and Configuring Databases

Learn how to install and configure Oracle Database 11.2 or Oracle Database 12.1 MAA databases in an Oracle SOA Suite enterprise deployment.

For recommendations and considerations for setting up Oracle databases that are used in an Oracle Fusion Middleware Disaster Recovery topology, see Database Considerations.

Installing and Configuring Oracle Database 11.2 or 12.1 MAA Environments

Parent topic: Setting Up a Site

4.1.3.1 Installing and Configuring Oracle Database 11.2 or 12.1 MAA Environments

Oracle Maximum Availability Architecture (MAA) is Oracle's comprehensive architecture to reduce downtime for scheduled outages, and to prevent, detect and recover, from unscheduled outages.

Real Application Clusters (RAC) and Data Guard provide the basis of the database MAA solution, where the primary site contains the RAC database, and the secondary site contains the RAC physical standby database.

This section contains the following topics:

Tip:

Alternatively, you can perform many of the tasks in this section by using Oracle Enterprise Manager Cloud Control (Cloud Control).

Setting up and managing databases using Cloud Control helps in controlling downtime and simplifies disaster recovery operations.

For information about installing Enterprise Manager Cloud Control 12c, see Oracle Enterprise Manager Cloud Control Basic Installation Guide.

For more information about setting up Oracle Data Guard using Cloud Control, see Set Up and Manage Oracle Data Guard using Oracle Enterprise Manager Cloud Control 12c.

Parent topic: Installing and Configuring Databases

4.1.3.1.1 Prerequisites and Assumptions

Ensure that the following prerequisites are met:

The Oracle RAC cluster and Automatic Storage Management (ASM) instances on the standby site have been created.
The Oracle RAC databases on the standby site and the production site are using a flash recovery area.
The Oracle RAC databases are running in archivelog mode.
The database hosts on the standby site already have Oracle software installed.
In a shared ORACLE_HOME configuration, the TNS_ADMIN directory must be a local, non-shared directory.

Parent topic: Installing and Configuring Oracle Database 11.2 or 12.1 MAA Environments

4.1.3.1.2 Oracle Data Guard Environment Description

The examples given in this section contain environment variables as described in Table 4-3.

Table 4-3 Variables Used by Primary and Standby Databases

Variable	Primary Database	Standby Database
Database names	`soa`	`soa`
SOA Database Host Names	`soadc1.dbhost1,` `soadc1.dbhost2`	`soadc2.dbhost1`, `soadc2.dbhost2`
Database unique names	`psoa`	`ssoa`
Instance names	`soa1`, `soa2`	`soa1`, `soa2`
Service names	`psoa`, `ssoa`	`psoa`, `ssoa`

Parent topic: Installing and Configuring Oracle Database 11.2 or 12.1 MAA Environments

4.1.3.1.3 Procedure for Duplicating the Primary Database

Follow these steps to prepare the primary database for setting up Oracle Data Guard:

Note:

For information about prerequisites for setting up Oracle Data Guard, see Prerequisites in Oracle Data Guard Broker.

Enable force logging on the primary database:

SQL> alter database force logging;
SQL> select name, log_mode, force_logging from v$database;
NAME LOG_MODE FOR
---------- ---------------- -------
PSOA ARCHIVELOG YES

Create standby redo logs that are the same size as the online redo logs, on the primary database. The Duplicate command automatically creates the standby redo log on the standby database.

Oracle recommends that you have the same number plus one additional redo logs for each thread as shown in **INTERNAL XREF ERROR**.
In the standby node 1, create and start a listener (for example, LISTENER_DUPLICATE) that offers a static SID entry for the standby database, and has the same value of ORACLE_SID as the primary (soa1), as shown in Example 4-1.

Provide the following path to the listener.ora file:
```
GRID_HOME/network/admin/listener.ora
```
Run the following command to start and verify the listeners by using lsnrctl from GRID_HOME/bin::
```
lsnrctl start listener_duplicate
lsnrctl status listener_duplicate
```
In the standby node 2, create and start a listener (for example, LISTENER_DUPLICATE) that offers a static SID entry for the standby database, and has the same value of ORACLE_SID as the primary (soa2), as shown in Example 4-2.

Provide the following path to the listener.ora file:
```
GRID_HOME/network/admin/listener.ora
```
Note:

Listeners are configured at the cluster level, and all nodes inherit the port and environment settings of the listener. Therefore, the TNS_ADMIN directory path have the same values on all nodes.

In a shared ORACLE_HOME configuration, the TNS_ADMIN directory must be a local, non-shared directory. These network files are included as IFILES.

Complete the following steps to set up TNS_ADMIN for a shared ORACLE_HOME in a two-node cluster, SOADC1.DBHOST1, and SOADC1.DBHOST2, with respective instances of SOA1 and SOA2:
1. Create a local network directory on each node. For example, /local_network_dir/network_admin.
2. Create a local listener.ora file in the location: /local_network_dir/network_admin on each node.
3. In the local listener.ora file, add the values for the LISTENER_duplicate parameter.
4. In the common listener.ora file, in GRID_HOME/network/admin, add the IFILE parameter values, as follows:
```
IFILE=/local_network_dir/network_admin/listener.ora
```

Run the following command to start and verify the listeners:

lsnrctl start listener_duplicate
lsnrctl status listener_duplicate

In the database home of the primary node, create an Oracle Net alias to connect to the listener that you created in step 3. See Example 4-3.

For example: GRID_HOME/bin/tnsping dup.
Configure the Oracle password file authentication for redo transport. Make sure it meets the following requirements:
- Set remote_login_passwordfile to EXCLUSIVE.
```
SQL> show parameter 
remote_login_passwordfile
NAME TYPE VALUE
--------------------------------- -------------- --------------------------------
remote_login_passwordfile string EXCLUSIVE
```
- Copy the primary password file to the auxiliary instance location.
In the ORACLE_HOME/dbs directory of the standby host, create a pfile initsoa1.ora, with the following parameters:
```
db_name=soa
db_unique_name=ssoa
sga_target=5g
```
Create the audit directory for the soa database on all standby hosts:
```
mkdir -p /u01/app/oracle/admin/soa/adump
```
Create an Oracle Net alias on all primary hosts to reach the ssoa database on the standby hosts.

Ensure that all hosts have an Oracle Net alias for psoa and ssoa. All the aliases that you create need to reference the scan listener and not the VIP node. Also, if the local_listener variable is set to an alias on the primary host, then enter the details of the variable on the standby site, that point to the local listener on the primary host. See Example 4-4 and Example 4-5.
Note:
- For primary node 2 (SOADC1.DBHOST2), update the HOST value in psoa_local_listener to point to the VIP of primary node 2, as follows:
```
psoa_local_listener = 
    (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL = TCP)
(HOST=prmy2-vip)(PORT = 1521))
    )
  )
```
- If you are using a shared Oracle home, add the VIPs of the two nodes to the local_listener parameter.
  
  For example:
```
psoa_local_listener =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL = TCP)
(HOST=prmy1-vip)(PORT = 1521))
    (ADDRESS=(PROTOCOL = TCP)
(HOST=prmy2-vip)(PORT = 1521))
    )
  )
```
- For standby node 2 (SOADC2.DBHOST2), update the HOST value in ssoa_local_listener to point to the VIP of standby node 2, as follows:
```
ssoa_local_listener = (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL = TCP)
(HOST=stby2-vip)(PORT = 1521))
    )
  )
```
- If you are using a shared Oracle home, add the VIPs of the two nodes to the local_listener parameter.
  
  For example:
```
ssoa_local_listener =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL = TCP)
(HOST=stby1-vip)(PORT = 1521))
    (ADDRESS=(PROTOCOL = TCP)
(HOST=stby2-vip)(PORT = 1521))
    )
  )
```
On the standby host, set the ORACLE_SID to the same value as that of the primary database (ORACLE_SID=soa1), and run the startup nomount command on the standby database with standby PFILE, as created in step 9.

To obtain the values use this query:

SQL> select p.inst_id,instance_name, name,value from gv$parameter p, gv$instance i where p.inst_id=i.inst_id and p.name='cluster_interconnects';

INST_ID	INSTANCE_NAME	NAME	VALUE
1	dbm1	cluster_interconnects	192.168.44.225
2	dbm2	cluster_interconnects	192.168.44.226

Disable the parameter cluster_interconnects on the primary host, if it is set:

SQL> alter system reset cluster_interconnects scope=spfile sid='soa1';
SQL> alter system reset cluster_interconnects scope=spfile sid='soa2';

On the primary host, run the Recovery Manager (RMAN) script to duplicate the primary database by using the command duplicate target database for standby from active database.

Note:

This command varies depending on your environment. For information about how to use the command, see Duplicating a Database in Oracle Database Backup and Recovery User's Guide.

See Example 4-6 to understand how to duplicate data between two systems with different disk-group names.

See Example 4-7 to understand how to duplicate data between two systems that have the same disk-group name +DATA.
If you have disabled the parameter cluster_interconnects on the primary host as described in step 14, then you must set it back to the original values in the spfile.
(Optional) Stop and remove the listener that you created in step 3.

Example 4-1 Script for Creating and Starting a Listener with Static SID

LISTENER_duplicate =
 (DESCRIPTION_LIST =
  (DESCRIPTION =
   (ADDRESS = (PROTOCOL = TCP)
   (HOST = soadc2.dbhost1)
   (PORT = 1521)(IP = FIRST))))
 
SID_LIST_LISTENER_duplicate =
 (SID_LIST =
  (SID_DESC =
   (SID_NAME = soa1)
   (ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1)))

Example 4-2 Script for Creating and Starting a Listener with Static SID

LISTENER_duplicate =
 (DESCRIPTION_LIST =
  (DESCRIPTION =
   (ADDRESS = (PROTOCOL = TCP)
   (HOST = soadc2.dbhost2)
   (PORT = 1521)(IP = FIRST))))

SID_LIST_LISTENER_duplicate =
 (SID_LIST =
  (SID_DESC =
   (SID_NAME = soa2)
   (ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1)))

Example 4-3 Script for Creating an Oracle Net Alias

dup =
(DESCRIPTION =
 (ADDRESS =
  (PROTOCOL = TCP)
   (HOST = soadc2.dbhost1)
   (PORT = 1521))
 (CONNECT_DATA =
  (SERVER = DEDICATED)
   (SID = soa1)))

Example 4-4 Sample tnsnames.ora File on Primary Node 1 (SOADC1.DBHOST1)

psoa =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL= TCP)
(HOST=prmy-scan)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = psoa)
    )
  )
 
ssoa =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL = TCP)
(HOST=stby-scan)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = ssoa)
    )
  )
 
psoa_local_listener = 
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL = TCP)
(HOST=prmy1-vip)(PORT = 1521))
    )
  )

Example 4-5 Sample tnsnames.ora File on Standby Node 1 (SOADC2.DBHOST1)

psoa =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL= TCP)(HOST=prmy-scan)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = psoa)
    )
  )
 
ssoa =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL = TCP)
(HOST=stby-scan)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = ssoa)
    )
  )
 
ssoa_local_listener = 
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL = TCP)
(HOST=stby1-vip)(PORT = 1521))
    )
  )

Example 4-6 Duplicating Data Between Two Systems with Different Disk-Group Names

rman <<EOF
connect target sys/password;
connect auxiliary sys/password@dup;
run { 
allocate channel prmy1 type disk; 
allocate channel prmy2 type disk; 
allocate channel prmy3 type disk; 
allocate channel prmy4 type disk; 
allocate auxiliary channel stby type disk; 
duplicate target database for standby from active database 
spfile 
parameter_value_convert '+DATA_prmy','+DATA_stby','+RECO_prmy','+RECO_stby' 
set db_file_name_convert '+DATA_prmy','+DATA_stby' 
set db_unique_name='ssoa' 
set db_create_online_log_dest_1='+DATA_stby' 
set db_create_file_dest='+DATA_stby' 
set db_recovery_file_dest='+RECO_stby' 
set log_file_name_convert '+DATA_prmy','+DATA_stby','+RECO_prmy','+RECO_stby' 
set control_files='+DATA_stby/ssoa/standby.ctl' 
set local_listener='ssoa_local_listener'
set remote_listener='stby-scan:1521'; 
} EOF

Example 4-7 Duplicating Data Between Two Systems with Same Disk-Group Name +DATA

rman <<EOF
connect target sys/password;
connect auxiliary sys/password@dup;
run {
allocate channel prmy1 type disk;
allocate channel prmy2 type disk;
allocate channel prmy3 type disk;
allocate channel prmy4 type disk;
allocate auxiliary channel stby type disk;
 
duplicate target database for standby from active database
spfile
set db_unique_name='ssoa'
set control_files='+DATA/ssoa/standby.ctl'
set local_listener='ssoa_local_listener'
set remote_listener='stby-scan:1521';
}
EOF

Example 4-8 Sample Redo Log

SQL> alter database add standby logfile thread 1
group 9 size 500M,
group 10 size 500M,
group 11 size 500M;
 
SQL> alter database add standby logfile thread 2
group 12 size 500M,
group 13 size 500M,
group 14 size 500M;

Parent topic: Installing and Configuring Oracle Database 11.2 or 12.1 MAA Environments

4.1.3.1.4 Procedure for Completing RAC Configuration for Standby Database

To complete the RAC configuration on the standby database, complete the steps given in Procedure for Duplicating the Primary Database. Then, perform the following steps:

Create a temporary parameter file in the standby database:
```
SQL> create pfile='/tmp/p.ora' from spfile;
```

Create an SPFILE in +DATA_stby for the standby database:

SQL> create spfile='+DATA_stby/ssoa/spfilessoa.ora' from pfile='/tmp/p.ora';

Remove the default file.

$rm /u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/spfilesoa1.ora

Create an initsoan.ora file on all the standby hosts. The file needs to point to the location of the SPFILE created in step 2.
In the soadc2.dbhost1 file, add the following:
```
cat /u01/app/oracle/product/11.2.0/dbs/initsoa1.ora
spfile='+DATA/ssoa/spfilesoa.ora
```
On the standby system, restart the instances in mount state:
```
startup mount
```

Register the RAC database with CRS as follows:

srvctl add database -d ssoa –o /u01/app/oracle/product/11.2.0/db_1
srvctl add instance -d ssoa -i soa1 -n soadc2.dbhost1
srvctl add instance -d ssoa -i soa2 -n soadc2.dbhost2
srvctl modify database –d ssoa –r physical_standby

Parent topic: Installing and Configuring Oracle Database 11.2 or 12.1 MAA Environments

4.1.3.1.5 Creating a Data Guard Broker Configuration

This section describes the basic steps for creating a Data Guard configuration.

For complete information about Data Guard Broker, see the Oracle Data Guard Installation guide.

To create a Data Guard Broker configuration, complete the following steps:

Add the values of static SID to the local node file, listener.ora, that is located in the grid infrastructure home on all hosts in the configuration:

LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2))))
LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3))))
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))) 

LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON 
SID_LIST_LISTENER = 
(SID_LIST = 
(SID_DESC = 
(GLOBAL_DBNAME = psoa_DGMGRL) 
(SID_NAME = soa1)
(ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1)))
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON

Note:

Ensure that you enter the appropriate values for the GLOBAL_DBNAME and SID_NAME parameters, for each host.
If IFILE is used for the shared ORACLE_HOME, modify the corresponding local listener.ora file that is specified in IFILE.

Bounce the listeners on the primary site and standby site to apply the modifications made to the files:
```
srvctl stop listener
srvctl start listener
```

Configure the Data Guard Broker metadata files, and enable the broker on the primary site and standby site as follows:

On the primary site

SQL> alter system set dg_broker_config_file1='+DATA_prmy/psoa/dr1.dat' scope=both;
SQL> alter system set dg_broker_config_file2='+DATA_prmy/psoa/dr2.dat' scope=both;
SQL> alter system set dg_broker_start=true scope=both;

On the standby site

SQL> alter system set dg_broker_config_file1='+DATA_stby/ssoa/dr1.dat' scope=both;
SQL> alter system set dg_broker_config_file2='+DATA_stby/ssoa/dr2.dat' scope=both;
SQL> alter system set dg_broker_start=true scope=both;

Access dgmgrl on the primary host, and create the configuration. See Example 4-9.

Example 4-9 Accessing dgmgrl to Create the Data Guard Broker Configuration on Primary Host

dgmgrl sys/password
 
DGMGRL> create configuration 'dg_config' as 
primary database is 'psoa'
connect identifier is psoa;
 
Configuration "dg_config" created with primary database "psoa"
 
DGMGRL> add database 'ssoa' as
connect identifier is ssoa;
 
Database "ssoa" added
 
DGMGRL> enable configuration;
 
Enabled.

Verifying the Data Guard Broker Configuration
Testing Database Switchover and Switchback
You can perform a database switchover and switchback.

Parent topic: Installing and Configuring Oracle Database 11.2 or 12.1 MAA Environments

4.1.3.1.5.1 Verifying the Data Guard Broker Configuration

Complete the following steps to verify that the Data Guard Broker configuration was created successfully.

Verify the Oracle Data Guard configuration by querying the V$ARCHIVED_LOG view to identify existing files in the archived redo log. For example:

SQL> SELECT SEQUENCE#, FIRST_TIME, NEXT_TIME 
  2> FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;

SEQUENCE# FIRST_TIME NEXT_TIME
---------- ------------------ ------------------
         8 11-JUL-13 17:50:45 11-JUL-13 17:50:53
         9 11-JUL-1317:50:53 11-JUL-1317:50:58
        10 11-JUL-13 17:50:58 11-JUL-13 17:51:03
 
3 rows selected

On the primary database, issue the following SQL statement to force a log switch and archive the current online redo log file group:
```
SQL> alter system archive log current;
```

On the standby database, query the V$ARCHIVED_LOG view to verify that the redo data was received and archived on the standby database:

SQL> SELECT SEQUENCE#, FIRST_TIME, NEXT_TIME 
  2> FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;

SEQUENCE# FIRST_TIME NEXT_TIME
---------- ------------------ ------------------
         8 11-JUL-13 17:50:45 11-JUL-13 17:50:53
         9 11-JUL-1317:50:53 11-JUL-1317:50:58
         10 11-JUL-13 17:50:58 11-JUL-13 17:51:03
         11 11-JUL-13 17:51:03 11-JUL-1318:34:11

4 rows selected

Use the show configuration command to verify that the configuration was created successfully on the standby site. See Example 4-10.

Example 4-10 Verifying the Data Guard Broker Configuration

DGMGRL> show configuration;

Configuration - dg_config

Protection Mode: MaxPerformance
Databases:
psoa- Primary database
ssoa- Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

Parent topic: Creating a Data Guard Broker Configuration

4.1.3.1.5.2 Testing Database Switchover and Switchback

You can perform a database switchover and switchback.

Performing a Switchover Operation by Using Oracle Data Guard Broker

To perform a switchover operation by using Oracle Data Guard Broker, complete the following tasks:

Verify the Oracle Data Guard Broker configuration that you created by using the instructions provided in Creating a Data Guard Broker Configuration.

To verify the configuration, run the following command:

DGMGRL> show configuration;
 
Configuration - dg_config
 
Protection Mode: MaxPerformance
Databases:
psoa- Primary database
ssoa- Physical standby database
 
Fast-Start Failover: DISABLED
 
Configuration Status:
SUCCESS

Swap the roles of the primary and standby databases by running the SWITCHOVER command. **INTERNAL XREF ERROR** shows how Data Guard Broker automatically shuts down and restarts the old primary database as part of the switchover operation.

DGMGRL> switchover to 'ssoa';
Performing switchover NOW, please wait...
New primary database "ssoa" is opening...
Operation requires shutdown of instance "psoa1" on database "psoa"
Shutting down instance "psoa1"...
ORA-01109: database not open
 
Database dismounted.
ORACLE instance shut down.
Operation requires startup of instance "psoa1" on database "psoa"
Starting instance "psoa1"...
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is "ssoa"

After the switchover is complete, use the SHOW CONFIGURATION command to verify that the switchover operation was successful:

DGMGRL> show configuration;
Configuration - dg_config
Protection Mode: MaxPerformance
Databases:
ssoa- Primary database
psoa- Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS

Performing a Switchover Operation Using SQL Plus

Perform the following operations to switchover or switchback databases correctly between the newly created physical standby database and the primary Oracle RAC databases:

Shut down all but one instance of the Oracle RAC databases (PSOA) on the primary site. For example, run the following command on SOADC1.DBHOST1 of the production site:
```
srvctl stop instance -d psoa -i soa2
```
Initiate the role transition to the physical standby on the current primary database. For example, run the following command on SOADC1.DBHOST1 of the production site:
```
SQL > ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY WITH SESSION SHUTDOWN;
```
Shut down the primary instance and mount the primary instance. For example, run the following command on SOADC1.DBHOST1 of the production site:
```
SQL > shutdown immediate
SQL > startup mount
```
At this point, both the databases are in Physical Standby mode.

Sign in to the instances to continue.
To verify that both the databases are in Physical Standby mode, run this SQL query on both the databases:
```
SQL> select database_role from v$database;
DATABASE_ROLE
----------------
PHYSICAL_STANDBY
```
Switch the physical standby database role to the primary role. For example, run the following command on SOADC2.DBHOST1 of the standby site:
```
SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WITH SESSION SHUTDOWN;
```
Now the physical standby database is the new primary database.
Shut down the new primary database and start up both the RAC nodes by using srvctl. For example, run the following command on the new primary site SOADC2.DBHOST1:
```
srvctl start database -d ssoa
```
On the new physical standby database (the old primary) start the managed recovery of the database. For example, run the following command on SOADC1.DBHOST1 of the primary site:
```
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;
```
Start sending the redo data to the new physical standby database. For example, run the following command on the new primary site SOADC2.DBHOST1:
```
SQL> ALTER SYSTEM SWITCH LOGFILE;
```
Check the new physical standby database to see if it is receiving the archive log files by querying the V$ARCHIVED_LOG view.
Perform a switchback operation. A switchback operation is a subsequent switchover operation to return the roles to their original state.

Note:

For information about switchover and failover operation of Oracle Data Guard Broker, see Switchover and Failover Operations in the Oracle Data Guard Broker.

Parent topic: Creating a Data Guard Broker Configuration

4.2 Creating a Production Site

Learn how to create a production site on an Oracle SOA enterprise deployment topology.

Before you create your production site:

Set up the host name aliases for the middle tier hosts, as described in Planning Host Names.
Create the required volumes on the shared storage on the production site, as described in Designing Directory Structure and Volumes.
Determine the Oracle Data Guard configuration to use based on the data loss requirements of the database and network considerations, such as the available bandwidth and latency when compared to the redo generation.

This section includes the following topics:

Creating a Production Site
Create a production site for a topology.
Configuring Data Sources for Oracle Fusion Middleware Active-Passive Deployment
Configure the data sources that Oracle Fusion Middleware uses to automate connections failover, in case of a failover or switchover of the primary database.

Parent topic: Setting Up and Managing Disaster Recovery Sites

4.2.1 Creating a Production Site

Create a production site for a topology.

Note:

This section provides information about how to create a production site for the Oracle SOA Suite topology. If you plan to create a production site for a different topology, see the appropriate Oracle Fusion Middleware Enterprise Deployment Guide listed under the Install a Production Environment: Plan, Install & Configure an Enterprise Deployment category.

Install and configure your production site as described in the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.

The following sections describe how to complete the installation and configuration of your production site:

Creating Volumes and Consistency Groups
Create volumes and consistency groups on the shared storage device.
Setting Up Physical Host Names and Alias Host Names
Set up physical host names on the production site and physical host names and alias host names on the standby site.
Installing and Configuring Oracle SOA Suite
Install and configure Oracle SOA Suite.

Parent topic: Creating a Production Site

4.2.1.1 Creating Volumes and Consistency Groups

Create volumes and consistency groups on the shared storage device.

To create volumes and consistency groups on the shared storage device, see Recommended Volume Design for Oracle SOA Suite.

Parent topic: Creating a Production Site

4.2.1.2 Setting Up Physical Host Names and Alias Host Names

Set up physical host names on the production site and physical host names and alias host names on the standby site.

For information about planning host names for the production and standby sites, see Planning Host Names.

Parent topic: Creating a Production Site

4.2.1.3 Installing and Configuring Oracle SOA Suite

Install and configure Oracle SOA Suite.

To install and configure Oracle SOA Suite, see Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite and apply the following modifications:

Install the Oracle SOA Suite components into the volumes created on the shared storage device.
Set up the production and standby sites by using the aliases to the physical and virtual host names.
Create a separate volume on each site for the JMS stores and transaction logs.
After the installation and configuration of the production site, turn off host name verification. See Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite for detailed instructions about turning off host name verification for an Administration Server and Managed Server.
If you do not plan on turning the host name verification off, follow the steps in Updating Self-Signed Certificates and Keys on the Standby Site to configure Node Manager communication.
Create SSL certificates by using the host name aliases on all of the Oracle Fusion Middleware hosts for proper Node Manager communication.

Parent topic: Creating a Production Site

4.2.2 Configuring Data Sources for Oracle Fusion Middleware Active-Passive Deployment

Configure the data sources that Oracle Fusion Middleware uses to automate connections failover, in case of a failover or switchover of the primary database.

Configure all the data sources that are used in the domain, including datasources used by jdbc persistence stores, leasing data sources, and custom datasources. The GridLink data sources must be modified to include information about the standby database.

Update the ONS configuration to include both production and standby site ONS.

The items in the list of ONS addresses must be separated by commas, as the following example illustrates:
```
prmy-scan:6200,stby-scan:6200
```
On the Test ONS Client Configuration page, review the connection parameters, and click Test All ONS Nodes. The following example illustrates a successful connection notification:
```
Connect test for prmy-scan:6200 succeeded.
```

Update the JDBC URL to include the appropriate services in both sites.

The following is a sample JDBC URL for the SOA Data source in an Oracle Fusion Middleware SOA Active or Passive configuration, where the database uses Data Guard.

jdbc:oracle:thin:@
(DESCRIPTION=
(CONNECT_TIMEOUT=15)
(RETRY_COUNT=5)
(RETRY_DELAY=5)
(ADDRESS_LIST=(LOAD_BALANCE=on)(ADDRESS=(PROTOCOL=TCP)(HOST=prmy-scan)(PORT=1521)))
(ADDRESS_LIST=(LOAD_BALANCE=on)(ADDRESS=(PROTOCOL=TCP)(HOST=stby-scan)(PORT=1521)))
(CONNECT_DATA=(SERVICE_NAME=soaedg.example.com))
)

In the Test GridLink Database Connection page, review the connection parameters, and click Test All Listeners. The following example illustrates a successful connection notification:

Connection test for jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=prmy-scan)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=soaedg.example.com))) succeeded.

Update the JDBC URL of the OPSS Security Stores to include the appropriate services in both sites.
Complete the following steps:
1. Sign in to APPHOST1 and go to the $ASERVER_HOME/config/fmwconfig folder.
2. Take a backup copy of the jps-config-jse.xml and jps-config.xml files.
3. Edit both files to update value of the property name="jdbc.url" with the appropriate JDBC URL used in the datasources.
4. Restart the Admin Server and the managed servers.
5. To verify that the JDBC URL is correctly updated, go to Enterprise Manager Console.
6. Navigate to WebLogic Domain > Security > Security Provider Configuration.
7. Expand Security Stores and verify that the Database URL is updated.
For BI only: The command synchronizes connection details to the mid-tier database and ensures that BI components can access the mid-tier database when connection details are changed. Location of the command:

DOMAIN_HOME/bitools/bin/sync_midtier_db.sh

Note:
On UNIX this must be performed on the primary host.
To synchronize the mid-tier database connection details:
- Execute the synchronisation script:
  
  DOMAIN_HOME/bitools/bin/sync_midtier_db.sh
- Script displays the data sources that are updated.
- Restart the Managed Server and BI system components.

Parent topic: Creating a Production Site

4.3 Creating a Standby Site

Learn how to create a standby site.

To create the standby site, the Oracle SOA enterprise deployment topology is used as an example.

This section includes the following topics:

Preparing the Standby Site
Prepare your standby site for operation.
Updating Self-Signed Certificates and Keys on the Standby Site
If you have enabled host name verification in the Oracle WebLogic Administration Server, update the appropriate trust and key stores with the certificates of the standby site.
Validating the Standby Site Setup
Validate your standby site.

Parent topic: Setting Up and Managing Disaster Recovery Sites

4.3.1 Preparing the Standby Site

Prepare your standby site for operation.

To prepare it for operation, on your standby site:

Set up the correct alias host names and physical host names by following the instructions in Planning Host Names.

Ensure that each standby site host has an alias host name that is the same as the physical host name of its peer host at the production site.
Create, on the shared storage, the same volumes that were created on the shared storage at the production site
Create the same mount points and symbolic links (if required) that you created at the production site. Note that symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes.

For more details about symbolic links, see Storage Replication.

Setting Up Middle Tier Hosts
Middle tier hosts on a standby site do not require installation nor configuration of Oracle Fusion Middleware or Oracle WebLogic Server software. When the production site storage is replicated to the standby site storage, the software installed on the production site is replicated at the standby site.

Parent topic: Creating a Standby Site

4.3.1.1 Setting Up Middle Tier Hosts

Middle tier hosts on a standby site do not require installation nor configuration of Oracle Fusion Middleware or Oracle WebLogic Server software. When the production site storage is replicated to the standby site storage, the software installed on the production site is replicated at the standby site.

To set up the middle tier hosts on the standby site:

Create a baseline snapshot copy of the shared storage on the production site, which sets up the replication between the storage devices. Create the initial baseline copy and subsequent snapshot copies by using asynchronous replication mode.
Synchronize the shared storage at the production site with the shared storage at the standby site. This transfers the initial baseline snapshot from the production site to the standby site.
Set up the frequency of subsequent copies of the production site shared storage, which is replicated at the standby site. When asynchronous replication mode is used, then at the requested frequency the changed data blocks at the production site shared storage (based on comparison to the previous snapshot copy) become the new snapshot copy, and the snapshot copy is transferred to the standby site shared storage.
After the baseline snapshot copy is performed, validate that all the directories inside the standby site volumes have the same content as the directories inside the production site volumes.

Parent topic: Preparing the Standby Site

4.3.2 Updating Self-Signed Certificates and Keys on the Standby Site

If you have enabled host name verification in the Oracle WebLogic Administration Server, update the appropriate trust and key stores with the certificates of the standby site.

Certificates must be specifically created for the nodes in the standby site.

For more information about creating certificates for nodes, see Enabling SSL Communication Between the Middle Tier and the Hardware Load Balancer in Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.

The examples in these sections show how to perform tasks for the Oracle SOA Suite enterprise topology shown in Figure 4-2.

Note:

When you set up the Oracle SOA Suite enterprise topology shown in Figure 4-2 as the production site for a Disaster Recovery topology, you must use the physical host names shown in Table 3-1 for the production site hosts instead of the host names shown in Figure 4-2.
The steps in this section must be performed on the application tier hosts on which Oracle WebLogic Server is installed.

This section includes the following topics:

Generating Self-Signed Certificates
Learn how to generate self-signed certificates.
Importing Certificates into a Key Store
Import certificates into your key store with the utils.ImportPrivateKey utility.
Creating a Trust Key Store
Create a trust key store and import certificates with the keytool utility.

Parent topic: Creating a Standby Site

4.3.2.1 Generating Self-Signed Certificates

Learn how to generate self-signed certificates.

To generate self-signed certificates:

Set up your environment by running the WL_HOME/server/bin/setWLSEnv.sh script:
In the Bourne shell, run the following command:
```
. setWLSEnv.sh
```
Verify that the CLASSPATH environment variable is set:
```
echo $CLASSPATH
```
Create a user-defined directory for the certificates.
```
mkdir home/user_projects/domains/SOADomain/certs
```

Change directory to the user-defined directory.

cd home/user_projects/domains/SOADomain/certs

Run the utils.CertGen tool from the user-defined directory to create the certificates for both the physical hostnames and the virtual hostnames used by the servers in the node:

java utils.CertGen key_passphrase cert_file_name key_file_name [export|domestic] [hostname]

For example:

java utils.CertGen password soadc1host1_cert soahost1_key domestic soahost1
java utils.CertGen password soadc1host2_cert soahost2_key domestic soahost2

Parent topic: Updating Self-Signed Certificates and Keys on the Standby Site

4.3.2.2 Importing Certificates into a Key Store

Import certificates into your key store with the utils.ImportPrivateKey utility.

To import certificates into a key store by using the utils.ImportPrivateKey utility:

Run the following utility to import the certificate and private key into the Identity Store, and make sure that you use a different alias for each of the certificate/key pairs imported:

java utils.ImportPrivateKey 
   keystore_file 
   keystore_password 
   certificate_alias_to_use private_key_passphrase 
   certificate_file private_key_file keystore_type

Note:

The default value for keystore_type is jks.

The following example illustrates the use of this utility:

java utils.ImportPrivateKey 
   appIdentityKeyStore.jks password
   appIdentity1 password
   ASERVER_HOME/certs/SOAHOST1.example.com_cert.pem 
   ASERVER_HOME/certs/SOAHOST1.example.com_key.pem

Repeat the preceding step for all the hosts on the primary and the standby sites.

Parent topic: Updating Self-Signed Certificates and Keys on the Standby Site

4.3.2.3 Creating a Trust Key Store

Create a trust key store and import certificates with the keytool utility.

To create a trust key store and to import certificates into it:

Copy the standard java key store to create the new trust key store since it already contains most of the root CA certificates needed. Oracle recommends not modifying the standard Java trust key store directly.
Copy the standard Java key store CA certificates located under the WL_HOME/server/lib directory to the same directory as the certificates, as illustrated in the following example:
```
cp $WL_HOME/server/lib/cacerts 
   $home/user_projects/domains/SOADomain/certs/appTrustKeyStore.jks
```

The default password for the standard Java key store is changeit. Oracle recommends that you change the default password by using the keytool utility:

keytool 
  -storepasswd 
  -new NewPassword 
  -keystore TrustKeyStore 
  -storepass OriginalPassword

The following example illustrates the use of this utility:

keytool 
  -storepasswd 
  -new passwrd 
  -keystore home//user_projects/domains/SOADomain/certs/appTrustKeyStore.jks 
  -storepass changeit

The CA certificate CertGenCA.der is used to sign all certificates that are generated by the utils.CertGen tool and is located at WL_HOME/server/lib directory. Import this CA certificate into the appTrustKeyStore by using the keytool utility:

keytool 
  -import 
  -v 
  -noprompt 
  -trustcacerts 
  -alias AliasName 
  -file CA_file_location 
  -keystore key_store_location 
  -storepass key_store_password

The following example illustrates the use of this utility to import certificates:

keytool 
  -import 
  -v 
  -noprompt 
  -trustcacerts 
  -alias clientCACert 
  -file $WL_HOME/server/lib/CertGenCA.der 
  -keystore appTrust.jks 
  -storepass password

Parent topic: Updating Self-Signed Certificates and Keys on the Standby Site

4.3.3 Validating the Standby Site Setup

Validate your standby site.

To validate a standby site:

Shut down any processes still running on the production site. These include the database instances in the data tier, Oracle Fusion Middleware instances, and any other processes in the application tier and web tier.
Stop the replication between the production site shared storage and the standby site shared storage.
Perform a switchback operation. A switchback operation is a subsequent switchover operation to return the roles to their original state.
On the standby site host, manually start all the processes. These include the database instances in the data tier, Oracle Fusion Middleware instances, and any other processes in the application tier and web tier.
Use a browser client to perform post-failover testing to confirm that requests are being resolved and redirected to the standby site.

Parent topic: Creating a Standby Site

4.4 Creating an Asymmetric Standby Site

Learn how to create an asymmetric Oracle Fusion Middleware Disaster Recovery topology.

An asymmetric topology is a disaster recovery configuration where the production and standby sites differ. In most asymmetric Disaster Recovery topologies, the standby site differs from the production site in that it has fewer resources than the production site.

Ensure that you understand how to set up a symmetric topology presented earlier in this document. Many of the concepts used to set up a symmetric topology are also valid when you set up an asymmetric topology.

This section include the following topics:

Creating an Asymmetric Standby Site
Create an asymmetric standby site for your Oracle Fusion Middleware Disaster Recovery topology.
Validating the Asymmetric Standby Site Setup
Validate your standby site.

Parent topic: Setting Up and Managing Disaster Recovery Sites

4.4.1 Creating an Asymmetric Standby Site

Create an asymmetric standby site for your Oracle Fusion Middleware Disaster Recovery topology.

The production site is assumed to be the Oracle SOA Suite enterprise deployment shown in Figure 4-2. An asymmetric standby site differs from the production site.

To create an asymmetric standby site:

Design the production site and the standby site. Determine the resources that are necessary at the standby site to ensure acceptable performance when the standby site assumes the production role.

Note:

The ports for the standby site instances must use the same port numbers as the peer instances at the production site. Therefore, ensure that all the port numbers that are required at the standby site are available (not in use at the standby site).
Create the Oracle Fusion Middleware Disaster Recovery production site by performing these operations:
1. Create volumes on the production site's shared storage system for the Oracle Fusion Middleware instances that are installed for the production site. See Designing Directory Structure and Volumes.
2. Create mount points and symbolic links on the production site hosts to the Oracle home directories for the Oracle Fusion Middleware instances on the production site's shared storage system volumes. Note that symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes.
  
  For more details about symbolic links, see Storage Replication.
  
  For more information about volume design, see Recommended Volume Design for Oracle SOA Suite.
3. Create mount points and symbolic links on the production site hosts to the Oracle Central Inventory directories for the Oracle Fusion Middleware instances on the production site's shared storage system volumes. Note that symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes.
  
  For more details about symbolic links, see Storage Replication.
  
  For more information about the Oracle Central Inventory directories, see Oracle Home and Oracle Inventory.
4. Create mount points and symbolic links on the production site hosts to the static HTML pages directories for the Oracle HTTP Server instances on the production site's shared storage system volumes, if applicable. Note that symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes.
  
  For more details about symbolic links, see Storage Replication.
5. Install the Oracle Fusion Middleware instances for the production site on the volumes in the production site's shared storage system. See Creating a Production Site.
Create the same volumes with the same file and directory privileges on the standby site's shared storage system as you created for the Oracle Fusion Middleware instances on the production site's shared storage system. This step is critical because it enables you to use storage replication later to create the peer Oracle Fusion Middleware instance installations for the standby site instead of installing them by using Oracle Universal Installer.

Note:

When you configure storage replication, ensure that all the volumes you set up on the production site's shared storage system are replicated to the same volumes on the standby site's shared storage system.

Even though some of the instances and hosts at the production site may not exist at the standby site, you must configure storage replication for all the volumes set up for the production site's Oracle Fusion Middleware instances.
Configure the shared storage to enable storage replication between the production site's shared storage system and the standby site's shared storage system. Configure storage replication to asynchronously copy the volumes in the production site's shared storage system to the standby site's shared storage system.
Create the initial baseline snapshot copy of the production site shared storage system to set up the replication between the production site and standby site shared storage systems. Create the initial baseline snapshot and subsequent snapshot copies by using the asynchronous replication mode. After the baseline snapshot copy is performed, validate that all the directories for the standby site volumes have the same contents as the directories for the production site volumes. Refer to the documentation for your shared storage vendor for information about creating the initial snapshot and enabling storage replication between the production site and standby site shared storage systems.
After the baseline snapshot has been taken, perform these steps for the Oracle Fusion Middleware instances for the standby site hosts:
1. Set up a mount point directory on the standby site host to the Oracle home directory for the Oracle Fusion Middleware instance on the standby site's shared storage system. The mount point directory that you set up for the peer instance on the standby site host must be the same as the mount point directory that you set up for the instance on the production site host.
2. Set up a symbolic link on the standby site host to the Oracle home directory for the Oracle Fusion Middleware instance on the standby site's shared storage system. Note that symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes.
  
  For more details about symbolic links, see Storage Replication. The symbolic link that you set up for the peer instance on the standby site host must be the same as the symbolic link that you set up for the instance on the production site host.
3. Set up a mount point directory on the standby site host to the Oracle Central Inventory directory for the Oracle Fusion Middleware instance on the standby site's shared storage system. The mount point directory that you set up for the peer instance on the standby site host must be the same as the mount point directory that you set up for the instance on the production site host.
4. Set up a symbolic link on the standby site host to the Oracle Central Inventory directory for the Oracle Fusion Middleware instance on the standby site's shared storage system. Note that symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes.
  
  For more details about symbolic links, see Storage Replication. The symbolic link you set up for the peer instance on the standby site host must be the same as the symbolic link that you set up for the instance on the production site host.
5. Set up a mount point directory on the standby site host to the Oracle HTTP Server static HTML pages directory for the Oracle HTTP Server instance on the standby site's shared storage system. The mount point directory that you set up for the peer instance on the standby site host must be the same as the mount point directory that you set up for the instance on the production site host.
6. Set up a symbolic link on the standby site host to the Oracle HTTP Server static HTML pages directory for the Oracle HTTP Server instance on the standby site's shared storage system. Note that symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes; see Storage Replication for more details about symbolic links. The symbolic link that you set up for the peer instance on the standby site host must be the same as the symbolic link that you set up for the instance on the production site host.

At this point, the Oracle Fusion Middleware instance installations for the production site have been replicated to the standby site. At the standby site, all of the following are true:

The Oracle Fusion Middleware instances are installed into the same Oracle home directories on the same volumes as at the production site, and the hosts use the same mount point directories and symbolic links for the Oracle home directories as at the production site. Note that symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes; see Storage Replication for more details about symbolic links.
The Oracle Central Inventory directories are located in the same directories on the same volumes as at the production site, and the hosts use the same mount point directories and symbolic links for the Oracle Central Inventory directories as at the production site. Note that symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes; see Storage Replication for more details about symbolic links.
The Oracle HTTP Server static HTML pages directories are located in the same directories on the same volumes as at the production site, and the hosts use the same mount point directories and symbolic links for the Oracle HTTP Server static HTML pages directories as at the production site. Note that symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes; see Storage Replication for more details about symbolic links.
The same ports are used for the standby site Oracle Fusion Middleware instances as were used for the same instances at the production site.

To further create an asymmetric standby with fewer hosts and instances, see the following topic:

Creating an Asymmetric Standby Site with Fewer Hosts and Instances
Create an asymmetric standby site that has fewer hosts and Oracle Fusion Middleware instances than the production site.

Parent topic: Creating an Asymmetric Standby Site

4.4.1.1 Creating an Asymmetric Standby Site with Fewer Hosts and Instances

Create an asymmetric standby site that has fewer hosts and Oracle Fusion Middleware instances than the production site.

The production site for this Oracle Fusion Middleware Disaster Recovery topology is assumed to be the Oracle SOA Suite enterprise deployment shown in Figure 4-2. Setting Up a Site through Designing Directory Structure and Volumes describe how to set up this production site and the volumes for its shared storage system, and how to create the necessary mount points.

Figure 4-4 shows the asymmetric standby site for the production site shown in Figure 4-2.

Figure 4-4 An Asymmetric Standby Site with Fewer Hosts and Instances

Description of "Figure 4-4 An Asymmetric Standby Site with Fewer Hosts and Instances"

The Oracle SOA Suite asymmetric standby site shown in Figure 4-4 has fewer hosts and instances than the Oracle SOA Suite production site shown in Figure 4-2.

The hosts WEBHOST2 and SOAHOST2 and the instances on those hosts exist at the production site in Figure 4-2, but these hosts and their instances do not exist at the asymmetric standby site in Figure 4-4. The standby site therefore has fewer hosts and fewer instances than the production site.

It is important to ensure that this asymmetric standby site has sufficient resources to provide adequate performance when it assumes the production role.

When you follow the steps in Creating an Asymmetric Standby Site to set up this asymmetric standby site, the standby site should be properly configured to assume the production role.

To set up the asymmetric standby site correctly, create the same volumes and consistency groups on the standby site shared storage as you did on the production site shared storage. For example, for the Oracle SOA Suite deployment, the volume design recommendations in Table 4-1 and the consistency group recommendations in Table 4-2 were used to set up the production site shared storage. Use these same volume design recommendations and consistency group recommendations that you used for the production site shared storage to set up the asymmetric standby site shared storage.

Note that at an asymmetric standby site, some hosts that exist at the production site do not exist at the standby site. For example, the asymmetric standby site for Oracle SOA Suite shown in Figure 4-4, WEBHOST2 and SOAHOST2 do not exist; therefore, it is not possible or necessary for you to create mount points on these hosts to the standby site shared storage volumes.

Parent topic: Creating an Asymmetric Standby Site

4.4.2 Validating the Asymmetric Standby Site Setup

Validate your standby site.

To validate the standby site:

Shut down any processes running on the production site. These include the database instances in the data tier, Oracle Fusion Middleware instances, and any other processes in the application tier and web tier.
Stop the replication between the production site shared storage and the standby site shared storage.
Use Oracle Data Guard to switchover the databases.
On the standby site host, manually start all the processes. These include Oracle Fusion Middleware instances and any other processes in the application tier and web tier.
Use a browser client to perform post-failover testing to confirm that requests are being resolved and redirected to the standby site.

Parent topic: Creating an Asymmetric Standby Site

4.5 Performing Site Operations and Administration

Learn how to operate and administer your Oracle Fusion Middleware Disaster Recovery topology.

This section includes the following topics:

Synchronizing the Production and Standby Sites
Learn how to force a synchronization of the production and standby sites when you introduce a change in the middle tier at the production site.
Performing a Switchover
A switchover operation sets the standby site as the production role.
Performing a Switchback
A switchback operation reverts the roles of the current production and standby sites.
Performing a Failover
A failover operation sets the standby site as the production role when the production site becomes unavailable.
Testing the Standby Site
Learn how to create a clone of the read-only standby site shared storage and use it for testing the standby site.
Using Peer-To-Peer File Copy for Testing
Use the utility rsync in test environments to replicate the middle tier file system data from a production site host to a standby site peer host in an Oracle Fusion Middleware Disaster Recovery topology.

Parent topic: Setting Up and Managing Disaster Recovery Sites

4.5.1 Synchronizing the Production and Standby Sites

Learn how to force a synchronization of the production and standby sites when you introduce a change in the middle tier at the production site.

In normal operations, the standby site shared storage receives snapshots transferred periodically from the production site shared storage. After the snapshots are applied, the standby site shared storage includes all the data up to and including the data contained in the last snapshot transferred from the production site before the failover or switchover.

Be sure to force a synchronization when you introduce a change to the middle tier at the production site such as, for example, when you deploy a new application at the production site. Follow the vendor-specific instructions to force a synchronization by using the storage replication technology.

The databases synchronization in an Oracle Fusion Middleware Disaster Recovery topology is managed by Oracle Data Guard.

Parent topic: Performing Site Operations and Administration

4.5.2 Performing a Switchover

A switchover operation sets the standby site as the production role.

This operation is needed when you plan to take down the production site (for example, to perform maintenance) and to make the current standby site the production site.

To perform a switchover operation:

Shut down any processes running on the production site. These include the database instances in the data tier, Oracle Fusion Middleware instances, and any other processes in the application tier and web tier.
Stop the replication between the production site shared storage and the standby site shared storage.
Unmount the shared storage volume with the middle tier artifacts, on the current production site, and mount the corresponding volumes on the current standby site, which is the new production site.
Use Oracle Data Guard to switch over the databases.
On the standby site host, manually start all the processes. These include the database instances in the data tier, Oracle Fusion Middleware instances, and any other processes in the application tier and web tier.
Ensure that all user requests are routed to the standby site by performing a global DNS push or something similar, such as updating the global load balancer.
Use a browser client to perform post-switchover testing to confirm that requests are being resolved and redirected to the standby site.

At this point, the former standby site is the new production site and the former production site is the new standby site.
Reestablish the replication between the two sites, but configure the replication so that the snapshot copies go in the opposite direction (from the current production site to the current standby site). See the documentation for your shared storage to learn how to configure the replication so that snapshot copies are transferred in the opposite direction.

At this point, the former standby site becomes the new production site, and you can perform maintenance at the original production site. After you have carried out the maintenance of the original production site, you can use it as either the production site or the standby site.

Note:

This note is applicable for BI-specific systems only.

After a switchover operation, the creation of an Essbase cube with CDS may fail with an error similar to the following:

oracle.essbase.cds.util.CDSException: oracle.essbase.cds.util.CDSException: java.sql.SQLException: ORA-25153:

To work around this issue and to create an Essbase cube, on the current primary database:

Identify the temporary tablespaces by using a select statement similar to the following (where BIS17V1 is the Oracle Business Intelligence RCU prefix):

select username,temporary_tablespace from dba_users where username like 'BIS17V1%'

Assume that the above command returns the following list of temporary tablespaces:

USERNAME.....TEMPORARY_TABLESPACE

BIS17V1_IAU_VIEWER.....BIS17V1_IAS_TEMP

BIS17V1_STB.....BIS17V1_IAS_TEMP

BIS17V1_IAU_APPEND.....BIS17V1_IAS_TEMP

BIS17V1_MDS.....BIS17V1_IAS_TEMP

BIS17V1_IAU.....BIS17V1_IAS_TEMP

BIS17V1_BIPLATFORM......BIS17V1_IAS_TEMP

BIS17V1_OPSS.....BIS17V1_IAS_TEMP
After the switchover, drop the tablespace BIS17V1_IAS_TEMP including contents and datafiles.
Create the temporary tablespace BIS17V1_IAS_TEMP, as a tempfile, in the location (for example) /work/primy/oradata/stnby/BIS17V1_IAS_TEMP.dbf, with size 250 m.
Issue the following alter commands (here is where you use the list temporary tablespaces):

alter user BIS17V1_OPSS temporary tablespace BIS17V1_IAS_TEMP ;

alter user BIS17V1_BIPLATFORM temporary tablespace BIS17V1_IAS_TEMP ;

alter user BIS17V1_IAU temporary tablespace BIS17V1_IAS_TEMP ;

alter user BIS17V1_MDS temporary tablespace BIS17V1_IAS_TEMP ;

alter user BIS17V1_IAU_APPEND temporary tablespace BIS17V1_IAS_TEMP ;

alter user BIS17V1_STB temporary tablespace BIS17V1_IAS_TEMP ;

alter user BIS17V1_IAU_VIEWER temporary tablespace BIS17V1_IAS_TEMP ;

To use the original production site as the production site, perform a switchback as explained in Performing a Switchback.

Parent topic: Performing Site Operations and Administration

4.5.3 Performing a Switchback

A switchback operation reverts the roles of the current production and standby sites.

To perform a switchback operation:

Shut down any processes running on the current production site. These include the database instances in the data tier, Oracle Fusion Middleware instances, and any other processes in the application tier and web tier.
Stop the replication between the current production site shared storage and standby site shared storage.
Unmount the shared storage volume with the middle tier artifacts, on the current production site, and mount the corresponding volumes on the current standby site, which is the new production site.
Use Oracle Data Guard to switch back the databases.
On the new production site hosts, manually start all the processes. These include Oracle Fusion Middleware instances and any other processes in the application tier and web tier.
Ensure that all user requests are routed to the new production site by performing a global DNS push or something similar, such as updating the global load balancer.
Use a browser client to perform post-switchback testing to confirm that requests are being resolved and redirected to the new production site.

At this point, the former standby site is the new production site and the former production site is the new standby site.
Reestablish the replication between the two sites, but configure the replication so that the snapshot copies go in the opposite direction (from the new production site to the new standby site). See the documentation for your shared storage to learn how to configure the replication so that snapshot copies are transferred in the opposite direction.

Parent topic: Performing Site Operations and Administration

4.5.4 Performing a Failover

A failover operation sets the standby site as the production role when the production site becomes unavailable.

To perform a failover operation:

Stop the replication between the production site shared storage and the standby site shared storage.
Mount the shared storage volume with the middle-tier artifacts, on the current standby site, which is the new production site.
From the standby site, use Oracle Data Guard to fail over the databases.
On the standby site hosts, manually start all the processes. These include the Oracle Fusion Middleware instances and any other processes in the application tier and web tier.
Ensure that all user requests are routed to the standby site by performing a global DNS push or something similar, such as updating the global load balancer.
Use a browser client to perform post-failover testing to confirm that requests are being resolved and redirected to the production site.

At this point, the standby site is the new production site. You can examine the issues that caused the former production site to become unavailable.
To use the original production site as the current standby site, you must reestablish the replication between the two sites, but configure the replication so that the snapshot copies go in the opposite direction (from the current production site to the current standby site). See the documentation for your shared storage system to learn how to configure the replication so that snapshot copies are transferred in the opposite direction.

Note:

After a failover operation, the creation of an Essbase cube with CDS may fail with an error similar to the following:

oracle.essbase.cds.util.CDSException: oracle.essbase.cds.util.CDSException: java.sql.SQLException: ORA-25153:

To work around this issue and to create an Essbase cube, on the current primary database:

Identify the temporary tablespaces by using a select statement similar to the following, where BIS17V1 is the Oracle Business Intelligence RCU prefix):

select username,temporary_tablespace from dba_users where username like 'BIS17V1%'

Assume that the above command returns the following list of temporary tablespaces:

USERNAME.....TEMPORARY_TABLESPACE

BIS17V1_IAU_VIEWER.....BIS17V1_IAS_TEMP

BIS17V1_STB.....BIS17V1_IAS_TEMP

BIS17V1_IAU_APPEND.....BIS17V1_IAS_TEMP

BIS17V1_MDS.....BIS17V1_IAS_TEMP

BIS17V1_IAU.....BIS17V1_IAS_TEMP

BIS17V1_BIPLATFORM......BIS17V1_IAS_TEMP

BIS17V1_OPSS.....BIS17V1_IAS_TEMP
After the failover, drop the tablespace BIS17V1_IAS_TEMP including contents and datafiles.
Create the temporary tablespace BIS17V1_IAS_TEMP, as a tempfile, in location. For example, /work/primy/oradata/stnby/BIS17V1_IAS_TEMP.dbf with size 250 m.
Issue the following alter commands (here is where you use the list temporary tablespaces):

alter user BIS17V1_OPSS temporary tablespace BIS17V1_IAS_TEMP ;

alter user BIS17V1_BIPLATFORM temporary tablespace BIS17V1_IAS_TEMP ;

alter user BIS17V1_IAU temporary tablespace BIS17V1_IAS_TEMP ;

alter user BIS17V1_MDS temporary tablespace BIS17V1_IAS_TEMP ;

alter user BIS17V1_IAU_APPEND temporary tablespace BIS17V1_IAS_TEMP ;

alter user BIS17V1_STB temporary tablespace BIS17V1_IAS_TEMP ;

alter user BIS17V1_IAU_VIEWER temporary tablespace BIS17V1_IAS_TEMP ;

To again use the original production site as the production site, perform a switchback as explained in Performing a Switchback.

Parent topic: Performing Site Operations and Administration

4.5.5 Testing the Standby Site

Learn how to create a clone of the read-only standby site shared storage and use it for testing the standby site.

A typical Oracle Fusion Middleware Disaster Recovery configuration uses:

Storage replication to copy Oracle Fusion Middleware middle tier file systems and data from the production site shared storage to the standby site shared storage. During normal operation, the production site is active and the standby site is passive. When the production site is active, the standby site is passive and the standby site shared storage is in read-only mode; the only write operations made to the standby site shared storage are the storage replication operations from the production site shared storage to the standby site shared storage.
Oracle Data Guard to copy database data for the production site Oracle databases to the standby databases at standby site. By default, the production site databases are active and the standby databases at the standby site are passive. The standby databases at the standby site are in Managed Recovery mode while the standby site is in the standby role (is passive). When the production site is active, the only write operations made to the standby databases are the database synchronization operations performed by Oracle Data Guard.
The standby site as the production role when the production site becomes unavailable. If the current production site becomes unavailable unexpectedly, then a failover operation (described in Performing a Failover) is performed to enable the standby site to assume the production role. Or, if the current production site is taken down intentionally (for example, for planned maintenance), then a switchover operation (described in Performing a Switchover) is performed to enable the standby site to assume the production role.

The usual method of testing a standby site is to shut down the current production site and perform a switchover operation to enable the standby site to assume the production role. However, some enterprises may want to perform periodic testing of their Disaster Recovery standby site without shutting down the current production site and performing a switchover operation.

Another method to test the standby site is to create a clone of the read-only standby site shared storage and then use the cloned standby site shared storage in testing.

To use this alternate testing method:

Use the cloning technology provided by the shared storage vendor to create a clone of the standby site's read-only volumes on the shared storage at the standby site. Ensure that the cloned standby site volumes are writable. If you want to test the standby site just once, then this can be a one-time clone operation. However, if you want to test the standby site regularly, you can set up periodic cloning of the standby site read-only volumes to the standby site's cloned read/write volumes.
Perform a backup of the standby site databases, then modify the Oracle Data Guard replication between the production site and standby site databases.
- For the 12c and later databases, complete the following steps to establish a snapshot standby database:
  1. Cancel Redo Apply, if active.
  2. Ensure that the database is mounted, but not open.
  3. Ensure that a fast recovery area has been configured. It is not necessary for flashback database to be enabled.
    
    To perform the conversion, run the following SQL command:
```
SQL> ALTER DATABASE CONVERT TO SNAPSHOT STANDBY;
```
  4. To open the snapshot standby in read/write mode, run following SQL command:
```
SQL> ALTER DATABASE OPEN READ WRITE;
```
- For the 11g database, complete the following steps to establish a snapshot standby database:
  1. Cancel Redo Apply, if active.
  2. Ensure that the database is mounted, but not open.
  3. To perform the conversion, run the following SQL command:
```
SQL> ALTER DATABASE CONVERT TO SNAPSHOT STANDBY;
```
    The database is dismounted after the conversion is complete.
  4. Restart the database.
Use Oracle Data Guard database recovery procedures to bring the standby databases online.
On the standby site computers, modify the mount commands to point to the volumes on the standby site's cloned read/write shared storage by following these steps:
1. Unmount the read-only shared storage volumes.
2. Mount the cloned read/write volumes at the same mount point.
Before you test the standby site, modify the host name resolution method for the computers that are used to perform the testing to ensure that the host names point to the standby site computers and not the production site computers. For example, on a Linux computer, change the /etc/hosts file to point to the virtual IP of the load balancer for the standby site.
Perform the standby site testing.

After you complete the standby site testing, follow these steps to begin using the original production site as the production site again:

Modify the mount commands on the standby site computers to point to the volumes on the standby site's read-only shared storage. In other words, reset the mount commands back to what they were before the testing was performed.
1. Unmount the cloned read/write shared storage volume.
2. Mount the read-only shared storage volumes.
At this point, the mount commands are reset to what they were before the standby site testing was performed.
Configure Oracle Data Guard to perform replication between the production site databases and standby databases at the standby site. Performing this configuration puts the standby database into Managed Recovery mode again:
- For Oracle Database 10.2 and later, follow these steps:
  1. Revert the activated database back to a physical standby database:
```
SQL> STARTUP MOUNT FORCE;
SQL> FLASHBACK DATABASE TO POINT standbytest;
SQL> ALTER DATABASE CONVERT TO PHYSICAL STANDBY;
SQL> STARTUP MOUNT FORCE;
```
  2. Restart managed recovery:
```
SQL> ALTER DATABASE RECOVER MANAGED STANDBY
     DATABASE USING CURRENT LOGFILE DISCONNECT;
```
  3. Reenable the standby destination and switch logs:
```
SQL> ALTER SYSTEM SET
     LOG ARCHIVE DEST STATE 2=ENABLE;
```
- For Oracle Database 12c, set up the replication again by following the steps in Managing Physical and Snapshot Standby Databases .
Before you use the original production site again, modify the host name resolution method for the computers that are used to access the production site to ensure that the host names point to the production site computers and not the standby site computers. For example, on a Linux computer, change the /etc/hosts file to point to the virtual IP of the load balancer for the production site.

Parent topic: Performing Site Operations and Administration

4.5.6 Using Peer-To-Peer File Copy for Testing

Use the utility rsync in test environments to replicate the middle tier file system data from a production site host to a standby site peer host in an Oracle Fusion Middleware Disaster Recovery topology.

An alternative to replicating middle tier components in test environments is to use the rsync utility (which uses peer-to-peer file copy) to replicate middle tier file system data from a production site host to a standby site peer host in an Oracle Fusion Middleware Disaster Recovery topology. The use of the rsync utility is explained in the context of symmetric topologies.

Ensure that you are familiar with storage replication and Oracle Data Guard in an Oracle Fusion Middleware Disaster Recovery topology, because there are many similarities between using storage replication and using the rsync utility for disaster protection and disaster recovery of your Oracle Fusion Middleware components.

Note:

Note the following important differences between using storage replication technologies and using the rsync utility to replicate middle tier file systems:

When you use storage replication, you can roll changes back to the point in time when any previous snapshot was taken at the production site.

When you use the rsync utility, replicated production site data overwrites the standby site data, and you cannot roll back a replication.
When you use storage replication, the volume that you set up for each host cluster in the shared storage systems ensures data consistency for that host cluster across the production site's shared storage system and the standby site's shared storage system.

When you use the rsync utility, data consistency is not guaranteed.

Because of these differences, the rsync utility is not supported in production environments but in test environments only.

This section includes the following topic:

Using rsync and Oracle Data Guard in Oracle Fusion Middleware Disaster Recovery Topologies
Learn how to use the UNIX rsync utility and Oracle Data Guard in your Oracle Fusion Middleware Disaster Recovery topology.

Parent topic: Performing Site Operations and Administration

4.5.6.1 Using rsync and Oracle Data Guard in Oracle Fusion Middleware Disaster Recovery Topologies

Learn how to use the UNIX rsync utility and Oracle Data Guard in your Oracle Fusion Middleware Disaster Recovery topology.

Note:

For information about how to set up Oracle Data Guard for Oracle database, see Database Considerations.

The following sections describe how to use the rsync utility and Oracle Data Guard to protect and force synchronization between your production and standby sites in an Oracle Fusion Middleware Disaster Recovery topology:

Using rsync for Oracle Fusion Middleware Middle Tier Components
Use the UNIX rsync utility to protect and recover your Oracle Fusion Middleware middle tier components.
Performing Failover and Switchover Operations
Learn how to perform failover or switchover operations when you use the rsync utility.

Parent topic: Using Peer-To-Peer File Copy for Testing

4.5.6.1.1 Using rsync for Oracle Fusion Middleware Middle Tier Components

Use the UNIX rsync utility to protect and recover your Oracle Fusion Middleware middle tier components.

To use the rsync utility:

Install and set up rsync to enable replication of files from a production site host to its standby site peer host. For instructions about how to install, set up, and use the utility, see the utility man pages and also http://rsync.samba.org.
For each production site host on which one or more Oracle Fusion Middleware components has been installed, set up rsync to copy the following directories and files to the same directories and files on the standby site peer host:
- The Oracle home directory, subdirectories, and all the files in them.
- The Oracle Central Inventory directory and files for the host, which includes the Oracle Universal Installer entries for the Oracle Fusion Middleware installations.
- If applicable, the Oracle Fusion Middleware static HTML pages directory for the Oracle HTTP Server installations on the host.
- If applicable, the .fmb and .fmx deployment files that are created by Oracle Forms on the host, and the .rdf deployment artifact files that are created by Oracle Reports on the host.
  
  Note:
  
  Run the rsync utility as root. If you want it to work without prompting users for a password, set up SSH keys between the production site host and standby site host, so that SSH does not prompt for a password.
Set up scheduled jobs, for example, cron jobs, for the production site hosts for which you set up rsync in the previous step. These scheduled jobs enable rsync to automatically perform replication of these files from the production site hosts to the standby site hosts on a regular interval. An interval of once a day is recommended for a production site where the Oracle Fusion Middleware configuration does not change very often.
Whenever a change is made to the configuration of an Oracle Fusion Middleware middle tier configuration on a production site host (for example, when a new application is deployed), perform a manual synchronization of that host with its standby site peer host by using rsync.
Whenever you use rsync to a manually synchronize an Oracle Fusion Middleware middle tier instance on a production site host with the peer standby site host, also use Oracle Data Guard to synchronize database repositories in the production site with the databases in the standby site. For information about synchronizing Oracle databases with Oracle Data Guard, see Synchronizing Databases Manually.

Parent topic: Using rsync and Oracle Data Guard in Oracle Fusion Middleware Disaster Recovery Topologies

4.5.6.1.2 Performing Failover and Switchover Operations

Learn how to perform failover or switchover operations when you use the rsync utility.

To perform a failover or switchover from the production site to the standby site when you use rsync:

Shut down any processes running on the production site (if applicable).
Stop rsync jobs between the production site hosts and standby site peer hosts.
Use Oracle Data Guard to failover the production site databases to the standby site.
On the standby site, manually start the processes for the Oracle Fusion Middleware Server instances.
Route all user requests to the standby site by performing a global DNS push or something similar, such as updating the global load balancer.
Use a browser client to perform post-failover or post-switchover testing to confirm that requests are being resolved at the standby site (current production site).

At this point, the standby site is the new production site and the production site is the new standby site.
Reestablish rsync between the two sites, but configure it so that replications go now in the opposite direction (from the current production site to the current standby site).

To use the original production site as the new production site, perform the preceding steps again but configure the rsync replications to go in the original direction (from the original production site to the original standby site).

Parent topic: Using rsync and Oracle Data Guard in Oracle Fusion Middleware Disaster Recovery Topologies

4.6 Using Oracle Site Guard for Disaster Recovery

Oracle Site Guard orchestrates switchover and failover between two disaster recovery sites.

Oracle Site Guard:

Ensures high availability, data protection, and disaster recovery for enterprise data.
Performs Oracle Site Guard operations such as switchover and failover. If the primary site becomes unavailable due to a planned or an unplanned outage, a Switchover or Failover process needs to be initiated using Oracle Site Guard.

For more information about how to use Oracle Site Guard, see Oracle Site Guard Administrator's Guide.

Parent topic: Setting Up and Managing Disaster Recovery Sites

4.7 Patching an Oracle Fusion Middleware Disaster Recovery Site

Apply an Oracle Fusion Middleware patch set to upgrade the Oracle homes that participate in an Oracle Fusion Middleware Disaster Recovery site.

It is assumed that the Oracle Central Inventory for any Oracle Fusion Middleware instance that you are patching is located on the production site shared storage, so that the Oracle Central Inventory for the patched instance can be replicated to the standby site.

To apply an Oracle Fusion Middleware patch:

Perform a backup of the production site to ensure that the starting state is secured.
Apply the patch set to upgrade the production site instances.
After you apply the patch set, manually force a synchronization of the production site shared storage and standby site shared storage. This replicates the production site's patched instance and Oracle Central Inventory in the standby site's shared storage.
After you apply the patch set, use Oracle Data Guard to manually force a synchronization of the Oracle databases at the production site and standby sites. Because some Oracle Fusion Middleware patch sets make updates to repositories, this step ensures that any changes made to production site databases are synchronized to the standby site databases.
The upgrade is now complete. Your Disaster Recovery topology is ready to resume processing.

Note:

Patches must be applied only at the production site for an Oracle Fusion Middleware 12c Disaster Recovery topology. If a patch is for an Oracle Fusion Middleware instance or for the Oracle Central Inventory, the patch is copied when the production site shared storage is replicated to the standby site shared storage. A synchronization operation should be performed when a patch is installed at the production site.

Similarly, if a patch is installed for a production site database, Oracle Data Guard copies the patch to the standby database at the standby site when a synchronization is performed.

Parent topic: Setting Up and Managing Disaster Recovery Sites