3 Setting Up and Managing Disaster Recovery Sites

Learn about the Oracle SOA Suite enterprise deployment topology that illustrates how to set up production and standby sites.

Note:

You can automate disaster recovery operations such as switchover and failover by using Oracle Site Guard. See Oracle Site Guard Administrator's Guide.

This chapter includes the following sections:

Setting Up a Site

Learn how to set up an Oracle Disaster Recovery site.

Before you start creating the production site, ensure that you:

  • Set up the host name aliases for the middle tier hosts, as described in Planning Host Names.

  • Create the required volumes on the shared storage on the production site, as described in Designing Directory Structure and Volumes.

  • Determine the Oracle Data Guard configuration to use based on the data loss requirements of the database and network considerations, such as the available bandwidth and latency when compared to the redo generation.

This section includes the following topics:

Designing Directory Structure and Volumes

Learn about the recommended directory structure in your disaster recovery topology.

You can choose a directory layout different from the one recommended in this document, but the model adopted enables maximum availability, provides the best isolation of components and symmetry in the configuration, and facilitates backup and disaster recovery.

The following list describes directories and directory environment variables:

  • ORACLE_BASE: This environment variable and related directory path refers to the base directory below which Oracle products are installed.

  • ORACLE_HOME: This related directory path refers to the location where Oracle Fusion Middleware resides.

  • WL_HOME: This environment variable and related directory path contains installed files that are necessary to host an Oracle WebLogic Server.

  • PROD_DIR: This environment variable and related directory path refers to the location where a product suite, such as Oracle SOA Suite, Oracle WebCenter Portal, or Oracle Identity Management is installed.

  • DOMAIN directory: This directory path refers to the location where the Oracle WebLogic Domain information (configuration artifacts) is stored. Different WebLogic Servers can use different domain directories even when in the same node.

  • ORACLE_INSTANCE: An Oracle instance contains one or more system components. An Oracle instance directory contains updatable files, such as configuration files, log files, and temporary files.

See Recommended Directory Structure for Oracle SOA Suite.

This section includes the following topic:

Recommended Directory Structure for Oracle SOA Suite

Learn about the recommended directory structures for Oracle SOA Suite.

Oracle Fusion Middleware allows you to create multiple SOA Managed Servers from a single binary installation. This allows the installation of binary files in a single location on a shared storage and the reuse of this installation by the servers in different nodes. However, for maximum availability, Oracle recommends that you use redundant binary installations. In this model, two Oracle homes (each of which has a WL_HOME and an ORACLE_HOME for each product suite) are installed in a shared storage. Additional servers (when scaling out or up) of the same type can use either one of these two locations without requiring more installations. Ideally, users should use two different volumes for redundant binary location, this isolating the failures as much as possible in each volume. For additional protection, Oracle recommends using storage replication for these volumes. If multiple volumes are not available, Oracle recommends that you use mount points to simulate the same mount location in a different directory in the shared storage. Although this does not guarantee the protection that multiple volumes provide, it does allow protection from user deletions and individual file corruption.

Oracle also recommends separating the domain directory that is used by the Administration Server from the domain directory that is used by Managed Servers. This allows a symmetric configuration for the domain directories that is used by Managed Servers, and isolates the failover of the Administration Server. The domain directory for the Administration Server must reside in a shared storage to allow failover to another node with the same configuration. In addition, Oracle recommends that you place the Managed Servers' domain directories on a shared storage, although having them on the local file system is also supported. This is especially important when you design a production site with the disaster recovery site in mind. Figure 3-1 image represents the directory structure layout for Oracle SOA Suite.

Figure 3-1 Recommended Shared Storage Directory Structure for an Enterprise Deployment

Recommended Shared Storage Directory Structure for an Enterprise Deployment

Figure 3-2 Recommended Local Storage Directory Structure for an Enterprise Deployment

Recommended Local Storage Directory Structure for an Enterprise Deployment

For information about setting up this directory structure, see Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.

For volume design, see the following section:

Recommended Volume Design for Oracle SOA Suite

Learn about the recommended volume design for Oracle SOA Suite.

Figure 3-3 and Figure 3-4 shows an Oracle SOA Suite topology diagram. The volume design described in this section is for this Oracle SOA Suite topology. Detailed instructions for installing and configuring this topology are provided in the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.

Figure 3-3 Oracle SOA Suite and Oracle Business Activity Monitoring Enterprise Topology Diagram

Description of Figure 3-3 follows
Description of "Figure 3-3 Oracle SOA Suite and Oracle Business Activity Monitoring Enterprise Topology Diagram"

Figure 3-4 Oracle SOA Suite and Oracle Service Bus Enterprise Deployment Reference Topology Diagram

Description of Figure 3-4 follows
Description of "Figure 3-4 Oracle SOA Suite and Oracle Service Bus Enterprise Deployment Reference Topology Diagram"

For disaster recovery of this Oracle SOA Suite topology, Oracle recommends the following volume design:

  • Provision two volumes for two Oracle homes that contain the redundant product binary files (VOLFMW1 and VOLFMW2 in Table 3-1).

  • Provision one volume for the Administration Server domain directory (VOLADMIN in Table 3-1).

  • Provision one volume on each node for the Managed Server domain directory (VOLSOA1 and VOLSOA2 in Table 3-1). This directory is shared among all the Managed Servers on that node.

  • Provision one volume on each node for the Oracle HTTP Server Oracle home (VOLWEB1 and VOLWEB2 in Table 3-1).

  • Provision one volume on each node for the Oracle HTTP Server Domain Directory (VOLOHS1 and VOLOHS2 in Table 3-1).

    Note:

    For WebTier hosts, local storage is usually recommended. You can replicate this configuration on a regular basis to one of the other app tier volumes to sync to standby or directly from the production webhost to the standby webhost.

Table 3-1 provides a summary of Oracle recommendations for volume design for the Oracle SOA Suite topology shown in Figure 3-3 and Figure 3-4.

Table 3-1 Volume Design Recommendations for Oracle SOA Suite

Tier Volume Name Mounted on Host Mount Point Comments

Web

VOLWEB1

WEBHOST1

/u02/oracle/products/fmw

Volume for Oracle HTTP Server installation

Web

VOLWEB2

WEBHOST2

/u02/oracle/products/fmw

Volume for Oracle HTTP Server installation

Web

VOLOHS1

WEBHOST1

/u02/oracle/config/domains/ohs_domain

Volume for Oracle HTTP Server domain directory 

Web

VOLOHS2

WEBHOST2

/u02/oracle/config/domains/ohs_domain

Volume for Oracle HTTP Server domain directory 

Web

VOLSTATIC1

WEBHOST1

/u02/oracle/config/static

(Optional) Volume for static HTML content

Web

VOLSTATIC2

WEBHOST2

/u02/oracle/config/static

(Optional) Volume for static HTML content

Application

VOLFMW1

SOAHOST1

/u01/oracle/products/fmw

Volume for the WebLogic Server and Oracle SOA Suite binary files

Application

VOLFMW2

SOAHOST2

/u01/oracle/products/fmw

Volume for the WebLogic Server and Oracle SOA Suite binary files

Application

VOLADMIN

SOAHOST1, SOAHOST2

/u01/oracle/config

Volume for Administration Server domain directory and other shared configurations, such as Deployment Plans, applications, and keystores

Application

VOLSOA1

SOAHOST1

/u02/oracle/config

Volume for Managed Server domain directory

Application

VOLSOA2

SOAHOST2

/u02/oracle/config

Volume for Managed Server domain directory

Application

VOLRUNTIME

SOAHOST1, SOAHOST2

/u01/oracle/runtime

Volume for shared runtime content.

For example, files used by file adapters, MFT transfers, and other runtime artifacts.

Note:

It is recommended to store JMS messages and TLOGS in the database, using JDBC persistent stores, instead of this folder.

For consistency group recommendations, see:

Recommended Consistency Groups for Oracle SOA Suite

Learn about the recommended consistency groups for Oracle SOA Suite.

Oracle recommends the following consistency groups for the Oracle SOA Suite topology:

  • Create one consistency group with the volumes that contains the domain directories for the Administration Server and Managed Servers as members (DOMAINGROUP in Table 3-2).

  • Create one consistency group with the volume that contains the JMS file store and transaction log data as members (RUNTIMEGROUP in Table 3-2).

  • Create one consistency group with the volume that contains the Oracle homes as members (FMWHOMEGROUP in Table 3-2).

Table 3-2 provides a summary of Oracle recommendations for consistency groups for the Oracle SOA Suite topology as shown in Figure 3-3.

Table 3-2 Consistency Groups for Oracle SOA Suite

Tier Group Name Members Comments

Application

DOMAINGROUP

VOLADMIN

VOLSOA1

VOLSOA2

Consistency group for the Administration Server, and the Managed Server domain directory

Application

RUNTIMEGROUP

VOLRUNTIME

Consistency group for the shared runtime content

Application

FMWHOMEGROUP

VOLFMW1

VOLFMW2

Consistency group for the Oracle homes

Setting Up Storage Replication

Learn how to set up storage replication for the Oracle Fusion Middleware Disaster Recovery topology using Storage level replication, Rsync and Database file system.

Storage Level Replication

The Oracle Fusion Middleware Disaster Recovery solution uses storage replication technology for disaster protection of Oracle Fusion Middleware middle tier components.

To set up storage replication for the Oracle Fusion Middleware Disaster Recovery topology:

  • On the standby site, ensure that the alias host names that are created are the same as the alias host names that are used for the peer hosts at the production site.

  • On the shared storage at the standby site, create the same volumes as were created on the shared storage at the production site.

  • On the standby site, create the same mount points and symbolic links that you created at the production site.

    Note:

    • The symbolic links only need to be set up on the standby site if you set up symbolic links at the production site.

    • The symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes. For more information about symbolic links, see Setting Up Storage.

  • It is not necessary to install the same Oracle Fusion Middleware instances at the standby site as were installed at the production site. When the production site storage is replicated to the standby site storage, the Oracle software installed on the production site volumes are replicated at the standby site volumes.

  • Create the baseline snapshot copy of the production site shared storage that sets up the replication between the production site and standby site shared storage. Create the initial baseline copy and subsequent snapshot copies by using asynchronous replication mode. After the baseline snapshot copy is performed, validate that all the directories inside the standby site volumes have the same contents as the directories inside the production site volumes.

  • Set up the frequency of subsequent copies of the production site shared storage, which is replicated at the standby site. When asynchronous replication mode is used, then at the requested frequency the changed data blocks at the production site shared storage (based on comparison to the previous snapshot copy) become the new snapshot copy, and the snapshot copy is transferred to the standby site shared storage.

  • Ensure that disaster protection for any database that is included in the Oracle Fusion Middleware Disaster Recovery production site is provided by Oracle Data Guard. Do not use storage replication technology to provide disaster protection for Oracle databases.

  • The standby site shared storage receives snapshots transferred periodically from the production site shared storage. After the snapshots are applied, the standby site shared storage includes all the data up to and including the data contained in the last snapshot transferred from the production site before the failover or switchover.

  • Oracle strongly recommends that you manually force a synchronization operation whenever a change is made to the middle tier at the production site (for example, when a new application is deployed at the production site). Follow the vendor-specific instructions for forcing a synchronization by using storage replication technology.

Rsync

Alternatively, rsync can be used for replication. Rsync is a versatile copying tool, that can copy locally, or to/from another host over any remote shell. It offers a large number of options that control every aspect of its behavior, and permit very flexible specification of the set of files to be copied.

Rsync implements delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Due to these advantages and easy to use ability, it is a widely used tool for backups and mirroring.

Although rsync is reliable and implements implicit retries, the network outages and other connectivity issues can still cause failures in the file synchronization. Hence, rsync can be used as an alternative to the storage level replication under these these conditions:

  • When the storage replication is not possible or feasible and/or does not meet the cost requirements.
  • When primary and standby use a reliable and secure network connection for the copy.
  • When checks are performed in the copy to make sure that it is valid.
  • When the disaster recovery site is validated on a regular basis.

For more details about how to use rsync to replicate the file system artifacts, see Using Peer-To-Peer File Copy.

Oracle Database File System

Oracle Database File System (DBFS) is an additional method that can be used for replicating the config. Conceptually, a database file system is a file system interface placed on top of files and directories that are stored in database tables.

DBFS is similar to NFS in that it provides a shared network file system that looks like a local file system and has both a server component and a client component. The DBFS filesystem can be mounted from the mid-tier hosts and accessed as a regular shared filesystem. It could be used as an intermediate location to replicate content: any content that is copied to the DBFS mount, as it resides in the database, is automatically replicated to the standby site through the underlying Data Guard replication.

This method takes advantage of the robustness of the Data Guard replica. It has good availability through Oracle Driver’s retry logic and provides a resilient behavior. It can be used in scenarios with medium or high latencies between the data centers. However, using DBFS for configuration replication has also additional implications from the setup, database storage and lifecycle perspectives:

  • It introduces some complexity due to the configuration and maintenance required by the DBFS mount. It requires the DB client to be installed in the host that is going to mount it, it requires an initial setup of some artifacts in the database (tablespace, user, and so on) and in the client (the wallet, the tnsnames.ora, and so on. ). See the script dbfs_dr_setup_root.sh (it can be downloaded from this location) as an example script that installs the database client, creates the DBFS schema in the database, configures the client artifacts, and mounts the DBFS filesystem in a mid-tier host.
  • It requires additional capacity in the Database, because the content copied to the DBFS mount is stored in the database.
  • It is not recommended to store the domain configuration or the binaries directly in the DBFS mount. This would create a very strong dependency between the FMW files and the database. Instead, it is recommended to use the DBFS as an "assistance" file system: an intermediate staging file system to place the info that is going to be replicated to the standby site. Any replication to standby would have two steps: from the primary's origin folder to the intermediate DBFS mount, and then, in the standby site, from the DBFS mount to the standby's destination folder.
  • The DBFS can be mounted only if the database is open. When the Data Guard is not an Active Data Guard (ADG), the standby database is in mount state. Hence, in order to access to the DBFS mount in the standby site in such cases, the database needs to be converted to snapshot standby. When ADG is used, however, the file system can be mounted for reads and there is no need to transition to snapshot.

Due to the above stated reasons, it is not recommended to use this approach as a general purpose solution to replicate all the artifacts to the standby. For example, using DBFS to replicate the binaries is an overkill. However, this approach is suitable to replicate some dynamic artifacts during the lifecycle, like the domain shared configuration, when other methods like the storage replication or rsync are not feasible. See the Oracle WebLogic Server for Oracle Cloud Infrastructure Disaster Recovery paper to see an example on how to use DBFS to replicate the domain configuration.

Configuring Oracle Data Guard for the FMW Database

Learn how to install and configure Oracle Data Guard for the FMW Database in an Oracle SOA Suite enterprise deployment.

For recommendations and considerations for setting up Oracle databases that are used in an Oracle Fusion Middleware Disaster Recovery topology, see Database Considerations.

Oracle Maximum Availability Architecture (MAA) is Oracle's comprehensive architecture to reduce downtime for scheduled outages, and to prevent, detect and recover, from unscheduled outages.

Real Application Clusters (RAC) and Data Guard provide the basis of the database MAA solution, where the primary site contains the RAC database, and the secondary site contains the RAC physical standby database.

Tip:

Alternatively, you can perform many of the tasks in this section by using Oracle Enterprise Manager Cloud Control.

Setting up and managing databases using Cloud Control helps in controlling downtime and simplifies disaster recovery operations.

For information about Installing Enterprise Manager Cloud Control 13c, see Cloud Control Basic Installation Guide .

For more information about Setting up Oracle Data Guard using Cloud Control, see Provisioning Oracle Standby Databases in Oracle Enterprise Manager Lifecycle Management Administrator's Guide.

Prerequisites and Assumptions

Ensure that the following prerequisites are met:

  • The Oracle RAC cluster and Automatic Storage Management (ASM) instances on the standby site have been created.

  • The Oracle RAC databases on the standby site and the production site are using a flash recovery area.

  • The Oracle RAC databases are running in archivelog mode.

  • The database hosts on the standby site already have Oracle software installed.

  • In a shared ORACLE_HOME configuration, the TNS_ADMIN directory must be a local, non-shared directory.

Oracle Data Guard Environment Description

The examples given in this section contain environment variables as described in Table 3-3.

Table 3-3 Variables Used by Primary and Standby Databases

Variable Primary Database Standby Database

Database names

soa

soa

SOA Database Host Names

soadc1.dbhost1, soadc1.dbhost2

soadc2.dbhost1, soadc2.dbhost2

Database unique names

psoa

ssoa

Instance names

soa1, soa2

soa1, soa2

Service names

psoa, ssoa

psoa, ssoa

Procedure for Duplicating the Primary Database

Follow these steps to prepare the primary database for setting up Oracle Data Guard:

Note:

For information about prerequisites for setting up Oracle Data Guard, see Prerequisites in Oracle Data Guard Broker.

  1. Enable force logging on the primary database:

    SQL> alter database force logging;
    SQL> select name, log_mode, force_logging from v$database;
    NAME LOG_MODE FOR
    ---------- ---------------- -------
    PSOA ARCHIVELOG YES
    
  2. Create standby redo logs that are the same size as the online redo logs, on the primary database. The Duplicate command automatically creates the standby redo log on the standby database.

    Oracle recommends that you have the same number plus one additional redo logs for each thread. See, Example 3-8.

  3. In the standby node 1, create and start a listener (for example, LISTENER_DUPLICATE) that offers a static SID entry for the standby database, and has the same value of ORACLE_SID as the primary (soa1), as shown in Example 3-1.

    Provide the following path to the listener.ora file:

    GRID_HOME/network/admin/listener.ora
  4. Run the following command to start and verify the listeners by using lsnrctl from GRID_HOME/bin::

    lsnrctl start listener_duplicate
    lsnrctl status listener_duplicate
    
  5. In the standby node 2, create and start a listener (for example, LISTENER_DUPLICATE) that offers a static SID entry for the standby database, and has the same value of ORACLE_SID as the primary (soa2), as shown in Example 3-2.

    Provide the following path to the listener.ora file:

    GRID_HOME/network/admin/listener.ora
    

    Note:

    Listeners are configured at the cluster level, and all nodes inherit the port and environment settings of the listener. Therefore, the TNS_ADMIN directory path have the same values on all nodes.

    In a shared ORACLE_HOME configuration, the TNS_ADMIN directory must be a local, non-shared directory. These network files are included as IFILES.

    Complete the following steps to set up TNS_ADMIN for a shared ORACLE_HOME in a two-node cluster, SOADC1.DBHOST1, and SOADC1.DBHOST2, with respective instances of SOA1 and SOA2:

    1. Create a local network directory on each node. For example, /local_network_dir/network_admin.

    2. Create a local listener.ora file in the location: /local_network_dir/network_admin on each node.

    3. In the local listener.ora file, add the values for the LISTENER_duplicate parameter.

    4. In the common listener.ora file, in GRID_HOME/network/admin, add the IFILE parameter values, as follows:

      IFILE=/local_network_dir/network_admin/listener.ora
  6. Run the following command to start and verify the listeners:

    lsnrctl start listener_duplicate
    lsnrctl status listener_duplicate
    
  7. In the database home of the primary node, create an Oracle Net alias to connect to the listener that you created in step 3. See Example 3-3.

    For example: GRID_HOME/bin/tnsping dup.

  8. Configure the Oracle password file authentication for redo transport. Make sure it meets the following requirements:

    • Set remote_login_passwordfile to EXCLUSIVE.

      SQL> show parameter 
      remote_login_passwordfile
      NAME TYPE VALUE
      --------------------------------- -------------- --------------------------------
      remote_login_passwordfile string EXCLUSIVE
      
    • Copy the primary password file to the auxiliary instance location.

  9. In the ORACLE_HOME/dbs directory of the standby host, create a pfile initsoa1.ora, with the following parameters:

    db_name=soa
    db_unique_name=ssoa
    sga_target=5g
    
  10. Create the audit directory for the soa database on all standby hosts:

    mkdir -p /u01/app/oracle/admin/soa/adump
    
  11. Create an Oracle Net alias on all primary hosts to reach the ssoa database on the standby hosts.

    Ensure that all hosts have an Oracle Net alias for psoa and ssoa. All the aliases that you create need to reference the scan listener and not the VIP node. Also, if the local_listener variable is set to an alias on the primary host, then enter the details of the variable on the standby site, that point to the local listener on the primary host. See Example 3-4 and Example 3-5.

    Note:

    • For primary node 2 (SOADC1.DBHOST2), update the HOST value in psoa_local_listener to point to the VIP of primary node 2, as follows:

      psoa_local_listener = 
          (ADDRESS_LIST =
            (ADDRESS=(PROTOCOL = TCP)
      (HOST=prmy2-vip)(PORT = 1521))
          )
        )
    • If you are using a shared Oracle home, add the VIPs of the two nodes to the local_listener parameter.

      For example:

      psoa_local_listener =
        (DESCRIPTION =
          (ADDRESS_LIST =
            (ADDRESS=(PROTOCOL = TCP)
      (HOST=prmy1-vip)(PORT = 1521))
          (ADDRESS=(PROTOCOL = TCP)
      (HOST=prmy2-vip)(PORT = 1521))
          )
        )
    • For standby node 2 (SOADC2.DBHOST2), update the HOST value in ssoa_local_listener to point to the VIP of standby node 2, as follows:

      ssoa_local_listener = (ADDRESS_LIST =
            (ADDRESS=(PROTOCOL = TCP)
      (HOST=stby2-vip)(PORT = 1521))
          )
        )
      
    • If you are using a shared Oracle home, add the VIPs of the two nodes to the local_listener parameter.

      For example:

      ssoa_local_listener =
        (DESCRIPTION =
          (ADDRESS_LIST =
            (ADDRESS=(PROTOCOL = TCP)
      (HOST=stby1-vip)(PORT = 1521))
          (ADDRESS=(PROTOCOL = TCP)
      (HOST=stby2-vip)(PORT = 1521))
          )
        )
  12. On the standby host, set the ORACLE_SID to the same value as that of the primary database (ORACLE_SID=soa1), and run the startup nomount command on the standby database with standby PFILE, as created in step 9.

  13. To obtain the values use this query:

    SQL> select p.inst_id,instance_name, name,value from gv$parameter p, gv$instance i where p.inst_id=i.inst_id and p.name='cluster_interconnects'; 
    INST_ID INSTANCE_NAME NAME VALUE
    1 dbm1 cluster_interconnects 192.168.44.225
    2 dbm2 cluster_interconnects 192.168.44.226
  14. Disable the parameter cluster_interconnects on the primary host, if it is set:

    SQL> alter system reset cluster_interconnects scope=spfile sid='soa1';
    SQL> alter system reset cluster_interconnects scope=spfile sid='soa2';
    
  15. On the primary host, run the Recovery Manager (RMAN) script to duplicate the primary database by using the command duplicate target database for standby from active database.

    Note:

    This command varies depending on your environment. For information about how to use the command, see Duplicating Databases in Oracle Database Backup and Recovery User's Guide.

    See Example 3-6 to understand how to duplicate data between two systems with different disk-group names.

    See Example 3-7 to understand how to duplicate data between two systems that have the same disk-group name +DATA.

  16. If you have disabled the parameter cluster_interconnects on the primary host as described in step 14, then you must set it back to the original values in the spfile.

  17. (Optional) Stop and remove the listener that you created in step 3.

Example 3-1 Sample for Creating and Starting a Listener with Static SID

LISTENER_duplicate =
 (DESCRIPTION_LIST =
  (DESCRIPTION =
   (ADDRESS = (PROTOCOL = TCP)
   (HOST = soadc2.dbhost1)
   (PORT = 1521)(IP = FIRST))))
 
SID_LIST_LISTENER_duplicate =
 (SID_LIST =
  (SID_DESC =
   (SID_NAME = soa1)
   (ORACLE_HOME = /u01/app/oracle/product/19.0.0.0/dbhome_1)))

Example 3-2 Sample for Creating and Starting a Listener with Static SID

LISTENER_duplicate =
 (DESCRIPTION_LIST =
  (DESCRIPTION =
   (ADDRESS = (PROTOCOL = TCP)
   (HOST = soadc2.dbhost2)
   (PORT = 1521)(IP = FIRST))))

SID_LIST_LISTENER_duplicate =
 (SID_LIST =
  (SID_DESC =
   (SID_NAME = soa2)
   (ORACLE_HOME = /u01/app/oracle/product/19.0.0.0/dbhome_1)))

Example 3-3 Sample for Creating an Oracle Net Alias

dup =
(DESCRIPTION =
 (ADDRESS =
  (PROTOCOL = TCP)
   (HOST = soadc2.dbhost1)
   (PORT = 1521))
 (CONNECT_DATA =
  (SERVER = DEDICATED)
   (SID = soa1)))

Example 3-4 Sample tnsnames.ora File on Primary Node 1 (SOADC1.DBHOST1)

psoa =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL= TCP)
(HOST=prmy-scan)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = psoa)
    )
  )
 
ssoa =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL = TCP)
(HOST=stby-scan)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = ssoa)
    )
  )
 
psoa_local_listener = 
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL = TCP)
(HOST=prmy1-vip)(PORT = 1521))
    )
  )

Example 3-5 Sample tnsnames.ora File on Standby Node 1 (SOADC2.DBHOST1)

psoa =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL= TCP)(HOST=prmy-scan)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = psoa)
    )
  )
 
ssoa =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL = TCP)
(HOST=stby-scan)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = ssoa)
    )
  )
 
ssoa_local_listener = 
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS=(PROTOCOL = TCP)
(HOST=stby1-vip)(PORT = 1521))
    )
  )

Example 3-6 Duplicating Data Between Two Systems with Different Disk-Group Names

rman <<EOF
connect target sys/password;
connect auxiliary sys/password@dup;
run { 
allocate channel prmy1 type disk; 
allocate channel prmy2 type disk; 
allocate channel prmy3 type disk; 
allocate channel prmy4 type disk; 
allocate auxiliary channel stby type disk; 
duplicate target database for standby from active database 
spfile 
parameter_value_convert '+DATA_prmy','+DATA_stby','+RECO_prmy','+RECO_stby' 
set db_file_name_convert '+DATA_prmy','+DATA_stby' 
set db_unique_name='ssoa' 
set db_create_online_log_dest_1='+DATA_stby' 
set db_create_file_dest='+DATA_stby' 
set db_recovery_file_dest='+RECO_stby' 
set log_file_name_convert '+DATA_prmy','+DATA_stby','+RECO_prmy','+RECO_stby' 
set control_files='+DATA_stby/ssoa/standby.ctl' 
set local_listener='ssoa_local_listener'
set remote_listener='stby-scan:1521'; 
} EOF

Example 3-7 Duplicating Data Between Two Systems with Same Disk-Group Name +DATA

rman <<EOF
connect target sys/password;
connect auxiliary sys/password@dup;
run {
allocate channel prmy1 type disk;
allocate channel prmy2 type disk;
allocate channel prmy3 type disk;
allocate channel prmy4 type disk;
allocate auxiliary channel stby type disk;
 
duplicate target database for standby from active database
spfile
set db_unique_name='ssoa'
set control_files='+DATA/ssoa/standby.ctl'
set local_listener='ssoa_local_listener'
set remote_listener='stby-scan:1521';
}
EOF

Example 3-8 Sample Redo Log

SQL> alter database add standby logfile thread 1
group 9 size 500M,
group 10 size 500M,
group 11 size 500M;
 
SQL> alter database add standby logfile thread 2
group 12 size 500M,
group 13 size 500M,
group 14 size 500M;
Procedure for Completing RAC Configuration for Standby Database

To complete the RAC configuration on the standby database, complete the steps given in Procedure for Duplicating the Primary Database. Then, perform the following steps:

  1. Create a temporary parameter file in the standby database:

    SQL> create pfile='/tmp/p.ora' from spfile;
    
  2. Create an SPFILE in +DATA_stby for the standby database:

    SQL> create spfile='+DATA_stby/ssoa/spfilessoa.ora' from pfile='/tmp/p.ora';
    
  3. Remove the default file.

    $rm /u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/spfilesoa1.ora
  4. Create an initsoan.ora file on all the standby hosts. The file needs to point to the location of the SPFILE created in step 2.

    In the soadc2.dbhost1 file, add the following:
    cat /u01/app/oracle/product/11.2.0/dbs/initsoa1.ora
    spfile='+DATA/ssoa/spfilesoa.ora
  5. On the standby system, restart the instances in mount state:

    startup mount
    
  6. Register the RAC database with CRS as follows:

    srvctl add database -d ssoa –o /u01/app/oracle/product/11.2.0/db_1
    srvctl add instance -d ssoa -i soa1 -n soadc2.dbhost1
    srvctl add instance -d ssoa -i soa2 -n soadc2.dbhost2
    srvctl modify database –d ssoa –r physical_standby
    
Creating a Data Guard Broker Configuration

This section describes the basic steps for creating a Data Guard configuration.

For complete information about Data Guard Broker, see the Oracle Data Guard Installation guide.

To create a Data Guard Broker configuration, complete the following steps:

  1. Add the values of static SID to the local node file, listener.ora, that is located in the grid infrastructure home on all hosts in the configuration:
    LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2))))
    LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3))))
    LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))) 
    
    LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))
    ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON
    ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON 
    SID_LIST_LISTENER = 
    (SID_LIST = 
    (SID_DESC = 
    (GLOBAL_DBNAME = psoa_DGMGRL) 
    (SID_NAME = soa1)
    (ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1)))
    ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON
    

    Note:

    • Ensure that you enter the appropriate values for the GLOBAL_DBNAME and SID_NAME parameters, for each host.

    • If IFILE is used for the shared ORACLE_HOME, modify the corresponding local listener.ora file that is specified in IFILE.

  2. Bounce the listeners on the primary site and standby site to apply the modifications made to the files:
    srvctl stop listener
    srvctl start listener
    
  3. Configure the Data Guard Broker metadata files, and enable the broker on the primary site and standby site as follows:

    On the primary site

    SQL> alter system set dg_broker_config_file1='+DATA_prmy/psoa/dr1.dat' scope=both;
    SQL> alter system set dg_broker_config_file2='+DATA_prmy/psoa/dr2.dat' scope=both;
    SQL> alter system set dg_broker_start=true scope=both;
    

    On the standby site

    SQL> alter system set dg_broker_config_file1='+DATA_stby/ssoa/dr1.dat' scope=both;
    SQL> alter system set dg_broker_config_file2='+DATA_stby/ssoa/dr2.dat' scope=both;
    SQL> alter system set dg_broker_start=true scope=both;
    
  4. Access dgmgrl on the primary host, and create the configuration. See Example 3-9.

Example 3-9 Accessing dgmgrl to Create the Data Guard Broker Configuration on Primary Host

dgmgrl sys/password
 
DGMGRL> create configuration 'dg_config' as 
primary database is 'psoa'
connect identifier is psoa;
 
Configuration "dg_config" created with primary database "psoa"
 
DGMGRL> add database 'ssoa' as
connect identifier is ssoa;
 
Database "ssoa" added
 
DGMGRL> enable configuration;
 
Enabled.
Verifying the Data Guard Broker Configuration

Complete the following steps to verify that the Data Guard Broker configuration was created successfully.

  1. Verify the Oracle Data Guard configuration by querying the V$ARCHIVED_LOG view to identify existing files in the archived redo log. For example:
    SQL> SELECT SEQUENCE#, FIRST_TIME, NEXT_TIME 
      2> FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;
    
    SEQUENCE# FIRST_TIME NEXT_TIME
    ---------- ------------------ ------------------
             8 11-JUL-13 17:50:45 11-JUL-13 17:50:53
             9 11-JUL-1317:50:53 11-JUL-1317:50:58
            10 11-JUL-13 17:50:58 11-JUL-13 17:51:03
     
    3 rows selected
    
  2. On the primary database, issue the following SQL statement to force a log switch and archive the current online redo log file group:
    SQL> alter system archive log current;
    
  3. On the standby database, query the V$ARCHIVED_LOG view to verify that the redo data was received and archived on the standby database:
    SQL> SELECT SEQUENCE#, FIRST_TIME, NEXT_TIME 
      2> FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;
    
    SEQUENCE# FIRST_TIME NEXT_TIME
    ---------- ------------------ ------------------
             8 11-JUL-13 17:50:45 11-JUL-13 17:50:53
             9 11-JUL-1317:50:53 11-JUL-1317:50:58
             10 11-JUL-13 17:50:58 11-JUL-13 17:51:03
             11 11-JUL-13 17:51:03 11-JUL-1318:34:11
    
    4 rows selected
    
  4. Use the show configuration command to verify that the configuration was created successfully on the standby site. See Example 3-10.

Example 3-10 Verifying the Data Guard Broker Configuration

DGMGRL> show configuration;

Configuration - dg_config

Protection Mode: MaxPerformance
Databases:
psoa- Primary database
ssoa- Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS
Testing Database Switchover and Switchback

You can perform a database switchover and switchback.

Performing a Switchover Operation by Using Oracle Data Guard Broker

To perform a switchover operation by using Oracle Data Guard Broker, complete the following tasks:

  1. Verify the Oracle Data Guard Broker configuration that you created by using the instructions provided in Creating a Data Guard Broker Configuration.

    To verify the configuration, run the following command:

    DGMGRL> show configuration;
     
    Configuration - dg_config
     
    Protection Mode: MaxPerformance
    Databases:
    psoa- Primary database
    ssoa- Physical standby database
     
    Fast-Start Failover: DISABLED
     
    Configuration Status:
    SUCCESS
    
  2. Swap the roles of the primary and standby databases by running the SWITCHOVER command. Example 3-10 shows how Data Guard Broker automatically shuts down and restarts the old primary database as part of the switchover operation.

    DGMGRL> switchover to 'ssoa';
    Performing switchover NOW, please wait...
    New primary database "ssoa" is opening...
    Operation requires shutdown of instance "psoa1" on database "psoa"
    Shutting down instance "psoa1"...
    ORA-01109: database not open
     
    Database dismounted.
    ORACLE instance shut down.
    Operation requires startup of instance "psoa1" on database "psoa"
    Starting instance "psoa1"...
    ORACLE instance started.
    Database mounted.
    Switchover succeeded, new primary is "ssoa"
    
  3. After the switchover is complete, use the SHOW CONFIGURATION command to verify that the switchover operation was successful:

    DGMGRL> show configuration;
    Configuration - dg_config
    Protection Mode: MaxPerformance
    Databases:
    ssoa- Primary database
    psoa- Physical standby database
    Fast-Start Failover: DISABLED
    Configuration Status:
    SUCCESS
    

Performing a Switchover Operation Using SQL Plus

Perform the following operations to switchover or switchback databases correctly between the newly created physical standby database and the primary Oracle RAC databases:

  1. Shut down all but one instance of the Oracle RAC databases (PSOA) on the primary site. For example, run the following command on SOADC1.DBHOST1 of the production site:
    srvctl stop instance -d psoa -i soa2
    
  2. Initiate the role transition to the physical standby on the current primary database. For example, run the following command on SOADC1.DBHOST1 of the production site:
    SQL > ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY WITH SESSION SHUTDOWN;
    
  3. Shut down the primary instance and mount the primary instance. For example, run the following command on SOADC1.DBHOST1 of the production site:
    SQL > shutdown immediate
    SQL > startup mount
    
  4. At this point, both the databases are in Physical Standby mode.

    Sign in to the instances to continue.

  5. To verify that both the databases are in Physical Standby mode, run this SQL query on both the databases:
    SQL> select database_role from v$database;
    DATABASE_ROLE
    ----------------
    PHYSICAL_STANDBY
    
  6. Switch the physical standby database role to the primary role. For example, run the following command on SOADC2.DBHOST1 of the standby site:
    SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WITH SESSION SHUTDOWN;
    
  7. Now the physical standby database is the new primary database.
  8. Shut down the new primary database and start up both the RAC nodes by using srvctl. For example, run the following command on the new primary site SOADC2.DBHOST1:
    srvctl start database -d ssoa
    
  9. On the new physical standby database (the old primary) start the managed recovery of the database. For example, run the following command on SOADC1.DBHOST1 of the primary site:
    SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;
    
  10. Start sending the redo data to the new physical standby database. For example, run the following command on the new primary site SOADC2.DBHOST1:
    SQL> ALTER SYSTEM SWITCH LOGFILE;
    
  11. Check the new physical standby database to see if it is receiving the archive log files by querying the V$ARCHIVED_LOG view.
  12. Perform a switchback operation. A switchback operation is a subsequent switchover operation to return the roles to their original state.

Note:

For information about switchover and failover operation of Oracle Data Guard Broker, see Switchover and Failover Operations in the Oracle Data Guard Broker.

Creating a Production Site

Learn how to create a production site on an Oracle SOA enterprise deployment topology.

Before you create your production site:

  • Set up the host name aliases for the middle tier hosts, as described in Planning Host Names.

  • Create the required volumes on the shared storage on the production site, as described in Designing Directory Structure and Volumes.

  • Determine the Oracle Data Guard configuration to use based on the data loss requirements of the database and network considerations, such as the available bandwidth and latency when compared to the redo generation.

This section includes the following topics:

Creating a Production Site

Create a production site for a topology.

Note:

This section provides information about how to create a production site for the Oracle SOA Suite topology. If you plan to create a production site for a different topology, see the appropriate Oracle Fusion Middleware Enterprise Deployment Guide listed under the Install a Production Environment: Plan, Install & Configure an Enterprise Deployment category.

Install and configure your production site as described in the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.

The following sections describe how to complete the installation and configuration of your production site:

Creating Volumes and Consistency Groups

Create volumes and consistency groups on the shared storage device.

To create volumes and consistency groups on the shared storage device, see Recommended Volume Design for Oracle SOA Suite.

Setting Up Physical Host Names and Alias Host Names

Set up physical host names on the production site and physical host names and alias host names on the standby site.

For information about planning host names for the production and standby sites, see Planning Host Names.

Installing and Configuring Oracle SOA Suite

Install and configure Oracle SOA Suite.

To install and configure Oracle SOA Suite, see Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite and apply the following modifications:

  1. Install the Oracle SOA Suite components into the volumes created on the shared storage device.
  2. Set up the production and standby sites by using the aliases to the physical and virtual host names.
  3. Create SSL certificates by using the host name aliases on all of the Oracle Fusion Middleware hosts for proper Node Manager communication.

Configuring Data Sources for Oracle Fusion Middleware Active-Passive Deployment

Configure the data sources that Oracle Fusion Middleware uses to automate connections failover, in case of a failover or switchover of the primary database.

As explained in the Setting Up DataSources in the Middle Tier, it is recommended to use a TNS alias in the datasources to access to the local database in each site. The TNS alias is the same name in production and standby, hence the datasources use the same db connect string. The TNS alias is resolved with a tnsnames.ora file that is stored separately from the WebLogic domain configuration, and not replicated between sites, so you can have a different tnsnames.ora content in each site. Each site will resolve the TNS alias with the appropriate connect string in each site, pointing to the local database only.

If you are not using this approach already in the production system, you can use the following steps to configure it. Configure it in all the data sources that are used in the domain, including datasources used by jdbc persistence stores, leasing data sources, and custom datasources, and in the JDBC URL of the OPSS Security Stores.

  1. The WebLogic servers must use the system property -Doracle.net.tns_admin=<tns_directory> , where <tns_directory> is the directory location of the tnsnames.ora file. This must be in a folder that is not replicated to the secondary site, and readable by the oracle user. For example: /home/oracle/tnsnames_dir.

    To set it as a java property for the servers, edit the files:
    $ASERVER_HOME/bin/setUserOverrides.sh 
    $MSERVER_HOME/bin/setUserOverrides.sh (Note: This is not shared, so it must be edited in all the soa mid-tier hosts.)

    and the following content :

    # For using tns alias in the datasources 
    export EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES} -Doracle.net.tns_admin=/home/oracle/tnsnames_dir"
  2. Create the tns folder in all the middle tier hosts:
    mkdir -p /home/oracle/tnsnames_dir 

    Note:

    Create this folder in the middle tier hosts in production and standby.
  3. Create a tnsnames.ora file in the tns folder, with the tns alias that will be used in the datasources. In the production middle tier hosts, the alias must point to the production database.

    For example:

    SOAEDG= 
    (DESCRIPTION=
      (ADDRESS_LIST=
        (LOAD_BALANCE=ON)
        (ADDRESS=(PROTOCOL=TCP)(HOST=prmy-scan)(PORT=1521))
      )
      (CONNECT_DATA=(SERVICE_NAME=soaedg.example.com))
    )

    In the standby middle tier hosts, the alias is the same name, but pointing to the standby database.

    For example:

    SOAEDG= 
    (DESCRIPTION=
      (ADDRESS_LIST=
        (LOAD_BALANCE=ON)
        (ADDRESS=(PROTOCOL=TCP)(HOST=stby-scan)(PORT=1521))
      )
      (CONNECT_DATA=(SERVICE_NAME=soaedg.example.com))
    )
  4. Use the WebLogic Console to modify the URL definition in the data sources, by replacing the connection string with the alias. The following is a sample JDBC URL using the tns alias: jdbc:oracle:thin:@soaedg.

    Alternatively, you can perform the modification directly in the files, as described here:
    1. Sign in to APPHOST1.
    2. Take a backup copy of the $ASERVER_HOME/config/jdbc, which contains the datasource config files.
    3. Run a command to replace the previous db connection string by the new one that uses. Example:
      cp -rf $ASERVER_HOME/config/jdbc   $ASERVER_HOME/config/jdbc_bck
      export PREV_STRING='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=prmy-scan)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=soaedg.example.com)))'
      export NEW_STRING='soaedg'
      cd $ASERVER_HOME/config/jdbc
      sed -i 's/${PREV_STRING}/${NEW_STRING}/g' *.xml
      
    4. Update also the JDBC URL of the OPSS Security Stores too. Complete the following steps:
      1. Sign in to APPHOST1 and go to the $ASERVER_HOME/config/fmwconfig folder.
      2. Take a backup copy of the jps-config-jse.xml and jps-config.xml files.
      3. Edit both files to update value of the property name="jdbc.url" with the appropriate JDBC URL used in the datasources.

      Example:

      export PREV_STRING='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=prmy-scan)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=soaedg.example.com)))' 
      export NEW_STRING='soaedg'
      cp -rf $ASERVER_HOME/config/fmwconfig $ASERVER_HOME/config/fmwconfig_bck
      cd $ASERVER_HOME/config/fmwconfig 
      sed -i's/${PREV_STRING}/${NEW_STRING}/g' *.xml

    Note:

    Starting with the 18.3 jdbc driver, the location of the tnsnames.ora file can be set as part of the connection string. Sample: jdbc:oracle:thin:@soapdb?TNS_ADMIN=/home/oracle/tnsnames_dir . That way, it is not required to set the property -Doracle.net.tns_admin in the startup scripts. Hence, the required modifications are performed at configuration level only. This applies to FMW 12.2.1.4 (which uses jdbc driver 19.3) and later versions.
  5. A restart of all the Weblogic Servers in the domain is needed for these changes to take effect.
    1. Stop all the WebLogic Servers in domain (Admin server and Managed servers).
    2. Start the Administration server in domain.
    3. Once it is running, start the Managed Servers.

    Note:

    Starting with the 18.3 jdbc driver, the location of the tnsnames.ora file can be set as part of the connection string. Sample: jdbc:oracle:thin:@soapdb?TNS_ADMIN=/home/oracle/tnsnames_dir . Since the required modifications are performed at configuration level , you don't have to set the property -Doracle.net.tns_admin in the startup scripts. This applies to Oracle Fusion Middleware 12.2.1.4 (which uses jdbc driver 19.3) and newer versions.
  6. Verify that the datasource connections are correctly established with the database.

    Verify that the JDBC URL is correctly updated in the OPSS store. To verify this:
    1. Go to the Enterprise Manager Console.
    2. Navigate to WebLogic Domain > Security > Security Provider Configuration.
    3. Expand Security Stores and verify that the Database URL is updated.
  7. Regarding the ONS Host and Port configuration in the datasources: these values are not required when you are using an Oracle 12c database or higher versions because the ONS list is automatically provided obtained from the database by from the client driver the database to the driver.

    Consistently with the recommendation in the EDG guide, use this feature instead of providing the scan address or the list of the rac nodes in the ONS configuration of each Datasource.

    Make sure that FAN is enabled and that the ONS nodes property is empty in each Datasource. This can be checked in the WebLogic Administration Console, in Services > Data Sources > Configuration > ONS.

Creating a Standby Site

Learn how to create a standby site.

To create the standby site, the Oracle SOA enterprise deployment topology is used as an example.

This section includes the following topics:

Preparing the Standby Site

Prepare your standby site for operation.

To prepare it for operation, on your standby site:

  • Set up the correct alias host names and physical host names by following the instructions in Planning Host Names.

    Ensure that each standby site host has an alias host name that is the same as the physical host name of its peer host at the production site.

  • Create, on the shared storage, the same volumes that were created on the shared storage at the production site

  • Create the same mount points and symbolic links (if required) that you created at the production site. Note that symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes.

    For more details about symbolic links, see Setting Up Storage.

Setting Up Middle Tier Hosts

Middle tier hosts on a standby site do not require installation nor configuration of Oracle Fusion Middleware or Oracle WebLogic Server software. When the production site storage is replicated to the standby site storage, the software installed on the production site is replicated at the standby site.

To set up the middle tier hosts on the standby site:

  1. Create a baseline snapshot copy of the shared storage on the production site, which sets up the replication between the storage devices. Create the initial baseline copy and subsequent snapshot copies by using asynchronous replication mode.
  2. Synchronize the shared storage at the production site with the shared storage at the standby site. This transfers the initial baseline snapshot from the production site to the standby site.
  3. Set up the frequency of subsequent copies of the production site shared storage, which is replicated at the standby site. When asynchronous replication mode is used, then at the requested frequency the changed data blocks at the production site shared storage (based on comparison to the previous snapshot copy) become the new snapshot copy, and the snapshot copy is transferred to the standby site shared storage.
  4. After the baseline snapshot copy is performed, validate that all the directories inside the standby site volumes have the same content as the directories inside the production site volumes.
About Self-Signed certificates and Keys on the Standby Site

Learn about certificates and keystores.

As recommended in Planning Host Names the FMW components use alias host names, that are resolvable to the IP addresses of the appropriate system in each site. If the primary self-signed certificates are created with these host names, the certificates are valid both for the production and for the standby systems.

For more information about creating primary self-signed certificates, see Common Configuration and Management Tasks for an Enterprise Deployment in Enterprise Deployment Guide for Oracle SOA Suite.

The certificates and keystores are replicated to standby along with the configuration, so there is no need to create specific self-signed certificates or keystores for the standby site.

Validating the Standby Site Setup

Validate your standby site.

To validate a standby site:

  1. Shut down any processes still running on the production site. These include the database instances in the data tier, Oracle Fusion Middleware instances, and any other processes in the application tier and web tier.
  2. Stop the replication between the production site shared storage and the standby site shared storage.
  3. Perform a switchback operation. A switchback operation is a subsequent switchover operation to return the roles to their original state.
  4. On the standby site host, manually start all the processes. These include the database instances in the data tier, Oracle Fusion Middleware instances, and any other processes in the application tier and web tier.
  5. Use a browser client to perform post-failover testing to confirm that requests are being resolved and redirected to the standby site.

Performing Site Operations and Administration

Learn how to operate and administer your Oracle Fusion Middleware Disaster Recovery topology.

This section includes the following topics:

Synchronizing the Production and Standby Sites

Learn how to force a synchronization of the production and standby sites when you introduce a change in the middle tier at the production site.

In normal operations, the standby site shared storage receives snapshots transferred periodically from the production site shared storage. After the snapshots are applied, the standby site shared storage includes all the data up to and including the data contained in the last snapshot transferred from the production site before the failover or switchover.

Be sure to force a synchronization when you introduce a change to the middle tier at the production site such as, for example, when you deploy a new application at the production site. Follow the vendor-specific instructions to force a synchronization by using the storage replication technology.

The databases synchronization in an Oracle Fusion Middleware Disaster Recovery topology is managed by Oracle Data Guard.

Performing a Switchover

A switchover operation sets the standby site as the production role.

This operation is needed when you plan to take down the production site (for example, to perform maintenance) and to make the current standby site the production site.

To perform a switchover operation:

  1. Shut down any processes running on the production site. These include the database instances in the data tier, Oracle Fusion Middleware instances, and any other processes in the application tier and web tier.
  2. Stop the replication between the production site shared storage and the standby site shared storage.
  3. Unmount the shared storage volume with the middle tier artifacts, on the current production site, and mount the corresponding volumes on the current standby site, which is the new production site.
  4. Use Oracle Data Guard to switch over the databases.
  5. On the standby site host, manually start all the processes. These include the database instances in the data tier, Oracle Fusion Middleware instances, and any other processes in the application tier and web tier.
  6. Ensure that all user requests are routed to the standby site by performing a global DNS push or something similar, such as updating the global load balancer.
  7. Use a browser client to perform post-switchover testing to confirm that requests are being resolved and redirected to the standby site.

    At this point, the former standby site is the new production site and the former production site is the new standby site.

  8. Reestablish the replication between the two sites, but configure the replication so that the snapshot copies go in the opposite direction (from the current production site to the current standby site). See the documentation for your shared storage to learn how to configure the replication so that snapshot copies are transferred in the opposite direction.

At this point, the former standby site becomes the new production site, and you can perform maintenance at the original production site. After you have carried out the maintenance of the original production site, you can use it as either the production site or the standby site.

Note:

This note is applicable for BI-specific systems only.

After a switchover operation, the creation of an Essbase cube with CDS may fail with an error similar to the following:

oracle.essbase.cds.util.CDSException: oracle.essbase.cds.util.CDSException: java.sql.SQLException: ORA-25153:

To work around this issue and to create an Essbase cube, on the current primary database:
  • Identify the temporary tablespaces by using a select statement similar to the following (where BIS17V1 is the Oracle Business Intelligence RCU prefix):

    select username,temporary_tablespace from dba_users where username like 'BIS17V1%'

    Assume that the above command returns the following list of temporary tablespaces:

    USERNAME.....TEMPORARY_TABLESPACE

    BIS17V1_IAU_VIEWER.....BIS17V1_IAS_TEMP

    BIS17V1_STB.....BIS17V1_IAS_TEMP     

    BIS17V1_IAU_APPEND.....BIS17V1_IAS_TEMP

    BIS17V1_MDS.....BIS17V1_IAS_TEMP

    BIS17V1_IAU.....BIS17V1_IAS_TEMP

    BIS17V1_BIPLATFORM......BIS17V1_IAS_TEMP

    BIS17V1_OPSS.....BIS17V1_IAS_TEMP

  • After the switchover, drop the tablespace BIS17V1_IAS_TEMP including contents and datafiles.

  • Create the temporary tablespace BIS17V1_IAS_TEMP, as a tempfile, in the location (for example) /work/primy/oradata/stnby/BIS17V1_IAS_TEMP.dbf, with size 250 m.

  • Issue the following alter commands (here is where you use the list temporary tablespaces):

    alter user BIS17V1_OPSS temporary tablespace  BIS17V1_IAS_TEMP ;

    alter user BIS17V1_BIPLATFORM temporary tablespace  BIS17V1_IAS_TEMP ;

    alter user BIS17V1_IAU temporary tablespace  BIS17V1_IAS_TEMP ;

    alter user BIS17V1_MDS temporary tablespace  BIS17V1_IAS_TEMP ;

    alter user BIS17V1_IAU_APPEND temporary tablespace  BIS17V1_IAS_TEMP ;

    alter user BIS17V1_STB temporary tablespace  BIS17V1_IAS_TEMP ;

    alter user BIS17V1_IAU_VIEWER temporary tablespace  BIS17V1_IAS_TEMP ;

To use the original production site as the production site, perform a switchback as explained in Performing a Switchback.

Performing a Switchback

A switchback operation reverts the roles of the current production and standby sites.

To perform a switchback operation:

  1. Shut down any processes running on the current production site. These include the database instances in the data tier, Oracle Fusion Middleware instances, and any other processes in the application tier and web tier.
  2. Stop the replication between the current production site shared storage and standby site shared storage.
  3. Unmount the shared storage volume with the middle tier artifacts, on the current production site, and mount the corresponding volumes on the current standby site, which is the new production site.
  4. Use Oracle Data Guard to switch back the databases.
  5. On the new production site hosts, manually start all the processes. These include Oracle Fusion Middleware instances and any other processes in the application tier and web tier.
  6. Ensure that all user requests are routed to the new production site by performing a global DNS push or something similar, such as updating the global load balancer.
  7. Use a browser client to perform post-switchback testing to confirm that requests are being resolved and redirected to the new production site.

    At this point, the former standby site is the new production site and the former production site is the new standby site.

  8. Reestablish the replication between the two sites, but configure the replication so that the snapshot copies go in the opposite direction (from the new production site to the new standby site). See the documentation for your shared storage to learn how to configure the replication so that snapshot copies are transferred in the opposite direction.

Performing a Failover

A failover operation sets the standby site as the production role when the production site becomes unavailable.

To perform a failover operation:

  1. Stop the replication between the production site shared storage and the standby site shared storage.
  2. Mount the shared storage volume with the middle-tier artifacts, on the current standby site, which is the new production site.
  3. From the standby site, use Oracle Data Guard to fail over the databases.
  4. On the standby site hosts, manually start all the processes. These include the Oracle Fusion Middleware instances and any other processes in the application tier and web tier.
  5. Ensure that all user requests are routed to the standby site by performing a global DNS push or something similar, such as updating the global load balancer.
  6. Use a browser client to perform post-failover testing to confirm that requests are being resolved and redirected to the production site.

    At this point, the standby site is the new production site. You can examine the issues that caused the former production site to become unavailable.

  7. To use the original production site as the current standby site, you must reestablish the replication between the two sites, but configure the replication so that the snapshot copies go in the opposite direction (from the current production site to the current standby site). See the documentation for your shared storage system to learn how to configure the replication so that snapshot copies are transferred in the opposite direction.

Note:

After a failover operation, the creation of an Essbase cube with CDS may fail with an error similar to the following:

oracle.essbase.cds.util.CDSException: oracle.essbase.cds.util.CDSException: java.sql.SQLException: ORA-25153:

To work around this issue and to create an Essbase cube, on the current primary database:
  • Identify the temporary tablespaces by using a select statement similar to the following, where BIS17V1 is the Oracle Business Intelligence RCU prefix):

    select username,temporary_tablespace from dba_users where username like 'BIS17V1%'

    Assume that the above command returns the following list of temporary tablespaces:

    USERNAME.....TEMPORARY_TABLESPACE

    BIS17V1_IAU_VIEWER.....BIS17V1_IAS_TEMP

    BIS17V1_STB.....BIS17V1_IAS_TEMP     

    BIS17V1_IAU_APPEND.....BIS17V1_IAS_TEMP

    BIS17V1_MDS.....BIS17V1_IAS_TEMP

    BIS17V1_IAU.....BIS17V1_IAS_TEMP

    BIS17V1_BIPLATFORM......BIS17V1_IAS_TEMP

    BIS17V1_OPSS.....BIS17V1_IAS_TEMP

  • After the failover, drop the tablespace BIS17V1_IAS_TEMP including contents and datafiles.

  • Create the temporary tablespace BIS17V1_IAS_TEMP, as a tempfile, in location. For example, /work/primy/oradata/stnby/BIS17V1_IAS_TEMP.dbf with size 250 m.

  • Issue the following alter commands (here is where you use the list temporary tablespaces):

    alter user BIS17V1_OPSS temporary tablespace  BIS17V1_IAS_TEMP ;

    alter user BIS17V1_BIPLATFORM temporary tablespace  BIS17V1_IAS_TEMP ;

    alter user BIS17V1_IAU temporary tablespace  BIS17V1_IAS_TEMP ;

    alter user BIS17V1_MDS temporary tablespace  BIS17V1_IAS_TEMP ;

    alter user BIS17V1_IAU_APPEND temporary tablespace  BIS17V1_IAS_TEMP ;

    alter user BIS17V1_STB temporary tablespace  BIS17V1_IAS_TEMP ;

    alter user BIS17V1_IAU_VIEWER temporary tablespace  BIS17V1_IAS_TEMP ;

To again use the original production site as the production site, perform a switchback as explained in Performing a Switchback.

Testing the Standby Site

Learn how to create a clone of the read-only standby site shared storage and use it for converting database secondary_db_unqname to snapshot standby.

A typical Oracle Fusion Middleware Disaster Recovery configuration uses:

  • Storage replication to copy Oracle Fusion Middleware middle tier file systems and data from the production site shared storage to the standby site shared storage. During normal operation, the production site is active and the standby site is passive. When the production site is active, the standby site is passive and the standby site shared storage is in read-only mode; the only write operations made to the standby site shared storage are the storage replication operations from the production site shared storage to the standby site shared storage.

  • Oracle Data Guard to copy database data for the production site Oracle databases to the standby databases at standby site. By default, the production site databases are active and the standby databases at the standby site are passive. The standby databases at the standby site are in Managed Recovery mode while the standby site is in the standby role (is passive). When the production site is active, the only write operations made to the standby databases are the database synchronization operations performed by Oracle Data Guard.

  • The standby site as the production role when the production site becomes unavailable. If the current production site becomes unavailable unexpectedly, then a failover operation (described in Performing a Failover) is performed to enable the standby site to assume the production role. Or, if the current production site is taken down intentionally (for example, for planned maintenance), then a switchover operation (described in Performing a Switchover) is performed to enable the standby site to assume the production role.

The usual method of testing a standby site is to shut down the current production site and perform a switchover operation to enable the standby site to assume the production role.

However, some enterprises may want to perform periodic testing of their Disaster Recovery standby site without shutting down the current production site and a complete switchover operation. This is possible by converting the standby database to snapshot standby. This allows the standby servers to be started in the standby site and verify the secondary system. Any change performed in the standby site database while it is in snapshot standby mode will be discarded once it is converted to physical standby again, so primary data will not be affected by secondary site validations.

To use this testing method:

  1. Use the cloning technology provided by the shared storage vendor to create a clone of the standby site's read-only volumes on the shared storage at the standby site. Ensure that the cloned standby site volumes are writable. If you want to test the standby site just once, then this can be a one-time clone operation. However, if you want to test the standby site regularly, you can set up periodic cloning of the standby site read-only volumes to the standby site's cloned read/write volumes.

  2. Convert the standby database into snapshot standby by the following steps:

    1. Use Data Guard broker in primary database host to convert the secondary database to snapshot standby.

      Example:

      [oracle@primarydbhost~]$dgmgrl sys/your_sys_password@primary_db_unqname
      DGMGRL> convert database “secondary_db_unqname” to snapshot standby
      
    2. . Use show configuration command to verify that the conversion has been correctly performed.

  3. On the standby site computers, modify the mount commands to point to the volumes on the standby site's cloned read/write shared storage by following these steps:

    1. Unmount the read-only shared storage volumes.

    2. Mount the cloned read/write volumes at the same mount point.

  4. Before you test the standby site, modify the host name resolution method for the computers that are used to perform the testing to ensure that the host names point to the standby site computers and not the production site computers. For example, on a Linux computer, change the /etc/hosts file to point to the virtual IP of the load balancer for the standby site.

  5. Perform the standby site testing.

Note:

This operation must be done with caution in SOA environments: if there are pending messages or composites in the database when it is converted into snapshot, the standby site's SOA servers will process them when they start. Check that there are no pending actions in primary database when converting to snapshot standby, otherwise, remove records from runtime SOA tables in the standby database after it is converted to snapshot standby database and before starting the secondary site's SOA servers. See, Removing Records from the Runtime Tables Without Dropping the Tables.

After you complete the standby site testing, follow these steps to begin using the original production site as the production site again:

  1. Modify the mount commands on the standby site computers to point to the volumes on the standby site's read-only shared storage. In other words, reset the mount commands back to what they were before the testing was performed.

    1. Unmount the cloned read/write shared storage volume.

    2. Mount the read-only shared storage volumes.

    At this point, the mount commands are reset to what they were before the standby site testing was performed.

  2. Convert the standby database into snapshot standby by the following steps:

    1. Use Data Guard broker in primary database host to convert the secondary database to physical standby again.

      Example:

      [oracle@primarydbhost~]$dgmgrl sys/your_sys_password@primary_db_unqname
      DGMGRL> convert database “secondary_db_unqname” to physical standby
      
    2. Use show configuration command to verify that the conversion has been correctly performed.

  3. Before you use the original production site again, modify the host name resolution method for the computers that are used to access the production site to ensure that the host names point to the production site computers and not the standby site computers. For example, on a Linux computer, change the /etc/hosts file to point to the virtual IP of the load balancer for the production site.

Using Peer-To-Peer File Copy

The rsync utility (which uses peer-to-peer file copy) can be used to replicate middle tier file system data from a production site host to a standby site peer host in an Oracle Fusion Middleware Disaster Recovery topology. The use of the rsync utility is explained in the context of symmetric topologies

For information about the conditions for using the rsync in Disaster Recovery environments, see the point rsync in the section Setting Up Storage Replication of this document.

Ensure that you are familiar with storage replication and Oracle Data Guard in an Oracle Fusion Middleware Disaster Recovery topology, because there are many similarities between using storage replication and using the rsync utility for disaster protection and disaster recovery of your Oracle Fusion Middleware components.

Note:

Note the following important differences between using storage replication technologies and using the rsync utility to replicate middle tier file systems:

  • When you use storage replication, you can roll changes back to the point in time when any previous snapshot was taken at the production site.

    When you use the rsync utility, replicated production site data overwrites the standby site data, and you cannot roll back a replication.

  • When you use storage replication, the volume that you set up for each host cluster in the shared storage systems ensures data consistency for that host cluster across the production site's shared storage system and the standby site's shared storage system.

    When you use the rsync utility, data consistency is not guaranteed.

Because of these differences, the rsync utility is not supported in production environments but in test environments only.

This section includes the following topic:

Using rsync and Oracle Data Guard in Oracle Fusion Middleware Disaster Recovery Topologies

Learn how to use the UNIX rsync utility and Oracle Data Guard in your Oracle Fusion Middleware Disaster Recovery topology.

Note:

For information about how to set up Oracle Data Guard for Oracle database, see Database Considerations.

The following sections describe how to use the rsync utility and Oracle Data Guard to protect and force synchronization between your production and standby sites in an Oracle Fusion Middleware Disaster Recovery topology:

Using rsync for Oracle Fusion Middleware Middle Tier Components

Use the UNIX rsync utility to protect and recover your Oracle Fusion Middleware middle tier components.

To use the rsync utility:

  1. Install and set up rsync to enable replication of files from a production site host to its standby site peer host. For instructions about how to install, set up, and use the utility, see the utility man pages and also http://rsync.samba.org.
  2. For each production site host on which one or more Oracle Fusion Middleware components has been installed, set up rsync to copy the following directories and files to the same directories and files on the standby site peer host:
    • The Oracle home directory, subdirectories, and all the files in them.

    • The Oracle Central Inventory directory and files for the host, which includes the Oracle Universal Installer entries for the Oracle Fusion Middleware installations.

    • The shared config folder, such as the WebLogic Administration Server domain directory, deployment plans, applications, keystores, and so on. As this folder is shared between the mid-tier hosts and you do not have to have a for each host.

    • The private config folders, such as the WebLogic managed server's domain, local node manager home on the host.

    • If applicable, the shared runtime folder.

    • If applicable, the Oracle Fusion Middleware static HTML pages directory for the Oracle HTTP Server installations on the host.

    • If applicable, the .fmb and .fmx deployment files that are created by Oracle Forms on the host, and the .rdf deployment artifact files that are created by Oracle Reports on the host.

      Note:

      Run the rsync utility as root or as the OS user that is the owner of the files. If you want it to work without prompting users for a password, set up SSH keys between the production site host and standby site host, so that SSH does not prompt for a password.

      For an example of using rsync utility to copy folders to a remote node, see soa-hybrid-dr-utils.zip. The zip contains a generic script (rsync_copy_and_validate.sh) that copies a folder to a remote node, and performs a checksum validation of the copy. The zip file also includes a few examples to use this generic script for copying the typical FMW folders.

  3. Set up scheduled jobs, for example, cron jobs, for the production site hosts for which you set up rsync in the previous step. These scheduled jobs enable rsync to automatically perform replication of these files from the production site hosts to the standby site hosts on a regular interval. An interval of once a day is recommended for a production site where the Oracle Fusion Middleware configuration does not change very often.
  4. Whenever a change is made to the configuration of an Oracle Fusion Middleware middle tier configuration on a production site host (for example, when a new application is deployed), perform a manual synchronization of that host with its standby site peer host by using rsync.
  5. Whenever you use rsync to a manually synchronize an Oracle Fusion Middleware middle tier instance on a production site host with the peer standby site host, also use Oracle Data Guard to synchronize database repositories in the production site with the databases in the standby site.
Performing Failover and Switchover Operations

Learn how to perform failover or switchover operations when you use the rsync utility.

To perform a failover or switchover from the production site to the standby site when you use rsync:

  1. Shut down any processes running on the production site (if applicable).
  2. Stop rsync jobs between the production site hosts and standby site peer hosts.
  3. Use Oracle Data Guard to failover or swithcover the production site databases to the standby site.
  4. On the standby site, manually start the processes for the Oracle Fusion Middleware Server instances.
  5. Route all user requests to the standby site by performing a global DNS push or something similar, such as updating the global load balancer.
  6. Use a browser client to perform post-failover or post-switchover testing to confirm that requests are being resolved at the standby site (current production site).

    At this point, the standby site is the new production site and the production site is the new standby site.

  7. Reestablish rsync between the two sites, but configure it so that replications go now in the opposite direction (from the current production site to the current standby site).

To use the original production site as the new production site, perform the preceding steps again but configure the rsync replications to go in the original direction (from the original production site to the original standby site).

Using Oracle Site Guard for Disaster Recovery

Oracle Site Guard is a disaster-recovery solution that enables administrators to automate complete site switchover or failover. It orchestrates the coordinated failover of Oracle Fusion Middleware, Oracle Fusion Applications, and Oracle Databases.

Oracle Site Guards offers the following benefits:

  • Fully automate disaster recovery operations and launch them with a single click
  • Minimizes disaster-recovery time
  • Reduces human errors
  • Flexible and customizable
  • Eliminates the need for special skills
  • Use a single pane of glass to manage disaster recovery
  • Assure disaster recovery readiness using on-demand or scheduled disaster recovery drills

For more information about how to use Oracle Site Guard, see Oracle Site Guard Administrator's Guide.

Patching an Oracle Fusion Middleware Disaster Recovery Site

Apply an Oracle Fusion Middleware patch set to upgrade the Oracle homes that participate in an Oracle Fusion Middleware Disaster Recovery site.

It is assumed that the Oracle Central Inventory for any Oracle Fusion Middleware instance that you are patching is located on the production site shared storage, so that the Oracle Central Inventory for the patched instance can be replicated to the standby site.

To apply an Oracle Fusion Middleware patch:

  1. Perform a backup of the production site to ensure that the starting state is secured.
  2. Apply the patch set to upgrade the production site instances.
  3. After you apply the patch set, manually force a synchronization of the production site shared storage and standby site shared storage. This replicates the production site's patched instance and Oracle Central Inventory in the standby site's shared storage.
  4. After you apply the patch set, use Oracle Data Guard to manually force a synchronization of the Oracle databases at the production site and standby sites. Because some Oracle Fusion Middleware patch sets make updates to repositories, this step ensures that any changes made to production site databases are synchronized to the standby site databases.
  5. The upgrade is now complete. Your Disaster Recovery topology is ready to resume processing.

Note:

Patches must be applied only at the production site for an Oracle Fusion Middleware 12c Disaster Recovery topology. If a patch is for an Oracle Fusion Middleware instance or for the Oracle Central Inventory, the patch is copied when the production site shared storage is replicated to the standby site shared storage. A synchronization operation should be performed when a patch is installed at the production site.

When patching the database, check the specific patch’s documentation on how to apply the patch in a Data Guard topology.