1. Disaster Recovery Guide
  2. Setting Up and Managing Disaster Recovery Sites

3 Setting Up and Managing Disaster Recovery Sites

Learn about the Oracle SOA Suite enterprise deployment topology that illustrates how to set up production and standby sites.

Note:

You can automate disaster recovery operations such as switchover and failover by using Oracle Site Guard. See Oracle Site Guard Administrator's Guide.

This chapter includes the following sections:

Setting Up a Site

Learn how to set up an Oracle Disaster Recovery site.

Before you start creating the production site, ensure that you:

  • Set up the host name aliases for the middle tier hosts, as described in Planning Host Names.

  • Create the required volumes on the shared storage on the production site, as described in Designing Directory Structure and Volumes.

  • Determine the Oracle Data Guard configuration to use based on the data loss requirements of the database and network considerations, such as the available bandwidth and latency when compared to the redo generation.

This section includes the following topics:

Parent topic: Setting Up and Managing Disaster Recovery Sites

Designing Directory Structure and Volumes

Learn about the recommended directory structure in your disaster recovery topology.

You can choose a directory layout different from the one recommended in this document, but the model adopted enables maximum availability, provides the best isolation of components and symmetry in the configuration, and facilitates backup and disaster recovery.

The following list describes directories and directory environment variables:

  • ORACLE_BASE: This environment variable and related directory path refers to the base directory below which Oracle products are installed.

  • ORACLE_HOME: This related directory path refers to the location where Oracle Fusion Middleware resides.

  • WL_HOME: This environment variable and related directory path contains installed files that are necessary to host an Oracle WebLogic Server.

  • PROD_DIR: This environment variable and related directory path refers to the location where a product suite, such as Oracle SOA Suite, Oracle WebCenter Portal, or Oracle Identity Management is installed.

  • DOMAIN directory: This directory path refers to the location where the Oracle WebLogic Domain information (configuration artifacts) is stored. Different WebLogic Servers can use different domain directories even when in the same node.

  • ORACLE_INSTANCE: An Oracle instance contains one or more system components. An Oracle instance directory contains updatable files, such as configuration files, log files, and temporary files.

See Recommended Directory Structure for Oracle SOA Suite.

This section includes the following topic:

Parent topic: Setting Up a Site

Recommended Directory Structure for Oracle SOA Suite

Learn about the recommended directory structures for Oracle SOA Suite.

Oracle Fusion Middleware allows you to create multiple SOA Managed Servers from a single binary installation. This allows the installation of binary files in a single location on a shared storage and the reuse of this installation by the servers in different nodes. However, for maximum availability, Oracle recommends that you use redundant binary installations. In this model, two Oracle homes (each of which has a WL_HOME and an ORACLE_HOME for each product suite) are installed in a shared storage. Additional servers (when scaling out or up) of the same type can use either one of these two locations without requiring more installations. Ideally, users should use two different volumes for redundant binary location, this isolating the failures as much as possible in each volume. For additional protection, Oracle recommends using storage replication for these volumes. If multiple volumes are not available, Oracle recommends that you use mount points to simulate the same mount location in a different directory in the shared storage. Although this does not guarantee the protection that multiple volumes provide, it does allow protection from user deletions and individual file corruption.

Oracle also recommends separating the domain directory that is used by the Administration Server from the domain directory that is used by Managed Servers. This allows a symmetric configuration for the domain directories that is used by Managed Servers, and isolates the failover of the Administration Server. The domain directory for the Administration Server must reside in a shared storage to allow failover to another node with the same configuration. In addition, Oracle recommends that you place the Managed Server's domain directories on a shared storage, although having them on the local file system is also supported. This is especially important when you design a production site with the disaster recovery site in mind. Figure 3-1 shows the directory structure layout for Oracle SOA Suite.

Figure 3-1 Recommended Shared Storage Directory Structure for an Enterprise Deployment

This image shows the recommended shared storage directory structure for an enterprise deployment.

Figure 3-2 Recommended Local Storage Directory Structure for an Enterprise Deployment

This image shows the recommended local storage directory structure for an enterprise deployment.

For information about setting up this directory structure, see Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.

Recommended Volume Design for Oracle SOA Suite

Learn about the recommended volume design for Oracle SOA Suite.

Figure 3-3 and Figure 3-4 shows an Oracle SOA Suite topology diagram. The volume design described in this section is for this Oracle SOA Suite topology. Detailed instructions for installing and configuring this topology are provided in the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.

Figure 3-3 Oracle SOA Suite and Oracle Business Activity Monitoring Enterprise Topology Diagram

This image shows the Oracle SOA, Business Process Management, and Oracle Service Bus enterprise deployment topology.

Figure 3-4 Oracle SOA Suite and Oracle Service Bus Enterprise Deployment Reference Topology Diagram

This image shows the SOA Topology with Oracle Access Manager and Business Process Management.

For disaster recovery of this Oracle SOA Suite topology, Oracle recommends the following volume design:

  • Provision two volumes for two Oracle homes that contain the redundant product binary files (VOLFMW1 and VOLFMW2 in Table 3-1).

  • Provision one volume for the Administration Server domain directory (VOLADMIN in Table 3-1).

  • Provision one volume on each node for the Managed Server domain directory (VOLSOA1 and VOLSOA2 in Table 3-1). This directory is shared among all the Managed Servers on that node.

  • Provision one volume on each node for the Oracle HTTP Server Oracle home (VOLWEB1 and VOLWEB2 in Table 3-1).

  • Provision one volume on each node for the Oracle HTTP Server Domain Directory (VOLOHS1 and VOLOHS2 in Table 3-1).

    Note:

    For WebTier hosts, local storage is usually recommended. You can replicate this configuration on a regular basis to one of the other app tier volumes to sync to standby or directly from the production webhost to the standby webhost.

Table 3-1 provides a summary of Oracle recommendations for volume design for the Oracle SOA Suite topology shown in Figure 3-3 and Figure 3-4.

Table 3-1 Volume Design Recommendations for Oracle SOA Suite

Tier Volume Name Mounted on Host Mount Point Comments

Web

VOLWEB1

WEBHOST1

/u02/oracle/products/fmw

Volume for Oracle HTTP Server installation

Web

VOLWEB2

WEBHOST2

/u02/oracle/products/fmw

Volume for Oracle HTTP Server installation

Web

VOLOHS1

WEBHOST1

/u02/oracle/config/domains/ohs_domain

Volume for Oracle HTTP Server domain directory 

Web

VOLOHS2

WEBHOST2

/u02/oracle/config/domains/ohs_domain

Volume for Oracle HTTP Server domain directory 

Web

VOLSTATIC1

WEBHOST1

/u02/oracle/config/static

(Optional) Volume for static HTML content

Web

VOLSTATIC2

WEBHOST2

/u02/oracle/config/static

(Optional) Volume for static HTML content

Application

VOLFMW1

SOAHOST1

/u01/oracle/products/fmw

Volume for the WebLogic Server and Oracle SOA Suite binary files

Application

VOLFMW2

SOAHOST2

/u01/oracle/products/fmw

Volume for the WebLogic Server and Oracle SOA Suite binary files

Application

VOLADMIN

SOAHOST1, SOAHOST2

/u01/oracle/config

Volume for Administration Server domain directory and other shared configurations, such as Deployment Plans, applications, and keystores

Application

VOLSOA1

SOAHOST1

/u02/oracle/config

Volume for Managed Server domain directory

Application

VOLSOA2

SOAHOST2

/u02/oracle/config

Volume for Managed Server domain directory

Application

VOLRUNTIME

SOAHOST1, SOAHOST2

/u01/oracle/runtime

Volume for shared runtime content.

For example, files used by file adapters, MFT transfers, and other runtime artifacts.

Note:

It is recommended to store JMS messages and TLOGS in the database, using JDBC persistent stores, instead of this folder.

For consistency group recommendations, see:

Parent topic: Designing Directory Structure and Volumes

Recommended Consistency Groups for Oracle SOA Suite

Learn about the recommended consistency groups for Oracle SOA Suite.

Oracle recommends the following consistency groups for the Oracle SOA Suite topology:

  • Create one consistency group with the volumes that contains the domain directories for the Administration Server and Managed Servers as members (DOMAINGROUP in Table 3-2).

  • Create one consistency group with the volume that contains the JMS file store and transaction log data as members (RUNTIMEGROUP in Table 3-2).

  • Create one consistency group with the volume that contains the Oracle homes as members (FMWHOMEGROUP in Table 3-2).

Table 3-2 provides a summary of Oracle recommendations for consistency groups for the Oracle SOA Suite topology as shown in Figure 3-3.

Table 3-2 Consistency Groups for Oracle SOA Suite

Tier Group Name Members Comments

Application

DOMAINGROUP

VOLADMIN

VOLSOA1

VOLSOA2

Consistency group for the Administration Server, and the Managed Server domain directory

Application

RUNTIMEGROUP

VOLRUNTIME

Consistency group for the shared runtime content

Application

FMWHOMEGROUP

VOLFMW1

VOLFMW2

Consistency group for the Oracle homes

Setting Up Storage Replication

Learn how to set up storage replication for the Oracle Fusion Middleware Disaster Recovery topology using Storage level replication, Rsync and Database file system.

  • Storage Level Replication
    The Oracle Fusion Middleware Disaster Recovery solution uses storage replication technology for disaster protection of Oracle Fusion Middleware middle tier components.
  • Rsync
    Alternatively, rsync can be used for replication. Rsync is a versatile copying tool, that can copy locally, or to/from another host over any remote shell. It offers a large number of options that control every aspect of its behavior, and permit very flexible specification of the set of files to be copied.
  • Oracle Database File System
    Oracle Database File System (DBFS) is an additional method that can be used for replicating the configuration. Conceptually, a database file system is a file system interface placed on top of files and directories that are stored in database tables.

Parent topic: Setting Up a Site

Storage Level Replication

The Oracle Fusion Middleware Disaster Recovery solution uses storage replication technology for disaster protection of Oracle Fusion Middleware middle tier components.

To set up storage replication for the Oracle Fusion Middleware Disaster Recovery topology:

  • On the standby site, ensure that the alias host names that are created are the same as the alias host names that are used for the peer hosts at the production site.

  • On the shared storage at the standby site, create the same volumes as created on the shared storage at the production site.

  • On the standby site, create the same mount points and symbolic links that you created at the production site.

    Note:

    • The symbolic links only need to be set up on the standby site if you set up symbolic links at the production site.

    • The symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes. For more information about symbolic links, see Setting Up Storage.

  • It is not necessary to install the same Oracle Fusion Middleware instances at the standby site as installed at the production site. When the production site storage is replicated to the standby site storage, the Oracle software installed on the production site volumes are replicated at the standby site volumes.

  • Create the baseline snapshot copy of the production site shared storage that sets up the replication between the production site and standby site shared storage. Create the initial baseline copy and subsequent snapshot copies by using asynchronous replication mode. After the baseline snapshot copy is performed, validate that all the directories inside the standby site volumes have the same contents as the directories inside the production site volumes.

  • Set up the frequency of subsequent copies of the production site shared storage, which is replicated at the standby site. When asynchronous replication mode is used, then at the requested frequency the changed data blocks at the production site shared storage (based on comparison to the previous snapshot copy) become the new snapshot copy, and the snapshot copy is transferred to the standby site shared storage.

  • Ensure that disaster protection for any database that is included in the Oracle Fusion Middleware Disaster Recovery production site is provided by Oracle Data Guard. Do not use storage replication technology to provide disaster protection for Oracle databases.

  • The standby site shared storage receives snapshots transferred periodically from the production site shared storage. After the snapshots are applied, the standby site shared storage includes all the data up to and including the data contained in the last snapshot transferred from the production site before the failover or switchover.

  • Oracle strongly recommends that you manually force a synchronization operation whenever a change is made to the middle tier at the production site (for example, when a new application is deployed at the production site). Follow the vendor-specific instructions for forcing a synchronization by using storage replication technology.

Rsync

Alternatively, rsync can be used for replication. Rsync is a versatile copying tool, that can copy locally, or to/from another host over any remote shell. It offers a large number of options that control every aspect of its behavior, and permit very flexible specification of the set of files to be copied.

Rsync implements delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Due to these advantages and easy to use ability, it is a widely used tool for backups and mirroring.

Although rsync is reliable and implements implicit retries, the network outages and other connectivity issues can still cause failures in the file synchronization. Hence, rsync can be used as an alternative to the storage level replication under these conditions:

  • When the storage replication is not possible or feasible and/or does not meet the cost requirements.
  • When primary and standby use a reliable and secure network connection for the copy.
  • When checks are performed in the copy to make sure that it is valid.
  • When the disaster recovery site is validated on a regular basis.

For more details about how to use rsync to replicate the file system artifacts, see Using Peer-To-Peer File Copy.

Oracle Database File System

Oracle Database File System (DBFS) is an additional method that can be used for replicating the configuration. Conceptually, a database file system is a file system interface placed on top of files and directories that are stored in database tables.

DBFS is similar to NFS in that it provides a shared network file system that looks like a local file system and has both a server component and a client component. The DBFS file system can be mounted from the mid-tier hosts and accessed as a regular shared file system. It could be used as an intermediate location to replicate content: any content that is copied to the DBFS mount, as it resides in the database, is automatically replicated to the standby site through the underlying Data Guard replication.

This method takes advantage of the robustness of the Data Guard replica. It has good availability through Oracle Driver’s retry logic and provides a resilient behavior. It can be used in scenarios with medium or high latencies between the data centers. However, using DBFS for configuration replication has also additional implications from the setup, database storage and lifecycle perspectives:

  • It introduces some complexity due to the configuration and maintenance required by the DBFS mount. It requires the DB client to be installed in the host that is going to mount it, it requires an initial setup of some artifacts in the database (tablespace, user, and so on) and in the client (the wallet, the tnsnames.ora, and so on ). See the script dbfs_dr_setup_root.sh (in Using the Application DR common scripts) as an example script that installs the database client, creates the DBFS schema in the database, configures the client artifacts, and mounts the DBFS file system in a mid-tier host.
  • It requires additional capacity in the Database, because the content copied to the DBFS mount is stored in the database.
  • Oracle does not recommend to store the domain configuration or the binaries directly in the DBFS mount. This would create a very strong dependency between the FMW files and the database. Instead, it is recommended to use the DBFS as an "assistance" file system: an intermediate staging file system to place the info that is going to be replicated to the standby site. Any replication to standby would have two steps: from the primary's origin folder to the intermediate DBFS mount, and then, in the standby site, from the DBFS mount to the standby's destination folder.
  • The DBFS can be mounted only if the database is open. When the Data Guard is not an Active Data Guard (ADG), the standby database is in mount state. Hence, in order to access to the DBFS mount in the standby site in such cases, the database needs to be converted to snapshot standby. When ADG is used, however, the file system can be mounted for reads and there is no need to transition to snapshot.

Due to the above stated reasons, it is not recommended to use this approach as a general purpose solution to replicate all the artifacts to the standby. For example, using DBFS to replicate the binaries is an overkill. However, this approach is suitable to replicate some dynamic artifacts during the lifecycle, like the domain shared configuration, when other methods like the storage replication or rsync are not feasible. See the Oracle WebLogic Server for Oracle Cloud Infrastructure Disaster Recovery paper to see an example on how to use DBFS to replicate the domain configuration.

Configuring Oracle Data Guard for the FMW Database

Learn how to install and configure Oracle Data Guard for the FMW Database in an Oracle SOA Suite enterprise deployment.

For recommendations and considerations for setting up Oracle databases that are used in an Oracle Fusion Middleware Disaster Recovery topology, see Database Considerations.

Oracle Maximum Availability Architecture (MAA) is Oracle's comprehensive architecture to reduce downtime for scheduled outages, and to prevent, detect and recover, from unscheduled outages.

Real Application Clusters (RAC) and Data Guard provide the basis of the database MAA solution, where the primary site contains the RAC database, and the secondary site contains the RAC physical standby database.

Tip:

Alternatively, you can perform many of the tasks in this section by using Oracle Enterprise Manager Cloud Control.

Setting up and managing databases using Cloud Control helps in controlling downtime and simplifies disaster recovery operations.

For information about Installing Enterprise Manager Cloud Control 13c, see Cloud Control Basic Installation Guide .

For more information about Setting up Oracle Data Guard using Cloud Control, see Provisioning Oracle Standby Databases in Oracle Enterprise Manager Lifecycle Management Administrator's Guide.

Parent topic: Setting Up a Site

Prerequisites and Assumptions

Ensure that the following prerequisites are met:

  • The Oracle RAC cluster and Automatic Storage Management (ASM) instances on the standby site have been created.

  • The Oracle RAC databases on the standby site and the production site are using a flash recovery area.

  • The Oracle RAC databases are running in archivelog mode.

  • The database hosts on the standby site already have Oracle software installed.

  • In a shared ORACLE_HOME configuration, the TNS_ADMIN directory must be a local, non-shared directory.

Oracle Data Guard Environment Description

The examples given in this section contain environment variables as described in Table 3-3.

Table 3-3 Variables Used by Primary and Standby Databases

Variable Primary Database Standby Database

Database names

soa

soa

SOA Database Host Names

dbhost1.example.com, dbhost2.example.com

dbhost1stby.example.com, dbhost2stby.example.com

Database unique names

soa_pri

soa_stby

Instance names

soa1, soa2

soa1, soa2

Service names

soa_pri.example.com , soaedg.example.com

soa_stby.example.com , soaedg.example.com
Procedure for Configuring the Data Guard

The configuration of a Data Guard involves several steps: you need to prepare the primary database with the recommended parameters, prepare the TNS aliases in primary and standby environments, create the physical standby database as a duplication of the primary database, configure the Data Guard Broker, and so on.

Oracle provides a set of sample scripts that can be used to automate most of these actions. These scripts are available here: https://github.com/oracle-samples/maa/raw/main/dg_setup_scripts/dg_setup_scripts.zip. These scripts are designed to setup a standby database for an existing primary database, using the restore from service feature and Data Guard Broker. They allow customization (OS user names and Oracle Homes are configurable values). They support databases with and without TDE encryption, and they work in environments with Read Only Oracle Home (ROOH) or with regular Oracle Homes. The scripts are validated in 12c (12.2), 18c, 19c and 21c RDBMS versions. Download the file and follow the instructions included in the README.md to configure a standby database.

If the scripts are not feasible for your specific environment, you can manually configure the Data Guard by following one of these documents:

  • Creating a Physical Standby database using RMAN restore database from service (Doc ID 2283978.1)
  • Creating a Physical Standby using RMAN Duplicate (RAC or Non-RAC) (Doc ID 1617946.1)

Note:

You can find the above documents in My Oracle Support.
Verifying the Data Guard Broker Configuration

Complete the following steps to verify that the Data Guard Broker configuration was created successfully.

  1. Verify the Oracle Data Guard configuration by querying the V$ARCHIVED_LOG view to identify existing files in the archived redo log. For example:
    SQL> SELECT SEQUENCE#, FIRST_TIME, NEXT_TIME 
  2> FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;

SEQUENCE# FIRST_TIME NEXT_TIME
---------- ------------------ ------------------
         8 11-DEC-21 17:50:45 11-DEC-21 17:50:53
         9 11-DEC-21 17:50:53 11-DEC-21 17:50:58
         10 11-DEC-21 17:50:58 11-DEC-21 17:51:03
3 rows selected
  2. On the primary database, issue the following SQL statement to force a log switch and archive the current online redo log file group:
    SQL> alter system archive log current;
  3. On the standby database, query the V$ARCHIVED_LOG view to verify that the redo data was received and archived on the standby database:
    SQL> SELECT SEQUENCE#, FIRST_TIME, NEXT_TIME 
  2> FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;

SEQUENCE# FIRST_TIME NEXT_TIME
---------- ------------------ ------------------
        8 11-DEC-21 17:50:45 11-DEC-21 17:50:53
        9 11-DEC-21 17:50:53 11-DEC-21 17:50:58
       10 11-DEC-21 17:50:58 11-DEC-21 17:51:03
       11 11-DEC-21 17:51:03 11-DEC-21 18:34:11

4 rows selected
  4. Use the show configuration command in Data Guard Broker command line to verify that the configuration was created successfully. See Example 3-1.

Example 3-1 Verifying the Data Guard Broker Configuration

[oracle@dbhost1 ~]$ dgmgrl sys/'<password>'
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Tue Feb 1 09:00:16 2022
Version 19.6.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.
Welcome to DGMGRL, type "help" for information.
Connected to "soa_pri"
Connected as SYSDBA.

DGMGRL> show configuration
Configuration - dg_config

Protection Mode: MaxPerformance
Databases:
soa_pri - Primary database
soa_stby - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS
Testing Database Switchover and Switchback

You can perform a database switchover and switchback.

Performing a Switchover Operation by Using Oracle Data Guard Broker

To perform a switchover operation by using Oracle Data Guard Broker, complete the following tasks:

  1. Verify the Oracle Data Guard Broker configuration by running the following command:

    DGMGRL> show configuration;
 
Configuration - dg_config
 
Protection Mode: MaxPerformance
Databases:
soa_pri  - Primary database
soa_stby - Physical standby database
 
Fast-Start Failover: DISABLED
 
Configuration Status:
SUCCESS

  2. Swap the roles of the primary and standby databases by running the SWITCHOVER command. Example 3-1 shows how Data Guard Broker automatically shuts down and restarts the old primary database as part of the switchover operation. 

    DGMGRL> switchover to 'soa_stby'
Performing switchover NOW, please wait...
Operation requires a connection to database "soa_stby"
Connecting ...
Connected to "soa_stby"
Connected as SYSDBA.
New primary database "soa_stby" is opening...
Oracle Clusterware is restarting database "soa_pri" ...
Connected to "soa_pri"
Connected to "soa_pri"
Switchover succeeded, new primary is "soa_stby"

  3. After the switchover is complete, use the SHOW CONFIGURATION command to verify that the switchover operation was successful: 

    DGMGRL> show configuration;
Configuration - dg_config
Protection Mode: MaxPerformance
Databases:
soa_stby - Primary database
soa_pri  - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS

Note:

For information about switchover and failover operation of Oracle Data Guard Broker, see Switchover and Failover Operations in the Oracle Data Guard Broker.

Creating a Production Site on an Oracle SOA Enterprise Topology

Learn how to create a production site on an Oracle SOA enterprise deployment topology.

Before you create your production site:

  • Set up the host name aliases for the middle tier hosts, as described in Planning Host Names.

  • Create the required volumes on the shared storage on the production site, as described in Designing Directory Structure and Volumes.

  • Determine the Oracle Data Guard configuration to use based on the data loss requirements of the database and network considerations, such as the available bandwidth and latency when compared to the redo generation.

This section includes the following topics:

Parent topic: Setting Up and Managing Disaster Recovery Sites

Creating a Production Site

This section provides information about how to create a production site for the Oracle SOA Suite topology.

If you plan to create a production site for a different topology, see the appropriate Oracle Fusion Middleware Enterprise Deployment Guide listed under the Install a Production Environment: Plan, Install & Configure an Enterprise Deployment category.

Install and configure your production site as described in the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.

The following sections describe how to complete the installation and configuration of your production site:

Parent topic: Creating a Production Site on an Oracle SOA Enterprise Topology

Creating Volumes and Consistency Groups

Create volumes and consistency groups on the shared storage device.

To create volumes and consistency groups on the shared storage device, see Recommended Volume Design for Oracle SOA Suite.

Setting Up Physical Host Names and Alias Host Names

Set up physical host names on the production site and set up the physical host names and alias hostnames on the standby site.

For information about planning host names for the production and standby sites, see Planning Host Names.

Installing and Configuring Oracle SOA Suite

Install and configure Oracle SOA Suite.

To install and configure Oracle SOA Suite, see Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite and apply the following modifications:

  1. Install the Oracle SOA Suite components into the volumes created on the shared storage device.
  2. Set up the production and standby sites by using the aliases to the physical and virtual host names.
  3. Create SSL certificates by using the host name aliases on all of the Oracle Fusion Middleware hosts for proper Node Manager communication.

Configuring Data Sources for Oracle Fusion Middleware Active-Passive Deployment

Configure the data sources that Oracle Fusion Middleware uses for active-passive disaster recovery.

As explained in the Setting Up DataSources in the Middle Tier, Oracle recommends that you use a TNS alias in the data sources to access each site's local database. The TNS alias is the same name in production and standby. Hence, the data sources use the same db connect string. The TNS alias is resolved with a tnsnames.ora file stored separately from the WebLogic domain configuration and not replicated between sites. Therefore, you can have different tnsnames.ora content in each site. Each site will resolve the TNS alias with the appropriate connection string in each site, pointing to the local database only.

Another advantage of the active-passive deployment approach is that changes (such as retries and timeouts) to the tnsnames.ora file can be done without requiring a WebLogic Server restart or a WebLogic Server data source restart. At the same time, this approach reduces the repetition of addresses and settings that are typically shared by many data sources, thereby reducing the configuration creation and maintenance footprint.

If you are not already using this approach in the production system, you can use the following steps to configure it. Configure a tns alias in all the data sources that are used in the domain, including the data sources used by the JDBC persistence stores, leasing data sources, and custom data sources, and in the JDBC URL of the OPSS security stores.

  1. Create the tns folder in all the middle-tier hosts.

    This folder must be readable by the Oracle user and placed in a file system that is not replicated between sites.

    For example:
    mkdir -p /home/oracle/tnsnames_dir

    Given that the tns folder is part of the configuration, you can also create it under the config folder that is shared by all the servers. But in that case, ensure that you exclude the tns folder when you copy the domain configuration from primary to standby or update tnsnames.ora in the standby system, after a failover or switchover, to point to the secondary database.

    Note:

    Create this folder in the middle-tier hosts in both production and standby.

  2. Create a tnsnames.ora file in the tns folder, with the tns alias that will be used in the data sources. In the production middle-tier hosts, the alias must point to the production database.

    For example: 

    SOAEDG= 
(DESCRIPTION=
  (ADDRESS_LIST=
    (LOAD_BALANCE=ON)
    (ADDRESS=(PROTOCOL=TCP)(HOST=prmy-scan)(PORT=1521))
  )
  (CONNECT_DATA=(SERVICE_NAME=soaedg.example.com))
)

    In the standby middle-tier hosts, the alias is the same name, but it is pointing to the standby database.

    For example:

    SOAEDG= 
(DESCRIPTION=
  (ADDRESS_LIST=
    (LOAD_BALANCE=ON)
    (ADDRESS=(PROTOCOL=TCP)(HOST=stby-scan)(PORT=1521))
  )
  (CONNECT_DATA=(SERVICE_NAME=soaedg.example.com))
)

    If you connect to additional databases or services, ensure that you add the appropriate tns alias to them too.

  3. Specify the oracle.net.tns_admin property pointing to the directory location of the tnsnames.ora file. Use one of the following methods:

    Note:

    Do not use a mix of these methods. It can cause unexpected behavior.

    Option 1: Set the property as a data source connection property. Oracle recommends this method.

    Specify the oracle.net.tns_admin=tns_directory property in the data source configuration. To specify this property in the WebLogic Administration Console, go to Services, click Data Sources, select a data source from the list, click Connection Pool, and then add it to the Properties text box. Repeat this step for each data source.

    For example:

    Add the following property to the Properties text box:
    oracle.net.tns_admin=/home/oracle/tnsnames_dir

    You must also specify this property in the OPSS security stores files jps-config-jse.xml and jps-config.xm available in the $ASERVER_HOME/config/fmwconfig folder. To modify these jps files, edit them and add the oracle.net.tns_admin property after the jdbc.url property.

    For example:
    …
<property name="jdbc.url" value="jdbc:oracle:thin:@soaedg"/>
<property name="oracle.net.tns_admin" value="tns_directory"/>
…

    Note:

    This property applies to the specific file (data source, jps file) in which it is specified.

    Option 2: Set the property as a java system property.

    Specify the -Doracle.net.tns_admin=tns_directory system property where tns_directory is the directory location of the tnsnames.ora file. To set it as a java property for the servers, edit the following files:
    • $ASERVER_HOME/bin/setUserOverrides.sh
    • $MSERVER_HOME/bin/setUserOverrides.sh (This file is not shared. Therefore, you should edit the file in all the SOA mid-tier hosts.)
    Add the following content to these files:
    # For using tns alias in the datasources 
export EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES} -Doracle.net.tns_admin=/home/oracle/tnsnames_dir

    Note:

    • This property applies to all the data sources and jps files in WebLogic Server.
    • Before you run some of the WLST commands and the Configuration Wizard, this approach requires that you set the property in the environment. Before running WLST, you should set the property in the WLST_PROPERTIES environment variable. Before running the Configuration Wizard, you should add the property to the JVM_ARGS environment variable of the config_internal.sh script.

    Option 3: Set the property in the jdbc URL.

    Specify the location of the tnsnames.ora file as part of the connection string in the data sources and jps files: 
    jdbc:oracle:thin:@alias?TNS_ADMIN=tns_directory

    Note:

    • This property applies to the specific file (data source, jps file) in which it is specified.
    • You can use this method with JDBC Driver 18.3 and later. This applies to Fusion Middleware 12.2.1.4 (which uses JDBC Driver 19.3) and later.

  4. Modify the URL defined in the data sources by replacing the connection string with the alias. The following is a sample JDBC URL using the tns alias jdbc:oracle:thin:@soaedg.

    You can use the WebLogic Administration Console to perform this change. Alternatively, you can perform the modification directly in the files, as described here:
    1. Sign in to APPHOST1.
    2. Take a backup of $ASERVER_HOME/config/jdbc which contains the data source config files.
    3. Run a command to replace the previous database connection string by the new one that uses the tns alias.
      For example:
      cp -rf $ASERVER_HOME/config/jdbc   $ASERVER_HOME/config/jdbc_bck
export PREV_STRING='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=prmy-scan)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=soaedg.example.com)))'
export NEW_STRING='soaedg'
cd $ASERVER_HOME/config/jdbc
find . -name '*.xml' | xargs sed -I 's|'${PREV_STRING}'|'${NEW_STRING}'|gI'
  5. Update the JDBC URL of the OPSS security stores too by using the following steps:
    1. Sign in to APPHOST1 and go to the $ASERVER_HOME/config/fmwconfig folder.
    2. Take a backup of the jps-config-jse.xml and jps-config.xml files.
    3. Edit both files to update the value of the property name="jdbc.url" with the appropriate JDBC URL used in the data sources.
      For example:
      cp -rf $ASERVER_HOME/config/fmwconfig $ASERVER_HOME/config/fmwconfig_bck
cd $ASERVER_HOME/config/fmwconfig 
export PREV_STRING='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=prmy-scan)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=soaedg.example.com)))' 
export NEW_STRING='soaedg' 
find . -name '*.xml' | xargs sed -i 's|'${PREV_STRING}'|'${NEW_STRING}'|gI'
  6. Restart all the WebLogic Servers in the domain for the changes to take effect.
    1. Stop all the WebLogic Servers in the domain (Administration Server and Managed Servers).
    2. Start the Administration Server in the domain.
    3. Start the Managed Servers.

  7. Verify that the data source connections are established correctly with the database.

    Verify that the JDBC URL is updated correctly in the OPSS store. To verify this:
    1. Go to the Enterprise Manager Console.
    2. Navigate to WebLogic Domain, select Security, and then click Security Provider Configuration.
    3. Expand Security Stores and verify that the Database URL is updated.

Note:

Regarding the ONS host and port configuration in the data sources, these values are not required when you are using Oracle Database 12c and later because the ONS list is automatically obtained from the database by the client driver.

As per Oracle's consistent recommendation in this guide, use this feature instead of providing the scan address or the list of the RAC nodes in the ONS configuration of each data source.

Ensure that Fast Application Notification (FAN) is enabled and that the ONS nodes property is empty in each data source. To check this property in the WebLogic Administration Console, go to Services, click Data Sources, select Configuration, and then click ONS.

Parent topic: Creating a Production Site on an Oracle SOA Enterprise Topology

Creating a Standby Site

Learn how to create a standby site.

To create the standby site, the Oracle SOA enterprise deployment topology is used as an example.

This section includes the following topics:

Parent topic: Setting Up and Managing Disaster Recovery Sites

Preparing the Standby Site

Prepare your standby site for operation.

To prepare it for operation, on your standby site:

  • Set up the correct alias host names and physical host names by following the instructions in Planning Host Names.

    Ensure that each standby site host has an alias host name that is the same as the physical host name of its peer host at the production site.

  • Create, on the shared storage, the same volumes that were created on the shared storage at the production site.

  • Create the same mount points and symbolic links (if required) that you created at the production site. Note that symbolic links are required only in cases where the storage system does not guarantee consistent replication across multiple volumes.

    For more details about symbolic links, see Setting Up Storage.

Parent topic: Creating a Standby Site

Setting Up Middle Tier Hosts

Middle tier hosts on a standby site do not require installation nor configuration of Oracle Fusion Middleware or Oracle WebLogic Server software. When the production site storage is replicated to the standby site storage, the software installed on the production site is replicated at the standby site.

To set up the middle tier hosts on the standby site:

  1. Create a baseline snapshot copy of the shared storage on the production site, which sets up the replication between the storage devices. Create the initial baseline copy and subsequent snapshot copies by using asynchronous replication mode.
  2. Synchronize the shared storage at the production site with the shared storage at the standby site. This transfers the initial baseline snapshot from the production site to the standby site.
  3. Set up the frequency of subsequent copies of the production site shared storage, which is replicated at the standby site. When asynchronous replication mode is used, then at the requested frequency the changed data blocks at the production site shared storage (based on comparison to the previous snapshot copy) become the new snapshot copy, and the snapshot copy is transferred to the standby site shared storage.
  4. After the baseline snapshot copy is performed, validate that all the directories inside the standby site volumes have the same content as the directories inside the production site volumes.
About Self-Signed certificates and Keys on the Standby Site

Learn about certificates and keystores.

As recommended in Planning Host Names the FMW components use alias host names, that are resolvable to the IP addresses of the appropriate system in each site. If the primary self-signed certificates are created with these host names, the certificates are valid both for the production and for the standby systems.

For more information about creating primary self-signed certificates, see Common Configuration and Management Tasks for an Enterprise Deployment in Enterprise Deployment Guide for Oracle SOA Suite.

The certificates and keystores are replicated to standby along with the configuration, so there is no need to create specific self-signed certificates or keystores for the standby site.

Validating the Standby Site Setup

Validate your standby site.

To validate a standby site:

  1. Shut down any processes still running on the production site. These include the database instances in the data tier, Oracle Fusion Middleware instances, and any other processes in the application tier and web tier.
  2. Stop the replication between the production site shared storage and the standby site shared storage.
  3. Perform a switchback operation. A switchback operation is a subsequent switchover operation to return the roles to their original state.
  4. On the standby site host, manually start all the processes. These include the database instances in the data tier, Oracle Fusion Middleware instances, and any other processes in the application tier and web tier.
  5. Use a browser client to perform post-failover testing to confirm that requests are being resolved and redirected to the standby site.

Parent topic: Creating a Standby Site

Performing Site Operations and Administration

Learn how to operate and administer your Oracle Fusion Middleware Disaster Recovery topology.

This section includes the following topics:

  • Synchronizing the Production and Standby Sites
    Learn how to force a synchronization of the production and standby sites when you introduce a change in the middle tier at the production site.
  • Performing a Switchover
    A switchover operation sets the standby site as the production role.
  • Performing a Switchback
    A switchback operation reverts the roles of the current production and standby sites.
  • Performing a Failover
    A failover operation sets the standby site as the production role when the production site becomes unavailable.
  • Testing the Standby Site
    Learn how to create a clone of the read-only standby site shared storage and use it for converting database secondary_db_unqname to snapshot standby.
  • Using Peer-To-Peer File Copy
    The rsync utility (which uses peer-to-peer file copy) can be used to replicate middle tier file system data from a production site host to a standby site peer host in an Oracle Fusion Middleware Disaster Recovery topology. The use of the rsync utility is explained in the context of symmetric topologies.

Parent topic: Setting Up and Managing Disaster Recovery Sites

Synchronizing the Production and Standby Sites

Learn how to force a synchronization of the production and standby sites when you introduce a change in the middle tier at the production site.

In normal operations, the standby site shared storage receives snapshots transferred periodically from the production site shared storage. After the snapshots are applied, the standby site shared storage includes all the data up to the data contained in the last snapshot transferred from the production site before the failover or switchover.

Be sure to force a synchronization when you introduce a change to the middle tier at the production site such as, for example, when you deploy a new application at the production site. Follow the vendor-specific instructions to force a synchronization by using the storage replication technology.

The databases synchronization in an Oracle Fusion Middleware Disaster Recovery topology is managed by Oracle Data Guard.

Parent topic: Performing Site Operations and Administration

Performing a Switchover

A switchover operation sets the standby site as the production role.

This operation is needed when you plan to take down the production site (for example, to perform maintenance) and to make the current standby site the production site.

To perform a switchover operation:

  1. Shut down any processes running on the production site. These include the database instances in the data tier, Oracle Fusion Middleware instances, and any other processes in the application tier and web tier.
  2. Stop the replication between the production site shared storage and the standby site shared storage.
  3. Unmount the shared storage volume with the middle tier artifacts, on the current production site, and mount the corresponding volumes on the current standby site, which is the new production site.
  4. Use Oracle Data Guard to switch over the databases.
  5. On the standby site host, manually start all the processes. These include the database instances in the data tier, Oracle Fusion Middleware instances, and any other processes in the application tier and web tier.
  6. Ensure that all user requests are routed to the standby site by performing a global DNS push or something similar, such as updating the global load balancer.
  7. Use a browser client to perform post-switchover testing to confirm that requests are being resolved and redirected to the standby site.

    At this point, the former standby site is the new production site and the former production site is the new standby site.

  8. Reestablish the replication between the two sites, but configure the replication so that the snapshot copies go in the opposite direction (from the current production site to the current standby site). See the documentation for your shared storage to learn how to configure the replication so that snapshot copies are transferred in the opposite direction.

At this point, the former standby site becomes the new production site, and you can perform maintenance at the original production site. After you have carried out the maintenance of the original production site, you can use it as either the production site or the standby site.

Note:

This note is applicable for BI-specific systems only.

After a switchover operation, the creation of an Essbase cube with CDS may fail with an error similar to the following:

oracle.essbase.cds.util.CDSException: oracle.essbase.cds.util.CDSException: java.sql.SQLException: ORA-25153:

To work around this issue and to create an Essbase cube, on the current primary database:

  • Identify the temporary tablespaces by using a select statement similar to the following (where BIS17V1 is the Oracle Business Intelligence RCU prefix):

    select username,temporary_tablespace from dba_users where username like 'BIS17V1%'

    Assume that the above command returns the following list of temporary tablespaces:

    USERNAME.....TEMPORARY_TABLESPACE

    BIS17V1_IAU_VIEWER.....BIS17V1_IAS_TEMP

    BIS17V1_STB.....BIS17V1_IAS_TEMP     

    BIS17V1_IAU_APPEND.....BIS17V1_IAS_TEMP

    BIS17V1_MDS.....BIS17V1_IAS_TEMP

    BIS17V1_IAU.....BIS17V1_IAS_TEMP

    BIS17V1_BIPLATFORM......BIS17V1_IAS_TEMP

    BIS17V1_OPSS.....BIS17V1_IAS_TEMP

  • After the switchover, drop the tablespace BIS17V1_IAS_TEMP including contents and datafiles.

  • Create the temporary tablespace BIS17V1_IAS_TEMP, as a tempfile, in the location (for example) /work/primy/oradata/stnby/BIS17V1_IAS_TEMP.dbf, with size 250 m.

  • Issue the following alter commands (here is where you use the list temporary tablespaces):

    alter user BIS17V1_OPSS temporary tablespace  BIS17V1_IAS_TEMP ;

    alter user BIS17V1_BIPLATFORM temporary tablespace  BIS17V1_IAS_TEMP ;

    alter user BIS17V1_IAU temporary tablespace  BIS17V1_IAS_TEMP ;

    alter user BIS17V1_MDS temporary tablespace  BIS17V1_IAS_TEMP ;

    alter user BIS17V1_IAU_APPEND temporary tablespace  BIS17V1_IAS_TEMP ;

    alter user BIS17V1_STB temporary tablespace  BIS17V1_IAS_TEMP ;

    alter user BIS17V1_IAU_VIEWER temporary tablespace  BIS17V1_IAS_TEMP ;

To use the original production site as the production site, perform a switchback as explained in Performing a Switchback.

Parent topic: Performing Site Operations and Administration

Performing a Switchback

A switchback operation reverts the roles of the current production and standby sites.

To perform a switchback operation:

  1. Shut down any processes running on the current production site. These include the database instances in the data tier, Oracle Fusion Middleware instances, and any other processes in the application tier and web tier.
  2. Stop the replication between the current production site shared storage and standby site shared storage.
  3. Unmount the shared storage volume with the middle tier artifacts, on the current production site, and mount the corresponding volumes on the current standby site, which is the new production site.
  4. Use Oracle Data Guard to switch back the databases.
  5. On the new production site hosts, manually start all the processes. These include Oracle Fusion Middleware instances and any other processes in the application tier and web tier.
  6. Ensure that all user requests are routed to the new production site by performing a global DNS push or something similar, such as updating the global load balancer.
  7. Use a browser client to perform post-switchback testing to confirm that requests are being resolved and redirected to the new production site.

    At this point, the former standby site is the new production site and the former production site is the new standby site.

  8. Reestablish the replication between the two sites, but configure the replication so that the snapshot copies go in the opposite direction (from the new production site to the new standby site). See the documentation for your shared storage to learn how to configure the replication so that snapshot copies are transferred in the opposite direction.

Parent topic: Performing Site Operations and Administration

Performing a Failover

A failover operation sets the standby site as the production role when the production site becomes unavailable.

To perform a failover operation:

  1. Stop the replication between the production site shared storage and the standby site shared storage.
  2. Mount the shared storage volume with the middle-tier artifacts, on the current standby site, which is the new production site.
  3. From the standby site, use Oracle Data Guard to fail over the databases.
  4. On the standby site hosts, manually start all the processes. These include the Oracle Fusion Middleware instances and any other processes in the application tier and web tier.
  5. Ensure that all user requests are routed to the standby site by performing a global DNS push or something similar, such as updating the global load balancer.
  6. Use a browser client to perform post-failover testing to confirm that requests are being resolved and redirected to the production site.

    At this point, the standby site is the new production site. You can examine the issues that caused the former production site to become unavailable.

  7. To use the original production site as the current standby site, you must reestablish the replication between the two sites, but configure the replication so that the snapshot copies go in the opposite direction (from the current production site to the current standby site). See the documentation for your shared storage system to learn how to configure the replication so that snapshot copies are transferred in the opposite direction.

Note:

After a failover operation, the creation of an Essbase cube with CDS may fail with an error similar to the following:

oracle.essbase.cds.util.CDSException: oracle.essbase.cds.util.CDSException: java.sql.SQLException: ORA-25153:

To work around this issue and to create an Essbase cube, on the current primary database:

  • Identify the temporary tablespaces by using a select statement similar to the following, where BIS17V1 is the Oracle Business Intelligence RCU prefix):

    select username,temporary_tablespace from dba_users where username like 'BIS17V1%'

    Assume that the above command returns the following list of temporary tablespaces:

    USERNAME.....TEMPORARY_TABLESPACE

    BIS17V1_IAU_VIEWER.....BIS17V1_IAS_TEMP

    BIS17V1_STB.....BIS17V1_IAS_TEMP     

    BIS17V1_IAU_APPEND.....BIS17V1_IAS_TEMP

    BIS17V1_MDS.....BIS17V1_IAS_TEMP

    BIS17V1_IAU.....BIS17V1_IAS_TEMP

    BIS17V1_BIPLATFORM......BIS17V1_IAS_TEMP

    BIS17V1_OPSS.....BIS17V1_IAS_TEMP

  • After the failover, drop the tablespace BIS17V1_IAS_TEMP including contents and datafiles.

  • Create the temporary tablespace BIS17V1_IAS_TEMP, as a tempfile, in location. For example, /work/primy/oradata/stnby/BIS17V1_IAS_TEMP.dbf with size 250 m.

  • Issue the following alter commands (here is where you use the list temporary tablespaces):

    alter user BIS17V1_OPSS temporary tablespace  BIS17V1_IAS_TEMP ;

    alter user BIS17V1_BIPLATFORM temporary tablespace  BIS17V1_IAS_TEMP ;

    alter user BIS17V1_IAU temporary tablespace  BIS17V1_IAS_TEMP ;

    alter user BIS17V1_MDS temporary tablespace  BIS17V1_IAS_TEMP ;

    alter user BIS17V1_IAU_APPEND temporary tablespace  BIS17V1_IAS_TEMP ;

    alter user BIS17V1_STB temporary tablespace  BIS17V1_IAS_TEMP ;

    alter user BIS17V1_IAU_VIEWER temporary tablespace  BIS17V1_IAS_TEMP ;

To again use the original production site as the production site, perform a switchback as explained in Performing a Switchback.

Parent topic: Performing Site Operations and Administration

Testing the Standby Site

Learn how to create a clone of the read-only standby site shared storage and use it for converting database secondary_db_unqname to snapshot standby.

A typical Oracle Fusion Middleware Disaster Recovery configuration uses:

  • Storage replication to copy Oracle Fusion Middleware middle tier file systems and data from the production site shared storage to the standby site shared storage. During normal operation, the production site is active and the standby site is passive. When the production site is active, the standby site is passive and the standby site shared storage is in read-only mode; the only write operations made to the standby site shared storage are the storage replication operations from the production site shared storage to the standby site shared storage.

  • Oracle Data Guard to copy database data for the production site Oracle databases to the standby databases at standby site. By default, the production site databases are active and the standby databases at the standby site are passive. The standby databases at the standby site are in Managed Recovery mode while the standby site is in the standby role (is passive). When the production site is active, the only write operations made to the standby databases are the database synchronization operations performed by Oracle Data Guard.

  • The standby site as the production role when the production site becomes unavailable. If the current production site becomes unavailable unexpectedly, then a failover operation (described in Performing a Failover) is performed to enable the standby site to assume the production role. Or, if the current production site is taken down intentionally (for example, for planned maintenance), then a switchover operation (described in Performing a Switchover) is performed to enable the standby site to assume the production role.

The usual method of testing a standby site is to shut down the current production site and perform a switchover operation to enable the standby site to assume the production role.

However, some enterprises may want to perform periodic testing of their Disaster Recovery standby site without shutting down the current production site and a complete switchover operation. This is possible by converting the standby database to snapshot standby. This allows the standby servers to be started in the standby site and verify the secondary system. Any change performed in the standby site database while it is in snapshot standby mode will be discarded once it is converted to physical standby again, so primary data will not be affected by secondary site validations.

To use this testing method:

  1. Use the cloning technology provided by the shared storage vendor to create a clone of the standby site's read-only volumes on the shared storage at the standby site. Ensure that the cloned standby site volumes are writable. If you want to test the standby site just once, then this can be a one-time clone operation. However, if you want to test the standby site regularly, you can set up periodic cloning of the standby site read-only volumes to the standby site's cloned read/write volumes.

  2. Convert the standby database into snapshot standby by the following steps:

    1. Use Data Guard broker in primary database host to convert the secondary database to snapshot standby.

      Example:

      [oracle@primarydbhost~]$dgmgrl sys/your_sys_password@primary_db_unqname
DGMGRL> convert database “secondary_db_unqname” to snapshot standby

    2. . Use show configuration command to verify that the conversion has been correctly performed.

  3. On the standby site computers, modify the mount commands to point to the volumes on the standby site's cloned read/write shared storage by following these steps:

    1. Unmount the read-only shared storage volumes.

    2. Mount the cloned read/write volumes at the same mount point.

  4. Before you test the standby site, modify the host name resolution method for the computers that are used to perform the testing to ensure that the host names point to the standby site computers and not the production site computers. For example, on a Linux computer, change the /etc/hosts file to point to the virtual IP of the load balancer for the standby site.

  5. Perform the standby site testing.

Note:

This operation must be done with caution in SOA environments: if there are pending messages or composites in the database when it is converted into snapshot, the standby site's SOA servers will process them when they start. Check that there are no pending actions in primary database when converting to snapshot standby, otherwise, remove records from runtime SOA tables in the standby database after it is converted to snapshot standby database and before starting the secondary site's SOA servers. See, Removing Records from the Runtime Tables Without Dropping the Tables.

After you complete the standby site testing, follow these steps to begin using the original production site as the production site again:

  1. Modify the mount commands on the standby site computers to point to the volumes on the standby site's read-only shared storage. In other words, reset the mount commands back to what they were before the testing was performed.

    1. Unmount the cloned read/write shared storage volume.

    2. Mount the read-only shared storage volumes.

    At this point, the mount commands are reset to what they were before the standby site testing was performed.

  2. Convert the standby database into snapshot standby by the following steps:

    1. Use Data Guard broker in primary database host to convert the secondary database to physical standby again.

      Example: 

      [oracle@primarydbhost~]$dgmgrl sys/your_sys_password@primary_db_unqname
DGMGRL> convert database “secondary_db_unqname” to physical standby

    2. Use show configuration command to verify that the conversion has been correctly performed.

  3. Before you use the original production site again, modify the host name resolution method for the computers that are used to access the production site to ensure that the host names point to the production site computers and not the standby site computers. For example, on a Linux computer, change the /etc/hosts file to point to the virtual IP of the load balancer for the production site.

Parent topic: Performing Site Operations and Administration

Using Peer-To-Peer File Copy

The rsync utility (which uses peer-to-peer file copy) can be used to replicate middle tier file system data from a production site host to a standby site peer host in an Oracle Fusion Middleware Disaster Recovery topology. The use of the rsync utility is explained in the context of symmetric topologies.

For information about the conditions for using the rsync in Disaster Recovery environments, see the point rsync in the section Setting Up Storage Replication of this document.

Ensure that you are familiar with storage replication and Oracle Data Guard in an Oracle Fusion Middleware Disaster Recovery topology, because there are many similarities between using storage replication and using the rsync utility for disaster protection and disaster recovery of your Oracle Fusion Middleware components.

Note the following important differences between using storage replication technologies and using the rsync utility to replicate middle tier file systems:

  • When you use storage replication, you can roll changes back to the point in time when any previous snapshot was taken at the production site.

    When you use the rsync utility, replicated production site data overwrites the standby site data, and you cannot roll back a replication.

  • When you use storage replication, the volume that you set up for each host cluster in the shared storage systems ensures data consistency for that host cluster across the production site's shared storage system and the standby site's shared storage system.

    When you use the rsync utility, data consistency is not guaranteed.

This section includes the following topic:

Parent topic: Performing Site Operations and Administration

Using rsync and Oracle Data Guard in Oracle Fusion Middleware Disaster Recovery Topologies

Learn how to use the UNIX rsync utility and Oracle Data Guard in your Oracle Fusion Middleware Disaster Recovery topology.

Note:

For information about how to set up Oracle Data Guard for Oracle database, see Database Considerations.

The following sections describe how to use the rsync utility and Oracle Data Guard to protect and force synchronization between your production and standby sites in an Oracle Fusion Middleware Disaster Recovery topology:

Parent topic: Using Peer-To-Peer File Copy

Using rsync for Oracle Fusion Middleware Middle Tier Components

Use the UNIX rsync utility to protect and recover your Oracle Fusion Middleware middle tier components.

To use the rsync utility:

  1. Install and set up rsync to enable replication of files from a production site host to its standby site peer host. For instructions about how to install, set up, and use the utility, see the utility man pages and also http://rsync.samba.org.
  2. For each production site host on which one or more Oracle Fusion Middleware components has been installed, set up rsync to copy the following directories and files to the same directories and files on the standby site peer host:

    • The Oracle home directory, subdirectories, and all the files in them.

    • The Oracle Central Inventory directory and files for the host, which includes the Oracle Universal Installer entries for the Oracle Fusion Middleware installations.

    • The shared config folder, such as the WebLogic Administration Server domain directory, deployment plans, applications, keystores, and so on. As this folder is shared between the mid-tier hosts and you do not have to have a for each host.

    • The private config folders, such as the WebLogic managed server's domain, local node manager home on the host.

    • If applicable, the shared runtime folder.

    • If applicable, the Oracle Fusion Middleware static HTML pages directory for the Oracle HTTP Server installations on the host.

    • If applicable, the .fmb and .fmx deployment files that are created by Oracle Forms on the host, and the .rdf deployment artifact files that are created by Oracle Reports on the host.

      Note:

      Run the rsync utility as root or as the OS user that is the owner of the files. If you want it to work without prompting users for a password, set up SSH keys between the production site host and standby site host, so that SSH does not prompt for a password.

      For an example of using rsync utility to copy folders to a remote node, see rsync_copy_and_validate.sh generic script (available here https://github.com/oracle-samples/maa/raw/main/hybrid_dr/hybrid_dr_rsync_scripts/hybrid_dr_rsync_scripts.zip). The rsync_copy_and_validate.sh copies a folder to a remote node, and performs a checksum validation of the copy. The zip file also includes a few examples to use this generic script for copying the typical FMW folders.

  3. Set up scheduled jobs, for example, cron jobs, for the production site hosts for which you set up rsync in the previous step. These scheduled jobs enable rsync to automatically perform replication of these files from the production site hosts to the standby site hosts on a regular interval. An interval of once a day is recommended for a production site where the Oracle Fusion Middleware configuration does not change very often.
  4. Whenever a change is made to the configuration of an Oracle Fusion Middleware middle tier configuration on a production site host (for example, when a new application is deployed), perform a manual synchronization of that host with its standby site peer host by using rsync.
  5. Whenever you use rsync to manually synchronize an Oracle Fusion Middleware middle tier instance on a production site host with the peer standby site host, also use Oracle Data Guard to synchronize database repositories in the production site with the databases in the standby site.
Performing Failover and Switchover Operations

Learn how to perform failover or switchover operations when you use the rsync utility.

To perform a failover or switchover from the production site to the standby site when you use rsync:

  1. Shut down any processes running on the production site (if applicable).
  2. Stop rsync jobs between the production site hosts and standby site peer hosts.
  3. Use Oracle Data Guard to failover or swithcover the production site databases to the standby site.
  4. On the standby site, manually start the processes for the Oracle Fusion Middleware Server instances.
  5. Route all user requests to the standby site by performing a global DNS push or something similar, such as updating the global load balancer.
  6. Use a browser client to perform post-failover or post-switchover testing to confirm that requests are being resolved at the standby site (current production site).

    At this point, the standby site is the new production site and the production site is the new standby site.

  7. Reestablish rsync between the two sites, but configure it so that replications go now in the opposite direction (from the current production site to the current standby site).

To use the original production site as the new production site, perform the preceding steps again but configure the rsync replications to go in the original direction (from the original production site to the original standby site).

Using Oracle Site Guard for Disaster Recovery

Oracle Site Guard is a disaster-recovery solution that enables administrators to automate complete site switchover or failover. It orchestrates the coordinated failover of Oracle Fusion Middleware, Oracle Fusion Applications, and Oracle Databases.

Oracle Site Guards offers the following benefits:

  • Fully automate disaster recovery operations and launch them with a single click
  • Minimizes disaster-recovery time
  • Reduces human errors
  • Flexible and customizable
  • Eliminates the need for special skills
  • Use a single pane of glass to manage disaster recovery
  • Assure disaster recovery readiness using on-demand or scheduled disaster recovery drills

For more information about how to use Oracle Site Guard, see Oracle Site Guard Administrator's Guide.

Parent topic: Setting Up and Managing Disaster Recovery Sites

Patching an Oracle Fusion Middleware Disaster Recovery Site

Apply an Oracle Fusion Middleware patch set to upgrade the Oracle homes that participate in an Oracle Fusion Middleware Disaster Recovery site.

It is assumed that the Oracle Central Inventory for any Oracle Fusion Middleware instance that you are patching is located on the production site shared storage, so that the Oracle Central Inventory for the patched instance can be replicated to the standby site.

To apply an Oracle Fusion Middleware patch:

  1. Perform a backup of the production site to ensure that the starting state is secured.
  2. Apply the patch set to upgrade the production site instances.
  3. After you apply the patch set, manually force a synchronization of the production site shared storage and standby site shared storage. This replicates the production site's patched instance and Oracle Central Inventory in the standby site's shared storage.
  4. After you apply the patch set, use Oracle Data Guard to manually force a synchronization of the Oracle databases at the production site and standby sites. Because some Oracle Fusion Middleware patch sets make updates to repositories, this step ensures that any changes made to production site databases are synchronized to the standby site databases.
  5. The upgrade is now complete. Your Disaster Recovery topology is ready to resume processing.

Note:

Patches must be applied only at the production site for an Oracle Fusion Middleware 12c Disaster Recovery topology. If a patch is for an Oracle Fusion Middleware instance or for the Oracle Central Inventory, the patch is copied when the production site shared storage is replicated to the standby site shared storage. A synchronization operation should be performed when a patch is installed at the production site.

When patching the database, check the specific patch’s documentation on how to apply the patch in a Data Guard topology.

Parent topic: Setting Up and Managing Disaster Recovery Sites

Accessing and Managing T3/T3s External Clients in DR Scenarios

This section discusses different approaches for accessing and managing external T3/T3s clients in the context of a Disaster Recovery scenario. 

Note:

This section does not apply to the internal T3/T3s clients, which run in the same domain as the T3/T3s servers. When connecting to a cluster in the same domain, internal clients can utilize the T3/T3s cluster syntax, such as cluster:t3://cluster_name. Only when the clusters are in the same domain can you utilize this cluster syntax.

WebLogic uses the T3/T3s protocol for Remote Method Invocation (RMI). It is used by several WebLogic services, such as JMS, EJB, JTA, JNDI, JMX, WLST, and so on. The external T3/T3s clients, which are those that do not run in the same domain as the T3/T3s servers, can use different ways to connect to a WebLogic cluster. For more information, see Using WebLogic RMI with T3 Protocol.

External T3/T3s clients can connect directly to WebLogic servers' channels (default or custom) that listen to the T3/T3s requests. The connection URL configured in the client's provider property contains the list of all the servers and ports of the cluster.

For example:

t3://host1.example.com:9073,host2.example.com:9073

The external T3/T3s clients can access to T3/T3s services through TCP Load Balancer (LBR) too. In this scenario, the client’s provider URL points to the load balancer service and port, and the requests are load balanced to the WebLogic Server's T3/T3s ports. In the initial contact for the JNDI context retrieval, the external clients connect to a WebLogic Server through the load balancer and download the stubs. These stubs contain the connect information that is used for the subsequent requests.

In general, the WebLogic T3/T3s is TCP/IP-based, so it can support TCP load balancing when services are homogeneous, such as JMS and EJB. For example, a JMS front-end can be configured in a WebLogic cluster in which remote JMS clients can connect to any cluster member. By contrast, a JTA subsystem cannot safely use TCP load balancing in transactions that span across multiple WebLogic domains. The JTA transaction coordinator must establish a direct RMI connection to the server instance that acts as the sub coordinator of the transaction when that transaction is either committed or rolled back. Due to this, the load balancer is normally used only for the JNDI initialContext creation. The WebLogic Server load balancing system controls the future T3/T3s requests, which connect to the WebLogic managed servers' addresses and ports (default or custom channels) indicated in the stubs retrieved during the initial context retrieval.

This section also explains how to use the load balancer for all communications (both initial and subsequent requests), which is applicable if JTA is not used.

Note:

External clients can access to T3 services using T3 tunneling over HTTP. But, this approach is not discussed in this document. This approach creates an HTTP session for each RMI session and uses standard HTTP protocols to transfer the session ID back and forth between the client and the server. This introduces some overhead and is less frequently used.
  • Different Approaches to Access T3/T3s Services
    A Disaster Recovery scenario presents specific aspects which affect the configuration of external T3/T3s clients. This topic explains different approaches that you can use when accessing T3/T3s services from external clients, as well as learn how to manage them in a disaster recovery scenario.
  • About T3/T3s Client’s DNS Cache
    All the approaches to access T3/T3s services mostly requires an DNS update. Since DNS update is required, you must set the limit for DNS Cache TTL (Time To Live) in DNS server and client's specific cache.

Parent topic: Setting Up and Managing Disaster Recovery Sites

Different Approaches to Access T3/T3s Services

A Disaster Recovery scenario presents specific aspects which affect the configuration of external T3/T3s clients. This topic explains different approaches that you can use when accessing T3/T3s services from external clients, as well as learn how to manage them in a disaster recovery scenario.

Note:

These approaches apply to a DR scenario that complies with the MAA guidelines for Fusion Middleware and PaaS DR, so the secondary domain configuration is a mirror replica of the primary system.
  • Direct T3/T3s Using Default Channels
    In this approach, the external T3/T3s client connects directly to the WebLogic managed servers’ default channels. These default channels listen in the Listen Address and Listen Port specified in the general configuration of each WebLogic managed server.
  • Direct T3/T3s Using Custom Channels
    In this approach, the external T3/T3s client connects directly to custom channels defined in the WebLogic servers of the cluster. The Listen Address, the External Listen Address, and the External Port are customizable values in the custom channels. These values can differ from the WebLogic server’s default listen values.
  • Using Load Balancer for Initial Lookup
    In this scenario, the client’s provider URL point to the Load Balancer service.
  • Using Load Balancer for All Traffic
    In this approach, not only the initial context lookup goes through the load balancer, but also the subsequent connections. There are no direct connections from the external T3/T3s client to the servers.

Parent topic: Accessing and Managing T3/T3s External Clients in DR Scenarios

Direct T3/T3s Using Default Channels

In this approach, the external T3/T3s client connects directly to the WebLogic managed servers’ default channels. These default channels listen in the Listen Address and Listen Port specified in the general configuration of each WebLogic managed server.

Figure 3-5 Direct T3/T3s Using Default Channels

This image shows the external T3/T3s client connecting directly to default channels.

Configuration

  • The provider URL in the T3 client uses the list of the WebLogic servers’ default listen addresses and ports.

    Example: In the following DR example, the listener addresses of the WebLogic servers are the primary hostnames: 

    t3://soahost1.example.com:8001,soahost2.example.com:8001

    In the case of using the T3s protocol, the port must be set to the server’s SSL Listen Port.

    Example:
    t3s://soahost1.example.com:8002,soahost2.example.com:8002

  • The external client must resolve these hostnames with the primary host’s IPs. These IPs must be reachable from the client. It is possible to use Network Address Translating (NAT), as long as there is a NAT IP address for each server. In that case, the client will resolve each server name with the appropriate NAT IP for the server.

    Example: Naming resolution at the external client side

    This is achievable though local /etc/hosts or formal DNS server resolution. 

    172.11.2.113	soahost1.example.com
172.11.2.114	soahost2.example.com

Switchover

In a switchover scenario, there is no need to update the client's provider URL. Instead, you need to perform an update of the entries in the DNS (or client's /etc/hosts) of the client. After the switchover, the names used to connect to the servers must resolve with the IPs of the secondary servers.

Example: Naming resolution at the external client side after a switchover
172.22.2.113	soahost1.example.com
172.22.2.114	soahost2.example.com

Advantages

  • You don't require additional configuration either at the server’s side or at the front-end tier. All you require is only opening the ports to the client.

Disadvantages

  • When a switchover happens, you need to update the DNS (or client's /etc/hosts) to alter the resolution of all the hostnames that the external client uses to connect to the managed servers. For more information about implications of the client’s cache, see About T3/T3s Client’s DNS Cache.
  • Clients must be able to resolve and reach the hostnames set in WebLogic's Listen Address. It is not possible to use alternate names, because the default channels use the WebLogic server’s default Listen Address.
  • The WebLogic default channels not only listen for T3(s) requests, but also for HTTP(s). You cannot disable this setting. If you open the default port for the clients, direct HTTP(s) to the server is also allowed, which can result in security concerns.
  • You need to modify the client’s provider URL if you have to either add or remove the WebLogic server nodes from the WebLogic cluster. First contact only uses the list in the client’s provider URL. So, even without updating the list, the subsequent requests can connect to any server of the cluster as the recovered stubs list all of the members. But it is a good practice that the client’s provider URL matches the real list of the servers, for failover purposes in the first contact.
Direct T3/T3s Using Custom Channels

In this approach, the external T3/T3s client connects directly to custom channels defined in the WebLogic servers of the cluster. The Listen Address, the External Listen Address, and the External Port are customizable values in the custom channels. These values can differ from the WebLogic server’s default listen values.

Figure 3-6 Direct T3/T3s Using Custom Channels

This image shows the external T3/T3s client connecting directly to custom channels.

Once the T3/T3s external client has connected to the server during the initial context retrieval, the subsequent T3 calls connects directly to one of the listen addresses and ports configured in the custom channels as External Listen address and External Port. These requests will be load balanced per the mechanism specified in the connection factory defined in the WebLogic and used by the client.

This approach is similar to Direct T3/T3s Using Default Channels, with the exception that you can customize the addresses and ports used in T3/T3s calls when using WebLogic custom channels.

Configuration

  • Each WebLogic server has the appropriate custom channels configured and these custom channel uses a unique External Listen address that points to that server node. Table 3-4 and Table 3-5 provides example of Custom Channel in Server 1 and Server 2.

    Table 3-4 Custom Channel in Server 1

    Name Protocol Enabled Listen Address Listen Port Public Address Public Port
    t3_external_channel t3 true soahost1.example.com 8111 soahost1-t3ext.example.com 8111

    Table 3-5 Custom Channel in Server 2

    Name Protocol Enabled Listen Address Listen Port Public Address Public Port
    t3_external_channel t3 true soahost2.example.com 8111 soahost2-t3ext.example.com 8111

  • The external T3 client’s provider URL contains the list of the external address and port of the custom channels.

    Example:

     t3://soahost1-t3ext.example.com:8111,soahost2-t3ext.example.com:8111

    If you are using the T3s protocol, you must create the custom channels with T3s protocol. Then the clients will connect using T3s protocol and appropriate port.

    Example:

     t3s://soahost1-t3ext.example.com:8112,soahost2-t3ext.example.com:8112

    In a T3s channel, you can add a specific SSL certificate for the name used as an External Listen address.

  • The external T3/T3s client must resolve the custom channels’ external hostnames with the primary host’s IPs. These hostnames must be reachable from the client. It is possible to use NAT, as long as there is a NAT address for each server. In that case, the client will resolve each server name with the appropriate NAT IP for the server.

    Example: Naming resolution at the external client side

    172.11.2.113	soahost1-t3ext.example.com
172.11.2.114	soahost2-t3ext.example.com

    With this approach, all the requests from the external client connect to soahost1-t3ext.example.com and soahost2-t3ext.example.com, both for the initial context retrieval and for the subsequent calls. You can control these requests by using the WebLogic Server load balancing mechanism.

Switchover

You don't have to update the client's provider URL if you have performed a switchover. Instead, you must update the entries in the DNS (or in the /etc/hosts) of the client. After the switchover, the names used to connect to the servers must resolve with the IPs of the secondary servers.

Example: Naming resolution at the external client side after a switchover
172.22.2.113	soahost1-t3ext.example.com
172.22.2.114	soahost2-t3ext.example.com

Advantages

  • This method allows using specific hostnames for the external T3/T3s communication, different from the server's default listen address. This is useful if you do not want to expose the default server's Listen Address to the external clients for security reason.

  • This method is also useful if you are using NAT IPs and you do not want to resolve the servers' default listen addresses with different IPs internally and externally.

  • This method is also useful in case you want to use different names for external T3/T3s accesses just for organizational purposes. You can also use this method to isolate protocols in different interfaces or network routes.

  • The protocol in the custom channel can be limited to T3/T3s only. The HTTP(s) protocol can be disabled in the custom channels.

Disadvantages

  • When a switchover happens, you need to update the DNS (or client's /etc/hosts) at the external client side for all the hostnames used to connect to the managed servers. For more information about the implications of the client’s cache, see About T3/T3s Client’s DNS Cache.
  • If you are using T3s, then you must create and configure specific SSL certificates for the external names in the custom channels to avoid SSL hostname verification errors in the client.
  • You have to modify the client’s provider URL if you have added or removed the WebLogic server nodes from the WebLogic cluster. First contact only uses the list in the client’s provider URL. So, even without updating the list, the subsequent requests can connect to any server of the cluster as the recovered stubs list all of the members. But in any case, it is a good practice that the client’s provider URL matches the real list of the servers, for failover purposes in the first contact.
Using Load Balancer for Initial Lookup

In this scenario, the client’s provider URL point to the Load Balancer service.

Direct T3/T3s Using Default Channels and Direct T3/T3s Using Custom Channels approaches can also use a load balancer for the initial context lookup.

However, the subsequent T3/T3s calls connects to WebLogic servers directly. If you use the default channels then the requests goes to the default channel’s listen address and port. Similarly, if you use the custom channels then the subsequent request goes to the external listen address and port defined in the customs channels.

Figure 3-7 Using Load Balancer for Initial Lookup

This image shows using load balancer for the initial lookup.

Configuration

  • You will need a TCP service in the load balancer to load balances the requests to the WebLogic servers' T3/T3s ports (either to the default channels or to the custom channels when used).

  • The external T3/T3s client’s provider uses the front-end name and port of the load balancer as the point of contact.

    Example:

    t3://t3lbr.example.com:8111

  • You can use default channels or custom channels as explained in Direct T3/T3s Using Default Channels and Direct T3/T3s Using Custom Channels scenarios. The external T3/T3s client must resolve the custom channels’ external hostnames (or the default server’s listeners hostnames if you are using default channels) with the primary host’s IPs. The client must be able to access them. It is possible to use NAT, as long as there is a NAT address for each server. In that case, the client will resolve each server name with the appropriate NAT IP for the server.

    Example: Naming resolution at the external client side
    111.111.111.111    t3lbr.example.com
172.11.2.113	soahost1-t3ext.example.com
172.11.2.114	soahost2-t3ext.example.com

Switchover

In case of a complete switchover, you don't have to update the client's provider URL. Instead, you must update the entries in the DNS (or in the /etc/hosts) used by the client so that they resolve with the IPs of the secondary site. The name used to connect to the load balancer must resolve with the IP of the secondary load balancer, and the server names used in subsequent requests must point to the IPs of the secondary servers.

Example: Naming resolution at the external client side after a switchover
222.222.222.222    t3lbr.example.com
172.22.2.113	soahost1-t3ext.example.com
172.22.2.113	soahost2-t3ext.example.com

Advantages

You don't have to modify the client’s provider URL if you add or remove the WebLogic server nodes from the WebLogic cluster.

Disadvantages

  • Despite using a load balancer in front, the client still needs to reach the servers directly.

  • When a switchover happens, you need to update the DNS (or /etc/hosts) of servers’ addresses at the external client side. For more information about the implications of the client’s cache, see About T3/T3s Client’s DNS Cache.

  • The complexity of this method is higher for the T3s cases. The client connects both through the front-end LBR and directly to the server using a secure protocol. In this case, you will need to either skip the hostname verification at the client side, or use an SSL certificate that is valid for front and back addresses (e.g. wildcard or SAN certificates).

Using Load Balancer for All Traffic

In this approach, not only the initial context lookup goes through the load balancer, but also the subsequent connections. There are no direct connections from the external T3/T3s client to the servers.

Oracle does not recommend using LBR for load balancing all types of T3/T3s communications. It is only recommended for initial context lookup. For more information, see, WebLogic RMI Integration with Load Balancers. However, there are T3/T3s use cases, where you can use Load Balancer for complete T3/T3s communication flow. WebLogic T3/T3s is TCP/IP-based protocols, so it can support TCP load balancing when services are homogeneous, such as JMS and EJB. For example, you can configure an LBR front-ended JMS subsystem in a WebLogic cluster in which remote JMS clients can connect to any cluster member.

This approach, however, will not work with external clients that use JTA connections. A JTA subsystem cannot safely use TCP load balancing in transactions that span across multiple WebLogic domains. When the transaction is either committed or rolled back, the JTA transaction coordinator must establish a direct RMI connection to the server instance that has been chosen as the transaction's sub coordinator.

This method is not suitable also for cases where you require direct connection to an specific server, like JMX or WLST when you want to connect to a particular server only.

Figure 3-8 Using Load Balancer for All Traffic

This image shows using load balancer for all traffic.

Configuration

  • Load balancer requires a TCP service to load balance the requests to the WebLogic servers’ T3/T3s ports defined in the custom channels.

  • The external T3/T3s client’s provider URL uses the front-end name and port of the load balancer as the point of contact.

    Example:
    t3://t3lbr.example.com:8111

  • The external client must resolve the load balancer address with the IP of the primary site’s load balancer.

    Example:
    111.111.111.111  t3lbr.example.com

  • WebLogic Server requires custom channels. Configure the external listen address and port of these custom channels with the Load Balancer's address and port. Table 3-6 and Table 3-7 provides example of Custom Channel in Server 1 and Server 2.

    Table 3-6 Custom Channel in Server 1

    Name Protocol Enabled Listen Address Listen Port Public Address Public Port
    t3_external_channel t3 true soahost1.example.com 8111 t3blr.example.com 8111

    Table 3-7 Custom Channel in Server 2

    Name Protocol Enabled Listen Address Listen Port Public Address Public Port
    t3_external_channel t3 true soahost2.example.com 8111 t3lbr.example.com 8111

Switchover

In case of a switchover, you don't have to update the client's provider URL. Instead, you must update the entries in the DNS (or in the /etc/hosts) of the client. After the switchover, the names used to connect to the servers resolves with the IP of the secondary LBR service.

Example: Naming resolution at the external client side after a switchover
222.222.222.222  t3lbr.example.com

Advantages

  • All the communication goes through the load balancer. The client only needs to know and reach the load balancer’s service.

  • You don't have to modify the client’s provider URL if you have to either add or remove the WebLogic server nodes from the WebLogic cluster.

  • In case of a switchover, you only have to update the load balancer’s front-end name in the DNS (or in the /etc/hosts).

    Note:

    Although only one DNS name is updated, you need to refresh the client’s DNS cache. For more information, see About T3/T3s Client’s DNS Cache.

  • In T3s cases, you can use the same SSL certificate in all the custom channels, associated to the load balancer service’s front-end name.

Disadvantages

  • This approach is not suitable for all the T3/T3s cases. Scenarios with JTA cannot use this approach, because JTA needs to explicitly connect to the server instance that acts as the sub coordinator of the transaction.

About T3/T3s Client’s DNS Cache

All the approaches to access T3/T3s services mostly requires an DNS update. Since DNS update is required, you must set the limit for DNS Cache TTL (Time To Live) in DNS server and client's specific cache.

The TTL limit in the DNS service is a setting that tells the DNS resolver how long to cache a query before requesting a new one. Note that the TTL value of the DNS entries will affect the effective RTO of the switchover; if the TTL is high (for example, 20 min), the DNS change will take that time to be effective in the clients. Using lower TTL values will make this switchover faster, however, this can cause overhead because the clients check the DNS more frequently. A good approach is to set the TTL to a low value temporarily (for example, one min), before the change in the DNS. Then, perform the change, and once the switchover procedure is completed, set the TTL to the normal value again.

Besides the DNS server's TTL, networkaddress.cache.ttl Java property controls the Java clients' cache TTL. This Java property indicates the caching policy for successful name lookups from the name service. Specify the value as an integer to indicate the number of seconds to cache the successful lookup. Ensure you set a limit to the networkaddress.cache.ttl Java property so the client's Java cache does not cache the DNS entries forever. Else, with each switchover, you might have to restart the client.

Parent topic: Accessing and Managing T3/T3s External Clients in DR Scenarios