2 Setting Up a Disaster Recovery Deployment

This chapter provides instructions about how to set up an Oracle Fusion Middleware Disaster Recovery topology for the Linux and UNIX operating systems. The procedures use the Oracle SOA Suite Enterprise Deployment (see Figure 2-1) in the examples to illustrate how to set up the Oracle Fusion Middleware Disaster Recovery solution for a specific case. After you understand how to set up the disaster recovery for the Oracle SOA Suite enterprise topology, use the same information to set up a disaster recovery system for the other enterprise deployments.

Figure 2-1 Deployment Used at Production and Secondary Sites for Oracle Fusion Middleware Disaster Recovery


This image shows the Oracle SOA Suite and Oracle Business Activity Monitoring Enterprise Deployment

Disaster Recovery Deployment Process Overview

Setting up a disaster protection system for Oracle Fusion Middleware involves the following procedures.

  1. Planning Host Names

    Determines how the listen addresses used by the different components are going to be configured and virtualized so that no changes are required on the secondary when the configuration is propagated from the primary. Deciding how these addresses will be resolved may impact your system’s manageability and the RTO of your disaster protection solutions.

  2. Planning a File System Replication Strategy

    Determines what replication technology and approach is going to be used to meet the RTO and RPO requirements for the different artifacts that need to be copied from the primary to the secondary.

  3. Preparing the Primary System for Disaster Protection

    For the optimum configuration, some changes may need to be applied in the primary in preparation for the disaster protection configuration. These changes primarily affect the storage configuration and database’s address configuration.

  4. Preparing the Secondary System for Disaster Protection

    The secondary system can also be set up based on the information provided in the Enterprise Deployment Guides. The infrastructure used by the Fusion Middleware system needs a specific configuration that will optimize the behavior of a disaster protection solution.

  5. Setting Up the Secondary System

    Setting up the secondary system involves the following three tasks:

    1. Replicating the database by setting up the Data Guard for the database used by Fusion Middleware.

    2. Replicating the primary Fusion Middleware file systems to the secondary storage.

    3. Updating the TNS alias for the secondary’s database address.

The following sections describe the procedures listed above in detail.

Planning Host Names

In a disaster recovery topology, the host name addresses used by the Fusion Middleware components must be resolvable to valid IP addresses in each site.

There are different types of host names that are used in this document:

  • Physical Host Names

    These are the host names used to uniquely identify a node in a network. This is the address used to establish an SSH connection to that node. A physical host name is mapped to a fixed IP that is attached to the main Network Interface Controller (NIC) that the node uses. Physical host names are attached to a specific host and cannot be moved to a different one because they provide the main access point to the machine they belong to.

  • Host Name Aliases or Virtual Hostnames

    These are host names that can float from node to node. They can be assigned to a node and then disabled in that node and assigned to a different one. Virtual hostnames are used as listen address for processes and components that can move across different nodes. They enable accessors to abstract themselves from the physical host name where the process or components runs at a precise point in time.

As mentioned in the Enterprise Deployment Guides, Oracle recommends creating host name aliases along with the existing physical host names to decouple the system’s configuration from those physical host names (which are different in each site and attached to specific nodes). Physical host names will typically differ between the primary and the secondary but the Fusion Middleware configuration uses virtual hostnames that are valid in both locations.

This section describes how to plan alias host names for the middle-tier hosts that use the Oracle Fusion Middleware instances at the production and secondary sites. It uses the Oracle SOA Suite Enterprise Deployment shown in Figure 2-1 for the host name examples. Each host at the production site has a peer host in the secondary site. The Fusion Middleware components of the peer host in the secondary site are configured with the same listen addresses and ports as their counterparts in the primary.

When you configure each component, use host-name-based configuration instead of IP-based configuration. For example, if you want an Oracle WebLogic Managed Server to listen on a specific IP address (for example, 172.11.2.113), then use the host name SOAHOST1.EXAMPLE.COM in its configuration and then use your OS to resolve this host name to IP 172.11.2.113. You must not include IPs directly in the configuration or in your WebLogic domain and applications or in the configuration of Fusion Middleware system components such as the Oracle HTTP Server.

The following section shows how to set up host names at the disaster recovery production and standby sites.

Note:

In the examples listed, IP addresses for hosts at the initial production site have the format 172.11.x.x and IP addresses for hosts at the initial standby site have the format 172.22.x.x.

Sample Host Names for the Oracle SOA Suite Production and Standby Site Hosts

Figure 2-2 shows the Oracle Fusion Middleware Disaster Recovery topology that uses the Oracle SOA Suite enterprise deployment in the primary site.

Figure 2-2 Deployment Used at Production and Secondary Sites for Oracle Fusion Middleware Disaster Recovery


This image shows the Oracle SOA Suite and Oracle Business Activity Monitoring Enterprise Deployment

Table 2-1 lists the IP addresses, physical host names, and aliases that are used for the Oracle SOA Suite Enterprise Deployment Guide (EDG) primary site hosts. Figure 2-2 shows the configuration of the Oracle SOA Suite EDG deployment at the production site.

Table 2-1 IP Addresses and Physical Host Names for SOA Suite Production Site Hosts

IP Address Physical Host Name Host Name Alias

172.11.2.111

prweb1.example.com

WEBHOST1

172.11.2.112

prweb2.example.com

WEBHOST2

172.11.2.113

prsoa1.example.com

SOAHOST1

172.11.2.114

prsoa2.example.com

SOAHOST2

Note:

This document uses WEBHOST1, WEBHOST2, SOAHOST1, SOAHOST2, and ADMINVHN as abstract placeholders for the actual alias host names. For example, in your system, SOAHOST1 may have a value of soahost1.example.com.

Figure 2-3 shows the physical host names that are used for the Oracle SOA Suite EDG deployment at the standby site.

Figure 2-3 Physical Host Names Used at Oracle SOA Suite Deployment Standby Site


Physical Host Names Used at Oracle SOA Suite Deployment Standby Site

Table 2-2 lists the IP addresses, physical host names, and aliases that are used for the Oracle SOA Suite Enterprise Deployment Guide (EDG) deployment standby site hosts.

Table 2-2 IP Addresses and Physical Host Names for SOA Suite Standby Site Hosts

IP Address Physical Host Name Host Name Alias

172.22.2.111

stbyweb1.example.com

WEBHOST1

172.22.2.112

stbyweb2.example.com

WEBHOST2

172.22.2.113

stbysoa1.example.com

SOAHOST1

172.22.2.114

stbysoa2.example.com

SOAHOST2

Note:

If you use separate DNS servers to resolve host names, then you can use the same physical host names for the production site hosts and standby site hosts and you do not need to define the alias host names on the standby site hosts. For more information about using separate DNS servers to resolve host names, see Resolving Host Names Using Separate DNS Servers.

Virtual IP Considerations

Besides using virtual hostnames to facilitate disaster protection configuration and in the context of local failures, some Oracle WebLogic servers may use a floating hostname as the listen address that is mapped to an IP that can be enabled in different nodes. These are called floating IPs and can be moved across different nodes inside the same region by disabling them in a network or virtual network interface in one node and enabling them in a different one. This allows failing over of the WebLogic servers and system components without reconfiguring their listen addresses which is the case for Oracle WebLogic Administration Server. In the Oracle WebLogic Cluster, VIPs and servers can also be moved across nodes by the WebLogic infrastructure and this mechanism is called WebLogic Server Migration.

Note:

The WebLogic Administration Server cannot be clustered and therefore cannot use server migration.

In Oracle Fusion Middleware 14c, most applications support Oracle WebLogic Automatic Service Migration (ASM) where only the services running inside WebLogic (JMS, JTA) are migrated to other running WebLogic Servers. This improves recovery time because instead of entire servers being restarted, just few WebLogic subsystems are failed over. As a result, it is no longer necessary to reserve Virtual IPs for the Managed Servers in the domain (required for server migration). Those products or applications that do not support ASM, may require a VIP to configure server migration for the WebLogic Server where they run. Refer to your specific Fusion Middleware component’s documentation to determine if it supports ASM. As indicated, a virtual hostname and its mapping VIP is needed for the administration server (Enterprise Deployment Guide recommendation). The WebLogic administration server cannot be clustered so it needs a mobile hostname that can be failed over to other nodes in a loss of host scenario.

Ensure that you provide the floating IP addresses for the administration server with the same virtual hostnames on the production site and the standby site.

Table 2-3 Floating IP Addresses

Virtual IP Address Virtual Name Alias Host Name

172.11.2.134

prsoa-vip.example.com

(in production site)

ADMINVHN

172.22.2.134

stbysoa-vip.example.com

(in standby site)

ADMINVHN

Host Name Resolution

Host name resolution means mapping a host name to the proper IP address required for accessing a node, a service, or an endpoint in a system. Host name resolution is a key aspect in all disaster protection systems because it allows virtualizing addresses and using primary sites' configuration in the secondary site without manipulations. A primary system may have its own host name resolution mechanics (Enterprise Deployment Guides do not prescribe a specific methodology in this domain). However, depending on the number of nodes, the number of virtual hostnames, and the components used in the Fusion Middleware topology, different host name resolution mechanisms based on a system’s need is required.

In a disaster protection system, hostname aliases or virtual hostnames are used in the configuration of Fusion Middleware so that the configuration can be reused without any modifications in a different site by simply remapping the virtual hostnames to IPs that are valid in the secondary site. Host name resolution enables this dynamic mapping of hostnames to IPs and can be configured using any of the following procedures:

  • Resolving Host Names Locally

    Local host name resolution uses the host name to IP address mapping that is specified by the /etc/hosts file on each host.

    For more information about using the /etc/hosts file to implement local host name file resolution, see Resolving Host Names Locally .

  • Resolving Host Names Using DNS

    A DNS server is a dedicated server or a service that provides DNS name resolution in an IP network. You can use global DNS services (common to primary and secondary) or use separate DNS services in each location.

    For more information about the two methods used for implementing DNS server host name resolution, see Resolving Host Names Using Separate DNS Servers and Resolving Host Names Using a Global DNS Server.

You must determine the method of host name resolution that you will use for your Oracle Fusion Middleware Disaster Recovery topology when you plan the deployment of the topology. Most site administrators use a combination of these resolution methods in a precedence order to manage host names.The Oracle Fusion Middleware hosts and the shared storage system for each site must be able to communicate with each other.

Host Name Resolution Precedence

To determine the host name resolution method used by a particular host, search for the value of the hosts parameter in the /etc/nsswitch.conf file on the host.

If you want to resolve host names locally on the host, make the files entry the first entry for the hosts parameter, as shown in Example 2-1. When files is the first entry for the hosts parameter, entries in the host /etc/hosts file are first used to resolve host names.

If you want to resolve host names by using DNS on the host, make the dns entry the first entry for the hosts parameter, as shown in Example 2-2. When dns is the first entry for the hosts parameter, DNS server entries are first used to resolve host names.

For simplicity and consistency, Oracle recommends that all the hosts within a site (production or standby site) should use the same host name resolution method (resolving host names locally or resolving host names using separate DNS servers or a global DNS server).

The recommendations in the following sections are high-level that you can adapt to meet the host name resolution standards used by your enterprise.

Example 2-1 Specifying the Use of Local Host Name Resolution

hosts:   files   dns   nis

Example 2-2 Specifying the Use of DNS Host Name Resolution

hosts:   dns    files   nis
Resolving Host Names Locally

Local host name resolution uses the host name for IP mapping that is defined in the /etc/hosts file of a host.

When you resolve host names for your disaster recovery topology in this way, use the following procedure:

  1. Ensure that the hosts parameter in the /etc/nsswitch.conf file on all the production site and standby site hosts is as shown below:
    hosts:   files   dns   nis
    
  2. The /etc/hosts file entries on the hosts of the production site should have their physical host names mapped to their IP addresses along with the hostname aliases used by the Fusion Middleware components. For simplicity and ease of maintenance, Oracle recommends you to provide the same entries on all the hosts of the production site. Example 2-3 shows the /etc/hosts file for the production site of a SOA enterprise deployment topology.
  3. The /etc/hosts file entries on the hosts of the standby site should have their physical host names mapped to their IP addresses along with the hostnames aliases of their corresponding peer on the production site defined as the alias host names. For simplicity and ease of maintenance, Oracle recommends that you provide the same entries on all the hosts of the standby site. Example 2-4 shows the /etc/hosts file for the standby site of a SOA enterprise deployment topology.
  4. After you set up host name resolution by using /etc/host file entries, use the ping command to test host name resolution. For a system configured with static IP addressing and the /etc/hosts file entries shown in Example 2-3, a ping webhost1 command on the production site returns the correct IP address (172.11.2.111) and indicates that the hostname is fully resolvable.
  5. Similarly, for a system configured with static IP addressing and the /etc/hosts file entries shown in Example 2-4, a ping webhost1 command on the standby site returns the correct IP address (172.22.2.111) and it shows that the name WEBHOST1 is associated with that IP address.

Example 2-3 Adding /etc/hosts File Entries for a Production Site Host

127.0.0.1        localhost.localdomain     localhost
172.11.2.134     prsoa-vip.example.com     prsoa-vip      ADMINVHN.EXAMPLE.COM    ADMINVHN
172.11.2.111     prweb1.example.com 	 prweb1 	 WEBHOST1.EXAMPLE.COM    WEBHOST1
172.11.2.112     prweb2.example.com 	 prweb2 	 WEBHOST2.EXAMPLE.COM    WEBHOST2
172.11.2.113     prsoa1.example.com 	 prsoa1 	 SOAHOST1.EXAMPLE.COM    SOAHOST1
172.11.2.114     prsoa2.example.com 	 prsoa2 	 SOAHOST2.EXAMPLE.COM    SOAHOST2

Example 2-4 Adding /etc/hosts File Entries for a Standby Site Host

127.0.0.1          localhost.localdomain       localhost
172.22.2.134       stbysoa-vip.example.com     stbysoa-vip   ADMINVHN.EXAMPLE.COM	 ADMINVHN
172.22.2.111       stbyweb1.example.com 	stbyweb1 	WEBHOST1.EXAMPLE.COM	WEBHOST1
172.22.2.112       stbyweb2.example.com 	stbyweb2 	WEBHOST2.EXAMPLE.COM	WEBHOST2
172.22.2.113       stbysoa1.example.com 	stbysoa1 	SOAHOST1.EXAMPLE.COM	SOAHOST1
172.22.2.114       stbysoa2.example.com 	stbysoa2 	SOAHOST2.EXAMPLE.COM	SOAHOST2 

Note:

The subnets in the production site and standby site are different.
Resolving Host Names Using Separate DNS Servers

You can also use separate DNS servers to resolve host names for your disaster recovery topology (one in the primary and another in the secondary site).

When you use separate DNS servers to resolve host names for your disaster recovery topology, use the following procedure:

  1. Ensure that the hosts parameter in the /etc/nsswitch.conf file on all the production site and standby site hosts is as shown below:
    hosts:   dns   files   nis
    
  2. The DNS servers on the production and standby site must not be aware of each other and must contain entries for host names used within their own site.
  3. The DNS server entries on the production site should have the physical and alias host names mapped to their IP addresses. Example 2-5 shows the DNS server entries for the production site of a SOA enterprise deployment topology.
  4. The DNS server entries on the standby site should have the physical and alias host names mapped to their IP addresses. Example 2-6 shows the DNS server entries for the standby site of a SOA enterprise deployment topology.
  5. Ensure that there are no entries in the /etc/hosts file for any host at the production or the standby site.
    1. Test the host name resolution by using the ping command. For a system configured with the production site DNS entries as shown in Example 2-5, a ping webhost1 command on the production site returns the correct IP address (172.11.2.111) and indicates that the host name is fully qualified.
    2. Similarly, for a system configured with the standby site DNS entries shown in Example 2-6, a ping webhost1 command on the standby site returns the correct IP address (172.22.2.111) and indicates that the host name is fully qualified.

Example 2-5 DNS Entries for a Production Site Host in a Separate DNS Servers Configuration

PRSOA-VIP.EXAMPLE.COM    IN    A    172.11.2.134
PRWEB1.EXAMPLE.COM       IN    A    172.11.2.111
PRWEB2.EXAMPLE.COM       IN    A    172.11.2.112
PRSOA1.EXAMPLE.COM       IN    A    172.11.2.113
PRSOA2.EXAMPLE.COM       IN    A    172.11.2.114

ADMINVHN.EXAMPLE.COM    IN    A    172.11.2.134
WEBHOST1.EXAMPLE.COM    IN    A    172.11.2.111
WEBHOST2.EXAMPLE.COM    IN    A    172.11.2.112
SOAHOST1.EXAMPLE.COM    IN    A    172.11.2.113
SOAHOST2.EXAMPLE.COM    IN    A    172.11.2.114

Example 2-6 DNS Entries for a Standby Site Host in a Separate DNS Servers Configuration

STBYSOA-VIP.EXAMPLE.COM   IN    A    172.22.2.134
STBYWEB1.EXAMPLE.COM      IN    A    172.22.2.111
STBYWEB2.EXAMPLE.COM      IN    A    172.22.2.112
STBYSOA1.EXAMPLE.COM      IN    A    172.22.2.113
STBYSOA2.EXAMPLE.COM      IN    A    172.22.2.114

ADMINVHN.EXAMPLE.COM       IN    A    172.22.2.134
WEBHOST1.EXAMPLE.COM       IN    A    172.22.2.111
WEBHOST2.EXAMPLE.COM       IN    A    172.22.2.112
SOAHOST1.EXAMPLE.COM       IN    A    172.22.2.113
SOAHOST2.EXAMPLE.COM       IN    A    172.22.2.114

Note:

If you use separate DNS servers to resolve host names, then you can use the same host names for the production site hosts and standby site hosts. You do not need to define the alias host names.
Resolving Host Names Using a Global DNS Server

You can use a global DNS server to resolve host names for your disaster recovery topology as an alternative to the two previous options.

The term global DNS server refers to a disaster recovery topology, where a single DNS server is used for both the production and standby site.

When you use a global DNS server to resolve host names for your disaster recovery topology, use the following procedure:

  1. Using a global DNS server, involves updating records on switchover. Notice that this may impact your RTO depending on how your DNS server is managed and how Time to Live (TTL) can be adjusted during switchover.
  2. When you use a global DNS server, use a combination of local host name resolution and DNS host name resolution.
  3. In this example, it is assumed that the production site uses DNS host name resolution and the standby site uses local host name resolution.
  4. The global DNS server should have the entries for both the production and the standby site hosts. Example 2-7 shows the entries for a SOA enterprise deployment topology.
  5. Ensure that the hosts parameter in the /etc/nsswitch.conf file on all the production site hosts is as shown below:
    hosts:   dns   files   nis
    
  6. Ensure that the hosts parameter in the /etc/nsswitch.conf file on all the standby site hosts is as shown below:
    hosts:   files   dns   nis
    
  7. The /etc/hosts file entries on the hosts of the standby site should have their physical host names mapped to their IP addresses along with the physical host names of their corresponding peer on the production site defined as the alias hostnames. For simplicity and ease of maintenance, Oracle recommends that you provide the same entries on all the hosts of the standby site. Example 2-9 shows the /etc/hosts file for the production site of a SOA enterprise deployment topology.
    1. Test the host name resolution by using the ping command. A ping webhost1 command on the production site returns the correct IP address (172.11.2.111) and indicates that the host name is fully qualified.
    2. Similarly, a ping webhost1 command on the standby site returns the correct IP address (172.22.2.111) and indicates that the host name is fully qualified.

Example 2-7 DNS entries for production site and standby site hosts when using a global DNS server configuration during normal operations (when the workload is sustained by the original primary)

PRSOA-VIP.EXAMPLE.COM      IN A 172.11.2.134
PRWEB1.EXAMPLE.COM         IN A 172.11.2.111
PRWEB2.EXAMPLE.COM         IN A 172.11.2.112
PRSOA1.EXAMPLE.COM         IN A 172.11.2.113
PRSOA2.EXAMPLE.COM         IN A 172.11.2.114

STBYSOA-VIP.EXAMPLE.COM    IN A 172.22.2.134
STBYWEB1.EXAMPLE.COM       IN A 172.22.2.111
STBYWEB2.EXAMPLE.COM       IN A 172.22.2.112
STBYSOA1.EXAMPLE.COM       IN A 172.22.2.113
STBYSOA2.EXAMPLE.COM       IN A 172.22.2.114

ADMINVHN.EXAMPLE.COM       IN A 172.11.2.134
WEBHOST1.EXAMPLE.COM       IN A 172.11.2.111
WEBHOST2.EXAMPLE.COM       IN A 172.11.2.112
SOAHOST1.EXAMPLE.COM       IN A 172.11.2.113
SOAHOST2.EXAMPLE.COM       IN A 172.11.2.114

Example 2-8 DNS entries for production site and standby site hosts when using a global DNS server configuration, after a switchover to standby site

PRSOA-VIP.EXAMPLE.COM      IN A 172.11.2.134
PRWEB1.EXAMPLE.COM         IN A 172.11.2.111
PRWEB2.EXAMPLE.COM         IN A 172.11.2.112
PRSOA1.EXAMPLE.COM         IN A 172.11.2.113
PRSOA2.EXAMPLE.COM         IN A 172.11.2.114

STBYSOA-VIP.EXAMPLE.COM    IN A 172.22.2.134
STBYWEB1.EXAMPLE.COM       IN A 172.22.2.111
STBYWEB2.EXAMPLE.COM       IN A 172.22.2.112
STBYSOA1.EXAMPLE.COM       IN A 172.22.2.113
STBYSOA2.EXAMPLE.COM       IN A 172.22.2.114

ADMINVHN.EXAMPLE.COM       IN A 172.22.2.134
WEBHOST1.EXAMPLE.COM       IN A 172.22.2.111
WEBHOST2.EXAMPLE.COM       IN A 172.22.2.112
SOAHOST1.EXAMPLE.COM       IN A 172.22.2.113
SOAHOST2.EXAMPLE.COM       IN A 172.22.2.114

Example 2-9 Production site /etc/hosts file entries when using a global DNS server configuration

127.0.0.1       localhost.localdomain     localhost
172.11.2.134    prsoa-vip.example.com     prsoa-vip    ADMINVHN.EXAMPLE.COM     ADMINVHN
172.11.2.111    prweb1.example.com 	  prweb1       WEBHOST1.EXAMPLE.COM     WEBHOST1
172.11.2.112    prweb2.example.com 	  prweb2       WEBHOST2.EXAMPLE.COM     WEBHOST2
172.11.2.113    prsoa1.example.com 	  prsoa1       SOAHOST1.EXAMPLE.COM     SOAHOST1
172.11.2.114    prsoa2.example.com 	  prsoa2       SOAHOST2.EXAMPLE.COM     SOAHOST2 

Example 2-10 Standby Site /etc/hosts File Entries when using a global DNS server configuration

127.0.0.1      localhost.localdomain        localhost
172.22.2.134   stbysoa-vip.example.com      stbysoa-vip    ADMINVHN.EXAMPLE.COM	ADMINVHN
172.22.2.111   stbyweb1.example.com 	    stbyweb1 	  WEBHOST1.EXAMPLE.COM	WEBHOST1
172.22.2.112   stbyweb2.example.com 	    stbyweb2 	  WEBHOST2.EXAMPLE.COM	WEBHOST2
172.22.2.113   stbysoa1.example.com 	    stbysoa1 	  SOAHOST1.EXAMPLE.COM	SOAHOST1
172.22.2.114   stbysoa2.example.com 	    stbysoa2 	  SOAHOST2.EXAMPLE.COM	SOAHOST2
Testing the Host Name Resolution

Validate the host name assignment by connecting to each host at the production site. Use the ping command to ensure that the host can locate the other hosts at the production site.

External T3/T3s Clients Considerations

Systems directly accessing the WebLogic servers in a topology need to be aware of the listen address that is used by those WebLogic servers. An appropriate host name resolution needs to be provided to those clients so that the hostname alias used by the servers as listen address is correctly resolved. This is also applicable when performing deployments from Oracle JDeveloper or other development tools and scripts. Using the example addresses provided in the EDG, the client hosting Oracle Jdeveloper needs to map the SOAHOSTx and ADMINVHN aliases (corresponding to different WebLogic servers and administration server) to correct IP addresses for management and RMI operations to succeed. Depending on how controllable and how many external clients are accessing the system, a different host name resolution approach may be better for an optimum RTO and simplified management.

Note:

This section does not apply to T3/T3s clients that reside in the same WebLogic domain as the T3/T3s servers. When connecting to a cluster in the same domain, internal clients can utilize the T3/T3s cluster syntax such as cluster:t3://cluster_name. You can utilize this cluster syntax, only when the clusters are in the same domain.

WebLogic uses the T3/T3s protocol for Remote Method Invocation (RMI). It is used by several WebLogic services such as JMS, EJB, JTA, JNDI, JMX and WLST. The external T3/T3s clients that do not run in the same domain as the T3/T3s servers, can use different ways to connect to a WebLogic cluster. For more information, see Using WebLogic RMI with T3 Protocol.

External T3/T3s clients can connect directly to WebLogic server channels (default or custom) that listen to the T3/T3s requests. The connection URL configured in the client's provider property contains the list of all the servers and ports of the cluster. For example,
t3://host1.example.com:9073,host2.example.com:9073

The external T3/T3s clients can also access the T3/T3s services through a TCP Load Balancer (LBR). In this scenario, the client provides URL points to the load balancer service and port. The requests are then load balanced to the WebLogic Server's T3/T3s ports. In the initial contact for the JNDI context retrieval, the external clients connect to a WebLogic Server through the load balancer and download the T3 client stubs. These stubs contain the connect information that is used for the subsequent requests.

In general, the WebLogic T3/T3s is TCP/IP based so it can support the TCP load balancing when services are homogeneous such as JMS and EJB. For example, a JMS front-end can be configured in a WebLogic cluster in which remote JMS clients can connect to any cluster member. By contrast, a JTA subsystem cannot safely use TCP load balancing in transactions that span across multiple WebLogic domains. The JTA transaction coordinator must establish a direct RMI connection to the server instance that acts as a sub coordinator of the transaction when that transaction is either committed or rolled back. Due to this, the load balancer is normally used only for the JNDI initial context retrieval. The WebLogic Server load balancing system controls the future T3/T3s requests, which connect to the WebLogic managed servers' addresses and ports (default or custom channels) indicated in the stubs retrieved during this initial context retrieval.

This section explains how to manage T3/T3s clients and its impact on disaster protection configuration. It also explains how to use the load balancer for all T3/T3s communications (both initial and subsequent requests) when JTA is not used.

Note:

External clients can access the T3 services using the T3 tunneling over HTTP. This approach creates a HTTP session for each RMI session and uses standard HTTP protocols to transfer the session ID back and forth between the client and the server. This introduces some overhead and is used less frequently.

Different Approaches for Accessing T3/T3s Services

A disaster recovery scenario presents specific aspects which affect the host name resolution configuration of external T3/T3s clients. This topic explains different approaches that you can use when accessing T3/T3s services from external clients and how to manage them in a disaster recovery scenario. For each approach a configuration example is provided and the advantages and disadvantages while managing clients and switchover are explained.

Note:

These approaches apply to a disaster recovery scenario that complies with the MAA guidelines for Fusion Middleware, where the secondary domain configuration is a replica of the primary system.
Direct T3/T3s Using Default Channels

In this approach, the external T3/T3s client connects directly to the WebLogic managed server's default channels. These default channels listen in the Listen Address and Listen Port specified in the general configuration of each WebLogic managed server.

Figure 2-4 Direct T3/T3s Using Default Channels

This image shows the external T3/T3s client connecting directly to default channels.

Configuration

  • The provider URL in the T3 client uses the list of the WebLogic servers' default listen addresses and ports.

    Example: In the following disaster recovery example, the listener addresses of the WebLogic servers are the primary hostnames:

    t3://soahost1.example.com:8001,soahost2.example.com:8001

    In case you are using the T3s protocol, the port must be set to the server’s SSL Listen Port.

    Example:
    t3s://soahost1.example.com:8002,soahost2.example.com:8002
  • External clients must resolve these hostnames with the primary host’s IPs. These IPs must be reachable from the client. It is possible to use Network Address Translating (NAT), if there is a NAT IP address for each server. In that case, the client will resolve each server name with the appropriate NAT IP for the server.

    Example: Naming Resolution at the External Client Side

    This is achievable through local /etc/hosts or formal DNS server resolution.

    172.11.2.113	soahost1.example.com
    172.11.2.114	soahost2.example.com

Switchover Behavior

In a switchover scenario, there is no need to update the client's provider URL. You need to perform an update of the entries in the DNS (or client's /etc/hosts) of the client. After the switchover, the names used to connect to the servers must resolve with the IPs of the secondary servers.

Example: Naming Resolution at the External Client Side After a Switchover
172.22.2.113	soahost1.example.com
172.22.2.114	soahost2.example.com

Advantages

  • You do not require additional configuration either at the server’s side or at the front-end tier. All you require is opening the specific ports to the client.

Disadvantages

  • When a switchover happens, you need to update the host resolver (DNS or the clients's /etc/hosts) to alter the resolution of all the hostnames that the external client uses to connect to the managed servers. For more information about implications of the client’s cache, see About T3/T3s Client’s DNS Cache.
  • Clients must be able to resolve and reach the hostnames set in WebLogic's Listen Address. It is not possible to use alternate names because the default channels use the WebLogic Server’s default Listen Address.
  • The WebLogic default channels listen for T3(s) and also HTTP(s) requests. You cannot disable this setting. If you open the default port for the clients, direct HTTP(s) to the server is also allowed which can result in security concerns.
  • You need to modify the client’s provider URL if you have to scale out or scale in a WebLogic cluster. The first contact in a T3/T3s invocation uses only the list in the client’s provider URL. So, even without updating the list, the subsequent requests can connect to any server of the cluster as the recovered stubs list all of the members. It is recommended that the client’s provider URL matches the real list of the servers for failover purposes in the first contact.
Direct T3/T3s Using Custom Channels

In this approach, the external T3/T3s client connects directly to custom channels defined in the WebLogic servers of the cluster. The Listen Address, the External Listen Address, and the External Port are customizable values in the custom channels. These values can differ from the WebLogic server’s default listen values.

Figure 2-5 Direct T3/T3s Using Custom Channels

This image shows the external T3/T3s client connecting directly to custom channels.

Once the T3/T3s external client is connected to the server during the initial context retrieval, the subsequent T3 calls connects directly to one of the listen addresses and ports configured in the custom channels as External Listen address and External Port. These requests will be load balanced according to the mechanism specified in the connection factory defined in the WebLogic and used by the client.

This approach is similar to Direct T3/T3s Using Default Channels with the exception that you can customize the addresses and ports used in T3/T3s calls when using WebLogic custom channels.

Configuration

  • Each WebLogic server has the appropriate custom channels configured and these custom channel uses a unique External Listen address that points to that server node. Table 2-4 and Table 2-5 provides example of Custom Channel in Server 1 and Server 2.

    Table 2-4 Custom Channel in Server 1

    Name Protocol Enabled Listen Address Listen Port Public Address Public Port
    t3_external_channel t3 true soahost1.example.com 8111 soahost1-t3ext.example.com 8111

    Table 2-5 Custom Channel in Server 2

    Name Protocol Enabled Listen Address Listen Port Public Address Public Port
    t3_external_channel t3 true soahost2.example.com 8111 soahost2-t3ext.example.com 8111
  • The external T3 client’s provider URL contains the list of the external address and port of the custom channels.

    Example:

     t3://soahost1-t3ext.example.com:8111,soahost2-t3ext.example.com:8111

    If you are using the T3s protocol, you must create the custom channels with T3s protocol. Then the clients will connect using T3s protocol and appropriate port.

    Example:

     t3s://soahost1-t3ext.example.com:8112,soahost2-t3ext.example.com:8112

    In a T3s channel, you can add a specific SSL certificate for the name used as an External Listen address.

  • The external T3/T3s client must resolve the custom channels’ external hostnames with the primary host’s IPs. These hostnames must be reachable from the client. It is possible to use NAT if there is a NAT address for each server. In that case, the client will resolve each server name with the appropriate NAT IP for the server.

    Example: Naming Resolution at the External Client Side

    172.11.2.113	soahost1-t3ext.example.com
    172.11.2.114	soahost2-t3ext.example.com

    With this approach, all the requests from the external client connect to soahost1-t3ext.example.com and soahost2-t3ext.example.com, both for the initial context retrieval and for the subsequent calls. You can control these requests by using the WebLogic Server load balancing mechanism.

Switchover

In a switchover scenario, you do not have to update the client's provider URL. You must update the entries in the DNS or in the /etc/hosts of the client. After the switchover, the names used to connect to the servers must resolve with the IPs of the secondary servers.

Example: Naming Resolution at the External Client Side After a Switchover
172.22.2.113	soahost1-t3ext.example.com
172.22.2.114	soahost2-t3ext.example.com

Advantages

  • This method allows using specific hostnames for the external T3/T3s communication different from the server's default listen address. This is useful if you do not want to expose the default server's Listen Address to the external clients for security reason.

  • This method is useful if you are using NAT IPs and you do not want to resolve the servers' default listen addresses with different IPs internally and externally.

  • This method is useful in case you want to use different names for external T3/T3s accesses just for organizational purposes. You can also use this method to isolate protocols in different interfaces or network routes.

  • The protocol in the custom channel can only be limited to T3/T3s. The HTTP(s) protocol can be disabled in the custom channels.

Disadvantages

  • When a switchover happens, you need to update the DNS or the clients's /etc/hosts at the external client side for all the hostnames used to connect to the managed servers. For more information about the implications of the client’s cache, see About T3/T3s Client’s DNS Cache.
  • If you are using T3s, then you must create and configure specific SSL certificates for the external names in the custom channels to avoid SSL hostname verification errors in the client.
  • You have to modify the client’s provider URL if you have to scale out or scale in a WebLogic cluster. The first contact only uses the list in the client’s provider URL. So, even without updating the list, the subsequent requests can connect to any server of the cluster as the recovered stubs list all of the members. It is always a good practice to ensure that the client’s provider URL matches the real list of the servers for failover purposes in the first contact.
Using Load Balancer for Initial Lookup

In this scenario, the client’s provider URL points to the load balancer address.

Direct T3/T3s Using Default Channels and Direct T3/T3s Using Custom Channels approaches can also use a load balancer for the initial context lookup.

However, the subsequent T3/T3s calls connects to WebLogic servers directly. If you use the default channel for these invocations, the request goes to the default channel’s listen address and port. Similarly, if you use the custom channels then the subsequent request goes to the external listen address and port defined in the customs channels.

Figure 2-6 Using Load Balancer for Initial Lookup

This image shows using load balancer for the initial lookup.

Configuration

  • You will need a TCP service in the load balancer to load balance the requests to the WebLogic Servers' T3/T3s ports (either to the default channels or to the custom channels when used).

  • The external T3/T3s client’s provider uses the front-end name and port of the load balancer as the point of contact.

    Example:

    t3://t3lbr.example.com:8111
  • You can use default channels or custom channels as explained in Direct T3/T3s Using Default Channels and Direct T3/T3s Using Custom Channels scenarios. The external T3/T3s client must resolve the custom channels’ external hostnames (or the default server’s listeners hostnames, if you are using default channels) with the primary host’s IPs. The client must be able to access them. It is possible to use NAT as long as there is a NAT address for each server. In that case, the client will resolve each server name with the appropriate NAT IP for the server.

    Example: Naming Resolution at the External Client Side
    111.111.111.111    t3lbr.example.com
    172.11.2.113	soahost1-t3ext.example.com
    172.11.2.114	soahost2-t3ext.example.com

Switchover

In a switchover scenario, you do not have to update the client's provider URL. You must update the entries in the DNS or in the /etc/hosts used by the client so that they resolve with the IPs of the secondary site. The name used to connect to the load balancer must resolve with the IP of the secondary load balancer and the server names used in subsequent requests must point to the IPs of the secondary servers.

Example: Naming Resolution at the External Client Side After a Switchover
222.222.222.222    t3lbr.example.com
172.22.2.113	soahost1-t3ext.example.com
172.22.2.113	soahost2-t3ext.example.com

Advantages

You do not have to modify the client’s provider URL if you add or remove the WebLogic Server nodes from the WebLogic cluster.

Disadvantages

  • Despite using a load balancer in front, the client still needs to reach the servers directly.

  • When a switchover happens, you need to update the DNS (or /etc/hosts) of servers’ addresses at the external client side. For more information about the implications of the client’s cache, see About T3/T3s Client’s DNS Cache.

  • The complexity of this method is higher for the T3s cases. The client connects both through the front-end LBR and directly to the server using a secure protocol. In this case, you will need to either skip the hostname verification at the client side or use an SSL certificate that is valid for front and back addresses (For example, wildcard or SAN certificates).

Using Load Balancer for All Traffic

In this approach, the initial context lookup goes through the load balancer and the subsequent connections. There are no direct connections from the external T3/T3s client to the servers.

Oracle does not recommend using LBR for load balancing all types of T3/T3s communications. It is only recommended for initial context lookup. For more information, see WebLogic RMI Integration with Load Balancers. However, there are T3/T3s use cases, where you can use the load balancer for complete T3/T3s communication flow. WebLogic T3/T3s are TCP/IP-based protocols, so it can support TCP load balancing when services are homogeneous such as JMS and EJB. For example, you can configure an LBR front-ended JMS subsystem in a WebLogic cluster in which remote JMS clients can connect to any cluster member.

However, this approach will not work with external clients that use JTA connections. A JTA subsystem cannot safely use TCP load balancing in transactions that span across multiple WebLogic domains. When the transaction is either committed or rolled back the JTA transaction coordinator must establish a direct RMI connection to the server instance that has been chosen as the transaction's sub coordinator.

This method is also not suitable for cases where you require direct connection to a specific server like JMX or WLST when you want to connect to a particular server only.

Figure 2-7 Using Load Balancer for All Traffic

This image shows using load balancer for all traffic.

Configuration

  • The load balancer requires a TCP service to load balance the requests to the WebLogic servers’ T3/T3s ports defined in the custom channels.

  • The external T3/T3s client’s provider URL uses the front-end name and port of the load balancer as the point of contact.

    Example:
    t3://t3lbr.example.com:8111
  • The external client must resolve the load balancer address with the IP of the primary site’s load balancer.

    Example:
    111.111.111.111  t3lbr.example.com
  • WebLogic Server requires custom channels. Configure the external listen address and port of these custom channels with the load balancer's address and port. Table 2-6 and Table 2-7 provides examples of Custom Channel in Server 1 and Server 2.

    Table 2-6 Custom Channel in Server 1

    Name Protocol Enabled Listen Address Listen Port Public Address Public Port
    t3_external_channel t3 true soahost1.example.com 8111 t3lbr.example.com 8111

    Table 2-7 Custom Channel in Server 2

    Name Protocol Enabled Listen Address Listen Port Public Address Public Port
    t3_external_channel t3 true soahost2.example.com 8111 t3lbr.example.com 8111

Switchover

In a switchover scenario, you do not have to update the client's provider URL. You must update the entries in the DNS (or in the /etc/hosts) of the client. After the switchover, the names used to connect to the servers resolves with the IP of the secondary LBR service.

Example: Naming Resolution at the External Client Side After a Switchover
222.222.222.222  t3lbr.example.com

Advantages

  • All the communication goes through the load balancer. The client only needs to know and reach the load balancer’s service.

  • If you have to either add or remove the WebLogic Server nodes from the WebLogic cluster, you do not have to modify the client’s provider URL

  • In case of a switchover, you only have to update the load balancer’s front-end name in the DNS (or in the /etc/hosts).

    Note:

    Although only one DNS name is updated, you need to refresh the client’s DNS cache. For more information, see About T3/T3s Client’s DNS Cache.
  • In T3s cases, you can use the same SSL certificate in all the custom channels associated with the load balancer service’s front-end name.

Disadvantages

  • This approach is not suitable for all the T3/T3s cases. It also cannot be used in scenarios with JTA as JTA needs to explicitly connect to the server instance that acts as the sub coordinator of the transaction.

About T3/T3s Client’s DNS Cache

All the approaches explained to access T3/T3s services mostly require a DNS update in disaster recovery switchover scenarios. Therefore, you must set the limit for DNS Cache TTL (Time to Live) in DNS server and client's specific cache.

The TTL limit in the DNS service is a setting that tells the DNS resolver how long to cache a query before requesting a new one. Note that the TTL value of the DNS entries will affect the effective RTO of the switchover. If the TTL value is high (for example, 20 minutes), the DNS change will take that time to be effective in the clients cache. Using lower TTL values will make this switchover faster, but this can cause overhead because the clients check the DNS more frequently. A good approach is to set the TTL to a low value temporarily (for example, one minute) before the DNS update. Once the change and switchover procedure is completed, set the TTL to the normal value again.

Besides the DNS server's TTL, networkaddress.cache.ttl Java property controls the Java clients' cache TTL. This Java property indicates the caching policy for successful name lookups from the name service. Specify the value as an integer to indicate the number of seconds to cache the successful lookup. Ensure you set a limit to the networkaddress.cache.ttl Java property so the client's Java cache does not cache the DNS entries forever or you might have to restart the client with each switchover.

Planning a File System Replication Strategy

Defining a file system replication strategy is important while designing a disaster protection system. Oracle does not recommend creating and managing the secondary system by repeating installations and configuration steps.

Applying changes first in the primary system (for example, deploying a data source) and then repeating the step in the secondary system is not a reliable approach. Even when using deployment scripts and automated pipelines, there is rarely a guarantee of transaction. This can lead to changes being applied only in the primary system and not propagated properly to the secondary system. Some configuration changes require WebLogic servers to be running and this may cause duplicated processing and undesired behavior. But more importantly, a dual change approach typically gives room to inconsistencies and management overhead that can affect the RTO and RPO of the disaster protection solution.

Instead, different file replication strategies can be used to replicate configuration changes across sites. File system replication is used in the initial setup and during the ongoing lifecycle of a system. Costs, management overhead, and RTO and RPO may have different implications during setup and ongoing lifecycle, so different replication approaches can be used in each case. Also, there are different data types involved in a Fusion Middleware topology and each one may also have different RTO and RPO needs depending on their size on the file system and how frequently they are generated and modified. You must first understand the different data types involved in a disaster protection topology to decide which replication strategy is suitable for your needs.

Data Tiers in a Fusion Middleware File System

All files involved in the Fusion Middleware system should be replicated from the primary to the secondary at the same point in time. However, different file types may need different replication frequency to simplify management costs and to reduce the total cost of ownership of a disaster protection system. This is important when you design the volumes and file systems that you want to use for replication. Some artifacts are static whereas others are dynamic.

Oracle Home and Oracle Inventory

An Oracle Home is the directory into which all the Oracle software is installed and is referenced by Fusion Middleware and WebLogic environment variables. Oracle Fusion Middleware allows you to create multiple Oracle WebLogic Server Managed Servers from one single binary file installation. The Oracle Fusion Middleware Enterprise Deployment Guides provide recommendations on how to configure storage to install and use Oracle Home so that there is no single point of failure in the Fusion Middleware system while optimizing the storage size and simplifying maintenance.

It is not necessary to install Oracle Fusion Middleware instances at the secondary site as it is installed at the production site. When the production site storage is replicated to the secondary site storage, the Oracle software installed on the production site volumes are replicated at the secondary site volumes.

A secondary system needs to behave exactly like the primary system when a failover or switchover occurs. It should admit patches and upgrades as a first-class installation. This means that when a failover or switchover occurs, the secondary system must use a standard inventory for patches and upgrades. To maintain consistency, it is required to replicate the Oracle Inventory with the same frequency as the Oracle Homes used by the different Fusion Middleware components (RTO, RPO, and consistency requirement of the Oracle Inventory must be the same as the one prescribed for Oracle Homes). The Oracle Inventory includes oraInst.loc and oratab files which are located in the /etc directory.

When an Oracle Home or a WebLogic home is shared by multiple servers in different nodes, Oracle recommends that you keep the Oracle Inventory and Oracle Home list in those nodes that are updated for consistency in the installations and application of patches. To update the inventory files in a node and attach an installation in a shared storage to it, use the ORACLE_HOME/oui/bin/attachHome.sh script.

Both Oracle Home and Oracle Inventory are static artifacts during runtime and typically require a low RTO (since they are the binaries from which Fusion Middleware domains run). It is not required to copy them across regions too frequently because they change only when patches and fixes are applied. From a consistency point of view, runtime artifacts are not usually too restrictive. For example, most patches will support previous configurations of a WebLogic Server domain. So even if you do not propagate a patch in a WebLogic Oracle Home across regions, the secondary domain configuration will work properly.

Configuration Artifacts

Configuration artifacts are files that change frequently and includes the following:

  • WebLogic Domain Home: Domain directories of the Administration Server and the Managed Servers.

  • Oracle instances of system components such as Oracle HTTP Server: Oracle Instance home directories.

  • Application artifacts like .ear or .war files.

  • Database artifacts like the MDS repository and the JDBC persistent stores definitions.

  • Deployment plans used for updating technology adapters like the file and JMS adapters. They need to be saved in a location that is accessible to all nodes in the cluster where the artifacts are being deployed to.

Configuration artifacts change frequently depending on application updates. They require a low RTO and a high replication frequency. It is important to maintain consistency for configuration artifacts in different stores, else the applications may stop working after a restore. For example, WebLogic domain configuration reflecting a new JMS server must be aligned with the database table that it uses as persistent store. Replicating only the WebLogic domain configuration without replicating the pertaining table will cause a WebLogic Server failure.

Runtime Artifacts

Runtime artifacts are files generated by applications at runtime. For example, files generated by SOA’s file or FTP Adapters, files managed by Oracle MFT, or any other information that applications generate through their business logic and that is stored directly on the file system. These files may change very frequently. Their RTO and RPO are purely driven by business needs. In some cases, these type of artifacts may need to be discarded after a short period of time. For example, a bid order that expires in a short period. In other cases, these files may contain transactional records of operations completed by an application that need to be preserved. How frequently they need to be replicated and how important it is to preserve this type of files in a disaster event is usually a business-driven decision.

Main Replication Options

Oracle verifies different approaches to replicate the different types of data used by Fusion Middleware deployments on the file system. Although there are other options, only the following ones are formally tested and can be utilized for Fusion Middleware deployments.

Storage Level Replication

The Oracle Fusion Middleware Disaster Recovery solution can use storage replication technology to replicate the secondary Oracle Fusion Middleware middle-tier primary system. This means using specific storage vendor technology to replicate storage volumes across regions. Typical solutions in this area involve creating an initial or baseline snapshot of the different volumes presented in the EDG and then manually, on schedule or continuously replicating changes incrementally to a secondary location.

Do not use storage replication technology to provide disaster protection for Oracle databases. Ensure that disaster protection for any database that is included in the Oracle Fusion Middleware Disaster Recovery production site is provided by Oracle Data Guard.

Ensure that your storage device's replication technology guarantees consistent replication across multiple volumes. Otherwise, problems may arise when different configuration and runtime artifacts used by the Fusion Middleware systems are replicated at different points in time. Most storage vendors guarantee consistency when multiple nodes are using different storage volumes and it is needed to take a backup of all of them (or replicate their contents) in a precise point in time. Volumes are grouped in a single logical unit frequently called consistency volume group, consistency group, or volume group. Volume groups provide a point in time copy of the different volumes used in an EDG so that a consistent snapshot of all of them is propagated to the secondary. Imagine a situation where a database module deployment resides in volume A while the application that uses it resides in volume B. Without consistency groups it could occur that a new version of the WAR for the application is replicated while the data source configuration it requires is not, thus causing a consistency problem in the secondary.

Storage level replication can be used as a general-purpose approach for all types of data in the EDG (Oracle Home/Oracle Inventory, configuration artifacts, and runtime artifacts). It is a valid approach both for the initial setup and ongoing replication cycles during the lifetime of a system.

Rsync

Rsync can be used as an alternative to using storage vendor-specific replication for replicating the primary Fusion Middleware system to the secondary system. Rsync is a versatile copying tool, that can copy locally or to/from another host over any remote shell. It offers a large number of options that control every aspect of its behavior and permits very flexible specification of the set of files to be copied.

Rsync implements delta-transfer algorithm which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Due to these advantages and convenience, it is a widely used tool for backups and mirroring. To guarantee a secure copy in all cases, it is recommended using rsync over SSH.

Although rsync is reliable and implements implicit retries, network outages and other connectivity issues can still cause failures in the file synchronization. Additionally, rsync does not guarantee consistency across different nodes. It does not provide a distributed point in time replication. Hence, rsync can be used as an alternative to the storage level replication under the following conditions:

  • When the storage replication is not possible or feasible and/or does not meet the cost requirements.

  • When primary and standby use a reliable and secure network connection for the copy.

  • When checks are performed across the different nodes involved in the copy to ensure that it is valid.

  • When the disaster recovery site is validated on a regular basis.

Rsync is present in most Linux distributions these days. You can install and set up rsync to enable replication of files from a production site host to its secondary site peer host. For instructions about how to install, set up, and use the utility, see the utility manual and rsync.

In a rsync replication approach, there are two main alternatives to copy from the primary to the secondary:

Figure 2-8 Peer-to-Peer Copy

replication approach for rsync

In this case, the copy can be done directly from each host to its remote peer. Each node has SSH connectivity to its peer and uses rsync commands over SSH to replicate the primary system.

shows an overview of using rsync commands over SSH to replicate the primary system.

In this approach, each node uses individual scripts to rsync its Fusion Middleware installation and configuration to the remote peer node. Each node can cron the scripts. Different data types can have different schedules for the cron jobs. For example:

  • Products once a month.

  • Configuration once a day.

This is an easy setup that does not need additional hardware resources. However, it requires maintenance across many nodes since scripts are not centralized (for example, large clusters add more complexity to the solution). It is also difficult to guarantee consistency at a point in time since each node stores a separate copy. The consistency and validation of this approach can be improved using a central host to launch the scripts in each node (acts like a single scheduler) and to validate the copies.

Staging Location

In this approach, a staging location is used and typically a node acts as a coordinator connecting to each individual host that needs to be replicated. This coordinator copies the required configuration and binary data to a staging location. This can be on a shared storage attached to all nodes or through a staging host. This staging location can be validated without affecting the primary or the secondary nodes. This approach offloads the individual nodes from the overhead of the copies and maintains consistency for binaries by using a single install that is distributed to the rest of the nodes. It is for this reason that this approach is typically preferred to the peer-to-peer copy. When there are strong security restrictions in terms of connection allowed across data centers, it is also possible to use a node local to the primary that stages the copy from the primary nodes and distributes them to the secondary staging location. This approach limits the access across regions to just two nodes and offloads the remote copies (typically incurring in longer latency and overhead) to the coordinator box.

Figure 2-9 Staging Location

image shows the node acting as a coordinator

Rsync can be used as a general-purpose approach for all types of data in the EDG but needs to be tested in each specific deployment when applications are generating large amounts of data (constantly changing) during runtime. Rsync may overload the nodes running the replication and workload tests need to be executed to verify its validity. For configuration and binaries, rsync should not cause any performance issues both for the initial setup and ongoing replications. For runtime artifacts it is required to consider the required RTO and RPO to calculate data modification rates and possible overhead caused by the rysnc copy.

Oracle Database File System

Oracle Database File System (DBFS) is an additional method that can be used for replicating primary to standby. It is mainly intended to deal with configuration or data that does not grow and changes very frequently (because of the overhead it generates in the database both at the connection and the storage level). Conceptually, a database file system is a file system interface for data that is stored in database tables.

DBFS is similar to NFS which provides a shared network file system that looks like a local file system and has both a server component and a client component. The DBFS file system can be mounted from the mid-tier hosts and accessed as a regular shared file system. Any content that is copied to the DBFS mount (as it resides in the database), is automatically replicated to the secondary site through the underlying Data Guard replication.

This method takes advantage of the robustness of the Data Guard replica. It has good availability through Oracle Driver’s retry logic and provides a resilient behavior. It can be used in scenarios with medium or high latencies between the data centers. However, using DBFS for configuration replication has additional implications from the setup, database storage, and lifecycle perspectives as follows:

  • It introduces some complexity due to the configuration and maintenance required by the DBFS mount. It requires the database client to be installed in the host that is going to mount it and requires an initial setup of some artifacts in the database (tablespace, user, and so on) and in the client (the wallet, the tnsnames.ora, and mount scripts).

  • It requires additional capacity in the database because the content copied to the DBFS mount is stored in the database.

  • Oracle does not recommend storing the WebLogic domain configuration or the binaries directly in the DBFS mount. This would create a very strong dependency between the Fusion Middleware files and the database. Instead, it is recommended to use the DBFS as an "assistance" file system like an intermediate staging file system to place the information that is going to be replicated to the secondary site. Any replication to standby implies two steps in this model — from the primary's origin folder to the intermediate DBFS mount and then in the secondary site from the DBFS mount to the standby's destination folder. The intermediate copies are done using rsync. As this is a low latency and local rsync copy, some of the problems that arise in a remote rsync copy operation are avoided with this model. The following diagram illustrates the replication flow

    Figure 2-10 Oracle Database File System (DBFS)

    DBFS mounts in primary and secondary sites
  • The DBFS directories can be mounted only if the database is open. When the Data Guard is not an Active Data Guard (ADG), the standby database is in mount state. Hence, to access the DBFS mount in the secondary site, the database needs to be converted to snapshot standby. When ADG is used, the file system can be mounted for reads and there is no need to transition to snapshot.

Due to the above stated reasons, it is not recommended to use DBFS as a general purpose solution to replicate all the artifacts (especially runtime files) to the standby. Using DBFS to replicate the binaries is an overkill. However, this approach is suitable to replicate (through a staging directory) few artifacts like the domain shared configuration where other methods like the storage replication or rsync do not fit the system needs. Cost or unreliable network connections between the primary and the secondary makes DBFS the best approach for file replication.

Comparing Different Replication Methods

Defining a file system replication strategy implies making a decision considering the RTO, RPO, management complexity, and total cost of ownership of each approach. Different aspects may drive the decision and the following is a list with the most relevant features. You must analyze your specific application needs and how these aspects may affect your disaster recovery design.

Management Complexity

The configuration replication procedures in the DBFS and rsync (using a staging location) methods are similar irrespective of the number of nodes in the WebLogic domain. Contrary to this, shared storage replication management gets more complicated as the number of replicated volumes increases. It requires a good lifecycle management of the different volumes and volume groups. Also, switchover and failover operations are more complex when using storage replication than in the rsync and DBFS methods. There may be additional pre and post switchover or failover steps including the activation and attachment of the replicated volumes. This complexity increases when each WebLogic managed server node uses private volumes that need to be replicated individually.

Cost Implication

In the DBFS approach, the increased costs are related to the additional storage required in the database to host the DBFS tables. Typical database tablespace and storage maintenance operations are required for an efficient recovery of allocated space. The network overhead in the cross-region copy is typically neglectable in comparison to the overall Data Guard traffic requirements.

With rsync, the storage requirement is low since file storage allocation for a WebLogic domain is typically below 100 gigabytes and the copy over rsync does not have a big network impact either.

With storage replication, once you enable replication for a volume or share, additional costs emerge in most storage vendors. Replication efficiency is also a factor affecting network and storage costs. As part of the replication process, all data being updated on the source volumes or shares is transferred to the volume replica, so volumes with continual updates incur higher network costs.

Replication Overhead

Copying through a staging directory in the DBFS and rsync scenarios does have an impact on the replication coordinator server (typically the Weblogic administration node). Additional memory and CPU resources may be required in it depending on the frequency of the copy, how big the WebLogic domain is, and whether other runtime files need to be replicated across regions in the same replication cycle.

Recovery Time Objective (RTO) for File System Data

The switchover RTO is similar in the DBFS, rsync, and storage replication approaches. For failover operations, additional steps are required by shared storage replication (activate snapshots, attach or mount volumes, and so on). These operations typically increment the downtime during a failover.

Recovery Point Objective (RPO) for File System Data

Storage replication process is continuous in most vendors with the typical Recovery Point Object (RPO) target rate being as low as a few minutes. However, depending on the change rate of data on the source volume the RPO can vary. In the DBFS and rsync methods, the user can have a finer control on the RPO for the WebLogic configuration because the information is replicated using a scheduled script and the amount of information replicated is lower than the entire storage volume. On the other hand, in the DBFS and rsync methods, it is typically the administration server’s node that acts as "manager" of the domain configuration received, so this node’s availability and capacity drives the speed of the configuration copy. When using storage replication, the replication keeps taking place regardless of the secondary nodes being up and running which can be used as an offline replication approach.

In summary, your system’s RTO, RPO, and cost needs will drive the decision for the best possible replication approach. Analyze the requirements of your applications to determine the best solution.

Preparing the Primary System

It is assumed that the primary site is created following the best practices prescribed in the Oracle Fusion Middleware Enterprise Deployment Guide. The network, hostnames, storage and database recommendations provided in the EDG need to be already present in the primary system.

This guarantees that before providing protection against disaster, local outages in the different layers (expected to happen much more frequently than disasters affecting an entire data center) will also be preempted.

The listen addresses, storage allocations, and directory considerations included in the EDG are already designed with disaster protection in mind. It is possible that some alternatives may have been adopted in the primary which can be corrected to implement disaster protection. The most common ones are using local storage for binaries and configuration information (instead of the shared storage recommended by the EDG) or not using the appropriate aliases as listen address for the different Fusion Middleware components. The following sections describe how to prepare the primary site for disaster recovery.

Preparing the Primary Storage for Storage Replication

Moving Oracle Homes To Shared Storage

When you use storage replication for Oracle Fusion Middleware Disaster Recovery, the Oracle Home and middle-tier configuration should reside on the shared storage according to the recommendations provided in the Enterprise Deployment Guide for your Fusion Middleware component. This facilitates maintenance of Oracle Homes while providing high availability since at least two Oracle Home locations are used (one for even nodes and one for odd nodes).

If the production site was initially created without disaster recovery in mind, the directories for the Oracle Fusion Middleware instances that comprise the site might be located on the local storage private to each node. In this scenario, Oracle recommends migrating the homes completely to shared storage to implement the Oracle Fusion Middleware Disaster Recovery solution. This operation may incur downtime on the primary system and needs to be planned properly to minimize its impact. This change is recommended because not only does it facilitate the DR strategy but also simplifies the Oracle Home and domain configuration management.

Follow these guidelines for migrating the production site from the local disks to the shared storage:

  • Perform offline backup of the folders that are going to be moved to the shared storage. If the backup is performed using OS commands, it must be done as the root user and the permissions must be preserved.

    See Types of Backups and Recommended Backup Strategy in Oracle Fusion Middleware Administering Oracle Fusion Middleware

  • Although you move the content to NFS, the path will be preserved. The current folder that is going to be moved to an NFS (for example, /u01/oracle/products) will become a mount point. In preparation for this, move or rename the current folder to another path so the mount point is empty.

  • Ensure that the mount point where the shared storage will be mounted exists, is empty, and has the correct ownership.

  • Mount the shared storage in the appropriate mount point. The directory structure on the shared storage must be set up as described in the Enterprise Deployment Guide.

  • Once the shared storage is mounted, copy the content from the backup (or from the renamed folder) to the folder, that is now in the shared storage.

For example, moving a products folder, that is in /u01/oracle/products, from local disk to an NFS folder.

  • To backup the content in the local folder, a copy is performed by user root, preserving the mode:
    [root@soahost1]# cp -a /u01/oracle/products /backups/products_backup
  • The current folder is moved because the folder that will become the mount point should be empty:

    [root@soahost1]# mv /u01/oracle/products /u01/oracle/products_local
  • After renaming the folder, check that the mount folder point exists (if it does not exist, it must be created) and verify that it is empty and with the correct ownership. The mount point is /u01/oracle/products in the following example:

    [root@soahost1]# mkdir -p /u01/oracle/products
    [root@soahost1]# ls /u01/oracle/products
    [root@soahost1]# chown oracle:oinstall /u01/oracle/products
  • The NFS volume is mounted in the mount point.

    [root@soahost1]# mount -t nfs nasfiler:VOL1/oracle/products/ /u01/oracle/products/
  • To make this persistent, add the mount to the /etc/fstab on each host.

  • Now the content from the backup is copied to the mount:

    [root@soahost1]# cp -a /backups/products_backup /u01/oracle/products 

You can use a similar approach to move private WebLogic domain directories to shared storage. This will facilitate the replication without requiring a staging location for the managed servers domain directory content. Refer to the EDG for details on how to use shared storage for the managed server private configuration directories.

Creating Volume Groups

Once the pertaining volumes on shared storage have been allocated according to the EDG recommendations, you need to prepare snapshots and volume groups for replication.

Most storage vendors guarantee consistency when multiple nodes are using different storage volumes and it is needed to take a backup of all of them (or replicate their contents) at a precise point in time. Volumes are grouped in a single logical unit frequently called consistency volume group, consistency group or volume group. Volume groups provide a point in time copy of the different volumes used in an Enterprise Deployment Guide so that a consistent snapshot of all of them is propagated to the secondary.

Imagine a situation where a database module deployment resides in volume A while the application that uses it resides in volume B. Without consistency groups it could occur that a new version of the WAR for the application is replicated while the data source configuration it requires is not, thus causing a consistency problem in the secondary. To avoid situations like this and while using the different volumes suggested in the Oracle Fusion Middleware Enterprise Deployment topology, Oracle recommends the following consistency groups. This example is using the precise case of Oracle Fusion Middleware SOA Suite but can be extrapolated to other Fusion Middleware systems:

  • Create one consistency group with the volumes that contains the domain directories for the Administration Server and Managed Servers as members (DOMAINGROUP in Table 2-8). In Oracle Fusion Middleware 14.1.2, the EDG places the Identity and Truststores required for configuring SSL for the different components (Node Manager, WebLogic Servers) in the same volume as the Administration Server’s domain configuration. If you are placing SSL certificates and stores in a different location, ensure that it is included as part of the DOMAINGROUP consistency group, since the SSL configuration for the WebLogic parts has a dependency on the KESYTORE_HOME location (refer to the EDG for details on configuring SSL for the different components).

  • Create one consistency group with the volume that contains the runtime artifacts generated by the applications running in the WebLogic Server domain (if any) (RUNTIMEGROUP in Table 2-8).

  • Create one consistency group with the volume that contains the Oracle Home as members (FMWHOMEGROUP in Table 2-8).

Table 2-8 provides a summary of Oracle recommendations for consistency groups for the Oracle SOA Suite topology as shown in Figure 2-1.

Table 2-8 Consistency Groups for Oracle SOA Suite

Tier Group Name Members Comments

Application

DOMAINGROUP

VOLADMIN

VOLSOA1

VOLSOA2

Consistency group for the Administration Server, and the Managed Server domain directory

Application

RUNTIMEGROUP

VOLRUNTIME

Consistency group for the shared runtime content

Application

FMWHOMEGROUP

VOLFMW1

VOLFMW2

Consistency group for the Oracle Homes

Preparing the Primary Storage for Rsync Replication

Using Peer-to-Peer

When using the rsync replication approach in a peer-to-peer model, preparing primary involves preparing the pertaining rsync scripts for the initial copy to each peer (one per node). It is recommended to offload the primary systems from performing the copy and running checksum validations. In terms of CPU, memory impact, and number of Disk I/O operations there is no significant difference whether secondary pulls information from the primary or the primary pushes the file system copy to the secondary. However, starting the rsync from secondary has other advantages as follows:

  • Cron jobs’ scheduling does not interfere with the primary (primary does not need to schedule anything and does not need to be manipulated to start rsync ops).

  • There is a better control on hard stop for errors during rsync operations.

  • Log and analysis of the operations is better offloaded to secondary.

It is for these reasons that in the peer-to-peer model, rsync scripts are typically prepared in secondary and not considered part of the primary preparation. The only aspect that needs to be accounted for is the appropriate SSH connectivity between the secondary and primary. Before running the rsync operations, ensure the SSH connectivity is allowed and working between each node in secondary and its peer in primary. It is recommended to use a SSH key for the access. Try a simple ssh -i path_to_keyfile primary_wlsnode1_ip_primary from the wlsnode1 peer in secondary. Repeat the verification for each OHS and WLS node participating in the topology. The scripts and first replications will be executed as part of the secondary setup.

Using a Staging Location

When using the staging approach, you should create a central location (ideally on shared storage) to host the staging copy and scripts to transfer from the primary nodes to the staging directory and scripts to transfer from the staging directory to the target secondary nodes. When using two staging locations (one in the primary and one in the secondary to reduce the connectivity requirements across sites), three sets of scripts are required: to copy from primary nodes to the staging location in primary, to copy from the staging location in primary to the staging location in secondary, and to copy from the staging location in secondary to the different nodes in secondary. This approach is also useful as an extension to a backup strategy. Each location has a backup copy that can be distributed to the different nodes. Prepare a script in the node responsible for the staging (typically the admin server node in secondary) per node in primary. You should characterize each nodes WebLogic domain copy. For example, using a structure as follows:

Figure 2-11 Staging Location

location of the staged inventory

Before running the rsync operations ensure SSH connectivity is allowed and working between each node in primary and the staging node in primary and between the staging node in primary and the staging one in secondary (if the two-staging-locations model is used). Try a simple ssh -i path_to_keyfile primary_stagenode_ip_primary from each WebLogic and OHS node in primary. If you are using staging locations both in primary and secondary, check SSH connectivity from the staging orchestrator in secondary also.

For each production site host on which one or more Oracle Fusion Middleware components have been installed, set up rsync to copy the following directories and files to the same directories and files on the secondary to the staging host:

  1. Oracle Home directory and subdirectories.

  2. Oracle Central Inventory directory which includes the Oracle Universal Installer entries for the Oracle Fusion Middleware installations.

  3. Shared config folder such as the WebLogic Administration Server domain directory, deployment plans, applications, keystores, and so on. Since this folder is shared by all the mid-tier hosts in the EDG, you do not have to have a copy for each host. Do not replicate the shared WebLogic domain directory only. In context of the EDG, there are dependencies in the /u01/oracle/config/applications and /u01/oracle/config/keystores directories that need to be copied with the WebLogic domain configuration.

  4. Private config folders such as the WebLogic managed server's domain and local node manager home on the host.

  5. Shared runtime folder (if applicable).

  6. Oracle Fusion Middleware static HTML pages directory for the Oracle HTTP Server installations on the host (if applicable).

You can use the scripts at GitHub as an example to prepare the primary system for the staging copy. The script rsync_for_WLS.sh is a wrapper that invokes rsync_copy_and_validate.sh. This second script contains the real logic to perform rsync copies with the recommended rsync configuration and executes a thorough validation of files after the copy is completed. If any differences are detected after several validations retries, these are logged so that they can be acted upon.

By default, the rsync_copy_and_validate.sh uses oracle as the user to perform the copies. If a different user owns the origin folders, customize the property USER in the script.

It also uses the environment variable DEST_FOLDER to determine the target location that will be used to rsync contents. If the variable is not set, the script copies contents to the node executing the script to the exact same directory path used in the source node. While using the staging approach, set the variable to the desired path (for example , for the first WLS node private configuration) /staging_share_storage/midtier/wls_private_config/wlsnode1_private_config.

  • To rsync any folder from primary nodes, use rsync_for_WLS.sh in the staging node in primary as the user owning the directory in each primary node . rsync_for_WLS.sh needs to be executed with three parameters when using SSH key file for SSH connections: the IP of the node from which the copy will be pulled, the path to be replicated, and the ssh key file. For example:

    ./rsync_for_WLS.sh 10.1.1.1 /u01/oracle/config/domains/soaedg/  /home/oracle/keys/SSH_KEY.priv

    It can also be executed with only two parameters (the IP of the node from which the copy will be pulled and the path to be replicated). In this case, the SSH connection used by rsync will prompt for SSH password (password-based SSH needs to be set up before running the script).

    ./rsync_for_WLS.sh 172.11.2.113 /u01/oracle/config/domains/soaedg/  
  • Replicate the entire primary administration server node first. Execute the script for the products folder, the shared configuration folder, and the private domain configuration folder as follows (using the examples provided in this book, 172.11.2.113 is the IP of the primary administration server):

    
    [oracle@staging_node]$ export DEST_FOLDER=/staging_share_storage/midtier/wls_products_home/wls_products_home1
    [oracle@staging_node]$./rsync_for_WLS.sh 172.11.2.113 /u01/oracle/products /home/oracle/keys/SSH_KEY.priv
    [oracle@staging_node]$ export DEST_FOLDER=/staging_share_storage/midtier/wls_shared_config
    [oracle@staging_node]$./rsync_for_WLS.sh 172.11.2.113 /u01/oracle/config /home/oracle/keys/SSH_KEY.priv
    [oracle@staging_node]$ export DEST_FOLDER=/staging_share_storage/midtier/wls_private_config/wlsnode1_private_config
    [oracle@staging_node]$./rsync_for_WLS.sh 172.11.2.113 /u02/oracle/config/ /home/oracle/keys/SSH_KEY.priv
  • Now replicate the second WLS server node. Execute the script for the products folder and the private domain configuration folder as follows (using the examples provided in this book 172.11.2.114 is the IP of the primary WLS server node 2):

    
    [oracle@staging_node]$ export DEST_FOLDER=/staging_share_storage/midtier/wls_products_home/wls_products_home2
    [oracle@staging_node]$./rsync_for_WLS.sh 172.11.2.114 /u01/oracle/products /home/oracle/keys/SSH_KEY.priv
    [oracle@staging_node]$ export DEST_FOLDER=/staging_share_storage/midtier/wls_private_config/wlsnode2_private_config
    [oracle@staging_node]$./rsync_for_WLS.sh 172.11.2.114 /u02/oracle/config /home/oracle/keys/SSH_KEY.priv
  • Replicate the two OHS nodes. Since the OHS instances is run in the standalone mode there is no shared configuration folder and only the private domain configuration needs to be replicated as follows (using the examples provided in this book, 172.11.2.111 is the primary OHS node 1):

    
    [oracle@staging_node]$ export DEST_FOLDER=/staging_share_storage/webtier/ohs_products_home/ohs_products_home1
    [oracle@staging_node]$./rsync_for_WLS.sh 172.11.2.111 /u02/oracle/products /home/oracle/keys/SSH_KEY.priv
    [oracle@staging_node]$ export DEST_FOLDER=/staging_share_storage/webtier/ohs_private_config/ohsnode1_private_config
    [oracle@staging_node]$./rsync_for_WLS.sh 172.11.2.111 /u02/oracle/config /home/oracle/keys/SSH_KEY.priv

After this, you should have a complete copy of all the OHS and WLS nodes in the staging location (in primary or secondary depending of the staging model used).

Preparing the Primary Storage for DBFS

In the case of DBFS, preparing the primary system requires installing the DB client in the hosts that will use the DBFS mount point. This requires creating few artifacts in the database (tablespace, user, and so on) and in the client nodes (the wallet, the tnsnames.ora, and so on ). Download the scripts from GitHub and perform the following steps to configure the required DBFS mount:

  1. Download the Oracle DB client from edelivery and upload it to the mid-tier host (do not install it yet). Ensure that you download the installer version, not the image-based version. Download the software and upload it to all the mid-tier hosts. For example, to /u01/install/V982064-01.zip.

  2. Locate the script dbfs_dr_setup_root.sh in the scripts you downloaded from GitHub and copy it to the mid-tier hosts. This script performs the required tasks to get the DBFS mount ready in the host. It installs the DB client and its required operating system packages, it configures the DBFS user and schema in the database, and it mounts the DBFS file system and creates a cron so the DBFS file system is mounted on host boot.

  3. Execute the script as root user using the following syntax:

    ./dbfs_dr_setup_root.sh <local_db_scan_name> <db_port> <local_PDB_service> <pdb_sys_password> <path_to_dbclient_installer>

    As input parameters, you need to provide the connection data used to connect to the local database used by the WLS. Provide primary PDB connection data in the primary site mid-tier nodes and provide secondary PDB connection data in the secondary site mid-tier nodes. You can get the local scan name, port, and PDB service name from the data sources of each domain.

    Note:

    Before running the script on the secondary hosts, you must convert the standby database to snapshot mode.

    You must provide primary PDB values. Example to run primary mid-tier hosts (it must be a single line):

    ./dbfs_dr_setup_root.sh drdba-scan.wlsdrvcnlon1ad2.wlsdrvcnlon1.oraclevcn.com 1521 PDB1.wlsdrvcnlon1ad2.wlsdrvcnlon1.oraclevcn.com mypassword /u01/install/V982064-01.zip

    Provide the appropriate service name, so DBFS will connect to a CRS service rather than the default PDB administration one. Example of execution in a RAC scenario:

    ./dbfs_dr_setup_root.sh drdbrac2a-scan.subnetlon1.myvcnlon.oraclevcn.com 1521 mypdbservice.mycompany.com mypassword /u01/install/V982064-01.zip

  4. Verify if the DBFS mount is now available in the mid-tier host:

    
    [root@ wlsociprefix-wls-1]# df -h | grep dbfs
    dbfs-@PDB1:/ 32G 248K 32G 1% /u02/data/dbfs_root
    [root@ wlsociprefix-wls-1]# ls /u02/data/dbfs_root
    dbfsdir

Repeat the following steps in all the mid-tier hosts:

  1. The db client software is installed in the mid-tier host, in the folder /u01/app/oracle/client. The required packages for the db client are installed using yum.

  2. A user is created in the PDB database for DBFS. The username is dbfsuser and its password is set to the same SYS password provided as input of the script.

  3. A DBFS tablespace is created in the PDB database for the DBFS mount (tablespace name is tbsdbfs) and a DBFS folder (name dbfsdir).

  4. A new folder is created in the domain, DOMAIN_HOME/dbfs. It contains the wallet to store the dbfsuser’s password and other artifacts needed (tnsnames.ora, sqlnet.ora). This wallet is used by the db client to mount the dbfs mount in the mid-tier host.

  5. The script dbfsMount.sh is also created in DOMAIN_HOME/dbfs. This script is used to mount the dbfs mount in the mid-tier. It is also added to the cron on reboot, so the script is executed when the machine is rebooted.

  6. The DBFS mount is mounted in /u02/data/dbfs_root mount point as the folder dbfsdir.

If there is a failure, the script can be executed again. In these re-executions, there are warnings because few artifacts may have been created already in the DB (db user, tablespace, and so on), but these messages can be ignored.

Preserving the Production Hosts Hostnames as Listener Address

When a primary site is created without considering disaster protection, it is possible that the Fusion Middleware components are not configured using host name aliases as listener addresses (as recommended in the EDG and explained in the previous sections of this document) and instead they may be using the physical host names of the nodes where they reside. In this case, you have the following two options:

  • Modify the WebLogic domain configuration in the existing site to use host aliases as listener addresses for the servers. This change has additional implications as all clients accessing the WLS servers’ endpoints (like JMS or RMI clients) directly need to be made aware of the new hostname.

    Note:

    This is not the case for HTTP access as they should already be using a front-end load balancer virtual server.
  • If modifying the production configuration is not feasible, you can preserve the configuration as it is (continue using the physical host names as listener addresses for the Fusion Middleware components) and adjust the secondary to these hostnames by manipulating host name resolution.

    This can be done when using separate DNS services in the primary and the secondary or manipulating local host resolution in the secondary nodes involved (For more details, see Host Name Resolution section). In this case, the primary physical host names must be added as aliases to the /etc/hosts of the secondary site hosts.

    Example:

    /etc/hosts entries in the production site hosts that do not use aliases for the components

    127.0.0.1     localhost.localdomain     localhost
    172.11.2.134  prsoa-vip.example.com     prsoa-vip
    172.11.2.111  prweb1.example.com        prweb1
    172.11.2.112  prweb2.example.com        prweb2
    172.11.2.113  prsoa1.example.com        prsoa1
    172.11.2.114  prsoa2.example.com        prsoa2

    /etc/hosts entries in the secondary site hosts

    127.0.0.1    localhost.localdomain    localhost
    172.22.2.134  scdrysoa-vip.example.com  scdrysoa-vip prsoa-vip.example.com prsoa-vip
    172.22.2.111  scdryweb1.example.com     scdryweb1    prweb1.example.com    prweb1
    172.22.2.112  scdryweb2.example.com     scdryweb2    prweb2.example.com    prweb2
    172.22.2.113  scdrysoa1.example.com     scdrysoa1    prsoa1.example.com    prsoa1
    172.22.2.114  scdrysoa2.example.com     scdrysoa2    prsoa2.example.com    prsoa2

Preparing Data Sources in the Primary Middle-Tier

There are several approaches that can be used to configure the connection strings used in WebLogic data sources. For Disaster Recovery and Maximum Availability purposes, Oracle recommends using TNS alias in these connection strings. Other approaches are valid alternatives to address needs in special cases like local database standbys or stretched clusters.

Using TNS alias is recommended for configuration maintenance and availability purposes. Instead of using long connection strings repeated in each WebLogic data source, you can use a TNS alias in the JDBC URL. For more information, see Use a TNS Alias Instead of a DB Connection String in Administering JDBC Data Sources for Oracle WebLogic Server. It is also prescribed in the Enterprise Deployment Guide. In Oracle Fusion Middleware 14.1.2, database client modules are used to deploy tnsnames.ora files so that they are managed directly by the WebLogic infrastructure. The TNS alias is the same in the primary and the secondary, hence data sources use the same DB connect string text in both locations. The TNS alias is resolved with a tnsnames.ora file that is managed by the WebLogic domain configuration. This file should not be replicated from primary to standby, so each site maps the alias to its local database.

  • If it is included in storage replication, it needs to be updated after each replication cycle with the appropriate local database service and address in the secondary.

  • If rsync or DBFS is used, it simply needs to be in the excluded file list in the replica scripts. This will avoid string replacements and unnecessary manipulations during the initial setup and the ongoing replications for the system’s lifecycle.

Each site will resolve the TNS alias with its appropriate connect string pointing to the local database only. When a new database connect string is added in primary’s data sources, the tnsnames.ora file needs to be updated again in the secondary with the appropriate new alias information.

For example, Connect string in data sources in the primary site:

jdbc:oracle:thin:@soaedg

Where, the tnsnames.ora file in the primary contains the following:


SOAEDG =
        (DESCRIPTION=
        (ADDRESS_LIST=
            (LOAD_BALANCE=ON)
            (ADDRESS=(PROTOCOL=TCP)(HOST=prmy-scan)(PORT=1521)))
            (CONNECT_DATA=(SERVICE_NAME=soaedg.example.com))
        )

Connect string in data sources in the secondary site:

jdbc:oracle:thin:@soaedg

Where, the tnsnames.ora file in the secondary contains the following:


SOAEDG =
        (DESCRIPTION=
        (ADDRESS_LIST=
            (LOAD_BALANCE=ON)
            (ADDRESS=(PROTOCOL=TCP)(HOST=stby-scan)(PORT=1521)))             
			(CONNECT_DATA=(SERVICE_NAME=soaedg.example.com))
        )

The advantage of this tnsnames approach is that since the same DB connect string is used in the WebLogic domain config, there is no need to alter the WebLogic configuration after replicating the config from primary to standby. Since each site points to the local database only, there is no risk of cross connections from the mid-tier to the remote database.

Before proceeding with the initial replication and setup of the disaster protection system, ensure that all the different data sources used by the Fusion Middlware system have been configured with a TNS alias. This simplifies configuration management and optimizies your system’s RTO.

Preparing the Secondary System

This section provides information on how to set up the secondary system.

Preparing Network Components in the Secondary Site

When you plan your disaster recovery solution, you need to allocate precise hostnames in the secondary, configure a load balancer with the same virtual servers as the primary, and prepare external clients to access the system in the event of a disaster.

Preparing Host Names

In the secondary location, the host name addresses used by the Fusion Middleware components must be resolvable to valid IP addresses in the secondary site.

Use your preferred host name resolution mechanism to create this mappings as described in the Host Name Resolution section.

As prescribed in the EDG, Oracle recommends creating aliases for physical host names to isolate the real physical node names (which are different in each site) from the host names used by the Fusion Middleware components (which are the same regardless of the site).

This section describes how to plan physical host names and alias host names for the middle-tier hosts that use the Oracle Fusion Middleware instances at the secondary site for an existing primary site. It uses the Oracle SOA Suite Enterprise Deployment shown in Figure 2-1 for the host name examples. Each host at the production site has a peer host at the secondary site. The peer WebLogic servers and processes use the same ports as their counterparts on the other site.

The following section shows how to set up host names at the secondary sites to map the primary system’s configuration.

Note:

In the examples listed, IP addresses for hosts at the initial production site have the format 172.11.x.x and IP addresses for hosts at the initial secondary site have the format 172.22.x.x.
Sample Host Names for the Oracle SOA Suite Production and Secondary Site Listen Addresses

Learn about the Oracle SOA Suite secondary sites host name mappings.

Table 2-9 lists the IP addresses, physical host names, and aliases that are used for the Oracle SOA Suite Enterprise Deployment Guide (EDG) deployment production site hosts. The WebLogic Server’s listen address and channels (if any) in the primary should use the hostname alias and not the IP address nor the physical host name.

Table 2-9 IP Addresses and Physical Host Names for SOA Suite Production Site Hosts

IP Address Physical Host Name Host Name Alias

172.11.2.111

prweb1.example.com

WEBHOST1

172.11.2.112

prweb2.example.com

WEBHOST2

172.11.2.113

prsoa1.example.com

SOAHOST1

172.11.2.114

prsoa2.example.com

SOAHOST2

Table 2-10 lists the IP addresses, physical host names, and aliases that are used for the Oracle SOA Suite Enterprise Deployment Guide (EDG) deployment secondary site hosts.

Table 2-10 IP Addresses and Physical Host Names for SOA Suite Secondary Site Hosts

IP Address Physical Host Name Host Name Alias

172.22.2.111

scdryweb1.example.com

WEBHOST1

172.22.2.112

scdryweb2.example.com

WEBHOST2

172.22.2.113

scdrysoa1.example.com

SOAHOST1

172.22.2.114

scdrysoa2.example.com

SOAHOST2

Note:

If you use separate DNS servers in the primary and the secondary to resolve host names, then you can use the same physical host names for the production site hosts and the secondary site hosts, and you do not need to define the alias host names on the secondary site hosts. For more information about using separate DNS servers to resolve host names, see Resolving Host Names Using Separate DNS Servers.

However, it is recommended to use different hostnames and same aliases to isolate the real physical host names (which are different in each site) from the host names used by the Fusion Middleware components (which are the same regardless the site).

Figure 2-12 shows the physical host names that are used for the Oracle SOA Suite EDG deployment at the secondary site.

Figure 2-12 Physical Host Names Used at Oracle SOA Suite Deployment Standby Secondary Site


Physical Host Names Used at Oracle SOA Suite Deployment Standby Secondary Site

Preparing the Required Virtual IPs

Besides the regular hostname listen addresses and as described in the Virtual IP Considerations section, additional virtual hostnames may be required for those components not supporting WebLogic Server migration. In the case of Oracle SOA Suite, only the WebLogic administration server requires this virtual hostname and mapping IP.

Table 2-11 shows the virtual IP addresses and virtual host names that were used for the Oracle SOA Suite EDG deployment production site hosts.

Table 2-11 Virtual IP Addresses and Virtual Host Names for the SOA Suite Production Site Hosts

Virtual IP Address Virtual Host Name Alias Host Name

172.11.2.134

prsoa-vip.example.com

ADMINVHN

Table 2-12 shows the virtual IP addresses, virtual hostnames, and alias hostnames that are used for the Oracle SOA Suite EDG deployment secondary site hosts. Figure 2-2 shows the physical host names that are used for the Oracle SOA Suite EDG deployment at the secondary site. The alias host names shown in Table 2-12 should be defined for the SOA Oracle Suite secondary site hosts, as shown in Figure 2-2.

Note:

If you use separate DNS servers to resolve host names, then you can use the same virtual IP addresses and virtual host names for the production site hosts and the secondary site hosts, and you do not need to define the alias host names.

For more information about using separate DNS servers to resolve host names, see Resolving Host Names Using Separate DNS Servers.

Table 2-12 Virtual IP Addresses, Virtual Host Names, and Alias Host Names for SOA Suite Secondary Site Hosts

Virtual IP Address Virtual Host Name Host Name Alias

172.22.2.134

stbysoa-vip.example.com

ADMINVHN

Note:

The subnets in the production site and the secondary site are different.

Note:

If you use separate DNS servers to resolve host names, then you can use the same host names for the production site hosts and the secondary site hosts, and you do not need to define the alias host names.
Preparing Load Balancer in the Secondary Site

In a disaster recovery topology, both the primary and the secondary systems use a hardware load balancer that acts as front-end for all HTTP/HTTPs requests with equivalent configuration. Each load balancer will balance the traffic between the servers of its local site.

The secondary load balancer must support the required features to act as front-end. For more information, see Hardware Load Balancer Requirements in Enterprise Deployment Guide for Oracle SOA Suite.

The virtual front-end name used by the production and the secondary load balancer must be the same. That virtual front-end name must be resolved in DNS with the IP of the load balancer of the site that has primary role in each moment.

You must configure the virtual servers and the associated ports on the load balancer for the different types of network traffic and monitoring as they were configured in the primary.

Configure the virtual servers’ backend pools using the secondary’s Oracle HTTP Servers listen address and using the same ports for the listeners. Also, just as in the primary, the secondary load balancer should be configured to monitor for availability so that the traffic to these is stopped as soon as possible when a service is down. This ensures that incoming traffic on a given virtual host is not directed to an unavailable service in the other tiers.

As prescribed in the EDG, Oracle recommends that you use two load balancers (or at least separate listeners) when you deal with external and internal traffic. In such a topology, one load balancer or listener is set up for external HTTP traffic and the other load balancer is set up for internal traffic. In all cases, the load balancer in secondary should be a replica of the one in the primary and it is highly recommended that it should be configured in fault tolerant mode (for more details, refer to your load balancer vendor).

Some of the virtual servers defined in the load balancer are used for inter-component communication. These virtual servers are used for internal traffic and are defined in the internal DNS of a company. When you use a single global DNS server to resolve host names, Oracle recommends you create aliases for these virtual servers. Creating aliases is not required when you use separate DNS servers to resolve host names.

The virtual servers required for the various Oracle Fusion Middleware products are described in the related EDG for each product.

Note:

A hardware load balancer is assumed to server as a front-end for each site. For more information about supported load balancers, see My Oracle Support.

Preparing File Systems in the Secondary Site

File System, Directories, and Volumes

Verify if the storage in the secondary nodes are configured according to the recommendations in the EDG replicating the one used in the primary system. The EDG provides specific details about the storage volumes and mount points that need to be created for an enterprise topology. The same volumes and mount points that are used in the primary need to be configured in the secondary or else additional manipulation of the system’s configuration will be needed on the secondary. This will increase the management overhead and also increase the RTO of your disaster protection solution. Create private and shared directories that exactly reflects the same structure in the secondary as in the primary.

Setting Up the Secondary System

After the primary and secondary systems are prepared based on the information provided in the previous sections at the network (host name resolution), storage, and database connections level, the following steps are required to set up the secondary system.

Configuring Oracle Data Guard for the Fusion Middleware Database

When you plan your disaster recovery solution, you need to synchronize the databases in your system with the Oracle Data Guard. This is the only Maximum Availability approach recommended for Fusion Middleware systems.

This section provides the recommendations and considerations to set up Oracle databases that are used in an Oracle Fusion Middleware Disaster Recovery topology.

  • Oracle recommends that you create Oracle Real Application Cluster (Oracle RAC) databases on both the production and secondary site as described in the Enterprise Deployment Guide.

  • Oracle Data Guard is the recommended disaster protection technology for the databases running the Fusion Middleware persistent stores and Fusion Middleware metadata repositories.

  • Ensure that your network is configured for low latency with sufficient bandwidth because synchronous redo transmission can affect the response time and throughput.

  • Oracle Data Guard provides three protection modes: Maximum Availability, Maximum Performance (default), and Maximum Protection. Oracle recommends using the protection mode that meets your availability, performance, and data protection requirements. For more information, see Oracle Data Guard Protection Modes in the Oracle Data Guard Administration documentation.

  • During normal operation the secondary site database should be placed in managed recovery mode. This ensures that the secondary site’s database is in a constant state of media recovery. Managed recovery mode is enabled for shorter failover times.

  • The tnsnames.ora file on the production and secondary site database nodes must have entries for databases on both the production and secondary sites.

    Note:

    This is not the case for the tnsnames.ora files in the middle-tiers.

  • Oracle recommends using Automatic Storage Management (ASM) as the volume manager for Oracle database files. ASM is an Oracle recommended storage management solution that provides an alternative to conventional volume managers, file systems, and raw device. It supports single instance Oracle Database and Oracle Real Application Clusters (Oracle RAC) configurations.

    For more information about Automatic Storage Management (ASM) and Real Application Clusters (RAC), see Oracle Automatic Storage Management and Real Application Clusters Installation Guide for Linux and UNIX.

  • The SERVICE_NAME used by applications in the primary should also be the same in the secondary database . However, each database can have additional services defined for other purposes. For example, specific read-only services can be created on the secondary to offload reporting and business intelligence operations to the secondary site.

Prerequisites and Assumptions

Ensure that the following prerequisites are met:

  • The Oracle RAC cluster and Automatic Storage Management (ASM) instances on the secondary site have been created.

  • The Oracle RAC databases on the secondary site and the production site are using a flash recovery area.

  • The Oracle RAC databases are running in archivelog mode.

  • The database hosts on the secondary site already have Oracle software installed.

  • In a shared ORACLE_HOME configuration, the TNS_ADMIN directory must be a local, non-shared directory.

Oracle Data Guard Environment Description

The examples given in this section contain environment variables as described in Table 2-13.

Table 2-13 Variables Used by Primary and Standby Databases

Variable Primary Database Standby Database

Database names

soa

soa

SOA database host names

dbhost1.example.com, dbhost2.example.com

dbhost1stby.example.com, dbhost2stby.example.com

Scan address

prmy-scan.example.com

stby-scan.example.com

Database unique names

soa_pri

soa_stby

Instance names

soa1, soa2

soa1, soa2

Service names

soa_pri.example.com , soaedg.example.com

soa_stby.example.com , soaedg.example.com

Procedure for Configuring the Data Guard

The configuration of a Data Guard involves several steps: you need to prepare the primary database with the recommended parameters, prepare the TNS aliases in primary and standby environments, create the physical standby database as a duplication of the primary database, configure the Data Guard Broker, and so on. You can use one of these methods to automate most of these actions:

  • Oracle provides a set of sample scripts available in the MAA repository at GitHub, in dg_setup_scripts. These scripts are designed to setup a standby database for an existing primary database, using the restore from service feature and Data Guard Broker. They allow customization (OS user names and Oracle Homes are configurable values). They support databases with and without TDE encryption, and work in environments with Read Only Oracle Home (ROOH) or with regular Oracle Homes. The scripts are validated in 12c (12.2), 18c, 19c, 21c, and 23ai RDBMS versions. Download the file and follow the instructions included in the README.md to configure a standby database.

  • If both primary and standby databases are in Oracle Cloud Infrastructure, use the Data Guard configuration option in OCI Console. This skips most of the complexity in the setup operation. For more information, see Enable Oracle Data Guard on a DB System in Oracle Base Database Service.

  • Using "Enterprise Manager Cloud Control". Setting up and managing databases using Cloud Control helps in controlling downtime and simplifies disaster recovery operations. For more information, see Provisioning Oracle Standby Databases in Oracle Enterprise Manager Lifecycle Management Administrator's Guide.

  • If none of the above methods are feasible for your specific environment, you can manually configure the Data Guard by following one of these documents:

Verifying the Data Guard Broker Configuration

Complete the following steps to verify that the Data Guard Broker configuration was created successfully.

  1. Verify the Oracle Data Guard configuration by querying the V$ARCHIVED_LOG view to identify existing files in the archived redo log. For example:
    SQL> SELECT SEQUENCE#, FIRST_TIME, NEXT_TIME 
      2> FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;
    
    SEQUENCE# FIRST_TIME NEXT_TIME
    ---------- ------------------ ------------------
             8 11-DEC-21 17:50:45 11-DEC-21 17:50:53
             9 11-DEC-21 17:50:53 11-DEC-21 17:50:58
             10 11-DEC-21 17:50:58 11-DEC-21 17:51:03
    3 rows selected
  2. On the primary database, issue the following SQL statement to force a log switch and archive the current online redo log file group:
    SQL> alter system archive log current;
    
  3. On the standby database, query the V$ARCHIVED_LOG view to verify that the redo data was received and archived on the standby database:
    SQL> SELECT SEQUENCE#, FIRST_TIME, NEXT_TIME 
      2> FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;
    
    SEQUENCE# FIRST_TIME NEXT_TIME
    ---------- ------------------ ------------------
            8 11-DEC-21 17:50:45 11-DEC-21 17:50:53
            9 11-DEC-21 17:50:53 11-DEC-21 17:50:58
           10 11-DEC-21 17:50:58 11-DEC-21 17:51:03
           11 11-DEC-21 17:51:03 11-DEC-21 18:34:11
    
    4 rows selected
    
  4. Use the show configuration command in Data Guard Broker command line to verify that the configuration was created successfully. See Example 2-11.

Example 2-11 Verifying the Data Guard Broker Configuration

[oracle@dbhost1 ~]$ dgmgrl sys/'<password>'
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Tue Feb 1 09:00:16 2022
Version 19.6.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.
Welcome to DGMGRL, type "help" for information.
Connected to "soa_pri"
Connected as SYSDBA.

DGMGRL> show configuration
Configuration - dg_config

Protection Mode: MaxPerformance
Databases:
soa_pri - Primary database
soa_stby - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS
Testing Database Switchover and Switchback
You can perform a database switchover and switchback.

Performing a Switchover Operation by Using Oracle Data Guard Broker

To perform a switchover operation by using Oracle Data Guard Broker, complete the following tasks:

  1. Verify the Oracle Data Guard Broker configuration by running the following command:

    DGMGRL> show configuration;
     
    Configuration - dg_config
     
    Protection Mode: MaxPerformance
    Databases:
    soa_pri  - Primary database
    soa_stby - Physical standby database
     
    Fast-Start Failover: DISABLED
     
    Configuration Status:
    SUCCESS
    
  2. Swap the roles of the primary and standby databases by running the SWITCHOVER command. Example 2-11 shows how Data Guard Broker automatically shuts down and restarts the old primary database as part of the switchover operation.

    DGMGRL> switchover to 'soa_stby'
    Performing switchover NOW, please wait...
    Operation requires a connection to database "soa_stby"
    Connecting ...
    Connected to "soa_stby"
    Connected as SYSDBA.
    New primary database "soa_stby" is opening...
    Oracle Clusterware is restarting database "soa_pri" ...
    Connected to "soa_pri"
    Connected to "soa_pri"
    Switchover succeeded, new primary is "soa_stby"
  3. After the switchover is complete, use the SHOW CONFIGURATION command to verify that the switchover operation was successful:

    DGMGRL> show configuration;
    Configuration - dg_config
    Protection Mode: MaxPerformance
    Databases:
    soa_stby - Primary database
    soa_pri  - Physical standby database
    Fast-Start Failover: DISABLED
    Configuration Status:
    SUCCESS
    

Note:

For information about switchover and failover operation of Oracle Data Guard Broker, see Switchover and Failover Operations in the Oracle Data Guard Broker.

Replicating the Primary File Systems to the Secondary Site

Storage Replication Approach

Using storage replication process to get a copy of the primary in the secondary involves activating the snapshots that have been created from the primary volumes. Some vendors provide a multiple step process where a snapshot is cloned and then activated so that it can be mounted by the secondary nodes. Refer to your precise storage vendor details to activate, attach, and mount the pertaining shares or volume to the secondary nodes. Each node in the secondary should mount the exact same volumes using the exact same paths as its peer in the primary.

Rsync Replication Approach

In the peer-to-peer model, rsync script should be executed in each secondary node to pull the content from each respective peer in primary. In the stage rsync approach, the scripts to transfer from the staging location to each individual node should be executed to a get a copy of Oracle Homes, WebLogic Domain configuration, and runtime artifacts if required. You can also use the scripts at GitHub as an example to replicate to the secondary system both in peer-to-peer and using staging location.

The script rsync_for_WLS.sh is a wrapper that invokes rsync_copy_and_validate.sh. This second script contains the real logic to perform rsync copies with the recommended rsync configuration and executes a thorough validation of files after the copy is completed. If any differences are detected after several validation retries, these are logged so that they can be acted upon.

By default, the rsync_copy_and_validate.sh uses oracle as the user to perform the copies. If a different user owns the origin folders, customize the property USER in the script .

It also uses the environment variable DEST_FOLDER to determine the target location that will be used to rsync contents. If the variable is not set, the script copies contents to the node executing the script to the exact same directory path used in the source node. While using the staging approach set the variable to the desired path (for example , for the first WLS node private configuration) /staging_share_storage/midtier/wls_private_config/wlsnode1_private_config.

Using Peer-to-Peer

For each host on the secondary site, set up rsync to copy the following directories and files to the same directories and files from each primary peer:

  1. Oracle Home directory and subdirectories.

  2. Oracle Central Inventory directory which includes the Oracle Universal Installer entries for the Oracle Fusion Middleware installations.

  3. Shared config folder such as the WebLogic Administration Server domain directory, deployment plans, applications, keystores, and so on. Since this folder is shared by all the mid-tier hosts in the EDG, you do not need a copy for each host. Do not replicate the shared WebLogic domain directory only. In context of the EDG, there are dependencies in the /u01/oracle/config/applications and /u01/oracle/config/keystores directories that need to be copied with the WebLogic domain configuration.

  4. Private config folders such as the WebLogic managed server's domain and local node manager home on the host.

  5. Shared runtime folder (if applicable).

  6. Oracle Fusion Middleware static HTML pages directory for the Oracle HTTP Server installations on the host (if applicable).

  • Replicate the entire primary Administration server node first. Execute the script for the products folder, the shared configuration folder, and the private domain configuration folder (do not replicate the domain directory for the shared configuration only) as follows. Using the examples provided in this book, 172.11.2.113 is the IP of the WLS administration server in primary, and 172.22.2.113 is the IP of the WLS administration server in secondary:

    
    [oracle@172.22.2.113]$./rsync_for_WLS.sh 172.11.2.113 /u01/oracle/products /home/oracle/keys/SSH_KEY.priv
    [oracle@172.22.2.113]$./rsync_for_WLS.sh 172.11.2.113 /u01/oracle/config /home/oracle/keys/SSH_KEY.priv
    [oracle@172.22.2.113]$./rsync_for_WLS.sh 172.11.2.113 /u02/oracle/config /home/oracle/keys/SSH_KEY.priv
  • Replicate the second WLS server node. Execute the script for the products folder and the private domain configuration folder as follows. Using the examples provided in this book, 172.11.2.114 is the IP of the WLS server node 2 in primary, and 172.22.2.114 is the IP of the WLS server node 2 in secondary:

    
    [oracle@172.22.2.114]$./rsync_for_WLS.sh 172.11.2.114 /u01/oracle/products /home/oracle/keys/SSH_KEY.priv
    [oracle@172.22.2.114]$./rsync_for_WLS.sh 172.11.2.114 /u02/oracle/config /home/oracle/keys/SSH_KEY.priv
  • Replicate the two OHS nodes. Since OHS instances run in standalone mode, there is no shared configuration folder, and only the private domain configuration needs to be replicated. Using the examples provided in this book, 172.11.2.111 is the IP of the OHS server node 1 in primary, and 172.22.2.111 is the IP of the OHS server node 1 in secondary:

    
    [oracle@172.22.2.111]$./rsync_for_WLS.sh 172.11.2.111 /u02/oracle/products /home/oracle/keys/SSH_KEY.priv
    [oracle@172.22.2.111]$./rsync_for_WLS.sh 172.11.2.111 /u02/oracle/config /home/oracle/keys/SSH_KEY.priv

After this, you should have a complete copy of all the primary OHS and WLS nodes in each peer node in secondary.

Using a Staging Location

  • If you are copying from a staging location on primary to a staging location in secondary, you do not need to set the DEST_FOLDER environment variable and you must replicate the same paths in the staging location in primary to the secondary staging location. For example:

    [oracle@staging_node_secondary]$./rsync_for_WLS.sh staging_node_primary_ip /staging_share_storage/midtier/wls_products_home/wls_products_home1 /home/oracle/keys/SSH_KEY.priv

    After you have the copy available in the secondary staging location, you must pull the information from the secondary nodes. For example (using the examples provided in this book, 172.22.2.113 is the IP of the WLS administration server in secondary):

    [oracle@172.22.2.113]$ export DEST_FOLDER=/u01/oracle/products
    [oracle@172.22.2.113]$./rsync_for_WLS.sh staging_node_secondary_ip /staging_share_storage/midtier/wls_products_home/wls_products_home1 /home/oracle/keys/SSH_KEY.priv
  • If you are copying directly to the secondary nodes from a single staging location, you must pull the information directly from the single staging location. For example (using the examples provided in this book, 172.22.2.113 is the IP of the WLS administration server in secondary):

    [oracle@172.22.2.113]$ export DEST_FOLDER=/u01/oracle/products
    [oracle@172.22.2.113]$./rsync_for_WLS.sh staging_node_ip /staging_share_storage/midtier/wls_products_home/wls_products_home1 /home/oracle/keys/SSH_KEY.priv
DBFS Replication Approach

The secondary system needs to receive the copy from the primary that can be mounted to the pertaining DBFS directory. To mount it in the secondary, use the same scripts and steps as described in Preparing the Primary Storage for DBFS. Here is an example to run the dbfs_dr_setup_root.sh in the secondary mid-tier hosts. You must provide secondary database services values:

./dbfs_dr_setup_root.sh drdbb-scan.wlsdrvcnfra1ad2.wlsdrvcnfra1.oraclevcn.com 1521 PDB1.wlsdrvcnfra1ad2.wlsdrvcnfra1.oraclevcn.com mypassword /u01/install/V982064-01.zip

Once the secondary’s DBFS mount is available, replicate the contents from it to each middle-tier node replicating the directory structure and contents as available in the primary peer.

Configuring Secondary Site’s Connect Strings with the Local Database

Whether you have replicated the tnsnames.ora from the primary or excluded it in the copy (either by placing the pertaining database client module outside the storage mounts that are replicated or explicitly excluding the file in rsync copies), you need to update the connect strings in the secondary location to point to the secondary database only. In remote disaster protection topologies, Oracle recommends configuring WebLogic data sources only with local database addresses (instead of using dual connect strings or replacing long connect strings directly in the different data sources). Update your tnsnames.ora file in the secondary with the service and scan the address used in the secondary database. With this configuration, the jdbc url information in the WebLogic data sources remains the same in the primary and the secondary (jdbc:oracle:thin:@soaedg) but the tnsnames.ora file in primary contains the following data:


SOAEDG =
        (DESCRIPTION=
        (ADDRESS_LIST=
            (LOAD_BALANCE=ON)
            (ADDRESS=(PROTOCOL=TCP)(HOST=prmy-scan)(PORT=1521)))
            (CONNECT_DATA=(SERVICE_NAME=soaedg.example.com))
        )

You need to update the secondary’s tnsnames.ora file with the following data:


SOAEDG =
        (DESCRIPTION=
        (ADDRESS_LIST=
            (LOAD_BALANCE=ON)
            (ADDRESS=(PROTOCOL=TCP)(HOST=stby-scan)(PORT=1521)))
                  (CONNECT_DATA=(SERVICE_NAME=soaedg.example.com))
         )

In this example, the service remains the same between the primary and the secondary but it could be different depending on the connection needs in each case. These updates should typically be done once in the initial setup. Only if additional services and databases are used by the primary system, additional alias would need to be replaced in the secondary.