1. Configure Oracle Fusion Middleware DR on Oracle Cloud with an autonomous database
  2. Configure the DR Topology

Configure the DR Topology

Set up the disaster recovery (DR) topology. Scripts are available to streamline the process.

Download the Scripts

Get the latest setup scripts from the GitHub repository.

Note:

Place all of the downloaded scripts in the same folder.
  1. Go to the GitHub repository.
  2. Download all of the scripts in the maa/fmw-wls-with-adb-dr directory.
  3. Download all of the scripts in the maa/app_dr_common directory.
    These scripts make calls to each other. Despite the specific operation being performed at a point in time, download the entire directories and place all of the scripts in the same folder. You will need the scripts in both the primary and secondary sites.
  4. Follow the instructions in this document and read the required variables in each script for each operation that you perform.

The downloaded file includes scripts to perform the following tasks:

  1. Configure a TNS Alias for datasources
  2. Set up the initial DR configuration
  3. Set up ongoing replication
  4. Change wallets for an Oracle WebLogic Server, Oracle SOA, or Oracle Fusion Middleware system.
Each script provides automation for different parts of the DR setup and lifecycle of a disaster protection system. The following table provides a summary of the utilities:
Script Name Description
fmwadb_config_replica.sh Replicate configuration between sites.
fmwadb_dr_prim.sh Prepares a primary site for the DR setup.
fmwadb_dr_stby.sh Prepares a secondary site for the DR setup.
fmwadb_rest_api_listabds.sh Obtain the Autonomous Database role base on the ADB ID and tenancy information.
fmwadb_switch_db_conn.sh Replaces the existing connect information with a new ADBS WALLET.
fmw_change_to_tns_alias.sh Replace the connect strings used by Oracle WebLogic datasources and jps config files with a tns alias.
fmw_dec_pwd.sh Decrypts a Oracle WebLogic-encrypted password.
fmw_enc_pwd.sh Encrypts a password using Oracle WebLogic encryption.
fmw_get_connect_string.sh Returns the connect string that a Oracle WebLogic, Oracle SOA, or Oracle Fusion Middleware datasource is using.
fmw_get_ds_property.sh Returns the value of a specific datasource property.

Prepare the Primary Mid-tier for the Virtual Front-end

If the primary mid-tier is not already configured with a virtual front-end name, then perform these actions to prepare it for the disaster recovery (DR) configuration.

  1. Add the virtual front-end name and IP to the /etc/hosts file in all primary mid-tier hosts.
    Each site should always resolve the front-end name to its local load balancer, regardless of client-facing resolution through the Domain Name System (DNS). As root user, edit the /etc/hosts file and map the primary load balancer public IP to the virtual front-end fully qualified domain name (FQDN). Repeat in all primary Oracle WebLogic hosts. For example: 
    [oracle@wlsociprefix-wls-0 ~]$ more /etc/hosts
(...)
# Front-end virtual name
111.111.111.111 mywebapps.example.com

    Note:

    The /etc/hosts file of the primary Oracle WebLogic hosts must not be altered when there is a switchover or failover. Primary Oracle WebLogic hosts will always resolve the virtual front-end name with its front-end IP. The DNS update that is needed during the switchover and failover procedures is performed in the DNS or host files used by the clients.

  2. Configure the front-end name as cluster front-end.
    1. Log into the Oracle WebLogic Console of your instance.
    2. Navigate to Environment, then Clusters, and then select the cluster.
    3. Go to Configuration, then HTTP.
    4. Set the Fronted host to the front-end FQDN.
      For example, mywebapps.example.com.
    5. Ensure that the Frontend ports for HTTP and HTTPS are correctly configured with values.
    6. Click Save, then click Activate.
  3. Restart the cluster to implement the changes.

Modify the Primary's Data Sources and JPS Configuration to Use a TNS Alias

Using a Transparent Network Substrate (TNS) alias in Java Database Connectivity (JDBC) URLs facilitates the reconfiguration of Oracle WebLogic Server for Oracle Cloud Infrastructure data sources by using remote refreshable clones to move between primary and standby.

Note:

Oracle SOA Suite on Marketplace instances provisioned with release 23.1.1 (February 2023) or later are configured with the TNS alias approach out-of-the-box. For this case, you can skip this task.

Using a TNS alias requires that the datasources and jps files include the variable oracle.net.tns_admin in the Oracle Fusion Middleware configuration files.

  1. Use the command grep to search for the oracle.net.tns_admin variable in the Oracle Fusion Middleware configuration file. 
    [oracle@soarefr-soa-0 ~]$ grep oracle.net.tns_admin ${DOMAIN_HOME}/config/fmwconfig/jps-config.xml 
<property name="oracle.net.tns_admin" value="/u01/data/domains/soarefr_domain/config/atp"/>

[oracle@soarefr-soa-0 ~]$ grep oracle.net.tns ${DOMAIN_HOME}/config/jdbc/opss-datasource-jdbc.xml  -A1 | grep value 
<value>/u01/data/domains/soarefr_domain/config/atp</value>
[oracle@soarefr-soa-0 ~]$
    The directory reflected in jps-config.xml (and datasources) must be available and accessible from all of the nodes in the WebLogic Server domain.
    • In Oracle SOA Suite on Marketplace this directory is under the $DOMAIN_HOME/config/atp directory (before release 23.1.1) or $DOMAIN_HOME/config/tnsadmin directory (for release 23.1.1 and later) and is automatically replicated by the WebLogic Server to all other nodes when the managed servers are started.

      Note:

      If your primary Oracle SOA Suite on Marketplace is before 23.1.1, it is recommended to move the folder to the $DOMAIN_HOME/config/tnsadmin directory to make it consistent with the standby.
    • In Oracle WebLogic Server for Oracle Cloud Infrastructure this is located under $DOMAIN_HOME/atpwallet.

      Note:

      If the wallet is not located under the DOMAIN_HOME/config directory, then changes to the contents of the wallet directory will NOT be replicated by the Oracle WebLogic Server infrastructure to other nodes automatically. In these cases, you must either change the wallet directory (and update the required datasources configurations), or manually copy it to other nodes when it gets updated.
    • In other configurations, you can place the directory on shared storage.
    In all cases, the same tns_admin directory must be reachable by all of the different members of a WebLogic Server domain. This directory will contain a tnsnames.ora file with different aliases for the different services created in the Oracle Autonomous Database.

    The following is a sample tnsnames.ora file for and Oracle Fusion Middleware configuration with Oracle Autonomous Database Serverless. 

    [oracle@soarefr-soa-0 ~]$ cat 
$DOMAIN_HOME}}/config/tnsadmin/tnsnames.ora
soaadb1_high = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbvcn.adb.ap-mumbai-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_high.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
 
soaadb1_low = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbvcn.adb.ap-mumbai-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_low.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
 
soaadb1_medium = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbvcn.adb.ap-mumbai-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_medium.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
 
soaadb1_tp = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbvcn.adb.ap-mumbai-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_tp.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
 
soaadb1_tpurgent = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbvcn.adb.ap-mumbai-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_tpurgent.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))

    Modifications to the tnsnames.ora file are dynamically consumed by WLS data sources. This means that you can alter the tnsnames.ora file (for example to point to a different service) without requiring the restart of the datasource.

  2. To modify a standard datasource configuration to use a TNS alias in a Oracle SOA Suite on Marketplace or an Oracle WebLogic Server for Oracle Cloud Infrastructure system that uses Autonomous Database, use the fmw_change_to_tns_alias.sh script.

    The alias should be used in all datasource files under $DOMAIN_HOME/config/jdbc and in the jps config files under $DOMAIN_HOME/config/fmwconfig.

    Note:

    This script will only change the connect string and tns_admin property. If you add other custom datasources, then those will also need to use a TNS alias just like the internal ones for Oracle WebLogic, Oracle SOA Suite on Marketplace, and Oracle Fusion Middleware.
    When Oracle WebLogic Server for Oracle Cloud Infrastructure or Oracle SOA Suite on Marketplace are provisioned using Oracle Autonomous Database, the database wallet's configuration is included in the datasources (trust store location, keystore location, and so on). These parameters are NOT modified by the fmw_change_to_tns_alias.sh script and are valid whether using TNS alias or not. A wallet directory is created by Oracle WebLogic Server for Oracle Cloud Infrastructure and Oracle SOA Suite on Marketplace during provisioning and already contains a tnsnames.ora file. You can also get the Oracle Autonomous Database wallet from the autonomous database UI in OCI.

    Note:

    The Oracle Autonomous Database wallet used by Oracle WebLogic Server for Oracle Cloud Infrastructure and Oracle SOA Suite on Marketplace is downloaded to the Administration server node when they are provisioned. You can update the wallet for the primary and standby systems by re-downloading it from the Oracle Autonomous Database UI and updating the wallet password in the $DOMAIN_HOME/config/jdbc directory and in the jps config files under $DOMAIN_HOME/config/fmwconfig. Using the same password for primary's, standby's and the refreshable clone's wallets will facilitate the manipulation of datasource configuration in both sites; however the scripts provided below will be able to handle different passwords. Use the fmwadb_switch_db_conn.sh script to update the system with a new wallet.

    When your primary system was provisioned, you chose (in the provisioning screens) one of the Oracle Autonomous Database service levels. Run the fmw_change_to_tns_alias.sh script with the same service level. The service levels for a DB service in Oracle Autonomous Database are: low, mid, high tp, tpurgent. The following is an example of changing Oracle Fusion Middleware to TNS alias: 

    [oracle@soarefr-soa-0 ~]$ grep url /u01/data/domains/soarefr_domain/config/fmwconfig/jps-config.xml
<property name="jdbc.url" value="jdbc:oracle:thin:@(description=(retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbvcn.adb.ap-mumbai-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_low.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))"/>
[oracle@soarefr-soa-0 ~]$ grep url /u01/data/domains/soarefr_domain/config/jdbc/opss-datasource-jdbc.xml
<url>jdbc:oracle:thin:@(description=(retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbvcn.adb.ap-mumbai-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_low.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))</url>
 
NOTICE the "low" label in g914a2540e8ab6d_soaadb1_low.adb.oraclecloud.com, that is the service flag that will allow you identify the tns alias that needs to be used
 
[oracle@soarefr-soa-0 good]$ cat $DOMAIN_HOME/config/atp/tnsnames.ora | grep low | awk -F '=' '{print $1}'
soaadb1_low
 
[oracle@soarefr-soa-0 good]$ ./fmw_change_to_tns_alias.sh soaadb1_low                                                                                          
Getting variables from current datasource ...............
An existing tns_admin property was found in /u01/data/domains/soarefr_domain/config/jdbc/opss-datasource-jdbc.xml and will be used
Found soaadb1_low  as tns_alias
No modifications will be required in /u01/data/domains/soarefr_domain/config/atp/tnsnames.ora
*******************WILL USE THESE SETTINGS********************
**************************************************************
TNS admin:........................./u01/data/domains/soarefr_domain/config/atp
TNS alias:........................ soaadb1_low
Current connect string:............(description=(retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=rcsoaadb1.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_rcsoaadb1_low.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
Current jps connect string:........(description=(retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=rcsoaadb1.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_rcsoaadb1_low.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
Current tnsnames.ora:..............
soaadb1_high = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=rcsoaadb1.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_rcsoaadb1_high.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
soaadb1_low = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=rcsoaadb1.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_rcsoaadb1_low.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
soaadb1_medium = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=rcsoaadb1.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_rcsoaadb1_medium.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
soaadb1_tp = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=rcsoaadb1.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_rcsoaadb1_tp.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
soaadb1_tpurgent = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=rcsoaadb1.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_rcsoaadb1_tpurgent.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
**************************************************************
Taking backup of existing config...
************** Replacing DB connect information **************
Replacing jdbc url in config/jdbc files...
Replacing jdbc url in config/fmwconfig files...
Replacement complete!
 
[oracle@soarefr-soa-0 ~]$ grep url $DOMAIN_HOME/config/fmwconfig/jps-config.xml
<property name="jdbc.url" value="jdbc:oracle:thin:@soaadb1_low "/>
[oracle@soarefr-soa-0 ~]$ grep url /u01/data/domains/soarefr_domain/config/jdbc/opss-datasource-jdbc.xml
    <url>jdbc:oracle:thin:@soaadb1_low </url>
<property name="jdbc.url" value="jdbc:oracle:thin:@soaadb1_low "/>
[oracle@soarefr-soa-0 ~]$
  3. Restart all Oracle WebLogic Servers in the domain to consume the changes performed by the script.

Create a VCN and Subnet in the Secondary Region

If you haven’t already done it, create a VCN in the standby region with a CIDR that doesn't conflict with the primary region's CIDR. For example, if the primary VCN uses 10.1.0.0/16, then the secondary VCN could use 10.2.0.0/16.

  1. Create a VCN and subnet in the secondary region.
    You can create a cloud stack to provision a group of related cloud services.
  2. Ensure that the security lists allow the appropriate communications between tiers.

    Verify the following communication:

    • From the OCI load balancer to the WebLogic Server
    • From the WebLogic Server to the database
    • From the WebLogic Server to the shared storage

    Also, include the required rules to allow communications between administration server nodes through the Dynamic Routing Gateway (DRG) once it is created.

Configure a DRG Between the Primary and Secondary VCNs

The disaster recovery set up requires that the primary and secondary Oracle WebLogic Server Administration Nodes communicate with each other to receive domain configuration through Oracle Cloud Infrastructure File Storage copies. For this, you must create a dynamic routing gateway (DRG) between the mid-tier VCNs.

  1. Create a DRG between the mid-tier VCNs.
  2. Verify that the new DRG appears on the Dynamic Routing Gateways page of the compartment or region that you chose.
    The DR set up scripts will verify that the required connections can be established.
  3. Configure the appropriate routes and security rules in the primary and standby VCNs to allow SSH connections between the midtiers.
  4. Verify that you can establish an SSH connection from the Oracle WebLogic Administration Server node in primary to the Oracle WebLogic Administration Server node in secondary.
    For example, if 10.2.1.104 is the secondary's Administration Server node's IP, then run the following from the primary's Administration Server node.
    [opc@soarefr-soa-0 ~]$ ssh -i KeySOAMAA.ppk opc@10.2.1.104
Last login: Wed Dec 14 16:59:15 2022 from 10.2.0.83
[opc@soarefr-soa-0 ~]$

Create an Oracle Autonomous Data Guard Standby in the Secondary Region

Create a standby database for the existing primary Oracle Autonomous Database.

  1. For Oracle Autonomous Database Serverless, create a standby Oracle Autonomous Database Serverless in the secondary region.
    1. In the OCI Console, click Oracle Database in the left navigation menu to navigate to the primary Oracle Autonomous Database.
    2. On the Autonomous Database Details page, under Resources, click Disaster Recovery, then click Add peer database.
    3. Use the VCN and private subnets that you created earlier.
  2. For Oracle Autonomous Database on Dedicated Exadata Infrastructure
    1. In the OCI Console, navigate to your Autonomous Container Database details page and select the Autonomous Data Guard associations resource in the Resources section.
    2. Click Enable Autonomous Data Guard.
    3. Enter the peer Autonomous VM Cluster information, protection mode, and automatic failover choice. After entering all the necessary information, click Enable Autonomous Data Guard.
      As part of this workflow, a new standby Autonomous Container Database is created with all of the standby autonomous databases in the selected Autonomous VM Cluster.

Prepare the Standby Autonomous Database for the DR Setup

This task depends on whether you are using the Snapshot Standby or Remote Refreshable Clone approach.

For the Snapshot Standby approach, see Convert the Standby into a Snapshot Standby.

For the Remote Refreshable Clone approach, see Create a Remote Refreshable Clone in the Secondary Region.

Convert the Standby into a Snapshot Standby

Using the Snapshot Standby approach, convert your standby autonomous database into Snapshot Standby.

  1. From the Oracle Cloud Infrastructure Console left navigation menu, click Autonomous Database.
  2. In secondary region, select the standby Autonomous Database.
  3. From the More Actions drop-down list, click Convert to snapshot standby database.
  4. If you're using Dedicated Infrastructure, select Use primary database services.

Note:

When a Snapshot Standby in Oracle Dedicated Exadata Infrastructure is not converted to physical standby within 7 days, the Snapshot Standby is automatically converted to physical standby.

When a Snapshot Standby in Oracle Autonomous Database Serverless is not converted to physical standby within 2 days, the Snapshot Standby is automatically converted to physical standby.

Create a Remote Refreshable Clone in the Secondary Region

Using the Remote Refreshable Clone approach, create a refreshable clone from the existing primary Oracle Autonomous Database Serverless in the remote region.

  1. Create a refreshable clone from the existing primary Oracle Autonomous Database Serverless.
    Place the refreshable clone in the secondary (remote) region inside the VCN and the private subnet that you created earlier.

    Note:

    You cannot use the same DB name as the primary's since that name is already in use by the Physical Standby.
  2. Use Private Endpoint Access Only to access the database.

    Once created, the refreshable clone will remain connected and in read-only mode. Oracle WebLogic Server, Oracle SOA, and Oracle Fusion Middleware systems cannot be provisioned against it UNLESS it is converted to read-write (disconnected from the primary).

  3. Disconnect the refreshable clone from the primary autonomous database and convert it to read/write mode.

    Note:

    The remote refreshable clone cannot remain disconnected for more than 24 hours. For example, you cannot take longer than 24 hours to create the secondary mid-tiers pointing to the remote refreshable clone database.
    1. From the Oracle Cloud Infrastructure left navigation menu, click Autonomous Database.
    2. Select the refreshable clone name.
    3. Select the source database, then click Disconnect.

Provision the Secondary System

Provision the secondary Oracle WebLogic Server for Oracle Cloud Infrastructure, Oracle SOA Suite on Marketplace, or other middle-tier Oracle Cloud Infrastructure (OCI) service that uses Oracle Fusion Middleware (system) pointing to the secondary database (for Snapshot Standby approach) or to the refreshable clone (for remote refreshable clone approach).

  1. Follow the standard subnet, CIDR, security rules and instance prefix recommendations for disaster recovery.

    The Stack Name can be different, but you must use the EXACT same resource name prefix that you used in your primary location. Oracle recommends that you use the exact same capacity and compute configuration on both primary and standby locations for the ideal failover/switchover behavior.

    Make sure that the Oracle WebLogic Server for OCI version and patch level to be provisioned in the secondary location matches the one running in the primary site.

    If you need more details, then see the table that summarizes the provisioning wizard options for the DR set up in the "Provision WLS for OCI in Secondary Site" section of Oracle WebLogic Server for Oracle Cloud Infrastructure Disaster Recovery or in the "Provision SOA Suite on Marketplace in Secondary Site" section of SOA Suite on Oracle Cloud Infrastructure Marketplace Disaster Recovery.

  2. Create the secondary middle tier system according to the standard provisioning process.
  3. Check the standard Oracle WebLogic Server for Oracle Cloud Infrastructure, Oracle SOA Suite on Marketplace, or any other middle-tier Oracle Cloud Infrastructure (OCI) service that uses Oracle Fusion Middleware URLs (for example, Console, soa-infra, and so on) to validate that the system is properly created.
  4. Once validated, stop the Oracle WebLogic processes in the standby: managed servers, admin server, and node managers.
  5. If you are using remote refreshable clone, then you can reconnect it to the source. If you're using snapshot standby, then you can convert it to physical standby again.
    Do not try to start the Oracle WebLogic processes in secondary until the DR setup is completed.

Modify the Secondary's TNS Alias Connect Strings

Modify the alias used in the secondary system to be the same alias as in the primary system.

Just as in the primary system, you should use a Transparent Network Substrate (TNS) alias in all datasource files under $DOMAIN_HOME/config/jdbc and in the jps config files under $DOMAIN_HOME/config/fmwconfig. The alias used in the secondary (standby) system will be the same one as in the primary system because datasources are replicated from the primary containing the alias created earlier.

This task depends on whether you are using a Snapshot Standby or Remote Refreshable Clone approach.

Modify the Secondary's TNS Alias Connect Strings for Snapshot Standby Approach

For the Snapshot Standby approach, it's expected that the aliases in the tnsnames.ora file are the same as in primary (although connect strings will be pointing to the standby database’s address). You don’t need to modify them.

  1. If the strings in the tnsnames.ora file are dual (where they contain both local and remote autonomous database hosts), then Oracle recommends that you modify them to point ONLY to the local database.
    This can happen when you are using Oracle Autonomous Database on Dedicated Infrastructure. You only need to make this change once, since the tnsnames.ora file for standby is not altered by configuration replication and other lifecycle operations.

    The following is an example where the entries in the tnsnames.ora file are dual: 

    adbd1_tp=(DESCRIPTION =(CONNECT_TIMEOUT = 90)(RETRY_COUNT = 50)(RETRY_DELAY = 3)(TRANSPORT_CONNECT_TIMEOUT = 3)(ADDRESS_LIST =(LOAD_BALANCE = ON)(ADDRESS = (PROTOCOL=TCP)(HOST= host-kyzm3-scan.secondarysubnet.secondaryvcn.oraclevcn.com)(PORT=1521)) (ADDRESS = (PROTOCOL=TCP)(HOST=host-xwgo9-scan.primarysubnet.primaryvcn.oraclevcn.com)(PORT=1521)) )(CONNECT_DATA = (SERVICE_NAME = ADBD1_tp.atp.oraclecloud.com)))

adbd1_medium=(DESCRIPTION =(CONNECT_TIMEOUT = 90)(RETRY_COUNT = 50)(RETRY_DELAY = 3)(TRANSPORT_CONNECT_TIMEOUT = 3)(ADDRESS_LIST =(LOAD_BALANCE = ON)(ADDRESS = (PROTOCOL=TCP)(HOST=host-kyzm3-scan.secondarysubnet.secondaryvcn.oraclevcn.com)(PORT=1521)) (ADDRESS = (PROTOCOL=TCP)(HOST=host-xwgo9-scan.primarysubnet.primaryvcn.oraclevcn.com)(PORT=1521)) )(CONNECT_DATA = (SERVICE_NAME = ADBD1_medium.atp.oraclecloud.com)))

adbd1_tpurgent=(DESCRIPTION =(CONNECT_TIMEOUT = 90)(RETRY_COUNT = 50)(RETRY_DELAY = 3)(TRANSPORT_CONNECT_TIMEOUT = 3)(ADDRESS_LIST =(LOAD_BALANCE = ON)(ADDRESS = (PROTOCOL=TCP)(HOST=host-kyzm3-scan.secondarysubnet.secondaryvcn.oraclevcn.com)(PORT=1521)) (ADDRESS = (PROTOCOL=TCP)(HOST=host-xwgo9-scan.primarysubnet.primaryvcn.oraclevcn.com)(PORT=1521)) )(CONNECT_DATA = (SERVICE_NAME = ADBD1_tpurgent.atp.oraclecloud.com)))
…
  2. If you have dual entries, then modify them to only point to the local Autonomous Database by removing the ADDRESS of the remote database.

    The tnsnames.ora file in secondary should include only the ADDRESS of the standby database. For example: 

    adbd1_tp=(DESCRIPTION =(CONNECT_TIMEOUT = 90)(RETRY_COUNT = 50)(RETRY_DELAY = 3)(TRANSPORT_CONNECT_TIMEOUT = 3)(ADDRESS_LIST =(LOAD_BALANCE = ON)(ADDRESS = (PROTOCOL=TCP)(HOST= host-kyzm3-scan.secondarysubnet.secondaryvcn.oraclevcn.com)(PORT=1521)))(CONNECT_DATA = (SERVICE_NAME = ADBD1_tp.atp.oraclecloud.com)))

adbd1_medium=(DESCRIPTION =(CONNECT_TIMEOUT = 90)(RETRY_COUNT = 50)(RETRY_DELAY = 3)(TRANSPORT_CONNECT_TIMEOUT = 3)(ADDRESS_LIST =(LOAD_BALANCE = ON)(ADDRESS = (PROTOCOL=TCP)(HOST=host-kyzm3-scan.secondarysubnet.secondaryvcn.oraclevcn.com)(PORT=1521)))(CONNECT_DATA = (SERVICE_NAME = ADBD1_medium.atp.oraclecloud.com)))

adbd1_tpurgent=(DESCRIPTION =(CONNECT_TIMEOUT = 90)(RETRY_COUNT = 50)(RETRY_DELAY = 3)(TRANSPORT_CONNECT_TIMEOUT = 3)(ADDRESS_LIST =(LOAD_BALANCE = ON)(ADDRESS = (PROTOCOL=TCP)(HOST=host-kyzm3-scan.secondarysubnet.secondaryvcn.oraclevcn.com)(PORT=1521)))(CONNECT_DATA = (SERVICE_NAME = ADBD1_tpurgent.atp.oraclecloud.com)))
…

Modify the Secondary's TNS Alias Connect Strings for the Remote Refreshable Clone Approach

For the Remote Refreshable Clone approach, modify the alias used in the secondary system to be the same alias as in the primary system.

The tnsnames.ora file included in the secondary Oracle WebLogic Server for Oracle Cloud Infrastructure, Oracle SOA Suite on Marketplace, or Oracle Fusion Middleware system creation will contain an alias based on the remote refreshable clone name. For example, if a remote refreshable clone is created with the name soaadb1rc2 the tnsnames.ora file (in the wallet directory created during provisioning) will contain the following alias: soaadb1rc2_high, soaadb1rc2_low, soaadb1rc2_medium, soaadb1rc2_tp, soaadb1rc2_tpurgent. To simplify the configuration, you should use the same alias name in tnsnames.ora file in both the primary and secondary systems, so modify the TNS alias that the Oracle WebLogic Server for Oracle Cloud Infrastructure, Oracle SOA Suite on Marketplace, or Oracle Fusion Middleware domain is configured with (remote refreshable clone) to use the same alias name as primary. You can obtain the alias name in primary from the $tns_admin/tsnames.ora file. Different aliases are created for different services and they all will infer a prefix from the DB name. Notice that you want to modify the alias name only, not the services. Don't use a global search and replace in the file as that will likely also alter the service names in the connect strings in tnsnames.ora file.

  • Modify the secondary system's tnsnames.ora file to use the same alias as the primary system.

    The following is a sample tnsnames.ora file for an Oracle Fusion Middleware configuration with Oracle Autonomous Database Serverless in the primary system. 

    [oracle@soarefr-soa-0 ~]$ cat $DOMAIN_HOME/config/tnsadmin/tnsnames.ora
soaadb1_high = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbvcn.adb.ap-mumbai-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_high.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
 
soaadb1_low = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbvcn.adb.ap-mumbai-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_low.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
 
soaadb1_medium = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbvcn.adb.ap-mumbai-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_medium.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
 
soaadb1_tp = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbvcn.adb.ap-mumbai-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_tp.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
 
soaadb1_tpurgent = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbvcn.adb.ap-mumbai-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_tpurgent.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))

    The following is a sample tnsnames.ora file for an Oracle Fusion Middleware configuration with Oracle Autonomous Database Serverless in the secondary with Refreshable Clone. 

    oracle@soarefr-soa-0 ~]$ cat $DOMAIN_HOME/config/tnsadmin/tnsnames.ora
rcsoaadb1_high = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=rcsoaadb1.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_rcsoaadb1_high.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
 
rcsoaadb1_low = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=rcsoaadb1.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_rcsoaadb1_low.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
 
rcsoaadb1_medium = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=rcsoaadb1.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_rcsoaadb1_medium.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
 
rcsoaadb1_tp = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=rcsoaadb1.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_rcsoaadb1_tp.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
 
rcsoaadb1_tpurgent = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=rcsoaadb1.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_rcsoaadb1_tpurgent.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))

    Since it is a one time operation, the easiest way to change the alias for the refreshable clone is to edit the file. You can modify the alias for the service level in use or all of them for consistency.

    Note:

    This must be done every time a new refreshable clone is used for testing or validations and a new wallet is used.

    The following is a sample tnsnames.ora file for an Oracle Fusion Middleware configuration with Oracle Autonomous Database Serverless in the secondary with Refreshable Clone using the Primary's alias. 

    [oracle@soarefr-soa-0 ~]$ cat $DOMAIN_HOME/config/tnsadmin/tnsnames.ora
soaadb1_high = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=rcsoaadb1.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_rcsoaadb1_high.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
 
soaadb1_low = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=rcsoaadb1.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_rcsoaadb1_low.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
 
soaadb1_medium = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=rcsoaadb1.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_rcsoaadb1_medium.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
 
soaadb1_tp = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=rcsoaadb1.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_rcsoaadb1_tp.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
 
soaadb1_tpurgent = (description= (retry_count=20)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=rcsoaadb1.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_rcsoaadb1_tpurgent.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))

Just like in the primary system, all datasource files under $DOMAIN_HOME/config/jdbc and in the jps config files under $DOMAIN_HOME/config/fmwconfig use the chosen alias. It is expected that the same service level was chosen with the remote refreshable clone as in the primary system (either low, mid, high, tp, or tpurgent). Because the alias and datasources are copied over from the primary system, you don't need to run the fmw_change_to_tns_alias.sh script in the secondary system. The disaster recovery set up will handle the required replacements.

If you have additional aliases to point to other databases in the primary tnsnames.ora file, add them accordingly in the tnsnames.ora file of the secondary system.

Update the Host Name Aliases and Front-End Address in the Primary and Standby Midtiers

The WebLogic domain configuration in the secondary will be a copy of the primary WebLogic domain once the DR setup is completed. Therefore, the host names used as listen addresses by the primary Oracle WebLogic Servers (which are the host names of the primary hosts) must be valid in the secondary location, but mapping to the secondary IPs.

The same front-end address must be used in both the primary and standby. During normal operation this front-end host name will map to the IP of the primary Oracle Cloud Infrastructure (OCI) Load Balancer. When running from secondary (after a switchover or failover) this front-end host name will map to the IP of the secondary OCI Load Balancer.

Note:

The /etc/hosts file of the midtier hosts must not be altered when there is a switchover or failover. The mid-tier hosts will always resolve the virtual front-end name with its front-end IP. The DNS update that is needed during the switchover and failover procedures is performed in the DNS or host files used by the clients.

You can implement this host aliasing in the following ways:

  • Add the host names as aliases to the /etc/hosts files of the Oracle WebLogic Server for OCI compute instances
  • Use a private DNS view in the secondary OCI VCN

Use /etc/hosts Files

The host names used by the primary Oracle WebLogic Server are added to the /etc/hosts files of the secondary Oracle WebLogic Server hosts, pointing to the IP addresses of the secondary Oracle WebLogic Server hosts. This mode is valid when the DNS server is the same on primary and secondary Oracle Cloud Infrastructure (OCI) sites, and also when separated DNS servers are used in the primary and secondary sites. The entries in the /etc/hosts file have precedence over the DNS resolution, because this is the precedence defined out-of-the-box in the directive “hosts” of the /etc/nsswitch.conf file.
  1. Edit the /etc/oci-hostname.conf file of each WebLogic Server compute instance and set the property PRESERVE_HOSTINFO=3 to preserve /etc/hosts entries across instance reboots.
  2. Use the command hostname --fqdn to identify the complete host names of the OCI WebLogic Server compute instances.
  3. Add the following entries to the /etc/hosts file of the mid-tier compute instances:
    #################################
# ALIASES on OCI for DR
#################################
apphost1_compute_instance_IP  apphost1_fqdn   apphost1_hostname   ALIAS_OF_APPHOST1 
apphost2_compute_instance_IP  apphost2_fqdn   apphost2_hostname   ALIAS_OF_APPHOST2 
#################################
# Frontend name 
#################################
IP_OF_LBR  frontend_name_fqdn
    The following is an example of the /etc/hosts file of the primary Oracle WebLogic Server hosts:
    # Aliases for DR in primary region
10.0.2.10 wlsociprefix-wls-0.subnetlon.vcnlon.oraclevcn.com wlsociprefix-wls-0 
10.0.2.11 wlsociprefix-wls-1.subnetlon.vcnlon.oraclevcn.com wlsociprefix-wls-1

# Front-end virtual name to primary LBR IP
111.111.111.111 mywebapps.mycompany.com

    The following is an example of the /etc/hosts file in the secondary OCI WebLogic Server compute instance.

    Note:

    Primary host names are added as aliases of the secondary nodes and that front-end virtual name points to secondary load balancer IP:

    # Aliases for DR in secondary region
10.1.2.5 wlsociprefix-wls-0.subnetfra.vcnfra.oraclevcn.com wlsociprefix-wls-0 wlsociprefix-wls-0.subnetlon.vcnlon.oraclevcn.com wlsociprefix-wls-0
10.1.2.4 wlsociprefix-wls-1. subnetfra.vcnfra.oraclevcn.com wlsociprefix-wls-1 wlsociprefix-wls-1.subnetlon.vcnlon.oraclevcn.com wlsociprefix-wls-0

# Front-end virtual name to secondary LBRIP
222.222.222.222 mywebapps.example.com

Use the OCI Domain Name System (DNS)

The host names used by the primary Oracle WebLogic Server hosts are added to the DNS resolver used by the VCN of the secondary mid-tier servers, pointing to the IP addresses of the secondary Oracle WebLogic Server hosts. This mode is valid when separate DNS servers are used on primary and secondary Oracle Cloud Infrastructure (OCI) sites. Otherwise, it can cause conflicts in naming resolution. The server of each site should resolve these names with their own IPs. The advantage of this method is that you can add all the entries to a private DNS view, instead of adding them to all the /etc/hosts of all the Oracle WebLogic Server hosts.

The following are the steps to create the private view in the secondary VCN and resolve the host names used by primary with the secondary IPs:

  1. In the OCI Console, go to secondary region and create the private view to resolve the names of the PRIMARY hosts WITH THE SECONDARY IP addresses.
    1. Click Networking, DNS Management, Private Views, then Create Private View.
      For example, you can name the private view: PRIMARY_HOSTNAMES_PRIVATE_VIEW
    2. Click Create Zone in the private view.
      For the zone name, you must use the complete domain of the hosts. For example: subnetpri.vcnpri.oraclevcn.com
    3. Add the primary host names to this zone (short name), but resolved with secondary Oracle WebLogic Server hosts’ IP address.
    4. Click Publish changes.
  2. Add the private view to the secondary VCN resolver.
    1. Click the DNS resolver resource in the VCN.
    2. Add the DNS private view created previously.
      The hosts in the secondary VCN will resolve the host names used by the primary Oracle WebLogic Server hosts using the private view.
  3. Validate the resolution SECONDARY hosts, by doing ping and nslookup of the host names.
    They must be resolved with the equivalent SECONDARY IPs.

    Note:

    You can find Terraform code to create this OCI private view and records in GitHub.

Configure the Secondary System

To configure the system as the secondary, you must copy the WebLogic Server domain configuration from the primary to the secondary WebLogic Server, Oracle SOA Suite, or Oracle Fusion Middleware system domain.
The disaster recovery (DR) setup replaces all of the secondary domain, except specific directories and files that must remain locally-valid. These directories are explicitly excluded by the scripts. In addition, the script replaces the datasource files so that the secondary uses the same DB schema names as in the primary but with the existing jdbc url pointing to the local database (refreshable clone during DR set up and standby in preparation for switchover) with the required wallets.
  1. Run the fmwadb_dr_prim.sh script in the primary's WebLogic Server Administration Node.
    The script will check connectivity between the WebLogic Server Administration Server nodes through the Dynamic Routing Gateway and will copy the domain to the staging directory in an Oracle Cloud Infrastructure File Storage mount in the secondary region.

    Use the following parameters:

    • REMOTE_ADMIN_NODE_IP

      The IP address of the secondary WebLogic Server Administration server node.

      This IP must be reachable from this HOST (the primary Oracle WebLogic Server Administration server node.)

      It is recommended that you use the Dynamic Routing Gateway to interconnect the primary and secondary sites, enabling you to provide the private IP address.

    • REMOTE_SSH_PRIV_KEYFILE

      The private ssh keyfile to connect to the remote Oracle WebLogic Administration server node.

    • FSS_MOUNT

      The OCI File Storage mounted directory that is used to stage the copy of the WebLogic Server domain configuration. 

    ./fmwadb_dr_prim.sh [REMOTE_ADMIN_NODE_IP] [REMOTE_SSH_PRIV_KEYFILE] [FSS_MOUNT]
    The following is an example of the command and output:
    [oracle@soarefr-soa-0 ~]$ ./fmwadbs_dr_prim.sh '10.2.1.104' '/u01/soacs/dbfs/share/KeyWithoutPassPhraseSOAMAA.ppk' /u01/soacs/dbfs/share/
 Checking ssh connectivity to remote WebLogic Administration server node....
    Connectivity to 10.2.1.104 is OK
    REMOTE_ADMIN_HOSTNAME...... soarefr-soa-0.sub09051735411.soaadbvcnstby.oraclevcn.com
 Checking local FSS mount folder readiness........
     The FSS is expected to be mounted in /u01/soacs/dbfs/share
     The folder for the copy of the domain is expected to be /u01/soacs/dbfs/share/domain_config_copy
     Local folder /u01/soacs/dbfs/share/domain_config_copy exists.
 Checking remote FSS mount folder readiness........
     Remote folder 10.2.1.104:/u01/soacs/dbfs/share/domain_config_copy exists.
 
----------- Rsyncing from local domain to local FSS folder ---------------
 
Local rsync output to /u01/soacs/dbfs/share/domain_config_copy/last_primary_update_local_06-10-2022-09-47-50.log ....
Source and target directories are in sync. ALL GOOD!
 
----------- Local rsync complete -----------------------------------------
 
----------- Rsyncing from local FSS folder to remote site... --------------
Remote rsync output to /u01/soacs/dbfs/share/domain_config_copy/last_primary_update_remote_06-10-2022-09-47-50.log ....
Source and target directories are in sync. ALL GOOD!
  2. When the fmwadb_dr_prim.sh script finishes running, if not already stopped, stop all WebLogic Server systems and Node managers in the secondary system.
  3. Log into the secondary's WebLogic Server Administration node and run the fmwadb_dr_stby.sh script.

    The script copies the WebLogic Server domain from the staging directory in an OCI File Storage mount to the secondary WebLogic Server domain directory and replaces the required wallet and password entries. The wallet to be provided is the refreshable clone's so that secondary can be validated after the initial DR set up.

    Use the following parameters:

    • WALLET_DIR

      The directory for an unzipped Oracle Autonomous Database wallet. WebLogic Server Administration server node.

      This directory should contain at least tnsnames.ora, keystore.jks and truststore.jks files.

      If you are using the snapshot standby approach, this is just the tnsadmin folder.

    • WALLET_PASSWORD

      The password provided when the wallet was downloaded from the Oracle Autonomous Database Serverless OCI UI.

      If the wallet is the initial one created by the WebLogic Server, Oracle SOA Suite, or Oracle Fusion Middleware system when provisioning. You can obtain it with the following command:

      Oracle SOA:
      python /opt/scripts/atp_db_util.py generate-atp-wallet-password
      Oracle WebLogic Server:
      python3 /opt/scripts/atp_db_util.py generate-atp-wallet-password
    • FSS_MOUNT

      The OCI File Storage mounted directory that is used to stage the WebLogic Server domain configuration. 

    ./fmwadb_dr_stby.sh [WALLET_DIR] [WALLET_PASSWORD] [FSS_MOUNT]
    The following is an example of the command run and output:
    [oracle@wsladbs2-wls-0 scripts]$ ./fmwadb_dr_stby.sh /u01/data/domains/wsladbs2_domain/config/atpwallet/  7f3e3ed8ee7921fed058b481298eca55 /u01/shared/
Backing up current domain...
Backup created at /u01/data/domains/wsladbs2_domain/ /u01/data/domains/wsladbs2_domain_backup_11_42_19-26-10-22
Rsyncing from FSS  mount to domain dir...
 Syncing the Weblogic Administration server node...
Rsync complete!
Switching config to /u01/data/domains/wsladbs2_domain/config/atpwallet/
The password provided for the wallet is valid. Proceeding...
Gathering Data Soure information...
The new wallet is the same as the one being used.
Will maintain existing wallet dir and just replace password
Backing up current config...
Backup created at  /u01/data/domains/wsladbs2_domain/DS_backup_11_42_33-26-10-22
Replacing values in datasources...
Replacement complete!
  4. Run the fmwadb_dr_stby.sh script in the other WebLogic Server managed server nodes in the secondary region.
  5. Ensure that the standby database is accessible in read-write mode.
    • If you're using the snapshot standby approach, ensure that the standby database is converted to the snapshot standby database, so it is accessible in read-write mode.
    • If you are using Remote Refreshable Clone approach, make sure it is disconnected from the source, so it is accessible in read-write mode.
  6. Start Node Manager and the Oracle WebLogic Administration Server in secondary. Access the Console and start the WebLogic Server managed servers in the domain.
  7. Validate that the existing applications and deployments in the primary are also available in the secondary.
  8. Shutdown the WebLogic Servers in secondary.
  9. Convert the standby into a physical standby or reconnect the refreshable clone.

    • If you are using snapshot standby approach, then convert the standby database into physical standby database.

    • If you are using Remote Refreshable Clone, then reconnect the refreshable clone. Reminder: The refreshable clone cannot remain disconnected from the primary for more than 24 hours.

Leave the System Ready for Switchover

After the secondary system is configured and validated with a refreshable clone, leave the system ready for switchover by pointing the TNS alias to the standby Oracle Autonomous Database Serverless.

Note:

This task is only needed when the secondary system is configured and validated with a Remote Refreshable Clone.

Download the wallet for the standby database from the secondary database user interface. Avoid dual connect strings (both primary and standby hosts) in remote DR cases because they cause unnecessary retries.

  1. Log into the Oracle Autonomous Database Serverless user interface for your standby database.
  2. Click Autonomous Database, select the autonomous_db_name, click DB Connection, then click Download Wallet.
  3. If the wallet contains dual strings pointing to the primary, then remove the primary description's entry from the tnsnames.ora file so that the connect string points ONLY to the local DB in the secondary region.

    The following is a sample modification of the tnsnames.ora file: 

    soaadb1_high = (description_list= (failover=on) (load_balance=off) (description= (retry_count=15)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbvcn.adb.ap-mumbai-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_high.adb.oraclecloud.com))(security=(ssl_server_cert_dn="CN=adb.ap-mumbai-1.oraclecloud.com, OU=Oracle ADB INDIA, O=Oracle Corporation, L=Redwood City, ST=California, C=US"))) (description= (retry_count=15)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbstby.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_high.adb.oraclecloud.com))(security=(ssl_server_dn_match=no))))
soaadb1_low = (description_list= (failover=on) (load_balance=off) (description= (retry_count=15)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbvcn.adb.ap-mumbai-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_low.adb.oraclecloud.com))(security=(ssl_server_cert_dn="CN=adb.ap-mumbai-1.oraclecloud.com, OU=Oracle ADB INDIA, O=Oracle Corporation, L=Redwood City, ST=California, C=US"))) (description= (retry_count=15)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbstby.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_low.adb.oraclecloud.com))(security=(ssl_server_dn_match=no))))
soaadb1_medium = (description_list= (failover=on) (load_balance=off) (description= (retry_count=15)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbvcn.adb.ap-mumbai-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_medium.adb.oraclecloud.com))(security=(ssl_server_cert_dn="CN=adb.ap-mumbai-1.oraclecloud.com, OU=Oracle ADB INDIA, O=Oracle Corporation, L=Redwood City, ST=California, C=US"))) (description= (retry_count=15)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbstby.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_medium.adb.oraclecloud.com))(security=(ssl_server_dn_match=no))))
soaadb1_tp = (description_list= (failover=on) (load_balance=off) (description= (retry_count=15)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbvcn.adb.ap-mumbai-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_tp.adb.oraclecloud.com))(security=(ssl_server_cert_dn="CN=adb.ap-mumbai-1.oraclecloud.com, OU=Oracle ADB INDIA, O=Oracle Corporation, L=Redwood City, ST=California, C=US"))) (description= (retry_count=15)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbstby.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_tp.adb.oraclecloud.com))(security=(ssl_server_dn_match=no))))
soaadb1_tpurgent = (description_list= (failover=on) (load_balance=off) (description= (retry_count=15)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbvcn.adb.ap-mumbai-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_tpurgent.adb.oraclecloud.com))(security=(ssl_server_cert_dn="CN=adb.ap-mumbai-1.oraclecloud.com, OU=Oracle ADB INDIA, O=Oracle Corporation, L=Redwood City, ST=California, C=US"))) (description= (retry_count=15)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbstby.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_tpurgent.adb.oraclecloud.com))(security=(ssl_server_dn_match=no))))

SHOULD BE

soaadb1_high = (description= (retry_count=15)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbstby.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_high.adb.oraclecloud.com))(security=(ssl_server_dn_match=no))) 
soaadb1_low = (description= (retry_count=15)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbstby.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_low.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
soaadb1_medium = (description= (retry_count=15)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbstby.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_medium.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
soaadb1_tp = (description= (retry_count=15)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbstby.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_tp.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
soaadb1_tpurgent = (description= (retry_count=15)(retry_delay=3)(address=(protocol=tcps)(port=1522)(host=soaadbstby.adb.ap-hyderabad-1.oraclecloud.com))(connect_data=(service_name=g914a2540e8ab6d_soaadb1_tpurgent.adb.oraclecloud.com))(security=(ssl_server_dn_match=no)))
  4. Copy the wallet to a temp folder in the secondary region's WebLogic Server Administration Server node and navigate to the directory.
    [oracle@wsladbs2-wls-0 scripts]$ cd /tmp
[oracle@wsladbs2-wls-0 tmp]$ mkdir wallet-stby
[oracle@wsladbs2-wls-0 tmp]$ cd wallet-stby/
  5. Unzip the wallet.
    [oracle@wsladbs2-wls-0 wallet-stby]$ unzip /tmp/Wallet_soaadb1-standby.zip
Archive:  ../Wallet_soaadb1-standby.zip
  inflating: ewallet.pem
  inflating: README
  inflating: cwallet.sso
  inflating: tnsnames.ora
  inflating: truststore.jks
  inflating: ojdbc.properties
  inflating: sqlnet.ora
  inflating: ewallet.p12
  inflating: keystore.jks
  6. Run the fmwadb_switch_db_conn.sh script to leave the middle tier ready with the secondary wallet and connect string.

    The following is an example of the script run to point to the standby Oracle Autonomous Database Serverless wallet: 

    [oracle@wsladbs2-wls-0 scripts]$ ./fmwadb_switch_db_conn.sh /tmp/wallet-stby/ wallet_password
The password provided for the wallet is valid. Proceeding...
Gathering Data Source information...
The new wallet is the same as the one being used.
Will maintain existing wallet dir and just replace password
Backing up current config...
Backup created at  /u01/data/domains/wsladbs2_domain/DS_backup_10_38_32-27-10-22
Replacing values in datasources...
Replacement complete!

Setup Ongoing Configuration Replication

When the system’s lifecycle involves frequents updates to the domain file system, you can use a script to automate the configuration replication process. The fmwadb_config_replica.sh script replicates changes through the Oracle Cloud Infrastructure File Storage (OCI File Storage) stage directory on a regular basis.

The configuration is ready (prepared) for a switchover after each replication.

When you're using Remote Refreshable Clone, it is assumed that the replicated configuration will be "adapted" for the physical standby (not the refreshable clone). If you need a validation or test using refreshable clones, then you can alter the standby domain configuration to point to the refreshable clone instead.

You must run the fmwadb_config_replica.sh script in the WebLogic Server Administration Nodes (in both the primary and standby) to replicate the configuration. Typically, this script is scheduled with a cron job to replicate the configuration between a primary and a standby WebLogic Server, Oracle SOA Suite, or Oracle Fusion Middleware system on the Oracle Autonomous Database system at regular intervals. This script checks the current role of the local database to determine if it is running in the primary or standby site.

  • When the script runs in the PRIMARY site, it copies the domain configuration from primary domain to local assistance folder (FSS) and then to the secondary site assistance folder (through rsync).
  • When the script runs in STANDBY site, it copies the domain configuration from the secondary assistance folder (OCI File Storage) to the secondary domain and makes the required replacements for datasources to work with the local database.

You must gather different parameters from the OCI Console and encrypt the password for accessing the Oracle Autonomous Database wallets for security reasons.

The following is a description of the variables used in the script and how to obtain them:

  • REMOTE_WLSADMIN_NODE_IP

    Peer and remote WebLogic Server Administration server node's IP.

    This is the IP of the node host in the WebLogic Server Administration Server in the peer site. It must be reachable from the local node. It is recommended to connect to the remote private IP of the node using a Dynamic Routing Gateway.

  • REMOTE_SSH_PRIV_KEYFILE

    The private SSH keyfile to connect to remote Oracle WebLogic Administration server node.

  • TENANCY_OCID

    The OCID of the tenancy where the Oracle Autonomous Database resides. You can get the OCID from the Oracle Cloud Infrastructure (OCI) user interface (UI).

  • USER_OCID

    The OCID of the user owning the autonomous database instance. You can find the OCID in the OCI UI.

  • PRIVATE_KEY

    Path to the private PEM format key for this user.

  • LOCAL_ADB_OCID

    The OCID of the Oracle Autonomous Database being inspected. You can find the OCID in the Oracle Autonomous Database screen in the OCI Console.

  • WALLET_DIR

    The directory for the local Oracle Autonomous Database wallet (unzip the wallet downloaded from the OCI Console). This directory should contain at least the tnsnames.ora, keystore.jks, and truststore.jks files. When you use Snapshot Standby approach, this is the tnsadmin folder

  • ENC_WALLET_PASSWORD

    The WLS ENCRYPTED incarnation of the password provided when the wallet was downloaded from the Oracle Autonomous Database OCI UI.

    If the wallet is the initial one created by WebLogic Server, Oracle SOA Suite, or Oracle Fusion Middleware during provisioning WebLogic Server, then you can use the following commands to get the password:

    For WebLogic:
    [oracle@wsladbs2-wls-1 ~]$ python3 /opt/scripts/atp_db_util.py generate-atp-wallet-password
    For SOA:
    [oracle@soarefr-soa-0 ~]$ python /opt/scripts/atp_db_util.py generate-atp-wallet-password
    To encrypt the password, whether the one provided in the OCI Console or the one used during provisioning, you can use the fmw_enc_pwd.sh script.
    ./fmw_enc_pwd.sh UNENC_WALLET_PASSWORD

    Use the obtained string for the ENC_WALLET_PASSWORD variable.

  • FSS_MOUNT

    The OCI File Storage Mounted directory that will be used to stage the WebLogic Server domain configuration.

Once you have collected the information for the variables, perform the following steps to copy and customize the scripts for your environment:

  1. Encrypt the password for accessing the Oracle Autonomous Database wallets for security reasons.
  2. Copy the script to the primary site, then edit the script completing the required variables for the primary site.
    See above for the description of the variables used in the script.
  3. Copy the script to the standby site, then edit the script filling in the required variables for the standby site.
    See above for the description of the variables used in the script.
  4. Once customized for EACH region (you must provide the LOCAL Database OCID for each region), perform the following tasks:
    1. Run the script in the Oracle WebLogic Administration host in primary.
    2. Run the script in the Oracle WebLogic Administration host in secondary region.
  5. Add the script to a cron job in each region.