1. Configure a hybrid DR solution for Oracle SOA Suite
  2. Configure the Environment

Configure the Environment

Setup the secondary system in OCI and configure it as the standby of the primary on-premises system.

Scripts are available to automate some of the steps. These scripts don't automate the complete setup, so you must complete the tasks and you can use the scripts when they are referenced.

Go to Download Code for the link to download the scripts referenced in this document.

Prepare the WebLogic Data Sources in the Primary Data Center

There are several approaches that you can use for the database connection string of the WebLogic data sources in a disaster recovery topology, such as dual string, different connect strings, and TNS alias. See the Oracle Fusion Middleware Disaster Recovery Guide for more details and comparison between the different approaches. We'll use TNS alias because it requires a one-time setup. Using a TNS alias means that you won't need to alter the WebLogic configuration every time you copy it to the secondary. Using a TNS alias in GridLink data sources is supported starting FMW version 12.2.1.3.

The TNS alias is the same name in the primary and secondary; therefore, the datasources use the same db connect string. It is resolved with a tnsnames.ora file that is not copied to standby, so you can have different tnsnames.ora content in each site. You can place it separately from the WebLogic domain configuration, in a file system that is not replicated between sites. Or, given that it is part of the configuration, you can also store it in a folder under the domain configuration. In this case, make sure that you exclude that folder when you copy the domain configuration from primary to standby. Each site will resolve the TNS alias with the appropriate connect string in each site, pointing to the local database only. For example: 

Connect string in data sources in primary site:
jdbc:oracle:thin:@soapdb

The tnsnames.ora file in primary contains:
SOAPDB =
(DESCRIPTION=
 (ADDRESS_LIST=
    (LOAD_BALANCE=ON)
    (ADDRESS=(PROTOCOL=TCP)(HOST=prmy-scan)(PORT=1521)))
    (CONNECT_DATA=(SERVICE_NAME=soapdb.example.com))
)

Connect string in data sources in secondary site:
jdbc:oracle:thin:@soapdb
The tnsnames.ora file in secondary: 
SOAPDB =
(DESCRIPTION=
 (ADDRESS_LIST=
    (LOAD_BALANCE=ON)
    (ADDRESS=(PROTOCOL=TCP)(HOST=stby-scan)(PORT=1521)))
    (CONNECT_DATA=(SERVICE_NAME=soapdb.example.com))
)

The following are the advantages of using TNS alias:

  • Since the same db connect string is used in the WebLogic domain config, you don't need to alter the WebLogic configuration after replicating the config from the primary to the standby.
  • As each site points to the local database only, there is no risk of cross-connections from the mid-tier to the remote database.

If you are not already using this approach in the primary SOA system, then perform the following steps to use a TNS alias in the data sources.

  1. Create the tns folder in the primary mid-tier hosts:

    This folder must be readable by the Oracle user and placed in a file system that is not replicated between sites.

    Given that the tns folder is part of the configuration, you can also create it under the config folder that is shared by all the servers. But in that case, ensure that you exclude the tns folder when you copy the domain configuration from primary to standby or update tnsnames.ora in the standby system, after a failover or switchover, to point to the secondary database. 

    [oracle@host3 ~]$ mkdir -p /home/oracle/tnsnames_dir
[oracle@host4 ~]$ mkdir -p /home/oracle/tnsnames_dir
  2. Create a tnsnames.ora file in the directory with the TNS alias that will be used in the data sources, as shown in the example.
    Oracle recommends entering the string in a single line.
    SOAPDB = 
(DESCRIPTION=
  (ADDRESS_LIST=
    (LOAD_BALANCE=ON)
    (ADDRESS=(PROTOCOL=TCP)(HOST=dbhost-scan.example.com)(PORT=1521))
  )
  (CONNECT_DATA=(SERVICE_NAME= soapdb.example.com))
)
  3. Specify the oracle.net.tns_admin property pointing to the directory location of the tnsnames.ora file. Use one of the following methods:

    • Option 1: Set the property as a data source connection property. Oracle recommends this method.

      1. Specify the oracle.net.tns_admin=tns_directory property in the data source configuration.

        To specify this property in the WebLogic Administration Console, go to Services, click Data Sources, select a data source from the list, click Connection Pool, and then add it to the Properties text box. Repeat this step for each data source.

        For example: oracle.net.tns_admin=/home/oracle/tnsnames_dir

      2. Specify this property in the OPSS security stores files jps-config-jse.xml and jps-config.xm available in the $ASERVER_HOME/config/fmwconfig folder.

        To modify these jps files, edit them and add the oracle.net.tns_admin property after the jdbc.url property. For example, 

        …
<property name="jdbc.url" value="jdbc:oracle:thin:@soaedg"/>
<property name="oracle.net.tns_admin" value="tns_directory"/>
…

        Note:

        This property applies to the specific file (data source, jps file) in which it is specified.

    • Option 2: Set the property as a java system property.

      1. Specify the -Doracle.net.tns_admin=tns_directory system property where tns_directory is the directory location of the tnsnames.ora file.

        To set it as a java property for the servers, edit the following files:
        • $ASERVER_HOME/bin/setUserOverrides.sh and
        • $MSERVER_HOME/bin/setUserOverrides.sh (This file is not shared. Therefore, you should edit the file in all of the SOA mid-tier hosts.)

      2. Add the following content to these files: 

        # For using tns alias in the datasources 
export EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES} -Doracle.net.tns_admin=/home/oracle/tnsnames_dir

        Note:

        This property applies to all the data sources and jps files in Oracle WebLogic Server. Before you run some of the WLST commands and the Configuration Wizard, this approach requires that you set the property in the environment.
        • Before running WLST, set the property in the WLST_PROPERTIES environment variable.
        • Before running the Configuration Wizard, add the property to the JVM_ARGS environment variable of the config_internal.sh script.
      3. Option 3: Set the property in the jdbc URL.

        Specify the location of the tnsnames.ora file as part of the connection string in the data sources and jps files: 

        jdbc:oracle:thin:@alias?TNS_ADMIN=tns_directory

      Note:

      This property applies to the specific file (data source, jps file) in which it is specified.

      You can use this method with JDBC Driver 18.3 and later. This applies to Fusion Middleware 12.2.1.4 (which uses JDBC Driver 19.3) and later.

  4. Use the alias in the data source definition URL by replacing the connection string with the alias.
    jdbc:oracle:thin:@soapdb
    Modify the following files:
    • In data source files, located in $ASERVER_HOME/config/jdbc/
    • In the jps-config.xml and jps-config-jse.xml files, located in $ASERVER_HOME/config/fmwconfig/
    You can use the Oracle WebLogic Server Administration Console to modify the data sources, but you must manually edit the jps-config xml files. You can use the update_dbconnect.sh script to perform the replacement in all of the files mentioned.

    Go to Download Code for the link to download the script.

  5. Restart each Oracle WebLogic Server in the domain.
    1. Stop all of the WebLogic Servers in the primary domain (Admin server and Managed servers).
    2. Start the Administration server in the primary domain.
    3. Once the Administration server is running, start the Managed Servers.
    4. Verify that the connections are correctly established with the database.
  6. Additional best practice: When you are using an Oracle Database 12c or later versions, the Oracle Notification Service (ONS) Host and Port configuration values aren't required in the GridLink data sources.
    The Oracle Notification Service list is automatically obtained from the database by the client driver. Oracle recommends using this feature instead of providing the scan address or the list of the Oracle RAC nodes in the ONS configuration of each data source. Ensure that FAN is enabled and that the ONS nodes property is empty in each data source.
    1. Open the Oracle WebLogic Server Administration Console.
    2. Go to Services, then Data Sources. Select the data source name, then Configuration, and ONS.
    3. Verify that the ONS nodes list is empty.

Configure the Network

Connectivity is required between the primary on-premises network and the secondary Oracle Cloud Infrastructure (OCI) network. Oracle recommends using Oracle Cloud Infrastructure FastConnect, which allows you to connect directly to your OCI virtual cloud network through a dedicated, private, high-bandwidth connection. You choose an appropriate port speed based on the amount of data. Alternatively, you can use Site-to-Site VPN, although it provides lower bandwidth, and variable latency, jitter, and cost.
  1. Create the VCN and subnets in the OCI compartment as defined in Determine the Resources Needed on OCI. Configure Oracle Cloud Infrastructure FastConnect (or Site-to-Site VPN) to allow communication between your on-premises network and the VCN. See FastConnect and Site-to-Site VPN for information.
  2. Create the required network rules in OCI and in the on-premises firewall and configure the source, targets, protocols and ports, as shown in the table.

    It is assumed that all the traffic is enabled inside each subnet.

    See the OCI Documentation for information about network security rules.

    Source Target Protocol and Port Usage
    On-premises network OCI mid-tier subnet SSH (port 22) Setup and lifecycle
    On-premises network OCI web-tier subnet SSH (port 22) Setup and lifecycle (when Oracle HTTP Server is used in OCI)
    On-premises network OCI db-tier subnet SSH (port 22) Setup and lifecycle
    On-premises DB hosts OCI db-tier subnet SQLNET (port 1521) For Oracle Data Guard redo transport
    On-premises network OCI web-tier subnet HTTPS/HTTP (443, 80, 7001, 8888) Access to Oracle SOA Suite services and administration consoles
    On-premises network OCI mid-tier subnet HTTP (7001,7010,8001,8011,8021,9001) For direct checks to Oracle WebLogic Servers
    OCI web-tier subnet OCI mid-tier subnet HTTP (7001,7010,8001,8011,8021,9001) For connectivity from the web-tier components (OCI Load Balancer, Oracle HTTP Server) to WebLogic Servers
    OCI mid-tier subnet OCI db-tier subnet SQLNET (1521), ONS (6200) For connectivity from WebLogic Servers to database
    OCI mid-tier subnet OCI fss-tier subnet See Configuring VCN Security Rules for File Storage for specific information. To mount the OCI File Storage file system exports with NFS.
    OCI mid-tier subnet On-premises mid-tier hosts SSH (port 22) For rsync
    OCI db-tier subnet On-premises db hosts SQLNET (1521) For Oracle Data Guard redo transport
    OCI mid-tier subnet OCI web-tier subnet HTTPS (443) For callbacks from the application to the front-end

    Note:

    You can find terraform code to create the VCN, the subnets, and the security rules in Download Code.

Configure Oracle Data Guard

Configure Oracle Data Guard between the primary on-premises database and the standby database in Oracle Cloud Infrastructure (OCI).
  1. Configure Oracle Data Guard between a primary, on-premises database and a standby in OCI, as described in Hybrid Data Guard to Oracle Cloud Infrastructure.
    To help configure Oracle Data Guard, scripts are available in GitHub and referenced in Download Code.
  2. Verify that the Oracle Data Guard setup is correct by using the Oracle Data Guard command-line interface (DGMGRL).
    DGMGRL> show configuration verbose
  3. Once you have completed configuring Oracle Data Guard (Step 2), create the same database services in the OCI DB system, as those used in the primary, on-premises system. Configure the service with the PRIMARY role and with the SNAPSHOT_STANDBY role.
    Configuring the service with both roles allows the service to be automatically started by DG Broker when the database is converted to snapshot standby, which is required when you want to validate the secondary system without performing a complete switchover.
    For example, if the primary database uses the database service soapdb.example.com to access to the PDB, then create the same service in the secondary OCI DB system. 
    srvctl add service -db $ORACLE_UNQNAME -service soapdb.example.com -preferred ORCL1,ORCL2 -pdb PDB1 -role "PRIMARY,SNAPSHOT_STANDBY"
srvctl modify service -db $ORACLE_UNQNAME -service soapdb.example.com -rlbgoal SERVICE_TIME -clbgoal SHORT
srvctl config service -db $ORACLE_UNQNAME -service soapdb.example.com
  4. If the service in the primary database was created with the role PRIMARY only (which is the default), then you can modify it to add the snapshot standby role.
    srvctl modify service -db $ORACLE_UNQNAME -s soapdb.example.com -role "PRIMARY,SNAPSHOT_STANDBY"
  5. Review the Oracle Recovery Manager (RMAN) policies in your primary database to prevent the archive logs from being deleted before they are applied to the standby database.
    Ensure that you have the clause applied on all standby in the archivelog deletion policy of RMAN, in the primary and standby databases.

About the database version and patch level

Oracle Home in the on-premises database and the standby database on OCI must be the same version and at the same patch level. You can achieve this in the following ways:

  • When choosing the Database Software image during the DB System provisioning in OCI, select Display all versions and select the same database version and patch set level as the on-premises database.
  • If the source database’s Oracle Home version is no longer available in OCI for provisioning, you will have to patch the source environment to the same database patch level as the database home in the cloud environment.

The following scenario is an real example for reference. The on-premises DB Home is 19.6 and the OCI DB Home is 19.11.

  1. Run the command $ORACLE_HOME/OPatch/opatch lspatches to identify the patches installed on both the Source and Target environments.
    $ORACLE_HOME/OPatch/opatch lspatches

    The following is the output in this example:

    DB Oracle Home Patches On-premises DB Oracle HOME Patches on OCI

    30676209;LNX64-20.1-RAC ASM HIT ORA-07445 EXCEPTION ENCOUNTERED CORE DUMP [KSXPOSDIFQRY()+556]

    30613937;IPCOR TOPO SERVICE FIX IP TYPE BUG IN IP SELECTION

    30484981;OJVM RELEASE UPDATE: 19.6.0.0.200114 (30484981)

    30489227;OCW RELEASE UPDATE 19.6.0.0.0 (30489227)

    30557433;Database Release Update : 19.6.0.0.200114 (30557433)

    29780459;INCREASE _LM_RES_HASH_BUCKET AND BACK OUT CHANGES FROM THE BUG 29416368 FIX

    30310195;DBSAT REPORTED DISABLED CONSTRAINTS FOR SHARDING STS_CHUNKS ON GSMADMIN_INTERNAL.SHARD_TS

    32327201;RDBMS - DSTV36 UPDATE - TZDATA2020E

    31335037;RDBMS - DSTV35 UPDATE - TZDATA2020A

    30432118;MERGE REQUEST ON TOP OF 19.0.0.0.0 FOR BUGS 28852325 29997937

    31732095;UPDATE PERL IN 19C DATABASE ORACLE HOME TO V5.32

    32490416;JDK BUNDLE PATCH 19.0.0.0.210420

    32399816;OJVM RELEASE UPDATE: 19.11.0.0.210420 (32399816)

    32579761;OCW RELEASE UPDATE 19.11.0.0.0 (32579761)

    32545013;Database Release Update : 19.11.0.0.210420 (32545013)
  2. Analyze the existing patches: determine which patches are one-offs, review if they are already fixed in the new RU patches or if new overlap patches are needed, determine which of them must be uninstalled, locate the appropriate patch files for RU, and so on.
  3. Based on the analysis, uninstall the one-off patches that are already fixed in the new RU before installing the RU update (otherwise, they will cause conflict). In this example, the on-off patches are fixed in 19.11, so the patches must be rolled back before installing the 19.11 RU.
    30676209;LNX64-20.1-RAC  ASM HIT ORA-07445  EXCEPTION ENCOUNTERED  CORE DUMP [KSXPOSDIFQRY()+556]
30613937;IPCOR TOPO SERVICE  FIX IP TYPE BUG IN IP SELECTION
  4. Locate, download, and install the RU patches. In this example, the 19.11 RU patches are located in the combo patch 32578973: COMBO OF OJVM RU COMPONENT 19.11.0.0.210420 + GI RU 19.11.0.0.210420 and are the following:
    32399816;OJVM RELEASE UPDATE: 19.11.0.0.210420 (32399816)  
32579761;OCW RELEASE UPDATE 19.11.0.0.0 (32579761)            
32545013;Database Release Update : 19.11.0.0.210420 (32545013)
  5. Locate, download, and install the overlays, one-offs, and other patches that the OCI DB Home has on top of the RU. For example: 
    29780459;INCREASE _LM_RES_HASH_BUCKET AND BACK OUT CHANGES FROM THE BUG 29416368 FIX
30310195;DBSAT REPORTED DISABLED CONSTRAINTS FOR SHARDING STS_CHUNKS ON GSMADMIN_INTERNAL.SHARD_TS
30432118;MERGE REQUEST ON TOP OF 19.0.0.0.0 FOR BUGS 28852325 (DSTV33 update) 29997937 (DSTV34 update)
31335037;RDBMS - DSTV35 UPDATE - TZDATA2020A
32327201;RDBMS - DSTV36 UPDATE - TZDATA2020E
32490416;JDK BUNDLE PATCH 19.0.0.0.210420
31732095;UPDATE PERL IN 19C DATABASE ORACLE HOME TO V5.32
  6. Perform a similar analysis for the GI patches.

Note:

This example includes only the RDBMS Oracle Home. Patching the on-premises GI installation to the same level as secondary may be not strictly necessary:
  • From Oracle Data Guard’s perspective, it's not strictly necessary to have the same GI versions in the primary and standby: Oracle Data Guard is completely independent from anything under the database, so you can run different versions of the operating system, Oracle Clusterware, hardware, or storage software across different sites with no restrictions on versions or time. (Doc ID 1265700.1)
  • Regardless of Oracle Data Guard, it is not required to have the same version in GI and RDBMS versions in a RAC DB: Starting from 18c, the Oracle Grid Infrastructure (GI) /Clusterware (CRS) version must be of equal or the highest version down to the first digit in the possible combinations at all times. For example: Grid infrastructure can be at 18.1.0.0 and Database can be at 18.3.0.0. (Doc ID 337737.1)

It's a good practice to patch GI at same level that the DB home. Once you have to patch the DB home to a newer Release Update (RU), many of the patches are common for DB and GI, and you can use OPatchAuto to both homes at the same time.