Validating the Setup

3 Validating the Setup

You must validate the secondary system on a regular basis especially after the initial setup. You can validate the standby site without performing a complete switchover by converting the Fusion Middleware standby database to snapshot standby. This allows the secondary Fusion Middleware servers to start in the standby site, so that you can verify that the secondary Fusion Middleware system has been setup properly. Testing the secondary system is a key part of a healthy and reliable disaster protection strategy.

In MAA, it is recommended to schedule periodic switchovers (every six to nine months) to run a reliable and complete verification of the secondary. This also allows preparing operations and teams for real disaster situations.

However, enterprises may want to perform periodic testing of their secondary site without shutting down the current production site and avoiding a complete switchover operation and validate behaviors for new changes, workload, and so on. Since most Fusion Middleware components (primarily the WebLogic Servers) require write access to the database (for persistent stores and service migration leasing) this is only possible by converting the standby database to snapshot standby (which makes the secondary database writable). This allows the standby middle-tier Fusion Middleware components to be started in the secondary site. Any change performed in the secondary site database while it is in the snapshot standby mode will be discarded once it is converted back to physical standby, so primary data will not be affected by secondary site validations. By using this approach with Oracle Data Guard snapshot standby, the primary system can continue processing workloads while the secondary database is actively used for verifications at the functional and performance level. Secondary systems can be used this way to verify new patches, new versions of applications, and changes in infrastructure and operational models while using an up-to-date copy of the production database.

These validation operations need to be performed with caution. If there are pending JMS messages, transactions or pending operations (like uncompleted SOA composites or business processes with faults) in the database, when it is converted into snapshot, the Fusion Middleware servers at the standby site will process them when they start. You must verify that there are no pending actions for Fusion Middleware components in the database when you convert to snapshot standby, otherwise remove records from the runtime tables in the standby database (such as JMS and JTA persistent stores) after you convert it to snapshot standby database and before you start the secondary site’s Fusion Middleware servers.

Testing the Setup

When using shared storage replications to synchronize middle-tier file systems, use the cloning technology provided by the shared storage vendor to create a clone of the secondary site's read-only volumes on the shared storage at the secondary site. Ensure that the cloned secondary site volumes are writable. If you want to test the secondary site just once, then this can be a one-time clone operation. However, if you want to test the secondary site regularly, you can set up periodic cloning of the secondary site read-only volumes to the secondary site's cloned read or write volumes. Modify the mounts accordingly in the secondary to mount the activated copies in each node using the shared storage.
If rsync is used, perform a refresh of contents from the primary to the secondary site using the example scripts.
If DBFS is used, the latest copy of information would be propagated to the secondary by Oracle Data Guard automatically.
Convert the standby database into snapshot standby by performing the following steps:
1. Use Data Guard broker in the primary database host to convert the secondary database to snapshot standby.
  
  Example:
```
[oracle@primarydbhost~]$dgmgrl sys/ your_sys_password@primary_db_unqname
DGMGRL> convert database “secondary_db_unqname” to snapshot standby
```
2. Use dgmgrl show configuration command to verify that the conversion has been correctly performed.
Verify that there are no pending actions in the secondary environment. If there were pending actions (transactions, messages) in the primary DB when the standby is converted to snapshot, the secondary Fusion Middleware servers will try to process them when they start. For Oracle Fusion Middleware SOA, you can use the SOA truncate script to remove the records from the SOA runtime tables in the secondary database to clean the runtime data before you start the secondary servers. See Removing Records from the Runtime Tables Without Dropping the Tables.
Configure hostname mappings for your validation. Since this is not a switchover and the primary site is still active, the front-end address will resolve to the primary site’s LBR IP address so that any browser access will by default be redirected to the active primary site. To access the secondary site services directly, you can use local host name resolutions by updating the /etc/hosts files in the WLS nodes and in controlled clients (laptop and so on) so that the front-end and access points resolve to the secondary addresses. For more information, see Host Name Resolution.

Note:

Verify that the client used for validations does not access the Fusion Middleware system via an HTTP proxy because some HTTP proxies may continue to resolve the virtual front-end name with the primary site’s LBR IP address regardless of which name is in the /etc/hosts of the client.

Note:

Non-Linux clients may require a reset of their local DNS cache before a browser will resolve the IP address with the customized host file entry.
Start the WebLogic servers in the secondary site:
1. Start the secondary admin server as shown in the following example:
```
$ cd /u01/app/oracle/middleware/oracle_common/common/bin
$ ./wlst.sh
wlst> nmConnect ('weblogic', 'password','sooahost1','5556','soaedgdomain','/u01/data/domains/soaedgdomain','SSL')
wlst> nmStart('Adminserver')
```
2. Start secondary managed servers (use the secondary WebLogic Remote Console or scripts).
  
  You can use the start or stop scripts provided by MAA in GitHub. These scripts provide more granularity and improved shutdown procedures.
  
  Note:
  For Fusion Middleware SOA, while you validate the secondary site as described (without performing a complete switchover that is just opening standby in snapshot standby mode), "ORA-01403: no data found ORA-06512" errors may show up in the logs of the standby SOA servers. These errors are related to the SOA auto purge job. These errors arise because jobs in the database may have DB role dependencies (they are defined to be enabled only when the database is in primary role). This is an expected and desired behavior that prevents jobs from being executed twice (once in primary and once in standby). The SOA auto purge job is defined with primary role, so it is not shown in DBA_SCHEDULER_JOBS view when the database is in snapshot standby mode. The database_role defined for each job can be seen in the view DBA_SCHEDULER_JOB_ROLE. In summary, these errors can be ignored as long as they appear in the standby system. The scheduler job for SOA auto purge will be executed on the DB only if the instance changes its role to primary.
  
  In summary, review each WebLogic Server to verify that all applications and subsystems come up correctly.
Validate your Fusion Middleware system. You must first test the WebLogic data sources and confirm that all the applications are in running state in the Weblogic Remote Console. For SOA, use the Enteprise Manager test composite utility and verify that your specific business logic works properly. Any information persisted to the database will be discarded once the DB is placed again on physical standby mode.
After you have finished validations on the secondary site, perform the following steps to revert it back to standby role again:
1. Stop managed servers and admin servers in the secondary.
  
  You can connect to the secondary WebLogic Remote Console and shutdown managed and administration servers in the secondary site.
2. Convert the standby DB into a physical standby again. Use DG broker in primary DB host and convert the secondary to physical standby again as an oracle user as follows:
```
[oracle@drdbA ~]$ dgmgrl sys/your_sys_password@primary_db_unqname
DGMGRL> convert database “secondary_db_unqname” to physical standby
```
  Use show configuration command to verify that the conversion has been correctly performed.
3. Revert any updated client's /etc/hosts file.
  
  If you updated the front-end name in the /etc/hosts file of a client to point to the secondary site, revert it back so that the virtual front-end name points to the primary front-end IP again.