Design Considerations for Active-Active Stretch Cluster Topology

6 Design Considerations for Active-Active Stretch Cluster Topology

In an active-active stretch cluster topology, cluster nodes can span data centers within a proximate geographical range, and usually with guaranteed, relatively low latency networking between the sites.

In addition to the general best practices recommended for all Oracle WebLogic Server Continuous Availability MAA architectures as described in Common Design Considerations for Continuous Availability, the following sections describe the design considerations and failure scenarios that apply to the MAA architecture shown in Active-Active Stretch Cluster with an Active-Passive Database Tier.

Active-Active Stretch Cluster with an Active-Passive Database Tier Design Considerations

Consider Oracle’s best practice design recommendations for continuous availability in an active-active stretch cluster topology with an active-passive database tier.

To take full advantage of continuous availability features in an active-active stretch cluster, consider the following:

In a multi-data center, active-active stretch cluster environment session replication across data centers can cause serious performance degradation in the system. Oracle recommends defining two different replication groups (one for each site) to minimize the possibility of replication occurring across the two sites.

Note:

Using replication groups is a best effort to replicate state only to servers in the same site, but is not a deterministic method. If one single server is available in one site, and other servers are available in the other site, replication occurs across the MAN and continues for that session even if servers come back online in the same site.
An active-active stretch cluster only works in the metro latency model. Latency should be no longer than 10-milliseconds round-trip time (RTT).
For contention and security reasons, Oracle does not recommend using shared storage across sites.
Each site uses individual shared storage. If this is used to store runtime data, disk mirroring and replication from Site1 to Site2, and in reverse, can be used to provide a recoverable copy of these artifacts in each site.
It is recommended to store the JMS and Transaction Logs (TLogs) in database-based stores. This allows cross-site service migration for the JMS and JTA services, because the TLogs and JMS messages are available in the database for all the servers in the cluster. And, in case of a complete site switchover, the Tlogs and JMS messages are automatically replicated to the other site with the underlying Data Guard replication.
Both sites are managed with a single Administration Server that resides in one of the two sites. A unique Oracle WebLogic Server Administration Console is used to configure and monitor servers running on both sites. The WebLogic Server infrastructure is responsible for copying configuration changes to all the different domain directories used in the domain.
If an Administration Server fails, the same considerations that apply to an Administration Server failure in a single data center topology apply to a multi-data Center active-active stretch cluster topology. Address node failures using the standard failover procedures described in Failing Over or Failing Back Administration Server in High Availability Guide (that is restarting the Administration Server in another node that resides in the same data center pointing to the shared storage that hosted the Administration Server domain directory). Also, deploy the appropriate backup and restore procedures to make regular copies of the Administration Server domain directory. If there is a failure that affects the site hosting the Administration Server (involving all nodes), you need to restart the server in a different site. To do so, use the existing storage replication technology to copy the Administration Server domain directory available in the failover site. Restore the server/ directory (including both the domain and applications directory) in the failover site so that the exact same domain directory structure is created for the Administration Server domain directory as in the original site. Restart Node Manager in the node where the Administration Server is restored.

Likely, the Administration Server is failed over to a different subnet requiring the use of a different virtual IP (VIP) that is reachable by other nodes. Make the appropriate changes in the hostname resolution system in this subnet so that this VIP maps to the original Virtual Hostname that the Administration Server used as the listen address in Site1. For example, in Site1, ADMINHOSTVHN1 maps to 10.10.10.1, while in Site2 either local /etc/hosts or DNS server has to be updated so that ADMINHOSTVHN1 maps to 20.20.20.1. All servers use ADMINHOSTVHN1 as the address to reach the Administration Server. If the Administration Server is front ended with an Oracle HTTP Server and LBR, clients are agnostic to this change. If clients directly access the Administration Server listen hostname, they must be updated in their DNS resolution also.

Also, if host name verification is enabled for the Administration Server, update the appropriate trust stores and key stores with new certificates. Use the instructions in the Oracle Fusion Middleware Enterprise Deployment Guide for Oracle SOA Suite.

Verify that the Administration Server is working properly by accessing both the Oracle WebLogic Server Administration Console and Oracle Enterprise Manager Fusion Middleware Control.
Servers that are remote to the Administration Server take longer to restart than the servers that are collocated. The reason is that all the communications with the Administration Server (for retrieving the domain configuration upon start) and initial connection pool creation and database access is affected by the latency across sites. See Administration Server High Availability Topology in High Availability Guide.
Automatic Server or Service Migration across sites is not recommended unless a database is used for JMS and TLog persistence, otherwise a constant replica of the appropriate persistent stores must be set up between the sites.
Oracle recommends using Service Migration instead of Server migration. Server migration uses Virtual IPs and in most scenarios the Virtual IPs used in one site are invalid for migration to the other. It requires additional intervention to enable a listen address, which is initially available in Site1 in Site2 and viceversa. This intervention can be automated in pre-migration scripts, but the RTO increases compared to a standard automated server migration (taking place in the scope of single data center). When compared, Service Migration does not require virtual IPs and the RTO is much better than in Server migration.
JMS and transaction recovery across sites is handled by using service or server migration inside a WebLogic Server stretch cluster. As explained in the Common Design Considerations: WebLogic Server section, Oracle recommends using Service Migration rather than Server Migration. For server or service migration of JMS or JTA, Oracle recommends using database leasing on a highly available database. When configured with consensus non-database leasing, servers in the stretch cluster could fail to reboot and require the entire domain to be restarted when the environment experiences network partition failure.
Zero Downtime Patching in an active-active stretch cluster topology orchestrates the updates to all servers in the stretch cluster across both sites. See Zero Downtime Patching.
In an active-active stretch cluster topology, only stretch the WebLogic Server cluster across the two sites. Coherence should have a Coherence cluster on each site using federated caching to replicate data across the two active sites. When a Coherence cluster is stretched across sites, it is susceptible to split brain.
Oracle Site Guard works for active-passive topology or components only. In an active-active stretch cluster topology, Oracle Site Guard can only failover/switchover the database.

Table 6-1 lists the recommended settings to configure database leasing in a stretch cluster topology. See the following MBean descriptions in MBean Reference for Oracle WebLogic Server:

Table 6-1 Recommended Settings for DataBase Leasing in a Stretch Cluster

Configuration Property	MBean/Command	Description	Recommended Setting
DatabaseLeasingBasisConnectionRetryCount	ClusterMBean	The maximum number of times Database Leasing tries to obtain a valid connection from the Data Source.	5 (Default value is 1)
DatabaseLeasingBasisConnectionRetryDelay	ClusterMBean	The length of time, in milliseconds, Database Leasing waits before attempting to obtain a new connection from the Data Source when a connection has failed.	2000 (Default value is 1000)
TestConnectionOnReserve	JDBCConnectionPoolParamsBean	Enables WebLogic Server to test a connection before giving it to a client.	Enabled. For leasing the Data Source. The servers remain in a running state during switchover.
-Dweblogic.cluster.jta.SingletonMasterRetryCount	Server start up command	Specifies the retry count for the singleton master to get elected and open its listen ports. The singleton master might not be immediately available on the first try to deactivate JTA.	4 (The default value is 20) Because DatabaseLeasingBasisConnectionRetryCount is set to 5, this property can be decreased to 4. This setting can reduce the time cost during database server booting.

Figure 6-1 and Figure 6-2 represent results found during benchmark testing that show the degradation observed in the overall system throughput (both sites working together) for different latencies. Figure 6-1 shows that for a latency of around 20-milliseconds round-trip time (RTT), the throughput decreases by almost 25%. Figure 6-2 shows the additional time (msecs) consumed for deploying SOA composites with increasing latencies (RTT in msecs.) between sites in a SOA Active-Active stretch cluster (as compared to a deployment with all servers and database in the same site).

Considering the data provided and the performance penalties observed in many tests, Oracle recommends not to exceed 10 ms of latency (RTT) for active-active stretch cluster topologies when the latency affects communication between the two sites.

Figure 6-1 Throughput Degradation With Latency in a Stretch Cluster

Description of "Figure 6-1 Throughput Degradation With Latency in a Stretch Cluster"

Figure 6-2 Deployment Delay Vs. Latency in a Stretch Cluster

Description of "Figure 6-2 Deployment Delay Vs. Latency in a Stretch Cluster"

Active-Active Stretch Cluster with an Active-Passive Database Tier Failure Scenarios

Learn how the Continuous Availability features are used in each of the different failure scenarios for an active-active stretch cluster topology.

Table 6-2 describes the different failure scenarios and how each Continuous Availability feature applies. For an explanation of the different failure scenarios, see Potential Failure Scenarios.

Table 6-2 Active-Active Stretch Cluster and Active/Passive Database Tier Failure Scenarios

Continuous Availability Features	Complete Site Failure	Partial Site/Mid-Tier Failure (WebLogic Server/Coherence/OTD)	Maintenance Outage	Network Partition Failure
Transaction Recovery Across Sites	Oracle Site Guard integrates with Oracle Data Guard broker to perform database failover/switchover. Site Guard calls Oracle Data Guard broker to perform the failover and Site Guard switches the roles from primary to secondary. JMS and Tlog persistent stores are replicated to database on Site 2. Migrates servers and services using WebLogic Server server and service migration features. Oracle recommends using Service Migration.	Either service migration occurs so that transactions can be recovered on an available server in the stretch cluster, or server migration occurs and transactions are recovered on the migrated server. Oracle recommends using Service Migration.	Application infrastructure in Site 1 is shutdown gracefully so that the servers can complete work before shutting down. Oracle Site Guard integrates with Oracle Data Guard to orchestrate database and application infrastructure switchover. JMS and Tlog persistent stores are replicated to database on Site 2. Either service migration occurs so that transactions can be recovered on an available server in the stretch cluster, or server migration occurs and transactions are recovered on the migrated server. Oracle recommends using Service Migration.	The site that is collocated with the database continues processing transactions. The site that is remote to the database gets connection exceptions and transactions fail. Site 2 cannot communicate to the database so transactions fail. If there is intra server transaction participation, transactions will fail and rollback or retry commit depending on the transaction state. When communication resumes, transactions eventually commit/rollback.
Oracle Traffic Director	Oracle Traffic Director on Site 2 continues to route traffic to the servers on its site.	Oracle Traffic Director on Site 2 continues to route traffic to the servers on its site.	Oracle Traffic Director on Site 1 is stopped. Oracle Traffic Director on Site 2 continues to route traffic to the servers on its site.	Oracle Traffic Director on both Sites 1 and 2 continue to route traffic to the servers on their respective sites.
Oracle Site Guard	Oracle Site Guard orchestrates database failover	No-op.	Oracle Site Guard orchestrates database switchover.	No-op.
Zero Downtime Patching	The behavior is dependent on the ZDT rollback setting. If rollback is enabled, then the workflow is rolled back. If rollback is not enabled, then ZDT rollout stops and waits for manual intervention from the end user.	The behavior is dependent on the ZDT rollback setting. If rollback is enabled, then the workflow is rolled back. If rollout is not enabled, then ZDT rollout stops and waits for manual intervention from the end user.	The behavior is dependent on the ZDT rollback setting. If rollback is enabled, then the workflow is rolled back. If rollback is not enabled, then ZDT rollout stops and waits for manual intervention from the end user.	The behavior is dependent on the ZDT rollback setting. If rollback is enabled, then the workflow is rolled back. If rollback is not enabled, then ZDT rollout stops and waits for manual intervention from the end user.
Coherence Federated Caching	Because replication is asynchronous, the cache data eventually becomes consistent either through Coherence Hot Cache or Read-Through cache, or when the other site comes back up.	Because replication is asynchronous, the cache data eventually becomes consistent either through Coherence Hot Cache or Read-Through cache, or when the other site comes back up.	Because replication is asynchronous, the cache data eventually becomes consistent either through Coherence Hot Cache or Read-Through cache, or when the other site comes back up.	Cache data eventually becomes consistent either through Coherence Hot Cache or Read-Through cache, or when the network connectivity is re-established between the two sites.