Configure Additional Optimizations for FMW Stretched Clusters
While not all are mandatory for this topology, they enhance performance and improve the behavior of certain features and scenarios.
Configure Timeouts in Oracle WebLogic Server (WLS)
There are timeout limits specified for transactions in the database, for transaction branches, Enterprise JavaBean (EJB) method invocations, web services, and so on. The Oracle Fusion Middleware stretched cluster deployments are especially sensitive to timeout settings because multiple operations need to access the database in a different location. Configure the timeouts such that:
- They account for the latency in the system.
You may need to increase the timeouts in these types of systems due to the latencies involved. In the stretched cluster model, domain settings are shared for both sites; hence, it is required to use timeouts that account for the worst-case scenario.
- They expire properly in the chain of invocations across different
layers.
Timeouts must be configured in the different layers of the system so that the appropriate embracing tiers behave properly. For example, if the database timeout is set to a value lower than the global WLS timeout, a transaction ID may be “removed” from the database before the work on other branches completes.
These are the main timeout parameters in different layers:
- Global transaction timeout
The Oracle WebLogic Server global transaction timeout determines the maximum duration (in seconds) that a distributed transaction (a global transaction) is allowed to remain active before it is automatically rolled back by Oracle WebLogic Server. Configure the global transaction timeout in the WebLogic Remote Consoleby selecting JTA (Java Transaction API) in the Services section of the Edit Tree and specifying Timeout Seconds.
- XA transaction timeout
The XA transaction timeout in the XA data sources is used in Oracle WebLogic Server to set a transaction branch timeout for data sources. By default, this value is set to 0 (zero) , so it uses the WebLogic global transaction timeout. But you may want to set it if you have long-running transactions that exceed the default timeout value on the XA resource. This value is configured for each XA data source, in the Transaction tab.
- Distributed lock timeout
The distributed lock timeout in the database specifies the amount of time (in seconds) for distributed transactions to wait for locked resources. You can modify the parameter (
distributed_lock_timeout) with the appropriateSQL ALTERstatements.
Set the database’s distributed locking timeout long enough for the slowest transaction, considering any network delays between sites. After that, set the XA Data Source and Global Transaction Timeouts to lower values by using the following rule:
distributed_lock_timeout >= XA DS Timeout >= Global Transaction TimeoutApplications may use additional timeouts parameters. For example, Oracle SOA and BPEL have the following additional timeout parameters you must consider:
- syncMaxWaitTime
The syncMaxWaitTime is the maximum time a synchronous client waits for a response. This setting defines how long a request and response operation can take before it times out. If the BPEL process does not get a reply within this time, the activity fails. To change this value in Oracle Enterprise Manager Fusion Middleware Control:
- Right-click soa-infra and then select SOA Administration.
- Select BPEL Properties.
- Select More BPEL Configuration Properties.
- Update the syncMaxWaitTime value (in seconds).
- Timeouts for EJBs
When the BPEL EJBs methods are involved, this specifies the transaction timeout in seconds (default is 300 for all BPEL EJBs). Modify the timeout as follows:
- In the WebLogic Remote Console Monitoring Tree, select Deployments and then select Application Management.
- Expand soa_infra application and then expand Configuration and Subdeployment.
- Expand the module and select the specific BPEL EJB.
Global Transaction Timeout > BPEL EJB's transaction timeout > syncMaxWaitTime - Web services clients timeouts
The web services client must be configured with a timeout that is permissive enough with worst latency. For example, if a call takes 3 seconds through site 1 but takes 10 seconds through site 2, the timeout should be set to at least 10 seconds to handle the slower site.
- BPEL processes timeouts
If the BPEL process is using specific request-reply (synchronous) and in-only receive (asynchronous) timeouts, you must define them based on the worst-case scenario: when the call goes to the site with the highest database latency. These timeout settings are defined within the BPEL process and cannot be set differently for each site. For more information on configuring events and timeouts in BPEL processes, refer to the Oracle Fusion Middleware Developer's Guide for Oracle SOA Suite. See the Explore More section.
Configure Session Replication
Some applications (web applications, consoles such as Oracle SOA composer, BPM composer, BPM workspace, and so on) make intensive use of HTTP session objects.
One of the advantages of Oracle Fusion Middleware (FMW) stretched cluster deployment is the ability to provide seamless failover between sites. However, session replication across regions may cause performance degradation in the system. Oracle recommends defining two different replication groups (one for each region) to minimize the replication across the two sites.
To configure session replication groups,
- In the WebLogic Remote Console Edit Tree, select Environment, then Servers.
- Select Server Name then select the Cluster tab.
- For each server in region 1, enter the same Replication Group name (for example, Region1RepGroup).
- Repeat the steps for the servers in region 2, using a common name for all of them but different from the one used in region 1 (for example, Region2RepGroup).
Note that using replication groups is a best effort to replicate the state only to servers in the same site but is not a deterministic method. If one single server is available in one site, and others are available in the other, replication will occur across the regions and will continue for that session even if servers come back online in the same site.
Configure In-Place Restart for JMS
The persistent stores are critical for the proper functioning of Oracle WebLogic Server (WLS) servers. With in-place restart, if the persistent store encounters an error, the JMS service will try to restart on the same server before attempting to migrate to another server. This method is faster than migrating the entire JMS service to another server and can often resolve issues quickly.
To configure in-place restart for JMS persistent stores, the related JMS server and persistent store must be targeted to a migratable target. This migratable target must use Restart on Failure, which is not enabled by default. To configure this property, follow these steps:
- Connect to the WebLogic Remote Console
- In the Edit Tree, select Environment and then Migratable Targets.
- Select a migratable target and enable Restart On Failure.
- Select Save and commit the changes.
Configure the Oracle FMW SOA Suite JMS Adapter
Oracle recommends using the cluster syntax (for example:
cluster:t3s://cluster_name) to simplify the configuration. This
approach simplifies the configuration by allowing the adapter to dynamically
retrieve the current list of active servers within the cluster, ensuring efficient
JNDI context retrieval.
If the system uses the JMS adapter intensively and with large payloads, it is recommended to configure the JNDI URL to include only the local servers within each region. This means specifying servers in region 1 for configurations in region 1, and servers in region 2 for configurations in region 2. This setup ensures site context affinity, optimizing performance and reducing latency.
Configure the Oracle FMW SOA Suite File Adapter
This mechanism guarantees that the same file is processed only by one server at a time, and that two adapter instances will not write to the same file concurrently.
In the Oracle Fusion Middleware (FMW) stretched cluster topology, each site operates independently, processing files using its own dedicated shared storage. However, by default, both sites utilize the same data source, schema, and tables for file locking and mutex mechanisms.
In outbound operations, utilizing a shared database table is appropriate because a unique sequence is employed as a mutex to prevent concurrent write operations from overwriting files.
However, “under processing” scenarios can arise for inbound operations. File Adapter instances in either site can mark a file as "blocked." If the same file name exists in both sites, this mechanism can block its processing in both locations, but the file will be consumed in only one of them. To avoid these situations, there are multiple alternatives:
- Implement site-specific file naming conventions. Use unique identifiers into file names based on the site, reducing the likelihood of name collisions and subsequent blocking.
- Configure separate schemas for each region for the File adapter lock and mutex tables, to ensure that file locking and mutex mechanisms operate independently, preventing cross-site blocking.
- Use a separate database for the File adapter lock and mutex tables to ensure that file processing metadata is managed separately.
To configure the File Adapter to use different databases or separate schemas in each region, you must create the appropriate schema owner and tables, and establish a new data source. The servers in region 1 will continue using the original data source, only the deployment plan in the region 2 will be modified to use the new data source.
The servers in region 1 will use the original data source while the servers in region 2 will use the new one.
Configure Oracle Fusion Middleware Oracle SOA Suite In-Memory SOA
This improves performance and scalability for these business processes as read and write operations are performed out of the cache.
The BPEL state information is stored to and pulled from the Oracle Coherence cache. The process state gets written to the database only when faulted, or at regular intervals using a write-behind thread.
In-memory SOA is not supported in the Oracle Fusion Middleware stretched cluster topology, where the usage of the coherence cache must be minimal, as it is very sensitive to delays.
Configure Tuning WebLogic Database Leasing
The database leasing is a core mechanism used to support Automatic Service Migration, particularly for clustered singleton services such as JMS servers or the JTA Transaction Recovery Service. It is used to manage and coordinate the ownership of these singleton services across multiple servers in a cluster. This mechanism uses a database as a central point to reliably track and control which server in a cluster "owns" or manages a specific singleton service at any given time. This is accomplished by storing a lease record in the database, which is periodically updated (renewed) by the owning server. See Leasing for Migratable Services in Administering Clusters for Oracle WebLogic Server.
The configuration for the GridLink data sources provided earlier in this guide guarantees automatic reconnections when a database failover or a switchover takes place. However, all the servers configured with database leasing or with JDBC persistent stores can shut down during a database outage, switchover or failover. When a critical subsystem (like the JTA service) fails, the server is set to the FAILED state, and it is automatically restarted by the node manager.
The time taken to go to the FAILED state is variable; it depends on when the relevant checks are triggered, and it can happen for various reasons:
- If the server is not able to update the leasing table after the total number of retries.
- If the server can’t access the JTA JDBC persistent store after various retries and in-place restarts.
A database switchover can take a few minutes. The automatic restart of WLS servers due to loss of access to the JTA JDBC persistent store takes longer to happen, approximately 8-9 minutes, so it is not triggered during a typical switchover or failover of the database. But a server can transition to FAILED state due to losing a lease in less than 2 minutes with the default leasing configuration. Hence, a Oracle WebLogic Server restart can be automatically triggered during a Oracle Data Guard switchover or failover if that WebLogic server is using database leasing.
There are two parameters to improve the resiliency to database outages for
servers using database leasing. These parameters are
database-leasing-basis-connection-retry-count (1 retry by default)
and database-leasing-basis-connection-retry-delay (1 second by
default). Increasing these parameters prolongs the time before a lost lease error
occurs. This allows the WebLogic servers to tolerate longer database outages without
automatically restarting; they simply reconnect to the database once it becomes
available again. Although connection attempts will fail while the database is down, the
WebLogic servers will not be restarted.
Hence, in a stretched cluster, it is recommended to increment the values of the Database Leasing Connection Retry Count and Timeout to make the WebLogic server more resilient to the database outages. To change these properties, use WLST commands. For example:
$ORACLE_COMMON_HOME/common/bin/wlst.sh
connect('weblogic','password','t3s://ADMINVHN:9002')
edit()
startEdit()
cd('/Clusters/' + 'My_Cluster')
cmo.setDatabaseLeasingBasisConnectionRetryCount(10)
cmo.setDatabaseLeasingBasisConnectionRetryDelay(10000L)
save()
activate()
disconnect()
exit()Tip:
When a server goes to a FAILED state, the node manager attempts two restarts for the server by default. If the server can’t start up correctly, it is marked as FAILED_NOT_RESTARTABLE. You can tune the number of restarts in the server's Health tab. Use the parameter Max Restarts Within Interval to define the number of times that the node manager can restart the server within the interval specified in Restart Interval.Configure Coherence
For example, Oracle SOA Suite uses Oracle Coherence for propagating composite deployments and metadata (MDS) updates across a cluster.
Oracle Coherence supports both multicast and unicast communication for cluster member discovery and messaging. Unicast is especially suitable in environments where multicast is unavailable or unsupported, such as across multiple data centers or cloud regions. Using the well known addresses (WKA) feature, cluster members can discover and join the cluster without relying on multicast. When WKA is configured, all multicast communication is disabled. By default, Oracle Fusion Middleware products utilize unicast with a predefined WKA list for Coherence.
Oracle Coherence is sensitive to delays during cluster formation and in responding to heartbeats from
cluster members. However, recent versions have enhanced tolerance to communication
delays through features like allowable variance. The allowable
variance defines the permissible timing variations in network
communications within a Coherence cluster. This configurable parameter
(maximum-time-variance), defaulting to 16 ms, sets the maximum
allowed latency for round-trip time (RTT) UDP communications. Coherence dynamically
adjusts this value in response to observed network latencies or inconsistencies,
maintaining cluster stability and performance by accommodating greater deviations in
expected timing. The message Increasing allowable variance to XX
indicates such an adjustment. See Operational Configuration
Elements in Developing Applications with Oracle
Coherence.
With this improvement in Oracle Coherence, no errors are reported on cluster formation if the latency remains within the recommended limits for the FMW stretched cluster topology (<10ms RTT). But still, long delays on cluster heartbeats can cause contention in deployments and bad reaction times to outages.
If you encounter errors in the Coherence cluster formation or operations, then you can adjust the following Coherence networking timeout parameters:
<multicast-listener><join-timeout-milliseconds>. Despite being originally designed for multicast configurations, join-timeout impact in unicast as well. It determines how long the initial node will take to form a cluster, and after that how long each node will spend looking for the cluster before trying to form their own (if they are on the WKA list). The default value is 3000, on an unreliable network you may need to increase this value, to try for a longer period of time to find a cluster before starting a new one.<packet-publisher><packet-delivery><resend-milliseconds>. The packet resend interval specifies the minimum amount of time, in milliseconds, that the packet publisher waits for a corresponding ACK packet, before resending a packet. This should be a few multiples of the RTT, 10x should be fine. The default value is 200.<packet-publisher><packet-delivery><timeout-milliseconds>. For packets which require confirmation, specifies the maximum amount of time, in milliseconds, that a packet is resent. After this timeout expires Coherence makes a determination if the recipient is to be considered terminated. The default value of 5 min should be enough, but you can increase it to avoid timeouts when joining the cluster.<packet-publisher><packet-delivery><heartbeat-milliseconds>This parameter is the heartbeat interval for the tcp-ring death detection. The default heartbeat value is 1 second. You can increase the value to alleviate the network traffic, but this also prolongs detection of failed members.<tcp-ring-listener><ip-timeout>and<ip-attempts>. These are the amount of time and attempts before determining that a computer that is hosting cluster members has become unreachable. Ip-timeout should be in the area of 10x the RTT or greater, with a minimum of 5s.<cluster-config><tcp-ring-listener><enabled>. You can even disable tcp-ring death detection to alleviate network traffic, but it also makes the detection of failed members take longer. To disable death detection, set it to false. If disabled, a cluster member uses the packet publisher's resend timeout interval to determine that another member has stopped responding to UDP packets (by default set to 5 minutes.)
To modify the default configuration settings, use a Coherence override file.
Create a file named custom-coherence-override-<name>.xml and
ensure that it is consistent and accessible to all servers within the cluster. Specify
this file in the java property tangosol.coherence.override. You can set
it in the EXTRA_JAVA_PROPERTIES java property in the
setUserOverrides.sh file. For example:
EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES} -Dtangosol.coherence.override=/u01/oracle/config/coherence_custom/custom-coherence-override.xml"Example of coherence override file to tune parameters:
<?xml version='1.0'?>
<!--
This is a custom operational configuration override
-->
<coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config coherence-operational-config.xsd"
>
<cluster-config>
<multicast-listener>
<join-timeout-milliseconds>5000</join-timeout-milliseconds>
</multicast-listener>
<tcp-ring-listener>
<ip-timeout>20s</ip-timeout>
<ip-attempts>3</ip-attempts>
</tcp-ring-listener>
<packet-publisher>
<packet-delivery>
<resend-milliseconds>200</resend-milliseconds>
<timeout-milliseconds>650000</timeout-milliseconds>
</packet-delivery>
</packet-publisher>
</cluster-config>
</coherence>
Example of coherence override file to tune the tcp-ring interval:
<?xml version='1.0'?>
<coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config coherence-operational-config.xsd">
<cluster-config>
<packet-publisher>
<packet-delivery>
<heartbeat-milliseconds>5000</heartbeat-milliseconds>
</packet-delivery>
</packet-publisher>
</cluster-config>
</coherence>Example of coherence override file to disable the tcp-ring death-detection:
<?xml version='1.0'?>
<coherence xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://xmlns.oracle.com/coherence/coherence-operational-config"
xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-operational-config coherence-operational-config.xsd">
<cluster-config>
<tcp-ring-listener>
<enabled>false</enabled>
</tcp-ring-listener>
</cluster-config>
</coherence>
Configure Net Service Performance
Some operating system default configurations may not be optimized, and it becomes very important to adjust some parameters to make data transfer more efficient. These adjustments become particularly significant when there is high latency between JDBC clients and servers. The servers experiencing the highest latency in database access should guide the configuration of network buffers and Oracle Net Services settings to optimize performance.
One of the key metrics in determining the optimum configuration is the bandwidth-delay product (BDP). It represents the maximum amount of data that can be in transit in a network at any given time. It is calculated by multiplying a network link's capacity (bandwidth) by its round-trip delay time (latency):
BDP = Bandwidth × Round-Trip Time (RTT)- RTT: In OCI, you can get the average RTT between regions from the Inter-region latency page in the OCI console. Alternatively, you can determine the round-trip time (RTT) with commands such as ping from one host to another over a few minutes and use the response times returned.
- Bandwidth: There isn't a guaranteed bandwidth between OCI regions, and OCI doesn't offer a specific OCI bandwidth measurement tool. For precise bandwidth measurements, you can use tools like iPerf to conduct tests. The tested bandwidth between Frankfurt and Amsterdam is around 2–2.5 Gbps.
The TCP socket buffer settings control how many packets are sent at one time over the network. To achieve optimal throughput, it's recommended to set the socket buffer size to at least the BDP. In many cases, setting the buffer size to twice the BDP can further enhance performance, especially in high-latency networks. However, excessively large buffers can lead to increased memory usage. Therefore, it's advisable to set the buffer size within the range of the BDP to three times the BDP:
BDP ≤ Socket Buffer Size ≤ 3 × BDPConfigure IO Buffer Sizes in the Database
SEND_BUF_SIZE parameter on the server side is typically enough.
If the database server is receiving large requests, then also set the
RECV_BUF_SIZE parameter. It is recommended to tune them at the Oracle Net Services level in the database instead of to the Operating System level, so that normal TCP sessions
such as SSH, etc. do not use additional memory.
To configure these parameters for the database server, set the buffer space size in the
listener.ora or sqlnet.ora files.
-
In the
listener.orafile, specify the buffer space parameters for a particular protocol address or for a description. The following is an example of the settings for a typical Oracle RAC listener configuration in a Base Database System in Oracle Cloud Infrastructure (OCI):LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))) LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))) LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))) (…) LISTENER=(DESCRIPTION=(SDU=65535)(SEND_BUF_SIZE=10485760)(RECV_BUF_SIZE=10485760)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))) -
If the parameters are set in the
sqlnet.ora, they apply globally to all the listeners. The following is an example of the settings in thesqlnet.orafile:RECV_BUF_SIZE=10485760 SEND_BUF_SIZE=10485760
Configure IO Buffer Sizes in the Middle Tier Nodes
The Linux operating system uses auto-tuning for both the receive and the sender buffer.
For the receive buffer, the auto-tune is determined by the
/proc/sys/net/ipv4/tcp_moderate_rcvbuf parameter. If the
parameter has a value of 1, then auto-tuning is enabled. With auto-tuning, the
receiver buffer size (and TCP window size) is dynamically updated for each
connection, with a 4 MB maximum buffer sizes.
For the sender buffer, the auto-tuning is also enabled by default. While
the receive buffer auto-tuning can be controlled via the
tcp_moderate_rcvbuf parameter, the send buffer auto-tuning does
not have a direct toggle. Just setting fixed buffer sizes disables the send buffer
auto-tuning.
These auto-tuning processes are governed by several kernel parameters that manage memory allocation for sending and receiving data:
- Per connection buffer sizes
The memory allocation for each TCP connection is defined by two parameters:
/proc/sys/net/ipv4/tcp_rmem, which specifies the memory reserved for TCP receive buffers./proc/sys/net/ipv4/tcp_wmem, which specifies the memory reserved for TCP send buffers.
Both parameters accept three values:
- Minimum Buffer Size: the smallest buffer size allocated for a TCP socket.
- Default Buffer Size: the initial buffer size assigned to a TCP socket upon creation.
- Maximum Buffer Size: the largest buffer size that can be automatically allocated for a TCP socket.
These settings establish the boundaries for auto-tuning mechanisms and help balance memory usage, especially during periods of memory stress.
- Per-application buffer sizes
Applications can request specific buffer sizes, but these requests are subject to limits defined by the following parameters:
/proc/sys/net/core/rmem_max, is the maximum receive window, which sets the upper limit for the receive buffer size that applications can request./proc/sys/net/core/wmem_max, is the maximum Send window, which sets the upper limit for the send buffer size that applications can request.
To adjust IO buffer sizes:
TCP auto-tuning remains enabled with boundaries sized for your network’s bandwidth-delay product, improving throughput without over-sizing system memory usage.
Configure the Session Data Unit Size
Oracle Net Services waits for these units to be filled before sending them across the network. Each of these buffers is a session data unit (SDU). Adjusting the size of the SDU to the amount of data provided to Oracle Net Services can improve performance, network utilization, and memory consumption in an Oracle Fusion Middleware stretched cluster with higher latencies than 5 msecs RTT.
The SDU size can be set from 512 bytes to 2 MB. In Oracle Database 23ai, the default SDU size is 64 KB (65,536 bytes). Earlier database releases set their default SDU size to 8 KB.
The actual SDU size used is negotiated between the client and the server at connect time and is the smaller of the two values. Configuring an SDU size different from the default requires configuring the SDU on both the client and server computers. Oracle recommends setting the SDU to the maximum value possible (64k).
Clients and servers negotiate the SDU at connect time; configuring both sides ensures the intended SDU is used.