B Using WebLogic Server with Oracle RAC

More and more customers are looking for solutions to make their back-end systems more scalable and more available. In response to these requests, WebLogic Server supports the use of Oracle Real Application Clusters (Oracle RAC).

The following sections describe the requirements and configuration tasks for using Oracle Real Application Clusters with WebLogic Server:

Both Oracle RAC and WebLogic Server are complex systems. To use them together requires specific configuration on both systems, as well as clustering software and a shared storage solution. This document describes the configuration required at a high level. For more details about configuring Oracle RAC, your clustering software, your operating system, and your storage solution, see the documentation from the respective vendors.

Overview of Oracle Real Application Clusters

Oracle Real Application Clusters (Oracle RAC) is a software component you can add to a high-availability solution that enables users on multiple machines to access a single database with increased performance. Oracle RAC comprises two or more Oracle database instances running on two or more clustered machines and accessing a shared storage device via cluster technology. To support this architecture, the machines that host the database instances are linked by a high-speed interconnect to form the cluster. The interconnect is a physical network used as a means of communication between the nodes of the cluster. Cluster functionality is provided by the operating system or compatible third party clustering software. Figure B-1 shows an Oracle RAC configuration.

Figure B-1 Oracle Real Application Clusters Configuration

Description of Figure B-1 follows
Description of "Figure B-1 Oracle Real Application Clusters Configuration"

Oracle RAC offers features in the following areas:

  • Scalability

  • Availability

  • Load balancing

  • Failover

Oracle RAC Scalability with WebLogic Server

An Oracle RAC installation appears like a single standard Oracle database and is maintained using the same tools and practices. All the nodes in the cluster execute transactions against the same database and Oracle RAC coordinates each node's access to the shared data to maintain consistency and ensure integrity. You can add nodes to the cluster easily and there is no need to partition data when you add them. This means that you can horizontally scale the database tier as usage and demand grows by adding Oracle RAC nodes, storage, or both. You can then scale WebLogic Server by adding a data source that maps to the new node.

Oracle RAC Availability with WebLogic Server

Depending upon your configuration, when an Oracle RAC node or instance fails, session requests are redirected to another node in the cluster either by WebLogic Server or by the Oracle Thin driver. Note that with WebLogic Server, Oracle RAC does not provide failover for database connections. In flight transactions usually roll back when the database is the transaction manager. When the WebLogic server is the transaction manager, in flight transactions are failed over in the sense that they are driven to completion or rolled back, based on the state of the transaction at the time of the failure.

Oracle RAC Load Balancing with WebLogic Server

If your application requires load balancing across Oracle RAC nodes, WebLogic Server supports this capability through use of JDBC multi data sources with Oracle RAC nodes. (A multi data source is an abstraction around a group of data sources that provides load balancing or failover processing at the time of connection requests, between the data sources associated with the multi data source.) The data sources that form a multi data source are accessed using a round-robin scheme. When switching connections, WebLogic Server selects a connection from the next data source in the order listed. For more information about using multi data sources with Oracle RAC, see Using Multi Data Sources with Oracle RAC.

For Oracle RAC deployment with Oracle Database 10g (10.2) and earlier, XA is supported only when using multi data sources. Therefore, load balancing for XA is supported only when using multi data sources. In a configuration without a multi data source, WebLogic Server relies on the connect-time failover feature provided by the Oracle Thin driver to work with Oracle RAC. As described in Oracle's Technical Note 235118.1, the Oracle Thin driver cannot guarantee that a transaction is initiated and concluded on the same Oracle RAC instance when the driver is configured for load balancing. As Oracle RAC requires that all database operations inside a global transaction be routed to the same Oracle instance, this known limitation means that you cannot use driver-level load balancing when using XA with Oracle RAC and therefore you cannot use a primary/primary Oracle RAC configuration.

Oracle Database 11g (11.1) includes support for Oracle RAC 11g as a high-availability database solution for WebLogic Server. However, in this release, WebLogic Server does not support new features in Oracle RAC 11g that support XA with load balancing. Instead, WebLogic Server will continue to use its well-proven integration architecture using multi data sources for XA with load balancing.

Oracle RAC Failover with WebLogic Server

Although Oracle RAC offers JDBC connect-time failover features, for most configurations, Oracle recommends using WebLogic JDBC multi data sources to handle failover instead. While connect-time failover does not provide the ability to pre-create connections to alternate Oracle RAC nodes, multi data sources have multiple connections available at all times to handle failover. For more information see Using Multi Data Sources with Oracle RAC.

Note:

Transparent Application Failover (TAF) is not supported for any WLS data source. TAF, as delivered via JDBC is currently not transparent. It is documented to affect some ongoing query results and PreparedStatements in unpredictable and unrecoverable ways. TAF JDBC requires specific recovery code at the application level and affects the integrity of statements that WebLogic might be caching.

Environment

When using WebLogic Server with Oracle RAC, consider the following requirements:

Hardware Requirements

A typical WebLogic Server/Oracle RAC system includes a WebLogic Server cluster, an Oracle RAC cluster, and hardware for shared storage.

WebLogic Server Cluster

The WebLogic Server cluster can be configured in many ways and with various hardware options. See Using Clusters for Oracle WebLogic Server for more details about configuring a WebLogic Server cluster.

Oracle RAC Cluster

For the latest hardware requirements for Oracle RAC, see the Oracle RAC documentation. However, to use Oracle RAC with WebLogic Server, you must run Oracle RAC instances on robust, production-quality hardware. The Oracle RAC configuration must deliver database processing performance appropriate for reasonably-anticipated application load requirements. Unusual database response delays can lead to unexpected behavior during database failover scenarios.

Shared Storage

In an Oracle RAC configuration, all data files, control files, and parameter files are shared for use by all Oracle RAC instances. An HA storage solution that uses one of the following architectures is recommended:

  • Direct Attached Storage (DAS), such as a dual ported disk array or a Storage Area Network (SAN)

  • Network Attached Storage (NAS)

For a complete list of supported storage solutions, see your Oracle documentation.

Software Requirements

To use WebLogic Server with Oracle RAC, you must install the following software on each Oracle RAC node:

  • Operating system patches required to support Oracle RAC. See the release notes from Oracle for details.

  • Oracle 10g or Oracle 11g database management system

  • Clustering software for your operating system. See the Oracle documentation for supported clustering software and cluster configurations.

  • Shared storage software, such as Veritas Cluster File System. Note that some clustering software includes a file storage solution, in which case additional shared storage software is not required.

    Note:

    See the Oracle Fusion Middleware Supported System Configurations page at http://www.oracle.com/technology/software/products/ias/files/fusion_certification.html for the latest WebLogic Server hardware platform and operating system support, and for the Oracle RAC versions supported by WebLogic Server versions and service packs. See the Oracle documentation for hardware and software requirements required for running the Oracle RAC software.

Configuration Considerations for Oracle

Once you have installed and configured Oracle RAC, you must configure the listener process for each Oracle RAC instance as described in the sections that follow. For information about installing and configuring your operating system and the Oracle software for Oracle RAC nodes see the Oracle documentation.

Configuring the Listener Process for Each Oracle RAC Instance

For Oracle RAC, the listener process establishes a communication pathway between a user process and an Oracle instance. When you use Oracle RAC with WebLogic Server, the user process is typically a JDBC data source.

When a data source is created, it attempts to create a pool of database connections for applications to borrow. If a pooled database connection becomes inoperative or if the data source is configured to grow in capacity, the data source attempts to create additional database connections up to the maximum specified in the configuration file. In all of these instances, the Oracle listener process handles the connection request on the Oracle RAC instance.

Figure B-2 shows the Oracle listener process functionality.

Figure B-2 Oracle Listener Process Functionality

Description of Figure B-2 follows
Description of "Figure B-2 Oracle Listener Process Functionality"

To enable this functionality, you have two options:

  • Use local listeners. Configure the listener process for each Oracle RAC instance in the Oracle cluster. WLS requires that you configure a local listener on each Oracle RAC instance. Each database instance should be configured to register only with its local listener.

    Oracle instances can be configured to register with the listener statically in the listener.ora file or registered dynamically using the instance initialization parameter local_listener, or both. Oracle recommends using dynamic registration.

    A listener can start either a shared dispatcher process or a dedicated process. When using with WebLogic Server, Oracle recommends using dedicated processes.

  • Use remote listeners. WLS requires that you specify both the SERVICE_NAME and the INSTANCE_NAME in the JDBC URL for the data sources in the multi data source. See Configuring Multi Data Sources When Remote Listeners are Enabled or Disabled.

Configuring Multi Data Sources When Remote Listeners are Enabled or Disabled

If the server-side load balancing feature has been enabled for the Oracle RAC backend (using remote_listeners), the JDBC URL that you use in the data sources of a multi data source configuration should include the INSTANCE_NAME. For example, the URL can be specified in the following format:

jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=host-vip)(PORT=1521))
(CONNECT_DATA=(SERVICE_NAME=dbservice)(INSTANCE_NAME=inst1)))

If specifying the INSTANCE_NAME in the URL is not possible, remote listeners must be disabled. To disable remote listeners, delete any listed remote listeners in spfile.ora on each Oracle RAC node. For example:

*.remote_listener="

In this case, the recommended URL that you use in the data sources of a multi data source configuration is:

jdbc:oracle:thin:@host-vip:port/dbservice

or

jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=host-vip)(PORT=1521))(
CONNECT_DATA=(SERVICE_NAME=dbservice)))

Configuration Options in WebLogic Server with Oracle RAC

When using WebLogic Server with Oracle RAC, you must configure your WebLogic Domain so that it can interact with Oracle RAC instances and so that it performs as expected. The following sections describe configuration options and requirements:

Choosing a WebLogic Server Configuration for Use with Oracle RAC

WebLogic Server supports several configuration options for using Oracle RAC:

  • To connect to multiple Oracle RAC 10g or Oracle RAC 11g instances when using global transactions (XA), Oracle recommends the use of transaction-aware WebLogic JDBC multi data sources, which support failover and load balancing, to connect to the Oracle RAC nodes. For more information see Using Multi Data Sources with Global Transactions.

  • To connect to multiple Oracle RAC 10g or Oracle RAC 11g instances when not using XA, Oracle recommends the use of (non-transaction-aware) multi data sources to connect to the Oracle RAC nodes. Use the standard multi data source configuration, which supports failover and load balancing. For more information see Using Multi Data Sources without Global Transactions.

  • To connect to multiple Oracle RAC nodes when multi data sources are not an option and when not using XA, use Oracle RAC 10g or Oracle RAC 11g with connect-time failover. Note that load balancing is not supported in this configuration. For more information see Using Connect-Time Failover without Global Transactions.

The following table may help you as you try to determine which configuration is right for your particular application:

Table B-1 Choosing Configurations to Use with Oracle RAC

Requires Load Balancing? Requires Failover? Requires Global Transactions (XA)? Requires JDBC Store? Oracle RAC Services See...

Yes

Yes

Yes

No

No

Using Multi Data Sources with Global Transactions

Yes

Yes

No

Yes

No

Using Multi Data Sources without Global Transactions

Yes

Yes

Yes

Yes

Yes

Configuring Connections to Services on Oracle RAC Nodes

No

Yes

No

Yes

Yes

Using Connect-Time Failover without Global Transactions


You can also configure data sources to connect to specific services that are running on Oracle RAC nodes. In this configuration, multi data sources are required. Both Load Balancing and Failover are supported. Both XA and non-XA drivers are provided. For more information, see Configuring Connections to Services on Oracle RAC Nodes.

Required JDBC Drivers

To use WebLogic Server with Oracle RAC, your WebLogic JDBC data sources must use the Oracle JDBC Thin driver 11g to create database connections.

Configuration Considerations for Failover

Consider the following information when configuring for failover.

Multi Data Source-Managed Failover and Load Balancing

Multi data sources offer failover and load balancing for global transactions without the limitations and known issues associated with a data source configuration with connect-time failover. For a description of multi data source failover features, see Multi Data Source Failover Enhancements.

With this configuration, pictured in Figure B-3, you get:

  • Faster failover controlled by the multi data source

  • Automatic failback by the WebLogic Server health monitor

The multi data source handles failover for database connections when an Oracle RAC node becomes unavailable. When WebLogic Server tests a connection and the connection fails, it attempts to recreate the connection. If that attempt fails, the server disables the data source and routes connection requests to other data sources (which correspond to other Oracle RAC nodes) in the multi data source. WebLogic Server periodically tries to recreate the database connections in the disabled data source. When WebLogic Server is successful in recreating the connections, it next re-enables the data source and begins routing connection requests to the data source again. Because of the connection request routing and automatic health checking features, there is minimal delay in satisfying connection requests after a failure compared to when relying on the Oracle Thin driver connect-time failover configuration.

Connect-Time Failover

When multi data sources are not an option, WebLogic Server relies on the connect-time failover feature of the Oracle Thin driver to handle connection failover when an Oracle RAC instance becomes unavailable and a primary/primary configuration is not an option. When WebLogic Server tests a connection and the connection fails, the server replaces it by getting a new connection, and the driver again determines which Oracle RAC instance to use based on instance availability. When a connection fails a connection test, WebLogic Server automatically closes all connections in the data source. WebLogic Server replaces the connection with a new one, relying on the driver to determine to which node it should connect. In this case, the primary Oracle RAC node has failed, so the new connection is to the secondary Oracle node. WebLogic Server tests the new connection before satisfying the request.

Delays During Failover

Occasionally, when one Oracle RAC node fails over to another, there may be a delay before the data associated with a transaction branch in progress on the now failed node is available throughout the cluster. This prevents incomplete transactions from being properly completed, which could further result in data locking in the database. To protect against the potential consequences of such a delay, WebLogic Server provides two configuration attributes that enable XA call retry for Oracle RAC: XARetryDurationSeconds and XARetryIntervalSeconds.

XARetryDurationSeconds controls the period of time during which WebLogic Server will repeatedly retry XA operations such as recover, commit and rollback for pending transactions. XARetryIntervalSeconds controls the frequency of the retry attempts within the established time period.

To enable XA call retries, add a value for XARetryDurationSeconds to all JDBC data sources in your WebLogic domain that connect to an Oracle RAC instance. For example:

<jdbc-data-source xmlns="http://xmlns.oracle.com/weblogic/jdbc-data-source"
  xmlns:sec="http://xmlns.oracle.com/weblogic/security"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:wls="http://xmlns.oracle.com/weblogic"
  xsi:schemaLocation="http://xmlns.oracle.com/weblogic/domain/1.0/domain.xsd">
  <name>oracleRACXAPool</name> 
  ...
  <jdbc-xa-params>
    ...
    <xa-retry-duration-seconds>300</xa-retry-duration-seconds> 
  </jdbc-xa-params>
</jdbc-data-source>

Use the following formula to determine the value for XARetryDurationSeconds:

XARetryDurationSeconds = (longest transaction timeout for transactions that use connections from the data source) + (delay before XIDs are available on all Oracle RAC nodes, typically less than 5 minutes)

For example, if your application sets the longest transaction timeout as 180 seconds, you should set XARetryDurationSeconds to 180 seconds + 300 seconds, for a total of 480 seconds.

Note:

It is generally better to set XARetryDurationSeconds higher than minimally necessary to make sure that all transactions are completed properly. Setting the value higher than minimally required should not affect application performance during normal operations. The additional processing only affects transactions that have been prepared but have failed to complete.

You can also optionally set a value for XARetryIntervalSeconds. This value determines the time between XA retry calls. By default, the value is 60 seconds. Decreasing the value will decrease the amount of time between XA retry attempts. The default value should suffice in most cases.

To enable XARetryDurationSeconds and XARetryIntervalSeconds from the Administration Console, use the following steps:

  1. If you have not already done so, in the Change Center of the Administration Console, click Lock & Edit.

  2. In the Domain Structure tree, expand Services > JDBC, then select Data Sources.

  3. On the Summary of Data Sources page, click the data source name.

  4. Select the Configuration: Connection Pool tab.

  5. Scroll down and click Advanced to show the advanced connection pool options.

  6. Update XA Retry Duration and XA Retry Interval.

  7. Click Save.

Optionally, you can use WebLogic Scripting Tool (WLST) or a JMX program.

Failure Handling Walkthrough for Global Transactions

What happens to in-flight transactions to a database node if that node fails? When the primary Oracle RAC node fails? Does WebLogic Server support transparent failover? To answer these and other questions about how WebLogic Server handles failures, let's walk through the transaction processing steps and describe how a failure would be handled at each stage along the way.

The first stage at which a failure may occur is before the application calls for the transaction to be committed. If a database or Oracle RAC node fails at this stage, the application receives an exception and must get a new connection and make a new attempt at processing the transaction. WebLogic Server does not support transparent failover.

If a failure occurs after the application has called for the transaction to be committed, the handling of any in-flight transaction depends upon whether the PREPARE operation is complete. If the PREPARE operation is not complete, the transaction manager rolls back the transaction and sends the application an exception for the failed transaction. If the PREPARE operation is complete, the transaction manager attempts to drive the in-flight transaction to completion using another node.

If a failure occurs during the COMMIT operation, the transaction manager attempts to retry the COMMIT operation several times. Note that the connection is blocked during these attempts. If the COMMIT operation is not successful during the first set of retry attempts, the application receives an exception. The transaction manager then continues to retry the COMMIT operation periodically until it is successful; if the transaction cannot be completed successfully within the abandon time period, the transaction is driven to completion heuristically.

Using Multi Data Sources with Oracle RAC

To connect WebLogic Server to multiple Oracle RAC nodes using multi data sources, first configure a JDBC data source for each Oracle RAC instance in your Oracle RAC cluster with the Oracle Thin driver. Then configure a multi data source, using either the algorithm for load balancing or the algorithm for failover, and add the data sources to it.

Figure B-3 shows a typical multi data source configuration.

Figure B-3 Multi Data Source Configuration

Description of Figure B-3 follows
Description of "Figure B-3 Multi Data Source Configuration"

You can use the Administration Console or any other means that you prefer to configure your domain, such as the WebLogic Scripting Tool (WLST) or a JMX program. For information about configuring a WebLogic JDBC multi data source see Chapter 4, "Configuring JDBC Multi Data Sources." For information on how to configure the data sources in a multi data source to connect to services running on Oracle RAC nodes, see Configuring Connections to Services on Oracle RAC Nodes.

To use a database connection in this configuration, your applications look up one multi data source on the JNDI tree and then request a connection. The multi data source determines which data source to use to satisfy the connection request based on the algorithm type specified in the configuration (that is, failover or load balancing).

Attributes of a Multi Data Source

The multi data source may have the following attributes, depending on the role of Oracle RAC in your system—load balancing or failover:

  • AlgorithmType="Load-Balancing" or AlgorithmType="Failover"

    With the Load-Balancing option, connection requests are distributed among available data sources; with the High-Availability option, connection requests are served by the first available pool in the list. When a data source becomes defunct, connection requests are served by the next data source in the list.

  • FailoverRequestIfBusy="true"

    With the Failover algorithm, this attribute enables failover when all connections in a data source are in use.

  • TestFrequencySeconds="120"

    This attribute controls the frequency at which WebLogic Server checks the health of data sources previously marked as unhealthy to see if connections can be recreated and if the data source can be re-enabled. For more details see Chapter 4, "Configuring JDBC Multi Data Sources."

    For fast failover of Oracle RAC nodes, set this value to a smaller interval, for example, 10 (seconds).

Using Multi Data Sources with Global Transactions

Note:

Release 11.1 includes support for Oracle RAC 11g as a high-availability database solution for WebLogic Server. However, this release does not support the new Oracle Services features in Oracle RAC 11g that support XA with load balancing. Instead, WebLogic Server will continue to use its well-proven integration architecture using multi data sources for XA with load balancing.

In this configuration, a multi data source "pins" a transaction to one and only one Oracle RAC instance. Individual transactions are load balanced with the initial connection request for the transaction. Failover is handled at the multi data source level when a Oracle RAC instance becomes unavailable. If there is a failure on a Oracle RAC instance before PREPARE, the operation is retried until the retry duration has expired. If there is a failure after PREPARE the transaction is failed over to another instance.

Rules for Data Sources within a Multi Data Source Using Global Transactions

The following rules apply to the XA data sources within a multi data source:

  • All the data sources must be homogeneous. In other words, either all of them must use an XA driver or none of them can use an XA driver.

  • If you choose to specify them, all XA-related attributes must be set to the same values for each data source. The attributes include the following:

    • XARetryDurationSeconds

    • SupportsLocalTransaction

    • KeepXAConnTillTxComplete

    • NeedTxCtxOnClose

    • XAEndOnlyOnce

    • NewXAConnForCommit

    • RollbackLocalTxUponConnClose

    • RecoverOnlyOnce

    • KeepLogicalConnOpenOnRelease

      Note:

      If you are not using XA, Oracle recommends the use of multi data sources for failover and load balancing across Oracle RAC instances, but the XA-specific configuration requirements above do not apply. For more information see Using Multi Data Sources without Global Transactions.

Required Attributes of Data Sources within a Multi Data Source Using Global Transactions

Each data source within the multi data source should have the following attributes:

  • Oracle JDBC Thin driver 11g. For example:

    <url>jdbc:oracle:thin:@lcqsol24:1521:SNRAC1</url> 
    <driver-name>oracle.jdbc.xa.client.OracleXADataSource</driver-name>
    
  • KeepXAConnTillTxComplete="true"

    • Forces the data source to reserve a physical database connection and provide the same connection to an application throughout transaction processing until the distributed transaction is complete.

    • Required for proper transaction processing with Oracle RAC.

  • XARetryDurationSeconds="300"

    • Enables the WebLogic Server transaction manager to retry XA recover, commit, and rollback calls for the specified amount of time.

  • TestConnectionsOnReserve="true"

    • Enables testing of a database connection when an application reserves a connection from the data source. See Test Connections on Reserve to Enable Fail-Over for more details about this attribute.

    • Required to enable failover to another Oracle RAC node.

  • TestTableName="name_of_small_table" The name of the table used to test a physical database connection. For more details about this attribute, see Connection Testing Options for a Data Source.

Sample Configuration Code

Sample configuration code for a multi data source and two associated data sources is shown below.

<jdbc-data-source xmlns="http://xmlns.oracle.com/weblogic/jdbc-data-source"
  xmlns:sec="http://xmlns.oracle.com/weblogic/security"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:wls="http://xmlns.oracle.com/weblogic"
  xsi:schemaLocation="http://xmlns.oracle.com/weblogic/domain/1.0/domain.xsd">
  <name>oracleRACXAPool</name> 
  <jdbc-driver-params>
    <url>jdbc:oracle:thin:@lcqsol24:1521:SNRAC1</url> 
    <driver-name>oracle.jdbc.xa.client.OracleXADataSource</driver-name> 
    <properties>
      <property>
        <name>user</name> 
        <value>wlsqa</value> 
      </property>
    </properties>
    <password-encrypted>{3DES}aP/xScCS8uI=</password-encrypted> 
  </jdbc-driver-params>
  <jdbc-connection-pool-params>
    <test-table-name>SQL SELECT 1 FROM DUAL</test-table-name> 
    <profile-type>0</profile-type> 
  </jdbc-connection-pool-params>
  <jdbc-data-source-params>
    <jndi-name>oracleRACXAJndiName</jndi-name>
    <global-transactions-protocol>TwoPhaseCommit
        </global-transactions-protocol>
  </jdbc-data-source-params>
  <jdbc-xa-params>
    <keep-xa-conn-till-tx-complete>true</keep-xa-conn-till-tx-complete> 
    <xa-end-only-once>true</xa-end-only-once> 
    <xa-set-transaction-timeout>true</xa-set-transaction-timeout> 
    <xa-transaction-timeout>120</xa-transaction-timeout> 
    <xa-retry-duration-seconds>300</xa-retry-duration-seconds> 
  </jdbc-xa-params>
</jdbc-data-source>
<jdbc-data-source xmlns="http://xmlns.oracle.com/weblogic/jdbc-data-source"
  xmlns:sec="http://xmlns.oracle.com/weblogic/security"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:wls="http://xmlns.oracle.com/weblogic"
  xsi:schemaLocation="http://xmlns.oracle.com/weblogic/domain/1.0/domain.xsd">
  <name>oracleRACXAPool2</name> 
  <jdbc-driver-params>
    <url>jdbc:oracle:thin:@lcqsol25:1521:SNRAC2</url> 
    <driver-name>oracle.jdbc.xa.client.OracleXADataSource</driver-name> 
    <properties>
      <property>
        <name>user</name> 
        <value>wlsqa</value> 
      </property>
    </properties>
    <password-encrypted>{3DES}aP/xScCS8uI=</password-encrypted> 
  </jdbc-driver-params>
  <jdbc-connection-pool-params>
    <test-table-name>SQL SELECT 1 FROM DUAL</test-table-name> 
    <profile-type>0</profile-type> 
  </jdbc-connection-pool-params>
  <jdbc-data-source-params>
    <jndi-name>oracleRACXAJndiName2</jndi-name> 
    <global-transactions-protocol>TwoPhaseCommit
        </global-transactions-protocol> 
  </jdbc-data-source-params>
  <jdbc-xa-params>
    <keep-xa-conn-till-tx-complete>true</keep-xa-conn-till-tx-complete> 
    <xa-end-only-once>true</xa-end-only-once> 
    <xa-set-transaction-timeout>true</xa-set-transaction-timeout> 
    <xa-transaction-timeout>120</xa-transaction-timeout> 
    <xa-retry-duration-seconds>300</xa-retry-duration-seconds> 
  </jdbc-xa-params>
</jdbc-data-source>
<jdbc-data-source xmlns="http://xmlns.oracle.com/weblogic/jdbc-data-source"
  xmlns:sec="http://xmlns.oracle.com/weblogic/security"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:wls="http://xmlns.oracle.com/weblogic"
  xsi:schemaLocation="http://xmlns.oracle.com/weblogic/domain/1.0/domain.xsd">
  <name>oracleRACXAMDS</name> 
  <jdbc-data-source-params>
    <jndi-name>oracleRACMDSJndiName</jndi-name> 
    <algorithm-type>Load-Balancing</algorithm-type> 
    <data-source-list>oracleRACXAPool,oracleRACXAPool2</data-source-list> 
  </jdbc-data-source-params>
</jdbc-data-source>

Using Multi Data Sources without Global Transactions

The following sections describe a configuration that uses Oracle RAC with multi data sources in an application that does not require global transactions.

Attributes of Data Sources within a Multi Data Source Not Using Global Transactions

Data sources must have the following attributes:

  • Oracle JDBC Thin driver 11g. For example:

     <url>jdbc:oracle:thin:@lcqsol24:1521:SNRAC1</url> 
     <driver-oracle.jdbc.OracleDriver/driver-name>
    
  • TestConnectionsOnReserve="true"

    • Enables testing of a database connection when an application reserves a connection from the data source. Test Connections on Reserve to Enable Fail-Over for more details about this attribute.

    • Required to enable failover and connection request routing within a multi data source (effectively, failover to another Oracle RAC node).

  • TestTableName="name_of_small_table"

Sample Configuration Code

Sample configuration code for a WebLogic JDBC multi data source and associated data sources is shown below.

<jdbc-data-source xmlns="http://xmlns.oracle.com/weblogic/jdbc-data-source"
  xmlns:sec="http://xmlns.oracle.com/weblogic/security"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:wls="http://xmlns.oracle.com/weblogic"
  xsi:schemaLocation="http://xmlns.oracle.com/weblogic/domain/1.0/domain.xsd">
  <name>jdbcPool</name> 
  <jdbc-driver-params>
    <url>jdbc:oracle:thin:@lcqsol24:1521:snrac1</url> 
    <driver-name>oracle.jdbc.OracleDriver</driver-name> 
    <properties>
      <property>
        <name>user</name> 
        <value>wlsqa</value> 
      </property>
    </properties>
    <password-encrypted>{3DES}aP/xScCS8uI=</password-encrypted> 
  </jdbc-driver-params>
  <jdbc-connection-pool-params>
    <test-connections-on-reserve>true</test-connections-on-reserve> 
    <test-table-name>SQL SELECT 1 FROM DUAL</test-table-name> 
  </jdbc-connection-pool-params>
  <jdbc-data-source-params>
    <jndi-name>jdbcDataSource</jndi-name> 
  </jdbc-data-source-params>
</jdbc-data-source>
<jdbc-data-source xmlns="http://xmlns.oracle.com/weblogic/jdbc-data-source"
  xmlns:sec="http://xmlns.oracle.com/weblogic/security"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:wls="http://xmlns.oracle.com/weblogic"
  xsi:schemaLocation="http://xmlns.oracle.com/weblogic/domain/1.0/domain.xsd">
  <name>jdbcPool2</name> 
  <jdbc-driver-params>
    <url>jdbc:oracle:thin:@lcqsol25:1521:SNRAC2</url> 
    <driver-name>oracle.jdbc.OracleDriver</driver-name> 
    <properties>
      <property>
        <name>user</name> 
        <value>wlsqa</value> 
      </property>
    </properties>
    <password-encrypted>{3DES}aP/xScCS8uI=</password-encrypted> 
  </jdbc-driver-params>
  <jdbc-connection-pool-params>
    <test-connections-on-reserve>true</test-connections-on-reserve> 
    <test-table-name>SQL SELECT 1 FROM DUAL</test-table-name> 
  </jdbc-connection-pool-params>
  <jdbc-data-source-params>
    <jndi-name>jdbcDataSource2</jndi-name> 
    <global-transactions-protocol>OnePhaseCommit
         </global-transactions-protocol> 
  </jdbc-data-source-params>
</jdbc-data-source>
<jdbc-data-source xmlns="http://xmlns.oracle.com/weblogic/jdbc-data-source"
  xmlns:sec="http://xmlns.oracle.com/weblogic/security"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:wls="http://xmlns.oracle.com/weblogic"
  xsi:schemaLocation="http://xmlns.oracle.com/weblogic/domain/1.0/domain.xsd">
  <name>jdbcNonXAMultiPool</name> 
  <jdbc-data-source-params>
    <jndi-name>jdbcDataSource</jndi-name> 
    <algorithm-type>Failover</algorithm-type> 
    <data-source-list>jdbcPool,jdbcPool2</data-source-list> 
    <failover-request-if-busy>true</failover-request-if-busy> 
  </jdbc-data-source-params>
</jdbc-data-source>

Note:

Line breaks added for readability.

Configuring Connections to Services on Oracle RAC Nodes

If you rely on Oracle services in your Oracle RAC cluster for workload management, you must use multi data sources to connect to those services instead of you using a Service ID (SID). A WebLogic Server data source can be configured to connect only to a specific service on a specific Oracle RAC node, providing both workload management and load balancing.

In general, to connect to Oracle RAC services, you need to:

  • Create a multi data source for each service to which you want to connect.

  • Within each multi data source, create one data source for each Oracle RAC node in the cluster on which the service will be configured, whether or not the service will be actively running on each node.

Configuring a Data Source to Connect to a Service, describes special considerations for configuring these data sources. Service Connection Configurations, shows example configurations for either load balancing or workload management.

Configuring a Data Source to Connect to a Service

You configure a data source to connect to a service running on an Oracle RAC node in the same way as you configure any data source (using WLST, the Administration Console, or the Configuration Wizard), with the following exceptions:

  • initial-capacity="0"

    This prevents pool creation failure for inactive pools at WLS startup, and enables WLS to create the data source even if it can't connect to the service on the node. Without setting this option to 0, data source creation will fail and the server may fail to boot normally.

    In the Administration Console, edit the data source after creating it, and set Initial Capacity to 0.

  • Oracle JDBC Thin (or Thin XA) driver for 11g. For example:

    For non-XA:

    driver-name="oracle.jdbc.OracleDriver"
    url="jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=RAC1)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=Service_1)(INSTANCE_NAME=DB_02)))"
    

    If configuring via the Administration Console, select Oracles's Driver (Thin) for RAC Service-Instance connections from the Database Driver drop-down and specify the service in the Service Name field.

    For XA:

    driver-name="oracle.jdbc.xa.client.OracleXADataSource"
    url="jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=RAC1)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=Service1)(INSTANCE_NAME=DBase1)))"
    

    If configuring via the Administration Console, select Oracle's Driver (Thin XA) for RAC Service-Instance connections from the Database Driver drop-down and specify the service in the Service Name field.

    Notes:

    The SERVICE_NAME must be the same for all data sources in a given multi data source.

    Specify a different HOST NAME and/or port for each data source in a given multi data source.

  • When specifying max-capacity (Maximum Capacity in the Administration Console) for the connection pool, you need to consider the connection capacity of each of the Oracle RAC nodes in your configuration, and the total number of connections from all data sources. See Connection Pool Capacity Planning, for more information.

Selecting the Appropriate Multi Data Source Algorithm

For service connection scenarios, Oracle recommends that you configure your multi data source with the Load Balancing algorithm. If the multi data source is configured with the Load Balancing algorithm, its connection pools are used in a round robin fashion. In this case, workload is load-balanced across all of the Oracle RAC nodes on which the associated service is currently active.

If the multi data source is configured with the Failover algorithm, the first data source is used to connect to the service on its associated Oracle RAC node, until a connection attempt fails for any reason (for example, the Oracle RAC node becomes unavailable or there are no more connections available in the data source). At that point, the second data source is used to connect to the service on its associated Oracle RAC node, and so on. In this case, the Oracle RAC node to which the first data source is connected will experience more use than the remaining nodes on which the service is running.

Service Connection Configurations

You can design your configuration to provide

  • Workload management

  • Load balancing

These configurations are described in the next two sections.

Workload Management

In a workload management configuration, each multi data source has one data source configured for a given service on each Oracle RAC node, regardless of whether the service you are connecting to is active or inactive on a given Oracle RAC node. This lets you quickly start an inactive service on a node and create connections to that service should another node become unavailable due to unplanned downtime or scheduled maintenance. It also lets you quickly increase or decrease the available capacity for a given service based on workload demands.

When you start the service on a node, the associated data source detects that the service is now active, and the data source will then start making connections to that node as needed. When you stop a service on a given node, the associated data source can no longer make connections to that node, and will become inactive until the service is restarted on that node.

The WLS data source performs connection testing. This lets the data source adjust to changes in the topology of the Oracle RAC configuration. The data source performs polling to see if its associated service is active or inactive. The connection test fails if the service is no longer available on the Oracle RAC node.

Figure B-4 Workload Management

Description of Figure B-4 follows
Description of "Figure B-4 Workload Management"

In this example, Service 1 is active on Oracle RAC Nodes 1, 2, and 3, while Service 2 is inactive on those nodes. Service 2 is active on Oracle RAC Nodes 4 and 5, while Service 1 is inactive on those nodes.

If Oracle RAC Node 1 becomes unavailable for any reason, you can start Service 1 on Oracle RAC Node 4. WebLogic Server will detect that the service is running on Node 4, and will begin making connections from the associated backup data source to Node 4 as needed.

Load Balancing

In a load balancing configuration, there are multiple services running concurrently on each Oracle RAC node. Each WLS multi data source has an active connection pool configured to connect to a given service on each of the nodes. In this scenario, you would configure each multi data source to use the Load Balancing algorithm.

Figure B-5 Load Balancing

Description of Figure B-5 follows
Description of "Figure B-5 Load Balancing"

In this example, Service 1 and Service 2 are each actively running on all of the available Oracle RAC nodes. As a result, all of the connection pools in each multi data source will actively make connections in a round-robin fashion, balancing workload among the five nodes.

Connection Pool Capacity Planning

It is important to note the Maximum Capacity value you specify for a data source can cause the connection capacity to a given Oracle RAC node to be exceeded. You must consider the following factors when determining how to set this value for each of your data sources:

  • The maximum number of simultaneous connections that a Oracle RAC node can support. This is based on the available memory on a given Oracle RAC node and the amount of memory consumed by each service connection (which can vary for each service). Memory consumption by each connection is a major limitation on the amount of work that can be generated from the WLS servers. Exceeding the amount of available memory by creating too many connections from your WLS data sources to a given Oracle RAC node can result in degraded performance on the Oracle RAC node, or could lead to failed connections.

    Available memory for a node should be based on the PGA target attribute (per session memory).

  • The maximum number of data sources that can potentially create connections to a given Oracle RAC node, and the number of WebLogic server instances to which each data source/multi data source is targeted. For example, if you have one data source that is targeted to three WLS servers, that data source counts as three data sources, as each server uses its own instance of the data source. This is the case whether the servers are part of a cluster or are independent server instances.

  • The maximum number of services that may be actively running on a given Oracle RAC node, and the memory consumed on the node by each connection to each service.

  • The expected relative workload for each service on a given node. For example, the expected workload of Service1 may be double that of the expected workload of Service2.

    Regardless of whether or not a service is always active on a node, you should allocate resources for that service in the event you have to start it on the node.

  • Always use the worst-case scenario when setting the Maximum Capacity value for your data sources. For example, assume that all available services will be actively running on the Oracle RAC node associated with each data source.

The following example explains how you could go about determining each data source's Maximum Capacity value. Keep in mind that this is a very simple example intended to illustrate the issue conceptually, and that real-world situations are much more complicated. In general, it is best to under-configure your data sources with a low Maximum Capacity value, monitor your Oracle RAC nodes for memory usage and performance, then adjust the Maximum Capacity values upward until you are approaching the maximum capacity of the associated Oracle RAC nodes.

Example

Suppose you have the following basic configuration:

  • Five Oracle RAC nodes, each with 16 GB of memory.

  • Two services actively running on each Oracle RAC node. Service1 uses 10MB per connection, Service2 uses 20MB per connection.

  • Workload for each service is the same, that is, each service will generate an equivalent number of connections on a given Oracle RAC node.

  • Two WebLogic Server clusters. Cluster1 has five servers. Cluster2 has four servers.

  • For a given Oracle RAC node, one data source is targeted to Cluster1 and is configured to connect to Service1.

  • For a given Oracle RAC node, one data source is targeted to Cluster 2 and is configured to connect to Service2.

Because Service2 uses twice as much memory per connection as Service1, you should allocate approximately 10GB of the node's memory for Service 2 and approximately 5GB for Service1.

Because Cluster1 has five WLS servers, there will be five data sources making connection requests to this Oracle RAC node. This gives you 1GB of memory available for connections from a given data source (5GB/5). Each connection requires 10MB of memory, so the Maximum Capacity value for each data source targeted to Cluster1 should be 100 or lower.

Because Cluster 2 has four WLS servers, there will be four data sources making connection requests to this Oracle RAC node. This gives you 2.5GB of memory available for connections from a given data source (10GB/4). Each connection requires 20MB, so the Maximum Capacity value for each data source targeted to Cluster2 should be 125 or lower.

If Service 2 generates more workload than Service1, you would have to adjust these values appropriately (increase the Maximum Capacity value for the data source connecting to Service2, decrease the value for the data source connecting to Service1). As long as:

(Max. connections to Service1 x memory used per connection) + (Max. connections to Service2 x memory used per connection) < Available memory

you can avoid the potential for performance degradation or connection failures.

Alternatively, in a simple configuration, such as is shown in Figure B-6, the Maximum Capacity value you specify for each of your data sources can be loosely determined using the following formula:

Maximum connection pool capacity = Maximum number of connections to Oracle RAC nodes/(Number of WebLogic Server instances x Nmber of data sources targeted to each instance x Number of active Oracle RAC services configured x Number of Oracle RAC Nodes)

where:

Maximum number of connections to Oracle RAC nodes is determined by total memory available on all nodes divided by the memory consumed by each connection.

Number of WebLogic Server instances is the number of server instances to which the data sources are targeted. If the data sources is targeted to a WLS cluster, this is the number of servers in the cluster.

In the example in Figure B-6:

  • assume that a maximum of 4000 total connections can be made to the group of Oracle RAC nodes, based on 8GB of available memory per Oracle RAC node, and 10MB of memory used per connection.

  • there are a total of five server instances to which the data sources are targeted

  • there are five data sources targeted to each server instance

  • there are two services running on each Oracle RAC node, and

  • there are five Oracle RAC nodes.

In this configuration, the Maximum Capacity value you would enter for each of your data sources would be:

Maximum connection pool capacity = 4000/(5 server instances x 5 data sources x 2 services x 5 Oracle RAC nodes)

which would give you a Maximum Capacity value of 16 for each of your data sources.

Figure B-6 Example Data Source Connection Configuration

Surrounding text describes Figure B-6 .

Keep in mind that this formula is just a general guideline for configuring your data sources, as many configurations will be too complex for you to use such a simple calculation.

When calculating the Maximum Capacity value you should use, always consider the worst-case scenario that you will have in your overall configuration. It is best to under-configure this value for normal operation than to have it over-configured when a worst-case situation develops. You can always monitor your Oracle RAC nodes to determine if it is safe to increase the Maximum Capacity value for any of your data sources.

Using Connect-Time Failover with Oracle RAC

When multi data sources are not an option in your application, you can configure your data sources to use connect-time failover and load balancing. To connect WebLogic Server to multiple Oracle RAC nodes using data sources configured for connect-time failover and load balancing, configure a JDBC data source for each Oracle RAC instance in your Oracle RAC cluster with the Oracle Thin driver, as described in the sections that follow. Figure B-7 shows an overview of the system.

Figure B-7 Data Source Configuration with Oracle Thin Driver Connect-Time Failover

Description of Figure B-7 follows
Description of "Figure B-7 Data Source Configuration with Oracle Thin Driver Connect-Time Failover"

You can use the Administration Console or any other means that you prefer to configure your domain, such as the Weblogic Scripting Tool (WLST) or a JMX program.

When connections are created in the data source, the Oracle Thin driver determines which Oracle RAC instance to use. When an application gets a connection, it looks up a data source on the JNDI tree and requests a connection from the data source. The data source delivers one of the available connections from the pool of connections in the data source.

The following sections describe configuration options and requirements:

Using Connect-Time Failover without Global Transactions

The following sections describe a configuration which uses Oracle RAC's connect-time failover features to handle connection failures. With this configuration, in some failure cases, the failover time is as long as the TCP timeout, which can be several minutes, depending on your environment.

Attributes of a Connect-Time Failover Configuration without Global Transactions

To use this configuration, create JDBC data sources in your WebLogic domain with the following attributes.

  • Oracle JDBC Thin driver 11g configured for connect-time failover. For example:

    <url>jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=
        (PROTOCOL=TCP)(HOST=lcqsol24)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)
        (HOST=lcqsol25)(PORT=1521))(FAILOVER=on)(LOAD_BALANCE=off))
        (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=snrac)))</url> 
    <driver-name>oracle.jdbc.OracleDriver</driver-name>
    
  • ConnectionReserveTimeoutSeconds="120"

    • Enables application requests for a connection to wait 120 seconds for a connection to become available.

  • TestConnectionsOnReserve="true"

    • Enables testing of a database connection when an application reserves a connection from the data source. See Test Connections on Reserve to Enable Fail-Over for more details about this attribute.

    • Required to enable failover to another Oracle RAC node.

  • TestTableName="name_of_small_table" The name of the table used to test a physical database connection. For more details about this attribute, see Connection Testing Options for a Data Source.

Sample Configuration Code

Sample configuration code is shown below.

<jdbc-data-source xmlns="http://xmlns.oracle.com/weblogic/jdbc-data-source"
  xmlns:sec="http://xmlns.oracle.com/weblogic/security"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:wls="http://xmlns.oracle.com/weblogic"
  xsi:schemaLocation="http://xmlns.oracle.com/weblogic/domain/1.0/domain.xsd">
  <name>oracleRACNonXAPool</name> 
  <jdbc-driver-params>
    <url>jdbc:oracle:thin:@(DESCRIPTION=
         (ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)
         HOST=lcqsol24)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)
         (HOST=lcqsol25)(PORT=152))(FAILOVER=on)
         (LOAD_BALANCE=off))(CONNECT_DATA=(SERVER=DEDICATED)
         (SERVICE_NAME=snrac)))</url> 
    <driver-name>oracle.jdbc.OracleDriver</driver-name> 
    <properties>
      <property>
        <name>user</name> 
        <value>wlsqa</value> 
      </property>
    </properties>
    <password-encrypted>{3DES}aP/xScCS8uI=</password-encrypted> 
  </jdbc-driver-params>
  <jdbc-connection-pool-params>
    <test-connections-on-reserve>true</test-connections-on-reserve> 
    <test-table-name>SQL SELECT 1 FROM DUAL</test-table-name> 
    <profile-type>4</profile-type> 
  </jdbc-connection-pool-params>
  <jdbc-data-source-params>
    <jndi-name>oracleRACJndiName</jndi-name> 
    <global-transactions-protocol>OnePhaseCommit
         </global-transactions-protocol> 
  </jdbc-data-source-params>
</jdbc-data-source> 

Note:

Line breaks added for readability.

Using Fast Connection Failover

WebLogic Server supports Fast Connection Failover, an Oracle feature which provides an application-independent method to implement Oracle RAC event notifications, such as detection and cleanup of invalid connections, load balancing of available connections, and work redistribution on active Oracle RAC instances.

For more information, see "Fast Connection Failover" at http://download-east.oracle.com/docs/cd/B19306_01/java.102/b14355/fstconfo.htm#CIHJBFFC in the Oracle® Database JDBC Developer's Guide and Reference.

Required JDBC Driver Configuration for use with Oracle Fast Connection Failover

To enable Fast Connection Failover on a data source, set the following connection pool properties:

  • In Driver Class Name—set the class name to oracle.jdbc.pool.OracleDataSource.

  • In Properties—set the ONS configuration string to remotely subscribe the Oracle RAC nodes to Oracle FAN/ONS events. For example: ONSConfiguration=nodes=hostname1:port1,hostname2:port2

    Note:

    Oracle's OracleDataSource class is not XA-capable, so the resulting data source does not implement a XA connection pool.

Because the Oracle driver uses ONS to implement Fast Connection Failover, you must add ons.jar to the WebLogic classpath. To do so, edit the setDomainEnv script in the domain's bin directory, and add ons.jar to the POST_CLASSPATH variable.

XA Considerations and Limitations with Oracle RAC

When using XA (global transactions) with Oracle RAC, consider the following requirements and limitations.

Oracle RAC XA Requirements

Oracle RAC has the following requirements when using XA.

A Global Transaction Must Be Initiated, Prepared, and Concluded in the Same Instance of the Oracle RAC Cluster

Global transactions must be initiated, prepared, and concluded in the same instance of the Oracle RAC cluster. WebLogic Server data sources manage this for you when you set KeepXAConnTillTxComplete="true" in the JDBC data source configuration.

Note:

For databases prior to 11.1, WebLogic Server relies on the connect-time failover feature in the Oracle Thin driver to work with Oracle RAC. As described in Oracle's Technical Note 235118.1, the Oracle Thin driver cannot guarantee that a transaction is initiated and concluded on the same Oracle RAC instance when the driver is configured for load balancing. As Oracle RAC requires that all database operations inside a global transaction be routed to the same Oracle instance, this known limitation means that you cannot use connect-time load balancing when using XA with Oracle RAC and, therefore, you cannot use a primary/primary Oracle RAC configuration.

This release of WebLogic Server includes support for Oracle RAC 11g as a high-availability database solution for WebLogic Server. However, this release does not support the new Oracle Services features in Oracle RAC 11g that support XA with load balancing. Instead, WebLogic Server will continue to use its well-proven integration architecture using multi data sources for XA with load balancing.

Transaction IDs Must Be Unique Within the Oracle RAC Cluster

When using global transactions, transaction IDs (XIDs) must be unique within the Oracle RAC cluster. However, neither the Oracle Thin driver nor an Oracle RAC instance can determine if an XID is unique within the Oracle RAC cluster. Transactions with the same XID can execute SQL code on different instances of the Oracle RAC cluster without any exception.

The WebLogic Server Transaction Manager generates unique transaction IDs. However, in some failover scenarios, a transaction can continue on a Oracle RAC instance other than the originating instance, which can cause data inconsistencies. See Potential for Inconsistent Transaction Completion (Data Loss) in Some Failure Conditions.

Known Limitations When Using Oracle RAC with WebLogic Server

The following sections describe known issues and limitations when using XA and WebLogic Server with Oracle RAC:

Potential for Inconsistent Transaction Completion (Data Loss) in Some Failure Conditions

In some failure conditions, when multi data sources are not being used, transaction processing (data changes) that occurred on a Oracle RAC instance other than the instance on which a transaction was initiated will be lost without any notification or exception.

For example, consider the following WebLogic Server configuration:

  • A WebLogic cluster containing two servers: server1 and server2.

  • A JDBC data source ds1 targeted to the cluster. (Identical instances of the data source are present on all instances in the cluster.)

  • A JDBC data source ds1 is configured to connect to an Oracle RAC cluster with connect-time failover enabled. RAC1 is set as the primary Oracle RAC instance; RAC2 is set as the secondary Oracle RAC instance. Data source ds1 is targeted to the cluster. (Identical instances of the data source are present on all nodes in the WebLogic cluster.)

In the following scenario, some data changes will be lost:

  1. Network connectivity between server2 and RAC1 is lost, which causes database connections in cp1 on server2 to fail over to RAC2. The same data source on server1 still has connections to RAC1.

  2. On server1, an application starts a transaction and uses a database connection from cp1 (a connection to RAC1) to make data changes.

  3. The application invokes an EJB on server2, which uses a database connection from cp1 on server2 (a connection to RAC2) to make data changes.

  4. The application completes the transaction on server1.

Result: Data changes on RAC1 are committed. Data changes on RAC2 are ignored. The WebLogic Server transaction manager calls prepare and commit on the resource. In this case, because the data sources have the same name, they are considered to be the same resource, so the calls are made on only one instance of the data source. Because the data sources contain connections to different Oracle RAC instances, the data changes are committed on one Oracle RAC instance, but the changes on the other Oracle RAC instance are lost.

Workaround: Provide redundant network hardware between the WebLogic Server instance and the Oracle RAC instance to avoid the network failure.

Potential for Data Deadlocks in Some Failure Scenarios

There is a window of time in which transaction IDs are not available across the Oracle RAC cluster. Because of this known limitation, after some failure conditions, some incomplete transactions cannot be properly completed, which can result in deadlocks in the database. To prevent these failure conditions from arising, WebLogic Server provides two configuration attributes that enable XA call retry for Oracle RAC: XARetryDurationSeconds and XARetryIntervalSeconds. For more information about these configuration options, see Delays During Failover.

Potential for Transactions Completed Out of Sequence

When using the Oracle DataBase Control, the order of transaction processing is not guaranteed. For example, if you implement a web service that uses DataBase Control do the following transaction sequence:

  1. Create a table

  2. Insert record 1

  3. Insert record 2

  4. Insert record 3

  5. Select records

If the primary node goes down momentarily after the table is created, it is possible that transactions submitted to the database are performed out of sequence.

Known Issue Occurring After Database Server Crash

If, while a transaction is being processed, the database server instance crashes after the PREPARE operation is complete but before the results of that operation have been written to the transaction log, a COMMIT call from a client for that transaction may hang for several minutes and possibly until the TCP timeout period has expired. The window of time in which this might occur is small and the problem occurs rarely. There is no workaround for the issue at this time.

JDBC Store Recovery with Oracle RAC

If you are using a JDBC Store with Oracle RAC, there are features and limitations to consider that concern Oracle RAC node failover. See the following sections:

For a list of the major services that use the JDBC store, see "Monitoring Store Connections" in Configuring Server Environments for Oracle WebLogic Server.

Configuring a JDBC Store for Use with Oracle RAC

The way that a JDBC Store works limits the options you have for configuring one for use with Oracle RAC. You cannot configure a JDBC store to use a JDBC data source that is configured to support global transactions. The JDBC store must use a JDBC data source that uses a non-XA JDBC driver. For more information about this configuration option, see Using Multi Data Sources without Global Transactions.

A JDBC Store holds on to a connection until that connection fails, at which point it moves on to the next connection and repeats the process. Therefore you cannot implement load balancing with a JDBC Store, including using a load balancing multi data source. You should configure a multi data source for a JDBC store to use the Failover algorithm.

In short, for a JDBC store:

  • Use a non-XA driver

  • Configure the multi data source for Failover mode.

Automatic Retry

JMS has a limited connection retry mechanism which enables it to silently react to the failure of the Oracle RAC node that hosts its database connection. If the database has experienced either a minor network 'hiccup' or a Oracle RAC database has failed over to another node, the second connection attempt (the retry) will succeed to the next Oracle RAC node.

The time within which this retry is attempted and the number of retries attempted are limited to minimize the negative effects that an extended connection retry time could cause. If the database connection remains unavailable for a long period of time, the delay can impede the ability of JMS to properly continue its processing (for example, to maintain proper message ordering). Also, the transaction manager could declare the JMS resource of a transaction to be dead if there is not enough processing progress made within this time period, or out-of memory conditions could arise. There are system-level tuning guidelines that can help minimize the Oracle RAC failover time frame which is critical to the success of the automatic retry.

The tight loop on the automatic retry is particularly important when JMS processing occurs with transactions. If an I/O failure occurs in the JDBC Store, the store record is in an unknown state which will put the message itself in an unknown state. To prevent the message from being committed in this unknown state, JMS will mark the transaction associated with the message as a "failedTransaction." Any future attempts by the transaction manager to finishing committing the message will cause JMS to throw a javax.transaction.xa.XAException with an errorCode set to XAException.XAER_RMERR. This exception is an indication to the transaction manager that a transient error has occurred in the resource manager (JMS) and that the transaction manager should retry commit processing. The retry logic provides a second attempt to establish the connection before JMS communicates any failure to the upper layer which would translate into an RMERR. If the RMERR is generated, then the only way to recover the message and complete the transaction is to restart WebLogic Server.

The automatic retry logic is currently governed by an option on WebLogic Server as follows:

-Dweblogic.store.jdbc.IORetryDelaySeconds=X

Where X is the number of seconds that should elapse before the connection to the database is retried. The default is one second. This value is restricted to the range 0 to 15, and the retry will only be attempted once. If a failure occurs on the second attempt, an exception will be propagated up the call stack and a manual restart will be required to recover the messages associated with the failed transaction.

Note:

In the event that an automatic retry attempt is not successful, you must restart WebLogic Server.