Using JDBC Multi Data Sources

A Multi Data Source (MDS) is an abstraction around a group of Generic data sources that is bound to the JNDI tree or local application context just like Generic data sources are bound to the JNDI tree. You can configure a MDS to provide load balancing or failover processing at the time of connection requests, between the Generic data sources associated with the MDS.

For information about Generic data sources, see Using Generic Data Sources.

Applications lookup a MDS on the JNDI tree or in the local application context (java:comp/env) just as they do for generic data sources, and then request a database connection. The MDS determines which generic data source to use to satisfy the request depending on the algorithm selected in the MDS configuration: load balancing or failover.

Note:

Active GridLink and Multi Data Source are designed to work with Oracle RAC clusters. Oracle does not recommend using Generic data sources with Oracle RAC clusters. See Generic Data Source Handling for Oracle RAC Outages.

• What is Multi Data Source
  Multi Data Source is used for failover or load balancing between nodes of a highly available database system such as Oracle Real Application Clusters (Oracle RAC). The Generic data source member list for a MDS source supports dynamic updates. This feature allows Oracle RAC environments to add and remove database nodes and corresponding Generic data sources without redeployment, grow and shrink RAC clusters in response to throughput, and shutdown Oracle RAC node for maintenance.
• Configuring Multi Data Sources
  Perform the steps mentioned in this topic to create and configure Multi Data Source.
• Multi Data Source Failover Enhancements
  Learn how to improve failover processing for Multi Data Sources.
• Planned Database Maintenance with a Multi Data Source
  Learn how to handle planned maintenance, without service interruption, on the database server used by a Multi Data Source.
• Shutting Down the Data Source
  Shutting down the data source involves first suspending the data source and then releasing the associated resources including the connections.

What is Multi Data Source

Multi Data Source is used for failover or load balancing between nodes of a highly available database system such as Oracle Real Application Clusters (Oracle RAC). The Generic data source member list for a MDS source supports dynamic updates. This feature allows Oracle RAC environments to add and remove database nodes and corresponding Generic data sources without redeployment, grow and shrink RAC clusters in response to throughput, and shutdown Oracle RAC node for maintenance.

Note:

Multi Data Sources do not provide any synchronization between databases. It is assumed that database synchronization is handled properly outside of WebLogic Server so that data integrity is maintained.

Adding and removing database nodes is a manual operation performed by the database administrator.

• Adding a Database Node
  You can add a database node and corresponding Generic data sources without redeployment. This capability provides you the ability to start a node after maintenance or grow a cluster.
• Removing a Database Node
  You can remove a database node and corresponding Generic data sources without redeployment. This capability provides you the ability to shutdown a node for maintenance or shrink a cluster.

Adding a Database Node

You can add a database node and corresponding Generic data sources without redeployment. This capability provides you the ability to start a node after maintenance or grow a cluster.

Use the following high-level steps to add a database node:

1. Restart the database node.
2. Restart the Generic data source. See Start JDBC data sources in Oracle WebLogic Server Administration Console Online Help.
3. Add the Generic data source back to the Multi Data Source. See Add or remove data sources in a JDBC Multi Data Sources in Oracle WebLogic Server Administration Console Online Help.

Removing a Database Node

You can remove a database node and corresponding Generic data sources without redeployment. This capability provides you the ability to shutdown a node for maintenance or shrink a cluster.

Use the following high-level steps to shutdown a database node:

Note:

Failure to follow these step may cause transaction roll-backs.

1. Remove the Generic data source from the Multi Data Source. See Add or remove data sources in a JDBC multi data sources in Oracle WebLogic Server Administration Console Online Help.
2. When all transactions have completed, suspend the Generic data source. See Suspend JDBC data sources in Oracle WebLogic Server Administration Console Online Help.
3. When all transactions have completed, shut down the Generic data source. See Shut down JDBC data sources in Oracle WebLogic Server Administration Console Online Help.
4. Shut down the database node.

Configuring Multi Data Sources

Perform the steps mentioned in this topic to create and configure Multi Data Source.

1. Create Generic data sources. See Using Generic Data Sources.

2. Create the Multi Data Source using either the WebLogic Server Administration Console or the WebLogic Scripting Tool. See, Configure JDBC Multi Data Sources in the Oracle WebLogic Server Administration Console Online Help.

3. Assign the Generic data sources to the Multi Data Source.

   For information about the configuration files created when configuring a Multi Data Source, see Understanding JDBC Resources in WebLogic Server and Creating a Multi Data Source Module.

Note:

In general, if a WebLogic Server data source setting of initial capacity is set to Zero, WebLogic Server makes no DBMS connections at startup. But to startup a Multi Data Source of LLR data sources, WebLogic Server makes a connection at startup to see if the DBMS is a RAC or not. For a generic LLR Multi Data Source, all the data sources need to be available, but if it is using RAC, only one node needs to be accessible for LLR processing.

• Choosing the Multi Data Source Algorithm
  Before you set up a Multi Data Source, you need to determine the primary purpose of the Multi Data Source—failover or load balancing. You can choose the algorithm that corresponds with your requirements.
• Multi Data Source Fail-Over Limitations and Requirements
  WebLogic Server provides a failover algorithm for Multi Data Sources so that if a Generic data source fails (for example, if the database management system crashes), your system can continue to operate. However, there are certain limitations and requirements you must consider when configuring the Multi Data Source.
• Controlling Multi Data Source Failover with a Callback
  You can register a callback handler with WebLogic Server that controls when a MDS with the Failover algorithm fails over connection requests from one JDBC Generic data source in the MDS to the next Generic data source in the list.
• Deploying JDBC Multi Data Sources on Servers and Clusters
  All Generic data sources used by a Multi Data Source to satisfy connection requests must be deployed on the same servers and clusters as the Multi Data Source. A Multi Data Source always uses a Generic data source deployed on the same server to satisfy connection requests. Multi Data Sources do not route connection requests to other servers in a cluster or in a domain.

Choosing the Multi Data Source Algorithm

Before you set up a Multi Data Source, you need to determine the primary purpose of the Multi Data Source—failover or load balancing. You can choose the algorithm that corresponds with your requirements.

• Failover
  
• Load Balancing
  

Failover

The Failover algorithm provides an ordered list of Generic data sources to use to satisfy connection requests. Normally, every connection request to this kind of Multi Data Source is served by the first Generic data source in the list. If a database connection test fails and the connection cannot be replaced, or if the Generic data source is suspended, a connection is sought sequentially from the next Generic data source on the list.

Note:

This algorithm requires that Test Reserved Connections (TestConnectionsOnReserve) on the Generic data source is enabled. If enabled, a connection in the first Generic data source is tested to verify if the Generic data source is healthy. If the connection fails the test, the Multi Data Source uses a connection from the next Generic data source listed in the Multi Data Source. See Connection Testing Options for a Data Source for information about configuring TestConnectionsOnReserve.

JDBC is a highly stateful client-DBMS protocol, in which the DBMS connection and transactional state are tied directly to the socket between the DBMS process and the client (driver). For this reason, failover of a connection while it is in use is not supported.

Load Balancing

Connection requests to a load-balancing Multi Data Source are served from any Generic data source in the list. The MDS selects Generic data sources to use to satisfy connection requests using a round-robin scheme. When the MDS provides a connection, it selects a connection from the Generic data source listed just after the last Generic data source that was used to provide a connection. Multi Data Sources that use the Load Balancing algorithm also fail over to the next Generic data source in the list if a database connection test fails and the connection cannot be replaced, or if the Generic data source is suspended.

Multi Data Source Fail-Over Limitations and Requirements

WebLogic Server provides a failover algorithm for Multi Data Sources so that if a Generic data source fails (for example, if the database management system crashes), your system can continue to operate. However, there are certain limitations and requirements you must consider when configuring the Multi Data Source.

• Test Connections on Reserve to Enable Fail-Over
  
• No Fail-Over for In-Use Connections
  

Test Connections on Reserve to Enable Fail-Over

Generic data sources rely on the Test Reserved Connections (TestConnectionsOnReserve) feature on the Generic data source to know when database connectivity is lost. Testing reserved connections must be enabled for the Generic data sources within the Multi Data Source. WebLogic Server will test each connection before giving it to an application. With the Failover algorithm, the Multi Data Source uses the results from connection test to determine when to fail over to the next Generic data source in the Multi Data Source. After a test failure, the Generic data source attempts to recreate the connection. If that attempt fails, the Multi Data Source fails over to the next Generic data source.

No Fail-Over for In-Use Connections

It is possible for a connection to fail after being reserved, in which case your application must handle the failure. WebLogic Server cannot provide fail-over for connections that fail while being used by an application. Any failure while using a connection requires that the application code close the failed connection, and the transaction must be restarted from the beginning with a new connection.

Controlling Multi Data Source Failover with a Callback

You can register a callback handler with WebLogic Server that controls when a MDS with the Failover algorithm fails over connection requests from one JDBC Generic data source in the MDS to the next Generic data source in the list.

You can use callback handlers to control if or when the failover occurs so that you can make any other system preparations before the failover, such as priming a database or communicating with a high-availability framework.

Callback handlers are registered using the Failover Callback Handler attribute of the MDS and are registered per MDS. You must register the callback handler for each MDS to which you want the callback handler to apply. And you can register different callback handlers for each MDS in your domain.

• Callback Handler Requirements
  
• Callback Handler Configuration
  
• How It Works—Failover
  

Callback Handler Requirements

A callback handler used to control the failover and failback within a Multi Data Source must include an implementation of the weblogic.jdbc.extensions.ConnectionPoolFailoverCallback interface. When the Multi Data Source needs to failover to the next Generic data source in the list or when a previously disabled Generic data source becomes available, WebLogic Server calls the allowPoolFailover() method in the ConnectionPoolFailoverCallback interface, and passes a value for the three parameters, currPool, nextPool, and opcode, as defined below. WebLogic Server then waits for the return from the callback handler before completing the task.

Your application must return OK, RETRY_CURRENT, or DONOT_FAILOVER as defined below. The application should handle failover and failback cases.

See the weblogic.jdbc.extensions.ConnectionPoolFailoverCallback interface.

Note:

Failover callback handlers are optional. If no callback handler is specified in the Multi Data Source configuration, WebLogic Server proceeds with the operation (failing over or re-enabling the disabled Generic data source).

Callback Handler Configuration

There are two Multi Data Source configuration attributes associated with the failover and failback functionality:

• Failover Callback Handler (ConnectionPoolFailoverCallbackHandler)—To register a failover callback handler for a Multi Data Source, you add a value for this attribute to the Multi Data Source configuration. The value must be an absolute name, such as com.bea.samples.wls.jdbc.MultiDataSourceFailoverCallbackApplication. You can set the Failover Callback Handler using the WebLogic Server Administration Console (see Register a failover callback handler in the Oracle WebLogic Server Administration Console Online Help) or on the JDBCDataSourceParamsBean for the Multi Data Source using WLST.

• Test Frequency (TestFrequencySeconds)—To control how often the Multi Data Source checks disabled (dead) Generic data sources to see if they are now available. See Automatic Re-enablement on Recovery of a Failed Generic Data Source within a Multi Data Source for more details.

How It Works—Failover

WebLogic Server attempts to failover connection requests to the next Generic data source in the list when the current Generic data source fails a connection test or, if you enabled FailoverRequestIfBusy, when all connections in the current Generic data source are busy.

To enable the callback feature, you register the callback handler with Weblogic Server using Failover Callback Handler in the Multi Data Source configuration.

With the Failover algorithm, connection requests are served from the first Generic data source in the list. If a connection from that Generic data source fails a connection test, WebLogic Server marks the Generic data source as dead and disables it. If a callback handler is registered, WebLogic Server calls the callback handler, passing the following information, and waits for a return:

• currPool—For failover, this is the name of Generic data source currently being used to supply database connections. This is the "failover from" Generic data source.

• nextPool—The name of next available Generic data source listed in the Multi Data Source. For failover, this is the "failover to" Generic data source.

• opcode—A code that indicates the reason for the call:

  • OPCODE_CURR_POOL_DEAD—WebLogic Server determined that the current Generic data source is dead and has disabled it.

  • OPCODE_CURR_POOL_BUSY—All database connections in the Generic data source are in use. (Requires FailoverIfBusy=true in the Multi Data Source configuration. See Enabling Failover for Busy Generic Data Sources in a Multi Data Source.)

Failover is synchronous with the connection request: Failover occurs only when WebLogic Server is attempting to satisfy a connection request.

The return from the callback handler can indicate one of three options:

• OK—proceed with the operation. In this case, that means to failover to the next Generic data source in the list.

• RETRY_CURRENT—Retry the connection request with the current Generic data source.

• DONOT_FAILOVER—Do not retry the current connection request and do not failover. WebLogic Server will throw a weblogic.jdbc.extensions.PoolUnavailableSQLException.

WebLogic Server acts according to the value returned by the callback handler.

If the secondary Generic data sources fails, WebLogic Server calls the callback handler again, as in the previous failover, in an attempt to failover to the next available Generic data source in the Multi Data Source, if there is one.

Note:

WebLogic Server does not call the callback handler when you manually disable a Generic data source.

For Multi Data Sources with the Load-Balancing algorithm, WebLogic Server does not call the callback handler when a Generic data source is disabled. However, it does call the callback handler when attempting to re-enable a disabled Generic data source. See the following section for more details.

Deploying JDBC Multi Data Sources on Servers and Clusters

All Generic data sources used by a Multi Data Source to satisfy connection requests must be deployed on the same servers and clusters as the Multi Data Source. A Multi Data Source always uses a Generic data source deployed on the same server to satisfy connection requests. Multi Data Sources do not route connection requests to other servers in a cluster or in a domain.

To deploy a Multi Data Source to a cluster or server, you select the server or cluster as a deployment target. When a Multi Data Source is deployed on a server, WebLogic Server creates an instance of the Multi Data Source on the server. When you deploy a Multi Data Source to a cluster, WebLogic Server creates an instance of the Multi Data Source on each server in the cluster.

For instructions, see Target and deploy JDBC Multi Data Sources in the Oracle WebLogic Server Administration Console Online Help.

Multi Data Source Failover Enhancements

Learn how to improve failover processing for Multi Data Sources.

• Connection Request Routing Enhancements When a Generic Data Source Fails
  
• Automatic Re-enablement on Recovery of a Failed Generic Data Source within a Multi Data Source
  
• Enabling Failover for Busy Generic Data Sources in a Multi Data Source
  
• Controlling Multi Data Source Failback with a Callback
  

Connection Request Routing Enhancements When a Generic Data Source Fails

To improve performance when a Generic data source within a Multi Data Source fails, WebLogic Server automatically disables the Generic data source when a pooled connection fails a connection test. After a Generic data source is disabled, WebLogic Server does not route connection requests from applications to the Generic data source. Instead, it routes connection requests to the next available Generic data source listed in the Multi Data Source.

This feature requires that Generic data source testing options are configured for allGeneric data sources in a Multi Data Source, specifically Test Table Name and Test Reserved Connections. See Connection Testing Options for a Data Source.

If a callback handler is registered for the Multi Data Source, WebLogic Server calls the callback handler before failing over to the next Generic data source in the list. See Controlling Multi Data Source Failover with a Callback for more details.

Automatic Re-enablement on Recovery of a Failed Generic Data Source within a Multi Data Source

After a Generic data source is automatically disabled because a connection failed a connection test, the Multi Data Source periodically tests a connection from the disabled Generic data source to determine when the Generic data source (or underlying database) is available again. When the Generic data source becomes available, the Multi Data Source automatically re-enables the Generic data source and resumes routing connection requests to the Generic data source, depending on the Multi Data Source algorithm and the position of the Generic data source in the list of included Generic data sources. Frequency of these tests is controlled by the Test Frequency Seconds attribute of the Multi Data Source. The default value for Test Frequency is 120 seconds, so if you do not specifically set a value for the option, the Multi Data Source will test disabled Generic data sources every 120 seconds. See JDBC Multi Data Source: Configuration: General in the Oracle WebLogic Server Administration Console Online Help.

WebLogic Server does not test and automatically re-enable Generic data sources that you manually disable. It only tests Generic data sources that are automatically disabled.

If a callback handler is registered for the Multi Data Source, WebLogic Server calls the callback handler before re-enabling the Generic data source. See Controlling Multi Data Source Failback with a Callback for more details.

Enabling Failover for Busy Generic Data Sources in a Multi Data Source

By default, for Multi Data Sources with the Failover algorithm, when the number of requests for a database connection exceeds the number of available connections in the current Generic data source in the Multi Data Source, subsequent connection requests fail.

To enable the Multi Data Source to failover when all connections in the current Generic data source are in use, you can enable the Failover Request if Busy option on the JDBC Multi Data Source: Configuration: General page in the WebLogic Server Administration Console. (Also available as the FailoverRequestIfBusy attribute in the JDBCDataSourceParamsBean). If enabled (set to true), when all connections in the current Generic data source are in use, application requests for connections will be routed to the next available Generic data source within the Multi Data Source. When disabled (set to false, the default), connection requests do not failover.

If a ConnectionPoolFailoverCallbackHandler is included in the Multi Data Source configuration, WebLogic Server calls the callback handler before failing over. See Controlling Multi Data Source Failover with a Callback for more details.

Controlling Multi Data Source Failback with a Callback

If you register a failover callback handler for a Multi Data Source, WebLogic Server calls the same callback handler when re-enabling a Generic data source that was automatically disabled. You can use the callback to control if or when the disabled Generic data source is re-enabled so that you can make any other system preparations before the Generic data source is re-enabled, such as priming a database or communicating with a high-availability framework.

See the following sections for more details about the callback handler:

• How It Works—Failback
  

How It Works—Failback

WebLogic Server periodically checks the status of Generic data sources in a Multi Data Source that were automatically disabled. (See Automatic Re-enablement on Recovery of a Failed Generic Data Source within a Multi Data Source.) If a disabled Generic data source becomes available and if a failover callback handler is registered, WebLogic Server calls the callback handler with the following information and waits for a return:

• currPool—For failback, this is the name of the Generic data source that was previously disabled and is now available to be re-enabled.

• nextPool—For failback, this is null.

• opcode—A code that indicates the reason for the call. For failback, the code is always OPCODE_REENABLE_CURR_POOL, which indicates that the Generic data source named in currPool is now available.

Failback, or automatically re-enabling a disabled Generic data source, differs from failover in that failover is synchronous with the connection request, but failback is asynchronous with the connection request.

The callback handler can return one of the following values:

• OK—proceed with the operation. In this case, that means to re-enable the indicated Generic data source. WebLogic Server resumes routing connection requests to the Generic data source, depending on the Multi Data Source algorithm and the position of the Generic data source in the list of included Generic data sources.

• DONOT_FAILOVER—Do not re-enable the currPool Generic data source. Continue to serve connection requests from the Generic data sources in use.

WebLogic Server acts according to the value returned by the callback handler.

If the callback handler returns DONOT_FAILOVER, WebLogic Server will attempt to re-enable the Generic data source during the next testing cycle as determined by the TestFrequencySeconds attribute in the Multi Data Source configuration, and will call the callback handler as part of that process.

The order in which Generic data sources are listed in a Multi Data Source is very important. A Multi Data Source with the Failover algorithm will always attempt to serve connection requests from the first available Generic data source in the list of Generic data sources in the Multi Data Source. Consider the following scenario:

1. MultiDataSource_1 uses the Failover algorithm, has a registered ConnectionPoolFailoverCallbackHandler, and includes three Generic data sources: DS1, DS2, and DS3, listed in that order.

2. DS1 becomes disabled, so MultiDataSource_1 fails over connection requests to DS2.

3. DS2 then becomes disabled, so MultiDataSource_1 fails over connection requests to DS3.

4. After some time, DS1 becomes available again and the callback handler allows WebLogic Server to re-enable the Generic data source. Future connection requests will be served by DS1 because DS1 is the first Generic data source listed in the Multi Data Source.

5. If DS2 subsequently becomes available and the callback handler allows WebLogic Server to re-enable the Generic data source, connection requests will continue to be served by DS1 because DS1 is listed before DS2 in the list of Generic data sources.

Planned Database Maintenance with a Multi Data Source

Learn how to handle planned maintenance, without service interruption, on the database server used by a Multi Data Source.

To avoid service interruption, multiple database instances must be available so that the database can be updated in a rolling fashion. Oracle RAC cluster and Oracle GoldenGate, or a combination of these products, can be used to help accomplish this goal. (Note that Oracle DataGuard cannot be used for planned maintenance without service interruption). Each database instance is configured as a Generic data source member of the Multi Data Source. This approach assumes that the application is returning connections to the pool on a regular basis.

Process Overview

The following steps provide a high-level overview of the planned maintenance process:

1. On mid-tier systems—Shutdown all member data sources associated with the Oracle RAC instance that will be shut down for maintenance. It is important that you do not shut down all data sources in each Multi Data Source list so that connections can be reserved for the other member(s). Wait for data source shutdown to complete. See:

   • Shutting Down the Data Source

   • JDBCDataSourceRuntimeMBean shutdown operation in MBean Reference for Oracle WebLogic Server

2. If required, you may want to reduce the remaining connections on the database side that are not associated with the WebLogic data source. For the Oracle database server, this might include stopping (or relocating) the application services at the instances that will be shut down for maintenance, stopping the listener, and/or issuing a transactional disconnect for the services on the database instance.
3. Shut down the database instance using your preferred tools.
4. Perform the planned maintenance
5. Restart the database instance using your preferred tools.
6. Startup the services when the database instances are ready for application use.
7. On mid-tier systems—Start the member data sources. See JDBCDataSourceRuntimeMBean start operation in MBean Reference for Oracle WebLogic Server.

Shutting Down the Data Source

Shutting down the data source involves first suspending the data source and then releasing the associated resources including the connections.

When a member data source in a Multi Data Source is marked as suspended, the Multi Data Source will not try to get connections from the suspended pool. Instead, to reserve connections, it will go to the next member data source. It is important that you do not shut down all member data sources in a Multi Data Source list at the same time. If all members are shut down or fail, then access to the Multi Data Source fails and the application will see failures.

When you gracefully suspend a data source, which is the first step of the shut down process, the following occurs:

• The data source is immediately marked as suspended at the beginning of the operation and no further connections are created on the data source.

• Idle (not reserved) connections are marked closed

• After a timeout period for the suspend operation, all remaining connections in the pool are marked as suspended and the following exception is thrown for any operations on the connection, indicating that the data source is suspended:

  java.sql.SQLRecoverableException: Connection has been administratively disabled. Try later.

• All the remaining connections are then closed. We won't know until the data source is resumed if they are good or not. In this case, we know that the database will be shut down and the connections in the pool will not be good if the data source is resumed. Instead, we are doing a data source shutdown which will close all of the disabled connections.

  The shutdown operation can be done synchronously or asynchronously. If you do a synchronous shutdown, the default timeout period is 60 seconds. You can change the value of this timeout period by configuring or dynamically setting Inactive Connection Timeout Seconds to a non-zero value. There is no upper limit on the inactive timeout period. Note that the processing actually checks for in-use (reserved) resources every tenth of a second so if the timeout value is set to 2 hours and all reserved resources are released a second later, the shut down will complete a second later. If you do an asynchronous operation, the timeout period is specified on the method itself. If set to 0, the default is used. The default is to use Inactive Connection Timeout Seconds if set or 60 seconds. If you want a minimal timeout, set the value to 1. If you want no timeout, set it to a large value (not recommended).

This shutdown operation runs synchronously; there is no asynchronous version of the MBean operation available.

You can also use this for Multi Data Sources configured with either Load-Balancing or Failover.

Example 4-1 WLST Example

The following WLST example script demonstrates how to edit the configuration to increase the suspend timeout period and then use the runtime MBean to shutdown a data source. This script must be integrated into the existing framework for all WebLogic Server servers and data sources.

import sys, socket, os
hostname = 'hostname'
datasource='ds'
svr='myserver'
connect("weblogic","password","t3://"+hostname+":7001")
# Shutdown the data source serverRuntime()
serverRuntime()
cd('/JDBCServiceRuntime/' + svr + '/JDBCDataSourceRuntimeMBeans/' +datasource )
task = cmo.shutdown(10000)
while (task.isRunning ()): 
      print 'SHUTTING DOWN' 
      java.lang.Thread.sleep(2000) 
      print 'Datasource task is in status' + task.getStatus()
exit()
$ java weblogic.WLST myscript2.py
Intializing Weblogic Scripting Tool (WLST)...
Welcome to WebLogic Server Administration Scripting Shell
....
Location changed to serverRuntime tree.
This is a read-only tree with ServerRuntimeMBean as the root. For more help, use help('serverRuntime').
SHUTTING DOWN
Datasource task is in status
SUCCESS
Datasource task is in status
SUCCESS
Exiting WebLogic Scripting Tool.

Important Considerations

The following list describes issues you should be aware of when performing planned maintenance:

• If the Multi Data Source is using a database service, you cannot stop or relocate the database service before suspending or shutting down the Multi Data Source. If you do, the Multi Data Source may attempt to create a connection to the now missing service and it will react as though the database is down and kill all connections, preventing a graceful shutdown. Because suspending a Multi Data Source ensures that no new connections are created at the associated instance, it is not necessary to stop the service. (Note that the Multi Data Source only creates connections on this instance. It will never create connections on another instance even if it is relocated). Also, suspending a Multi Data Source ceases operations on all connections, therefore no further progress occurs on any sessions (the transactions will not complete) that remain in the Multi Data Source pool.

• You may encounter an issue related to XA affinity that is enforced by the Multi Data Source algorithms. When an XA branch is created on an Oracle RAC instance, all additional branches are created on the same instance. While Oracle RAC supports XA across instances, there are some significant limitations that applications run into before the prepare phase, and the Multi Data Source enforces that all operations are on the same instance. As soon as the graceful suspend operation starts, the member data source is marked as suspended and no further connections are allocated there. If an application using global transactions tries to start another branch on the suspending data source, it will fail to get a connection and the transaction fails. In the case of an XA transaction spanning multiple WebLogic servers, the suspend is not graceful. This issue does not apply to Emulate Two-Phase Commit or one-phase commit, which use a single connection for all work, and Logging Last Resource (LLR).

• If for some reason you must separate suspending the data source, at which point all connections are disabled, from releasing the resources, you can perform a suspend followed by forceShutdown. You must use a forced shutdown to avoid going through the waiting period a second time. Oracle does not recommend using this process.

• To get a graceful shutdown of the data source when shutting down the database, the data source must be involved. This process of shutting down the data source followed by shutdown of the database requires coordination between the mid-tier and the database server processing. Processing is simplified by using Active GridLink instead of Multi Data Source. See Using Active GridLink Data Sources.

• When using the Oracle database, Oracle recommends that an application service be configured for each database so that it can be configured for high availability. By using an application service, you can start up the database on its own without the data source starting to use it. Once the application service is explicitly started, the administrator can make the database available to the data source.