Applications lookup a multi data source on the JNDI tree or in the local application context (java:comp/env) just as they do for generic data sources, and then request a database connection. The multi data source determines which generic data source to use to satisfy the request depending on the algorithm selected in the multi data source configuration: load balancing or failover.
Note:
Active GridLink and Multi Data Source are designed to work with Oracle RAC clusters. Oracle does not recommend using generic data sources with Oracle RAC clusters. See Comparing AGL and Multi Data Sources.
This chapter includes the following sections:
See:
Note:
Multi data sources do not provide any synchronization between databases. It is assumed that database synchronization is handled properly outside of WebLogic Server so that data integrity is maintained. Adding and removing database nodes is a manual operation performed by the database administrator.
You can remove a database node and corresponding generic data sources without redeployment. This capability provides you the ability to shutdown a node for maintenance or shrink a cluster. Use the following high-level steps to shutdown a database node:
Note:
Failure to follow these step may cause transaction roll-backs.
You can add a database node and corresponding generic data sources without redeployment. This capability provides you the ability to start a node after maintenance or grow a cluster. Use the following high-level steps to add a database node:
For instructions to create a multi data source, see Configure JDBC multi data sources in the Oracle WebLogic Server Administration Console Online Help.
For information about the configuration files created when configuring a multi data source, see Understanding JDBC Resources in WebLogic Server. Also see Creating a Multi Data Source Module.
Note:
In general, if a WebLogic Server data source setting of initial capacity is set to 0, WebLogic Server makes no DBMS connections at startup. But to startup a MultiDataSource of LLR data sources, WebLogic Server makes a connection at startup to see if the DBMS is a RAC or not. For a generic LLR MultiDatasource, all the data sources need to be available, but if it is using RAC, only one node needs to be accessible for LLR processing.
Before you set up a multi data source, you need to determine the primary purpose of the multi data source—failover or load balancing. You can choose the algorithm that corresponds with your requirements.
The Failover algorithm provides an ordered list of generic data sources to use to satisfy connection requests. Normally, every connection request to this kind of multi data source is served by the first generic data source in the list. If a database connection test fails and the connection cannot be replaced, or if the generic data source is suspended, a connection is sought sequentially from the next generic data source on the list.
Note:
This algorithm requires that Test Reserved Connections (TestConnectionsOnReserve) on the generic data source is enabled. If enabled, a connection in the first generic data source is tested to verify if the generic data source is healthy. If the connection fails the test, the multi data source uses a connection from the next generic data source listed in the multi data source. See Connection Testing Options for a Data Source for information about configuring TestConnectionsOnReserve.
JDBC is a highly stateful client-DBMS protocol, in which the DBMS connection and transactional state are tied directly to the socket between the DBMS process and the client (driver). For this reason, failover of a connection while it is in use is not supported.
Connection requests to a load-balancing multi data source are served from any generic data source in the list. The multi data source selects generic data sources to use to satisfy connection requests using a round-robin scheme. When the multi data source provides a connection, it selects a connection from the generic data source listed just after the last generic data source that was used to provide a connection. Multi data sources that use the Load Balancing algorithm also fail over to the next generic data source in the list if a database connection test fails and the connection cannot be replaced, or if the generic data source is suspended.
Generic data sources rely on the Test Reserved Connections (TestConnectionsOnReserve) feature on the generic data source to know when database connectivity is lost. Testing reserved connections must be enabled for the generic data sources within the multi data source. WebLogic Server will test each connection before giving it to an application. With the Failover algorithm, the multi data source uses the results from connection test to determine when to fail over to the next generic data source in the multi data source. After a test failure, the generic data source attempts to recreate the connection. If that attempt fails, the multi data source fails over to the next generic data source.
It is possible for a connection to fail after being reserved, in which case your application must handle the failure. WebLogic Server cannot provide fail-over for connections that fail while being used by an application. Any failure while using a connection requires that the application code close the failed connection, and the transaction must be restarted from the beginning with a new connection.
The following enhancements improve failover processing for multi data sources:
Connection request routing enhancements to avoid requesting a connection from an automatically disabled (dead) generic data source within a multi data source. See Connection Request Routing Enhancements When a Generic Data Source Fails.
Automatic failback on recovery of a failed generic data source within a multi data source. See Automatic Re-enablement on Recovery of a Failed Generic Data Source within a Multi Data Source.
Failover for busy generic data sources within a multi data sources. See Enabling Failover for Busy Generic Data Sources in a Multi Data Source.
Failover callbacks for multi data sources with the Failover algorithm. See Controlling Multi Data Source Failover with a Callback.
Failback callbacks for multi data sources with either algorithm. See Controlling Multi Data Source Failback with a Callback.
To improve performance when a generic data source within a multi data source fails, WebLogic Server automatically disables the generic data source when a pooled connection fails a connection test. After a generic data source is disabled, WebLogic Server does not route connection requests from applications to the generic data source. Instead, it routes connection requests to the next available generic data source listed in the multi data source.
This feature requires that generic data source testing options are configured for all generic data sources in a multi data source, specifically Test Table Name and Test Reserved Connections. See Connection Testing Options for a Data Source.
If a callback handler is registered for the multi data source, WebLogic Server calls the callback handler before failing over to the next generic data source in the list. See Controlling Multi Data Source Failover with a Callback for more details.
After a generic data source is automatically disabled because a connection failed a connection test, the multi data source periodically tests a connection from the disabled generic data source to determine when the generic data source (or underlying database) is available again. When the generic data source becomes available, the multi data source automatically re-enables the generic data source and resumes routing connection requests to the generic data source, depending on the multi data source algorithm and the position of the generic data source in the list of included generic data sources. Frequency of these tests is controlled by the Test Frequency Seconds attribute of the multi data source. The default value for Test Frequency is 120 seconds, so if you do not specifically set a value for the option, the multi data source will test disabled generic data sources every 120 seconds. See JDBC Multi Data Source: Configuration: General in the Oracle WebLogic Server Administration Console Online Help.
WebLogic Server does not test and automatically re-enable generic data sources that you manually disable. It only tests generic data sources that are automatically disabled.
If a callback handler is registered for the multi data source, WebLogic Server calls the callback handler before re-enabling the generic data source. See Controlling Multi Data Source Failback with a Callback for more details.
By default, for multi data sources with the Failover algorithm, when the number of requests for a database connection exceeds the number of available connections in the current generic data source in the multi data source, subsequent connection requests fail.
To enable the multi data source to failover when all connections in the current generic data source are in use, you can enable the Failover Request if Busy option on the JDBC Multi Data Source: Configuration: General page in the WebLogic Server Administration Console. (Also available as the FailoverRequestIfBusy attribute in the JDBCDataSourceParamsBean). If enabled (set to true), when all connections in the current generic data source are in use, application requests for connections will be routed to the next available generic data source within the multi data source. When disabled (set to false, the default), connection requests do not failover.
If a ConnectionPoolFailoverCallbackHandler is included in the multi data source configuration, WebLogic Server calls the callback handler before failing over. See Controlling Multi Data Source Failover with a Callback for more details.
You can register a callback handler with WebLogic Server that controls when a multi data source with the Failover algorithm fails over connection requests from one JDBC generic data source in the multi data source to the next generic data source in the list.
You can use callback handlers to control if or when the failover occurs so that you can make any other system preparations before the failover, such as priming a database or communicating with a high-availability framework.
Callback handlers are registered via the Failover Callback Handler attribute of the multi data source and are registered per multi data source. You must register the callback handler for each multi data source to which you want the callback handler to apply. And you can register different callback handlers for each multi data source in your domain.
A callback handler used to control the failover and failback within a multi data source must include an implementation of the weblogic.jdbc.extensions.ConnectionPoolFailoverCallback interface. When the multi data source needs to failover to the next generic data source in the list or when a previously disabled generic data source becomes available, WebLogic Server calls the allowPoolFailover() method in the ConnectionPoolFailoverCallback interface, and passes a value for the three parameters, currPool, nextPool, and opcode, as defined below. WebLogic Server then waits for the return from the callback handler before completing the task.
Your application must return OK, RETRY_CURRENT, or DONOT_FAILOVER as defined below. The application should handle failover and failback cases.
See the weblogic.jdbc.extensions.ConnectionPoolFailoverCallback interface.
Note:
Failover callback handlers are optional. If no callback handler is specified in the multi data source configuration, WebLogic Server proceeds with the operation (failing over or re-enabling the disabled generic data source).
There are two multi data source configuration attributes associated with the failover and failback functionality:
Failover Callback Handler (ConnectionPoolFailoverCallbackHandler)—To register a failover callback handler for a multi data source, you add a value for this attribute to the multi data source configuration. The value must be an absolute name, such as com.bea.samples.wls.jdbc.MultiDataSourceFailoverCallbackApplication. You can set the Failover Callback Handler using the WebLogic Server Administration Console (see Register a failover callback handler in the Oracle WebLogic Server Administration Console Online Help) or on the JDBCDataSourceParamsBean for the multi data source using WLST.
Test Frequency (TestFrequencySeconds)—To control how often the multi data source checks disabled (dead) generic data sources to see if they are now available. See Automatic Re-enablement on Recovery of a Failed Generic Data Source within a Multi Data Source for more details.
WebLogic Server attempts to failover connection requests to the next generic data source in the list when the current generic data source fails a connection test or, if you enabled FailoverRequestIfBusy, when all connections in the current generic data source are busy.
To enable the callback feature, you register the callback handler with Weblogic Server using Failover Callback Handler in the multi data source configuration.
With the Failover algorithm, connection requests are served from the first generic data source in the list. If a connection from that generic data source fails a connection test, WebLogic Server marks the generic data source as dead and disables it. If a callback handler is registered, WebLogic Server calls the callback handler, passing the following information, and waits for a return:
currPool—For failover, this is the name of generic data source currently being used to supply database connections. This is the "failover from" generic data source.
nextPool—The name of next available generic data source listed in the multi data source. For failover, this is the "failover to" generic data source.
opcode—A code that indicates the reason for the call:
OPCODE_CURR_POOL_DEAD—WebLogic Server determined that the current generic data source is dead and has disabled it.
OPCODE_CURR_POOL_BUSY—All database connections in the generic data source are in use. (Requires FailoverIfBusy=true in the multi data source configuration. See Enabling Failover for Busy Generic Data Sources in a Multi Data Source.)
Failover is synchronous with the connection request: Failover occurs only when WebLogic Server is attempting to satisfy a connection request.
The return from the callback handler can indicate one of three options:
OK—proceed with the operation. In this case, that means to failover to the next generic data source in the list.
RETRY_CURRENT—Retry the connection request with the current generic data source.
DONOT_FAILOVER—Do not retry the current connection request and do not failover. WebLogic Server will throw a weblogic.jdbc.extensions.PoolUnavailableSQLException.
WebLogic Server acts according to the value returned by the callback handler.
If the secondary generic data sources fails, WebLogic Server calls the callback handler again, as in the previous failover, in an attempt to failover to the next available generic data source in the multi data source, if there is one.
Note:
WebLogic Server does not call the callback handler when you manually disable a generic data source.
For multi data sources with the Load-Balancing algorithm, WebLogic Server does not call the callback handler when a generic data source is disabled. However, it does call the callback handler when attempting to re-enable a disabled generic data source. See the following section for more details.
If you register a failover callback handler for a multi data source, WebLogic Server calls the same callback handler when re-enabling a generic data source that was automatically disabled. You can use the callback to control if or when the disabled generic data source is re-enabled so that you can make any other system preparations before the generic data source is re-enabled, such as priming a database or communicating with a high-availability framework.
See the following sections for more details about the callback handler:
WebLogic Server periodically checks the status of generic data sources in a multi data source that were automatically disabled. (See Automatic Re-enablement on Recovery of a Failed Generic Data Source within a Multi Data Source.) If a disabled generic data source becomes available and if a failover callback handler is registered, WebLogic Server calls the callback handler with the following information and waits for a return:
currPool—For failback, this is the name of the generic data source that was previously disabled and is now available to be re-enabled.
nextPool—For failback, this is null.
opcode—A code that indicates the reason for the call. For failback, the code is always OPCODE_REENABLE_CURR_POOL, which indicates that the generic data source named in currPool is now available.
Failback, or automatically re-enabling a disabled generic data source, differs from failover in that failover is synchronous with the connection request, but failback is asynchronous with the connection request.
The callback handler can return one of the following values:
OK—proceed with the operation. In this case, that means to re-enable the indicated generic data source. WebLogic Server resumes routing connection requests to the generic data source, depending on the multi data source algorithm and the position of the generic data source in the list of included generic data sources.
DONOT_FAILOVER—Do not re-enable the currPool generic data source. Continue to serve connection requests from the generic data source(s) in use.
WebLogic Server acts according to the value returned by the callback handler.
If the callback handler returns DONOT_FAILOVER, WebLogic Server will attempt to re-enable the generic data source during the next testing cycle as determined by the TestFrequencySeconds attribute in the multi data source configuration, and will call the callback handler as part of that process.
The order in which generic data sources are listed in a multi data source is very important. A multi data source with the Failover algorithm will always attempt to serve connection requests from the first available generic data source in the list of generic data sources in the multi data source. Consider the following scenario:
MultiDataSource_1 uses the Failover algorithm, has a registered ConnectionPoolFailoverCallbackHandler, and includes three generic data sources: DS1, DS2, and DS3, listed in that order.
DS1 becomes disabled, so MultiDataSource_1 fails over connection requests to DS2.
DS2 then becomes disabled, so MultiDataSource_1 fails over connection requests to DS3.
After some time, DS1 becomes available again and the callback handler allows WebLogic Server to re-enable the generic data source. Future connection requests will be served by DS1 because DS1 is the first generic data source listed in the multi data source.
If DS2 subsequently becomes available and the callback handler allows WebLogic Server to re-enable the generic data source, connection requests will continue to be served by DS1 because DS1 is listed before DS2 in the list of generic data sources.
To deploy a multi data source to a cluster or server, you select the server or cluster as a deployment target. When a multi data source is deployed on a server, WebLogic Server creates an instance of the multi data source on the server. When you deploy a multi data source to a cluster, WebLogic Server creates an instance of the multi data source on each server in the cluster.
For instructions, see Target and deploy JDBC multi data sources in the Oracle WebLogic Server Administration Console Online Help.
Learn how to handle planned maintenance, without service interruption, on the database server used by a multi data source.
Process Overview
The following steps provide a high-level overview of the planned maintenance process:
This topic describes the process of shutting down a Data Source.
Shutting down the data source involves first suspending the data source and then releasing the associated resources including the connections. When a member data source in a Multi Data Source is marked as suspended, the Multi Data Source will not try to get connections from the suspended pool. Instead, to reserve connections, it will go to the next member data source. It is important that you do not shut down all member data sources in a Multi Data source list at the same time. If all members are shut down or fail, then access to the Multi Data Source fails and the application will see failures.
When you gracefully suspend a data source, which is the first step of the shut down process, the following occurs:
The data source is immediately marked as suspended at the beginning of the operation and no further connections are created on the data source.
Idle (not reserved) connections are marked closed
After a timeout period for the suspend operation, all remaining connections in the pool are marked as suspended and the following exception is thrown for any operations on the connection, indicating that the data source is suspended:
java.sql.SQLRecoverableException: Connection has been administratively disabled. Try later.
All the remaining connections are then closed. We won't know until the data source is resumed if they are good or not. In this case, we know that the database will be shut down and the connections in the pool will not be good if the data source is resumed. Instead, we are doing a data source shutdown which will close all of the disabled connections.
The shutdown operation can be done synchronously or asynchronously. If you do a synchronous shutdown, the default timeout period is 60 seconds. You can change the value of this timeout period by configuring or dynamically setting Inactive Connection Timeout Seconds to a non-zero value. There is no upper limit on the inactive timeout period. Note that the processing actually checks for in-use (reserved) resources every tenth of a second so if the timeout value is set to 2 hours and all reserved resources are released a second later, the shut down will complete a second later.If you do an asynchronous operation, the timeout period is specified on the method itself. If set to 0, the default is used. The default is to use Inactive Connection Timeout Seconds if set or 60 seconds. If you want a minimal timeout, set the value to 1. If you want no timeout, set it to a large value (not recommended).
This shutdown operation runs synchronously; there is no asynchronous version of the MBean operation available.
You can also use this for Multi Data Sources configured with either Load-Balancing or Failover.
Example 5-1 WLST Example
The following WLST example script demonstrates how to edit the configuration to increase the suspend timeout period and then use the runtime MBean to shutdown a data source. This script must be integrated into the existing framework for all WebLogic Server servers and data sources.
import sys, socket, os
hostname = socket.gethostname()
datasource='ds'
svr='myserver'
connect("weblogic","password","t3://"+hostname+":7001")
# Shutdown the data source serverRuntime()
cd('/JDBCServiceRuntime/' + svr + '/JDBCDataSourceRuntimeMBeans/' + datasource )
task = cmo.shutdown(10000)
while (task.isRunning ()):
print 'SHUTTING DOWN';
java.lang.Thread.sleep(2000);
print 'Datasource task is in status' + task.getStatus();
exit()
$ java weblogic.WLST myscript2.py
Intializing Weblogic Scripting Tool (WLST)...
Welcome to WebLogic Server Administration Scripting Shell
....
Location changed to serverRuntime tree.
This is a read-only tree with ServerRuntimeMBean as the root. For more help, use help('serverRuntime').
SHUTTING DOWN
Datasource task is in status
SUCCESS
Datasource task is in status
SUCCESS
Exiting WebLogic Scripting Tool.Important Considerations
The following list describes issues you should be aware of when performing planned maintenance:
If the Multi Data Source is using a database service, you cannot stop or relocate the database service before suspending or shutting down the Multi Data Source. If you do, the Multi Data Source may attempt to create a connection to the now missing service and it will react as though the database is down and kill all connections, preventing a graceful shutdown. Because suspending a Multi Data Source ensures that no new connections are created at the associated instance, it is not necessary to stop the service. (Note that the Multi Data Source only creates connections on this instance. It will never create connections on another instance even if it is relocated). Also, suspending a Multi Data Source ceases operations on all connections, therefore no further progress occurs on any sessions (the transactions will not complete) that remain in the Multi Data Source pool.
You may encounter an issue related to XA affinity that is enforced by the Multi Data Source algorithms. When an XA branch is created on an Oracle RAC instance, all additional branches are created on the same instance. While Oracle RAC supports XA across instances, there are some significant limitations that applications run into before the prepare phase, and the Multi Data Source enforces that all operations are on the same instance. As soon as the graceful suspend operation starts, the member data source is marked as suspended and no further connections are allocated there. If an application using global transactions tries to start another branch on the suspending data source, it will fail to get a connection and the transaction fails. In the case of an XA transaction spanning multiple WebLogic servers, the suspend is not graceful. This issue does not apply to Emulate Two-Phase Commit or one-phase commit, which use a single connection for all work, and Logging Last Resource (LLR).
If for some reason you must separate suspending the data source, at which point all connections are disabled, from releasing the resources, you can perform a suspend followed by forceShutdown. You must use a forced shutdown to avoid going through the waiting period a second time. Oracle does not recommend using this process.
To get a graceful shutdown of the data source when shutting down the database, the data source must be involved. This process of shutting down the data source followed by shutdown of the database requires coordination between the mid-tier and the database server processing. Processing is simplified by using Active GridLink instead of Multi Data Source. See Using Active GridLink Data Sources.
When using the Oracle database, Oracle recommends that an application service be configured for each database so that it can be configured for high availability. By using an application service, you can start up the database on its own without the data source starting to use it. Once the application service is explicitly started, the administrator can make the database available to the data source.