Programming WebLogic JMS

Recovering from a WebLogic Server Failure

These sections discuss the JMS client reconnect feature, which automatically refreshes most JMS clients after a network failure, how to terminate a JMS application gracefully if a server fails, and how to migrate JMS data after a server failure.

Automatic Failover for JMS Clients

With the automatic JMS client reconnect feature, if a server or network failure occurs, some JMS client objects will transparently failover to use a another server instance, as long as one is available. For example, if a fatal server failure occurs, JMS clients automatically attempt to reconnect to the server when it becomes available. A network connection failure could be due to transient reasons (a temporary blip in the network connection) or non-transient reasons (a server bounce or network failure). In such cases, some JMS client objects will attempt to automatically operate with another server instance in a cluster, or possibly with the host server.

Automatic Failover for JMS Producers

By default, JMS producer session objects automatically attempt to reconnect to an available server instance without any manual configuration or modifications to existing client code. If you do not want your JMS producers to be automatically reconnected, then you must explicitly disable this feature either programmatically or administratively.

The following WebLogic JMS producer objects will attempt to automatically reconnect to an available sever instance without any manual configuration or modification to existing client code:

Connection
Session
MessageProducer

As shown in the sample client code inListing 13-1, the WebLogic JMS client code reconnects in the event of a network failure during Steps 3-8.

Listing 13-1 Sample JMS Client Code for Message Production

0.   Context ctx = create WebLogic JNDI context with credentials etc.
1.   ConnectionFactory cf = ctx.lookup(JNDI name of connection factory)
2.   Destination dest     = ctx.lookup(JNDI name of destination)
     // the following operations recover from network failures
3.   Connection  con      = cf.createConnection()
4.   Session     sess     = con.createSession(no transactions, ack mode)
5.   MessageProducer prod = sess.createProducer(dest)
6.   Loop over:
7.      Message msg = sess.createMessage() 
8.      prod.send(msg)
9.   con.close(); ctx.close()

The JMS client will transparently failover to another server instance if feasible. This keeps the client code as simple as listed above and eliminates the need for client code for retrying across network failures.

Re-usable Connection Factory Objects

Since Weblogic Server 8.1, a ConnectionFactory object looked up via JNDI (see Step 1 in Listing 13-1) is re-usable after a server or network failure without requiring a re-lookup. A network failure could be between the JMS client JVM and the remote WebLogic Server instance it is connected to as part of the JNDI lookup, or between the JMS client JVM and any remote WebLogic Server instance in the same cluster that the JMS client subsequently connects to.

Re-usable Destination Objects

A Destination object (Queue or Topic) looked up via JNDI (see Step 2 in Listing 13-1) is re-usable after a server or network failure without requiring a re-lookup. The same principle applies to producers that send to a distributed destinations, since the client looks up the distributed destination in JNDI, and not the unavailable distributed member.

A network failure could be between the client JVM and the WebLogic Server instance it is connected to, or between that WebLogic Server instance and the WebLogic Server instance that actually hosts the destination. The Destination object will also be robust after restarting the WebLogic Server instance hosting the destination.

Reconnected Connection Objects

The JMS Connection object is used to map one-to-one to a physical network connection between the client JVM and a remote WebLogic Server instance. With the JMS client reconnect feature, the JMS Connection object that the client gets from the ConnectionFactory.createConnection() method (see Step 3 in Listing 13-1) maps in a one-to-one-at-a-time fashion to the physical network connection. One consequence is that while the JMS client continues to use the same Connection object, it could be actually communicating with a different WebLogic Server instance after an implicit failover.

If there is a network disconnect and a subsequent implicit refresh of the connection, all objects derived from the connection (such as javax.jms.Session and javax.jms.MessageProducer objects) are also implicitly refreshed. During the refresh, any synchronous operation on the connection or its derived objects that go to the server (such as producer.send() or connection.createSession()), may block for a period of time before giving up on the connection refresh. This time is controlled by the setReconnectBlockingMillis(long) method. in the weblogic.jms.extension.WLConnection interface.

The reconnect feature keeps trying to reconnect to the Weblogic Server instance's ConnectionFactory object in the background until the application calls connection.close(). The ReconnectBlockingMillis parameter is the time-out for a synchronous caller trying to use the connection when the connection in being retried in the background.

If a synchronous call does time out without seeing a refreshed connection, it then behaves in exactly the same way (that is, throws the same Exceptions) as without the implicit reconnect (that is, it will behave as if it was called on a stale connection without the reconnect feature).

The caller can then decide to simply retry the synchronous call (with a potentially lower quality of service, like duplicate messages), or decide to call connection.close(), which will terminate the background retries for that connection.

Special Cases for Reconnected Connections

These sections discuss special cases that can occur when Connections are refreshed.

Connections with a Client ID for Durable Subscribers

If a JMS Connection has a Client ID specified at the time of a network/server failure, then the Connection will not be automatically refreshed. The reason for this restriction is backward compatibility, and so avoids breaking existing JMS applications that try to re-create a JMS Connection with the same connection name after a failure. If implicit failover also occurs on a network failure, then the application's creation of the connection will fail due to a duplicate client ID.

Closed Objects Are Not Refreshed

When the application calls javax.jms.Connection.close(), javax.jms.Session.close(), etc., that object and it descendents are not refreshed. Similarly, when the JMS client is told its Connection has been administratively destroyed, it is not refreshed.

Connection with Registered Exception Listener

If the JMS Connection has an application ExceptionListener registered on it, that ExceptionListener's onException() callback will be invoked even if the connection is implicitly refreshed. This notifies the application code of the network disconnect event. The JMS client application code might normally call connection.close() in onException; however, if it wants to take advantage of the reconnect feature, it may choose not to call connection.close(). The registered ExceptionListener is also migrated transparently to the internally refreshed connection to listen for exceptions on the refreshed connection.

Multiple Connections

If there are multiple JMS Connections created off the same ConnectionFactory object, each connection will behave independently of the other connections as far as the reconnect feature is concerned. Each connection will have its own connection status, its own connection retry machinery, etc.

Reconnected Session Objects

As described in Reconnected Connection Objects on page 13-3, JMS Session objects that are part of JMS Connection objects are implicitly refreshed when the JMS Connection is refreshed (see Step 4 in Listing 13-1). Session states, such as acknowledge mode and transaction mode, are preserved across each refresh occurrence. The same session object can be used for calls, like createMessageProducer(), after a refresh.

Special Cases for Reconnected Sessions

These sections discuss special cases that can occur when Sessions are refreshed.

Transacted Sessions With Pending Commits or Rollbacks

Similar to non-transacted JMS Sessions, transacted JMS sessions are automatically refreshed. However, if there were send or receive operations on a Session pending a commit or rollback at the time of the network disconnect, then the first commit call after the Session refresh will fail throwing a javax.jms.TransactionRolledBackException. This occurs because when a JMS Session transaction spans a network refresh, the commit for that transaction cannot vouch for the operations done as part of that transaction (from an application code perspective) prior to the refresh.

After a Session refresh, operations like send or receive will not throw an exception; it is only the first commit after a refresh that will throw an exception. However, the first commit after a Session refresh will not throw an exception if there were no pending transactional operations in that JMS session at the time of the network disconnect. In case of Session.commit() throwing the exception, the client application code can simply retry all the operations in the transaction again with the same (implicitly refreshed) JMS objects.

Pending Unacknowledged Messages

If a Session had unacknowledged messages prior to the Session refresh, then the first WLSession.acknowledge() call after a refresh will throw a weblogic.jms.common.LostServerException. This indicates that the acknowledge() call may have not removed messages from the server. As a result, the refreshed Session may receive duplicate messages.

Reconnected Message Producer Objects

As described in Reconnected Connection Objects on page 13-3, JMS MessageProducer objects are refreshed when their associated JMS connection gets refreshed (see Step 5 in Listing 13-1). If producers are non-anonymous, that is, they are specific to a Destination object (standalone or distributed destination), then the producer's destination is also implicitly refreshed, as described in Reconnected Destination Objects. If a producer is anonymous, that is not specific to a Destination object, then the possibly-stale Destination object specified on the producer's send() operation is implicitly refreshed.

Special Case for Reconnected MessageProducers and Distributed Destinations

It is possible that a producer can send a message at the same time that a distributed destination member becomes unavailable. If WebLogic JMS can determine that the distributed destination member is not available, or was not available when the message was sent, the system will retry sending the message to another distributed member. If there is no way to determine if the message made it through the connection all the way to the distributed member before it went down, the system will not attempt to resend the message because doing so may create a duplicate message. In that case, WebLogic JMS will throw an exception. It is up to the application to catch that exception and decide whether or not to resend the message.

Limitations for Automatic JMS Client Failover

Implicit refresh of the following objects is not supported WebLogic Server 9.1:

javax.jms.MessageConsumer
javax.jms.QueueBrowser
The WebLogic JMS Thin client on will not reconnect For more information, see "WebLogic JMS Thin Client" in Programming Stand-alone Clients.
Client statistics are be reset on each reconnect, which results in the loss historical data for the client.
Temporary destinations: javax.jms.TemporaryQueue and javax.jms.TemporaryTopic

Note: Temporary destinations may still be accessible after a sever/network failure. This is because temporary destinations are not always on the same server instance as the local connection factory due to server load balancing. Therefore, if a temporary destination survives a server/network failure and a producer continues sending messages to it, an auto-reconnected consumer may or may not be able consume messages from the same temporary destination it was connected to before the failure occurred.

Turning off Automatic Reconnect Programatically

If you do not want your JMS clients to be automatically reconnected, then your applications should call the following code:

ConnectionFactory cf = javax.jms.ConnectionFactory)ctx.lookup(JNDI name of connection factory)
javax.jms.Connection con = cf.createConnection();
((weblogic.jms.extensions.WLConnection)con).setReconnectPolicy("none")

Best Practices for JMS Clients Using Automatic Failover

BEA recommends the following best practices for JMS clients when using the Automatic JMS Client Reconnect feature:

Use Transactions to Group Message Work

Use transacted sessions (JMS) or user transactions (JTA) to group related or dependent work, including messaging work, so that either all of the work is completed or none of it is. If a server instance goes down and a message is lost in the middle of a transaction, the entire transaction is rolled back and the application does not need to make a decision for each message after a failure.

Note: Be aware of transaction commit failures after a server reconnect, which may occur if the transaction subsystem cannot reach all the participants involved in the transaction.

JMS Clients Should Always Call the close() Method

As a best practice, your applications should not rely on the JVM's garbage collection to clean up JMS connections because the JMS automatic reconnect feature keeps a reference to the JMS connection. Therefore, always use connection.close() to clean up your connections. Also consider using a Finally block to ensure that your connection resources are cleaned up. Otherwise, the WebLogic Server allocates system resources to keep the connection available.

Programming Considerations for WebLogic Server 9.0 or Earlier Failures

JMS client applications running on Weblogic Server 9.0 or earlier had to reestablish javax.jms objects after a server failure. If you are still running release 9.0 or earlier JMS clients, you may want to program your JMS clients to terminate gracefully in the event of a server failure. For example:

Table 13-1 Programming Considerations for Server Failures

If a WebLogic Server Instance Fails and...	Then...
You are connected to the failed WebLogic Server instance	A `JMSException` is delivered to the connection exception listener. You must restart the application once the server is restarted or replaced.
A JMS Server is targeted on the failed WebLogic Server instance	A `ConsumerClosedException` is delivered to the session exception listener. You must re-establish any message consumers that have been lost.

Migrating JMS Data to a New Server

WebLogic JMS uses the migration framework implemented in the WebLogic Server core, which allows WebLogic JMS respond properly to migration requests and bring a WebLogic JMS server online and offline in an orderly fashion. This includes both scheduled migrations as well as migrations in response to a WebLogic Server failure.

Once properly configured, a JMS server and all of its destinations can migrate to another WebLogic Server within a cluster.

You can recover JMS data from a failed WebLogic Server by starting a new server and doing one or more of the tasks in Table 13-2.

Note: There are special considerations when you migrate a service from a server instance that has crashed or is unavailable to the Administration Server. If the Administration Server cannot reach the previously active host of the service at the time you perform the migration, see Migrating a Service When Currently Active Host is Unavailable in Using WebLogic Server Clusters.

Table 13-2 Migration Task Guide

If Your JMS Application Uses. . .	Perform the Following Task. . .
Persistent messaging—JDBC Store	If the JDBC database store physically exists on the failed server, migrate the database to a new server and ensure that the JDBC connection pool URL attribute reflects the appropriate location reference. If the JDBC database does not physically exist on the failed server, access to the database has not been impacted, and no changes are required.
Persistent messaging—File Store	Migrate the file to the new server, ensuring that the pathname within the WebLogic Server home directory is the same as it was on the original server.
Transactions	To facilitate recovery after a crash, WebLogic Server provides the Transaction Recovery Service, which automatically attempts to recover transactions on system startup. The Transaction Recovery Service owns the transaction log for a server. For detailed instructions on recovering transactions from a failed server, see "Transaction Recovery After a Server Fails" in Programming WebLogic JTA.

If Your JMS Application Uses. . .

Perform the Following Task. . .

Persistent messaging—JDBC Store

If the JDBC database store physically exists on the failed server, migrate the database to a new server and ensure that the JDBC connection pool URL attribute reflects the appropriate location reference.

If the JDBC database does not physically exist on the failed server, access to the database has not been impacted, and no changes are required.

Persistent messaging—File Store

Migrate the file to the new server, ensuring that the pathname within the WebLogic Server home directory is the same as it was on the original server.

Transactions

To facilitate recovery after a crash, WebLogic Server provides the Transaction Recovery Service, which automatically attempts to recover transactions on system startup. The Transaction Recovery Service owns the transaction log for a server.

For detailed instructions on recovering transactions from a failed server, see "Transaction Recovery After a Server Fails" in Programming WebLogic JTA.

Note: JMS persistent stores can increase the amount of memory required during initialization of WebLogic Server as the number of stored messages increases. When rebooting WebLogic Server, if initialization fails due to insufficient memory, increase the heap size of the Java Virtual Machine (JVM) proportionally to the number of messages that are currently stored in the JMS persistent store and try the reboot again.

For information about starting a new WebLogic Server, see the "Starting and Stopping Servers: Quick Reference". For information about recovering a failed server, refer to Avoiding and Recovering From Server Failure in Managing Server Startup and Shutdown.

For more information about defining migratable services, see "Service Migration" in Using WebLogic Server Clusters.