Recovering from a Server Failure

14 Recovering from a Server Failure

Understand how WebLogic JMS client applications reconnect or recover from a server/network failure and learn how to migrate JMS data after a server failure.

Automatic JMS Client Failover

WebLogic JMS automatic client reconnection is deprecated and disabled by default. Applications should implement explicit recovery from connection failures.

Note:

The deprecated reconnect settings do not provide automatic reconnection in the default configuration of this release. Applications must handle messaging exceptions and recover their connections, sessions, producers, and consumers as appropriate. See Client Resiliency Best Practices in Administering JMS Resources for Oracle WebLogic Server.

The following automatic-reconnect subsections describe legacy behavior only when the deprecated reconnect mechanism is active. They do not describe the default recovery behavior of this release. Setting ReconnectPolicy on a connection factory or calling setReconnectPolicy() is not, by itself, sufficient to activate that mechanism in the default configuration.

A client connection failure and migration of a JMS server are separate events. Applications must handle failures even when the destination service is configured for high availability. For new application recovery logic, follow the linked client-resiliency guidance.

Automatic Reconnect Limitations

A connection failure can leave the application uncertain whether a send completed. Handle the exception and determine whether retrying is safe under the application's transaction and duplicate-handling design. Retrying can produce another copy of a message.

For retryable business operations, use a stable application identifier and duplicate-aware or idempotent processing. Do not use a queue browse or the absence of a message as proof that the send failed; the message may already have been consumed or expired.

Note:

An unavailable destination can delay investigation or recovery. Its return to service does not, by itself, resolve whether an earlier send completed.

The deprecated reconnect mechanism has additional limitations:

Queue browsers: jakarta.jms.QueueBrowser
The legacy IIOP JMS thin client, wljmsclient.jar, does not participate in this automatic reconnect mechanism. Do not confuse it with the T3 thin client, wlthint3client.jar.
Client statistics are reset on each reconnect, which results in the loss of historical data for the client.
Under some circumstances, automatic reconnect is not possible. If it is not possible, an exception is reported.
Temporary destinations (jakarta.jms.TemporaryQueue and jakarta.jms.TemporaryTopic).

Note:

Do not rely on a temporary destination surviving connection recovery. Its lifetime is associated with its creating connection, and a server or connection failure can invalidate it. Recreate temporary destinations and update any request/reply references as required by the application's recovery design.

Automatic Failover for JMS Producers

When the deprecated automatic-reconnect mechanism is active and its effective policy includes producers, eligible Connection, Session, and MessageProducer objects can be refreshed after a connection failure. Recovery is not guaranteed for every failure, and operations can still throw exceptions. Automatic producer reconnection is disabled by default in this release. See Automatic JMS Client Failover.

Sample Producer Code

Example 14-1 is pseudocode illustrating producer resource use and failure handling. It is not executable Java and does not implement a complete recovery loop. The deprecated automatic-reconnect mechanism is disabled by default.

Example 14-1 Sample JMS Client Code for Message Production

1. Create a JNDI context using the application's connection settings.
2. Look up the connection factory and destination.
3. Create a connection, a nontransacted session, and a producer.
4. Register an exception listener that signals the application's recovery coordinator.
5. For each message:
   a. Create and send the message.
   b. If a JMS operation fails, stop using the failed resources and invoke
      the application's recovery and uncertain-send handling.
      Do not blindly resend a message whose outcome is unknown.
6. Close the connection and JNDI context on completion or failure.

Application recovery logic must coordinate resource cleanup, reconnection attempts, and safe handling of an uncertain send outcome. See Automatic Reconnect Limitations and Client Resiliency Best Practices in Administering JMS Resources for Oracle WebLogic Server.

Reusable ConnectionFactory Objects

A ConnectionFactory object looked up using JNDI is reusable after a server or network failure without requiring a re-lookup. A network failure could be between the JMS client JVM and the remote WebLogic Server instance it is connected to as part of the JNDI lookup, or between the JMS client JVM and any remote WebLogic Server instance in the same cluster where the JMS client subsequently connects. See Example 14-1 and Example 14-2. Reusing the object does not guarantee that the corresponding service is currently available; handle failures from subsequent operations.

Reusable Destination Objects

A destination object (queue or topic) looked up using JNDI is reusable after a server or network failure without requiring another lookup. The same principle applies to producers that send to a distributed destination, because the client looks up the distributed destination in JNDI, and not the unavailable distributed member. See Example 14-1 and Example 14-2. Reusing the object does not guarantee that the corresponding service is currently available; handle failures from subsequent operations.

A network failure could be between the client JVM and the WebLogic Server instance it is connected to, or between that WebLogic Server instance and the WebLogic Server instance that actually hosts the destination. The Destination object will also be robust after restarting the WebLogic Server instance hosting the destination.

Note:

For information on how consumers of distributed destinations behave with automatic JMS client reconnect, see Consumers of Distributed Destinations.

Reconnected Connection Objects

A JMS Connection is a logical messaging connection; it does not necessarily correspond to a dedicated physical network connection. When the deprecated reconnect mechanism is active and refresh succeeds, the same application-visible Connection object can communicate through a replacement server-side connection.

Eligible derived objects are refreshed according to the effective reconnect policy and the limitations described in this section. A synchronous operation can wait for reconnect processing according to ReconnectBlockingMillis, and can still fail. This setting is not a general timeout for every messaging or network operation.

Background retry duration is governed by TotalReconnectPeriodMillis. Its legacy default of -1 permits indefinite retries; a finite setting limits the retry period. Closing the connection stops its reconnect attempts.

Handle failures and uncertain operation outcomes even when a refresh succeeds. See Automatic Reconnect Limitations and Configuring Automatic Client Refresh Options.

Special Cases for Reconnected Connections

There are special cases that can occur when producer connections are refreshed:

Connections with a ClientID for Durable Subscribers – If your Reconnect Policy field is set to None or Producer, and a JMS Connection has a Client ID specified at the time of a network/server failure, then the Connection will not be automatically refreshed. The reason for this restriction is backward compatibility, which avoids breaking existing JMS applications that try to re-create a JMS Connection with the same connection name after a failure. If implicit failover also occurs on a network failure, then the application's creation of the connection will fail due to a duplicate ClientID.

Note:

For information on how a consumer connection with a ClientID behaves, see Consumer Connections with a ClientID for Durable Subscriptions.
Closed Objects Are Not Refreshed – When the application calls jakarta.jms.Connection.close(), jakarta.jms.Session.close(), etc., that object and it descendents are not refreshed. Similarly, when the JMS client is told its Connection has been administratively destroyed, it is not refreshed.
Connection with Registered Exception Listener – If the JMS Connection has an application ExceptionListener registered on it, that ExceptionListener's onException() callback will be invoked even if the connection is implicitly refreshed. This notifies the application code of the network disconnect event. The JMS client application code might normally call connection.close() in onException; however, if it wants to take advantage of the reconnect feature, it may choose not to call connection.close(). The registered ExceptionListener is also migrated transparently to the internally refreshed connection to listen for exceptions on the refreshed connection.
Multiple Connections – If there are multiple JMS Connections created off the same ConnectionFactory object, each connection will behave independently of the other connections as far as the reconnect feature is concerned. Each connection will have its own connection status, its own connection retry machinery, etc.

Reconnected Session Objects

As described in Reconnected Connection Objects, JMS Session objects are refreshed when their associated JMS connection gets refreshed . Session states, such as acknowledge mode and transaction mode, are preserved across each refresh occurrence. The same session object can be used for calls, like createProducer(Destination), after a refresh. See Example 14-1 and Example 14-2.

Special Cases for Reconnected Sessions

These sections discuss special cases that can occur when Sessions are reconnected.

Transacted sessions with pending work — When a local JMS session is refreshed with pending transactional work, its first commit can throw jakarta.jms.TransactionRolledBackException. Handle this together with other messaging and transaction-completion exceptions. A refreshed connection does not guarantee that subsequent sends or receives succeed.

After a confirmed rollback, retry the logical unit of work only when doing so is safe for all effects of that work. Do not treat every commit exception as proof of rollback or blindly repeat external effects that did not participate in the JMS transaction.
Pending unacknowledged messages — A refreshed session with pending unacknowledged work can report weblogic.jms.common.LostServerException on acknowledgement. The exception indicates a connection-loss condition; handle possible redelivery and do not assume that the acknowledgement removed the earlier deliveries.

Reconnected MessageProducer Objects

As described in Reconnected Connection Objects, JMS MessageProducer objects are refreshed when their associated JMS connection gets refreshed . If producers are non-anonymous, that is, they are specific to a destination object (standalone or distributed destination), then the producer's destination is also implicitly refreshed, as described in Reusable Destination Objects. If a producer is anonymous, that is not specific to a destination object, then the possibly stale destination object specified on the producer's send() operation is implicitly refreshed. See Example 14-1.

Special Case for Distributed Destinations

It is possible that a producer can send a message at the same time that a distributed destination member becomes unavailable. If WebLogic JMS can determine that the distributed destination member is not available, or was not available when the message was sent, the system will retry sending the message to another distributed member. If there is no way to determine if the message made it through the connection all the way to the distributed member before it went down, the system will not attempt to resend the message because doing so may create a duplicate message. In that case, WebLogic JMS will throw an exception. It is up to the application to catch that exception and decide whether or not to resend the message.

Configuring Automatic Failover for JMS Consumers

When the deprecated mechanism is active, an effective ReconnectPolicy of all allows eligible MessageConsumer objects to participate in connection refresh. Calling WLConnection.setReconnectPolicy("all") alone does not enable automatic reconnection in the default configuration of this release. See Automatic JMS Client Failover.

Explicitly enabling automatic refresh of consumers also refreshes connections with a configured client ID for a durable subscriber, as described in Consumer Connections with a ClientID for Durable Subscriptions. However, consumer refresh does not include QueueBrowser clients, which are never refreshed, as described in Automatic Reconnect Limitations.

Sample Consumer Client Code

Example 14-2 is pseudocode illustrating consumer resource use and failure handling. It is not executable Java and does not implement a complete recovery loop. The deprecated automatic-reconnect mechanism is disabled by default.

Example 14-2 Sample JMS Client Code for Message Consumption

1. Create a JNDI context using the application's connection settings.
2. Look up the connection factory and destination.
3. Create a connection and session using the required acknowledgement
   or transaction mode.
4. Register an exception listener that signals the application's recovery coordinator.
5. Create the consumer; register a message listener for asynchronous delivery.
6. Start the connection.
7. Receive and process messages, completing acknowledgements or transactions
   according to the selected mode.
8. On failure, coordinate cleanup and recreation of the failed resources.
   Restore the intended subscription identity and handle possible redelivery.
9. Close the connection and JNDI context on completion or failure.

Use the application's recovery logic to restore consumer resources after a connection failure. Do not assume that calling setReconnectPolicy("all") activates recovery in the default configuration. See Client Resiliency Best Practices in Administering JMS Resources for Oracle WebLogic Server.

Configuring Automatic Client Refresh Options

The following deprecated settings describe the legacy reconnect mechanism. They do not enable automatic reconnection in this release's default configuration. The defaults shown are descriptor defaults, not a promise that reconnect is active.

Table 14-1 Automatic JMS Client Reconnect Options

MBean Attribute Legacy values or default Description

MBean Attribute	Legacy values or default	Description
Reconnect Policy `ReconnectPolicy`	`none` `producer` (descriptor default) `all`	When the deprecated mechanism is active, controls which eligible connection, session, producer and consumer objects participate in refresh. QueueBrowser objects are not refreshed. The current default effective policy is none because automatic reconnection is disabled.
Reconnect Blocking Time `ReconnectBlockingMillis`	60000 (60 seconds)	Limits a synchronous caller's wait for reconnect processing when the deprecated mechanism is active. A value of 0 disables that wait; -1 permits an indefinite wait. It is not a general end-to-end operation timeout.
`TotalReconnectPeriodMillis`	-1	Controls the total period during which the deprecated mechanism attempts reconnection. The default value of -1 permits retries without a configured time limit. Closing the connection stops reconnect attempts.

Reconnect Policy

ReconnectPolicy

none
producer (descriptor default)
all

When the deprecated mechanism is active, controls which eligible connection, session, producer and consumer objects participate in refresh. QueueBrowser objects are not refreshed. The current default effective policy is none because automatic reconnection is disabled.

Reconnect Blocking Time

ReconnectBlockingMillis

60000 (60 seconds)

Limits a synchronous caller's wait for reconnect processing when the deprecated mechanism is active. A value of 0 disables that wait; -1 permits an indefinite wait. It is not a general end-to-end operation timeout.

TotalReconnectPeriodMillis

-1

Controls the total period during which the deprecated mechanism attempts reconnection. The default value of -1 permits retries without a configured time limit. Closing the connection stops reconnect attempts.

Note:

The configuration options in the table are not supported in the WebLogic Remote Console, as these options are for an already deprecated feature.

For more information about these parameters, see ClientParamsBean in the MBean Reference for Oracle WebLogic Server.

Common Cases for Reconnected Consumers

This section describes the common scenarios when refreshing synchronous and asynchronous consumers.

Synchronous Consumers

Use MessageConsumer.receive(), receive(timeout), or receiveNoWait() according to the application's message-waiting requirements. In normal operation, receive(timeout) can return null when no message arrives within the requested interval, and receiveNoWait() can return null when no message is immediately available. A connection failure can instead cause a JMSException.

When the deprecated automatic-reconnect mechanism is active, reconnect processing can add waiting before the receive operation proceeds. Do not use receive(timeout) as a strict end-to-end bound on a call that encounters reconnection, or assume that receiveNoWait() bypasses all reconnect processing. ReconnectBlockingMillis controls reconnect waiting rather than the normal wait for a message.

After recovery, consumers must handle possible redelivery. Handle synchronous exceptions as well as asynchronous connection notifications. For acknowledgement and transaction completion after legacy refresh, see Special Cases for Reconnected Sessions.

Asynchronous Consumers

In the context of a reconnect, the behavior for asynchronous consumers will be governed by the setting on the Total Reconnect Period property. The JMS consumer's registered message listener's onMessage() method will continue to be invoked if the reconnect framework is able to successfully re-establish a connection within the Total Reconnect Period setting after a connection failure. If the user explicitly calls a close() on the JMS Connection (or on the JMS Session corresponding to the asynchronous Consumer), then the reconnect framework will not invoke any further onMessage() calls for that Consumer. The onMessage() should expect post recover behavior (like redelivered messages) if the Connection ExceptionListener's onException is invoked with a LostServerException.

Special Cases for Reconnected Consumers

These sections discuss special cases that can occur when consumers are refreshed.

Consumers of Distributed Destinations

A consumer attached to a distributed destination receives from a selected physical member. If a consumer is recreated during recovery, member selection can choose the same member or a different member, depending on the destination and configuration. Do not rely on recovery to preserve member selection.

Create durable subscriptions on physical distributed-topic members, or use MDBs or documented helpers to manage those member subscriptions. Direct creation of a durable consumer on the logical distributed-topic object is not supported. See Considerations When Using Standard Shared Subscriptions.

Message-Driven EJBs

Message-driven EJBs (MDBs) are a special sub case of asynchronous consumers that have their own behavior requirements and their own refresh framework. As such, MDBs are not expected to participate in MessageConsumer refreshes, and are not expected to be affected in any other way by the JMS client reconnect framework.

Consumer Connections with a ClientID for Durable Subscriptions

When the deprecated mechanism is active, a connection with a client ID is eligible for refresh only when the effective reconnect policy includes consumers. Successful recovery of a durable consumer depends on the availability of its durable subscription. Preserve its identity when recreating it, and handle exceptions and possible redelivery.

Note:

A Restricted client ID can delay reconnection while the previous connection's registration is being released. Client ID Policy and subscription identity affect recovery; see Shared Subscriptions and Client ID Policy.

Durable subscriptions on different physical distributed-topic members remain separate subscriptions, even when their names and client IDs match. Recovery does not move a member's retained messages into another member's subscription.

Non-Durable Subscriptions and Possible Missed Messages

A non-durable subscription does not retain messages after the subscription ceases to exist. Messages published while a failed subscription is absent can be missed when it is recreated. For a shared non-durable subscription, failure of one consumer does not by itself remove the subscription if other consumers remain attached. Recovery must account for both the subscription's lifetime and the messages eligible for that subscription.

Duplicate Messages

Connection recovery can cause previously delivered messages to be delivered again, even when the application did not explicitly call recover(). Design processing for the acknowledgement and transaction mode in use, including its redelivery possibilities. Recovery does not establish an exactly-once processing guarantee.

A non-durable RDT consumer that reconnects to a different member can also encounter a replicated copy of a message it received earlier. Such a copy need not be marked as redelivered to that new subscription. Do not rely solely on JMSRedelivered to identify duplicate business work.

Variations Due to Acknowledge Modes

The acknowledgement or transaction mode determines how an application completes its messaging work. After legacy refresh, an acknowledgement can fail when the session had pending unacknowledged work. Locally transacted sessions instead complete work through commit or rollback. See Special Cases for Reconnected Sessions.

Reconnecting with Migrated JMS Destinations In a Cluster

Migrating a JMS server moves the destination service; it does not migrate application-side Connection, Session, or MessageConsumer objects. A client can receive a synchronous JMSException or an asynchronous ExceptionListener notification when its connection fails. Coordinate cleanup and recreation of the affected client resources after the service becomes available, preserving the intended durable-subscription identity. See Client Resiliency Best Practices in Administering JMS Resources for Oracle WebLogic Server.

Explicitly Disabling Automatic Failover on JMS Clients

Automatic JMS client reconnection is disabled by default in this release. The following settings apply only to environments where the deprecated mechanism is active.

Programmatically

For an existing WebLogic JMS Connection named con, the following legacy API call requests a policy of none. The enclosing method must have created con and must handle connection-creation and messaging errors.

((weblogic.jms.extensions.WLConnection) con).setReconnectPolicy("none");

For more information about the setReconnectPolicy method, see the weblogic.jms.extensions.WLConnection API.

Administratively

Where the deprecated reconnect mechanism is active, set the connection factory's ReconnectPolicy attribute to none. Connection-factory changes apply to newly created connections. These settings are not exposed by WebLogic Remote Console.

For more information about the other JMS connection factory client parameters, see ClientParamsBean in the MBean Reference for Oracle WebLogic Server.

Best Practices for JMS Clients Using Automatic Failover

Oracle recommends the following best practices for JMS clients when using the Automatic JMS Client Reconnect feature:

Always Catch Exceptions

There are some connection failure scenarios where the result of a message operation is undetermined and WebLogic Server throws an exception. Your application must deal with the exception appropriately. See the following:

Use Transactions to Group Message Work

Use a local JMS transaction to group sends and receives on one session, or an appropriate JTA transaction to coordinate enlisted resources. Only operations participating in that transaction share its completion outcome. Handle completion exceptions and resolve uncertain outcomes before retrying work; a server failure does not by itself prove that an earlier commit failed.

See Using Transactions with WebLogic JMS.

Note:

Transaction recovery can continue after a server or client connection is restored. A local JMS TransactionRolledBackException reports rollback, but other completion failures can require further investigation. For JTA, follow the transaction manager's recovery and heuristic-outcome guidance.

JMS Clients Should Always Call the close() Method

As a best practice, your applications should not rely on the JVM's garbage collection to clean up JMS connections because the JMS automatic reconnect feature keeps a reference to the JMS connection. Therefore, always use connection.close() to clean up your connections. Also consider using a finally block to ensure that your connection resources are cleaned up. Otherwise, WebLogic Server allocates system resources to keep the connection available.

For more information about closing JMS client connections, see Best Practice: Always Close Failed JMS ClientIDs.

Manually Migrating JMS Data to a New Server

WebLogic JMS uses the migration framework to allow WebLogic JMS to respond properly to migration requests and bring a WebLogic JMS server online and offline in an orderly fashion. This includes both scheduled migrations as well as migrations in response to a WebLogic Server failure.

A JMS server can move between eligible WebLogic Server hosts when its targeting, migration policy, persistent store, and recovery configuration support that move. Use the service-migration procedure appropriate to that configuration.

Before activating the recovered service, preserve access to its existing persistent-store data and ensure that the previous host cannot concurrently activate the same service or write to the same store. The Table 14-2 summarizes recovery checks; it does not replace the linked migration and persistent-store procedures.

Note:

There are special considerations when you migrate a service from a server instance that has crashed or is unavailable to the Administration Server. If the Administration Server cannot reach the previously active host of the service at the time you perform the migration, see Migrating a Service From an Unavailable Server in Administering Clusters for Oracle WebLogic Server.

Table 14-2 Migration Task Guide

If Your JMS Application Uses	Perform the Following Task
Persistent messaging—JDBC Store	Ensure that the recovery host can access the existing JDBC store's database and backing table through the correctly configured data source. If the database endpoint changes, update the data-source configuration as required. A database that remained available still requires the JMS service, store targeting, access and ownership checks.
Persistent messaging—File Store	Ensure that the recovery host can access the existing store's data files through the configured store directory. If files must be copied, first ensure that the store is inactive and cannot restart during the copy, and preserve a consistent complete set of its files. Configure the recovery location according to the persistent-store recovery procedure before activating the service. See Using Custom File Stores and File Locations in Administering the WebLogic Persistent Store.
Transactions	Coordinate JMS store recovery with transaction recovery. The Transaction Recovery Service needs access to the failed server's transaction-log records to resolve outstanding transactions. For detailed instructions about recovering transactions from a failed server, see Transaction Recovery After a Server Fails in Developing JTA Applications for Oracle WebLogic Server.

If Your JMS Application Uses

Perform the Following Task

Persistent messaging—JDBC Store

Ensure that the recovery host can access the existing JDBC store's database and backing table through the correctly configured data source. If the database endpoint changes, update the data-source configuration as required. A database that remained available still requires the JMS service, store targeting, access and ownership checks.

Persistent messaging—File Store

Ensure that the recovery host can access the existing store's data files through the configured store directory. If files must be copied, first ensure that the store is inactive and cannot restart during the copy, and preserve a consistent complete set of its files. Configure the recovery location according to the persistent-store recovery procedure before activating the service. See Using Custom File Stores and File Locations in Administering the WebLogic Persistent Store.

Transactions

Coordinate JMS store recovery with transaction recovery. The Transaction Recovery Service needs access to the failed server's transaction-log records to resolve outstanding transactions.

For detailed instructions about recovering transactions from a failed server, see Transaction Recovery After a Server Fails in Developing JTA Applications for Oracle WebLogic Server.

Note:

Recovery of a persistent store containing many messages can require substantial memory. If startup fails because of insufficient memory, investigate the failure and review JVM heap sizing and available host memory before retrying. Recovery requirements depend on the stored workload and configuration; do not size the heap solely in proportion to message count.

See Starting and Stopping Servers: Quick Reference. For information about recovering a failed server, refer to Avoiding and Recovering From Server Failure in Administering Server Startup and Shutdown for Oracle WebLogic Server.

For more information about defining migratable services, see Service Migration in Administering Clusters for Oracle WebLogic Server.