Review a checklist of items to consider when tuning WebLogic JMS.
Always configure quotas, see Defining Quota.
Verify that default paging settings apply to your needs, see Paging Out Messages To Free Up Memory. Paging lowers performance but may be required if JVM memory is insufficient.
Avoid large message backlogs. See Handling Large Message Backlogs.
Create and use custom connection factories with all applications instead of using default connection factories, including when using MDBs. Default connection factories are not tunable, while custom connection factories provide many options for performance tuning.
Write applications so that they cache and re-use JMS client resources, including JNDI contexts and lookups, and JMS connections, sessions, consumers, or producers. These resources are relatively expensive to create. For information on detecting when caching is needed, as well as on built-in pooling features, see Cache and Re-use Client Resources.
For asynchronous consumers and MDBs, tune
MessagesMaximum on the connection factory. Increasing
MessagesMaximum can improve performance, decreasing
MessagesMaximum to its minimum value can lower performance, but helps ensure that messages do not end up waiting for a consumer that's already processing a message. See Tuning MessageMaximum.
Avoid single threaded processing when possible. Use multiple concurrent producers and consumers and ensure that enough threads are available to service them.
Tune server-side applications so that they have enough instances. Consider creating dedicated thread pools for these applications. See Tuning Message-Driven Beans.
For client-side applications with asynchronous consumers, tune client-side thread pools using Client-side Thread Pools.
Tune persistence as described in Tuning the WebLogic Persistent Store. In particular, it's normally best for multiple JMS servers, destinations, and other services to share the same store so that the store can aggregate concurrent requests into single physical I/O requests, and to reduce the chance that a JTA transaction spans more than one store. Multiple stores should only be considered once it's been established that the a single store is not scaling to handle the current load.
If you have large messages, see Tuning for Large Messages.
Prevent unnecessary message routing in a cluster by carefully configuring connection factory targets. Messages potentially route through two servers, as they flow from a client, through the client's connection host, and then on to a final destination. For server-side applications, target connection factories to the cluster. For client-side applications that work with a distributed destination, target connection factories only to servers that host the distributed destinations members. For client-side applications that work with a singleton destination, target the connection factory to the same server that hosts the destination.
If JTA transactions include both JMS and JDBC operations, consider enabling the JDBC LLR optimization. LLR is a commonly used safe "ACID" optimization that can lead to significant performance improvements, with some drawbacks. See Tuning Transactions.
If you are using Java clients, avoid thin Java clients except when a small jar size is more important than performance. Thin clients use the slower IIOP protocol even when T3 is specified so use a full java client instead. See Developing Standalone Clients for Oracle WebLogic Server.
Tune JMS Store-and-Forward according to Tuning WebLogic JMS Store-and-Forward.
Tune a WebLogic Messaging Bridge according Tuning WebLogic Message Bridge.
If you are using non-persistent non-transactional remote producer clients, then consider enabling one-way calls. See Using One-Way Message Sends.
Consider using JMS distributed queues. See Using Distributed Queues in Developing JMS Applications for Oracle WebLogic Server.
If you are already using distributed queues, see Tuning Distributed Queues.
Consider using advanced distributed topic features (PDTs). See Developing Advanced Pub/Sub Applications in Developing JMS Applications for Oracle WebLogic Server.
If your applications use Topics, see Tuning Topics.
Avoid configuring sorted destinations, including priority sorted destinations. FIFO or LIFO destinations are the most efficient. Destination sorting can be expensive when there are large message backlogs, even a backlog of a few hundred messages can lower performance.
Use careful selector design. See Filtering Messages in Developing JMS Applications for Oracle WebLogic Server.
Run applications on the same WebLogic Servers that are also hosting destinations. This eliminates networking and some or all marshalling overhead, and can heavily reduce network and CPU usage. It also helps ensure that transactions are local to a single server. This is one of the major advantages of using an application server's embedded messaging.
When message senders inject messages faster than consumers, messages accumulate into a message backlog.
Large backlogs can be problematic for a number of reasons, for example:
Indicates consumers may not be capable of handling the incoming message load, are failing, or are not properly load balanced across a distributed queue.
Can lead to out-of-memory on the server, which in turn prevents the server from doing any work.
Can lead to high garbage collection (GC) overhead. A JVM's GC overhead is partially proportional to the number of live objects in the JVM.
One area for investigation is to improve overall message processing performance. Here are some suggestions:
Follow the JMS tuning recommendations as described in JMS Performance & Tuning Check List.
Check for programming errors in newly developed applications. In particular, ensure that non-transactional consumers are acknowledging messages, that transactional consumers are committing transactions, that plain
javax.jms applications called
javax.jms.Connection.start(), and that transaction timeouts are tuned to reflect the needs of your particular application. Here are some symptoms of programming errors: consumers are not receiving any messages (make sure they called
start()), high "pending" counts for queues, already processed persistent messages re-appearing after a shutdown and restart, and already processed transactional messages re-appearing after a delay (the default JTA timeout is 30 seconds, default transacted session timeout is one hour).
Check WebLogic statistics for queues that are not being serviced by consumers. If you're having a problem with distributed queues, see Tuning Distributed Queues.
Check WebLogic statistics for topics with high pending counts. This usually indicates that there are topic subscriptions that are not being serviced. There may be a slow or unresponsive consumer client that's responsible for processing the messages, or it's possible that a durable subscription may no longer be needed and should be deleted, or the messages may be accumulating due to delayed distributed topic forwarding. You can check statistics for individual durable subscriptions on the WebLogic Server Administration Console. A durable subscription with a large backlog may have been created by an application but never deleted. Unserviced durable subscriptions continue to accumulate topic messages until they are either administratively destroyed, or unsubscribed by a standard JMS client.
Understand distributed topic behavior when not all members are active. In distributed topics, each produced message to a particular topic member is forwarded to each remote topic member. If a remote topic member is unavailable then the local topic member will store each produced message for later forwarding. Therefore, if a topic member is unavailable for a long period of time, then large backlogs can develop on the active members. In some applications, this backlog can be addressed by setting expiration times on the messages. See Defining a Message Expiration Policy.
In certain applications it may be fine to automatically delete old unprocessed messages. See Handling Expired Messages.
For transactional MDBs, consider using MDB transaction batching as this can yield a 5 fold improvement in some use cases.
Leverage distributed queues and add more JVMs to the cluster (in order to add more distributed queue member instances). For example, split a 200,000 message backlog across 4 JVMs at 50,000 messages per JVM, instead of 100,000 messages per JVM.
For client applications, use asynchronous consumers instead of synchronous consumers when possible. Asynchronous consumers can have a significantly lower network overhead, lower latency, and do not block a thread while waiting for a message.
For synchronous consumer client applications, consider: enabling
CLIENT_ACKNOWLEDGE to enable acknowledging multiple consumed messages at a time, and using
DUPS_OK_ACKNOWLEDGE instead of
For asynchronous consumer client applications, consider using
DUPS_OK_ACKNOWLEDGE instead of
Leverage batching. For example, include multiple messages in each transaction, or send one larger message instead of many smaller messages.
For non-durable subscriber client-side applications handling missing ("dropped") messages, investigate
MULTICAST_NO_ACKNOWLEDGE. This mode broadcasts messages concurrently to subscribers over UDP multicast.
Another area for investigation is to slow down or even stop message production. Here are some suggestions:
Use fewer producer threads.
Tune a sender blocking timeout that occurs during a quota condition, as described Blocking Senders During Quota Conditions. The timeout is tunable on connection factory.
Tune producer flow control, which automatically slows down producer calls under threshold conditions. See Controlling the Flow of Messages on JMS Servers and Destinations.
Consider modifying the application to implement flow-control. For example, some applications do not allow producers to inject more messages until a consumer has successfully processed the previous batch of produced messages (a windowing protocol). Other applications might implement a request/reply algorithm where a new request isn't submitted until the previous reply is received (essentially a windowing protocol with a window size of 1). In some cases, JMS tuning is not required as the synchronous flow from the RMI/EJB/Servlet is adequate.
Slowing down or stopping message processing has at least two potential drawbacks:
It puts back-pressure on the down-stream flow that is calling the producer. Sometimes the down-stream flow cannot handle this back-pressure, and a hard-to-handle backlog develops behind the producer. The location of the backlog depends on what's calling the producer. For example, if the producer is being called by a servlet, the backlog might manifest as packets accumulating on the incoming network socket or network card.
Blocking calls on server threads can lead to thread-starvation, too many active threads, or even dead-locks. Usually the key to address this problem is to ensure that the producer threads are running in a size limited dedicated thread pool, as this ensures that the blocking threads do not interfere with activity in other thread pools. For example, if an EJB or servlet is calling a "send" that might block for a significant time: configure a custom work manager with a
max threads constraint, and set the
dispatch-policy of the EJB/servlet to reference this work-manager.
JMS client resources are relatively expensive to create in comparison with sending and receiving messages. These resources should be cached or pooled for re-use rather than recreating them with each message. They include contexts, destinations, connection factories, connections, sessions, consumers, or producers.
In addition, it is important for applications to close contexts, connections, sessions, consumers, or producers once they are completely done with these resources. Failing to close unused resources leads to a memory leak, which lowers overall JVM performance and eventually may cause the JVM to fail with an out-of-memory error. Be aware that JNDI contexts have
close() method, and that closing a JMS connection automatically efficiently closes all sessions, consumers, and producers created using the connection.
For server-side applications, WebLogic automatically wraps and pools JMS resources that are accessed using a resource reference. See Enhanced Support for Using WebLogic JMS with EJBs and Servlets in Developing JMS Applications for Oracle WebLogic Server.
To check for heavy JMS resource allocation or leaks, you can monitor mbean stats and/or use your particular JVM's built in facilities. You can monitor mbean stats using the console, WLST, or java code.
Check JVM heap statistics for memory leaks or unexpectedly high allocation counts.
Similarly, check WebLogic statistics for memory leaks or unexpectedly high allocation counts.
Each distributed queue member is individually advertised in JNDI as
jms-server-name@distributed-destination-jndi-name. If produced messages are failing to load balance evenly across all distributed queue members, you may wish to change the configuration of your producer connection factories to disable server affinity (enabled by default).
Once created, a JMS consumer remains pinned to a particular queue member. This can lead to situations where consumers are not evenly load balanced across all distributed queue members, particularly if new members become available after all consumers have been initialized. If consumers fail to load balance evenly across all distributed queue members, the best option is to use an MDB that's targeted to a cluster designed to process the messages. WebLogic MDBs automatically ensure that all distributed queue members are serviced. If MDBs are not an option, here are some suggestions to improve consumer load balancing:
Ensure that your application is creating enough consumers and the consumer's connection factory is tuned using the available load balancing options. In particular, consider disabling the default server affinity setting.)
Change applications to periodically close and recreate consumers. This forces consumers to re-load balance.
Consume from individual queue members instead of from the distributed queues logical name.
Configure the distributed queue to enable forwarding. Distributed queue forwarding automatically internally forwards messages that have been idled on a member destination without consumers to a member that has consumers. This approach may not be practical for high message load applications.
Queue forwarding is not compatible with the WebLogic JMS Unit-of-Order feature, as it can cause messages to be delivered out of order.
Review information on how to tune WebLogic Topics.
You may want to convert singleton topics to distributed topics. A distributed topic with a
Partitioned policy generally outperforms the
Replicated policy choice.
Oracle highly recommends leveraging MDBs to process Topic messages, especially when working with Distributed Topics. MDBs automate the creation and servicing of multiple subscriptions and also provide high scalability options to automatically distribute the messages for a single subscription across multiple Distributed Topic members.
There is a
Sharable subscription extension that allows messages on a single topic subscription to be processed in parallel by multiple subscribers on multiple JVMs. WebLogic MDBs leverage this feature when they are not in
If the application can tolerate the deletion of old messages without having them be processed by a consumer, consider using message expirations or subscription limits. See Defining an Expiration Logging Policy and Subscription Message Limits.
If produced messages are failing to load balance evenly across the members of a Partitioned Distributed Topic, you may need to change the configuration of your producer connection factories to disable server affinity (enabled by default).
Before using any of these previously mentioned advanced features, Oracle recommends fully reviewing the following related documentation:
Configuring and Deploying MDBs Using Distributed Topics in Developing Message-Driven Beans for Oracle WebLogic Server
Developing Advanced Pub/Sub Applications in Administering JMS Resources for Oracle WebLogic Server
Advanced Programming with Distributed Destinations Using the JMS Destination Availability Helper API in Administering JMS Resources for Oracle WebLogic Server
Since WebLogic Server 9.0, a non-durable topic message publish request may block until the message is pushed to all consumers that are currently ready to process the message. This may cause non-durable topic publishers with large numbers of consumers to take longer to publish a message than expected. To revert to a publish that does not wait for consumers and waits only until it's confirmed the message arrived on a JMS server, use the following property:
Learn how to improve JMS performance when handling large messages.
WebLogic JMS pipelines messages that are delivered to asynchronous consumers (otherwise known as message listeners) or prefetch-enabled synchronous consumers. This action aids performance because messages are aggregated when they are internally pushed from the server to the client. The messages backlog (the size of the pipeline) between the JMS server and the client is tunable by configuring the
MessagesMaximum setting on the connection factory. See Asynchronous Message Pipeline in Developing JMS Applications for Oracle WebLogic Server.
In some circumstances, tuning the
MessagesMaximum parameter may improve performance dramatically, such as when the JMS application defers acknowledges or commits. In this case, Oracle suggests setting the
MessagesMaximum value to:
2 * (ack or commit interval) + 1
For example, if the JMS application acknowledges 50 messages at a time, set the
MessagesMaximum value to 101.
MessagesMaximum value too high can cause:
Increased memory usage on the client.
Affinity to an existing client as its pipeline fills with messages. For example: If
MessagesMaximum has a value of 10,000,000, the first consumer client to connect will get all messages that have already arrived at the destination. This condition leaves other consumers without any messages and creates an unnecessary backlog of messages in the first consumer that may cause the system to run out of memory.
Packet is too large exceptions and stalled consumers. If the aggregate size of the messages pushed to a consumer is larger than the current protocol's maximum message size (default size is 10 MB and is configured on a per WebLogic Server instance basis using the console and on a per client basis using the
-Dweblogic.MaxMessageSize command line property), the message delivery fails.
You may need to configure WebLogic clients in addition to the WebLogic Server instances, when sending and receiving large messages.
For most protocols, including T3, WLS limits the size of a network call to 10MB by default. If individual JMS message sizes exceed this limit, or if a set of JMS messages that is batched into the same network call exceeds this limit, this can lead to either "packet too large exceptions" and/or stalled consumers. Asynchronous consumers can cause multiple JMS messages to batch into the same network call, to control this batch size, see Tuning MessageMaximum Limitations.
To set the maximum message size on a server instance, tune the maximum message size for each supported protocol on a per protocol basis for each involved default channel or custom channel. In this context the word 'message' refers to all network calls over the given protocol, not just JMS calls.
To set the maximum message size on a client, use the following command line property:
This setting applies to all WebLogic Server network packets delivered to the client, not just JMS related packets.
A message compression threshold can be set programmatically using a JMS API extension to the
WLMessageProducer interface, or administratively by either specifying a Default Compression Threshold value on a connection factory or on a JMS SAF remote context. Compressed messages may actually inadvertently affect destination quotas since some message types actually grow larger when compressed
For instructions on configuring default compression thresholds using the WebLogic Server Administration Console, see:
Connection factories — Configure default delivery parameters in the Oracle WebLogic Server Administration Console Online Help.
Store-and-Forward (SAF) remote contexts — Configure SAF remote contexts in the Oracle WebLogic Server Administration Console Online Help.
Once configured, message compression is triggered on producers for client sends, on connection factories for message receives and message browsing, or through SAF forwarding. Messages are compressed using GZIP. Compression only occurs when message producers and consumers are located on separate server instances where messages must cross a JVM boundary, typically across a network connection when WebLogic domains reside on different machines. Decompression automatically occurs on the client side and only when the message content is accessed, except for the following situations:
Using message selectors on compressed XML messages can cause decompression, since the message body must be accessed in order to filter them. For more information on defining XML message selectors, see Filtering Messages in Developing JMS Applications for Oracle WebLogic Server.
Interoperating with earlier versions of WebLogic Server can cause decompression. For example, when using the Messaging Bridge, messages are decompressed when sent from the current release of WebLogic Server to a receiving side that is an earlier version of WebLogic Server.
On the server side, messages always remains compressed, even when they are written to disk.
With the message paging feature, JMS servers automatically attempt to free up virtual memory during peak message load periods. This feature can greatly benefit applications with large message spaces. Message paging is always enabled on JMS servers, and so a message paging directory is automatically created without having to configure one. You can, however, specify a directory using the Paging Directory option, then paged-out messages are written to files in this directory.
In addition to the paging directory, a JMS server uses either a file store or a JDBC store for persistent message storage. The file store can be user-defined or the server's default store. Paged JDBC store persistent messages are copied to both the JDBC store as well as the JMS Server's paging directory. Paged file store persistent messages that are small are copied to both the file store as well as the JMS Server's paging directory. Paged larger file store messages are not copied into the paging directory. See Best Practices When Using Persistent Stores.
However, a paged-out message does not free all of the memory that it consumes, since the message header with the exception of any user properties, which are paged out along with the message body, remains in memory for use with searching, sorting, and filtering. Queuing applications that use selectors to select paged messages may show severely degraded performance as the paged out messages must be paged back in. This does not apply to topics or to applications that select based only on message header fields (such as
CorrelationID). A good rule of thumb is to conservatively assume that messages each use 512 bytes of JVM memory even when paged out.
If a paging directory is not specified, then paged-out message bodies are written to the default
\tmp directory inside the servername subdirectory of a domain's root directory. For example, if no directory name is specified for the default paging directory, it defaults to:
domainname is the root directory of your domain, typically
c:\Oracle\Middleware\Oracle_Home\user_projects\domains\domainname, which is parallel to the directory in which WebLogic Server program files are stored, typically
To configure the Message Paging Directory attribute, see "Configure general JMS server properties" in Oracle WebLogic Server Administration Console Online Help.
The Message Buffer Size option specifies the amount of memory that will be used to store message bodies in memory before they are paged out to disk. The default value of Message Buffer Size is approximately one-third of the maximum heap size for the JVM, or a maximum of 512 megabytes. The larger this parameter is set, the more memory JMS will consume when many messages are waiting on queues or topics. Once this threshold is crossed, JMS may write message bodies to the directory specified by the Paging Directory option in an effort to reduce memory usage below this threshold.
It is important to remember that this parameter is not a quota. If the number of messages on the server passes the threshold, the server writes the messages to disk and evicts the messages from memory as fast as it can to reduce memory usage, but it will not stop accepting new messages. It is still possible to run out of memory if messages are arriving faster than they can be paged out. Users with high messaging loads who wish to support the highest possible availability should consider setting a quota, or setting a threshold and enabling flow control to reduce memory usage on the server.
It is highly recommended to always configure message count quotas. Quotas help prevent large message backlogs from causing out-of-memory errors, and WebLogic JMS does not set quotas by default.
There are many options for setting quotas, but in most cases it is enough to simply set a
Messages Maximum quota on each JMS Server rather than using destination level quotas. Keep in mind that each current JMS message consumes JVM memory even when the message has been paged out, because paging pages out only the message bodies but not message headers. A good rule of thumb for queues is to assume that each current JMS message consumes 512 bytes of memory. A good rule of thumb for topics is to assume that each current JMS message consumes 256 bytes of memory plus an additional 256 bytes of memory for each subscriber that hasn't acknowledged the message yet. For example, if there are 3 subscribers on a topic, then a single published message that hasn't been processed by any of the subscribers consumes 256 + 256*3 = 1024 bytes even when the message is paged out. Although message header memory usage is typically significantly less than these rules of thumb indicate, it is a best practice to make conservative estimates on memory utilization.
In prior releases, there were multiple levels of quotas: destinations had their own quotas and would also have to compete for quota within a JMS server. In this release, there is only one level of quota: destinations can have their own private quota or they can compete with other destinations using a shared quota.
In addition, a destination that defines its own quota no longer also shares space in the JMS server's quota. Although JMS servers still allow the direct configuration of message and byte quotas, these options are only used to provide quota for destinations that do not refer to a quota resource.
A quota is a named configurable JMS module resource. It defines a maximum number of messages and bytes, and is then associated with one or more destinations and is responsible for enforcing the defined maximums. Multiple destinations referring to the same quota share available quota according to the sharing policy for that quota resource.
Quota resources include the following configuration parameters:
Table 13-1 Quota Parameters
Bytes Maximum and Messages Maximum
The Messages Maximum/Bytes Maximum parameters for a quota resource defines the maximum number of messages and/or bytes allowed for that quota resource. No consideration is given to messages that are pending; that is, messages that are in-flight, delayed, or otherwise inhibited from delivery still count against the message and/or bytes quota.
The Shared parameter for a quota resource defines whether multiple destinations referring to the same quota resource compete for resources with each other.
The Policy parameter defines how individual clients compete for quota when no quota is available. It affects the order in which send requests are unblocked when the Send Timeout feature is enabled on the connection factory, as described in Tuning for Large Messages.
For more information about quota configuration parameters, see
QuotaBean in the MBean Reference for Oracle WebLogic Server. For instructions on configuring a quota resource using the WebLogic Server Administration Console, see Create a quota for destinations in the Oracle WebLogic Server Administration Console Online Help.
Destinations no longer define byte and messages maximums for quota, but can use a quota resource that defines these values, along with quota policies on sharing and competition.
The Quota parameter of a destination defines which quota resource is used to enforce quota for the destination. This value is dynamic, so it can be changed at any time. However, if there are unsatisfied requests for quota when the quota resource is changed, then those requests will fail with a
Outstanding requests for quota will fail at such time that the quota resource is changed. This does not mean changes to the message and byte attributes for the quota resource, but when a destination switches to a different quota.
In some cases, there will be destinations that do not configure quotas. JMS Server quotas allow JMS servers to limit the resources used by these quota-less destinations. All destinations that do not explicitly set a value for the Quota attribute share the quota of the JMS server where they are deployed. The behavior is exactly the same as if there were a special Quota resource defined for each JMS server with the Shared parameter enabled.
The interfaces for the JMS server quota are unchanged from prior releases. The JMS server quota is entirely controlled using methods on the JMSServerMBean. The quota policy for the JMS server quota is set by the Blocking Send Policy parameter on a JMS server, as explained in Specifying a Blocking Send Policy on JMS Servers. It behaves just like the Policy setting of any other quota.
The Send Timeout feature provides more control over message send operations by giving message produces the option of waiting a specified length of time until space becomes available on a destination. For example, if a producer makes a request and there is insufficient space, then the producer is blocked until space becomes available, or the operation times out. See Controlling the Flow of Messages on JMS Servers and Destinations for another method of flow control.
To use the WebLogic Server Administration Console to define how long a JMS connection factory will block message requests when a destination exceeds its maximum quota.
If sufficient space becomes available before the timeout period ends, the operation continues.
If sufficient space does not become available before the timeout period ends, you receive a resource allocation exception.
If you choose not to enable the blocking send policy by setting this value to 0, then you will receive a resource allocation exception whenever sufficient space is not available on the destination.
For more information about the Send Timeout field, see JMS Connection Factory: Configuration: Flow Control in the Oracle WebLogic Server Administration Console Online Help.
The Blocking Send policies enable you to define the JMS server's blocking behavior on whether to deliver smaller messages before larger ones when multiple message producers are competing for space on a destination that has exceeded its message quota.
To use the WebLogic Server Administration Console to define how a JMS server will block message requests when its destinations are at maximum quota.
FIFO — All send requests for the same destination are queued up one behind the other until space is available. No send request is permitted to complete when there another send request is waiting for space before it.
Preemptive — A send operation can preempt other blocking send operations if space is available. That is, if there is sufficient space for the current request, then that space is used even if there are previous requests waiting for space.
For more information about the Blocking Send Policy field, see JMS Server: Configuration: Thresholds and Quota in the Oracle WebLogic Server Administration Console Online Help.
In Oracle WebLogic JMS 220.127.116.11.0 and later, you can help prevent overloaded subscriptions from using all the available resources by configuring a message limit for a topic or a template. To configure a message limit, set the
MessagesLimitOverride attribute on a destination template, a standalone topic, or a uniform distributed topic.
When a subscription reaches its specified limit and receives a new message, the head message of the subscription is deleted to provide space for the new message. For a default FIFO subscription, the head message is the oldest. Messages are deleted only from subscriptions that have reached their limit. If a message exists on multiple subscriptions and is deleted on one subscription, then the message can still be received by the other subscriptions.
A subscription limit differs from a quota in multiple ways. A topic that has reached its quota disallows new messages until existing messages have been processed or expired; on the other hand, a subscription that has reached its subscription limit allows the new message and makes room for it by deleting current messages. Also, a topic that has reached its quota affects all subscriptions on the topic, as this disallows new messages from being added to any subscription. By contrast, a subscription limit only affects subscriptions that have reached their limits.
Subscription limits are not substitutes for quotas. Oracle always recommends configuring quotas, even when a subscription limit is also configured. For information on quotas, see Blocking Senders During Quota Conditions.
Regardless of subscription limits, subscription messages are not deleted if they are participating in a pending transaction, are part of a Unit-of-Work that is still waiting to accumulate all of its messages, or have already been passed to a consumer and are awaiting acknowledgement.
If a topic has not reached its quota, and all messages are immune from deletion, then a new message is accepted regardless of whether this causes a subscription to exceed its limit.
To configure a subscription limit, set the MessagesLimitOverride attribute on a destination template, stand-alone topic, or uniform distributed topic. You can see whether a topic’s runtime MBean has a subscription limit configured via its SubscriptionMessagesLimit attribute (“-1” indicates that no limit has been configured). You can monitor the number of messages that have been deleted due to a subscription limit on a durable subscription by checking its SubscriptionLimitDeletedCount attribute.
See JMS Topic: Configuration: Thresholds and Quotas and JMS Templates: Configuration: Thresholds and Quotas in Administration Console Online Help.
With the Flow Control feature, you can direct a JMS server or destination to slow down message producers when it determines that it is becoming overloaded.
See Compressing Messages.
Specifically, when either a JMS server or it's destinations exceeds its specified byte or message threshold, it becomes armed and instructs producers to limit their message flow (messages per second).
Producers will limit their production rate based on a set of flow control attributes configured for producers via the JMS connection factory. Starting at a specified
flow maximum number of messages, a producer evaluates whether the server/destination is still armed at prescribed intervals (for example, every 10 seconds for 60 seconds). If at each interval, the server/destination is still armed, then the producer continues to move its rate down to its prescribed flow minimum amount.
As producers slow themselves down, the threshold condition gradually corrects itself until the server/destination is unarmed. At this point, a producer is allowed to increase its production rate, but not necessarily to the maximum possible rate. In fact, its message flow continues to be controlled (even though the server/destination is no longer armed) until it reaches its prescribed flow maximum, at which point it is no longer flow controlled.
Producers receive a set of flow control attributes from their session, which receives the attributes from the connection, and which receives the attributes from the connection factory. These attributes allow the producer to adjust its message flow.
Specifically, the producer receives attributes that limit its flow within a minimum and maximum range. As conditions worsen, the producer moves toward the minimum; as conditions improve; the producer moves toward the maximum. Movement toward the minimum and maximum are defined by two additional attributes that specify the rate of movement toward the minimum and maximum. Also, the need for movement toward the minimum and maximum is evaluated at a configured interval.
Flow Control options are described in following table:
Table 13-2 Flow Control Parameters
Flow Control Enabled
Determines whether a producer can be flow controlled by the JMS server.
The maximum number of messages per second for a producer that is experiencing a threshold condition.
If a producer is not currently limiting its flow when a threshold condition is reached, the initial flow limit for that producer is set to Flow Maximum. If a producer is already limiting its flow when a threshold condition is reached (the flow limit is less than Flow Maximum), then the producer will continue at its current flow limit until the next time the flow is evaluated.
Once a threshold condition has subsided, the producer is not permitted to ignore its flow limit. If its flow limit is less than the Flow Maximum, then the producer must gradually increase its flow to the Flow Maximum each time the flow is evaluated. When the producer finally reaches the Flow Maximum, it can then ignore its flow limit and send without limiting its flow.
The minimum number of messages per second for a producer that is experiencing a threshold condition. This is the lower boundary of a producer's flow limit. That is, WebLogic JMS will not further slow down a producer whose message flow limit is at its Flow Minimum.
An adjustment period of time, defined in seconds, when a producer adjusts its flow from the Flow Maximum number of messages to the Flow Minimum amount, or vice versa.
The number of steps used when a producer is adjusting its flow from the Flow Minimum amount of messages to the Flow Maximum amount, or vice versa. Specifically, the Flow Interval adjustment period is divided into the number of Flow Steps (for example, 60 seconds divided by 6 steps is 10 seconds per step).
Also, the movement (that is, the rate of adjustment) is calculated by dividing the difference between the Flow Maximum and the Flow Minimum into steps. At each Flow Step, the flow is adjusted upward or downward, as necessary, based on the current conditions, as follows:
The downward movement (the decay) is geometric over the specified period of time (Flow Interval) and according to the specified number of Flow Steps. (For example, 100, 50, 25, 12.5).
The movement upward is linear. The difference is simply divided by the number of Flow Steps.
For more information about the flow control fields, and the valid and default values for them, see JMS Connection Factory: Configuration: Flow Control in the Oracle WebLogic Server Administration Console Online Help.
The attributes used for configuring bytes/messages thresholds are defined as part of the JMS server and/or its destination.Table 13-3 defines how the upper and lower thresholds start and stop flow control on a JMS server and/or JMS destination.
Table 13-3 Flow Control Threshold Parameters
Bytes/Messages Threshold High
When the number of bytes/messages exceeds this threshold, the JMS server/destination becomes armed and instructs producers to limit their message flow.
Bytes/Messages Threshold Low
When the number of bytes/messages falls below this threshold, the JMS server/destination becomes unarmed and instructs producers to begin increasing their message flow.
Flow control is still in effect for producers that are below their message flow maximum. Producers can move their rate upward until they reach their flow maximum, at which point they are no longer flow controlled.
For detailed information about other JMS server and destination threshold and quota fields, and the valid and default values for them, see the following pages in the Oracle WebLogic Server Administration Console Online Help:
Active message expiration ensures that expired messages are cleaned up immediately. Expired message auditing gives you the option of tracking expired messages, either by logging when a message expires or by redirecting expired messages to a defined error destination.
The following sections describe two message expiration features, the message Expiration Policy and the Active Expiration of message, which provide more control over how the system searches for expired messages and how it handles them when they are encountered.
Use the message Expiration Policy feature to define an alternate action to take when messages expire. Using the Expiration Policy attribute on the Destinations node, an expiration policy can be set on a per destination basis. The Expiration Policy attribute defines the action that a destination should take when an expired message is encountered: discard the message, discard the message and log its removal, or redirect the message to an error destination.
Also, if you use JMS templates to configure multiple destinations, you can use the Expiration Policy field to quickly configure an expiration policy on all your destinations. To override a template's expiration policy for specific destinations, you can modify the expiration policy on any destination.
For instructions on configuring the Expiration Policy, click one of the following links:
Follow these directions if you are configuring an expiration policy on topics without using a JMS template. Expiration policies that are set on specific topics will override the settings defined on a JMS template.
Discard — Expired messages are removed from the system. The removal is not logged and the message is not redirected to another location.
Log — Removes expired messages and writes an entry to the server log file indicating that the messages were removed from the system. You define the actual information that will be logged in the Expiration Logging Policy field in next step.
Redirect — Moves expired messages from their current location into the Error Destination defined for the topic.
For more information about the Expiration Policy options for a topic, see JMS Topic: Configuration: Delivery Failure in the Oracle WebLogic Server Administration Console Online Help.
Follow these directions if you are configuring an expiration policy on queues without using a JMS template. Expiration policies that are set on specific queues will override the settings defined on a JMS template.
Discard — Expired messages are removed from the system. The removal is not logged and the message is not redirected to another location.
Log — Removes expired messages from the queue and writes an entry to the server log file indicating that the messages were removed from the system. You define the actual information that will be logged in the Expiration Logging Policy field described in the next step.
Redirect — Moves expired messages from the queue and into the Error Destination defined for the queue.
For more information about the Expiration Policy options for a queue, see JMS Queue: Configuration: Delivery Failure in the Oracle WebLogic Server Administration Console Online Help.
Since JMS templates provide an efficient way to define multiple destinations (topics or queues) with similar attribute settings, you can configure a message expiration policy on an existing template (or templates) for your destinations.
Discard — Expired messages are removed from the messaging system. The removal is not logged and the message is not redirected to another location.
Log — Removes expired messages and writes an entry to the server log file indicating that the messages were removed from the system. The actual information that is logged is defined by the Expiration Logging Policy field described in the next step.
Redirect — Moves expired messages from their current location into the Error Destination defined for the destination.
For more information about the Expiration Policy options for a template, see JMS Template: Configuration: Delivery Failure in the Oracle WebLogic Server Administration Console Online Help.
The following section provides information on the expiration policy.
The Expiration Logging Policy parameter has been deprecated in this release of WebLogic Server. In its place, Oracle recommends using the Message Life Cycle Logging feature, which provide a more comprehensive view of the basic events that JMS messages will traverse through once they are accepted by a JMS server, including detailed message expiration data. For more information about message life cycle logging options, see Message Life Cycle Logging in Administering JMS Resources for Oracle WebLogic Server.
For example, you could specify one of the following values:
JMSPriority, Name, Address, City, State, Zip
%header%, Name, Address, City, State, Zip
JMSMessageID field is always logged and cannot be turned off. Therefore, if the Expiration Policy is not defined (that is, none) or is defined as an empty string, then the output to the log file contains only the
JMSMessageID of the message.
When an expired message is logged, the text portion of the message (not including timestamps, severity, thread information, security identity, etc.) conforms to the following format:
<ExpiredJMSMessage JMSMessageId='$MESSAGEID' > <HeaderFields Field1='Value1' [Field2='Value2'] … ] /> <UserProperties Property1='Value1' [Property='Value2'] … ] /> </ExpiredJMSMessage>
MESSAGEID is the exact string returned by
<ExpiredJMSMessage JMSMessageID='ID:P<851839.1022176920343.0' > <HeaderFields JMSPriority='7' JMSRedelivered='false' /> <UserProperties Make='Honda' Model='Civic' Color='White'Weight='2680' /> </ExpiredJMSMessage>
If no header fields are displayed, the line for header fields is not be displayed. If no user properties are displayed, that line is not be displayed. If there are no header fields and no properties, the closing
</ExpiredJMSMessage> tag is not necessary as the opening tag can be terminated with a closing bracket (
<ExpiredJMSMessage JMSMessageID='ID:N<223476.1022177121567.1' />
All values are delimited with double quotes. All string values are limited to 32 characters in length. Requested fields and/or properties that do not exist are not displayed. Requested fields and/or properties that exist but have no value (a null value) are displayed as null (without single quotes). Requested fields and/or properties that are empty strings are displayed as a pair of single quotes with no space between them.
<ExpiredJMSMessage JMSMessageID='ID:N<851839.1022176920344.0' > <UserProperties First='Any string longer than 32 char ...' Second=null Third='' /> </ExpiredJMSMessage>
Use the Active Expiration feature to define the timeliness in which expired messages are removed from the destination to which they were sent or published. Messages are not necessarily removed from the system at their expiration time, but they are removed within a user-defined number of seconds. The smaller the window, the closer the message removal is to the actual expiration time.
Follow these directions to define how often a JMS server will actively scan its destinations for expired messages. The default value is 30 seconds, which means the JMS server waits 30 seconds between each scan interval.
To disable active scanning, enter a value of 0 seconds. Expired messages are passively removed from the system as they are discovered.
For more information about the Expiration Scan Interval attribute, see JMS Server: Configuration: General in the Oracle WebLogic Server Administration Console Online Help.
There are a number of design choices that impact performance of JMS applications. Some others include reliability, scalability, manageability, monitoring, user transactions, message driven bean support, and integration with an application server. In addition, there are WebLogic JMS extensions and features have a direct impact on performance.
For more information on designing your applications for JMS, see Best Practices for Application Design in Developing JMS Applications for Oracle WebLogic Server.
Message Unit-of-Order is a WebLogic Server value-added feature that enables a stand-alone message producer, or a group of producers acting as one, to group messages into a single unit with respect to the processing order (a sub-ordering). This single unit is called a Unit-of-Order (or UOO) and requires that all messages from that unit be processed sequentially in the order they were created.
UOO replaces the following complex design patterns:
A dedicated consumer with a unique selector per each sub-ordering
A new destination per sub-ordering, one consumer per destination.
See Using Message Unit-of-Order in Developing JMS Applications for Oracle WebLogic Server.
The following sections provide best practice information when using UOO:
Ideal for applications that have strict message ordering requirements. UOO simplifies administration and application design, and in most applications improves performance.
Use MDB batching to:
Speed-up processing of the messages within a single sub-ordering.
Consume multiple messages at a time under the same transaction.
You can configure a default UOO for the destination. Only one consumer on the destination processes messages for the default UOO at a time.
To ensure strict ordering when using distributed destinations, each different UOO is pinned to a specific physical destination instance. There are two options for automatically determining the correct physical destination for a given UOO:
Hashing – Is generally faster and the UOO setting. Hashing works by using a hash function on the UOO name to determine the physical destination. It has the following drawbacks:
It doesn't correctly handle the administrative deleting or adding physical destinations to a distributed destination.
If a UOO hashes to an unavailable destination, the message send fails.
Path Service – Is a single server UOO directory service that maps the physical destination for each UOO. The Path Service is generally slower than hashing if there are many differently named UOO created per second. In this situation, each new UOO name implicitly forces a check of the path service before sending the message. If the number of UOOs created per second is limited, Path Service performance is not an issue as the UOO paths are cached throughout the cluster.
For releases prior to WebLogic Server 9.0, applications that had strict message ordering requirements were required to do the following:
Use a single physical destination with a single consumer
Ensure the maximum asynchronous consumer message backlog (The
MessagesMaximum parameter on the connection factory) was set to a value of 1.
UOO relaxes these requirements significantly as it allows for multiple consumers and allows for a asynchronous consumer message backlog of any size. To migrate older applications to take advantage of UOO, simply configure a default UOO name on the physical destination. See Configure connection factory unit-of-order parameters in Oracle WebLogic Server Administration Console Online Help and Ordered Redelivery of Messages in Developing JMS Applications for Oracle WebLogic Server.
One-way message sends can greatly improve the performance of applications that are bottle-necked by senders, but do so at the risk of introducing a lower QOS (quality-of-service). By enabling the One-Way Send Mode options, you allow message producers created by a user-defined connection factory to do one-way message sends, when possible.
Typical message sends from a JMS producer are termed two-way sends because they include both an internal request and an internal response. When an producer application calls
send(), the call generates a request that contains the application's message and then waits for a response from the JMS server to confirm its receipt of the message. This call-and-response mechanism regulates the producer, since the producer is forced to wait for the JMS server's response before the application can make another send call. Eliminating the response message eliminates this wait, and yields a one-way send. WebLogic Server supports a configurable one-way send option for non-persistent, non-transactional messaging; no application code changes are required to leverage this feature.
When the One-Way Send Mode is active, the associated producers can send messages without internally waiting for a response from the target destination's host JMS server. You can choose to allow queue senders and topic publishers to do one-way sends, or to limit this capability to topic publishers only. You must also specify a One-Way Window Size to determine when a two-way message is required to regulate the producer before it can continue making additional one-way sends.
You configure one-way message send parameters on a connection factory by using the WebLogic Server Administration Console, as described in Configure connection factory flow control in the Oracle WebLogic Server Administration Console Online Help. You can also use the WebLogic Scripting Tool (WLST) or JMX via the
One-way message sends are disabled if your connection factory is configured with "XA Enabled". This setting disables one-way sends whether or not the sender actually uses transactions.
To ensure one-way send support in a cluster with a single destination, verify that the connection factory and the JMS server hosting the destination are targeted to the same WebLogic server. The connection factory must not be targeted to any other WebLogic Server instances in the cluster.
To ensure one-way send support in a cluster with multiple destinations that share the same name, special care is required to ensure the WebLogic Server instance that hosts the client connection also hosts the destination. One solution is the following:
This solution disables RMI-level load balancing for clustered RMI objects, which includes EJB homes and JMS connection factories. Effectively, the client will obtain a connection and destination based only on the network address used to establish the JNDI context. Load balancing can be achieved by leveraging network load balancing, which occurs for URLs that include a comma-separated list of WebLogic Server addresses, or for URLs that specify a DNS name that resolves to a
round-robin set of IP addresses (as configured by a network administrator).
For more information on Server Affinity for clusters, see Load Balancing for EJBs and RMI Objects in Administering Clusters for Oracle WebLogic Server.
This section defines when one-way sends are not supported. When one-ways are not supported, the send QOS is automatically upgraded to standard two-ways.
One-way sends are supported when the client producer's connection host and the JMS server hosting the target destination are the same WebLogic Server instance; otherwise, the one-way mode setting will ignored and standard two-way sends will be used instead.
One-way message sends are disabled if the client's host connection factory is configured with XA Enabled. This setting disables one-way sends whether or not the sender actually uses transactions.
When the following higher QOS features are detected, then the one-way mode setting will be ignored and standard two-way sends will be used instead:
When the specified quota is exceeded on the targeted destination, then standard two-way sends will be used until the quota clears.
One-way messages that exceed quota are silently deleted, without immediately throwing exceptions back to the client. The client will eventually get a quota exception if the destination is still over quota at the time the next two-way send occurs. (Even in one-way mode, clients will send a two-way message every One Way Send Window Size number of messages configured on the client's connection factory.)
A workaround that helps avoid silently-deleted messages during quota conditions is to increase the value of the Blocking Send Timeout configured on the connection factory, as described in Compressing Messages. The one-way messages will not be deleted immediately, but instead will optimistically wait on the JMS server for the specified time until the quota condition clears (presumably due to messages getting consumed or by messages expiring). The client sender will not block until it sends a two-way message. For each client, no more than One Way Window Size messages will accumulate on the server waiting for quota conditions to clear.
A change in the server-side security policy could prevent one-way message sends without notifying the JMS client of the change in security status.
One-way sends can be disabled when a host JMS server or target destination is administratively undeployed, or when message production is paused on either the JMS server or the target destination using the "Production Pause/Resume" feature. See Production Pause and Production Resume in Administering JMS Resources for Oracle WebLogic Server.
One-way message sends work with distributed destinations provided the client looks up the physical distributed destination members directly rather than using the logical distributed destination's name. See Using Distributed Destinations in Developing JMS Applications for Oracle WebLogic Server.
A hardware or network failure will disable one-way sends. In such cases, the JMS producer is notified by an
OnException or by the next two-way message send. (Even in one-way mode, clients will send a two-way message every One Way Send Window Size number of messages configured on the client's connection factory.) The producer will be closed. The worst-case scenario is that all messages can be lost up to the last two-way message before the failure occurred.
Use the following QOS-related guidelines when using the one-way send mode for typical non-persistent messaging.
When used in conjunction with the Blocking Sends feature, then using one-way sends on a well-running system should achieve similar QOS as when using the two-way send mode.
One-way send mode for topic publishers falls within the QOS guidelines set by the JMS Specification, but does entail a lower QOS than two-way mode (the WebLogic Server default mode).
One-way send mode may not improve performance if JMS consumer applications are a system bottleneck, as described in Asynchronous vs. Synchronous Consumers in Developing JMS Applications for Oracle WebLogic Server.
Consider enlarging the JVM's heap size on the client and/or server to account for increased batch size (the Window) of sends. The potential memory usage is proportioned to the size of the configured Window and the number of senders.
The sending application will not receive all quota exceptions. One-way messages that exceed quota are silently deleted, without throwing exceptions back to the sending client. See Destination Quota Exceeded for more information and a possible work around.
Configuring one-way sends on a connection factory effectively disables any message flow control parameters configured on the connection factory.
By default, the One-way Window Size is set to "1", which effectively disables one-way sends as every one-way message will be upgraded to a two-way send. (Even in one-way mode, clients will send a two-way message every One Way Send Window Size number of messages configured on the client's connection factory.) Therefore, you must set the one-way send window size much higher. It is recommended to try setting the window size to "300" and then adjust it according to your application requirements.
The client application will not immediately receive network or server failure exceptions, some messages may be sent but silently deleted until the failure is detected by WebLogic Server and the producer is automatically closed. See Hardware Failure for more information.
The Messaging Performance Preference tuning option on JMS destinations enables you to control how long a destination should wait (if at all) before creating full batches of available messages for delivery to consumers.
This is an advanced option for fine tuning. It is normally best to explore other tuning options first.
At the minimum value, batching is disabled. Tuning above the default value increases the amount of time a destination is willing to wait before batching available messages. The maximum message count of a full batch is controlled by the JMS connection factory's Messages Maximum per Session setting.
Using the WebLogic Server Administration Console, this advanced option is available on the General Configuration page for both standalone and uniform distributed destinations (or via the
DestinationBean API), as well as for JMS templates (or via the
Specifically, JMS destinations include internal algorithms that attempt to automatically optimize performance by grouping messages into batches for delivery to consumers. In response to changes in message rate and other factors, these algorithms change batch sizes and delivery times. However, it isn't possible for the algorithms to optimize performance for every messaging environment. The Messaging Performance Preference tuning option enables you to modify how these algorithms react to changes in message rate and other factors so that you can fine-tune the performance of your system.
The Message Performance Preference option includes the following configuration parameters:
Table 13-4 Message Performance Preference Values
|WebLogic Server Administration Console Value||MBean Value||Description|
Do Not Batch Messages
Effectively disables message batching. Available messages are promptly delivered to consumers.
This is equivalent to setting the value of the connection factory's Messages Maximum per Session field to "1".
Batch Messages Without Waiting
Less-than-full batches are immediately delivered with available messages.
This is equivalent to the value set on the connection factory's Messages Maximum per Session field.
Low Waiting Threshold for Message Batching
Wait briefly before less-than-full batches are delivered with available messages.
Medium Waiting Threshold for Message Batching
Possibly wait longer before less-than-full batches are delivered with available messages.
High Waiting Threshold for Message Batching
Possibly wait even longer before less-than-full batches are delivered with available messages.
It may take some experimentation to find out which value works best for your system. For example, if you have a queue with many concurrent message consumers, by selecting the WebLogic Server Administration Console's Do Not Batch Messages value (or specifying "0" on the
DestinationBean MBean), the queue will make every effort to promptly push messages out to its consumers as soon as they are available. Conversely, if you have a queue with only one message consumer that doesn't require fast response times, by selecting the console's High Waiting Threshold for Message Batching value (or specifying "100" on the
DestinationBean MBean), then the queue will strongly attempt to only push messages to that consumer in batches, which will increase the waiting period but may improve the server's overall throughput by reducing the number of sends.
For instructions on configuring Messaging Performance Preference parameters on a standalone destinations, uniform distributed destinations, or JMS templates using the WebLogic Server Administration Console, see the following sections in the Administration Console Online Help:
The Message Performance Preference option is compatible with asynchronous consumers using the Asynchronous Message Pipeline, and is also compatible with synchronous consumers that use the Prefetch Mode for Synchronous Consumers feature, which simulates the Asynchronous Message Pipeline. However, if the value of the Maximum Messages value is set too low, it may negate the impact of the destination's higher-level performance algorithms (e.g., Low, Medium, and High Waiting Threshold for Message Batching). For more information on the Asynchronous Message Pipeline, see Receiving Messages in Developing JMS Applications for Oracle WebLogic Server.
WebLogic client thread pools are configured differently than WebLogic server thread-pools, and are not self tuning. Use the
-Dweblogic.ThreadPoolSize=n command-line property to configure the thread pools.
With most Java client side applications, the default client thread pool size of 5 threads is sufficient. If, however, the application has a large number of asynchronous consumers, then it is often beneficial to allocate slightly more threads than asynchronous consumers. This allows more asynchronous consumers to run concurrently.
WebLogic clients have a specific thread pool that is used for handling incoming requests from the server, such as JMS MessageListener invocations. This pool can be configured via the command-line property:
where n is the number of threads
You can force a client-side thread dump to verify that this setting is taking effect.
Review a short list of performance related best practices to use when creating a JMS .NET client application.
Always register a connection exception listener using an IConnection if the application needs to take action when an idle connection fails.
Have multiple .NET client threads share a single context to ensure that they use a single socket.
Cache and reuse frequently accessed JMS resources, such as contexts, connections, sessions, producers, destinations, and connection factories. Creating and closing these resources consumes significant CPU and network bandwidth.
Use DNS aliases or comma separated addresses for load balancing JMS .NET clients across multiple JMS .NET client host servers in a cluster.
For more information on best practices and other programming considerations for JMS .NET client applications, see Programming Considerations in Developing JMS .NET Client Applications for Oracle WebLogic Server.
Review the configuration considerations for a WebLogic JMS environment that includes Oracle Data Guard.
For more information on Oracle Data Guard, see http://www.oracle.com/us/products/database/options/active-data-guard/overview/index.html.
For planned maintenance windows, pause the impacted JMS destinations before initiating the switch from the production database instance to standby instance. When the standby database has transitioned to production, resume the JMS destinations. See Pause JMS server message operations at runtime in Oracle WebLogic Server Administration Console Online Help.
For unexpected service outages, implement JMS Service migration with the Restart on Failure option. Should the amount of time required to switch from the production to standby database exceed the value of the Store
IORetryDelaySeconds attribute and the JMS Services fails, the JMS service and associated store are restarted in-place. See In-Place Restarting of Failed Migratable Services in Administering Clusters for Oracle WebLogic Server.