WebLogic Server Performance and Tuning

Tuning WebLogic JMS

The following sections explain how to get the most out of your applications by implementing the administrative performance tuning features available with WebLogic JMS:

Defining Quota

In prior releases, there were multiple levels of quotas: destinations had their own quotas and would also have to compete for quota within a JMS server. In this release, there is only one level of quota: destinations can have their own private quota or they can compete with other destinations using a shared quota.

In addition, a destination that defines its own quota no longer also shares space in the JMS server’s quota. Although JMS servers still allow the direct configuration of message and byte quotas, these options are only used to provide quota for destinations that do not refer to a quota resource.

Quota Resources

A quota is a named configurable JMS module resource. It defines a maximum number of messages and bytes, and is then associated with one or more destinations and is responsible for enforcing the defined maximums. Multiple destinations referring to the same quota share available quota according to the sharing policy for that quota resource.

Quota resources include the following configuration parameters:

For more information about quota configuration parameters, see QuotaBean in the WebLogic Server MBean Reference. For instructions on configuring a quota resource using the Administration Console, see Create a quota for destinations in the Administration Console Online Help.

Destination-Level Quota

Destinations no longer define byte and messages maximums for quota, but can use a quota resource that defines these values, along with quota policies on sharing and competition.

The Quota parameter of a destination defines which quota resource is used to enforce quota for the destination. This value is dynamic, so it can be changed at any time. However, if there are unsatisfied requests for quota when the quota resource is changed, then those requests will fail with a javax.jms.ResourceAllocationException.

JMS Server-Level Quota

In some cases, there will be destinations that do not configure quotas. JMS Server quotas allow JMS servers to limit the resources used by these quota-less destinations. All destinations that do not explicitly set a value for the Quota attribute share the quota of the JMS server where they are deployed. The behavior is exactly the same as if there were a special Quota resource defined for each JMS server with the Shared parameter enabled.

The interfaces for the JMS server quota are unchanged from prior releases. The JMS server quota is entirely controlled using methods on the JMSServerMBean. The quota policy for the JMS server quota is set by the Blocking Send Policy parameter on a JMS server, as explained in Specifying a Blocking Send Policy on JMS Servers. It behaves just like the Policy setting of any other quota.

Blocking Senders During Quota Conditions

Defining a Send Timeout on Connection Factories

Blocking producers during quota conditions (by defining a send timeout) can dramatically improve the performance of applications and benchmarks that continuously retry message sends on quota failures. The Send Timeout feature provides more control over message send operations by giving message produces the option of waiting a specified length of time until space becomes available on a destination. For example, if a producer makes a request and there is insufficient space, then the producer is blocked until space becomes available, or the operation times out. See Controlling the Flow of Messages on JMS Servers and Destinations for another method of flow control.

To use the Administration Console to define how long a JMS connection factory will block message requests when a destination exceeds its maximum quota.

Specifying a Blocking Send Policy on JMS Servers

The Blocking Send policies enable you to define the JMS server’s blocking behavior on whether to deliver smaller messages before larger ones when multiple message producers are competing for space on a destination that has exceeded its message quota.

To use the Administration Console to define how a JMS server will block message requests when its destinations are at maximum quota.

Tuning for Large Messages

The following sections provide information on how to improve JMS performance when handling large messages:

Tuning MessageMaximum

WebLogic JMS pipelines messages that are delivered to asynchronous consumers (otherwise known as message listeners) or prefetch-enabled synchronous consumers. This action aids performance because messages are aggregated when they are internally pushed from the server to the client. The messages backlog (the size of the pipeline) between the JMS server and the client is tunable by configuring the MessagesMaximum setting on the connection factory. See Asynchronous Message Pipeline in Programming WebLogic JMS.

In some circumstances, tuning the MessagesMaximum parameter may improve performance dramatically, such as when the JMS application defers acknowledges or commits. In this case, BEA suggests setting the MessagesMaximum value to:

   2 * (ack or commit interval) + 1

For example:

   If the JMS application acknowledges 50 messages at a time, set the MessagesMaximum value to 101.

Tuning MessageMaximum Limitations

Tuning the MessagesMaximum value too high can cause:

Setting Maximum Message Size on a Client

You may need to configure WebLogic clients in addition to the WebLogic Server instance, when sending and receiving large messages. To set the maximum message size on a client, use the following command line property:

   -Dweblogic.MaxMessageSize

Compressing Messages

You may improve the performance of sending large messages traveling across JVM boundaries and help conserve disk space by specifying the automatic compression of any messages that exceed a user-specified threshold size. Message compression can help reduce network bottlenecks by automatically reducing the size of messages sent across network wires. Compressing messages can also conserve disk space when storing persistent messages in file stores or databases.

A message compression threshold can be set programmatically using a JMS API extension to the WLMessageProducer interface, or administratively by either specifying a Default Compression Threshold value on a connection factory or on a JMS SAF remote context.

For instructions on configuring default compression thresholds using the Administration Console, see:

Once configured, message compression is triggered on producers for client sends, on connection factories for message receives and message browsing, or through SAF forwarding. Messages are compressed using GZIP. Compression only occurs when message producers and consumers are located on separate server instances where messages must cross a JVM boundary, typically across a network connection when WebLogic domains reside on different machines. Decompression automatically occurs on the client side and only when the message content is accessed, except for the following situations:

On the server side, messages always remains compressed, even when they are written to disk.

Paging Out Messages To Free Up Memory

With the message paging feature, JMS servers automatically attempt to free up virtual memory during peak message load periods. This feature can greatly benefit applications with large message spaces. Message paging is always enabled on JMS servers, and so a message paging directory is automatically created without having to configure one. You can, however, specify a directory using the Paging Directory option, then paged-out messages are written to files in this directory.

JMS message paging saves memory for persistent messages, as even persistent messages cache their data in memory. If a JMS server is associated with a file store (either user-defined or the server’s default store), paged persistent messages are generally written to that file store, while non-persistent messages are always written to the JMS server’s paging directory. If a JMS server is associated with a JDBC store, then both paged persistent and non-persistent messages are always written to the JMS server’s paging directory. See Best Practices When Using Persistent Stores.

However, a paged-out message does not free all of the memory that it consumes, since the message header with the exception of any user properties, which are paged out along with the message body, remains in memory for use with searching, sorting, and filtering. Queuing applications that use selectors to select paged messages may show severely degraded performance as the paged out messages must be paged back in. This does not apply to topics or to applications that select based only on message header fields (such as CorrelationID).

Specifying a Message Paging Directory

If a paging directory is not specified, then paged-out message bodies are written to the default \tmp directory inside the servername subdirectory of a domain’s root directory. For example, if no directory name is specified for the default paging directory, it defaults to:

bea_home\user_projects\domains\domainname\servers\servername\tmp

where domainname is the root directory of your domain, typically c:\bea\user_projects\domains\domainname, which is parallel to the directory in which WebLogic Server program files are stored, typically c:\bea\weblogic92.

Tuning the Message Buffer Size Option

The Message Buffer Size option specifies the amount of memory that will be used to store message bodies in memory before they are paged out to disk. The default value of Message Buffer Size is approximately one-third of the maximum heap size for the JVM, or a maximum of 512 megabytes. The larger this parameter is set, the more memory JMS will consume when many messages are waiting on queues or topics. Once this threshold is crossed, JMS may write message bodies to the directory specified by the Paging Directory option in an effort to reduce memory usage below this threshold.

It is important to note that this parameter is not a quota. If the number of messages on the server passes the threshold, the server writes the messages to disk and evicts the messages from memory as fast as it can to reduce memory usage, but it will not stop accepting new messages. It is still possible to run out of memory if messages are arriving faster than they can be paged out. Users with high messaging loads who wish to support the highest possible availability should consider setting a quota, or setting a threshold and enabling flow control to reduce memory usage on the server.

Controlling the Flow of Messages on JMS Servers and Destinations

With the Flow Control feature, you can direct a JMS server or destination to slow down message producers when it determines that it is becoming overloaded. See Compressing Messages.

The following sections describe how flow control feature works and how to configure flow control on a connection factory.

How Flow Control Works

Specifically, when either a JMS server or it’s destinations exceeds its specified byte or message threshold, it becomes armed and instructs producers to limit their message flow (messages per second).

Producers will limit their production rate based on a set of flow control attributes configured for producers via the JMS connection factory. Starting at a specified flow maximum number of messages, a producer evaluates whether the server/destination is still armed at prescribed intervals (for example, every 10 seconds for 60 seconds). If at each interval, the server/destination is still armed, then the producer continues to move its rate down to its prescribed flow minimum amount.

As producers slow themselves down, the threshold condition gradually corrects itself until the server/destination is unarmed. At this point, a producer is allowed to increase its production rate, but not necessarily to the maximum possible rate. In fact, its message flow continues to be controlled (even though the server/destination is no longer armed) until it reaches its prescribed flow maximum, at which point it is no longer flow controlled.

Configuring Flow Control

Producers receive a set of flow control attributes from their session, which receives the attributes from the connection, and which receives the attributes from the connection factory. These attributes allow the producer to adjust its message flow.

Specifically, the producer receives attributes that limit its flow within a minimum and maximum range. As conditions worsen, the producer moves toward the minimum; as conditions improve; the producer moves toward the maximum. Movement toward the minimum and maximum are defined by two additional attributes that specify the rate of movement toward the minimum and maximum. Also, the need for movement toward the minimum and maximum is evaluated at a configured interval.

Flow Control options are described in following table:

For more information about the flow control fields, and the valid and default values for them, see JMS Connection Factory: Configuration: Flow Control in the Administration Console Online Help.

Flow Control Thresholds

The attributes used for configuring bytes/messages thresholds are defined as part of the JMS server and/or its destination. Table 13-3 defines how the upper and lower thresholds start and stop flow control on a JMS server and/or JMS destination.

For detailed information about other JMS server and destination threshold and quota fields, and the valid and default values for them, see the following pages in the Administration Console Online Help:

Handling Expired Messages

The following sections describe two message expiration features, the message Expiration Policy and the Active Expiration of message, which provide more control over how the system searches for expired messages and how it handles them when they are encountered.

Active message expiration ensures that expired messages are cleaned up immediately. Moreover, expired message auditing gives you the option of tracking expired messages, either by logging when a message expires or by redirecting expired messages to a defined error destination.

Defining a Message Expiration Policy

Use the message Expiration Policy feature to define an alternate action to take when messages expire. Using the Expiration Policy attribute on the Destinations node, an expiration policy can be set on a per destination basis. The Expiration Policy attribute defines the action that a destination should take when an expired message is encountered: discard the message, discard the message and log its removal, or redirect the message to an error destination.

Also, if you use JMS templates to configure multiple destinations, you can use the Expiration Policy field to quickly configure an expiration policy on all your destinations. To override a template’s expiration policy for specific destinations, you can modify the expiration policy on any destination.

For instructions on configuring the Expiration Policy, click one of the following links:

Configuring an Expiration Policy on Topics

Follow these directions if you are configuring an expiration policy on topics without using a JMS template. Expiration policies that are set on specific topics will override the settings defined on a JMS template.

Configuring an Expiration Policy on Queues

Follow these directions if you are configuring an expiration policy on queues without using a JMS template. Expiration policies that are set on specific queues will override the settings defined on a JMS template.

Configuring an Expiration Policy on Templates

Since JMS templates provide an efficient way to define multiple destinations (topics or queues) with similar attribute settings, you can configure a message expiration policy on an existing template (or templates) for your destinations.

Defining an Expiration Logging Policy

When the Expiration Policy is set to Log, the Expiration Logging Policy defines what information about the message is logged. Valid values for Expiration Logging Policy properties include%header%, %properties%, JMS header properties as defined in the JMS specification, the WebLogic JMS-specific extended header fields JMSDeliveryTime and JMSRedeliveryLimit, and any user-defined property. Each property must be separated by a comma.

The %header% value indicates that all header fields should be logged. The %properties% value indicates that all user properties should be logged. Neither values are case sensitive. However, the enumeration of individual JMS header fields and user properties are case sensitive.

For example, you could specify one of the following values:

The JMSMessageID field is always logged and cannot be turned off. Therefore, if the Expiration Policy is not defined (that is, none) or is defined as an empty string, then the output to the log file contains only the JMSMessageID of the message.

Expiration Log Output Format

When an expired message is logged, the text portion of the message (not including timestamps, severity, thread information, security identity, etc.) conforms to the following format:

<ExpiredJMSMessage JMSMessageId=’$MESSAGEID’ >
    <HeaderFields Field1=’Value1’ [Field2=’Value2’] … ] />
    <UserProperties Property1=’Value1’ [Property=’Value2’] … ] />
</ExpiredJMSMessage>

where $MESSAGEID is the exact string returned by Message.getJMSMessageID().

For example:

<ExpiredJMSMessage JMSMessageID=’ID:P<851839.1022176920343.0’ >
    <HeaderFields JMSPriority=’7’ JMSRedelivered=’false’ />
    <UserProperties Make=’Honda’ Model=’Civic’ Color=’White’
     Weight=’2680’ />
</ExpiredJMSMessage>

If no header fields are displayed, the line for header fields is not be displayed. If no user properties are displayed, that line is not be displayed. If there are no header fields and no properties, the closing </ExpiredJMSMessage> tag is not necessary as the opening tag can be terminated with a closing bracket (/>).

For example:

<ExpiredJMSMessage JMSMessageID=’ID:N<223476.1022177121567.1’ />

All values are delimited with double quotes. All string values are limited to 32 characters in length. Requested fields and/or properties that do not exist are not displayed. Requested fields and/or properties that exist but have no value (a null value) are displayed as null (without single quotes). Requested fields and/or properties that are empty strings are displayed as a pair of single quotes with no space between them.

For example:

<ExpiredJMSMessage JMSMessageID=’ID:N<851839.1022176920344.0’ >
    <UserProperties  First=’Any string longer than 32 char ...’
     Second=null Third=’’ />
</ExpiredJMSMessage>

Tuning Active Message Expiration

Use the Active Expiration feature to define the timeliness in which expired messages are removed from the destination to which they were sent or published. Messages are not necessarily removed from the system at their expiration time, but they are removed within a user-defined number of seconds. The smaller the window, the closer the message removal is to the actual expiration time.

Configuring a JMS Server to Actively Scan Destinations for Expired Messages

Follow these directions to define how often a JMS server will actively scan its destinations for expired messages. The default value is 30 seconds, which means the JMS server waits 30 seconds between each scan interval.

There are a number of design choices that impact performance of JMS applications. Some others include reliability, scalability, manageability, monitoring, user transactions, message driven bean support, and integration with an application server. In addition, there are WebLogic JMS extensions and features have a direct impact on performance.

For more information on designing your applications for JMS, see Best Practices for Application Design in Programming WebLogic JMS

Tuning Applications Using Unit-of-Order

Message Unit-of-Order is a WebLogic Server value-added feature that enables a stand-alone message producer, or a group of producers acting as one, to group messages into a single unit with respect to the processing order (a sub-ordering). This single unit is called a Unit-of-Order (or UOO) and requires that all messages from that unit be processed sequentially in the order they were created. UOO replaces the following complex design patterns:

See Using Message Unit-of-Order in Programming WebLogic JMS.

Best Practices

The following sections provide best practice information when using UOO:

Using UOO and Distributed Destinations

To ensure strict ordering when using distributed destinations, each different UOO is pinned to a specific physical destination instance. There are two options for automatically determining the correct physical destination for a given UOO:

Migrating Old Applications to Use UOO

For releases prior to WebLogic Server 9.0, applications that had strict message ordering requirements were required to do the following:

UOO relaxes these requirements significantly as it allows for multiple consumers and allows for a asynchronous consumer message backlog of any size. To migrate older applications to take advantage of UOO, simply configure a default UOO name on the physical destination. See Configure connection factory unit-of-order parameters in Administration Console Online Help and Ordered Redelivery of Messages in Programming WebLogic JMS.

Using One-Way Message Sends For Improved Non-Persistent Messaging Performance

One-way message sends can greatly improve the performance of applications that are bottle-necked by senders, but do so at the risk of introducing a lower QOS (quality-of-service). Typical message sends from a JMS producer are termed two-way sends because they include both an internal request and an internal response. When an producer application calls send(), the call generates a request that contains the application’s message and then waits for a response from the JMS server to confirm its receipt of the message. This call-and-response mechanism regulates the producer, since the producer is forced to wait for the JMS server’s response before the application can make another send call. Eliminating the response message eliminates this wait, and yields a one-way send. WebLogic Server supports a configurable one-way send option for non-persistent, non-transactional messaging; no application code changes are required to leverage this feature.

By enabling the One-Way Send Mode options, you allow message producers created by a user-defined connection factory to do one-way message sends, when possible. When active, the associated producers can send messages without internally waiting for a response from the target destination’s host JMS server. You can choose to allow queue senders and topic publishers to do one-way sends, or to limit this capability to topic publishers only. You must also specify a One-Way Window Size to determine when a two-way message is required to regulate the producer before it can continue making additional one-way sends.

Configure One-Way Sends On a Connection Factory

You configure one-way message send parameters on a connection factory by using the Administration Console, as described in Configure connection factory flow control in the Administration Console Online Help. You can also use the WebLogic Scripting Tool (WLST) or JMX via the FlowControlParamsBean MBean.

One-Way Send Support In a Cluster With a Single Destination

To ensure one-way send support in a cluster with a single destination, verify that the connection factory and the JMS server hosting the destination are targeted to the same WebLogic server. The connection factory must not be targeted to any other WebLogic Server instances in the cluster.

One-Way Send Support In a Cluster With Multiple Destinations

To ensure one-way send support in a cluster with multiple destinations that share the same name, special care is required to ensure the WebLogic Server instance that hosts the client connection also hosts the destination. One solution is the following:

This solution disables RMI-level load balancing for clustered RMI objects, which includes EJB homes and JMS connection factories. Effectively, the client will obtain a connection and destination based only on the network address used to establish the JNDI context. Load balancing can be achieved by leveraging network load balancing, which occurs for URLs that include a comma-separated list of WebLogic Server addresses, or for URLs that specify a DNS name that resolves to a round-robin set of IP addresses (as configured by a network administrator).

For more information on Server Affinity for clusters, see Load Balancing for EJBs and RMI Objects in Using WebLogic Server Clusters.

When One-Way Sends Are Not Supported

This section defines when one-way sends are not supported. When one-ways are not supported, the send QOS is automatically upgraded to standard two-ways.

Different Client and Destination Hosts

One-way sends are supported when the client producer’s connection host and the JMS server hosting the target destination are the same WebLogic Server instance; otherwise, the one-way mode setting will ignored and standard two-way sends will be used instead.

XA Enabled On Client’s Host Connection Factory

One-way message sends are disabled if the client’s host connection factory is configured with XA Enabled. This setting disables one-way sends whether or not the sender actually uses transactions.

Higher QOS Detected

When the following higher QOS features are detected, then the one-way mode setting will be ignored and standard two-way sends will be used instead:

Destination Quota Exceeded

When the specified quota is exceeded on the targeted destination, then standard two-way sends will be used until the quota clears.

One-way messages that exceed quota are silently deleted, without immediately throwing exceptions back to the client. The client will eventually get a quota exception if the destination is still over quota at the time the next two-way send occurs. (Even in one-way mode, clients will send a two-way message every One Way Send Window Size number of messages configured on the client’s connection factory.)

A workaround that helps avoid silently-deleted messages during quota conditions is to increase the value of the Blocking Send Timeout configured on the connection factory, as described in Compressing Messages. The one-way messages will not be deleted immediately, but instead will optimistically wait on the JMS server for the specified time until the quota condition clears (presumably due to messages getting consumed or by messages expiring). The client sender will not block until it sends a two-way message. For each client, no more than One Way Window Size messages will accumulate on the server waiting for quota conditions to clear.

Change In Server Security Policy

A change in the server-side security policy could prevent one-way message sends without notifying the JMS client of the change in security status.

Change In JMS Server or Destination Status

One-way sends can be disabled when a host JMS server or target destination is administratively undeployed, or when message production is paused on either the JMS server or the target destination using the “Production Pause/Resume” feature. See Production Pause and Production Resume in Configuring and Managing WebLogic JMS.

Looking Up Logical Distributed Destination Name

One-way message sends work with distributed destinations provided the client looks up the physical distributed destination members directly rather than using the logical distributed destination’s name. See Using Distributed Destinations in Programming WebLogic JMS.

Hardware Failure

A hardware or network failure will disable one-way sends. In such cases, the JMS producer is notified by an OnException or by the next two-way message send. (Even in one-way mode, clients will send a two-way message every One Way Send Window Size number of messages configured on the client’s connection factory.) The producer will be closed. The worst-case scenario is that all messages can be lost up to the last two-way message before the failure occurred.

One-Way Send QOS Notes

Use the following QOS-related guidelines when using the one-way send mode for typical non-persistent messaging.

Tuning Destination Performance

The Messaging Performance Preference tuning option on JMS destinations enables you to control how long a destination should wait (if at all) before creating full batches of available messages for delivery to consumers. At the minimum value, batching is disabled. Tuning above the default value increases the amount of time a destination is willing to wait before batching available messages. The maximum message count of a full batch is controlled by the JMS connection factory’s Messages Maximum per Session setting.

Using the Administration Console, this advanced option is available on the General Configuration page for both standalone and uniform distributed destinations (or via the DestinationBean API), as well as for JMS templates (or via the TemplateBean API).

Specifically, JMS destinations include internal algorithms that attempt to automatically optimize performance by grouping messages into batches for delivery to consumers. In response to changes in message rate and other factors, these algorithms change batch sizes and delivery times. However, it isn’t possible for the algorithms to optimize performance for every messaging environment. The Messaging Performance Preference tuning option enables you to modify how these algorithms react to changes in message rate and other factors so that you can fine-tune the performance of your system.

Messaging Performance Configuration Parameters

The Message Performance Preference option includes the following configuration parameters:

It may take some experimentation to find out which value works best for your system. For example, if you have a queue with many concurrent message consumers, by selecting the Administration Console’s Do Not Batch Messages value (or specifying “0” on the DestinationBean MBean), the queue will make every effort to promptly push messages out to its consumers as soon as they are available. Conversely, if you have a queue with only one message consumer that doesn’t require fast response times, by selecting the console’s High Waiting Threshold for Message Batching value (or specifying “100” on the DestinationBean MBean), then the queue will strongly attempt to only push messages to that consumer in batches, which will increase the waiting period but may improve the server’s overall throughput by reducing the number of sends.

For instructions on configuring Messaging Performance Preference parameters on a standalone destinations, uniform distributed destinations, or JMS templates using the Administration Console, see the following sections in the Administration Console Online Help:

For more information about these parameters, see DestinationBean and TemplateBean in the WebLogic Server MBean Reference.

Compatibility With the Asynchronous Message Pipeline

The Message Performance Preference option is compatible with asynchronous consumers using the Asynchronous Message Pipeline, and is also compatible with synchronous consumers that use the Prefetch Mode for Synchronous Consumers feature, which simulates the Asynchronous Message Pipeline. However, if the value of the Maximum Messages value is set too low, it may negate the impact of the destination’s higher-level performance algorithms (e.g., Low, Medium, and High Waiting Threshold for Message Batching). For more information on the Asynchronous Message Pipeline, see Receiving Messages in Programming WebLogic JMS.