WebLogic Server Performance and Tuning

Tuning WebLogic Server

The following sections describe how to tune WebLogic Server to match your application needs.

Setting Java Parameters for Starting WebLogic Server

Java parameters must be specified whenever you start WebLogic Server. For simple invocations, this can be done from the command line with the weblogic.Server command. However, because the arguments needed to start WebLogic Server from the command line can be lengthy and prone to error, BEA recommends that you incorporate the command into a script. To simply this process, you can modify the default values in the sample scripts that are provided with the WebLogic distribution to start WebLogic Server, as described in "Specifying Java Options for a WebLogic Server Instance".

If you used the Configuration Wizard to create your domain, the WebLogic startup scripts are located in the domain-name directory where you specified your domain. By default, this directory is BEA_HOME\user_projects\domain\domain-name, where BEA_HOME is the directory that contains the product installation, and domain-name is the name of the domain directory defined by the selected configuration template. For more information about creating domains using the Configuration Wizard, see "Creating Domains Using the Configuration Wizard".

You need to modify some default Java values in these scripts to fit your environment and applications. The important performance tuning parameters in these files are the JAVA_HOME parameter and the Java heap size parameters:

Change the value of the variable JAVA_HOME to the location of your JDK. For example:

set JAVA_HOME=C:\bea\jdk141_03

For higher performance throughput, set the minimum java heap size equal to the maximum heap size. For example:

"%JAVA_HOME%\bin\java" -hotspot -Xms512m -Xmx512m -classpath %CLASSPATH% -
See Specifying Heap Size Values for details about setting heap size options.

Setting Performance-Related Configuration Parameters

The WebLogic Server configuration file (config.xml) contains a number of performance-related parameters that can be fine-tuned depending on your environment and applications. Tuning these parameters based on your system requirements (rather than running with default settings) can greatly improve both single-node performance and the scalability characteristics of an application.

Within a WebLogic Server domain, the configuration file is located on the machine that hosts the Administration Server, and provides persistent storage of WebLogic MBean attribute values. The Administration Server serves as a central point of contact for server instances and system administration tools. A domain may also include additional WebLogic Server instances called Managed Servers, which are used mainly for servicing applications.

When the Administration Server starts, it reads the domain configuration file and overrides the default attribute values of the administration MBeans with any attribute values found in the configuration file. Every time you change an attribute using the system administration tools (using either the command-line interface or the Administration Console), its value is stored in the appropriate administration MBean and written to the configuration file.

For more information about system administration infastructure, see "Overview of WebLogic Server System Administration" in the Administration Guide.

Overview of WebLogic Server System Administration

Table 4-1 lists the config.xml file parameters that affect server performance.

Table 4-1 Performance-Related config.xml Elements

Element	Attributes	Console Field	For information
`Server`	`NativeIOEnabled`	Native IO Enabled	See Using WebLogic Server "Native IO" Performance Packs.
`ExecuteQueue`	`ThreadCount`	Thread Count	See Tuning the Default Execute Queue Threads.
`ExecuteQueue`	`QueueLength` `QueueLengthThresholdPercent` `ThreadsIncrease` `ThreadsMaximum` `Thread Priority`	Queue Length Queue Length Threshold Percent Threads Increase Threads Maximum Thread Priority	See Tuning Execute Queues for Overflow Conditions.
`Server`	`StuckThreadMaxTime` `StuckThreadTimerInterval`	Stuck Thread Max Time Stuck Thread Timer Interval	See Tuning the Execute Thread Detection Behavior.
`Server`	`ThreadPoolPercentSocketReaders`	Socket Readers	See Allocating Execute Threads to Act as Socket Readers.
`Server`	`AcceptBacklog`	Accept Backlog	See Tuning Connection Backlog Buffering.
`JDBCConnectionPool`	`InitialCapacity` `MaxCapacity`	Initial Capacity Max Capacity	See How JDBC Connection Pools Enhance Performance.
`JDBCConnectionPool`	`StatementCacheSize`	Statement Cache Size	See Caching Prepared and Callable Statements.

Development vs. Production Mode Default Tuning Values

You can indicate whether a domain is to be used in a development environment or a production environment. WebLogic Server uses different default values for various services depending on the type of environment you specify.

Table 4-2 lists the performance-related configuration parameters that differ when switching from development to production startup mode.

Table 4-2 Development and Production Startup Mode Tuning Defaults

Tuning Parameter	Development Mode Default	Production Mode Default
Execute Queue: `ThreadCount`	15 threads	25 threads
JDBC Connection Pool: `MaxCapacity`	15 connections	25 connections

The tuning defaults discussed in throughout WebLogic Performance and Tuning Guide refer to the "development mode" defaults, which is the default startup mode when WebLogic Server is installed. For information on switching the startup mode from development to production, see Changing the Runtime Mode in the Administration Console Online Help.

For a complete listing of the differences between development and production startup modes, see the "Differences Between Configuration Startup Modes" section in Creating WebLogic Configurations Using the Configuration Wizard.

Using WebLogic Server "Native IO" Performance Packs

Benchmarks show major performance improvements when you use native performance packs on machines that host WebLogic Server instances. Performance packs use a platform-optimized, native socket multiplexor to improve server performance. For example, the native socket reader multiplexor threads have their own execute queue and do not borrow threads from the default execute queue, which frees up default execute threads to do application work.

However, if you must use the pure-Java socket reader implementation for host machines, you can still improve the performance of socket communication by configuring the proper number of socket reader threads for each server instance and client machine.

Which Platforms Have Performance Packs?

To see which supported platforms currently have performance packs available:

Go to Supported Configurations for WebLogic Server .

From the list of supported configurations, click on the link for the platform that you need.

The ensuing page contains tables of information for each supported WebLogic Server releases (including service packs). Within each release table there is a "Performance Pack" entry that indicates whether a performance pack is "Included" in the release.

To verify performance pack information, you can either click on a specific WebLogic Server release at the top of the page and scan the corresponding table, or use your browser's Edit —> Find feature to search for all instances of "Performance Pack" on the page.

Enabling Performance Packs

The use of native performance packs are enabled by default in the config.xml shipped with your distribution. To verify this setting in your configuration file, check that the NativeIOEnabled attribute of the Server element is set to "true" (NativeIOEnabled=true).

You can also use the Administration Console to verify that performance packs are enabled:

Start the Administration Server if it is not already running.

Access the Administration Console for the domain.

Expand the Servers node in the left pane to display the servers configured in your domain.

Click the name of the server instance that you want to configure.

Select the Configuration —> Tuning tab.

If the Enable Native IO check box is not selected, select the check box.

Click Apply.

Restart the server.

Tuning the Default Execute Queue Threads

The value of the ThreadCount attribute of an ExecuteQueue element in the config.xml file equals the number of simultaneous operations that can be performed by applications that use the execute queue. As work enters an instance of WebLogic Server, it is placed in an execute queue. This work is then assigned to a thread that does the work on it. Threads consume resources, so handle this attribute with care—you can degrade performance by increasing the value unnecessarily.

By default, a new WebLogic Server instance is configured with a development mode execute queue, weblogic.kernel.default, that contains 15 threads. In addition, WebLogic Server provides two other pre-configured queues:

weblogic.admin.HTTP—Available only on Administration Servers, this queue is reserved for communicating with the Administration Console; you cannot reconfigure it.
weblogic.admin.RMI—Both Administration Servers and Managed Servers have this queue; it is reserved for administrative traffic; you cannot reconfigure it.

Unless you configure additional execute queues, and assign applications to them, Web applications and RMI objects use weblogic.kernel.default.

Note: If native performance packs are not being used for your platform, you may need to tune the default number of execute queue threads and the percentage of threads that act as socket readers to achieve optimal performance. For more information, see Allocating Execute Threads to Act as Socket Readers.

Should You Modify the Default Thread Count?

Adding more threads to the default execute queue does not necessarily imply that you can process more work. Even if you add more threads, you are still limited by the power of your processor. Because threads consume memory, you can degrade performance by increasing the value of the ThreadCount attribute unnecessarily. A high execute thread count causes more memory to be used and increases context switching, which can degrade performance.

The value of the ThreadCount attribute depends very much on the type of work your application does. For example, if your client application is thin and does a lot of its work through remote invocation, that client application will spend more time connected — and thus will require a higher thread count — than a client application that does a lot of client-side processing.

If you do not need to use more than 15 threads (the development default) or 25 threads (the production default) for your work, do not change the value of this attribute. As a general rule, if your application makes database calls that take a long time to return, you will need more execute threads than an application that makes calls that are short and turn over very rapidly. For the latter case, using a smaller number of execute threads could improve performance.

Scenarios for Modifying the Default Thread Count

To determine the ideal thread count for an execute queue, monitor the queue's throughput while all applications in the queue are operating at maximum load. Increase the number of threads in the queue and repeat the load test until you reach the optimal throughput for the queue. (At some point, increasing the number of threads will lead to enough context switching that the throughput for the queue begins to decrease.)

Note: The WebLogic Server Administration Console displays the cumulative throughput for all of a server's execute queues. To access this throughput value, follow steps 1-6 in Modifying the Default Thread Count.

Table 4-3 shows default scenarios for adjusting available threads in relation to the number of CPUs available in the WebLogic Server domain. These scenarios also assume that WebLogic Server is running under maximum load, and that all thread requests are satisfied by using the default execute queue. If you configure additional execute queues and assign applications to specific queues, monitor results on a pool-by-pool basis.

Table 4-3 Scenarios for Modifying the Default Thread Count

When...	Results	Do This:
Thread Count < number of CPUs	Your thread count is too low if: CPU is waiting to do work, but there is work that could be done. Cannot get 100 percent CPU utilization rate.	Increase the thread count.
Thread Count = number of CPUs	Theoretically ideal, but the CPUs are still under-utilized.	Increase the thread count.
Thread Count > number of CPUs (by a moderate number of threads)	Practically ideal, with a moderate amount of context switching and a high CPU utilization rate.	Tune the moderate number of threads and compare performance results.
Thread Count > number of CPUs (by a large number of threads)	Too much context switching, which can lead to significant performance degradation. Your performance may increase as you decrease the number of threads.	Reduce the number of threads so that it equals the number of CPUs, and then add only the number of "stuck" threads that you have determined. For example, if you have four processors, then four threads can be running concurrently with the number of stuck threads. So, you want the execute threads to be 4 + the number of stuck threads. To determine the amount of stuck threads, see Tuning the Execute Thread Detection Behavior. Note: This recommendation is highly application-dependent. For instance, the length of time the application might block threads can invalidate the formula.

Modifying the Default Thread Count

To modify the default execute queue thread count using the Administration Console:

Start the Administration Server if it is not already running.

Access the Administration Console for the domain.

Expand the Servers node in the left pane to display the servers configured in your domain.

Right-click the name of the server instance that contains the execute queue you want to configure, and then select View Execute Queues on the pop-up menu to display a table of execute queues that can be modified.

Note: You can only modify the default execute queue for the server or a user-defined execute queue.

In the Name column, click directly on the default execute queue name to display the Configuration tab for modifying execute queues.

Locate the Thread Count value and increase or decrease it, as appropriate.

Click Apply to save your changes.

Reboot the selected server to enable the new execute queue settings.

Assigning Applications to Execute Queues

Although you can configure the default execute queue to supply the optimal number threads for all WebLogic Server applications, configuring multiple execute queues can provide additional control for key applications. By using multiple execute queues, you can guarantee that selected applications have access to a fixed number of execute threads, regardless of the load on WebLogic Server. See Using Execute Queues to Control Thread Usage for more information on assigning applications to configured execute queues.

Allocating Execute Threads to Act as Socket Readers

For best socket performance, BEA recommends that you use the native socket reader implementation, rather than the pure-Java implementation, on machines that host WebLogic Server instances (see Using WebLogic Server "Native IO" Performance Packs). However, if you must use the pure-Java socket reader implementation for host machines, you can still improve the performance of socket communication by configuring the proper number of execute threads to act as socket reader threads for each server instance and client machine.

The ThreadPoolPercentSocketReaders attribute sets the maximum percentage of execute threads that are set to read messages from a socket. The optimal value for this attribute is application-specific. The default value is 33, and the valid range is 1-99.

Allocating execute threads to act as socket reader threads increases the speed and the ability of the server to accept client requests. It is essential to balance the number of execute threads that are devoted to reading messages from a socket and those threads that perform the actual execution of tasks in the server.

Setting the Number of Socket Reader Threads For a Server Instance

To use the Administration Console to set the maximum percentage of execute threads that read messages from a socket:

Start the Administration Server if it is not already running.

Access the Administration Console for the domain.

Expand the Servers node in the left pane to display the servers configured in your domain.

Click the name of the server that you want to configure.

Select the Configuration —> Tuning tab.

Edit the percentage of Java reader threads in the Socket Readers attribute field. The number of Java socket readers is computed as a percentage of the number of total execute threads (as shown in the Thread Count field for the Execute Queue).

Apply the changes.

Setting the Number of Socket Reader Threads on Client Machines

On client machines, you can configure the number of available socket reader threads in the JVM that runs the client. Specify the socket readers by defining the following parameters in the java command line for the client:

-Dweblogic.ThreadPoolSize=value-Dweblogic.ThreadPoolPercentSocketReaders=value

Tuning Execute Queues for Overflow Conditions

You can configure WebLogic Server to detect and optionally address potential overflow conditions in the default execute queue or any user-defined execute queue. WebLogic Server considers a queue to have a possible overflow condition when its current size reaches a user-defined percentage of its maximum size. When this threshold is reached, the server changes its health state to "warning" and can optionally allocate additional threads to perform the outstanding work in the queue, thereby reducing its size.

To automatically detect and address overflow conditions in a queue, you configure the following items:

The threshold at which the server indicates an overflow condition. This value is set as a percentage of the configured size of the execute queue (the QueueLength value).
The number of threads to add to the execute queue when an overflow condition is detected. These additional threads work to reduce the size of the queue to its normal operating size.
The maximum number of threads available to the queue. In particular, setting the maximum number of threads prevents the server from assigning an overly high thread count in response to overload conditions.

To tune an execute queue using the WebLogic Server Administration Console:

Start the Administration Server if it is not already running.

Access the Administration Console for the domain.

Expand the Servers node in the left pane to display the servers configured in your domain.

Right-click the name of the server instance that contains the execute queue you want to configure, and then select View Execute Queues from the pop-up menu to display a table of execute queues that can be modified.

Note: You can only modify the default execute queue for the server or a user-defined execute queue.

In the Name column, directly click the default execute queue name (or the user-defined execute queue) that you want to configure.

On the execute queue Configuration tab, specify how the server instance should detect an overflow condition for the selected queue by modifying the following attributes:

Queue Length: Specifies the maximum number of simultaneous requests that the server can hold in the queue. The default of 65536 requests represents a very large number of requests; outstanding requests in the queue should rarely, if ever reach this maximum value. Always leave the Queue Length at the default value of 65536 entries.
Queue Length Threshold Percent: The percentage (from 1-99) of the Queue Length size that can be reached before the server indicates an overflow condition for the queue. All actual queue length sizes below the threshold percentage are considered normal; sizes above the threshold percentage indicate an overflow. By default, the Queue Length Threshold Percent is set to 90 percent.
Thread Priority: The priority of the threads associated with the queue. By default, the Thread Priority is set to 5.

To specify how this server should address an overflow condition for the selected queue, modify the following attribute:

Threads Increase: The number of threads WebLogic Server should add to this execute queue when it detects an overflow condition. If you specify zero threads (the default), the server changes its health state to "warning" in response to an overflow condition in the execute queue, but it does not allocate additional threads to reduce the workload.

To limit the maximum number of threads that can be added to the selected queue, modify the following attribute:

Threads Maximum: The maximum number of threads that this execute queue can have; this value prevents WebLogic Server from creating an overly high thread count in the queue in response to continual overflow conditions. By default, Threads Maximum is set to 400.

Click Apply to save your changes.

Reboot the selected server to enable the new execute queue settings.

Tuning the Execute Thread Detection Behavior

WebLogic Server automatically detects when a thread in an execute queue becomes "stuck." Because a stuck thread cannot complete its current work or accept new work, the server logs a message each time it diagnoses a stuck thread. If all threads in an execute queue become stuck, the server changes its health state to either "warning" or "critical" depending on the execute queue:

If all threads in the default queue become stuck, the server changes its health state to "critical." (You can set up the Node Manager application to automatically shut down and restart servers in the critical health state. For more information, see "Node Manager Capabilities" in Configuring and Managing WebLogic Server.)
If all threads in weblogic.admin.HTTP, weblogic.admin.RMI, or a user-defined execute queue become stuck, the server changes its health state to "warning."

WebLogic Server diagnoses a thread as stuck if it is continually working (not idle) for a set period of time. You can tune a server's thread detection behavior by changing the length of time before a thread is diagnosed as stuck, and by changing the frequency with which the server checks for stuck threads.

Note: Although you can change the criteria WebLogic Server uses to determine whether a thread is stuck, you cannot change the default behavior of setting the "warning" and "critical" health states when all threads in a particular execute queue become stuck. For more information, see Overview of WebLogic Logging Services.

To configure WebLogic Server stuck thread detection behavior:

Start the Administration Server if it is not already running.

Access the Administration Console for the domain.

Expand the Servers node in the left pane to display the servers configured in your domain.

Click the name of the server instance that you want to modify for improved stuck thread detection.

Note: You configure stuck thread detection parameters on a per-server basis, rather than on a per-execute queue basis.

Select the Configuration —> Tuning tab in the right pane.

Modify the following attributes as necessary to tune thread detection behavior for the server:

Stuck Thread Max Time: Enter the number of seconds, that a thread must be continually working before this server diagnoses the thread as being stuck. By default, WebLogic Server considers a thread to be "stuck" after 600 seconds of continuous use.
Stuck Thread Timer Interval: Enter the number of seconds, after which WebLogic Server periodically scans threads to see if they have been continually working for the length of time specified by Stuck Thread Max Time. By default, WebLogic Server sets this interval to 600 seconds.

Click Apply to save your changes.

Reboot the server to use the new settings.

Tuning Connection Backlog Buffering

Use the AcceptBacklog attribute of the Server element in the config.xml file to set the number of connection requests the WebLogic Server instance will accept before refusing additional requests. The AcceptBacklog attribute specifies how many Transmission Control Protocol (TCP) connections can be buffered in a wait queue. This fixed-size queue is populated with requests for connections that the TCP stack has received, but the application has not accepted yet. The default value is 50 and the maximum value is operating system dependent.

To tune the Accept Backlog value from the Administration Console:

Start the Administration Server if it is not already running.

Access the Administration Console for the domain.

Expand the Servers node in the left pane to display the servers configured in your domain.

Click the name of the server instance that you want to configure.

Select the Configuration —> Tuning tab.

Modify the default Accept Backlog value as necessary to tune how many TCP connections can be buffered in a wait queue:

During operations, if many connections are dropped or refused at the client, and no other error messages are on the server, the Accept Backlog value might be set too low.
If you are getting "connection refused" messages when you try to access WebLogic Server, raise the Accept Backlog value from the default by 25 percent. Continue increasing the value by 25 percent until the messages cease to appear.

Click Apply to save your changes.

How JDBC Connection Pools Enhance Performance

Establishing a JDBC connection with a DBMS can be very slow. If your application requires database connections that are repeatedly opened and closed, this can become a significant performance issue. WebLogic connection pools offer an efficient solution to the problem.

When WebLogic Server starts, connections from the connection pools are opened and are available to all clients. When a client closes a connection from a connection pool, the connection is returned to the pool and becomes available for other clients; the connection itself is not closed. There is little cost to opening and closing pool connections.

How many connections should you create in the pool? A connection pool can grow and shrink according to configured parameters, between a minimum and a maximum number of connections. The best performance occurs when the connection pool has as many connections as there are concurrent client sessions.

In addition to the following subsections, see "Performance Tuning Your JDBC Application" in Programming WebLogic JDBC.

Tuning JDBC Connection Pool Initial Capacity

The InitialCapacity attribute of the JDBCConnectionPool element enables you to set the number of physical database connections to create when configuring the pool. If the server cannot create this number of connections, the creation of this connection pool will fail.

During development, it may be convenient to set the value of the InitialCapacity attribute to a low number to help the server start up faster. In production systems, consider setting the InitialCapacity value equal to the MaxCapacity attribute's default production mode setting of 25. This way, all database connections are acquired during server start-up. And if you need to tune the MaxCapacity value, make sure to set the InitialCapacity so that it equals the MaxCapacity value.

If InitialCapacity is less than MaxCapacity, the server needs to create additional database connections when its load is increased. When the server is under load, all resources should be working to complete requests as fast as possible, rather than creating new database connections.

Tuning JDBC Connection Pool Maximum Capacity

The MaxCapacity attribute of the JDBCConnectionPool element allows you to set the maximum number of physical database connections that a connection pool can contain. Different JDBC drivers and database servers might limit the number of possible physical connections.

The default settings for development and production mode are equal to the default number of execute threads: 15 for development mode; 25 for production mode. However, in production, it is advisable that the number of connections in the pool equal the number of concurrent client sessions that require JDBC connections. The pool capacity is independent of the number of execute threads in the server. There may be many more ongoing user sessions than there are execute threads.

Caching Prepared and Callable Statements

When you use a prepared statement or callable statement in an application or EJB, there is considerable processing overhead for the communication between the application server and the database server and on the database server itself. To minimize the processing costs, WebLogic Server can cache prepared and callable statements used in your applications. When an application or EJB calls any of the statements stored in the cache, WebLogic Server reuses the statement stored in the cache. Reusing prepared and callable statements reduces CPU usage on the database server, improving performance for the current statement and leaving CPU cycles for other tasks. For more details, see "Increasing Performance with the Statement Cache" in the Administration Console Online Help.

Using the statement cache can dramatically increase performance, but you must consider its limitations before you decide to use it. For more details, see "Usage Restrictions for the Statement Cache" in the Administration Console Online Help.

Setting Your Java Compiler

The standard Java compiler for compiling JSP servlets is javac. You can improve performance significantly by setting your server's java compiler to sj or jikes instead of javac. The following sections discuss this procedure and other compiler considerations.

Changing Compilers in the Administration Console

To change your compiler in the Administration Console:

Start the Administration Server if it is not already running.

Access the Administration Console for the domain.

Expand the Servers node in the left pane to display the servers configured in your domain.

Click the name of the server instance that you want to configure.

Select the Configuration —> General tab and enter the full path of the compiler in the Java Compiler field. For example:

c:\visualcafe31\bin\sj.exe

Click Show on the Advanced Options bar to display additional attributes.

Enter the full path to the JRE rt.jar library in the Append to the Classpath field. For example:

BEA_HOME\jdk141_02\jre\lib\rt.jar

Click Apply.

Restart your server for the new Java Compiler and Append to Classpath values to take effect.

Setting Your Compiler in weblogic.xml

In the weblogic.xml file, the jsp-descriptor element defines parameter names and values for servlet JSPs.

Use the compileCommand parameter to specify the Java compiler for compiling the generated JSP servlets.
Use the precompile parameter to configure WebLogic Server to precompile your JSPs when WebLogic Server starts up.

For more information about setting your server's java compiler in the weblogic.xml file, see the jsp-descriptor element.

Compiling EJB Container Classes

Use the weblogic.appc utility to compile EJB 2.0 and 1.1 container classes. If you compile Jar files for deployment into the EJB container, you must use weblogic.appc to generate the container classes. By default, ejbc uses the javac compiler. For faster performance, specify a different compiler (such as Symantec sj) using the -compiler flag.

For more information, see "Implementing EJBs" in Programming WebLogic Enterpise JavaBeans.

Compiling on UNIX

If you receive the following error message received when compiling JSP files on a UNIX machine:

failed: java.io.IOException: Not enough space

Try any or all of the following solutions:

Add more RAM if you have only 256 MB.
Raise the file descriptor limit, for example:

set rlim_fd_max = 4096

set rlim_fd_cur = 1024

Use the -native flag to use native threads when starting the JVM.

Using WebLogic Server Clusters to Improve Performance

A WebLogic Server cluster is a group of WebLogic Servers instances that together provide fail-over and replicated services to support scalable high-availability operations for clients within a domain. A cluster appears to its clients as a single server but is in fact a group of servers acting as one to provide increased scalability and reliability.

A domain can include multiple WebLogic Server clusters and non-clustered WebLogic Server instances. Clustered WebLogic Server instances within a domain behave similarly to non-clustered instances, except that they provide failover and load balancing. The Administration Server for the domain manages all the configuration parameters for the clustered and non-clustered instances.

For more information about clusters, see "Introduction to WebLogic Server Clustering".

Scalability and High Availability

Scalability is the ability of a system to grow in one or more dimensions as more resources are added to the system. Typically, these dimensions include (among other things), the number of concurrent users that can be supported and the number of transactions that can be processed in a given unit of time.

Given a well-designed application, it is entirely possible to increase performance by simply adding more resources. To increase the load handling capabilities of WebLogic Server, add another WebLogic Server instance to your cluster—without changing your application. Clusters provide two key benefits that are not provided by a single server: scalability and availability.

WebLogic Server clusters bring scalability and high-availability to J2EE applications in a way that is transparent to application developers. Scalability expands the capacity of the middle tier beyond that of a single WebLogic Server or a single computer. The only limitation on cluster membership is that all WebLogic Servers must be able to communicate by IP multicast. New WebLogic Servers can be added to a cluster dynamically to increase capacity.

A WebLogic Server cluster guarantees high-availability by using the redundancy of multiple servers to insulate clients from failures. The same service can be provided on multiple servers in a cluster. If one server fails, another can take over. The ability to have a functioning server take over from a failed server increases the availability of the application to clients.

Caution: Provided that you have resolved all application and environment bottleneck issues, adding additional servers to a cluster should provide linear scalability. When doing benchmark or initial configuration test runs, isolate issues in a single server environment before moving to a clustered environment.

How to Ensure Scalability for WebLogic Clusters

In general, any operation that requires communication between the servers in a cluster is a potential scalability hindrance. The following sections provide information on issues that impact the ability to linearly scale clustered WebLogic servers:

Database Bottlenecks

In many cases where a cluster of WebLogic servers fails to scale, the database is the bottleneck. In such situations, the only solutions are to tune the database or reduce load on the database by exploring other options. See JDBC Application Tuning.

Session Replication

User session data can be stored in two standard ways in a J2EE application: stateful session EJBs or HTTP sessions. By themselves, they are rarely a impact cluster scalability. However, when coupled with a session replication mechanism required to provide high-availability, bottlenecks are introduced. If a J2EE application has Web and EJB components, you should store user session data in HTTP sessions rather than stateful session EJBs as HTTP session management provides more replication options than stateful session EJBs. See Managing Sessions.

Invalidation of Entity EJBs

This applies to entity EJBs that use a concurrency strategy of Optimistic or ReadOnly with a read-write pattern.

Optimistic—When an Optimistic concurrency bean is updated, the EJB container sends a multicast message to other cluster members to invalidate their local copies of the bean. This is done to avoid optimistic concurrency exceptions being thrown by the other servers and hence the need to retry transactions. If updates to the EJBs are frequent, the work done by the servers to invalidate each other's local caches become a serious bottleneck.

ReadOnly with a read-write pattern—In this pattern, persistent data that would otherwise be represented by a single EJB are actually represented by two EJBs: one read-only and the other updateable. When the state of the updateable bean changes, the container automatically invalidates corresponding read-only EJB instance. If updates to the EJBs are frequent, the work done by the servers to invalidate the read-only EJBs becomes a serious bottleneck.

Invalidation of HTTP sessions

Similar to Invalidation of Entity EJBs, HTTP sessions can also be invalidated. This is not as expensive as entity EJB invalidation, since only the session data stored in the secondary server needs to be invalidated. BEA advises users to not invalidate sessions unless absolutely required.

JNDI Binding, Unbinding and Rebinding

In general, JNDI binds, unbinds and rebinds are expensive operations. However, these operations become a bigger bottleneck in clustered environments because JNDI tree changes have to be propagated to all members of a cluster. If such operations are performed too frequently, they can reduce cluster scalability significantly.

Performance Considerations When Running Multiple Server Instances on Multi-CPU Machines

With multi-processor machines, additional consideration must be given to the ratio of the number of available CPUs to clustered WebLogic Server instances. Because WebLogic Server has no built-in limit to the number of server instances that reside in a cluster, large, multi-processor servers, such as Sun Microsystems' Sun Enterprise 10000, can potentially host very large clusters or multiple clusters.

In order to determine the optimal ratio of CPUs to WebLogic server instances, you must first ensure that an application is truly CPU-bound, rather than network or disk I/O-bound. Use the following steps to determine the optional ratio of CPUs to server instances:

Test your application to determine the Network Requirements.

If you discover that an application is primarily network I/O-bound, consider measures to increase network throughput before increasing the number of available CPUs. For truly network I/O-bound applications, installing a faster network interface card (NIC) may increase performance more than additional CPUs, because most CPUs would remain idle while waiting to read available sockets.

Test your application to determine the Disk I/O Requirements.

If you discover that an application is primarily disk I/O-bound, consider upgrading the number of disk spindles or individual disks and controllers before allocating additional CPUs.

Begin performance tests using a ratio of one WebLogic Server instance for every available CPU.

If CPU utilization is consistently at or near 100 percent, increase the ratio of CPUs to servers by adding an additional CPU. Add additional CPUs until utilization reaches an acceptable level. Remember, always reserve some spare CPU cycles on your production systems to perform any administration tasks that may occur.

Monitoring a WebLogic Server Domain

The tool for monitoring the health and performance of your WebLogic Server domain is the Administration Console. Using the Administration Console, you can view status and statistics for WebLogic Server resources such as servers, HTTP, the JTA subsystem, JNDI, security, CORBA connection pools, EJB, JDBC, and JMS.

For more details, see "Monitoring a WebLogic Server Domain in Configuring and Managing WebLogic Server.

For example, there is a Server —> Monitoring —> Performance tab on the Administration Console that provides performance metrics related to pending and processed requests for the current server instance. It includes the following information:

The number of idle threads assigned to the queue.
The oldest pending request in the queue.
Throughput, as measured by the number of requests already processed by the queue.
Queue Length, as measured by the number waiting requests in the queue.
The amount of memory available in the JVM heap.

For more details, see "Server -> Monitoring -> Performance in the Administration Console Online Help.