WebLogic Server Performance and Tuning

Top Tuning Recommendations for WebLogic Server

Performance tuning WebLogic Server and your WebLogic Server application is a complex and iterative process. To get you started, we have created a "top ten" list of recommendations to help you optimize your application's performance. These tuning techniques are applicable to nearly all WebLogic applications. Although we highly recommend performing these tasks in the sequence they are presented, this isn't a requirement.

Understand Your Performance Objectives

Gather information about the level of activity expected on your server, the anticipated number of users, the number of requests, acceptable response time, and an optimal hardware configuration (e.g., fast CPU, disk size vs. speed, sufficient memory, and so on.).

There is no single formula for determining your hardware requirements. The process of determining what type of hardware and software configuration is required to meet application needs adequately is called capacity planning. Capacity planning requires assessment of your system performance goals and an understanding of your application. Capacity planning for server hardware should focus on maximum performance requirements.

For more information about capacity planing for WebLogic Server, see the BEA WebLogic Server Capacity Planning Guide.

Tune the Operating System

Each operating system sets default tuning parameters differently. For Windows platforms, the default settings are usually sufficient. However, the UNIX and Linux operating systems usually need to be tuned appropriately.

UNIX Tuning Parameters

Use the following guidelines when tuning UNIX operating systems supported by WebLogic Server.

Solaris TCP Tuning Parameters

For better TCP (transmission control protocol) socket performance, set the tcp_time_wait_interval parameter as follows:

ndd -set /dev/tcp tcp_time_wait_interval 60000

This parameter determines the time interval that a TCP socket is kept alive after issuing a close call. The default value of this parameter on Solaris is four minutes. When a large number of clients connect for a short amount of time, holding these socket resources can have a significant negative impact on performance. Setting this parameter to a value of 60000 (60 seconds) has shown a significant throughput enhancement when running benchmark JSP tests on Solaris. You might want to reduce this setting further if the server gets backed up with a queue of half-opened connections.

Note: Prior to Solaris 2.7, the tcp_time_wait_interval parameter was called tcp_close_wait_interval.

For additional recommended Solaris tunable settings, see:

For more information about Solaris tuning options, see:

Solaris Tunable Parameters Reference Manual (Solaris 8)
Solaris Tunable Parameters Reference Manual (Solaris 9)

HP-UX Tuning Parameters

For HP-UX tuning information, see:

Tunable Kernel Parameters reference documentation.
Java Performance Tuning on HP-UX

AIX Tuning Parameters

See the AIX 5L Version 5.2 Performance Management Guide.

Linux Tuning Parameters

For better packet transfer performance, set the /sbin/ifconfig lo mtu parameter as follows:

/sbin/ifconfig lo mtu 1500

The mtu (maximum transfer unit) parameter refers to largest number of bytes that a packet can carry over the network. If the packet size is set too low, then your network performance will decrease due to fragmented data.

For additional recommended Linux tunable settings for WebLogic Server, see Linux Tuning Parameters. For general information about Linux tuning, consult your Linux vendor's documentation. Also, the Ipsysctl Tutorial 1.0.4 describes all of the IP options provided by Linux.

Windows Tuning Parameters

For Windows platforms, the default settings are usually sufficient. For more information about Windows 2000 tuning options, see:

The Microsoft Windows 2000 TCP/IP Implementation Details white paper.
The Windows 2000 Performance Tuning white paper.

Optimize Your Database

Your database can be a major enterprise-level bottleneck. Configure your database for optimal performance by following the tuning guidelines in this section and in the product documentation for the database you are using.

General Suggestions

Here are some general database tuning suggestions:

Good database design — Distribute the database workload across multiple disks to avoid or reduce disk overloading. Good design also includes proper sizing and organization of tables, indexes, logs, and so on.
Disk I/O optimization — Disk I/O optimization is related directly to throughput and scalability. Access to even the fastest disk is orders of magnitude slower than memory access. Whenever possible, optimize the number of disk accesses. In general, selecting a larger block/buffer size for I/O reduces the number of disk accesses and might substantially increase throughput in a heavily loaded production environment.
Checkpointing — This mechanism periodically flushes all dirty cache data to disk, which increases the I/O activity and system resource usage for the duration of the checkpoint. Although frequent checkpointing can increase the consistency of on-disk data, it can also slow database performance. Most database systems have checkpointing capability, but not all database systems provide user-level controls. Oracle, for example, allows administrators to set the frequency of checkpoints while users have no control over SQLServer 7.x checkpoints. For recommended settings, see the product documentation for the database you are using.

Database-Specific Tuning

Here are some basic tuning suggestions for Oracle, SQL Server, and Sybase. Again, you should also check the tuning guidelines in your database-specific vendor documentation.

Oracle

This section describes performance tuning for Oracle 8.1.7.

Number of processes — On most operating systems, each connection to the Oracle server spawns a shadow process to service the connection. Thus, the maximum number of processes allowed for the Oracle server must account for the number of simultaneous users, as well as the number of background processes used by the Oracle server. The default number is usually not big enough for a system that needs to support a large number of concurrent operations. For platform-specific issues, see your Oracle administrator's guide. The current setting of this parameter can be obtained with the following query:

SELECT name, value FROM v$parameter WHERE name = 'processes';

Shared pool size — The share pool in an important part of the Oracle server system global area (SGA). The SGA is a group of shared memory structures that contain data and control information for one Oracle database instance. If multiple users are concurrently connected to the same instance, the data in the instance's SGA is shared among the users. The shared pool portion of the SGA caches data for two major areas: the library cache and the dictionary cache. The library cache stores SQL-related information and control structures (for example, parsed SQL statement, locks). The dictionary cache stores operational metadata for SQL processing.

For most applications, the shared pool size is critical to Oracle performance. If the shared pool is too small, the server must dedicate resources to managing the limited amount of available space. This consumes CPU resources and causes contention because Oracle imposes restrictions on the parallel management of the various caches. The more you use triggers and stored procedures, the larger the shared pool must be. The SHARED_POOL_SIZE initialization parameter specifies the size of the shared pool in bytes.
The following query monitors the amount of free memory in the share pool: SELECT * FROM v$sgastat WHERE name = 'free memory' AND pool = 'shared pool';

Maximum opened cursor — To prevent any single connection taking all the resources in the Oracle server, the OPEN_CURSORS initialization parameter allows administrators to limit the maximum number of opened cursors for each connection. Unfortunately, the default value for this parameter is too small for systems such as WebLogic Server. Cursor information can be monitored using the following query:

SELECT name, value FROM v$sysstat
WHERE name LIKE 'opened cursor%';

Database block size — A block is Oracle's basic unit for storing data and the smallest unit of I/O. One data block corresponds to a specific number of bytes of physical database space on disk. This concept of a block is specific to Oracle RDBMS and should not be confused with the block size of the underlying operating system. Note that since the block size affects physical storage, this value can be set only during the creation of the database; it cannot be changed once the database has been created. The current setting of this parameter can be obtained with the following query:

SELECT name, value FROM v$parameter WHERE name = 'db_block_size';

Sort area size — Increasing the sort area increases the performance of large sorts because it allows the sort to be performed in memory during query processing. This can be important, as there is only one sort area for each connection at any point in time. The default value of this init.ora parameter is usually the size of 6-8 data blocks. This value is usually sufficient for OLTP operations but should be increased for decision support operation, large bulk operations, or large index-related operations (for example, recreating an index). When performing these types of operations, you should tune the following init.ora parameters (which are currently set for 8K data blocks):

sort_area_size = 65536
sort_area_retained_size = 65536

Microsoft SQL Server

The following guidelines pertain to performance tuning parameters for Microsoft SQL Server databases. For more information about these parameters, see your Microsoft SQL Server documentation.

Store tempdb on a fast I/O device.
Increase the recovery interval if perfmon shows an increase in I/O.
Use an I/O block size larger than 2 KB.

Sybase

The following guidelines pertain to performance tuning parameters for Sybase databases. For more information about these parameters, see your Sybase documentation.

Lower recovery interval setting results in more frequent checkpoint operations, resulting in more I/O operations.
Use an I/O block size larger than 2 KB.
Sybase controls the number of engines in a symmetric multiprocessor (SMP) environment. They recommend configuring this setting to equal the number of CPUs minus 1.

Identify the Best JVM Settings

Tune your JVM's heap garbage collection and heap size parameters to get the best performance out of your JVM. The Sun HotSpot and WebLogic JRockit JVM parameters that most significantly affect performance are listed below. For more detailed information, consult your JVM vendor's tuning documentation, as well as the JVM-related reading material at Java Virtual Machine (JVM) Information.

Sun JDK

When using the HotSpot VM option (-server or -client), experiment with the following garbage collection parameters:

-Xms and -Xmx (use equal settings at start up)
-XX:NewSize and -XX:MaxNewSize
-XX:SurvivorRatio
-XX:+UseISM -XX:+AggressiveHeap

For more information about tuning the HotSpot JVM, see JVM Heap Size and Garbage Collection.

JRockit JDK

When using JRockit's JVM, experiment with the following garbage collection parameters:

-Xms and -Xmx (use equal settings at startup)
-Xns
-Xgc: parallel
-XXenablefatspin

Also, see WebLogic JRockit Documentation.

Tune WebLogic Server Performance Parameters

The WebLogic Server configuration file (config.xml) contains a number of OOTB (out-of-the-box) performance-related parameters that can be fine-tuned depending on your environment and applications. Tuning these parameters based on your system requirements (rather than running with default settings) can greatly improve both single-node performance and the scalability characteristics of an application.

Try experimenting with the following WebLogic Server configuration tuning parameters to determine your system's "sweet spot" for optimal performance:

Modify the value of the execute queue's Thread Count, as described in Tuning the Default Execute Queue Threads.
If possible, use native performance packs (NativeIOEnabled=true), as described in Using WebLogic Server "Native IO" Performance Packs.
Use application-specific execute queues, as described in Using Execute Queues to Control Thread Usage.
When using a JDBC Connection Pool, modify the following attributes:

DriverName: Use the thin driver or jDriver, as described in "Using JDBC Drivers with WebLogic Server" in Programming WebLogic JDBC.
InitialCapacity: Set this to equal the MaxCapacity value, as described in Tuning JDBC Connection Pool Initial Capacity.
MaxCapacity: Set the MaxCapacity value to at least equal the Thread Count value, and then, if necessary, increase it again until you find the right number, as described in Tuning JDBC Connection Pool Maximum Capacity.

Set the connection pool size to equal the execute queue's Thread Count, as described in How JDBC Connection Pools Enhance Performance.
Set the statement cache as described in Caching Prepared and Callable Statements.
Use multiple execute queues for servlets and JSPs, as described in Assigning Servlets and JSPs to Execute Queues, and for EJBs and RMI, as described in Assigning EJBs and RMI Objects to Execute Queues.
Consider switching the default Java compiler for JSP compilation, javac, which is significantly slower than jikes or sj, as described in Setting Your Java Compiler.

Monitor Disk and CPU Utilization

After following the previous steps, run your application under a high load while monitoring the:

Application server (disk and CPU utilization)
Database server (disk and CPU utilization)

To check your disk utilization on Solaris or Linux, use the iostat -D <interval> command, where the interval value determines how many seconds you want to elapse between monitoring cycles. To check your CPU utilization, simply leave off the -D flag (iostat <interval>).

For Windows, use the Performance Monitor tool (perfmon), to monitor both your disk and CPU utilization.

The goal is to get to a point where the application server becomes 100 percent utilized. If you find that the application server CPU is not close to 100 percent, confirm whether the database is bottlenecked. If the database CPU is 100 percent utilized, then check your application SQL calls query plans. For example, are your SQL calls using indexes or doing linear searches? Also, confirm whether there are too many ORDER BY clauses used in your application that are affecting the database CPU.

If you discover that the database disk is the bottleneck (for example, if the disk is 100 percent utilized), try moving to faster disks or to a RAID (redundant array of independent disks) configuration, assuming the application is not doing more writes then required.

Once you know the database server is not the bottleneck, determine whether the application server disk is the bottleneck. Some of the disk bottlenecks for application server disks are:

JMS file store writes
Transaction logging (tlogs)
HTTP logging
Server logging

The disk I/O on an application server can be optimized using faster disks or RAID, disabling synchronous JMS writes, using JTA direct writes for tlogs, or increasing the HTTP log buffer.

Monitor Data Transfers Across the Network

Check the amount of data transferred between the application and the application server, and between the application server and the database server. This amount should not exceed your network bandwidth; otherwise, your network becomes the bottleneck. To verify this, monitor the network statistics for retransmission and duplicate packets. This can be done using the following command:

netstat -s -P tcp

For instructions on viewing other TCP parameters using the netstat -s -P command, see Setting TCP Parameters With the ndd Command.

Check For Frequent Standard I/O or Logging

Make sure your application is not doing too much standard I/O or excessive logging. Either situation could significantly slow system performance. In production environments, remove all system.out.println statements from your code, as these statements should only be used in development environments for debugging purposes.

Locate Bottlenecks in Your Applications

If you determine that neither the network nor the database server is the bottleneck, start looking at your WebLogic Server applications. Most importantly, is the machine running WebLogic Server able to get 100 percent CPU utilization with a high client load? If the answer is no, then check if there is any locking taking place in the application. You should profile your application using a commercially available tool (for example, JProbe or OptimizeIt) to pinpoint bottlenecks and improve application performance.

Tip: Even if you find that the CPU is 100 percent utilized, you should profile your application for performance improvements.

For more information about application profiling tools, see Using Performance Analysis Tools.

Tune Your Application

This section contains recommended application-specific tuning suggestions for performance improvement.

EJBs

Stateless session beans and MDBs (message-driven beans) — For maximum concurrency, the pool sizes should be at least as large as the thread count of the execute queue that handles requests to such beans.
Use concurrency strategy.
Experiment with EJB pool settings.
Use Call-by-reference.
Cache EJBs.
Increase the MDB pool size for asynchronous message consumption.

See Tuning WebLogic Server EJBs.

JSPs and Servlets

Disable checks for JSP page checks and servlet reloading.
Use in-memory session persistence, as described in Managing Session Persistence.
Precompile JSPs, as described in "Precompiling JSPs" in Programming WebLogic JSP

See "Introduction to Programming" in Programming WebLogic HTTP Servlets.

JMS

Avoid JMS message selectors and use multiple queues/topics to do message selection.
Use asynchronous (onMessage) JMS consumers instead of synchronous receivers.
Defer JMS acknowledgments and commits.

See the "WebLogic JMS Performance Guide" white paper on BEA dev2dev. For administrative tuning guidelines, see "JMS Tuning" in the Administration Console Online Help.

JDBC

Tune your JDBC connection pool's Initial Capacity and Max Capacity settings to complete database requests as fast as possible, rather than creating new connections.
Cache prepared and callable statements used in your applications to minimize processing costs.
Make your transactions single-batch by collecting a set of data operations and submitting an update transaction in one statement in the form.

See How JDBC Connection Pools Enhance Performance and "Performance Tuning Your JDBC Application" in Programming WebLogic JDBC.