Chapter 4 Performance Testing and Fine-Tuning Your System

This chapter explains how to run performance tests against Netscape Application Server (NAS), analyze the test results, and then, based on the results, how to fine-tune your system to improve performance.

• Goals of Performance Testing
• A Typical Application Test Scenario
• Running Performance Tests
• Analyzing Test Results
• Fine-Tuning Netscape Application Server
• Fine-Tuning Worksheet

• How the server responds to a given realistic load and where the bottlenecks occur before you take the system into production
• How your applications perform
• If your current NAS configuration—made up of web servers, back-end data sources, NAS machines, and a certain number of CPUs—can sustain your peak load with an expected level of performance.

1. The user enters the URL of the shopping site's home page to navigate to the application.
2. The user pauses 3 seconds (think time) to view the home page and determine which link or button to click.
3. The user clicks Sign In to navigate to the Customer Sign In page.
4. The user pauses 3 seconds (think time) to view the Sign In page.
5. If the user is already registered, she enters her email address and password, and clicks Sign In.
6. If she is not yet registered, she does the following:

1. Clicks Register Here.
2. Enters the required data on the Registration Page.
3. Clicks Save.
7. The user pauses 3 seconds to visually locate the link she will click to search the Book database. (This, plus Step 5—entering email address and password—or Step 6.2—entering the required data on the Registration page—equals think time.)
8. The user clicks Browse Subjects to navigate to the shopping site's list of book subjects.
9. The user pauses 3 seconds (think time) to view the Browse Subjects page.
10. The user clicks a subject name, for example, Literature or Computer, to navigate to the first page of the resulting list.
11. The user pauses 5 seconds (think time) to view and scroll through the page.
12. The user clicks the Add to Cart button next to an item on the list to add it to the Shopping Cart and simultaneously navigate to the Shopping Cart page.
13. The user pauses 3 seconds to view the Shopping Cart page, which displays the total quantity and cost of the items currently in the shopping cart.
14. The user repeats Steps 7 through 13 three times.
15. The user clicks Continue Checkout, which navigates to the Checkout page. The checkout page displays a summary of the committed order.
16. The user pauses 6 seconds (think time) to view the page and ascertain that the ordering information is correct.
17. The user clicks Logout to exit the application.

1. Prepare your test environment, and make sure it includes the following:

• Hardware that mirrors your production environment.
• Firewalls that mirror those in your production environment.
• A testing tool that predicts and measures system behavior and performance. You want to use a tool that can mimic your enterprise activity by emulating the number of users and requests you expect on your system. After you run the test, such a tool provides statistics and graphs of system behavior to help you analyze performance.
• The Netscape Application Server Administrator tool. You can use this tool while running tests to measure NAS performance. For information about how to install Netscape Application Server Administrator, see the Installation Guide. For information about how to measure NAS performance using the Netscape Application Server Administrator tool, see Chapter 3, "Monitoring Server Activity," in the Administration Guide.

For performance test results to be accurate, it is very important that your test environment closely match, if not exactly reproduce, your production environment.

2. Decide in advance how many users you expect at peak load and at steady state load.
3. Decide in advance your response time expectations at peak load and at steady state load.
4. Decide in advance your expectations for requests per minute (or per second) at peak load and at steady state loads.
5. Know in advance the durations (start and end times) of peak load and steady state load.
6. Know in advance your user behaviors and the kinds of requests they submit. Your applications determine the mix of requests, based on how they are designed, so know your applications and how your users work with them.
7. Create a script that automates your application scenario.

• The scenario should map as closely to your user behavior as possible, including think times and the typical mix of requests your users submit.
• The script should have the ability to simulate multiple users.
8. Run one or both of the following types of tests, depending on the benchmarking or performance testing tool you use and on what you want to measure:

Peak User Load Test: This test measures how many users your system can handle.

• Run the script with the number of users you expect at steady state load as a parameter.
• Make sure to vary the iterations so that in half of them, the user performs a Registration request by registering on the application for the first time, while in others, the user performs a Sign In request by signing on to the application.
• Increment the number of users over a period of time until you reach the peak load amount. For example, if the initial number of users at steady state is 100, start with that number of simultaneous script iterations and increment this by 20 users every 10 minutes until you reach your peak load of, for example, 300 iterations. Then, continue running the iterations for as long as your peak load typically lasts.

Free Running Threads Test: This test measures how many requests per minute your system can handle, in other words, it validates the system's throughput.

• Create an application scenario that does not include think time. It should consist of a variety of raw requests on free-running threads. As soon as a request is submitted, another is submitted immediately following it, and so on, without any think time between requests.
• This test is useful because you don't need multiple machines representing multiple clients and you can run the test in less time than a test that includes think time. The normal throughput restrictions found in the peak user load test are eliminated.
9. Using a web site performance and benchmarking tool, track the following information:

• Response time of requests at each phase of the test: What is the response time for requests when there's a load of 100 concurrent users? What is the response time when you increase the number to 120 concurrent users? And so on.
• Requests per minute or per second at each phase of the test.
• At what point does performance start to degrade? In other words, how many users are on the system concurrently when response time starts to increase and requests per minute start to decrease?

Tracking this information will help you determine peak capacity information; in other words, information about the maximum number of concurrent users your system can realistically sustain, versus what you'd like it to ideally sustain at peak load.

In this diagram, the ideal patterns of server behavior, as already explained in

• Does the response time for the number of users at a given point in the test cycle meet your response time expectations?
• In particular, does the response time for peak capacity meet your expectations?
• Does the server handle peak load (the expected maximum number of concurrent users) without a drop in requests per minute?
• Is the number of concurrent users acceptable when response time starts to increase and requests per minute start to decrease?

• mpstat
• vmstat
• iostat
• netstat
• sar
• proctool

• Is the bottleneck occurring on the client side?

Although unlikely, sometimes the bottleneck may be in the client that generates requests to the web server. When you run performance tests, always verify if the script or software that simulates user requests has problems at high loads.

• Is the web server performing at near maximum capacity?

Use the operating system tools on the web server machine to determine if a bottleneck exists in the CPU, memory, disk I/O, network I/O, lock usage, and so on. Additionally, look at the web server administration documentation, or consult with your web server system administrator, for information about how to identify performance bottlenecks.

• Is the back-end data source the problem?

Use the operating system tools on the back-end data source machine to determine if a bottleneck exists in the CPU, memory, disk I/O, network I/O, lock usage, and so on. For example, if the CPU on the back-end data source is at maximum capacity, while the NAS CPU is, for example, 50 percent idle, then the back-end data source is the bottleneck.

Additionally, look at the back-end data source administration documentation, or consult with your data source system administrator, for information about how to identify performance bottlenecks.

• Is the application the source of the bottleneck?

Use an application profiling tool to measure where CPU usage, memory leaks, disk I/O, network I/O, lock usage and other such factors take place in your NAS applications. You may discover that your applications need to be optimized. Refer to the Programmer's Guide for information on how to optimize NAS applications.

• Is NAS performing at near maximum capacity?

Use the operating system tools on the NAS machine to determine if a bottleneck exists in the CPU, memory, disk I/O, network I/O, lock usage, and so on. The Netscape Application Server Administrator tool can help monitor several NAS-related statistics and fine-tune NAS. For more information about monitoring and fine-tuning the server, see Chapter 3, "Monitoring Server Activity," in the Administration Guide.

If you determine that the bottleneck exists on NAS, begin by isolating requests that you ran in the tests and fine-tuning different aspects of NAS, according to the information in the following section, "Fine-Tuning Netscape Application Server."

• Java Server Engines (KJSs)
• C++ Server Engines (KCSs)
• Executive Server (KXS)
• Sessions and Session Data
• Request and Response Objects
• Input and Output of Application Components
• Sticky Application Components
• Load Balancing
• Sync Backups
• Database Connections
• Multiple CPUs
• Cluster Sizing
• Resizing for Throughput
• Web Servers

Typically, for optimal performance, specify between 32 and 48 threads per engine, depending on the request load you expect for your applications. Since a thread can only be used by one application component at a time, when a component is running in it, the thread becomes busy until the component completes running. Therefore, if your applications use many components, you should configure the number of threads accordingly. Specifying more than 48 threads has its drawbacks: a greater amount of memory is required and a greater amount of thread context switching occurs.

• Large numbers of requests for small application components that spend very little time in the KXS engine. This condition causes the KXS to work harder and more frequently at redirecting requests.
• Application component result caching. The KXS must update and search the cache to fulfill requests, so when caching is enabled, it affects KXS performance. The amount of data that can be cached is limited by how much physical memory is available. However, caching large amounts of data degrades performance. Typically, the number of cache entries for any instance of an application component shouldn't be more than 100.
• Streaming large amounts of data, such as images and video clips, from the application components. KXS and the Web Connector plug-in performance are affected because all output from an application component must pass through the KXS before reaching the Web Connector plug-in and finally returning to the requesting client.
• Huge request and response streams. When the list of name-value pairs sent to and from an application component (such as a servlet) is large, the KXS becomes busy. For more details about request and response objects, see the section "Request and Response Objects" on page 93.

• Keep the number of entries in the list to a minimum.

Even if the name-value pairs in the list are distinct entries, try to bundle them together. NAS does not interpret the data itself when the KXS engine deconstructs the list, so bundle the entries together, using a protocol, and store the bundled entry as a single item in the list. You can then code the application so that it separates the data when it receives the list.

• Keep the number of cookies to a minimum.

Every cookie becomes a distinct entry in a name-value pair list. Avoid using too many cookies as a way of reducing the number of entries on the list.

• How frequently should a NAS server update its load-balancing information?
• How frequently should every NAS installation broadcast its load-balancing information?

Update Interval . A minimum value of 5 seconds and a maximum value of 10 seconds is appropriate in most cases. In general, set the Update Intervals criteria for each server to be twice the response time, under stable conditions, of the most frequently used application component. For example, on a system where the most frequently used application component returns requests in 5 seconds, set the update interval to 10 seconds. Setting it to a more frequent update rate causes the server to do more work and could even alter load-balancing characteristics. Use caution with this calculation; if the response time of a heavily used application component is only 1.5 seconds, do not set the Update Interval to 3 seconds.

Broadcast Interval. As mentioned earlier, broadcasting load-balancing information too frequently will not only increase network traffic, it will also increase the work load of your NAS system as all the servers work to post and gather the information. In general, set the Broadcast Intervals criteria for a server to be twice the value of its Update Interval.

• The nature of the queries and how long they spend in the database
• The speed of the database
• The size of the data you are retrieving from the database
• The efficiency of the database when handling multiple connections

• You have a single NAS machine system with the following configuration

• 4 CPUs
• 4 KJS engines (one KJS per CPU)
• 32 threads per engine
• On average, 120 requests are submitted concurrently on your system.
• These requests are lightweight (see "Complexity of Requests" on page 68).
• The requests should take no more than 2 seconds to complete.

• Use operating system tools on the web server machine to verify the specific location of the bottleneck on that machine.
• Deploy more web servers.
• Use a web server load balancer, such as Cisco's LocalDirector to optimally distribute requests across all your to the web servers.
• Configure the number of worker threads to be as low as possible.
• See the web server administration documentation.
• Consult with your web server administrator.

• Use operating system tools on the back-end data source machine to verify the specific location of the bottleneck on that machine.
• If your back-end data source is a database, limit the number of database connections to no more than 32. For a configuration of less than 32 threads, set the number of connections from the application to equal the number of threads per KJS or KCS engine that is processing the particular application component.
• For information about implementing database connectivity in your applications, see the

• Programmer's Guide, Chapter 8, "Handling Transactions with EJBs"
• Programmer's Guide, Chapter 9, "Using JDBC for Database Access"
• Sun Java Database Connectivity (JDBC) API Specification. All specifications are accessible from installdir/nas/docs/index.htm, where installdir is the location in which you installed NAS.
• See the back-end data source administration documentation.
• Consult with your data source administrator.

• Use operating system tools on the back-end data source machine to verify the specific location of the bottleneck on that machine.
• Use Netscape Application Server Administrator to monitor and fine-tune the server. For more information about this, see Chapter 3, "Monitoring Server Activity," in the Administration Guide.

• Avoid developing applications that generate large numbers of requests for small application components that spend very little time in the KXS engine.
• Avoid caching large amounts of data. Keep the number of cache entries in any application component at no greater than 100.
• Avoid streaming large amounts of data, such as images and video clips, from your application components.
• Avoid creating huge name-value pair lists in the request and response objects that get sent to and from your application components.
• Configure no more than 128 threads. If you configure less than this number, make sure the number of KXS request manager threads is equal to or less than the sum of all threads in all the KJS and KCS engines that the KXS controls. For details, see Chapter 7, "Increasing Fault Tolerance and Server Resources," in the Administration Guide.

• Adjust the number of KJS engines per CPU:

• A typical KXS can serve approximately 8 KJS engines.
• For maximum performance, limit the number of KJS engines on a four CPU machine to 4 or 6. If you exceed this number, do not use more than 2 engines per CPU.
• If your application components are CPU intensive, configure one engine per CPU.
• If your application components are I/O intensive, configure between two and four engines per CPU, two being the ideal.
• Adjust the number of threads per KJS:

• If your application uses many components, configure the number of threads accordingly.
• Try not to specify more than 48 threads per engine.
• For details about specifying more KJS engines and adjusting the number of threads, see Chapter 7, "Increasing Fault Tolerance and Server Resources," in the Administration Guide.

• Adjust the number of engines per CPU:

• A typical KXS can serve approximately 8 KCS engines.
• For maximum performance, limit the number of KCS engines on a four CPU machine to 4 or 6. If you exceed this number, do not use more than 2 engines per CPU.
• If your application components are CPU intensive, configure one engine per CPU.
• If your application components are I/O intensive, configure between two and four engines per CPU, two being the ideal.
• Adjust the number of threads per KCS engine (between 32 and 128 threads).
• For details about specifying more KCS engines and adjusting the number of threads, see Chapter 7, "Increasing Fault Tolerance and Server Resources," in the Administration Guide.

• Set the Update Intervals criteria for each server to be twice the response time, under stable conditions, of the most frequently used application component. A minimum value of 5 seconds and a maximum value of 10 seconds is appropriate in most cases.
• Set the Broadcast Intervals criteria for a server to be twice the value of its Update Interval.
• For more information about load balancing and using Netscape Application Server Administrator to set load-balancing criteria, see Chapter 13, "Balancing User-Request Loads," in the Administration Guide.

• Evaluate the number of machines in your cluster and decide if you should add more.
• See Chapter 2, "Planning Your Environment" for more information.

• Use only if absolutely necessary.
• For more information, see "Resizing for Throughput" on page 100, and Chapter 2, "Planning Your Environment."

• Decide how much memory to deploy based on the number of CPUs in your NAS machine.

• For example, a four CPU machine performs well with a total of 512 MB of memory, which amounts to 128 MB per CPU.
• If your applications require more memory, deploy more, such as 256 MB per CPU on a four CPU machine.

• Use an application profiling tool to measure where CPU usage, memory leaks, disk I/O, network I/O, lock usage and other such factors take place in your NAS applications.
• Refer to the Programmer's Guide for information on how to optimize NAS applications

• When using data synchronization, keep sessions small. They should be no greater than 1 to 2 KB in size.
• For information about creating sessions, see Chapter 11, "Creating and Managing User Sessions," in the Programmer's Guide.

• Keep the number of entries in the request and response objects to a minimum.
• Keep the number of cookies to a minimum.
• For more information about request and response objects, see Chapter 3, "Controlling Applications with Servlets," in the Programmer's Guide.

• Improve response time by making sure that all the information an application component requires and returns can fit into a single packet.
• Instead of fitting all the information into one packet, send several packets, but not exceed 8 packets.
• Limit packet sizes to 31 KB.

• Decide whether or not to mark application components as sticky to improve performance.
• For details, see Chapter 13, "Balancing User-Request Loads," in the Administration Guide.