Chapter 6   Performance Tuning and Monitoring


After installing the iPlanet Market Maker software, a number of performance tuning steps can be carried out to ensure the best performance of the application. This chapter includes:

• System configuration.

• Database configuration.

• Tuning the iPlanet Web Server.

• Performance Monitoring.

System Configuration

---

Solaris Kernel Parameter Tuning

To achieve better performance and scalability, the Solaris operating system can be tuned. These settings should be made on the clients - web server / iPlanet Market Maker hosts. The following three settings should be made to the system file so that the settings become effective with the current values whenever the server is booted.

Set rlim_fd_max=8192

Set rlim_fd_cur=8192

Set sq_max_size=0

For details refer to the URL : http://home.netscape.com/eng/server/webserver/4.0/tuning/perf.htm#1019260.

The following settings are required to be run from the command line whenever the system is rebooted. For easy execution, they can be grouped and run as a script file.

ndd -set /dev/tcp tcp_time_wait_interval 60000

ndd -set /dev/tcp tcp_conn_req_max_q 1024 (Only for Solaris 2.8)

ndd -set /dev/tcp tcp_conn_req_max_q0 4096

ndd -set /dev/tcp tcp_ip_abort_interval 60000

ndd -set /dev/tcp tcp_keepalive_interval 900000

ndd -set /dev/tcp tcp_rexmit_interval_initial 3000

ndd -set /dev/tcp tcp_rexmit_interval_max 10000

ndd -set /dev/tcp tcp_rexmit_interval_min 3000

ndd -set /dev/tcp tcp_smallest_anon_port 1024

ndd -set /dev/tcp tcp_slow_start_initial 2

ndd -set /dev/tcp tcp_xmit_hiwat 32768

ndd -set /dev/tcp tcp_recv_hiwat 32768

Memory Configuration

Sizing the RAM for your configuration depends on the implementation scenario you choose. The only reliable way to estimate your memory requirements is through stress testing the system with the user loads and user actions you would expect in a real world scenario.

Each server type in the production systems have different forces that drive the memory requirements. Consider the following factors to configure memory across different servers and perform stress test to monitor the memory utilization.

• The proxy tier utilizes memory for caching and SSL decryption. The decryption proxy servers utilize memory for decrypting the content sent to and returning from the web server layer. The size of the memory which can only be determined via stress and performance testing should be tailored to the throughput of the proxy server layer.

• The static server is responsible for the caching of the static content of the market place. The more static content you have that can be cached, the more you will benefit from more memory.

• The web server tier derives its memory requirements more directly from the application itself. This is determined by the marketmaker in addition to the customization that is made to the product in your installation. As you increase the number of users on the system, the memory requirements for the web server increase. As a rule of thumb, most web servers are configured for 512MB to 1 GB of RAM per CPU. Your selection of the appropriate amount of RAM will be contingent on your performance testing results.

• The iPlanet Directory Server benefits a great deal from memory and so it is important that the memory be sized appropriately for your installation. The memory in the Directory Server used for caching is able to deliver very high levels of performance when ample memory is available. Refer to the iPlanet Directory Server Guides for latest sizing instructions.

• The Oracle 8i server also benefits greatly from appropriate amounts of memory. Since the sizing of the Oracle 8i server can be very complex, consult your database administrator (DBA) for information on sizing the memory on the server.

Refer to "Performance Monitoring" to learn how to observe memory utilization. A full listing on all the processes with the processor numbers and their memory utilizations can be helpful in determining the memory configuration. A few tips to configure memory are:

• Running vmstat will give you the information about the free and available memory that can be utilized optimally.

• At the web server layer, if you are running short of memory, you can down size JVM heap, taking care not to go below 256 Megs.

• On the database machine, give Oracle as much memory as you can to limit or avoid swapping.

For more details on configuring the memory, refer to the administrator's guides of the individual server applications.

Disk Configuration

Depending on the size of the database, read / write loads can be heavy. Placing the Oracle tablespaces and LDAP database in a single disk will result in queueing the read and write operations, causing undue delay in response. Separating and distributing the Oracle tablespaces among multiple disks or an array of disks enables simultaneous (parallel) access to the database, which will solve the problem. The iPlanet Market Maker software is configured to work with separate Oracle data and index tablespaces for each module, such that the tablespaces can be distributed among separate disks or disk arrays. For example, the Auction, and Catalog modules have separate data and index tablespaces and while bidding on auctions, a user accesses the Auction tables, the Catalog tables, and the Common tables individually. If the tablespaces for these data and their indices are placed in separate disks or disk arrays, they can be accesssed simltaneously which will reduce the response time considerably. Similarly, LDAP database files, index files and transaction logs can also be distributed among multiple disks to achieve better response times during Community-Registration activity.

Based on our performance analysis, the following general guidelines are recommended:

1. Use disk arrays striping the group of disks and creating a logical mount point. These striped logical disk units can be used in place of singular disks for better performance. Hardware striping (or Raid 0) can be used to get optimal performance by striping the groups of disks. The disk arrays can be mirrored by using the operating system's mirroring or Raid Solutions.

2. Distribute the I/O evenly among the controllers.

For specific instructions on how to distribute Oracle and LDAP database among multiple disks, refer to the following sections :

"Distributing the Write Load Among Multiple Disks" and "Distributing the Tablespaces Among Multiple-Disks" respectively.

Database Configuration

---

To improve response times, read and write performance of the LDAP and Oracle database must be configured. The following sections describe how LDAP and Oracle can be configured to achieve better performance. Also refer to "iPlanet Directory Server 4.12 Documentation" and "Oracle 8i Designing and Tuning for Performance Release 2 (8.1.6)".

To Improve the LDAP Read Performance

LDAP maintains two levels of cache. Improved LDAP performance can be acheived by tuning these two-level caches :

• Database cache.

• Entry cache.

The basic rule for sizing these caches is :

Dedicate 75% of your cache memory to dbcachesize and 25% to the entry cache size. For example, if you have 500 MB RAM available for caching the directory, tune the caches as descibed below.

Tuning the database cache

The database cache is the lower-level cache, which caches the pages from the database. The size of the database cache is set in terms of memory size in the file slapd.ldbm.conf in the config directory of the LDAP instance home.

1. Calculate the database cache as follows:

   The memory used by the database cache = 500 * 75% = 375MB, and the overhead associated with cache management = 25%; therefore the setting should be = 375MB / 1.25 = 300 MB

2. Apply the caveats as follows before accepting the results:
   • If the result is larger than 1.6 GB, reduce it to 1.6 GB (the maximum dbcache size).

   • If the result is smaller than the total size of the database indices (calculated by adding the sizes of all "*.db2" files in the database directory and deducting the size of the file, id2entry.db2), increase it to match the size of the indices, taking care not to exceed available memory.

3. Set the size in the file :

      <Netscape-Home>\slapd-<instance name>\config\ slapd.ldbm.conf

Tuning the entry cache

The entry cache is the higher-level cache, which caches the most recently used entries from the directory. The size is set in the slapd.ldbm.conf file in the config directory of the LDAP instance home, in terms of the number of entries it should hold rather than a memory size.

1. Calculate the entry cache as follows:

   Required Entry size = 1000 bytes per entry.

   The memory used by entry cache = 500 * 25% = 125MB;

   the available memory including overhead = 125 / 1.25 = 100MB;

   therefore the setting should be = 100,000,000 bytes / 1,000 bytes per entry = 100,000 entries.

2. Set the size in the file :

      <Netscape-Home>\slapd-<instance name>\config\ slapd.ldbm.conf

To Improve the LDAP Write Performance

LDAP write performance can be improved by four different methods:

• Setting the db_home_directory.

• Distributing the write load among multiple disk subsystems.

• Disabling durable transactions.

• Setting the `Allids' threshold.

Setting the db_home_directory

Making this setting reduces the LDAP-write related response times considerably. The standard deviations in the response times would decrease considerably. Though the average response times are not considerably affected, this setting would reduce isolated peaks observed in the LDAP write timers thus reducing the chances of a few users being timed out during the registration process under load.

This method can be used if the following symptoms are observed from vmstat and iostat on the LDAP host :

• The operating system seems to flush pages endlessly. This flushing can be so excessive that performance of the entire system is severely degraded.

• The disk is heavily used (more than 1MB per second of data transfer).

• There is a long service time (more than 100ms).

Two additional conditions where this setting may be necessitated are:

• The database cache size is around 100 MB or more.

• There is mostly write activity going on on the LDAP host.

To set the db_home_directory:

The directory referenced by db_home_directory must be a subdirectory of a filesystem of type tempfs (such as /tmp).

1. Create a subdirectory manually under /tmp.

2. Set db_home_directory to /tmp/<subdirectory> in the file:

      <Netscape-Home>\slapd-<instance name>\config\ slapd.ldbm.conf

   The above setting causes the internal directory server database files to be moved to the directory referenced by the parameter.

   Caution : It is possible, but unlikely, that the server will no longer start after the files have been moved, because enough memory cannot be committed. This is a symptom of an overly large database cache size being configured for the database server. If this happens, reduce the size of the database cache size to a value at which the server will start again.

   Making this setting will considerably smoothen the LDAP-write related response times. Though there will not be any considerable change in the average response times, it would reduce isolated peaks in the LDAP-write timers, thus reducing the chances of a few users being timed out.

Distributing the Write Load Among Multiple Disks

Heavy LDAP write loads can occur if there is heavy registration activity. Placing the LDAP database on a single disk can cause long write queues (serial access). Under heavy write loads, this can result in long response times. A multiple-disk system facilitates fast and simultaneous (parallel) access to the database and will reduce the write queues. In "Disk Configuration", you will find the guidelines to configure multiple disks or disk arrays. After configuring the multiple-disk system, you can distribute the LDAP database files, index files, and transaction logs across disks to achieve better response times for the registration activity.

The default installation of the directory places all directory data files under a single subdirectory. These files include:

• Database and index files.

• Transaction logs used mainly for recovery purposes.

• LDAP access, error, and audit logs.

To store the database and log files on different disk volumes, set the directives in the data files slapd.conf and slapd.ldbm.conf to different disks.

1. Modify the "directory" setting in the file <NSHOME>/slapd-<server ID>/config/slapd.ldbm.conf to point to a different location from its default setting of <NSHOME>/slapd-<serverID>/db.

2. The default setting for the transaction log files is: <NSHOME>/slapd-<serverID>/db.

   Modify the db_logdirectory setting in the file <NSHOME>/slapd-<server ID>/config /slapd.conf to point to a different location on a different disk from the one on which the database and index files are located.

3. In the file <NSHOME>/slapd-<server ID>/config /slapd.conf, modify the accesslog setting to point to a different location on a file system mounted on a disk different from that on which the database, index, and transaction log files are set up.

4. When the installation achieves some stability, consider turning off LDAP access logs, because LDAP is very heavy in the writes to the access logs.

   In the file <NSHOME>/slapd-<server ID>/config /slapd.conf turn off access logging by setting the parameter accesslog-logging-enabled to Off.

Disabling Durable Transactions

To reduce the response time for the registration activity and also to reduce the write intensiveness on the disk to which the the LDAP transaction logs are mapped, set "db_durable_transactions" to Off in the file <NSHOME>/ slapd-<server ID>/config/slapd.ldbm.conf.

The above setting is expected to buy some performance gains in terms of better response times under conditions of high load of the Registration activity on the site.

Quoting the Administrator's guide for iPlanet Directory Server 4.1,

"By default, durable database transaction logging is enabled. This means that every time a write is performed on the directory, a corresponding entry is physically written to the database transaction log disk. To improve performance, you can disable durable transaction logging. When you do so, every directory database operation is logically written to the database transaction log file, but it may not be physically written to disk immediately. That means that if a directory change was written to the logical database transaction log file but not physically written to disk at the time of a system crash, you cannot recover the change. When durable transactions are disabled, the recovered database is consistent, but does not reflect the results of any LDAP write operations that completed just before the system crash."

Setting the ALL_IDS_THRESHOLD

If the community size (number of companies) exceeds the value set for ALL_IDS_THRESHOLD, a search with a filter such as ou=immOrg turns into an unindexed search, causing the search times on some community pages to be inordinately long. The default value for this setting is 4000. If the community size grows beyond 4000, considerable deterioration in performance can be expected in terms of response time for login.

The solution to this is to set the ALL_IDS_THRESHOLD to a value equal to the community size that is expected to be reached at steady state for the site.

Refer to the iPlanet Directory Server 4.1 Administrator's Guide for more details on setting the appropriate value for ALL_IDS_THRESHOLD.

1. Set the ALL_IDS_THRESHOLD to the required value in the file <NSHOME>/slapd-<server ID>/config/slapd.ldbm.conf.

2. Refer to the following for more information about tuning the LDAP and make the appropriate changes in settings to improve write performance on LDAP:
   • "Adminstrators reference" at : http://docs.iplanet.com/docs/manuals/directory/41/admin/

   • "LDAP Performance Tuning Primer" at : http://mc.red.iplanet.com/launches/directory/dir4launch/performance/PerfTuningPrimer.html

LDAP Indices

While importing LDAP schema and data during the installation of iPlanet Market Maker software, LDAP indices are created. The file slapd.ldbm.conf holds the list of indices that are created on the LDAP database. While logging in or while performing any such task involving the LDAP database, indexed searches will be made if the indices are present. Absence of any index will make it an unindexed search, and unindexed searches are very slow. The Login page and also some other pages where calls to LDAP are made for authenticating/selecting available privileges for a user will become much slower if the search is unindexed.

If the indices on any of the following attributes are missing, "notes=U" entries will show up in the LDAP access logs corresponding to LDAP searches where the search filter includes any of the unidexed attributes. The table below gives a list of indices that are created and you can verify whether they are present in the file slapd.ldbm.conf.

Table 6-1    LDAP Indices

index aci pres

index changenumber eq

index cn pres,eq,sub

index copiedfrom pres

index displayName sub

index dncomp eq

index entrydn eq

index givenName pres,eq,sub

index immBusinessRole eq

index immMembershipStatus eq

index immroletype eq

index iplanetecguid eq

index iplanetecparentorgguid eq

index mail pres,eq,sub

index mailAlternateAddress eq

index mailHost eq

index member eq

index memberof eq

index nsactionname eq

index nsallowedprincipal eq

index nsCalXItemId pres,eq,sub

index nsLIProfileName eq

index nsresourcename eq

index nswcalCALID pres,eq

index ntGroupDomainId pres,eq,sub

index ntUserDomainId pres,eq,sub

index numsubordinates pres

index o eq,sub

index objectclass eq

index ou eq

index owner eq

index parentid eq

index pipstatus eq

index pipuid pres,eq,sub

index seeAlso eq

index sn pres,eq,sub

index telephoneNumber pres,eq,sub

index uid pres,eq,sub

index uniquemember eq

( pres - presence ; eq - equality and sub - substring index)

To add an index on an attribute in LDAP when LDAP already has data loaded, you could either use the Directory Server Console or make suitable manual additions to the file slapd.ldbm.conf.

Adding indices through the Directory Server Console is the suggested method because addition of indices through the console ensures that the result takes effect on the existing LDAP database retrospectively. Note that while adding indices from the console, the server is placed in a read only mode. During this process, we can neither make any configuration changes to the server nor modify the contents of the directory.

The other method of making manual edits to the file slapd.ldbm.conf is as follows:

1. Add these attributes manually to slapd.ldbm.conf.

2. Create indices on these attributes for the existing data in LDAP if it has any entries that is unindexed with respect to these attributes. This can be done by one of the two methods given below:

3. Export the database as LDIF and import it again. This will cause the indices to be updated and applied on all the new additions in the file slapd.ldbm.conf.

OR

• Run the db2index command line tool.
  1. Stop the server or place it in read only mode. To place the LDAP in the read only mode :
     1. On the Directory Server Console select the Configuration tab.

     2. Select the Database icon in the navigation tree in the left panel.

     3. Select the Settings tab in the right panel.

     4. Select the "Make Database Read-Only" check box.

     5. Click Save.

  2. Change the directory to <NSHOME>/bin/slapd/server,

  where NSHOME is the directory where you installed the directory server.

  1. In this directory, you can find a utility called ns-slapd. Run this utility as follows:

           ns-slapd db2index -f <slapd.conf> -t

           attributeName[:indexTypes]

     where attributeName is the name of the attribute to be indexed, indexTypes is a comma-separated list of indices to be created for the attribute, and slapd.conf is the full path to the slapd.conf file.

  2. Restart the server.

     If the directory server was running at the time of index creation, stop the server, take it out of read only mode, and restart it. If it was stopped for index creation, simply restart it.

Tuning Oracle

Distributing the Tablespaces Among Multiple-Disks

The iPlanet Market Maker software is configured to work with separate Oracle data and index tablespaces for each module, such that the tablespaces can be distributed among separate disks or disk arrays. In "Disk Configuration", you can find instructions on configuring a multiple-disk subsystem. After configuring multiple disks or disk arrays, the data and index tablespaces can be distributed across multiple disks or disk stripes. Given below are some recommendations based on the tests carried out in our performance labs.

• Place data tablespaces and index tablespaces on separate disks or a disk stripe.

• Separate heavily used tablespaces, such as catalog, auction, and common (cmn), so that the tables can be accessed simultaneously.

• Place `redo' logs on a disk separate from other disks.

• Create rollback segments based on the concurrent transactions and separate rollback tablespaces.

• Place temporary tablespaces on disks separate from other tablespaces.

• Place system tablespaces on disks separate from other tablespaces.

During the installation, the Oracle data files are installed in a single location (for example, "/u01/oracle/oradata/<SID>"). As recommended above, the most active datafiles and index files should be moved to other disk locations (for example, "/u02/oracle/oradata/<SID>").

Assume that an open database has a tablespace named USERS that is made up of data files located on the same disk of a computer. To relocate the data files of the USERS tablespace to a different disk drive, execute the following steps :

1. Locate the data files that need to be relocated.

   The following query of the data dictionary view DBA_DATA_FILES lists the data file names and sizes (in bytes) of the USERS tablespace:

      SELECT file_name, bytes FROM sys.dba_data_files WHERE

      tablespace_name = 'USERS';

This will return :

   FILE_NAME                                  BYTES

------------------------------------------ ----------------

   /U02/ORACLE/RBDB1/USERS01.DBF                                  102400000

   /U02/ORACLE/RBDB1/USERS02.DBF                                  102400000

2. Back up the database. Before making any structural change to a database, such as renaming and relocating the data files, it is always recommended that the entire database is backed up.

3. Take the tablespace containing the data files offline, or shut down the database and restart and mount it, leaving it closed. Both the options close the data files.

4. Copy the data files to their new locations and rename them using operating system commands. You can also execute an operating system command to copy a file by using the HOST command.

5. Rename the datafiles within Oracle.

   Now, the datafile pointers for the files that make up the USERS tablespace, recorded in the control file of the associated database must be changed from the old names to the new names. If the tablespace is offline but the database is open, use the ALTER TABLESPACE...RENAME DATAFILE statement. If the database is mounted but closed, use the ALTER DATABASE...RENAME FILE statement.

      ALTER TABLESPACE users

   RENAME DATAFILE '/u02/oracle/rbdb1/users01.dbf',

   '/u02/oracle/rbdb1/users02.dbf'

   TO '/u03/oracle/rbdb1/users01.dbf',

   '/u04/oracle/rbdb1/users02.dbf';

6. Bring the tablespace online, or shut down and restart the database.

   If the USERS tablespace is offline and the database is open, bring the tablespace back online. If the database is mounted but closed, open the database.

7. Back up the database. After making any structural changes to a database, always perform an immediate and complete backup.

   Reference: Oracle8i Administrator's Guide Release 2 (8.1.6)

Tuning the iPlanet Web Server

---

The two settings to be made for the iPlanet Web Server other than the settings defined in "Installing and Configuring Resonate" are as follows :

• Setting the maximum heap size for iPlanet Web Server JVM.

• Turning the JIT Compiler `ON'.

Setting the Maximum Heap Size for the iPlanet Web Server JVM

• Set the maximum heap size for the iPlanet Web Server JVM to 1 GB in the file <NSHOME>/https-<serverID>/config/jvm12.conf.

  jvm.maxHeapSize= 1000000000

  This heap size has been seen to be sufficient for the maximum loads that we put through an iPlanet Web Server JVM, which is about 300 on a single web server instance. If much larger loads are expected through an iPlanet Web Server JVM, this setting might need to be increased. It is not recommended to set this value lower than 256 MB. You can set this to as high a value as possible depending on the available RAM; this helps in avoiding garbage collection.

Turning the JIT Compiler `ON'

• Comment out the following line in the file <NSHOME>/https-<serverID>/config/jvm12.conf :

  java.compiler=NONE

  This ensures that the JIT compiler is ON, which considerably improves the performance on the Web Server Instance.

Performance Monitoring

---

Monitoring performance will be helpful in sizing and performance tuning. A well- tuned system should indicate :

• An even usage of CPU time across all of the servers.

• An even usage of CPU time across all processors in each server.

• A fairly low percentage of system time (0 - 25%), especially if the workflow is computationally intensive.

If you find requests waiting while the processors are not 100 % busy, it indicates that the server is not tuned optimally. Check the active database connections to find the appropriate number of connections for the DB connection pooling.

A variety of command line tools are available to monitor various performance parameters on Solaris.

Monitoring Tools

To monitor I/O usage

Execute the following command to get I/O statistics :

/usr/bin/iostat

To monitor CPU utilization

Execute the following command to get virtual memory statistics regarding process, virtual memory, disk, trap, and CPU activity :

vmstat

To monitor process resource utilization

Execute the following command to get process statistics :

prstat

(Available only on Solaris 8. You can use Top as an alternate to prstat)

To monitor multi-processor CPU usage

Execute the following command to get statistics on each processor :

/usr/bin/mpstat

To observe memory usage

Execute the following command to get a report on process status :

# ps -Pefly | grep "\.k"

This gives a long full listing about every process with the processor number and memory utilization.

To monitor network activity

To watch network activity between IP nodes (server1 and server2) execute the following command :

# snoop from <server1> to <server2>

Log Files

Activities can be logged at different components of the iPlanet Market Maker installation. Logging can be very useful in trouble shooting and solving problems. The web server logs will give you a good idea of activities taking place at a lower level, (for example, JVM issues). The iPlanet Market Maker logs will give you a great deal of application-specific messages, which will be invaluable as you debug a problem. Fresh logs are created every time the web server JVM restarts. While debugging a problem, it is a good idea to `tail' the output of the iPlanet Market Maker log file to watch for errors as `jsp' pages are returned.

You can also adjust the debug settings for the iPlanet Market Maker installation to get a more detailed internal view of the application.

iPlanet Market Maker Logs

The iPlanet Market Maker logs are located under the directory iMM/logs. You can set the debug level by a setting in the file VortexConfiguration.properties.

#CFG_DEBUG_LEVEL (off=3, terse=7, verbose=15, vverbose=31)

# - Debug level for the Market Maker system.

5=3

The default setting of `OFF' means that only errors will be logged. It is recommended not to increase the debug level to any point above terse, because the log volumes will quickly be overwhelming. Moreover, be sure to turn off debug when running under a load, as it will have a dramatic negative impact on performance if the debug remains `ON'. Everytime you recycle the web server, it creates a new log file with a time stamp. The location of the debug log files is specified in the file VortexConfiguration.properties as follows :

#CFG_DEBUG_DESTINATION

# - Location of the debug log file.

6=@IMM_HOME@/logs/imm.log

iPlanet Web Server Logs

Log files can help you monitor your server's activity. You can use these logs to monitor your server and troubleshoot problems.

To configure logging options for the web administration server, perform the following steps:

1. Access the administration server and choose the Preferences tab.

2. Click the Logging Options link.

3. Make the desired changes and click OK.

To view the access log file, perform the following steps:

1. Access the administration server and choose the Preferences tab.

2. Click the View Access Log link and click OK.

To view the error log file, perform the following steps:

1. Access the administration server and choose the Preferences tab.

2. Click the View Error Log link and click OK.

For more details on logging options, refer to the iPlanet Web Server 4.1 Administrator's Guide.

iPlanet Directory Server Logs (LDAP)

Access, audit, and error logs are generated by LDAP and a number of settings related to logging can be made in the file <NSHOME>/slapd-<server ID>/config /slapd.conf.

When the installation achieves some stability, turn off access logging by setting accesslog-logging-enabled to Off.