Mailstone Utility

This document contains the following sections:

• About the Mailstone Utility
• Running Mailstone on Unix Platforms
• Running Mailstone on Windows NT
• Setting up a Mailstone Test
• Reviewing Results

About the Mailstone Utility

What Mailstone Is

You should use the Mailstone utility when you first set up your messaging system. Based on how well your mail server handles the loads generated by Mailstone, you can determine whether your current deployment configuration is adequate, or whether you need more powerful or additional server machines.

How Mailstone Works

Figure 1. Mailstone testbed

On the mailmaster client machine, Mailstone execution starts with the mailmaster Perl script, which

• reads the mailmaster configuration file to determine the location of the mailclient commands file and the length of time to run the test
• parses the mailclient commands file to determine the testbed configuration and to build the command-line arguments for invoking  the mailclient program
• uses rsh to start the execution of mailclient processes across the testbed
• collects all results and displays them in an HTML page

The main utility program for Mailstone is mailclient. It accepts several command-line arguments (described below), including the location of command information and test-configuration information.  mailclient executes on each client machine and sends its results back to mailmaster for collection and summary.

Mailstone measures both throughput and response time of the mail server being tested. Mailstone reports the number and size of messages sent, the rate at which messages (and bytes) pass through the system, and the average time taken to both transmit a message to the server and receive a message from the server. Mailstone can present its results in both text and graphical format (see Figure 2 and Figure 3).

The Mailstone Files

Table 1. Mailstone files

 

File	Description
mailclient (Unix) mailclient.exe (NT)	The principal Mailstone program--the one that talks to the mail server.
mailmaster (Unix)	The Perl script used to start all the mailclient instances and collect results.
doc/bench doc/bench2k doc/bench3k  doc/bench10k	The "filler" message bodies that Mailstone uses when it sends messages during testing. See Sample Messages in Standard Sizes.
admin/create_accounts_ldif  admin/create_broadcast_ldif	Script tools used to create test mail accounts. See The User-Account LDIF Scripts.
loadclient (Unix)	A shell script that facilitates directly running the mailclient program. See Running Mailstone on Unix Platforms.
conf/commands	An example mailclient commands file. The commands file tells clients which protocols to use and how. (A mailclient commands file must be on each client machine.) See The mailclient Commands File.
conf/CONFIG	An example mailmaster configuration file. This file is read by the mailmaster program, and contains generation information about the test (for example, the amount of time the test should run for ).  See The mailmaster Configuration File.

 

Running Mailstone on Unix platforms

• rsh,  the remote shell program (for remotely executing mailclient on client machines)
• .rhosts, the remote hosts file (for configuring the client machines to accept rsh from the mailmaster client)
• Perl, the Practical Extraction and Report Language (for running mailmaster; the mailmaster client machine must have Perl version 5.0 or later)

Running Mailstone from a Unix mailmaster Client

1. Pick one Unix machine to be the master from which you will run the mailmaster program. Install Mailstone onto this machine by creating a directory /mailstone and unpacking the archive file (mailstone.zip) into it. The unpacking places the proper files in each subdirectory, as shown in Table 1.
 
2. Create user accounts on your mail server:

 
1. Create ldif files that define users test0, test1, and so on by editing and running the scripts create_accounts_ldif and create_broadcast_ldif. See The User Account LDIF Scripts for details.

 
2. Load the ldif file created by the scripts into your directory. (As noted in The User Account LDIF Scripts, be careful not to overwrite your existing directory entries.)

 
3. Send a broadcast message to spam@yourServer.com. This message instructs the mail server to create a mailbox for each user. (Sending the broadcast message is not absolutely necessary since the first piece of mail sent to any user will create the user's mailbox. However, using the broadcast message to create all the mailboxes at the beginning of the test produces better results.)

 
Note that the mail server must be running when you send the broadcast message. Note also that, if you have a large number of test users, it may take a significant amount of time for the broadcast message to be sent to everyone and for all the mailboxes to be created.
 
4. Specify your client configuration and test parameters:
 
1. On the mailmaster client machine, edit the mailclient commands file to specify the client machines that you will use for the test and to define other test parameters. See The mailclient Commands File for details.

 
2. On the mailmaster client machine, edit the mailmaster configuration file so that it defines the appropriate input to mailmaster. See The mailmaster Configuration File for details.
 
5. Create a /mailstone directory on every client. (See Using Windows NT as a Client Machine for special information about Windows NT clients.) You can install the entire set of Mailstone files into this directory, just as on the mailmaster client. As a minimum, the directory should contain
• the mailclient program (select the appropriate one for the client's specific operating system)
• any message bodies (such as bench2k, bench3k, and so on) to be used during the test (place them in the /doc subfolder)
• the mailclient commands file
 
6. Verify that the user running mailmaster has rsh privilege on every client machine in the testbed.

 
7. Run the mailmaster program using this command:

 
         ./mailmaster -c ./conf/CONFIG

(Replace CONFIG with the name of your mailmaster configuration file, if you have changed the name. Replace the pathname with the correct path, if you have moved the file.)
 

8. If necessary, edit the mailmaster configuration file and rerun the test.  Try short test times of 1 minute until everything works properly and the configuration reflects what you want to test.

Running mailclient Directly

mailclient [-d] -s|-u filename -t timeInMins -n numClients

-d : Turn on verbose debugging.; each client process writes to a /tmp/debug.$$

-s : Use stdin to provide commands information (mailmaster uses this by default to rcp the single central copy of the mailclient commands file to all Unix clients)

-u : Use the local mailclient commands file specified in filename; for example /mailstone/commands

-t : Run the test for the number of minutes specified in timeInMins

-n : Spawn the number of child processes specified in numClients
 

NOTE: When you run mailclient by itself, its raw results are sent to standard output rather than being sent to mailmaster for further processing.

---

Running Mailstone on Windows NT

Using Windows NT as a Client Machine

• The mailmaster client (the machine that runs the mailmaster script) cannot be Windows NT.
• Each Windows NT client must be configured to accept rsh commands from the mailmaster client.
• You must manually load the mailclient commands file onto each Windows NT client.

To use a Windows NT machine as a client that works with a Unix master, follow these steps:

1. Create a C:\MAILSTONE directory on your NT system. Into this directory, place
• the rshd.exe file
• the mailclient commands file
• the mailclient.exe program
• any message bodies (such as bench2k, bench3k, and so on) to be used during the test (place them in the /doc subfolder)

 
2. Run the rshd.exe program with the -r option (to disable access control), or as an NT service, on the client machine. You must do this before starting the test, or the mailmaster script will not be able to communicate with this client.

Using Windows NT as a Single Stand-alone Client Machine

Follow these steps to run Mailstone on a single-client Windows NT configuration:

1. Install Mailstone on the machine, into the following directory structure. Create a  C:\mailstone directory. The directory should contain at least these files:
• the mailclient.exe program
• the mailclient commands file (start with a copy of the sample file commands that comes with Mailstone)
• the test message bodies (bench2k, bench3k, and so on) that Mailstone will use
• the scripts create_accounts_ldif and create_broadcast_ldif

 
2. Create user accounts and passwords on your mail server:

 
1. Create ldif files that define users test0, test1, and so on. If you have a version of Perl that runs on Windows NT, you can do this by configuring the scripts create_accounts_ldif and create_broadcast_ldif. See The User Account LDIF Scripts for details.

 
2. Load the ldif file you have created into your directory. (As noted in The User Account LDIF Scripts, be careful not to overwrite your existing directory entries.)
 
3. Send a broadcast message to spam@yourServer.com. This message instructs the mail server to create a mailbox for each user. (Sending the broadcast message is not absolutely necessary since the first piece of mail sent to any user will create the user's mailbox. However, using the broadcast message to create all the mailboxes at the beginning of the test produces better results.)

 
Note that the mail server must be running when you send the broadcast message. Also, if you have a large number of test users, it may take a significant amount of time for the broadcast message to be sent to everyone and for all the mailboxes to be created.
 
4. Specify your test parameters. Modify the mailclient commands file to define the commands, number of threads, time, and other test parameters.  See The mailclient Commands File for details.

 
5. Run the mailclient program as shown here:

mailclient [-d] -u filename -t timeInMins -n numClients
 
where the options have the following meanings:
 
-d : Turn on verbose debugging; each client thread writes to  .\debug.$$

-u : Use the local mailclient commands file specified in filename; for example C:\mailstone\commands

-t : Run the test for the number of minutes specified in timeInMins

-n : Spawn the number of  NT threads specified in numClients
 

NOTE: When you run mailclient by itself, its raw results are sent to standard output rather than to mailmaster for further processing.

---

Setting Up a Mailstone Test

How Many Client Machines to Use?

The mailmaster Configuration File

# Turn on debugging.
DEBUG=1

# Test time, in minutes
TIME=12
 
# path to gnuplot
GNUPLOT=./admin/bin/gnuplot

# path to ppmtogif
PPMTOGIF=./admin/bin/ppmtogif
 

The DEBUG Directive

CAUTION: debugging files can grow very large; if necessary, shorten the test time (TIME) accordingly.

The THROTTLE Directive

The GNUPLOT Directive

ftp://prep.ai.mit.edu./pub/gnu/gnuplot-3.5.tar.gz

The PPMTOGIF Directive

The mailclient Commands File

The format of the mailclient commands file is similar to that of the the obj.conf file on Netscape servers. It includes InitHost directives, read by mailmaster, which describe the testbed. It also includes the command tags (<SMTP>, <POP3>, <IMAP4>, and <Default>), which describe each command to be executed and which machines it should run on. The command tags occur in pairs, some tags have attributes embedded within them, and the opening and closing tags of a pair surround command-specific directives.
 

An Example mailclient Commands File

InitHost Directive

 

host	(required) Host name of client machine (must be identical to name returned by hostname command)
clients	(required) Number of simultaneous Unix processes (or NT threads) to execute on that host
type	(required) Operating system (unix or nt)
ramp	(optional) Number of seconds between forking of processes (threads) on that host

 

InitHost host=mingus clients=5 type=unix ramp=60

The rest of the mailclient commands file is parsed by the mail clients on each host. The mailclient program looks for command tags that apply to the given host it is running on.  For example, one machine might be used solely for IMAP commands while another is used solely for SMTP sends.

The mail client determines the set of commands that apply to its host and then begins selecting commands to execute at random. The client generates a random number to determine which user account to either send mail to or retrieve the mailbox contents of; the client also generates a weighted random number to determine which command to execute. Each command has a weight factor that determines its relative frequency of occurrence.  For example, if the weights of all commands for a host sum to 100, and a particular command has a weight of 18, then that command has an 18% chance of being chosen every time a command is executed. By adjusting the weight factors for the commands, you can balance the sends and retrieves appropriately for your clients.
 

Descriptions of the Command Tags

<Default>...</Default>

A convenience tag to set values default for directives for the rest of the command tags.  For example, each tag may use the same mail server or the number of test accounts may be the same. Directives defined here needn't be defined in the other tags.

 
These are some directives that make useful defaults:

 

Directive	Description	Example
server	The mail server	mail.foo.com
smtpMailFrom	Email address of sender (all messages come from one sender)	mailstone
accountFormat	Mail address convention for test users	test%d@mingus.mcom.com    (for user accounts test0, test1, etc.)
passwdFormat	Password convention for test users	password    or password%d    (for user accounts test0, test1, etc.)
numAccounts	Number of test user accounts	5000
beginAccounts	An optional number at which to begin numbering user accounts (if you don't want to start at 0)	100
throttle	Seconds-per-operation throttle, to artificially limit the rate at which clients access the server	10

 

 
Note: Redefining a directive in one of the other command tags overrides--for that command alone--the default defined here.

 

<SMTP HOSTS=machA,machB,machC,...>...</SMTP>

Tells a mail client running on a host listed in the HOSTS attribute to send a message by SMTP. (Note that host names listed in the HOSTS attribute here must exactly match one or more of the host names specified in the InitHost directives at the beginning of the mailclient commands file.)

 
Mailstone performs these tasks when it executes an SMTP message send:
 

1. It connects to the mail server, noting the time required to achieve connection.
2. It sends a greeting message to the server.
3. It constructs a message with MAIL FROM and RCPT TO fields, appends a message body, and sends it to the mail server.
4. It quits the connection, noting the time required to send the message and the total time since attempting the connection.

 
SMTP messages go to the user account specified with the accountFormat directive (mailclient applies a random number as shown to get the actual address). The sender is the mailclient program itself (the message header lists the sender as the string defined by the smtpMailFrom directive).
 
These are the directives specific to the <SMTP> tag:

 

Directive	Description	Example
server*	The mail server	mail.foo.com
smtpMailFrom*	Email address of sender	mailstone
accountFormat*	Mail address convention for test users	test%d@mingus.mcom.com    (for user accounts test0, test1, etc.)
numAccounts*	Number of test user accounts	5000
weight*	Command weighting   (compared to other commands for a given host)	20
file*	File to send as the message body	/mailstone/bench2
numrecips	Number of mail recipients per send (regardless of the number of recipients, each message counts as one SMTP send)	3
throttle	Seconds-per-operation throttle	10
*Required if not defined with <Default>

 

<POP3 HOSTS=machA,machB,machC,...>...</POP3>

Tells a mail client running on a host listed in the HOSTS attribute to execute a POP download. (Note that host names listed in the HOSTS attribute here must exactly match one or more of the host names specified in the InitHost directives at the beginning of the mailclient commands file.)

 
 Mailstone performs these tasks when it executes a POP3 download:
 

1. It connects to the mail server, noting the time required to achieve connection.
2. It submits a user's name password, and status to the mail server.
3. It loops through all messages in the user's mailbox, retrieving each one and then deleting it.
4. It quits the connection, noting the time required to retrieve all messages and the total time since attempting the connection.

 
Messages are downloaded from the mailbox specified with the accountFormat directive (mailclient applies a random number as shown to get the actual address). The client connects to the user account's mailbox and downloads all new messages.
 
These are the directives specific to the <POP3> tag:

 

Directive	Description	Example
server*	The mail server	mail.foo.com
accountFormat*	Mail address convention for test users	test%d@mingus.mcom.com   (for user accounts test0, test1, etc.)
passwdFormat*	Password convention for test users	password
numAccounts*	Number of test user accounts	5000
weight*	Command weighting   (compared to other commands for a given host)	20
leaveMailOnServer	Don't delete downloaded mail from server (Boolean)	1
throttle	Seconds-per-operation throttle	10
*Required if not defined with <Default>

 

<IMAP4 HOSTS=machA,machB,machC,...>...</IMAP4>

Tells a mail client running on a host listed in the HOSTS attribute to execute an IMAP session. (Note that host names listed in the HOSTS attribute here must exactly match one or more of the host names specified in the InitHost directives at the beginning of the mailclient commands file.)

 
 Mailstone performs these tasks when it executes an IMAP session:
 

1. It connects to the mail server, noting the time required to achieve connection.
2. It logs in with a user's name and password.
3. It selects a mailbox and loops through all messages, retrieving each one and then deleting it.
4. It closes the mailbox, noting the time required to retrieve all messages.
5. It quits the connection, noting the total time since attempting the connection.

 
The client connects to the mailbox of the user account specified with the accountFormat directive (mailclient applies a random number as shown to get the actual address) and downloads all new messages.
 
These are the directives specific to the <IMAP4> tag:

 

Directive	Description	Example
server*	The mail server	mail.foo.com
accountFormat*	Mail address convention for test users	test%d@mingus.mcom.com   (for user accounts test0, test1, etc.)
passwdFormat*	Password convention for test users	password
numAccounts*	Number of test user accounts	5000
weight*	Command weighting   (compared to other commands for a given host)	20
throttle	Seconds-per-operation throttle	10
checkMailInterval	Interval (in seconds) between checks for new mail	120
searchFolder	Folder on server in which to search for user's mailbox  (currently, only INBOX is supported)	INBOX
searchPattern	String pattern to use in searching for the message subject	test
searchRate	Number of searches to perform per 8-hour day. Mailstone  randomly chooses a search schedule based on the search rate and spreads the searches over the entire test run. With a search rate of 12, an 8-hour  run would execute all searches, whereas a 30-minute run may not execute any at all.	12
*Required if not defined with <Default>

 

The User-Account LDIF Scripts

Mailstone includes the two Perl scripts create_accounts_ldif and create_broadcast_ldif, which you can use to facilitate the creation of the user accounts needed to run a test. You create the accounts in your user directory (Mailstone requires that you have a functioning  LDAP directory server in place).

User accounts are of a form similar to  usernameN, where username is a common string, and N varies from zero to one less than the number of users in the test.  Account names are defined by the accountFormat attribute in the mailclient commands file. The examples in this document use account names of test0, test1, and so on (accountFormat = test%d).

Passwords for the accounts can be the same for all users (the examples shown here all use the single password password), or you can define a password format that, as in the account name, includes a number. For example, the format could be pwd0, pwd1, and so on (passwdFormat = pwd%d).

You need to modify the following lines in the create_accounts_ldif and create_broadcast_ldif scripts before using the scripts:
 

 

Default value	Modification needed
#!/usr/local/bin/perl	Make sure this is the correct path to your Perl binary. (Note that Mailmaster requires Perl 5.0.)
$server = "mingus.mcom.com";	The fully qualified domain name (FQDN) of the mail server being tested, as listed in your LDAP directory.
$basedn = "o=netscape, c=US";	The base distinguished name (DN) for directory searches on your directory.
$usernamebase = "test";	The base string from which to construct user account names for the test. (You can use this default as is; if you change it here, you must also change it in the accountFormat attribute in the mailclient commands file.)
$userpassword = "netscape";	The password for all user accounts. Make sure this is the same value that you specify in the passwdFormat attribute in the commands configuration file.    NOTE: If you use unique passwords with a replaceable-number format, such as pwd%d, you'll need to bypass or modify this script to accommodate that format.
$num_accounts = 2000;	The number of test accounts to create. This should be at least as large as the maximum number of users you expect to use in subsequent tests.
$mailboxSubDirs =  $num_accounts/500;	The maximum number of mailboxes per subdirectory on the mail server. For most situations, the default is reasonable and you can keep it. If you have particular reasons for changing it and understand the implications of doing so, you can use a different value.

 

You first run the create_accounts_ldif script  to create a test LDIF file that contains directory entries for all the user accounts:
 

# ./create_accounts_ldif > accounts.ldif
 

You then run the create_broadcast_ldif script (it creates the spam broadcast address that initially constructs the user mailboxes) and append the results to your test LDIF file:
 

# ./create_broadcast_ldif >> accounts.ldif
 

You then add the test LDIF file's entries to your directory.

IMPORTANT: Do not overwrite your existing directory entries by importing the test LDIF file and creating a completely new LDAP database. Either use the Directory Server's Add Entries user interface, or export the current directory as LDIF, combine it with the test LDIF file, and reimport the combined file. See the Directory Server Administrator's Guide and online Help for more information.

Sample Messages in Standard Sizes

The Mailstone distribution also includes three standard-sized message bodies that you can use for sending and retrieval during the tests. The files bench2k, bench3k, bench10k, and bench consist of meaningless text data occupying 2K, 3K, 10K, and 250K bytes, respectively. Mailstone constructs a message that it sends to a user mailbox by adding header information to the body data.

You specify which of the body files to use when you define the file attribute of the SMTP tag in the mailclient commands file. (You can specify your own file in place of the provided files, if you wish.)
 

Reviewing Results

The top of the page lists various summary statistics such as total number of connections and total messages, connections per second and messages per second, and server throughput. A table shows results broken down by protocol (SMTP, POP3, and IMAP4).

If you use the gnuplot and ppmtogif programs (see The mailmaster Configuration File), the results page also includes a graphical plot of the results, as shown in Figure 3.
 
Figure 3. Plot of Mailstone results
 

Consider these two points when reviewing the results of a Mailstone performance analysis:

• Did the test accurately simulate the type and volume of mail traffic for your enterprise?
• Did the server--both software and machine--handle the load within an acceptable margin of performance?

Mailstone displays the results of a run in terms of the statistics for each protocol:  SMTP, POP3, and IMAP4. With this information, you can  determine (1) whether enough SMTP connections were made to the server during the run and (2) how many messages were downloaded over how many POP3/IMAP4 connections.

If the number of SMTP connections is not in the acceptable range--that is, not close to the projected usage for your installation--consider adding more client processes or machines or checking the server performance during the run.  You should also check the message/connection ratio for POP3/IMAP4 for consistency with your projections, and make any necessary adjustments adjustments to the mailboxes before running the next test.

Monitoring the server performance during test runs is crucial in understanding the results.  If you have a sufficient number of clients for the test, and yet the desired number of client connections is not being achieved and the server CPU usage and process run queue is not settling down after the initial spike, you may need to modify your server architecture to increase capacity.

The analysis of Mailstone results is an iterative process of client (Mailstone client) and server tuning.  The bottom line is to determine whether the messaging solution can handle the type of load expected in an acceptable manner.