Chapter 5   MTA Concepts


This chapter provides a conceptual description of the MTA. It consists of the following sections:

• "The MTA Functionality"

• "MTA Architecture and Message Flow Overview"

• "The Dispatcher"

• "Rewrite Rules"

• "Channels"

• "The MTA Directory Information"

• "The Job Controller"

The MTA Functionality

---

The Message Transport Agent, or MTA (Figure 5-2), is a component of the Messaging Server (Figure 5-1)that performs the following functions:

• Message Routing - accepts messages and routes it to: A) another SMTP host, B) a local message store, or C) to the a program for processing (example: virus checking).

• Message Blocking - blocks or accepts messages based on specified source and/or destination IP address, mail address, port, channel, or by header strings.

• Address Rewriting - rewrite incoming From: or To: addresses to a more desired form.

• Message Processing - perform different types of message processing. For example:
  • Expand aliases

  • Control SMTP command and protocol support

  • Provide SASL support

  • Hold messages when the number of addresses exceeds a specified limit

  • Deliver messages to site-supplied program such as virus checkers and mail filing programs

  • Perform message-part-by-message-part conversion on messages

  • Customize of delivery status notification messages

Figure 5-1    iPlanet Messaging Server, Simplified Components View (Messenger Express not Shown)

Figure 5-2    MTA Architecture

MTA Architecture and Message Flow Overview

---

This section provides a short overview of MTA architecture and message flow (Figure 5-2).

Dispatcher and SMTP Server (Slave Program)
Messages enter the MTA from the internet or intranet via SMTP sessions. When the MTA receives a request for an SMTP connection, the MTA dispatcher (a multithreaded connection dispatching agent), executes a slave program (tcp_smtp_server) to handle the SMTP session. As additional sessions are requested, the dispatcher activates an SMTP server program to handle each session. Together the dispatcher and slave program perform a number of different functions on each incoming message. Three primary functions are:

• Message blocking - messages from specified IP addresses, mail addresses, ports, channels, header strings and so on, may be blocked (Chapter 10 "Mail Filtering and Access Control.")

• Address changing. Incoming From: or To: addresses may be rewritten to a different form.

• Channel enqueueing. Addresses are run through the rewrite rules to determine which channel the message should be sent.

For more information see "The Dispatcher"

Enqueue
A number of tasks are achieved during this phase of delivery, but the primary tasks are:

• Alias expansion.

• Running the addresses through the rewrite rules to determine to which channel the message should be enqueued and to rewriting the domain part of addresses into their proper or desired format.

• Channel keyword processing.

• Sending the message to the appropriate channel queue.

Channels
The channel is the fundamental MTA component used for message processing. A channel represents a message connection with another system (for example, another MTA, another channel, or the local message store). As mail comes in, different messages require different routing and processing depending on the message's source and destination. For example, mail to be delivered to a local message store will be processed differently from mail to be delivered to the internet which will be processed differently from mail to be sent to another MTA within the mail system. Channels provide the mechanism for customizing the processing and routing required for each connection. In a default installation, the majority of messages go to a channels handling internet, intranet, and local messages.

Specialized channels for specific situations can also be created. For example, suppose that a certain internet domain processes mail very slowly causing mail addressed to this domain to clog up the MTA. A special channel could be created to provide special handling for messages addressed to the slow domain, thus relieving the system of this domain bottleneck.

The domain part of the address determines to what channel the message is enqueued. The mechanism for reading the domain and determining the appropriate channel is called the rewrite rules (see "Rewrite Rules").

Channels typically consist of a channel queue and a channel processing program called a master program. After the slave program delivers the message to the appropriate channel queue, the master program performs the desired processing and routing. Channels, like rewrite rules, are specified and configured in the imta.cnf file. An example of a channel entry is shown below:

tcp_intranet smtp mx single_sys subdirs 20 noreverse maxjobs 7 SMTP_POOL maytlsserver allowswitchchannel saslswitchchannel tcpauth
tcpintranet-daemon

The first word, in this case tcp_intranet is the channel name. The last word is called the channel tag. The words in between are called channel keywords and specify how messages are to be processed. Hundreds of different keywords allow messages to be processed in many ways. A complete description of channel keywords is provided in the iPlanet Messaging Server Reference Manual and Chapter 8 "Configuring Channel Definitions."

Message Delivery
After the message is processed, the master program sends the message to the next stop along the message's delivery path. This may be the intended recipient's mailbox, another MTA, or even a different channel. Forwarding to another channel is not shown in the picture, but is common occurrence.

The Dispatcher

---

The Dispatcher is a multithreaded dispatching agent that permits multiple multithreaded server processes to share responsibility for SMTP connection services. When using the Dispatcher, it is possible to have several multithreaded SMTP server processes running concurrently, all handling connections to the same port. In addition, each server may have one or more active connections.

The Dispatcher acts as a central receiver for the TCP ports listed in its configuration. For each defined service, the Dispatcher may create one or more SMTP server processes to handle the connections after they are established.

In general, when the Dispatcher receives a connection for a defined TCP port, it checks its pool of available worker processes for the service on that port and chooses the best candidate for the new connection. If no suitable candidate is available and the configuration permits it, the Dispatcher may create a new worker process to handle this and subsequent connections. The Dispatcher may also create a new worker process in expectation of future incoming connections. There are several configuration options which may be used to tune the Dispatcher's control of its various services, and in particular, to control the number of worker processes and the number of connections each worker process handles.

See "Dispatcher Configuration File" for more information.

Creation and Expiration of Server Processes

Automatic housekeeping facilities within the Dispatcher control the creation of new and expiration of old or idle server processes. The basic options that control the Dispatcher's behavior are MIN_PROCS and MAX_PROCS. MIN_PROCS provides a guaranteed level of service by having a number of server processes ready and waiting for incoming connections. MAX_PROCS, on the other hand, sets an upper limit on how many server processes may be concurrently active for the given service.

It is possible that a currently running server process might not be able to accept any connections because it is already handling the maximum number of connections of which it is capable, or because the process has been scheduled for termination. The Dispatcher may create additional processes to assist with future connections.

The MIN_CONNS and MAX_CONNS options provide a mechanism to help you distribute the connections among your server processes. MIN_CONNS specifies the number of connections that flags a server process as "busy enough" while MAX_CONNS specifies the "busiest" that a server process can be.

In general, the Dispatcher creates a new server process when the current number of server processes is less than MIN_PROCS or when all existing server processes are "busy enough" (the number of currently active connections each has is at least MIN_CONNS).

If a server process is killed unexpectedly, for example, by the UNIX system kill command, the Dispatcher still creates new server processes as new connections come in.

For information about configuring the Dispatcher, see "Dispatcher Configuration File".

To Start and Stop the Dispatcher

To start the Dispatcher, execute the command:

imsimta start dispatcher

This command subsumes and makes obsolete any other imsimta start command that was used previously to start up a component of the MTA that the Dispatcher has been configured to manage. Specifically, you should no longer use imsimta start smtp. An attempt to execute any of the obsoleted commands causes the MTA to issue a warning.

To shut down the Dispatcher, execute the command:

imsimta stop dispatcher

What happens with the server processes when the Dispatcher is shut down depends upon the underlying TCP/IP package. If you modify your MTA configuration or options that apply to the Dispatcher, you must restart the Dispatcher so that the new configuration or options take effect.

To restart the Dispatcher, execute the command:

imsimta restart dispatcher

Restarting the Dispatcher has the effect of shutting down the currently running Dispatcher, then immediately starting a new one.

Rewrite Rules

---

Rewrite rules determine the following:

• How to rewrite the domain part of an address into its proper or desired format.

• To which channel the message should be enqueued after the address is rewritten.

Each rewrite rule consists of a pattern and a template. The pattern is a string to match against the domain part of an address. The template specifies the actions to take if the domain part matches the pattern. It consists of two things: 1) a set of instructions (that is, a string of control characters) specifying how the address should be rewritten and 2) the name of the channel to which the message shall be sent. After the address is rewritten, the message is enqueued to the destination channel for delivery to the intended recipient.

An example of a rewrite rule is shown below:

siroe.com             $U%$D@tcp_siroe-daemon

siroe.com is the domain pattern. Any message with the address containing siroe.com will be rewritten as per the template instructions ($U%$D). $U specifies that the rewritten address use the same user name. % specifies that the rewritten address use the same that the domain separator. $D specifies that the rewritten address use the same domain name that was matched in the pattern. @tcp_siroe-daemon specifies that the message with its rewritten address be sent to the the channel called tcp_siroe-daemon. See Chapter 7 "Configuring Rewrite Rules" for more details.

For more information about configuring rewrite rules, see "The MTA Configuration File" and Chapter 7 "Configuring Rewrite Rules."

Channels

---

The channel is the fundamental MTA component that processes a message. A channel represents a connection with another computer system or group of systems. The actual hardware connection or software transport or both may vary widely from one channel to the next.

Channels perform the following functions:

• Transmit messages to remote systems, deleting them from their queue after they are sent.

• Accept messages from remote systems, placing them in the appropriate channel queues.

• Deliver messages to the local message store.

• Deliver messages to programs for special processing.

Messages are enqueued by channels on the way into the MTA and dequeued on the way out. Typically, a message enters via one channel and leaves by another. A channel might dequeue a message, process the message, or enqueue the message to another MTA channel.

Master and Slave Programs

Generally (but not always), a channel is associated with two programs: master and slave. The slave program accepts messages from another system and adds them to a channel's message queue. The master program transfers messages from the channel to another system.

For example, an SMTP channel has a master program that transmits messages and a slave program that receives messages. These are, respectively, the SMTP client and server.

The master channel program is typically responsible for outgoing connections where the MTA has initiated the operation. The master channel program:

• Runs in response to a local request for processing.

• Dequeues the message from the channel message queue.

• If the destination format is not the same format as the queued message, performs conversion of addresses, headers, and content, as necessary.

• Initiates network transport of the message.

The slave channel program typically accepts incoming connections where the MTA is responding to an external request. The slave channel program:

• Runs in response to an external event or upon local demand.

• Enqueues a message to a channel. The target channel is determined by passing envelope addresses through a rewrite rule.

For example, Figure 5-3 shows two channel programs, Channel 1 and Channel 2. The slave program in Channel 1 receives a message from a remote system. It looks at the address, applies rewrite rules as necessary, then based on the rewritten address enqueues the message to the appropriate channel message queue.

The master program dequeues the message from the queue and initiates network transport of the message. Note that the master program can only dequeue messages from its own channel queue.

Figure 5-3    Master and Slave Programs

Although a typical channel has both a master and a slave program, it is possible for a channel to contain only a slave program or a master program. For example, the ims-ms channel supplied with Messaging Server contains only a master program because this channel is responsible only for dequeuing messages to the local message store, as shown in Figure 5-4.

Figure 5-4    ims-ms Channel

Channel Message Queues

All channels have an associated message queue. When a message enters the messaging system, a slave program determines to which message queue the message is enqueued. The enqueued messages are stored in message files in the channel queue directories. By default, these directories are stored at the following location: /server_instance/imta/queue/channel/*.

Channel Definitions

Channel definitions appear in the lower half of the MTA configuration file, imta.cnf, following the rewrite rules (see "The MTA Configuration File"). The first blank line to appear in the file signifies the end of the rewrite rules section and the start of the channel definitions.

A channel definition contains the name of the channel followed by an optional list of keywords that define the configuration of the channel, and a unique channel tag, which is used in rewrite rules to route messages to the channel. Channel definitions are separated by single blank lines. Comments, but no blank lines, may appear inside a channel definition.

[blank line] ! sample channel definition Channel_Name keyword1 keyword2 Channel_Tag [blank line]

Collectively, the channel definitions are referred to as the channel host table. An individual channel definition is called a channel block. For example, in Figure 5-5 the channel host table contains three channel definitions or blocks.

Figure 5-5    Simple Configuration File - Channel Definitions

! test.cnf - An example configuration file. ! ! Rewrite Rules       .       .       . ! BEGIN CHANNEL DEFINITIONS ! FIRST CHANNEL BLOCK l local-host ! SECOND CHANNEL BLOCK a_channel defragment charset7 usascii a-daemon ! THIRD CHANNEL BLOCK b_channel noreverse notices 1 2 3 b-daemon

A typical channel entry will look something like this:

tcp_intranet smtp mx single_sys subdirs 20 noreverse maxjobs 7 SMTP_POOL maytlsserver allowswitchchannel saslswitchchannel tcpauth
tcpintranet-daemon

The first word, in this case tcp_intranet, is the channel name. The last word, in this case tcpintranet-daemon, is called the channel tag. The channel tag is the name used by rewrite rules to direct messages. The words in between the channel name and channel tag are called channel keywords and specify how the message is to be processed. Hundreds of different keywords allow messages to processed in many ways. A complete listing of channel keywords is listed and described in the iPlanet Messaging Server Reference Manual and Chapter 8 "Configuring Channel Definitions."

The channel host table defines the channels Messaging Server can use and the names of the systems associated with each channel.

On UNIX systems, the first channel block in the file always describes the local channel, l. (An exception is a defaults channel, which can appear before the local channel.) The local channel is used to make routing decisions and for sending mail sent by UNIX mail tools.

You can also set global options for channels in the MTA Option file, option.dat, or set options for a specific channel in a channel option file. For more information on the option files, see "Option File", and "TCP/IP (SMTP) Channel Option Files". For details on configuring channels, see Chapter 8 "Configuring Channel Definitions." For more information about creating MTA channels, see "The MTA Configuration File".

The MTA Directory Information

---

For each message that it processes, the MTA needs to access directory information about the users, groups, and domains that it supports. This information is stored in an LDAP directory service. The MTA has two methods of accessing this information. The first is by directly accessing the LDAP directory. This is called the direct LDAP mode and is fully described in Appendix B "MTA Direct LDAP Operation." The second and default method for accessing directory information is through the directory cache. This is called the dirsync mode.

In the dirsync mode, directory information about users and groups used by the MTA is accessed through a number of files and databases collectively called the directory cache. The data itself is stored in the LDAP directory, but actual information is accessed from the cache. Data in the cache is updated by the dirsync program which monitors changes to the LDAP directory and updates the files and databases accordingly.

Details on dirsync operation and configuration are described in "dirsync Configuration"

The Job Controller

---

Each time a message is enqueued to a channel, the Job Controller ensures that there is a job running to deliver the message. This might involve starting a new job process, adding a thread, or simply noting that a job is already running. If a job cannot be started because the job limit for the channel or pool has been reached, the Job Controller waits until another job has exited. When the job limit is no longer exceeded, the Job Controller starts another job.

Channel jobs run inside processing pools within the Job Controller. A pool can be thought of a "place" where the channel jobs are run. The pool provides a computing area where a set of jobs can operate without vying for resources with jobs outside of the pool. For more information on pools see "Job Controller File" and "Processing Pools for Channel Execution Jobs".)

Job limits for the channel are determined by the maxjobs channel keyword. Job limits for the pool are determined by the JOB_LIMIT option for the pool.

Messaging Server normally attempts to deliver all messages immediately. If a message cannot be delivered on the first attempt, however, the message is delayed for a period of time determined by the appropriate backoff keyword. As soon as the time specified in the backoff keyword has elapsed, the delayed message is available for delivery, and if necessary, a channel job is started to process the message.

The Job Controller's in-memory data structure of messages currently being processed and awaiting processing typically reflects the full set of message files stored on disk in the MTA queue area. However, if a backlog of message files on disk builds up enough to exceed the Job Controller's in-memory data structure size limit, then the Job Controller tracks in memory only a subset of the total number of messages files on disk. The Job Controller processes only those messages it is tracking in memory. After a sufficient number of messages have been delivered to free enough in-memory storage, the Job Controller automatically refreshes its in-memory store by scanning the MTA queue area to update its list of messages. The Job Controller then begins processing the additional message files it just retrieved from disk. The Job Controller performs these scans of the MTA queue area automatically.

If your site routinely experiences heavy message backlogs, you might want to tune the Job Controller by using the MAX_MESSAGES option. By increasing the MAX_MESSAGES option value to allow Job Controller to use more memory, you can reduce the number of occasions when message backlogs overflow the Job Controller's in-memory cache. This reduces the overhead involved when the Job Controller must scan the MTA queue directory. Keep in mind, however, that when the Job Controller does need to rebuild the in-memory cache, the process will take longer because the cache is larger. Note also that because the Job Controller must scan the MTA queue directory every time it is started or restarted, large message backlogs mean that starts or restarts of the Job Controller will incur more overhead than starts or restarts when no such backlog exists.

The Job Controller also runs a number of periodic jobs. These jobs are configured in the Job Controller configuration rather than using a more general facility such as cron so that the scheduling of these jobs is dependent on the Job Controller being up and running. This is an important point for high availability configurations where failover is a consideration.

For information about pools and configuring the Job Controller, see "Job Controller File" and "Configuring Message Processing and Delivery".

To Start and Stop the Job Controller

To start the Job Controller, execute the command:

imsimta start job_controller

To shut down the Job Controller, execute the command:

imsimta stop job_controller

To restart the Job Controller, execute the command:

imsimta restart job_controller

Restarting the Job Controller has the effect of shutting down the currently running Job Controller, then immediately starting a new one.