Chapter 16 LMTP Delivery

The Sun Java System Messaging Server MTA can use LMTP (Local Mail Transfer Protocol, defined in RFC 2033) for delivery to the message store in situations where a multi-tier messaging server deployment is used. In these scenarios, where you are using inbound relays and back end message stores, the relays become responsible for address expansion and delivery methods such as autoreply and forwarding and also for mailing list expansion. Delivery to the back end stores historically has been over SMTP which requires the back end system to look up the recipient addresses in the LDAP directory again, thereby engaging the full machinery of the MTA. For speed and efficiency, the MTA can use LMTP rather than SMTP to deliver messages to the back end store. The Sun Java System Messaging Server’s LMTP server is not intended as a general purpose LMTP server, but rather as a private protocol between the relays and the back end message stores. For simplicity of discussion, examples involving two-tier deployments are used.

By design, LMTP is intended for use in multi-tier deployments. It is not possible to use LMTP with single-system deployments. Also, the Messaging Server’s LMTP service as implemented is not designed to work with other LMTP servers or other LMTP clients.

This chapter consists of the following sections:

• 16.1 LMTP Delivery Features

• 16.2 Messaging Processing in a Two-Tier Deployment Without LMTP

• 16.3 Messaging Processing in a Two-Tier Deployment With LMTP

• 16.4 LMTP Overview

• 16.5 To Configure LMTP Delivery

• 16.6 LMTP Protocol as Implemented

16.1 LMTP Delivery Features

The MTA’s LMTP server is more efficient for delivering to the back end message store because it:

• Reduces the load on the back end stores.

  Because relays are horizontally scalable and back end stores are not, it is good practice to push as much processing to the relays as possible

• Reduces the load on the LDAP servers.

  The LDAP infrastructure is often a limiting factor in large messaging deployments.

• Reduces the number of message queues.

  Having queues on both the relay and the back end store makes finding a lost message that much harder for the administrator of a messaging deployment.

16.2 Messaging Processing in a Two-Tier Deployment Without LMTP

Figure 16–1 presents in pictorial form the following discussion of message processing in a two-tier deployment scenario without LMTP.

Figure 16–1 Two Tier Deployment Without LMTP

Without LMTP, in a two level deployment with relays in front of the store systems, the processing of an inbound message begins with a connection on the SMTP port picked up by the dispatcher on the relay machine and handed off to a tcp_smtp_server process. This process does a number of things with the inbound message including:

• Looking up the user in the directory

• Determining if the user is within a domain hosted by this email deployment

• Determining if the user is a valid user in the domain

• Rewriting the envelope address as @mailhost:user@domain

• Enqueuing the message for delivery to the mailhost

The smtp_client process then picks up the mail message from the queue and sends it to the mailhost. On the mailhost, some very similar processing takes place. A connection on the SMTP port is picked up by the dispatcher and handed off to a tcp_smtp_server process. This process does a number of things to the message, including:

• Looking up the user in the directory

• Determining if the user is within a domain hosted by this email deployment

• Determining if the user is a valid user in the domain

• Rewriting the envelope address to direct the message to the ims_ms channel

• Enqueuing the message for delivery to the store

Then the ims_ms process picks up the mail message and attempts to deliver it to the store. In this scenario, the enqueuing processing is performed twice, and the MTAs each perform an LDAP lookup.

16.3 Messaging Processing in a Two-Tier Deployment With LMTP

16.3 Messaging Processing in a Two-Tier Deployment With LMTP presents in pictorial form the following discussion of message processing in a two-tier deployment scenario with LMTP.

Figure 16–2 Two Tier Deployment With LMTP

With LMTP in place, a connection on the SMTP port of the relay machine is picked up by the dispatcher and handed off to a tcp_smtp_server process. This process does a number of things with the inbound message including:

• Looking up the user in the directory

• Determining if the user is within a domain hosted by this email deployment

• Determining if the user is a valid user in the domain

• Determining which back end message store machine hosts the mailbox for the user

• Enqueuing the message for delivery to the mailhost

On the store machine, a connection to the LMTP port is received by the dispatcher and handed off to the lmtp_server process. The LMTP server then inserts the message into the user's mailbox or into the UNIX native mailbox. If message delivery is successful, the message is dequeued on the relay machine. If unsuccessful, the message remains on the relay machine. Note that the LMTP process on the message store does not engage any MTA machinery for processing addresses or messages.

16.4 LMTP Overview

For the most part, the MTA itself can be basically absent from the back end server. The only necessary MTA components are:

• The dispatcher

• libimta

• The LMTP server

• The imta.cnf file

• The mappings file

• The imta.tailor file

While the dispatcher requires MTA configuration files, these files can be very short. The dispatcher must run on the back end server so that it can start the LMTP servers which run under it. Because the dispatcher and the LMTP server use various functions of libimta, this needs to be present on the back end server as well.

The LMTP server does not perform any of the usual MTA enqueuing or dequeuing functions, header processing, or address translations. The relay system performs all the manipulation of the content of the messages and addresses which then presents to the LMTP server the message in exactly the form to be delivered to the message store and with the delivery address already in the form required by the store. Additional recipient information that is usually available as a message is delivered to the store, such as the user’s quota, is presented along with the recipient address as LMTP parameters. Should a delivery attempt fail, the message is left enqueued in the LMTP queue on the relay system.

16.5 To Configure LMTP Delivery

Configuring the LMTP delivery mechanism requires configuration on both the relay machines and on the back end stores. On the relays, the DELIVERY_OPTIONS MTA option (in option.dat) has to be changed so that messages being delivered to the stores are passed to the LMTP channel. The back end store must be configured with the dispatcher, but does not need the job controller. The dispatcher must be configured to run the LMTP server.

In a typical multi-tier deployment, users are provisioned on different backend message store machines. One or more of these backend machines may not have LMTP turned on and therefore the front-end relays need to be aware of which store machines are LMTP aware. This is achieved by using the General Database facility to explicitly name those message stores which are configured to accept LMTP delivery.

To Configure the Inbound MTA Relays with LMTP

To configure inbound MTA relays to use LMTP, do the following:

1. Modify your imta.cnf file and change the LMTP rewrite rules to read:

   ! lmtp .lmtp $E$F$U%$H.lmtp@lmtpcs-daemon .lmtp $B$F$U%$H@$H@lmtpcs-daemon !

2. Set the mailbox DELIVERY_OPTIONS to:

   #*mailbox=@$X.LMTP:$M%$\$2I$_+$2S@lmtpcs-daemon

3. Add the channel keywords multigate connectcanonical to each of the tcp_lmtp* channel blocks.

4. Add the following channel keywords to the tcp_lmtpcs channel:

   fileinto @$4O:$U+$S@$D

   Note that the 'O' in the keyword above is a capital letter O, not a zero.

5. The incoming MTA relay configuration settings should look like this:

   The option.dat entry for DELIVERY_OPTIONS should look like this:

   !------------------------------------------ ! Modified DELIVERY_OPTIONS to activate LMTP ! delivery from a frontend to the backend store !-------------------------------------------- ! DELIVERY_OPTIONS=\ #*mailbox=@$X.LMTP:$M%$\$2I$_+$2S@lmtpcs-daemon,\ #&members=*,\ #&@members_offline=*,\ #/hold=@hold-daemon:$A,\ #program=$M%$P@pipe-daemon,\ #forward=**,\ #*^!autoreply=$M+$D@bitbucket !

   After your changes the modified imta.cnf rewrite rules should look like this:

   ! lmtp .lmtp $E$F$U%$H.lmtp@lmtpcs-daemon .lmtp $B$F$U%$H@$H@lmtpcs-daemon !

   The changed channel blocks should look like this:

   ! ! tcp_lmtpcs (LMTP client - store) tcp_lmtpcs defragment lmtp multigate connectcanonical \ fileinto @$4O:$U+$S@$D port 225 nodns single_sys \ subdirs 20 maxjobs 7 pool SMTP_POOL dequeue_removeroute lmtpcs-daemon

16.5.1 To Configure Back End Stores with LMTP and a Minimal MTA

The back end stores require only a minimal MTA if they are receiving messages over LMTP. They require a dispatcher, a job controller and a simple MTA configuration. In particular they need a dispatcher.cnf, job_controller.cnf and a mappings file which comprise the only significant part of the MTA configuration.

The dispatcher.cnf file must contain the following:

! VERSION=1.1
! IMTA default dispatcher configuration file
!
! Global defaults
!
MIN_PROCS=1
MAX_PROCS=10
MIN_CONNS=30
MAX_CONNS=50
MAX_SHUTDOWN=2
MAX_LIFE_TIME=86400
MAX_LIFE_CONNS=10000
MAX_IDLE_TIME=600
HISTORICAL_TIME=0
!
! rfc 2033 LMTP server - store
!
[SERVICE=LMTPSS]
PORT=225
IMAGE=IMTA_BIN:tcp_lmtp_server
LOGFILE=IMTA_LOG:tcp_lmtpss_server.log
PARAMETER=CHANNEL=tcp_lmtpss
STACKSIZE=2048000
! Uncomment the following line and set INTERFACE_ADDRESS to an
! appropriate host IP (dotted quad) if the dispatcher needs to
! listen on a specific interface (e.g. in a HA environment).
! INTERFACE_ADDRESS=!
! rfc 2033 LMTP server - native
!

Note that by default, the LMTP services in the dispatcher.cnf file are commented out. You must uncomment them to get LMTP to work.

The normal dispatcher options of MAX_CONNS, MAX_PROCS, MAX_LIFE_CONNS, and MAX_LIFE_TIME can also be set, but need to be set appropriately for your hardware.

The PORT_ACCESS mapping is important. The LMTP implementation for the back end servers is intended as a private protocol between Sun Java System Messaging Server relays and back end stores. You must use the PORT_ACCESS mapping to make sure that only such relays can connect to these services. Your mapping file should look like this:

PORT_ACCESS

  TCP|*|225|192.18.74.206|* $Y
  TCP|*|226|192.18.74.206|* $Y
  TCP|*|225|192.18.74.129|* $Y
  TCP|*|226|192.18.74.129|* $Y
  TCP|*|*|*|*   $N500$ Do$ not$ connect$ to$ this$ machine

The IP address above are LMTP server and client IP address. You should replace the sample IP addresses specified in the PORT_ACCESS mapping table here with the IP addresses of your relays on the network that connect to the back end stores.

There has to be an imta.cnf file, but it is there merely to make the configuration complete. A minimal imta.cnf file consists of the following channel definitions:

!
! IMTA configuration file
!
! tcp_lmtpss (LMTP server - store)
tcp_lmtpss lmtp flagtransfer
tcp_lmtpss-daemon

Note that by default, the LMTP channel definitions are commented out. You must uncomment them if you want LMTP to work.

You can use the default job_controller.cnf file created on installation. No modification of this file is required.

16.5.2 Configuring Relays for Sending Messages Via LMTP to Back End Systems with Message Stores and Full MTAs

There are situations where you might want the back end stores to have the full capabilities of the MTA but still to have the load savings of using LMTP. For example, you might want program delivery on the back end store. In this case the relays should be configured as described above in To Configure the Inbound MTA Relays with LMTP

16.5.3 Configuring LMTP on Back End Message Store Systems Having Full MTAs

The only changes from the configuration of a back end store messaging system to one with LMTP direct delivery to the store are that the following lines need to be added to the end of the dispatcher.cnf file:

! rfc 2033 LMTP server - store
[SERVICE=LMTPSS]
PORT=225
IMAGE=IMTA_BIN:tcp_lmtp_server
LOGFILE=IMTA_LOG:tcp_lmtpss_server.log
PARAMETER=CHANNEL=tcp_lmtpss
STACKSIZE=2048000
! Uncomment the following line and set INTERFACE_ADDRESS to an 
! appropriate host IP (dotted quad) if the dispatcher needs to 
! listen on a specific interface (e.g. in a HA environment).
!INTERFACE_ADDRESS=

         

Note that by default, the LMTP services in the dispatcher.cnf file are commented out. You must uncomment them to get LMTP to work. Also, the LMTP port numbers are just examples, and can be anything you choose.

This is the same as the whole dispatcher.cnf file described above for when the back end store is configured only for LMTP. The mappings file also requires the PORT_ACCESS mappings as described for LMTP only back end stores.

16.5.4 Handling 4.2.1 Mailbox Busy Error in Response to LMTP Message Data

If the LMTP channel option MAILBOX_BUSY_FAST_RETRY is set to 1 (the default) a 4.2.1 Mailbox busy error in response to LMTP message data is handled by retrying the message after a random but short interval; normal message backoff values do not apply. Setting the option to 0 disables this behavior.

16.6 LMTP Protocol as Implemented

This section provides a sample LMTP dialogue with an explanation of what is seen in that dialogue. The LMTP client on the relay uses standard LMTP protocol to talk to the LMTP server on the back end store. However the protocol is used in specific ways. For example:

---> LHLO
<--- 250 OK

No action is taken on the LHLO message. The reply is always 250 OK.

---> MAIL FROM: address size=messageSizeInBytes
<--- 250 OK

No checks or conversions are made on the originator address. The size= parameter gives a size in bytes for the message that is to be delivered. This is the size of the message exactly as it appears in the protocol. It is not necessarily the exact size of the message, but the actual message size will not exceed this size. The LMTP server allocates a memory buffer of this size to receive the message.

---> RCPT TO: uid+folder@domain xquota=size,number xdflg=xxx
<--- 250 OK

No checks are made on the recipient addresses at the time they are received, but a list of recipients is built for later use. Note that the @domain part of the address is omitted for uids in the primary domain, and that the +folder part is optional. This is the same address format used by the message store channel in the MTA.

The xquota= parameter gives the user’s message quotas which consist of the maximum total size and the maximum number of messages. The MTA provides this information which it retrieves while performing an LDAP lookup on the user to do the address translation. This information is used to keep the quota information in the message store synchronized with the directory. Getting the quota information does not result in an additional performance hit.

The xdflg= parameter specifies a number which is interpreted as a bit field. These bits control how the message is delivered. For example, the bit whose value is 2, if set, guarantees delivery of the message even if the user is over quota. (Note that xdflg is an internal parameter and the bits in it are subject to change or addition without notice. We do not support other clients using this extension with our server, nor do we support using our client with some other server and this parameter.)

This interaction may be repeated many times, once for each recipient.

--->DATA
---> <the message text>
--->.

The LMTP client then sends the entire message, dot-stuffed, just as SMTP does. The message finishes with a dot (.) alone on a line. If the message size is exceeded the LMTP server sends:

<--- 500 message too big

and ends the connection.

Assuming that the message is received correctly, the LMTP server then sends back to the LMTP client the status for each recipient given in the RCPT TO: lines. For instance, if the message is delivered successfully, the response is:

<--- 250 2.5.0 address OK

where address is exactly as it appeared on the RCPT TO: line.

The conversation can either repeat with another MAIL FROM: line or end with the following interaction:

---> quit
<--- 221 OK

Table 16–1 shows the possible status codes for each recipient. This three-column table shows the short code in the first column, its long-code equivalent in the second column and the status text in the third column. 2.x.x status codes are success codes, 4.x.x codes are retryable errors, 5.x.x codes are non retryable errors.

Otherwise, there are changes to the delivery options for mailbox, native (and, therefore, UNIX), and file. The object of these rules is to generate addresses that will cause the messages to be sent through the appropriate LMTP channel to the back end servers. The addresses generated are source routed addresses of the form:

@sourceroute:localpart@domain