11 LMTP Delivery

This chapter provides an overview of the Local Mail Transfer Protocol (LMTP), and describes how to configure both the LMTP client and server.

Overview of LMTP

The Oracle Communications Messaging Server MTA can use Local Mail Transfer Protocol (LMTP), as defined in RFC 2033, to deliver messages to the message store in a multi-tiered Messaging Server deployment. In this scenario, the front-end relays become responsible for address expansion and delivery methods such as autoreply and forwarding, and also for mailing list expansion. Delivery to the back-end stores historically has been over SMTP, which requires the back-end system to look up the recipient addresses in the LDAP directory again, thereby engaging the full machinery of the MTA. For speed and efficiency, the MTA can use LMTP rather than SMTP to deliver messages to the back-end store. The Messaging Server's LMTP server is not intended as a general purpose LMTP server, but rather as a private protocol between the relays and the back-end message stores. For simplicity of discussion, examples involving two-tiered deployments are used.

Note:

LMTP is recommended for use in multi-tiered deployments. Also, the Messaging Server's LMTP service as implemented is not designed to work with third-party LMTP servers or third-party LMTP clients.

LMTP Delivery Features

The MTA's LMTP server is more efficient for delivering to the back-end message store because it:

  • Reduces the load on the back-end stores. Because relays are horizontally scalable and back-end stores are not, it is good practice to push as much processing to the relays as possible.

  • Reduces the load on the LDAP servers. The LDAP infrastructure is often a limiting factor in large messaging deployments.

  • Reduces the number of message queues. Having queues on both the relay and the back-end store makes finding a lost message that much harder for administering a messaging deployment.

Support for LMTP Client and Server to Detect and Respond to Certain Conditions

Support has been added to the LMTP client and server to detect and respond to the condition where a given host's LMTP server is responding but isn't associated with the master store replica. When this happens the LMTP server produces a banner or MAIL FROM response of the form:

423 4.3.2 Host not master for store; correct master host is <HOSTNAME>

or, if the correct master host is not known:

421 4.3.2 Host not master for store; cannot determine correct master host

When the LMTP client sees the 423 banner it will immediately disconnect and reconnect to the correct host. The affinity subsystem is also informed of the new master host.

The 421 banner is treated like any other 4YZ banner response; the client disconnects and tries the next host in the affinity group.

A 423 MAIL FROM response now engages the LMTP client's fast retry logic; the client will abort the delivery attempt and disconnect but schedule the message for fast retry.

The handling of a 421 MAIL FROM response is unchanged.

Two new LMTP channel-specific parameters have been added to support this new capability:

  • WRONG_MASTER_FAST_RETRY - Controls the actual value used to override the normal backoff value. The default is 1, meaning to ask the Job Controller for a fairly quick retry. Setting the option to 0 disables backoff override in this case. Positive values greater than 1 result in a retry, but not quite as quick of one. The formula used is n + rand() % (15 * n) seconds, where n is the WRONG_MASTER_FAST_RETRY value.

  • WRONG_MASTER_FALLBACK_ATTEMPTS - Controls the number of times the LMTP client will honor a 423 banner redirect. The default is 1, meaning a single redirect will be honored for a given message.

Messaging Processing in a Two-Tiered Deployment Without LMTP

Figure 11-1 shows message processing in a two-tiered deployment without LMTP.

Figure 11-1 Two-Tiered Deployment Without LMTP

Description of Figure 11-1 follows
Description of ''Figure 11-1 Two-Tiered Deployment Without LMTP''

Without LMTP, in a two-tiered deployment with relays "in front" of the store systems, inbound message processing begins with a connection on the SMTP port picked up by the dispatcher on the relay machine and handed off to a tcp_smtp_server process. This process does a number of things with the inbound message including:

  • Looking up the user in the directory

  • Determining if the user is within a domain hosted by this email deployment

  • Determining if the user is a valid user in the domain

  • Rewriting the envelope address as @mailhost:user@domain

  • Enqueuing the message for delivery to the mailhost

The smtp_client process then picks up the mail message from the queue and sends it to the mailhost. On the mailhost, some very similar processing takes place. A connection on the SMTP port is picked up by the dispatcher and handed off to a tcp_smtp_server process. This process does a number of things to the message, including:

  • Looking up the user in the directory

  • Determining if the user is within a domain hosted by this email deployment

  • Determining if the user is a valid user in the domain

  • Rewriting the envelope address to direct the message to the ims_ms channel

  • Enqueuing the message for delivery to the store

Then the ims_ms process picks up the mail message and attempts to deliver it to the store. In this scenario, the enqueuing processing is performed twice, and the MTAs each perform an LDAP lookup.

Messaging Processing in a Two-Tiered Deployment With LMTP

Figure 11-2 shows message processing in a two-tiered deployment scenario with LMTP.

Figure 11-2 Two-Tiered Deployment With LMTP

Description of Figure 11-2 follows
Description of ''Figure 11-2 Two-Tiered Deployment With LMTP''

With LMTP in place, a connection on the SMTP port of the relay machine is picked up by the dispatcher and handed off to a tcp_smtp_server process. This process does a number of things with the inbound message including:

  • Looking up the user in the directory

  • Determining if the user is within a domain hosted by this email deployment

  • Determining if the user is a valid user in the domain

  • Determining which back end message store machine hosts the mailbox for the user

  • Enqueuing the message for delivery to the mailhost

On the store machine, a connection to the LMTP port is received by the dispatcher and handed off to the lmtp_server process. The LMTP server then inserts the message into the user's mailbox or into the UNIX native mailbox. If message delivery is successful, the message is dequeued on the relay machine. If unsuccessful, the message remains on the relay machine. Note that the LMTP process on the message store does not engage any MTA machinery for processing addresses or messages.

LMTP Architecture

An LMTP configuration consists of setting up LMTP channels, one for the LMTP client and one for the LMTP server. LMTP channels are special cases of TCP/IP channels. The general SMTP-over-TCP/IP channel is often configured for bidirectional use. An LMTP channel, on the other hand, is configured to be dedicated for either client or server use. You configure the LMTP client channel on the MTA front-end relay host and the LMTP server channel on the back-end message store. Running commpkg install on both the front-end relay and back-end store system will configure LMTP properly.

For the most part, the MTA itself can be basically absent from the back-end server. The following items are the only necessary MTA components on the back end:

  • The dispatcher

  • The LMTP server

  • A simple MTA configuration including mappings

The dispatcher must run on the back-end server so that it can start the LMTP servers that run under it. Because the dispatcher and the LMTP server use various functions of libimta, this needs to be present on the back-end server as well.

The LMTP server does not perform any of the usual MTA enqueuing or dequeuing functions, header processing, or address translations. The front-end relay system performs all the manipulation of the content of the messages and addresses, which then presents to the LMTP server the message in exactly the form to be delivered to the message store and with the delivery address already in the form required by the store. Additional recipient information that is usually available as a message that is delivered to the store, such as the user's quota, is presented along with the recipient address as LMTP options. Should a delivery attempt fail, the message is left enqueued in the LMTP queue on the relay system.

Configuring LMTP

Setting up LMTP requires that you configure both the front-end relay hosts and back-end message store hosts. On the relays, you must change the mta.delivery_options option so that messages being delivered to the stores are passed to the LMTP channel. The back-end store must be configured with the dispatcher, but does not need the job controller. On the LMTP client, you must configure the job controller to run the LMTP client.

This section describes how to configure the LMTP front ends and back ends by using Unified Configuration recipes. You could also perform the same configuration manually by using the msconfig command, however, using recipes is faster and makes the task repeatable. You run one recipe on the LMTP front ends and one on the LMTP back ends. For more information on Unified Configuration recipes, see the discussion on recipe language in the Messaging Server Reference and "Using Recipes."

Topics in this section:

Before You Begin

  • To see if any LMTP options are already enabled on your Messaging Server hosts, run the following command:

    msconfig show mta.delivery_options -default

To Configure the Front-end MTA Relay with LMTP

Use the following "Example Recipe to Configure LMTP Front-end Relay with LMTP" to configure the inbound MTA relay for LMTP.

  1. Make a copy of the example recipe file and save it as recipe.rcp in the config/recipes directory.

  2. In this example, replace the following items with your site-specific values:

    • myIP

    • myNetmaskbits

  3. To run the recipe, type the following command:

    cd MessagingServer_home/bin
msconfig run MessagingServer_home/config/recipes/<recipe_name>

  4. Recompile if running a compiled configuration

    imsimta cnbuild

  5. Restart Messaging Server.

    cd MessagingServer_home/bin
stop-msg
start-msg

Example Recipe to Configure LMTP Front-end Relay with LMTP

# -*- mode: sieve; -*-
description("frontendMTA to backend LMTP store");
keywords(["frontend",,"MTA", "LMTP"]);
#
# Sample recipe for a frontend MMP/MTA to a backend LMTP store
#   the corresponding recipe for the backend store/LMTP is backendLMTP.rcp
#
###########################################################################
# !!!!!!!!!!!!!!!!!!!!!
# CHANGE THESE
# !!!!!!!!!!!!!!!!!!!!!
#
# constants - supplied input
#
# my IP address
myIP = "10.133.158.10";
# network portion of IP address in number of bits
myNetmaskbits = "8";

###########################################################################
#
# configure LMTP frontend
#

#
# disable store
#
set_option("store.enable", "0");
set_option("imap.enable", "0");
set_option("pop.enable", "0");

#
# add to rewrite rules
# .lmtp $E$F$U%$H.lmtp@lmtpcs-daemon
# .lmtp $B$F$U%$H@$H@lmtpcs-daemon
#
# should really check to see if the rewrite rule exists instead
# of unilaterally appending it
#
append_rewrites([".lmtp", "$E$F$U%$H.lmtp@lmtpcs-daemon"]);
append_rewrites([".lmtp", "$B$F$U%$H@$H@lmtpcs-daemon"]);

#
# add channel tcp_lmtpcs
# ,----
# | tcp_lmtpcs defragment lmtp multigate connectcanonical fileinto @$4O:$U+$S@$D f\
# | lagtransfer multigate connectcanonical port 225 nomx sin\
# | gle_sys pool SMTP_POOL dequeue_removeroute
# | lmtpcs-daemon
# `----
#
set_option("channel:tcp_lmtpcs.connectcanonical");
set_option("channel:tcp_lmtpcs.defragment");
set_option("channel:tcp_lmtpcs.fileinto", "@$4O:$U+$S@$D");
set_option("channel:tcp_lmtpcs.flagtransfer");
set_option("channel:tcp_lmtpcs.lmtp");
set_option("channel:tcp_lmtpcs.multigate");
set_option("channel:tcp_lmtpcs.nomx");
set_option("channel:tcp_lmtpcs.pool", "SMTP_POOL");
set_option("channel:tcp_lmtpcs.port", "225");
set_option("channel:tcp_lmtpcs.single_sys");
set_option("channel:tcp_lmtpcs.dequeueremoveroute");
set_option("channel:tcp_lmtpcs.official_host_name", "lmtpcs-daemon");

#
# ! The modified DELIVERY_OPTIONS which activate LMTP
# ! delivery from a frontend relay to the backend
#
# DELIVERY_OPTIONS=\
#       #*mailbox=@$X.LMTP:$M%$\$2I$_+$2S@lmtpcs-daemon,\
#       #&members=*,\
#       #*native=@$X.LMTPN:$M+$2S@native-daemon,\
#       #*unix=@$X.LMTPN:$M,\
#       #*file=@$X.LMTPN:+$F,\
#       #&@members_offline=*,\
#       #/hold=@hold-daemon:$A,\
#       #program=$M%$P@pipe-daemon,\
#       #forward=**,\
#       #*^!autoreply=$M+$D@bitbucket
#
# NOTE NOTE NOTE - have to escape the backslash in the "mailbox=..."
set_option("mta.delivery_options", "#*mailbox=@$X.LMTP:$M%$\\$2I$_+$2S@lmtpcs-daemon,#&members=*,#*native=@$X.LMTPN:$M+$2S@native-daemon,#*unix=@$X.LMTPN:$M,#*file=@$X.LMTPN:+$F,#&@members_offline=*,#/hold=@hold-daemon:$A,#program=$M%$P@pipe-daemon,#forward=**,#*^!autoreply=$M+$D@bitbucket");

###########################################################################
write("use \"write -remark frontendLMTP.rcp\" to write out the changes\n");

This recipe performs the following configuration on the LMTP front end:

  1. Disables the message store, as it is not needed.

  2. Enables the necessary rewrite rules. The recipe language append_rewrites function adds entries to existing rewrite rules, or, if none exists, adds them as new rules.

  3. Adds the tcp_lmtpcs channel with the appropriate options:

    • connectcanonical: Tells the MTA to compare the recipient envelope address domain with the channel host proper names, and if the domain name matches one of the channel's host proper names, then connect to the host name corresponding to that host proper name.

    • defragment: Any message/partial messages queued to the channel are placed in the defragmentation channel queue instead. Once all the parts have arrived, the message is rebuilt and sent on its way.

    • fileinto, value of @$4O:$U+$S@$D: Specifies how to alter an address when a Sieve filter "fileinto" action is applied. In $4O, the O is the capital or majuscule letter "o", not the numeral zero 0. The effect is that the explicit source route to the mailhost should be preserved if present, and the foldername should be inserted as a subaddress into the original address, replacing any originally present subaddress.

    • flagtransfer: Enables SMTP client support of the XDFLG private SMTP extension command.

    • lmtp: Specifies that channel supports LMTP protocol.

    • multigate: Instructs the MTA to route the message to the daemon mailbox specified by the daemon channel option on the system specified in the message's To: address.

    • nomx: Disables MX lookups.

    • pool, value of SMTP_POOL: Specifies the SMTP_POOL pool where the jobs are created for this channel.

    • port, value of 225: Specifies dispatcher port number.

    • single_sys: Creates a single copy of the message for each destination system (more precisely, each destination domain name) associated with a recipient address.

    • dequeueremoveroute: Causes source routes to be stripped from envelope recipient addresses when the channel dequeues messages (but after the channel has determined how to route the message).

    • official_host_name, value of lmtpcs-daemon: Specifies the "name" of the system with which this channel communicates. (In legacy configuration, the official host name is specified as the first name on the second line of a channel definition.)

  4. Finally, the delivery options are set, which activates LMTP delivery from the front-end relay to the back-end message store. In the *mailbox=@$X.LMTP:$M%$\$2I$_+$2S@lmtpcs-daemon, portion of the delivery options, the script "escapes" the forward slash, so that the actual entry in the recipe is as follows:

    ..."#*mailbox=@$X.LMTP:$M%$\\$2I$_+$2S@lmtpcs-daemon,#&members=*,...

    Note:

    You need to add the full delivery_options as shown in the example. You cannot override just the default for one option. If you specify delivery_options at all, you must define them all.

To Configure Back-End Stores with LMTP and a Minimal MTA

Use the following "Example Recipe to Configure LMTP Back-end Store with LMTP" to configure the back-end store for LMTP. This recipe:

  • Prompts for myIP and feIPs, if you do not specify them in the recipe (but does not validate the IP addresses)

  • Checks if the tcp_lmtpss channel exists before creating it

  • Checks if the dispatcher group exists before creating it

  • Checks if PORT_ACCESS entries exist before creating them

  • Treats feIPs as a list to allow multiple front ends

  • Can be run multiple times correctly

To create and run the recipe, follow these steps:

  1. Make a copy of the example recipe file and save it as recipe.rcp in the config/recipes directory.

  2. In this example, replace the following items with your site-specific values:

    • myIP

    • feIPs

  3. To run the recipe, type the following command:

    cd MessagingServer_home/bin
msconfig run MessagingServer_home/config/recipes/<recipe_name>

  4. Recompile if running a compiled configuration.

    imsimta cnbuild

  5. Restart Messaging Server.

    cd MessagingServer_home/bin
stop-msg
start-msg

Example Recipe to Configure LMTP Back-end Store with LMTP

# -*- mode: sieve; -*-
description("backend store via LMTP");
keywords(["backend", "store", "LMTP"]);
#
# Sample recipe for a backend store via LMTP
#   the corresponding recipe for the frontend MMP/MTA is frontendLMTP.rcp
###########################################################################
# !!!!!!!!!!!!!!!!!!!!!
# CHANGE THESE
# !!!!!!!!!!!!!!!!!!!!!
#
# constants - supplied input
# if you leave it blank, the script will prompt for it
#  sample entry
#myIP = "10.133.152.193";
myIP = "";

#
# list of frontend machines that access this store via LMTP
# if you leave it blank, the script will prompt for it
#   sample entry:
#feIPs = ["10.133.152.192", "10.133.152.193"];
feIPs = [];

###########################################################################
# prompt for myIP and feIP if needed


if (length(myIP) <= 0) {
  myIP = read("Enter the IP address of this host: ");
}

if (length(feIPs) <= 0) {
  loop {
    ip = read("Enter the IP address of a frontend machine (<RET> if no more): ");
    exitif (ip == "");
    push(feIPs, ip);
  }
}

###########################################################################
#
# configure LMTP backend

#
# create tcp_lmtpss channel. In legacy config, this would show up as:
#   tcp_lmtpss lmtp flagtransfer identnonenumeric
#   tcp_lmtpss-daemon
if exists_channel("tcp_lmtpss") {
  warn("-- WARNING: tcp_lmtpss channel already exists.");
} else {
  # This creates the channel by using individual options
  #set_option("channel:tcp_lmtpss.flagtransfer");
  #set_option("channel:tcp_lmtpss.identnonenumeric");
  #set_option("channel:tcp_lmtpss.lmtp");
  #set_option("channel:tcp_lmtpss.official_host_name", "tcp_lmtpss-daemon");
  # an alternative way of doing it as a one-liner
  print("-- INFO: Adding tcp_lmtpss channel\n");
  add_channel("tcp_lmtpss",
              ["flagtransfer", "",
               "identnonenumeric", "",
               "lmtp", "",
               "official_host_name", "tcp_lmtpss-daemon"]);
}

#
# dispatcher.cnf
#  uncomment [SERVICE=LMTPSS] block
#
if exists_group("dispatcher.service:LMTPSS") {
  warn("-- WARNING: dispatcher.service:LMTPSS group already exists");
} else {
  print("-- INFO: Creating dispatcher.service:LMTPSS group\n");
  # This creates the group by using individual options
  #set_option("dispatcher.service:LMTPSS.image", "IMTA_BIN:tcp_lmtp_server");
  #set_option("dispatcher.service:LMTPSS.logfilename", "IMTA_LOG:tcp_lmtpss_server.log");
  #set_option("dispatcher.service:LMTPSS.option", "CHANNEL=tcp_lmtpss");
  #set_option("dispatcher.service:LMTPSS.tcp_ports", "225");
  #set_option("dispatcher.service:LMTPSS.stacksize", "2048000");
  #set_option("dispatcher.service:LMTPSS.enable", "1");

  # alternate way of doing this in one line
  add_group("dispatcher.service:LMTPSS",
            ["image", "IMTA_BIN:tcp_lmtp_server",
             "logfilename", "IMTA_LOG:tcp_lmtpss_server.log",
             "option", "CHANNEL=tcp_lmtpss",
             "tcp_ports", "225",
             "stacksize", "2048000",
             "enable", "1"]);
}

#
# add PORT_ACCESS mapping entries
#
#   allow frontends (feIPs) to access LMTP port
# TCP|*|225|10.133.152.192|*  $Y
# TCP|*|226|10.133.152.192|*  $Y
#
# ! Allow 'msprobe' on this host (myIP) to connnect to the LMTP ports
# !
# TCP|*|225|10.133.152.193|*  $Y
# TCP|*|226|10.133.152.193|*  $Y

portAccess_optlist =  get_mapping("PORT_ACCESS");
#print ("\n -- DEBUG optlist for PORT_ACCESS" . portAccess_optlist . "\n");

# list of IP addresses
ipaddrs = [feIPs];
push(ipaddrs, myIP);
numips = length(ipaddrs);
ports = ["225", "226"];
numports = length(ports);

i = 1;
loop {                          #loop over all ipaddrs
  p = numports;
  loop {
    tmp = "TCP|*|" + ports[p] + "|" + ipaddrs[i] + "|*";
    if exists_optlist(get_mapping("PORT_ACCESS"), tmp) {
      warn ( "-- WARNING: optlist for " . tmp . " exists in PORT_ACCESS");
    } else {
      print("-- INFO: Adding IP " + ipaddrs[i] + " port " + ports[p] + " to PORT_ACCESS mapping\n");
      prepend_mapping("PORT_ACCESS", [tmp, "$Y"]);
    }
    exitif(p==1);
    p--;
  }
  exitif(i==numips);
  i++;
}

###########################################################################
write("use \"write -remark backendLMTP.rcp\" to write out the changes\n");

LMTP Protocol as Implemented

This section provides a sample LMTP dialogue with an explanation of what is seen in that dialogue. The LMTP client on the relay uses standard LMTP protocol to talk to the LMTP server on the back end store. However the protocol is used in specific ways. For example:

---> LHLO
<--- 250 OK

No action is taken on the LHLO message. The reply is always 250 OK.

---> MAIL FROM: address size=messageSizeInBytes
<--- 250 OK

No checks or conversions are made on the originator address. The size= option gives a size in bytes for the message that is to be delivered. This is the size of the message exactly as it appears in the protocol. It is not necessarily the exact size of the message, but the actual message size will not exceed this size. The LMTP server allocates a memory buffer of this size to receive the message.

---> RCPT TO: uid+folder@domain xquota=size,number xdflg=xxx
<--- 250 OK

No checks are made on the recipient addresses at the time they are received, but a list of recipients is built for later use. Note that the @domain part of the address is omitted for uids in the primary domain, and that the +folder part is optional. This is the same address format used by the message store channel in the MTA.

The xquota= option gives the user's message quotas which consist of the maximum total size and the maximum number of messages. The MTA provides this information which it retrieves while performing an LDAP lookup on the user to do the address translation. This information is used to keep the quota information in the message store synchronized with the directory. Getting the quota information does not result in an additional performance hit.

The xdflg= option specifies a number which is interpreted as a bit field. These bits control how the message is delivered. For example, the bit whose value is 2, if set, guarantees delivery of the message even if the user is over quota. (Note that xdflg is an internal option and the bits in it are subject to change or addition without notice. Oracle does not support other clients using this extension with Messaging Server, nor does Oracle support using the Messaging Server LMTP client with some other server and this option.)

This interaction may be repeated many times, once for each recipient.

--->DATA
---> <the message text>
--->.

The LMTP client then sends the entire message, dot-stuffed, just as SMTP does. The message finishes with a dot (.) alone on a line. If the message size is exceeded the LMTP server sends:

<--- 500 message too big

and ends the connection.

Assuming that the message is received correctly, the LMTP server then sends back to the LMTP client the status for each recipient given in the RCPT TO: lines. For instance, if the message is delivered successfully, the response is:

<--- 250 2.5.0 address OK

where address is exactly as it appeared on the RCPT TO: line.

The conversation can either repeat with another MAIL FROM: line or end with the following interaction:

---> quit
<--- 221 OK

Table 11-1 shows the possible status codes for each recipient. This three-column table shows the short code in the first column, its long-code equivalent in the second column and the status text in the third column. 2.x.x status codes are success codes, 4.x.x codes are retryable errors, and 5.x.x codes are non-retryable errors.

Table 11-1 LMTP Status Codes for Recipients

Short Code Long Code Status Text

250

2.5.0

OK

420

4.2.0

Mailbox Locked

422

4.2.2

Quota Exceeded

420

4.2.0

Mailbox Bad Formats

420

4.2.0

Mailbox not supported

430

4.3.0

IMAP IOERROR

522

5.2.2

Persistent Quota Exceeded

523

5.2.3

Message too large

511

5.1.1

mailbox nonexistent

560

5.6.0

message contains null

560

5.6.0

message contains nl

560

5.6.0

message has bad header

560

5.6.0

message has no blank line

Otherwise, there are changes to the delivery options for mailbox, native (and, therefore, UNIX), and file. The object of these rules is to generate addresses that will cause the messages to be sent through the appropriate LMTP channel to the back end servers. The addresses generated are source routed addresses of the form:

@sourceroute:_localpart_@_domain_