12.8.2 Automatic Defragmentation of Message/Partial Messages

Keywords: defragment, nodefragment

The MIME standard provides the message/partial content type for breaking up messages into smaller parts. This is useful when messages have to traverse networks with size limits, or traverse unreliable networks where message fragmentation can provide a form of “checkpointing,” allowing for less subsequent duplication of effort when network failures occur during message transfer. Information is included in each part so that the message can be automatically reassembled after it arrives at its destination.

The defragment channel keyword and the defragmentation channel provide the means to reassemble messages in the MTA. When a channel is marked defragment, any partial messages queued to the channel are placed in the defragmentation channel queue instead. After all the parts have arrived, the message is rebuilt and sent on its way. The nodefragment disables this special processing. The keyword nodefragment is the default.

12.8.2.1 Defragmentation Channel

Messages are routed to the defragment channel if the defragment keyword is on a destination channel. That is, when the defragment keyword is present on a destination channel to which the MTA would normally enqueue a message, the MTA "looks inside" (MIME parses) the message structure and if the MTA sees that the structure is a MIME message fragment, then the MTA automatically routes the message to the defragment channel rather than straight to the (normal) destination channel.

The defragment database contains information about message fragments coming into the MTA including information indicating the host on which each message fragment is received. Once an initial fragment has been received and noted in the defragment database, any other parts of the message that are received on any other systems using the same defragment database will get routed to the host that received the very first part. For instance:

1. message/partial; id=123; part=x arrives on host 1 and is routed to the defragment channel on host 1 due to the defragment keyword being present on what would otherwise be the destination/outbound channel.

2. Defragment channel on host 1 checks the defragment database for whether any other parts of this message have yet arrived. If none have, the defragment channel (on host 1) enters this part into the defragment database marking the part as being on host 1.

3. message/partial; id=123; part=y arrives on host 2 and is routed to the defragment channel on host 2 due to the defragment keyword being present on what would otherwise be the destination/outbound channel.

4. Defragment channel on host 2 checks the defragment database, sees that part x of this message is already present and stored on host 1. The defragment channel redirects the message fragment to host 1 (source routes the address with @host1).

5. message/partial' id=123; part=y arrives on host 1, is routed to the defragment channel, the defragment channel runs and enters it into the database, and so on.

All remaining parts of a fragmented message are redirected to the host that received the first (first receiving, not necessarily part=1) part of the message. They are reassembled by that host's defragment channel and eventually a reassembled, defragmented message (or if the defragmentation efforts times out, due to exceeding notices, the individual fragments are sent on as-is) is sent onwards to the real destination channel. You get some load-balancing of the defragmentation of messages depending upon which host happens to receive the "first" part of each message.

12.8.2.2 Defragmentation Channel Retention Time

Messages are retained in the defragment channel queue only for a limited time. When one half of the time before the first nondelivery notice is sent has elapsed, the various parts of a message will be sent on without being reassembled. This choice of time value eliminates the possibility of a nondelivery notification being sent about a message in the defragment channel queue.

The notices channel keyword controls the amount of time that can elapse before nondelivery notifications are sent, and hence also controls the amount of time messages are retained before being sent on in pieces. Set the notices keyword value to twice the amount of time you wish to retain messages for possible defragmentation. For example, a notices value of 4 would cause retention of message fragments for two days:

defragment notices 4 
DEFRAGMENT-DAEMON

12.8.2.3 Using NFS-based File Systems for Defragmentation and Vacation Caching

NFS-based file systems are often used for defragmentation and vacation caching. One application is to share the defragmentation database between the multiple MTA systems by having them all share the same defragment cache. This is done by making a link from the msg-svr-base/config/defragment_cache on each system to the file on which you wish to have be the shared defragment database, which will be on the shared NFS disk.

In any case, NFS servers that support proper NFS file semantics (specifically those that honor lock requests, like Solaris NFS) can be used for vacation and defragmentation caches. If NFS is used, use a soft mount option. (A hard mount is the default.) Setting a relatively short timeout value controlled by the mount timeo option (see the mount_nfs(1M) man page) is also a good idea.

With an NFS hard mount and NFS going down, what one would see is the defragment channels on the various systems hanging. With a soft mount, the defragment channels wouldn't hang, but since they would be failing to open the defragment cache, they wouldn't be able to coordinate with defragment channels on other hosts. In the unlikely case that all of a message's fragments happened to first arrive at the same host, then that host's defragment channel would presumably be able to reassemble the message and send it onwards, properly reassembled. More likely, fragments would be on different hosts, would never get reassembled, and would get sent onwards as separate fragments once a relevant defragment channel's retention time expired.