This chapter describes how to implement greylisting by using MeterMaid as well as enhancing its functionality.
Greylisting is a technique used by some MTAs as a way to reduce the number of undesirable spam messages they receive. Simply put, greylisting initially gives a temporary rejection to all incoming mail the first time it sees it, but then permits it upon subsequent attempts. It works by making the assumption that most spam is sent by spambots, PCs that have been compromised by a virus or trojan software, that act as mass-mailing clients. In order to send out as much spam as possible, these systems will connect to a mail server and attempt to deliver the spam to the recipient. If it should encounter a failure, it is extremely unlikely to retry delivery. Proper MTA clients will reschedule and reattempt delivery upon receiving a temporary failure, thus allowing the greylisting mail server a second chance to permit the message to be received.
Greylisting matches messages by using a combination of the source IP address, the envelope FROM: address, and the envelope TO: address. By using this triplet, greylisting works before the message body is sent during the SMTP transaction, making the temporary rejections happen in response to the RCPT TO: command. When the same triplet is presented to the mail server during a subsequent delivery attempt, the RCPT TO: command returns a successful response and the mail server will then accept the message for delivery.
In some setups, greylisting has been seen to reduce the number of spam messages received by 80-90%.
The downside to greylisting, however, is that it introduces an artificial delay to incoming mail from previously unknown triplets. The length of this delay varies depending on the originating MTA, but could range from thirty minutes to several hours. It is now expected by many that e-mail is nearly instantaneous, and greylisting can have a negative impact on customer's expectations. Table 15-1 provides an example of greylisting in action.
Table 15-1 Greylisting at a Glance: Example
| Time | Action | SMTP Result | Explanation |
|---|---|---|---|
|
9:45 |
Incoming SMTP transaction from john@example.com to susan@example.com (local user) |
451 4.5.1 Temporary failure - retry later |
This is the first time that Oracle Communications Messaging Server has seen mail from john@example.com going to susan@example.com so Messaging Server responds with a temporary rejection. |
|
9:47 |
Another transaction from john@example.com to susan@example.com |
451 4.5.1 Temporary failure - retry later |
Because this attempt happened within a short window after the first attempt, it is also temporarily rejected. |
|
10:15 |
A bit later, the same transaction is retried |
250 OK |
Now that a subsequent attempt was made within the resubmit window, Messaging Server consider this combination of sender and recipient permitted. |
|
10:20 |
Mail from stephen@example.com to susan@example.com |
451 4.5.1 Temporary failure - retry later |
This is a different sender, so it is handled independently from the previously permitted combination. This combination would be permitted after the block period has passed. |
|
The next day |
A new message from john@example.com to susan@example.com |
250 OK |
Since this combination is permitted, it remains valid for an extended period. |
MeterMaid supports a greylisting table with additional related tuning options.
Table 15-2 describes the different implementations of greylisting.
Table 15-2 Greylisting Features
| Feature | Implementation |
|---|---|
|
Rejects previously unseen triplet |
Yes |
|
Continues rejecting for an initial blocking period (to block spambots that automatically retry within a very short period) |
Yes |
|
Once accepted, continues to accept messages from that triplet |
Yes |
|
Requires subsequent attempt within a specified period of time to register the triplet as permitted |
Yes |
|
Allows pre-registration of triplets based on outgoing mail, permitting wildcard source IP address |
Yes |
|
Expiration of existing triplets can be extended by recent use |
Yes |
Greylisting has more functionality than using a throttle table.
This section contains information on using Greylisting with Messaging Server.
Setting up greylisting with Messaging Server is easy due to MeterMaid's built-in support for greylisting tables. Instead of calling the throttle routine in check_metermaid.so, you can use the greylisting routine which handles the appropriate return values for allowing Messaging Server to block transactions when the routine returns success.
First, the MeterMaid table definition:
metermaid.table.greylist.type = greylisting metermaid.table.greylist.data_type = string metermaid.table.greylist.max_entries = 50000 metermaid.table.greylist.options = nocase metermaid.table.greylist.block_time = pt5m metermaid.table.greylist.resubmit_time = pt4h metermaid.table.greylist.inactivity_time = p7d
Note that the time formats can now be in ISO 8601 duration format.
This table defines a greylisting table with the following characteristics:
All triplets are rejected during the first 5 minutes, even if multiple attempts occur.
In order for a triplet to be recognized and permitted, a subsequent attempt must be made after that 5 minute window, but within 4 hours of the initial attempt.
Once a triplet is permitted, it will remain as permitted in the table for 7 days after its last use.
The corresponding access control mapping table is simpler when using a greylisting table as no special handling by the mapping table is required:
ORIG_MAIL_ACCESS ! Check the source IP address, sender, and recipient in MeterMaid's greylist table. ! If the call to greylisting() returns success, then Messaging Server should return ! a temporary rejection. If the call fails, then the greylisting check has passed ! and other access control checks can continue. TCP|$@*|$@*|*|$@*|SMTP$@*|MAIL|tcp_local|*|l|* \ $C$[IMTA_LIB:check_metermaid.so,greylisting,greylist,$0|$1|$2]\ $N$X4.5.1|Temporary$ failure$ -$ retry$ later$E
(Note that the suggested form of the string being used in a greylisting table is source-ip|env-sender|env-recipient.)
After the MTA has received the MAIL FROM: and RCPT TO: SMTP commands, it will use the envelope addresses as well as the source IP address in a greylisting call to MeterMaid. Using the table configuration above, MeterMaid will determine whether or not this particular transaction should be permitted. If the MTA should send a temporary rejection at this point, the call to check_metermaid.so will succeed, and the $N part of this entry will be returned indicating a rejection. The $X4.5.1 flags this rejection as a temporary condition so that a 4xx SMTP response will be given.
However greylisting is configured, there are additional steps one can take to make further improvements to its functionality. Several possibilities are listed here. They can be used individually or combined together to make more powerful setups.
This section contains the following topics:
Since it is often the case that one can expect to receive mail from addresses to which the local users are already sending, it may be useful to preload such address combinations into the greylist table. Since the future source IP address is not known, MeterMaid supports a special address of * that will match any other supplied address.
To preload the address combinations, one needs to add an entry to the access control mapping table:
X-IS_INTERNAL_CHANNEL tcp_intranet $Y tcp_submit $Y tcp_auth $Y * $N ORIG_MAIL_ACCESS ! For mail that is coming from a local user and going to an external recipient, we ! can save that user/recipient combination and store it into the greylist table for ! future permission. TCP|$@*|$@*|$@*|$@*|SMTP$@*|MAIL|*|*|tcp_local|* $C$|X-IS_INTERNAL_CHANNEL;$0|\ $[IMTA_LIB:check_metermaid.so,store,greylist,*|$2|$1,1]
(Note that the store routine requires a value although it is not used by a greylisting table. Any value may be specified here and is ignored by MeterMaid.)
This mapping table entry first checks to see whether the source channel is considered a channel used by our local users. If the channel is in the list provided by the X-IS_INTERNAL_CHANNEL mapping table, then processing continues with the call to the store routine of check_metermaid.so to store this new combination into the greylist table. The combination is stored so that it will match incoming messages from the current recipient going to the current sender, and these messages may come from any source IP address and be permitted.
A complication that can occur with greylisting is dealing with remote MTAs that use several different hosts to process deliveries. This can mean that one attempt may occur from 192.168.12.1, but a subsequent attempt may come from a different host like 192.168.12.5. Additional delays may be introduced until an attempt is repeated from a host that had tried it previously.
One way to help address this is to limit IP address matching to the first three octets, allowing more hosts to be considered to be the same source. This would match addresses coming from the same A.B.C.D/24 (class C) subnet. The mapping table setup would be very similar to the examples above, but with a change to the IP address wildcard matching.
ORIG_MAIL_ACCESS ! When checking the source IP address, only use the first three octets in the string ! passed to MeterMaid. TCP|$@*|$@*|$D*.$D*.$D*.$@*|$@*|SMTP$@*|MAIL|tcp_local|*|l|* \ $C$[IMTA_LIB:check_metermaid.so,greylisting,greylist,$0.$1.$2|$3|$4]\ $N$X4.5.1|Temporary$ failure$ -$ retry$ later$E
Some sender addresses will be more complex than a simple user@example.com including such features as subaddresses or VERP (variable envelope return path) notation. For more information see https://en.wikipedia.org/wiki/Variable_envelope_return_path. It may be useful to help greylisting recognize the base form of the address using some basic canonicalization in order to keep track of the basic, simplified address form. This simplification can be done by using a nested mapping table call out to perform the canonicalization.
X-CORRESPONDENT ! Subsidiary mapping for removing any subaddress or VERP style material from ! the local-part of an address. $_*$[+=\-]%*@* $0@$3$Y * $0$Y ORIG_MAIL_ACCESS TCP|$@*|$@*|*|$@*|SMTP$@*|MAIL|tcp_local|*|l|* \ $C$[IMTA_LIB:check_metermaid.so,greylisting,greylist,$0|$|X-CORRESPONDENT;$1||$2]\ $N$X4.5.1|Temporary$ failure$ -$ retry$ later$E
Here, the X-CORRESPONDENT table is used to reconstruct the sender address into the simpler user@example.com form. The result from this is then used in the call to the greylisting function.
Note:
This section requires Messaging Server 7 Update 2 or later for the necessary INCLUDE_SPARES option.It may be desirable to allow users to choose whether to have MeterMaid perform greylisting on their incoming mail. This could be especially useful when considering some local mail recipients who may not wish to be subject to delays in receiving incoming mail from unknown senders, such as recipients like sales or customer_service. For these local users, one can set up additional LDAP attributes to be used in conjunction with the existing ORIG_MAIL_ACCESS mapping table processing.
For this example, let us assume that one creates a new LDAP attribute mailUserGreyListOptIn that will be set to true or false. Those users who have this attribute set to true will have their incoming mail checked with greylisting, while those who have it set for false will skip this check and receive their mail immediately.
In order to have the MTA look at this extra attribute, it must be configured into the option.dat configuration file.
LDAP_SPARE_5=mailUserGreyListOptIn ! Include LDAP_SPARE_5 in ORIG_MAIL_ACCESS probes by setting bit 22 (counting from 0) ! of INCLUDE_SPARES. Bit 22 has the value 4194304. INCLUDE_SPARES=4194304
This will add the value of the user's mailUserGreyListOptIn attribute to the probe string used in the ORIG_MAIL_ACCESS mapping table.
ORIG_MAIL_ACCESS ! This example assumes INCLUDE_SPARES=4194304 is set, so that probes corresponding ! to submissions from remote (tcp_local) senders to local recipients have the form: ! ! TCP|host-ip|host-port|source-ip|source-port|SMTP-app-info|MAIL|tcp_local| ! remote-sender-address|l|local-recipient-address|recipient-mailUserGreyListOptIn ! TCP|$@*|$@*|*|$@*|SMTP$@*|MAIL|tcp_local|$_*|l|$_*|true \ $C$[IMTA_LIB:check_metermaid.so,greylisting,greylist,$0|$1|$2]\ $N$X4.5.1|Temporary$ failure$ -$ retry$ later$E
The key difference in this mapping table entry is the addition of |true to the matching string. Since the INCLUDE_SPARES option will append the content of the mailUserGreyListOptIn attribute to the probe string, this mapping entry can match against only those where the recipient's mailUserGreyListOptIn attribute has been set to true, thus skipping others who may be opting out of greylisting.
The goal of greylisting is to allow mail from remote senders to local recipients once they are known. In most cases, this happens when a transaction presents this combination on a subsequent delivery attempt. It is also possible to use the recipient's LDAP-based address book to check for the sender's address to determine whether to bypass greylisting for an already recognized address. In order to do this, another LDAP attribute must be added to the ORIG_MAIL_ACCESS probe string by including these values into the option.dat file:
LDAP_SPARE_6=psroot ! Include LDAP_SPARE_6 in ORIG_MAIL_ACCESS probes by setting bit 23 (counting from 0) ! of INCLUDE_SPARES. Bit 23 has the value 8388608. INCLUDE_SPARES=8388608
Furthermore, if appropriate, the MTA's LDAP_PAB_xyz options should be set to the proper values for accessing the PAB LDAP server. (However, the usual pab.* (Unified Configuration) or local.service.pab.* (legacy configuration) settings are usually adequate, and usually do not need to be overridden for MTA purposes via the MTA-specific LDAP_PAB_xyz options.)
ORIG_MAIL_ACCESS ! This example assumes INCLUDE_SPARES=4194304 is set, so that probes corresponding ! to submissions from remote (tcp_local) senders to local recipients have the form: ! ! TCP|host-ip|host-port|source-ip|source-port|SMTP-app-info|MAIL|tcp_local| ! remote-sender-address|l|local-recipient-address|recipient-psroot ! ! Matches on this line mean that the sender was found in the recipient's address book. ! "Whitelist" those addresses, bypassing the greylisting check. ! TCP|$@*|$@*|*|$@*|SMTP$@*|MAIL|tcp_local|$_*|l|$_*|* \ $C$]pabldap:///$3?piEmail1?sub?(|(piEmail1=$1)(piEmail2=$1)(piEmail3=$1))[$E$Y ! ! Now, for all other senders, do the normal greylisting check. ! TCP|$@*|$@*|*|$@*|SMTP$@*|MAIL|tcp_local|$_*|l|$_*|* \ $C$[IMTA_LIB:check_metermaid.so,greylisting,greylist,$0|$1|$2]\ $N$X4.5.1|Temporary$ failure$ -$ retry$ later$E
This mapping table example shows an LDAP callout being done to check to see whether the sender is already known to the local recipient. This allows users to add their correspondents to their address book as a way to whitelist those entries, allowing those senders to bypass the greylisting when sending mail to these local recipients.
It is possible to combine many of these elements together into a much more comprehensive setup. This example makes use of the preloading, opt-in, and address book whitelisting features together.
First, the two LDAP attributes must be available to the mapping table probe. They can be added with these options in option.dat:
LDAP_SPARE_5=mailUserGreyListOptIn LDAP_SPARE_6=psroot ! Include LDAP_SPARE_5 and LDAP_SPARE_6 in ORIG_MAIL_ACCESS probes by ! setting bits 22 and 23 (counting from 0) of INCLUDE_SPARES; that is, ! INCLUDE_SPARES=12582912=4194304+8388608=(1<<22)+(1<<23) INCLUDE_SPARES=12582912
Then, the mappings file excerpt below shows how the above elements may be combined.
! Subsidiary mapping for checking incoming port and channel against a
! list of "internal submission" channels.
!
! Probe format is
! port.channel
!
X-INTERNAL-CHANNELS
587.tcp_submit $Y
25.tcp_auth $Y
25.tcp_intranet $Y
! Subsidiary mapping for removing any subaddress or VERP style
! material from the local-part of an address.
! This mapping also performs LDAP URL style quoting of the retained
! portion of the address.
!
X-CORRESPONDENT
$_*$[+=\-]%*@* $=$0@$3$_$Y
* $=$0$_$Y
ORIG_MAIL_ACCESS
! This example assumes INCLUDE_SPARES=12582912 (or some superset of bits) is
! set, so that probes corresponding to submissions from local senders to
! remote recipients have a form of:
!
! TCP|host-ip|host-port|source-ip|source-port|SMTP-app-info|MAIL|source-channel|
! local-sender-address|tcp_local|remote-recipient-address|
! sender-mailUserGreyListOptIn|sender-psroot
!
! while probes corresponding to SMTP MAIL submissions from remote
! (tcp_local) senders to local recipients have the form:
!
! TCP|host-ip|host-port|source-ip|source-port|SMTP-app-info|MAIL|tcp_local|
! remote-sender-address|l|local-recipient-address|
! recipient-mailUserGreyListOptIn|recipient-psroot
!
! The overall logic includes pre-population of the "greylist" table at (1)
! with *|remote-correspondent|local-user on outgoing messages from local users
! who have opted-in to greylisting (have mailUserGreyListOptIn: true) (0),
! and then checks of incoming messages to local users who want greylisting (2)
! against:
! (i) the local-user's PAB (3)
! (ii) the pre-populated entries in the "greylist" table (4) or (5)
! (iii) the "greylist" table tracking "recent" submission attempts from
! not-otherwise-known (not pre-populated due to local-user sending
! to them, nor recognized in local-user's PAB) remote senders (4) or (5)
! Note that (ii) and (iii) are done by one probe to the greylist table, as
! MeterMaid first performs the (ii) check automatically due to the format of
! probe. This probe the greylist table is either done at (4) (for IPv4
! source IPs) or at (5) (for IPv6 source IPs).
!
! For outgoing messages, from local users to remote correspondents,
! pre-populate the "greylist" table using the "store" entry point with
! *|simplified-quoted-remote-correspondent|local-user
! This is so that replies from that remote-correspondent (from whatever
! source-IP) to that local-user will be accepted.
! The subsidiary mapping table X-INTERNAL-CHANNELS is used to check
! (based on the host-port and source-channel) whether the message is
! one from a local user to a remote correspondent. The subsidiary
! mapping table X-CORRESPONDENT is used to canonicalize the
! recipient-address.
! (0)
!
TCP|$@*|*|$@*|$@*|SMTP$@*|MAIL|*|*|tcp_local|*|true|* \
$C$|X-INTERNAL-CHANNELS;$0.$1|PREPOPULATE|$|X-CORRESPONDENT;$3||$2
!
! If the message was indeed from a local user who wants grey-listing,
! then the above entry matched and reset the probe to now be:
! PREPOPULATE|quoted-simplified-remote-correspondent|local-user
! Then the entry below pre-populates the greylist table with an
! entry for
! *|quoted-simplified-remote-correspondent|local-user
! (1)
!
PREPOPULATE|*|* \
$C$[IMTA_LIB:check_metermaid.so,store,greylist,*|$0|$1,1]$E
!
! For incoming submission attempts from remote correspondents (tcp_local
! submission attempts):
! (2) Entry matches remote senders to recipients that
! want grey-listing (mailUserGreyListOptIn: true). Entry constructs
! a new GREYLIST... probe retaining relevant fields, namely:
! GREYLIST|source-ip|quoted-simplified-sender|recipient|psroot
! where the quoted-simplified-sender field is processed (simplified
! and LDAP quoted) using the subsidiary X-CORRESPONDENT mapping table
! (3) For the recipients who want grey-listing, the probe is now
! GREYLIST|source-ip|quoted-simplified-sender|recipient|psroot
! Look up the (simplified) sender address in the
! recipients PAB. Accept submission if found, fall through otherwise.
! (4) If the sender address wasn't found at (3), fall-through and now
! attempt a MeterMaid "greylist" table lookup. The probe to this
! "greylist" table will have the form:
! source-IP-subnet|quoted-simplified-sender|recipient
! This entry matches on IPv4 incoming source IPs, and ignores the last
! eight bits to give an IPv4 subnet.
! Because the probe has the form A|B|C, and it is a probe to a
! greylisting entrypoint, MeterMaid will automatically initially attempt
! a *|B|C probe, only bothering with the A|B|C probe if its initial,
! automatic probe fails. Thus any pre-populated, generic source IP
! entry will match first, prior to MeterMaid attempting a lookup of
! the specific source IP subnet. If the specific triad is
! found in the "greylist" table as being due to be greylisted (rejected
! temporarily), then the probe "succeeds" -- set a new probe string
! FIRSTATTEMPT and continue so that (6) will match and the greylist
! response will be issued.
! Otherwise, the MeterMaid probe "fails" -- as for the cases
! where the probe matches a "good" (pre-populated, or resubmitted
! after block_time) entry.
! (5) The same as (4), but matching on IPv6 incoming source IPs, ignoring
! the last 64 bits to give an IPv6 subnet.
! (6) Issue the temporary rejection when the greylist probe of (4) or (5)
! "succeeded".
!
! For remote senders (source channel tcp_local) to local recipients with the
! LDAP_SPARE_5 attribute "true", reset the probe to the GREYLIST|...form
! (2)
!
TCP|$@*|$@*|*|$@*|SMTP$@*|MAIL|tcp_local|$_*|l|$_*|true|* \
$CGREYLIST|$0|$|X-CORRESPONDENT;$1||$2|$=$3$_
!
! If a recipient wants grey-listing, the probe has been rebuilt to be:
! GREYLIST|ip-source|simplified-sender-address|recipient-address|psroot
! where simplified-sender-address omits any subaddress/VERP-y sorts of fluff
! and has had any LDAP URL required quoting applied, and where psroot has also
! had any LDAP URL required quoting applied.
! So next check whether the simplified-sender-address can be found in
! the recipient-address user's PAB (found under psroot); if the sender is
! found, then accept this message -- this sender is "known".
! (3)
!
GREYLIST|*|*|*|* \
$C$]pabldap:///$3?piEmail1?sub?(|(piEmail1=$1)(piEmail2=$1)(piEmail3=$1))[$E$Y
!
! Otherwise, if the sender was not known to this recipient, then fall down
! to the subsequent entry which performs the MeterMaid grey-list check.
!
! Match on IPv4 addresses and probe the greylist table.
! If this sender matches an entry in the greylist table, whether
! pre-populated or due to a recent sending attempt, then let their message in.
! If the greylist table probe says the sender needs greylisting,
! (that is, never seen before, or seen before but only within block_time),
! then continue with the probe changed to "FIRSTATTEMPT" so that it'll fall
! through and match the temporary rejection entry below at (6).
! Otherwise, if the sender was in the greylist table but after block_time,
! then MeterMaid "fails" this probe, so the check ends; the table processing
! "falls-through" and, if no other entry matches, the submission is
! permitted.
! (4)
!
GREYLIST|$D*.$D*.$D*.$D*|*|*|* \
$[IMTA_LIB:check_metermaid.so,\
greylisting,greylist,$0.$1.$2|$4|$5]$CFIRSTATTEMPT
!
! (5)
!
GREYLIST|$H*:$H*:$H*:$H*:$@H*:$@H*:$@H*:$@H*|*|*|* \
$[IMTA_LIB:check_metermaid.so,\
greylisting,greylist,$0:$1:$2:$3|$4|$5]$CFIRSTATTEMPT
!
! Must be a "new" sending attempt -- give it a temporary rejection
! (6)
!
FIRSTATTEMPT $N$X4.5.1|Temporary$ failure$ -$ retry$ later
The mapping tables above use several features that may be unfamiliar to casual users of the MTA's mapping table, including these:
| Strings | Explanation |
|---|---|
$@ |
Disables saving the following wildcard match that will not be needed for right-hand side substitutions. This permits sufficient saved wildcards (of which there can be at most ten) to be available for matching fields of more interest. |
$_ |
Specifies "non-greedy" (minimal) matching of the portion of the string; used for the local-part of the sender address prior to the first occurrence of a special character possibly indicating a subaddress or VERP address variation. |
$[+=\-]% |
Matches an occurrence of any one of the specified characters. Note that the hyphen character must be quoted with a backslash character to be interpreted as a literal hyphen character rather than indicating a character range. |
$D* |
Matches only decimal digits; used for parsing IP addresses. |
$H* |
Matches only hexadecimal digits; used for parsing IPv6 addresses. |