Chapter 8 Filtering Unsolicited Bulk Email

This chapter describes the unsolicited bulk email (UBE) plug-in and the filters it uses to help you screen out unsolicited email. It also describes the anti-relay filter and plug-in that you can use to prevent your site from being used to relay UBE to others. For information on Messaging Server plug-ins in general, see Chapter 7, Working with SMTP Plug-Ins.

• About UBE
• About the UBE Plug-In


• UBE Filter Format


• Managing Filters with Netscape Console


• Creating Filters Manually


• Extending the UBE Plug-In


• Interface Reference: UBE Filters

The UBE plug-in is a post-accept SMTP plug-in to Netscape Messaging Server. SMTP plug-ins are dynamically loaded software modules that access the Messaging Server Plug-in API, an application programming interface. The plug-ins function by intercepting incoming messages, analyzing them for certain characteristics, and taking appropriate action before possibly passing them on. SMTP Plug-ins are described in general in Chapter 7, Working with SMTP Plug-Ins.

A UBE filter is a line in a text file (default name = UBEfilter.cfg). When you create a filter, you define a criterion to which email messages are compared. Whenever an email message meets the criterion defined in the filter definition, the filter triggers the action that you designated in the filter definition.

[:label] messageField matchCriterion action [argument]

:DontSend Subject "Bad mail" REJECT "Do not send mail"

The label is an identifying name for the filter. This part is optional and is used only if this filter is the destination of another filter's JUMP action. (For a description of the JUMP action, see Table 8.1.)

The message field is the portion of the message header or envelope that the filter analyzes. Each arriving message has header fields that hold information about that message. The filter can use any of the fields in deciding how to handle the mail. There is no complete list of possible header fields since the format is open (see Internet Draft RFC 822 for a list of the defined standard fields), but common fields include To, From, Sender, Reply-To, Content-type, and so on. For descriptions of some of the most common fields, see Table 8.3.

The case tag causes the matching criterion (which follows the message field in the filter) to be treated as case sensitive. By default, filters are not case sensitive. Here's an example of two filters, identical except for the case tag:

Subject:case "Bad mail" REJECT "Do not send mail"

Subject "Bad mail" REJECT "Do not send mail"

The envonly tag instructs the filter to consider only envelope information, rather than all message-header information, for the message field. Because header information is more easily altered by a mail sender than is envelope information, you can use this tag to help resist spoofing attacks based on header information.

The match criterion is the string (or regular expression) that the filter looks for in the contents of the message field to see whether a match is achieved. The combination of message field and match criterion is called the filter's predicate; the state of the predicate (matched or unmatched) determines whether or not the filter is to be applied.

The action field defines the action that the UBE plug-in performs if a match occurs. As soon as a match occurs with any filter predicate, the action associated with that filter is taken. Filter processing stops when a terminal action (one that stops processing) is taken, or when the end of the filter file is reached. For descriptions of the actions, see Table 8.1.

The argument field contains any additional information that may be needed to perform the specified action. Some actions do not take an argument, whereas others do. For descriptions of the arguments used by each of the actions, see Table 8.1.

Table 8.1 lists the actions that you can specify in a UBE filter. Each action is classified either as terminal (filter processing stops as soon as it is taken) or non-terminal (processing continues after it is taken).

The UBE filter supports extended regular expressions compliant with POSIX 1003.2. There are many sources of information on regular expressions; this document is not intended as a reference.

Every message has two types of fields that you can include in the message-field part of your filters. Header fields are created by a mail client when it sends a message. Envelope fields are created by mail servers that send or resend the message during its transit from sender to receiver. By default, the UBE plug-in examines envelope fields only. This restriction exists for performance reasons, because the header can contain any number of fields.

user@xyzcorp.com

parseheader: 1

You can use special field names to combine filters and narrow your criteria. Table 8.4 describes these names, using the following filter as an example:

Subject "This is ."   REJECT "This is bad mail"

:handleFrom   User-From   (.*)@airius.com   !JUMP   handleregular   
              $1          "postmaster"       JUMP   handleAIRIUSpost   
              ""          ""                 JUMP   handleregular   

• The first line (labeled :handleFrom) looks at the senders of the message, using parentheses (as described in Table 8.2) to match against the user name. Only if the mail is from someone at Airius corporation does execution continue to the next line. (See the next section for an explanation of the exclamation point modifier.)
• The second line states that, if the message is exactly from the postmaster account at Airius, execution should jump to the filter line that handles Airius postmaster messages.
• The third line causes processing of other messages to jump to another location. The message field and match criterion are not needed for this filter, so they are replaced by empty quotes; for an explanation, see Omitting Parts of a Filter.

You can modify an action by applying the negation operator to it. Normally, the action specified in a filter is taken if the match criterion is matched. If, however, the negation operator is applied to the action, the action is not taken if the criterion is matched; conversely, the action is taken if the criterion is not matched.

Sender ".*airius\.com" EXIT

Sender ".*airius\.com" !REJECT "local mail only"

To make the UBE plug-in available for use, use Netscape Console to access the SMTP Plugins form (see Activating and Deactivating Plug-Ins). Turn on the UBE plug-in in the Plugin Configuration table. (Its name is libUBEfilter.so or libUBEfilter.sl for Unix; libUBEfilter.dll for Windows NT.)

Follow these steps to create a new UBE filter from Netscape Console:

1. In Netscape Console, open the Messaging Server whose UBE plug-in you want to add a new filter to.
2. Follow either of these steps to access the UBE Configuration form:

• Click the Tasks tab, then click "Configure Unsolicited Bulk Email Filters".
• Click the Configuration tab, open the Services folder in the left pane, and open the SMTP icon beneath the Services folder. Open the Plugins icon beneath the SMTP folder icon, and select UBE beneath the Plugins icon. Click the Unsolicited Bulk Email tab in the right pane.
3. Click "Add a filter." The "Add a UBE Filter" window opens.
4. (Optional) Assign a label to the new filter. The label is needed only in certain circumstances; see Label.
5. Specify a message field to use for matching. Either select a field from the drop-down list in the "If" field or enter the field name directly into the "If" field. The field names in the drop-down list correspond to the parts of the header or envelope information attached to an email message. See Message Field.
6. Choose equals (=) or does not equal (!=). If you choose =, the UBE plug-in acts only when a match between a message and this filter occurs. If you choose !=, the plug-in acts only when a match between a message and this filter does not occur. See Negation Modifier.
7. Type a value (the match criterion) in the Value field. This is the value (generally a string or regular expression) that the UBE plug-in compares with the contents of the message field you specified above. See Match Criteria.
8. Select an action from the "Then" field. This is the action the UBE plug-in will perform on any email that matches this filter (or does not match, if you chose (!=) in Step 4). See Action.
9. If required by the action you selected, enter an argument for that action in the Argument field. Some actions require arguments, such as addresses to forward mail to. See Argument.
10. When you have finished creating the UBE filter, click OK. The new UBE filter appears in the list of UBE filters in the UBE Configuration form.
11. Back in the UBE Configuration form, click Save to save the new UBE filter.

Note: Newly created filters are not saved until you click Save, even though they may appear in the UBE Configuration form.

Follow these steps to edit an existing UBE filter from Netscape Console:

1. Access the UBE Configuration tab, as described in Creating a New Filter.
2. In the Filters field, click to highlight the UBE filter that you want to edit.
3. Click the Edit button. The "Edit a UBE Filter" window opens, displaying the parts of the UBE filter you're editing.
4. Make any desired changes to the filter, specifying the contents of the fields, as described in Creating a New Filter.
5. When you have finished editing the filter, click OK. The revised filter appears in the same place it previously occupied in the UBE configuration form.

Caution: When you edit a UBE filter, the edited filter replaces the original one. You cannot create a new filter by editing an existing filter and, for example, saving it with a new name (label). To create a new filter, see Creating a New Filter.

6. Back in the UBE Configuration form, click Save to save the edited filter.

Note: Changes made to filters are not saved until you click Save, even though they may appear changed in the UBE Configuration form.

Active filters are designated with a check in the Active box next to the filter definition in the UBE Configuration tab. You can activate or deactivate filters individually, and you can activate or deactivate all filters at once. Follow these steps:

1. Access the UBE Configuration tab, as described in Creating a New Filter.
2. Do any of the following:

• To activate all filters, click "Activate all filters."
• To deactivate all filters, click "Deactivate all filters."
• To activate a single inactive filter, check the Active box next to it.
• To deactivate a single active filter, uncheck the Active box next to it.
3. When you have finished activating or deactivating filters, click Save.

Note: The changes you make to the active state of filters are not saved until you click Save.

Incoming email messages are compared to UBE filters in the order in which the filters appear in the UBE Configuration tab. Depending on how you design your filters, the order in which they are applied to incoming email messages may be important. Follow these steps to change the order of the filters in the UBE Configuration tab:

1. Access the UBE Configuration tab, as described in Creating a New Filter.
2. Select the filter you want to move.
3. Click the Move up or Move down button to move the selected filter up or down. Repeat as necessary.
4. When you have finished changing the order of your UBE filters, click Save.

Note: The changes you make in the order of your UBE filters are not saved until you click Save, even though they may appear reordered in the UBE Configuration tab.

You can specify that the UBE plug-in examine header fields as well as envelope fields when applying the filters to a message. (By default, the plug-in looks only at envelope fields.) Follow these steps:

1. Access the UBE Configuration tab, as described in Creating a New Filter.
2. Check the "Parse message header" box to specify that the UBE plug-in should parse message headers as well as envelopes. For more information, see Envelope Fields and Header Fields.

The UBE plug-in is an SMTP plug-in that operates at the PostSmtpAccept point of message handling. Messaging Server uses the file plugins.cfg to configure the UBE plug-in. A typical UBE configuration in plugins.cfg (for a Unix Solaris installation) might look like this (for more information about plugins.cfg, see Chapter 7, Working with SMTP Plug-Ins):

PostSmtpAccept \
/export2/server4/msg-Airius1/smtp-bin/plugins/libUBEfilter.so \
funcs=filter_msg_plugin \
init=filter_msg_init \
config=/export2/server4/msg-Airius1/smtp-bin/plugins/UBEfilter.cfg \
option=/export2/sever4/msg-Airius1/smtp-bin/plugins/UBEfilter.opt 

• Use the file libUBEfilter.so (the library that contains the UBE plug-in)
• Call the initialization function filter_msg_init at server startup


• Call the function filter_msg_plugin to start filter processing


• Use the filter configuration file named UBEfilter.cfg


• Use the filter options file named UBEfilter.opt

• The filter configuration file (UBEfilter.cfg by default) contains the list of filters to apply to any incoming mail. When you create, edit, delete, activate, or deactivate filters, whether manually or through the Netscape Console interface, you modify this file.
• The filter options file (UBEfilter.opt by default) controls whether to parse the message header file. You modify this option by editing the file directly or by using the Netscape Console interface (see Parsing Header Fields).

To create and manipulate UBE filters without using Netscape Console, you need to manipulating several configuration files.

1. Start by ensuring that the UBE plug-in is activated.

Examine the file plugins.cfg (see previous section) to make sure the proper configuration lines are in place to activate the plug-in.

2. If necessary, edit the file UBEfilter.opt or its equivalent to specify message-header parsing.
3. Create your set of UBE filters by editing the file UBEfilter.cfg or its equivalent. You can use any text editor (Notepad for Windows NT or vi for Unix, for example).

Enter your filters in order, one per line, in the configuration file. To enter a comment use the # symbol at the beginning of the line. (For more information on comments, see Entering Comments.)

4. When you are satisfied with your edits, save the file in its proper location as defined in plugins.cfg.
5. To make changes later, open, edit, and resave the configuration file.

All parts of each filter line (other than the optional label) must be present, but you can use empty quotes ("") as a placeholder, if you wish to.

           Channel-To  (.*)@airius.com  ""      
  :ceo     $1           ceo             COPY    postmaster@airius.com 
  :cfo     $1           cfo             COPY    finance@airius.com     

 :cfo     $1           cfo           COPY    finance@airius.com    
          ""           ""            JUMP     othermail          

Any line that starts with the number sign (#) or tilde (~) (leading spaces or tabs are ignored) is considered a comment and is ignored during execution by the UBE plug-in.

• You use the number sign (#) to mark comments if you edit the filter file manually. Lines starting with a number sign do not show up in the UBE Configuration tab and are not processed during execution.
• The UBE configuration tab uses the tilde (~) as a special comment marker. If you disable a filter in the tab, the software adds a leading tilde to that filter line in the configuration file. If you enable a previously disabled filter, the software removes the tilde from the line. Both disabled and enabled filters appear in the UBE Configuration tab (with their status marked appropriately), but during execution lines starting with a tilde are considered comments and are ignored.

• Use only the number sign to create comments. The tilde is reserved for use by the UBE Configuration tab only.
• Do not use the comment symbol anywhere but at the beginning of a line. Any other location is invalid syntax.

You can include either the number sign or tilde as a regular character in any part of your filter by enclosing the character in quotes. For example, if you want to filter based on a message field named X-Accept#, your filter line could be:

:Label "X-Accept#" "Free stuff" REJECT "Please don't send this mail"

The following example, however, has invalid syntax:

:Label X-Accept# "Free stuff" REJECT "Please don't send this mail"

Filter examples are shown throughout this section as illustrations of filter format. This section provides additional illustrations, including an example of a complexly interacting set of filters that performs many functions.

        Channel-To   "louisr@xyzcorp\.com" COPY     "watch@domain.com"
        Subject      "weapons for sale"    DROP     "weap@xxx.gov"
        Channel-To   "CEO.*"               JUMP     "DoCEO"
:xCEO   $#           "50"                  REJECT   "No bulk mail"
        Subject      "May contain a virus" RUN      "VirusScan.exe"
        $&           "1"                   REJECT   "This had a virus"
        Content-Type "multipart/mixed"     JUMP     "DoMime"
        Client       "Netscape.*"         !JUMP     "TstCli"
:xCli   Subject      ".*"                  EXIT
:DoCEO  Subject      "Postmaster Eval"     HOLDCOPY "postmaster eval"
        $ANY         ".*"                  JUMP     "xCEO"
:DoMime Channel-To   ".*_.*@xyzcorp\.com"  REJECT   "Can't read MIME "
        $ANY         ".*"                  EXIT
:TstCli Host-From    ".*\.xyzcorp\.com"    COPY     "IS_department"
        $ANY         ".*"                  JUMP     "xCli"

Channel-To: CEO@domain.com

Subject "Postmaster Eval"

1. This message is matched by the Channel-To "CEO.*" filter (line 3).
2. Processing therefore jumps to the filter labeled :DoCEO (line 10). The filter on that line again matches the message (because the subject is "Postmaster Eval") and so this message is copied to the postmaster, who can then decide whether or not the CEO should see it.

Channel-To: CEO@domain.com

Subject "Shareholders meeting"

1. As in the previous case, this message is matched by the Channel-To "CEO.*" filter (line 3), so processing then jumps to the filter labeled :DoCEO (line 10).
2. But this time the filter on line 10 doesn't match the message (the subject is not "Postmaster Eval"), so processing passes to the next line (11).
3. The filter on line 11 matches the message (because its predicate can match any string in any field), so processing jumps back to the label :xCEO (line 4).
4. At this point the rest of the filters can be applied to this message. If none of lines 4 through 8 match the message, processing will halt with the EXIT action at line 9, because the Subject fields will match. The mail will then be forwarded to its addressee.

Channel-To "louisr@xyzcorp.com"

Subject "illegal stock trade"

1. If the Client field of the message doesn't start with "Netscape", the filter on line 8 transfers processing to the filter labeled :TstCli (line 14).
2. That filter tests whether the message was sent from any host in the domain xyzcorp.com. If it was, the sender did not use a Netscape client to send the mail, so the plug-in notifies the IS department of that fact by sending them a copy of the message.
3. If the message was not sent locally, processing passes to the last line (15), where it jumps back up to line 9 and continues.

Channel-To r_francisco 
Content-Type "multipart/mixed"

1. The filter on line 7 matches the content type of the message, so processing jumps to the filter labeled :DoMime (line 12).
2. That filter checks the account name. Because the name contains an underscore, the plug-in rejects the mail.

The RUN action is built into the UBE plug-in. You can use this action to invoke an external program that can process messages in conjunction with the UBE filters. External programs called in this way can perform tasks such as scanning for viruses, matching message-field content against DNS or other databases of names, matching message-body text, and gathering statistics.

The UBE plug-in supports use of an extension library that is separate from the plug-in itself. You can write extensions to verify host name, reject relaying, perform DNS lookup, and perform virus checking, and any other task required. An extension library may be more appropriate than a program executed by the RUN action in situations where you want to alter the fundamental nature of a UBE plug-in action, or where you need to add a new kind of action to it.

1. It tries to locate the action name in the extension library. If an entry point with that action name is found, the plug-in calls it.
2. If the plug-in finds no entry point with that action name in the extension library, the plug-in executes its own built-in default action.
3. If the plug-in has no built in default action of that name, it does nothing and continues processing the filter configuration file.

int (*ExtpFuncAct) (
      const char *arg, 
      char *control_file, 
      char *msg_file, 
      int * result
     );

• A UBE plug-in filter can be created to prevent others from relaying their messages through your site to other sites or to your own users. See Creating an Anti-Relay Filter for details.
• A ready-to-run, protocol-level, anti-relay plug-in that you can use to prevent others from relaying their messages through your site to other sites. See Using the Anti-Relay Plug-In for details.

UBE filters block unsolicited messages according to information in the message header and envelope as described in About the UBE Plug-In.

Channel-To ".*@xyzcorp\.com" EXIT
$ANY ".*" REJECT "We accept mail for XYZ Corporation only"

Host-From "123.45.67.*" EXIT 
Channel-To ".*@xyzcorp\.com" EXIT 
$ANY ".*" REJECT "We accept mail for XYZ Corporation only"

Netscape supplies a protocol-level plug-in that you can use to prevent your servers from being used to relay UBE to other sites. Because this anti-relay plug-in operates at the protocol level, it intercepts and rejects unauthorized messages before the server accepts them and writes them to disk. This saves both disk space and processing resources. Netscape provides both the ready to use anti-relay plug-in, and the plug-in's source code so that you can modify or enhance it to meet your special requirements.

• Delivery rules identify the users at your site for whom Messaging Server will accept incoming messages. Messages addressed to those identified by a delivery rule are accepted and delivered regardless of who submits the messages.
• Submission rules identify the hosts from whom Messaging Server will accept messages for initial transmission, relay, or delivery.

Messaging Server will only pass along messages that it receives from hosts identified by a submission rule or messages from any source that are addressed to those identified by a delivery rule. (Note however, that by default the anti-relay plug-in is also configured to accept messages from users who have been authenticated by SMTP regardless of whether or not they are using a host identified by a submission rule.)

If you use the anti-relay plug-in, the only users who will be able to send email to outside destinations (that is, destinations not identified by a delivery rule) are those who have been authenticated by SMTP (if you are using that option) or who are sending a message from a host identified by a submission rule.

To enable the anti-relay plug-in:

1. Use configutil to specify that SMTP is to load the anti-relay plug-in when it starts up.

The anti-relay plug-in must be loaded with the configutil -l option to specify that it run locally. A full path to the anti-relay plug-in must be specified.

In Unix environments, enter the command:

configutil -l -o service.smtp.protplugmodules -v \
'/usr/netscape/suitespot4/plugins/antirelay.so'

In NT environments, enter the command:

configutil -l -o service.smtp.protplugmodules -v
C:\Netscape\Server4\plugins\antirelay.dll
2. Configure the anti-relay plug-in as described in the next section.

You control how the anti-relay plug-in operates, and what relaying is permitted, through configuration options. The plug-in configuration options are specified in a plain text file named antirelay.conf that is stored in the server-instance configuration directory.

• In Unix environments, the configuration directory is specified by the CONFIGROOT environment variable and is typically
  server-root/config/smtp/config.
• In NT environments, the configuration directory is typically
  server-root\msg-instance\config.

# Anti-relay configuration file for server nsmail7
  resolvehostnames:0
  useauthinfo:0
  advertiseauthinfo:1

# We accept mail addressed to these recipients:
  delivery:*@airius.com

# We relay messages coming from these hosts:
  submission:*.airius.com
  submission:mailserver.ourfriend.com
  submission:127.0.0.1

The anti-relay plug-in configuration options are:

• delivery: Destination email addresses for which relayed messages will be accepted for delivery by Messaging Server. In other words, if a relayed message is addressed to a user identified in a delivery: statement, it will be delivered to the recipient's mailbox. Normally, your own users are identified so that their incoming messages are delivered. You can have multiple delivery: statements in an antirelay.conf file. You can use patterns such as *@airius.com in delivery statements. (For more detailed information on how to specify email addresses in delivery: statements, see Specifying Anti-Relay User and Domain Names .)
• submission: A host or domain from which messages will be relayed by Messaging Server. In other words, if a message to be relayed comes from a source identified in a submission: statement, it will be relayed. Normally, your own hosts and domains are listed so that your users' outgoing messages are dispatched.You can have multiple submission: statements in an antirelay.conf file. You can use patterns such as *.airius.com in delivery statements. (For more detailed information on how to specify host and domain names in submission: statements, see Specifying Anti-Relay User and Domain Names .)
• resolvehostnames: By default, the anti-relay plug-in resolves host names if Messaging Server is configured to resolve peer addresses into host names. If Messaging Server is not configured to resolve host names, you can only use IP addresses in delivery: and submission: statements. If your server is configured to resolve host names, you can choose to allow the anti-relay plug-in to use host names or force it to require IP addresses by setting the value of the resolvehostnames option. The allowed values are:

• resolvehostnames:0
  Forces the anti-relay plug-in to us IP addresses
• resolvehostnames:1
  Allows anti-relay plug-in to use host names. (This is the default value.)
• useauthinfo: By default, the anti-relay plug-in trusts all SMTP authenticated users as if they are connected from a valid submission domain. To change this, reset the useauthinfo: value. The allowed values are:

• useauthinfo:0
  Do not automatically trust SMTP authenticated users.
• useauthinfo:1
  Trust all users authenticated by SMTP. (This is the default value.)
• advertiseauthinfo: By default, the anti-relay plug-in does not provide non-authenticated clients who are denied relay privileges any indication that they might be allowed to relay if they were authenticated. To change this, reset the advertiseauthinfo: value. The allowed values are:

• advertiseauthinfo:0
  Rejected users get the message: 551 delivery not allowed to non-local recipient. (This is the default value.)
• advertiseauthinfo:1
  Rejected users get the message: 530 delivery not allowed to non-local recipient, try authenticating.

The submission: statements in the anti-relay plug-in configuration file identify the hosts and domains authorized to submit messages to the server for relaying. Hosts and domains are specified in submission: statements with IP addresses or DNS domain names. If your server is not configured to resolve peer addresses into host names (configuration parameter service.smtp.doclientdnslookup) all hosts and domains in submission: statements must be specified using IP addresses.

• submission:*.airius.com only permits hosts in that domain to submit messages for relay.
• submission:*.airius.com specifies that messages will be relayed from all hosts in airius.com, and all hosts in subdomains of airius.com.
• submission:123.123.123.* specifies that messages will be relayed from all hosts with IP addresses that begin with 123.123.123.
• delivery:*@*.airius.com specifies that messages will be relayed to all users in airius.com and its subdomains.
• delivery:*@airius.com only permits delivery to users in that domain; messages will not be delivered to hosts in subdomains of airius.com unless they are identified in other submission: statements.

• Creating a New Filter
• Editing an Existing Filter


• Activating and Deactivating Filters


• Changing the Order of Filters


• Envelope Fields and Header Fields

Save. Click this button to commit any settings you have made in the Unsolicited Bulk Email tab.

• Creating a New Filter
• Editing an Existing Filter

OK. Click this button to commit the entries you have made. If you have been creating a new filter, the filter is added to the end of the list of filters. If you have been editing an existing filter, the edited filter replaces the existing version at its current location in the list of filters.