Chapter 13 Planning Messaging Server Security

This chapter describes how to plan for and protect the various component of your Messaging Server deployment.

Protecting Messaging Components in Your Deployment

Protecting MTAs

Secure MTAs to protect processing resources and server availability. When messages are relayed from unauthorized users or large quantities of spam are delivered, response time is reduced, disk space is used up, and processing resources, which are reserved for end users, are consumed. Not only does spam waste server resources, it is also a nuisance for your end users.


Note	With each component, you should use the chroot function to limit the number of available commands on each machine.


Note	Not only must you protect your deployment from external unauthorized users, but you might also have to protect your system from internal users as well.

Table 13-1 Common MTA Security Threats
Threat	Description
UBE (Unsolicited Bulk Email) or spam	Refers to the practice of sending electronic junk mail to millions of users.
Unauthorized relaying	Uses another company’s SMTP server to relay your email. Spammers often use this technique to cover their tracks. End-users might send complaints back to the sending relay, not to the spammer.
Mail bombs	Characterized by abusers who repeatedly send an identical message to a particular address. The goal is to exceed mailbox quotas with the message.
Email spoofing	Creates email that appears to have originated from one source when it actually was sent from another source.
Denial of service attacks	Prevents legitimate users of a service from using that service. For example, an attacker attempts to flood a network, thereby preventing legitimate network traffic.

This section on MTA relays describes security options you can use in your deployment:

Access Controls

Conversion Channels and Third Party Filtering Tools

RBL Checking

Client Access Filters

Monitoring Your Security Strategy

Access Controls

You can use access controls to reject messages from (or to) certain users at a system level. In addition, you can institute more complex restrictions of message traffic between certain users. Also, you might allow users to set up filters on their own incoming messages (including rejecting messages based on contents of the message headers).

If you want to control access with envelope-level controls, use mapping tables to filter mail. If you want to control access with header-based controls, or if users wish to implement their own personalized controls, use the more general mailbox filters approach with server-side rules.

Mapping Table Overview

You can control access to your mail services by configuring certain mapping tables. Many components of the MTA employ table lookup-oriented information. This type of table is used to transform, that is, map, an input string into an output string. Mapping tables are usually presented as two columns. The first (left-hand) column provides possible input strings against which to match (pattern), and the second (right-hand) column gives the resulting output string for which the input string is mapped (template).

The following table describes these mapping tables, which enable you to control who can or cannot send mail, receive mail, or both. See the Sun Java System Messaging Server Adminstration Guide for more information:

Table 13-2 Access Control Mapping Tables
Mapping Table	Description
SEND_ACCESS	Used to block incoming connections based on envelope From: address, envelope To: address, source and destination channels. The To: address is checked after rewriting, alias expansion, and so on, have been performed.
ORIG_SEND_ACCESS	Used to block incoming connections based on envelope From: address, envelope To: address, source and destination channels. The To: address is checked after rewriting but before alias expansion.
MAIL_ACCESS	Used to block incoming connections based on combined information found in SEND_ACCESS and PORT_ACCESS tables: that is, the channel and address information found in SEND_ACCESS combined with the IP address and port number information found in PORT_ACCESS.
ORIG_MAIL_ACCESS	Used to block incoming connections based on combined information found in ORIG_SEND_ACCESS and PORT_ACCESS tables: that is, the channel and address information found in ORIG_SEND_ACCESS combined with the IP address and port number information found in PORT_ACCESS.
FROM_ACCESS	Used to filter mail based on envelope From: addresses. Use this table if the To: address is irrelevant.
PORT_ACCESS	Used to block incoming connections based on IP number.

The following figure illustrates where mapping tables are activated in the mail acceptance process.

For all the network ports controlled by the MTA service dispatcher, a PORT_ACCESS rejection response, if warranted, takes place at the initial connection from a remote host. A FROM_ACCESS rejection occurs in response to the MAIL FROM: command, before the sending side can send the recipient information or the message data. A SEND_ACCESS or MAIL_ACCESS rejection occurs in response to a RCPT TO: command, before the sending side gets to send the message data. If an SMTP message is rejected, your Messaging Server never accepts or sees the message data, thus minimizing the overhead of performing such rejections. If multiple access control mapping tables exist, Messaging Server checks them all.

Configuring Anti-Relaying with Mapping Tables

You can also use access control mappings to prevent people from relaying SMTP mail through your Messaging Server system. For example, someone might try to use your mail system to relay junk mail to thousands of mailboxes on your system or on other systems.


Note	If the message is accepted, it can still be filtered by way of conversion channels and user defined filters.

By default, Messaging Server prevents all SMTP relaying activity, including relaying by local POP and IMAP mail clients. If clients do not authenticate by using SMTP AUTH, as described in Enabling Authenticated SMTP, and attempt to submit messages to external addresses via Messaging Server’s SMTP server, their submission attempts are rejected. Thus, you will likely want to modify your configuration so that it recognizes your own internal systems and subnets from which relaying should always be accepted.

To Prevent Relaying From Outside Hosts

To prevent hosts that reside outside your domain from relaying to other hosts outside your domain:

Split incoming mail into different channels. For example:

IP addresses within your domain go to the tcp_internal channel.

Authenticated sessions go to the tcp_auth channel.

All other mail is sent to the tcp_local channel.

Recognize and allow mail from your POP and IMAP clients by using an INTERNAL_IP mapping table, fully explained in the chapter on Mail Filtering and Access Control in the Sun Java System Messaging Server Administration Guide.

http://docs.sun.com/doc/819-0105

Using Mailbox Filters

A filter consists of one or more conditional actions to apply to a message. Messaging Server filters are stored on the server and evaluated by the server. They are sometimes called server-side rules (SSR).

You can create channel-level filters and MTA-wide filters to prevent the delivery of unwanted mail. You can also create filter templates and make them available to end users by using Messenger Express. End users use the templates to build personal mailbox filters to prevent delivery of unwanted mail message to their mailboxes. The server applies filters in the following priority. See the Sun Java System Messaging Server Administration Guide for more information.

Per-user filters

Per-user filters apply to messages destined for a particular user’s mailbox. You can create filter templates and make them available to end users by using the Messenger Express client. End users use the templates to build personal server filters to manipulate the delivery of mail messages to their mailboxes. The filers reject unwanted messages, redirect mail, filter messages into mailbox folders, and so on.

If a personal mailbox filter explicitly accepts or rejects a message, then filter processing for that message finishes.

A filter template generalizes a Sieve script by replacing “hard-coded” elements of the Sieve script with prompts and input fields. A Java servlet is used to parse the Sieve templates and generate the user interface in the browser. When an end user supplies values in the input fields, the servlet takes those values and saves them in a Sieve script in the user’s directory profile entry. The prompts and input fields are presented to the end user through the Messenger Express interface.

If the recipient user had no mailbox filter, or if the user’s mailbox filter did not explicitly apply to the message in question, Messaging Server next applies the channel-level filter.

Channel-level filter

Channel-level filters apply to each message enqueued to a channel. A typical use for this type of filter is to block messages going through a specific channel.

To create a channel-level filter, you must write the filter using Sieve. See the Mail Filtering and Access Control chapter in the Sun Java System Messaging Server Administration Guide for specific instructions on creating filters with Sieve:

http://docs.sun.com/doc/819-0105

If the channel-level filter explicitly accepts or rejects a message, then filter processing for that message finishes. Otherwise, Messaging Server next applies the MTA-wide filter, if there is one.

MTA-wide filter

MTA-wide filters apply to all messages enqueued to the MTA. A typical use for this type of filter is to block unsolicited bulk email or other unwanted messages regardless of the messages’ destinations.

To create an MTA-wide filter, you must write the filter using Sieve. See the Mail Filtering and Access Control chapter in the Sun Java System Messaging Server Administration Guide for specific instructions on creating filters with Sieve:

http://docs.sun.com/doc/819-0105

By default, each user has no mailbox filter. When a user accesses Messenger Express interface to create one or more filters, then their filters are stored in the LDAP Directory.

Conversion Channels and Third Party Filtering Tools

The conversion channel performs body-part-by-body-part conversions on messages through the MTA. This processing can be done by any site-supplied programs or command procedures. The conversion channel can do such things such as convert text or images from one format to another, scan for viruses, translate languages, and so forth. Various message types of the MTA traffic are selected for conversion, and specific processes and programs can be specified for each type of message body part. If you are looking to use the conversion channel with a virus scanning program, you can either disinfect, hold, or reject messages. A special conversion channel configuration is consulted to choose an appropriate conversion for each body part. For more information, see the Using Pre-defined Channels chapter in the Sun Java System Messaging Server Administration Guide.


Note	Using specialized processing like a conversion channel puts additional load on your system. Be sure to account for it when you plan your sizing strategy.

With the conversion channel, you can use third-party anti-spam and anti-virus software solutions. You can also use the MTA API to create a channel to invoke a remote scanning engine. For more information on the MTA API, see the Sun Java System Messaging Server Developer’s Reference:

In general, it is best that these third-party solutions are shielded from external sites and are only used on back-end or intermediate relays.

The Brightmail solution consists of the Brightmail server and real-time anti-spam and anti-virus (for service providers only) rule updates that are downloaded to your messaging servers. When the Brightmail Logistics and Operations Center (BLOC) receives spam from email probes, operators immediately create appropriate anti-spam rules. These rules are then downloaded to Brightmail customer machines. Similarly, the Symantec Security Response real-time virus rules are also sent from Brightmail. These rules are used by customer’s Brightmail servers to catch spam and viruses.

Messaging Server also supports the use of SpamAssassin, a freeware mail filter used to identify spam. SpamAssassin calculates a score for every message. Scores are calculated by performing a series of tests on message header and body information. Each test either succeeds or fails, and the score is adjusted accordingly. Scores are real numbers and may be positive or negative. Scores that exceed a certain threshold are considered to be spam.

For more information on configuring Brightmail and SpamAssassin for Messaging Server, see the Sun Java System Messaging Server Administration Guide:

RBL Checking

The Mail Abuse Protection System’s Real-time Blackhole List (MAPS RBL) is a list of host and networks that are known to be friendly or neutral to abusers who use these hosts and networks to either originate or relay spam, or to provide spam support services.

You can configure your MTAs to compare incoming connections against the MAPS RBL. You can also use DNS-based databases used to determine incoming SMTP connections that might send unsolicited bulk mail.

For more information, see the Mail Filtering and Access Control chapter in the Sun Java System Messaging Server Administration Guide.

Client Access Filters

Messaging Server supports sophisticated access control on a service-by-service basis for POP, IMAP, and HTTP. The Messaging Server access-control facility is a program that listens at the same port as the TCP daemon it serves. The access-control facility uses access filters to verify client identity, and it gives the client access to the daemon if the client passes the filtering process.

If you are managing messaging services for a large enterprise or for a service provider, these capabilities can help you to exclude spammers and DNS spoofers from your system and improve the general security of your network.

As part of its processing, the Messaging Server TCP client access-control system performs (when necessary) the following analyses of the socket end-point addresses:

Reverse DNS lookups of both end points (to perform name-based access control)

Forward DNS lookups of both end points (to detect DNS spoofing)

Identd callback (to check that the user on the client end is known to the client host)

The system compares this information against access-control statements called filters to decide whether to grant or deny access. For each service, separate sets of Allow filters and Deny filters control access. Allow filters explicitly grant access; Deny filters explicitly forbid access.

When a client requests access to a service, the access-control system compares the client’s address or name information to each of that service’s filters—in order—using these criteria:

The search stops at the first match. Because Allow filters are processed before Deny filters, Allow filters take precedence.

Access is granted if the client information matches an Allow filter for that service.

Access is denied if the client information matches a Deny filter for that service.

If no match with any Allow or Deny filter occurs, access is granted. The exception is the case where there are Allow filters but no Deny filters, in which case lack of a match means that access is denied.

The filter syntax described here is flexible enough that you should be able to implement many different kinds of access-control policies in a simple and straightforward manner. You can use both Allow filters and Deny filters in any combination, even though you can probably implement most policies by using almost exclusively Allows or almost exclusively Denies.

Client access filters are particularly helpful if troublesome domains are a known quantity. While UBE filters must store and process every spam message, client access filters free Messaging Server from having to process any spammed messages. Because client access filters block mail from entire domains, this feature should be used with caution.

An SMTP client is required to log in before relaying a message.

Client access filters do not scale well for large deployments.

For more information on client access filters, see the Configuring Security and Access Control chapter in the Sun Java System Messaging Server Administration Guide.

Monitoring Your Security Strategy

Monitoring your server is an important part of your security strategy. To identify attacks on your system, monitor message queue size, CPU utilization, disk availability, and network utilization. Unusual growth in the message queue size or reduced server response time may identify some of these attacks on MTA relays. Also, investigate unusual system load patterns and unusual connections. Review logs on a daily basis for any unusual activity.

Protecting the Message Store

The most important data in a messaging server is the user’s mail in the message store. Note that the mail messages are stored as individual files, which are not encrypted. Consequently, physical access and root access to the message store must be protected.

To secure the Message Store, restrict access to the machine where the store is installed. You can enable CRAM-MD5 or Digest-MD5 passwords instead of using unencrypted, plaintext passwords. For more information on passwords, see Planning Messaging User Authentication.

Not only should you create password authentication to the store machine, you might also use tools like VPN access, ssh, or pam, which lists valid users that are allowed to login to the machine.

In addition, a two-tiered architecture is recommended over a one-tiered architecture. Because the Message Store performs the most disk intensive work of any components in a messaging system, do not have filtering, virus scanning, and other disk-intensive security processes on the same machine. In a two-tiered architecture, you don’t have to run UBE filters, anti-relay, and client access filters on the same machine as the message store, which can add load to your system. Instead, the MTAs handle that processing. In addition, user access to the store is limited to through an MMP or an MEM in a two-tiered deployment, potentially adding an extra security layer to the message store.

If you deploy a one-tiered architecture, be sure to account for the additional security processing and load (like SSL and virus scanning) that you will need. For more information, see Chapter 10, "Planning a Messaging Server Sizing Strategy."

For additional Message Store security processing, set disk quotas per user to limit disk usage. Also, use administrator alarms if free space thresholds are fast approaching their limits. Like the MTA, be sure to monitor the server state, disk space, and service response times. For more information, see the Managing Your Message Store chapter in the Sun Java System Messaging Server Administration Guide.

Protecting MMPs and MEMs

Because the MMP serves as a proxy for the Message Store, it needs to protect access to end user data and guard against unauthorized access. User IDs and passwords provide basic authentication capabilities. In addition, you can use client access filters to limit user login to specific domains or IP address ranges. SMTP Authentication, or SMTP AUTH (RFC 2554) is the preferred method of providing SMTP relay server security. SMTP AUTH allows only authenticated users to send mail through the MTA. For more information, see Enabling Authenticated SMTP.

Locate the MMP on a different machine (or under a different userID) in front of your POP or IMAP services. You can have front-end machines with just MMP and MTAs, and then have a physically secure network between those front-end machines, the mail stores, and the LDAP servers.

Special security considerations need to be given to Messenger Express access to the Message Store when your users are logging in from the Internet. In general, you want to make sure that the stores are separated from the outside world by a firewall. In addition, you might consider adding a Messenger Express Multiplexor (MEM), a specialized server that acts as a single point of connection to the HTTP access service. Like the MMP, the MEM supports both unencrypted and encrypted (SSL) communication with mail clients. The MEM also has to protect access to end user data and guard against unauthorized access.

Planning Messaging User Authentication

User authentication enables your users to log in through their mail clients to retrieve their mail messages. Methods for user authentication include:

Plain Text and Encrypted Password Login

Authentication with Simple Authentication and Security Layer (SASL)

Enabling Authenticated SMTP

Certificate-based Authentication with Secure Sockets Layer (SSL)

Plain Text and Encrypted Password Login

User IDs and passwords are stored in your LDAP directory. Password security criteria, such as minimum length, are determined by directory policy requirements. Password security criteria is not part of Messaging Server administration. To understand directory server password policies, see the Sun Java System Directory Server Deployment Planning Guide:

An administrator can set a messaging configuration parameter to determine if plain passwords are allowed or if passwords must be encrypted. For more information, see the service.xxx.plaintextminciper (where xxx is http, pop, or imap) parameter in the Sun Java System Messaging Server Administration Reference:

Both plain text and encrypted password login can be used with POP, IMAP, and Messenger Express user access protocols.

Authentication with Simple Authentication and Security Layer (SASL)

SASL (RFC 2222) provides additional authentication mechanisms for POP, IMAP, and SMTP user access protocols. Messaging Server has SASL support for the user access protocols listed in the following table:

Table 13-3 SASL Authentication User Access Protocols Support Matrix
	Plain	Login	CRAM-MD5	Digest-MD5	Certificate	APOP
SMTP AUTH	Yes	Yes	Yes	Yes	-	-
POP	Yes	-	Yes	Yes	-	Yes
IMAP	Yes	-	Yes	Yes	-	-
HTTP (Messenger Express)	Yes	-	-	-	Yes	-


NoteS	When using CRAM-MD5, passwords must be stored in plain text format in the LDAP directory server. Digest-MD5 is not yet supported in the MMP, but it is supported if you choose not to use the MMP. When using POP, passwords must be stored in plain text format in the LDAP directory server.

If you use SASL, user name and passwords are not encrypted unless SSL is used for the session. (For more information on SSL, see Encryption with SSL). The SASL mechanisms, PLAIN and LOGIN, encode authentication information, but can be easily decoded if captured. Despite this limitation, SASL is useful because it can be combined with SMTP AUTH (described in Enabling Authenticated SMTP) to allow only authenticated users to send mail to relay mail through your system. For example, legitimate users can authenticate to the SMTP server, and the SMTP server can then be configured to switch to a different channel. In this way, the message from an authenticated session can come from a different TCP channel than a user that did not authenticate. A message from a user in your internal network can also be switched to differentiate it from a message coming from other sources just based on the IP address of the incoming connection.

For more information on SASL, see the Configuring Security and Access Control chapter in the Sun Java System Messaging Server Administration Guide:

Enabling Authenticated SMTP

By default, users need not submit a password when they connect to the SMTP service of Messaging Server to send a message. You can, however, enable password login to SMTP in order to enable authenticated SMTP.

Authenticated SMTP (also referred to as SMTP AUTH) is an extension to the SMTP protocol. Authenticated SMTP allows clients to authenticate to the server. The authentication accompanies the message. The primary use of authenticated SMTP is to enable local users who are not in their office to submit mail without creating an open relay that others could abuse. The AUTH command is used by the client to authenticate to the server.

Authenticated SMTP provides security in sending messages with the SMTP protocol. To use authenticated SMTP, you do not need to deploy a certificate-based infrastructure. (Certificates authentication is described in Certificate-based Authentication with Secure Sockets Layer (SSL).)

With authenticated SMTP, the client can indicate an authentication mechanism to the server, perform an authentication protocol exchange, and optionally negotiate a security layer for subsequent protocol interactions.

If you require SMTP AUTH for mail submission, turn on appropriate logging, so any mail abuse can be traced.

For more information on authenticated SMTP, see the MTA chapters of the Sun Java System Messaging Server Administration Guide:

Certificate-based Authentication with Secure Sockets Layer (SSL)

Messaging Server uses the SSL protocol for encrypted communications and for certificate-based authentication of clients and servers. This section describes certificate-based SSL authentication. For information on SSL Encryptions, see Encryption with SSL.

SSL is based on the concepts of public-key cryptography. Although TLS (Transport Layer Security) is functionally a superset of SSL, the names are used interchangeably.

At a high-level, a server which supports SSL needs to have a certificate, a public key, a private key, certificate, key, and security databases. This helps assure message authentication, privacy, and integrity.

The following table describes the SSL authentication support with each client access protocol.

Table 13-4 SSL Authentication Support Matrix
	SSL with MMP	SSL with MMP on Alternate Port	SSL	SSL on Alternate Port
SMTP	Yes	Yes	Yes	Yes
POP	-	Yes	-	Yes
IMAP	Yes	Yes	-	Yes
Messenger Express (HTTP)	Yes (through Messenger Express Multiplexor)	Yes (through Messenger Express Multiplexor)	-	Yes

The SMTP, POP, and IMAP protocols provide a way for the client and server to start communication without SSL, and then switch to it by using an equivalent “start TLS” command. The SMTP, POP, and IMAP servers can also be configured to use SSL on an alternate port, for clients which do not implement “start TLS.”

To authenticate with SSL, the mail client establishes an SSL session with the server and submits the user’s certificate to the server. The server then evaluates if the submitted certificate is genuine. If the certificate is validated, the user is considered authenticated.

If you use SSL for authentication, you need to obtain a server certificate for your Messaging Server. The certificate identifies your server to clients and to other servers. Your server can also have any number of certificates of trusted Certificate Authorities (CAs) that it uses for client authentication.

For more information on SSL, see the Security and Access Control chapter in the Sun Java System Messaging Server Administration Guide:

Planning Message Encryption Strategies

This section describes encryption and privacy solutions. The following topics are covered:

Encryption with SSL

Signed and Encrypted S/MIME

Encryption with SSL

SSL functions as a protocol layer beneath the application layers of IMAP, HTTP, and SMTP. If transmission of messages between a Messaging Server and its clients and between the servers and other servers is encrypted, there is little chance for eavesdropping on the communications. If connecting clients and servers are authenticated, there is little chance for intruders to spoof them.

End-to-end encryption of message transmission requires the use of S/MIME. See Signed and Encrypted S/MIME for more information.

The SSL connection process between client and server using HTTP/SSL (HTTPS) is as follows:


Note	The extra performance overhead in setting up an SSL connection can put a burden on the server. In designing your messaging installation and in analyzing performance, you need to balance security needs against server capacity. If you use SSL for encryption, you can improve server performance by installing a hardware encryption accelerator. An encryption accelerator typically consists of a hardware board, installed permanently in your server machine, and a software driver.

The client initiates contact using HTTPS. The client specifies which secret-key algorithms it can use.

The server sends its certificate for authentication and specifies which secret-key algorithm should be used. It will specify the strongest algorithm which it has in common with the client. If there is no match (for example, client is 40 bit only, server requires 128 bits), the connection will be refused.

If the server has been configured to require client authentication, it will ask the client for its certificate at this point.

The client checks the validity of the server certificate to make sure that it has:

Not expired

A known signed Certification Authority

A valid signature

A host name in the certificate matches the same of the server in the HTTPS request

SSL Ciphers

A cipher is the algorithm used to encrypt and decrypt data in the encryption process. Some ciphers are stronger than others, meaning that a message encrypted by a stronger cipher is more difficult for an unauthorized person to decrypt.

A cipher operates on data by applying a key to the data. Generally, the longer the key the cipher users during encryption, the more difficult it is to decrypt the data without the proper decryption key.

When a client initiates an SSL connection with Messaging Server, the client lets the server know what ciphers and key lengths it prefers to use for encryption. In any encrypted communication, both parties must use the same ciphers. Because there are a number of cipher-and-key combinations in common use, a server should be flexible in its support for encryption. For more information on ciphers, see the chapter on Configuring Security and Access Control in the Sun Java System Messaging Server Administration Guide:

Signed and Encrypted S/MIME

Messages that are signed and encrypted are referred to as Secure/Multipurpose Internet Mail Extensions (S/MIME) messages. S/MIME is a means of securing client to client communication.

With S/MIME, senders can encrypt messages prior to sending them. The recipients can store the encrypted messages after receipt, decrypting them only to read them.

Communications Express Mail now includes the security advantages of S/MIME. Communications Express Mail users who are set up to use S/MIME can exchange signed or encrypted messages with other Communications Express Mail users, and with users of the Microsoft Outlook mail system or other mail clients that support S/MIME. See Requirements for Using S/MIME with Communications Express Mail for more information.

For other clients that support S/MIME, see that client documentation for information on S/MIME configuration.

Previous Contents Index Next
Sun Java System Communications Services 6 2005Q1 Deployment Planning Guide