Sun Java logo     Previous      Contents      Index      Next     

Sun logo
Sun Java System Messaging Server 6 2004Q2 Administration Guide 

Chapter 19
Configuring Security and Access Control

Messaging Server supports a full range of flexible security features that allow you to keep messages from being intercepted, prevent intruders from impersonating your users or administrators, and permit only specific people access to specific parts of your messaging system.

The Messaging Server security architecture is part of the security architecture of Sun Java System servers as a whole. It is built on industry standards and public protocols for maximum interoperability and consistency. To implement Messaging Server security policies, therefore, you will need not only this chapter but several other documents as well. In particular, information in Sun ONE Server Console 5.2 Server Management Guide is required for setting up Messaging Server security.

This chapter contains the following sections:

About Server Security

Server security encompasses a broad set of topics. In most enterprises, ensuring that only authorized people have access to the servers, that passwords or identities are not compromised, that people do not misrepresent themselves as others when communicating, and that communications can be held confidential when necessary are all important requirements for a messaging system.

Perhaps because the security of server communication can be compromised in many ways, there are many approaches to enhancing it. This chapter focuses on setting up encryption, authentication, and access control. It discusses the following security-related Messaging Server topics:

Not all security and access issues related to Messaging Server are treated in this chapter. Security topics that are discussed elsewhere include the following:

There are a large number of documents that cover a variety of security topics. For additional background on the topics mentioned here and for other security-related information, see documentation web site at

About HTTP Security

Messaging Server supports user ID/password authentication, client certificate authentication, and Identity Server. There are some differences, however, in how the protocols handle network connections between client and server.

When a POP, IMAP, or SMTP client logs in to Messaging Server, a connection is made and a session is established. The connection lasts for the duration of the session; that is, from login to logout. When establishing a new connection, the client must reauthenticate to the server.

When an HTTP client logs in to Messaging Server, the server provides a unique session ID to the client. The client uses the session ID to establish multiple connections during a session. The HTTP client need not reauthenticate for each connection; the client need only reauthenticate if the session is dropped and the client wants to establish a new session. (If an HTTP session remains idle for a specified time period, the server will automatically drop the HTTP session and the client is logged out; the default time period is 2 hours.)

The following techniques are used to improve the security of HTTP sessions:

For information about specifying configuration parameters for improved connection performance, see Chapter 5, "Configuring POP, IMAP, and HTTP Services".”

For information on Identity Server see Chapter 6, "Enabling Single Sign-On (SSO)".

Configuring Authentication Mechanisms

An authentication mechanism is a particular method for a client to prove its identity to a server. Messaging Server supports authentication methods defined by the Simple Authentication and Security Layer (SASL) protocol and it supports certificate-based authentication. The SASL mechanisms are described in this section. For more information about certificate-based authentication, see Configuring Encryption and Certificate-Based Authentication.

Messaging Server supports the following SASL authentication methods for password-based authentication.

With a challenge/response authentication mechanism, the server sends a challenge string to the client. The client responds with a hash of that challenge and the user's password. If the client's response matches the server's own hash, the user is authenticated. The hash isn't reversible, so the user's password isn't exposed when sent over the network.


The POP, IMAP, and SMTP services support all SASL mechanisms. The HTTP service supports only the plaintext password mechanism.

Table 19-1 shows some SALS and SASL-related configutil parameters. For the latest and most complete listing of configutil parameters, see the Sun Java System Messaging Server Administration Reference.

Table 19-1  Some SASL and SASL-related configutil Parameters




Boolean to indicate that directory stores plaintext passwords which enables APOP, CRAM-MD5 and DIGEST-MD5.

Default: False


No longer supported or used. See sasl.default.auto_transition.


Boolean. When set and a user provides a plain text password, the password storage format will be transitioned to the default password storage method for the directory server. This can be used to migrate from plaintext passwords to APOP, CRAM-MD5 or DIGEST-MD5.

Default: False


This enables the SASL ANONYMOUS mechanism for use by IMAP.

Default: False


If this is > 0, then disable use of plaintext passwords unless a security layer (SSL or TLS) is activated. This forces users to enable SSL or TLS on their client to login which prevents exposure of their passwords on the network. The MMP has an equivalent option "RestrictPlainPasswords".

NOTE: the 5.2 release of messaging server would actually check the value against the strength of the cipher negotiated by SSL or TLS. That feature has been eliminated to simplify this option and better reflect common-case usage.

Default: 0


A space-separated list of SASL mechanisms to enable. If non-empty, this overrides the sasl.default.ldap.has_plain_passwords option as well as the service.imap.allowanonymouslogin option. This option applies to all protocols (imap, pop, smtp).

Default: False


This is the default search filter used to look up users when one is not specified in the inetDomainSearchFilter for the domain. The syntax is the same as inetDomainSearchFilter (see schema guide).

Default: (&(uid=%U)(objectclass=inetmailuser))


By default, the authentication system looks up the domain in LDAP following the rules for domain lookup (ref. needed) then looks up the user. However, if this option is set to "0" rather than the default value of "1", then the domain lookup does not happen and a search for the user (using the sasl.default.ldap.searchfilter) occurs directly under the LDAP tree specified by local.ugldapbasedn. This is provided for compatibility with legacy single-domain schemas, but use is not recommended for new deployments as even a small company may go through a merger or name change which requires support for multiple domains.

To Configure Access to Plaintext Passwords

To work, the CRAM-MD5, DIGEST-MD5, or APOP SASL authentication methods require access to the users’ plaintext passwords. You need to perform the following steps:

  1. Configure Directory Server to store passwords in cleartext.
  2. Configure Messaging Server so that it knows Directory Server is using cleartext passwords.

To Configure Directory Server to Store Passwords

To enable CRAM-MD5, DIGEST-MD5, or APOP mechanisms, you must configure the Directory Server to store passwords in cleartext as follows:

  1. In Console, open the Directory Server you want to configure.
  2. Click the Configuration tab.
  3. Open Data in the left pane.
  4. Click Passwords in the right pane.
  5. From the Password encryption drop-down list, choose “cleartext”.


    This change only impacts users created in the future. Existing users will have to transition or have their password reset after this change.

To Configure Messaging Server

You can now configure Messaging Server so that it knows the Directory Server is able to retrieve cleartext passwords. This makes it safe for Messaging Server to advertise APOP, CRAM-MD5, and DIGEST-MD5:

configutil -o sasl.default.ldap.has_plain_passwords -v 1

You can disable these challenge/response SASL mechanisms by setting the value to 0 or null (““).


Existing users cannot use APOP, CRAM-MD5, or DIGEST-MD5 until their password is reset or migrated (see to Transition Users).

Note that MMP has an equivalent option: CRAMs.

To Transition Users

You can use configutil to specify information about transitioning users. An example would be if a user password changes or if a client attempts to authenticate with a mechanism for which they do not have a proper entry.

configutil -o sasl.default.auto_transition -v value

For value, you can specify one of the following:

To successfully transition users, you must set up ACIs in the Directory Server that allow Messaging Server write access to the user password attribute. To do this, perform the following steps:

  1. In Console, open the Directory Server you want to configure.
  2. Click the Directory tab.
  3. Select the base suffix for the user/group tree.
  4. From the Object menu, select Access Permissions.
  5. Select (double click) the ACI for “Messaging Server End User Administrator Write Access Rights”.
  6. Click ACI Attributes.
  7. Add the userpassword attribute to the list of existing attributes.
  8. Click OK.

sasl.default.mech_list can be used to enable a list of SASL mechanisms. If non-empty, this overrides the sasl.default.ldap.has_plain_passwords option as well as the service.imap.allowanonymouslogin option. This option applies to all protocols (imap, pop, smtp).

User Password Login

Requiring password submission on the part of users logging into Messaging Server to send or receive mail is a first line of defense against unauthorized access. Messaging Server supports password-based login for its IMAP, POP, HTTP, and SMTP services.

IMAP, POP, and HTTP Password Login

By default, internal users must submit a password to retrieve their messages from Messaging Server. You enable or disable password login separately for POP, IMAP, and HTTP services. For more information about password login for POP IMAP, and HTTP Services, see Password-Based Login.

User passwords can be transmitted from the user’s client software to your server as cleartext or in encrypted form. If both the client and your server are configured to enable SSL and both support encryption of the required strength (as explained in To Enable SSL and Selecting Ciphers), encryption occurs.

User IDs and passwords are stored in your installation’s LDAP user directory. Password security criteria, such as minimum length, are determined by directory policy requirements; they are not part of Messaging Server administration.

Certificate-based login is an alternative to password-based login. It is discussed in this chapter along with the rest of SSL; see To Set Up Certificate-Based Login.

Challenge/response SASL mechanisms are another alternative to plaintext password login.

SMTP Password Login

By default, users need not submit a password when they connect to the SMTP service of Messaging Server to send a message. You can, however, enable password login to SMTP in order to enable authenticated SMTP.

Authenticated SMTP is an extension to the SMTP protocol that allows clients to authenticate to the server. The authentication accompanies the message. The primary use of authenticated SMTP is to allow local users who are travelling (or using their home ISP) to submit mail (relay mail) without creating an open relay that others can abuse. The “AUTH” command is used by the client to authenticate to the server.

For instructions on enabling SMTP password login (and thus Authenticated SMTP), see SMTP Authentication, SASL, and TLS.

You can use Authenticated SMTP with or without SSL encryption.

Configuring Encryption and Certificate-Based Authentication

This section contains the following subsections:

Messaging Server uses the Transport Layer Security (TLS) protocol, otherwise known as the Secure Sockets Layer (SSL) protocol, for encrypted communications and for certificate-based authentication of clients and servers. Messaging Server supports SSL versions 3.0 and 3.1. TLS is fully compatible with SSL and includes all necessary SSL functionality.

For background information on SSL, see the Introduction to SSL (reproduced as an appendix to Managing Servers with iPlanet Console). SSL is based on the concepts of public-key cryptography, described in Introduction to Public-Key Cryptography (also reproduced as an appendix to Managing Servers with iPlanet Console).

If transmission of messages between a Messaging Server and its clients and between the server and other servers is encrypted, there is little chance for eavesdropping on the communications. If connecting clients are authenticated, there is little chance for intruders to impersonate (spoof) them.

SSL functions as a protocol layer beneath the application layers of IMAP4, HTTP, POP3, and SMTP. SMTP and SMTP/SSL use the same port; HTTP and HTTP/SSL require different ports; IMAP and IMAP/SSL, and POP and POP/SSL can use the same port or different ports. SSL acts at a specific stage of message communication, as shown in Figure 19-1, for both outgoing and incoming messages.

Figure 19-1  Encrypted Communications with Messaging Server

This graphic depicts encrypted incoming and outgoing messages.

SSL provides hop-to-hop encryption, but the message is not encrypted on each intermediate server.


To enable encryption for outgoing messages, you must modify the channel definition to include the tls channel keywords, such as maytls, musttls, and so on. For more information, see the Transport Layer Security and the Messaging Server Reference Manual.

Keep in mind that the extra overhead in setting up an SSL connection can put a performance burden on the server. In designing your messaging installation and in analyzing performance, you may need to balance security needs against server capacity.


Because all Sun Java System servers support SSL, and the interface for enabling and configuring SSL through Console is nearly identical across many servers, several of the tasks described in this section are documented more completely in the SSL chapter of Managing Servers with iPlanet Console. For those tasks, this chapter gives summary information only.

Obtaining Certificates

Whether you use SSL for encryption or for authentication, you need to obtain a server certificate for your Messaging Server. The certificate identifies your server to clients and to other servers.

To Manage Internal and External Modules

A server certificate establishes the ownership and validity of a key pair, the numbers used to encrypt and decrypt data. Your server’s certificate and key pair represent your server’s identity. They are stored in a certificate database that can be either internal to the server or on an external, removable hardware card (smartcard).

Sun Java System servers access a key and certificate database using a module conforming to the Public-Key Cryptography System (PKCS) #11 API. The PKCS #11 module for a given hardware device is usually obtained from its supplier and must be installed into the Messaging Server before the Messaging Server can use that device. The pre-installed “Netscape Internal PKCS # 11 Module” supports a single internal software token that uses the certificate database that is internal to the server.

Setting up the server for a certificate involves creating a database for the certificate and its keys and installing a PKCS #11 module. If you do not use an external hardware token, you create an internal database on your server, and you use the internal, default module that is part of Messaging Server. If you do use an external token, you connect a hardware smartcard reader and install its PKCS #11 module.

You can manage PKCS #11 modules, whether internal or external, through Console. To install a PKCS #11 module:

  1. Connect a hardware card reader to the Messaging Server host machine and install drivers.
  2. Use the PKCS #11 Management interface in Console to install the PKCS #11 module for the installed driver.

(For more complete instructions, see the chapter on SSL in Managing Servers with iPlanet Console.)

Installing Hardware Encryption Accelerators     If you use SSL for encryption, you may be able to improve server performance in encrypting and decrypting messages by installing a hardware encryption accelerator. An encryption accelerator typically consists of a hardware board, installed permanently in your server machine, plus a software driver. Messaging Server supports accelerator modules that follow the PKCS #11 API. (They are essentially hardware tokens that do not store their own keys; they use the internal database for that.) You install an accelerator by first installing the hardware and drivers as specified by the manufacturer, and then completing the installation—as with hardware certificate tokens—by installing the PKCS #11 module.

To Request a Server Certificate

You request a server certificate by opening your server in Console and running the Certificate Setup Wizard. You can access the Wizard from the Console menu or from the Messaging Server Encryption tab. Using the Wizard, you perform the following tasks:

  1. Generate a certificate request.
  2. Send the request by email to the certificate authority (CA) that is to issue the certificate.

When the email response from the CA arrives, you save it as a text file and install it using the Certificate Setup Wizard.

(For more complete instructions, see the chapter on SSL in Managing Servers with iPlanet Console.)

To create a certificate database

  1. Log in as or become superuser (root).
  2. Specify the certificate database password for certutil in /opt/SUNWmsgsr/config/sslPassword. For example:
  3. # echo "password" > /opt/SUNWmsgsr/config/sslpassword

    where password is your specific password.

  4. Move to the sbin directory and generate the certificate database (cert7.db) and key database (key3.db). For example:
  5. # cd /opt/SUNWmsg/sbin
    # ./certutil -N -d /opt/SUNWmsgsr/config -f /opt/SUNWmsgsr/config/sslpassword

  6. Generate a default self-signed root Certificate Authority certificate. Example:

    # . /certutil -S -n SampleRootCA -x -t "CTu,CTu,CTu"
    -s "CN=My Sample Root CA," -m 25000
    -o /opt/SUNWmsgsr/config/SampleRootCA.crt
    -d /opt/SUNWmsgsr/config

    -f /opt/SUNWmsgsr/config/sslpassword -z /etc/passwd

  7. Generate a certificate for the host. For example:

    ../certutil -S -n Server-Cert -c SampleRootCA -t "u,u,u"
    -s "," -m 25001
    -o /opt/SUNWmsgsr/config/SampleSSLServer.crt
    -d /opt/SUNWmsgsr/config -f /opt/SUNWmsgsr/config/sslpassword
    -z /etc/passwd

  8. where is the server host name.

  9. Validate the certificates. For example:
  10. # ./certutil -V -u V -n SampleRootCA -d /var/opt/SUNWmsgsr/config
    # ./certutil -V -u V -n Server-Cert -d /opt/SUNWmsgsr/config

  11. List the certificates. For example:
  12. # ./certutil -L -d /opt/SUNWmsgsr/config
    # ./certutil -L -n Server-Cert -d /opt/SUNWmsgsr/config

  13. Use modutil to list the available security modules (secmod.db). For example:
  14. # . ./modutil -list -dbdir /opt/SUNWmsgsr/config

  15. Change the owner of the alias file to icsuser and icsgroup (or the user and group identity under which Calendar Server will run). For example:
  16. find /opt/SUNWmsgsr/config/cert7.db -exec chown mailsrv {} \;
    find /opt/SUNWmsgsr/config/key3.db -exec chown mailsrv {} \;
    find /opt/SUNWmsgsr/config/cert7.db -exec chgrp mail {} \;
    find /opt/SUNWmsgsr/config/key3.db -exec chgrp mail {} \;

    where mailsrv is the mail server UID and mail is the mail server GID.

  17. Restart the messaging services to enable the SSL.

To Install the Certificate

Installing is a separate process from requesting. Once the email response to your request for a certificate has arrived from the CA and been saved as a text file, run the Certificate Setup Wizard once more to install the file as a certificate:

  1. Specify that you are installing a certificate that you have already obtained.
  2. Paste the text of your certificate into a field when prompted to do so.
  3. Change the certificate nickname from server-cert to Server-Cert.
  4. If you do not want to change the certificate nickname, you can change what the system wants the certificate nickname to be by setting the configutil parameter encryption.rsa.nssslpersonalityssl.

(For more complete instructions, see the chapter on SSL in Managing Servers with iPlanet Console.)


This is also the process you follow to install a CA certificate (described next), which your server uses to determine whether to trust the certificates presented by clients.

To Install Certificates of Trusted CAs

You also use the Certificate Setup Wizard to install the certificates of certificate authorities. A CA certificate validates the identity of the CA itself. Your server uses these CA certificates in the process of authenticating clients and other servers.

If, for example, you set up your enterprise for certificate-based client authentication in addition to password-based authentication (see “Setting Up Certificate-Based Login” on page 157), you need to install the CA certificates of all CAs that are trusted to issue the certificates that your clients may present. These CAs may be internal to your organization or they may be external, representing commercial or governmental authorities or other enterprises. (For more details on the use of CA certificates for authentication, see Introduction to Public-Key Cryptography in Managing Servers with iPlanet Console.)

When installed, Messaging Server initially contains CA certificates for several commercial CAs. If you need to add other commercial CAs or if your enterprise is developing its own CA for internal use (using Sun Java System Certificate Server), you need to obtain and install additional CA certificates.


The CA certificates automatically provided with Messaging Server are not initially marked as trusted for client certificates. You need to edit the trust settings if you want to trust client certificates issued by these CAs. For instructions, see “Managing Certificates and Trusted CAs” on page 153.

To request and install a new CA certificate, you:

  1. Contact the certificate authority (possibly through the Web or by email) and download its CA certificate.
  2. Save the received text of the certificate as a text file.
  3. Use the Certificate Setup Wizard, as described in the previous section, to install the certificate.

For more complete instructions, see the chapter on SSL in Managing Servers with iPlanet Console.

Managing Certificates and Trusted CAs

Your server can have any number of certificates of trusted CAs that it uses for authentication of clients.

You can view, edit the trust settings of, or delete any of the certificates installed in your Messaging Server by opening your server in Console and choosing the Certificate Management Command in the Console menu. For instructions, see the chapter on SSL in Managing Servers with iPlanet Console.

Creating a Password File

On any Sun Java System server, when you use the Certificate Setup Wizard to request a certificate, the wizard creates a key pair to be stored in either the internal module’s database or in an external database (on a smartcard). The wizard then prompts you for a password, which it uses to encrypt the private key. Only that same password can later be used to decrypt the key. The wizard does not retain the password nor store it anywhere.

On most Sun Java System servers for which SSL is enabled, the administrator is prompted at startup to supply the password required to decrypt the key pair. On Messaging Server, however, to alleviate the inconvenience of having to enter the password multiple times (it is needed by at least three server processes), and to facilitate unattended server restarts, the password is read from a password file.

The password file is named sslpassword.conf and is in the directory msg_svr_base/config/. Entries in the file are individual lines with the format


where moduleName is the name of the (internal or external) PKCS #11 module to be used, and password is the password that decrypts that module’s key pair. The password is stored as clear (unencrypted) text.

Messaging Server provides a default version of the password file, with the following single entry (for the internal module and default password):

Internal (Software) Token:netscape!

If you specify anything but the default password when you install an internal certificate, you need to edit the above line of the password file to reflect the password you specified. If you install an external module, you need to add a new line to the file, containing the module name and the password you specified for it.


Because the administrator is not prompted for the module password at server startup, it is especially important that you ensure proper administrator access control to the server and proper physical security of the server host machine and its backups.

To Enable SSL and Selecting Ciphers

You can use Console to enable SSL and to select the set of encryption ciphers that Messaging Server can use in its encrypted communications with clients.

About Ciphers

A cipher is the algorithm used to encrypt and decrypt data in the encryption process. Some ciphers are stronger than others, meaning that a message they have scrambled is more difficult for an unauthorized person to unscramble.

A cipher operates on data by applying a key—a long number—to the data. Generally, the longer the key the cipher uses during encryption, the harder it is to decrypt the data without the proper decryption key.

When a client initiates an SSL connection with a Messaging Server, the client lets the server know what ciphers and key lengths it prefers to use for encryption. In any encrypted communication, both parties must use the same ciphers. Because there are a number of cipher-and-key combinations in common use, a server should be flexible in its support for encryption. Messaging Server can support up to 6 combinations of cipher and key length.

Table 6.1 lists the ciphers that Messaging Server supports for use with SSL 3.0. The table summarizes information that is available in more detail in the Introduction to SSL section of Managing Servers with iPlanet Console.

Table 19-2   SSL Ciphers for Messaging Server



RC4 with 128-bit encryption and MD5 message authentication

The fastest encryption cipher (by RSA) and a very high-strength combination of cipher and encryption key.

Triple DES with 168-bit encryption and SHA message authentication

A slower encryption cipher (a U.S. government-standard) but the highest-strength combination of cipher and encryption key.

DES with 56-bit encryption and SHA message authentication

A slower encryption cipher (a U.S. government-standard) and a moderate-strength combination of cipher and encryption key.

RC4 with 40-bit encryption and MD5 message authentication

The fastest encryption cipher (by RSA) and a lower-strength combination of cipher and encryption key.

RC2 with 40-bit encryption and MD5 message authentication

A slower encryption cipher (by RSA) and a lower-strength combination of cipher and encryption key.

No encryption, only MD5 message authentication

No encryption; use of a message digest for authentication alone.

Unless you have a compelling reason for not using a specific cipher, you should support them all. However, note that export laws restrict the use of certain encryption ciphers in certain countries. Also, some client software produced before the relaxation of United States Export Control laws cannot use the higher strength encryption. Be aware that while the 40-bit ciphers might hinder the casual eavesdropper, they are not secure and therefore will not stop a motivated attack.

To enable SSL and select encryption ciphers, follow these command line steps:

To enable or disable SSL:

configutil -o nsserversecurity -v [ on | off ]

To enable or disable RSA ciphers:

configutil -o encryption.rsa.nssslactivation -v [ on | off ]

To specify a token:

configutil -o encryption.rsa.nsssltoken -v tokenname

To specify a certificate:

configutil -o encryption.rsa.nssslpersonalityssl -v certname

Note that if you enable RSA ciphers, you must also specify a token and a certificate.

To choose a cipher preference:

configutil -o encryption.nsssl3ciphers -v cipherlist

where cipherlist is a comma-separated list of ciphers.


To enable SSL encryption for outgoing messages, you must modify the channel definition to include the tls channel keywords, such as maytls, musttls, and so on. For more information, see the Transport Layer Security and the Messaging Server Reference Manual.

To Set Up Certificate-Based Login

In addition to password-based authentication, Sun Java System servers support authentication of users through examination of their digital certificates. In certificate-based authentication, the client establishes an SSL session with the server and submits the user’s certificate to the server. The server then evaluates whether the submitted certificate is genuine. If the certificate is validated, the user is considered authenticated.

To set up your Messaging Server for certificate-based login:

  1. Obtain a server certificate for your server. (For details, see Obtaining Certificates.)
  2. Run the Certificate Setup Wizard to install the certificates of any trusted certificate authorities that will issue certificates to the users your server will authenticate. (For details, see To Install Certificates of Trusted CAs.)
  3. Note that as long as there is at least one trusted CA in the server’s database, the server requests a client certificate from each connecting client.

  4. Turn on SSL. (For details, see To Enable SSL and Selecting Ciphers.)
  5. (Optional) Edit your server’s certmap.conf file so that the server appropriately searches the LDAP user directory based on information in the submitted certificates.
  6. Editing the certmap.conf file is not necessary if the email address in your users’ certificates matches the email address in your users’ directory entries, and you do not need to optimize searches or validate the submitted certificate against a certificate in the user entry.

    For details of the format of certmap.conf and the changes you can make, see the SSL chapter of Managing Servers with iPlanet Console.

Once you have taken these steps, when a client establishes an SSL session so that the user can log in to IMAP or HTTP, the Messaging Server requests the user’s certificate from the client. If the certificate submitted by the client has been issued by a CA that the server has established as trusted, and if the identity in the certificate matches an entry in the user directory, the user is authenticated and access is granted (depending on access-control rules governing that user).

There is no need to disallow password-based login to enable certificate-based login. If password-based login is allowed (which is the default state), and if you have performed the tasks described in this section, both password-based and certificate-based login are supported. In that case, if the client establishes an SSL session and supplies a certificate, certificate-based login is used. If the client does not use SSL or does not supply a certificate, the server requests a password.

How to Optimize SSL Performance Using the SMTP Proxy

Most sites should not use the SMTP proxy as it adds additional latency to the SMTP protocol. However, a large-scale site which makes heavy use of SSL to protect SMTP connections may wish to maximize their investment in SSL accelerator hardware by performing all SSL operations for all protocols on a server which does nothing other than SSL and proxy. The SMTP proxy allows SSL to be processed by a front end proxy server while the mail queues are on a separate MTA machine. This way hardware optimized for each task can be separately configured and purchased.

See To Install the SMTP Proxy for instructions on how to install the SMTP Proxy.

Configuring Administrator Access to Messaging Server

This section mostly pertains to the Sun Java System LDAP Schema v. 1. This section contains the following subsections:

This section describes how to control the ways in which server administrators can gain access to Messaging Server. Administrative access to a given Messaging Server and to specific Messaging Server tasks occurs within the context of delegated server administration.

Delegated server administration is a feature of most Sun Java System servers; it refers to the capability of an administrator to provide other administrators with selective access to individual servers and server features. This chapter briefly summarizes delegated server tasks. For more detailed information, see the chapter on delegating server administration in Managing Servers with iPlanet Console.

Hierarchy of Delegated Administration

When you install the first Sun Java System server on your network, the installation program automatically creates a group in the LDAP user directory called the Configuration Administrators group. By default, the members of the Configuration Administrators group have unrestricted access to all hosts and servers on your network.

The Configuration Administrators group is at the top of an access hierarchy, such as the following, that you can create to implement delegated administration (if Sun Java System LDAP Schema v. 1 is used) for Messaging Server:

  1. Configuration administrator. The “super user” for the network of Sun Java System servers. Has complete access to all resources.
  2. Server administrator. A domain administrator might create groups to administer each type of server. For example, a Messaging Administrators group might be created to administer all Messaging Servers in an administrative domain or across the whole network. Members of that group have access to all Messaging Servers (but no other servers) in that administrative domain.
  3. Task administrator. Finally, any of the above administrators might create a group, or designate an individual user, with restricted access to a single Messaging Server or a set of Messaging Servers. Such a task administrator is permitted to perform only specific, limited server tasks (such as starting or stopping the server only, or accessing logs of a given service).

Console provides convenient interfaces that allow an administrator to perform the following tasks:

To Provide Access to the Server as a Whole

To give a user or group permission to access a given instance of Messaging Server, you:

  1. Log in to Console as an administrator with access to the Messaging Server you want to provide access to.
  2. Select that server in the Console window.
  3. From the Console menu, choose Object, then choose Set Access Permissions.

  4. Add or edit the list of users and groups with access to the server.

(For more complete instructions, see the chapter on delegating server administration in Managing Servers with iPlanet Console.)

Once you have set up the list of individuals and groups that have access to the particular Messaging Server, you can then use ACIs, as described next, to delegate specific server tasks to specific people or groups on that list.

To Restrict Access to Specific Tasks

An administrator typically connects to a server to perform one or more administrative tasks. Common administrative tasks are listed in the Messaging Server Tasks form in Console.

By default, access to a particular Messaging Server means access to all of its tasks. However, each task in the Task form can have an attached set of access-control instructions (ACIs). The server consults those ACIs before giving a connected user (who must already be a user with access permissions to the server as a whole) access to any of the tasks. In fact, the server displays in the Tasks form only those tasks to which the user has permission.

If you have access to a Messaging Server, you can create or edit ACIs on any of the tasks (that is, on any of the tasks to which you have access), and thus restrict the access that other users or groups can have to them.

To restrict the task access that a connected user or group can have, you:

  1. Log in to the Console as an administrator with access to the Messaging Server you want to provide restricted access to.
  2. Open the server and select a task in the server’s Tasks form by clicking on the Task text.
  3. From the Edit menu, choose Set Access Permissions, and add or edit the list of access rules to give a user or group the kind of access you want them to have.
  4. Repeat the process for other tasks, as appropriate.

ACIs and how to create them are described more fully in the chapter on delegating server administration in Managing Servers with iPlanet Console.

Configuring Client Access to POP, IMAP, and HTTP Services

This section contains the following subsections:

Messaging Server supports sophisticated access control on a service-by-service basis for its IMAP, POP, and HTTP services so that you can exercise far-ranging and fine-grained control over which clients can gain access to your server.

If you are managing messaging services for a large enterprise or an Internet service provider, these capabilities can help you to exclude spammers and DNS spoofers from your system and improve the general security of your network. For control of unsolicited bulk email specifically, see also Chapter 17, "Mail Filtering and Access Control".”


If controlling access by IP address is not an important issue for your enterprise, you do not have to create any of the filters described in this section. If minimal access control is all you need, see the section Mostly Allowing for instructions on setting it up.

How Client Access Filters Work

The Messaging Server access-control facility is a program that listens at the same port as the TCP daemon it serves; it uses access filters to verify client identity, and it gives the client access to the daemon if the client passes the filtering process.

As part of its processing, the Messaging Server TCP client access-control system performs (when necessary) the following analyses of the socket end-point addresses:

The system compares this information against access-control statements called filters to decide whether to grant or deny access. For each service, separate sets of Allow filters and Deny filters control access. Allow filters explicitly grant access; Deny filters explicitly forbid access.

When a client requests access to a service, the access-control system compares the client’s address or name information to each of that service’s filters—in order—using these criteria:

The filter syntax described here is flexible enough that you should be able to implement many different kinds of access-control policies in a simple and straightforward manner. You can use both Allow filters and Deny filters in any combination, even though you can probably implement most policies by using almost exclusively Allows or almost exclusively Denies.

The following sections describe filter syntax in detail and give usage examples. The section To Create Access Filters for Services gives the procedure for creating access filters.

Filter Syntax

Filter statements contain both service information and client information. The service information can include the name of the service, names of hosts, and addresses of hosts. The client information can include host names, host addresses, and user names. Both the server and client information can include wildcard names or patterns.

The very simplest form of a filter is:

service: hostSpec

where service is the name of the service (such as smtp, pop, imap, or http) and hostSpec is the host name, IP address, or wildcard name or pattern that represents the client requesting access. When a filter is processed, if the client seeking access matches client, access is either allowed or denied (depending on which type of filter this is) to the service specified by service. Here are some examples:


pop: ALL

http: ALL

If these are Allow filters, the first one grants the host access to the IMAP service, and the second and third grant all clients access to the POP and HTTP services, respectively. If they are Deny filters, they deny those clients access to those services. (For descriptions of wildcard names such as ALL, see Wildcard Names.)

Either the server or the client information in a filter can be somewhat more complex than this, in which case the filter has the more general form of:

serviceSpec: clientSpec

where serviceSpec can be either service or service@hostSpec, and clientSpec can be either hostSpec or user@hostSpec. user is the user name (or a wildcard name) associated with the client host seeking access. Here are two examples: ALL


If these are Deny filters, the first filter denies all clients access to the SMTP service on the host The second filter denies the user srashad at the host access to the IMAP service. (For more information on when to use these expanded server and client specifications, see Server-Host Specification and Client User-Name Specification.)

Finally, at its most general, a filter has the form:

serviceList: clientList

where serviceList consists of one or more serviceSpec entries, and clientList consists of one or more clientSpec entries. Individual entries within serviceList and clientList are separated by blanks and/or commas.

In this case, when a filter is processed, if the client seeking access matches any of the clientSpec entries in clientList, then access is either allowed or denied (depending on which type of filter this is) to all the services specified in serviceList. Here is an example:

pop, imap, http:

If this is an Allow filter, it grants access to POP, IMAP, and HTTP services to all clients in either of the domains and For information on using a leading dot or other pattern to specify domains or subnet, see Wildcard Patterns.

You can also use the following syntax:

"+" or "-" serviceList:*$next_rule

+ (allow filter) means the daemon list services are being granted to the client list.

- (deny filter) means the services are being denied to the client list.

* (wildcard filter) allow all clients to used these services.

$ separates rules.

This example enables multiple services on all clients.


This example shows multiple rules, but each rule is simplified to have only one service name and uses wildcards for the client list. (This is the most commonly used method of specifying access control in LDIF files.)


An example of how to disallow all services for a user is:


Wildcard Names

You can use the following wildcard names to represent service names, host names or addresses, or user names:

Table 19-3   Wildcard Names for Service Filters

Wildcard Name


ALL, *

The universal wildcard. Matches all names.


Matches any local host (one whose name does not contain a dot character). However, if your installation uses only canonical names, even local host names will contain dots and thus will not match this wildcard.


Matches any user whose name is unknown, or any host whose name or address is unknown.

Use this wildcard name carefully:

Host names may be unavailable due to temporary DNS server problems—in which case all filters that use UNKNOWN will match all client hosts.

A network address is unavailable when the software cannot identify the type of network it is communicating with—in which case all filters that use UNKNOWN will match all client hosts on that network.


Matches any user whose name is known, or any host whose name and address are known.

Use this wildcard name carefully:

Host names may be unavailable due to temporary DNS server problems—in which case all filters that use KNOWN will fail for all client hosts.

A network address is unavailable when the software cannot identify the type of network it is communicating with—in which case all filters that use KNOWN will fail for all client hosts on that network.


Matches any host whose DNS name does not match its own IP address.

Wildcard Patterns

You can use the following patterns in service or client addresses:

EXCEPT Operator

The access-control system supports a single operator. You can use the EXCEPT operator to create exceptions to matching names or patterns when you have multiple entries in either serviceList or clientList. For example, the expression:

list1 EXCEPT list2

means that anything that matches list1 is matched, unless it also matches list2.

Here is an example:


If this were a Deny filter, it would deny access to all services to all clients except those on the host machine

EXCEPT clauses can be nested. The expression:

list1 EXCEPT list2 EXCEPT list3

is evaluated as if it were:

list1 EXCEPT (list2 EXCEPT list3)

Server-Host Specification

You can further identify the specific service being requested in a filter by including server host name or address information in the serviceSpec entry. In that case the entry has the form:


You might want to use this feature when your Messaging Server host machine is set up for multiple internet addresses with different internet host names. If you are a service provider, you can use this facility to host multiple domains, with different access-control rules, on a single server instance.

Client User-Name Specification

For client host machines that support the identd service as described in RFC 1413, you can further identify the specific client requesting service by including the client’s user name in the clientSpec entry in a filter. In that case the entry has the form:


where user is the user name as returned by the client’s identd service (or a wildcard name).

Specifying client user names in a filter can be useful, but keep these caveats in mind:

The user-name lookup capability can in some cases help you guard against attack from unauthorized users on the client’s host. It is possible in some TCP/IP implementations, for example, for intruders to use rsh (remote shell service) to impersonate trusted client hosts. If the client host supports the ident service, you can use user-name lookups to detect such attacks.

Filter Examples

The examples in this section show a variety of approaches to controlling access. In studying the examples, keep in mind that Allow filters are processed before Deny filters, the search terminates when a match is found, and access is granted when no match is found at all.

The examples listed here use host and domain names rather than IP addresses. Remember that you can include address and netmask information in filters, which can improve reliability in the case of name-service failure.

Mostly Denying

In this case, access is denied by default. Only explicitly authorized hosts are permitted access.

The default policy (no access) is implemented with a single, trivial deny file:


This filter denies all service to all clients that have not been explicitly granted access by an Allow filter. The Allow filters, then, might be something like these:

ALL: LOCAL @netgroup1


The first rule permits access from all hosts in the local domain (that is, all hosts with no dot in their host name) and from members of the group netgroup1. The second rule uses a leading-dot wildcard pattern to permit access from all hosts in the domain, with the exception of the host

Mostly Allowing

In this case, access is granted by default. Only explicitly specified hosts are denied access.

The default policy (access granted) makes Allow filters unnecessary. The unwanted clients are listed explicitly in Deny filters such as these:



The first filter denies all services to a particular host and to a specific domain. The second filter permits nothing but POP access from a particular host and from a specific domain.

Denying Access to Spoofed Domains

You can use the DNSSPOOFER wildcard name in a filter to detect host-name spoofing. When you specify DNSSPOOFER, the access-control system performs forward or reverse DNS lookups to verify that the client’s presented host name matches its actual IP address. Here is an example for a Deny filter:


This filter denies all services to all remote hosts whose IP addresses don’t match their DNS host names.

Controlling Access to Virtual Domains

If your messaging installation uses virtual domains, in which a single server instance is associated with multiple IP addresses and domain names, you can control access to each virtual domain through a combination of Allow and Deny filters. For example, you can use Allow filters like:


coupled with a Deny filter like:


Each Allow filter permits only hosts within domainN to connect to the service whose IP address corresponds to All other connections are denied.

To Create Access Filters for Services

You can create Allow and Deny filters for the IMAP, POP, or HTTP services. You can also create them for SMTP services, but they have little value because they only apply to authenticated SMTP sessions. See Chapter 17, "Mail Filtering and Access Control"” for how to control access to unauthenticated SMTP sessions.

Console     To create filters by using Console, follow these steps:

  1. In Console, open the Messaging Server that you want to create access filters for.
  2. Click the Configuration tab.
  3. Open the Services folder in the left pane and select IMAP, POP, or HTTP beneath the Services folder.
  4. Click the Access tab in the right pane.
  5. The Allow and Deny fields in the tab show the existing Allow and Deny filters for that service. Each line in the field represents one filter. For either of the fields, you can specify the following actions:

    1. Click Add to create a new filter. An Allow Filter window or Deny filter window opens; enter the text of the new filter into the window, and click OK.
    2. Select a filter and click Edit to modify the filter. An Allow Filter window or Deny filter window opens; edit the text of the filter displayed in the window, and click OK.
    3. Select a filter and click Delete to remove the filter.

Note that if you need to rearrange the order of Allow or Deny filters, you can do so by performing a series of Delete and Add actions.

For a specification of filter syntax and a variety of examples, see Filter Syntax. For additional examples, see Filter Examples.

Command Line     You can also specify access and deny filters at the command line as follows:

To create or edit access filters for services:

configutil -o service.service.domainallowed -v filter

where service is pop, imap, or http and filter follows the syntax rules described in Filter Syntax.

To create or edit deny filters for services:

configutil -o service.service.domainnotallowed -v filter

where service is pop, imap, or http and filter follows the syntax rules described in Filter Syntax.

To Create Access Filters for HTTP Proxy Authentication

Any store administrator can proxy authenticate to any service. (For more information about store administrators, see Specifying Administrator Access to the Store.) For the HTTP service only, any user can proxy authenticate to the service if their client host is granted access via a proxy authentication access filter.

Proxy authentication allows other services, such as a portal site, to authenticate users and pass the authentication credentials to the HTTP login service. For example, assume a portal site offers several services, one of which is Messenger Express web-based email. By using the HTTP proxy authentication feature, end users need only authenticate once to the portal service; they need not authenticate again to access their email. The portal site must configure a login server that acts as the interface between the client and the service. To help configure the login server for Messenger Express authentication, Sun Java System offers an authentication SDK for Messenger Express.

This section describes how to create allow filters to permit HTTP proxy authentication by IP address. This section does not describe how to set up your login server or how to use the Messenger Express authentication SDK. For more information about setting up your login server for Messenger Express and using the authentication SDK, contact your Sun Java System representative.

Console     To create access filters for proxy authentication to the HTTP service:

  1. In Console, open the Messaging Server that you want to create access filters for.
  2. Click the Configuration tab.
  3. Open the Services folder in the left pane and select HTTP beneath the Services folder.
  4. Click the Proxy tab in the right pane.
  5. The Allow field in the tab shows the existing Allow filters for proxy authentication.

  6. To create a new filter, click Add.
  7. An Allow filter window opens. Enter the text of the new filter into the window and click OK.

  8. To edit an existing filter, select the filter and click Edit.
  9. An Allow filter window opens. Edit the text of the filter display in the window, and click OK.

  10. To delete an existing filter, select a field from the Allow field, and click Delete.
  11. When you are finished making changes to the Proxy tab, click Save.

For more information about allow filter syntax, see Filter Syntax.

Command Line     You can also specify access filters for proxy authentication to the HTTP service at the command line as follows:

configutil -o service.service.proxydomainallowed -v filter

where filter follows the syntax rules described in Filter Syntax.

Enabling POP Before SMTP

SMTP Authentication, or SMTP Auth (RFC 2554) is the preferred method of providing SMTP relay server security. SMTP Auth allows only authenticated users to send mail through the MTA. However, some legacy clients only provide support for POP before SMTP. If this is the case for your system, you may enable POP before SMTP as described below. If possible, however, encourage your users to upgrade their POP clients rather than using POP before SMTP. Once POP before SMTP is deployed at a site users will become dependent on clients which fail to follow Internet security standards, putting end users at greater risk of hacking and slowing your site with the unavoidable performance penalty because of the necessity of having to track and coordinate IP addresses of recent successful POP sessions.

The Messaging Server implementation of POP before SMTP is completely different from either SIMS or Netscape Messaging Server. POP before SMTP is supported by configuring a Messaging Multiplexor (MMP) to have both a POP and SMTP proxy. When an SMTP client connects to the SMTP proxy, the proxy will check an in-memory cache of recent POP authentications. If a POP authentication from the same client IP address is found, the SMTP proxy will inform the SMTP server that it should permit messages directed to both local and non-local recipients.

To Install the SMTP Proxy

  1. Install a Messaging Multiplexor (MMP) as described in the Sun ONE Messaging Server Installation Guide.
  2. Enable the SMTP proxy on the MMP.
  3. Add the string:


    to the ServiceList option in the msg_svr_base/config/AService.cfg file. That option is one long line and can’t contain line breaks.


    When the MMP is upgraded, four new files which correspond to the existing four configuration files for the MMP. The new files are:

    AService-def.cfg, ImapProxyAService-def.cfg, PopProxyAService-def.cfg, and SmtpProxyAService-def.cfg

    These files are created by the installer, the four configuration files described in the docs are not created or affected by the install process. When the MMP starts up, it will look for the normal configuration file (as currently documented). If it doesn’t find the normal configuration file, it will attempt to copy the respective *AService-def.cfg file to the corresponding *AService.cfg file name.

  4. Set the PROXY_PASSWORD option in the SMTP channel option file tcp_local_option at each SMTP relay server.
  5. When the SMTP proxy connects to the SMTP server, it has to inform the SMTP server of the real client IP address and other connection information so that the SMTP server can correctly apply relay blocking and other security policy (including POP before SMTP authorization). This is a security sensitive operation and must be authenticated. The proxy password configured on both the MMP SMTP Proxy and the SMTP server assures that a third party cannot abuse the facility.

    Example: PROXY_PASSWORD=A_Password

  6. Make sure the IP address that the MMP uses to connect to the SMTP server is not treated as “internal” by the INTERNAL_IP mapping table.
  7. See the To Add SMTP Relaying of the Chapter 17, "Mail Filtering and Access Control"for information about the INTERNAL_IP mapping table.

  8. Configure the SMTP proxy to Support POP before SMTP.
    1. Edit the msg_svr_base/config/SmtpProxyAService.cfg configuration file.
    2. The following SMTP proxy options operate identically to the same options for the IMAP and POP proxies (see the appendix entitled, “Installing the Messaging Multiplexor” in the Messaging Server Installation Guide and the description of these options in the Encryption (SSL) Option section in the Messaging Server Reference Manual or more information):

      LdapURL, LogDir, LogLevel, BindDN, BindPass, Timeout, Banner, SSLEnable, SSLSecmodFile, SSLCertFile, SSLKeyFile, SSLKeyPasswdFile, SSLCipherSpecs, SSLCertNicknames, SSLCacheDir, SSLPorts, CertMapFile, CertmapDN, ConnLimits, TCPAccess

      Other MMP options not listed above (including the BacksidePort option) do not currently apply to the SMTP Proxy.

      Add the following five options:

      SmtpRelays is a space-separated list of SMTP relay server hostnames (with optional port) to use for round-robin relay. These relays must support the XPROXYEHLO extension. This option is mandatory with no default. Example: default:SmtpRelays  manatee:485 gonzo mothra

      SmtpProxyPassword is a password used to authorize source channel changes on the SMTP relay servers. This option is mandatory with no default and must match the PROXY_PASSWORD option on the SMTP servers.
      Example: default:SmtpProxyPassword  A_Password

      EhloKeywords option provides a list of EHLO extension keywords for the proxy to pass through to the client, in addition to the default set. The MMP will remove any unrecognized EHLO keywords from the EHLO list returned by an SMTP relay. EhloKeywords specifies additional EHLO keywords which should not be removed from the list. The default is empty, but the SMTP proxy will support the following keywords, so there is no need to list them in this option: 8BITMIME, PIPELINING, DSN, ENHANCEDSTATUSCODES, EXPN, HELP, XLOOP, ETRN, SIZE, STARTTLS, AUTH

      The following is an example that might be used by a site which uses the rarely used “TURN” extension:
      Example: default:EhloKeywords   TURN

      PopBeforeSmtpKludgeChannel option is set to the name of an MTA channel to use for POP before SMTP authorized connections. The default is empty and the typical setting for users who want to enable POP before SMTP is tcp_intranet. This option is not required for optimizing SSL performance (see How to Optimize SSL Performance Using the SMTP Proxy).
      Example: default:PopBeforeSmtpKludgeChannel  tcp_intranet

      ClientLookup option defaults to no. If set to yes, a DNS reverse lookup on the client IP address will be performed unconditionally so the SMTP relay server doesn’t have to do that work This option may be set on a per hosted domain domain basis.
      Example: default:ClientLookup  yes

    3. Set the PreAuth option and the AuthServiceTTL option in PopProxyAService.cfg configuration file. This option is not required for optimizing SSL performance. (See How to Optimize SSL Performance Using the SMTP Proxy.)


      AuthServiceTTL must not be set in IMAP or SMTP proxy configuration files in order for POP before SMTP to work.

    4. These options specify how long in seconds a user is authorized to submit mail after a POP authentication. The typical setting is 900 to 1800 (15-30 minutes).


      default:PreAuth    yes
      default:AuthServiceTTL    900

    5. You may optionally specify how many seconds the MMP will wait for an SMTP Relay to respond before trying the next one in the list.
    6. The default is 10 (seconds). If a connection to an SMTP Relay fails, the MMP will avoid trying that relay for a number of minutes equivalent to the failover time-out (so if the failover time-out is 10 seconds, and a relay fails, the MMP won’t try that relay again for 10 minutes).
      Example: default:FailoverTimeout  10

Configuring Client Access to SMTP Services

For information about configuring client access to SMTP services, see Chapter 17, "Mail Filtering and Access Control".”

Previous      Contents      Index      Next     

Copyright 2004 Sun Microsystems, Inc. All rights reserved.