test

Sun Java System Messaging Server 6 2005Q4 Administration Guide

Configuring Encryption and Certificate-Based Authentication

This section contains the following subsections:

Messaging Server uses the Transport Layer Security (TLS) protocol, otherwise known as the Secure Sockets Layer (SSL) protocol, for encrypted communications and for certificate-based authentication of clients and servers. Messaging Server supports SSL versions 3.0 and 3.1. TLS is fully compatible with SSL and includes all necessary SSL functionality.

For background information on SSL, see the Introduction to SSL (reproduced as an appendix to Managing Servers with iPlanet Console). SSL is based on the concepts of public-key cryptography, described in Introduction to Public-Key Cryptography (also reproduced as an appendix to Managing Servers with iPlanet Console).

If transmission of messages between a Messaging Server and its clients and between the server and other servers is encrypted, there is little chance for eavesdropping on the communications. If connecting clients are authenticated, there is little chance for intruders to impersonate (spoof) them.

SSL functions as a protocol layer beneath the application layers of IMAP4, HTTP, POP3, and SMTP. SMTP and SMTP/SSL use the same port; HTTP and HTTP/SSL require different ports; IMAP and IMAP/SSL, and POP and POP/SSL can use the same port or different ports. SSL acts at a specific stage of message communication, as shown in Figure 19–1, for both outgoing and incoming messages.

Figure 19–1 Encrypted Communications with Messaging Server

Graphic depicts encrypted incoming and outgoing messages.

SSL provides hop-to-hop encryption, but the message is not encrypted on each intermediate server.

Note –

To enable encryption for outgoing messages, you must modify the channel definition to include the tls channel keywords, such as maytls, musttls, and so on. For more information, see the Transport Layer Security Manual.

Keep in mind that the extra overhead in setting up an SSL connection can put a performance burden on the server. In designing your messaging installation and in analyzing performance, you may need to balance security needs against server capacity.

Note –

Because all Sun Java System servers support SSL, and the interface for enabling and configuring SSL through Console is nearly identical across many servers, several of the tasks described in this section are documented more completely in the SSL chapter of Managing Servers with iPlanet Console. For those tasks, this chapter gives summary information only.

Obtaining Certificates through the Administration Console

Whether you use SSL for encryption or for authentication, you need to obtain a server certificate for your Messaging Server. The certificate identifies your server to clients and to other servers. If you want to obtain certificates through the Administration Console, follow the steps in this section. If you want to create self-signed certificates in command-line mode, see To Create Self-signed Certificates

To Manage Internal and External Modules

A server certificate establishes the ownership and validity of a key pair, the numbers used to encrypt and decrypt data. Your server’s certificate and key pair represent your server’s identity. They are stored in a certificate database that can be either internal to the server or on an external, removable hardware card (smartcard).

Sun Java System servers access a key and certificate database using a module conforming to the Public-Key Cryptography System (PKCS) #11 API. The PKCS #11 module for a given hardware device is usually obtained from its supplier and must be installed into the Messaging Server before the Messaging Server can use that device. The pre-installed “Netscape Internal PKCS # 11 Module” supports a single internal software token that uses the certificate database that is internal to the server.

Setting up the server for a certificate involves creating a database for the certificate and its keys and installing a PKCS #11 module. If you do not use an external hardware token, you create an internal database on your server, and you use the internal, default module that is part of Messaging Server. If you do use an external token, you connect a hardware smartcard reader and install its PKCS #11 module.

You can manage PKCS #11 modules, whether internal or external, through Console. To install a PKCS #11 module:

  1. Connect a hardware card reader to the Messaging Server host machine and install drivers.

  2. Use the PKCS #11 Management interface in Console to install the PKCS #11 module for the installed driver.

(For more complete instructions, see the chapter on SSL in Managing Servers with iPlanet Console.)

Installing Hardware Encryption Accelerators. If you use SSL for encryption, you may be able to improve server performance in encrypting and decrypting messages by installing a hardware encryption accelerator. An encryption accelerator typically consists of a hardware board, installed permanently in your server machine, plus a software driver. Messaging Server supports accelerator modules that follow the PKCS #11 API. (They are essentially hardware tokens that do not store their own keys; they use the internal database for that.) You install an accelerator by first installing the hardware and drivers as specified by the manufacturer, and then completing the installation—as with hardware certificate tokens—by installing the PKCS #11 module.

ProcedureTo Request a Server Certificate

You request a server certificate by opening your server in Console and running the Certificate Setup Wizard. You can access the Wizard from the Console menu or from the Messaging Server Encryption tab. Using the Wizard, you perform the following tasks:

Steps

  1. Generate a certificate request.

  2. Send the request by email to the certificate authority (CA) that is to issue the certificate.

    When the email response from the CA arrives, you save it as a text file and install it using the Certificate Setup Wizard.

    (For more complete instructions, see the chapter on SSL in Managing Servers with iPlanet Console.)

ProcedureTo Install the Certificate

Installing is a separate process from requesting. Once the email response to your request for a certificate has arrived from the CA and been saved as a text file, run the Certificate Setup Wizard once more to install the file as a certificate:

Steps

  1. Specify that you are installing a certificate that you have already obtained.

  2. Paste the text of your certificate into a field when prompted to do so.

  3. Change the certificate nickname from server-cert to Server-Cert.

    If you do not want to change the certificate nickname, you can change what the system wants the certificate nickname to be by setting the configutil parameter encryption.rsa.nssslpersonalityssl.

    (For more complete instructions, see the chapter on SSL in Managing Servers with iPlanet Console.)

    Note –

    This is also the process you follow to install a CA certificate (described next), which your server uses to determine whether to trust the certificates presented by clients.

To Install Certificates of Trusted CAs

You also use the Certificate Setup Wizard to install the certificates of certificate authorities. A CA certificate validates the identity of the CA itself. Your server uses these CA certificates in the process of authenticating clients and other servers.

If, for example, you set up your enterprise for certificate-based client authentication in addition to password-based authentication (see “Setting Up Certificate-Based Login” on page 157), you need to install the CA certificates of all CAs that are trusted to issue the certificates that your clients may present. These CAs may be internal to your organization or they may be external, representing commercial or governmental authorities or other enterprises. (For more details on the use of CA certificates for authentication, see Introduction to Public-Key Cryptography in Managing Servers with iPlanet Console.)

When installed, Messaging Server initially contains CA certificates for several commercial CAs. If you need to add other commercial CAs or if your enterprise is developing its own CA for internal use (using Sun Java System Certificate Server), you need to obtain and install additional CA certificates.

Note –

The CA certificates automatically provided with Messaging Server are not initially marked as trusted for client certificates. You need to edit the trust settings if you want to trust client certificates issued by these CAs. For instructions, see “Managing Certificates and Trusted CAs” on page 153.

To request and install a new CA certificate, you:

ProcedureTo Request and Install a New CA Certificate

Steps

  1. Contact the certificate authority (possibly through the Web or by email) and download its CA certificate.

  2. Save the received text of the certificate as a text file.

  3. Use the Certificate Setup Wizard, as described in the previous section, to install the certificate.

    For more complete instructions, see the chapter on SSL in Managing Servers with iPlanet Console.

Managing Certificates and Trusted CAs

Your server can have any number of certificates of trusted CAs that it uses for authentication of clients.

You can view, edit the trust settings of, or delete any of the certificates installed in your Messaging Server by opening your server in Console and choosing the Certificate Management Command in the Console menu. For instructions, see the chapter on SSL in Managing Servers with iPlanet Console.

Creating a Password File

On any Sun Java System server, when you use the Certificate Setup Wizard to request a certificate, the wizard creates a key pair to be stored in either the internal module’s database or in an external database (on a smartcard). The wizard then prompts you for a password, which it uses to encrypt the private key. Only that same password can later be used to decrypt the key. The wizard does not retain the password nor store it anywhere.

On most Sun Java System servers for which SSL is enabled, the administrator is prompted at startup to supply the password required to decrypt the key pair. On Messaging Server, however, to alleviate the inconvenience of having to enter the password multiple times (it is needed by at least three server processes), and to facilitate unattended server restarts, the password is read from a password file.

The password file is named sslpassword.conf and is in the directory msg_svr_base/config/. Entries in the file are individual lines with the format

moduleName:password

where moduleName is the name of the (internal or external) PKCS #11 module to be used, and password is the password that decrypts that module’s key pair. The password is stored as clear (unencrypted) text.

Messaging Server provides a default version of the password file, with the following single entry (for the internal module and default password):

Internal (Software) Token:netscape!

If you specify anything but the default password when you install an internal certificate, you need to edit the above line of the password file to reflect the password you specified. If you install an external module, you need to add a new line to the file, containing the module name and the password you specified for it.

Caution – Caution –

Because the administrator is not prompted for the module password at server startup, it is especially important that you ensure proper administrator access control to the server and proper physical security of the server host machine and its backups.

To Create Self-signed Certificates

If you want to create self-signed certificates in command-line mode, follow the instructions in this section. To create certificates with the certificate wizard, see Obtaining Certificates through the Administration Console.

ProcedureTo Create Self-signed Certificates

Steps

  1. Log in as or become superuser (root).

  2. Specify the certificate database password for certutil in /opt/SUNWmsgsr/config/sslpassword. For example:


    # echo "password" > /opt/SUNWmsgsr/config/sslpassword

    where password is your specific password.

  3. Move to the sbin directory and generate the certificate database (cert8.db) and key database (key3.db). For example:


    # cd /opt/SUNWmsgsr/sbin
# ./certutil -N -d /opt/SUNWmsgsr/config -f /opt/SUNWmsgsr/config/sslpassword

  4. Generate a default self-signed root Certificate Authority certificate. Example:


    # ./certutil -S -n SampleRootCA -x -t "CTu,CTu,CTu"
-s "CN=My Sample Root CA, O=sesta.com" -m 25000
-o /opt/SUNWmsgsr/config/SampleRootCA.crt
-d /opt/SUNWmsgsr/config
-f /opt/SUNWmsgsr/config/sslpassword   -z /etc/passwd

  5. Generate a certificate for the host. For example:


    ../certutil -S -n Server-Cert -c SampleRootCA -t "u,u,u"
-s "CN=hostname.sesta.com, o=sesta.com" -m 25001
-o /opt/SUNWmsgsr/config/SampleSSLServer.crt
-d /opt/SUNWmsgsr/config -f /opt/SUNWmsgsr/config/sslpassword
-z /etc/passwd

    where hostname.sesta.com is the server host name.

  6. Validate the certificates. For example:


    # ./certutil -V -u V -n  SampleRootCA -d /opt/SUNWmsgsr/config
# ./certutil -V -u V -n  Server-Cert -d /opt/SUNWmsgsr/config

  7. List the certificates. For example:


    # ./certutil -L -d /opt/SUNWmsgsr/config
# ./certutil -L -n Server-Cert -d /opt/SUNWmsgsr/config

  8. Use modutil to list the available security modules (secmod.db). For example:


    # ./modutil -list -dbdir /opt/SUNWmsgsr/config

  9. Change the owner of the certificate database files to the mail server user and group, as shown in the example.


    chown mailsrv:mail /opt/SUNWmsgsr/config/cert8.db
chown mailsrv:mail /opt/SUNWmsgsr/config/key3.db

  10. Restart the messaging services to enable the SSL.

    Note –

    Previously, certificates and key files were always located in the Messaging Server configuration directory. It is now possible to specify the location of the these files using local.ssldbpath (specifies the location of the certificate and key files) and local.ssldbprefix (specifies the prefixes of the certificate and key files.)

To Enable SSL and Selecting Ciphers

You can use Console to enable SSL and to select the set of encryption ciphers that Messaging Server can use in its encrypted communications with clients.

About Ciphers

A cipher is the algorithm used to encrypt and decrypt data in the encryption process. Some ciphers are stronger than others, meaning that a message they have scrambled is more difficult for an unauthorized person to unscramble.

A cipher operates on data by applying a key—a long number—to the data. Generally, the longer the key the cipher uses during encryption, the harder it is to decrypt the data without the proper decryption key.

When a client initiates an SSL connection with a Messaging Server, the client lets the server know what ciphers and key lengths it prefers to use for encryption. In any encrypted communication, both parties must use the same ciphers. Because there are a number of cipher-and-key combinations in common use, a server should be flexible in its support for encryption. Messaging Server can support up to 6 combinations of cipher and key length.

Table 19–2 lists the ciphers that Messaging Server supports for use with SSL 3.0. The table summarizes information that is available in more detail in the Introduction to SSL section of Managing Servers with iPlanet Console.

Table 19–2 SSL Ciphers for Messaging Server

Cipher  

Description  

RC4 with 128-bit encryption and MD5 message authentication 

The fastest encryption cipher (by RSA) and a very high-strength combination of cipher and encryption key. 

Triple DES with 168-bit encryption and SHA message authentication 

A slower encryption cipher (a U.S. government-standard) but the highest-strength combination of cipher and encryption key. 

DES with 56-bit encryption and SHA message authentication 

A slower encryption cipher (a U.S. government-standard) and a moderate-strength combination of cipher and encryption key. 

RC4 with 40-bit encryption and MD5 message authentication 

The fastest encryption cipher (by RSA) and a lower-strength combination of cipher and encryption key. 

RC2 with 40-bit encryption and MD5 message authentication 

A slower encryption cipher (by RSA) and a lower-strength combination of cipher and encryption key. 

No encryption, only MD5 message authentication 

No encryption; use of a message digest for authentication alone. 

Unless you have a compelling reason for not using a specific cipher, you should support them all. However, note that export laws restrict the use of certain encryption ciphers in certain countries. Also, some client software produced before the relaxation of United States Export Control laws cannot use the higher strength encryption. Be aware that while the 40-bit ciphers might hinder the casual eavesdropper, they are not secure and therefore will not stop a motivated attack.

To enable SSL and select encryption ciphers, follow these command line steps:

To enable or disable SSL:

configutil -o nsserversecurity -v [ on | off ]

To enable or disable RSA ciphers:

configutil -o encryption.rsa.nssslactivation -v [ on | off ]

To specify a token:

configutil -o encryption.rsa.nsssltoken -v tokenname

To specify a certificate:

configutil -o encryption.rsa.nssslpersonalityssl -v certname

Note that if you enable RSA ciphers, you must also specify a token and a certificate.

To choose a cipher preference:

configutil -o encryption.nsssl3ciphers -v cipherlist

where cipherlist is a comma-separated list of ciphers.

Note –

To enable SSL encryption for outgoing messages, you must modify the channel definition to include the tls channel keywords, such as maytls, musttls, and so on. For more information, see the Transport Layer Security Manual.

To Set Up Certificate-Based Login

In addition to password-based authentication, Sun Java System servers support authentication of users through examination of their digital certificates. In certificate-based authentication, the client establishes an SSL session with the server and submits the user’s certificate to the server. The server then evaluates whether the submitted certificate is genuine. If the certificate is validated, the user is considered authenticated.

To set up your Messaging Server for certificate-based login:

ProcedureTo Set Up Certificate-Based Login

Steps

  1. Obtain a server certificate for your server. (For details, see Obtaining Certificates through the Administration Console

  2. Run the Certificate Setup Wizard to install the certificates of any trusted certificate authorities that will issue certificates to the users your server will authenticate. (For details, see To Install Certificates of Trusted CAs

    Note that as long as there is at least one trusted CA in the server’s database, the server requests a client certificate from each connecting client.

  3. Turn on SSL. (For details, see To Enable SSL and Selecting Ciphers

  4. (Optional) Edit your server’s certmap.conf file so that the server appropriately searches the LDAP user directory based on information in the submitted certificates.

    Editing the certmap.conf file is not necessary if the email address in your users’ certificates matches the email address in your users’ directory entries, and you do not need to optimize searches or validate the submitted certificate against a certificate in the user entry.

    For details of the format of certmap.conf and the changes you can make, see the SSL chapter of Managing Servers with iPlanet Console.

    Once you have taken these steps, when a client establishes an SSL session so that the user can log in to IMAP or HTTP, the Messaging Server requests the user’s certificate from the client. If the certificate submitted by the client has been issued by a CA that the server has established as trusted, and if the identity in the certificate matches an entry in the user directory, the user is authenticated and access is granted (depending on access-control rules governing that user).

    There is no need to disallow password-based login to enable certificate-based login. If password-based login is allowed (which is the default state), and if you have performed the tasks described in this section, both password-based and certificate-based login are supported. In that case, if the client establishes an SSL session and supplies a certificate, certificate-based login is used. If the client does not use SSL or does not supply a certificate, the server requests a password.

How to Optimize SSL Performance Using the SMTP Proxy

Most sites should not use the SMTP proxy as it adds additional latency to the SMTP protocol. However, a large-scale site which makes heavy use of SSL to protect SMTP connections may wish to maximize their investment in SSL accelerator hardware by performing all SSL operations for all protocols on a server which does nothing other than SSL and proxy. The SMTP proxy allows SSL to be processed by a front end proxy server while the mail queues are on a separate MTA machine. This way hardware optimized for each task can be separately configured and purchased.

See Enabling POP Before SMTP for instructions on how to install the SMTP Proxy.