8 Certificate-Based Authentication for Messaging Server

This chapter describes the installation, configuration, and standards for certificate-based authentication for Oracle Communications Messaging Server using SSL/TLS.

Introduction: SSL/TLS, Client Certificates and CRLs

The following discusses issues related to configuring Messaging Server to use client certificates with CRLs and SSL/TLS.

Authentication Technology Overview

Messaging Server supports three types of end-user authentication:

  1. Passwords in the clear

  2. Passwords over SSL/TLS

  3. Client certificates over SSL/TLS

Option 1 is for very low security sites, option 2 is current best practice, and option 3 is usually for very high security sites. Some parts of Messaging Server have additional options. CRAM-MD5/APOP passwords are slightly better on the wire, but worse on the back-end than those in the clear. Therefore, CRAM-MD5/APOP passwords are not recommended. HTTP-only WebSSO systems most commonly use passwords over SSL/TLS.

By default, only passwords in the clear are enabled for Messaging Server. When SSL/TLS is configured, both passwords in the clear and passwords over SSL/TLS are enabled.

SSL/TLS Overview

Transport Layer Security (TLS) is the Internet proposed standard most widely used to secure application protocols such as POP, IMAP, SMTP, and HTTP. TLS is a newer and more secure version of the Secure Sockets Layer (SSL) protocol designed by Netscape. SSLv2 is an insecure legacy version of SSL which is unsupported. SSLv3 is also a legacy version which is still used, though it is less secure than TLS.

SSL provides session encryption, server certificate authentication and optional client certificate authentication. The actual security algorithms used by SSL/TLS are controlled by a negotiated cipher suite and by the authentication certificates.

STARTTLS is the command used in an application protocol (e.g., POP, IMAP, SMTP, SMTP Submission, LDAP) to start SSL/TLS. SSL/TLS is started on the application's standard port rather than a special port reserved for SSL/TLS usage only. Clients discover the availability of SSL/TLS by default and cache this information. Thus, clients are secure-by-default without requiring explicit configuration.

Note:

Disable TLS 1.0 with option.

Per bug 19849287, a new security compliance rule, OSSA-03-01-SCS/TLS-M.1, requires TLS 1.0 to be disabled by default. This means TLS 1.1 or 1.2 must be used when connecting to Messaging Server security in the default configuration. The tlsminversion option can be set to TLS1.0 to restore the previous default behavior.

Certificate Authentication Overview

Certificate Authentication involves several components:

  • a public key (information shared with other parties)

  • an associated private key (secret information)

  • a certificate authority (an authority that asserts someone in possession of a particular private key has a certain identity)

  • a certificate (a public key with one or more identities signed by itself and/or a certificate authority)

A Trust Anchor is the certificate for a well known certificate authority (CA) that is distributed with the software. Messaging Server does not enable any trust anchors by default. For authentication, the public key and its associated public key algorithm (typically RSA) are used to verify that the other party knows the associated private key.

A certificate's primary identity is called the certificate subject and is represented in attribute=value,attribute=value format. Typical certificates use a convention to encode a domain name or email address in the certificate subject. A certificate can also have a subjectAltName with an explicit DNS name or email syntax.

A certificate revocation list (CRL) is a list of certificates which are no longer valid signed by the relevant CA.

Certificate and Key Storage Overview

Certificates and keys must be stored somewhere. By default, Messaging Server stores certificates in a file in the product's configuration directory called cert8.db or cert9.db. Keys are stored in an encrypted file in the configuration directory, called either key3.db or key4.db. The encryption password to the key file is stored in sslpassword.conf so the server can reboot without administrative intervention. In general, the certutil tool manages certificates and the pk12util tool manages keys and certificate/key pairs.

SSL/TLS/Certificate Standards Overview

The following standards (or a subset thereof) are implemented in Messaging Server:

  • TLS 1.0 (RFC 2246) defines the underlying transport layer security protocol independent of how it integrates with various applications. The most recent version of TLS is 1.2 (RFC 5246).

  • SSLv3 is a less secure and less extensible predecessor to TLS 1.0 and later.

  • TLS Renegotiation Indication Extension (RFC 5746) fixes a security bug present in all versions of SSL and TLS prior to TLS 1.3.

  • SASL EXTERNAL (RFC 2222 section 7.4) is a protocol authentication mechanism typically used by POP, IMAP, and SMTP clients to explicitly tell a server to use external authentication, such as client certificates. As our servers support administrative proxy authentication (where an administrator's credentials are used to impersonate a regular user), this mechanism is required for that feature to work correctly with client certificate authentication. EXTERNAL occurs automatically when the following conditions are met:

    • A CA certificate is installed and is marked as trusted in order to validate peer certificates in the certificate DB.

    • certmap.conf is configured.

    • The client offers a valid certificate during a successful SSL negotiation.

    • The server can locate the user account associated with the certificate based on the certmap.conf. If verifycert is on the certificate, it has to match the one in the user's LDAP directory entry.

  • IMAP STARTTLS (RFC 3501 sections 6.2.1 & 11.1) is the standard for use of TLS with IMAP; and it requires use of SASL EXTERNAL.

Note:

IMAPS on port 993 is not a standard and no rules exist for its use with client certificates beyond what can be inferred from RFC 3501.
  • POP STARTTLS (RFC 2595 sections 2 & 4) is the standard for use of TLS with POP; and it requires use of SASL EXTERNAL.

    Note:

    POPS on port 995 is not a standard and no rules exist for its use with client certificates beyond what can be inferred from RFC 2595 and RFC 1939 (section 4 is particularly relevant and states that a POP session starts in AUTHORIZATION state and remains there until a mechanism such as SASL EXTERNAL is used to transition to TRANSACTION state).
  • SMTP STARTTLS (RFC 3207) is the standard for use of TLS with SMTP (RFC 2821) and SMTP Submission (RFC 4409). It does not describe how client certificates are used for submission authentication; so we follow the model of the POP/IMAP standards that are explicit. Use of SSL on port 465 for SMTP submission is a non-standard protocol with no documented interoperability rules; we follow the same rules that we do on the standard submission port with STARTTLS when this is configured.

  • HTTPS (RFC 2818) is a de-facto standard for use of SSL/TLS on the HTTPS port (443). The mshttpd daemon that runs on port 8991 by default when SSL is enabled implements SSL/TLS, but with the safer wild-carding rules present in this de-facto standard.

  • PKCS#12 is an industry standard binary format used to store a certificate and a private key, optionally encrypted by a password. It typically has a .p12 or .pk12 file extension when stored in a file.

  • PEM is an ASCII encoding used to store certificates and/or keys and/or CRLs and/or PKCS#12. It typically has a .pem file extension.

  • PKCS#11 is an industry standard that defines how a security library talks to software or hardware that can implement certificate and key storage as well as encryption algorithms. The modutil tool controls use of alternate PKCS#11 modules with Messaging Server. There is a Solaris-specific PKCS#11 module present in Solaris 10 with support for cryptographic hardware acceleration present in newer Sparc chips that can be used instead of the default PKCS#11 software module.

  • PKIX (RFC 3280) is a standard for verifying certificates and using CRLs. Messaging Server supports a commonly used subset of PKIX.

  • Online Certificate Status Protocol (OCSP, RFC 2560) is a protocol to check the status of a client certificate with a remote server, removing the need to fetch and copy CRLs. Messaging Server does not support this protocol.

SSL/TLS Tools Available in Messaging Server Installer

This section outlines the available SSL/TLS tools in Messaging Server Installer.

SUNWtlsu

The SUNWtlsu (Solaris) and sun-nss (Linux) packages installed by the Messaging Server Installer include utilities that can be used to manage keys and certificates for Messaging Server. On Solaris, the utilities are installed in /usr/sfw/bin by default. On Linux they are installed in /opt/sun/private/bin by default. Make sure to supply the option -d /var/opt/sun/comms/messaging64/config (substituting the appropriate path for your installation) to use these tools with the Messaging Server's configuration directory. If you run these utilities as root, you must manually chown the appropriate output files (cert8.db, key3.db, secmod.db or cert9.db, key4.db, pkcs11.txt) so that they are owned by the Messaging Server user.

Utilities Used to Manage Certificates

  • The certutil tool is used to manage certificates. When creating a certificate or certificate request, include -d followed by a database directory with a sql: prefix, and -Z sha256 -g 2048 to enable modern security.

  • The pk12util tool is used to manage keys or certificate and key pairs. To import or export PKCS#12 files as well as PEM-encoded PKCS#12, use the -a option.

  • The modutil tool is used to manage PKCS#11 modules.

  • The crlutil tool is used to manage CRLs.

  • The msgcert tool available in the Messaging Server's bin is no longer available because it used an inadequate key size and deprecated signature algorithm. Use the certutil tool with the previously mentioned options instead.

  • The configure tool used to configure the Messaging Server has a -ssl option which allows configuration against an LDAPS(LDAP over SSL on port 636) server. However, use of this tool requires the administrator to create the Messaging Server's configuration directory manually, in advance, and to use certutil to import the LDAP server's certificate and mark it as a trusted peer.

Certificate and Key Storage

The command msconfig set base.ssldblegacy (Unified Configuration) or configutil -o local.ssldblegacy -v 0 (legacy configuration) enables support for a newer format (cert9.db, key4.db, pkcs11.txt) which allows concurrent modification of the locally-stored certificates, keys, CRLs, and PKCS#11 modules without requiring that the server be shut down.

Modifying the Certificate Format

The aforementioned tools use the newer format only if the administrator modifies his environment as follows:

~/.bashrc:  export NSS_DEFAULT_DB_TYPE="sql"
 ~/.cshrc:   setenv NSS_DEFAULT_DB_TYPE "sql"

If a site is using a legacy certificate format, the following commands will upgrade to the new format. (Substitute appropriate path for your site to the configuration):

  1. cd /opt/sun/comms/messaging64/config

  2. Create file containing just the password for the key database with no extra newline. Call this pwfile.

  3. certutil --merge -d sql:. -Z sha256 -g 2048 -f pwfile --source-dir . -@ pwfile

  4. Create a directory for the old format databases: mkdir old_certdb

  5. mv cert8.db key3.db secmod.db old_certdb

  6. Recompile the MTA configuration if you are running a compiled configuration and restart the Messaging Server.

    imsimta cnbuild
    stop-msg
    start-msg
    

Note:

Executing the latter two commands prevent the certutil and pk12util utilities from using the old databases, if you happen to forget to set the environment variable. Remove the old format databases once a successful migration is verified.

Checking the NSS version

The new certificate and key storage format was introduced in NSS version 3.12. To check the version of NSS, do the following:

imsimta version

SSL/TLS Configuration

This section summarizes the configuration options related to the use of SSL/TLS with Messaging Server.

SSL-Related Settings

Table 8-1 provides a summary of the available SSL-related settings, which do not apply to the MMP unless explicitly mentioned.

Table 8-1 SSL-Related Settings

Unified Configuration Legacy Configuration Description

base.sslnicknames

encryption.rsa.nssslpersonalityssl

Sets the certificate nickname of the server certificate used by the Messaging Server by default.

Not applicable

local.defdomain

Default domain for authentication if one not explicitly specified; including AUTH EXTERNAL.

http.sslnicknames

imap.sslnicknames

Not applicable

mta.sslnicknames

local.http.sslnicknames

local.imap.sslnicknames

local.smtp.sslnicknames

local.imta.sslnicknames

Specify a certificate nickname of a server certificate that will be used by a particular Messaging Server service.

Not applicable

local.ldapusessl

If set, use LDAPS (LDAP-over-SSL default port 636) when contacting an LDAP server.

base.proxyimapssl

local.service.proxy.imapssl

If set, use IMAPS (IMAP-over-SSL default port 993) when proxying to another IMAP back-end for shared folders. Defaults to 1 if IMAP port is 993.

http.sslusessl

imap.sslusessl

pop.sslusessl

service.http.sslusessl

service.imap.sslusessl

service.pop.sslusessl

Enable use of SSL/STARTTLS for these services.

http.sslport

imap.sslport

pop.sslport

service.http.sslport

service.imap.sslport

service.pop.sslport

Specify a non-default SSL port.

http.enablesslport

imap.enablesslport

pop.enablesslport

service.http.enablesslport

service.imap.enablesslport

service.pop.enablesslport

Enable use of SSL on a separate port for this service.

http.plaintextmincipher

imap.plaintextmincipher

pop.plaintextmincipher

Not applicable

service.http.plaintextmincipher

service.imap.plaintextmincipher

service.pop.plaintextmincipher

service.imta.plaintextmincipher

If set to 1, these utilities require use of SSL or STARTTLS prior to use of a plaintext password. In previous versions of SMTP, this behavior had to be simulated with multiple channels, as well as the use of tlsswitchchannel and saslswitchchannel. The use of tlsswitchchannel is no longer necessary.

http.sslcachesize

imap.sslcachesize

pop.sslcachesize

Not applicable

service.http.sslcachesize

service.imap.sslcachesize

service.pop.sslcachesize

service.imta.sslcachesize

Specifies the number of SSL sessions to cache (a cached session reconnects clients faster).

base.ldapcheckcert

local.ldapcheckcert

Verifies that the LDAP server's certificate is valid when using LDAPSSL. Default = 1 and is used by MMP.

base.ldaptrace

local.ldaptrace

Enables additional LDAP diagnostics.

base.ssladjustciphersuites

local.ssladjustciphersuites

Determines which SSL cipher suites are enabled for use.

base.sslcompress

local.sslcompress

Enables SSL/TLS compression (per RFC 3749). Default = 1 and is used by MMP.

base.ssldbpath

local.ssldbpath

Determines where SSL certificate and key databases are stored (defaults to configuration directory). Using another directory is discouraged.

base.ssldblegacy

local.ssldblegacy

When set to 0, enables use of the new certificate and key database format. Used by MMP.

base.ssldbprefix

local.ssldbprefix

A prefix used in the certificate and key database file names. Changing this is discouraged.

base.sslpkix

local.sslpkix

Setting this makes your system unsupported. It enables untested experimental code with unknown scalability, timeout, and failure handling characteristics to perform full RFC 3280 PKIX validation of certificates. Used by MMP.

base.sslrequiresafenegotiate

local.sslrequiresafenegotiate

Requires use of the SSL/TLS renegotiate extension (per RFC 5746). Default = 0 and is used by MMP.

base.sslv3enable

local.sslv3enable

When set to 0, disables support for SSLv3. Default = 1 and is used by MMP.

imap.indexer.sslusessl

service.imap.indexer.sslusessl

If set to 1, the IMAP server uses SSL to authenticate with the Indexing and Search Service using the imap.indexer.port option (Unified Configuration) or service.imap.indexer.port option (legacy configuration).


Dispatcher SSL-Related Settings

When a dispatcher service configuration block includes TLS_PORT, the server will negotiate SSL/TLS immediately prior to starting the specified application. This is typically used to enable the non-standard SMTPS submission port 465 in the SERVICE=SMTP_SUBMIT section and that is presently a commented-out option in the default configuration (older versions used to incorrectly include that as an option in the SERVICE=SMTP section although it is rarely used for SMTP relay).

Messaging Transfer Agent (MTA) SSL-Related Channel Options

Table 8-2 shows SSL-related channel options.

Table 8-2 SSL-Related Channel Options

Option Description

maytlsserver

Allow use of TLS connecting to the server for that channel

musttlsserver

Require use of TLS connecting to the server for that channel

tlsswitchchannel

Change channels if TLS is successfully negotiated

maytlsclient

Use TLS if available for outbound mail sent via that channel

musttlsclient

Require TLS for outbound mail sent via that channel

maysaslserver

Allow use of SMTP AUTH (including AUTH EXTERNAL)

mustsaslserver

Require use of SMTP AUTH (including AUTH EXTERNAL)

saslswitchchannel

Change channels if SASL authentication is successful


SMTP Channel Options

Table 8-3 shows the SMTP channel options.

Table 8-3 SMTP Channel Options

SMTP Channel Option Description

IGNORE_BAD_CERT

The maytls channel keywords and SMTPS ignore errors with bad client and server certificates. Use the musttls channel keywords to control whether the SMTP client or STARTTLS command processor on the server will ignore bad certificates as follows:

  • Set bit 0 (value 1) to ignore bad client certificates.

  • Set bit 1 (value 2) to ignore bad server certificates.

  • Default setting is 3 (ignore bad certificates in SMTP).

SSL_CLIENT

Set this option to 1 to negotiate SSL/TLS on outbound client connections from this channel.

EXTERNAL_IDENTITY

Use to enable support for SASL AUTH EXTERNAL for outbound client connections. The value may be the empty string which is the identity the remote server implicitly associates with this client. When an SMTP Server's TLS implementation asks the Messaging Server SMTP client for a client certificate, the SMTP TLS client only supports providing the default server certificate for that Messaging Server installation.

AUTH_USERNAME

For outbound client SMTP connections, use this username with SASL PLAIN authentication.

AUTH_PASSWORD

For outbound client SMTP connections, use this password with SASL PLAIN authentication.

PORT_ACCESS SSL-Related Fields

The right hand side of PORT_ACCESS supports some additional fields on a successful match:

{$Yruleset|realm|tls-cert-nickname

Additional fields between realm and tls-cert-nickname are enabled by LOG_CONNECTION bit 4 and $D)

While ruleset is not currently used, the other two fields are typically used to change SSL/TLS and authentication behavior based on the server IP address to which the client is connected. The realm is the default domain appended to an unqualified authentication ID, and the tls-cert-nickname is the nickname of an alternate TLS server certificate to use.

BURL_ACCESS Mapping Table SSL-Related Input Flags

  • $:A Test if SASL authentication complete

  • $:T Test if TLS is in use


MMP SSL-Related Settings

The following MMP options appear in PopProxyAService.cfg, ImapProxyAService.cfg, and/or SmtpProxyAService.cfg. Note that all of these configuration files that specify SSLEnable MUST have the same settings. If different settings are used in different files, the MMP SSL subsystem will initialize with the settings from only one of the files. Use of non-default values is discouraged.

Table 8-4 MMP SSL-Related Settings

MMP SSL-Related Option Description

SSLCacheDir

The directory to search for certificate & key databases.

SSLCertPrefix

Filename prefix used when locating the cert8.db or cert9.db.

SSLKeyPasswordFile

Filename to use instead of sslpassword.conf for SSL key passwords.

SSLKeyPrefix

Filename prefix used when locating the key3.db or key4.db.

SSLSecmodFile

Specifies alternate file name/path for secmod.db.


Table 8-5 shows the MMP options that can be included in PopProxyAService.cfg, ImapProxyAService.cfg and SmtpProxyAService.cfg. The options can also be included in vdmap.cfg and may have different settings for each service or virtual domain:

Table 8-5 Optional MMP SSL-Related Options

Optional MMP SSL-Related Options Description

DebugKeys

A list of keywords used to enable additional diagnostics. The TLS keyword enables some additional TLS debugging features.

DefaultDomain

Default domain for authentication identities without an explicit domain, including AUTH EXTERNAL.

RestrictPlainPasswords

Requires the use of SSL/TLS prior to allowing plaintext password authentication.

SSLAdjustCipherSuites

Determines the availability of the SSL/TLS cipher suites.

SSLCertNicknames

Assigns certificate nickname(s) to use for server.

StoreAdmin

Determines which user identity to use when proxying to back-end server. Required to support client certificates with MMP.

StoreAdminPass

Assigns password for user identity to use when proxying to back-end server. Required to support client certificates with MMP.


Table 8-6 shows the MMP SSL-related options that may have different settings for each service.

Table 8-6 MMP SSL-Related Options that May Have Different Settings for Each Service

MMP SSL-Related Option with Different Settings Per Service Description

CertMapFile

Names the certificate mapping file.

LdapUrl

Uses LDAPS: instead of LDAP: with SSL when talking to LDAP.

SSLBacksidePort

Use this port when communicating to the back-end over SSL only if the connection to the MMP is also over SSL.

SSLEnable

Enables STARTTLS and SSL for this service.

SSLPorts

Assigns one or more ports to negotiate SSL immediately. The assigned ports must also be listed in the appropriate ServiceList element in AService.cfg.

UserGroupDN

All user entries appear below this LDAP DN (Distinguished Name) in the LDAP DIT (Directory Information Tree). This is used both by the client certificate authentication mapping subsystem (regardless of schema level) as well as by schema 2.


Table 8-7 shows the MMP option that is only available in vdmap.cfg.

Table 8-7 MMP SSL-Related Option Only Available in vdmap.cfg

MMP SSL-Related Option Only Available in vdmap.cfg Description

CertMap

Specifies which certmap.conf settings are used by default for this virtual domain.


certmap.conf Settings

For client certificate authentication to work, a client certificate must be translated to a Messaging Server user with an entry in LDAP. The certmap.conf configuration file is required to perform this function. The default installation creates a certmap.conf.sample file in the configuration directory which can be copied to certmap.conf. certmap.conf has named sections - a default section is mandatory - so different rules can be applied to different client certificate issuer DNs. Table 8-8 shows the available options:

Table 8-8 certmap.conf Options

certmap.conf Option Description

DNComps

commented out - take the user's DN from the certificate as is

empty - search the entire LDAP tree (DN == suffix)

attr names - a comma separated list of attributes to form DN

FilterComps

commented out - set the filter to objectclass=

empty - set the filter to objectclass=

attr names - a comma separated list of attributes to form the filter

verifycert

The user's LDAP entry must have a userCertificate;binary field that matches the certificate used by the client.

CmapLdapAttr

If not empty, search the entire tree for an entry with the CmapLdapAttr attribute that matches the client certificate subject.


SSL/TLS Tasks

This section includes procedures to create, install, and test the different types of certificates and keys used by SSL/TLS.

How to Create and Install a Self-signed CA Certificate and Key

Using the following procedure, you can create a self-signed CA certificate used to sign client and server certificates. (In a production environment you would generate a certificate request (-R) and have that signed by a CA certificate.)

% cd /opt/sun/comms/messaging64/config
% certutil -d sql:. -g 2048 -Z sha256 -N

This prompts for a password. Save that password to a file called pwfile with no trailing newline. Also save the password to a file called sslpassword.conf which should contain the single line:

Internal (Software) Token:password

Where the password selected is after the ':'. You may include a newline in this file.

Make sure that the certificate, key, and pkcs11.txt files are owned by the messaging server user by performing the following:

% certutil -S -d sql:. -g 2048 -Z sha256 -n CA-Cert -s "cn=CA Cert for Messaging" -x -t CT -f pwfile

How to Create and Install a CA-signed Server Certificate and Key

To create and install a CA-signed server certificate and key, perform the following:

% certutil -S -d sql:. -g 2048 -Z sha256 -n Server-Cert -s "cn=mail.example.com" -c CA-Cert -f pwfile -t P

How to Create a CA-signed Client Certificate and Key

To create a CA-signed client certificate and key, perform the following:

% certutil -S -d sql:. -g 2048 -Z sha256 -m 1 -n client-cert -s "e=user@example.com" -c CA-Cert -f pwfile -t u

How to Test a CA-signed Client Certificate and Key

To test a CA-signed client certificate and key, perform the following:

msconfig set imap.sslusessl 1 (Unified Configuration) or configutil - o service.imap.sslusessl -v 1 (legacy configuration).

% start-msg
% /opt/sun/comms/messaging64/lib/sslconnect -r -c client-cert mail.example.com 143

Next, type the following command: A AUTHENTICATE EXTERNAL =

and the following appears:

A OK User logged in

How to Create and Install a CRL for a Client Certificate

To create and install a CRL for a client certificate, perform the following:

% crlutil -d sql:. -G -n CA-Cert << EOF
update=20100510200000Z
addcert 1 20100510200000Z
EOF

Note that the 1 in the addcert above is the serial number of the client certificate, which was specified by the -m option used when creating the client certificate.

How to Test a CRL for a Client Certificate

To test a CRL for a client certificate, perform the following:

% /opt/sun/comms/messaging64/lib/sslconnect -r -c client-cert mail.example.com 143

Next, type the following command: A AUTHENTICATE EXTERNAL =

and the following appears:

A NO Mechanism not Available

The following message should appear in the IMAP log:

[10/May/2010:20:29:20 -0700] nifty-silver imapd[12720]: General Notice: Bad 
certificate from [127.0.0.1:64215]: errno -8180 (Peer's Certificate has been 
revoked.)

How to Look Up Numeric SSL/TLS Error Codes

To look up numeric error codes related to SSL/TLS, see:

http://www.mozilla.org/projects/security/pki/nss/ref/ssl/sslerr.html

String error text and numeric error codes are outputted in order to debug log files. Often, this site contains additional information that is not in the string error text that we output.

Sample Protocol Sessions with Client Certificate Authentication

This section includes example protocol transcripts where client certificate authentication uses both standard and non-standard protocols.

Standard IMAP (STARTTLS) default port 143

The following is how standard IMAP client certificate authentication executes:

====

S: * OK CommSuite7:CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS 
CHILDREN BINARY UNSELECT SORT CATENATE URLAUTH LANGUAGE ESEARCH ESORT 
THREAD=ORDEREDSUBJECT THREAD=REFERENCES ENABLE CONTEXT=SEARCH CONTEXT=SORT WITHIN 
SASL-IR SEARCHRES XSENDER X-NETSCAPE XSERVERINFO X-SUN-SORT ANNOTATE-EXPERIMENT-1 
X-UNAUTHENTICATE X-SUN-IMAP X-ANNOTATEMORE XUM1 ID STARTTLS IDLE XREFRESH 
AUTH=PLAIN nifty-silver.west.sun.com IMAP4 service (Oracle Communications 
Messaging Server 7u5-0.01 32bit (built Apr 8 2010))

C: a STARTTLS

S: a OK Completed

...TLS-negotiation-with-client-cert...

C: b CAPABILITY

S: * CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS CHILDREN 
BINARY UNSELECT SORT CATENATE URLAUTH LANGUAGE ESEARCH ESORT THREAD=ORDEREDSUBJECT 
THREAD=REFERENCES ENABLE CONTEXT=SEARCH CONTEXT=SORT WITHIN SASL-IR SEARCHRES 
XSENDER X-NETSCAPE XSERVERINFO X-SUN-SORT ANNOTATE-EXPERIMENT-1 X-UNAUTHENTICATE 
X-SUN-IMAP X-ANNOTATEMORE XUM1 ID IDLE XREFRESH AUTH=EXTERNAL AUTH=PLAIN

S: b OK Completed

C: c AUTHENTICATE EXTERNAL =

S: c OK User logged in

====

For a standard IMAP client certificate authentication to execute successfully, the following requirements must be met:

  1. There has to be a valid server certificate installed.

  2. The client has to connect to the fully-qualified domain name of the server using the fully-qualified hostname present in the server certificate.

  3. There has to be a CA certificate trusted to sign client certificates installed.

  4. service.imap.sslusessl must be enabled.

  5. There must be a valid certmap.conf with correct permissions.

  6. The client must supply a valid client certificate during the SSL exchange, and the certmap code must map the certificate to a valid mail user.

AUTH=EXTERNAL appears in the capability list when the above conditions are met.

Note that this can also be used for administrative proxy authentication. If the client certificate is a store administrator who wants to authenticate as user cnewman, the command would be:

====

C: c AUTHENTICATE EXTERNAL Y25ld21hbg==

S: c OK User logged in

====

An attempt to proxy authenticate inappropriately:

====

c AUTHENTICATE EXTERNAL YWRtaW4=

c NO Not authorized to login as specified user

====

Standard Submission (STARTTLS) Default port 587

====

S: 220 nifty-silver.west.sun.com - Server ESMTP (Oracle Communications Messaging Server 7u5-0.01 32bit (built Apr 20 2010))

C: EHLO nifty-silver.west.sun.com

S: 250-nifty-silver.west.sun.com

S: 250-8BITMIME

S: 250-PIPELINING

S: 250-CHUNKING

S: 250-DSN

S: 250-ENHANCEDSTATUSCODES

S: 250-EXPN

S: 250-HELP

S: 250-XADR

S: 250-XSTA

S: 250-XCIR

S: 250-XGEN

S: 250-XLOOP 8DA8D338B89F8CEC5FED34564D95F616

S: 250-STARTTLS

S: 250-AUTH PLAIN LOGIN

S: 250-AUTH=LOGIN PLAIN

S: 250-NO-SOLICITING

S: 250 SIZE 0

C: STARTTLS

S: 220 2.5.0 Go ahead with TLS negotiation.

...TLS-negotiation-with-client-cert...

C: EHLO nifty-silver.west.sun.com

S: 250-nifty-silver.west.sun.com

S: 250-8BITMIME

S: 250-PIPELINING

S: 250-CHUNKING

S: 250-DSN

S: 250-ENHANCEDSTATUSCODES

S: 250-EXPN

S: 250-HELP

S: 250-XADR

S: 250-XSTA

S: 250-XCIR

S: 250-XGEN

S: 250-XLOOP 8DA8D338B89F8CEC5FED34564D95F616

S: 250-AUTH EXTERNAL PLAIN LOGIN

S: 250-AUTH=LOGIN PLAIN

S: 250-NO-SOLICITING

S: 250 SIZE 0

C: AUTH EXTERNAL =

S: 235 2.7.0 EXTERNAL authentication successful.

====

For the authentication to execute successfully the following requirements must be met:

  1. There must be a valid server certificate installed.

  2. The client must connect to the fully-qualified domain name of the server using the fully-qualified hostname present in the server certificate.

  3. There must be a CA certificate trusted to sign client certs installed.

  4. The relevant channel must include the maytlsserver/musttlsserver channel option. For submission on port 587 with a factory configuration, the relevant channel is tcp_submit and has these settings by default.

  5. The relevant channel must include the maysaslserver/mustsaslserver channel option.

  6. The relevant channel typically includes saslswitchchannel tcp_auth.

  7. There must be a valid certmap.conf file with correct permissions.

  8. The client must supply a valid client certificate during the SSL exchange, and the certmap code has to successfully map that to a valid mail user.

The standards are not clear whether the AUTH EXTERNAL is required or not. Currently, administrative proxy authentication is allowed via standard protocol.

Standard POP (STLS) default port 110

====

S: +OK nifty-silver.west.sun.com POP3 service (Oracle Communications Messaging Server 7u5-0.01 32bit (built Apr 8 2010))

C: STLS

S: +OK

C: CAPA

S: +OK list follows

S: TOP

S: PIPELINING

S: UIDL

S: RESP-CODES

S: AUTH-RESP-CODE

S: SASL EXTERNAL PLAIN

S: USER

S: IMPLEMENTATION POPD-7.5p0.01 Apr 28 2010

S: .

C: AUTH EXTERNAL =

S: +OK Maildrop ready

====

The above executes in the same way as the standard IMAP (STARTTLS) with the same requirements.

Non-standard IMAPS typical port 993

====

...TLS-negotiation-with-client-cert...

S: * PREAUTH CommSuite7:CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE 
UIDPLUS CHILDREN BINARY UNSELECT SORT CATENATE URLAUTH LANGUAGE ESEARCH ESORT 
THREAD=ORDEREDSUBJECT THREAD=REFERENCES ENABLE CONTEXT=SEARCH CONTEXT=SORT WITHIN 
SASL-IR SEARCHRES XSENDER X-NETSCAPE XSERVERINFO X-SUN-SORT ANNOTATE-EXPERIMENT-1 
X-UNAUTHENTICATE X-SUN-IMAP X-ANNOTATEMORE XUM1 ID IDLE XREFRESH 
nifty-silver.west.sun.com IMAP4 service (Oracle Communications Messaging Server 
7u5-0.01 32bit (built Apr 8 2010))

====

The * PREAUTH means the client certificate supplied during SSL negotiation was valid, was successfully certmapped to a specific user, and the specific user has been logged in already. If the server sends * OK then this has failed and the client must proceed with standard password authentication.

Non-standard SMTPS typical port 465

====

...TLS-negotiation-with-client-cert...

S: 220 nifty-silver.west.sun.com - Server ESMTP (Oracle Communications Messaging Server 7u5-0.01 32bit (built Apr 20 2010))

C: EHLO nifty-silver.west.sun.com

S: 250-nifty-silver.west.sun.com

S: 250-8BITMIME

S: 250-PIPELINING

S: 250-CHUNKING

S: 250-DSN

S: 250-ENHANCEDSTATUSCODES

S: 250-EXPN

S: 250-HELP

S: 250-XADR

S: 250-XSTA

S: 250-XCIR

S: 250-XGEN

S: 250-XLOOP 28B98A3E76A6F4C19EDF069FBDC26E97

S: 250-AUTH EXTERNAL PLAIN LOGIN

S: 250-AUTH=LOGIN PLAIN

S: 250-NO-SOLICITING

S: 250 SIZE 0

C: AUTH EXTERNAL =

S: 235 2.7.0 EXTERNAL authentication successful.

====

As with standard SMTP STARTTLS: clients do not know if the server accepted the client certificate unless the EXTERNAL mechanism is available. Some third-party clients fail to look for and use the EXTERNAL mechanism.

Non-standard POPS typical port 995

====

...TLS-negotiation-with-client-cert...

S: +OK nifty-silver.west.example.com POP3 service (Oracle Communications Messaging Server 7u5-0.01 32bit (built Apr 8 2010))

C: CAPA

S: +OK list follows

S: TOP

S: PIPELINING

S: UIDL

S: RESP-CODES

S: AUTH-RESP-CODE

S: SASL EXTERNAL PLAIN

S: USER

S: IMPLEMENTATION POPD-7.5p0.01 Apr 28 2010

S: .

C: AUTH EXTERNAL =

S: +OK Maildrop ready

====

This execution differs from non-standard IMAPS because POP3 cannot determine if client certificate authentication succeeded. (There is nothing equivalent to IMAP's * PREAUTH greeting.) The Messaging Server implementation uses the only standard protocol mechanism (AUTH EXTERNAL), in compliance with RFC 1939 section 4 (RFC 1939 section 4 declares a POP3 session starts in AUTHORIZATION state). Some third-party clients fail to use the EXTERNAL mechanism and assume the client certificate worked if the connection remains open.

SSL/TLS Best Practices

The following are best practices when using SSL/TLS with Messaging Server:

  • Make sure that the installed NSS version is 3.16 or later.

  • Use a 2048-bit RSA certificate with a SHA256 signature. Do not use the deprecated msgcert tool to generate a server certificate as it does not support these two options.

  • Enable use of SSL between end-user clients and the server whenever possible.

  • Enable use of SSL between any geographically-disparate back-end servers. For example, if you have an off-site failover LDAP master configured, enable SSL when talking to LDAP. Make sure local.ldapcheckcert = 1 with earlier versions of Messaging Server.

  • Set RestrictPlainPassword and plaintextmincipher whenever possible for your deployed clients. If you can identify just the clients lacking this support, put them on a separate MMP virtual domain.

  • Enable the separate SSL-only ports (993, 995, 465) which can be used in addition to SSL/TLS on any of the regular ports (143,110, 587). With the RestrictPlainPassword and plaintextmincipher options, there is no significant security difference between the regular and SSL-only ports.

  • Use ssladjustciphersuites to enable:

    TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA.

  • Set tlsv12enable to 1.

  • If possible, disable weaker cipher suites with ssladjustciphersuites (particularly the RC4 cipher suites).

  • Disable SSLv3 by setting local.sslv3enable to 0.

Client Certificate SSL/TLS Best Practices

  • Set IGNORE_BAD_CERT to 0, at least for submission service.

  • If using CRLs, use the new format certificate database.

  • If using CRLs, make sure the CRLs stored in the certificate database are updated periodically, perhaps via cron or similar mechanism.

Messaging Server and SSL/TLS: Known Limitations

The following is not an exhaustive list of product limitations. There are additional limitations in ancillary utilities outside the core product feature set.

Protocols with Unspecified Client Certificate Behavior

Both IMAP4 and POP3 have well documented STARTTLS standards that cover client certificate behavior (published in June 1999). Messaging Server interoperates with standards compliant clients. Due to IMAP4's standard PREAUTH feature, Messaging Server's client certificate support on the undocumented IMAPS port is likely to interoperate with standards-compliant IMAP4 clients implementing IMAPS. However, administrative proxy authentication is made unavailable on that port as a result. Neither SMTP nor POP3s have well defined standards for client certificates.

Administrative Proxy with a Certificate

Except for the IMAPS service, all of the Messaging Server servers (IMAP, POP3, SMTP Submission) support administrative proxy using a client certificate. If the administrator provides a certificate that maps to a known store administrator identity using certmap, that administrator can authenticate as another user via the AUTH EXTERNAL mechanism. However, the feature is not presently available on the MMP.

Proxy IMAP Authentication Limitations

When Messaging Server's mshttpd or IMAPD daemons contact an IMAP back-end server, they only support administrative proxy using plain text passwords, which are optional over the SSL port. Support for administrative proxy via a client certificate is not currently supported. STARTTLS is not currently supported either.

The MTA IMAP URL resolver used by the MTA's BURL feature does not support SSL.

LDAP Client Authentication Limitations

The Messaging Server LDAP client only supports plaintext password authentication optionally over the SSL port for LDAP (636). The client does not support LDAP STARTTLS, client certificate authentication to LDAP, or other authentication mechanisms for LDAP.

Proxy MMP (IMAP/POP/SMTP-Submission) Authentication Limitations

The MMP's server components support client certificate authentication but do not support administrative proxy authentication using a mechanism other than plaintext passwords. The MMP's client components only support password replay or administrative proxy via plaintext passwords (optionally over SSL if the client also used SSL). They do not support client certificate authentication to the back-end and also do not support STARTTLS.

Internal Protocols Lacking Support for SSL and/or Authentication

The LMTP server only supports authentication using the PORT_ACCESS mapping to filter based on IP address. Basic SSL/TLS (without client certificates) should work using standard MTA channel keywords. The ENS server only supports authentication using service.ens.domainallowed and service.ens.domainnotallowed, where filtering is based on the IP address.

Disabling Passwords-Over-SSL

You cannot disable passwords-over SSL.

Hosting Multiple Domains with SSL

The MMP's IMAP, POP, and Submission proxies can be used to host multiple SSL domains with different certificates as long as each domain has a separate IP address and the appropriate vdmap.cfg settings. The MTA's PORT_ACCESS mapping supports standard SMTP submission with STARTTLS, although non-standard SMTPS via the MTA does not have this feature. The back-end IMAP and POP servers do not have this support.

CRL Updates and OCSP

The default certificate validation algorithm checks a client certificate against CRLs stored in the local NSS certificate database. There is no automated procedure to update those CRLs and fetch CRLs. OCSP is not supported.

In order to use up-to-date CRLs for SSL, you must use a new certificate format (cert9.db, key4.db, pkcs11.txt) or a third-party PKCS#11 module that supports concurrent read/write access, as well write your own cron or equivalent script to fetch and update this information.

Time delay for updates to CRLs or New Certificates

When a CRL is updated in the new certificate format (cert9.db, key4.db, pkcs11.txt), it can take up to 10 minutes for running processes to notice the database has changed and update their internal cache of certificates and CRLs. If a CRL change must take effect immediately, the relevant servers must be restarted.

References

Technical Documentation for New Database Format in NSS:

https://wiki.mozilla.org/NSS_Shared_DB

NSS Shared DB Howto (Primarily for Firefox/Thunderbird):

https://wiki.mozilla.org/NSS_Shared_DB_Howto

Documentation for certutil:

https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/tools/NSS_Tools_certutil

Documentation for crlutil:

https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/tools/NSS_Tools_crlutil

Key Manager (add on for Firefox to provide GUI key management for NSS):

https://addons.mozilla.org/en-US/firefox/addon/key-manager/

Error Codes Returned by Mozilla APIs:

https://developer.mozilla.org/en-US/docs/Mozilla/Errors