4 Implementing Messaging Server Security

This chapter explains the security features of Oracle Communications Messaging Server. It also provides links to security topics that provide more in-depth information for configuring and administering Messaging Server securely.

Security Features

Messaging Server supports security features that enable you to keep messages from being intercepted, prevent intruders from impersonating your users or administrators, and permit only specific people access to specific parts of your messaging system. The Messaging Server security architecture is part of the security architecture of Oracle servers as a whole. It is built on industry standards and public protocols for maximum interoperability and consistency. For a general overview of Messaging Server security strategies, see "Planning Messaging Server Security."

This section describes additional features, considerations, and recommendations for securing your Messaging Server deployment:

Messaging Server Security Strategy for your Deployment

Creating a security strategy is one of the most important steps in planning your deployment. Your strategy should meet your organization's security needs and provide a secure messaging environment without being overbearing to your users.

How you set up the following topics impacts your security strategy guidelines:

Creating a Security Strategy

Your security strategy needs to be simple enough to administer. A complex security strategy can lead to mistakes that prevent users from accessing their mail, or it can allow users and unauthorized intruders to modify or retrieve information that you do not want them to access.

The five steps to developing a security strategy, as listed in RFC 2196, the Site Security Handbook, include:

  • Identifying what you are trying to protect. For example, your list might include hardware, software, data, people, documentation, network infrastructure, or your organization's reputation.

  • Determining what you are trying to protect it from. For example: unauthorized users, spammers, or denial of service attacks.

  • Estimating how likely threats are to your system. If you are a large service provider, your chances of security threats could be greater than a small organization. In addition, the nature of your organization could provoke security threats.

  • Implementing measures that will protect your assets in a cost-effective manner. For example, the extra overhead in setting up an SSL connection can put a performance burden on your Messaging deployment. In designing your security strategy, you need to balance security needs against server capacity.

  • Continuously reviewing your strategy and making improvements each time a weakness is found. Conduct regular audits to verify the efficiency of your overall security policy. You can do this by examining log files and information recorded by the SNMP agents. For more information on SNMP, see the Messaging Server System Administrator's Guide.

Your security strategy should also plan for:

Physical Security

Limit physical access to important parts of your infrastructure. For example, place physical limits on routers, servers, wiring closets, server rooms, or data centers to prevent theft, tampering, or other misuse. Network and server security become a moot point if any unauthorized person can walk into your server room an unplug your routers.

Server Security

Limiting access to important operating system accounts and data is also part of any security strategy. Protection is achieved through the authentication and access control mechanisms available in the operating system.

In addition, you should install the most recent operating environment security patches and set up procedures to update the patches once every few months and in response to security alerts from the vendor.

Operating System Security

Reduce potential risk of security breaches in the operating environment by performing the following, often termed system hardening:

  • Minimize the size of the operating environment installation. When installing an Oracle server in an environment that is exposed to the Internet, or any untrusted network, reduce the Oracle Solaris OS software installation to the minimum number of packages necessary to support the applications to be hosted. Achieving minimization in services, libraries, and applications helps increase security by reducing the number of subsystems that must be maintained.

    The Oracle Solaris Security Toolkit provides a flexible and extensible mechanism to minimize, harden, and secure Oracle Solaris systems. The primary goal behind the development of this toolkit is to simplify and automate the process of securing Oracle Solaris systems.

  • Track and monitor file system changes. Within systems that require inclusion of security, a file change control and audit tool is indispensable as it tracks changes in files and detects possible intrusion. You can use a product such as Tripwire for Servers, or Oracle Solaris Fingerprint Database.

Network Security

The recommended deployment configuration, to support both horizontal scalability and service security, is to place the access layer of the architecture behind a firewall. In a two-tiered architecture, use two firewalls, creating a DMZ. This enables access to the information delivery elements, the calendar and messaging front ends, while protecting the main service elements on the internal network behind a second firewall. Such a configuration also enables the access layer and data layer elements to be scaled independently, accommodating traffic and storage elements.

Limiting access to your network is an important part of your security strategy. Normally, overall access to networks is limited through the use of firewalls. However, email must be made available outside your site. SMTP is one such service that allows for this.

To secure your network, you should:

  • Turn off all operating system-provided services that listen on ports that you do not use.

  • Replace telnet with sshd, if possible.

  • Place your application servers behind a packet filter, which drops external packets with an internal source IP address. A packet filter forbids all connections from the outside except for those ports that you explicitly specify.

Messaging Security

Messaging Server offers the following sets of security features:

  • Protecting Messaging Components in Your Deployment. With this set of options, you can secure your MTA relays, message stores, Webmail clients, and multiplexing services. In addition, you'll learn about third-party spam filter options.

  • Planning User Authentication. These options enable you to determine how your users will authenticate to your mail servers, preventing unauthorized users from gaining access to your system.

  • Understanding Security Misconceptions. Using this set of options, you can perform user authentication and protect the message itself by using authenticated SMTP and certificates for digital signatures, encryption, and Secure Sockets Layer (SSL).

See "Planning Messaging Server Security" for more information.

Application Security

Messaging Server provides features that ensure the security and integrity of business communications. Messaging Server offers extensive built-in security features, such as:

  • Authentication

  • Message and session encryption

  • Virus and spam protection

  • Archiving and auditing of communications

Implementing Secure Connections

Messaging Server supports security standards such as SSL/TLS, S/MIME, and SASL. SSL/TLS enables all communication between clients and servers to take place inside an encrypted session.

If you want to implement public key data security, you must select mail clients that support public key infrastructure and key choice. Messaging Server is capable out of the box, of participating in the transmission and storage of encrypted messages. Secure/Multipurpose Internet Mail Extension (S/MIME) is available on the Convergence Mail clients. Users who are set up to use S/MIME can exchange signed or encrypted messages with other users of Convergence Mail, Microsoft Outlook, and Mozilla mail systems.

A more commonly used mechanism for data security protects the data across the wire only (that is, from client to server) by using SSL encryption on the connections used to transmit data between various messaging agents. This solution is not as complete as public key encryption, but is far easier to implement and is supported by many more products and service providers.

What problem does using SSL from client to server solve? An organization assumes that it controls its own corporate network and that data transmitted on that network is safe from non-employees. Mail sent to anyone from outside the corporate network using the corporation's infrastructure transmits the data over an encrypted connection to the corporation's network. Likewise, all mail picked up by a corporate user from outside the corporate network will be transmitted over an encrypted connection. Thus, if the enterprise's assumption about the safety of the internal network is true, and its employees use only sanctioned servers for transmission between themselves and other employees, mail between employees is safe from external attack.

What problem doesn't this solution solve? First of all, this approach does not protect the data from unintended viewing by non-intended recipients of the data who have access to the organization's internal network. Secondly, there is no protection offered for data being transmitted between employees and their external partners, customers, or suppliers. The data travels across the public Internet in a completely insecure fashion.

However, this problem can be remedied by configuring SSL encryption between MTA routers at both the enterprise's and customer's network. This type of solution requires setup for each private connection you want to use. In so doing, you add an important additional layer of security with customer or partner data being sent or received via email. Using MTAs and SSL, companies can save money by using the public Internet as the transport, but force the MTAs to use SSL for their partners. This solution does not take into account other network traffic to and from partners. Nevertheless, mail is usually a large proportion of the traffic, and because companies can pay based on data transmitted, using the public Internet is usually cheaper.

Implementing Secure Connections Using Two Different Certificate Authorities (CAs)

You can implement SSL connections between server and clients, for example, from Messaging Server, and to other servers in your deployment as well (Web Server, Calendar Server, Directory Server). If desired, you can use two different Certificate Authorities (CAs), one for the server and one for the client.

In such a scenario, you can use one CA to issue server certificates, and another CA to issue client certificates. If you want the client to accept the server's certificate as genuine, you will need to load the CA certificate for the server into the client's certificate DB. If you want the server to accept the client as genuine, you will need to load the CA certificate for the client into the server's certificate DB.

Identifying Password Policy Requirements

The user password login process is described in the Messaging Server System Administrator's Guide.

Review the following additional password policy recommendations:

  1. Select a password policy system that meets your requirements and any additional requirements you might add at a later time.

  2. Require your users to create high quality passwords on your site identity system's password change web page. Do not require your users to change passwords too frequently as it cause users to write their passwords on paper.

  3. Directory Server has password policy capabilities, but if you enable password expiration, be sure the administrator and service accounts used by Messaging Server are exempt from expiration.

  4. Keep a strong administration password.

  5. Maintain administrative access policies for Messaging Server hosts.

  6. Create Delegated Administrator access policies for domains.

  7. If needed by Oracle Support, plan how to gather configuration files, excluding passwords. If you use Unified Configuration, this task is made much easier.

Verifying File Ownership for Configuration Files

Related topics include:

  • The discussion on ownership of a mail server user account in the Messaging Server System Administrator's Guide.

  • The discussion on identifying, analyzing, and resolving user mailbox directory problems in the Messaging Server System Administrator's Guide.

Securely Monitoring and Auditing Your Messaging Server Deployment

Monitoring your server is an important part of your security strategy. To identify attacks on your system you should monitor message queue size, CPU utilization, disk availability, and network utilization. Unusual growth in the message queue size or reduced server response time may identify some of these attacks on MTA relays. Also, investigate unusual system load patterns and unusual connections. Review logs on a daily basis for any unusual activity.

Refer to the discussion on monitoring the Messaging Server for signs of problems in the Messaging Server System Administrator's Guide, which includes topics on:

  • Automatic Monitoring and Restart

  • Daily Monitoring Tasks

  • Utilities and Tools for Monitoring

  • Monitoring User Access to the Message Store

  • Monitoring System Performance

  • Monitoring Disk Space

  • Monitoring the MTA

  • Monitoring LDAP Directory Server

  • Monitoring the Message Store

  • Messaging Server Monitoring Framework Support

  • SNMP Support

  • How to Monitor MeterMaid

Additional guidelines for secure monitoring:

  • Ensure you have the right monitoring and auditing tools for your specific deployment and that you have contingency plans in place.

  • Enable MTA logging.

Tracking Security Patches

Be sure to install the most recent operating environment security patches and set up procedures to update the patches once every few months and in response to security alerts from the vendor. Be sure to pay close attention to NSS patches.

Identifying Legal-intercept Requirements

The following topics provide an overview for message archiving for legal and compliance purposes. For more information, see:

  • The discussion on the legal obligation to maintain strict retrievable email records in the Messaging Server System Administrator's Guide.

  • The discussion on the imarchive utility in the Messaging Server System Administrator's Guide.

  • The discussion on how to archive messages coming into and out of the Messaging Server using the Sun Compliance and Content Management Solution (for legacy systems only) in the Messaging Server System Administrator's Guide.

Determine which Messaging Server capture mechanism is best to meet those requirements in your jurisdiction prior to responding to a compliance request.

Securing Your Archiving Needs

Once you have satisfied legal requirements, use your third-party archiving system in your jurisdiction so that it can be configured to delete messages from the archive (or make them unreadable by discarding encryption keys). Refer to "Identifying Legal-intercept Requirements" for message archiving options.

Disabling Users in Response to Abuse/Appeal Process

The following topics describe enabling and disabling users, accounts, and services in response to the abuse/appeal process. See:

  • The discussion on mailAllowedServiceAccess in the Schema Reference.

  • The discussion on SEND_ACCESS and ORIG_SEND_ACCESS mappings in the Messaging Server Reference.

  • The discussion on enabling and disabling services at different levels in the Messaging Server System Administrator's Guide.

Utilizing a Disk Consumption Growth Plan

Unusual disk consumption may identify some attacks on MTA relays.

For more information, see:

  • The discussion on configuration options for monitoring disk and partition usage in the Messaging Server System Administrator's Guide.

  • The discussion on Message Store partitions and adding storage in the Messaging Server System Administrator's Guide.

  • The discussion on MTA performance tuning in the Messaging Server Reference.

Preventing Unrelated Usage of Messaging Server Hosts and Virtual Machines

Oracle recommends that you do not use Messaging Server hosts or virtual machines for unrelated tasks. Single purpose hosts and virtual machines are better for securing your deployment. Be sure to turn off any unused Messaging Server services.

Determining Security Capabilities of Your Supported Mail Clients

For information on security and access control for mail clients and mail client infrastructure, refer to "Security and Access Control in Messaging Server" where the following topics are covered:

Consider these questions when designing your Messaging Server security strategy:

  • Do your mail clients support SMTP Authentication?

  • Do your mail clients support standard SSL (STARTTLS on port 143 for IMAP, STLS on port 110 for POP, STARTTLS on port 587 for SMTP Submission)?

    • If not, do they support legacy non-standard SSL (IMAPS on port 993, POPS on port 995, SSL SMTP Submission on non-standard port 465)?

  • Do you have a plan in place to handle accidental/inappropriate blacklisting of your site by reputation services?

MTA Security Guidelines

Following secure guidelines protect your MTAs from unauthorized users, large quantities of spam, reduced response time, and used up disk space and resources. "Protecting MTAs" outlines general guidelines to protect your MTAs. This section provides additional details in the following topics:

About Messaging Server Anti-spam and Anti-virus Solutions

Refer to the following discussions on anti-spam and anti-virus solutions:

  • Planning a Messaging Server Anti-spam and Anti-virus Strategy

  • Integrating spam and virus filtering programs in the Messaging Server System Administrator's Guide.

  • Using Milter in the discussion about integrating spam and virus filtering programs in the Messaging Server System Administrator's Guide.

  • Using the Sender Policy Framework to detect and reject forged email in the Messaging Server System Administrator's Guide.

  • Protecting Against Email Spammers

  • Blocking emails based on DNS Realtime Blocklists (RBL) data in the Messaging Server System Administrator's Guide.

In addition, consider these guidelines:

  • Make sure your domain's MX records point directly to Messaging Server's MTA and have the Messaging Server call out to anti-spam/anti-virus systems preferably through the spam plug-in or Milter mechanism.

  • Filter both inbound and outbound mail.

  • Consider restricting outbound port 25 to outbound MTAs only.

Creating a Narrow Scope of MTA Relay Blocking in INTERNAL_IP Mapping Table

To use the INTERNAL_IP mapping table for MTA Relay Blocking, refer to the following discussions:

  • On preventing unauthorized users from relaying SMTP mail through your system in the Messaging Server System Administrator's Guide.

  • On mail filtering and access control in the Messaging Server System Administrator's Guide and the mailfromdnsverify channel option in the Messaging Server Reference.

  • On using access control mappings to prevent users from relaying SMTP mail through your system in the Messaging Server System Administrator's Guide.

You can accomplish STMP relay blocking during initial configuration. The configure command prompts for a list of IP addresses of internal systems that are allowed to relay. If the list of such systems is empty, then the default settings from the configure command are used.

Using LMTP to Connect to Inbound MTAs and in Multi-tier Deployments

Using LMTP between the relays and the back end Message Stores simplifies the deployment, which means there are fewer points of attack. For more information, see:

  • The discussion on configuring the LMTP delivery mechanism in the Messaging Server System Administrator's Guide.

  • The discussion on channel configuration in the Messaging Server Reference.

  • The discussion on implementing Local Message Transfer Protocol (LMTP) for Messaging Server in the Messaging Server System Administration Guide.

Forbidding Emailing Executable Code

See the discussion of the predefined conversion channel in the Messaging Server System Administrator's Guide.

Using and Configuring MeterMaid for Access Control

MeterMaid is a server that can provide centralized metering and management of connections and transactions through monitoring IP addresses and SMTP envelope addresses. Functionally, MeterMaid can be used to limit how often a particular IP address can connect to the MTA. Limiting connections by particular IP addresses is useful for preventing excessive connections used in denial-of-service attacks. MeterMaid supplants conn_throttle.so by providing similar functionality, but extending it across the Messaging Server installation. No new enhancements are planned for conn_throttle.so and MeterMaid is its more effective replacement.

For more information, see:

  • The discussion on using MeterMaid to limit how often a particular IP address can connect to the MTA in the Messaging Server System Administrator's Guide.

  • The MeterMaid Reference in the Messaging Server System Administrator's Guide.

Using and Configuring memcache for Access Control

memcache is a server that can provide functionality that is similar to MeterMaid. It allows you to access and manipulate data using the memcache protocol. For more information, see the discussions about memcache in the Messaging Server Reference.

Setting MTA Recipient Limits

For more information, see the discussion on channel configuration in the Messaging Server Reference.

Using Sieve Securely

Review your MAX_* options settings relevant to Sieve filter limits, especially MAX_NOTIFYS.

Note:

Notify, Forward, and Redirect can potentially increase the load of generating new messages. You need to consider if abusers could exploit such features by generating message loops or exponential growth of messages.

For Sieve external lists, enable setup carefully only allowing specific criteria. Some Sieve filter user education/Sieve filter creation interface guidelines to consider:

  • Discourage users from attempting to personally block spam by using Sieve.

  • Check that the interface generates efficient Sieves (for example: lists, wildcard matches, and so on)

Review:

  • The discussion on the extensions that Messaging Server supports in the Messaging Server System Administrator's Guide.

Using the MTA to Fix Messages from Bad Clients

If users use email clients that are especially vulnerable to buffer overruns, malicious embedding in malformed header lines, and so on, consider configuring the MTA with maximal MTA MIME processing and fixing up messages passing through the MTA with the inner MTA channel option.

Configuring Secure ETRN Command Support

Consider explicitly configuring the ETRN commands that the MTA honors. See the ETRN_ACCESS mapping table, the *etrn channel options, and the ALLOW_ETRNS_PER_SESSION TCP/IP channel option.

For more information, see:

  • The discussion on the ETRN_ACCESS mapping table in the Messaging Server Reference.

  • The discussion on channel configuration in the Messaging Server Reference.

ENS Security Guidelines

This section describes securing ENS Server (7997) with firewall and/or TCP Access Control Filters.

Note:

You can turn on ENS SSL and password based authentication. For additional information, see the discussion about ENS SSL and ENS password based authentication in the Messaging Server System Administrator's Guide.

To deploy the Event Notification Service (ENS) with Messaging Server, see the Messaging Server System Administrator's Guide.

Note:

The current implementation of ENS does not provide security on events that can be subscribed to. Thus, a user could register for all events, and portions of all other users' mail. Because of this it is strongly recommended that the ENS subscriber be on the safe side of the firewall at the very least.

A firewall system generally controls what TCP/IP communications are allowed between internal networks and the external world. Firewalls prevent packets considered to be unsafe from passing through.

Message Store Security Guidelines

The most important data in the Messaging Server is the data in the Message Store. Physical access and root access to the Message Store must be protected. "Protecting Messaging Components in Your Deployment" outlines general guidelines to protect your Message Store. In addition, you should review the discussion on managing the Message Store in the Messaging Server System Administrator's Guide. This section provides additional details in the following topics:

Securing Your Backup System

The process for backing up and restoring the Messaging Server is described in the Messaging Server System Administrator's Guide.

Some security guidelines to consider for Message Store backup:

  • Be sure that such a system does not leave unneeded data.

  • Backup systems that encrypt data improve your security if you manage the encryption keys properly.

Options for Securing Messaging Server

The http.feedback.notspam (Unified Configuration) or service.feedback.notspam (legacy configuration), http.feedback.spam (Unified Configuration) or service.feedback.spam (legacy configuration), and http.ipsecurity (Unified Configuration) or service.http.ipsecurity (legacy configuration) options are used to secure Messaging Server. See:

  • http.feedback.notspam (Unified Configuration) or service.feedback.notspam (legacy configuration) and http.feedback.spam (Unified Configuration) or service.feedback.spam (legacy configuration) in the Convergence System Administrator's Guide.

  • http.ipsecurity (Unified Configuration) or service.http.ipsecurity (legacy configuration) in "Recovering From Phishing Attacks That Have Compromised User Accounts."

Being Aware of IMAP ACLs

See the following discussions:

  • Granting permission for other users to access folders in the Messaging Server System Administrator's Guide

  • Describing the tasks that you use to administer shared folders in the Messaging Server System Administrator's Guide

Disabling IMAP Shared Folders if Not Needed

Disable shared folders if not in use. See the discussion on disabling shared folders in the Messaging Server System Administrator's Guide.

MMP Security Guidelines

The MMP serves as a proxy for the Message Store, therefore, it needs to protect access to end user data and guard against unauthorized access. "Protecting MMPs" outlines general guidelines.

You can use the server machine on which the multiplexor is installed as a firewall machine. By routing all client connections through this machine, you can restrict access to the internal Message Store machines by outside computers. The multiplexors support both unencrypted and encrypted communications with clients.

For more information, see the discussion on MMP badguy throttling and MMP connection limits in the Messaging Server Reference.

User Authentication Guidelines

User authentication allows end users to securely log in through their mail clients to retrieve their mail messages. "Planning Messaging User Authentication" outlines general guidelines. This section adds the following topics:

Acquiring SSL Server Certificates for the Server Domains

Refer to "Certificate-Based Authentication for Messaging Server" for more information.

Note the following recommendations:

  • Acquire SSL server certificates for server domains to which your users will connect from a third-party CA. If you also wish to secure inter-deployment connections, recommended for a geographically distributed deployment as well as to meet legal requirements in some jurisdictions, get certificates for your Directory Servers and back-end IMAP/POP storage servers.

  • Purchasing Certificate Authority (CA) service or software for your enterprise may be cost effective if you have many hosts in your deployment. Be sure to use at least 2048-bit RSA with SHA 256 signatures per current guidelines unless your jurisdiction does not permit that or some of your mail clients do not support that.

  • The certutil provided with Solaris and Messaging Server Installer can be used with the -g 2048 and -Z sha256 switches. Once enabled, you can configure SSL.

Some guidelines for SSL include:

  • Having a plan for SSL certificate or CA expiration.

  • Turning on SSL where required (external services, possible internal services)

  • Requiring SSL where possible (RestrictPlainPasswords, plaintextmincipher)

Requiring SMTP Authentication for Mail Submission

SMTP Authentication, or SMTP Auth (RFC 2554) is the preferred method of providing SMTP relay server security. SMTP Auth allows only authenticated users to send mail through the MTA. For more information, see:

  • The discussion on channel configuration in the Messaging Server Reference.

  • The discussion on SMTP authentication and SASL in the Messaging Server Reference.

  • The discussion on requiring password submission to login to Messaging Server in the Messaging Server System Administrator's Guide.

  • The discussion on the authrewrite channel option in the Messaging Server Reference.

Message Encryption Guidelines

"Planning Message Encryption Strategies" covers S/MIME and Encryption with SSL for encryption and privacy solutions. Review the following guidelines and recommendations:

Determining SSL Cipher Suites

See "Configuring Encryption and Certificate-Based Authentication."

Review your SSL Cipher framework to determine the following:

  • Do your mail clients work if the mail server only supports modern AES cipher suites, modern AES and slow-but-secure 3DES? Or, are legacy RC4 cipher suites required?

  • Is SSL3 required, or can you restrict to TLS?

Using Solaris Crypto Framework in Place of NSS Default Software Token

If you use SSL for encryption, you can improve server performance by installing a hardware encryption accelerator.

Security Considerations for Developers

For secure programming best practices, refer to the Messaging Server MTA Developer's Reference available in the Messaging Server 8.0 documentation set.