Chapter 12 Securing Instant Messaging Using TLS and Legacy SSL

Instant Messaging supports TLS (Transport Layer Security) and legacy SSL (Secure Sockets Layer) for secure communications. This chapter provides instructions on setting up security for Instant Messaging using these protocols in the following sections:

• Overview of Using TLS and Legacy SSL in Instant Messaging

• Setting Up TLS for the Instant Messaging Server

• Activating TLS on the Instant Messaging Server

• Setting Up Legacy SSL for the Multiplexor and Instant Messenger

• Invoking the Secure Version of Instant Messenger

Overview of Using TLS and Legacy SSL in Instant Messaging

Instant Messaging uses a startTLS extension to the Transport Layer Security (TLS) 1.0 protocol for client-to-server and server-to-server encrypted communications and for certificate-based authentication between servers. In addition, Instant Messaging supports a legacy implementation of the SSL protocol (version 3.0) for encrypted communications between Instant Messenger and the multiplexor. In the latter case, a certificate is used to validate the identity of the server to which the client connects, but certificates are not used for authentication.

Communication between multiplexor and server is over an unsecured transport. When you use TLS for client-to-server communication, the multiplexor simply passes the bytes from the client to the server and back and does not perform any encryption or decryption.

TLS is fully compatible with SSL and includes all necessary SSL functionality. TLS and SSL function as protocol layers beneath the application layers of XMPP and HTTP.

If you set up the multiplexor to only use legacy SSL, Instant Messenger will only connect to the multiplexor using SSL and will disregard any information returned from the server about TLS availability. However, if you choose to use legacy SSL with the multiplexor, all XMPP/HTTP Gateway instances should be configured to communicate directly with the server and not the multiplexor. The gateway does not support legacy SSL. Third-party clients that connect to the multiplexor over legacy SSL and then request a TLS connection are permitted to do so.

In addition, the multiplexor connects to the server over an unsecured transport. If you want to secure communications from end-to-end (client through multiplexor to server and back), use TLS instead of legacy SSL.

You must use Java 1.5 (minimum) in order to use TLS with the Instant Messaging server.

For information on TLS and StartTLS in XMPP, see “Use of TLS” in RFC 3920, Extensible Messaging and Presence Protocol: Core. For an overview of certificates, SSL, and TLS, see Introduction to Certificates and SSL in Sun Java System Application Server Enterprise Edition 8.2 Administration Guide. The procedures in this section assume you are using the Sun Java^TM System Application Server to generate certificates. If you are using another web container, you will need to refer to that web container's documentation for specific instructions on generating keystores and certificates.

Setting Up TLS for the Instant Messaging Server

Enabling TLS for Instant Messaging server-to-server and client-to-server communication requires the following general steps:

1. Creating a Java keystore (JKS) and a private key using the keytool utility.

   For an overview of the keytool utility, see Tools for Managing Security in Sun Java System Application Server Enterprise Edition 8.2 Administration Guide. For instructions on generating the JKS using Sun Java System Application Server, see Working with Certificates and SSL in Sun Java System Application Server Enterprise Edition 8.2 Administration Guide.

2. Using the private key to generate a server certificate for the Instant Messaging server.

   See Generating a Certificate Using the keytool Utility in Sun Java System Application Server Enterprise Edition 8.2 Administration Guide for instructions.

3. Getting the Instant Messaging server certificate signed by a Certificate Authority (CA).

   See Signing a Digital Certificate Using the keytool Utility in Sun Java System Application Server Enterprise Edition 8.2 Administration Guide for instructions. Replace Application Server with Instant Messaging where applicable.

4. Restart the Instant Messaging server.

   See Starting Instant Messaging Components for details.

5. Obtaining the CA's root certificate.

   Contact your CA for instructions on obtaining the CA's root certificate.

6. Import the certificates into the keystore.

   You import the CA root certificate and the signed server certificate into the keystore using the keytool utility as described in Using the keytool Utility in Sun Java System Application Server Enterprise Edition 8.2 Administration Guide.

7. Activating TLS in the server by setting the appropriate parameters in iim.conf.

   For instructions see Activating TLS on the Instant Messaging Server.

8. For server-to-server communication over TLS, you need to repeat these steps for each server that will be communicating over TLS. You do not need to perform anything to configure Instant Messenger to use TLS. You also do not need to configure the multiplexor for TLS, however you must not set up the multiplexor to use legacy SSL if you intend to use TLS.

9. If you are using the XMPP/HTTP Gateway in your deployment, configure the gateway to communicate directly with the Instant Messaging server and not the multiplexor.

If you are using the Sun Java System Application Server, steps 1 through 5 are documented in Working with Certificates and SSL in Sun Java System Application Server Enterprise Edition 8.2 Administration Guide of the Sun Java System Application Server Enterprise Edition 8.2 Administration Guide. Step 6 is described in Activating TLS on the Instant Messaging Server.

Activating TLS on the Instant Messaging Server

Before you can activate TLS on the server, you must create a JKS, obtain and install a signed server certificate, and trust the CA’s certificate as described in Setting Up TLS for the Instant Messaging Server. You activate TLS on the server when you want to use TLS for server-to-server and/or client-to-server communication.

Table 12–1 lists the parameters in iim.conf used to enable TLS in an Instant Messaging server. It also contains the description and the default value of these parameters.

/im-cfg-base/server-keystore.jks

To Activate TLS Communication in the Instant Messaging Server

Use this procedure to configure the Instant Messaging server to use secure communication over TLS in the following ways:

• Require TLS for all client and server connections.

• Require TLS only for specific server-to-server connections.

• Allow TLS connections for clients and servers that request a secure transport after the initial communication session has been set up.

• A Combination of requiring TLS for specific server-to-server connections and allowing TLS connections for other clients and servers.

Before You Begin

Ensure that you have created a JKS, obtained and installed a server certificate, and configured the server to trust the CA’s certificate as described in Setting Up TLS for the Instant Messaging Server.

For server-to-server TLS communication, you must complete this procedure on each server you want to configure to use TLS.

1. Add values for the following parameters in iim.conf.

   If the parameters are not already present in iim.conf, add them.

   iim_server.sslkeystore=server-keystore.jks iim_server.keystorepasswordfile=sslpassword.conf

   The server will now respond to a connection request from any client or another Instant Messaging server with the information that it is able to communicate over TLS. The requesting client or server then chooses whether or not to establish a secure connection over TLS.

2. If you want the server to require TLS for all connections from clients, and remote and peer servers, add the following parameter to iim.conf:

   iim_server.requiressl=true

   If you set this parameter to true, the server will terminate a connection with any client or remote or peer server that does not support TLS. Use this parameter to require secure client-server communication over TLS.

   See Chapter 8, Federating Deployment of Multiple Instant Messaging Servers for more information about server-to-server communication.

3. If you want to require TLS for communication with a specific remote or peer server, add the following parameter to iim.conf:

   iim_server.coserver1.requiressl=true

   Set this parameter for each coserver for which you want to require TLS.

   If you set iim_server.requiressl to true, the server will require a TLS connection for any server with which it communicates. In this case, you do not need to set this parameter for specific coservers.

4. (Optional) If you want the server to trust all certificates it receives, and to add certificate information to the log files, add the following parameter to iim.conf:

   iim_server.trust_all_cert=true

   ---

   Caution –

   You might need to use this feature to test your deployment before you go live. However, you typically should not do this on a deployed system as it presents severe security risks. If this value is true, the server will trust all certificates, including expired and self-signed certificates, and will also add the certificate information into the log files. If false, the server will not log certificate information and will only trust valid certificates signed by a CA.

   ---

5. Refresh the server configuration using imadmin.

   imadmin refresh server

6. Verify that TLS is working properly.

   You can do this a number of ways, for example by following the steps in Invoking the Secure Version of Instant Messenger.

Example 12–1 TLS Configuration in iim.conf

The following is an example section of an iim.conf file with the required TLS configuration for server-to-server and client-to-server communication. Values for the parameters in this example will be different in your deployment.

! Server to server communication port.
iim_server.port = "5269"
! Should the server listen on the server to server
! communication port
iim_server.useport = "True”
iim_server.coservers=coserver1
iim_server.coserver1.serverid=Iamcompany22
iim_server.coserver1.password=secretforcompany22
iim_server.coserver1.host=iim.i-zed.com:5269
iim_server.serverid=Iami-zed
iim_server.password=secret4i-zed
iim_server.trust_all_cert=true
iim_server.sslkeystore=/var/im/server_keystore.jks
iim_server.keystorepasswordfile=/var/im/sslpassword.conf

Setting Up Legacy SSL for the Multiplexor and Instant Messenger

If you are using an Instant Messaging client that does not support TLS, you can still use SSL instead of TLS for client-to-multiplexor communication. If you configure the multiplexor to use SSL, you cannot use TLS for client-to-server communication. All communication between the multiplexor and the server will be in clear text over an unsecured transport.

If you set up legacy SSL on the multiplexor and are using the XMPP/HTTP Gateway, you must configure the gateway to communicate directly with the server, not the multiplexor. The gateway does not support legacy SSL.

Enabling SSL between the multiplexor and Instant Messenger requires the following:

1. Requesting an SSL Certificate for the Instant Messaging Multiplexor from the CA.

2. Installing the Certificate.

3. Enabling Legacy SSL Between the Multiplexor and Instant Messenger.

4. Activating TLS on the Instant Messaging Server.

5. Invoking the Secure Version of Instant Messenger.

Requesting an SSL Certificate for the Instant Messaging Multiplexor from the CA

To enable SSL in the multiplexor, you need to request a certificate.

To Request a Certificate for the Instant Messaging Multiplexor

This section assumes you are requesting the certificate using either the Sun Java System Web Server or Sun Java System Application Server as your web container.

The multiplexor uses NSS for certificate management, so you can use the NSS utilities to create, manage, and use certificates and the certificate database.

1. In a web browser, type the following URL to start the web container's administration server:

   http://hostname.domain-name:administration-port

   A window prompting you for a user name and password appears.

2. Type the administration user name and password you specified during the Web Server or Application Server installation.

   The Administration Server page appears.

3. Create a separate Web Server or Application Server instance.

   For more information on installing multiple instances of the Application Server, see the Sun Java System Application Server Enterprise Edition 8.2 Installation Guide. For information about installing multiple instances of Web Server, see the Sun Java Communications Suite 5 Installation Guide.

4. Create a trust database to store the public and private keys, referred as the key-pair file.

   The key-pair file is used for SSL encryption.

   For information on creating a trust database, see Chapter 9, Configuring Security, in Sun Java System Application Server Enterprise Edition 8.2 Administration Guide for Application Server and Chapter 6, Certificates and Keys, in Sun Java System Web Server 7.0 Administrator’s Guide for Web Server.

5. Request a certificate from the CA.

   For more information on requesting a certificate, see Chapter 9, Configuring Security, in Sun Java System Application Server Enterprise Edition 8.2 Administration Guide for Application Server and Chapter 6, Certificates and Keys, in Sun Java System Web Server 7.0 Administrator’s Guide for Web Server.

Installing the Certificate

After you receive the signed server certificate from your Certificate Authority, you need to install the certificate and create databases for secure communication.

To Install the Certificate for the Instant Messaging Multiplexor

1. In a web browser, type the following URL to start the administration server:

   http://hostname.domain-name:administration-port

   A window appears, prompting you for a user name and password.

2. Type the administration user name and password you specified during the Web Server or Application Server installation.

   The Administration Server page appears.

3. Install the server certificate.

   For more information on installing the certificate, see the Web Server or Application Server product documentation at http://docs.sun.com

4. Change to your Web Server or Application Server’s /alias directory.

5. Copy the database files from the /alias directory to the Instant Messaging server's im-cfg-base directory.

   For example, on Solaris:

   cp https-serverid-hostname-cert8.db /etc/opt/SUNWiim/default/config/cert8.db

   cp https-serverid-hostname-key3.db /etc/opt/SUNWiim/default/config/key3.db

   cp secmod.db /etc/opt/SUNWiim/default/config/secmod.db

   and on Linux:

   cp https-serverid-hostname-cert8.db /etc/opt/sun/im/default/config/cert8.db

   cp https-serverid-hostname-key3.db /etc/opt/sun/im/default/config/key3.db

   cp secmod.db /etc/opt/sun/im/default/config/secmod.db

   ---

   Note –

   You need to allow Read permission on the cert7.db, key3.db, and secmod.db files for the system user used by the multiplexor. In addition, if you created multiple instances of Instant Messaging, the name of the /default directory will vary depending on the instance.

   ---

   See Table 3–1 for default locations for im-cfg-base.

6. Change to your im-cfg-base on the multiplexor's host.

   See Instant Messaging Server Directory Structure for information on locating im-cfg-base.

7. Create a file named sslpassword.conf using a text editor of your choice.

8. Enter the following line in sslpassword.conf.

   Internal (Software) Token:password

   Where password is the password you specified when you created the trust database.

9. Save and close sslpassword.conf.

10. Ensure that all Instant Messenger end users have Ownership and Read permission on sslpassword.conf.

11. Restart the multiplexor.

12. Verify that SSL is working properly.

    You can do this a number of ways, for example by following the steps in Invoking the Secure Version of Instant Messenger.

13. Log in to the Web Server or Application Server as an administrator.

14. Remove the server instance that you created while requesting the certificate.

Enabling Legacy SSL Between the Multiplexor and Instant Messenger

You enable SSL for client-to-multiplexor communication by modifying parameters in iim.conf and then connecting to the multiplexor using the secure version of the Instant Messenger client.

Table 12–2 lists the parameters in iim.conf for enabling SSL between Instant Messenger and the multiplexor. It also lists the description and the default value of these parameters.

To Enable SSL Between Instant Messenger and the Multiplexor

1. Open iim.conf.

   See iim.conf File Syntax for instructions on locating and modifying iim.conf.

2. Add the values from Table 12–2 to the multiplexor configuration parameters in iim.conf.

Example 12–2 Legacy SSL Multiplexor Configuration in iim.conf

The following is an example of iim.conf with the multiplexor configuration parameters included:

! IIM multiplexor configuration
! =============================
!
! Multiplexor specific options

! IP address and listening port for the multiplexor.
! WARNING: If this value is changed, the port value of ’-server’
! argument in the client’s im.html and im.jnlp files should 
! also be changed to match this.
iim_mux.listenport = "siroe.com:5222"

! The IM server and port the multiplexor talks to.
iim_mux.serverport = "siroe.com:45222"

! Number of instances of the multiplexor.
iim_mux.numinstances = "1"

! Maximum number of threads per instance
iim_mux.maxthreads = "10"

! Maximum number of concurrent connections per multiplexor process
iim_mux.maxsessions = "1000"

                         
iim_mux.usessl = "on"
iim_mux.secconfigdir = "/etc/opt/SUNWiim/default/config"
iim_mux.keydbprefix = "This-Database"
iim_mux.certdbprefix = "Secret-stuff"
iim_mux.secmodfile = "secmod.db"
iim_mux.certnickname = "Multiplexor_Cert"
iim_mux.keystorepasswordfile = "sslpassword.conf"

Invoking the Secure Version of Instant Messenger

Instant Messenger automatically supports TLS. If you have configured the server to use TLS as described in Activating TLS on the Instant Messaging Server then the server will communicate to the client that it can support a TLS session when Instant Messenger connects to the server. Instant Messenger can then request that the connection be changed to use TLS.

You invoke the legacy SSL version of Instant Messenger by accessing imssl.html or imssl.jnlp from your web browser. These files are located under the resource directory, the base directory under which all the Instant Messenger resources are stored.

The links to these applet descriptor files can also be added to index.html.

Once you have configured legacy SSL for the multiplexor or TLS for the server, you can verify that the Instant Messenger client has made a secure connection.

To Verify a Secure Instant Messenger Connection

1. Log in to Instant Messenger.

   If you are using legacy SSL, access imssl.html or imssl.jnlp from your web browser. If you are using TLS, access the client normally. See Invoking Instant Messenger for information.

   Instant Messenger will always use TLS if it is available and if the multiplexor is not set up for legacy SSL.

2. On the Instant Messenger Main Window, ensure the lock icon is visible.

   The lock icon appears on the bottom right corner of the Main Window when Instant Messenger is using a secured transport, either SSL or TLS.