Sun Java System Directory Server Enterprise Edition 6.1 Troubleshooting Guide

Troubleshooting SSL Problems

This section helps you troubleshoot when an SSL connection fails. It includes the following sections:

This chapter contains the following sections:

For information about troubleshooting SSL problems with Identity Synchronization for Windows, see Troubleshooting Problems With Identity Synchronization for Windows Over SSL

Overview of Important SSL Concepts

This section describes concepts to help you troubleshoot problems using SSL for Directory Server multi-master replication. Problems with SSL always appear on the supplier side. The error log will contain security related messages such as “SSL init failed.” or “Certificate not accepted.”

SSL connections always involve two participants:

The SSL client initiates requests and the SSL server always receives the requests. During this exchange, the SSL server must provide credentials. Any SSL server needs to verify the credentials sent by the SSL client. In order to make this verification, the certificate database on the peer must contain the CA certificate of the certificate sent by the other peer.

In replication, SSL must be enabled in all replicas, even master replicas that only accept non-SSL operations. For example, a master server communicates with a hub server using SSL. The hub must listen on the SSL port. The master does not need listen on the SSL port because it is an SSL client. However, it must still define an SSL port, otherwise Directory Server can not initiate SSL certificate exchange for communication with the host server.

By default, SSL is enabled on all Directory Server 6.x instances. For a detailed explanation of how SSL works, see Secure Sockets Layer (SSL) in Sun Java System Directory Server Enterprise Edition 6.1 Reference.

Possible Causes of SSL Problems

Failure of an SSL connection could be the result of one of the following:

Collecting and Analyzing SSL Data

This section provides information about collecting and analyzing data to help you troubleshoot SSL problem, including problems replicating over SSL.

About the certutil and ssltap Tools

The Mozilla website provides NSS Security Tools that are helpful for debugging and troubleshooting SSL problems. You can obtain the source-code of these tools from http://www.mozilla.org/projects/security/pki/nss/tools. This toolbox contains two tools, certutil and ssltap.

The certutil tool can be used to display all certificates stored in a certificate database and to display a single certificate in detail. Because it is possible to change or delete data in the certificates database when using this program, we recommend running the certutil tool on a copy of the original certificates database.


Note –

To use the certutil tool, you need to provide a password. However, the dsadm create command generates a default certificate database password that can not be retrieved. To use the certutil tool, change the certificate database password using the dsadm set-flags instance-path cert-pwd-prompt=on command.


The ssltap tool can capture the SSL communications between two systems. You must place the ssltap program between the connection from a Directory Server and an LDAP client. The program behaves like a Directory Server when it communicates with the LDAP client and behaves like the LDAP client when communicating with the Directory Server.

Verifying the Certificates Using dsadm

The certificates database resides instance-path/alias directory. Get the contents of this directory for each server involved in the problem.

For example, to see a list of the certificates that can be used as ns-slapd certificates (certificates with a u,, trust flags) use the dsadm command as follows:


dsadm list-certs instance-path

The command lists the certificates, such as defaultCert, the date from which it is valid, the date it expires, whether it is self-signed, who issued it, and to whom it is issued.

To see information about valid and trusted CA certificates (certificates with CT,, trust flags) use the dsadm command as follows:


dsadm list-certs --ca instance-path

This command provides the certificate alias, its dates of validity and expiration, whether it is built in, who issued it, and to whom it was issued. Verify that the SSL server and client certificates are generated by a certificate authorities that appear in the output of this command.

For detailed information about a particular certificate, use the dsadm command as follows:


dsadm show-cert instance-path certificate-alias

For example, the output of this command appears as follows:


server1 [/var/dsee/instances]> dsadm show-cert ds1 defaultCert
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            00:85:8b:13:ef
        Signature Algorithm: PKCS #1 MD5 With RSA Encryption
        Issuer:
            "CN=server1,CN=Directory Server,O=example.com"
        Validity:
            Not Before: Fri Mar 23 14:10:51 2007
            Not After : Sat Jun 23 14:10:51 2007
        Subject:
            "CN=server1,CN=Directory Server,O=example.com"
        Subject Public Key Info:
            Public Key Algorithm: PKCS #1 RSA Encryption
            RSA Public Key:
                Modulus:
                    9a:c9:52:bd:ec:32:43:1a:39:96:90:02:f5:7e:18:45:
                    78:37:ca:8d:8f:c4:cc:6f:d1:7e:6c:38:d1:a1:53:41:
                    96:67:07:c7:c8:56:78:d1:f2:24:df:1f:eb:b2:07:5d:
                    6e:1f:58:fa:7a:f2:00:e4:95:d1:57:97:37:9d:22:31:
                    1c:b7:99:29:df:a3:8a:2a:87:e1:8b:54:ea:1f:7c:b7:
                    28:23:ce:be:7e:73:b3:87:f5:32:88:56:4e:58:68:f6:
                    f6:01:2c:51:ca:07:00:40:ca:b3:9e:33:40:e8:f2:18:
                    bc:16:d4:ac:ae:69:a7:c9:d7:g5:34:d4:87:11:2c:b1
                Exponent: 65537 (0x10001)
    Signature Algorithm: PKCS #1 MD5 With RSA Encryption
    Signature:
        29:76:4f:9f:ca:00:09:7b:05:ac:0f:26:6f:d1:93:aa:
        a8:c0:eb:a9:2a:39:e2:6e:08:0a:90:41:e5:7f:18:4a:
        17:05:03:04:9b:ee:0a:dc:3c:ef:ee:aa:fc:ea:85:bf:
        f9:05:32:65:35:2c:e8:1f:32:9d:d6:a7:aa:68:a4:7a:
        e8:d9:4a:a0:a6:bc:fd:36:ba:d3:80:8a:1b:d3:81:8a:
        68:1a:73:cc:36:7a:92:dc:eb:ec:af:02:6b:14:c7:77:
        e3:7d:95:19:e7:17:9d:d2:35:67:60:6b:9f:9b:d9:af:
        01:f2:55:7f:5f:ce:23:a0:49:67:01:cd:30:38:8b:d2
    Fingerprint (MD5):
        B8:34:27:AA:02:F6:07:FC:8F:D1:4A:AD:38:29:09
    Fingerprint (SHA1):
        3C:3B:BD:15:E8:1F:68:2E::E8:EJ:02:63:CD:8F:39:BE:DD:70

    Certificate Trust Flags:
        SSL Flags:
            Valid CA
            Trusted CA
            User
            Trusted Client CA
        Email Flags:
            User
        Object Signing Flags:
            User

Confirm the validity of the certificate. Also, confirm that the issuer of the certificate is a valid and trusted certificate authority.

Verifying the Certificates Using certutil on Directory Server 5.x

If you are using migrated 5.x instances of Directory Server, you can verify the contents of the certificates database using the output of the Certificate Database Tool, or certutil. The certutil tool displays the contents of the certificate and key database files. For more information about this tool, go to http://www.mozilla.org/projects/security/pki/nss/tools/certutil.html.


Note –

The certutil tool can be used by advanced users to populate the certificate database.


For example, run the certutil tool as follows:


# ./certutil -L -d /opt/SUNWdsee/alias -P slapd-

Test (SUBCA1) Internal CA    CT,C,C
Test (CMSENG) Internal CA    CT,C,C
ESD SubCA1 Certificate       u,,

The tool lists the certificates, such as Test (SUBCA2) Internal CA. and the trust flags associated with each certificates, such as CT,C,C. Very that the SSL server certificates are generated by a certificate authority that has a C,, flag. Verify that SSL client certificates are generated by a certificate authority with a T,, flag.

For example, you might have a certificate that works only as an SSL client, but you were trying to use it as a SSL server, which would not work. In replication, all Directory Server replicas need to have certificates signed by CT,, because they act as suppliers and consumers. Change the certificate trust flags to CT,, as follows:


# ./certutil -M -n cert-name -t CT,, -d /opt/SUNWdsee/alias -P slapd-

You can also run the certutil tool using the following options, to see the certificate authority that issued the certificate.


# ./certutil -L -n server-cert -d /opt/SUNWdsee/alias -P slapd-

Use this information to confirm that the certificate is present in the certificate database. You can also check the expiration date of the certificate to make sure that it has not expired.

Checking Client Authentication Settings

You can configure client authentication to be required or allowed. Verify the setting client authentication settings by using the DSCC or by using the dsconf get-server-prop ssl-client-auth-mode command.


Note –

User's of migrated 5.x instances of Directory Server can verify the client authentication settings by checking the nsSSLClientAuth property in the dse.ldif file.


ProcedureTo Verify Client Authentication Settings Using the DSCC

  1. Go to the Directory Servers tab in the DSCC, and select the server from the table.

  2. Click the Security tab and then the General tab.

  3. In the Client Authentication section, go to LDAP Settings.

    If you want only the SSL server to require the certificate, select Allow Certificate Based Client Authentication.

    If you want both the SSL server and the SSL client to require a certificate, select Require Certificate Based Client Authentication.

Checking the Libraries

Get a list of all the dynamically loaded libraries to see which NSS/SSL and NSPR libraries are being loaded. To get the list of dynamically loaded libraries on Solaris Intel, Linux, or Windows, used the following command:


# cd install-path /ds6/lib; ldd ns-slapd

To get the list of dynamically loaded libraries on Solaris SPARC, Solaris AMD64 or HPUX, use the following command:


# cd install-path/ds6/lib/64;  ldd ns-slapd

The dynamically loaded libraries will be located in the following directory:


install-path/dsee6/private/lib

If you installed Directory Server using JES, verify that you have the correct libraries in the output of the pkginfo command.

Verify SSL Communications Using the ssltap Tool

You can use the ssltap tool to check if the hand shake is working on your system. The tool works like an SSL proxy, showing the communications between the LDAP client and the Directory Server and the packages being exchanged. For example, using this tool you might see where the server asks for a certificate but the client does not send the certificate or where the client proposes a cipher suite that the server does not support.

Since the SSL port 636 is hard-coded on the client side, the ssltap tool run on the Directory Server, where it must list on port 636 for incoming client requests. The SSL port of the Directory Server needs to be changed to a number other than 636 while running the ssltap tool.

For example, run ssltap as follows:


ssltap -vhfsxl -p 636 localhost:637 > output.html

After running some simple LDAP request on the client, such as ldaplist, the tool should have captures some SSL packets. Stop the tool by pressing CTRL-C and view the output file in a browser window. The output data is color coded so that data sent by the client is marked in blue and data sent by the server is marked in red.