Chapter 9 Troubleshooting

This chapter provides information to help you troubleshoot problems you may encounter while using Identity Synchronization for Windows. The information is organized as follows:

Troubleshooting Checklist


Note	Administrators: When you are debugging problems, adjust the logging level (as described in Configuring Your Log Files) to ensure the log reflects all events that may be causing problems. Some events (such as the program failing to synchronize a user change because the user was not included in the SUL) are not included in a log file until you adjust the log level to FINE or higher. The log level should be left at INFO during all idsync resync operations. The idsync printstat command can be a useful tool as you install and configure Identity Synchronization for Windows. When you run printstat (see Using printstat), it displays a list of the remaining steps you have to perform to complete the installation and configuration process.

Are there any problems reported in the central error.log?

isw-<hostname>/logs/central/error.log

Almost all errors will be reported in the central error log file. Also, additional information about any error is usually available in the audit.log file. To ease correlation of related log entries, the audit.log file also includes all entries in the error log.

The Release Notes document many known issues. Is this problem explained there?

Was the installation performed on a clean machine? Problems might occur when this product is reinstalled if the uninstallation of the previous configuration was not complete. Please refer to Chapter 8, "Removing the Software" for more instructions about how to clean up previous installations.

Was the Core properly installed? If Core installation completed successfully, then log files will exist in the isw-<hostname>/logs/central/ directory.

Was the Directory Server running during resource configuration?

Is the Core, including the Message Queue and the System Manager, currently running? On Windows, check for the appropriate service name. On Solaris, check for the appropriate daemon name. Use the idsync printstat command to verify that the Message Queue and System Manager are active.

Was a configuration saved successfully? If the idsync printstat command lists connectors, then a configuration was saved successfully.

Were all connectors installed? One connector must be installed for each directory source being synchronized.

Were all subcomponents installed? Directory Server and Windows NT Connectors require subcomponents to be installed after the Connector installation. The Directory Server Plugin must be installed in each Directory Server replica.

Were post-installation procedures followed? The Directory Server must be restarted after the Directory Server Plugin is installed. The Windows NT Primary Domain Controller must be restarted after the Windows NT subcomponents are installed.

Was synchronization started either from the Console or command line?

Are all connectors currently running?

Verify that all connectors are in the SYNCING state using the Console or idsync printstat.

Are the directory sources being synchronized currently running?

Verify using the Console that modifications and/or creates are synchronized in the expected direction(s).

If synchronizing users that existed in only one directory source, were these users created in the other directory source using the idsync resync command?



Note	You must run idsync resync whenever there are existing users. If you do not resynchronize existing users, resynchronization behavior remains undefined.

If synchronizing users that existed in both directory sources, were these users linked using the idsync resync command?

If user creates fail from Active Directory or Windows NT to the Sun Java System Directory Server, verify that all mandatory attributes in the Directory Server objectclass are specified as creation attributes and values for the corresponding attributes are present in the original user entry.

If synchronizing creates from Directory Server to Windows NT and the user creation succeeded, but the account is unusable, verify that the user name does not violate Windows NT requirements.

For example, if you specify a name that exceeds the maximum allowable length for Windows NT, the user will be created on NT but will remain unusable and uneditable until you rename the user (User > Rename).

For the Windows NT SAM Change Detector subcomponent to be effective, you must turn on the NT audit log. Select Start > Programs > Administrative Tools > User Manager, and then select Policies > Audit Policies.
Select Audit These Events and then both the Success and Failure boxes for User and Group Management.

Select Event Log Settings in the Event Viewer >Event Log Wrapping, and then select Overwrite Events as Needed.

Are the users that fail to synchronize within a Synchronization User List? For example, do they match the base DN and filter of a Synchronization User List? In deployments that include Active Directory, on-demand password synchronization fails silently if the Sun Java System Directory Server entry is not in any Synchronization User List. This most often occurs because the filter on the Synchronization User List is incorrect.

Were the synchronization settings changed? If the synchronization settings changed from only synchronizing users from Active Directory to the Sun Java System Directory Server to synchronizing users from the Directory Server to Active Directory, then the Active Directory SSL CA certificate must be added to the connector’s certificate database. The idsync certinfo command reports what SSL certificates must been installed based on the current SSL settings.

Are all host names properly specified and resolvable in DNS? The Active Directory domain controller should be DNS-resolvable from the machine where the Active Directory Connector is running and the machine where the Sun Java System Directory Server Plugin is running.

Does the IP address of the Active Directory domain controller resolve to the same name that the connector uses to connect to it?

Does the source connector detect the change to the user? Use the central audit.log to determine if the connector for the directory source where the user was added or modified detects the modification.

Does the destination connector process this modification?

Are multiple Synchronization User Lists configured? If so, are these in conflict? More specific Synchronization User Lists should be ordered before less specific ones using the Console.

If flow is set to bidirectional or from Sun to Windows and there are Active Directory data sources in your deployment, are the connectors configured to use SSL communication?

If memory problems are suspected on Solaris environments check the processes. To view which components are running as different processes, enter

/usr/ucb/ps -gauxwww | grep com.sun.directory.wps

The output gives the full details including the ID of connectors, system manager and central logger. This can be useful to see if any of the processes are consuming excessive memory.

If you are creating or editing the Sun Java System Directory source, and the Directory Server does not display in the Choose a known server drop-down list, check that the Directory Server is running. The Directory Server must be running to appear in the drop down list of available hosts.

If the server in question is down temporarily, type the host and port into the Specify a server by providing a hostname and port field.



Note	Identity Synchronization for Windows uses a short host name by default; however, the default host name may not work with your configuration. We recommend using a fully qualified name whenever you are asked to provide a host name.

Do you receive the following error while running uninstaller program?

./runInstaller.sh

IOException while making /tmp/SolarisNativeToolkit_5.5.1_1 executable:java.io.IOException: Not enough space

java.io.IOException: Not enough space

Increase the size of the swap file mounted at /tmp.

Troubleshooting Connectors

Use the information in this section to troubleshoot problems with your connectors. The information is organized as follows:

How to Determine the ID of a Connector Managing a Directory Source

How to Determine a Connector’s Current State

How to Determine the ID of a Connector Managing a Directory Source

Using the Central Logs

Using idsync printstat

Using the Central Logs

Determine the connector IDs of the directory sources being synchronized by looking in the central audit.log. At startup, the central logger logs the IDs of each connector and the directory source that it manages. Look for the last instance of the startup banner for the most recent information.

CNN101 is a Sun Directory Connector that manages dc=airius,dc=com

CNN100 is an Active Directory Connector that manages the airius.com domain

[2003/03/19 00:00:00.722 -0600] INFO 16 "System Component Information: SysMgr_100 is the system manager (CORE); console is the Product Console User Interface; CNN101 is the connector that manages [dc=airius,dc=com (ldap://host1.airius.com:389)]; CNN100 is the connector that manages [airius.com (ldaps://host2.airius.com:636)];"

Using idsync printstat

The connector IDs and status are also available from the idsync printstat command (see Using printstat).

How to Determine a Connector’s Current State

You can determine the current state of the connectors involved in synchronization, using the Status pane in the Console, the idsync printstat command (as shown previously), or by looking in the central audit.log.

Search for the last message in the audit.log that reports the connector state.
For example, in the following log message you can see that connector CNN101 is in the READY state.

What to Do if the Connector is in the UNINSTALLED State

What to Do if the Connector Install Failed but You Cannot Reinstall

If the connector installation failed, but the Identity Synchronization for Windows installation program thinks that the connector is installed, the installation program will not allow you to reinstall the connector.

Table 9-1 Connector State Meanings
State	Meaning
UNINSTALLED	The connector has not been installed.
INSTALLED	The connector has been installed, but it has not received its configuration.
READY	The connector has been installed and has received its configuration, but it has not started to synchronize.
SYNCING	The connector has been installed, has received its configuration, and has attempted to start synchronizing.

Run idsync resetconn (as described in Using resetconn) to reset the connector’s state to UNINSTALLED, and then re-install the connector.

What to Do if the Connector is in the INSTALLED State

If a connector remains in the installed state for a long period of time, then most likely it is not running, or it is unable to communicate with the Message Queue.

At the machine where the connector was installed, look in the connector’s logs (audit.log and error.log) for potential errors. If the connector cannot connect to the Message Queue, then that error will be reported here. If this is the case, see Troubleshooting Message Queue for possible causes.

If the most recent messages in the audit log are old, then perhaps the connector is not running. See Troubleshooting Components.

What to Do if the Connector is in the READY State

A connector remains in the READY state until synchronization has been started and all of its subcomponents have been installed and have connected to the connector. If synchronization has not been started, then start it using the Console or command line utility.

If synchronization has been started, but a connector does not enter the SYNCING state, then there is likely a problem with subcomponent. See Troubleshooting Subcomponents.

What to Do if the Connector is in the SYNCING State

If all connectors are in the SYNCING state, but modifications are not being synchronized, then verify that the synchronization settings are correct:

Using the Console, verify that modifications and/or creates are synchronized in the expected direction (for example, from Windows to the Sun Java System Directory Server).

Using the Console, verify that the attribute being modified is a synchronized attribute (note: passwords are always synchronized). If created user entries are not being synchronized, then verify that user creation flow is enabled in the Console.

What to Do if the Active Directory Connector Fails to Contact Active Directory Over SSL

If the Active Directory Connector fails to contact Active Directory over SSL and the following error message displays, restart the AD domain controller.

Troubleshooting Components

Use the information in this section to troubleshoot components. The information is organized as follows:

On Solaris

On Windows

Examining WatchList.properties

On Solaris

The command /usr/ucb/ps -auxww | grep com.sun.directory.wps will list all of the Identity Synchronization for Windows processes running. This table shows which processes should be running.

If the expected number of processes are not running, then issue the following commands to restart all Identity Synchronization for Windows processes.

Table 9-2 Identity Synchronization for Windows Processes
Java Process Class Name	Component	When Present
com.sun.directory.wps.watchdog.server.WatchDog	System Watchdog	Always
com.sun.directory.wps.centrallogger.CentralLoggerManager	Central Logger	Only where Core is installed
com.sun.directory.wps.manager.SystemManager	System Manager	Only where Core is installed
com.sun.directory.wps.controller.AgentHarness	Connector	One for each connector installed

If the WatchDog process is running, but the expected number of java.exe processes are not running, then see the “Examining WatchList.properties” section to verify that all components were installed properly.

Like other system components, the Sun Java System Directory Server Plugin sends log records over the bus that are managed by the central logger for end-user viewing. However, the Plugin also logs some messages that may not show up over the bus (for instance when the subcomponent cannot contact the connector). In this case the log messages only show up in the Plugin’s logs directory on the file system, which should look something like the following:

Because the Plugin runs with the Directory Server process, there could potentially be a problem for the Plugin’s ability to write into its logs directory. This happens if the directory server runs as a different user than the owner of the logs directory. In this case, it may be necessary to give the Plugin permission explicitly by changing the directories permission or owner using native operating system commands.

On Windows

Using the Service control panel, check that the “Sun Java System Identity Synchronization for Windows service is started. If it is not started, then Identity Synchronization for Windows is not running on that machine, and should be started. If the service is started, then verify using the Task Manager that pswwatchdog.exe (the Watchdog process) is running and that the expected number of java.exe processes are running:

One for the Message Queue broker only if the Core is installed

One for the System Manager only if the Core is installed

One for the Central Logger only if the Core is installed

One for each Connector installed on that machine

Examining WatchList.properties


Note	There might be other active java processes, such as the Directory Server Console. If pswwatchdog.exe is not running, then restart the “Sun Java System Identity Synchronization for Windows” service. If it is running but the expected number of java.exe processes are not running, then see Examining WatchList.properties to verify that all components were installed properly.

On each machine where a Identity Synchronization for Windows component is installed, the isw-<machine_name>/resources/WatchList.properties file enumerates the components that should run on that machine. The process.name[n] properties name the components that should be running.

On machines where Core is installed, WatchList.properties will include entries for the Central Logger and System Manager:

On machines where connectors are installed, WatchList.properties will include a separate entry for each connector. The process.name property is the connector ID:

If there is a mismatch between the entries in WatchList.properties and the actively running processes, then restart the Identity Synchronization for Windows daemon or service.

If there are fewer than expected entries in WatchList.properties (e.g. only one connector entry even though two were installed), then examine the installation logs for possible installation failures.

On Solaris systems: Installation logs are written to /var/sadm/install/logs/

On Windows systems: Installation logs are written to the %TEMP% directory, which is a subdirectory of the Local Settings folder located under

C:\Documents and Settings\Administrator



Note	On some Windows systems (such as Windows 2000 Advanced Server), the Local Settings folder is a hidden folder. To view this folder and the Temp subdirectory: Open your Windows Explorer and select Tools > Folder Options from the menu bar. When the Folder Options dialog box is displayed, select the View tab and enable the Show Hidden Files option.

Troubleshooting Subcomponents

Have all subcomponents been installed?

Subcomponent installation must be done after the connector is installed:

For Active Directory Connectors, no subcomponents are installed.

For Sun Java System Directory Server Connectors, the Directory Server Plugin must be installed on the Sun Java System Directory Server being synchronized.

For Windows NT Connectors, the Windows Change Detector and Password Filter subcomponents must be installed on the primary domain controller for each Windows NT domain being synchronized. These two subcomponents are installed together after the Windows NT Connector has been installed.



Note	For the Windows NT SAM Change Detector subcomponent to be effective, you must turn on the NT audit log. Select Start > Programs > Administrative Tools > User Manager, and then select Policies > Audit Policies. Select Audit These Events and then both the Success and Failure boxes for User and Group Management. Select Event Log Settings in the Event Viewer > Event Log Wrapping, and then select Overwrite Events as Needed.

Have the subcomponent post-installation steps been followed?

After the Directory Server Plugin has been installed at the Sun Java System Directory Server, the server must be restarted. After the NT Change Detector and Password Filter have been installed on the primary domain controller, the server must be restarted.

Are the subcomponents running?

Is the Directory Server where the Plugin was installed running? Is the Primary Domain Controller where the Change Detector and Password Filter were installed running?

Have the subcomponents established a network connection to the connector?

On the machine where the connector is running, verify that the connector is listening for the subcomponent’s connection by running netstat -n -a. The following examples show the results of this command for three different scenarios. (The connector was configured to listen on port 9999.)

The connector is listening for incoming connections, and the subcomponent has successfully connected, which is the expected result:

netstat –n –a | grep 9999

*.9999 *.* 0 0 65536 0 LISTEN

12.13.1.2.44397 12.13.1.2.9999 73620 0 73620 0 ESTABLISHED

12.13.1.2.9999 12.13.1.2.44397 73620 0 73620 0 ESTABLISHED

The connector is listening for incoming connections, but the subcomponent has not connected:

# netstat –n –a | grep 9999

*.9999 *.* 0 0 65536 0 LISTEN

After verifying that the subcomponent is running, examine the subcomponent’s local logs for potential problems.

The connector is not listening for incoming connections:

# netstat –n –a | grep 9999

Verify that the correct port number was specified. Verify that the connector is running and is in the READY state. Examine the connector’s local logs for potential problems.

Troubleshooting Message Queue

Verify that the Sun Java System Message Queue broker is running. Issuing a telnet command to the machine and port where the Message Queue broker is running will return a list of the active Message Queue services:

If the “ssljms tcp NORMAL” service is not listed in the output, then examine the Message Queue logs for potential problems. If the Core was installed on Solaris, then the Message Queue broker’s log is:

/var/imq/instances/psw-broker/log/log.txt

If the Core was installed on Windows, then the broker’s log is:

<installation_root>\isw-<machine_name>\imq\var\instances\isw-broker\ log\log.txt

If the telnet command fails, then either the broker is not running or the wrong port was specified. Check the port number in the broker’s log. The broker’s port is specified in the following line

If the broker is not running, then it can be started on Solaris by running /etc/init.d/imq start and on Windows by starting the iMQ Broker Windows service.

If you are installing Message Queue on Solaris 8, and you will be running mquinstall to install all of the packages, be sure to set IMQ_JAVAHOME before running mqinstall to ensure the software picks the correct version of Java.

If you have not yet installed Core, you do not have to set IMQ_JAVAHOME because the Identity Synchronization for Windows installation program tells the Message Queue broker which JVM to use.

Troubleshooting Broker Configuration Directory Communication

The Message Queue broker authenticates clients against the Directory Server that stores the Identity Synchronization for Windows configuration. If the broker cannot connect to this Directory Server, no clients will be able to connect to the Message Queue, and the broker log will mention some javax.naming exception, such as “javax.naming.CommunicationException” or “javax.naming.NameNotFoundException”.

Verify that all imq.user_repository.ldap properties in /var/imq/instances/isw-broker/props/config.properties have the correct values. If any of these is incorrect, stop the Message Queue broker, correct and save the file, and restart the broker. The directory server host name must be resolvable from the broker’s machine.

Verify that the imq.user_repository.ldap.password property in /etc/imq/passfile is correct.

In some cases, the broker cannot search for entries if the root suffix contains spaces.

Troubleshooting Broker Memory Settings

During normal operation, the Message Queue broker consumes a modest amount of memory. However, during idsync resync operations, the broker’s memory requirements increase. If the broker reaches its memory limit, undelivered messages will accumulate, the idsync resync operation will slow down dramatically (or stop completely), and Identity Synchronization for Windows might be unresponsive afterwards.

When the broker enters a low-memory state, the following messages will appear in its log:

Increase the broker’s memory limit to 1 or 2 GB, as explained in the Sun Java System 1 2004Q3 Identity Synchronization for Windows Release Notes.

During an idsync resync operation, keep the log level set to INFO. Changing the log level to FINE or higher increases the load at the broker as more log messages are sent to the central logger.

Run idsync resync for a single Synchronization User List at a time.

Verify that the broker has a backlog of undelivered messages by examining its persistent message store in the appropriate directory.

On Solaris: /var/imq/instances/psw-broker/filestore/message/

On Windows: <installation_root>\isw-<machine_name>\imq\var\ instances\isw-broker\filestore\message\

Each file in this directory contains a single undelivered message. If there are more than 10000 files in this directory, then the broker has a backlog of messages.1 Otherwise, there is another problem with the broker.

The message backlog is probably only log files related to an idsync resync operation and you can safely remove them.

Stop the Message Queue broker as described in Starting and Stopping Services.

Remove all files in the persistent message store. The easiest way to remove these files is by recursively removing the message/ directory and then re-creating it.

Restart the Message Queue broker.

Follow the instructions in this section to ensure the broker does not run out of memory again.

Troubleshooting SSL Problems

When diagnosing problems with SSL, also see the Chapter 11, "Configuring Security," which describes how to setup SSL between components in Identity Synchronization for Windows. This section contains:

SSL Between Core Components

SSL between Connectors and Directory Server or Active Directory

SSL between the Directory Server Plugin and Active Directory

SSL Between Core Components

The Identity Synchronization for Windows installation program cannot verify that the SSL port provided during Core installation is correct. If you type the SSL port incorrectly during Core installation, then the Core components will not be able to communicate properly. You may not notice a problem until you try to save your configuration for the first time. The Console will display the following warning:

In this situation, uninstall the Core and install it again with the correct SSL port number.

SSL between Connectors and Directory Server or Active Directory

If a connector is unable to connect over SSL to the Directory Server or Active Directory, then this message will appear in the central error log:

Open the Console and check the Specifying Advanced Security Options panel
(see (more...) ).

Untrusted Certificates

More information will be available in the central audit log. For example, if the LDAP server’s SSL certificate is not trusted this message will be logged

In most situations, the CA certificate has not been added to the connector’s certificate database. This can be confirmed by running the certutil program that ships with the Directory Server.2

In the following example, the certificate database contains only the Active Directory CA certificate:


Note	Certificate management utilities such as certutil are provided in the SUNWtlsu package, which is not bundled with the Directory Server. (You can download this package from Sun Microsystems at no cost.) After downloading this package, you will find certutil in: /usr/sfw/bin/certutil

As shown here, the trust flags of the CA certificate must be “C,,”. If the certificate exists and the trust flags are set properly, but the connector still cannot connect, then first verify that the connector was restarted after adding the certificate, and then use the ldapsearch command that ships with the Sun Java System Directory to help diagnose the problem. If ldapsearch does not accept the certificate, then neither will the connector. For example, ldapsearch can reject certificates if they are not trusted

The -P option directs ldapsearch to use connector CNN100’s certificate database for SSL certificate validation. After the correct certificate is added to the connector’s certificate database, verify that ldapsearch accepts the certificate, and then restart the connector.

Mismatched Hostnames

When Identity Synchronization for Windows tries to establish SSL connections (with the trust all certificates setting disabled), the Identity Synchronization for Windows’ Connectors verify that the server’s hostname matches the hostname in the certificate that is presented by the server during the SSL negotiation phase. If the hostnames do not match, the connector will refuse to establish the connection.

The directory source hostname in the Identity Synchronization for Windows configuration must always match the hostname embedded in the certificate used by that directory source.

If there is a mismatch between the hostname in the command line (host2.example.com) and the hostname embedded in the certificate, then the following error message is displayed:

If the hostnames match, the ldapsearch command is successful and displays the contents of the root DSE.

Expired Certificates

SSL between the Directory Server Plugin and Active Directory

By default, Directory Server does not communicate with Active Directory over SSL when performing on-demand password synchronization. If the default is overridden to protect this communication with SSL, then the Active Directory CA certificate must be added to the Directory Server certificate database of each master replica as described in Chapter 11, "Configuring Security." If this certificate is not added, users will fail to bind to Directory Server with the error “DSA is unwilling to perform.”, and the Plugin’s log (for example, isw-<hostname>/logs/SUBC100/pluginwps_log_0.txt) will report:

In this situation, you must add the Active Directory CA certificate to Directory Server’s certificate database and restart Directory Server.

Troubleshooting Controller Problems

Some counters will not be reset when you restore an Active Directory domain controller from back-up files.

To ensure all counters will be reset appropriately, resynchronize all users after restoring the an Active Directory domain controller.

1Even if all messages have been delivered, the broker might maintain up to 10000 message files to avoid the performance penalty of creating and deleting files.

2Before running this command on Solaris, you must add the <installation_root>/lib directory to the LD_LIBRARY_PATH environment variable.

3The default certificate databases for the Sun Java System Directory Server and Windows NT Connectors include two certificates, saint-cert100 and saintRootCA. These certificates are not used in this release.

Previous Contents Index Next
Sun Java System Identity Synchronization for Windows 1 2004Q3 Installation and Configuration Guide