CHAPTER 13

SIMS Troubleshooting






Topic/Task
Description
Page

Troubleshooting the Admin Console  

- Netscape Applet Loading Problem
- To Restart the Admin Server
- Preventing the "Warning Applet" Banner
- Forgetting the Admin Password  

270  

Troubleshooting the Administration Server  

- Java Exceptions When Administration Server Starts
- java.lang.OutOfMemoryError While Administration Services Starts
- Admin Console Displays "Failed to locate SIMS Administration Server"  

271  

Troubleshooting the Message Store  

- User Not Able to Access INBOX
- Problems Turning Message Store Quota Enforcement Off and On
- Message Purge Failure
- User Can't Perform Internationalized String Search on Mail Messages  

274  

Troubleshooting the IMTA  

- SMTP Connection Aborted
- Message Queue Problems
- Logging and Tracking Messages in the IMTA
- Address Unknown to IMTA
- Multiple Reprocess Jobs Generated
- IMTA Error Messages  

278  

Troubleshooting the Directory Service  

- Diagnosing SIMS Problems Caused by Improper Directory Entries

- Re-generating the Sun Directory Service Directory Index  

295  

Crash Recovery  

- SIMS Crash Recovery
- Message Store Crash Recovery
- Admin Console Crash Recovery  

297  

Error Messages  

Go to Appendix D, "Error Messages."  

329  


Troubleshooting the Admin Console

Netscape Applet Loading Problem  

270  

To Restart the Admin Server  

271  

Preventing the "Warning Applet" Banner  

270  

Forgetting the Admin Password  

271  


Netscape Applet Loading Problem

This indicates that the Netscape Navigator does not reload the applets properly. If you are in any component page, and you want to reload the page, use the browser back arrow to go back to the Console main page and reselect the component icon. If you are in the Admin Console main page, logout, and re-login again. If none of this works, you may need to stop and restart the browser, and reconnect to the SIMS admin server again.

If applet appears to not be loading. You may need to move the mouse or select some menu buttons from the browser to activate the process.


Preventing the "Warning Applet" Banner

When accessing the Admin Console using a Netscape browser, you may see a "Warning Applet" banner in all dialogs. To prevent this banner from appearing, first close all browsers and do the following:

If you are using the Netscape browser (4.X and above), you MUST add the following line must be in the preferences.js file in the .netscape directory for the Admin Console to run correctly:

user_pref("signed.applets.codebase_principal_support",true);


Note - It is always a good idea to make these changes as some Admin Console functions won't work otherwise.

Forgetting the Admin Password

If you forget any SIMS Administrator password, you can use the
admin-modify-user utility (you'll need to provide the binddn and password of another administrator) to change the password. If you have only one SIMS administrator and you forget this password, you'll need to contact your Sun Microsystems support person.


Troubleshooting the Administration Server

Java Exceptions When Administration Server Starts  

271  

java.lang.OutOfMemoryError While Administration Services Starts  

272  

Admin Console Displays "Failed to locate SIMS Administration Server"  

273  


To Restart the Admin Server

  1. On rare occasions the Admin Server will crash and you may need to restart it using the following command:

# /opt/SUNWmail/sbin/adm.server stop
# /opt/SUNWmail/sbin/adm.server start


Java Exceptions When Administration Server Starts

Exceptions 1: If the following Java exception is thrown when the administration server is starting:

java.rmi.server.ExportException: Port already in use: 1099; nested exception is:
java.net.BindException: Address already in use
at java.lang.Throwable.<init>(Compiled Code)
at java.lang.Exception.<init>(Compiled Code)
at java.io.IOException.<init>(Compiled Code)

...

This could be caused by one of two possible reasons:

  1. Some other JAVA application is using RMI, and the RMI default port was used.
  2. You stopped, then started the admin server too quickly and the RMI registry did not completely end the process. The procedure to handle these two situation is:
Run /opt/SUNWmail/admin/sbin, type adm.server stop
Wait 30 seconds and start the admin server again: type adm.server start.
If you still see the Exception, go to /opt/SUNWmail/html, edit the simsadmin_port.html file, and change the number 1099 to another port number.
Stop and start the admin server.

Exceptions 2: Another JAVA exception thrown when the admin server is starting:

AdminServerImpl err: Server RemoteException; nested exception is:
java.rmi.AccessException: Registry.rebind
java.rmi.ServerException: Server RemoteException;
nested exception is:
java.rmi.AccessException: Registry.rebind
AdminServerImpl fails to register

This may be caused by the DNS setup. View the /etc/nsswitch.conf file. Is the format configured as "hosts: dns files" or "hosts: files dns?"

If a hostname is not understood by the DNS table, then the RMI will have problems. If this occurs, change the order of the host to "files dns" then the admin server can be started.


java.lang.OutOfMemoryError While Administration Services Starts

If the Java program runs out of memory you will receive the following error:

# java.lang.OutOfMemoryError: /var/opt/SUNWmail/ims/user/080
at java.io.File.list(Compiled Code)
at COM.Sun.sunsoft.sims.admin.ms.MSUserFolderInfo.<init>(Compiled Code)
at COM.Sun.sunsoft.sims.admin.ms.MSManagedObjectImpl.<init>(Compile Code)
at COM.Sun.sunsoft.sims.admin.console.RegistryLoader.loadFiles(Compiled Code)
at COM.Sun.sunsoft.sims.admin.console.RegistryLoader.<init>(Compiled Code)
at COM.Sun.sunsoft.sims.admin.console.AdminServerImpl.init(Compiled Code)
at COM.Sun.sunsoft.sims.admin.console.AdminServerImpl.main(Compiled Code)

To fix the problem go into /opt/SUNWmail/sbin/adm.server and change

$JAVA -Ddirectoryhost=${DIRECTORYHOST} \

to

$JAVA -oss2m -ms32m -mx64m -Ddirectoryhost=${DIRECTORYHOST} \

and then stop and start the admin server


Admin Console Displays "Failed to locate SIMS Administration Server"

Your admin server did not fully start up because of a component configuration error. Start the admin server in debug mode:

/opt/SUNWmail/admin/sbin/adm.server stop
/opt/SUNWmail/admin/sbin/adm.server -d All -l 6 start

If there are JAVA exceptions displayed, it means the admin server did not come up. Write down the exception information and send it to your SIMS support personal for more help.

If there aren't any JAVA exceptions, then it could be a browser connection time-out. Usually, an http proxy is set in the browser's preferences window. JAVA applets are downloaded from the server machine through the proxy server, and then to the client machine. This could take a long time depending on how the network is setup. Try removing the http proxy setting in your browser environment, and then try to reload the SIMS page again.


Troubleshooting the Message Store

User Not Able to Access INBOX  

274  

Verifying Password and Login  

274  

Problems Turning Message Store Quota Enforcement Off and On  

276  

Message Purge Failure  

277  

User Can't Perform Internationalized String Search on Mail Messages  

277  


User Not Able to Access INBOX

If an administrator created a new /var/mail account on SIMS using the Admin Console, and she forgot to create a UNIX account, the user will not be able to read the INBOX. The following error message will appear:

Unknown uid: <username>

The solution is to create the UNIX login account and the home directory:

  1. All /var/mail store users must have a shell account and HOME directory setting.
 
  2. Create the UNIX shell account using the desired operating system tools (for example, admintool).
  3. Using the Admin Console set the Home Directory field.

Verifying Password and Login

If a user cannot log in to his mailbox, you can determine if the correct login and password is being used by doing the following:

  1. Obtain the password and login which the POP/IMAP user is using.
  This corresponds to the userPassword and uid attributes in the LDAP record for that person. userPassword is stored as an encrypted string (unless APOP is used, in which case it might be clear text). If you dump the record it will display as encrypted just as it would in the /etc/shadow file. Note that the password will only display if you can bind as that user successfully, or if you bind as the administrator distinguished name.
  2. Use the uid to get the distinguished name for that person's record.

The example below is for an entry on an LDAP server "fork", using a search-base appropriate to bridge.

% /opt/SUNWconn/bin/ldapsearch -h fork -L -b "dc=bridge,\
dc=com,o=internet" uid=rja dn

dn: cn=Rob Albert (rja),ou=People,dc=bridge,dc=com,o=internet

  3. Using the value of the distinguished name retrieved in the previous step (but just the value, not the attribute tag "dn:") try to bind to the directory as that user, using the password (in cleartext) that the user is trying to login as:

Example 1 (bad password, good DN):

% /opt/SUNWconn/bin/ldapsearch -h fork -L -b "o=internet" \
-D "cn=Rob Albert (rja),ou=People,dc=bridge,dc=com,o=internet"\
-w badpass uid=rja dn

ldap_bind: Invalid credentials

Example 2 (good password, good DN):

% /opt/SUNWconn/bin/ldapsearch -h thestork -L -b "o=internet" \
-D "cn=Rob Albert (rja),ou=People,dc=bridge,dc=com,o=internet" -w\
goodpass uid=rja dn

dn: cn=Rob Albert (rja),ou=People,dc=bridge,dc=com,o=internet

Example 3 (good password, bad DN):

% /opt/SUNWconn/bin/ldapsearch -h thestork -L -b "o=internet" \
-D "cn=Rob Albert (rja),ou=People,dc=bridge,dc=com,o=internet" -w \
goodpass uid=rja dn

ldap_bind: No such object
ldap_bind: matched: ou=People,dc=bridge,dc=com,o=internet

Example 4 (good password, good DN, but the wrong "uid=" value):

% /opt/SUNWconn/bin/ldapsearch -h thestork -L -b "o=internet" \
-D "cn=Rob Albert (rja),ou=People,dc=bridge,dc=com,o=internet" -w \
goodpass uid=fred

dn: CN=Frederic Herrmann (fred),ou=People,dc=bridge,dc=com,o=internet

In the above examples you first bind anonymously to the directory to get the distinguished name of whom you want to bind as. Then you bind as that person, using their distinguished name and password. If you cannot bind as the user, then either their password or their distinguished name is wrong. If you can bind as the user, but they still can't log in, then either their LDAP record is damaged (you can verify this by comparing their record to a known undamaged record), or possibly the DITs are misconfigured (they may be confusing where the user data is to be looked up or the ims-basedn value in /etc/opt/SUNWmail/ims/ims.cnf does not match the DIT structure). This is not a complete list, but it covers the most common misconfigurations).


Problems Turning Message Store Quota Enforcement Off and On

If for some reason you turn the message store quota enforcement off, modify a user's quota, and then turn message store quota enforcement back on again, you should shut down SIMS. If you don't, it's possible that for a short period incoming mail to the modified user will be temporarily not delivered and returned to the sender. We absolutely recommend that you NOT turn the quota enforcement off to modify a user's quota, but in the very rare and unusual circumstance that you do, use the following procedure:

  1. Shut down SIMS.
  2. Do whatever work you need to do with quota turned off.
  3. Turn the quota back on.
  4. Run iminitquota -a.
  5. Restart SIMS.

Here's an example of how a mail store full situation could occur, even though the mail store is not full:

A user, Galaxion, has a user quota of 20 Mbytes and is currently using 15 Mbytes.

The message store quota enforcement is turned off, and while it is turned off, three things happen:

1) Galaxion's quota is reduced to 10 Mbytes.

2) Galaxion deletes a large amount of mail and reduces her mail store usage to 7 Mbytes.

3) An incremental directory synchronization occurs.

When quota enforcement is turned back on, iminitquota -a must be run to tally up mail storage usage. As soon as an incremental directory synchronization occurs, the system starts to enforce the new quota--in this case, 10 Mbytes.

However, the system reads the old mail store usage value of 15 Mbytes, until iminitquota -a is finished running. For 10,000 users, this might take 30 minutes. For that 30-minute period, the new quota of 10 Mbytes might be used, but the old mail store usage value of 15 Mbytes is read. This causes the enforcement to stop accepting new mail.


Message Purge Failure

The impurge command fails if the file system is full. You will see the following error message in the syslog file:

Jul 11 12:52:27 mcm-charmed SUNWmail.ims.impurge[17436]: PURGE erro: Cannot create expungedir tmp file
Jul 11 12:52:27 mcm-charmed SUNWmail.ims.impurge[17436]: PURGE erro: Cannot create ADM expungedir tmp file

To recover from this scenario, one can use the impurge -f dirname and follow the sequence of steps below:

  1. Create a clean file system with sufficient disk space, (this is twice as much space as the largest amount of data residing in the daily message store bucket, that is, an amount larger than the message data stored on the system's busiest day).
  2. Mount the clean file system with a name of your choice, for example, /backup
  3. Run the impurge command as impurge -f /backup. This will allow the purge command to complete the purging operations.
  4. Upon completion, the /backup directory will be empty and the messages will be purged.

If the file system is 100% full, you must run the impurge -f command as root.

In case of a system crash during the middle of a purging operation, you should make sure the /backup file system is mounted before running the command again.


User Can't Perform Internationalized String Search on Mail Messages

SIMS supports internationalized string searches in mail messages using any major character set. Search strings are no longer limited to ASCII/English.

Some mail clients, however, perform searches locally. That is, they use client code to perform searches. This client code might or might not support an internationalized search. To perform an internationalized search using the SIMS search code, mail clients must send an IMAP SEARCH command to the server.

If a mail user cannot do an internationalized search, make sure that the particular mail client used sends an IMAP SEARCH command to the server.


Troubleshooting the IMTA

Standard IMTA Troubleshooting Procedures  

278  

Changes to Configuration Files or IMTA Databases Do Not Take Effect  

283  

IMTA Sends Outgoing Mail But Does Not Receive Incoming Mail  

283  

Time Outs on Incoming SMTP Connections  

283  

Message Queue Growing Because a Recipient Address is Slow Accepting Email  

284  

SMTP Connection Aborted  

285  

Sent Message Can't Find Server Name  

286  

Message Queue Problems  

286  

Received Message is Encoded  

291  

From: Address Missing in Notifications from the IMTA  

292  

Address Unknown to IMTA  

293  

Multiple Reprocess Jobs Generated  

293  

Addresses Not Reversed  

294  

SMTP Access Restrictions Not Working As Expected  

294  

IMTA Error Messages  

331  


Standard IMTA Troubleshooting Procedures

In tracking down problems with the IMTA you should do the following:

Determine whether the problem occurs before or after a message is entered into the message queue
Determine whether messages were never accepted because of configuration problems or environmental problems (e.g., disk space or quota problems) or the absence of IMTA servers such as the Dispatcher and its services that can prevent the IMTA from accepting messages
Determine if network connectivity or routing problems could mean that the messages are stuck or mis-routed on a remote system.

If messages do not get placed in a queue directory at all, or get put into the wrong queue directory, you probably have a configuration problem.

To track an errant mail message, you can start by examining each step of the process for errors. The subsections below discuss investigating these processing steps.


Check the IMTA Configuration

Use the imta test -rewrite utility (see the imta-test-rewrite man page), to test the response of your configuration to addresses. Certain basic sorts of problems in the IMTA configuration, such as clear syntax errors in the IMTA configuration, will cause the utility to issue an error message. Otherwise, the utility will show address rewritings that will be applied as well as the channel to which messages would ultimately be queued. If the output is not what you expect you may need to modify your configuration.


Check Message Queue Directories

Check whether messages are present under the IMTA message queue directory,
/var/opt/SUNWmail/imta/queue/. Shell commands such as imta qm utility's directory command may be used to check for the presence of expected message files under the IMTA message queue directory.

If the imta test -rewrite output looks correct, check that messages are actually being placed in the IMTA message queue subdirectories. If not, you may have a problem with file space on that disk.


Check the Ownership of Critical Files

You should have an administrator (inetmail by default) account, created when you installed the IMTA. The directories /var/opt/SUNWmail/imta/queue/, /var/opt/SUNWmail/imta/log, and /var/opt/SUNWmail/imta/queue_cache, and all subdirectories and files under them should be owned by the inetmail account. If the protection and ownership are not correct for the queue cache database, messages may not be entered into the queue cache, and the queue cache will not be synchronized with the directory. Commands such as the following may be used to check the protection and ownership of these directories:

# ls -l -p -d /var/opt/SUNWmail/imta/queue
drwx------ 6 inetmail bin 512 Feb 7 09:32 /var/opt/SUNWmail/imta/queue/
# ls -l -p -d /var/opt/SUNWmail/imta/log
drwx------ 2 inetmail bin 1536 Mar 10 20:00 /var/opt/SUNWmail/imta/log/
# ls -l -p -d /var/opt/SUNWmail/imta/queue_cache
drwx------ 2 inetmail bin 512 Mar 10 15:03 /var/opt/SUNWmail/imta/ queue_cache/

Then check that any files and subdirectories of /var/opt/SUNWmail/imta/queue and /var/opt/SUNWmail/imta/log are owned by the IMTA account using commands such as:

# ls -l -p -R /var/opt/SUNWmail/imta/queue
# ls -l -p -R /var/opt/SUNWmail/imta/log


Checking that the Job Controller and Dispatcher are Present

The IMTA Job Controller handles the execution of the IMTA processing jobs, including most outgoing (master) channel jobs.

Some IMTA channels, such as the IMTA's multi-threaded SMTP channels, include resident server processes that process incoming messages---such servers handle the slave (incoming) direction for the channel.

The IMTA Dispatcher handles the creation of such IMTA servers. Dispatcher configuration options control whether such servers are available at all, and if available, how many such servers are created and when, and how many connections each server can handle. The Dispatcher always keeps at least one SMTP server process resident.

The command

# imta process

can be used to check that the IMTA Job Controller and IMTA Service Dispatcher are present, and to see if there are IMTA servers and processing jobs running. Under idle conditions the command should result in job_controller, dispatcher and tcp_smtp_server.


Check Processing Log Files

If the IMTA processing jobs run properly but the message stays in the message queue directory, you can examine the log files to see what is happening. All log files are created in the directory /var/opt/SUNWmail/imta/log. Log file name formats for various IMTA processing jobs are shown below.

TABLE  13-1   IMTA log files on UNIX
File name
Log file contains

channel_master.log-uniqueid  

Output of master program (usually client) for channel  

channel_slave.log-uniqueid  

Output of slave program (usually server) for channel  

dispatcher.log-uniqueid  

Dispatcher logging, if the Dispatcher DEBUG option has been set  

job_controller.log-uniqueid  

Job controller logging, if the Job Controller option DEBUG=1 has been set  

server-name_server.log-uniqueid  

Logging for the server server-name  

server-name_thread.log-uniqueid  

Per thread errors for server-name  

post.log- uniqueid  

Log output for the periodic IMTA delivery job  

return.log- uniqueid  

Log output for the periodic IMTA message bouncer job  

Channel log files are not created unless an error occurs or unless debugging output is enabled for the channel with the master_debug channel keyword or slave_debug channel keyword.

Each new log file is created with a unique id to avoid overwriting an earlier log written by the same channel. You can use the imta find utility to aid in finding the desired "version" of a log file. You can purge back older log files using the imta purge command.


Running a Channel Program Manually

While diagnosing an IMTA delivery problem it may be useful to run an IMTA delivery job by hand, particularly after enabling debugging output for one or more channels. The command

# imta submit channel-name

will notify the IMTA Job Controller to run the channel. If debugging is enabled for the channel in question, imta submit will create a log file in
/var/opt/SUNWmail/imta/log. See "Logging Messages Passing Through the IMTA" on page 254.

The command

# imta run channel-name

will perform outbound delivery for the channel channel-name under the currently active process, with output directed to your terminal. This may be more convenient than submitting a job, particularly if you suspect problems with job submission itself.


 

To Start and Stop Individual Channels

This section tells how to stop message enqueueing and dequeueing so that message queue problems may be more easily diagnosed and debugged. Freezing the message queue allows you to examine queued messages to determine the existence of loops, spam attacks, or mail bombs.

  1. To stop outbound processing (dequeueing) for a specific channel, edit the channel's channel block in imta.cnf and add the slave channel keyword. Then run imta cnbuild.
  To resume processing, remove the keyword and run
imta restart job.controller as root.
  2. To stop inbound processing (enqueuing to a channel) for a specific channel, without bouncing messages, redirect messages destined to this channel to the hold channel by replacing the channel's routing system with hold-daemon in the rewrite rules. Then restart the IMTA by executing imta restart. Messages are enqueued to the hold channel.
  To resume inbound processing, change the routing system back to its original setting and restart the IMTA. Then run
  /opt/SUNWmail/imta/sbin/hold_master -u "*" -d "*"
  Note that you should not be using the hold channel for any other purpose (such as moving a user) while you are performing this process.
  3. To stop inbound processing (enqueuing to a channel) while returning temporary SMTP errors to client hosts, add the following access rule in the SEND_ACCESS mapping table in the IMTA mappings file:
  SEND_ACCESS
  *|*|your_channel|* $X4.2.1|$Ndestination$ channel$ temporarily$ disabled

Changes to Configuration Files or IMTA Databases Do Not Take Effect

If changes to your configuration, mapping, conversion, security, option, or alias files or to IMTA databases do not seem to be taking effect, check to see if you have restarted your mail user agent session, and that you have restarted the IMTA.


IMTA Sends Outgoing Mail But Does Not Receive Incoming Mail

Most IMTA channels depend upon a slave, or server, channel program to receive incoming messages. For some transports supported by the IMTA, in particular
TCP/IP and UUCP, you need to make sure that the transport activates the IMTA slave program rather than its standard server. Replacing the native sendmail SMTP server with the IMTA SMTP server is performed as a part of installation task.

For the multi-threaded SMTP server, the startup of the SMTP server is controlled via the IMTA Dispatcher. The IMTA Dispatcher controls the starting up of an SMTP server or servers, according to your Dispatcher configuration. If the Dispatcher is configured to use a MIN_PROCS value greater than or equal to one for the SMTP service, then there should always be at least one SMTP server process running (and potentially more, according to the MAX_PROCS value for the SMTP service). The imta process command may be used to check for the presence of SMTP server processes. Use dispatcher_stats_tty to check SMTP processes connection.


Time Outs on Incoming SMTP Connections

Time outs on incoming SMTP connections are most often related to system resources and the allocation thereof.

Check how many simultaneous incoming SMTP connections you allow. This is controlled by the MAX_PROCS and MAX_CONNECTIONS Dispatcher settings for the SMTP service; the number of simultaneous connections allowed is MAX_PROCS*MAX_CONNECTIONS. If you can afford the system resources, consider raising this number if it is too low for your usage.

Try putting the slave_debug keyword on the channels handling incoming SMTP over TCP/IP mail, usually tcp_local. Then take a look at the resulting tcp_local_slave.log-uniqueid files, and try to spot any particular characteristics of the messages that time out. For instance, if incoming messages with large numbers of recipients are timing out, consider using the expandlimit keyword on the channel.

Of course, if your system is extremely overloaded and overextended, time outs will be difficult to avoid entirely.


Message Queue Growing Because a Recipient Address is Slow Accepting Email

Sometimes, it becomes necessary to separate the messages going to one or more specific email sites because the remote SMTP servers at those sites are slow or not responding and it holds up all the outgoing mail. This can be done by creating a new channel, similar to the external SMTP channel and a new processing queue, associated with it, for all mails going to those domains.

For example, if you want to separate out all mail going to slow.com, first you create a new channel and it's rewrite rules. That is, in the IMTA configuration file,
/etc/opt/SUNWmail/imta/imta.cnf add lines similar to the following:

!Rewrite rules
..........
! tcp_slow
.slow.com $E$U%$H.slow.com@tcp_slow-daemon
.slow $U%$H.slow.com@tcp_slow-daemon
!
...........
!Channel definitions
...........
!
! tcp_slow
tcp_slow queue slow_queue smtp single_sys copywarnpost copysendpost postheadonly
switchchannel subdirs 20 immnonurgent logging notices 1 2 4 7 remotehost inner
tcp_slow-daemon mailserver.acme.com
!
..........
Note the usage of the queue keyword to separate the message processing queue.
Also edit the job controller configuration file, /etc/opt/ SUNWmail/imta/job_controller.cnf
and add the following lines, in the queue definitions section :
...........
!
[QUEUE=SLOW_QUEUE]
job_limit=10
capacity=200
!
..............


SMTP Connection Aborted

If the SMTP connection aborts, check whether the IMTA is running (SMTP server included):

% imta process

The result should list the three following processes:

job_controller
dispatcher
tcp_smtp_server

Restart the IMTA if it is not running.

If the IMTA continues to abort, look at the tcp_smtp_server log files to determine the problem.

  /var/opt/SUNWmail/imta/log

To debug the IMTA problem, set debug=1 in
/etc/opt/SUNWmail/imta/dispatcher.cnf. Also enable the diagnostic output for the slave program in the internet channel. (From the IMTA Property Book double-click the relevant SMTP channel and go to the Diagnostics Output Section. Check the box for Enable diagnostic output for slave program.

Try to start the IMTA again, and look at the debug output in the tcp_smtp_server log files at /var/opt/SUNWmail/imta/log/tcp_local_slave*.


Sent Message Can't Find Server Name

If DNS is not working, the administration server will display the following warning message in a console window:

***Can't find server name for address <ip_address>: No response from server. ***Default servers are not available

In addition, a Java exception stack will be displayed, and users will not be able to send mail until the DNS is once again operating.

The administration server does not itself depend on the DNS server--it can continue to operate and an Admin Console can connect to it. But the DNS server needs to be returned to normal operation before mail can be sent.


Message Queue Problems

Undeliverable messages are probably either not being dequeued from the IMTA, being saved in .HELD file because it is looping between another server or channel, or stuck at another server. This section describes various message queue problems.


Unjamming a Message Queue

If the IMTA stops processing messages in a queue, enter the following command as root for the queue that appears jammed:

# /opt/SUNWmail/sbin/imta run <channel name>

where <channel name> is specified in imta.cnf


Message Not Being Dequeued

Errors encountered during TCP/IP delivery are quite often transient in nature and the IMTA will generally retain messages when problems are encountered and retry them periodically. It is quite normal on very large networks to experience periodic outages to certain hosts while other host connections work fine. You can examine the log files for errors relating to delivery attempts. You may see error messages such as "Fatal error from smtp_open." Such errors are not uncommon and are usually associated with a transient network problem. Your TCP/IP package may contain tools such as ping, traceroute, and nslookup to aid in debugging TCP/IP network problems.

The example below shows the steps you might use to see why a message is sitting in the queue awaiting delivery to xtel.co.uk. The basic idea is to duplicate the steps the IMTA uses to deliver SMTP mail on TCP/IP.

% nslookup -query=mx xtel.co.uk (1)
Server: LOCALHOST
Address: 127.0.0.1

Non-authoritative answer:
XTEL.CO.UK preference = 10, mail exchanger = nsfnet-relay.ac.uk (2)

% /usr/sbin/ping nsfnet-relay.ac.uk (3)
PING NSFNET-RELAY.AC.UK (128.86.8.6): 56 data bytes
64 bytes from 128.86.8.6: icmp_seq=0 time=490 ms
CANCEL

% telnet nsfnet-relay.ac.uk 25 (4)
Trying... [128.86.8.6]
telnet: Unable to connect to remote host: Connection refused

  1. First use the NSLOOKUP utility to see what MX records, if any, exist for this host. If no MX records exist, then you should try connecting directly to the host. If MX records do exist, then you must test by connecting to the designated MX relays since the IMTA is required to honor MX information preferentially.
  2. In this example, the Domain Name Service returned the name of the designated MX relay for xtel.co.uk. This is the host that the IMTA will actually connect to. If more than one MX relay is listed, the IMTA would try each in succession.
  3. A simple way to test connectivity to the host is with a PING utility. If no response is received then you have a network routing or configuration problem. If the problem is on some router over which you have no control, there is not anything you can do except to wait until it is fixed.
  4. If you do have connectivity to the remote host, the next step is to see if it is accepting inbound SMTP connections by using TELNET to the SMTP server port, port 25. If you use TELNET without specifying the port, you may merely discover that the remote host accepts normal TELNET connections. This by no means indicates that it accepts SMTP connections: many systems may accept regular TELNET connections but refuse SMTP connections or vice versa. Thus, you should always do your testing against the SMTP port.
  In this example, the remote host is currently refusing connections to the SMTP port. This is undoubtedly why the IMTA fails to deliver the message. The connection may be refused due to a misconfiguration of the remote host or some sort of resource exhaustion, again, on the remote host. There is absolutely nothing you can do locally to solve the problem. Typically, you should just let the IMTA continue to retry the message.

If you are running on a TCP/IP network which does not use the Domain Name Service, then you can skip steps (1) and (2) and use PING and TELNET directly to the host in question. Be careful to use precisely the host name that the IMTA would use, which can be ascertained by examination of the relevant log file from the IMTA's last attempt.

Note that if you test connectivity to a TCP/IP host and encounter no problems using interactive tests, it is quite likely that the problem has simply been resolved since the IMTA last tried delivering the message. This is not an indication of a problem with the IMTA.


IMTA Messages are Not Delivered

In addition to message transport problems, there are two other common problems which can lead to messages sitting around unprocessed in the message queues:

  1. The message has a low priority. By default, the IMTA pays attention to Priority: headers in scheduling message delivery jobs: only messages of normal or urgent Priority: get an immediate delivery attempt, while messages of non-urgent Priority: wait until the next run of the IMTA periodic delivery job.
  2. The queue cache database is not synchronized with the messages in the queue directories.
  Message files in the IMTA queue subdirectories which are awaiting delivery are entered into the queue cache database. When channel programs run in order to deliver messages in their queues they consult the queue cache to determine what messages to process. There are circumstances which can lead to message files in the queue that do not have a corresponding queue cache entry. For example, if the queue cache database has incorrect ownership and protection, see "Check the Ownership of Critical Files" on page 279. Channel programs will ignore queued messages which do not have a cache entry. You can use the imta cache -view utility to check if a particular file is in the queue cache; if it is not, then the queue cache needs to be synchronized.
  The queue cache is normally synchronized daily. If required, you can manually resynchronize the cache using the UNIX command
  # imta cache -synchronize
  Once synchronized, upon the next running of the IMTA periodic delivery job the channel programs should process all messages in their queue.
  There is a more drastic step, rebuilding the queue cache database, which should only be performed as a last resort, e.g., if disk problems have corrupted your queue cache database, as it will cause loss of some information from the queue cache database. (The sort of information lost includes, but is not limited to, message creation dates, message deferral dates, message expiration dates, and the original message owner information.)
  To rebuild the queue cache database, use the UNIX commands,

# imta cache -rebuild
# imta cache -close
# imta cache -synchronize

  3. Channel processing programs fail to run because they cannot create their execution log file.
  Check the access permissions, disk space and quotas.

.HELD Messages

If the IMTA detects a mail loop, that is, messages bounce between servers/channels, delivery is halted and the messages are stored in a file with the suffix .HELD in
/var/opt/SUNWmail/imta/queue/<channel>. (A mail loop occurs because each server/channel thinks the other is responsible for delivery to an address.)

The message is ignored by the IMTA and no further delivery is attempted. Look at the headers in the message to determine which server/channel is bouncing the message. Fix the entry as needed and run the command:

% imta queue -retry_delivery <channel-name>


Messages are Looping

If the IMTA detects that a message is looping, that message will be sidelined as a .HELD file. But certain cases can lead to message loops which the IMTA can not detect. Some of the more common cases include:

  1. A postmaster address is broken.
  2. Stripping of Received: headers is preventing the IMTA from detecting the message loop.
  3. Incorrect handling of notification messages by other mail systems, that are generating reencapsulated messages in response to notification messages.

The first step in dealing with looping messages is to determine why the messages are looping. Useful things to look at are a copy of the problem message file while it is in the IMTA queue area, IMTA mail log entries (if you have the logging channel keyword enabled in your IMTA configuration file for the channels in question) relating to the problem message, and IMTA channel debug log files for the channels in question. Determining the From: and To: addresses for the problem message, seeing the Received: headers, and seeing the message structure (type of encapsulation of the message contents), can all help pinpoint which sort of message loop case you are encountering.

For case (1), note that mail systems such as the IMTA require that the postmaster address be a functioning address that can receive e-mail. If a message to the postmaster is looping, check that your configuration has a proper postmaster address pointing to an account that can receive messages.

For case (2), note that normal detection of message loops is based on various Received: headers. If Received: headers are being stripped---either explicitly on the IMTA system itself, or more likely on some other system such as a firewall---that interferes with proper detection of message loops. There will likely be two issues to resolve: check that no undesired stripping of Received: headers is occurring so that if a loop does occur it can be short-circuited, and check for the underlying reason why the messages were looping. Possible underlying reasons for the occurrence of the message loop in the first place include: a problem in the assignment of system names or a system not configured to recognize a variant of its own name, a DNS problem, a lack of authoritative addressing information on the system(s) in question, or a user address forwarding error.

For case (3), note that Internet standards require that notification messages (reports of messages being delivered, or messages bouncing) have an empty envelope From: address to prevent message loops. However, some mail systems do not correctly handle such notification messages; such mail systems may, when forwarding or bouncing such a notification message, insert a new envelope From: address of their own. This can then lead to message loops. The solution is to fix the mail system that is incorrectly handling the notification messages.


Received Message is Encoded

Messages sent by the IMTA are received in an encoded format. For example:

Date: Sun, 04 Jul 1993 11:59:56 -0700 (PDT)
From: "Wally C. Bouy" <wallyboy@bridge.com>
To: Padraic.Fanning@stream.edu
Subject: test message with 8bit data
MIME-Version: 1.0
Content-type: TEXT/PLAIN; CHARSET=ISO-8859-1
Content-transfer-encoding: QUOTED-PRINTABLE
2=00So are the Bo=F6tes Void and the Coal Sack the same?=

Such messages appear unencoded when read with a MIME-aware user agent such as Pine or when decoded with a decoder such as imta decode.

The SMTP protocol as set forth by RFC 821 only allows the transmission of ASCII characters. As ASCII is a seven-bit character set, the transmission via SMTP of eight bit characters is illegal. As a practical matter, the transmission of eight bit characters over SMTP is known to cause a variety of problems with some SMTP servers (for example, cause SMTP servers to go into compute bound loops, cause mail messages to be sent over and over again, crash SMTP servers, wreak havoc with user agents or mailboxes which cannot handle eight bit data, and so on).

Until the advent of RFC 1425 and RFC 1426, an SMTP client had only three alternatives when presented with a message containing eight bit data: return the message to the sender as undeliverable, encode the message, or send it anyhow in direct violation of RFC 821. None of these alternatives were pleasant. With the recent advent of MIME (first specified in RFCs 1521 and 1522, and updated in RFCs 2045--2049) and the SMTP extensions work (RFC 1425 and RFC 1426), there are now standard encodings which may be used to encode eight bit data using the ASCII character set and mechanisms to negotiate, between the SMTP client and server, whether or not eight bit data will be accepted as is by the server without first being encoded.

When recipients receive encoded messages such as those shown above with a MIME content type of TEXT/PLAIN, then invariably the original message contained eight-bit characters and the remote SMTP server to which the IMTA SMTP client transferred the message did not support the transfer of eight bit data. The IMTA then had to encode the message.


From: Address Missing in Notifications from the IMTA

Occasionally users or postmasters on other mail systems will complain that the IMTA is omitting the envelope From: address in messages it sends. You may be presented with a message header fragment like the one shown below

From Thu Jan 13 11:50:23 1994
Received: from vulcan.ajax.com by monster.ajax.com via SMTP
(930416.SGI/931108.SGI.ANONFTP) for xxxx id AA21154;
Thu, 13 Jan 94 11:50:23 +1100
Date: Thu, 13 Jan 1994 11:49:26 +1000
From: IMTA Mail Server <postmaster@vulcan.ajax.com>

Note how in the first line there is a noticeable blank space between the From and date? This header line is often referred to as the colonless From line and it gives the envelope From: address for the message. That blank space indicates that the message had no envelope From: address; that is, it had what is called in the mail business a null return path. Note further that this was an automatically generated mail message as suggested by the RFC 822 From: address of postmaster@vulcan.ajax.com.

The relevant standards require that automatically generated messages such as non-delivery notifications and delivery receipts use a null return path. As mailers are supposed to bounce mail to the envelope From: address1, this helps to prevent mail loops from occurring.

If someone complains about the missing the From: address, ask them to send you a sample offending message. Determine if it was an automatically generated message. If it was, then explain to them that if their mailer or user agent is incapable of handling null return paths then it is incompliant with RFC 821 and 1123. Refer them to Paragraph 8 of Section 3.6 and the second paragraph of the MAIL command description in Section 4.1.1 in RFC 821. Further point out that were you to change your mailer to use a non-null return path for automatically generated notifications, then you would be violating the Internet Host Requirements; specifically, you would be in violation of Section 5.3.3 of RFC 1123.

If for some reason you absolutely must generate non-null return paths in your notification messages, then you may do so with the RETURN_ENVELOPE option of the IMTA option file (see SIMS Reference Manual). Or to generate non-null return paths in notification messages only for a particular channel or channels, you may use the returnenvelope channel keyword (refer to the keywords section in the SIMS Reference Manual). Be warned: Use of either the option or the channel keyword will put you in violation of the Internet Host Requirements and, more importantly, may lead to looping mail. Looping mail will not only inconvenience you but may cause serious problems for some unfortunate site which gets into a loop with your system. Also, keep in mind that changing the IMTA's behavior so as not to cause problems for a broken mailer which cannot handle null return paths does not really fix anything: Other mailers over which you have no control will continue to send the broken mailer messages with null return paths. The only satisfactory solution in this situation is to fix the broken mailer.


Address Unknown to IMTA

If a sender sends a message to a valid address and receives a returned message with the error "User unknown," verify the address by using the imta test -rewrite command. Most likely the user entry in the directory is not correct. Retrieve the full directory entry for this user, and verify it. IMTA dirsync can also check directory entry errors. Enter the command:

# imta dirsync -t -F

If no invalid entries are reported, try running dirsync again with the -v switch, and look at the newest /var/opt/SUNWmail/log/dirsync.trc-* for more information.

Once you've fixed the directory user info, you may run dirsync again, or wait until the periodic incremental dirsync runs. Run the command:

# imta dirsync [ other options ... ]

Finally, check that the ownership of all files in /var/opt/SUNWmail/imta/db is set to inetmail. Verify that all the files are writable by the owner.


Multiple Reprocess Jobs Generated

If you are having severe performance problems, check if multiple process jobs are being generated. The imta process<ret> command should show only a single process running. If more are showing, look in /etc/opt/SUNWmail/imta/aliases, and verify that it contains the following line:

postmaster: <user-name>@FQDN

Make sure the postmaster address is valid and is receiving mail. If it doesn't, add this line and do a fresh restart of the IMTA.

# imta restart


Addresses Not Reversed

The IMTA has the ability to reverse envelope from: and header addresses. Generally, this is used to turn the address form username@host.domain to the more public form first_name.last_name@domain so that recipients outside of the domain only see the latter form.

This functionality is not always activated. For instance, it is not active if the routability scope is set to Local System Users Only. To enable, set the option USE REVERSE DATABASE to 5 in /etc/opt/SUNWmail/imta/option.dat, and make sure the list of channel keywords for the delivery channel does not contain noreverse.

Another cause of reverse address failure is the absence or incorrect configuration of the mail LDAP attribute.

To further diagnose the problem, set MM_DEBUG to 5 in option.dat and activate the slave diagnostic output for the queuing channel. Restart the IMTA and reproduce the problem. Examine the debug output file created in
/var/opt/SUNWmail/imta/log. For information on how to use diagnostic output, see "To Configure Diagnostics Output" on page 105.


SMTP Access Restrictions Not Working As Expected

Common reasons for such problems include:

  1. Interference between access control rules and rewrite rules. The address is rewritten before it is processed by the access rules, and the access rules handle the original and rewritten addresses differently.
  2. Interference between access rules. A typical case arises when a message is blocked using IP addresses only. Other rules involving addresses are never applied since IP level envelope information is sufficient to make an access decision.

The problem can best be diagnosed using the debugging functionality. See "To Configure Diagnostics Output" on page 105. For further details on access restriction, refer to the SIMS Reference Manual.


Troubleshooting the Directory Service

Diagnosing SIMS Problems Caused by Improper Directory Entries  

295  

 

295  


Diagnosing SIMS Problems Caused by Improper Directory Entries

Because SIMS relies completely on the presence and values of certain attributes in the directory, the single most common set of problems installing and maintaining SIMS is with LDIF generated by sources other than imldifsync, or by ldapmodify if accurate information is not entered into the SIMS install HTTP forms. This section discusses some common problems caused by incorrectly configuring attributes, and provides hints in diagnosing them.


General Hints

LDIF-generating scripts should be careful not to accidentally include nonprinting characters or white-space characters in attribute values. This is most commonly seen when a space (` `) is added at the end of an attribute value when not appropriate. For example, you should enter the value for mailFolderMap as

"mailFolderMap: SUN-MS"

:

and not as:

"mailFolderMap: SUN-MS "

.

where the quotes show the inappropriate spaces.

It is always best to try and scope a particular problem before trying to diagnose it. This may sound obvious, but within the context of SIMS this means:

If some users are having problems, dump the LDIF for users that are known to work (using ldapsearch(1) or the Admin Console) and compare this with the entries of users that do work.
If data was added with an LDIF generating script, or even with imldifsync, for users that don't work, try adding the same information via the Admin Console. If this works then look for white space or unprintable characters and determine if they are being added by the LDIF-generating scripts.

Users Can't Log In to Their IMAP Mail Server

Check the following attributes:

userPassword - Should match the encrypted password entry in /etc/passwd or the equivalent. Note that this will not be true if the attribute is changed directly some time after the directory has been in use for a while.
uid - Should match the uid in /etc/passwd or the equivalent.
mailFolderMap - This currently has only one of two valid values: "Sun-MS" for users receiving mail via the SIMS messages store, and "UNIX V7" for users who use the /var/mail inbox format.

Mail Inbound to the SIMS MTA Bounces

Check the following attributes:

rfc822Mailbox - Should have valid user@FQDN (fully qualified DNS domain name) value.
mailDeliveryOption - Mailbox, native, program, forward, and file for users; shared, program, and file for groups.
mailFolderMap - Must be set to UNIX V7 (/var/mail) or Sun-MS as appropriate.

Mail Delivered Does Not Arrive

mailQuota - Set incorrectly.
mailFolderMap - Must be set to UNIX V7 (/var/mail) or Sun-MS as appropriate.

Mail Forwarded between SIMS and Other Servers Isn't Received

mailForwardingAddr - Not set or set incorrectly.

Re-generating the Sun Directory Service Directory Index

Index re-generation is done by running the dsidxgen tool. The command takes the following arguments:

# dsidxgen [backend_directory] [-a attribute_name [attribute_name...]]

If backend_directory is not specified then dsidxgen regenerates index for all data stores defined in the SunDS 3.1 configuration files. The list of attributes for which to regenerate indexes are specified by -a attribute_nam'. If not supplied then all indexes are generated apart from id2children.dbb, dn2id.dbb, id2entry.dbb, dn.dbb and objectclass.dbb.


Crash Recovery

SIMS Crash Recovery  

297  

Message Store Crash Recovery  

298  

Admin Console Crash Recovery  

299  


 

SIMS Crash Recovery

If your mail server becomes nonfunctional, you must perform the following procedure:

  1. Enter the following command as root:

# imta queue-recover_crash

This command invokes a utility that rebuilds the queue caches for the Sun Message Store and /var/mail. These caches may become corrupted during the time that the mail server becomes nonfunctional. For example, a message may be partially written to the Sun Message Store cache during the time that the mail server becomes nonfunctional. Running the utility will enable the mail server to clean up these types of corruption.

  2. Stop any components that might still be running by entering the following:

# /etc/init.d/im.server stop

  3. Start all components by entering the following:

# /etc/init.d/im.server start


 

Message Store Crash Recovery

In the event of a catastrophic system failure, the message store may be left in an inconsistent state and in some instances require data recovery from backup media. If you see the following message, it means that SIMS did not start and the store was shut down abnormally:

/var/log/syslog.1:Oct 2 16:40:24 mcm-nitro SUNWmail.ims.imaccessd[1373]:Message store may be inconsistent. Please run imcheck -c

Follow the procedure below to recover the message store.

  1. Change to the proper directory:

% cd /opt/SUNWmail/sbin/imta

  2. Make sure IMTA is not running. To stop the IMTA, run:

% imta stop

  3. Make sure imaccessd is not running. To stop the imaccessd, run:

% mt.scheduler stop


Note - The imaccessd process should never be killed using the kill -9 command. If this should accidently occur, run imcheck -c before restarting imaccessd.
  4. Run the following as root:

% imcheck -c -f <filename>

  a. If imcheck completes successfully, check the
/var/opt/SUNWmail/ims/adm/restore_log file. If this file is present, restore the users whose names are in this file, and then remove the file. For example:

% imrestore -i
% rm /var/opt/SUNWmail/ims/adm/restore_log

  b. If imcheck fails, save the syslog file and the store report, then contact your Sun Service Provider. The store report is at
/var/opt/SUNWmail/ims/adm/<filename>. Do not restart the MTA and imaccessd.

 

Admin Console Crash Recovery

If the Admin Console hangs, kill the browser process, restart the browser, and reconnect to the Admin server. If the Admin Console crashes or vanishes, restart the browser and reconnect to the Admin Server. Note that switching back and forth between pages may cause Admin Console to hang or crash while communicating with the Admin server. To kill a browser process on Solaris, perform the following steps:

  1. Find the process id.
  2. Bring up a UNIX shell window and type the following command as root:

# ps -ef | grep netscape <cr>

  You will see something like this:

myuid 350 10767 0 hh:mm:ss pts/x 0:00 grep netscape
root 26145 1 0 hh:mm:ss pts/x 0:46 /usr/netscape,v4.51/bin/netscape
-Dlegacy.host= -Ddirectoryhost=motmot
-Dconsole.domain=eng.bridge
myuid 336 10767 24 hh:mm:ss pts/x 0:15 /usr/dt/appconfinetscape/
runtime/bin/sparc/green_threads/jre -classpath ..

  3. Kill the browser process using the following command:

# kill <browser process number (in this case 336)> <cr>

  4. On rare occasions the Admin Server will crash and you may need to restart the administration server using the following command:

# /opt/SUNWmail/sbin/adm.server stop
# /opt/SUNWmail/sbin/adm.server start


Note - For Win 95 and NT environments, please refer to the corresponding administrator's guides for instructions on killing a browser process.



1

Some mailers will preferentially send notifications to the address specified with the non-standard Errors-to: or Warnings-to: header lines. By default, the IMTA itself sends notifications to the envelope From: address, unless configured otherwise via the USE_ERRORS_TO and USE_WARNINGS_TO IMTA options.


Copyright© 1999 Sun Microsystems, Inc. All Rights Reserved.