This chapter provides guidelines for troubleshooting the Oracle Communications Messaging Server message store as well as recovery procedures for when the message store becomes corrupted or unexpectedly shuts down.
See also:
If one or more mailboxes become corrupt, use the "reconstruct" utility to rebuild the mailboxes or the mailbox database. You can use this utility to recover from almost any form of data corruption in the mail store. See "Error Messages Signifying reconstruct Is Needed" and "reconstruct" for more details.
Message store problems can occur if the mboxlist database cache is too small. Specifically, Message store performance can slow to unacceptable levels and can even dump core. Refer to the discussion on performance tuning considerations in Messaging Server Installation and Configuration Guide.
Red Hat Linux - Messaging Server Patch 120230-08 IMAP, POP and HTTP Servers Not Starting Due to Over Sessions Per Process
After installing this patch, when you try to start Messaging Server, the IMAP, POP and HTTP servers do not start and may send the following example error logs:
http server - log: [29/May/2006:17:44:37 +051800] usg197 httpd[6751]: General Critical: Not enough file descriptors to support 6000 sessions per process; Recommend ulimit -n 12851 or 87 sessions per process. pop server - log: [29/May/2006:17:44:37 +051800] usg197 popd[6749]: General Critical: Not enough file descriptors to support 600 sessions per process; Recommend ulimit -n 2651 or 58 sessions per process. imap server - log: [29/May/2006:17:44:37 +051800] usg197 imapd[6747]: General Critical: Not enough file descriptors to support 4000 sessions per process; Recommend ulimit -n 12851 or 58 sessions per process.
Set the appropriate number of file descriptors for all three server sessions. Additional file descriptors are available by adding a line similar to the following to the /etc/sysctl.conf file and using sysctl -p to reread that file:
fs.file-max = 65536
You must also add a line like the following to the /etc/security/limits.conf file:
* soft nofile 65536 * hard nofile 65536
If users accessing their mail with web clients, like Convergence, cannot load pages, the problem might be that the data is getting corrupted after compression. This can sometimes happen if the system has deployed a outdated proxy server. To solve this problem, try setting the msconfighttp.gzipstatic and http.gzipdynamic options to 0 to disable data compression. If this solves the problem, you may want to update the proxy server.
Some UNIX shells may require quotes around wildcard options and some will not. For example, the C shell tries to expand arguments containing wildcards ( *, ?) as files and will fail if no match is found. These pattern matching arguments may need to be enclosed in quotes to be passed to commands like mboxutil.
For example:
mboxutil -l -p user/usr44*
works in the Bourne shell, but fails with tsch and the C shell. These shells would require the following:
mboxutil -l -p "user/usr44*"
If a command using a wildcard pattern does not work, verify whether you need to use quotes around wildcards for that shell.
A user can get the message "Unknown/invalid partition" in Messenger Express if their mailbox was moved to a new partition that was just created and Messaging Server was not refreshed or restarted. This problem only occurs on new partitions. If you now add additional user mailboxes to this new partition, you will not have to do a refresh/restart of Messaging Server.
A user mailbox problem exists when the damage to the message store is limited to a small number of users and there is no global damage to the system. The following guidelines suggest a process for identifying, analyzing, and resolving a user mailbox directory problem:
Review the log files, the error messages, or any unusual behavior that the user observes.
To keep debugging information and history, copy the entire store_root/mboxlist/ user directory to another location outside the message store.
To find the user folder that might be causing the problem, run the command reconstruct -r -n. If you are unable to find the folder using reconstruct, the folder might not exist in the folder.db.
If you are unable to find the folder using the reconstruct -r -n command, use the "hashdir" command to determine the location.
Once you find the folder, examine the files, check permissions, and verify the proper file sizes.
Use reconstruct -r (without the -n option) to rebuild the mailbox.
If reconstruct does not detect a problem that you observe, you can force the reconstruction of your mail folders by using the reconstruct -r -f command.
If the folder does not exist in the mboxlist directory (store_root/mboxlist), but exists in the partition directory store_root/partition), there might be a global inconsistency. In this case, you should run the reconstruct -m command.
If the previous steps do not work, you can remove the store.idx file and run the reconstruct command again.
Caution:
You should only remove the store.idx file if you are sure there is a problem in the file that the reconstruct command is unable to find.If the issue is limited to a problematic message, you should copy the message file to another location outside of the message store and run the command reconstruct -r on the mailbox/ directory.
If you determine the folder exists on the disk (store_root/partition/ directory), but is apparently not in the database (store_root/mboxlist/ directory), run the command reconstruct -m to ensure message store consistency.
See "Repairing Mailboxes and the Mailboxes Database (reconstruct Command)" for more information on the reconstruct command.
If stored does not start and returns the following error message:
MessagingServer_home/bin/start-msg MessagingServer_home: Starting STORE daemon ...Fatal error: Cannot find group in name service
This indicates that the UNIX group configured in local.servergid cannot be found. Stored and others need to set their gid to that group. Sometimes the group defined by local.servergid gets inadvertently deleted. In this case, create the deleted group, add mailsrv to the group, change ownership of the instance_root and its files to mailsrv and the group.
The message store has a hard limit of two gigabytes for a store.idx file, which is equivalent to about one million messages in a single mailbox (folder). If a mailbox grows to the point that the store.idx file will attempt to exceed two gigabytes, the user will stop receiving any new email. In addition, other processes that handle that mailbox, such as imapd, popd, mshttpd, could also experience degraded performance.
If this problem arises, you will see errors in mail.log_current such as this:
05-Oct-2005 16:09:09.63 ims-ms Q 7 ... System I/O error. Administrator, check server log for details. System I/O error.
In addition, the MTA log file will have an errors such as the following:
[05/Oct/2005:16:09:09 +0900] jmail ims_master[20745]: Store Error: Unable to append cache for user/admin: File too large
You can determine this problem conclusively by looking at the file in the user's message store directory, or by looking in the imta log file to see a more detailed message.
The immediate action is to reduce the size of the file. Either delete some mail, or move some of it to another mailbox. You could also use mboxutil -r to rename the folder out of the way, or mboxutil -d to delete the folder (see "mboxutil").
Long-term, you will need to inform the user of mailbox size limitations, implement an aging policy (see "Configuring Message Expiration (Tasks)"), a quota policy (see "Message Store Quota Overview"), set a mailbox limit by setting store.maxmessages (see Messaging Server Reference), set up an archiving system, or make an adjustment to keep the mailbox size under control.
Symptom: After working fine for a short period of time, many IMAP events become unreasonably slow, with some events taking over a second.
Diagnosis: You have the Event Notification Service (ENS) plugin, libibiff, configured, but ENS is not running or not reachable. See "Administering Event Notification Service" for ENS details.
Solution: If you want ENS notifications, verify that the ENS is enabled and configured correctly. If you do not want ENS notifications, make sure that libibiff is not being loaded. Typical incorrect configuration:
notifytarget = /opt/sun/comms/messaging/lib/libibiff ens.enable = 0
Instead, use one of the following configurations:
notifytarget = ens.enable = 0
or
notifytarget = /opt/sun/comms/messaging/lib/libibiff ens.enable = 1