Chapter 15 Managing the Message Store

This chapter describes the message store and its administration interface. This chapter contains the following sections:

Overview

The message store contains the user mailboxes for a particular Messaging Server instance. The size of the message store increases as the number of mailboxes, folders, and log files increase. You can control the size of the store by specifying limits on the size of mailboxes (disk quotas), by specifying limits on the total number of messages allowed, and by setting aging policies for messages in the store.

As you add more users to your system, your disk storage requirements increase. Depending on the number of users your server supports, the message store might require one physical disk or multiple physical disks. There are two ways to integrate this additional disk space into your system. The easiest way is to add additional message store partitions (see "Configuring Message Store Partitions").

Likewise, if you are supporting multiple hosted domains, you might want to dedicate a server instance to a single, large domain. With this configuration, you can designate a store administrator for a particular domain. You can also expand the message store by adding more partitions.

To manage the message store, Messaging Server provides a set of command-line utilities in addition to the Sun ONE Console interface. Table 15-1 describes these command-line utilities. For information about using these utilities, see Performing Message Store Maintenance Procedures and the Messaging Server Reference Manual.

Table 15-1 Message Store Command-line Utilities
Utility	Description
configutil	Sets and modifies configuration parameters for the store.
deliver	Delivers mail directly to the message store accessible by IMAP or POP mail clients.
hashdir	Identifies the directory that contains the message store for a particular user.
imsconnutil	Monitors user access of the message store.
imexpire	Automatically removes messages from the message store based on administrator-specified criteria like age.
iminitquota	Reinitializes the quota limit from the LDAP directory and recalculates the disk space being used.
imsasm	Handles the saving and recovering of user mailboxes.
imsbackup	Backs up stored messages.
imsexport	Exports Certificate Management System mailboxes into UNIX /var/mail format folders.
imsrestore	Restores messages that have been backed up.
imscripter	The IMAP server protocol scripting tool. Executes a command or sequence of commands.
mboxutil	Lists, creates, deletes, renames, or moves mailboxes; reports quota usage.
mkbackupdir	Creates and synchronizes the backup directory with the information in the message store.
MoveUser	Moves a user’s account from one messaging server to another.
imquotacheck	Calculates the total mailbox size for each user in the message store and compares the size with their assigned quota. Localized versions of imquotacheck notification incorrectly convert the % and the $ signs. To correct the encoding, replace every $ with \24 and replace every % with \25 in the message file.
readership	Collects readership information on shared IMAP folders.
reconstruct	Reconstructs mailboxes that have been damaged or corrupted.
stored	Performs background and daily tasks, expunges, and erases messages stored on disk.

Message Store Directory Layout

Figure 15-1 shows the message store directory layout for a server instance. The message store is designed to provide fast access to mailbox contents. The store directories are described in Table 15-2.

The message store consists of a number of mailbox databases and the user mailboxes. The mailbox databases consists of information about users, mailboxes, partitions, quotas and other message store related data. The user mailboxes contain the user’s messages and folders. Mailboxes are stored in a message store partition, an area on a disk partition specifically devoted to storing the message store. See "Configuring Message Store Partitions" for details. Message store partitions are not the same as disk partitions, though for ease of maintenance, we recommend having one disk partition for each message store partition.

Mailboxes such as INBOX are located in the store_root. For example, a sample directory path might be:

Table 15-2 Message Store Directory Description
Location	Content/Description
msg_svr_base	Default: /opt/SUNWmsgsr The directory on the Messaging Server machine that holds the server program, configuration, maintenance, and information files.
store_root	msg_svr_base/data/store Top-level directory of the message store. Contains the mboxlist, user, and partition subdirectories.
./store.expirerule	Contains the automatic message removal rules (expire rules). This optional file can be at different locations. See "To Set the Automatic Message Removal (Expire and Purge) Feature".
store_root/dbdata/snapshots	Message store database backup snapshots.
store_root/mboxlist/	Contains mailbox database, database (Berkeley DB) that stores information about the mailboxes and quota information. folder.db contains information about mailboxes, including the name of the partition where the mailbox is stored, the ACL, and a copy of some of the information in store.idx. There is one entry in folder.db per mailbox quota.db contains information about quotas and quota usage. There is one entry in quota.db per user. lright.db - an index for the folders by acl lookup rights. peruser.db contains information about per-user flags. The flags indicate whether a particular user has seen or deleted a message. subscr.db contains information about user subscriptions.
store_root/session/	Contains active message store process information.
store_root/user/	Not used.
store_root/partition/	Contains the message store partitions. A default primary partition is created. Place any other partitions you define in this directory.
store_root/partition/primary/ =user/	Contains all the user mailboxes in the subdirectory of the partition. The mailboxes are stored in a hash structure for fast searching. To find the directory that contains a particular user’s mailbox, use the hashdir utility.
.../=user/hashdir/hashdir/ userid/	The top-level mail folder for the user whose ID is userid. This is the user’s INBOX. For the default domain, userid is uid. For hosted domains, userid is uid@domain. Incoming messages are delivered to this mail folder.
.../userid/folder	A user-defined folder on the messaging server.
.../userid/store.idx	An index that provides the following information about mail stored in the /userid/ directory: number of messages, disk quota used by this mailbox, the time the mailbox was last appended, message flags, variable-length information for each message including the headers and the MIME structure, and the size of each message. The index also includes a backup copy of mboxlist information for each user and a backup copy of quota information for each user.
.../userid/store.usr	Contains a list of users who have accessed the folder. For each user listed, contains information about the last time the user accessed the folder, the list of messages the user has seen, and the list of messages the user has deleted.
.../userid/store.sub	Contains information about user subscriptions.
.../userid/store.exp	Contains a list of message files that have been expunged, but not removed from disk. This file appears only if there are expunged messages.
.../userid/nn/ or .../userid/folder/nn/	nn is a hash directory that contains messages in the format message_id.msg; nn can be a number from 00 to 99. message_id is also a number. Example: messages 1 through 99 are stored in the .../00 directory. The first message is 1.msg, the second is 2.msg, thrid 3.msg, and so on. Messages 100 through 199 are stored in the 01 directory; messages 9990 through 9999 are stored in the 99 directory; messages 10000 through 10099 are in the 00 directory, and so on.

How the Message Store Removes Messages

Delete. A client sets a message flag to delete. At this point, the message is marked for removal, but client can still restore the message by removing the delete flag. If there is a second client, the deleted flag may not be recognized immediately by that second client. You can set the configutil parameter local.imap.immediateflagupdate to enable immediate flag update.

Expunge. Messages are removed from the mailbox. Technically, they are removed from the message store index file, store.idx. The message itself is still on disk, but once messages are expunged, clients can no longer restore them.

Purge. The stored utility purges from the disk any messages that have been expunged at 11PM everyday by default. This can be configured with local.schedule.purge, which controls the message purge schedule, and store.cleanup, which controls the purge grace period (period of time before which the message will not be purged).

Specifying Administrator Access to the Store

Message store administrators can view and monitor user mailboxes and specify access control for the message store. Store administrators have proxy authentication privileges to any service (POP, IMAP, HTTP, or SMTP), which means they can authenticate to any service using the privileges of any user. These privileges allow store administrators to run certain utilities for managing the store. For example, using MoveUser, store administrators can move user accounts and mailboxes from one system to another.

This section discusses how to grant store privileges to the message store for your Messaging Server installation.


Note	Other users might also have administrator privileges to the store. For example, some administrators may have these privileges.

To Add an Administrator

To Modify an Administrator Entry

To Delete an Administrator Entry

To Add an Administrator

From Console, open the Messaging Server you want to configure.

Click the Configuration tab and select Message Store in the left pane.

Click the Administrator tab.

Click the Add button beside the Administrator UID window.

In the Administrator UID field, type the user ID of the administrator you want to add.

Click OK to add the administrator ID to the list displayed in the Administrator tab.

Click Save in the Administrator tab to save the newly modified Administrator list.

where adminlist is a space-separated list of administrator IDs. If you specify more than one administrator, you must enclose the list in quotes. In addition, the administrator must be a member of the Service Administrator Group (in the LDAP user entry: memberOf: cn=Service Administrators,ou=Groups,o=usergroup).

To Modify an Administrator Entry

Console To modify an existing entry in the message store Administrator UID list at the Console:

Click the Administrator tab.

Click the Edit button beside the Administrator UID window.

Enter your changes to the Administrator UID field.

Click OK to submit your changes and dismiss the Edit Administrator window.

Click Save in the Administrator tab to submit and preserve the modified Administrator list.

Command Line To modify an existing entry in the message store Administrator UID list at the command line:

To Delete an Administrator Entry

Console To delete an entry from the message store Administrator UID list by using the Console:

Click the Administrator tab.

Select an item in the Administrator UID list.

Click Delete to delete the item.

Click Save to submit and preserve your changes to the Administrator list.

Command Line To delete store administrators at the command line, you can edit the administrator list as follows:

About Shared Folders

A shared folder is a folder that can be accessed and read by a group of users. In other words, access rights to shared folders are granted to multiple users. For example, a user can create a folder called golf and allow others to view the contents of that folder.

By default, Messaging Server creates a folder called Shared Folders/Users in all email accounts. Users create and access shared folders in this folder. An example of how shared folders would appear in a client is shown in Figure 15-2. This example will be described further in "To Set Up Distributed Shared Folders".

Users can create private shared folders and provide access rights to those folders with their email client, if that client supports shared folders. These shared folders will appear in the Shared Folders of other users who are given access rights.

Shared folders are useful for starting, sharing, and archiving an ongoing conversation on a particular topic. For example, a group of software developers can create a shared folder for discussing development of a particular project. When a message is sent to the shared folder, everyone who is subscribed to the shared folder (subscribers can be added by individual address or by a group address) can open this mailbox and read the message.

Private – A private shared folder is a shared folder owned by a specific user. The owner of a folder grants access to others.

Public – A public shared folder does not have an owner. Administrators create a public user account, which can be used to host public folders. The email address of a public folder looks like this:

public+foldername@domain

Normally shared folders are only available to users on a particular message store. Messaging Server, however, allows you to create special shared folders that can be accessed across multiple message stores. These are called distributed shared folders. See "To Set Up Distributed Shared Folders" for details.

Shared Folder Access Rights

Access rights are maintained in Access Control Lists (ACLs) stored in folder.db. Granting access rights is done by setting an ACL using either the IMAP SETACL command, the -s option with the readership command line utility (see "To Change a Public Folder’s Access Control Rights"), or using the Messenger Express interface.

ACL Identifiers

Each ACL entry has an identifier which specifies the user or group of users for which the entry applies. Identifiers starting with a dash (“-”) represent negative rights (those denied to the user or group).

anyone is a special identifier. The access rights for anyone apply to all users. Similarly, the access rights for anyone@domain apply to all users in the same domain.

ACL Rights Characters

Each ACL entry has a set of rights represented by a string of characters. The string of characters is defined by RFC 2086. To calculate the set of rights for a user, the server adds all the rights granted to this user and all of the groups this user belongs to, then it subtracts all of the rights denied the user and the groups this user belongs to.

The table that follows lists the characters recognized by Messaging Server and gives their names, a brief description of each, and shows the IMAP commands users with this permission are allowed to issue.

Table 15-3 ACL Rights Characters
Character	Description
l	lookup – User can see and subscribe to the shared folder. (IMAP commands allowed: LIST and LSUB)
r	read – Users can read the shared folder. (IMAP commands allowed: SELECT, CHECK, FETCH, PARTIAL, SEARCH, COPY from the folder)
s	seen – Directs the system to keep seen information across sessions. (Set IMAP STORE SEEN flag)
w	write – Users can mark as read, and delete messages. (Set IMAP STORE flags, other than SEEN and DELETED)
i	insert – Users can copy and move email from one folder to another. (IMAP commands allowed: APPEND, COPY into folder)
p	post – Users can send mail to the shared folder email address. (No IMAP command needed)
c	create – Users can create new sub-folders. (IMAP command allowed: CREATE)
d	delete – Users can delete entries from the shared folder. (IMAP commands allowed: EXPUNGE, set STORE DELETED flag)
a	administer – Users have administrative privileges. (IMAP command allowed: SETACL)

Group ACL

The identifier of an ACL entry can specify a group name. The access rights of this entry apply to all of the members of this group. The server determines group memberships by the aclGroupAddr attribute of the inetMailUser object class. A group is represented by a dynamic mailing list with a filter on the aclGroupAddr attribute. The following example shows an LDIF record that defines a group, including the aclGroupAddr attribute:

dn: cn=lee-staff,ou=Groups, o=sesta.com

cn: lee-staff

mailHost: mail.sesta.com

inetMailGroupStatus: active

mgrpErrorsTo: lee.jones@sesta.com

description: Dynamic Group of Lee’s staff

objectClass: top

objectClass: groupofuniquenames

objectClass: inetmailgroup

objectClass: inetmailgroupmanagement

objectClass: inetlocalmailrecipient

objectClass: groupofurls

mail: lee-staff@sesta.com

memberURL: ldap:///o=sesta.com??sub?

(&(aclGroupAddr=lee-staff@sesta.com)(objectclass=inetmailuser))

It is not necessary for a group to be created for the group email address to be used in the folder’s ACL. In practice, it makes sense to create such a dynamic group and set the aclGroupAddr attribute on user entries when adding members to the group. Once such a group is created, static external members can be added by using their email addresses in the attribute mgrpRfc822MailMember. Members should not be added using the uniqueMember attribute, nor by creating additional values of the memberURL attribute. Doing so would cause a disconnect between what the MTA sees as a member of the mailing list and what the IMAP server sees as a group member.

When a user logs in to an IMAP server or logs in with an HTTP access service client such as Messenger Express, the server fetches the aclGroupAddr attribute (along with other message store related attributes) and caches the group names in memory. The server uses this information to determine the user’s access rights whenever the client issues a command (such as LIST or SELECT) that requires access right verification.

Shared Folder Tasks

"To Create a Public Folder"

"To Change a Public Folder’s Access Control Rights"

"To Enable or Disable Listing of Shared Folders"

"To Set Up Distributed Shared Folders"

"To Monitor and Maintain Shared Folder Data"

To Create a Public Folder

Because public folders require access to the LDAP database as well as the readership command, they must be created by system administrators.

Add an LDAP user entry that will act as a container for all public folders, for example, one called public:

dn: cn=public,ou=people,o=sesta.com,o=ISP

objectClass: person

objectClass: organizationalPerson

objectClass: inetOrgPerson

objectClass: inetUser

objectClass: ipUser

objectClass: inetMailUser

objectClass: inetLocalMailRecipient

objectClass: nsManagedPerson

objectClass: userPresenceProfile

cn: public

mail: public@sesta.com

mailDeliveryOption: mailbox

mailHost: manatee.siroe.com

uid: public

inetUserStatus: active

mailUserStatus: active

mailQuota: -1

mailMsgQuota: 100

Create folders within the public account by using the mboxutil command line utility. For example:

mboxutil -c user/public/golftournament

Using the readership command line utility, set the appropriate ACL for this folder.

In order to make this folder public, it must be assigned a group of users that can access it. This is done by setting an ACL using the readership command. For instructions on how to set the ACL, see "To Change a Public Folder’s Access Control Rights" that follows.

To Change a Public Folder’s Access Control Rights

From time to time you might need to change the access control for a public folder, or you will need to set the access control for a public folder you have newly created.

To do this, use the readership command line utility. The command has the following form:

where foldername is the name of the public folder for which you are setting rights, userid is the person or group to whom you are assigning the rights, and rights_chars are the rights you are assigning (these are RFC 2086 compliant access rights characters). For the meaning of each character, see "ACL Rights Characters". You can also change the access control for a public folder using the Messenger Express interface.

Examples

For example, if you wish everyone at the sesta domain to have lookup, read and email marking (but not posting) access to the public folder, golftournament, issue the command as follows:.

To assign lookup, read, email marking and posting rights to a group, issue the command as follows:

If you want to assign administrator and posting rights for this folder to an individual, jdoe, issue the command as follows:

To deny an individual or group access to a public folder, prefix the userid with a dash. For example, to deny lookup, read and write rights to jsmith, issue the command as follows:

To Enable or Disable Listing of Shared Folders

The server will or will not return shared folders when responding to a LIST command depending on the setting in the configuration option local.store.sharedfolders. Setting the option to off disables it. The setting is enabled by default (set to on).

SELECT and LSUB commands are not affected by this option. The LSUB command returns every subscribed folder, including shared folders. Users can SELECT the shared folders they own or are subscribed to.

To Set Up Distributed Shared Folders

Normally shared folders are only available to users on a particular message store. Messaging Server, however, allows you to create distributed shared folders that can be accessed across multiple message stores. That is, access rights to distributed shared folders can be granted to any users within the group of message stores. Note, however, that web mail clients (HTTP access clients like Messenger Express) do not support remote shared folders access. Users can list and subscribe to the folders, but they can’t view or alter the contents.

The message store userids must be unique across the group of message stores.

The directory data across the deployment must be identical.

The remote message stores (that is the message stores that do not hold the shared folder) must be configured as proxy servers by setting the configuration variables listed in Table 15-4.

Table 15-4 Variables for Configuring Distributed Shared Folders
Name	Value	Data Format
local.service.proxy.serverlist	message store server list	space-separated strings
local.service.proxy.admin	default store admin login name	string
local.service.proxy.adminpass	default store admin password	string
local.service.proxy.admin.hostname	store admin login name for a specific host	string
local.service.proxy.adminpass.hostname	store admin password for a specific host	string

Setting Up Distributed Shared Folders—Example

Figure 15-3 shows a disturbed folder example of three message store servers called StoreServer1, StoreServer2, and StoreServer3.

These servers are connected to each other as peer proxy message stores by setting the variables shown in Table 15-4. Each server has a private shared folder—golf (owned by Han), tennis (owned by Kat), and hurling (owned by Luke). In addition there are two public shared folders called press_releases and Announcements. Users on any of the three servers can access any of these three shared folders. Figure 15-2 shows Ed’s shared folder list. Below is an example of the ACLs for each server in this configuration.

To Monitor and Maintain Shared Folder Data

The readership command line utility allows you to monitor and maintain shared folder data which is held in the folder.db, peruser.db, and lright.db files. folder.db has a record for each folder that holds a copy of the ACLs.The peruser.db has an entry per user and mailbox that lists the various flags settings and the last date the user accessed any folders. The lright.db has a list of all the users and the shared folders for which they have lookup rights.

Table 15-5 readership Options
Options	Description
-d days	Returns a report, per shared folder, of the number of users who have selected the folder within the specified days.
-p months	Removes data from the peruser.db for those users who have not selected their shared folders within the specified months.
-l	List the data in lright.db.
-s folder_identifier_rights	Set access rights for the specified folder. This updates the lright.db as well as the folder.db.

"To Monitor Shared Folder Usage"

"To List Users and Their Shared Folders"

"To Remove Inactive Users"

"To Set Access Rights"

To Monitor Shared Folder Usage

To find out how many users are actively accessing shared folders, issue the command:

where days is the number of days to check. Note that this option returns the number of active users, not a list of the active users.

Example: To find out the number of users who have selected shared folders within the last 30 days, issue the following command:

To List Users and Their Shared Folders

To list users and the shared folders to which they have access, issue the command:

$ readership -l
group=lee-staff@siroe.com: user/user2/lee-staff
richb: user/golf user/user10/Drafts user/user2/lee-staff user/user10/Trash
han1: user/public+hurling@siroe.com user/golf
gregk: user/public+hurling@siroe.com user/heaving user/tennis

To Remove Inactive Users

If you want to remove inactive users (those who have not accessed shared folders in a specified time period) issue the command:

Example: Remove users who have not accessed shared folders for the past six months:

To Set Access Rights

You can assign access rights to a new public folder, or change access rights on a current public folder.

About Message Store Quotas

"User Quotas"

"Domain Quotas"

"Exceptions for Telephony Application Servers"

User Quotas

You can limit the size of the message store by specifying limits on the size of user mailboxes. You can specify the following types of quotas.

Disk quotas allow you to limit the amount of disk space allotted to each user. Disk quotas apply to the total size of all the user’s messages, regardless of how many mail folders the user has or to the total number of user messages. If disk space is limited, you might want to set user disk quotas.

Message quotas allow you to limit the number of messages stored in a user’s mailbox.

Quota information is stored as LDAP attributes and configuration variables. If quota enforcement is enabled, Messaging Server checks the quota cache and configuration file to ensure quotas have not been exceeded before inserting messages into the message store. If quota notification is enabled, users are sent an error message when they have reached their disk quota. You can also enable the server to send a warning message when users are nearing their quota limit.

You can set default quotas for all users or set quotas for individual users. To determine if a user is over quota, Messaging Server first checks to see if a quota has been set for the individual user. If no quota has been set, Messaging Server then looks at the default quota set for all users.

If a user’s messages exceed their quota, incoming messages remain in the MTA queue until one of the following occurs:

(1) The size or number of the user’s messages no longer exceeds the quota, at which time the MTA delivers the messages to the user. (2) The undelivered message remains in the MTA queue longer than the specified grace period. See "To Set a Grace Period".

Disk space becomes available when a user deletes and expunges messages or when the server deletes messages according to the aging policies you have established.

Domain Quotas

You can also set quotas for a particular domain using the command imquotacheck -f. When a domain exceeds its quota, the maildomainstatus attribute is set to overquotam, which halts all delivery to this domain. If a domain is not overquota, the value is set to active.

Exceptions for Telephony Application Servers

To support unified messaging requirements, Messaging Server provides the ability to override quota limitations imposed by the message store. This guarantees the delivery of messages that have been accepted by certain agents, namely telephony application servers (TAS). Messages accepted by a TAS can be routed through a special MTA channel that will ensure the message is delivered to the store regardless of quota limits. For more information about configuring the TAS channel, see Chapter 10, "Configuring Channel Definitions".

Configuring Message Store Quotas

Set default quotas for all users by using Sun ONE Console or by using the configutil command. You can also set quotas for individual users, family groups, and hosted domains.

"To Specify a Default User Quota"

"To Enabling Quota Enforcement and Notification"

"To Set a Grace Period"

From Console, open the Messaging Server you want to configure.

Click the Configuration tab and select Message Store in the left pane.

Click the Quota tab in the right pane.

To Specify a Default User Quota

The default quota applies to users who do not already have individual quotas set for them. A quota set for an individual user overrides the default quota.

Click the Quota tab.

To specify a default user disk quota, for the “Default user disk quota” field, select one of the following options:

Unlimited. Select this option if you do not want to set a default disk quota.

Size specification. Select this option if you want to restrict the default user disk quota to a specific size. In the field beside the button, type a number, and from the drop-down list, choose Mbytes or Kbytes.

To specify a message number quota, in the “Default user message quota” box, type a number.

Click Save.

To Enabling Quota Enforcement and Notification

You can enable or disable quota enforcement and quota notification. The action the server takes depends on how these configuration variables are set, as shown in Table 15-6.

Table 15-6 Quota Enforcement and Notification
	Enforcement On	Enforcement Off
Notification On	Messages are deferred for specified grace period; rejected if grace period expires. Messages cannot be appended to mailbox. IMAP SELECT, IMAP APPEND, SMTP sendmail mechanism and deliver command will display error message.	Messages are delivered to the store. Messages can be appended to mailbox. IMAP SELECT, IMAP APPEND, SMTP sendmail mechanism and deliver command do not display error messages.
Notification Off	Messages are deferred for specified grace period; rejected if grace period expires. Messages cannot be appended to mailbox. IMAP SELECT command, deliver command, and SMTP sendmail mechanism do not display error message. IMAP APPEND command will display error message.	Messages are delivered to the store. Messages can be appended to mailbox. IMAP SELECT, IMAP APPEND, SMTP sendmail mechanism and deliver command do not display error message.

Enabling Quota Enforcement

Click the Quota tab.

Check the “Enable quota enforcement” box.

This box acts as a toggle. To disable quota enforcement, uncheck this box.

Click Save.

To start enforcement after quota is exceeded (that is, allow the message that will push the message store over quota before starting quota enforcement) set the above value to on. The default is off.

Enabling Quota Notification

Click the Quota tab.

Check the “Enable quota notification” box.

This box acts as a toggle. To disable quota notification, uncheck this box.

Define the quota warning messages

See "Defining a Quota Warning Message".

Click Save.

configutil -o store.quotanotification -v [ yes | no ]
configutil -o store.quotaexceededmsg -v message

If the message is not set, then no quota warning message will be sent to the user. See the next section for an example of quota warning message format.

Defining a Quota Warning Message

You can define the message that will be sent to users who have exceeded their disk quota as follows. Messages are sent to the user’s mailbox.

Click the Quota tab.

From the drop-down list, choose the language you want to use.

Type the message you want to send in the message text field below the drop-down list.

Click Save.

The message must be in RFC 822 format. It must contain a header (with at least a subject line), follow by $$, then the message body. ’$’ represents a new line. Example:

configutil -o store.quotaexceededmsg -v ‘Subject: WARNING: User quota exceeded$$User quota threshold exceeded - reduce space used.’

where number indicates a number of days. For example, 3 would mean the message is sent every 3 days.

Specifying a Quota Threshold

You can send a warning message to IMAP users before they reach their disk quota by specifying a quota threshold. When a user’s disk usage exceeds the specified threshold, the server sends a warning message to the user.

For IMAP users whose clients support the IMAP ALERT mechanism, the message is displayed on the user’s screen each time the user selects a mailbox (a message is also written to the IMAP log).

Click the Quota tab.

In the “Quota warning threshold” field, enter a number for the warning threshold.

This number represents a percentage of the allowed quota. For example, if you specify 90%, the user is warned after using 90% of the allowed disk quota. The default is 90%. To turn off this feature, enter 100%.

Click Save.

To Set a Grace Period

The grace period specifies how long the mailbox can be over the quota (disk space or number of messages) before messages are bounced back to sender. Messages are accepted by the MTA, but remain in the MTA queue and are not delivered to the message store until one of the following occurs:

The mailbox no longer exceeds the quota, at which time messages are delivered to the mailbox.

The user has remained over quota longer than the specified grace period, at which time the server will bounce all messages including those in the queue.

The message has remained in the message queue longer than the maximum message queue time.

For example, if your grace period is set for two days, and you exceed quota for one day, new messages will continue to be received and held in the message queue, and delivery attempts will continue. After the second day, messages bounce.


Note	Grace period is NOT how long the message will held in the message queue, it’s how long the mailbox is over quota before all incoming messages, including those in the message queue, are bounced. The grace period starts when the user has reached the quota threshold (see "Specifying a Quota Threshold") and been warned.

Console To set a grace period for how long messages are held in the queue at the Console:

Click the Quota tab.

In the “Over quota grace period” field, enter a number.

From the drop-down list, specify Day(s) or Hour(s).

Click Save.

To Set the Automatic Message Removal (Expire and Purge) Feature

The automatic message removal feature (also known as expire and purge) automatically removes messages from the message store based on a set of administrator-defined criteria.This feature can automatically remove old or overly large messages, seen/deleted messages, messages with certain Subject: lines and so on. This feature allows the following removal criteria:

By folder (mailboxes), users, domains, the entire message store, or specific partitions

Number of messages in the mailbox

Total size of the mailbox

Age, in days, that a message has been in the mailbox

Size of message and grace period (days that an oversized message will remain in the message store before purging)

Whether a message has been flagged as seen or deleted

Header strings

This feature is performed by the imexpire utility, which expunges and purges messages. See "How the Message Store Removes Messages" for details on the message removal process.

imexpire Theory of Operation


Note	The server removes messages without warning, so it is important to inform users about automatic message removal policies. Unexpected message removal can be a source of consternation for users and administrators.

imexpire can be invoked from the command line or scheduled to run automatically by the imsched daemon. The administrator configures the global expiration rules (that is rules for the entire message store) with either Console or the configutil command line utility. Local expire rules (rules that apply to folders or users) can be configured by creating expire rule files (store.expire) in a message store partition, user or mailbox directories.

imexpire loads all of the expire rules at start up. By default, imexpire creates one thread per partition. Each thread goes through the list of user folders under its assigned partition and loads the local expire rule files as it goes. The expire function checks each folder against the expire rules applicable to this folder and expunges messages as needed. If there is a store.exp file that exists under the mailbox directory, and there are messages that have been expunged/expired for longer than the time specified by the store.cleanupage configuration parameter, the purge function will permanently remove the message files under the message hash directories and remove the userID records from the store.exp files.

To Deploy the Automatic Message Removal Feature

Automatic message removal can be deployed by command line or by using the Console GUI. The process requires three steps:

Define automatic message removal policy: Which messages will be automatically removed? What users, domains and partitions will have messages automatically removed? What size, message age, headers will define the removal criteria. See "To Define Automatic Message Removal Policy".

Specify theimexpire rules to implement this policy. See "To Set Rules Implementing Automatic Message Removal Policy".

Specify the imexpire scheduling. See "To Schedule Automatic Message Removal and Logging Level".

To Define Automatic Message Removal Policy

Define your automatic message removal policy by specifying the criteria for removal. imexpire allows for removal using the following criteria:

Age of Message. Automatically remove messages older than X days. Attribute: messagedays.

Message Count. Automatically remove messages in a folder exceeding X messages. Attribute: messagecount.

Age of Oversized Message. Automatically remove messages that exceed X bytes after Y days grace period. Attributes: messagesize and messagesizedays.

Seen and Deleted Message Flag. Automatically remove messages with the Seen or Deleted flag set. These criteria can be set to “and” or “or.” If set to or, the message’s Seen/Delete flag will cause automatic deletion regardless of other criteria. If set to and, the message’s Seen/Delete flag must be set along with passing all other specified criteria. Attributes: seen and deleted.

Header Field of Message. Allows you specify a header and string as criteria for removing a message. For example, removing all messages with the header “Subject: Work from Home!”

Folder of Messages. Allows you to specify the folder on which to remove messages. Attribute: folderpattern

Examples of Automatic Message Removal Policy

Example 1: Remove all messages 365 days old in a folder exceeding 1,000 messages.


Note	imexpire does not allow you to delete or preserve messages based on how long it has been since that message was read. For example, you cannot specify that messages that have not been read for 200 days will be removed.

Example 4: Remove messages in sesta.com that have been marked as seen, are older than 30 days, are larger than 100 kilobytes, from folders exceeding 1,000 messages, with the header X-spam.

To Set Rules Implementing Automatic Message Removal Policy

To implement the automatic message removal policy defined in the previous section, you must set the imexpire rules. Rules can be set in the following ways:

By GUI (see Figure 15-4)

By setting the store.expirerule.attribute configutil parameters

By putting them into a store.expirerule file. An example of two store.expirerule rules is shown below:

Rule1.folderpatter: user/.*/trash
Rule1.messagedays: 2
Rule2.folderpattern: user/.*
Rule2.messagedays: 14

In this example, Rule 1 specifies that all messages in the trash folder will be remove after two days. Rule 2 specifies that all messages in the message store will be removed after 14 days.

Expiration Rules Guidelines

This section sets the guidelines for the store.expirerule.attribute configutil parameters and the store.expirerule file rules.

Rules are specified in a file called store.expirerule, or by using the configutil parameter store.expirerule.rulename.attribute.

Multiple expiration criteria can be specified with the same rule. (See example above.)

Rules can apply to the entire message store (global rules), a message store partition, a user, or a folder. Non-global rules can only be created with the store.expirerule rules.

Global rules are created by using the configutil parameter store.expirerule.rulename.attribute or by specifying rules in msg_svr_base/config/store.expirerule

Partition rules can be created by specifying rules in store_root/partition/partition_name/store.expirerule.

User rules can be created by specifying rules in store_root/partition/partition_name/userid/store.expirerule or by specifying the folderpattern rule to be user/userid/.*

Folder rules can be created by specifying rules in store_root/partition/partition_name/userid/folder/store.expirerule or by specifying the folderpattern rule to be user/userid/folder



Note	User and folder rules can also be placed in the global expire file (msg_svr_base/config/store.expirerule) by specifying the folderpattern attribute.

Multiple expire rules can be applied to a mailbox at the same time. An expire policy for a mailbox consists of global rules and local rules. Local rules apply to the mailbox under the same directory and all of its sub-folders.

imexpire unifies all of the expiration rules applying to a mailbox, unless there is an exclusive rule specified for this mailbox (see Table 15-7). The resulting rule set represents the most restrictive expiration policy based on all applicable rules. For example, if rule X expires messages such that the maximum message life is 10 days, and rule Y specifies 5 days, the union will be 5 days.

Table 15-7 imexpire Attributes
Attribute	Description (Attribute Value)
exclusive	Specifies whether or not this is an exclusive rule. If specified as exclusive, only this rule applies to the specified mailbox(s) and all other rules are ignored. If more than one exclusive rule exists, the last exclusive rule loaded will be used. For example, if a global and a local exclusive rule are specified, the local rule will be used. If there is more than one global exclusive rule, the last global rule listed by configutil is used. (yes/no)
folderpattern	Specifies the folders affected by this rule. The format must start with a user/, which represents the directory store_root/partition/*/ See Figure 15-4 and Table 15-8. (POSIX regular expression)
messagecount	Maximum number of messages in a folder. Oldest messages are expunged as additional messages are delivered. (integer)
foldersize	Maximum size of folder before the oldest messages are expunged when additional messages are delivered. (integer in bytes)
messagedays	Age of message in days before being expunged. (integer)
messagesize	Maximum size of message in bytes before it is marked to be expunged. (integer)
messagesizedays	Grace period. Days an over-sized message should remain in a folder. (integer)
message header field	Specifies a header field and string that marks a message for removal. Values are not case-sensitive and regular expressions are not recognized. Example: Rule1.Subject: Get Rich Now! For the header Expires and Expiry-Date, imexpire will remove the message if the date value specified with these header fields is older than the messagedays attribute. If multiple expiration header fields are specified, the earliest expiration date will be used. (string).
regexp	Enable UNIX regular expressions in rules creation. (1 or 0)
seen	seen is a message status flag set by the system when the user opens a message. If the attribute seen is set to and, then the message must be seen and other criteria must be met before the rule is fulfilled. If the attribute seen is set to or, then the message only needs to be seen or another criteria be met before the rule is fulfilled. (and/or).
deleted	deleted is a message status flag set by the system when the user deletes a message. If the attribute deleted is set to and, then the message must be deleted and another criteria must be met before the rule is fulfilled. If the attribute deleted is set to or, then the message only needs to be seen or another criteria be met before the rule is fulfilled. (and/or)

Setting imexpire Rules Textually

Automatic message removal rules can be set textually by using the configutil parameter store.expirerule.rulename.attribute or by specifying rules in a store.expirerule file.

The store.expirerule file contains one expire criteria per line. An expire criteria of the global rule configuration file (msg_svr_base/data/store/store.expirerule) has the following format:

Code Example 15-1 shows a set of expiration rules in msg_svr_base/config/store.expirerule.

Rule 1 sets the global expiration policy (that is, policy that applies to all messages), as follows:

Enable UNIX regular expressions in rules creation.

Removes messages larger than 100,000 bytes after 3 days.

Removes messages deleted by the user.

Removes any message with the strings “Viagra Now!” or “XXX Porn!” in the Subject: header.

Limits all folders to 1,000 messages. After 1,000 messages, the system removes the oldest messages on a folder to keep the total to 1,000.

Removes all messages older than 365 days.

Rule 2 sets the automatic message removal policy for users at the hosted domain siroe.com. It limits mailbox sizes to 1 megabyte, removes messages that have been deleted, and removes messages older than 14 days.

Rule 3 sets the automatic message removal policy for messages in the inbox folder of user f.dostoevski. It removes messages with a subject line having the expression “On-line Casino.”

Code Example 15-1 Example imexpire Rules


Rule1.regexp: 1
Rule1.folderpattern: user/.*
Rule1.messagesize: 100000
Rule1.messagesizedays: 3
Rule1.deleted: or
Rule1.Subject: Viagra Now!
Rule1.Subject: XXX Porn!
Rule1.messagecount: 1000
Rule1.messagedays: 365
Rule2.regexp: 1
Rule2.folderpattern: user/.@siroe.com/.
Rule2.exclusive: yes
Rule2.deleted: or
Rule2.messagedays: 14
Rule2.messagecount: 1000
Rule3.folderpattern: user/f.dostoevski/inbox
Rule3.Subject: On-line Casino

% configutil store.expirerule.rule1.regexp 1
% configutil store.expirerule.rule1.messagesizedays 3
% configutil store.expirerule.rule1.deleted or
% configutil store.expirerule.rule1.Subject Viagra Now!
% configutil store.expirerule.rule1.Subject XXX Porn!
% configutil store.expirerule.rule1.messagecount 1000
% configutil store.expirerule.rule1.messagedays 365
% configutil store.expirerule.rule1.messagesize 100000

Setting imexpire Folder Patterns

Folder patterns can be specified using POSIX regular expressions. The format must start with a user/, which represents the directory store_root/partition/*/ (Table 15-8 shows the folder pattern for various folders.)

Table 15-8 imexpire Folder Patterns
Folder Pattern	Scope
user/userid/.*	Apply rule to all messages in all folders of userid.
user/userid/Sent	Apply rule to messages of userid in folder Sent:
user/.*	Apply rule to entire message store.
user/.*/trash	Apply rule to the trash folder of all users.
user/.@siroe.com/.	Apply rule to folders in hosted domain siroe.com:
user/[^@]/.	Apply rule to folders in default domain.
user/partition_name/.*	Apply rule to specific parition.

To Set Automatic Message Removal Rules with the Console

Bring up the automatic message removal GUI as follows:

Main Console > Server Group > Messaging Server (Open) >Messaging Server Console > Configuration Tab > Message Store > Expire/Purge >Add

A very rough drawing of the GUI is shown on Figure 15-4.

Figure 15-4 Automatic Message Removal (Expire/Purge) GUI—Rough Drawing
Graphic shows an approximate drawing of the Automatic Message Removal GUI.

Enter a name for the new rule.

Enter the folders from which messages will be automatically removed.

See "Setting imexpire Folder Patterns" above.

If this rule is the exclusive rule for folders matching the specified criteria, then check the Exclusive box.

If this box is checked, then this rule takes precedence over all other rules matching the specified pattern. See Table 15-7 for details on the exclusive checkbox.

To create a rule based on folder size, do the following:

Check the Folder Size constraint checkbox. In the Message count field, specify the maximum number of messages that will be retained in a folder before the oldest messages are removed. In the Folder size field, specify the maximum folder size, in bytes, before the oldest messages are removed.

To create a rule based on message age, check the Message age constraint checkbox:

In the Number of days field, specify the age, in days, of how long messages should remain in the folder.

To create a rule based on message size:

Check the Message size limit constraint checkbox. In the Message size limit field, enter the maximum size of a message allowed in the folder. In the “Grace period” field, enter the how long over-sized messages will remain in the folder before being removed.

To create a rule based on whether the Seen or Deleted message flags are set:

Check the Message flags constraint checkbox.

For the Seen: field, select “and” to specify that the message must be seen and another criteria must be met before the rule is fulfilled. Select “or” to specify that the message need only be seen or another criteria be met before the rule is fulfilled.

For the Deleted: field, select “and” to specify that the message must be deleted and another criteria must be met before the rule is fulfilled. Select “or” to specify that the message need only be deleted or another criteria be met before the rule is fulfilled.

To create a rule based on a header fields and their values:

Check the Header constraint checkbox.

Enter a comma separated list of headers and values in the following format:

header1: value1, header2: value2

Example: Subject: Work at Home!,From: virus@sesta.com

For the header Expires and Expiry-Date, the system will remove the message if their date value is older than the Message age constraint. If multiple expiration header fields are specified, the earliest expiration date will be used. (string).

Click OK to add the new rule to the Automatic Message Removal list.

To Schedule Automatic Message Removal and Logging Level

Automatic message removal is activated by the imsched scheduling daemon. By default, imsched invokes imexpire at 23:00 every day, and messages are both expunged and purged. This schedule can be customized by setting the configutil parameters local.schedule.expire, local.schedule.purge, and store.cleanupage described in Table 15-9.

Expire and purge can take a long time to complete on a large message store, so you may wish to experiment and decide how often to run these processes. For example, if an expire/purge cycle takes 10 hours, you may not want the default schedule of running expire and purge once a day. Schedule expire and purge by using local.schedule.purge to specify a separate schedule for purge. If local.schedule.purge is not set, imexpire will perform purge after an expire.

Table 15-9 Expire and Purge configutil Log and Scheduling Parameters
Parameter	Description
local.schedule.expire	Interval for running imexpire. Uses UNIX crontab format: minute hour day-of-month month-of-year day-of-week The values are separated by a space or tab and can be 0-59, 0-23, 1-31, 1-12 or 0-6 (with 0=Sunday) respectively. Each time field can be either an asterisk (meaning all legal values), a list of comma-separated values, or a range of two values separated by a hyphen. Note that days can be specified by both day of the month and day of the week, however, it is not typical to use them both since the number of such occurrences are very small. If they are both specified, then both will be required. For example, setting the 17th day of the month and Tuesday will require both values to be true. Interval Examples: 1) Run imexpire at 12:30am, 8:30am, and 4:30pm: 30 0,8,16 * * * 2) Run imexpire at weekday morning at 3:15 am: 15 3 * * 1-5 3) Run imexpire only on Mondays: 0 0 * * 1 Default: 0 23 * * *
local.schedule.purge	Interval for running purge. Uses UNIX crontab format: minute hour day-of-month month-of-year day-of-week. Default: 0 0,4,8,12,16,20 * * * /opt/SUNWmsgsr/lib/purge -num=5 (Every four hours.)
store.cleanupage	Age (in hours) of expired or expunged message before purge will permanently remove it. Default: None
local.store.expire.loglevel	Specify a log level: 1 = log summary for the entire expire session. 2 = log one message per mailbox expired. 3 = log one message per message expired. Default: 1

imexpire Scheduling Using the Console

Main Console > Server Group > Messaging Server (Open) >Messaging Server Console > Configuration Tab > Message Store > Expire/Purge

This Console page lists the expire rules on the top and the expire and purge schedule on the bottom. To schedule expire and purge, use the pull down menus in the Expire/Purge Schedule to set the month, day of month, day of week (with 0=Sunday), hours of day and minute of hours for both expire and purge.

Setting imexpire Logging Levels


Note	The day value may be set by both day of the month and day of the week. Both are adhered to if both are set. If you set the 3rd day of the week (Wednesday) and the 17th day of the month, then a purge/expire will only occur on the 17th day of each month that falls on a Wednesday).

imexpire will log a summary to the default log file upon completion. If expire is invoked from the commmand line, the -v (verbose) and -d (debug) option can be used to instruct imexpire to log detail status/debug messages to stderr. If imexpire is invoked by imsched, the configutil parameter local.store.expire.loglevel can be set to 1,2 or 3 for different levels of logging. Loglevel 1 is the default, it will log a summary for the entire expire session. Loglevel 2 will log one message per mailbox expired. Loglevel 3 will log one message per message expired.

Configuring Message Store Partitions

Mailboxes are stored in message store partitions, an area on a disk partition specifically devoted to storing the message store. Message store partitions are not the same as disk partitions, though for ease of maintenance, it is recommended that you have one disk partition and one file system for each message store partition. Message store partitions are directories specifically designated as a message store.

User mailboxes are stored by default in the store_root/partition/ directory (see Figure 15-1). The partition directory is a logical directory that might contain a single partition or multiple partitions. At start-up time, the partition directory contains one subpartition called the primary partition.

You can add partitions to the partition directory as necessary. For example, you might want to partition a single disk to organize your users as follows:

store_root/partition/mkting/
store_root/partition/eng/
store_root/partition/sales/

As disk storage requirements increase, you might want to map these partitions to different physical disk drives.

You should limit the number of mailboxes on any one disk. Distributing mailboxes across disks improves message delivery time (although it does not necessarily change the SMTP accept rate). The number of mailboxes you allocate per disk depends on the disk capacity and the amount of disk space allocated to each user. For example, you can allocate more mailboxes per disk if you allocate less disk space per user.

If your message store requires multiple disks, you can use RAID (Redundant Array of Inexpensive Disks) technology to ease management of multiple disks. With RAID technology, you can spread data across a series of disks but the disks appear as one logical volume so disk management is simplified. You might also want to use RAID technology for redundancy purposes; that is, to duplicate the store for failure recovery purposes.

To Add a Partition

When adding a partition, you specify both an absolute physical path where the partition is stored on disk and a logical name, called the partition nickname.

The partition nickname allows you to map users to a logical partition name regardless of the physical path. When setting up user accounts and specifying the message store for a user, you can use the partition nickname. The name you enter must be an alphanumeric name and must use lowercase letters.

To create and manage the partition, the user ID used to run the server must have permission to write to the location specified in the physical path.


Note	To improve disk access, the message store and the message queue should reside on separate disks.


Note	After adding a partition, you must stop then restart the server to refresh the configuration information.

From Console, open the Messaging Server you want to configure.

Click the Configuration tab and select Message Store in the left pane.

Click the Partition tab in the right pane.

Click the Add button.

Enter the Partition nickname.

This is the logical name for the specified partition.

Enter the Partition path.

This is the absolute path name for the specified partition.

To specify this as the default partition, click the selection box labeled Make This the Default Partition.

Click OK to submit this partition configuration entry and dismiss the window.

Click Save to submit and preserve the current Partition list.

where nickname is the logical name of the partition and path indicates the absolute path name where the partition is stored.

To Move Mailboxes to a Different Disk Partition

By default, mailboxes are created in the primary partition. If the partition gets full, additional messages cannot be stored. There are several ways to address the problem:

Reduce the size of user mailboxes

If you are using volume management software, add additional disks

Create additional partitions ("To Add a Partition") and move mailboxes to the new partitions

If possible, we recommend adding additional disk space to a system using volume management software since this procedure is the most transparent for the user. However, you may also move mailboxes to a different partition by doing the following:

Make sure user is disconnected from their mailbox during the migration process. This can be done by informing the user to log off and stay off during mailbox move, or, by setting the mailAllowedServiceAccess attribute so that POP, IMAP and HTTP services are disallowed after they are logged off. (See Sun ONE Messaging Server Schema Reference Manual.



Note	Setting mailAllowedServiceAccess to disallow POP, IMAP,HTTP access does not disconnect any open connections to the mailbox. You must make sure that all connections are closed prior to the moving mailboxes.

Move the user mailbox with the following command:

mboxutil -r user/<userid>/INBOX user/<userid>/INBOX <partition_name>

Example:

mboxutil -r user/ofanning/INBOX user/ofanning/INBOX secondary

Set the mailMessageStore attribute in the moved user’s LDAP entry to the name of the new partition.

Example: mailMessageStore: secondary

Inform the user that message store connection is now allowed. If applicable, change the mailAllowedServiceAccess attribute to allow POP, IMAP and HTTP services.

Performing Message Store Maintenance Procedures

This section provides information about the utilities you use to perform maintenance and recovery tasks for the message store. You should always read your postmaster mail for warnings and alerts that the server might send. You should also monitor the log files for information about how the server is performing. For more information about log files, see Chapter 17, "Logging and Log Analysis".

"To Manage Mailboxes"

"To Monitor Quota Limits"

"To Monitor Disk Space"

"Using the stored Utility"

To Manage Mailboxes

This section describes the following utilities for managing and monitoring mailboxes: mboxutil, hashdir, readership.

The mboxutil Utility

You use the mboxutil command to perform typical maintenance tasks on mailboxes. Note that you should not to kill the mboxutil process in the middle of execution. If it is killed with SIGKILL (kill -9), it may potentially require that every server gets restarted and a recovery be done.

List mailboxes

Create mailboxes

Delete mailboxes

Rename mailboxes

Move mailboxes from one partition to another

You can also use the mboxutil command to view information about quotas. For more information, see To Monitor Quota Limits.

Table 15-10 lists the mboxutil commands. For detailed syntax and usage requirements, see the Messaging Server Reference Manual.

Table 15-10 mboxutil Options
Option	Description
-a	Lists all user quota information.
-c mailbox	Creates the specified mailbox.
-d mailbox	Deletes the specified mailbox.
-f file	Creates, deletes, or locks the mailbox or mailboxes listed in the specified data file.
-k mailbox cmd	Locks the specified mailbox at the folder level; runs the specified command; after command completes, unlocks the mailbox.
-l	Lists all of the mailboxes on a server.
-p pattern	When used in conjunction with the -l option, lists only those mailboxes with names that match pattern. You can use POSIX regular expressions
-q domain	Lists quota information for the specified domain.
-r oldname newname [partition]	Renames the mailbox from oldname to newname. To move a folder from one partition to another, specify the new partition with the partition option. This option can be used to rename a user. For example, mboxutil -r user/user1/INBOX user/user2/INBOX moves all mail and mailboxes from user1 to user2, and new messages will appear in the new INBOX. (If user2 already exists, this operation will fail.)
-u user	Lists user information such as current size of mail store, quota (if one has been set), and percentage of quota currently in use.
-x	When used in conjunction with the -l option, shows the path and access control for a mailbox.

Mailbox Naming Conventions


Note	POSIX regular expressions can be used in the mboxutil command.

You must specify mailbox names in the following format: user/userid/mailbox, where userid is the user that owns the mailbox and mailbox is the name of the mailbox. For hosted domains, userid is uid@domain.

For example, the following command creates the mailbox named INBOX for the user whose user ID is crowe. INBOX is the default mailbox for mail delivered to the user crowe.

Important: The name INBOX is reserved for each user’s default mailbox. INBOX is the only folder name that is case-insensitive. All other folder names are case-sensitive.

Examples

To delete the default mailbox named INBOX and all mail folders for the user druscilla:

To move the mail folder named personal for the user dimitria to a new partition:

The hashdir Utility

The mailboxes in the message store are stored in a hash structure for fast searching. Consequently, to find the directory that contains a particular user’s mailbox, use the hashdir utility.

This utility identifies the directory that contains the message store for a particular account. This utility reports the relative path to the message store, such as d1/a7/. The path is relative to the directory level just before the one based on the user ID. The utility sends the path information to the standard output.

The readership Utility

The readership utility reports on how many users other than the mailbox owner have read messages in a shared IMAP folder.

An owner of a IMAP folder may grant permission for others to read mail in the folder. A folder that others are allowed to access is called a shared folder. Administrators can use the readership utility to see how many users other than the owner are accessing a shared folder.

This utility scans all mailboxes and produces one line of output per shared folder, reporting the number of readers followed by a space and the name of the mailbox.

Each reader is a distinct authentication identity that has selected the shared folder within the past specified number of days. Users are not counted as reading their own personal mailboxes. Personal mailboxes are not reported unless there is at least one reader other than the folder’s owner.

For example, the following command counts as a reader any identity that has selected the shared IMAP folder within the last 15 days:

To Monitor Quota Limits

You can monitor quota usage and limits by using the mboxutil utility. The mboxutil utility generates a report that lists defined quotas and limits, and provides information on quota usage. Quotas and usage figures are reported in kilobytes.

To Monitor Disk Space

You can specify how often the system should monitor disk space and under what circumstances the system should send a warning. To configure disk space monitoring and notification, you use the configutil command to set the alarm space attributes, which are described in Table 15-11.

For example, if you want the system to monitor disk space every 600 seconds, specify the following command:

If you want to receive a warning whenever available disk space falls below 20%, specify the following command:

Table 15-11 Disk Space Alarm Attributes
Disk Space Attributes	Default Value
alarm.diskavail.msgalarmstatinterval	3600 seconds
alarm.diskavail.msgalarmthreshold	10%
alarm.diskavail.msgalarmwarninginterval	24 hours

For more information about setting alarm attributes, see the Messaging Server Reference Manual and "Monitoring Disk Space".

Using the stored Utility

The stored utility performs the following monitoring and maintenance tasks for the server:

Background and daily messaging tasks.

Deadlock detection and rollback of deadlocked database transactions.

Cleanup of temporary files on startup.

Implementation of aging policies.

Periodic monitoring of server state, disk space, service response times, and so on (see "stored").

Issuing of alarms if necessary.

Database recovery as necessary (see "Message Store Startup and Recovery").

The stored utility automatically performs cleanup and expiration operations once a day at 11 PM. You can choose to run additional cleanup and expiration operations.

Table 15-12 lists some of the stored options. Some common usage examples follow the table. For detailed syntax and usage requirements, see the Messaging Server Reference Manual.

Table 15-12 stored Options
Option	Description
-d	OBSOLETE. Use start-msg store to start up stored which will run as daemon, performing system checks and activating alarms, deadlock detection, and database repair.
-t	Checks the status of stored. The return code of this command indicates the status.
-v	Verbose output.
-v -v	More verbose output.

If you want to change the time of the automatic cleanup and expiration operations, use the configutil utility as follows:

Occasionally, you might need to restart the stored utility; for example, if the mailbox list database becomes corrupted. To restart stored on UNIX, use the following commands at the command line:

If any server daemon crashes, you must stop all daemons and restart all daemons including stored.

Backing Up and Restoring the Message Store

Message store backup and restore is one of the most common and important administrative tasks. It consists of backing up all the messages and folders on a message store. You must implement a backup and restore policy for your message store to ensure that data is not lost if problems such as the following occur:

System crashes

Hardware failure

Accidental deletion of messages or mailboxes

Problems when reinstalling or upgrading a system

Natural disasters (for example, earthquakes, fire, hurricanes)

Migrating users

You can do message store back up and restore using command-line utilities imsbackup and imsrestore, or the integrated solution that uses Legato Networker®.

Messaging Server provides a single-copy backup procedure. Regardless of how many user folders contain a particular message, during backup, the message file is backed up only once using the first message file found. The second message copy is backed up as a link to the name of the first message file, and so on. imsbackup maintains a hash table of all messages using the device and inode of the message files as the index. This method does have implications when restoring data, however. For more information, see Considerations for Partial Restore.

Creating a Mailbox Backup Policy

To Create Backup Groups

Messaging Server Backup and Restore Utilities

Considerations for Partial Restore

To Use Legato Networker

Creating a Mailbox Backup Policy

Peak Business Loads

Full and Incremental Backups

Parallel or Serial Backups

Peak Business Loads

You need to take into account peak business loads when scheduling backups for your system as this can reduce system load during peak hours. For example, backups are probably best scheduled for early morning hours such as 2:00 AM.

Full and Incremental Backups

Incremental backups will scan the store for changed data and back up only what has changed. Full backups will back up the entire message store. You need to determine how often the system should perform full as opposed to incremental backups. You’ll probably want to perform incremental backups as a daily maintenance procedure and full backups once a week.

Parallel or Serial Backups

When user data is stored on multiple disks, you can back up user groups in parallel if you wish. Depending on system resources, parallel backups can speed up the overall backup procedure. However, you might want to use serial backups if you want to reduce backup impact on the server’s performance. Whether to use parallel or serial backups can depend on many factors, including system load, hardware configuration, how many tape drives are available, and so on.

To Create Backup Groups

A backup group is an arbitrary set of user mailboxes defined by regular expressions. By organizing user mailboxes into backup groups, you can define more flexible backup management.

For example, you could create three backup groups, the first containing user IDs starting with the letters A through L, the second with users whose user IDs begin with M-Z, and the third with users whose user IDs begin with a number. Administrators could use these backup groups to backup mailboxes in parallel, or perhaps only certain groups on one day and other groups on another.

These are arbitrary virtual groups of mail users. They do not precisely map to the message store directory (Figure 15-1), although it may look like it.

They are defined by the administrator using UNIX regular expressions.

The regular expressions are defined in the configuration file
msg_svr_base/config/backup-groups.conf

When backup groups are referenced in imsbackup and imsrestore, they use the path format: /partition_name/backup_group

Using the example described in the paragraph above, the following definitions would be used to create the three backup groups:

You can now scope imsbackup and imsrestore at several levels.You can backup/restore the whole message store using the backup command:

Pre-defined Backup Group

Messaging Server includes one predefined backup group that is available without creating the backup-groups configuration file. This group is called user; it includes all users. For example, the following will backup all users on the primary partition:

Messaging Server Backup and Restore Utilities

To back up and restore your data, Messaging Server provides the imsbackup and imsrestore utilities. Note that the imsbackup and imsrestore utilities do not have the advanced features found in general purpose tools like Legato Networker. For example, the utilities have only very limited support for tape auto-changers, and they cannot write a single store to multiple concurrent devices. Comprehensive backup will be achieved via plug-ins to generalized tools like Legato Networker. For more information about using Legato Networker, see To Use Legato Networker.

The imsbackup Utility

With imsbackup, you can write selected contents of the message store to any serial device, including magnetic tape, a UNIX pipe, or a plain file. The backup or selected parts of the backup may later be recovered by using the imsrestore utility. The output of imsbackup can be piped to imsrestore.

This example backs up all the mailboxes of all the users defined in the backup group groupA to backupfile (see "To Create Backup Groups"):

This command uses the default blocking factor of 20. For a complete syntax description of the imsbackup command, see the Messaging Server Reference Manual.

The imsrestore Utility

To restore messages from the backup device, use the imsrestore command. For example, the following command restores messages for user1 from the file backupfile.

For a complete syntax description of the imsbackup command, see the Messaging Server Reference Manual.

Considerations for Partial Restore

The message store uses a single-copy message system. That is, only a single copy of any message is saved in the store as a single file. Any other instances of that message (like when a message is sent to multiple mailboxes) are stored as links to that copy. Because of this, there are implications when restoring messages. For example:

Full Restore. During a full restore, linked messages will still point to the same inode as the message file to which they are linked.

Partial Backup/Restore. During a partial backup and partial restore, however, the single-copy characteristic of the message store might not be preserved.

The following examples demonstrate what happens to a message used by multiple users when a partial restore is performed. Assume there are three messages, all the same, belonging to three users A, B, and C, as follows:

Example 1. In the first example, the system performs a partial backup and full restore procedure as follows:

Back up mailboxes for users B and C.

Delete mailboxes of users B and C.

Restore the backup data from step 1.

In this example, B/INBOX/1 and C/INBOX/1 are assigned a new inode number and the message data is written to a new place on the disk. Only one message is restored; the second message is a hard link to the first message.

Example 2. In this example, the system performs a full backup and a partial restore as follows:

Perform full backup.

Delete mailboxes for user A.

Restore mailboxes for user A.

Example 3. In this example, partial restore might require more than one attempt:

Perform full backup.

B/INBOX/1 and C/INBOX/1 are backed up as links to A/INBOX/1.

Delete mailboxes for users A and B.

Restore mailboxes for user B.

The restore utilities ask the administrator to restore A/INBOX first.

Restore mailboxes for users A and B.

Delete mailboxes for user A (optional).



Note	If you want to ensure that all messages are restored for a partial restore, you can run the imsbackup command with the -i option. The -i option backs up every message multiple times if necessary. If the backup device is seekable (example: a drive or tape), imsrestore seeks to the position containing A/INBOX/1 and restore it as B/INBOX/1. If the backup device is non-seekable example: a UNIX pipe), imsrestore logs the object ID and the ID of the depending (linked) object to a file, and the administrator must invoke imsrestore again with the -r option to restore the missing message references.

To Use Legato Networker

Messaging Server includes a backup API that provides an interface with third-party backup tools, such as Legato Networker. The physical message store structure and data format are encapsulated within the backup API. The backup API interacts directly with the message store. It presents a logical view of the message store to the backup service. The backup service uses the conceptual representation of the message store to store and retrieve the backup objects.

Messaging Server provides an Application Specific Module (ASM) that can be invoked by the Legato Networker’s save and recover commands to back up and restore the message store data. The ASM then invokes the Messaging Server imsbackup and imsrestore utilities.

Backing Up Data Using Legato Networker

To perform backups of the Messaging Server message store using Legato Networker, you must perform the following preparatory steps before invoking the Legato interface:


Note	This section provides information about how to use Legato Networker with the Messaging Server message store. To understand the Legato Networker interface, see your Legato documentation.

Create a symbolic link from /usr/lib/nsr/imsasm to msg_srv_base/lib/msg/imsasm

From Sun or Legato, obtain a copy of the nsrfile binary and copy it to the following directory:

/usr/lib/nsr/nsrfile

If you want to back up users by groups, perform the following steps:

Create a backup group file as described in To Create Backup Groups.

To verify your configuration, run mkbackupdir.sh.

Look at the directory structure created by mkbackupdir.sh.The structure should look similar to that shown in Table 15-4.

Note that if you do not specify a backup-groups.conf file, the backup process will use the default backup group ALL for all users.

In the directory /nsr/res/, create a res file for your save group to invoke the mkbackupdir.sh script before the backup. See Table 15-4 for an example.



Note	Earlier versions of Legato Networker have a limitation of 64 characters for the save set name. If the name of this directory plus the logical name of the mailbox (for example, /primary/groupA/fred) is greater than 64 characters, then you must run mkbackupdir.sh -p. Therefore, you should use a short path name for the -p option of mkbackupdir.sh. For example the following command will create the backup image under the directory /backup: mkbackupdir.sh -p /backup Important: The backup directory must be writable by the message store owner (example: inetuser).

The example below shows a sample res file named IMS.res in the /nsr/res directory:

Create the Messaging Server save group if necessary.

Run nwadmin.

Select Customize | Group | Create.

Create a backup client using savepnpc as the backup command:

Set the save set to the directory created by mkbackupdir.

For a single session backup, use /backup

For parallel backups, use /backup/server/group

Be sure you’ve already created group as defined in To Create Backup Groups.

You must also set the parallelism to the number of backup sessions.

See Example. Creating A Backup Client in Networker:.

Select Group Control | Start to test your backup configuration.

To create a backup client in Networker. From nwadmin, select Client | Client Setup | Create

Restoring Data Using Legato Networker

To recover data, you can use the Legato Networker nwrecover interface or the recover command-line utility. The following example recovers user a1’s INBOX:

To Use a Third Party Backup Software (Besides Legato)

Messaging Server provides two message store backup solutions, the command line imsbackup and the Solstice Backup (Legato Networker). A large message store running a single imbackup to backup the entire message store can take a significant amount of time. The Legato solution supports concurrent backup sessions on multiple backup devices. Concurrent backup can shorten backup time dramatically (backups of 25GB of data per hour have been achieved).

If you are using another third party concurrent backup software (for example, Netbackup), you may use the following method to integrate your backup software with the Messaging Server.

Divide your users into groups (see "To Create Backup Groups") and create a backup-groups.conf file under the directory msg_svr_base/config/.



Note	This backup solution requires additional disk space. To backup all the groups concurrently, the disk space requirement is 2 times the message store size. If you do not have that much disk space, divide your users into smaller groups, and then backup a set of groups at a time. For example group1 - group5, group6 - group10. Remove the group data files after backup.

Run imsbackup to backup each group into files under a staging area.

The command is imsbackup -f <device> /<instance>/<group>

You can run multiple imsbackup processes simultaneously. For example:

# imsbackup -f- /primary/groupA > /bkdata/groupA &
# imsbackup -f- /primary/groupB > /bkdata/groupB &

. . .

imsbackup does not support large files, if the backup data is larger than 2 GB, you need to use the -f- option to write the data to stdout and then pipe the output to a file.

Use your third party backup software to backup the group data files in the staging area (in our example that is /bkdata).

To restore a user, identify the group filename of the user, restore that file from tape, and then use imsrestore to restore the user from the data file.

Note that imsrestore does not support large files. If the data file is larger than 2GB. Use the following command:

# cat /bkdata/groupA | imsrestore -f- /primary/groupA/andy

Monitoring User Access

Messaging Server provides the command, imsconnutil, which allows you to monitor user’s message store access via IMAP, POP and http. You can also determine the last log in and log out of users. This command works on a per message store basis and will not work across message stores.

This command requires root access by the system user (default: inetuser), and you must set the configuration variables local.imap.enableuserlist, local.http.enableuserlist, local.enablelastaccess to 1.

To list users currently logged on via IMAP or any web mail client, use the following command:

To list the last IMAP, POP, or Messenger Express access (log in and log out) of every user on the message store use:

The following command does two things: 1) it determines whether the specified user is currently logged on via IMAP or Messenger Express or any client that connects via mshttp (note that this does not work for POP because POP users generally do not stay connected), and 2) it lists the last time the users have logged on and off:

Note that a list of users can be input from a file, one user per line, using the following command:

You can also specify a particular service (imap or http) using the -s flag. For example, to list whether a particular user ID is logged onto IMAP or not, use the following command:

For a complete description of the imsconnutil syntax, refer to the Sun ONE Messaging Server Reference Guide.

$ ./imsconnutil -a -u soroork

UID IMAP last access HTTP last access POP last access
=========================================================================
soroork 08/Jul/2003:10:49:05 10/Jul/2003:14:55:52 ----NOT-RECORDED----

$ ./imsconnutil -c

IMAP
UID  TIME   AUTH          TO           FROM
===========================================================================
ed  17/Jun/2003:11:24:03  plain  172.58.73.45:193  129.157.12.73:2631
bill  17/Jun/2003:04:28:43  plain  172.58.73.45:193  129.158.16.34:2340
mia   17/Jun/2003:09:36:54  plain  172.58.73.45:193  192.18.184.103:3744
jay  17/Jun/2003:05:38:46  plain  172.58.73.45:193  129.159.18.123:3687
paul  17/Jun/2003:12:23:28  plaintext  172.58.73.45:193  192.18.194.83:2943
tony  17/Jun/2003:05:38:46  plain  172.58.73.45:193  129.152.18.123:3688
anil  17/Jun/2003:12:26:40  plaintext  172.58.73.45:193  192.18.164.17:1767
anil  17/Jun/2003:12:25:17  plaintext  172.58.73.45:193  129.150.17.34:3117
jack  17/Jun/2003:12:26:32  plaintext  172.58.73.45:193  129.150.17.34:3119
toni  17/Jun/2003:12:25:32  plaintext  172.58.73.45:193  192.18.148.17:1764

===========================================================================
10 users were logged in to imap.
Feature is not enabled for http.
------------------------------------------------------------------------------

Troubleshooting the Message Store

This section provides guidelines for actively maintaining your message store. In addition, this section describes other message store recovery procedures you can use if the message store becomes corrupted or unexpectedly shuts down. Note that the section on these additional message store recovery procedures is an extension of "Repairing Mailboxes and the Mailboxes Database".

Prior to reading this section, it is strongly recommended that you review this chapter as well as the command-line utility and configutil chapters in the Sun ONE Messaging Server Reference Manual. Topics covered in this section include:

"Standard Message Store Monitoring Procedures"

"Common Problems and Solutions"

"Message Store Startup and Recovery"

"Repairing Mailboxes and the Mailboxes Database"

Standard Message Store Monitoring Procedures

This section outlines standard monitoring procedures for the message store. These procedures are helpful for general message store checks, testing, and standard maintenance.

Check Hardware Space

A message store should have enough additional disk space and hardware resources. When the message store is near the maximum limit of disk space and hardware space, problems might occur within the message store.

Inadequate disk space is one of the most common causes of the mail server problems and failure. Without space to write to the message store, the mail server will fail. In addition, when the available disk space goes below a certain threshold, there will be problems related to message delivery, logging, and so forth. Disk space can be rapidly depleted when the clean up function of the stored process fails and deleted messages are not expunged from the message store.

Check Log Files

Check the log files to make sure the message store processes are running as configured. Messaging Server creates a separate set of log files for each of the major protocols, or services, it supports: SMTP, IMAP, POP, and HTTP. You can look at the log files through the Console or in directory msg_svr_base/log/. You should monitor the log files on a routine basis.

Be aware that logging can impact server performance. The more verbose the logging you specify, the more disk space your log files will occupy for a given amount of time. You should define effective but realistic log rotation, expiration, and backup policies for your server. For information about defining logging policies for your server, see Chapter 17, "Logging and Log Analysis".

Check User IMAP/POP Session

Messaging Server provides a feature called telemetry that can capture a user’s entire IMAP or POP session into a file. This feature is useful for debugging client problems. For example, if a user complains that their message access client is not working as expected, this feature can be used to trace the interaction between the access client and Messaging Server.

Messaging Server will create one file per session in that directory. Example output is shown below.

LOGIN redb 2003/11/26 13:03:21
>0.017>1 OK User logged in
<0.047<2 XSERVERINFO MANAGEACCOUNTURL MANAGELISTSURL MANAGEFILTERSURL
>0.003>* XSERVERINFO MANAGEACCOUNTURL {67}
http://redb@cuisine.blue.planet.com:800/bin/user/admin/bin/enduser MANAGELISTSURL NIL MANAGEFIL
TERSURL NIL
2 OK Completed
<0.046<3 select "INBOX"
>0.236>* FLAGS (\Answered lagged raft eleted \Seen $MDNSent Junk)
* OK [PERMANENTFLAGS (\Answered lagged raft eleted \Seen $MDNSent Junk \*)]
* 1538 EXISTS
* 0 RECENT
* OK [UNSEEN 23]
* OK [UIDVALIDITY 1046219200]
* OK [UIDNEXT 1968]
3 OK [READ-WRITE] Completed
<0.045<4 UID fetch 1:* (FLAGS)
>0.117>* 1 FETCH (FLAGS (\Seen) UID 330)
* 2 FETCH (FLAGS (\Seen) UID 331)
* 3 FETCH (FLAGS (\Seen) UID 332)
* 4 FETCH (FLAGS (\Seen) UID 333)
* 5 FETCH (FLAGS (\Seen) UID 334)
<etc>

Check stored Processes

The stored function performs a variety of important tasks such as deadlock and transaction operations of the message database, enforcing aging policies, and expunging and erasing messages stored on disk. If stored stops running, Messaging Server will eventually run into problems. If stored doesn’t start when start-msg is run, no other processes will start.

Check that the stored process is running. Run stored -t -v

Check for the log file build up in store_root/mboxlist.

Check for stored messages in the default log file msg_svr_base/log/default/default

Check that the time stamps of the following files (in directory msg_svr_base/config/) are updated whenever one of the following functions are attempted by the stored process:

Table 15-13 stored Operations
stored Operation	Function
stored.ckp	Touched when a database checkpoint was initiated. Stamped approximately every 1 minute.
stored.lcu	Touched at every database log cleanup. Time stamped approximately every 5 minutes.
stored.per	Touched at every spawn of peruser db write out. Time stamped once an hour.

For more information on the stored process, see "Using the stored Utility" and the stored utility in the Messaging Server Command-line Utilities chapter of the Messaging Server Reference Manual.

Check Database Log Files

Database log files refer to sleepycat transaction checkpointing log files (in directory store_root/mboxlist). If log files accumulate, then database checkpointing is not occurring. In general, there are two or three database log files during a single period of time. If there are more files, it could be a sign of a problem.

Check User Folders

If you want to check the user folders, you might run the command
reconstruct -r -n (recursive no fix) which will review any user folder and report errors. For more information on the reconstruct command, see "Repairing Mailboxes and the Mailboxes Database".

Check for Core Files

Core files only exist when processes have unexpectedly terminated. It is important to review these files, particularly when you see a problem in the message store. On Solaris, use coreadmin to configure core file location.

Message Store Startup and Recovery

Message store data consists of the messages, index data, and the message store database. While this data is fairly robust, on rare occasions there may be message store data problems in the system. These problems will be indicated in the default log file, and almost always will be fixed transparently. In rare cases an error message in the log file may indicate that you need to run the reconstruct utility. In addition, as a last resort, messages are protected by the backup and restore processes described in "Backing Up and Restoring the Message Store". This section will focus on the automatic startup and recovery process of stored.

The message store automates many recovery operations which were previously the responsibility of the administrator. These operations are performed by message store daemon stored during startup and include database snapshots and automatic fast recovery as necessary. stored thoroughly checks the message store’s database and automatically initiates repairs if it detects a problem.

stored also provides a comprehensive analysis of the state of the database via status messages to the default log, reporting on repairs done to the message store and automatic attempts to bring it into operation.

Automatic Startup and Recovery—Theory of Operations

The stored daemon starts before the other message store processes. It initializes and, if necessary, recovers the message store database. The message store database keeps folder, quota, subscription, and message flag information. The database is logging and transactional, so recovery is already built in. In addition, some database information is copied redundantly in the message index area for each folder.

Although the database is fairly robust, on the rare occasions that it breaks, in most cases stored recovers and repairs it transparently. However, whenever stored is restarted, you should check the default log files to make sure that additional administrative intervention is not required. Status messages in the log file will tell you to run reconstruct if the database requires further rebuilding.

Before opening the message store database, stored analyzes its integrity and sends status messages to the default log under the category of warning. Some messages will be useful to administrators and some messages will consists of coded data to be used for internal analysis. If stored detects any problems, it will attempt to fix the database and try starting it again.

When the database is opened, stored will signal that the rest of the services may start. If the automatic fixes failed, messages in the default log will specify what actions to take. See "Error Messages Signifying that reconstruct -m is Needed" for details.

In previous releases, stored could start a recovery process which would take a very long time leaving the administrator wondering if stored was “stuck.” This type of long recovery has been removed and stored should determine a final state in less than a minute. However, if stored needs to employ recovery techniques such as recovering from a snapshot, the process may take a few minutes.

After most recoveries, the database will usually be up to date and nothing else will be required. However, some recoveries will require a reconstruct -m in order to synchronize redundant data in the message store. Again, this will be stated in the default log, so it is important to monitor the default log after a startup. Even though the message store will seem to be up and running normally, it is important to run any requested operations such as reconstruct.

Another reason for reading the log file is to determine what caused damage to the database in the first place. Although stored is designed to bring up the message store regardless of any problem on the system, you will still want to try to ascertain cause of the database damage as this may be a sign of a larger hidden problem.

Error Messages Signifying that reconstruct -m is Needed

This section describes the type of error messages that require reconstruct -m to be run.

When the error message indicates mailbox error, run reconstruct <mailbox>. Example:

"Removing extra database logs. Run reconstruct -m soon after startup to resync redundant data"

"Recovering database from snapshot. Run reconstruct -m soon after startup to resync redundant data"

Database Snapshots

A snapshot is a hot backup of the database and is used by stored to restore a broken database transparently in couple minutes. This is much quicker than using reconstruct, which relies on the redundant information stored in other areas.

Message Store Database Snapshot—Theory of Operations

Snapshots of the database, located in the mboxlist directory, are taken automatically, by default, once every 24 hours. Snapshots are copied by default into a subdirectory of the store directory. By default, there are five snapshots kept at any given time: one live database, three snapshots, and one database/removed copy. The database/removed copy is newer and is an emergency copy of the database which is thrown into a subdirectory removed of the mboxlist database directory.

If the recovery process decides to remove the current database because it is determined to be bad, stored will move it into the removed directory if it can. This allows the database to be analyzed if desired.

The data move will only happen once a week. If there is already a copy of the database there, stored will not replace it every time the store comes up. It will only replace it if the data in the removed directory is older than a week. This is to prevent the original database which had the problem of being replaced too soon by successive startups.

To Specify Message Store Database Snapshot Interval and Location

There should be five times as much space for the database and snapshots combined. It is highly recommended that the administrator reconfigure snapshots to run on a separate disk, and that it is tuned to the system’s needs.

If stored detects a problem with the database on startup, the best snapshot will automatically be recovered. Three snapshot variables can set the following parameters: the location of the snapshot file, the interval for taking snapshots, number of snapshots saved. These configutil parameters are shown in Table 15-14.

Having a snapshot interval which is too small will result in a frequent burden to the system and a greater chance that a problem in the database will be copied as a snapshot. Having a snapshot interval too large can create a situation where the database will hold the state it had back when the snapshot was taken.

A snapshot interval of a day is recommended and a week or more of snapshots can be useful if a problem remains on the system for a number of days and you wish to go back to a period prior to point at which the problem existed.

stored monitors the database and is intelligent enough to refuse the latest snapshot if it suspects the database is not perfect. It will instead retrieve the latest most reliable snapshot. Despite the fact that a snapshot may be retrieved from a day ago, the system will use more up to date redundant data and override the older snapshot data, if available.

Thus, the ultimate role the snapshot plays is to get the system as close to up-to-date and ease the burden of the rest of the system trying to rebuild the data on the fly.

Repairing Mailboxes and the Mailboxes Database

The reconstruct utility rebuilds one or more mailboxes, or the master mailbox file, and repairs any inconsistencies. You can use this utility to recover from almost any form of data corruption in the mail store. Note that low-level database repair, such as completing transactions and rolling back incomplete transactions is performed by automatically on startup.

Table 15-15 lists the reconstruct options. For detailed syntax and usage requirements, see the Messaging Server Reference Manual.

Table 15-15 reconstruct Options
Option	Description
-e	Remove store.exp file upon reconstruct.
-i	Initialize store.idx file upon reconstruct.
-f	Forces reconstruct to perform a fix on the mailbox or mailboxes.
-m	Repairs and performs a consistency check of the mailboxes database. This option examines every mailbox it finds in the spool area, adding or removing entries from the mailbox’s database as appropriate. The utility prints a message to the standard output file whenever it adds or removes an entry from the database.
-n	Checks the message store only, without performing a fix on the mailbox or mailboxes. The -n option cannot be used by itself unless a mailbox name is provided. When a mailbox name is not provided, the -n option must be used with the -r option. The -r option may be combined with the -p option. For example, any of the following commands are valid: reconstruct -n user/dulcinea/INBOX reconstruct -n -r reconstruct -n -r -p primary reconstruct -n -r user/dulcinea/
-o	Checks for orphaned accounts. This option searches for inboxes in the current messaging server host which do not have corresponding entries in LDAP. For example, the -o option finds inboxes of owners who have been deleted from LDAP or moved to a different server host. For each orphaned account it finds, reconstruct writes the following command to the standard output: mboxutil-d user/userid/INBOX
-o -d filename	If -d filename is specified with the -o option, reconstruct opens the specified file and writes the mboxutil -d commands into that file. The file may then be turned into a script file to delete the orphaned accounts.
-p partition	Specifies a partition name; do not use a full path name. If this option is not specified, reconstruct defaults to all partitions.
-q	Fixes any inconsistencies in the quota subsystem, such as mailboxes with the wrong quota root or quota roots with the wrong quota usage reported. The -q option can be run while other server processes are running.
-r [mailbox]	Repairs and performs a consistency check of the partition area of the specified mailbox or mailboxes. The -r option also repairs all sub-mailboxes within the specified mailbox. If you specify -r with no mailbox argument, the utility repairs the spool areas of all mailboxes within the user partition directory.

To Rebuild Mailboxes

Accessing a mailbox returns one of the following errors: “System I/O error” or “Mailbox has an invalid format”.

Accessing a mailbox causes the server to crash.

Files have been added to or removed from the spool directory.

With the 5.0 release, reconstruct -r first runs a consistency check. It reports any consistencies and rebuilds only if it detects any problems. Consequently, performance of the reconstruct utility is improved with this release.

To rebuild the spool area for the mailboxes belonging to the user daphne, use the following command:

You must use this option with caution, however, because rebuilding the spool area for all mailboxes listed in the mailbox database can take a very long time for large message stores. (See reconstruct Performance.) A better method for failure recovery might be to use multiple disks for the store. If one disk goes down, the entire store does not. If a disk becomes corrupt, you need only rebuild a portion of the store by using the -p option as follows:

To rebuild mailboxes listed in the command-line argument only if they are in the primary partition:

If you want to force reconstruct to rebuild a folder without performing a consistency check, use the -f option. For example, the following command forces a reconstruct of the user folder daphne:

Checking and Repairing Mailboxes

One or more directories were removed from the store spool area, so the mailbox database entries also need to be removed.

One or more directories were restored to the store spool area, so the mailbox database entries also need to be added.

The stored -d option is unable to make the database consistent.

To Remove Orphan Accounts

To search for orphaned accounts (orphaned accounts are mailboxes that do not have corresponding entries in LDAP) use the following command:

Use the following command to create a file listing orphaned mailboxes that can be turned into a script file that deletes the orphaned mailboxes (file is named orphans.cmd):

reconstruct Performance

The time it takes reconstruct to perform an operation depends on the following factors:

The kind of operation being performed and the options chosen

Disk performance

The number of folders when running reconstruct -m

The number of messages when running reconstruct -r

The overall size of the message store

What other processes the system is running and how busy the system is

Whether or not there is ongoing POP, IMAP, HTTP, or SMTP activity

The reconstruct -r option performs an initial consistency check; this check improves reconstruct performance depending on how many folders must be rebuilt.

The following performance was found with a system with approximately 2400 users, a message store of 85GB, and concurrent POP, IMAP, or SMTP activity on the server:

reconstruct -m took about 1 hour

reconstruct -r -f took about 18 hours



Note	A reconstruct operation may take significantly less time if the server is not performing ongoing POP, IMAP, HTTP, or SMTP activity.

Common Problems and Solutions

"Command Using Wildcard Pattern Doesn’t Work"

"Unknown/invalid Partition"

"User Mailbox Directory Problems"

Command Using Wildcard Pattern Doesn’t Work

Some UNIX shells may require quotes around wildcard parameters and some will not. For example, the C shell tries to expand arguments containing wild cards (*, ?) as files and will fail if no match is found. These pattern matching arguments may need to be enclosed in quotes to be passed to commands like mboxutil.

will work in the Bourne shell, but will fail with tsch and the C shell. These shells would require the following:

If a command using a wildcard pattern doesn’t work, verify whether or not you need to use quotes around wildcards for that shell.

Unknown/invalid Partition

A user can get the message “Unknown/invalid partition” in Messenger Express if their mailbox was moved to a new partition which was just created and Messaging Server was not refreshed or restarted. This problem only occurs on new partitions. If you now add additional user mailboxes to this new partition, you will not have to do a refresh/restart of Messaging Server.

User Mailbox Directory Problems

A user mailbox problem exists when the damage to the message store is limited to a small number of users and there is no global damage to the system. The following guidelines suggest a process for identifying, analyzing, and resolving a user mailbox directory problem:

Review the log files, the error messages, or any unusual behavior that the user observes.

To keep debugging information and history, copy the entire store_root/mboxlist/ user directory to another location outside the message store.

To find the user folder that might be causing the problem, run the command reconstruct -r -n. If you are unable to find the folder using reconstruct, the folder might not exist in the folder.db.

If you are unable to find the folder using the reconstruct -r -n command, use the hashdir command to determine the location. For more information on hashdir, see "The hashdir Utility" and the hashdir utility in the Messaging Server Command-line Utilities chapter of the Messaging Server Reference Manual.

Once you find the folder, examine the files, check permissions, and verify the proper file sizes.

Use reconstruct -r (without the -n option) to rebuild the mailbox.

If reconstruct does not detect a problem that you observe, you can force the reconstruction of your mail folders by using the reconstruct -r -f command.

If the folder does not exist in the mboxlist directory (store_root/mboxlist), but exists in the partition directory store_root/partition), there might be a global inconsistency. In this case, you should run the reconstruct -m command.

If the previous steps do not work, you can remove the store.idx file and run the reconstruct command again.



Caution	You should only remove the store.idx file if you are sure there is a problem in the file that the reconstruct command is unable to find.

If the issue is limited to a problematic message, you should copy the message file to another location outside of the message store and run the command reconstruct -r on the mailbox/ directory.

If you determine the folder exists on the disk (store_root/partition/ directory), but is apparently not in the database (store_root/mboxlist/ directory), run the command reconstruct -m to ensure message store consistency.

Store Daemon Not Starting

# msg_svr_base/sbin/start-msg

msg_svr_base: Starting STORE daemon ...Fatal error: Cannot find group in name service

This indicates that the UNIX group configured in local.servergid cannot be found. Stored and others need to set their gid to that group. Sometimes the group defined by local.servergid gets inadvertently deleted. In this case, create the deleted group, add inetuser to the group, change ownership of the instance_root and its files to inetuser and the group.

Parameter	Description
local.store.snapshotpath	Location of message store database snapshot files. Either existing absolute path or path relative to the store directory. Default: dbdata/snapshots
local.store.snapshotinterval	Minutes between snapshots. Valid values: 1 - 46080 Default: 1440 (1440 minutes = 1 day)
local.store.snapshotdirs	Number of different snapshots kept. Valid values: 2 -367 Default: 3