Sun Java System Messaging Server 6 2005Q4 Administration Guide

Chapter 18 Managing the Message Store

This chapter describes the message store and its administration interface. This chapter contains the following sections:

Overview

The message store contains the user mailboxes for a particular Messaging Server instance. The size of the message store increases as the number of mailboxes, folders, and log files increase. You can control the size of the store by specifying limits on the size of mailboxes (disk quotas), by specifying limits on the total number of messages allowed, and by setting aging policies for messages in the store.

As you add more users to your system, your disk storage requirements increase. Depending on the number of users your server supports, the message store might require one physical disk or multiple physical disks. There are two ways to integrate this additional disk space into your system. The easiest way is to add additional message store partitions (see Configuring Message Store Partitions)

Likewise, if you are supporting multiple hosted domains, you might want to dedicate a server instance to a single, large domain. With this configuration, you can designate a store administrator for a particular domain. You can also expand the message store by adding more partitions.

To manage the message store, Messaging Server provides a set of command-line utilities in addition to the Sun Java System Console interface. Table 18–1 describes these command-line utilities. For information about using these utilities, see Performing Message Store Maintenance Procedures and the Sun Java System Messaging Server 6 2005Q4 Administration Reference.

Table 18–1 Message Store Command-line Utilities

Utility  

Description  

configutil

Sets and modifies configuration parameters for the store. 

deliver

Delivers mail directly to the message store accessible by IMAP or POP mail clients. 

hashdir

Identifies the directory that contains the message store for a particular user. 

imsconnutil

Monitors user access of the message store. 

imexpire

Automatically removes messages from the message store based on administrator-specified criteria like age. 

iminitquota

Reinitializes the quota limit from the LDAP directory and recalculates the disk space being used. 

imsasm

Handles the saving and recovering of user mailboxes. 

imsbackup

Backs up stored messages. 

imsexport

Exports Messaging Server mailboxes into UNIX /var/mail format folders.

imsrestore

Restores messages that have been backed up. 

imscripter

The IMAP server protocol scripting tool. Executes a command or sequence of commands. 

mboxutil

Lists, creates, deletes, renames, or moves mailboxes; reports quota usage. 

mkbackupdir

Creates and synchronizes the backup directory with the information in the message store. 

MoveUser

Moves a user’s account from one messaging server to another. 

imquotacheck

Calculates the total mailbox size for each user in the message store and compares the size with their assigned quota. Localized versions of imquotacheck notification incorrectly convert the % and the $ signs. To correct the encoding, replace every $ with \24 and replace every % with \25 in the message file.

readership

Collects readership information on shared IMAP folders. 

reconstruct

Reconstructs mailboxes that have been damaged or corrupted. 

stored

Performs background and daily tasks, expunges, and erases messages stored on disk. 

Message Store Directory Layout

Figure 18–1 shows the message store directory layout for a server instance. The message store is designed to provide fast access to mailbox contents. The store directories are described in Table 18–2.

Figure 18–1 Message Store Directory Layout

Graphic shows Message Store directory layout.

The message store consists of a number of mailbox databases and the user mailboxes. The mailbox databases consists of information about users, mailboxes, partitions, quotas and other message store related data. The user mailboxes contain the user’s messages and folders. Mailboxes are stored in a message store partition, an area on a disk partition specifically devoted to storing the message store. See Configuring Message Store Partitions for details. Message store partitions are not the same as disk partitions, though for ease of maintenance, we recommend having one disk partition for each message store partition.

Mailboxes such as INBOX are located in the store_root. For example, a sample directory path might be:

store_root/partition/primary/=user/53/53/=mack1

The table below describe the message store directory.

Table 18–2 Message Store Directory Description

Location  

Content/Description  

msg_svr_base

Default: /opt/SUNWmsgsr

The directory on the Messaging Server machine that holds the server program, configuration, maintenance, and information files. 

store_root

msg_svr_base/data/store

Top-level directory of the message store. Contains the mboxlist, user, and partition subdirectories.

./store.expirerule

Contains the automatic message removal rules (expire rules). This optional file can be at different locations. See To Set the Automatic Message Removal (Expire and Purge) Feature

store_root/dbdata/snapshots

Message store database backup snapshots. 

store_root/mboxlist/

Contains mailbox database, database (Berkeley DB) that stores information about the mailboxes and quota information. 

folder.db contains information about mailboxes, including the name of the partition where the mailbox is stored, the ACL, and a copy of some of the information in store.idx. There is one entry in folder.db per mailbox

quota.db contains information about quotas and quota usage. There is one entry in quota.db per user.

lright.db - an index for the folders by acl lookup rights.

peruser.db contains information about per-user flags. The flags indicate whether a particular user has seen or deleted a message.

subscr.db contains information about user subscriptions.

store_root/session/

Contains active message store process information. 

store_root/user/

Not used. 

store_root/partition/

Contains the message store partitions. A default primary partition is created. Place any other partitions you define in this directory.

store_root/partition/primary/=user/

Contains all the user mailboxes in the subdirectory of the partition. The mailboxes are stored in a hash structure for fast searching. To find the directory that contains a particular user’s mailbox, use the hashdir utility.

.../=user/hashdir/hashdir/userid/

The top-level mail folder for the user whose ID is userid. This is the user’s INBOX. For the default domain, userid is uid. For hosted domains, userid is uid@domain. Incoming messages are delivered to this mail folder.

.../userid/folder

A user-defined mailbox on the messaging server. 

.../userid/store.idx

An index that provides the following information about mail stored in the /userid/ directory: number of messages, disk quota used by this mailbox, the time the mailbox was last appended, message flags, variable-length information for each message including the headers and the MIME structure, and the size of each message. The index also includes a backup copy of mboxlist information for each user and a backup copy of quota information for each user.

.../userid/store.usr

Contains a list of users who have accessed the folder. For each user listed, contains information about the last time the user accessed the folder, the list of messages the user has seen, and the list of messages the user has deleted. 

.../userid/store.sub

Contains information about user subscriptions. 

.../userid/store.exp

Contains a list of message files that have been expunged, but not removed from disk. This file appears only if there are expunged messages. 


.../userid/nn/
or
.../userid/folder/nn/

nn is a hash directory that contains messages in the format message_id.msg; nn can be a number from 00 to 99. message_id is also a number. Example: messages 1 through 99 are stored in the .../00 directory. The first message is 1.msg, the second is 2.msg, third 3.msg, and so on. Messages 100 through 199 are stored in the 01 directory; messages 9990 through 9999 are stored in the 99 directory; messages 10000 through 10099 are in the 00 directory, and so on.

How the Message Store Removes Messages

Messages are removed from the message store in three stages:

  1. Delete. A client sets a message flag to delete. At this point, the message is marked for removal, but client can still restore the message by removing the delete flag. If there is a second client, the deleted flag may not be recognized immediately by that second client. You can set the configutil parameter local.imap.immediateflagupdate to enable immediate flag update.

  2. Expunge. Messages are removed from the mailbox. Technically, they are removed from the message store index file, store.idx. The message itself is still on disk, but once messages are expunged, clients can no longer restore them.

    Expire is a special case of expunge. Messages that conform to a set of administrator-defined removal criteria such as message size, age and so forth, are expunged. See To Set the Automatic Message Removal (Expire and Purge) Feature

  3. Purge. The imexpire utility purges from the disk any messages that have been expunged at 11PM everyday by default. This can be configured with local.schedule.expire, which controls the message expire schedule, and store.cleanupage, which controls the purge grace period (period of time before which the message will not be purged). Note that this is different from the imsimta purge command which purges older versions of the MTA log files.

Specifying Administrator Access to the Store

Message store administrators can view and monitor user mailboxes and specify access control for the message store. Store administrators have proxy authentication privileges to any service (POP, IMAP, HTTP, or SMTP), which means they can authenticate to any service using the privileges of any user. These privileges allow store administrators to run certain utilities for managing the store. For example, using MoveUser, store administrators can move user accounts and mailboxes from one system to another.

This section discusses how to grant store privileges to the message store for your Messaging Server installation.


Note –

Other users might also have administrator privileges to the store. For example, some administrators may have these privileges.


You can perform administrator tasks as described in the following subsections:

ProcedureTo Add an Administrator Entry

Administrators can be added at the Console or by command line.

Steps
  1. From Console, open the Messaging Server you want to configure.

  2. Click the Configuration tab and select Message Store in the left pane.

  3. Click the Administrator tab.

    The tab contains a list of existing administrator IDs.

  4. Click the Add button beside the Administrator UID window.

  5. In the Administrator UID field, type the user ID of the administrator you want to add.

    The user ID you type must be known to the Sun Java System Directory Server.

  6. Click OK to add the administrator ID to the list displayed in the Administrator tab.

  7. Click Save in the Administrator tab to save the newly modified Administrator list.

    Command Line:To add an administrator entry at the command line:

    configutil -o store.admins -v "adminlist"

    where adminlist is a space-separated list of administrator IDs. If you specify more than one administrator, you must enclose the list in quotes. In addition, the administrator must be a member of the Service Administrator Group (in the LDAP user entry: memberOf: cn=Service Administrators,ou=Groups,o=usergroup).

ProcedureTo Modify an Administrator Entry

This section explains how to modify an existing entry in the message store Administrator UID list at the Console.

Steps
  1. Click the Administrator tab.

  2. Click the Edit button beside the Administrator UID window.

  3. Enter your changes to the Administrator UID field.

  4. Click OK to submit your changes and dismiss the Edit Administrator window.

  5. Click Save in the Administrator tab to submit and preserve the modified Administrator list.

    Command Line

    To modify an existing entry in the message store Administrator UID list at the command line:


    configutil -o store.admins -v "adminlist"

ProcedureTo Delete an Administrator Entry

Administrators can be deleted at the Console or by command line.

Steps
  1. Click the Administrator tab.

  2. Select an item in the Administrator UID list.

  3. Click Delete to delete the item.

  4. Click Save to submit and preserve your changes to the Administrator list.

    Command Line.To delete store administrators at the command line, you can edit the administrator list as follows:


    configutil -o store.admins -v "adminlist"

To Protect Mailboxes from Deletion or Renaming Except by an Administrator

You may wish to protect some mailboxes from deletion or modification except by the Administrator. The following procedures describes how to do this. If someone other than an Administrator attempts to delete, modify or rename a protected mailbox, the error message mailbox is pinned is displayed.

Set the local.store.pin configutil variable. using the following format:


configutil -o local.store.pin -v "mailbox1"%"mailbox2"%"mailbox 3"

where mailbox1, mailbox2, and mailbox 3 are the mailboxes to be protected (note that spaces can be used in mailbox names), and % is the separator between each mailbox.

About Shared Folders

A group or shared folder is like any other mail folder except that other users and groups can read, delete, or add messages to it depending on the permissions given. Messages can be added to shared folders by normal drag and drop, by Sieve filters, or by sending messages directly using the form: uid+folder@domain. For example:

carol.fanning+crafts_club@siroe.com

Shared folders are useful for starting, sharing, and archiving an ongoing email conversation on a particular topic. For example, a group of software developers can create a shared folder for discussing development of a particular project called mosaic_voices. When a message is sent or dropped into the folder mosaic_voices, anyone who has permissions to the shared folder (permissions can added by individual addresses or by group addresses) can open this mailbox and read the message.

User’s shared folders are stored in their mail folder called Shared Folders/Users. Users create and access shared folders in this mail folder. An example is shown in About Shared Folders.

Figure 18–2 Mail Client Example of Ed’s Shared Mail Folder List

Graphic shows example of client shared mail folder list.

There are two kinds of shared folders:

Normally shared folders are only available to users on a particular message store. Messaging Server, however, allows you to create special shared folders that can be accessed across multiple message stores. These are called distributed shared folders. See To Set Up Distributed Shared Folders for details.

Shared Folder Tasks

This section describes the shared folder administrator tasks:

ProcedureTo Create a Public Folder

Public folders must be created by system administrators because they require access to the LDAP database as well as the readership command.

Steps
  1. Create an LDAP user entry called public that will act as a container for all public folders (see About Shared Folders).

    Example:


    dn: cn=public,ou=people,o=sesta.com,o=ISP
    objectClass: person
    objectClass: organizationalPerson
    objectClass: inetOrgPerson
    objectClass: inetUser
    objectClass: ipUser
    objectClass: inetMailUser
    objectClass: inetLocalMailRecipient
    objectClass: nsManagedPerson
    objectClass: userPresenceProfile
    cn: public
    mail: public@sesta.com
    mailDeliveryOption: mailbox
    mailHost: manatee.siroe.com
    uid: public
    inetUserStatus: active
    mailUserStatus: active
    mailQuota: -1
    mailMsgQuota: 100
                      
  2. Create folders within the public account by using the mboxutil command line utility.

    For example, create a public folder called gardening:


    mboxutil -c user/public/gardening
  3. Specify the users and their access rights to the shared folder.

    Use the readership command to specify users and their access rights. For example the following command gives everyone at sesta.com lookup, read, and posting access to the public folder gardening:

    readership -s user/public/gardening anyone@sesta.com lrp

    For detailed instructions on how to user readership, see To Set or Change a Shared Folder’s Access Control Rights

To Add Shared Folders with an Email Group

Shared folders are typically created by adding users to a shared folder list with Communications Express, or by creating public shared folders as described earlier. Sometimes, however, users may wish to add an email group (mail distribution list) to a shared folder list so that everyone in the group will have access to the shared folder. For example, a group called tennis@sesta.com has 25 members and the members have decided that they would like to create a shared folder to store all email going to this group address.

ProcedureTo Adding an Email Group to a Shared Folder

Adding an email group to a shared folder requires System Administrator privileges.

Steps
  1. Create a folder. (If this has already been done, then skip this step. )

    Typically this should be done by one of the members of the group. If it’s not, you can create it for them using the following command:

    mboxutil -c user/gregk/gardening

    gregk is the uid of the shared folder owner. gardening is the name of the shared folder.

  2. Add the attribute-value pair aclGroupAddr group_name@domain to the user entry of every member who will have access to the group shared folder.

    Using the example above, add the following attribute-value pair to each user entry receiving access to the shared folder:

    aclGroupAddr: tennis@sesta.com

    Note that members will already have this attribute if the group was created dynamically using the memberURL attribute in the group entry. URL value for this attribute would look like this:


    memberURL: ldap:///o=sesta.com??sub?(&(aclGroupAddr=tennis@sesta.com)
    (objectclass=inetmailuser))

    (The sample entry line has been wrapped for typographic reasons. The actual entry should appear on one physical line.)

  3. Specify the group and the access rights to the shared folder.

    Use the readership command to do this. Using the example above the following command gives members of tennis@sesta.com lookup, read, and posting access to the public folder gardening:

    readership -s user/gregk/tennis tennis@sesta.com lrp

    For detailed instructions on how to user readership, see To Set or Change a Shared Folder’s Access Control Rights

To Set or Change a Shared Folder’s Access Control Rights

Users can set or change the access control for a shared folder using the Communications Express interface. Administrators can set or change the access control for a shared folder using the readership command line utility. The command has the following form:

readership -s foldername identifier rights_chars

where foldername is the name of the public folder for which you are setting rights, identifier is the person or group to whom you are assigning the rights, and rights_chars are the rights you are assigning. For the meaning of each character, see Table 18–3.


Note –

anyone is a special identifier. The access rights for anyone apply to all users. Similarly, the access rights for anyone@domain apply to all users in the same domain.


Table 18–3 ACL Rights Characters

Character  

Description  

l

lookup– User can see and subscribe to the shared folder. (IMAP commands allowed: LIST and LSUB)

r

read– Users can read the shared folder. (IMAP commands allowed: SELECT, CHECK, FETCH, PARTIAL, SEARCH, COPY from the folder)

s

seen– Directs the system to keep seen information across sessions. (Set IMAP STORE SEEN flag)

w

write– Users can mark as read, and delete messages. (Set IMAP STORE flags, other than SEEN and DELETED)

i

insert– Users can copy and move email from one folder to another. (IMAP commands allowed: APPEND, COPY into folder)

p

post– Users can send mail to the shared folder email address. (No IMAP command needed) 

c

create– Users can create new sub-folders. (IMAP command allowed: CREATE)

d

delete– Users can delete entries from the shared folder. (IMAP commands allowed: EXPUNGE, set STORE DELETED flag)

a

administer– Users have administrative privileges. (IMAP command allowed: SETACL)

Examples

If you wish everyone at the sesta domain to have lookup, read and email marking (but not posting) access to the public folder called golftournament, issue the command as follows:

readership -s User/public/golftournament anyone@sesta lwr

To assign the same access to everyone on the message store issue the following:

readership -s User/public/golftournament anyone lwr

To assign lookup, read, email marking and posting rights to a group, issue the command as follows:

readership -s User/public/golftournament group=golf@sesta.com lwrp

If you want to assign administrator and posting rights for this folder to an individual, jdoe, issue the command as follows:

readership -s User/public/golftournament jdoe@sesta.com lwrpa

To deny an individual or group access to a public folder, prefix the userid with a dash. For example, to deny lookup, read and write rights to jsmith, issue the command as follows:

readership -s User/public/golftournament -jsmith@sesta.com lwr

To deny an individual or group an access right, prefix the ACL rights character with a dash. For example, to deny posting rights to jsmith, issue the command as follows:

readership -s User/public/golftournament jsmith@sesta.com -p


Note –

Posting messages to a shared folder using the uid+folder@domain address requires that the p (post) access right be used with the readership command. See To Set or Change a Shared Folder’s Access Control Rights


To Enable or Disable Listing of Shared Folders

The server will or will not return shared folders when responding to a LIST command depending on the setting in the configuration option local.store.sharedfolders. Setting the option to off disables it. The setting is enabled by default (set to on).

SELECT and LSUB commands are not affected by this option. The LSUB command returns every subscribed folder, including shared folders. Users can SELECT the shared folders they own or are subscribed to.

To Set Up Distributed Shared Folders

Normally shared folders are only available to users on a particular message store. Messaging Server, however, allows you to create distributed shared folders that can be accessed across multiple message stores. That is, access rights to distributed shared folders can be granted to any users within the group of message stores. Note, however, that web mail clients (HTTP access clients like Messenger Express) do not support remote shared folders access. Users can list and subscribe to the folders, but they can’t view or alter the contents.

Distributed shared folders require the following:

The remote message stores (that is the message stores that do not hold the shared folder) must be configured as proxy servers by setting the configuration variables listed in Table 18–4.

Table 18–4 Variables for Configuring Distributed Shared Folders

Name  

Value  

Data Format  

local.service.proxy.serverlist

message store server list 

space-separated strings 

local.service.proxy.admin

default store admin login name 

string 

local.service.proxy.adminpass

default store admin password 

string 

local.service.proxy.admin.hostname

store admin login name for a specific host 

string 

local.service.proxy.adminpass.hostname

store admin password for a specific host 

string 

Setting Up Distributed Shared Folders—Example

Figure 18–3 shows a disturbed folder example of three message store servers called StoreServer1, StoreServer2, and StoreServer3.

Figure 18–3 Distributed Shared Folders—Example

Graphic shows example of distributed shared folders.

These servers are connected to each other as peer proxy message stores by setting the variables shown in Table 18–4. Each server has a private shared folder—golf (owned by Han), tennis (owned by Kat), and hurling (owned by Luke). In addition there are two public shared folders called press_releases and Announcements. Users on any of the three servers can access any of these three shared folders. Figure 18–2shows Ed's shared folder list. Below is an example of the ACLs for each server in this configuration.


$ StoreServer1 :> readership -l
Ed: user/Han/golf 
Ian: user/Han/golf 
anyone: user/public/press_releases

            

$ StoreServer2 :> readership -l
Jan: user/Kat/tennis
Ann: user/Kat/tennis
anyone: user/public+Announcements user/public+press_releases

            

$ StoreServer3 :> readership -l
Tuck: user/Ian/hurling
Ed: user/Ian/hurling 
Jac: user/Ian/hurling 
anyone: user/public/Announcements

            

To Monitor and Maintain Shared Folder Data

The readership command line utility allows you to monitor and maintain shared folder data which is held in the folder.db, peruser.db, and lright.db files. folder.db has a record for each folder that holds a copy of the ACLs. The peruser.db has an entry per user and mailbox that lists the various flags settings and the last date the user accessed any folders. The lright.db has a list of all the users and the shared folders for which they have lookup rights.

The readership command line utility takes the following options:

Table 18–5 readership Options

Options  

Description  

-d days

Returns a report, per shared folder, of the number of users who have selected the folder within the specified days. 

-p months

Removes data from the peruser.db for those users who have not selected their shared folders within the specified months.

-l

List the data in lright.db.

-s folder_identifier_rights

Set access rights for the specified folder. This updates the lright.db as well as the folder.db.

Using the various options, you can perform the following functions:

To Monitor Shared Folder Usage

To find out how many users are actively accessing shared folders, issue the command:

readership -d days

where days is the number of days to check. Note that this option returns the number of active users, not a list of the active users.

Example: To find out the number of users who have selected shared folders within the last 30 days, issue the following command:

readership -d 30

To List Users and Their Shared Folders

To list users and the shared folders to which they have access, issue the command:

readership -l

Example output:

$ readership -l
group=lee-staff@siroe.com: user/user2/lee-staff
richb: user/golf user/user10/Drafts user/user2/lee-staff user/user10/Trash
han1: user/public+hurling@siroe.com user/golf
gregk: user/public+hurling@siroe.com user/heaving user/tennis

To Remove Inactive Users

If you want to remove inactive users (those who have not accessed shared folders in a specified time period) issue the command:

readership -p months

where months is the number of months to check for.

Example: Remove users who have not accessed shared folders for the past six months:

readership -p 6

To Set Access Rights

You can assign access rights to a new public folder, or change access rights on a current public folder.

For an example of how to set access rights with this command, see To Set or Change a Shared Folder’s Access Control Rights

About Message Store Quotas

A message store quota is a way of setting a limit or quota for how much disk space or how many messages can be used by users or domains. This section contains information about the following:

For further information, see To Monitor Quota Limits

User Quotas

You can specify user quotas by disk space or by number of messages. Disk space quotas specify, in bytes, the amount of disk space for each user. Disk quotas apply to the total size of all the user’s messages, regardless of how many mail folders the user has or the total number of user messages. Message quotas allow you to limit the number of messages stored in a user’s mailbox.

Quota information is stored in user LDAP attributes (Table 18–6) and configutil variables (Table 18–7). (See the Sun Java System Communications Services 6 2005Q4 Schema Reference for latest and complete information.) In addition to setting the quota itself, Messaging Server allows you to control the following features:

Domain Quotas

As for users, quotas can also be set for domains by either number of bytes or number of messages. This quota is for all the cumulative bytes or messages of all the users in a particular domain.

Exceptions for Telephony Application Servers

To support unified messaging requirements, Messaging Server provides the ability to override quota limitations imposed by the message store. This guarantees the delivery of messages that have been accepted by certain agents, namely telephony application servers (TAS). Messages accepted by a TAS can be routed through a special MTA channel that ensures the message is delivered to the store regardless of quota limits. For more information about configuring the TAS channel, see Chapter 12, Configuring Channel Definitions.

Message Store Quota Attributes and Parameters

This section contains the message store quota attributes and parameters. For detailed information on these attributes and parameters, refer to the Sun Java System Communications Services 6 2005Q4 Schema Reference.

Table 18–6 Message Store Quota Attributes

Attribute  

Description  

mailQuota

Bytes of disk space allowed for the user’s mailbox. Special values:  

0 - No space allowed for user’s mailbox. 

–1 - No limit on space usage allowed.  

-2 - Use system default quota. (configutilparameter store.defaultmailboxquota)

mailMsgQuota

Maximum number of messages permitted for a user. This is a cumulative count for all folders in the store. Special values:  

0 - No message allowed for user’s mailbox. 

-1 - No limit on number of messages allowed.  

-2 - Use system default quota. (configutil parameter store.defaultmessage.quota.)

mailUserStatus

Status of the mail user. Can be one of the following values: 

active - mail is processed as normal. Default is active.

inactive - user’s mail account is inactive. A transient failure is returned.

deleted - Account marked deleted and ready for purge. Permanent failure returned. Access to mailbox blocked.

hold - Mail sent to the hold queue and access to the mailbox is disallowed

overquota - The MTA will not deliver mail to a mailbox with this status. This is the status set when the configutil parameter, store.overquotastatus is on.

mailDomainDiskQuota

Bytes of disk space allowed for the cumulative count of all the mailboxes in a domain. A value of -1 means no limit on space usage. (Default) To enforce the domain disk quota run the command: imquotacheck -f -d domain

mailDomainMsgQuota

Maximum number of messages permitted for a domain, that is, the total count for all mailboxes in the store. A value of -1 means no limit. (Default). To enforce the domain message quota run the command: imquotacheck -f -d domain

mailDomainStatus

Status of the mail domain. Values and default are the same as mailUserStatus.

Table 18–7 Message Store configutil parameters

Parameter  

Description  

store.quotaenforcement

Enable quota enforcement When off, the quota database is still updated, but messages are always delivered. Default: On 

store.quotanotification

Enable quota notification. Default: On 

store.defaultmailboxquota

Store default quota by number of bytes. Default: -1 (unlimited) 

store.defaultmessagequota

Store default quota by number of messages. Numeric. Default: -1 (unlimited) 

store.quotaexceededmsg

Quota warning message. If none, notification is not sent. Default: None. 

store.quotaexceededmsginterval

Interval, in days, for sending overquota notification. Default: 7 

store.quotagraceperiod

Time, in hours, a mailbox has been overquota before messages to the mailbox will bounce back to the sender. Number of hours. Default: 120 

store.quotawarn

Quota warning threshold. Percentage of quota exceeded before clients are sent an over quota warning. Default: 90 

local.store.quotaoverdraft

Used to provide compatibility with systems that migrated from the Netscape Messaging Server. When ON, allow delivery of one message that puts disk usage over quota. After the user is over quota, messages are deferred or bounced, the quota warning message is sent, and the quota grace period timer starts. (The default is that the quota warning messages are sent when the message store reaches the threshold.) Default: Off, but is treated as on if store.overquotastatus is set, otherwise the user can never go over quota and the overquotastatus is never used.

local.store.overquotastatus

Enable quota enforcement before messages are enqueued in the MTA. This prevents the MTA queues from filling up. When set, and a user is not yet over quota, but an incoming message pushes the user over quota, then the message is delivered, but the mailuserstatus LDAP attribute is set to overquota so no more messages will be accepted by the MTA. Default: off 

Configuring Message Store Quotas

This section describes the following tasks:

ProcedureTo Specify a Default User Quota

A default quota that applies to users who do not have individual quotas set can be done by Console or command line.

Steps
  1. Click the Configuration tab and select Message Store in the left pane.

  2. Click the Quota tab.

  3. To specify a default user disk quota for the “Default user disk quota” field, select one of the following options:

    Unlimited. Select this option if you do not want to set a default disk quota.

    Size specification. Select this option if you want to restrict the default user disk quota to a specific size. In the field beside the button, type a number, and from the drop-down list, choose Kbytes or Mbytes.

  4. To specify a message number quota, in the “Default user message quota” box, type a number.

  5. Click Save.

  6. Set the Mbytes attribute to -1 in the user entries that use the default message store quota.

    Command Line

    To specify a default user quota at the command line:

    To specify a default user disk quota for total message size:

    configutil -o store.defaultmailboxquota -v [ -1 | number ]

    where -1 indicates no quota and number indicates a number of bytes.

    To specify a default user quota for total number of messages:

    configutil -o store.defaultmessagequota -v [ -1 | number ]

    where -1 indicates no quota and number indicates the number of messages.

    Set the mailQuota attribute to -2 in the user entries that use the default message store quota. See Table 18–6.

To Specify Individual User Quotas

Each user can have individualized quotas. To set user-specific quotas, set the mailQuota or mailmsgquota attributes in the user’s LDAP entry. (See Table 18–6.) To enforce the quota, set the configutil store.quotaenforcement to on.

To Specify Domain Quotas

You can set disk space quotas or message quotas for particular domains. These quotas are for the cumulative bytes or messages of all users in a particular domain. To set domain quotas, set the mailDomainDiskQuota or mailDomainMsgQuota attributes in the user’s LDAP entry (see Table 18–6) and run imquotacheck -f.

To Deploy Quota Notification

Quota notification is the process of sending users a warning message when they are getting close to their quota. Using this feature requires three procedures:

ProcedureTo Enable Quota Notification

Quota notification can be enabled by Console or command line.

Steps
  1. Click the Quota tab.

  2. Check the “Enable quota notification” box. To disable quota notification, uncheck this box.

  3. Define the quota warning messages. See To Define a Quota Warning Message.

  4. Click Save.

    Command Line

    To enable or disable quota notification at the command line:

    configutil -o store.quotanotification -v [ yes | no ]

    If the message is not set, no quota warning message is sent to the user. See the next section for an example of quota warning message format.

ProcedureTo Define a Quota Warning Message

Define the message that will be sent to users who are close to exceeding their disk quota as follows. Messages are sent to the user’s mailbox.

Steps
  1. Click the Quota tab.

  2. From the drop-down list, choose the language you want to use.

  3. Type the message you want to send in the message text field below the drop-down list.

  4. Click Save.

    Command Line

    To define a quota warning message at the command line:

    configutil -o store.quotaexceededmsg -v ’message

    The message must be in RFC 822 format. It must contain a header with at least a subject line, follow by $$, then the message body. ’$’ represents a new line. Depending on the shell that you are using, it might be necessary to append a \ before $ to escape the special meaning of $. ($ is often the escape character for the shell.) Example:

    configutil -o store.quotaexceededmsg -v ”Subject: WARNING: User quota exceeded$$User quota threshold exceeded - reduce space used.’

    In addition, there is support for the following variables:

    [ID] - userid

    [DISKUSAGE] - disk usage

    [NUMMSG] - number of messages

    [PERCENT] - store.quotawarn percentage

    [QUOTA] - mailquota attribute

    [MSGQUOTA] - mailmsgquota attribute

    Here’s an example, using these variables:

    configutil -o store.quotaexceededmsg -v ”Subject: Overquota Warning$$[ID],$$Your mailbox size has exceeded [PERCENT] of its alloted quota.$Disk Usage: [DISKUSAGE]$Number of Messages: [NUMMSG]$Mailquota: [QUOTA]$Message Quota: [MSGQUOTA]$$-Postmaster’

    To define how often the warning message is sent:

    configutil -o store.quotaexceededmsginterval -v number

    where number indicates a number of days. For example, 3 would mean the message is sent every 3 days.

ProcedureTo Specify a Quota Threshold

A quota threshold is a percentage of a quota that is exceeded before clients are sent a warning. When a user’s disk usage exceeds the specified threshold, the server sends a warning message to the user.


Note –

When local.store.quotaoverdraft=on email notifications are not triggered until the user’s disk usage exceeds 100% of the quota regardless of the threshold set with store.quotawarn.


For IMAP users whose clients support the IMAP ALERT mechanism, the message is displayed on the user’s screen each time the user selects a mailbox and a message is also written to the IMAP log.

Steps
  1. Click the Quota tab.

  2. In the “Quota warning threshold” field, enter a number for the warning threshold.

    This number represents a percentage of the allowed quota. For example, if you specify 90%, the user is warned after using 90% of the allowed disk quota. The default is 90%. To turn off this feature, enter 100%.

  3. Click Save.

    Command Line

    To specify a quota threshold at the command line:

    configutil -o store.quotawarn -v number

    where number indicates a percentage of the allowed quota.

To Enable or Disable Quota Enforcement

By default, users or domains can exceed their quotas with no effect except for receiving an over quota notification (if set). Quota enforcement locks the mailboxes from receiving further messages until the disk usage is reduced below the quota level.

ProcedureTo Enable Quota Enforcement

Quota enforcement can be enabled at the Console or by command line.

Steps
  1. Click the Quota tab.

  2. Check the “Enable quota enforcement” box. To disable quota enforcement, uncheck this box.

  3. Click Save.

    Command Line

    To enable or disable quota enforcement:


    configutil -o store.quotaenforcement -v [ on | off]

    Note that over quota messages are saved in the MTA queues and a notification is sent to the sender stating that their messages was not delivered but that a redelivery attempt will be made later. Delivery retries will continue until the grace period expires and all messages are sent back to the senders, or the disk usage falls below the quota and messages can be dequeued from the MTA and delivered to the message store. If you want to return messages that are over quota before they get to the message queues, use the following command line:


    configutil -o store.overquotastatus -v on

Enabling Quota Enforcement at the Domain Level

To enforce quotas for a particular domain, use the command:

imquotacheck -f -d domain

To enable for all domains exclude the -d option. When a domain exceeds its quota, the maildomainstatus attribute is set to overquota, which halts all delivery to this domain. If a domain is not overquota, the value is set to active.

Disabling Quota Enforcement

If it appears that user quotas are being enforced, even when you have disabled them, check the following parameters:

These configutil parameters should be off or not set:

Note that when store.overquotastatus is on, it always treats store.quotaoverdraft as on, otherwise the user will never go over quota to trigger the rejection. Also, when store.quotaoverdraft is on, the user is allowed one message which is smaller than the quota only. That is, it will never accept a message that is greater than the user’s quota.

After making changes to these parameters, be sure to restart your messaging services.

These Message Store attributes should be active:

Note that messages will bounce if they are larger than the mailbox quota, regardless of quota enforcement configuration.

To Set a Grace Period

The grace period specifies how long the mailbox can be over the quota (disk space or number of messages) before messages are bounced back to sender. Messages are accepted by the MTA, but remain in the MTA queue and are not delivered to the message store until one of the following occurs:

For example, if your grace period is set for two days, and you exceed quota for one day, new messages continue to be received and held in the message queue, and delivery attempts continue. After the second day, the messages bounce back to the sender.


Note –

The grace period is not how long the message is held in the message queue, it’s how long the mailbox is over quota before all incoming messages, including those in the message queue, are bounced. The grace period starts when the user has reached the quota threshold. See To Specify a Quota Threshold and been warned.


ProcedureTo Set a Grace Period for Messages Held in the Queue

Steps
  1. Click the Quota tab.

  2. In the “Over quota grace period” field, enter a number.

  3. From the drop-down list, specify Day(s) or Hour(s).

  4. Click Save.

    Command Line

    To specify a quota grace period at the command line:

    configutil -o store.quotagraceperiod -v number

    where number indicates number of hours.

Netscape Messaging Server Quota Compatibility Mode

After disk usage exceeded the quota in the Netscape Messaging Server, the server deferred or bounced message delivery, sent an over quota notification, and started the grace period. Messaging Server provides a parameter, local.store.quotaoverdraft, which retains this behavior.

When set to ON, messages are delivered until disk usage is over quota. At that time, messages are deferred (messages stay in the MTA message queue but are not delivered to the message store), an over quota warning message is sent to the user, and a grace period starts. The grace period determines how long a mailbox is overquota before the overquota messages bounce. (The default is that the quota warning messages are sent when the message store reaches the threshold.) The default for this parameter is Off.

To Set the Automatic Message Removal (Expire and Purge) Feature

The automatic message removal feature (also known as expire and purge) automatically removes messages from the message store based on a set of administrator-defined criteria. This feature can automatically remove old or overly large messages, seen/deleted messages, messages with certain Subject: lines and so on. This feature allows the following removal criteria:

This feature is performed by the imexpire utility, which expunges and purges messages. See How the Message Store Removes Messages for details on the message removal process.


Note –

The server removes messages without warning, so it is important to inform users about automatic message removal policies. Unexpected message removal can be a source of consternation for users and administrators.


imexpire Theory of Operation

imexpire can be invoked from the command line or scheduled to run automatically by the imsched daemon. The administrator specifies a set of expiration rules in a file called store.expirerule. This file specifies the criteria by which messages are removed. There can be multiple files with each put in the directory that pertains to the scope of the rules. That is, rules that apply globally to the entire message store are put in one directory, rules that apply to a partition in another, rules that apply to users in yet another, and so on.


Note –

Although global expiration rules can be specified with either the Console or the configutil command, using store.expirerule is the preferred method. If too many rules are created using the Console or configutil, performance problems can result.


imexpire loads all of the expire rules at start up. By default, imexpire creates one thread per partition. Each thread goes through the list of user folders under its assigned partition and loads the local expire rule files as it goes. The expire function checks each folder against the expire rules applicable to this folder and expunges messages as needed. If there is a store.exp file that exists under the mailbox directory, and there are messages that have been expunged/expired for longer than the time specified by the store.cleanupage configuration parameter, the purge function will permanently remove the message files under the message hash directories and remove the UID records from the store.exp files.

It is also possible to exclude specified users from the expire rules by adding their user ID, one per line, in a file called expire_exclude_list in msg_svr_base/config/.

To Deploy the Automatic Message Removal Feature

Automatic message removal requires three steps:

  1. Define automatic message removal policy: Which messages will be automatically removed? What users, folders, domains and partitions will have messages automatically removed? What size, message age, headers will define the removal criteria. Define the scope of messages to be removed. See To Define Automatic Message Removal Policy

  2. Specify theimexpire rules to implement this policy. See To Set Rules Implementing Automatic Message Removal Policy

  3. Specify the imexpire scheduling. See To Schedule Automatic Message Removal and Logging Level

To Define Automatic Message Removal Policy

Define your automatic message removal policy by specifying the criteria for removal. imexpire allows for removal using the following criteria:

Age of Message. Automatically remove messages older than X days. Attribute: messagedays.

Message Count. Automatically remove messages in a folder exceeding X messages. Attribute: messagecount.

Age of Oversized Message. Automatically remove messages that exceed X bytes after Y days grace period. Attributes: messagesize and messagesizedays.

Seen and Deleted Message Flag. Automatically remove messages with the Seen or Deleted flag set. These criteria can be set to “and” or “or.” If set to or, the message’s Seen/Delete flag will cause automatic deletion regardless of other criteria. If set to and, the message’s Seen/Delete flag must be set along with passing all other specified criteria. Attributes: seen and deleted.

Header Field of Message. Allows you specify a header and string as criteria for removing a message. For example, removing all messages with the header “Subject: Work from Home!”

Folder of Messages. Allows you to specify the folder on which to remove messages. Attribute: folderpattern


Note –

imexpire does not allow you to delete or preserve messages based on how long it has been since that message was read. For example, you cannot specify that messages that have not been read for 200 days will be removed.


Examples of Automatic Message Removal Policy

Example 1: Remove all messages 365 days old in a folder exceeding 1,000 messages.

Example 2: Remove messages in domain siroe.com that are older than 180 days.

Example 3: Remove all messages that have been marked as deleted.

Example 4: Remove messages in sesta.com that have been marked as seen, are older than 30 days, are larger than 100 kilobytes, from folders exceeding 1,000 messages, with the header X-spam.

To Set Rules Implementing Automatic Message Removal Policy

To implement the automatic message removal policy defined in the previous section, you must set the imexpire rules. Rules are set by putting them into a store.expirerule file. An example of two global store.expirerule rules is shown below:


Rule1.regexp: 1
Rule1.folderpattern: user/.*/trash
Rule1.messagedays: 2
Rule2:regexp: 1
Rule2.folderpattern: user/.*
Rule2.messagedays: 14

            

In this example, Rule 1 specifies that all messages in the trash folder will be remove after two days. Rule 2 specifies that all messages in the message store will be removed after 14 days.

This section consists of the following subsections:

Expiration Rules Guidelines

This section sets the guidelines for the store.expirerule file rules.


Note –

In earlier Messaging Server releases, expiration rules could be set with configutil parameters store.expirerule.attribute (see configutil Parameters in Sun Java System Messaging Server 6 2005Q4 Administration Reference.) This is still true, but expire rules using header constraints (example: expiring a message with a specific subject line) are not supported. In any case, it is best to use store.expirerule to specify all expiration rules.


Table 18–8 imexpire Attributes

Attribute 

Description (Attribute Value) 

exclusive

Specifies whether or not this is an exclusive rule. If specified as exclusive, only this rule applies to the specified mailbox(s) and all other rules are ignored. If more than one exclusive rule exists, the last exclusive rule loaded will be used. For example, if a global and a local exclusive rule are specified, the local rule will be used. If there is more than one global exclusive rule, the last global rule listed by configutil is used. (1/0)

folderpattern

Specifies the folders affected by this rule. The format must start with a user/, which represents the directory store_root/partition/*/ See Figure 18–4 and Table 18–9. (POSIX regular expression)

messagecount

Maximum number of messages in a folder. Oldest messages are expunged as additional messages are delivered. (integer) 

foldersize

Maximum size of folder before the oldest messages are expunged when additional messages are delivered. (integer in bytes) 

messagedays

Age of message in days before being expunged. (integer) 

messagesize

Maximum size of message in bytes before it is marked to be expunged. (integer) 

messagesizedays

Grace period. Days an over-sized message should remain in a folder. (integer) 

message header field

Specifies a header field and string that marks a message for removal. Values are not case-sensitive and regular expressions are not recognized.Example: Rule1.Subject: Get Rich Now!

For the header Expires and Expiry-Date, imexpire will remove the message if the date value specified with these header fields is older than the messagedays attribute. If multiple expiration header fields are specified, the earliest expiration date will be used. (string).

regexp

Enable UNIX regular expressions in rules creation. (1 or 0). If not specified, IMAP expressions will be used. 

seen

seen is a message status flag set by the system when the user opens a message. If the attribute seen is set to and, then the message must be seen and other criteria must be met before the rule is fulfilled. If the attribute seen is set to or, then the message only needs to be seen or another criteria be met before the rule is fulfilled. (and/or).

deleted

deleted is a message status flag set by the system when the user deletes a message. If the attribute deleted is set to and, then the message must be deleted and another criteria must be met before the rule is fulfilled. If the attribute deleted is set to or, then the message only needs to be seen or another criteria be met before the rule is fulfilled. (and/or)

Setting imexpire Rules Textually

Automatic message removal rules are set by specifying rules in a store.expirerule file. The store.expirerule file contains one expire criteria per line. An expire criteria of the global rule configuration file (msg_svr_base/data/store/store.expirerule) has the following format:

rule_name.attribute: value

An expiration rule for a user or mailbox rule configuration file has this format:

attribute: value

Example 18–1 shows a set of global expiration rules in msg_svr_base/config/store.expirerule.

Rule 1 sets the global expiration policy (that is, policy that applies to all messages), as follows:

Rule 2 sets the automatic message removal policy for users at the hosted domain siroe.com. It limits mailbox sizes to 1 megabyte, removes messages that have been deleted, and removes messages older than 14 days.

Rule 3 sets the automatic message removal policy for messages in the inbox folder of user f.dostoevski. It removes messages with a subject line having the expression “On-line Casino.”


Example 18–1 Example imexpire Rules


Rule1.regexp: 1
Rule1.folderpattern: user/.*
Rule1.messagesize: 100000
Rule1.messagesizedays: 3
Rule1.deleted: or
Rule1.Subject: Vigara Now!
Rule1.Subject: XXX Porn!
Rule1.messagecount: 1000
Rule1.messagedays: 365
Rule2.regexp: 1
Rule2.folderpattern: user/.*@siroe.com/.*Rule2.exclusive: 1
Rule2.deleted: or
Rule2.messagedays: 14
Rule2.messagecount: 1000
Rule3.folderpattern: user/f.dostoevski/inboxRule3.Subject: *On-line Casino*
                  

Setting imexpire Folder Patterns

Folder patterns can be specified using POSIX regular expressions by setting the imexpire attribute regex to 1. If not specified, IMAP expressions will be used. The format must start with a user/ followed by a pattern. Table 18–9 shows the folder pattern for various folders.)

Table 18–9 imexpire Folder Patterns Using Regular Expressions

Folder Pattern 

Scope 

user/userid/.*

Apply rule to all messages in all folders of userid.

user/userid/Sent

Apply rule to messages of userid in folder Sent:

user/.*

Apply rule to entire message store. 

user/.*/trash

Apply rule to the trash folder of all users.

user/.*@siroe.com/.*

Apply rule to folders in hosted domain siroe.com: 

user/[^@]*/.*

Apply rule to folders in default domain. 

ProcedureTo Set Global Automatic Message Removal Rules with the Console

Note that while it is possible to set global expiration rules using the Console, using the using store.expirerule is the preferred method. If too many rules are created using the Console or configutil, performance problems can result.

Steps
  1. Bring up the automatic message removal GUI as follows:

    Main Console > Server Group > Messaging Server (Open) >Messaging Server Console > Configuration Tab > Message Store > Expire/Purge >Add

    A very rough drawing of the GUI is shown on Figure 18–4.

    Figure 18–4 Automatic Message Removal (Expire/Purge) GUI—Rough Drawing

    Graphic shows an appropriate drawing of the automatic message
removal GUI.

  2. Enter a name for the new rule.

  3. Enter the folders from which messages will be automatically removed.

    See Setting imexpire Folder Patterns above.

  4. If this rule is the exclusive rule for folders matching the specified criteria, then check the Exclusive box.

    If this box is checked, then this rule takes precedence over all other rules matching the specified pattern. See Table 18–8 for details on the exclusive checkbox.

  5. To create a rule based on folder size, do the following:

    • Check the Folder Size constraint checkbox. In the Message count field, specify the maximum number of messages that will be retained in a folder before the oldest messages are removed. In the Folder size field, specify the maximum folder size, in bytes, before the oldest messages are removed.

  6. To create a rule based on message age, check the Message age constraint checkbox:

    In the Number of days field, specify the age, in days, of how long messages should remain in the folder.

  7. To create a rule based on message size:

    • Check the Message size limit constraint checkbox. In the Message size limit field, enter the maximum size of a message allowed in the folder. In the “Grace period” field, enter the how long over-sized messages will remain in the folder before being removed.

  8. To create a rule based on whether the Seen or Deleted message flags are set:

    • Check the Message flags constraint checkbox.

    • For the Seen: field, select “and” to specify that the message must be seen and another criteria must be met before the rule is fulfilled. Select “or” to specify that the message need only be seen or another criteria be met before the rule is fulfilled.

    • For the Deleted: field, select “and” to specify that the message must be deleted and another criteria must be met before the rule is fulfilled. Select “or” to specify that the message need only be deleted or another criteria be met before the rule is fulfilled.

  9. To create a rule based on a header fields and their values:

    • Check the Header constraint checkbox.

    • Enter a comma separated list of headers and values in the following format:

      header1: value1, header2: value2

      Example: Subject: Work at Home!,From: virus@sesta.com

      For the header Expires and Expiry-Date, the system will remove the message if their date value is older than the Message age constraint. If multiple expiration header fields are specified, the earliest expiration date will be used. (string).

  10. Click OK to add the new rule to the Automatic Message Removal list.

To Schedule Automatic Message Removal and Logging Level

Automatic message removal is activated by the imsched scheduling daemon. By default, imsched invokes imexpire at 23:00 every day, and messages are both expunged and purged. This schedule can be customized by setting the configutil parameters local.schedule.expire, local.schedule.purge, and store.cleanupage described in Table 18–10.

Expire and purge can take a long time to complete on a large message store, so you may wish to experiment and decide how often to run these processes. For example, if an expire/purge cycle takes 10 hours, you may not want the default schedule of running expire and purge once a day. Schedule expire and purge by using local.schedule.purge to specify a separate schedule for purge. If local.schedule.purge is not set, imexpire will perform purge after an expire.

Table 18–10 Expire and Purge configutil Log and Scheduling Parameters

Parameter  

Description 

local.schedule.expire

Interval for running imexpire. Uses UNIX crontab format:minute hour day-of-month month-of-year day-of-week

The values are separated by a space or tab and can be 0-59, 0-23, 1-31, 1-12 or 0-6 (with 0=Sunday) respectively. Each time field can be either an asterisk (meaning all legal values), a list of comma-separated values, or a range of two values separated by a hyphen. Note that days can be specified by both day of the month and day of the week, however, it is not typical to use them both since the number of such occurrences are very small. If they are both specified, then both will be required. For example, setting the 17th day of the month and Tuesday will require both values to be true. 

Interval Examples:

1) Run imexpire at 12:30am, 8:30am, and 4:30pm: 30 0,8,16 * * * /opt/SUNWmsgsr/lib/imexpire

2) Run imexpire at weekday morning at 3:15 am: 15 3 * * 1-5 /opt/SUNWmsgsr/lib/imexpire

3) Run imexpire only on Mondays: 0 0 * * 1 /opt/SUNWmsgsr/lib/imexpire

Default: 0 23 * * * /opt/SUNWmsgsr/lib/imexpire

local.schedule.purge

Interval for running purge. Uses UNIX crontab format:minute hour day-of-month month-of-year day-of-week.

Default: 0 0,4,8,12,16,20 * * * /opt/SUNWmsgsr/lib/purge -num=5 (Every four hours.) 

store.cleanupage

Age (in hours) of expired or expunged message before purge will permanently remove it.

Default: None 

local.store.expire.loglevel

Specify a log level: 

1 = log summary for the entire expire session.  

2 = log one message per mailbox expired.  

3 = log one message per message expired. 

Default: 1 

imexpire Scheduling Using the Console

Bring up the automatic message removal GUI as follows:

Main Console > Server Group > Messaging Server (Open) >Messaging Server Console > Configuration Tab > Message Store > Expire/Purge

This Console page lists the expire rules on the top and the expire and purge schedule on the bottom. To schedule expire and purge, use the pull down menus in the Expire/Purge Schedule to set the month, day of month, day of week (with 0=Sunday), hours of day and minute of hours for both expire and purge.


Note –

The day value may be set by both day of the month and day of the week. Both are adhered to if both are set. If you set the 3rd day of the week (Wednesday) and the 17th day of the month, then a purge/expire will only occur on the 17th day of each month that falls on a Wednesday).


Setting imexpire Logging Levels

imexpire will log a summary to the default log file upon completion. If expire is invoked from the command line, the -v (verbose) and -d (debug) option can be used to instruct imexpire to log detail status/debug messages to stderr. If imexpire is invoked by imsched, the configutil parameter local.store.expire.loglevel can be set to 1,2 or 3 for different levels of logging. Loglevel 1 is the default, it will log a summary for the entire expire session. Loglevel 2 will log one message per mailbox expired. Loglevel 3 will log one message per message expired.

Excluding Specified Users from Automatic Message Remove

Exclude specified users from the expire rules by adding their user ID, one per line, in a file called expire_exclude_list in msg_svr_base/config/.

Configuring Message Store Partitions

Mailboxes are stored in message store partitions, an area on a disk partition specifically devoted to storing the message store. Message store partitions are not the same as disk partitions, though for ease of maintenance, it is recommended that you have one disk partition and one file system for each message store partition. Message store partitions are directories specifically designated as a message store.

User mailboxes are stored by default in the store_root/partition/ directory (see Message Store Directory Layout). The partition directory is a logical directory that might contain a single partition or multiple partitions. At start-up time, the partition directory contains one subpartition called the primary partition.

You can add partitions to the partition directory as necessary. For example, you might want to partition a single disk to organize your users as follows:


store_root/partition/mkting/
store_root/partition/eng/
store_root/partition/sales/

As disk storage requirements increase, you might want to map these partitions to different physical disk drives.

You should limit the number of mailboxes on any one disk. Distributing mailboxes across disks improves message delivery time (although it does not necessarily change the SMTP accept rate). The number of mailboxes you allocate per disk depends on the disk capacity and the amount of disk space allocated to each user. For example, you can allocate more mailboxes per disk if you allocate less disk space per user.

If your message store requires multiple disks, you can use RAID (Redundant Array of Inexpensive Disks) technology to ease management of multiple disks. With RAID technology, you can spread data across a series of disks but the disks appear as one logical volume so disk management is simplified. You might also want to use RAID technology for redundancy purposes; that is, to duplicate the store for failure recovery purposes.


Note –

To improve disk access, the message store and the message queue should reside on separate disks.


To Add a Partition

When adding a partition, you specify both an absolute physical path where the partition is stored on disk and a logical name, called the partition nickname.

The partition nickname allows you to map users to a logical partition name regardless of the physical path. When setting up user accounts and specifying the message store for a user, you can use the partition nickname. The name you enter must be an alphanumeric name and must use lowercase letters.

To create and manage the partition, the user ID used to run the server must have permission to write to the location specified in the physical path.


Note –

After adding a partition, you must stop then restart the server to refresh the configuration information.


Console

To add a partition to the store by using the Console:

ProcedureTo Add a Message Store Partition

Steps
  1. From Console, open the Messaging Server you want to configure.

  2. Click the Configuration tab and select Message Store in the left pane.

  3. Click the Partition tab in the right pane.

  4. Click the Add button.

  5. Enter the Partition nickname.

    This is the logical name for the specified partition.

  6. Enter the Partition path.

    This is the absolute path name for the specified partition.

  7. To specify this as the default message store partition, click the selection box labeled Make This the Default Partition.


    Note –

    The default partition is the partition used when a user is created and the mailMessageStore LDAP attribute is not specified in the user entry. The mailMessageStore LDAP attribute should be specified in all user entries so that a default partition is not necessary.


  8. Click OK to submit this partition configuration entry and dismiss the window.

  9. Click Save to submit and preserve the current Partition list.

    Command Line

    To add a partition to the store at the command line:

    configutil -o store.partition.nickname.path -v path

    where nickname is the logical name of the partition and path indicates the absolute path name where the partition is stored.

    To specify the path of the default primary partition:


    configutil -o store.partition.primary.path -v path
    

To Move Mailboxes to a Different Disk Partition

By default, mailboxes are created in the primary partition. If the partition gets full, additional messages cannot be stored. There are several ways to address the problem:

If possible, we recommend adding additional disk space to a system using volume management software since this procedure is the most transparent for the user. However, you may also move mailboxes to a different partition.

ProcedureTo Move Mailboxes to a Different Disk Partition

Steps
  1. Make sure user is disconnected from their mailbox during the migration process. This can be done by informing the user to log off and stay off during mailbox move, or, by setting the mailAllowedServiceAccess attribute so that POP, IMAP and HTTP services are disallowed after they are logged off. (See mailAllowedServiceAccess in Sun Java System Communications Services 6 2005Q4 Schema Reference.


    Note –

    Setting mailAllowedServiceAccess to disallow POP, IMAP, HTTP access does not disconnect any open connections to the mailbox. You must make sure that all connections are closed prior to the moving mailboxes.


  2. Move the user mailbox with the following command:

    mboxutil -r user/<userid>/INBOX user/<userid>/INBOX <partition_name>

    Example:

    mboxutil -r user/ofanning/INBOX user/ofanning/INBOX secondary

  3. Set the mailMessageStore attribute in the moved user’s LDAP entry to the name of the new partition.

    Example: mailMessageStore: secondary

  4. Inform the user that message store connection is now allowed. If applicable, change the mailAllowedServiceAccess attribute to allow POP, IMAP and HTTP services.

Changing the Default Message Store Partition Definition

The default partition is the partition used when a user is created and the mailMessageStore LDAP attribute is not specified in the user entry. The mailMessageStore LDAP attribute, which specifies a user’s message store partition, should be specified in all user entries so that a default partition is not necessary. In addition, the default partition should not be changed for load balancing or any other reason. It is invalid and dangerous to change the default partition while there are still users depending on the default partition definition.

If it is absolutely necessary to change the default partition, make sure that all users on the old default partition (the one being left behind) have their mailMessageStore attribute set to their current partition (which will no longer be the default), before changing the definition of default with the configutil parameter store.defaultpartition.

Performing Message Store Maintenance Procedures

This section provides information about the utilities you use to perform maintenance and recovery tasks for the message store. You should always read your postmaster mail for warnings and alerts that the server might send. You should also monitor the log files for information about how the server is performing. For more information about log files, see Chapter 21, Managing Logging

This section contains the following:

Adding More Physical Disks to the Message Store

The Messaging Server message store contains the user mailboxes for a particular Messaging Server instance. The size of the message store increases as the number of mailboxes, folders, and log files increase.

As you add more users to your system, your disk storage requirements increase. Depending on the number of users your server supports, the message store might require one physical disk or multiple physical disks. Messaging Server enables you an add more stores as needed. One approach to adding more stores is by using storage appliances. See for information on how to configure Network Appliance storage appliances with Messaging Server.

To Manage Mailboxes

This section describes the following utilities for managing and monitoring mailboxes: mboxutil, hashdir, readership.

The mboxutil Utility

Use the mboxutil command to perform typical maintenance tasks on mailboxes. mboxutil tasks include the following:


Note –

Note that you should not to kill the mboxutil process in the middle of execution. If it is killed with SIGKILL (kill -9), it may potentially require that every server get restarted and a recovery be done.


Table 18–11 lists the mboxutil commands. For detailed syntax and usage requirements, see the Sun Java System Messaging Server 6 2005Q4 Administration Reference.

Table 18–11 mboxutil Options

Option  

Description  

-a

Obsolete. Used to list all user quota information. Use. imquotacheck

-c mailbox

Creates the specified mailbox. Can be used with -f. 

A mailbox must exist before creating a secondary mailbox. 

-d mailbox

Delete the specified mailbox. 

To delete a user from the message store, use the following value for -d mailbox:

user/userid/INBOX

For example, to delete the user john from the message store, use -d user/john/INBOX. To delete the mm folder in the user john’s mailbox, use -d user/john/mm.

The recommended method to delete a user is to mark the user status as deleted in the LDAP directory (by using the Delegated Administrator utility commadmin user delete command or the Delegated Administrator console.) Next, use the commadmin user purge command to purge the users that have been marked as deleted for a period longer than the specified number of days.

If you use the Delegated Administrator utility as described in the preceding paragraph, you do not have to use the mboxutil -d command to delete a mailbox.

-e

Expunges all deleted messages in the message store. This option also can be used with the -p pattern option to expunge all deleted mailboxes with names that match pattern.

-f file

Specifies a file that stores mailbox names. The -f option can be used with the -c, -d, or -r options.

The file contains a list of mailboxes on which the mboxutil command is executed. The following is an example of entries in a data file:

user/daphne/INBOXuser/daphne/projxuser/daphne/mm

-k mailbox cmd

Obsolete. Locks the specified mailbox at the folder level; runs the specified command; after command completes, unlocks the mailbox. 

-l

Lists all of the mailboxes on a server. 

If you create multi-byte folders for different language locales, you should edit: msg_svr_base/sbin/bundles/encbylang.properties to associate the appropriate character set with the LANG environment variable.

-o

Checks for orphaned accounts. This option searches for inboxes in the current messaging server host which do not have corresponding entries in LDAP. For example, the -o option finds inboxes of owners who have been deleted from LDAP or moved to a different server host. For each orphaned account it finds, mboxutil writes the following command to the standard output:

mboxutil-d user/userid/INBOX

unless -w is specified

-p MUTF7_IMAP_pattern

When used with the -l option, lists only those mailboxes with names that match MUTF7_IMAP_pattern.

Can also be used with the -d or -e option to delete or expunge mailboxes with names that match MUTF7_IMAP_pattern.

You can use IMAP wildcards. This option expects a pattern in IMAP M-UTF-7 format. This is not the recommended way to search for non ascii mailboxes. To search for non ascii mailboxes, use the -P option. 

-P regexp

Lists only those mailboxes with names that match the specified POSIX regular expression. This option expects the regexp in the local language

-q domain

Obsolete. Use imquotacheck -d domain

-r oldname newname[partition]

Renames the mailbox from oldname to newname. To move a folder from one partition to another, specify the new partition with the partition option. Can be used with the -f flag to use a file.

This option can be used to rename a user. For example, mboxutil -r user/user1/INBOX user/user2/INBOX moves all mail and mailboxes from user1 to user2, and new messages will appear in the new INBOX. (If user2 already exists, this operation will fail.)

-R mailbox

Restores deleted messages that have not yet been purged. 

When a mailbox is expunged or expired, the uids of the deleted messages are stored in a store.exp file. The messages are physically removed by imexpire after the cleanup page has passed. When expunge or expire is issued by mistake, this option can by used to restore the deleted messages that have not been purged by imexpire into the original mailbox.

-s

When used with the -l option, displays only the mailbox name. No other data is displayed.

-t num

Lists the mailboxes that have not been accessed in a specified number of days (num). The -t option must be used with the -o option, which identifies orphaned mailboxes.

Thus, the -t option identifies inactive mailboxes (based on last-accessed date) together with orphaned mailboxes (mailboxes that do not have corresponding user entries in the LDAP directory).

To identify (list) the orphaned and inactive mailboxes, use mboxutil -o -w file -t num.

To mark these orphaned and inactive mailboxes for deletion, use mboxutil -d -f file, where file is the same file as the one used in the preceding -w file.

To use this feature, the config variable local.enablelastaccess must be enabled for at least the number of days specified with the -t option.

-u user

Obsolete. Used to list user information. Use imquotacheck -u user

-w file

Used with the -o option. Writes to a file the mailbox names generated by the -o option (which identifies orphaned accounts).

-x

When used with the -l option, shows the path and access control for a mailbox.


Note –

POSIX regular expressions can be used in the mboxutil command.


Mailbox Naming Conventions

You must specify mailbox names in the following format: user/userid/mailbox, where userid is the user that owns the mailbox and mailbox is the name of the mailbox. For hosted domains, userid is uid@domain.

For example, the following command creates the mailbox named INBOX for the user whose user ID is crowe. INBOX is the default mailbox for mail delivered to the user crowe.

mboxutil -c user/crowe/INBOX

Important: The name INBOX is reserved for each user’s default mailbox. INBOX is the only folder name that is case-insensitive. All other folder names are case-sensitive.

Examples

To list all mailboxes for all users:

mboxutil -l

To list all mailboxes and also include path and ACL information:

mboxutil -l -x

To create the default mailbox named INBOX for the user daphne:

mboxutil -c user/daphne/INBOX

To delete a mail folder named projx for the user delilah:

mboxutil -d user/delilah/projx

To delete the default mailbox named INBOX and all mail folders for the user druscilla:

mboxutil -d user/druscilla/INBOX

To rename the mail folder memos to memos-april for the user desdemona:

mboxutil -r user/desdemona/memos user/desdemona/memos-april

To move the mail account for the user dimitria to a new partition:

mboxutil -r user/dimitria/INBOX user/dimitria/INBOX partition

where partition specifies the name of the new partition.

To move the mail folder named personal for the user dimitria to a new partition:

mboxutil -r user/dimitria/personal user/dimitria/personal partition

To Remove Orphan Accounts

To search for orphaned accounts (orphaned accounts are mailboxes that do not have corresponding entries in LDAP) use the following command:


mboxutil -o

Command output follows:

  mboxutil: Start checking for orphaned mailboxes
  user/annie/INBOX
  user/oliver/INBOX
  mboxutil: Found 2 orphaned mailbox(es)
  mboxutil: Done checking for orphaned mailboxes

Use the following command to create a file listing orphaned mailboxes that can be turned into a script file that deletes the orphaned mailboxes (example filename is orphans.cmd):


mboxutil -o -w orphans.cmd

The command output is as follows:

  mboxutil: Start checking for orphaned mailboxes
  mboxutil: Found 2 orphaned mailbox(es)
  mboxutil: Done checking for orphaned mailboxes

Delete the orphan files with the following command:


mboxutil -d -f orphans.cmd

The hashdir Utility

The mailboxes in the message store are stored in a hash structure for fast searching. Consequently, to find the directory that contains a particular user’s mailbox, use the hashdir utility.

This utility identifies the directory that contains the message store for a particular account. This utility reports the relative path to the message store, such as d1/a7/. The path is relative to the directory level just before the one based on the user ID. The utility sends the path information to the standard output.

For example, to find the relative path to the mailbox for user crowe:

hashdir crowe

The readership Utility

The readership utility reports on how many users other than the mailbox owner have read messages in a shared IMAP folder.

An owner of a IMAP folder may grant permission for others to read mail in the folder. A folder that others are allowed to access is called a shared folder. Administrators can use the readership utility to see how many users other than the owner are accessing a shared folder.

This utility scans all mailboxes and produces one line of output per shared folder, reporting the number of readers followed by a space and the name of the mailbox.

Each reader is a distinct authentication identity that has selected the shared folder within the past specified number of days. Users are not counted as reading their own personal mailboxes. Personal mailboxes are not reported unless there is at least one reader other than the folder’s owner.

For example, the following command counts as a reader any identity that has selected the shared IMAP folder within the last 15 days:

readership -d 15

To Monitor Quota Limits

Monitor quota usage and limits by using imqutoacheck, which generates a report listing defined quotas and limits, and provides information on quota usage. Quotas and usage figures are reported in kilobytes. This utility can also compare mailbox size with a user’s assigned quota. As an option, you can email a notification to users who have exceeded a set percentage of their assigned quota.


Note –

Certain functions have changed in imquotacheck. (In Messaging Server 6.x, imquotacheck utility has superseded the quotacheck utility.) In Messaging Server 5.x, when you used the quotacheck utility to retrieve a list of users, quotacheck searched the local mboxlist database. This function duplicated the search function in the mboxutil utility.

In Messaging Server 6.x, this duplicate function was removed from the imquotacheck utility. If you perform a user search with imquotacheck, the search is performed against the LDAP directory, not the local mboxlist database. To retrieve a list of users from the local mboxlist database, use the mboxutil utility.


To list the usage of all users whose quota exceeds the least threshold in the rule file:

imquotacheck

List quota information for a the domain siroe.com:

imquotacheck -d siroe.com

To send a notification to all users in accordance to the default rule file:

imquotacheck -n

To send a notification to all users in accordance to a specified rulefile, myrulefile, and to a specified mail template file, mytemplate.file (for more information, refer to imquotacheck in Sun Java System Messaging Server 6 2005Q4 Administration Reference):

imquotacheck -n -r myrulefile -t mytemplate.file

To list the usage of all users and (will ignore the rule file):

imquotacheck -i

To list per folder usages for users user1 (will ignore the rule file):

imquotacheck -u user1 -e

To Monitor Disk Space

You can specify how often the system should monitor disk space and partition usage, and under what circumstances the system should send a warning. See Monitoring Disk Space for details.

Using the stored Utility

The stored utility performs the following monitoring and maintenance tasks for the server:

The stored utility automatically performs cleanup and expiration operations once a day at 11 PM. You can choose to run additional cleanup and expiration operations.

Table 18–12 lists some of the stored options. Some common usage examples follow the table. For detailed syntax and usage requirements, see the Sun Java System Messaging Server 6 2005Q4 Administration Reference.

Table 18–12 stored Options

Option  

Description  

-d

OBSOLETE. Use start-msg store to start up stored which will run as daemon, performing system checks and activating alarms, deadlock detection, and database repair.

-t

Checks the status of stored. The return code of this command indicates the status.

-v

Verbose output. 

-v -v

More verbose output. 

To print the status, enter:

stored -t -v

If you want to change the time of the automatic cleanup and expiration operations, use the configutil utility as follows:

configutil -o store.expirestart -v 21

Occasionally, you might need to restart the stored utility; for example, if the mailbox list database becomes corrupted. To restart stored on UNIX, use the following commands at the command line:

msg_svr_base/sbin/stop-msg store
msg_svr_base/sbin/start-msg store

If any server daemon crashes, you must stop all daemons and restart all daemons including stored.

Reducing Message Store Size Due to Duplicate Storage of Identical Messages

When a message is sent to multiple recipients, that message is placed in each recipient’s mailbox. Some messaging systems store separate copies of the same message in each recipient’s mailbox. By contrast, the Sun Java System Messaging Server strives to retain a single copy of a message regardless of the number of mailboxes in which that message resides. It does this by creating hard links to that message in the mailboxes containing that message.

When other messaging systems are migrated to the Sun Java Messaging Server, these multiple message copies may be copied over with the migration process. With a large message store, this mean that a lot of messages are duplicated unnecessarily. In addition, multiple copies of the same message can be accumulated in normal server operation, for example, from IMAP append operations or other sources.

Messaging Server provides a new command called relinker that removes the excess message copies and replaces them with hard links to a single copy.

Relinker Theory of Operations

The relinking function can be run in the command or realtime mode. When the relinker command is run, it scans through the message store partitions, creates or updates the MD5 message digest repository (as hard links), deletes excess message files, and creates the necessary hard links.

The digest repository consists of hard links to the messages in the message store. It is stored in the directory hierarchy partition_path/=md5. This directory is parallel to the user mailbox hierarchy partition_path/=user (see Figure 18–1). Messages in the digest repository are uniquely identified by their MD5 digest. For example, if the digest for fredb/00/1.msg is 4F92E5673E091B43415FFFA05D2E47, then partition/=user/hashdir/hashdir/=fredb/00/1.msg is linked to partition/=md5/hashdir/hashdir/4F92E5673E091B43415FFFA05D2E47EA.msg. If another mailbox has this same message, for example, partition_path/=user/hashdir/hashdir/gregk/00/17.msg, that message will also be hard linked to partition_path/=md5/4F/92/4F92E5673E091B43415FFFA05D2E47EA.msg. This is shown in Figure 18–5.

Figure 18–5 Message Store Digest Repository

Graphic depicts message store repository.

For this message, the link count will be three. If both messages are deleted from the mailboxes of fredb and gregk, then the link count will be one and the message can be purged.

The relinker process can also be run in the realtime mode for similar functionality. See Using Relinker in the Realtime Mode for details.

Using Relinker in the Command Line Mode

relinker scans through a message store partition, creates or updates the MD5 message repository (as hard links) and deletes excess message files. After relinker scans a store partition, it outputs statistics on the number of unique messages and size of the partition before and after relinking. To run more quickly on an already hashed store, relinker only computes the digest of the messages not yet present in =md5. It also has an option to erase the entire digest repository (which doesn’t affect the user mailboxes).

The syntax for the command is as follows:

relinker [-p partitionname] [-d]

where partitionname specifies the partition to be processed (default: all partitions) and -d specifies that the digest repository be deleted. Sample output is shown below:


# relinker

Processing partition: primary
Scanning digest repository...
Processing user directories..............................
---------------------------------------------------------
Partition statistics           Before            After 
---------------------------------------------------------
Total messages                 4531898         4531898
Unique messages                4327531         3847029
Message digests in repository        0         3847029
Space used                       99210Mb         90481Mb
Space savings from single-copy    3911Mb         12640Mb
---------------------------------------------------------


# relinker -d 
Processing partition: primary
Purging digest repository...
---------------------------------------------------------
Partition statistics                 Before         After
---------------------------------------------------------
Message digests in repository       3847029             0
---------------------------------------------------------

Relinker can take a long time to run, especially for the first time if there are no messages are in the repository. This is because it has to calculate the digest for every message (if the relinker criteria is configured include all messages—see Configuring Relinker for information on configuring relinker criteria.) For example, it could take six hours to process a 100 Gigabyte message store. However, if run-time relinking is enabled (see Using Relinker in the Realtime Mode

If the relinker command line mode is used exclusively, and not the run-time option, it is necessary to purge the digest repository (=md5), otherwise messages purged in the store (=user) will not become available disk space since they still have a link in the digest repository (they become orphaned). If you are just performing a one-time optimization of the store—for example after a migration—you can run relinker once, then delete the entire repository with relinker -d. For repeated purging—during migration—it is sufficient to just run the relinker command repeatedly, since each time it runs it also purges the expired or orphaned messages from the repository.

It is safe to run multiple instances of relinker in parallel with each processing a different partition (using the -p option). Messages are only relinked inside the same partition.

Using Relinker in the Realtime Mode

The relinker function can be enabled in the realtime mode by setting the configutil parameter local.store.relinker.enabled to yes. Using relinker in the realtime mode will compute the digest of every message delivered (or restored, IMAP appended, and so forth) which matches the configured relinker criteria (Configuring Relinker), then look in the repository to see if that digest is already present. If the digest is present, it creates a link to it in the destination mailbox instead of a creating new copy of the message. If there is no digest, it creates the message and adds a link to it in the repository afterwards.

stored scans the digest repositories of each partition and purges the messages having a link count of 1, or which don’t match the relinker criteria. The scan is done one directory at a time over a configurable time period. This is so that the I/O load is evenly distributed and doesn’t noticeably impact other server operations. By default the purge cycle is 24 hours, which means messages can still be present on the disk for up to 24 hours after they’ve been deleted from the store or have exceeded the configured maximum age. This task is enabled when the relinker realtime mode is enabled.

Configuring Relinker

Table 18–13 shows the parameters used to set relinker criteria.

Table 18–13 relinker configutil Parameters

Parameter 

Description  

local.store.relinker.enabled

Enables real-time relinking of messages in the append code and stored purge. The relinker command-line tool may be run even if this option is off, however since stored will not purge the repository, relinker -d must be used for this task. Turning this option on affects message delivery performance in exchange for the disk space savings.

Default: no

local.store.relinker.maxage

Maximum age in hours for messages to be kept in the repository, or considered by the relinker command-line. -1 means no age limit, that is, only purge orphaned messages from the repository. For relinker it means process existing messages regardless of age. Shorter values keep the repository smaller thus allow relinker or stored purge to run faster and reclaim disk space faster, while longer values allow duplicate message relinking over a longer period of time, for example, when users copy the same message to the store several days apart, or when running a migration over several days or weeks.

Default: 24 

local.store.relinker.minsize

Minimum size in kilobytes for messages to be considered by run-time or command-line relinker. Setting a non-zero value gives up the relinker benefits for smaller messages in exchange for a smaller repository.

Default: 0 

local.store.relinker.purgecycle

Approximate duration in hours of an entire stored purge cycle. The actual duration depends on the time it takes to scan each directory in the repository. Smaller values will use more I/O and larger values will not reclaim disk space as fast. 0 means run purge continuously without any pause between directories. -1 means don’t run purge in stored (then purge must be performed using the relinker -d command).

Default: 24 

Backing Up and Restoring the Message Store

Message store backup and restore is one of the most common and important administrative tasks. It consists of backing up all the messages and folders on a message store. You must implement a backup and restore policy for your message store to ensure that data is not lost if problems such as the following occur:

You can do message store back up and restore using command-line utilities imsbackup and imsrestore, or the integrated solution that uses Legato NetworkerTM.

Messaging Server provides a single-copy backup procedure. Regardless of how many user folders contain a particular message, during backup, the message file is backed up only once using the first message file found. The second message copy is backed up as a link to the name of the first message file, and so on. imsbackup maintains a hash table of all messages using the device and inode of the message files as the index. This method does have implications when restoring data, however. For more information, see Considerations for Partial Restore


Note –

It is also possible to do message store backup and restore by backing up all the message files and directories. Refer to Message Store Disaster Backup and Recovery.


This section contains the following subsections:

Creating a Mailbox Backup Policy

Your backup policy will depend on several factors, such as:

Peak Business Loads

You need to take into account peak business loads when scheduling backups for your system as this can reduce system load during peak hours. For example, backups are probably best scheduled for early morning hours such as 2:00 AM.

Full and Incremental Backups

Incremental backups (see Incremental Backup) will scan the store for changed data and back up only what has changed. Full backups will back up the entire message store. You need to determine how often the system should perform full as opposed to incremental backups. You’ll probably want to perform incremental backups as a daily maintenance procedure and full backups once a week.

Parallel or Serial Backups

When user data is stored on multiple disks, you can back up user groups in parallel if you wish. Depending on system resources, parallel backups can speed up the overall backup procedure. However, you might want to use serial backups if you want to reduce backup impact on the server’s performance. Whether to use parallel or serial backups can depend on many factors, including system load, hardware configuration, how many tape drives are available, and so on.

To Create Backup Groups

A backup group is an arbitrary set of user mailboxes defined by regular expressions. By organizing user mailboxes into backup groups, you can define more flexible backup management.

For example, you could create three backup groups, the first containing user IDs starting with the letters A through L, the second with users whose user IDs begin with M-Z, and the third with users whose user IDs begin with a number. Administrators could use these backup groups to backup mailboxes in parallel, or perhaps only certain groups on one day and other groups on another.

There are several things to remember about backup groups:

  1. These are arbitrary virtual groups of mail users. They do not precisely map to the message store directory (Figure 18–1), although it may look like it.

  2. They are defined by the administrator using UNIX regular expressions.

  3. The regular expressions are defined in the configuration filemsg_svr_base/config/backup-groups.conf

  4. When backup groups are referenced in imsbackup and imsrestore, they use the path format: /partition_name/backup_group

The format of backup-groups.conf is as follows:


group_name=definition
group_name=definition
.
.
.

Using the example described in the paragraph above, the following definitions would be used to create the three backup groups:


groupA=[a-l].*
groupB=[m,-z].*
groupC=[0-9].*

You can now scope imsbackup and imsrestore at several levels. You can backup/restore the whole message store using the backup command:

imsbackup -f device /

To backup all mailboxes for all users in groupA use the following:

imsbackup -f device /partition/groupA

The default partition is called primary.

Pre-defined Backup Group

Messaging Server includes one predefined backup group that is available without creating the backup-groups configuration file. This group is called user; it includes all users. For example, the following will backup all users on the primary partition:

imsbackup -f backupfile /primary/user

Messaging Server Backup and Restore Utilities

To back up and restore your data, Messaging Server provides the imsbackup and imsrestore utilities. Note that the imsbackup and imsrestore utilities do not have the advanced features found in general purpose tools like Legato Networker. For example, the utilities have only very limited support for tape auto-changers, and they cannot write a single store to multiple concurrent devices. Comprehensive backup will be achieved via plug-ins to generalized tools like Legato Networker. For more information about using Legato Networker, see To Use Legato Networker

The imsbackup Utility

With imsbackup, you can write selected contents of the message store to any serial device, including magnetic tape, a UNIX pipe, or a plain file. The backup or selected parts of the backup may later be recovered by using the imsrestore utility. The output of imsbackup can be piped to imsrestore.

The following example backs up the entire message store to /dev/rmt/0:


imsbackup -f /dev/rmt/0 /

This backs up the mailboxes of user ID joe to /dev/rmt/0:


imsbackup -f /dev/rmt/0 /primary/user/joe
            

This example backs up all the mailboxes of all the users defined in the backup group groupA to backupfile (see To Create Backup Groups):


imsbackup -f- /primary/groupA > backupfile
            

Incremental Backup

The following example will back up messages stored from May 1, 2004 at 1:10 pm to the present. The default is to backup all the messages regardless of their dates:


imsbackup -d 20040501:13100
               

This command uses the default blocking factor of 20. For a complete syntax description of the imsbackup command, see the Sun Java System Messaging Server 6 2005Q4 Administration Reference.

The imsrestore Utility

To restore messages from the backup device, use the imsrestore command. For example, the following command restores messages for user1 from the file backupfile.

imsrestore -f backupfile /primary/user1

For a complete syntax description of the imsbackup command, see the Sun Java System Messaging Server 6 2005Q4 Administration Reference.

Excluding Bulk Mail When You Perform Backups

When you perform a backup operation, you can specify mailboxes that will be excluded from being backed up. By excluding bulk or trash mailboxes that can accrue large numbers of unimportant messages, you can streamline the backup session, reduce the time to complete the operation, and minimize the disk space required to store the backup data.

To exclude mailboxes, specify a value for the configutil parameter local.store.backup.exclude.

You can specify a single mailbox or a list of mailboxes separated by the ”%’ character. (”%’ is an illegal character in a mailbox name.) For example, you could specify the following values:

Trash

Trash%Bulk Mail%Third Class Mail

In the first example, the folder Trash is excluded. In the second example, the folders Trash, Bulk Mail, and Third Class Mail are excluded.

The backup utility backs up all folders in a user mailbox except those folders specified in the local.store.backup.exclude parameter.

This feature works with the Messaging Server backup utility, Legato Networker, and third-party backup software.

You can override the local.store.backup.exclude setting and back up an excluded mailbox by specifying its full logical name during the operation. Suppose the Trash folder has been excluded. You can still back up Trash by specifying, for example:

/primary/user/user1/trash

However, if you specify

/primary/user/user1

the Trash folder is excluded.

Considerations for Partial Restore

A partial restore is when only a part of the message store is restored. A full restore is when the entire message store is restore. The message store uses a single-copy message system. That is, only a single copy of any message is saved in the store as a single file. Any other instances of that message (like when a message is sent to multiple mailboxes) are stored as links to that copy. Because of this, there are implications when restoring messages. For example:

The following examples demonstrate what happens to a message used by multiple users when a partial restore is performed. Assume there are three messages, all the same, belonging to three users A, B, and C, as follows:


A/INBOX/1
B/INBOX/1
C/INBOX/1

Example 1. In the first example, the system performs a partial backup and full restore procedure as follows:

  1. Back up mailboxes for users B and C.

  2. Delete mailboxes of users B and C.

  3. Restore the backup data from step 1.

In this example, B/INBOX/1 and C/INBOX/1 are assigned a new inode number and the message data is written to a new place on the disk. Only one message is restored; the second message is a hard link to the first message.

Example 2. In this example, the system performs a full backup and a partial restore as follows:

  1. Perform full backup.

  2. Delete mailboxes for user A.

  3. Restore mailboxes for user A.

A/INBOX/1 is assigned a new inode number.

Example 3. In this example, partial restore might require more than one attempt:

  1. Perform full backup.

    B/INBOX/1 and C/INBOX/1 are backed up as links to A/INBOX/1.

  2. Delete mailboxes for users A and B.

  3. Restore mailboxes for user B.

    The restore utilities ask the administrator to restore A/INBOX first.

  4. Restore mailboxes for users A and B.

  5. Delete mailboxes for user A (optional).


    Note –

    If you want to ensure that all messages are restored for a partial restore, you can run the imsbackup command with the -i option. The -i option backs up every message multiple times if necessary.

    If the backup device is seekable (example: a drive or tape), imsrestore seeks to the position containing A/INBOX/1 and restore it as B/INBOX/1. If the backup device is non-seekable example: a UNIX pipe), imsrestore logs the object ID and the ID of the depending (linked) object to a file, and the administrator must invoke imsrestore again with the -r option to restore the missing message references.


To Restore Messages from a Mailbox that has Been Incrementally Backed-up

If you are restoring messages from a mailbox that has been incrementally backed-up, and if that mailbox exists on the server on which you wish to restore the messages, then restoring the messages requires a straightforward imesrestore. However, if you wish to restore messages from a mailbox that has been incrementally backed-up, and if that mailbox no longer exists, you must follow different restore procedures.

Use one of the following procedures to restore messages to a mailbox that does not exist on the message store server:

The reason why these instructions must be followed for restoring an incremental backup is as follows: When a mailbox has been deleted or is being migrated, the imsrestore utility recreates the mailbox with the mailbox unique identification validity and message unique identifications (UIDs) stored in the backup archive.

In the past, when imsrestore would recreate a deleted or migrated mailbox, it would assign a new UID validity to the mailbox and new UIDs to the messages. In that situation, a client with cached messages would have to resynchronize the mailbox UID validity and message UIDs. The client would have to download the new data again, increasing the workload on the server.

With the new imsrestore behavior, the client cache remains synchronized, and the restore process operates transparently with no negative impact on performance.

If a mailbox exists, imsrestore assigns new UIDs to the restored messages so that the new UIDs remain consistent with the UIDs already assigned to existing messages. To ensure UID consistency, imsrestore locks the mailbox during the restore operation. However, because imsrestore now uses the mailbox UID validity and message UIDs from the backup archive instead of assigning new UID values, UIDs could become inconsistent if you perform incremental backups and restores.

If you perform incremental backups with the -d date option of the imsbackup utility, you might have to invoke imsrestore multiple times to complete the restore operation. If incremental backups were performed, you must restore the latest full backup and all subsequent incremental backups.

New messages can be delivered to the mailbox between the restore operations, but in this case, the message UIDs can become inconsistent. To prevent inconsistency in the UIDs, you need to take one of the actions describe above.

To Use Legato Networker

Messaging Server includes a backup API that provides an interface with third-party backup tools, such as Legato Networker. The physical message store structure and data format are encapsulated within the backup API. The backup API interacts directly with the message store. It presents a logical view of the message store to the backup service. The backup service uses the conceptual representation of the message store to store and retrieve the backup objects.

Messaging Server provides an Application Specific Module (ASM) that can be invoked by the Legato Networker’s save and recover commands to back up and restore the message store data. The ASM then invokes the Messaging Server imsbackup and imsrestore utilities.


Note –

This section provides information about how to use Legato Networker with the Messaging Server message store. To understand the Legato Networker interface, see your Legato documentation.


ProcedureTo Back Up Data Using Legato Networker

Steps
  1. Create a symbolic link from /usr/lib/nsr/imsasm to msg_srv_base/lib/msg/imsasm

  2. From Sun or Legato, obtain a copy of the nsrfile binary and copy it to the following directory:

    /usr/bin/nsr

    Note that this is required only if you are using an older version of Networker (5.x). With Networker 6.0 and above, nsrfile is automatically installed under /usr/bin/nsr.

  3. If you want to back up users by groups, perform the following steps:

    1. Create a backup group file as described in To Create Backup Groups

    2. To verify your configuration, run mkbackupdir.sh.

      Look at the directory structure created by mkbackupdir.sh.The structure should look similar to that shown in Table 18–4.

      Note that if you do not specify a backup-groups.conf file, the backup process will use the default backup group ALL for all users.

  4. In the directory /nsr/res/, create a res file for your save group to invoke the mkbackupdir.sh script before the backup. See Table 18–4 for an example.


    Note –

    Earlier versions of Legato Networker have a limitation of 64 characters for the save set name. If the name of this directory plus the logical name of the mailbox (for example, /primary/groupA/fred) is greater than 64 characters, then you must run mkbackupdir.sh -p. Therefore, you should use a short path name for the -p option of mkbackupdir.sh. For example the following command will create the backup image under the directory /backup:

    mkbackupdir.sh -p /backup

    Important: The backup directory must be writable by the message store owner (example: inetuser).


    Table 18–6 shows a sample backup groups directory structure.


    /backup/primary/groupA/amy
                          /bob
                          /carly
                   /groupB/mary
                          /nancy
                          /zelda
                   /groupC/123go
                          /1bill
                          /354hut

    The example below shows a sample res file named IMS.res in the /nsr/res directory:


    type: savepnpc;
    precmd: "echo mkbackupdir started",
       "/usr/siroe/server5/msg-siroe/bin/mkbackupdir.sh -p /backup";
    pstcmd: "echo imsbackup Completed";
    timeout: "12:00 pm";
    
                         

    You are now ready to run the Legato Networker interface as follows:

  5. Create the Messaging Server save group if necessary.

    1. Run nwadmin.

    2. Select Customize | Group | Create.

  6. Create a backup client using savepnpc as the backup command:

    1. Set the save set to the directory created by mkbackupdir.

      For a single session backup, use /backup

      For parallel backups, use /backup/server/group

      Be sure you’ve already created group as defined in To Create Backup Groups

      You must also set the parallelism to the number of backup sessions.

      See To Back Up Data Using Legato Networker.

  7. Select Group Control | Start to test your backup configuration.

    Example. Creating A Backup Client in Networker:

    To create a backup client in Networker. From nwadmin, select Client | Client Setup | Create


    Name: siroe
    Group: IMS
    Savesets:/backup/primary/groupA
       /backup/secondary/groupB
       /backup/tertiary/groupC
             .
             .
    Backup Command:savepnpc
    Parallelism: 4
    
                         

Restoring Data Using Legato Networker

To recover data, you can use the Legato Networker nwrecover interface or the recover command-line utility. The following example recovers user a1’s INBOX:

recover -a -f -s siroe /backup/siroe/groupA/a1/INBOX

The next example recovers the entire message store:

recover -a -f -s siroe /backup/siroe

To Use a Third Party Backup Software (Besides Legato)

Messaging Server provides two message store backup solutions, the command line imsbackup and the Solstice Backup (Legato Networker). A large message store running a single imbackup to backup the entire message store can take a significant amount of time. The Legato solution supports concurrent backup sessions on multiple backup devices. Concurrent backup can shorten backup time dramatically (backups of 25GB of data per hour have been achieved).

If you are using another third party concurrent backup software (for example, Netbackup), you may use the following method to integrate your backup software with the Messaging Server.

ProcedureTo Use a Third Party Backup Software (Besides Legato)

Steps
  1. Divide your users into groups (see To Create Backup Groups) and create a backup-groups.conf file under the directory msg_svr_base/config/.


    Note –

    This backup solution requires additional disk space. To backup all the groups concurrently, the disk space requirement is 2 times the message store size. If you do not have that much disk space, divide your users into smaller groups, and then backup a set of groups at a time. For example group1 - group5, group6 - group10. Remove the group data files after backup.


  2. Run imsbackup to backup each group into files under a staging area.

    The command is imsbackup -f <device> /<instance>/<group>

    You can run multiple imsbackup processes simultaneously. For example:


    # imsbackup -f- /primary/groupA > /bkdata/groupA &
    # imsbackup -f- /primary/groupB > /bkdata/groupB & 
    . . .

    imsbackup does not support large files, if the backup data is larger than 2 GB, you need to use the -f- option to write the data to stdout and then pipe the output to a file.

  3. Use your third party backup software to backup the group data files in the staging area (in our example that is /bkdata).

  4. To restore a user, identify the group filename of the user, restore that file from tape, and then use imsrestore to restore the user from the data file.

    Note that imsrestore does not support large files. If the data file is larger than 2GB. Use the following command:

    # cat /bkdata/groupA | imsrestore -f- /primary/groupA/andy

Troubleshooting Backup and Restore Problems

This section describes common backup and restore problems and their solutions.

Message Store Disaster Backup and Recovery

A disaster refers to a catastrophic failure of the entire message store. That is, a situation where all data on the message store servers are lost. A complete message store disaster restore will consist of restoring the following lost data:

Monitoring User Access

Messaging Server provides the command, imsconnutil, which allows you to monitor user’s message store access via IMAP, POP and http. You can also determine the last log in and log out of users. This command works on a per message store basis and will not work across message stores.


Note –

Use of this function or other Messaging Server functions to monitor, read or otherwise access user’s email may constitute a potential source of liability if used in violation of applicable laws or regulations or if used in violation of the customer’s own policies or agreements.


This command requires root access by the system user (default: inetuser), and you must set the configuration variables local.imap.enableuserlist, local.http.enableuserlist, local.enablelastaccess to 1.

To list users currently logged on via IMAP or any web mail client, use the following command:

# imsconnutil -c

To list the last IMAP, POP, or Messenger Express access (log in and log out) of every user on the message store use:

# imsconnutil -a

The following command does two things: 1) it determines whether the specified user is currently logged on via IMAP or Messenger Express or any client that connects via mshttp (note that this does not work for POP because POP users generally do not stay connected), and 2) it lists the last time the users have logged on and off:

# imsconnutil -c -a -u user_ID

Note that a list of users can be input from a file, one user per line, using the following command:

# imsconnutil -c -a -f filename

You can also specify a particular service (imap or http) using the -s flag. For example, to list whether a particular user ID is logged onto IMAP or not, use the following command:

# imsconnutil -c -s imap -u user_ID

For a complete description of the imsconnutil syntax, refer to imsconnutil in Sun Java System Messaging Server 6 2005Q4 Administration Reference.

Here is some example output:


$ ./imsconnutil -a -u soroork

UID     IMAP last access    HTTP last access    POP last access
=========================================================================
ed   08/Jul/2003:10:49:05   10/Jul/2003:14:55:52  ---NOT-RECORDED---

$ ./imsconnutil -c

IMAP
UID    TIME                AUTH            TO                 FROM
===========================================================================
ed   17/Jun/2003:11:24:03  plain     172.58.73.45:193   129.157.12.73:2631
bil  17/Jun/2003:04:28:43  plain     172.58.73.45:193   129.158.16.34:2340
mia  17/Jun/2003:09:36:54  plain     172.58.73.45:193   192.18.184.103:3744
jay  17/Jun/2003:05:38:46  plain     172.58.73.45:193   129.159.18.123:3687
pau  17/Jun/2003:12:23:28  plaintext 172.58.73.45:193   192.18.194.83:2943
ton  17/Jun/2003:05:38:46  plain     172.58.73.45:193   129.152.18.123:3688
ani  17/Jun/2003:12:26:40  plaintext 172.58.73.45:193   192.18.164.17:1767
ani  17/Jun/2003:12:25:17  plaintext 172.58.73.45:193   129.150.17.34:3117
jac  17/Jun/2003:12:26:32  plaintext 172.58.73.45:193   129.150.17.34:3119
ton  17/Jun/2003:12:25:32  plaintext 172.58.73.45:193   192.18.148.17:1764
===========================================================================
10 users were logged in to imap.
Feature is not enabled for http.
---------------------------------------------------------------------------

Troubleshooting the Message Store

This section provides guidelines for actively maintaining your message store. In addition, this section describes other message store recovery procedures you can use if the message store becomes corrupted or unexpectedly shuts down. Note that the section on these additional message store recovery procedures is an extension of Repairing Mailboxes and the Mailboxes Database.

Prior to reading this section, it is strongly recommended that you review this chapter as well as the command-line utility and configutil chapters in the Sun Java System Messaging Server Administration Reference. Topics covered in this section include:

Standard Message Store Monitoring Procedures

This section outlines standard monitoring procedures for the message store. These procedures are helpful for general message store checks, testing, and standard maintenance.

For additional information, see Monitoring the Message Store.

Check Hardware Space

A message store should have enough additional disk space and hardware resources. When the message store is near the maximum limit of disk space and hardware space, problems might occur within the message store.

Inadequate disk space is one of the most common causes of the mail server problems and failure. Without space to write to the message store, the mail server will fail. In addition, when the available disk space goes below a certain threshold, there will be problems related to message delivery, logging, and so forth. Disk space can be rapidly depleted when the clean up function of the stored process fails and deleted messages are not expunged from the message store.

For information on monitoring disk space, see To Monitor Disk Space and Monitoring the Message Store.

Check Log Files

Check the log files to make sure the message store processes are running as configured. Messaging Server creates a separate set of log files for each of the major protocols, or services, it supports: SMTP, IMAP, POP, and HTTP. You can look at the log files through the Console or in directory msg_svr_base/log/. You should monitor the log files on a routine basis.

Be aware that logging can impact server performance. The more verbose the logging you specify, the more disk space your log files will occupy for a given amount of time. You should define effective but realistic log rotation, expiration, and backup policies for your server. For information about defining logging policies for your server, see Chapter 21, Managing Logging.

Check User IMAP/POP Session by Using Telemetry

Messaging Server provides a feature called telemetry that can capture a user’s entire IMAP, POP or webmail session into a file. This feature is useful for debugging client problems. For example, if a user complains that their message access client is not working as expected, this feature can be used to trace the interaction between the access client and Messaging Server.

To capture a session, simply create the following directory:

msg_svr_base/data/telemetry/pop_or_imap/userid

Messaging Server will create one file per session in that directory. Example output is shown below.


LOGIN redb 2003/11/26 13:03:21
>0.017>1 OK User logged in
<0.047<2 XSERVERINFO MANAGEACCOUNTURL MANAGELISTSURL MANAGEFILTERSURL
>0.003>* XSERVERINFO MANAGEACCOUNTURL {67}
http://redb@cuisine.blue.planet.com:800/bin/user/admin/bin/enduser 
MANAGELISTSURL NIL MANAGEFILTERSURL NIL
2 OK Completed
<0.046<3 select "INBOX"
>0.236>* FLAGS (\Answered flagged draft deleted \Seen $MDNSent Junk)
* OK [PERMANENTFLAGS (\Answered flag draft deleted \Seen $MDNSent Junk \*)]
* 1538 EXISTS
* 0 RECENT
* OK [UNSEEN 23]
* OK [UIDVALIDITY 1046219200]
* OK [UIDNEXT 1968]
3 OK [READ-WRITE] Completed
<0.045<4 UID fetch 1:* (FLAGS)
>0.117>* 1 FETCH (FLAGS (\Seen) UID 330)
* 2 FETCH (FLAGS (\Seen) UID 331)
* 3 FETCH (FLAGS (\Seen) UID 332)
* 4 FETCH (FLAGS (\Seen) UID 333)
* 5 FETCH (FLAGS (\Seen) UID 334)
<etc>

Check stored Processes

The stored function performs a variety of important tasks such as deadlock and transaction operations of the message database, enforcing aging policies, and expunging and erasing messages stored on disk. If stored stops running, Messaging Server will eventually run into problems. If stored doesn’t start when start-msg is run, no other processes will start.

Table 18–14 stored Operations

stored Operation  

Function  

stored.ckp

Touched when a database checkpoint was initiated. Stamped approximately every 1 minute. 

stored.lcu

Touched at every database log cleanup. Time stamped approximately every 5 minutes. 

stored.per

Touched at every spawn of peruser db write out. Time stamped once an hour. 

For more information on the stored process, see Using the stored Utility chapter of the Sun Java System Messaging Server 6 2005Q4 Administration Reference.

For additional information on monitoring the stored function, see Monitoring the Message Store

Check Database Log Files

Database log files refer to sleepycat transaction checkpointing log files (in directory store_root/mboxlist). If log files accumulate, then database checkpointing is not occurring. In general, there are two or three database log files during a single period of time. If there are more files, it could be a sign of a problem.

Check User Folders

If you want to check the user folders, you might run the command reconstruct -r -n (recursive no fix) which will review any user folder and report errors. For more information on the reconstruct command, see Repairing Mailboxes and the Mailboxes Database

Check for Core Files

Core files only exist when processes have unexpectedly terminated. It is important to review these files, particularly when you see a problem in the message store. On Solaris, use coreadm to configure core file location.

Message Store Startup and Recovery

Message store data consists of the messages, index data, and the message store database. While this data is fairly robust, on rare occasions there may be message store data problems in the system. These problems will be indicated in the default log file, and almost always will be fixed transparently. In rare cases an error message in the log file may indicate that you need to run the reconstruct utility. In addition, as a last resort, messages are protected by the backup and restore processes described in Backing Up and Restoring the Message Store. This section will focus on the automatic startup and recovery process of stored.

The message store automates many recovery operations which were previously the responsibility of the administrator. These operations are performed by message store daemon stored during startup and include database snapshots and automatic fast recovery as necessary. stored thoroughly checks the message store’s database and automatically initiates repairs if it detects a problem.

stored also provides a comprehensive analysis of the state of the database via status messages to the default log, reporting on repairs done to the message store and automatic attempts to bring it into operation.

Automatic Startup and Recovery—Theory of Operations

The stored daemon starts before the other message store processes. It initializes and, if necessary, recovers the message store database. The message store database keeps folder, quota, subscription, and message flag information. The database is logging and transactional, so recovery is already built in. In addition, some database information is copied redundantly in the message index area for each folder.

Although the database is fairly robust, on the rare occasions that it breaks, in most cases stored recovers and repairs it transparently. However, whenever stored is restarted, you should check the default log files to make sure that additional administrative intervention is not required. Status messages in the log file will tell you to run reconstruct if the database requires further rebuilding.

Before opening the message store database, stored analyzes its integrity and sends status messages to the default log under the category of warning. Some messages will be useful to administrators and some messages will consists of coded data to be used for internal analysis. If stored detects any problems, it will attempt to fix the database and try starting it again.

When the database is opened, stored will signal that the rest of the services may start. If the automatic fixes failed, messages in the default log will specify what actions to take. See Error Messages Signifying that reconstruct -m is Needed

In previous releases, stored could start a recovery process which would take a very long time leaving the administrator wondering if stored was “stuck.” This type of long recovery has been removed and stored should determine a final state in less than a minute. However, if stored needs to employ recovery techniques such as recovering from a snapshot, the process may take a few minutes.

After most recoveries, the database will usually be up-to-date and nothing else will be required. However, some recoveries will require a reconstruct -m in order to synchronize redundant data in the message store. Again, this will be stated in the default log, so it is important to monitor the default log after a startup. Even though the message store will seem to be up and running normally, it is important to run any requested operations such as reconstruct.

Another reason for reading the log file is to determine what caused damage to the database in the first place. Although stored is designed to bring up the message store regardless of any problem on the system, you will still want to try to ascertain cause of the database damage as this may be a sign of a larger hidden problem.

Error Messages Signifying that reconstruct -m is Needed

This section describes the type of error messages that require reconstruct -m to be run.

When the error message indicates mailbox error, run reconstruct <mailbox>. Example:

"Invalid cache data for msg 102 in mailbox user/joe/INBOX. Needs reconstruct"

"Mailbox corrupted, missing fixed headers: user/joe/INBOX"

"Mailbox corrupted, start_offset beyond EOF: user/joe/INBOX"

When the error message indicates a database error, run reconstruct -m. Example:

"Removing extra database logs. Run reconstruct -m soon after startup to resync redundant data"

"Recovering database from snapshot. Run reconstruct -m soon after startup to resync redundant data"

Database Snapshots

A snapshot is a hot backup of the database and is used by stored to restore a broken database transparently in couple minutes. This is much quicker than using reconstruct, which relies on the redundant information stored in other areas.

Message Store Database Snapshot—Theory of Operations

Snapshots of the database, located in the mboxlist directory, are taken automatically, by default, once every 24 hours. Snapshots are copied by default into a subdirectory of the store directory. By default, there are five snapshots kept at any given time: one live database, three snapshots, and one database/removed copy. The database/removed copy is newer and is an emergency copy of the database which is thrown into a subdirectory removed of the mboxlist database directory.

If the recovery process decides to remove the current database because it is determined to be bad, stored will move it into the removed directory if it can. This allows the database to be analyzed if desired.

The data move will only happen once a week. If there is already a copy of the database there, stored will not replace it every time the store comes up. It will only replace it if the data in the removed directory is older than a week. This is to prevent the original database which had the problem of being replaced too soon by successive startups.

To Specify Message Store Database Snapshot Interval and Location

There should be five times as much space for the database and snapshots combined. It is highly recommended that the administrator reconfigure snapshots to run on a separate disk, and that it is tuned to the system’s needs.

If stored detects a problem with the database on startup, the best snapshot will automatically be recovered. Three snapshot variables can set the following parameters: the location of the snapshot file, the interval for taking snapshots, number of snapshots saved. These configutil parameters are shown in Table 18–15.

Having a snapshot interval which is too small will result in a frequent burden to the system and a greater chance that a problem in the database will be copied as a snapshot. Having a snapshot interval too large can create a situation where the database will hold the state it had back when the snapshot was taken.

A snapshot interval of a day is recommended and a week or more of snapshots can be useful if a problem remains on the system for a number of days and you wish to go back to a period prior to point at which the problem existed.

stored monitors the database and is intelligent enough to refuse the latest snapshot if it suspects the database is not perfect. It will instead retrieve the latest most reliable snapshot. Despite the fact that a snapshot may be retrieved from a day ago, the system will use more up to date redundant data and override the older snapshot data, if available.

Thus, the ultimate role the snapshot plays is to get the system as close to up-to-date and ease the burden of the rest of the system trying to rebuild the data on the fly.

Table 18–15 Message Store Database Snapshot Parameters

Parameter  

Description  

local.store.snapshotpath

Location of message store database snapshot files. Either existing absolute path or path relative to the store directory.

Default: dbdata/snapshots

local.store.snapshotinterval

Minutes between snapshots. Valid values: 1 - 46080 

Default: 1440 (1440 minutes = 1 day) 

local.store.snapshotdirs

Number of different snapshots kept. Valid values: 2 -367 

Default: 3 

Repairing Mailboxes and the Mailboxes Database

If one or more mailboxes become corrupt, you can use the reconstruct utility to rebuild the mailboxes or the mailbox database and repair any inconsistencies.

The reconstruct utility rebuilds one or more mailboxes, or the master mailbox file, and repairs any inconsistencies. You can use this utility to recover from almost any form of data corruption in the mail store. See Error Messages Signifying that reconstruct -m is Needed

Table 18–16 lists the reconstruct options. For detailed syntax and usage requirements, see the reconstruct in Sun Java System Messaging Server 6 2005Q4 Administration Reference.

Table 18–16 reconstruct Options

Option  

Description  

-e

Removes the store.exp file before reconstructing. This eliminates any internal store record of removed messages which have not been cleaned out by the store process. It would also be useful to use the -f option when using -i or -e, because these options only work if the folder is actually reconstructed. Similarly, if you use the -n option (which performs a check, not a reconstruction), the -i and -e options do not work.

Running a reconstruct -e will not recover removed messages if reconstruct does not detect damage. An -f will force the reconstruct.

-i

Sets the store.idx file length to zero before reconstructing. It would also be useful to use the -f option when using -i or -e, because these options only work if the folder is actually reconstructed. Similarly, if you use the -n option (which performs a check, not a reconstruction), the -i and -e options do not work.

-f

Forces reconstruct to perform a fix on the mailbox or mailboxes.

-l

Reconstruct lright.db.

-m

Performs a consistency check and, if needed, repairs the mailboxes database. This option examines every mailbox it finds in the spool area, adding or removing entries from the mailbox’s database as appropriate. The utility prints a message to the standard output file whenever it adds or removes an entry from the database. Specifically it fixes folder.db, quota.db, and lright.db

-n

Checks the message store only, without performing a fix on the mailbox or mailboxes. The -n option cannot be used by itself unless a mailbox name is provided. When a mailbox name is not provided, the -n option must be used with the -r option. The -r option may be combined with the -p option. For example, any of the following commands are valid:

reconstruct -n user/dulcinea/INBOX

reconstruct -n -r

reconstruct -n -r -p primary

reconstruct -n -r user/dulcinea/

-o

Obsolete, see mboxutil -o

-o -d filename

Obsolete, see mboxutil -o

-p partition

The -p option is used with the -m option and limits the scope of the reconstruction to the specified partition. If the -p option is not specified, reconstruct defaults to all partitions. Specifically it fixes folder.db and, quota.db, but not lright.db. This is because fixing the lright.db requires scanning the acls for every user in the message store. Performing this for every partition is not very efficient. To fix lright.db run reconstruct -l.

Specify a partition name; do not use a full path name. 

-q

Fixes any inconsistencies in the quota subsystem, such as mailboxes with the wrong quota root or quota roots with the wrong quota usage reported. The -q option can be run while other server processes are running.

-r [mailbox]

Repairs and performs a consistency check of the partition area of the specified mailbox or mailboxes. The -r option also repairs all sub-mailboxes within the specified mailbox. If you specify -r with no mailbox argument, the utility repairs the spool areas of all mailboxes within the user partition directory.

-u user

The -u option is used with the -m option and limits the scope of the reconstruction to the specified user. The -u option must be used with the -p option. If the -u option is not specified, reconstruct defaults to all partitions or to the partition specified with the -p option.

Specify a user name; do not use a full path name. 

To Rebuild Mailboxes

To rebuild mailboxes, use the -r option. You should use this option when:

reconstruct -r first runs a consistency check. It reports any consistencies and rebuilds only if it detects any problems. Consequently, performance of the reconstruct utility is improved with this release.

You can use reconstruct as described in the following examples:

To rebuild the spool area for the mailboxes belonging to the user daphne, use the following command:

reconstruct -r user/daphne

To rebuild the spool area for all mailboxes listed in the mailbox database:

reconstruct -r

You must use this option with caution, however, because rebuilding the spool area for all mailboxes listed in the mailbox database can take a very long time for large message stores. (See reconstruct Performance.) A better method for failure recovery might be to use multiple disks for the store. If one disk goes down, the entire store does not. If a disk becomes corrupt, you need only rebuild a portion of the store by using the -p option as follows:

reconstruct -r -p subpartition

To rebuild mailboxes listed in the command-line argument only if they are in the primary partition:

reconstruct -p primary mbox1 mbox2 mbox3

If you do need to rebuild all mailboxes in the primary partition:

reconstruct -r -p primary

If you want to force reconstruct to rebuild a folder without performing a consistency check, use the -f option. For example, the following command forces a reconstruct of the user folder daphne:

reconstruct -f -r user/daphne

To check all mailboxes without fixing them, use the -n option as follows:

reconstruct -r -n

Checking and Repairing Mailboxes

To perform a high-level consistency check and repair of the mailboxes database:

reconstruct -m

To perform a consistency check and repair of the primary partition:

reconstruct -p primary -m

Note –

Running reconstruct with the -p and -m flags together will not fix lright.db. This is because fixing the lright.db requires scanning the ACLs for every user in the message store. Performing this for every partition is not very efficient. To fix the lright.db run reconstruct -l


To perform a consistency check and repair of an individual user’s mailbox named john:

reconstruct -p primary -u john -m

You should use the -m option when:

reconstruct Performance

The time it takes reconstruct to perform an operation depends on the following factors:

The reconstruct -r option performs an initial consistency check; this check improves reconstruct performance depending on how many folders must be rebuilt.

The following performance was found with a system with approximately 2400 users, a message store of 85GB, and concurrent POP, IMAP, or SMTP activity on the server:


Note –

A reconstruct operation may take significantly less time if the server is not performing ongoing POP, IMAP, HTTP, or SMTP activity.


Common Problems and Solutions

This section lists common message store problems and solutions:

Messenger Express or Communications Express Not Loading Mail Page

If the user cannot load any Messenger Express pages or the Communications Express mail page, the problem may be that the data is getting corrupted after compression. This can sometimes happen if the system has deployed a outdated proxy server. To solve this problem, try setting local.service.http.gzip.static and local.service.http.gzip.dynamic to 0 to disable data compression. If this solves the problem, you may want to update the proxy server.

Command Using Wildcard Pattern Doesn’t Work

Some UNIX shells may require quotes around wildcard parameters and some will not. For example, the C shell tries to expand arguments containing wild cards (*, ?) as files and will fail if no match is found. These pattern matching arguments may need to be enclosed in quotes to be passed to commands like mboxutil.

For example:

mboxutil -l -p user/usr44*

will work in the Bourne shell, but will fail with tsch and the C shell. These shells would require the following:

mboxutil -l -p "user/usr44*"

If a command using a wildcard pattern doesn’t work, verify whether or not you need to use quotes around wildcards for that shell.

Unknown/invalid Partition

A user can get the message “Unknown/invalid partition” in Messenger Express if their mailbox was moved to a new partition which was just created and Messaging Server was not refreshed or restarted. This problem only occurs on new partitions. If you now add additional user mailboxes to this new partition, you will not have to do a refresh/restart of Messaging Server.

User Mailbox Directory Problems

A user mailbox problem exists when the damage to the message store is limited to a small number of users and there is no global damage to the system. The following guidelines suggest a process for identifying, analyzing, and resolving a user mailbox directory problem:

  1. Review the log files, the error messages, or any unusual behavior that the user observes.

  2. To keep debugging information and history, copy the entire store_root/mboxlist/ user directory to another location outside the message store.

  3. To find the user folder that might be causing the problem, run the command reconstruct -r -n. If you are unable to find the folder using reconstruct, the folder might not exist in the folder.db.

    If you are unable to find the folder using the reconstruct -r -n command, use the hashdir command to determine the location. For more information on hashdir, see The hashdir Utility and the hashdir utility in the Messaging Server Command-line Utilities chapter of the Sun Java System Messaging Server 6 2005Q4 Administration Reference.

  4. Once you find the folder, examine the files, check permissions, and verify the proper file sizes.

  5. Use reconstruct -r (without the -n option) to rebuild the mailbox.

  6. If reconstruct does not detect a problem that you observe, you can force the reconstruction of your mail folders by using the reconstruct -r -f command.

  7. If the folder does not exist in the mboxlist directory (store_root/mboxlist), but exists in the partition directory store_root/partition), there might be a global inconsistency. In this case, you should run the reconstruct -m command.

  8. If the previous steps do not work, you can remove the store.idx file and run the reconstruct command again.


    Caution – Caution –

    You should only remove the store.idx file if you are sure there is a problem in the file that the reconstruct command is unable to find.


  9. If the issue is limited to a problematic message, you should copy the message file to another location outside of the message store and run the command reconstruct -r on the mailbox/ directory.

  10. If you determine the folder exists on the disk (store_root/partition/ directory), but is apparently not in the database (store_root/mboxlist/ directory), run the command reconstruct -m to ensure message store consistency.

For more information on the reconstruct command, see Repairing Mailboxes and the Mailboxes Database

Store Daemon Not Starting

If stored won’t start and returns the following error message:


# msg_svr_base/sbin/start-msg

msg_svr_base: Starting STORE daemon ...Fatal error: Cannot
find group in name service

This indicates that the UNIX group configured in local.servergid cannot be found. Stored and others need to set their gid to that group. Sometimes the group defined by local.servergid gets inadvertently deleted. In this case, create the deleted group, add inetuser to the group, change ownership of the instance_root and its files to inetuser and the group.

Migrating or Moving Mailboxes to a New System

It is sometimes necessary to move existing mailboxes from one messaging server system to a another messaging server system. This commonly occurs in the following circumstances:

Messaging Server provides several ways to move mailboxes from one system to another. Each method has its advantages and disadvantages. These methods are described in Migrating User Mailboxes.