Sun Java System Messaging Server 6.3 Administration Guide

Chapter 20 Managing the Message Store

This chapter describes the message store and its administration interface. This chapter contains the following sections:

20.1 Overview

The message store contains the user mailboxes for a particular Messaging Server instance. The size of the message store increases as the number of mailboxes, folders, and log files increase. You can control the size of the store by specifying limits on the size of mailboxes (disk quotas), by specifying limits on the total number of messages allowed, and by setting aging policies for messages in the store.

As you add more users to your system, your disk storage requirements increase. Depending on the number of users your server supports, the message store might require one physical disk or multiple physical disks. There are two ways to integrate this additional disk space into your system. The easiest way is to add additional message store partitions (see 20.10 Configuring Message Store Partitions)

Likewise, if you are supporting multiple hosted domains, you might want to dedicate a server instance to a single, large domain. With this configuration, you can designate a store administrator for a particular domain. You can also expand the message store by adding more partitions.

To manage the message store, Messaging Server provides a set of command-line utilities described in Table 20–1. For information about using these utilities, see 20.11 Performing Message Store Maintenance Procedures and the Sun Java System Messaging Server 6.3 Administration Reference.

Table 20–1 Message Store Command-line Utilities

Utility 

Description  

configutil

Sets and modifies configuration parameters for the store. 

deliver

Delivers mail directly to the message store accessible by IMAP or POP mail clients. 

hashdir

Identifies the directory that contains the message store for a particular user. 

imsconnutil

Monitors user access of the message store. 

imexpire

Automatically removes messages from the message store based on administrator-specified criteria like age. 

iminitquota

Reinitializes the quota limit from the LDAP directory and recalculates the disk space being used. 

imsasm

Handles the saving and recovering of user mailboxes. 

imsbackup

Backs up stored messages. 

imsexport

Exports Messaging Server mailboxes into UNIX /var/mail format folders.

imsrestore

Restores messages that have been backed up. 

imscripter

The IMAP server protocol scripting tool. Executes a command or sequence of commands. 

mboxutil

Lists, creates, deletes, renames, or moves mailboxes; reports quota usage. 

mkbackupdir

Creates and synchronizes the backup directory with the information in the message store. 

MoveUser

Moves a user’s account from one messaging server to another. 

imquotacheck

Calculates the total mailbox size for each user in the message store and compares the size with their assigned quota. Localized versions of imquotacheck notification incorrectly convert the % and the $ signs. To correct the encoding, replace every $ with \24 and replace every % with \25 in the message file.

readership

Collects readership information on shared IMAP folders. 

reconstruct

Reconstructs mailboxes that have been damaged or corrupted. 

stored

Performs background and daily tasks, expunges, and erases messages stored on disk. 

20.2 Message Store Directory Layout

Figure 20–1 shows the message store directory layout for a server instance. The message store is designed to provide fast access to mailbox contents. The store directories are described in Table 20–2.

Figure 20–1 Message Store Directory Layout

Graphic shows Message Store directory layout.

The message store consists of a number of mailbox databases and the user mailboxes. The mailbox databases consists of information about users, mailboxes, partitions, quotas and other message store related data. The user mailboxes contain the user’s messages and folders. Mailboxes are stored in a message store partition, an area on a disk partition specifically devoted to storing the message store. See 20.10 Configuring Message Store Partitions for details. Message store partitions are not the same as disk partitions, though for ease of maintenance, we recommend having one disk partition for each message store partition.

Mailboxes such as INBOX are located in the store_root. For example, a sample directory path might be:

store_root/partition/primary/=user/53/53/=mack1

The table below describe the message store directory.

Table 20–2 Message Store Directory Description

Location  

Content/Description  

msg-svr-base

Default: /opt/SUNWmsgsr

The directory on the Messaging Server machine that holds the server program, configuration, maintenance, and information files. 

store_root

msg-svr-base/data/store

Top-level directory of the message store. Contains the mboxlist, user, and partition subdirectories.

./store.expirerule

Contains the automatic message removal rules (expire rules). This optional file can be at different locations. See 20.9 To Set the Automatic Message Removal (Expire and Purge) Feature

store_root/dbdata/snapshots

Message store database backup snapshots that stored makes periodically.

store_root/mboxlist/

Contains mailbox database, database (Berkeley DB) that stores information about the mailboxes and quota information. 

folder.db contains information about mailboxes, including the name of the partition where the mailbox is stored, the ACL, and a copy of some of the information in store.idx. There is one entry in folder.db per mailbox

quota.db contains information about quotas and quota usage. There is one entry in quota.db per user.

lright.db - an index for the folders by acl lookup rights.

peruser.db contains information about per-user flags. The flags indicate whether a particular user has seen or deleted a message.

subscr.db contains information about user subscriptions.

store_root/session/

Contains active message store process information. 

store_root/user/

Not used. 

store_root/partition/

Contains the message store partitions. A default primary partition is created. Place any other partitions you define in this directory.

store_root/partition/primary/=user/

Contains all the user mailboxes in the subdirectory of the partition. The mailboxes are stored in a hash structure for fast searching. To find the directory that contains a particular user’s mailbox, use the hashdir utility.

.../=user/hashdir/hashdir/userid/

The top-level mail folder for the user whose ID is userid. This contains the user’s INBOX. For the default domain, userid is uid. For hosted domains, userid is uid@domain. A user's incoming messages are delivered to the INBOX here.

.../userid/folder

A user-defined mailbox on the messaging server. 

.../userid/store.idx

An index that provides the following information about mail stored in the /userid/ directory: number of messages, disk quota used by this mailbox, the time the mailbox was last appended, message flags, variable-length information for each message including the headers and the MIME structure, and the size of each message. The index also includes a backup copy of mboxlist information for each user and a backup copy of quota information for each user.

.../userid/store.usr

Contains a list of users who have accessed the folder. For each user listed, contains information about the last time the user accessed the folder, the list of messages the user has seen, and the list of messages the user has deleted. 

.../userid/store.sub

Contains information about user subscriptions. 

.../userid/store.exp

Contains a list of message files that have been expunged, but not removed from disk. This file appears only if there are expunged messages. 


.../userid/nn/
or
.../userid/folder/nn/

nn is a hash directory that contains messages in the format message_id.msg; nn can be a number from 00 to 99. message_id is also a number. Example: messages 1 through 99 are stored in the .../00 directory. The first message is 1.msg, the second is 2.msg, third 3.msg, and so on. Messages 100 through 199 are stored in the 01 directory; messages 9990 through 9999 are stored in the 99 directory; messages 10000 through 10099 are in the 00 directory, and so on.

20.2.1 Valid UIDs and Folder Names

The following definitions will assist in this discussion:

message store user ID — A mail user's unique identifier in the message store. In the default domain, this is the same as the user's uid attribute in LDAP. In hosted domains this is uid@domain where uid is the uid LDAP attribute and domain is the canonical domain name.

message store mailbox name for commands — Some message store commands require that you specify a mailbox name. The required form of the name is user/userid/mailbox where userid is the message store user ID (see above) and mailbox is a user's mailbox. Specifying INBOX sometimes implies all the user's mailboxes in the message store. For example, the following command:

mboxutil -d user/joe/INBOX

will remove the INBOX and all the folders of user joe. Note that in the context of message stores, folders and mailboxes are synonymous.

Valid UIDs. Valid and invalid UID characters are controlled separately by the MTA and message store mechanisms. That means UID character limitations are specified by the union of MTA and message store limitations. The following characters and strings are invalid as UIDs in the message store:

The following characters are invalid in UIDs in the MTA:

<space> $~=#*+%!@,{}()/\\<>;:\"`[]&?"

The list of characters forbidden by the MTA can be modified by setting the option.dat parameter LDAP_UID_INVALID_CHARS with a string of the forbidden characters using decimal ASCII values, however, you are strongly advised not to change the default constraint. The default setting is as follows and reflect the characters listed above:


DAP_UID_INVALID_CHARS=32,33,34,35,36,37,38,40,41,42,43,44,47,58,59,60,61,
62,63,64,91,92,93,96,123,125,126

LDAP_UID_INVALID_CHARS=32,33,34,35,36,37,38,40,41,42,43,44,47,58,59,60,61,
62,63,64,91,92,93,96,123,125,126

Valid mail folder names. The following characters are invalid as folder names:

% * ? and ASCII values less than 20 or greater than 7E hexidecimal (see man ascii).

In addition, folder names must be valid MUTF-7 sequences.

20.3 How the Message Store Removes Messages

Messages are removed from the message store in three stages:

  1. Delete. A client sets a message flag to delete. At this point, the message is marked for removal, but client can still restore the message by removing the delete flag. If there is a second client, the deleted flag may not be recognized immediately by that second client. You can set the configutil parameter local.imap.immediateflagupdate to enable immediate flag update.

  2. Expunge. Messages are removed from the mailbox. Technically, they are removed from the message store index file, store.idx. The message itself is still on disk, but once messages are expunged, clients can no longer restore them.

    Expire is a special case of expunge. Messages that conform to a set of administrator-defined removal criteria such as message size, age and so forth, are expunged. See 20.9 To Set the Automatic Message Removal (Expire and Purge) Feature

  3. Purge. The imexpire utility purges from the disk any messages that have been expunged at 11PM everyday by default. This can be configured with local.schedule.expire, which controls the message expire schedule, and store.cleanupage, which controls the purge grace period (period of time before which the message will not be purged). Note that this is different from the imsimta purge command and configutil parameter local.schedule.purge which purges older versions of the MTA log files.

20.4 Specifying Administrator Access to the Store

Message store administrators can view and monitor user mailboxes and specify access control for the message store. Store administrators have proxy authentication privileges to any service (POP, IMAP, HTTP, or SMTP), which means they can authenticate to any service using the privileges of any user. These privileges allow store administrators to run certain utilities for managing the store. For example, using MoveUser, store administrators can move user accounts and mailboxes from one system to another.

This section discusses how to grant store privileges to the message store for your Messaging Server installation.


Note –

Other users might also have administrator privileges to the store. For example, some administrators may have these privileges.


You can perform administrator tasks as described in the following subsections:

ProcedureTo Add an Administrator Entry

  1. Command Line:To add an administrator entry at the command line:

    configutil -o store.admins -v "adminlist"

    where adminlist is a space-separated list of administrator IDs. If you specify more than one administrator, you must enclose the list in quotes. In addition, the administrator must be a member of the Service Administrator Group (in the LDAP user entry: memberOf: cn=Service Administrators,ou=Groups,o=usergroup). You must restart imapd for the system to recognize the change in store.admins.

ProcedureTo Modify an Administrator Entry

  1. Command Line. To modify an existing entry in the message store Administrator UID list at the command line:


    configutil -o store.admins -v "adminlist"

    where adminlist is a space-separated list of administrator IDs. If you specify more than one administrator, you must enclose the list in quotes. In addition, the administrator must be a member of the Service Administrator Group (in the LDAP user entry: memberOf: cn=Service Administrators,ou=Groups,o=usergroup).

    You must restart imapd for the system to recognize the change in store.admins.

ProcedureTo Delete an Administrator Entry

  1. Command Line.To delete store administrators at the command line, you can edit the administrator list as follows:


    configutil -o store.admins -v "adminlist"

    where adminlist is a space-separated list of administrator IDs. If you specify more than one administrator, you must enclose the list in quotes. In addition, the administrator must be a member of the Service Administrator Group (in the LDAP user entry: memberOf: cn=Service Administrators,ou=Groups,o=usergroup).

    You must restart imapd for the system to recognize the change in store.admins.

20.4.1 To Protect Mailboxes from Deletion or Renaming Except by an Administrator

You may wish to protect some mailboxes from deletion or modification except by the Administrator. The following procedures describes how to do this. If someone other than an Administrator attempts to delete, modify or rename a protected mailbox, the error message mailbox is pinned is displayed.

Set the local.store.pin configutil variable. using the following format:


configutil -o local.store.pin -v "mailbox1"%"mailbox2"%"mailbox 3"

where mailbox1, mailbox2, and mailbox 3 are the mailboxes to be protected (note that spaces can be used in mailbox names), and % is the separator between each mailbox.

20.5 About Shared Folders

A group or shared folder is like any other mail folder except that other users and groups can read, delete, or add messages to it depending on the permissions given. Messages can be added to shared folders by normal drag and drop, by Sieve filters, or by sending messages directly using the form: uid+folder@domain.

The example below shows the address for sending email to a private shared folder owned by carol.fanning@siroe.com called crafts_club:

carol.fanning+crafts_club@siroe.com

This example shows the address for sending email to a public shared folder called tennis@siroe.com.

public+tennis@siroe.com

Shared folders are useful for starting, sharing, and archiving an ongoing email conversation on a particular topic. For example, a group of software developers can create a shared folder for discussing development of a particular project called mosaic_voices. When a message is sent or dropped into the folder mosaic_voices, anyone who has permissions to the shared folder (permissions can added by individual addresses or by group addresses) can open this mailbox and read the message.

Shared folders are displayed in user's mailbox tree under a folder called Shared Folders. An example is shown below.

Figure 20–2 Example of Shared Mail Folder List as Seen from a Mail Client

Graphic shows example of client shared mail folder list.

There are two kinds of shared folders:

Normally shared folders are only available to users on a particular message store. Messaging Server, however, allows you to create special shared folders that can be accessed across multiple message stores. These are called distributed shared folders. See 20.6.4 To Set Up Distributed Shared Folders for details.

20.6 Shared Folder Tasks

This section describes the shared folder administrator tasks:

ProcedureTo Specify Sharing Attributes for Private Shared Folders

  1. Private shared folders are created by the user.

    Many mail clients support the creation of private shared folders. You can try this out onCommunications Express.

  2. Set the sharing parameters for private shared folders.

    The following configutil parameters are supported:

    store.privatesharedfolders.restrictanyone - If enabled (1), disallow regular users from setting the permission on private shared folders to anyone. Default: 0

    store.privatesharedfolders.restrictdomain - If enabled (1), disallow regular users sharing private folders to users outside of their domain. Default: 0

    store.privatesharedfolders.shareflags - If 0, flags cannot be shared across users. If 1, flags can be shared across users. Default: 0

    store.publicsharedfolders.user - Public shared folder owner's userid. Typically, this is simply public. Default: NULL (unset)

ProcedureTo Create a Public Shared Folder

Public folders must be created by system administrators because they require access to the LDAP database as well as the readership command.

  1. Create an LDAP user entry called public that will act as a container for all public folders (see 20.5 About Shared Folders).

    Example:


    dn: cn=public,ou=people,o=sesta.com,o=ISP
    objectClass: person
    objectClass: organizationalPerson
    objectClass: inetOrgPerson
    objectClass: inetUser
    objectClass: ipUser
    objectClass: inetMailUser
    objectClass: inetLocalMailRecipient
    objectClass: nsManagedPerson
    objectClass: userPresenceProfile
    cn: public
    mail: public@sesta.com
    mailDeliveryOption: mailbox
    mailHost: manatee.siroe.com
    uid: public
    inetUserStatus: active
    mailUserStatus: active
    mailQuota: -1
    mailMsgQuota: 100
                      
  2. Create folders within the public account by using the mboxutil command line utility.

    For example, create a public folder called gardening:


    mboxutil -c user/public/gardening
  3. Set the name of the folder.

    Typically, this is public. Here's the command for setting the folder name to public:

    configutil -o store.publicsharedfolders.user —v public

  4. Specify the users and their access rights to the shared folder.

    Use the readership command to specify users and their access rights. For example the following command gives everyone at sesta.com lookup, read, and posting access to the public folder gardening:

    readership -s user/public/gardening anyone@sesta.com lrp

    For detailed instructions on how to user readership, see 20.6.2 To Set or Change a Shared Folder’s Access Control Rights

20.6.1 To Add Shared Folders with an Email Group

Shared folders are typically created by adding users to a shared folder list with Communications Express, or by creating public shared folders as described earlier. Sometimes, however, users may wish to add an email group (mail distribution list) to a shared folder list so that everyone in the group will have access to the shared folder. For example, a group called tennis@sesta.com has 25 members and the members have decided that they would like to create a shared folder to store all email going to this group address.

ProcedureTo Add an Email Group to a Shared Folder

Adding an email group to a shared folder requires System Administrator privileges.

  1. Create a folder. (If this has already been done, then skip this step. )

    Typically this should be done by one of the members of the group. If it’s not, you can create it for them using the following command:

    mboxutil -c user/gregk/gardening

    gregk is the uid of the shared folder owner. gardening is the name of the shared folder.

  2. Add the attribute-value pair aclGroupAddr group_name@domain to the user entry of every member who will have access to the group shared folder.

    Using the example above, add the following attribute-value pair to each user entry receiving access to the shared folder:

    aclGroupAddr: tennis@sesta.com

    Note that members will already have this attribute if the group was created dynamically using the memberURL attribute in the group entry. URL value for this attribute would look like this:


    memberURL: ldap:///o=sesta.com??sub?(&(aclGroupAddr=tennis@sesta.com)
    (objectclass=inetmailuser))

    (The sample entry line has been wrapped for typographic reasons. The actual entry should appear on one physical line.)

  3. Specify the group and the access rights to the shared folder.

    Use the readership command to do this. Using the example above the following command gives members of tennis@sesta.com lookup, read, and posting access to the public folder gardening:

    readership -s user/gregk/gardening tennis@sesta.com lrp

    For detailed instructions on how to user readership, see 20.6.2 To Set or Change a Shared Folder’s Access Control Rights

20.6.2 To Set or Change a Shared Folder’s Access Control Rights

Users can set or change the access control for a shared folder using the Communications Express interface. Administrators can set or change the access control for a shared folder using the readership command line utility. The command has the following form:

readership -s foldername identifier rights_chars

where foldername is the name of the public folder for which you are setting rights, identifier is the person or group to whom you are assigning the rights, and rights_chars are the rights you are assigning. For the meaning of each character, see Table 20–3.


Note –

anyone is a special identifier. The access rights for anyone apply to all users. Similarly, the access rights for anyone@domain apply to all users in the same domain.


Table 20–3 ACL Rights Characters

Character  

Description  

l

lookup– User can see and subscribe to the shared folder. (IMAP commands allowed: LIST and LSUB)

r

read– Users can read the shared folder. (IMAP commands allowed: SELECT, CHECK, FETCH, PARTIAL, SEARCH, COPY from the folder)

s

seen– Directs the system to keep seen information across sessions. (Set IMAP STORE SEEN flag)

w

write– Users can mark as read, and delete messages. (Set IMAP STORE flags, other than SEEN and DELETED)

i

insert– Users can copy and move email from one folder to another. (IMAP commands allowed: APPEND, COPY into folder)

p

post– Users can send mail to the shared folder email address. (No IMAP command needed) 

c

create– Users can create new sub-folders. (IMAP command allowed: CREATE)

d

delete– Users can delete entries from the shared folder. (IMAP commands allowed: EXPUNGE, set STORE DELETED flag)

a

administer– Users have administrative privileges. (IMAP command allowed: SETACL)

20.6.2.1 Examples

If you wish everyone at the sesta domain to have lookup, read and email marking (but not posting) access to the public folder called golftournament, issue the command as follows:

readership -s User/public/golftournament anyone@sesta lwr

To assign the same access to everyone on the message store issue the following:

readership -s User/public/golftournament anyone lwr

To assign lookup, read, email marking and posting rights to a group, issue the command as follows:

readership -s User/public/golftournament group=golf@sesta.com lwrp

If you want to assign administrator and posting rights for this folder to an individual, jdoe, issue the command as follows:

readership -s User/public/golftournament jdoe@sesta.com lwrpa

To deny an individual or group access to a public folder, prefix the userid with a dash. For example, to deny lookup, read and write rights to jsmith, issue the command as follows:

readership -s User/public/golftournament -jsmith@sesta.com lwr

To deny an individual or group an access right, prefix the ACL rights character with a dash. For example, to deny posting rights to jsmith, issue the command as follows:

readership -s User/public/golftournament jsmith@sesta.com -p


Note –

Posting messages to a shared folder using the uid+folder@domain address requires that the p (post) access right be used with the readership command. See 20.6.2 To Set or Change a Shared Folder’s Access Control Rights


20.6.3 To Enable or Disable Listing of Shared Folders

The server will or will not return shared folders when responding to a LIST command depending on the setting in the configuration option local.store.sharedfolders. Setting the option to off disables it. The setting is enabled by default (set to on).

SELECT and LSUB commands are not affected by this option. The LSUB command returns every subscribed folder, including shared folders. Users can SELECT the shared folders they own or are subscribed to.

20.6.4 To Set Up Distributed Shared Folders

Normally shared folders are only available to users on a particular message store. Messaging Server, however, allows you to create distributed shared folders that can be accessed across multiple message stores. That is, access rights to distributed shared folders can be granted to any users within the group of message stores. Note, however, that web mail clients (HTTP access clients like Messenger Express) do not support remote shared folders access. Users can list and subscribe to the folders, but they can’t view or alter the contents.

Distributed shared folders require the following:

The remote message stores (that is the message stores that do not hold the shared folder) must be configured as proxy servers by setting the configuration variables listed in Table 20–4.

Table 20–4 Variables for Configuring Distributed Shared Folders

Name  

Value  

Data Format  

local.service.proxy.serverlist

message store server list 

space-separated strings 

local.service.proxy.admin

default store admin login name 

string 

local.service.proxy.adminpass

default store admin password 

string 

local.service.proxy.admin.hostname

store admin login name for a specific host 

string 

local.service.proxy.adminpass.hostname

store admin password for a specific host 

string 

20.6.4.1 Setting Up Distributed Shared Folders—Example

Figure 20–3 shows a distributed folder example of three message store servers called StoreServer1, StoreServer2, and StoreServer3.

Figure 20–3 Distributed Shared Folders—Example

Graphic shows example of distributed shared folders.

These servers are connected to each other as peer proxy message stores by setting the variables shown in Table 20–4. Each server has a private shared folder—golf (owned by Han), tennis (owned by Kat), and hurling (owned by Luke). In addition there are two public shared folders called press_releases and Announcements. Users on any of the three servers can access any of these three shared folders. Figure 20–2shows Ed's shared folder list. Below is an example of the ACLs for each server in this configuration.


$ StoreServer1 :> imcheck -d lright.db
Ed: user/Han/golf 
Ian: user/Han/golf 
anyone: user/public/press_releases

            

$ StoreServer2 :> imcheck -d lright.db
Jan: user/Kat/tennis
Ann: user/Kat/tennis
anyone: user/public+Announcements user/public+press_releases

            

$ StoreServer3 :> imcheck -d lright.db
Tuck: user/Ian/hurling
Ed: user/Ian/hurling 
Jac: user/Ian/hurling 
anyone: user/public/Announcements

            

20.6.5 To Monitor and Maintain Shared Folder Data

The readership command line utility allows you to monitor and maintain shared folder data which is held in the folder.db, peruser.db, and lright.db files. folder.db has a record for each folder that holds a copy of the ACLs. The peruser.db has an entry per user and mailbox that lists the various flags settings and the last date the user accessed any folders. The lright.db has a list of all the users and the shared folders for which they have lookup rights.

The readership command line utility takes the following options:

Table 20–5 readership Options

Options  

Description  

-d days

Returns a report, per shared folder, of the number of users who have selected the folder within the specified days. 

-p months

Removes data from the peruser.db for those users who have not selected their shared folders within the specified months.

-l

List the data in lright.db.

-s folder_identifier_rights

Set access rights for the specified folder. This updates the lright.db as well as the folder.db.

Using the various options, you can perform the following functions:

20.6.5.1 To Monitor Shared Folder Usage

To find out how many users are actively accessing shared folders, issue the command:

readership -d days

where days is the number of days to check. Note that this option returns the number of active users, not a list of the active users.

Example: To find out the number of users who have selected shared folders within the last 30 days, issue the following command:

readership -d 30

20.6.5.2 To List Users and Their Shared Folders

To list users and the shared folders to which they have access, issue the command:

imcheck -d lright.db

Example output:

$ imcheck -d lright.db
group=lee-staff@siroe.com: user/user2/lee-staff
richb: user/golf user/user10/Drafts user/user2/lee-staff user/user10/Trash
han1: user/public+hurling@siroe.com user/golf
gregk: user/public+hurling@siroe.com user/heaving user/tennis

20.6.5.3 To Remove Inactive Users

If you want to remove inactive users (those who have not accessed shared folders in a specified time period) issue the command:

readership -p months

where months is the number of months to check for.

Example: Remove users who have not accessed shared folders for the past six months:

readership -p 6

20.6.5.4 To Set Access Rights

You can assign access rights to a new public folder, or change access rights on a current public folder.

For an example of how to set access rights with this command, see 20.6.2 To Set or Change a Shared Folder’s Access Control Rights

20.7 Managing Message Types

This section includes the following topics:

20.7.1 Message Type Overview

A unified messaging application can receive, send, store, and administer messages of many types, including text messages, voice mail, fax mail, image data, and other data formats. The message store allows you to define up to 63 different message types.

One method of manipulating messages by type is to organize the messages by their types into individual folders.

With the introduction of the message type feature, you do not have to maintain different message types in individual mailbox folders. Once you configure a message type, the message store can identify it, no matter where it is stored. Thus, you can store heterogeneous message types in the same folder. You also can perform the following tasks:

20.7.1.1 Planning the Message-Type Configuration

In a unified messaging application, data of heterogeneous formats are given standard internet message headers so that Messaging Server can store and manage the data. For example, when voice mail is sent to an end-user's phone, a telephone front-end system adds a message header to the incoming voice mail and delivers it to the message store.

In order to recognize and administer messages of different types, all components of the unified messaging system must use the same message-type definitions and the same header fields to identify the messages.

Before you configure the message store to support message types, you must

For example, if your application includes phone messages, you can define this message type as “multipart/voice-message” and use the Content-Type header field to identify message types.

You would then configure the telephone front-end system to add the following header information to each phone message to be delivered to the message store:

Content-Type: multipart/voice-message

Next, you would configure the message store to recognize the multipart/voice-message message type, as described in the sections that follow.

20.7.1.2 Defining and Using Message Types

You define a message type by giving it a unique definition such as multipart/voice-message. By default, the message store reads the Content-Type header field to determine the message-type. If you prefer, you can configure a different header field to identify the message types.

The message store reads the Content-Type (or other specified) header field, ignoring case. That is, the message store accepts the header field as valid even if the header's combination of uppercase and lowercase letters differs from the expected combination.

The message store reads only the message-type name in the header field. It ignores additional arguments or parameters.

To define a message type, use the configutil utility to set values for the store.messagetype parameters. For instructions, see To Configure Message Types.

Configuring a message type allows the message store to identify and manipulate messages of the specified type. It is the first, essential step in administering message types in a unified messaging application.

To take full advantage of the message-type features provided by the message store, you also should perform some or all of the following tasks:

These tasks are summarized in the following sections:

ProcedureTo Configure Message Types

To configure a message type, use the configutil utility to set the store.messagetype parameter values that define and identify the message type.

  1. Enable message types by setting the store.messagetype.enable parameter to on.

    This configutil parameter allows the message store to identify and manipulate message types. You must set this parameter before you can configure an individual message type.

    For example, enter the following command:


    configutil -o store.messagetype.enable -v 1
  2. Define and identify the message type by setting the store.messagetype.x parameter.

    The variable x identifies this particular message type in the message store. The variable x must be an integer greater than zero and less than 64. You can define up to 63 message types by iteratively configuring this parameter with unique integers.

    You define the value of the message type with a text string that describes the type.

    For example, to define a text message type, you could enter the following command:


    configutil -o store.messagetype.1 -v text/plain

    To define a voice message type, you could enter the following command:


    configutil -o store.messagetype.2 -v multipart/voice-message
  3. Provide a flag name for the message type by setting the store.messagetype.x.flagname parameter.

    This parameter creates a unique flag that identifies the message type. The flag is automatically set whenever a message of this type first arrives in the message store and remains associated with the message until it is purged. The flag name value is a text string that describes the message type. It does not have to be the same as the value set with the store.messagetype.x parameter.

    The variable x is the integer ID of the message type defined with the store.messagetype.x parameter.

    For example, to define flag names for the message types configured in the preceding step, enter the following commands:


    configutil -o store.messagetype.1.flagname -v text
    
    configutil -o store.messagetype.2.flagname -v voice_message
  4. Configure a quota root name for the message type by setting the store.messagetype.x.quotaroot parameter.

    This parameter enables the quota function to identify and manage a quota root for this message type. The parameter value is a name—a text string that describes the message type. It does not have to be the same as the value set with the store.messagetype.x parameter.

    The variable x is the integer ID of the message type defined with the store.messagetype.x parameter.

    When this parameter is configured, you can set a quota that applies to the specified message type. For more information, see 20.7.4 Administering Quotas by Message Type.

    For example, to enable the use of quota roots for the message types configured in the preceding steps, enter the following commands:


    configutil -o store.messagetype.1.quotaroot -v text
    
    configutil -o store.messagetype.2.quotaroot -v voice
  5. To configure an alternate header field for identifying the message type, set the store.messagetype.header parameter.

    By default, the message store reads the Content-Type header field to determine the message type. Configure the store.messagetype.header parameter only if you want to use a different header field for identifying the message type. The value of this parameter is a text string.

    For example, to use a field called X-Message-Type, enter the following command:


    configutil -o store.messagetype.header -v X-Message-Type

20.7.2 Message Types in IMAP Commands

When you configure the store.messagetype.x.flagname parameter for a message type, you create a unique flag that identifies the message type. This flag cannot be modified by end users.

Messaging Server presents the message-type flag as a user flag to IMAP clients. Mapping the message type to a user flag allows mail clients to use simple IMAP commands to manipulate messages by message type.

For example, you can perform the following operations:

The message-type user flag is read only. It cannot be modified by IMAP commands.

The following examples assume you configure the message-type configutil parameters with the values shown here:


store.messagetype.enable = yes

store.messagetype.1 = text/plain
store.messagetype.1.flagname = text
store.messagetype.1.quotaroot = text

store.messagetype.2 = multipart/voice-message
store.messagetype.2.flagname = voice_message
store.messagetype.2.quotaroot = voice

Example 20–1 IMAP FETCH Session Based on the Message-Type configutil Configurations

The following IMAP session fetches messages for the currently selected mailbox:


2 fetch 1:2 (flags rfc822)
* 1 FETCH (FLAGS (\Seen text) RFC822 {164}

Date: Wed, 8 July 2006 03:39:57 -0700 (PDT)
From: bob.smith@siroe.com
To: john.doe@siroe.com
Subject:  Hello
Content-Type: TEXT/plain; charset=us-ascii


* 2 FETCH (FLAGS (\Seen voice_message) RFC822 {164}

Date: Wed, 8 July 2006 04:17:22 -0700 (PDT)
From: sally.lee@siroe.com
To: john.doe@siroe.com
Subject:  Our Meeting
Content-Type: MULTIPART/voice-message; ver=2.0

2 OK COMPLETED

In the preceding example, two messages are fetched, one text message and one voice mail.

The message-type flags are displayed in the format configured with the store.messagetype.*.flagname parameter.

The Content-Type header fields identify the message types. The message-type names are displayed as they were received in the incoming messages. They use mixed uppercase and lowercase letters and include the message-type arguments such as charset=us-ascii.



Example 20–2 IMAP SEARCH Session Based on the Message-Type configutil Configurations

The following IMAP session searches for voice messages for the currently selected mailbox:


3 search keyword voice_message
* SEARCH 2 4 6 
3 OK COMPLETED

In the preceding example, messages 2, 4, and 6 are voice messages. The keyword used in the search is voice_message, the value of the store.messagetype.2.flagname parameter.


20.7.3 Sending Notification Messages for Message Types

Notifications can deliver status information about messages of different types, such as text messages, voice mail, and image data. Messaging Server uses Sun Java System Message Queue to send notification information for message types. For information about configuring the JMQ notification plug-in for Message Queue, see Chapter 22, Configuring the JMQ Notification Plug-in to Produce Messages for Message Queue

To enable the JMQ notification plug-in to recognize a particular message type, you must configure the store.messagetype parameters, including the store.messagetype.x.flagname parameter. For details, see To Configure Message Types.

Once the message types have been configured, JMQ notification messages can identify the particular message types. You can write a Message Queue client to interpret notification messages by message type and deliver status information about each type to the mail client.

The JMQ notification function counts the number of messages currently in the mailbox, by message type. Instead of sending one count, an array specifying the count for each message type is sent with the notification message.

For example, a NewMsg notification message can carry data to tell the user that, for example, there are seven new voice mail messages and four new text messages in the user's inbox.

For more information about sending notifications by message type, see 22.3.3 Notifications for Particular Message Types

20.7.4 Administering Quotas by Message Type

When you set a quota for a message type, you include that value in a quota root. A quota root specifies quotas for a user. It can specify different quotas for particular message types and mailbox folders, and it can specify a default quota that applies to all remaining message types, folders, and messages not defined by type.

For complete information about setting and managing quotas, see 20.8.2 Quota Theory of Operations

20.7.4.1 Before You Set Message-Type Quotas

Before you can set quotas for message types, you must configure the following parameters:

20.7.4.2 Methods of Setting Message-Type Quotas

Use one of the following methods to set quotas for message types:

When you set a quota for the message type with a configutil parameter or LDAP attribute shown above, you must use the quota root specified with the store.messagetype.x.quotaroot parameter.

20.7.4.3 Example of a Message-Type Quota Root

The example described in this section sets the following quotas for the user joe:

This quota root permits greater storage in the Archive folder (100 M) than in all the other folders and message types combined (60 M). Also, no message limit is set for the Archive folder; in this example, only storage limits matter for archiving.

The message types have both storage and number-of-message quotas.

The message-type quotas apply to the sum of all messages of those types, whether they are stored in the Archive folder or in any other folder.

The default mailbox quotas apply to all messages that are not text or voice message types and are not stored in the Archive folder. That is, the message-type quotas and Archive quota are not counted as part of the default mailbox quotas.

To set the quota root in this example, you would take the following steps:

  1. Configure the store.messagetype.x.quotaroot parameter as follows:


    store.messagetype.1.quotaroot = text
    
    store.messagetype.2.quotaroot = voice
  2. Configure the mailQuota attribute for the user joe as follows:


    mailQuota: 20M;#text%10M;#voice%10M;Archive%100M
  3. Configure the mailMsgQuota attribute for the user joe as follows:


    mailMsgQuota: 5000;#text%2000;#voice%200

When you run the getquotaroot IMAP command, the resulting IMAP session displays all quota roots for the user joe's mailbox, as shown here:


1 getquotaroot INBOX
* QUOTAROOT INBOX user/joe user/joe/#text user/joe/#voice
* QUOTA user/joe (STORAGE 12340 20480 MESSAGE 148 5000)
* QUOTA user/joe/#text (STORAGE 1966 10240 MESSAGE 92 2000)
* QUOTA user/joe/#voice (STORAGE 7050 10240 MESSAGE 24 200)

2 getquotaroot Archive
* QUOTAROOT user/joe/Archive user/joe/#text user/joe/#voice
* QUOTA user/joe/Archive (STORAGE 35424 102400)
* QUOTA user/joe/#text (STORAGE 1966 10240 MESSAGE 92 2000)

* QUOTA user/joe/#voice (STORAGE 7050 10240 MESSAGE 24 200)

20.7.5 Expiring Messages by Message Type

The expire and purge feature allows you to move messages from one folder to another, archive messages, and remove messages from the message store, according to criteria you define in expire rules. You perform these tasks with the imexpire utility.

Because the imexpire utility is run by the administrator, it bypasses quota enforcement.

For information about how to write expire rules and use the imexpire utility, see 20.9 To Set the Automatic Message Removal (Expire and Purge) Feature

You can write expire rules so that messages of different types are expired according to different criteria.

The expire feature is extremely flexible, offering many choices for setting expire criteria. This section describes one example in which text and voice messages are expired according to different criteria.

The example assumes you have configured text and voice message types as follows:


store.messagetype.1 = text/plain

store.messagetype.2 = multipart/voice-message

Assume also that the message store is configured to read the Content-Type header field to determine the message type.


Example 20–3 Sample Rules for Expiring Different Message Types


TextInbox.folderpattern: user/%/INBOX
TextInbox.messageheader.Content-Type: text/plain
TextInbox.messagedays: 365
TextInbox.action: fileinto:Archive


VoiceInbox.folderpattern: user/%/INBOX
VoiceInbox.messageheader.Content-Type: multipart/voice-message
VoiceInbox.savedays: 14
VoiceInbox.action: fileinto:OldMail

VoiceOldMail.folderpattern: user/%/OldMail
VoiceOldMail.messageheader.Content-Type: multipart/voice-message
VoiceOldMail.savedays: 30
VoiceOldMail.action: fileinto:Trash

Trash.folderpattern: user/%/Trash
Trash.savedays: 7
Trash.action: discard

In this example, text messages and voice mail are expired in different ways, and they follow different schedules, as follows:

Note: The savedays rule causes a message to be expired the specified number of days after the message is saved. In a typical voice mail system, a user can save voice mail on the voice mail menu. For text messages, a message is saved when it is moved to a folder. The messagedays rule causes a message to be expired the specified number of days after it first arrives in the message store, no matter which folder it is stored in or how often it is moved.


20.8 About Message Store Quotas

With the explosion of email and voice mail, IMAP mailboxes can grow very large. Message store quotas limit, or quota, for how much disk space or how many messages can be held by a user or domain, are in a specific folder, or are of a specific message type. Quotas are used to limit or reduce message store usage. This section contains information about the following:

For further information, see 20.11.4 To Monitor Quota Limits

20.8.1 Quota Overview

Quotas can be set for specific users or domains and can be set in terms of number of messages or number of bytes. It can also be set for specific folders and message types. Message type quotas allow you to specify limits for message type. For example, voice mail and email. Folder quotas set limits to the size of a user's folder in bytes or messages. For example, a quota can be set on the Trash folder. Messaging Server allows you to set default quotas for domains and users as well as customized quotas.

Once a quota is set, how the system responds to users or domains that are either over quota or approaching the quota is also configurable. One response is to send users an over quota notification. Another response is to halt delivery of messages into the message store when quota is exceeded. This is called quota enforcement and usually occurs after a specified grace period. A grace period is how long the mailbox can be over the quota before enforcement occurs. If message delivery is halted due to over quota, incoming messages remain in the MTA queue until one of the following occurs:

Disk space becomes available when a user deletes and expunges messages or when the server deletes messages according to expiration policies established (see 20.9 To Set the Automatic Message Removal (Expire and Purge) Feature).

20.8.1.1 Exceptions for Telephony Application Servers

To support unified messaging requirements, Messaging Server provides the ability to override quota limitations imposed by the message store. This guarantees the delivery of messages that have been accepted by certain agents, namely telephony application servers (TAS). Messages accepted by a TAS can be routed through a special MTA channel that ensures the message is delivered to the store regardless of quota limits. This is a fairly esoteric usage, but can be used to telephony applications. For more information about configuring an TAS channel, contact your Sun messaging representative.

Quota by message type is useful for telephony applications that use unified messaging. For example, if a mix of messages, say text and voice mail, is stored in a user's mailbox, then the administrator can set different quotas for different types of messages. The user's email can have one quota, and their voice mail can have a different quota.

20.8.2 Quota Theory of Operations

Customized user and domain quotas are specified by adding quota attributes to LDAP user and domain entries. Quota defaults, notification policy, enforcement, and grace period are specified in configutil parameters or by using the imquotacheck utility.

To determine if a user is over quota, Messaging Server first checks to see if a quota has been set for the individual user. If no quota has been set, Messaging Server looks at the default quota set for all users. For a user, the quota is for all the cumulative bytes or messages in all of the user's folders. For a domain, the quota is for all the cumulative bytes or messages of all the users in a particular domain. For a message type, the quota is for all the cumulative bytes or messages for that message type. For a folder, the quota is for all the cumulative bytes or messages for user's folder.

You can specify the following quota values for a user's mailbox tree:

The following guidelines apply when you assign multiple quota values for a user:

Changes made to the quota attributes and configutil parameters will take effect automatically, but not immediately as information is stored in caches and it may take a little time before the changes fully take effect. Messaging Server provides a command, iminitquota in Sun Java System Messaging Server 6.3 Administration Reference that updates the changed immediately.

The imquotacheck utility allows you to check message store usage against assigned quotas.

20.8.3 Message Store Quota Attributes and Parameters

This section lists the major the message store quota attributes and configutil parameters. The intention is to provide you with an overview of the functionality interface. For detailed information on these attributes and parameters, refer to the appropriate reference documentation.

The following table lists the quota attributes. Refer to the Sun Java Communications Suite 5 Schema Reference

Table 20–6 Message Store Quota Attributes

Attribute  

Description  

mailQuota

Bytes of disk space allowed for the user’s mailbox.  

mailMsgQuota

Maximum number of messages permitted for a user. This is a cumulative count for all folders in the store.  

mailUserStatus

Status of the mail user. Some of the possible values are active, inactive, deleted, hold, and overquota.

mailDomainDiskQuota

Bytes of disk space allowed for the cumulative count of all the mailboxes in a domain.  

mailDomainMsgQuota

Maximum number of messages permitted for a domain, that is, the total count for all mailboxes in the store. 

mailDomainStatus

Status of the mail domain. Values and default are the same as mailUserStatus.

The following table lists the quota parameters. Refer to the Chapter 3, Messaging Server Configuration, in Sun Java System Messaging Server 6.3 Administration Reference for the latest and most detailed information.

Table 20–7 Message Store configutil parameters

Parameter  

Description  

store.quotaenforcement

Enable quota enforcement When off, the quota database is still updated, but messages are always delivered. Default: On 

store.quotanotification

Enable quota notification. Default: OFF 

store.defaultmailboxquota

Store default quota by number of bytes. Default: -1 (unlimited) 

store.defaultmessagequota

Store default quota by number of messages. Numeric. Default: -1 (unlimited) 

store.quotaexceededmsg

Quota warning message. If none, notification is not sent. Default: None. 

store.quotaexceededmsginterval

Interval, in days, for sending overquota notification. Default: 7 

store.quotagraceperiod

Time, in hours, a mailbox has been overquota before messages to the mailbox will bounce back to the sender. Number of hours. Default: 120 

store.quotawarn

Quota warning threshold. Percentage of quota exceeded before clients are sent an over quota warning. Default: 90 

local.store.quotaoverdraft

Used to provide compatibility with systems that migrated from the Netscape Messaging Server. When ON, allow delivery of one message that puts disk usage over quota. After the user is over quota, messages are deferred or bounced, the quota warning message is sent, and the quota grace period timer starts. (The default is that the quota warning messages are sent when the message store reaches the threshold.) Default: Off, but is treated as on if local.store.overquotastatus is set, otherwise the user can never go over quota and the overquotastatus is never used.

local.store.overquotastatus

Enable quota enforcement before messages are enqueued in the MTA. This prevents the MTA queues from filling up. When set, and a user is not yet over quota, but an incoming message pushes the user over quota, then the message is delivered, but the mailuserstatus LDAP attribute is set to overquota so no more messages will be accepted by the MTA. Default: off 

Message store quota also includes a couple of utilities. iminitquota in Sun Java System Messaging Server 6.3 Administration Reference initializes quota settings. In other words, quota attributes and configutil parameters will take effect automatically after running this command. The changes would take effect without running this, but not immediately, as information is stored in caches and it will take a little time before the changes take effect.

The imquotacheck utility allows you to check message store usage against assigned quotas.

20.8.4 Configuring Message Store Quotas

This section describes the following tasks:

20.8.4.1 To Specify a Default User Quota

A default quota applies to users who do not have individual quotas set in their LDAP entries. The process consists of two steps: 1) Specifying a user default quota and 2) Specifying which users are bound to the default quota. The following examples show how to set default user quotas. Refer to Chapter 3, Messaging Server Configuration, in Sun Java System Messaging Server 6.3 Administration Reference for detailed parameter information.

To specify a default user disk quota for message size in bytes:

configutil -o store.defaultmailboxquota -v [ -1 | number ]

where -1 indicates no quota (unlimited message usage) and number indicates a number of bytes.

To specify a default user quota for total number of messages:

configutil -o store.defaultmessagequota -v [ -1 | number ]

where -1 indicates no quota (unlimited messages) and number indicates the number of messages.

To specify the default quota for specific users:

Set the mailQuota attribute to -2 in the user entries that use the default message store quota. Note that if mailQuotais not specified, the system default quota is used.

20.8.4.2 To Specify Individual User Quotas

Each user can have individualized quotas. To set user-specific quotas, set the mailQuota in Sun Java Communications Suite 5 Schema Reference or mailMsgQuota in Sun Java Communications Suite 5 Schema Reference attributes in the user’s LDAP entry (see configutil Parameters in Sun Java System Messaging Server 6.3 Administration Reference for complete details). The following examples show how to set user quotas.

To specify the system default quota, do not add mailQuota to the LDAP entry, or set it to –2.

To set the quota to 1,000 messages set mailMsgQuota to 1000.

To set the quota to two megabytes set mailQuota to 2M or 2000000.

To set the quota to two gigabytes, set mailQuota to 2G or 2000000000 or 2000M.

To specify a 2 Gigabyte quota; a 20 Megabyte voice mail quota; and a 100 Megabyte quota for the Archive folder:

mailQuota: 2G;#voice%20M;Archive%100M

The two gigabyte quota represents all folders in the user's mailbox that are not explicitly assigned quotas. In this example, that excludes messages in the Archive folder, and messages of type voice. The 100 Megabyte quota includes messages in any folders within the Archive folder

20.8.4.3 To Specify Domain Quotas

You can set disk space or message quotas for domains. These quotas are for the cumulative bytes or messages of all users in a particular domain. To set domain quotas, set the mailDomainDiskQuota in Sun Java Communications Suite 5 Schema Reference or mailDomainMsgQuota in Sun Java Communications Suite 5 Schema Reference attributes in the desired LDAP domain entry. .

To set the quota to 1,000 messages set mailDomainMsgQuota to 1000.

To set the quota to two megabytes set mailDomainDiskQuota to 2M or 2000000.

To set the quota to two gigabytes, set mailDomainDiskQuota to 2G or 2000000000 or 2000M.

ProcedureTo Setup Quota Notification

Quota notification is the process of sending users a warning message when they are getting close to their quota. Using this feature requires three steps.

  1. Enable Quota Notification

    Run the following at the command line:

    configutil -o store.quotanotification -v [ yes | no ]

    If the message is not set, no quota warning message is sent to the user.

  2. Define a Quota Warning Message

    The Warning Message is the message that will be sent to users who are close to exceeding their disk quota. To define a quota warning message at the command line:

    configutil -o store.quotaexceededmsg -v ’message

    The message must be in RFC 822 format. It must contain a header with at least a subject line, follow by $$, then the message body. ’$’ represents a new line. Depending on the shell that you are using, it might be necessary to append a \ before $ to escape the special meaning of $. ($ is often the escape character for the shell.) Example:

    configutil -o store.quotaexceededmsg -v ”Subject: WARNING: User quota exceeded$$User quota threshold exceeded - reduce space used.’

    In addition, there is support for the following variables:

    [ID] - userid

    [DISKUSAGE] - disk usage

    [NUMMSG] - number of messages

    [PERCENT] - store.quotawarn percentage

    [QUOTA] - mailquota attribute

    [MSGQUOTA] - mailmsgquota attribute

    Here’s an example, using these variables:

    configutil -o store.quotaexceededmsg -v ”Subject: Overquota Warning$$[ID],$$Your mailbox size has exceeded [PERCENT] of its alloted quota.$Disk Usage: [DISKUSAGE]$Number of Messages: [NUMMSG]$Mailquota: [QUOTA]$Message Quota: [MSGQUOTA]$$-Postmaster’

  3. Specify how often the warning message is sent.

    Set the following parameter:

    configutil -o store.quotaexceededmsginterval -v number

    where number indicates a number of days. For example, 3 would mean the message is sent every 3 days.

  4. Specify a Quota Threshold

    A quota threshold is a percentage of a quota that is exceeded before clients are sent a warning. When a user’s disk usage exceeds the specified threshold, the server sends a warning message to the user.


    Note –

    When local.store.quotaoverdraft=on email notifications are not triggered until the user’s disk usage exceeds 100% of the quota regardless of the threshold set with store.quotawarn.


    For IMAP users whose clients support the IMAP ALERT mechanism, the message is displayed on the user’s screen each time the user selects a mailbox and a message is also written to the IMAP log.

    To specify a quota threshold at the command line:

    configutil -o store.quotawarn -v number

    where number indicates a percentage of the allowed quota.

20.8.4.4 To Enable or Disable Quota Enforcement

By default, users or domains can exceed their quotas with no effect except for receiving an over quota notification (if set). Quota enforcement locks the mailboxes from receiving further messages until the disk usage is reduced below the quota level.

To enable or disable quota enforcement:


configutil -o store.quotaenforcement -v [ on | off]

Note that over quota messages are saved in the MTA queues and a notification is sent to the sender stating that their messages was not delivered, but that a redelivery attempt will be made later. Delivery retries will continue until the grace period expires and all messages are sent back to the senders, or the disk usage falls below the quota and messages can be dequeued from the MTA and delivered to the message store. If you want to return messages that are over quota before they get to the message queues, use the following command line:


configutil -o local.store.overquotastatus -v on

To Enable Quota Enforcement at the Domain Level

To enforce quotas for a particular domain, use the command:

imquotacheck -f -d domain

To enable for all domains exclude the -d option. When a domain exceeds its quota, the maildomainstatus attribute is set to overquota, which halts all delivery to this domain. If a domain is not overquota, the value is set to active.

Disabling Quota Enforcement

If it appears that user quotas are being enforced, even when you have disabled them, check the following parameters:

These configutil parameters should be off or not set:

Note that when local.store.overquotastatus is on, it always treats store.quotaoverdraft as on, otherwise the user will never go over quota to trigger the rejection. Also, when store.quotaoverdraft is on, the user is allowed one message which is smaller than the quota only. That is, it will never accept a message that is greater than the user’s quota.

After making changes to these parameters, be sure to restart your messaging services.

These Message Store attributes should be active:

Note that messages will bounce if they are larger than the mailbox quota, regardless of quota enforcement configuration.

20.8.4.5 To Set a Grace Period

The grace period specifies how long the mailbox can be over the quota (disk space or number of messages) before messages are bounced back to sender. The grace period is not how long the message is held in the message queue, it’s how long the mailbox is over quota before all incoming messages, including those in the message queue, are bounced. (see 20.1 Overview for more details.) The grace period starts when the user has reached the quota threshold and been warned. See To Setup Quota Notification.

To specify a quota grace period at the command line:

configutil -o store.quotagraceperiod -v number

where number indicates number of hours.

20.8.4.6 Netscape Messaging Server Quota Compatibility Mode

After disk usage exceeded the quota in the Netscape Messaging Server, the server deferred or bounced message delivery, sent an over quota notification, and started the grace period. Messaging Server provides a parameter, local.store.quotaoverdraft, which retains this behavior.

When set to ON, messages are delivered until disk usage is over quota. At that time, messages are deferred (messages stay in the MTA message queue but are not delivered to the message store), an over quota warning message is sent to the user, and a grace period starts. The grace period determines how long a mailbox is overquota before the overquota messages bounce. (The default is that the quota warning messages are sent when the message store reaches the threshold.) The default for this parameter is Off.

20.9 To Set the Automatic Message Removal (Expire and Purge) Feature

The automatic message removal feature (also known as expire and purge) automatically removes messages from the message store based on a set of administrator-defined criteria. This feature can automatically remove old or overly large messages, seen/deleted messages, messages with certain Subject: lines and so on. This feature allows the following removal criteria:

This feature is performed by the imexpire utility, which expunges and purges messages. See 20.3 How the Message Store Removes Messages for details on the message removal process.


Note –

The server removes messages without warning, so it is important to inform users about automatic message removal policies. Unexpected message removal can be a source of consternation for users and administrators.


20.9.1 imexpire Theory of Operation

imexpire can be invoked from the command line or scheduled to run automatically by the imsched daemon. The administrator specifies a set of expiration rules in a file called store.expirerule. This file specifies the criteria by which messages are removed. There can be multiple files with each put in the directory that pertains to the scope of the rules. That is, rules that apply globally to the entire message store are put in one directory, rules that apply to a partition in another, rules that apply to users in yet another, and so on.


Note –

Although global expiration rules can be specified with the configutil command and store.expire.attribute parameters, it is better to use store.expirerule to specify these rules. If too many rules are created using configutil, performance problems can result.


imexpire loads all of the expire rules at start up. By default, imexpire creates one thread per partition. Each thread goes through the list of user folders under its assigned partition and loads the local expire rule files as it goes. The expire function checks each folder against the expire rules applicable to this folder and expunges messages as needed. If there is a store.exp file that exists under the mailbox directory, and there are messages that have been expunged/expired for longer than the time specified by the store.cleanupage configuration parameter, the purge function will permanently remove the message files under the message hash directories and remove the UID records from the store.exp files.

It is also possible to exclude specified users from the expire rules by adding their user ID, one per line, in a file called expire_exclude_list in msg-svr-base/config/.

20.9.2 To Deploy the Automatic Message Removal Feature

Automatic message removal requires three steps:

  1. Define automatic message removal policy: Which messages will be automatically removed? What users, folders, domains and partitions will have messages automatically removed? What size, message age, headers will define the removal criteria. Define the scope of messages to be removed. See 20.9.2.1 To Define Automatic Message Removal Policy

  2. Specify theimexpire rules to implement this policy. See 20.9.2.2 To Set Rules Implementing Automatic Message Removal Policy

  3. Specify the imexpire scheduling. See 20.9.2.3 To Schedule Automatic Message Removal and Logging Level

20.9.2.1 To Define Automatic Message Removal Policy

Define your automatic message removal policy by specifying the criteria for removal. imexpire allows for removal using the following criteria:

Age of Message. Automatically remove messages older than X days. Attribute: messagedays.

Message Count. Automatically remove messages in a folder exceeding X messages. Attribute: messagecount.

Age of Oversized Message. Automatically remove messages that exceed X bytes after Y days grace period. Attributes: messagesize and messagesizedays.

Seen and Deleted Message Flag. Automatically remove messages with the Seen or Deleted flag set. These criteria can be set to “and” or “or.” If set to or, the message’s Seen/Delete flag will cause automatic deletion regardless of other criteria. If set to and, the message’s Seen/Delete flag must be set along with passing all other specified criteria. Attributes: seen and deleted.

Header Field of Message. Allows you specify a header and string as criteria for removing a message. For example, removing all messages with the header “Subject: Work from Home!”

Folder of Messages. Allows you to specify the folder on which to remove messages. Attribute: folderpattern. Note that this attribute only uses the modified UTF-7 character set.


Note –

imexpire does not allow you to delete or preserve messages based on how long it has been since that message was read. For example, you cannot specify that messages that have not been read for 200 days will be removed.


Examples of Automatic Message Removal Policy

Example 1: Remove all messages 365 days old in a folder exceeding 1,000 messages.

Example 2: Remove messages in domain siroe.com that are older than 180 days.

Example 3: Remove all messages that have been marked as deleted.

Example 4: Remove messages in sesta.com that have been marked as seen, are older than 30 days, are larger than 100 kilobytes, from folders exceeding 1,000 messages, with the header X-spam.

20.9.2.2 To Set Rules Implementing Automatic Message Removal Policy

To implement the automatic message removal policy defined in the previous section, you must set the imexpire rules. Rules are set by putting them into a store.expirerule file. An example of two global store.expirerule rules is shown below:


Rule1.regexp: 1
Rule1.folderpattern: user/.*/trash
Rule1.messagedays: 2
Rule2:regexp: 1
Rule2.folderpattern: user/.*
Rule2.messagedays: 14

            

In this example, Rule 1 specifies that all messages in the trash folder will be remove after two days. Rule 2 specifies that all messages in the message store will be removed after 14 days.

This section consists of the following subsections:

Expiration Rules Guidelines

This section sets the guidelines for the store.expirerule file rules.


Note –

In earlier Messaging Server releases, expiration rules could be set with configutil parameters store.expirerule.attribute (see configutil Parameters in Sun Java System Messaging Server 6.3 Administration Reference.) This is still true, but expire rules using header constraints (example: expiring a message with a specific subject line) are not supported. Also, regular expressions in the expire rules created with configutil need to be POSIX compliant rules. If you want to use UNIX compliant regular expressions you will have to use the store.expire file. In addition, using both configutil options and the global store.expirerule configuration file is not supported. If the configuration file is present, configutil options are not used. In any case, it is best to use store.expirerule to specify all expiration rules.


Table 20–8 imexpire Attributes

Attribute 

Description (Attribute Value) 

action

Specifies an action to perform on the messages caught by the expire rules. The possible values are: 

discard discards the message. This is the default.

report action prints the mailbox name, uid-validity and uid to stdout.

archive archives the message with Sun Compliance and Content Management System and then discards the message.

fileinto: folder action folders the message into the specific folder. The shared folder prefix can be used to folder messages to folders owned by another user.

exclusive

Specifies whether or not this is an exclusive rule. If specified as exclusive, only this rule applies to the specified mailbox(s) and all other rules are ignored. If more than one exclusive rule exists, the last exclusive rule loaded will be used. For example, if a global and a local exclusive rule are specified, the local rule will be used. If there is more than one global exclusive rule, the last global rule listed by configutil is used. (1/0)

folderpattern

Specifies the folders affected by this rule. The format must start with a user/, which represents the directory store_root/partition/*/ See Table 20–9. (POSIX regular expression)

messagecount

Maximum number of messages in a folder. Oldest messages are expunged as additional messages are delivered. (integer) 

foldersize

Maximum size of folder before the oldest messages are expunged when additional messages are delivered. (integer in bytes) 

messagedays

Number of days in the message store before being expunged. (integer) 

messagesize

Maximum size of message in bytes before it is marked to be expunged. (integer) 

messagesizedays

Grace period. Days an over-sized message should remain in a folder. (integer) 

messageheader.header

Specifies a header field and string that marks a message for removal. Values are not case-sensitive and regular expressions are not recognized.Example: Rule1.messageheader.Subject: Get Rich Now!

For the header Expires and Expiry-Date, imexpire will remove the message if the date value specified with these header fields is older than the messagedays attribute. If multiple expiration header fields are specified, the earliest expiration date will be used. (string).

regexp

Enable UNIX regular expressions in rules creation. (1 or 0). If not specified, IMAP expressions will be used. 

savedays

Number of days the messages are saved in a folder until they are expunged. 

seen

seen is a message status flag set by the system when the user opens a message. If the attribute seen is set to and, then the message must be seen and other criteria must be met before the rule is fulfilled. If the attribute seen is set to or, then the message only needs to be seen or another criteria be met before the rule is fulfilled. (and/or).

sieve

A Sieve rule specifying message selection criteria. Example: Rule17.sieve: header :contains “Subject” “Vigara”

deleted

deleted is a message status flag set by the system when the user deletes a message. If the attribute deleted is set to and, then the message must be deleted and another criteria must be met before the rule is fulfilled. If the attribute deleted is set to or, then the message only needs to be seen or another criteria be met before the rule is fulfilled. (and/or)

Localized Mailbox Names

The IMAP protocol specifies that mailbox names use modified UTF-7 encoding. Messaging Server supports localized character sets on external interfaces so that mailbox names can be localized. Internally, however, the system converts the localized name to UTF-7. Thus, a folder that has a localize mailbox name on a client will have a corresponding mailbox file name in UTF-7. (Note that IMAP error messages will output mailbox names in UTF-7 and not the localized character set.)

In general, most message store utilities that require mailbox names expect the names in the localized character set, although they may have an option flag that allows a different character set to be used. These utilities include reconstruct, mboxutil, imsbackup, imsrestore, and hashdir. However, imexpire requires that the mailbox name, specified as the attribute folderpattern,be in UTF-7. Using a localized name will not work.

To obtain the appropriate folderpattern for imexpire it may be necessary to convert a localized mailbox name to the modified UTF-7 equivalent. This can be done using the mboxutil -E command as follows:

mboxutil showing localized filename and modified UTF-7
filename.

The first mboxutil shows the localized filename. The second mboxutil shows the filename in modified UTF-7. It is also possible to use the IMAP list command:

IMAP list command

To convert the local charset to modified UTF-7 encoding, use the mboxutil command with the -E option:

mboxutil with —E option.

Note that mboxutil -E can be used for any command that requires the use of a UTF-7 mailbox name including imexpire.

Setting imexpire Rules Textually

Automatic message removal rules are set by specifying rules in a store.expirerule file. The store.expirerule file contains one expire criteria per line. An expire criteria of the global rule configuration file (msg-svr-base/data/store/store.expirerule) has the following format:

rule_name.attribute: value

An expiration rule for a user or mailbox rule configuration file has this format:

attribute: value

Example 20–4 shows a set of global expiration rules in msg-svr-base/config/store.expirerule.

Rule 1 sets the global expiration policy (that is, policy that applies to all messages), as follows:

Rule 2 sets the automatic message removal policy for users at the hosted domain siroe.com. It limits mailbox sizes to 1 megabyte, removes messages that have been deleted, and removes messages older than 14 days.

Rule 3 sets the automatic message removal policy for messages in the inbox folder of user f.dostoevski. It removes messages with a subject line having the expression “On-line Casino.”


Example 20–4 Example imexpire Rules


Rule1.regexp: 1
Rule1.folderpattern: user/.*
Rule1.messagesize: 100000
Rule1.messagesizedays: 3
Rule1.deleted: or
Rule1.Subject: Vigara Now!
Rule1.Subject: XXX Porn!
Rule1.messagecount: 1000
Rule1.messagedays: 365
Rule2.regexp: 1
Rule2.folderpattern: user/.*@siroe.com/.*
Rule2.exclusive: 1
Rule2.deleted: or
Rule2.messagedays: 14
Rule2.messagecount: 1000
Rule3.folderpattern: user/f.dostoevski/inbox
Rule3.Subject: *On-line Casino*
                  

Setting imexpire Folder Patterns

Folder patterns can be specified using POSIX regular expressions by setting the imexpire attribute regex to 1. If not specified, IMAP expressions will be used. The format must start with a user/ followed by a pattern. Table 20–9 shows the folder pattern for various folders.)

Table 20–9 imexpire Folder Patterns Using Regular Expressions

Folder Pattern 

Scope 

user/userid/.*

Apply rule to all messages in all folders of userid.

user/userid/Sent

Apply rule to messages of userid in folder Sent:

user/.*

Apply rule to entire message store. 

user/.*/trash

Apply rule to the trash folder of all users.

user/.*@siroe.com/.*

Apply rule to folders in hosted domain siroe.com: 

user/[^@]*/.*

Apply rule to folders in default domain. 

20.9.2.3 To Schedule Automatic Message Removal and Logging Level

Automatic message removal is activated by the imsched scheduling daemon. By default, imsched invokes imexpire at 23:00 every day and messages are both expunged and purged. This schedule can be customized by setting the configutil parameter local.schedule.expire and store.cleanupage described in Table 20–10.

Expire and purge can take a long time to complete on a large message store, so you may wish to experiment and decide how often to run these processes. For example, if an expire/purge cycle takes 10 hours, you may not want the default schedule of running expire and purge once a day. Schedule expire and purge using the imexpire command and the automatic task scheduling parameter (see4.6 To Schedule Automatic Tasks). For example:


configutil -o local.schedule.expire -v "0 1 * * 6 /opt/SUNWmsgsr/sbin/imexpire -e"
configutil -o local.schedule.mspurge -v "0 23 * * * /opt/SUNWmsgsr/sbin/imexpire -c"

In this example, messages are expired at 1AM Saturdays and purged every night at 11PM. If no purge schedule is set, , imexpire will perform purge after an expire.

Table 20–10 Expire and Purge configutil Log and Scheduling Parameters

Parameter  

Description 

local.schedule.expire

Interval for running imexpire. Uses UNIX crontab format: minute hour day-of-month month-of-year day-of-week

The values are separated by a space or tab and can be 0-59, 0-23, 1-31, 1-12 and 0-6 (with 0=Sunday) respectively. Each time field can be either an asterisk (meaning all legal values), a list of comma-separated values, or a range of two values separated by a hyphen. Note that days can be specified by both day of the month and day of the week, however, it is not typical to use them both since the number of such occurrences are very small. If they are both specified, then both will be required. For example, setting the 17th day of the month and Tuesday will require both values to be true. 

Note that you can also use the -e and -c flags with imexpire to and expire only or purge only respectively. See imexpire in Sun Java System Messaging Server 6.3 Administration Reference.

Interval Examples:

1) Run imexpire at 12:30am, 8:30am, and 4:30pm:

30 0,8,16 * * * /opt/SUNWmsgsr/sbin/imexpire

2) Run imexpire at weekday morning at 3:15 am:

15 3 * * 1-5 /opt/SUNWmsgsr/sbin/imexpire

3) Run imexpire only on Mondays:

0 0 * * 1 /opt/SUNWmsgsr/sbin/imexpire

Default:  

0 23 * * * /opt/SUNWmsgsr/sbin/imexpire

To disable: set local.schedule.expire.enable to NO.

store.cleanupage

Age (in hours) of expired or expunged message before purge will permanently remove it.

Default: None 

local.store.expire.loglevel

Specify a log level: 

1 = log summary for the entire expire session.  

2 = log one message per mailbox expired.  

3 = log one message per message expired. 

Default: 1 

Setting imexpire Logging Levels

imexpire will log a summary to the default log file upon completion. If expire is invoked from the command line, the -v (verbose) and -d (debug) option can be used to instruct imexpire to log detail status/debug messages to stderr. If imexpire is invoked by imsched, the configutil parameter local.store.expire.loglevel can be set to 1,2 or 3 for different levels of logging. Loglevel 1 is the default, it will log a summary for the entire expire session. Loglevel 2 will log one message per mailbox expired. Loglevel 3 will log one message per message expired.

Excluding Specified Users from Automatic Message Remove

Exclude specified users from the expire rules by adding their user ID, one per line, in a file called expire_exclude_list in msg-svr-base/config/. Or, configure a dummy exclusive expire rule under the user's mailbox.

20.10 Configuring Message Store Partitions

Mailboxes are stored in message store partitions, an area on a disk partition specifically devoted to storing the message store. Message store partitions are not the same as disk partitions, though for ease of maintenance, it is recommended that you have one disk partition and one file system for each message store partition. Message store partitions are directories specifically designated as a message store.

User mailboxes are stored by default in the store_root/partition/ directory (see 20.2 Message Store Directory Layout). The partition directory is a logical directory that might contain a single partition or multiple partitions. At start-up time, the partition directory contains one subpartition called the primary partition.

You can add partitions to the partition directory as necessary. For example, you might want to partition a single disk to organize your users as follows:


store_root/partition/mkting/
store_root/partition/eng/
store_root/partition/sales/

As disk storage requirements increase, you might want to map these partitions to different physical disk drives.

You should limit the number of mailboxes on any one disk. Distributing mailboxes across disks improves message delivery time (although it does not necessarily change the SMTP accept rate). The number of mailboxes you allocate per disk depends on the disk capacity and the amount of disk space allocated to each user. For example, you can allocate more mailboxes per disk if you allocate less disk space per user.

If your message store requires multiple disks, you can use RAID (Redundant Array of Inexpensive Disks) technology to ease management of multiple disks. With RAID technology, you can spread data across a series of disks but the disks appear as one logical volume so disk management is simplified. You might also want to use RAID technology for redundancy purposes; that is, to duplicate the store for failure recovery purposes.


Note –

To improve disk access, the message store and the message queue should reside on separate disks.


20.10.1 To Add a Partition

When adding a partition, you specify both an absolute physical path where the partition is stored on disk and a logical name, called the partition nickname.

The partition nickname allows you to map users to a logical partition name regardless of the physical path. When setting up user accounts and specifying the message store for a user, you can use the partition nickname. The name you enter must be an alphanumeric name and must use lowercase letters.

To create and manage the partition, the user ID used to run the server must have permission to write to the location specified in the physical path.


Note –

After adding a partition, you must stop then restart the server to refresh the configuration information.


ProcedureTo Add a Message Store Partition

  1. Command Line, To add a partition to the store at the command line:

    configutil -o store.partition.nickname.path -v path

    where nickname is the logical name of the partition and path indicates the absolute path name where the partition is stored.

    To specify the path of the default primary partition:


    configutil -o store.partition.primary.path -v path
    

20.10.2 To Move Mailboxes to a Different Disk Partition

By default, mailboxes are created in the primary partition. If the partition gets full, additional messages cannot be stored. There are several ways to address the problem:

If possible, we recommend adding additional disk space to a system using volume management software since this procedure is the most transparent for the user. However, you may also move mailboxes to a different partition.

ProcedureTo Move Mailboxes to a Different Disk Partition

  1. The user does not have to be disconnected from their mailbox during this migration process.

    mboxutil -r should lock everything that needs to be locked. Delivery may be stalled while that happens, and POP and IMAP and therefore webmail clients, may experience a delay, but there should be no problem.

  2. Move the user mailbox with the following command:

    mboxutil -r user/<userid>/INBOX user/<userid>/INBOX <partition_name>

    Example:

    mboxutil -r user/ofanning/INBOX user/ofanning/INBOX secondary

  3. Set the mailMessageStore attribute in the moved user’s LDAP entry to the name of the new partition.

    Example: mailMessageStore: secondary

  4. Inform the user that message store connection is now allowed. If applicable, change the mailAllowedServiceAccess attribute to allow POP, IMAP and HTTP services.

20.10.3 Changing the Default Message Store Partition Definition

The default partition is the partition used when a user is created and the mailMessageStore LDAP attribute is not specified in the user entry. The mailMessageStore LDAP attribute, which specifies a user’s message store partition, should be specified in all user entries so that a default partition is not necessary. In addition, the default partition should not be changed for load balancing or any other reason. It is invalid and dangerous to change the default partition while there are still users depending on the default partition definition.

If it is absolutely necessary to change the default partition, make sure that all users on the old default partition (the one being left behind) have their mailMessageStore attribute set to their current partition (which will no longer be the default), before changing the definition of default with the configutil parameter store.defaultpartition.

20.11 Performing Message Store Maintenance Procedures

This section provides information about the utilities you use to perform maintenance and recovery tasks for the message store. You should always read your postmaster mail for warnings and alerts that the server might send. You should also monitor the log files for information about how the server is performing. For more information about log files, see Chapter 25, Managing Logging

This section contains the following:

20.11.1 Adding More Physical Disks to the Message Store

The Messaging Server message store contains the user mailboxes for a particular Messaging Server instance. The size of the message store increases as the number of mailboxes, folders, and log files increase.

As you add more users to your system, your disk storage requirements increase. Depending on the number of users your server supports, the message store might require one physical disk or multiple physical disks. Messaging Server enables you an add more stores as needed. One approach to adding more stores is by using storage appliances. See Using NetApp Filers with Sun Java System Messaging Server Message Store for information on how to configure Network Appliance storage appliances with Messaging Server.

20.11.2 To Manage Mailboxes

This section describes the following utilities for managing and monitoring mailboxes: mboxutil, hashdir, readership.

20.11.2.1 The mboxutil Utility

Use the mboxutil command to perform typical maintenance tasks on mailboxes. mboxutil tasks include the following:


Note –

Note that you should not kill the mboxutil process in the middle of execution. If it is killed with SIGKILL (kill -9), it may potentially require that every server get restarted and a recovery be done.


See mboxutil in Sun Java System Messaging Server 6.3 Administration Reference for detailed syntax and usage requirements.

Examples

To list all mailboxes for all users:

mboxutil -l

To list all mailboxes and also include path and ACL information:

mboxutil -l -x

To create the default mailbox named INBOX for the user daphne:

mboxutil -c user/daphne/INBOX

To delete a mail folder named projx for the user delilah:

mboxutil -d user/delilah/projx

To delete the default mailbox named INBOX and all mail folders for the user druscilla:

mboxutil -d user/druscilla/INBOX

To rename the mail folder memos to memos-april for the user desdemona:

mboxutil -r user/desdemona/memos user/desdemona/memos-april

To move the mail account for the user dimitria to a new partition:

mboxutil -r user/dimitria/INBOX user/dimitria/INBOX partition

where partition specifies the name of the new partition.

To move the mail folder named personal for the user dimitria to a new partition:

mboxutil -r user/dimitria/personal user/dimitria/personal partition

20.11.2.2 To Remove Orphan Accounts

To search for orphaned accounts (orphaned accounts are mailboxes that do not have corresponding entries in LDAP) use the following command:


mboxutil -o

Command output follows:

  mboxutil: Start checking for orphaned mailboxes
  user/annie/INBOX
  user/oliver/INBOX
  mboxutil: Found 2 orphaned mailbox(es)
  mboxutil: Done checking for orphaned mailboxes

Use the following command to create a file listing orphaned mailboxes that can be turned into a script file that deletes the orphaned mailboxes (example filename is orphans.cmd):


mboxutil -o -w orphans.cmd

The command output is as follows:

  mboxutil: Start checking for orphaned mailboxes
  mboxutil: Found 2 orphaned mailbox(es)
  mboxutil: Done checking for orphaned mailboxes

Delete the orphan files with the following command:


mboxutil -d -f orphans.cmd

20.11.2.3 The hashdir Utility

The mailboxes in the message store are stored in a hash structure for fast searching. Consequently, to find the directory that contains a particular user’s mailbox, use the hashdir utility.

This utility identifies the directory that contains the message store for a particular account. This utility reports the relative path to the message store, such as d1/a7/. The path is relative to the directory level just before the one based on the user ID. The utility sends the path information to the standard output.

For example, to find the relative path to the mailbox for user crowe:

hashdir crowe

20.11.2.4 The readership Utility

The readership utility reports on how many users other than the mailbox owner have read messages in a shared IMAP folder.

An owner of a IMAP folder may grant permission for others to read mail in the folder. A folder that others are allowed to access is called a shared folder. Administrators can use the readership utility to see how many users other than the owner are accessing a shared folder.

This utility scans all mailboxes and produces one line of output per shared folder, reporting the number of readers followed by a space and the name of the mailbox.

Each reader is a distinct authentication identity that has selected the shared folder within the past specified number of days. Users are not counted as reading their own personal mailboxes. Personal mailboxes are not reported unless there is at least one reader other than the folder’s owner.

For example, the following command counts as a reader any identity that has selected the shared IMAP folder within the last 15 days:

readership -d 15

20.11.3 Maximum Mailbox Size

A mailbox has a maximum size of about one million messages. More than this will cause messages to stop being delivered to the user and could cause message store performance problems. See 20.14.4.8 User Mail Not Delivered Due to Mailbox Overflow for details.

20.11.4 To Monitor Quota Limits

Monitor quota usage and limits by using imquotacheck, which generates a report listing defined quotas and limits, and provides information on quota usage. Quotas and usage figures are reported in kilobytes. This utility can also compare mailbox size with a user’s assigned quota. As an option, you can email a notification to users who have exceeded a set percentage of their assigned quota.


Note –

Certain functions have changed in imquotacheck. (In Messaging Server 6.x, imquotacheck utility has superseded the quotacheck utility.) In Messaging Server 5.x, when you used the quotacheck utility to retrieve a list of users, quotacheck searched the local mboxlist database. This function duplicated the search function in the mboxutil utility.

In Messaging Server 6.x, this duplicate function was removed from the imquotacheck utility. If you perform a user search with imquotacheck, the search is performed against the LDAP directory, not the local mboxlist database. To retrieve a list of users from the local mboxlist database, use the mboxutil utility.


To list the usage of all users whose quota exceeds the least threshold in the rule file:

imquotacheck

List quota information for a the domain siroe.com:

imquotacheck -d siroe.com

To send a notification to all users in accordance to the default rule file:

imquotacheck -n

To send a notification to all users in accordance to a specified rulefile, myrulefile, and to a specified mail template file, mytemplate.file (for more information, refer to imquotacheck in Sun Java System Messaging Server 6.3 Administration Reference):

imquotacheck -n -r myrulefile -t mytemplate.file

To list per folder usages for one user user1 (will ignore the rule file):

imquotacheck -u user1 -e

20.11.5 To Monitor Disk Space

You can specify how often the system should monitor disk space and partition usage, and under what circumstances the system should send a warning. See 27.3.2 Monitoring Disk Space for details.

20.11.6 The stored Daemon

The stored daemon performs the following maintenance tasks for the message store:

If any server daemon crashes, you must stop all daemons and restart all daemons including stored.

20.11.7 Reducing Message Store Size Due to Duplicate Storage of Identical Messages

When a message is sent to multiple recipients, that message is placed in each recipient’s mailbox. Some messaging systems store separate copies of the same message in each recipient’s mailbox. By contrast, the Sun Java System Messaging Server strives to retain a single copy of a message regardless of the number of mailboxes in which that message resides. It does this by creating hard links to that message in the mailboxes containing that message.

When other messaging systems are migrated to the Sun Java Messaging Server, these multiple message copies may be copied over with the migration process. With a large message store, this means that a lot of messages are duplicated unnecessarily. In addition, multiple copies of the same message can be accumulated in normal server operation, for example, from IMAP append operations or other sources.

Messaging Server provides a new command called relinker that removes the excess message copies and replaces them with hard links to a single copy.

20.11.7.1 Relinker Theory of Operations

The relinking function can be run in the command or realtime mode. When the relinker command is run, it scans through the message store partitions, creates or updates the MD5 message digest repository (as hard links), deletes excess message files, and creates the necessary hard links.

The digest repository consists of hard links to the messages in the message store. It is stored in the directory hierarchy partition_path/=md5. This directory is parallel to the user mailbox hierarchy partition_path/=user (see Figure 20–1). Messages in the digest repository are uniquely identified by their MD5 digest. For example, if the digest for fredb/00/1.msg is 4F92E5673E091B43415FFFA05D2E47, then partition/=user/hashdir/hashdir/=fredb/00/1.msg is linked to partition/=md5/hashdir/hashdir/4F92E5673E091B43415FFFA05D2E47EA.msg. If another mailbox has this same message, for example, partition_path/=user/hashdir/hashdir/gregk/00/17.msg, that message will also be hard linked to partition_path/=md5/4F/92/4F92E5673E091B43415FFFA05D2E47EA.msg. This is shown in Figure 20–4.

Figure 20–4 Message Store Digest Repository

Graphic depicts message store repository.

For this message, the link count will be three. If both messages are deleted from the mailboxes of fredb and gregk, then the link count will be one and the message can be purged.

The relinker process can also be run in the realtime mode for similar functionality. See 20.11.7.3 Using Relinker in the Realtime Mode for details.

20.11.7.2 Using Relinker in the Command Line Mode

relinker scans through a message store partition, creates or updates the MD5 message repository (as hard links) and deletes excess message files. After relinker scans a store partition, it outputs statistics on the number of unique messages and size of the partition before and after relinking. To run more quickly on an already hashed store, relinker only computes the digest of the messages not yet present in =md5. It also has an option to erase the entire digest repository (which doesn’t affect the user mailboxes).

The syntax for the command is as follows:

relinker [-P. partitionname] [-d]

where partitionname specifies the partition to be processed (default: all partitions) and -d specifies that the digest repository be deleted. Sample output is shown below:


# relinker

Processing partition: primary
Scanning digest repository...
Processing user directories..............................
---------------------------------------------------------
Partition statistics           Before            After 
---------------------------------------------------------
Total messages                 4531898         4531898
Unique messages                4327531         3847029
Message digests in repository        0         3847029
Space used                       99210Mb         90481Mb
Space savings from single-copy    3911Mb         12640Mb
---------------------------------------------------------


# relinker -d 
Processing partition: primary
Purging digest repository...
---------------------------------------------------------
Partition statistics                 Before         After
---------------------------------------------------------
Message digests in repository       3847029             0
---------------------------------------------------------

Relinker can take a long time to run, especially for the first time if there are no messages are in the repository. This is because it has to calculate the digest for every message (if the relinker criteria is configured to include all messages—see 20.11.7.4 Configuring Relinker for information on configuring relinker criteria.) For example, it could take six hours to process a 100 Gigabyte message store. However, if run-time relinking is enabled see 20.11.7.3 Using Relinker in the Realtime Mode.

If the relinker command line mode is used exclusively, and not the run-time option, it is necessary to purge the digest repository (=md5), otherwise messages purged in the store (=user) will not become available disk space since they still have a link in the digest repository (they become orphaned). If you are just performing a one-time optimization of the store—for example after a migration—you can run relinker once, then delete the entire repository with relinker -d. For repeated purging during migration, it is sufficient to just run the relinker command repeatedly, since each time it runs it also purges the expired or orphaned messages from the repository.

It is safe to run multiple instances of relinker in parallel with each processing a different partition (using the -p option). Messages are only relinked inside the same partition.

20.11.7.3 Using Relinker in the Realtime Mode

The relinker function can be enabled in the realtime mode by setting the configutil parameter local.store.relinker.enabled to yes. Using relinker in the realtime mode will compute the digest of every message delivered (or restored, IMAP appended, and so forth) which matches the configured relinker criteria (20.11.7.4 Configuring Relinker), then look in the repository to see if that digest is already present. If the digest is present, it creates a link to it in the destination mailbox instead of creating a new copy of the message. If there is no digest, it creates the message and adds a link to it in the repository afterwards.

stored scans the digest repositories of each partition and purges the messages having a link count of 1, or which don’t match the relinker criteria. The scan is done one directory at a time over a configurable time period. This is so that the I/O load is evenly distributed and doesn’t noticeably impact other server operations. By default the purge cycle is 24 hours, which means messages can still be present on the disk for up to 24 hours after they’ve been deleted from the store or have exceeded the configured maximum age. This task is enabled when the relinker realtime mode is enabled.

20.11.7.4 Configuring Relinker

Table 20–11 shows the parameters used to set relinker criteria.

Table 20–11 relinker configutil Parameters

Parameter 

Description  

local.store.relinker.enabled

Enables real-time relinking of messages in the append code and stored purge. The relinker command-line tool may be run even if this option is off. However since stored will not purge the repository, relinker -d must be used for this task. Turning this option on affects message delivery performance in exchange for the disk space savings.

Default: no

local.store.relinker.maxage

Maximum age in hours for messages to be kept in the repository, or considered by the relinker command-line. -1 means no age limit, that is, only purge orphaned messages from the repository. For relinker it means process existing messages regardless of age. Shorter values keep the repository smaller thus allow relinker or stored purge to run faster and reclaim disk space faster, while longer values allow duplicate message relinking over a longer period of time, for example, when users copy the same message to the store several days apart, or when running a migration over several days or weeks.

Default: 24 

local.store.relinker.minsize

Minimum size in kilobytes for messages to be considered by run-time or command-line relinker. Setting a non-zero value gives up the relinker benefits for smaller messages in exchange for a smaller repository.

Default: 0 

local.store.relinker.purgecycle

Approximate duration in hours of an entire stored purge cycle. The actual duration depends on the time it takes to scan each directory in the repository. Smaller values will use more I/O and larger values will not reclaim disk space as fast. 0 means run purge continuously without any pause between directories. -1 means don’t run purge in stored (then purge must be performed using the relinker -d command).

Default: 24 

20.12 Backing Up and Restoring the Message Store

Message store backup and restore is one of the most common and important administrative tasks. It consists of backing up all the messages and folders on a message store. You must implement a backup and restore policy for your message store to ensure that data is not lost if problems such as the following occur:

You can do message store back up and restore using command-line utilities imsbackup and imsrestore, or the integrated solution that uses Legato NetworkerTM.

Messaging Server provides a single-copy backup procedure. Regardless of how many user folders contain a particular message, during backup, the message file is backed up only once using the first message file found. The second message copy is backed up as a link to the name of the first message file, and so on. imsbackup maintains a hash table of all messages using the device and inode of the message files as the index. This method does have implications when restoring data, however. For more information, see 20.12.5 Considerations for Partial Restore


Note –

It is also possible to do message store backup and restore by backing up all the message files and directories. Refer to 20.12.9 Message Store Disaster Backup and Recovery.


This section contains the following subsections:

20.12.1 Creating a Mailbox Backup Policy

Your backup policy will depend on several factors, such as:

20.12.1.1 Peak Business Loads

You need to take into account peak business loads when scheduling backups for your system as this can reduce system load during peak hours. For example, backups are probably best scheduled for early morning hours such as 2:00 AM.

20.12.1.2 Full and Incremental Backups

Incremental backups (see Incremental Backup) will scan the store for changed data and back up only what has changed. Full backups will back up the entire message store. You need to determine how often the system should perform full as opposed to incremental backups. You’ll probably want to perform incremental backups as a daily maintenance procedure and full backups once a week.

20.12.1.3 Parallel or Serial Backups

When user data is stored on multiple disks, you can back up user groups in parallel if you wish. Depending on system resources, parallel backups can speed up the overall backup procedure. However, you might want to use serial backups if you want to reduce backup impact on the server’s performance. Whether to use parallel or serial backups can depend on many factors, including system load, hardware configuration, how many tape drives are available, and so on.

20.12.2 To Create Backup Groups

A backup group is an arbitrary set of user mailboxes defined by regular expressions. By organizing user mailboxes into backup groups, you can define more flexible backup management.

For example, you could create three backup groups, the first containing user IDs starting with the letters A through L, the second with users whose user IDs begin with M-Z, and the third with users whose user IDs begin with a number. Administrators could use these backup groups to backup mailboxes in parallel, or perhaps only certain groups on one day and other groups on another.

There are several things to remember about backup groups:

  1. These are arbitrary virtual groups of mail users. They do not precisely map to the message store directory (Figure 20–1), although it may look like it.

  2. They are defined by the administrator using UNIX regular expressions.

  3. The regular expressions are defined in the configuration filemsg-svr-base/config/backup-groups.conf

  4. When backup groups are referenced in imsbackup and imsrestore, they use the path format: /partition_name/backup_group

The format of backup-groups.conf is as follows:


group_name=definition
group_name=definition
.
.
.

Using the example described in the paragraph above, the following definitions would be used to create the three backup groups:


groupA=[a-l].*
groupB=[m,-z].*
groupC=[0-9].*

You can now scope imsbackup and imsrestore at several levels. You can backup/restore the whole message store using the backup command:

imsbackup -f device /

To backup all mailboxes for all users in groupA use the following:

imsbackup -f device /partition/groupA

The default partition is called primary.

20.12.2.1 Pre-defined Backup Group

Messaging Server includes one predefined backup group that is available without creating the backup-groups configuration file. This group is called user; it includes all users. For example, the following will backup all users on the primary partition:

imsbackup -f backupfile /primary/user

20.12.3 Messaging Server Backup and Restore Utilities

To back up and restore your data, Messaging Server provides the imsbackup and imsrestore utilities. Note that the imsbackup and imsrestore utilities do not have the advanced features found in general purpose tools like Legato Networker. For example, the utilities have only very limited support for tape auto-changers, and they cannot write a single store to multiple concurrent devices. Comprehensive backup will be achieved via plug-ins to generalized tools like Legato Networker. For more information about using Legato Networker, see 20.12.6 To Use Legato Networker

20.12.3.1 The imsbackup Utility

With imsbackup, you can write selected contents of the message store to any serial device, including magnetic tape, a UNIX pipe, or a plain file. The backup or selected parts of the backup may later be recovered by using the imsrestore utility. The output of imsbackup can be piped to imsrestore.

The following example backs up the entire message store to /dev/rmt/0:


imsbackup -f /dev/rmt/0 /

This backs up the mailboxes of user ID joe to /dev/rmt/0:


imsbackup -f /dev/rmt/0 /primary/user/joe
            

This example backs up all the mailboxes of all the users defined in the backup group groupA to backupfile (see 20.12.2 To Create Backup Groups):


imsbackup -f- /primary/groupA > backupfile
            

Incremental Backup

The following example will back up messages stored from May 1, 2004 at 1:10 pm to the present. The default is to backup all the messages regardless of their dates:


imsbackup -f /dev/rmt/0 -d 20040501:131000 /
               

This command uses the default blocking factor of 20. For a complete syntax description of the imsbackup command, see the Sun Java System Messaging Server 6.3 Administration Reference.

20.12.3.2 The imsrestore Utility

To restore messages from the backup device, use the imsrestore command. For example, the following command restores messages for user1 from the file backupfile.

imsrestore -f backupfile /primary/user1

For a complete syntax description of the imsbackup command, see the Sun Java System Messaging Server 6.3 Administration Reference.

20.12.4 Excluding Bulk Mail When You Perform Backups

When you perform a backup operation, you can specify mailboxes that will be excluded from being backed up. By excluding bulk or trash mailboxes that can accrue large numbers of unimportant messages, you can streamline the backup session, reduce the time to complete the operation, and minimize the disk space required to store the backup data.

To exclude mailboxes, specify a value for the configutil parameter local.store.backup.exclude.

You can specify a single mailbox or a list of mailboxes separated by the ”%’ character. (”%’ is an illegal character in a mailbox name.) For example, you could specify the following values:

Trash

Trash%Bulk Mail%Third Class Mail

In the first example, the folder Trash is excluded. In the second example, the folders Trash, Bulk Mail, and Third Class Mail are excluded.

The backup utility backs up all folders in a user mailbox except those folders specified in the local.store.backup.exclude parameter.

This feature works with the Messaging Server backup utility, Legato Networker, and third-party backup software.

You can override the local.store.backup.exclude setting and back up an excluded mailbox by specifying its full logical name during the operation. Suppose the Trash folder has been excluded. You can still back up Trash by specifying, for example:

/primary/user/user1/trash

However, if you specify

/primary/user/user1

the Trash folder is excluded.

20.12.5 Considerations for Partial Restore

A partial restore is when only a part of the message store is restored. A full restore is when the entire message store is restore. The message store uses a single-copy message system. That is, only a single copy of any message is saved in the store as a single file. Any other instances of that message (like when a message is sent to multiple mailboxes) are stored as links to that copy. Because of this, there are implications when restoring messages. For example:

The following examples demonstrate what happens to a message used by multiple users when a partial restore is performed. Assume there are three messages, all the same, belonging to three users A, B, and C, as follows:


A/INBOX/1
B/INBOX/1
C/INBOX/1

Example 1. In the first example, the system performs a partial backup and full restore procedure as follows:

  1. Back up mailboxes for users B and C.

  2. Delete mailboxes of users B and C.

  3. Restore the backup data from step 1.

In this example, B/INBOX/1 and C/INBOX/1 are assigned a new inode number and the message data is written to a new place on the disk. Only one message is restored; the second message is a hard link to the first message.

Example 2. In this example, the system performs a full backup and a partial restore as follows:

  1. Perform full backup.

  2. Delete mailboxes for user A.

  3. Restore mailboxes for user A.

A/INBOX/1 is assigned a new inode number.

Example 3. In this example, partial restore might require more than one attempt:

  1. Perform full backup.

    B/INBOX/1 and C/INBOX/1 are backed up as links to A/INBOX/1.

  2. Delete mailboxes for users A and B.

  3. Restore mailboxes for user B.

    The restore utilities ask the administrator to restore A/INBOX first.

  4. Restore mailboxes for users A and B.

  5. Delete mailboxes for user A (optional).


    Note –

    If you want to ensure that all messages are restored for a partial restore, you can run the imsbackup command with the -i option. The -i option backs up every message multiple times if necessary.

    If the backup device is seekable (example: a drive or tape), imsrestore seeks to the position containing A/INBOX/1 and restore it as B/INBOX/1. If the backup device is non-seekable example: a UNIX pipe), imsrestore logs the object ID and the ID of the depending (linked) object to a file, and the administrator must invoke imsrestore again with the -r option to restore the missing message references.


20.12.5.1 To Restore Messages from a Mailbox that has Been Incrementally Backed-up

If you are restoring messages from a mailbox that has been incrementally backed-up, and if that mailbox exists on the server on which you wish to restore the messages, then restoring the messages requires a straightforward imesrestore. However, if you wish to restore messages from a mailbox that has been incrementally backed-up, and if that mailbox no longer exists, you must follow different restore procedures.

Use one of the following procedures to restore messages to a mailbox that does not exist on the message store server:

The reason why these instructions must be followed for restoring an incremental backup is as follows: When a mailbox has been deleted or is being migrated, the imsrestore utility recreates the mailbox with the mailbox unique identification validity and message unique identifications (UIDs) stored in the backup archive.

In the past, when imsrestore would recreate a deleted or migrated mailbox, it would assign a new UID validity to the mailbox and new UIDs to the messages. In that situation, a client with cached messages would have to resynchronize the mailbox UID validity and message UIDs. The client would have to download the new data again, increasing the workload on the server.

With the new imsrestore behavior, the client cache remains synchronized, and the restore process operates transparently with no negative impact on performance.

If a mailbox exists, imsrestore assigns new UIDs to the restored messages so that the new UIDs remain consistent with the UIDs already assigned to existing messages. To ensure UID consistency, imsrestore locks the mailbox during the restore operation. However, because imsrestore now uses the mailbox UID validity and message UIDs from the backup archive instead of assigning new UID values, UIDs could become inconsistent if you perform incremental backups and restores.

If you perform incremental backups with the -d date option of the imsbackup utility, you might have to invoke imsrestore multiple times to complete the restore operation. If incremental backups were performed, you must restore the latest full backup and all subsequent incremental backups.

New messages can be delivered to the mailbox between the restore operations, but in this case, the message UIDs can become inconsistent. To prevent inconsistency in the UIDs, you need to take one of the actions describe above.

20.12.6 To Use Legato Networker

Messaging Server includes a backup API that provides an interface with third-party backup tools, such as Legato Networker. The physical message store structure and data format are encapsulated within the backup API. The backup API interacts directly with the message store. It presents a logical view of the message store to the backup service. The backup service uses the conceptual representation of the message store to store and retrieve the backup objects.

Messaging Server provides an Application Specific Module (ASM) that can be invoked by the Legato Networker’s save and recover commands to back up and restore the message store data. The ASM then invokes the Messaging Server imsbackup and imsrestore utilities.


Note –

This section provides information about how to use Legato Networker with the Messaging Server message store. To understand the Legato Networker interface, see your Legato documentation.


ProcedureTo Back Up Data Using Legato Networker

  1. Create a symbolic link from /usr/lib/nsr/imsasm to msg-srv-base/lib/msg/imsasm

  2. From Sun or Legato, obtain a copy of the nsrfile binary and copy it to the following directory:

    /usr/bin/nsr

    Note that this is required only if you are using an older version of Networker (5.x). With Networker 6.0 and above, nsrfile is automatically installed under /usr/bin/nsr.

  3. If you want to back up users by groups, perform the following steps:

    1. Create a backup group file as described in 20.12.2 To Create Backup Groups

    2. To verify your configuration, run mkbackupdir.sh.

      Look at the directory structure created by mkbackupdir.sh.The structure should look similar to that shown in Table 20–4.

      Note that if you do not specify a backup-groups.conf file, the backup process will use the default backup group ALL for all users.

  4. In the directory /nsr/res/, create a res file for your save group to invoke the mkbackupdir.sh script before the backup. See Table 20–4 for an example.


    Note –

    Earlier versions of Legato Networker have a limitation of 64 characters for the save set name. If the name of this directory plus the logical name of the mailbox (for example, /primary/groupA/fred) is greater than 64 characters, then you must run mkbackupdir.sh -p. Therefore, you should use a short path name for the -p option of mkbackupdir.sh. For example the following command will create the backup image under the directory /backup:

    mkbackupdir.sh -p /backup

    Important: The backup directory must be writable by the message store owner (example: mailsrv).


    Table 20–6 shows a sample backup groups directory structure.


    /backup/primary/groupA/amy
                          /bob
                          /carly
                   /groupB/mary
                          /nancy
                          /zelda
                   /groupC/123go
                          /1bill
                          /354hut

    The example below shows a sample res file named IMS.res in the /nsr/res directory:


    type: savepnpc;
    precmd: "echo mkbackupdir started",
       "/usr/siroe/server5/msg-siroe/bin/mkbackupdir.sh -p /backup";
    pstcmd: "echo imsbackup Completed";
    timeout: "12:00 pm";
    
                         

    You are now ready to run the Legato Networker interface as follows:

  5. Create the Messaging Server save group if necessary.

    1. Run nwadmin.

    2. Select Customize | Group | Create.

  6. Create a backup client using savepnpc as the backup command:

    1. Set the save set to the directory created by mkbackupdir.

      For a single session backup, use /backup

      For parallel backups, use /backup/server/group

      Be sure you’ve already created group as defined in 20.12.2 To Create Backup Groups

      You must also set the parallelism to the number of backup sessions.

      See To Back Up Data Using Legato Networker.

  7. Select Group Control | Start to test your backup configuration.

    Example. Creating A Backup Client in Networker:

    To create a backup client in Networker. From nwadmin, select Client | Client Setup | Create


    Name: siroe
    Group: IMS
    Savesets:/backup/primary/groupA
       /backup/secondary/groupB
       /backup/tertiary/groupC
             .
             .
    Backup Command:savepnpc
    Parallelism: 4
    
                         

20.12.6.1 Restoring Data Using Legato Networker

To recover data, you can use the Legato Networker nwrecover interface or the recover command-line utility. The following example recovers user a1’s INBOX:

recover -a -f -s siroe /backup/siroe/groupA/a1/INBOX

The next example recovers the entire message store:

recover -a -f -s siroe /backup/siroe

20.12.7 To Use a Third Party Backup Software (Besides Legato)

Messaging Server provides two message store backup solutions, the command line imsbackup and the Solstice Backup (Legato Networker). A large message store running a single imbackup to backup the entire message store can take a significant amount of time. The Legato solution supports concurrent backup sessions on multiple backup devices. Concurrent backup can shorten backup time dramatically (backups of 25GB of data per hour have been achieved).

If you are using another third party concurrent backup software (for example, Netbackup), you may use the following method to integrate your backup software with the Messaging Server.

ProcedureTo Use a Third Party Backup Software (Besides Legato)

  1. Divide your users into groups (see 20.12.2 To Create Backup Groups) and create a backup-groups.conf file under the directory msg-svr-base/config/.


    Note –

    This backup solution requires additional disk space. To backup all the groups concurrently, the disk space requirement is 2 times the message store size. If you do not have that much disk space, divide your users into smaller groups, and then backup a set of groups at a time. For example group1 - group5, group6 - group10. Remove the group data files after backup.


  2. Run imsbackup to backup each group into files under a staging area.

    The command is imsbackup -f <device> /<instance>/<group>

    You can run multiple imsbackup processes simultaneously. For example:


    # imsbackup -f- /primary/groupA > /bkdata/groupA &
    # imsbackup -f- /primary/groupB > /bkdata/groupB & 
    . . .

    imsbackup does not support large files, if the backup data is larger than 2 GB, you need to use the -f- option to write the data to stdout and then pipe the output to a file.

  3. Use your third party backup software to backup the group data files in the staging area (in our example that is /bkdata).

  4. To restore a user, identify the group filename of the user, restore that file from tape, and then use imsrestore to restore the user from the data file.

    Note that imsrestore does not support large files. If the data file is larger than 2GB. Use the following command:

    # cat /bkdata/groupA | imsrestore -f- /primary/groupA/andy

20.12.8 Troubleshooting Backup and Restore Problems

This section describes common backup and restore problems and their solutions.

20.12.9 Message Store Disaster Backup and Recovery

A disaster refers to a catastrophic failure of the entire message store as opposed to a mailbox or set of mailboxes. That is, a situation where all data on the message store servers are lost. A complete message store disaster restore will consist of restoring the following lost data:

If you want to backup your Message Store for disaster recovery, you can use a file system snapshot tools to take a snapshot of the file system. The snapshot must be a point-in-time file system snapshot. Otherwise, the mboxlist backup cannot be used (the mboxlist database must restored from a complete database snapshot).

It is best to capture all the data (message store partitions, database files and so on) at the same point-in-time, however, if this cannot be done, then you must backup the data in this order:

  1. Database snapshots

  2. Database files

  3. Message store partitions

  4. Configuration data

If the message store partitions and the database files are not backed up with the same point-in-time snapshot, run reconstruct -m after restoring the file system snapshots. This will synchronize the database and the store partitions.

20.13 Monitoring User Access

Messaging Server provides the command, imsconnutil, which allows you to monitor user’s message store access via IMAP, POP and http. You can also determine the last log in and log out of users. This command works on a per message store basis and will not work across message stores.


Note –

Use of this function or other Messaging Server functions to monitor, read or otherwise access user’s email may constitute a potential source of liability if used in violation of applicable laws or regulations or if used in violation of the customer’s own policies or agreements.


This command requires root access by the system user (default: mailsrv), and you must set the configuration variables local.imap.enableuserlist, local.http.enableuserlist, local.enablelastaccess to 1.

To list users currently logged on via IMAP or any web mail client, use the following command:

# imsconnutil -c

To list the last IMAP, POP, or Messenger Express access (log in and log out) of every user on the message store use:

# imsconnutil -a

The following command does two things: 1) it determines whether the specified user is currently logged on via IMAP or Messenger Express or any client that connects via mshttp (note that this does not work for POP because POP users generally do not stay connected), and 2) it lists the last time the users have logged on and off:

# imsconnutil -c -a -u user_ID

Note that a list of users can be input from a file, one user per line, using the following command:

# imsconnutil -c -a -f filename

You can also specify a particular service (imap or http) using the -s flag. For example, to list whether a particular user ID is logged onto IMAP or not, use the following command:

# imsconnutil -c -s imap -u user_ID

Note that the -k option may only work if IMAP IDLE is configured. For a complete description of the imsconnutil syntax, refer to imsconnutil in Sun Java System Messaging Server 6.3 Administration Reference.

Here is some example output:


$ ./imsconnutil -a -u soroork

UID     IMAP last access    HTTP last access    POP last access
=========================================================================
ed   08/Jul/2003:10:49:05   10/Jul/2003:14:55:52  ---NOT-RECORDED---

$ ./imsconnutil -c

IMAP
UID    TIME                AUTH            TO                 FROM
===========================================================================
ed   17/Jun/2003:11:24:03  plain     172.58.73.45:193   129.157.12.73:2631
bil  17/Jun/2003:04:28:43  plain     172.58.73.45:193   129.158.16.34:2340
mia  17/Jun/2003:09:36:54  plain     172.58.73.45:193   192.18.184.103:3744
jay  17/Jun/2003:05:38:46  plain     172.58.73.45:193   129.159.18.123:3687
pau  17/Jun/2003:12:23:28  plaintext 172.58.73.45:193   192.18.194.83:2943
ton  17/Jun/2003:05:38:46  plain     172.58.73.45:193   129.152.18.123:3688
ani  17/Jun/2003:12:26:40  plaintext 172.58.73.45:193   192.18.164.17:1767
ani  17/Jun/2003:12:25:17  plaintext 172.58.73.45:193   129.150.17.34:3117
jac  17/Jun/2003:12:26:32  plaintext 172.58.73.45:193   129.150.17.34:3119
ton  17/Jun/2003:12:25:32  plaintext 172.58.73.45:193   192.18.148.17:1764
===========================================================================
10 users were logged in to imap.
Feature is not enabled for http.
---------------------------------------------------------------------------

20.14 Troubleshooting the Message Store

This section provides guidelines for actively maintaining your message store. In addition, this section describes other message store recovery procedures you can use if the message store becomes corrupted or unexpectedly shuts down. Note that the section on these additional message store recovery procedures is an extension of 20.14.3 Repairing Mailboxes and the Mailboxes Database.

Prior to reading this section, it is strongly recommended that you review this chapter as well as the command-line utility and configutil chapters in the Sun Java System Messaging Server Administration Reference. Topics covered in this section include:

20.14.1 Standard Message Store Monitoring Procedures

This section outlines standard monitoring procedures for the message store. These procedures are helpful for general message store checks, testing, and standard maintenance.

For additional information, see 27.7 Monitoring the Message Store.

20.14.1.1 Check Hardware Space

A message store should have enough additional disk space and hardware resources. When the message store is near the maximum limit of disk space and hardware space, problems might occur within the message store.

Inadequate disk space is one of the most common causes of the mail server problems and failure. Without space to write to the message store, the mail server will fail. In addition, when the available disk space goes below a certain threshold, there will be problems related to message delivery, logging, and so forth. Disk space can be rapidly depleted when the clean up function of the stored process fails and deleted messages are not expunged from the message store.

For information on monitoring disk space, see 20.11.5 To Monitor Disk Space and 27.7 Monitoring the Message Store.

20.14.1.2 Check Log Files

Check the log files to make sure the message store processes are running as configured. Messaging Server creates a separate set of log files for each of the major protocols, or services, it supports: SMTP, IMAP, POP, and HTTP. You can look at the log files in the directory msg-svr-base/log/. You should monitor the log files on a routine basis.

Be aware that logging can impact server performance. The more verbose the logging you specify, the more disk space your log files will occupy for a given amount of time. You should define effective but realistic log rotation, expiration, and backup policies for your server. For information about defining logging policies for your server, see Chapter 25, Managing Logging.

20.14.1.3 Check User IMAP/POP/Webmail Session by Using Telemetry

Messaging Server provides a feature called telemetry that can capture a user’s entire IMAP, POP or HTTP session into a file. This feature is useful for debugging client problems. For example, if a user complains that their message access client is not working as expected, this feature can be used to trace the interaction between the access client and Messaging Server.

To capture a POP session, create the following directory:

msg-svr-base/data/telemetry/pop_or_imap_or_http/userid

To capture a POP session, create the following directory:

msg-svr-base/data/telemetry/pop/userid

To capture an IMAP session, create the following directory:

msg-svr-base/data/telemetry/imap/userid

To capture a Webmail session, create the following directory:

msg-svr-base/data/telemetry/http/userid

Note that the directory must be owned or writable by the messaging server userid.

Messaging Server will create one file per session in that directory. Example output is shown below.


LOGIN redb 2003/11/26 13:03:21
>0.017>1 OK User logged in
<0.047<2 XSERVERINFO MANAGEACCOUNTURL MANAGELISTSURL MANAGEFILTERSURL
>0.003>* XSERVERINFO MANAGEACCOUNTURL {67}
http://redb@cuisine.blue.planet.com:800/bin/user/admin/bin/enduser 
MANAGELISTSURL NIL MANAGEFILTERSURL NIL
2 OK Completed
<0.046<3 select "INBOX"
>0.236>* FLAGS (\Answered flagged draft deleted \Seen $MDNSent Junk)
* OK [PERMANENTFLAGS (\Answered flag draft deleted \Seen $MDNSent Junk \*)]
* 1538 EXISTS
* 0 RECENT
* OK [UNSEEN 23]
* OK [UIDVALIDITY 1046219200]
* OK [UIDNEXT 1968]
3 OK [READ-WRITE] Completed
<0.045<4 UID fetch 1:* (FLAGS)
>0.117>* 1 FETCH (FLAGS (\Seen) UID 330)
* 2 FETCH (FLAGS (\Seen) UID 331)
* 3 FETCH (FLAGS (\Seen) UID 332)
* 4 FETCH (FLAGS (\Seen) UID 333)
* 5 FETCH (FLAGS (\Seen) UID 334)
<etc>

To disable the telemetry logging, move or remove the directory that you created.

20.14.1.4 Check stored Processes

The stored function performs a variety of important tasks such as deadlock and transaction operations of the message database, enforcing aging policies, and expunging and erasing messages stored on disk. If stored stops running, Messaging Server will eventually run into problems. If stored doesn’t start when start-msg is run, no other processes will start.

Table 20–12 stored Operations

stored Operation 

Function  

stored.ckp

Touched when a database checkpoint was initiated. Stamped approximately every 1 minute. 

stored.lcu

Touched at every database log cleanup. Time stamped approximately every 5 minutes. 

stored.per

Touched at every spawn of peruser db write out. Time stamped once an hour. 

For more information on the stored process, see 20.11.6 The stored Daemon chapter of the Sun Java System Messaging Server 6.3 Administration Reference.

For additional information on monitoring the stored function, see 27.7 Monitoring the Message Store

20.14.1.5 Check Database Log Files

Database log files refer to sleepycat transaction checkpointing log files (in directory store_root/mboxlist). If log files accumulate, then database checkpointing is not occurring. In general, there are two or three database log files during a single period of time. If there are more files, it could be a sign of a problem.

20.14.1.6 Check User Folders

If you want to check the user folders, you might run the command reconstruct -r -n (recursive no fix) which will review any user folder and report errors. For more information on the reconstruct command, see 20.14.3 Repairing Mailboxes and the Mailboxes Database

20.14.1.7 Check for Core Files

Core files only exist when processes have unexpectedly terminated. It is important to review these files, particularly when you see a problem in the message store. On Solaris, use coreadm to configure core file location.

20.14.2 Message Store Startup and Recovery

Message store data consists of the messages, index data, and the message store database. While this data is fairly robust, on rare occasions there may be message store data problems in the system. These problems will be indicated in the default log file, and almost always will be fixed transparently. In rare cases an error message in the log file may indicate that you need to run the reconstruct utility. In addition, as a last resort, messages are protected by the backup and restore processes described in 20.12 Backing Up and Restoring the Message Store. This section will focus on the automatic startup and recovery process of stored.

The message store automates many recovery operations which were previously the responsibility of the administrator. These operations are performed by message store daemon stored during startup and include database snapshots and automatic fast recovery as necessary. stored thoroughly checks the message store’s database and automatically initiates repairs if it detects a problem.

stored also provides a comprehensive analysis of the state of the database via status messages to the default log, reporting on repairs done to the message store and automatic attempts to bring it into operation.

20.14.2.1 Automatic Startup and Recovery—Theory of Operations

The stored daemon starts before the other message store processes. It initializes and, if necessary, recovers the message store database. The message store database keeps folder, quota, subscription, and message flag information. The database is logging and transactional, so recovery is already built in. In addition, some database information is copied redundantly in the message index area for each folder.

Although the database is fairly robust, on the rare occasions that it breaks, in most cases stored recovers and repairs it transparently. However, whenever stored is restarted, you should check the default log files to make sure that additional administrative intervention is not required. Status messages in the log file will tell you to run reconstruct if the database requires further rebuilding.

Before opening the message store database, stored analyzes its integrity and sends status messages to the default log under the category of warning. Some messages will be useful to administrators and some messages will consists of coded data to be used for internal analysis. If stored detects any problems, it will attempt to fix the database and try starting it again.

When the database is opened, stored will signal that the rest of the services may start. If the automatic fixes failed, messages in the default log will specify what actions to take. See Error Messages Signifying that reconstruct is Needed

In previous releases, stored could start a recovery process which would take a very long time leaving the administrator wondering if stored was “stuck.” This type of long recovery has been removed and stored should determine a final state in less than a minute. However, if stored needs to employ recovery techniques such as recovering from a snapshot, the process may take a few minutes.

After most recoveries, the database will usually be up-to-date and nothing else will be required. However, some recoveries will require a reconstruct -m in order to synchronize redundant data in the message store. Again, this will be stated in the default log, so it is important to monitor the default log after a startup. Even though the message store will seem to be up and running normally, it is important to run any requested operations such as reconstruct.

Another reason for reading the log file is to determine what caused damage to the database in the first place. Although stored is designed to bring up the message store regardless of any problem on the system, you will still want to try to ascertain cause of the database damage as this may be a sign of a larger hidden problem.

Error Messages Signifying that reconstruct is Needed

This section describes the type of error messages that require reconstruct to be run.

When the error message indicates mailbox error, run reconstruct <mailbox>. Example:

"Invalid cache data for msg 102 in mailbox user/joe/INBOX. Needs reconstruct"

"Mailbox corrupted, missing fixed headers: user/joe/INBOX"

"Mailbox corrupted, start_offset beyond EOF: user/joe/INBOX"

When the error message indicates a database error, run reconstruct -m. Example:

"Removing extra database logs. Run reconstruct -m soon after startup to resync redundant data"

"Recovering database from snapshot. Run reconstruct -m soon after startup to resync redundant data"

Database Snapshots

A snapshot is a hot backup of the database and is used by stored to restore a broken database transparently in a few minutes. This is much quicker than using reconstruct, which relies on the redundant information stored in other areas.

Message Store Database Snapshot—Theory of Operations

Snapshots of the database, located in the mboxlist directory, are taken automatically, by default, once every 24 hours. Snapshots are copied by default into a subdirectory of the store directory. By default, there are five snapshots kept at any given time: one live database, three snapshots, and one database/removed copy. The database/removed copy is newer and is an emergency copy of the database which is put into a subdirectory removed of the mboxlist database directory.

If the recovery process decides to remove the current database because it is determined to be bad, stored will move it into the removed directory if it can. This allows the database to be analyzed if desired.

The data move will only happen once a week. If there is already a copy of the database there, stored will not replace it every time the store comes up. It will only replace it if the data in the removed directory is older than a week. This will prevent the original database which had the problem from being replaced too soon by successive startups.

To Specify Message Store Database Snapshot Interval and Location

There should be five times as much space for the database and snapshots combined. It is highly recommended that the administrator reconfigure snapshots to run on a separate disk, and that it is tuned to the system’s needs.

If stored detects a problem with the database on startup, the best snapshot will automatically be recovered. Three snapshot variables can set the following parameters: the location of the snapshot file, the interval for taking snapshots, number of snapshots saved. These configutil parameters are shown in Table 20–13.

Having a snapshot interval which is too small will result in a frequent burden to the system and a greater chance that a problem in the database will be copied as a snapshot. Having a snapshot interval too large can create a situation where the database will hold the state it had back when the snapshot was taken.

A snapshot interval of a day is recommended and a week or more of snapshots can be useful if a problem remains on the system for a number of days and you wish to go back to a period prior to the point at which the problem existed.

stored monitors the database and is intelligent enough to refuse the latest snapshot if it suspects the database is not perfect. It will instead retrieve the latest most reliable snapshot. Despite the fact that a snapshot may be retrieved from a day ago, the system will use more up to date redundant data and override the older snapshot data, if available.

Thus, the ultimate role the snapshot plays is to get the system as close to up-to-date and ease the burden of the rest of the system trying to rebuild the data on the fly.

Table 20–13 Message Store Database Snapshot Parameters

Parameter  

Description  

local.store.snapshotpath

Location of message store database snapshot files. Either existing absolute path or path relative to the store directory.

Default: dbdata/snapshots

local.store.snapshotinterval

Minutes between snapshots. Valid values: 1 - 46080 

Default: 1440 (1440 minutes = 1 day) 

local.store.snapshotdirs

Number of different snapshots kept. Valid values: 2 -367 

Default: 3 

20.14.3 Repairing Mailboxes and the Mailboxes Database

If one or more mailboxes become corrupt, you can use the reconstruct utility to rebuild the mailboxes or the mailbox database and repair any inconsistencies.

The reconstruct utility rebuilds one or more mailboxes, or the master mailbox file, and repairs any inconsistencies. You can use this utility to recover from almost any form of data corruption in the mail store. See Error Messages Signifying that reconstruct is Needed

Table 20–14 lists the reconstruct options. For detailed syntax and usage requirements, see the reconstruct in Sun Java System Messaging Server 6.3 Administration Reference.

Table 20–14 reconstruct Options

Option  

Description  

-e

Removes the store.exp file before reconstructing. This eliminates any internal store record of removed messages which have not been cleaned out by the store process. It would also be useful to use the -f option when using -i or -e, because these options only work if the folder is actually reconstructed. Similarly, if you use the -n option (which performs a check, not a reconstruction), the -i and -e options do not work.

Running a reconstruct -e will not recover removed messages if reconstruct does not detect damage. An -f will force the reconstruct.

-i

Sets the store.idx file length to zero before reconstructing. It would also be useful to use the -f option when using -i or -e, because these options only work if the folder is actually reconstructed. Similarly, if you use the -n option (which performs a check, not a reconstruction), the -i and -e options do not work.

-f

Forces reconstruct to perform a fix on the mailbox or mailboxes.

-l

Reconstruct lright.db.

-m

Performs a consistency check and, if needed, repairs the mailboxes database. This option examines every mailbox it finds in the spool area, adding or removing entries from the mailbox’s database as appropriate. The utility prints a message to the standard output file whenever it adds or removes an entry from the database. Specifically it fixes folder.db, quota.db, and lright.db

-n

Checks the message store only, without performing a fix on the mailbox or mailboxes. The -n option cannot be used by itself unless a mailbox name is provided. When a mailbox name is not provided, the -n option must be used with the -r option. The -r option may be combined with the -p option. For example, any of the following commands are valid:

reconstruct -n user/dulcinea/INBOX

reconstruct -n -r

reconstruct -n -r -p primary

reconstruct -n -r user/dulcinea/

-o

Obsolete, see mboxutil -o

-o -d filename

Obsolete, see mboxutil -o

-p partition

The -p option is used with the -m option and limits the scope of the reconstruction to the specified partition. If the -p option is not specified, reconstruct defaults to all partitions. Specifically it fixes folder.db and, quota.db, but not lright.db. This is because fixing the lright.db requires scanning the acls for every user in the message store. Performing this for every partition is not very efficient. To fix lright.db run reconstruct -l.

Specify a partition name; do not use a full path name. 

-q

Fixes any inconsistencies in the quota subsystem, such as mailboxes with the wrong quota root or quota roots with the wrong quota usage reported. The -q option can be run while other server processes are running.

-r [mailbox]

Repairs and performs a consistency check of the partition area of the specified mailbox or mailboxes. The -r option also repairs all sub-mailboxes within the specified mailbox. If you specify -r with no mailbox argument, the utility repairs the spool areas of all mailboxes within the user partition directory.

-u user

The -u option is used with the -m option and limits the scope of the reconstruction to the specified user. The -u option must be used with the -p option. If the -u option is not specified, reconstruct defaults to all partitions or to the partition specified with the -p option.

Specify a user name; do not use a full path name. 

20.14.3.1 To Rebuild Mailboxes

To rebuild mailboxes, use the -r option. You should use this option when:

reconstruct -r first runs a consistency check. It reports any inconsistencies and rebuilds only if it detects any problems. Consequently, performance of the reconstruct utility is improved with this release.

You can use reconstruct as described in the following examples:

To rebuild the spool area for the mailboxes belonging to the user daphne, use the following command:

reconstruct -r user/daphne

To rebuild the spool area for all mailboxes listed in the mailbox database:

reconstruct -r

You must use this option with caution, however, because rebuilding the spool area for all mailboxes listed in the mailbox database can take a very long time for large message stores. (See 20.14.3.3 reconstruct Performance.) A better method for failure recovery might be to use multiple disks for the store. If one disk goes down, the entire store does not. If a disk becomes corrupt, you need only rebuild a portion of the store by using the -p option as follows:

reconstruct -r -p subpartition

To rebuild mailboxes listed in the command-line argument only if they are in the primary partition:

reconstruct -p primary mbox1 mbox2 mbox3

If you do need to rebuild all mailboxes in the primary partition:

reconstruct -r -p primary

If you want to force reconstruct to rebuild a folder without performing a consistency check, use the -f option. For example, the following command forces a reconstruct of the user folder daphne:

reconstruct -f -r user/daphne

To check all mailboxes without fixing them, use the -n option as follows:

reconstruct -r -n

20.14.3.2 Checking and Repairing Mailboxes

To perform a high-level consistency check and repair of the mailboxes database:

reconstruct -m

To perform a consistency check and repair of the primary partition:

reconstruct -p primary -m

Note –

Running reconstruct with the -P. and -m flags together will not fix lright.db. This is because fixing the lright.db requires scanning the ACLs for every user in the message store. Performing this for every partition is not very efficient. To fix the lright.db run reconstruct -l


To perform a consistency check and repair of an individual user’s mailbox named john:

reconstruct -p primary -u john -m

You should use the -m option when:

20.14.3.3 reconstruct Performance

The time it takes reconstruct to perform an operation depends on the following factors:

The reconstruct -r option performs an initial consistency check; this check improves reconstruct performance depending on how many folders must be rebuilt.

The following performance was found with a system with approximately 2400 users, a message store of 85GB, and concurrent POP, IMAP, or SMTP activity on the server:


Note –

A reconstruct operation may take significantly less time if the server is not performing ongoing POP, IMAP, HTTP, or SMTP activity.


20.14.4 Common Problems and Solutions

This section lists common message store problems and solutions:

20.14.4.1 Reduced Message Store Performance

Message store problems can occur if the mboxlist database cache is too small. Specifically, Message store performance can slow to unacceptable levels and can even dump core. Refer to Tuning the mboxlist Database Cache.

20.14.4.2 Linux - Messaging Server Patch 120230-08 IMAP, POP and HTTP Servers Not Starting Due to Over Sessions Per Process

After installing this patch, when you try to start Messaging Server, the IMAP, POP and HTTP servers do not start and may send the following example error logs:


http server - log:
[29/May/2006:17:44:37 +051800] usg197 httpd[6751]: General Critical: Not enough file 
descriptors to support 6000 sessions per process; Recommend ulimit -n 12851 or 87 
sessions per process.

pop server - log:
[29/May/2006:17:44:37 +051800] usg197 popd[6749]: General Critical: Not enough file 
descriptors to support 600 sessions per process; Recommend ulimit -n 2651 or 58 
sessions per process.

Once these values setting in /opt/sun/messaging/sbin/configutil then imap server 
failed to start

imap server - log: 
[29/May/2006:17:44:37 +051800] usg197 imapd[6747]: General Critical: Not enough 
file descriptors to support 4000 sessions per process; Recommend ulimit -n 12851 
or 58 sessions per process.

Set the appropriate number of file descriptors for all three server sessions. Additional file descriptors are available by adding a line similar to the following to /etc/sysctl.conf and using sysctl -p to reread that file:


fs.file-max = 65536 

You must also add a line like the following to /etc/security/limits.conf:


*   soft  nofile  65536  
*   hard  nofile  65536

20.14.4.3 Messenger Express or Communications Express Not Loading Mail Page

If the user cannot load any Messenger Express pages or the Communications Express mail page, the problem may be that the data is getting corrupted after compression. This can sometimes happen if the system has deployed a outdated proxy server. To solve this problem, try setting local.service.http.gzip.static and local.service.http.gzip.dynamic to 0 to disable data compression. If this solves the problem, you may want to update the proxy server.

20.14.4.4 Command Using Wildcard Pattern Doesn’t Work

Some UNIX shells may require quotes around wildcard parameters and some will not. For example, the C shell tries to expand arguments containing wild cards (*, ?) as files and will fail if no match is found. These pattern matching arguments may need to be enclosed in quotes to be passed to commands like mboxutil.

For example:

mboxutil -l -p user/usr44*

will work in the Bourne shell, but will fail with tsch and the C shell. These shells would require the following:

mboxutil -l -p "user/usr44*"

If a command using a wildcard pattern doesn’t work, verify whether or not you need to use quotes around wildcards for that shell.

20.14.4.5 Unknown/invalid Partition

A user can get the message “Unknown/invalid partition” in Messenger Express if their mailbox was moved to a new partition which was just created and Messaging Server was not refreshed or restarted. This problem only occurs on new partitions. If you now add additional user mailboxes to this new partition, you will not have to do a refresh/restart of Messaging Server.

20.14.4.6 User Mailbox Directory Problems

A user mailbox problem exists when the damage to the message store is limited to a small number of users and there is no global damage to the system. The following guidelines suggest a process for identifying, analyzing, and resolving a user mailbox directory problem:

  1. Review the log files, the error messages, or any unusual behavior that the user observes.

  2. To keep debugging information and history, copy the entire store_root/mboxlist/ user directory to another location outside the message store.

  3. To find the user folder that might be causing the problem, run the command reconstruct -r -n. If you are unable to find the folder using reconstruct, the folder might not exist in the folder.db.

    If you are unable to find the folder using the reconstruct -r -n command, use the hashdir command to determine the location. For more information on hashdir, see 20.11.2.3 The hashdir Utility and the hashdir utility in the Messaging Server Command-line Utilities chapter of the Sun Java System Messaging Server 6.3 Administration Reference.

  4. Once you find the folder, examine the files, check permissions, and verify the proper file sizes.

  5. Use reconstruct -r (without the -n option) to rebuild the mailbox.

  6. If reconstruct does not detect a problem that you observe, you can force the reconstruction of your mail folders by using the reconstruct -r -f command.

  7. If the folder does not exist in the mboxlist directory (store_root/mboxlist), but exists in the partition directory store_root/partition), there might be a global inconsistency. In this case, you should run the reconstruct -m command.

  8. If the previous steps do not work, you can remove the store.idx file and run the reconstruct command again.


    Caution – Caution –

    You should only remove the store.idx file if you are sure there is a problem in the file that the reconstruct command is unable to find.


  9. If the issue is limited to a problematic message, you should copy the message file to another location outside of the message store and run the command reconstruct -r on the mailbox/ directory.

  10. If you determine the folder exists on the disk (store_root/partition/ directory), but is apparently not in the database (store_root/mboxlist/ directory), run the command reconstruct -m to ensure message store consistency.

For more information on the reconstruct command, see 20.14.3 Repairing Mailboxes and the Mailboxes Database

20.14.4.7 Store Daemon Not Starting

If stored won’t start and returns the following error message:


# msg-svr-base/sbin/start-msg

msg-svr-base: Starting STORE daemon ...Fatal error: Cannot
find group in name service

This indicates that the UNIX group configured in local.servergid cannot be found. Stored and others need to set their gid to that group. Sometimes the group defined by local.servergid gets inadvertently deleted. In this case, create the deleted group, add mailsrv to the group, change ownership of the instance_root and its files to mailsrv and the group.

20.14.4.8 User Mail Not Delivered Due to Mailbox Overflow

The message store has a hard limit of two gigabytes for a store.idx file, which is equivalent to about one million messages in a single mailbox (folder). If a mailbox grows to the point that the store.idx file will attempt to exceed two gigabytes, the user will stop receiving any new email. In addition, other processes that handle that mailbox, such as imapd, popd, mshttpd, could also experience degraded performance.

If this problem arises, you will see errors in mail.log_current such as this:

05-Oct-2005 16:09:09.63 ims-ms Q 7 ... System I/O error. Administrator, check server log for details. System I/O error.

In addition, the MTA log file will have an errors such as this:

[05/Oct/2005:16:09:09 +0900] jmail ims_master[20745]: Store Error: Unable to append cache for user/admin: File too large

You can determine this problem conclusively by looking at the file in the user's message store directory, or by looking in the imta log file to see a more detailed message.

The immediate action is to reduce the size of the file. Either delete some mail, or move some of it to another mailbox. You could also use mboxutil -r to rename the folder out of the way, or mboxutil -d to delete the folder (see 20.11.2.1 The mboxutil Utility.

Long-term, you will need to inform the user of mailbox size limitations, implement an aging policy (see 20.9 To Set the Automatic Message Removal (Expire and Purge) Feature), a quota policy (see 20.8 About Message Store Quotas), set a mail box limit by setting local.store.maxmessages(see configutil Parameters in Sun Java System Messaging Server 6.3 Administration Reference), set up an archiving system, or do something to keep the mailbox size under control.

20.14.4.9 IMAP Events Become Slow

Symptom: After working fine for a short period of time, many IMAP events become unreasonably slow, with some events taking over a second.

Diagnosis: You have the Event Notification Service (ENS) plugin, libibiff, configured, but ENS is not running or not reachable. See Appendix B, Administering Event Notification Service in Messaging Server for ENS details.

Solution: If you want ENS notifications, make sure the ENS is enabled and configured correctly. If you do not want ENS notifications, make sure that libibiff is not being loaded. Typical bad configuration:

local.store.notifyplugin = /opt/sun/comms/messaging/lib/libibiff
local.ens.enable = 0

Use either of the following for solution configurations:

local.store.notifyplugin = 
local.ens.enable = 0

or

local.store.notifyplugin = /opt/sun/comms/messaging/lib/libibiff
local.ens.enable = 1

20.15 Migrating or Moving Mailboxes to a New System

It is sometimes necessary to move existing mailboxes from one Messaging Server system to a another messaging server system. This commonly occurs in the following circumstances:

Messaging Server provides several ways to move mailboxes from one system to another. Each method has its advantages and disadvantages, which are described in the sections below. These methods are described in following sections:

20.15.1 Migrating User Mailboxes to Another Messaging Server While Online

You can use this procedure to migrate the message store from an older version of Messaging Server to a newer version or to move mailboxes from one Sun Messaging Server message store to another. This procedure should work for iPlanet Messaging Server 5.0 and later. It cannot be used to move messages from earlier versions of Messaging Server or a non-Sun Microsystems message store.

The advantages of moving mailboxes using this procedure are as follows:

The disadvantages of moving mailboxes using this procedure are as follows:

20.15.1.1 Incremental Mailbox Migration

Incremental migration provides numerous advantages for safely and effectively moving your message store to a different system or upgrading to a new system, incremental migration allows you to build a new back-end message store system alongside the old back-end message store. You can then test the new system, migrate a few friendly users, then test the new system again. Once you are comfortable with the new system and configuration, and you are comfortable with the migration procedure, you can start migrating real commercial users. These users can be split into discrete backup groups so that during migration, only members of this group are offline, and only for a short time.

Another advantage of on-line incremental migration is that you do not have to plan for a system-wide back out in case your upgrade fails. A back out is a procedure for reverting changes you have made to a system to return the system to the original working state. When doing a migration, you have to plan for failure, which means that for every step in the migration requires a plan to return your system back to its previous operational state.

The problem with offline migrations is that you can't be sure your migration is successful until you've completed all the migration steps and switched the service back on. If the system doesn't work and cannot be quickly fixed, you'll need a back out procedure for all the steps performed. This can be stressful and take some time, during which your users will remain offline.

With an on-line incremental migration you perform the following basic steps:

1. Build the new system alongside the old one so that both can operate independently.

2. Configure the old system for the coexistence with the new.

3. Migrate a group of friendly users and test the new system and its coexistence with the old system.

4. Divide the users on the old system into groups and migrate group by group to the new one as desired.

5. Disassemble the old system.

Because both systems will co-exist, you will have time to test and get comfortable with the new system before migrating to it. If you do have to perform a back out procedure--which should be very unlikely--you only have to plan for steps 2 and 4. Step 2 is easy to revert since you don't ever touch user's data. In step 4, the back out is to revert the user's state back to active and their mailhost attribute back to the old host. No system-wide back out is required.

20.15.1.2 On-line Migration Overview

Migrating mailboxes while remaining online is a straightforward process. Complications arise when you try to ensure that messages in transit to the mailbox (sitting in an MTA channel queue waiting for delivery) are not lost in the migration process. One solution is to hold messages sent during the migration process in a held state and wait for the messages in the various channel queues to be delivered. However, messages can get stuck in queues because of system problems or because a particular user is over quota. In this case, you must address this situation before migrating the mailboxes.

You can take various measures to reduce the likelihood of lost messages and to verify that messages are not stuck in a channel queue, but at a cost of increased complexity of the procedure.

The order and necessity of steps in the procedure vary depending upon your deployment and whether every message addressed to every mailbox must not be lost. This section describes the theory and concepts behind the steps. It is incumbent on you to understand each step and decide which to take and in which order, given your specific deployment. Following is an overview of the process of moving mailboxes. This process might vary depending upon your deployment.

  1. Block user access to the mailboxes being moved.

  2. Temporarily hold messages addressed to the mailbox being moved.

  3. Verify that messages are not stuck in the channel queues.

  4. Change the user's mailhost attribute to the new mailbox location.

  5. Move the mailboxes to the new location.

  6. Release held mail to be delivered to the new mailbox and enable incoming messages to be delivered to the migrated mailboxes.

  7. Examine the old message store to see if any messages were delivered after the migration.

  8. Unblock user access to mailbox.

ProcedureTo Migrate User Mailboxes from One Messaging Server to Another While Online

Before You Begin

The requirements for this type of migration are as follows:


Note –

Some steps apply only if you are upgrading the messaging server from an earlier version to a later version. These steps might not apply if you are only migrating mailboxes from one message store to another. The steps that apply to migrating entire systems are noted.


  1. On the source system, split your user entries to be moved into equal backup groups using the backup-groups.conf file.

    This step is in preparation for the mailbox migration, Step 8, that occurs later in this procedure. See 20.12.2 To Create Backup Groups for detailed instructions.

    You can also place the user names into files and use the -u option in the imsbackup command.

  2. Notify users to be moved that they will not have access to their mailboxes until the move is completed.

    Ensure that users to be moved are logged out of their mail systems before the data move occurs. (See 20.13 Monitoring User Access.)

  3. Set the authentication cache timeout to 0 on the back-end message store and MMP systems, and ALIAS_ENTRY_CACHE_TIMEOUToption to 0 on the MTAs.

    1. On the back end message stores containing the mailboxes to be moved, set the authentication cache timeout to 0.


      configutil -o service.authcachettl -v 0
      

      This step and Step 7 (changing mailUserStatus to hold) immediately prevents users from accessing their mailboxes during migration.

    2. On all MMPs, set the LDAP and authentication cache timeout to 0.

      In ImapProxyAService.cfg and PopProxyAService.cfg set both LdapCacheTTLand AuthCacheTTL to 0.

    3. On any Messaging Server that hosts an MTA that inserts messages into mailboxes that are to be migrated, set the ALIAS_ENTRY_CACHE_TIMEOUT option to 0.

      Messaging Servers hosting an MTA that inserts messages into the migrating mailboxes will typically be the back end message store. However, if the system is using LMTP, then that system will be the inbound MTAs. Check your configuration to make sure.

      Resetting ALIAS_ENTRY_CACHE_TIMEOUT in /msg_svr_base/config/option.dat forces the MTA to bypass the cache and look directly at the LDAP entry so that intermediate channel queues (for example, the conversion or reprocess channels) see the new mailUserStatus (hold) of the users being moved rather then the out-of-date cached information. ALIAS_ENTRY_CACHE_TIMEOUT is in option.dat.

    4. Restart the systems on which the caches were reset.

      You must restart the system for these changes to take place. See 4.4 Starting and Stopping Services for instructions.

  4. Ensure that both your source Messaging Server and destination Messaging Server are up and running.

    The source Messaging Server must be able to route incoming messages to the new destination server.

  5. Change the LDAP attribute mailUserStatus on all user entries whose mailboxes will be moved from active to hold.

    Changing the attribute holds incoming messages in the hold queue and prevents access to the mailboxes over IMAP, POP, and HTTP. Typically, users will be moved in groups of users. If you are moving all the mailboxes of a single domain, you can use the mailDomainStatus attribute.

    For more information on mailUserStatus, see the mailUserStatus in Sun Java Communications Suite 5 Schema Reference.

  6. Make sure that messages addressed to mailboxes being migrated are not stuck in the ims-ms or tcp_lmtp* channel queues (if LMTP has been deployed).

    Use the following commands to see if messages exist in the channel queue directory tree and in the held state (to see .HELD files) addressed to a user to be migrated:


    imsimta qm directory -to=<user_address_to_be_migrated> -directory_tree
    
    imsimta qm directory -to=<user_address_to_be_migrated> -held -directory_tree

    If there are messages in the queue, run these same commands later to see if the MTA has dequeued them. If there are messages that are not being dequeued, then you must address this problem before migrating. This should be a rare occurrence, but possible causes are recipient mailboxes being over quota, mailboxes being locked perhaps because users are logged in and moving messages, the LMTP backend server is not responding, network or name server problems, and so on).

  7. Change the LDAP attribute mailHost in the user entries to be moved as well as in any mail group entries*.

    Use the ldapmodify command to change the entries to the new mail server. Use the ldapmodify that comes with Messaging or Directory Server. Do NOT use the Solaris OS ldapmodify command.

    * You only need to change the mailHost attribute in the mail group entry if the old mail host is being shut down. You can either change this attribute to the new mail host name or just eliminate the attribute altogether. It is optional for mail groups to have a mailHost. Having a mailHost means that only that host can do the group expansion; omitting a mailHost (which is the more common case) means all MTAs can do the group expansion. Note that mail group entries do not have mailboxes to be migrated and typically do not even have the mailhost attribute.

    For more information on mailhost, see mailHost in Sun Java Communications Suite 5 Schema Reference.

  8. Move the mailbox data from the source Messaging Server message store to the destination Messaging Server message store and record the time when started.

    Back up the mailboxes with the imsbackup utility and restore them to the new Messaging Server with the imsrestore utility. For example, to migrate mailboxes from a Messaging Server 5.2 system called oldmail.siroe.com to newmail.siroe.com, run the following command on oldmail.siroe.com:


    /server-root/bin/msg/store/bin/imsbackup -f- /instance/group     \
    | rsh newmail.siroe.com /opt/SUNWmsgsr/lib/msg/imsrestore.sh   \
    -f- -c y -v 1
    

    You can run multiple concurrent backup and restore sessions (one per group) to maximize the transfer rate into the new message store. See the Command Descriptions in Sun Java System Messaging Server 6.3 Administration Reference for more information about the imsbackup and imsrestore utilities as well as 20.12 Backing Up and Restoring the Message Store.


    Note –

    When imsrestore or any processing intensive operation takes significantly more system resources than normal, and continues doing so longer than the msprobe interval, there may be a temporary backlog of DB transaction log files to be cleared. If there are more files than specified in local.store.maxlog, then msprobe may erroneously restart all the processes during a restore. To prevent this from happening, disable msprobe during the imsrestore.



    Note –

    Record the timestamp of when imsbackup is run for later delivery validation.


  9. (Conditional Step for System Upgrades) If your mailbox migration is part of the process of upgrading from an earlier version of Messaging Server to the current version, set this current version of Messaging Server to be the new default Messaging Server for the system.

    Change the DNS A record of oldmail.siroe.com to point to newmail.siroe.com (the server responsible for domain(s) previously hosted on oldmail.siroe.com).

  10. Enable user access to the new message store.

    Set the LDAP attribute mailUserStatus or mailDomainStatus, if applicable, to whatever value it had been before it was changed to hold (for example,active).

  11. Release the messages in the held state on all source Messaging Servers.

    Any system that may be holding incoming messages needs to run the following command to release all the user messages:


    imsimta qm release -channel=hold -scope
    

    where scope can be all, which releases all messages; user, which is the user ID; or domain which is the domain where the user resides.

  12. Reset the authentication cache timeout and the ALIAS_ENTRY_CACHE_TIMEOUT option to the default or desired values and restart the system.

    At this point, you've migrated all the user mailboxes that need to be migrated. Before proceeding, make sure that no new entries in LDAP have been created with the old system as the mailhost, and if some have, migrate them. Also, make sure that no such entries can be created by modifying the provisioning systems.

    You will also want to change the preferredmailhost attribute to the name of the new mail host.

    For back-end messages stores, set authentication cache timeout as follows:


    configutil -o service.authcachettl -v 900
    

    For the MMPs, in ImapProxyAService.cfg and PopProxyAService.cfg set the LdapCacheTTL and AuthCacheTTL options to 900.

    For MTAs, set the ALIAS_ENTRY_CACHE_TIMEOUT option to 600. ALIAS_ENTRY_CACHE_TIMEOUT is in option.dat.

    You must restart the system for these changes to take place. See 4.4 Starting and Stopping Services for instructions.

  13. Ensure that the user clients are pointing to the new mail server.

    After the upgrade finishes, have the users point to the new server through their mail client program (in this example, users would point to newmail.siroe.com from oldmail.siroe.com).

    An alternative is to use a messaging multiplexor (MMP) which obviates the need to have users point their clients directly to the new mail server. The MMP gets that information from the mailHost attribute which is stored in the LDAP user entries and automatically redirects the client to the new server.

  14. After everything works, verify that no messages were delivered to the old message store after the migration.

    Go to the old message store and run mboxutil -l to list the mailboxes. Check the last message delivery timestamp. If a message was delivered after the migration timestamp (the date stamp when you ran the imsbackup command), then migrate those messages with a backup and restore command. Because of the preparatory steps provided, it would be exceedingly rare to see a message delivered after migration.

    Theoretically, a message could be stuck in a queue for the number of days or hours specified by the notices channel keywords (see 10.10.4.3 To Set Notification Message Delivery Intervals).

  15. Remove duplicate messages on the new message store, run the relinker command.

    This command might free disk space on the new message store. See 20.11.7 Reducing Message Store Size Due to Duplicate Storage of Identical Messages.

  16. Remove the old messages from the store you migrated from and delete users from the database on the old store.

    Run the mboxutil -d command. (See 20.11.2.1 The mboxutil Utility).

ProcedureTo Move Mailboxes Using an IMAP client

This procedure can be used anytime messages need to be migrated from one messaging server to a different messaging server. Consider the advantages and disadvantages before moving mailboxes using this method.

The advantages of moving mailboxes using IMAP clients are as follows:

The disadvantages of moving mailboxes using IMAP clients are as follows:

  1. Install and configure the new Messaging Server.

  2. Set local.store.relinker.enabled to yes.

    This will reduce the message store size on the new system caused by duplicate storage of identical messages. See 20.11.7 Reducing Message Store Size Due to Duplicate Storage of Identical Messages for more information.

  3. Provision users on the new Messaging Server.

    You can use Delegated Administrator to do this. As soon as users are provisioned on the new system, newly arriving mail will be delivered to the new INBOX.

  4. Have users configure their mail client to view both new and old Messaging Server mailboxes.

    This may involve setting up a new email account on the client. See mail client documentation for details.

  5. Instruct users to drag folders from their old Messaging Server to their new Messaging Server.

  6. Verify with users that all mailboxes are migrated to the new system, then shut down the user account on the old system.

ProcedureTo Move Mailboxes Using the MoveUser Command

This procedure can be used anytime messages need to be migrated from one messaging server to a different messaging server. It is useful for migrating IMAP mailboxes from a non-Sun Messaging Server to the Sun Java System Messaging Server. Consider the advantages and disadvantages before moving mailboxes using this method.

The advantages of moving mailboxes using the MoveUser command are as follows:

The disadvantages of moving mailboxes using the MoveUser command are as follows:

  1. Install and configure the new Messaging Server.

  2. Set local.store.relinker.enabled to yes.

    This will reduce the message store size on the new system caused by duplicate storage of identical messages. See 20.11.7 Reducing Message Store Size Due to Duplicate Storage of Identical Messages for more information.

  3. Halt incoming mail to the messaging servers.

    Set the user attribute mailUserStatus to hold .

  4. Provision users on the new Messaging Server if needed.

    If you are migrating from a previous version of messaging server, you can use the same LDAP directory and server. MoveUser changes the mailhost attribute in each user entry.

  5. Run the MoveUser command.

    To move all users from host1 to host2, based on account information in the Directory Server siroe.com:


    MoveUser -l \
    "ldap://siroe.com:389/o=siroe.com???(mailhost=host1.domain.com)" \
    -D "cn=Directory Manager" -w password -s host1 -x admin \
    -p password -d host2 -a admin -v password
    

    See the MoveUser in Sun Java System Messaging Server 6.3 Administration Reference for details on the MoveUser command.

  6. Enable user access to the new messaging store.

    Set the mailUserStatus LDAP attribute to active.

  7. Shut down the old system.

ProcedureTo Move Mailboxes Using the imsimport Command

This procedure is specifically used to move mailboxes from UNIX /var/mail format folders into a Sun Java System Messaging Server message store. However, if the messaging server from which you are migrating can convert the IMAP message stores to UNIX /var/mail format, then you can use the imsimport command to migrate messages to the Sun Java System Messaging Server. Consider the advantages and disadvantages before moving mailboxes using this method.

The advantage of moving mailboxes using the imsimport command is as follows:

The disadvantages of moving mailboxes using the imsimport command are as follows:

  1. Install and configure the new Messaging Server.

  2. Set local.store.relinker.enabled to yes.

    This will reduce the message store size on the new system caused by duplicate storage of identical messages. See 20.11.7 Reducing Message Store Size Due to Duplicate Storage of Identical Messages for more information.

  3. Provision users on the new Messaging Server if needed.

    You can use Delegated Administrator to do this. Do not switch over to the new system yet.

  4. Disable user access to both the new and old messaging store.

    Set the mailUserStatus LDAP attribute to hold. User’s mail is sent to the hold queue and access to the mailbox over IMAP, POP, and HTTP is disallowed. MTA and Message Access Servers on the store server must comply with this requirement. This setting overrides any other mailDeliveryOption settings.

  5. If the mail store from the existing mail server is not already in the /var/mail format, convert the mail store to /var/mail files.

    Refer to the third-party mail server documentation.

  6. Run the imsimport command.

    For example:


    imsimport -s /var/mail/joe -d INBOX -u joe
    

    See the imsimport in Sun Java System Messaging Server 6.3 Administration Reference for details on the imsimport command.

  7. Enable user access to the message store.

    Set the mailUserStatus LDAP attribute to active.

  8. Enable user access to the new messaging store.

  9. Shut down the old system.