The automatic message removal feature (also known as expire and purge) automatically removes messages from the message store based on a set of administrator-defined criteria. This feature can automatically remove old or overly large messages, seen/deleted messages, messages with certain Subject: lines and so on. This feature allows the following removal criteria:
By folder (mailboxes), users, domains, the entire message store, or specific partitions
Age, in days, that a message has been in the mailbox
Size of message and grace period (days that an oversized message will remain in the message store before purging)
Whether a message has been flagged as seen or deleted
Header strings
This feature is performed by the imexpire utility, which expunges and purges messages. See 20.3 How the Message Store Removes Messages for details on the message removal process.
The server removes messages without warning, so it is important to inform users about automatic message removal policies. Unexpected message removal can be a source of consternation for users and administrators.
imexpire can be invoked from the command line or scheduled to run automatically by the imsched daemon. The administrator specifies a set of expiration rules in a file called store.expirerule. This file specifies the criteria by which messages are removed. There can be multiple files with each put in the directory that pertains to the scope of the rules. That is, rules that apply globally to the entire message store are put in one directory, rules that apply to a partition in another, rules that apply to users in yet another, and so on.
Although global expiration rules can be specified with the configutil command and store.expire.attribute parameters, it is better to use store.expirerule to specify these rules. If too many rules are created using configutil, performance problems can result.
imexpire loads all of the expire rules at start up. By default, imexpire creates one thread per partition. Each thread goes through the list of user folders under its assigned partition and loads the local expire rule files as it goes. The expire function checks each folder against the expire rules applicable to this folder and expunges messages as needed. If there is a store.exp file that exists under the mailbox directory, and there are messages that have been expunged/expired for longer than the time specified by the store.cleanupage configuration parameter, the purge function will permanently remove the message files under the message hash directories and remove the UID records from the store.exp files.
It is also possible to exclude specified users from the expire rules by adding their user ID, one per line, in a file called expire_exclude_list in msg-svr-base/config/.
Automatic message removal requires three steps:
Define automatic message removal policy: Which messages will be automatically removed? What users, folders, domains and partitions will have messages automatically removed? What size, message age, headers will define the removal criteria. Define the scope of messages to be removed. See 20.9.2.1 To Define Automatic Message Removal Policy
Specify theimexpire rules to implement this policy. See 20.9.2.2 To Set Rules Implementing Automatic Message Removal Policy
Specify the imexpire scheduling. See 20.9.2.3 To Schedule Automatic Message Removal and Logging Level
Define your automatic message removal policy by specifying the criteria for removal. imexpire allows for removal using the following criteria:
Age of Message. Automatically remove messages older than X days. Attribute: messagedays.
Message Count. Automatically remove messages in a folder exceeding X messages. Attribute: messagecount.
Age of Oversized Message. Automatically remove messages that exceed X bytes after Y days grace period. Attributes: messagesize and messagesizedays.
Seen and Deleted Message Flag. Automatically remove messages with the Seen or Deleted flag set. These criteria can be set to “and” or “or.” If set to or, the message’s Seen/Delete flag will cause automatic deletion regardless of other criteria. If set to and, the message’s Seen/Delete flag must be set along with passing all other specified criteria. Attributes: seen and deleted.
Header Field of Message. Allows you specify a header and string as criteria for removing a message. For example, removing all messages with the header “Subject: Work from Home!”
Folder of Messages. Allows you to specify the folder on which to remove messages. Attribute: folderpattern. Note that this attribute only uses the modified UTF-7 character set.
imexpire does not allow you to delete or preserve messages based on how long it has been since that message was read. For example, you cannot specify that messages that have not been read for 200 days will be removed.
Example 1: Remove all messages 365 days old in a folder exceeding 1,000 messages.
Example 2: Remove messages in domain siroe.com that are older than 180 days.
Example 3: Remove all messages that have been marked as deleted.
Example 4: Remove messages in sesta.com that have been marked as seen, are older than 30 days, are larger than 100 kilobytes, from folders exceeding 1,000 messages, with the header X-spam.
To implement the automatic message removal policy defined in the previous section, you must set the imexpire rules. Rules are set by putting them into a store.expirerule file. An example of two global store.expirerule rules is shown below:
Rule1.regexp: 1 Rule1.folderpattern: user/.*/trash Rule1.messagedays: 2 Rule2:regexp: 1 Rule2.folderpattern: user/.* Rule2.messagedays: 14 |
In this example, Rule 1 specifies that all messages in the trash folder will be remove after two days. Rule 2 specifies that all messages in the message store will be removed after 14 days.
This section consists of the following subsections:
This section sets the guidelines for the store.expirerule file rules.
In earlier Messaging Server releases, expiration rules could be set with configutil parameters store.expirerule.attribute (see configutil Parameters in Sun Java System Messaging Server 6.3 Administration Reference.) This is still true, but expire rules using header constraints (example: expiring a message with a specific subject line) are not supported. Also, regular expressions in the expire rules created with configutil need to be POSIX compliant rules. If you want to use UNIX compliant regular expressions you will have to use the store.expire file. In addition, using both configutil options and the global store.expirerule configuration file is not supported. If the configuration file is present, configutil options are not used. In any case, it is best to use store.expirerule to specify all expiration rules.
Rules are specified in a file called store.expirerule.
Multiple expiration criteria can be specified with the same rule. (Example above.)
Rules can apply to the entire message store (global rules), a partition, a user, or a folder.
The global rules are stored in msg-svr-base/config/store.expirerule
Each global rule will be checked against every mailbox, which can cause some processing overhead depending on the number of global rules you specify. For this reason, you should not specify partition, mailbox or user rules in the global rules file. In general, you should try not to put any more expiration rules than necessary in this file.
Partition rules are stored in store_root/partition/partition_name/store.expirerule.
User rules are specified in store_root/partition/partition_name/userid/store.expirerule or by specifying the folderpattern rule to be user/userid/.*
Folder rules are specified in store_root/partition/partition_name/userid/folder/store.expirerule or by specifying the folderpattern rule to be user/userid/folder
Note that multiple non-global rules (user, folder, partition) using rule_name was only implemented in the Messaging Server 6.2p4 release and later.
Multiple expire rules can be applied to a mailbox at the same time. An expire policy for a mailbox consists of global rules and local rules. Local rules apply to the mailbox under the same directory and all of its sub-folders.
imexpire unifies all of the expiration rules applying to a mailbox, unless there is an exclusive rule specified for this mailbox (see Table 20–8). The resulting rule set represents the most restrictive expiration policy based on all applicable rules. For example, if rule X expires messages such that the maximum message life is 10 days, and rule Y specifies 5 days, the union will be 5 days.
Attribute |
Description (Attribute Value) |
---|---|
Specifies an action to perform on the messages caught by the expire rules. The possible values are: discard discards the message. This is the default. report action prints the mailbox name, uid-validity and uid to stdout. archive archives the message with Sun Compliance and Content Management System and then discards the message. fileinto: folder action folders the message into the specific folder. The shared folder prefix can be used to folder messages to folders owned by another user. |
|
Specifies whether or not this is an exclusive rule. If specified as exclusive, only this rule applies to the specified mailbox(s) and all other rules are ignored. If more than one exclusive rule exists, the last exclusive rule loaded will be used. For example, if a global and a local exclusive rule are specified, the local rule will be used. If there is more than one global exclusive rule, the last global rule listed by configutil is used. (1/0) |
|
Specifies the folders affected by this rule. The format must start with a user/, which represents the directory store_root/partition/*/ See Table 20–9. (POSIX regular expression) |
|
Maximum number of messages in a folder. Oldest messages are expunged as additional messages are delivered. (integer) |
|
Maximum size of folder before the oldest messages are expunged when additional messages are delivered. (integer in bytes) |
|
Number of days in the message store before being expunged. (integer) |
|
Maximum size of message in bytes before it is marked to be expunged. (integer) |
|
Grace period. Days an over-sized message should remain in a folder. (integer) |
|
messageheader.header |
Specifies a header field and string that marks a message for removal. Values are not case-sensitive and regular expressions are not recognized.Example: Rule1.messageheader.Subject: Get Rich Now! For the header Expires and Expiry-Date, imexpire will remove the message if the date value specified with these header fields is older than the messagedays attribute. If multiple expiration header fields are specified, the earliest expiration date will be used. (string). |
regexp |
Enable UNIX regular expressions in rules creation. (1 or 0). If not specified, IMAP expressions will be used. |
Number of days the messages are saved in a folder until they are expunged. |
|
seen is a message status flag set by the system when the user opens a message. If the attribute seen is set to and, then the message must be seen and other criteria must be met before the rule is fulfilled. If the attribute seen is set to or, then the message only needs to be seen or another criteria be met before the rule is fulfilled. (and/or). |
|
A Sieve rule specifying message selection criteria. Example: Rule17.sieve: header :contains “Subject” “Vigara” |
|
deleted is a message status flag set by the system when the user deletes a message. If the attribute deleted is set to and, then the message must be deleted and another criteria must be met before the rule is fulfilled. If the attribute deleted is set to or, then the message only needs to be seen or another criteria be met before the rule is fulfilled. (and/or) |
The IMAP protocol specifies that mailbox names use modified UTF-7 encoding. Messaging Server supports localized character sets on external interfaces so that mailbox names can be localized. Internally, however, the system converts the localized name to UTF-7. Thus, a folder that has a localize mailbox name on a client will have a corresponding mailbox file name in UTF-7. (Note that IMAP error messages will output mailbox names in UTF-7 and not the localized character set.)
In general, most message store utilities that require mailbox names expect the names in the localized character set, although they may have an option flag that allows a different character set to be used. These utilities include reconstruct, mboxutil, imsbackup, imsrestore, and hashdir. However, imexpire requires that the mailbox name, specified as the attribute folderpattern,be in UTF-7. Using a localized name will not work.
To obtain the appropriate folderpattern for imexpire it may be necessary to convert a localized mailbox name to the modified UTF-7 equivalent. This can be done using the mboxutil -E command as follows:
The first mboxutil shows the localized filename. The second mboxutil shows the filename in modified UTF-7. It is also possible to use the IMAP list command:
To convert the local charset to modified UTF-7 encoding, use the mboxutil command with the -E option:
Note that mboxutil -E can be used for any command that requires the use of a UTF-7 mailbox name including imexpire.
Automatic message removal rules are set by specifying rules in a store.expirerule file. The store.expirerule file contains one expire criteria per line. An expire criteria of the global rule configuration file (msg-svr-base/data/store/store.expirerule) has the following format:
rule_name.attribute: value
An expiration rule for a user or mailbox rule configuration file has this format:
attribute: value
Example 20–4 shows a set of global expiration rules in msg-svr-base/config/store.expirerule.
Rule 1 sets the global expiration policy (that is, policy that applies to all messages), as follows:
Enable UNIX regular expressions in rules creation.
Removes messages larger than 100,000 bytes after 3 days.
Removes messages deleted by the user.
Removes any message with the strings “Vigara Now!” or “XXX Porn!” in the Subject: header.
Limits all folders to 1,000 messages. After 1,000 messages, the system removes the oldest messages on a folder to keep the total to 1,000.
Removes all messages older than 365 days.
Rule 2 sets the automatic message removal policy for users at the hosted domain siroe.com. It limits mailbox sizes to 1 megabyte, removes messages that have been deleted, and removes messages older than 14 days.
Rule 3 sets the automatic message removal policy for messages in the inbox folder of user f.dostoevski. It removes messages with a subject line having the expression “On-line Casino.”
Rule1.regexp: 1 Rule1.folderpattern: user/.* Rule1.messagesize: 100000 Rule1.messagesizedays: 3 Rule1.deleted: or Rule1.Subject: Vigara Now! Rule1.Subject: XXX Porn! Rule1.messagecount: 1000 Rule1.messagedays: 365 Rule2.regexp: 1 Rule2.folderpattern: user/.*@siroe.com/.* Rule2.exclusive: 1 Rule2.deleted: or Rule2.messagedays: 14 Rule2.messagecount: 1000 Rule3.folderpattern: user/f.dostoevski/inbox Rule3.Subject: *On-line Casino* |
Folder patterns can be specified using POSIX regular expressions by setting the imexpire attribute regex to 1. If not specified, IMAP expressions will be used. The format must start with a user/ followed by a pattern. Table 20–9 shows the folder pattern for various folders.)
Table 20–9 imexpire Folder Patterns Using Regular Expressions
Folder Pattern |
Scope |
---|---|
user/userid/.* |
Apply rule to all messages in all folders of userid. |
user/userid/Sent |
Apply rule to messages of userid in folder Sent: |
user/.* |
Apply rule to entire message store. |
user/.*/trash |
Apply rule to the trash folder of all users. |
user/.*@siroe.com/.* |
Apply rule to folders in hosted domain siroe.com: |
user/[^@]*/.* |
Apply rule to folders in default domain. |
Automatic message removal is activated by the imsched scheduling daemon. By default, imsched invokes imexpire at 23:00 every day and messages are both expunged and purged. This schedule can be customized by setting the configutil parameter local.schedule.expire and store.cleanupage described in Table 20–10.
Expire and purge can take a long time to complete on a large message store, so you may wish to experiment and decide how often to run these processes. For example, if an expire/purge cycle takes 10 hours, you may not want the default schedule of running expire and purge once a day. Schedule expire and purge using the imexpire command and the automatic task scheduling parameter (see4.6 To Schedule Automatic Tasks). For example:
configutil -o local.schedule.expire -v "0 1 * * 6 /opt/SUNWmsgsr/sbin/imexpire -e" configutil -o local.schedule.mspurge -v "0 23 * * * /opt/SUNWmsgsr/sbin/imexpire -c" |
In this example, messages are expired at 1AM Saturdays and purged every night at 11PM. If no purge schedule is set, , imexpire will perform purge after an expire.
Table 20–10 Expire and Purge configutil Log and Scheduling Parameters
Parameter |
Description |
---|---|
Interval for running imexpire. Uses UNIX crontab format: minute hour day-of-month month-of-year day-of-week The values are separated by a space or tab and can be 0-59, 0-23, 1-31, 1-12 and 0-6 (with 0=Sunday) respectively. Each time field can be either an asterisk (meaning all legal values), a list of comma-separated values, or a range of two values separated by a hyphen. Note that days can be specified by both day of the month and day of the week, however, it is not typical to use them both since the number of such occurrences are very small. If they are both specified, then both will be required. For example, setting the 17th day of the month and Tuesday will require both values to be true. Note that you can also use the -e and -c flags with imexpire to and expire only or purge only respectively. See imexpire in Sun Java System Messaging Server 6.3 Administration Reference. Interval Examples: 1) Run imexpire at 12:30am, 8:30am, and 4:30pm: 30 0,8,16 * * * /opt/SUNWmsgsr/sbin/imexpire 2) Run imexpire at weekday morning at 3:15 am: 15 3 * * 1-5 /opt/SUNWmsgsr/sbin/imexpire 3) Run imexpire only on Mondays: 0 0 * * 1 /opt/SUNWmsgsr/sbin/imexpire Default: 0 23 * * * /opt/SUNWmsgsr/sbin/imexpire To disable: set local.schedule.expire.enable to NO. |
|
Age (in hours) of expired or expunged message before purge will permanently remove it. Default: None |
|
Specify a log level: 1 = log summary for the entire expire session. 2 = log one message per mailbox expired. 3 = log one message per message expired. Default: 1 |
imexpire will log a summary to the default log file upon completion. If expire is invoked from the command line, the -v (verbose) and -d (debug) option can be used to instruct imexpire to log detail status/debug messages to stderr. If imexpire is invoked by imsched, the configutil parameter local.store.expire.loglevel can be set to 1,2 or 3 for different levels of logging. Loglevel 1 is the default, it will log a summary for the entire expire session. Loglevel 2 will log one message per mailbox expired. Loglevel 3 will log one message per message expired.
Exclude specified users from the expire rules by adding their user ID, one per line, in a file called expire_exclude_list in msg-svr-base/config/. Or, configure a dummy exclusive expire rule under the user's mailbox.