20.11.7 Reducing Message Store Size Due to Duplicate Storage of Identical Messages
When a message is sent to multiple recipients, that message is placed in each recipient’s mailbox. Some messaging systems store separate copies of the same message in each recipient’s mailbox. By contrast, the Sun Java System Messaging Server strives to retain a single copy of a message regardless of the number of mailboxes in which that message resides. It does this by creating hard links to that message in the mailboxes containing that message.
When other messaging systems are migrated to the Sun Java Messaging Server, these multiple message copies may be copied over with the migration process. With a large message store, this means that a lot of messages are duplicated unnecessarily. In addition, multiple copies of the same message can be accumulated in normal server operation, for example, from IMAP append operations or other sources.
Messaging Server provides a new command called relinker that removes the excess message copies and replaces them with hard links to a single copy.
20.11.7.1 Relinker Theory of Operations
The relinking function can be run in the command or realtime mode. When the relinker command is run, it scans through the message store partitions, creates or updates the MD5 message digest repository (as hard links), deletes excess message files, and creates the necessary hard links.
The digest repository consists of hard links to the messages in the message store. It is stored in the directory hierarchy partition_path/=md5. This directory is parallel to the user mailbox hierarchy partition_path/=user (see Figure 20–1). Messages in the digest repository are uniquely identified by their MD5 digest. For example, if the digest for fredb/00/1.msg is 4F92E5673E091B43415FFFA05D2E47, then partition/=user/hashdir/hashdir/=fredb/00/1.msg is linked to partition/=md5/hashdir/hashdir/4F92E5673E091B43415FFFA05D2E47EA.msg. If another mailbox has this same message, for example, partition_path/=user/hashdir/hashdir/gregk/00/17.msg, that message will also be hard linked to partition_path/=md5/4F/92/4F92E5673E091B43415FFFA05D2E47EA.msg. This is shown in Figure 20–4.
Figure 20–4 Message Store Digest Repository
For this message, the link count will be three. If both messages are deleted from the mailboxes of fredb and gregk, then the link count will be one and the message can be purged.
The relinker process can also be run in the realtime mode for similar functionality. See 20.11.7.3 Using Relinker in the Realtime Mode for details.
20.11.7.2 Using Relinker in the Command Line Mode
relinker scans through a message store partition, creates or updates the MD5 message repository (as hard links) and deletes excess message files. After relinker scans a store partition, it outputs statistics on the number of unique messages and size of the partition before and after relinking. To run more quickly on an already hashed store, relinker only computes the digest of the messages not yet present in =md5. It also has an option to erase the entire digest repository (which doesn’t affect the user mailboxes).
The syntax for the command is as follows:
relinker [-P. partitionname] [-d]
where partitionname specifies the partition to be processed (default: all partitions) and -d specifies that the digest repository be deleted. Sample output is shown below:
# relinker Processing partition: primary Scanning digest repository... Processing user directories.............................. --------------------------------------------------------- Partition statistics Before After --------------------------------------------------------- Total messages 4531898 4531898 Unique messages 4327531 3847029 Message digests in repository 0 3847029 Space used 99210Mb 90481Mb Space savings from single-copy 3911Mb 12640Mb --------------------------------------------------------- # relinker -d Processing partition: primary Purging digest repository... --------------------------------------------------------- Partition statistics Before After --------------------------------------------------------- Message digests in repository 3847029 0 --------------------------------------------------------- |
Relinker can take a long time to run, especially for the first time if there are no messages are in the repository. This is because it has to calculate the digest for every message (if the relinker criteria is configured to include all messages—see 20.11.7.4 Configuring Relinker for information on configuring relinker criteria.) For example, it could take six hours to process a 100 Gigabyte message store. However, if run-time relinking is enabled see 20.11.7.3 Using Relinker in the Realtime Mode.
If the relinker command line mode is used exclusively, and not the run-time option, it is necessary to purge the digest repository (=md5), otherwise messages purged in the store (=user) will not become available disk space since they still have a link in the digest repository (they become orphaned). If you are just performing a one-time optimization of the store—for example after a migration—you can run relinker once, then delete the entire repository with relinker -d. For repeated purging during migration, it is sufficient to just run the relinker command repeatedly, since each time it runs it also purges the expired or orphaned messages from the repository.
It is safe to run multiple instances of relinker in parallel with each processing a different partition (using the -p option). Messages are only relinked inside the same partition.
20.11.7.3 Using Relinker in the Realtime Mode
The relinker function can be enabled in the realtime mode by setting the configutil parameter local.store.relinker.enabled to yes. Using relinker in the realtime mode will compute the digest of every message delivered (or restored, IMAP appended, and so forth) which matches the configured relinker criteria (20.11.7.4 Configuring Relinker), then look in the repository to see if that digest is already present. If the digest is present, it creates a link to it in the destination mailbox instead of creating a new copy of the message. If there is no digest, it creates the message and adds a link to it in the repository afterwards.
stored scans the digest repositories of each partition and purges the messages having a link count of 1, or which don’t match the relinker criteria. The scan is done one directory at a time over a configurable time period. This is so that the I/O load is evenly distributed and doesn’t noticeably impact other server operations. By default the purge cycle is 24 hours, which means messages can still be present on the disk for up to 24 hours after they’ve been deleted from the store or have exceeded the configured maximum age. This task is enabled when the relinker realtime mode is enabled.
20.11.7.4 Configuring Relinker
Table 20–11 shows the parameters used to set relinker criteria.
Table 20–11 relinker configutil Parameters|
Parameter |
Description |
|---|---|
|
Enables real-time relinking of messages in the append code and stored purge. The relinker command-line tool may be run even if this option is off. However since stored will not purge the repository, relinker -d must be used for this task. Turning this option on affects message delivery performance in exchange for the disk space savings. Default: no |
|
|
Maximum age in hours for messages to be kept in the repository, or considered by the relinker command-line. -1 means no age limit, that is, only purge orphaned messages from the repository. For relinker it means process existing messages regardless of age. Shorter values keep the repository smaller thus allow relinker or stored purge to run faster and reclaim disk space faster, while longer values allow duplicate message relinking over a longer period of time, for example, when users copy the same message to the store several days apart, or when running a migration over several days or weeks. Default: 24 |
|
|
Minimum size in kilobytes for messages to be considered by run-time or command-line relinker. Setting a non-zero value gives up the relinker benefits for smaller messages in exchange for a smaller repository. Default: 0 |
|
|
Approximate duration in hours of an entire stored purge cycle. The actual duration depends on the time it takes to scan each directory in the repository. Smaller values will use more I/O and larger values will not reclaim disk space as fast. 0 means run purge continuously without any pause between directories. -1 means don’t run purge in stored (then purge must be performed using the relinker -d command). Default: 24 |
- © 2010, Oracle Corporation and/or its affiliates