62 Upgrading the Message Store

This chapter describes the data components and tools of message store, and how they affect data upgrades. It provides the technical background for upgrade planning. If you are using the upgrade procedure described in the Messaging Server Installation and Configuration Guide, you do not need to concern yourself with this level of message store technical detail. However, if you need to customize your message store upgrade process, use this information as a guideline.

For specific information about significant design changes in the message store between versions that can impact disk I/O and performance, see "Significant Changes in the Message Store Between Versions."

Topics:

Architecture and Components

The following figure shows the message store architecture and its components. See "Message Store Directory Layout" for a view of the message store file structure.

Figure 62-1 Message Store and Components

Description of Figure 62-1 follows
Description of ''Figure 62-1 Message Store and Components''

The message store uses the Berkeley Database provided by Sleepycat Software and, since 2006, Oracle Corporation. The message store database files are stored in a directory called mboxlist (see "Message Store Directory Layout," and so it is often called the mboxlist database.

The message store has used a number of versions of the Berkeley Database over the course of its existence. Thus, when you upgrade your message store, the Berkeley Database may also be upgraded. The database engine upgrade has complexities and implications for the data upgrade. (These complexities and implications are handled in the Message Store upgrade tools and instructions described in the Messaging Server Installation and Configuration Guide, but these details are described here for custom upgrade plans.)

The mboxlist database is stored in the BTREE database files. The BTREE database files store information about the message store users and mailboxes (see "Message Store Directory Layout"). The number and types of files, as well as the structure of files themselves have varied over the life of the Berkeley Database.

The Berkeley Database is transactional, so each transaction is logged in a database log file. Database log files are used for recovery. All database changes are recorded in the database log file. When the server is crashed and restarted, it uses the log file to bring the database back to a consistent state. Before you do an upgrade, make sure you have a clean shutdown and recovery by running stored -r.

Message Store Component Version Compatibilities

Table 62-1 contains the version number of various database components in the message store with respect to upgrade.

Table 62-1 Message Store Database Components

Messaging Server Mailbox Berkeley Database Database BTREE Database Log Files

4.15

1_1

2.6

6

2

5.x

1_2

2.6

6

2

6.0

1_2

3.2.9

8

3

6.1

1_2

4.2

9

8

6.2

1_2

4.2

9

8

6.3

1_2

4.4

9

11

7.0

1_3

4.4

9

11

7 Update 1

1_3

4.7.25

9

14

7 Update 2

1_3

4.7.25

9

14

7 Update 3

1_3

4.7.25

9

14

7 Update 4

1_3

4.7.25

9

14

7 Update 5

1_4

5.3.21

9

19

8.0

1_5

6.1.19

10

22

8.0.1

1_5

6.1.26

10

22

In general, when you upgrade Oracle Communications Messaging Server from one version to another, you also have to upgrade the components that have a different version. Certain message components are not compatible with other components. The following section describes the various compatibilities and incompatibilities.

Upgrading the Mailboxes

The message store upgrades mailboxes from version 1_1-1_2 and 1_2-1_4 to the current version automatically. Usually, a mailbox is upgraded when the end users select their mailboxes, or when messages are delivered to the mailboxes after the message store software is upgraded. The message store checks the mailbox version in the mailbox header and upgrades the mailbox if needed.

Mailbox upgrade increases the load on the server. Disable the non-essential tasks during the transition period, such as:.

  • Disable peruser.db archive:

    msconfig -o store.seenckpinterval -v 0 (Unified Configuration)

    or

    configutil -o local.store.seenckpinterval -v 0 (legacy configuration)

  • Disable last access time tracking:

    msconfig -o base.enablelastaccess -v 0 (Unified Configuration)

    or

    configutil -o local.enablelastaccess -v no (legacy configuration)

  • Decrease DB snapshot frequency

  • Decrease log level

  • Temporarily disable delivery

  • Temporarily disable impurge

You can upgrade the mailboxes manually with imcheck -H during off-peak hours. The imcheck command opens every mailbox, which triggers the upgrade. You can run this command to make sure all the mailboxes are upgraded.

To upgrade from mail version 1_1 to 1_4, you have to use a migration tool such as imsbackup and imsrestore.

Downgrading mailbox versions is not automatic. You have to use a migration tool such as imsbackup and imsrestore to downgrade a mailbox.

Upgrading and Downgrading the Berkeley Database (BDB)

The Message Store uses the Berkeley Database to store various data. The email messages themselves are not part of the data stored in the BDB. If the database is upgraded to a version that is incompatible with the previous version, the database files will become incompatible with older versions of the Message Store. We recommend, therefore, that you make a backup copy of the database before the upgrade in case you want to back out of the upgrade.

The primary database which needs to be backed up is located by default in the mboxlist/ directory, and consists of .dbfiles, andlog. files. The __* files are temp files for the database and do not need to be copied. A correct copy of the database ensures data is in a consistent state. The message store and utilities must be shut down. Running stored -r will make sure the cache files are synced to the database files. Both database and log files are required.

If you need to back out of a patch or update which upgrades the Berkeley Database to a version that is incompatible with previous versions and you did not make a backup, you may be able to rely on a previous database snapshot. Database snapshots are located by default in the dbdata/ directory. A valid database snapshot directory will have a .verified file. The.verified file indicates that the snapshot has been recovered, verified, and is ready to be used.

Normally when the store is brought up, the stored process replaces the mboxlist directory with any snapshots needed. In the case where you have backed out of a database upgrade, stored may not take into account that a downgrade has occurred. It may therefore be necessary to replace the valid snapshot manually. To do this, move the current database files in mboxlist/ out of the directory and move all the files from the chosen snapshot directory into the mboxlist/ directory. Be sure to remove the __* tmp files as well. Note that if the store is configured with store.dbtmpdir, the tmp files will be in a different location.

If you have no database backup and no valid snapshots, it may be necessary to move the upgraded database out of the way, and rebuild it from scratch. Under normal circumstances, the Message Store rebuilds the database while allowing users to access their mail. Since doing this puts a heavier load on the system, you should create proper database backups instead.

Normal reparation of the database should be done after putting an older version in place by running the reconstruct -m and reconstruct -r commands.

Some of these manual requirements might be addressed in future releases.

Note that the Berkeley database consist of a number of BTREE files, LOG files and temporary files (tmp file location is configurable with store.dbtmpdir). In order to upgrade the database, run "stored -r" before replacing the new libraries and binaries. Note that "stored -r" runs automatically during the proper upgrade process.

In the unlikely event that stored -r fails, check the store log files to determine the cause, run stored alone to perform a database recovery, and run stored -r again. In normal circumstances, this should not be a problem unless there is some underlying system problem which you should resolve before upgrading.

Database BTREE File

BTREE version 8, 9, and 10 are compatible. Upgrade is not needed.

To upgrade the BTREE files from version 6 (BDB 2.6) to a higher version, copy the database files from the old location (example: /usr/iplanet/server5/msg-store/store/mboxlist for 5.2) to the new mboxlist location (example: /var/opt/SUNWmsgsr/store/mboxlist), then run ims_db_upgrade.

Database Log Files

Database log files cannot be upgraded. When the log version changes, run the legacy version of "stored -r" to process the log files (recover the database) before upgrading. Do not remove the old log files.

The following message is logged when the server restarts.

'Skipping log file...historic log version'

The store daemon creates a new log file with the new version.

IMAPD, MSHTTPD and Convergence

The webmail server (mshttpd) uses imapd to access the Message Store. imapd and mshttpd in 6.3 and 7.x are fully compatible, so simultaneous upgrade is not required. To prevent memory problems due to redundant IMAP sessions from mshttpd, you should run with only one mshttpd process. If you serve a lot of concurrent webmail users, this might require an upgrade to 64-bit so that you have enough virtual address space for the process.

Convergence requires webmail server. To use Convergence, you must upgrade mshttpd to the current version.

Upgrading From Messaging Server 32-bit to 64-bit

Run the legacy version of "stored -r" before upgrading the Messaging Server software from 32-bit to 64-bit.

If you run the 64-bit version, then it is more efficient to run with only one mshttpd process.

You might also want to reduce the number of imapd process.

Migrating from x86 to SPARC

The message store data formats are architecture dependent. You cannot transfer any data components in the matrix from one architecture to another directly. To migrate the message store from an x86 machine to a SPARC machine (or from SPARC to x86), use imsbackup and imsrestore or rehostuser.

stored -r

stored -r performs a final recovery and cleanly removes the database temp files to prepare the database for an upgrade.

For example:

/opt/SUNWmsgsr/lib/stored -r
removing mboxlist environment ... done
removing lock environment ... done
removing session db ... done

Make sure stored -r completes successfully.

The stored -r command requires the watcher process. Make sure that the watcher is running before you perform stored -r.

ims_db_upgrade

ims_db_upgrade copies the database files to a backup directory, upgrades the database files to the current version and validates the new database. If upgrade is not successful, the backup files are restored. If the database is validated, the backup files are removed.

Downgrading

Mailbox and BDB downgrade are not supported. To downgrade a message store to an older version with incompatible mailboxes or databases, you must restore the mailboxes and mboxlist database from backup.

Significant Changes in the Message Store Between Versions

This document describes the significant design changes in the message store that can impact disk I/O and performance. Administrators and deployment designers may need to be aware of these changes for deployments where mboxlist, index, and message data are on different file systems or pools.

Topics:

Changes from Messaging Server 6.3 to Messaging Server 7.0

The following changes were made between Messaging Server 6.3 and Messaging Server 7.0:

Changes to store.idx

The store.idx file is separated into two or more files. The store.idx file contains the mailbox's meta data and a 128 bytes fixed length record for every message in the folder. The store.cN files contain the variable length cache records. The average cache record size is approximately 2 kilobytes.

Benefits

  • Mailbox expunge is much faster

  • Supports very large mailboxes (up to 33554431 messages)

I/O Impact

  • More files (inodes) on the index partitions

  • More I/O for very small mailboxes

  • Initial mailbox access triggers mailbox upgrade automatically

Message Store Maintenance Queue and impurge

A BDB queue is added to the message store to manage maintenance tasks. Cache file purge and message file cleanup tasks are queued and executed by impurge which runs continuously.

Benefits

  • Less I/O overall (especially write)

  • Event driven message store maintenance

  • Higher performance

  • Eliminates sudden increase in load during nightly cleanup

I/O Impact

  • Less write access

  • Higher disk space usage (approximately 10%)

Mailbox Self-Healing (Auto-Repair)

Mailbox corruption is detected and scheduled for repair automatically by impurge.

Benefits

  • Increases availability

  • Reduces mailbox administration cost

I/O Impact

  • Minor I/O increase due to error detection and repair.

Changes from Messaging Server 7 to Messaging Server 7 Update 1

The following change was made between Messaging Server 7 and Messaging Server 7 Update 1:

Berkeley Database Upgrade

The Berkeley Database was upgrade from version 4.4 to version 4.7.25.

Benefits

  • Higher throughput

I/O Impact

  • None

Changes from Messaging Server 7 Update 1 to Messaging Server 7 Update 5

The following changes were made between Messaging Server 7 Update 1 and Messaging Server 7 Update 5:

Changes to the Owner's Seen and Deleted Flags

The owner's seen and deleted flags have been moved from the Berkeley Database to the store.idx file.

Benefits

  • Higher throughput by reducing Berkeley Database lock contentions

  • Performance bottlenecks can be fixed by adding more spindles

I/O Impact

  • Less I/O on the mboxlist file system

  • More I/O on the index file system

  • Initial mailbox access triggers mailbox upgrade automatically

Immediate flag update and state sharing

Flags and ACL changes are flushed to the disk and shared immediately across IMAP sessions. New message arrival are notified immediately.

Benefits

  • IMAP sessions are more up-to-date

I/O Impact

  • More I/O on the index file system

Change to the service.imap.capability.condstore option

The service.imap.capability.condstore option is enabled by default.

Benefits

  • IMAP clients can utilize CONDSTORE

  • Improve overall performance (when CONDSTORE is utilized)

I/O Impact

  • Minimal (because flags and MODSEQ are updated together)

Changes to the Berkeley Database

The Berkeley Database was upgrade from version 4.7.25 to version 5.3.21. The default cache size increased from 16 MB to 64 MB. Moved the maintenance queue to the mboxlist environment.

Benefits

  • Increase performance and scalability

  • Simplify store maintenance tasks

I/O Impact

  • A small increase in database environment (tmp file) size.

Changes to mboxlist and lockdir BDB environments

Mboxlist and lockdir DB environments (store.dbtmpdir and local.lockdir) default to tmpfs.

Benefits

  • Better performance

I/O Impact

  • Less I/O on the data root file system.