test

Sun Java System Messaging Server 6.3 Administration Guide

20.12 Backing Up and Restoring the Message Store

Message store backup and restore is one of the most common and important administrative tasks. It consists of backing up all the messages and folders on a message store. You must implement a backup and restore policy for your message store to ensure that data is not lost if problems such as the following occur:

You can do message store back up and restore using command-line utilities imsbackup and imsrestore, or the integrated solution that uses Legato NetworkerTM.

Messaging Server provides a single-copy backup procedure. Regardless of how many user folders contain a particular message, during backup, the message file is backed up only once using the first message file found. The second message copy is backed up as a link to the name of the first message file, and so on. imsbackup maintains a hash table of all messages using the device and inode of the message files as the index. This method does have implications when restoring data, however. For more information, see 20.12.5 Considerations for Partial Restore

Note –

It is also possible to do message store backup and restore by backing up all the message files and directories. Refer to 20.12.9 Message Store Disaster Backup and Recovery.

This section contains the following subsections:

20.12.1 Creating a Mailbox Backup Policy

Your backup policy will depend on several factors, such as:

20.12.1.1 Peak Business Loads

You need to take into account peak business loads when scheduling backups for your system as this can reduce system load during peak hours. For example, backups are probably best scheduled for early morning hours such as 2:00 AM.

20.12.1.2 Full and Incremental Backups

Incremental backups (see Incremental Backup) will scan the store for changed data and back up only what has changed. Full backups will back up the entire message store. You need to determine how often the system should perform full as opposed to incremental backups. You’ll probably want to perform incremental backups as a daily maintenance procedure and full backups once a week.

20.12.1.3 Parallel or Serial Backups

When user data is stored on multiple disks, you can back up user groups in parallel if you wish. Depending on system resources, parallel backups can speed up the overall backup procedure. However, you might want to use serial backups if you want to reduce backup impact on the server’s performance. Whether to use parallel or serial backups can depend on many factors, including system load, hardware configuration, how many tape drives are available, and so on.

20.12.2 To Create Backup Groups

A backup group is an arbitrary set of user mailboxes defined by regular expressions. By organizing user mailboxes into backup groups, you can define more flexible backup management.

For example, you could create three backup groups, the first containing user IDs starting with the letters A through L, the second with users whose user IDs begin with M-Z, and the third with users whose user IDs begin with a number. Administrators could use these backup groups to backup mailboxes in parallel, or perhaps only certain groups on one day and other groups on another.

There are several things to remember about backup groups:

  1. These are arbitrary virtual groups of mail users. They do not precisely map to the message store directory (Figure 20–1), although it may look like it.

  2. They are defined by the administrator using UNIX regular expressions.

  3. The regular expressions are defined in the configuration filemsg-svr-base/config/backup-groups.conf

  4. When backup groups are referenced in imsbackup and imsrestore, they use the path format: /partition_name/backup_group

The format of backup-groups.conf is as follows:


group_name=definition
group_name=definition
.
.
.

Using the example described in the paragraph above, the following definitions would be used to create the three backup groups:


groupA=[a-l].*
groupB=[m,-z].*
groupC=[0-9].*

You can now scope imsbackup and imsrestore at several levels. You can backup/restore the whole message store using the backup command:

imsbackup -f device /

To backup all mailboxes for all users in groupA use the following:

imsbackup -f device /partition/groupA

The default partition is called primary.

20.12.2.1 Pre-defined Backup Group

Messaging Server includes one predefined backup group that is available without creating the backup-groups configuration file. This group is called user; it includes all users. For example, the following will backup all users on the primary partition:

imsbackup -f backupfile /primary/user

20.12.3 Messaging Server Backup and Restore Utilities

To back up and restore your data, Messaging Server provides the imsbackup and imsrestore utilities. Note that the imsbackup and imsrestore utilities do not have the advanced features found in general purpose tools like Legato Networker. For example, the utilities have only very limited support for tape auto-changers, and they cannot write a single store to multiple concurrent devices. Comprehensive backup will be achieved via plug-ins to generalized tools like Legato Networker. For more information about using Legato Networker, see 20.12.6 To Use Legato Networker

20.12.3.1 The imsbackup Utility

With imsbackup, you can write selected contents of the message store to any serial device, including magnetic tape, a UNIX pipe, or a plain file. The backup or selected parts of the backup may later be recovered by using the imsrestore utility. The output of imsbackup can be piped to imsrestore.

The following example backs up the entire message store to /dev/rmt/0:


imsbackup -f /dev/rmt/0 /

This backs up the mailboxes of user ID joe to /dev/rmt/0:


imsbackup -f /dev/rmt/0 /primary/user/joe

This example backs up all the mailboxes of all the users defined in the backup group groupA to backupfile (see 20.12.2 To Create Backup Groups):


imsbackup -f- /primary/groupA > backupfile

Incremental Backup

The following example will back up messages stored from May 1, 2004 at 1:10 pm to the present. The default is to backup all the messages regardless of their dates:


imsbackup -f /dev/rmt/0 -d 20040501:131000 /

This command uses the default blocking factor of 20. For a complete syntax description of the imsbackup command, see the Sun Java System Messaging Server 6.3 Administration Reference.

20.12.3.2 The imsrestore Utility

To restore messages from the backup device, use the imsrestore command. For example, the following command restores messages for user1 from the file backupfile.

imsrestore -f backupfile /primary/user1

For a complete syntax description of the imsbackup command, see the Sun Java System Messaging Server 6.3 Administration Reference.

20.12.4 Excluding Bulk Mail When You Perform Backups

When you perform a backup operation, you can specify mailboxes that will be excluded from being backed up. By excluding bulk or trash mailboxes that can accrue large numbers of unimportant messages, you can streamline the backup session, reduce the time to complete the operation, and minimize the disk space required to store the backup data.

To exclude mailboxes, specify a value for the configutil parameter local.store.backup.exclude.

You can specify a single mailbox or a list of mailboxes separated by the ”%’ character. (”%’ is an illegal character in a mailbox name.) For example, you could specify the following values:

Trash

Trash%Bulk Mail%Third Class Mail

In the first example, the folder Trash is excluded. In the second example, the folders Trash, Bulk Mail, and Third Class Mail are excluded.

The backup utility backs up all folders in a user mailbox except those folders specified in the local.store.backup.exclude parameter.

This feature works with the Messaging Server backup utility, Legato Networker, and third-party backup software.

You can override the local.store.backup.exclude setting and back up an excluded mailbox by specifying its full logical name during the operation. Suppose the Trash folder has been excluded. You can still back up Trash by specifying, for example:

/primary/user/user1/trash

However, if you specify

/primary/user/user1

the Trash folder is excluded.

20.12.5 Considerations for Partial Restore

A partial restore is when only a part of the message store is restored. A full restore is when the entire message store is restore. The message store uses a single-copy message system. That is, only a single copy of any message is saved in the store as a single file. Any other instances of that message (like when a message is sent to multiple mailboxes) are stored as links to that copy. Because of this, there are implications when restoring messages. For example:

The following examples demonstrate what happens to a message used by multiple users when a partial restore is performed. Assume there are three messages, all the same, belonging to three users A, B, and C, as follows:


A/INBOX/1
B/INBOX/1
C/INBOX/1

Example 1. In the first example, the system performs a partial backup and full restore procedure as follows:

  1. Back up mailboxes for users B and C.

  2. Delete mailboxes of users B and C.

  3. Restore the backup data from step 1.

In this example, B/INBOX/1 and C/INBOX/1 are assigned a new inode number and the message data is written to a new place on the disk. Only one message is restored; the second message is a hard link to the first message.

Example 2. In this example, the system performs a full backup and a partial restore as follows:

  1. Perform full backup.

  2. Delete mailboxes for user A.

  3. Restore mailboxes for user A.

A/INBOX/1 is assigned a new inode number.

Example 3. In this example, partial restore might require more than one attempt:

  1. Perform full backup.

    B/INBOX/1 and C/INBOX/1 are backed up as links to A/INBOX/1.

  2. Delete mailboxes for users A and B.

  3. Restore mailboxes for user B.

    The restore utilities ask the administrator to restore A/INBOX first.

  4. Restore mailboxes for users A and B.

  5. Delete mailboxes for user A (optional).

    Note –

    If you want to ensure that all messages are restored for a partial restore, you can run the imsbackup command with the -i option. The -i option backs up every message multiple times if necessary.

    If the backup device is seekable (example: a drive or tape), imsrestore seeks to the position containing A/INBOX/1 and restore it as B/INBOX/1. If the backup device is non-seekable example: a UNIX pipe), imsrestore logs the object ID and the ID of the depending (linked) object to a file, and the administrator must invoke imsrestore again with the -r option to restore the missing message references.

20.12.5.1 To Restore Messages from a Mailbox that has Been Incrementally Backed-up

If you are restoring messages from a mailbox that has been incrementally backed-up, and if that mailbox exists on the server on which you wish to restore the messages, then restoring the messages requires a straightforward imesrestore. However, if you wish to restore messages from a mailbox that has been incrementally backed-up, and if that mailbox no longer exists, you must follow different restore procedures.

Use one of the following procedures to restore messages to a mailbox that does not exist on the message store server:

The reason why these instructions must be followed for restoring an incremental backup is as follows: When a mailbox has been deleted or is being migrated, the imsrestore utility recreates the mailbox with the mailbox unique identification validity and message unique identifications (UIDs) stored in the backup archive.

In the past, when imsrestore would recreate a deleted or migrated mailbox, it would assign a new UID validity to the mailbox and new UIDs to the messages. In that situation, a client with cached messages would have to resynchronize the mailbox UID validity and message UIDs. The client would have to download the new data again, increasing the workload on the server.

With the new imsrestore behavior, the client cache remains synchronized, and the restore process operates transparently with no negative impact on performance.

If a mailbox exists, imsrestore assigns new UIDs to the restored messages so that the new UIDs remain consistent with the UIDs already assigned to existing messages. To ensure UID consistency, imsrestore locks the mailbox during the restore operation. However, because imsrestore now uses the mailbox UID validity and message UIDs from the backup archive instead of assigning new UID values, UIDs could become inconsistent if you perform incremental backups and restores.

If you perform incremental backups with the -d date option of the imsbackup utility, you might have to invoke imsrestore multiple times to complete the restore operation. If incremental backups were performed, you must restore the latest full backup and all subsequent incremental backups.

New messages can be delivered to the mailbox between the restore operations, but in this case, the message UIDs can become inconsistent. To prevent inconsistency in the UIDs, you need to take one of the actions describe above.

20.12.6 To Use Legato Networker

Messaging Server includes a backup API that provides an interface with third-party backup tools, such as Legato Networker. The physical message store structure and data format are encapsulated within the backup API. The backup API interacts directly with the message store. It presents a logical view of the message store to the backup service. The backup service uses the conceptual representation of the message store to store and retrieve the backup objects.

Messaging Server provides an Application Specific Module (ASM) that can be invoked by the Legato Networker’s save and recover commands to back up and restore the message store data. The ASM then invokes the Messaging Server imsbackup and imsrestore utilities.

Note –

This section provides information about how to use Legato Networker with the Messaging Server message store. To understand the Legato Networker interface, see your Legato documentation.

ProcedureTo Back Up Data Using Legato Networker

  1. Create a symbolic link from /usr/lib/nsr/imsasm to msg-srv-base/lib/msg/imsasm

  2. From Sun or Legato, obtain a copy of the nsrfile binary and copy it to the following directory:

    /usr/bin/nsr

    Note that this is required only if you are using an older version of Networker (5.x). With Networker 6.0 and above, nsrfile is automatically installed under /usr/bin/nsr.

  3. If you want to back up users by groups, perform the following steps:

    1. Create a backup group file as described in 20.12.2 To Create Backup Groups

    2. To verify your configuration, run mkbackupdir.sh.

      Look at the directory structure created by mkbackupdir.sh.The structure should look similar to that shown in Table 20–4.

      Note that if you do not specify a backup-groups.conf file, the backup process will use the default backup group ALL for all users.

  4. In the directory /nsr/res/, create a res file for your save group to invoke the mkbackupdir.sh script before the backup. See Table 20–4 for an example.

    Note –

    Earlier versions of Legato Networker have a limitation of 64 characters for the save set name. If the name of this directory plus the logical name of the mailbox (for example, /primary/groupA/fred) is greater than 64 characters, then you must run mkbackupdir.sh -p. Therefore, you should use a short path name for the -p option of mkbackupdir.sh. For example the following command will create the backup image under the directory /backup:

    mkbackupdir.sh -p /backup

    Important: The backup directory must be writable by the message store owner (example: mailsrv).

    Table 20–6 shows a sample backup groups directory structure.


    /backup/primary/groupA/amy
                      /bob
                      /carly
               /groupB/mary
                      /nancy
                      /zelda
               /groupC/123go
                      /1bill
                      /354hut

    The example below shows a sample res file named IMS.res in the /nsr/res directory:


    type: savepnpc;
precmd: "echo mkbackupdir started",
   "/usr/siroe/server5/msg-siroe/bin/mkbackupdir.sh -p /backup";
pstcmd: "echo imsbackup Completed";
timeout: "12:00 pm";

    You are now ready to run the Legato Networker interface as follows:

  5. Create the Messaging Server save group if necessary.

    1. Run nwadmin.

    2. Select Customize | Group | Create.

  6. Create a backup client using savepnpc as the backup command:

    1. Set the save set to the directory created by mkbackupdir.

      For a single session backup, use /backup

      For parallel backups, use /backup/server/group

      Be sure you’ve already created group as defined in 20.12.2 To Create Backup Groups

      You must also set the parallelism to the number of backup sessions.

      See To Back Up Data Using Legato Networker.

  7. Select Group Control | Start to test your backup configuration.

    Example. Creating A Backup Client in Networker:

    To create a backup client in Networker. From nwadmin, select Client | Client Setup | Create


    Name: siroe
Group: IMS
Savesets:/backup/primary/groupA
   /backup/secondary/groupB
   /backup/tertiary/groupC
         .
         .
Backup Command:savepnpc
Parallelism: 4

20.12.6.1 Restoring Data Using Legato Networker

To recover data, you can use the Legato Networker nwrecover interface or the recover command-line utility. The following example recovers user a1’s INBOX:

recover -a -f -s siroe /backup/siroe/groupA/a1/INBOX

The next example recovers the entire message store:

recover -a -f -s siroe /backup/siroe

20.12.7 To Use a Third Party Backup Software (Besides Legato)

Messaging Server provides two message store backup solutions, the command line imsbackup and the Solstice Backup (Legato Networker). A large message store running a single imbackup to backup the entire message store can take a significant amount of time. The Legato solution supports concurrent backup sessions on multiple backup devices. Concurrent backup can shorten backup time dramatically (backups of 25GB of data per hour have been achieved).

If you are using another third party concurrent backup software (for example, Netbackup), you may use the following method to integrate your backup software with the Messaging Server.

ProcedureTo Use a Third Party Backup Software (Besides Legato)

  1. Divide your users into groups (see 20.12.2 To Create Backup Groups) and create a backup-groups.conf file under the directory msg-svr-base/config/.

    Note –

    This backup solution requires additional disk space. To backup all the groups concurrently, the disk space requirement is 2 times the message store size. If you do not have that much disk space, divide your users into smaller groups, and then backup a set of groups at a time. For example group1 - group5, group6 - group10. Remove the group data files after backup.

  2. Run imsbackup to backup each group into files under a staging area.

    The command is imsbackup -f <device> /<instance>/<group>

    You can run multiple imsbackup processes simultaneously. For example:


    # imsbackup -f- /primary/groupA > /bkdata/groupA &
# imsbackup -f- /primary/groupB > /bkdata/groupB & 
. . .

    imsbackup does not support large files, if the backup data is larger than 2 GB, you need to use the -f- option to write the data to stdout and then pipe the output to a file.

  3. Use your third party backup software to backup the group data files in the staging area (in our example that is /bkdata).

  4. To restore a user, identify the group filename of the user, restore that file from tape, and then use imsrestore to restore the user from the data file.

    Note that imsrestore does not support large files. If the data file is larger than 2GB. Use the following command:

    # cat /bkdata/groupA | imsrestore -f- /primary/groupA/andy

20.12.8 Troubleshooting Backup and Restore Problems

This section describes common backup and restore problems and their solutions.

20.12.9 Message Store Disaster Backup and Recovery

A disaster refers to a catastrophic failure of the entire message store as opposed to a mailbox or set of mailboxes. That is, a situation where all data on the message store servers are lost. A complete message store disaster restore will consist of restoring the following lost data:

If you want to backup your Message Store for disaster recovery, you can use a file system snapshot tools to take a snapshot of the file system. The snapshot must be a point-in-time file system snapshot. Otherwise, the mboxlist backup cannot be used (the mboxlist database must restored from a complete database snapshot).

It is best to capture all the data (message store partitions, database files and so on) at the same point-in-time, however, if this cannot be done, then you must backup the data in this order:

  1. Database snapshots

  2. Database files

  3. Message store partitions

  4. Configuration data

If the message store partitions and the database files are not backed up with the same point-in-time snapshot, run reconstruct -m after restoring the file system snapshots. This will synchronize the database and the store partitions.