Sun StorageTek Storage Archive Manager Troubleshooting Guide

This chapter provides the backup and dump processes and information you need in order to keep your data safe and prepare for any disaster. For more information on planning for disaster recovery, see Planning for Disaster Recovery.

This chapter contains the following sections:

• Guarding Against or Troubleshooting Data Loss
• Troubleshooting an Inaccessible File System
• Backup and Recovery Commands and Tools
• Files Requiring Backup
• Preparing a Disaster Recovery Plan
• Using Archiver Logs
• Guidelines for Performing Metadata Dumps
• Metadata Used in Disaster Recovery
• Guidelines for Performing Metadata Dumps
• Backing Up the Metadata in SAM-QFS File Systems
• Creating samfsdump Dump Files
• Using Archiver Logs
• Storing Copies of Disaster Recovery Files and Metadata

---

Guarding Against or Troubleshooting Data Loss

TABLE 4-1 shows common causes of data loss, with notes and suggestions about how to avoid or respond to each type of loss.

---

Troubleshooting an Inaccessible File System

Some apparent data losses are actually caused by cabling problems or configuration changes. Be sure to eliminate the fundamental causes for a failure before commencing a data recovery process. Back up anything you change before you change it, if possible.

Caution - Do not reformat a disk, relabel a tape, or make other irreversible changes until you are convinced that the data on the disk or tape is completely unrecoverable.

1. Check cables and terminators.

2. If you cannot read a tape or magneto-optical cartridge, try cleaning the heads in the drive, or try reading the cartridge in a different drive.

3. Check the current state of your hardware configuration against the documented hardware configuration.

Go to Step 4 only when you are certain that a configuration error is not to blame.

4. Unmount the file system, and run samfsck(1M).

For example:

# umount file-system-name

# samfsck file-system-name

5. If you find the file system is still inaccessible, follow the procedures in the other chapters in this manual to restore the file system.

---

Backup and Recovery Commands and Tools

The following sections provide information about some of the commands and tools you can use to back up your data.

Disaster Recovery Commands

TABLE 4-2 summarizes the commands used most frequently in disaster recovery efforts.

For more information about these commands, see their man(1) pages. Other scripts and helpful sample files are located /opt/SUNWsamfs/examples or are available from Sun Microsystems.

Disaster Recovery Utilities

TABLE 4-3 describes some disaster recovery utilities in the /opt/SUNWsamfs/examples directory and explains their purpose. You must modify all of the listed shell scripts, except for recover.sh(1M), to suit your configuration before using them. See the comments in the files.

The samexplorer Script

The samexplorer script (called info.sh in software versions before 4U1) creates a file containing all the configuration information needed for complete reconstruction of a SAM-QFS installation should you ever need to rebuild the system. You can use the crontab(1) command with the -e option to create a cron(1M) job to run the samexplorer script at desired intervals. The script writes the reconfiguration information to /tmp/SAMreport.hostname.YYYYMMDD.HHMMZ.tar.gz (where Z indicates the time zone).

Although the /opt/SUNWsamfs/sbin/samexplorer script is not a backup utility, it should be run whenever changes are made to the system's configuration.

Make sure that the SAMreport file is moved from the /tmp directory after creation to a fixed disk that is separate from the configuration files and outside the SAM-QFS environment. For more information about managing the SAMreport file, see the samexplorer(1M) man page.

---

Files Requiring Backup

TABLE 4-4 lists the files that should be backed up and the recommended frequency of backups to a location outside the file system environment.

Except where specified otherwise, use whatever backup procedures you choose.

---

Preparing a Disaster Recovery Plan

For SAM-QFS file systems, you should have the following in place in anticipation of needing them for disaster recovery:

• Up-to-date archive copies

The effectiveness of any of the SAM-QFS recovery methods relies primarily on frequent archiving.

See Guidelines for Performing Metadata Dumps

• Up-to-date metadata backups

See Metadata Used in Disaster Recovery.

• Archiver logs

If recent metadata is not available, archiver logs can help you re-create the file system directly from archive media.

See Using Archiver Logs.

In addition, consider the following questions when preparing your site's disaster recovery plan:

• What is the right number of samfsdump or qfsdump files to retain at your site?
• For a Sun StorageTek QFS file system, use the qfsdump(1M) command. This command generates a dump of both metadata and data.

See the Sun StorageTek QFS Installation and Upgrade Guide for how to back up Sun StorageTek QFS metadata.

• For a SAM-QFS file system, you can use the samfsdump(1M) command with or without the -u option.

The samfsdump(1M) command with the -u option dumps file data for files that do not have a current archive copy. The dump files are substantially larger with than without the -u option, and the command takes longer to complete. However, restoration of the output from samfsdump with -u restores the file system back to its state when the dump was taken.

The samfsdump(1M) command without the -u option generates a metadata dump file. A metadata dump file is relatively small, so you should be able to store many more metadata dump files than data dump files. Restoration of the output of samfsdump without the -u option is quicker than with the -u option, because the data is not restored until accessed by a user.

Retain enough data and metadata to ensure that you can restore the file systems according to your site's needs. The appropriate number of dumps to save depends, in part, on how actively the system administrator monitors the dump output. If an administrator is monitoring the system daily to make sure the samfsdump(1M) or qfsdump(1M) dumps are succeeding and that there are enough tapes available, as well as investigating dump errors, then keeping a minimum number of dump files to cover vacations, long weekends, and other absences might be enough.

• If you are archiving data, are you actively recycling archive media? If so, be sure to schedule metadata copies to occur after recycling is complete.

If your site is using the sam-recycler(1M) command to reclaim space on archive media, it is critical that you make metadata copies after sam-recycler has completed its work. If a metadata dump is created before sam-recycler exits, the information in the metadump about archive copies becomes out of date as soon as sam-recycler runs. Also, some archive copies may be made inaccessible because the sam-recycler command may cause archive media to be relabeled.

Check root's crontab(1) entry to find out if and when the sam-recycler command is being run, and then, if necessary, schedule the creation of metadump files around the sam-recycler execution times. For more about recycling, see the Sun StorageTek Storage Archive Manager Archive Configuration and Administration Guide.

• How much data should you store off site, and in what format?

Off-site data storage is an essential part of a disaster recovery plan. In the event of a disaster, the only safe data repository might be an off-site vault. Beyond the recommended two copies of all files and metadata that you should be keeping in house as a safeguard against media failure, consider making a third copy on removable media and storing it off site.

Sun SAM-Remote offers you the additional alternative of making archive copies in remote locations on a local area network (LAN) or wide area network (WAN). Multiple Sun SAM-Remote servers can be configured as clients to one another in a reciprocal disaster recovery strategy.

• Is it sufficient to restore only the metadata to a predisaster state, or do you need also to restore all files that were online when the disaster happened?
• The samfsrestore(1M) command can restore a SAM-QFS file or file system to the state reflected in the samfsdump file. After the samfsrestore(1M) command is run, the metadata is restored, but the file data remains offline.

If you need to restore all files that were online, you must run the samfsrestore command with the -g option.

The log file generated by the samfsrestore command's -g option contains a list of all files that were on the disk when the samfsdump(1M) command was run. This log file can be used in conjunction with the restore.sh shell script to restore the files on disk to their predisaster state. The restore.sh script takes the log file as input and generates stage requests for files listed in the log. By default, the restore.sh script restores all files listed in the log file.

If your site has thousands of files that need to be staged, consider splitting the log file into manageable chunks and running the restore.sh script against each of those chunks separately to ensure that the staging process does not overwhelm the system. You can also use this approach to ensure that the most critical files are restored first. For more information, see the comments in /opt/SUNWsamfs/examples/restore.sh.

---

Using SAM-QFS Archiving Features

The features of SAM-QFS file systems described in TABLE 4-5 streamline and speed up data restoration and minimize the risk of losing data in the case of an unplanned system outage.

---

Metadata Used in Disaster Recovery

Metadata consists of information about files, directories, access control lists, symbolic links, removable media, segmented files, and the indexes of segmented files. Metadata must be restored before lost data can be retrieved.

With up-to-date metadata, the data can be restored as follows:

• File data can be restored even if the file has been removed from the file system.
• Individual files or entire file systems can be moved from one file system to another, or even from one server to another.

.inodes File Characteristics

In Sun StorageTek QFS file systems, the .inodes file contains all the metadata except for the directory namespace (which consists of the path names to the directories where the files are stored). The.inodes file is located in the root (/) directory of the file system. For a file system to be restored, the .inodes file is needed, along with the additional metadata.

FIGURE 4-1 illustrates some characteristics of the .inodes file. The arrows indicate that the .inodes file points to file contents on disk and to the directory namespace, and that the namespace also points back to the .inodes file. In SAM-QFS file systems where archiving is being done, the .inodes file also points to archived copies.

FIGURE 4-1 The .inodes File in Sun StorageTek QFS File Systems

The .inodes file is not archived. For more about protecting the .inodes file in these types of file systems, see Guidelines for Performing Metadata Dumps and Backing Up the Metadata in SAM-QFS File Systems.

Directory Path Name Synchronization

As indicated in FIGURE 4-1, the namespace (in the form of directories) does not point to the archive media. The directory path names for each archived file are copied into the headers of the tar(1) files on the archive media that contain the files, but the directory path names in the tar file headers may get out of sync with the actual locations of the files on the disk.

One reason the two path names can get out of sync is that the path names in the tar file header do not show the originating file system. For example, the full directory path name /samfs1/dir1/filea would appear in the tar file header one of the following forms, without the component that shows the name of the originating file system /samfs1:

• dir1/
• dir1/filea

Another cause of path name inconsistency is illustrated by a scenario in which a file is saved to disk, archived, then later moved, either by use of the mv(1) command or by restoration from a samfsdump(1M) output file using samfsrestore(1M) into an alternate path or file system.

This scenario results in the following:

• The archive copy is still valid.
• The .inodes file still points to the archive media.
• The path name in the tar file header no longer matches the namespace on disk.
• The name of the file system is not available in the tar file header.

To prevent this kind of situation, keep the data from each file system on its own unique set of tapes or other archive media, and do not mix data from multiple file systems.

The potential for inconsistency does not interfere with recovery in most cases, because the directory path names in the tar headers are not used when data is being recovered from an archive. The directory path names in the tar headers on the archive media are only used in an unlikely disaster recovery situation in which no metadata is available and the file system must be completely reconstructed with the tar command.

---

Guidelines for Performing Metadata Dumps

Follow these guidelines when performing metadata dumps:

• Perform dumps with the file system mounted.
• Perform metadata dumps at a time when files are not being created or modified.

At any given time, some files need to be archived because they are new, while others need to be rearchived because they are modified or because their archive media is being recycled. TABLE 4-6 defines the terms that apply to files archived onto archive media.

Dumping metadata during a time when files are not being created or modified avoids the dumping of metadata for files that are stale and minimizes the creation of damaged files.

• If an error message identifies a file as damaged, run the samfsdump(1M) command again after the specified file is archived.

When any stale files exist while metadata and file data are being dumped, the samfsdump command generates a warning message. The following warning message is displayed for any files that do not have an up-to-date archive copy:

/pathname/filename: Warning! File data will not be recoverable (file will be marked damaged).

Caution - If you see the above message and do not rerun the samfsdumpcommand after the specified file is archived, the file will not be retrievable.

If samfsrestore(1M) is later used to attempt to restore the damaged file, the following message is displayed:

/pathname/filename: Warning! File data was previously not recoverable (file is marked damaged).

---

Backing Up the Metadata in SAM-QFS File Systems

In SAM-QFS file systems, the archiver(1M) command can copy both file data and metadata, other than the .inodes file, to archive media. For example, if you create a SAM-QFS file system with a family-set name of samfs1, you can tell the archiver command to create an archive set also called samfs1. (See the archiver.cmd(4) man page for more information.) You can later retrieve damaged or destroyed file systems, files, and directories as long as the archive media onto which the archive copy was written has not been erased and as long as a recent metadata dump file is available.

The samfsdump(1M) command enables you to back up metadata separately from the file system data. The samfsdump command creates metadata dumps (including the .inodes file) either for a complete file system or of a portion of a file system. A cron(1M) job can be set up to automate the process.

If you dump metadata often enough using samfsdump, the metadata is always available to restore file data from the archives through samfsrestore(1M).

In summary, the samfsdump method to dump metadata has the following advantages:

• The samfsdump command saves the relative path for each file.
• The samfsdump command is run on mounted file systems.
• The metadata dump file generated by the samfsdump command contains all information required for restoring a SAM-QFS file system. The metadata dump file contains the .inodes file, directory information, and symbolic links.
• The samfsdump and samfsrestore method is flexible. This process enables you to restore an entire file system, a directory hierarchy, or a single file. With samfsdump(1M) and samfsrestore(1M), you can split an existing file system into multiple file systems, or you can join multiple file systems into a single file system.
• The samfsrestore command defragments the .inodes file, the file system namespace, and file data.

During a file system restoration, files and directories are assigned new inode numbers based on directory location; only the required number of inodes are assigned. Inodes are assigned as the samfsrestore process restores the directory structure.

File data is defragmented because files that were written in a combination of small disk allocation units (DAUs) and large DAUs are staged back to the disk with appropriately sized DAUs.

• When the samfsrestore process is complete, all directories and symbolic links are online and files are ready to be accessed.

---

Creating samfsdump Dump Files

If you have multiple SAM-QFS file systems, make sure that you routinely dump the metadata for every file system. Look in /etc/vfstab for all file systems of type samfs.

Be sure to save the dump for each file system in a separate file.

The following procedures describe how to find all the samfs type file systems and to dump metadata using samfsdump(1M):

• To Find Sun StorageTek QFS File Systems
• To Create a Sun StorageTek SAM Metadata Dump File Manually Using File System Manager
• To Create a Sun StorageTek SAM Metadata Dump File Manually Using the Command Line
• To Create a Recovery Point File Automatically From File System Manager
• To Create a Sun StorageTek SAM Metadata Dump File Automatically Using cron

Using samfsdump With the -u Option

The samfsdump(1M) command -u option causes unarchived file data to be interspersed with the metadata. Note the following about the use of the -u option:

• A samfsdump command run with the -u option on a version 3.5 or 4.x SAM-QFS file system cannot restore the file system to an earlier version (3.3.x) file system of the same type because versions 3.5 and 4.x have new data structures. Dumps from a 4.x version of either file system type can be restored on a 3.5 version and vice versa.
• A samfsdump dump taken using the -u option can be very large. The samfsdump command does not have any tape management or estimations such as those associated with ufsdump(1M). When using the -u option, as when setting up any data protection procedures, you need to weigh the amount of dump storage space available against the risks of having unarchived data. For more information, see also the samfsdump(1M) and ufsdump(1M) man pages.

Look in the vfstab file to find mount points for all samfs-type file systems.

CODE EXAMPLE 4-1 shows three file systems of type samfs with the file system names samfs1, samfs2, and samfs3. The mount points are /sam1, /sam2, and /sam3.

# vi /etc/vfstab

samfs1 -       /sam1 samfs   -       no high=80,low=70,partial=8

samfs2 -       /sam2 samfs   -       no high=80,low=50

samfs3 -       /sam3 samfs   -       no high=80,low=50

In the File System Manager interface, creating a metadata dump file is called creating a "recovery point." It is the equivalent of using the samfsdump command from the command line. You can create a recovery point from the File System Manager interface at any time.

Follow these steps to create a recovery point:

1. From the Servers menu, select the server that contains the file system that you want to administer.

The File Systems Summary page is displayed.

2. Navigate to the Recovery Points node under File Browsing & Recovery in the navigation tree.

3. Select the file system for which you want to create the recovery point from the drop down menu.

4. Click Create Recovery Point Now....

The Take Recovery Point pop-up window is displayed.

5. In the Fully Qualified Recovery Point Name field, type the path and the name of the recovery point file that you want to create.

6. Click Submit.

See the File System Manager online help for more information on creating recovery points.

Additional File System Manager Tools

Beginning with File System Manager version 2.1, compressed recovery point files created by File System Manager can be indexed without being uncompressed. In order to take advantage of this feature, you should select the gzip compression method for any scheduled recovery points.

If you have existing compressed recovery point files that are not in the gzip format, you can use the gznew command to convert them to gzip format.

In addition, indexing for recovery point files was also improved in version 2.1 of File System Manager. Additional information was added to the index, including information about damaged or online files. To take advantage of this improvement, you should delete any indexes created with earlier software versions and recreate them.

You can also use File System Manager to specify a retention policy for recovery point files. Recovery point files can be deleted after a specified number of months or marked for permanent retention.

When restoring from a recovery point file, the status of the file at the time the recovery point was created is provided and you can opt to restore files to the same state. You can also select a replacement strategy for determining which files to keep in case a file of the same name already exists. The following options are available:

• Do not restore
• Replace with the restored version
• Keep the newer version

1. Log in as root.

2. Go to the mount point for the samfs type file system mount point or to the directory that you are dumping.

# cd /sam1

See To Find Sun StorageTek QFS File Systems.

3. Enter the samfsdump(1M) command to create a metadata dump file.

CODE EXAMPLE 4-2 shows a SAM-QFS file system metadata dump file being created on February 14, 2004, in a dumps subdirectory in dump file system /dump_sam1/dumps. The output of the ls(1) command line shows that the date is assigned in yymmdd format as the dump file's name, 040214.

# samfsdump -f /dump_sam1/dumps/`date +\%y\%m\%d`

# ls /dump_sam1/dumps

040214

If you have a specific list of files that you would like to dump, you can use the
-I include_file option.This only dumps the files and directories that are listed in the include_file. The include_file must have one relative or absolute path per line.

Scheduling a recovery point through the File System Manager interface is the equivalent of creating a crontab(1) entry that automates the Sun StorageTek SAM software samfsdump(1M) process.

To schedule a recovery point:

1. From the Servers menu, select the server on which the archiving file system that you want to administer is located.

The File Systems Summary page is displayed.

2. Select the radio button next to the archiving file system for which you want to schedule a recovery point.

3. From the Operations menu, choose Schedule Recovery Points.

The Schedule Recovery Points page is displayed.

4. Specify values on the Schedule Recovery Points page.

For detailed instructions on using this page, see the File System Manager online help.

5. Click Save.

1. Log in as root.

2. Enter the crontab(1M) command with the -e option to make an entry to dump the metadata for each file system.

The crontab entry in CODE EXAMPLE 4-3 runs at 10 minutes past 2 a.m. every day and does the following:

• In the dump file system's dumps directory (/dump_sam1/dumps), removes files older than three days.
• Dumps the metadata from/sam1.
• Assigns the date of the metadata dump as the file's name in yymmdd format.


CODE EXAMPLE 4-3 Crontab Entry
# crontab -e 10 2 * * * ( find /dump_sam1/dumps -type f -mtime +72 -print | xargs -l1 rm -f; cd /sam1 ; /opt/SUNWsamfs/sbin/samfsdump -f /dump_sam1/dumps/'date +\%y\%m\%d') :wq

If the crontab entry in the previous code example had run on March 20, 2005, the full path name of the dump file would be /dump_sam1/dumps/050320.

---

Using Archiver Logs

Archiver logging should be enabled in the archiver.cmd file. Because archiver logs list all the files that have been archived and their locations on cartridges, archiver logs can be used to recover lost files that were archived after the last set of metadata dumps and backup copies were created.

Be aware of the following considerations:

• Processes writing to the archiver log continue to do so until they complete.
• If a log file is not found, the SAM-QFS file system creates a new log file when a process initiates a new write to the log.
• If a log file exists, data is appended to the existing file.
• Archiver log files grow over time, so they must be managed.

Set up and manage the archive logs by performing the described procedures in the following sections:

• To Set Up Archiver Logging
• To Save Archiver Logs

Enable archive logging in the archiver.cmd file, which is in the /etc/opt/SUNWsamfs directory.

The archiver log files are typically written to /var/adm/logfilename. The directory to which you direct the logs to be written should reside on a disk outside the SAM-QFS environment. For more information, see the archiver.cmd(4) man page.

Ensure that archiver log files are cycled regularly by creating a cron(1M) job that moves the current archiver log files to another location.

The screen example below shows how to create a dated copy of an archiver log named /var/adm/archlog every day at 3:15 a.m. The dated copy is stored in /var/archlogs.

---

Storing Copies of Disaster Recovery Files and Metadata

Consider writing scripts to create tar(1) files that contain copies of all the relevant disaster recovery files and metadata described in this chapter and to store the copies outside the file system. Depending on your site's policies, put the files into one or more of the locations described in the following list:

• On another file system of any type.
• Directly on removable media files.

For information on removable media files, see the request(1) man page.

• If running the archiver(1M) on a SAM-QFS file system, store the files on a separate SAM-QFS file system that is being archived on a separate set of cartridges.

This approach ensures that the disaster recovery files and metadata are archived separately from the file system to which they apply. You might also consider archiving multiple backup copies for additional redundancy.

Observe the following precautions:

• Keep a written (nonelectronic) listing of where the disaster recovery files are kept.

You can obtain lists of all directories containing removable media files by using the sls(1M) command. These listings can be emailed. For more information about obtaining file information, see the sls(1M) man page.

• Keep a written record of your hardware configuration.
• Do not assign the cartridges used to hold the removable media files to the archiver.

^{1 (Footnote) Protect these files only if you modify them.}

Sun StorageTek Storage Archive Manager Troubleshooting Guide

819-7933-10

Copyright © 2007, Sun Microsystems, Inc. All Rights Reserved.