Sun StorEdge SAM-FS Troubleshooting Guide

This chapter provides the backup and dump processes and information you need in order to keep your data safe and prepare for any disaster.

This chapter includes the following subsections:

• Guarding Against or Troubleshooting Data Loss
• Precautions Before Starting Data Restoration
• Prerequisites for Data Recovery
• Metadata Used in Disaster Recovery
• SAM-QFS Disaster Recovery Features
• Guidelines for Performing Dumps
• Backing Up the Metadata in SAM-QFS File Systems
• Creating samfsdump Dump Files
• Disaster Recovery Commands and Tools
• The samexplorer Script
• What to Back Up and How Often
• Additional Backup Considerations
• Using Archiver Logs
• How and Where to Keep Copies of Disaster Recovery Files and Metadata

---

Guarding Against or Troubleshooting Data Loss

TABLE 4-1 shows the usual causes of data loss, with notes and suggestions about how to avoid or respond to each type of loss.

---

Precautions Before Starting Data Restoration

Some apparent data losses are actually caused by cabling problems or configuration changes.

Caution - Do not reformat a disk, relabel a tape, or make other irreversible changes until you are convinced that the data on the disk or tape is completely unrecoverable. Make sure to eliminate the fundamental causes for a failure before making irreversible changes. Back up anything you change before you change it, if possible.

Take the steps in the following procedure, "To Troubleshoot an Inaccessible File System," before commencing a data recovery process.

1. Check cables and terminators.

2. If you cannot read a tape or magneto-optical cartridge, try cleaning the heads in the drive, or try reading the cartridge in a different drive.

3. Check the current state of your hardware configuration against the documented hardware configuration.

Go to Step 4 only when you are certain that a configuration error is not to blame.

4. Unmount the file system, and run samfsck(1M).

For example:

# umount file_system_name

# samfsck file_system_name

5. If you find the file system is still inaccessible, use the procedures in the other chapters in this manual to restore the file system.

---

Prerequisites for Data Recovery

For SAM-QFS file systems, the following are prerequisites for disaster recovery:

• Up-to-date archive copies

The effectiveness of any of the SAM-QFS recovery methods relies primarily on frequent archiving being done.

• Up-to-date metadata dumps

See Metadata Used in Disaster Recovery.

• Archiver logs

If recent metadata is not available, archiver logs can help you recreate the filesystem directly from archive media.

See Using Archiver Logs.

---

Metadata Used in Disaster Recovery

Metadata consists of information about files, directories, access control lists, symbolic links, removable media, segmented files, and the indexes of segmented files. Metadata must be restored before lost data can be retrieved.

With the up-to-date metadata, the data can be restored as follows:

• File data can be restored even if the file has been removed from the file system.
• Individual files or entire file systems can be moved from one file system to another, or even from one server to another.

.inodes File Characteristics

In Sun StorEdge QFS file systems, the .inodes file contains all the metadata except for the directory namespace (which consists of the pathnames to the directories where the files are stored). The.inodes file is located in the root (/) directory of the file system. For a file system to be restored, the .inodes file is needed along with the additional metadata.

FIGURE 4-1 illustrates some characteristics of the .inodes file. The arrows with the dashed lines indicate that the .inodes file points to file contents on disk and to the directory namespace. The namespace also points back to the .inodes file. Also indicated is that, in SAM-QFS file systems where archiving is being done, the .inodes file also points to archived copies.

Figure showing relationship between QFS nad SAM-QFS file systems.

The .inodes file is not archived. For more about protecting the .inodes file in these types of file systems, see SAM-QFS Disaster Recovery Features and Backing Up the Metadata in SAM-QFS File Systems.

More About Directory Pathnames

As indicated in FIGURE 4-1, the namespace (in the form of directories) does not point to the archive media. The directory pathnames for each archived file are copied into the headers of the tar(1) files on the archive media that contain the files, but for reasons illustrated elsewhere (in TABLE 4-3), the directory pathnames in the tar file headers may get out of sync with the actual locations of the files on the disk.

One reason why the two pathnames can get out of sync is that the pathnames in the tar file header do not show the originating file system. TABLE 4-2 shows how the directory pathname shown in the left column would appear in the tar file header in the right column, without the component that shows the name of the originating file system /samfs1.

TABLE 4-3 summarizes an example scenario, shows the result, and suggests a precaution.

The potential for inconsistency does not interfere with recovery in most cases, because the directory pathnames in the tar headers are not used when data is being recovered from an archive. The directory pathnames on the tar headers on the archive media are only used in an unlikely disaster recovery scenario where no metadata is available and the file system must be reconstructed from scratch using the tar command.

---

SAM-QFS Disaster Recovery Features

The features of SAM-QFS file systems described in TABLE 4-4 streamline and speed up data restoration and minimize the risk of losing data in the case of unplanned system outage.

---

Guidelines for Performing Dumps

• Perform dumps with the file system mounted.
• Perform metadata dumps at a time when files are not being created or modified.

At any given time, some files need to be archived because they are new, while others need to be rearchived because they are modified or because their archive media is being recycled. TABLE 4-5 defines the terms that apply to files archived onto archive media.

Dumping metadata during a time when files are not being created or modified avoids the dumping of metadata for files that are stale and minimizes the creation of damaged files.

• If an error message identifies a file as damaged, run the samfsdump(1M) command again after the specified file is archived.

When any stale files exist while metadata and file data are being dumped, the samfsdump command generates a warning message. The following warning message is displayed for any files that do not have an up-to-date archive copy:

/pathname/filename: Warning! File data will not be recoverable (file will be marked damaged).

Caution - If you see the above message and do not rerun the samfsdumpcommand after the specified file is archived, the file will not be retrievable.

If samfsrestore(1M) is later used to attempt to restore the damaged file, the following message is displayed:

/pathname/filename: Warning! File data was previously not recoverable (file is marked damaged).

---

Backing Up the Metadata in SAM-QFS File Systems

In SAM-QFS file systems, the archiver(1M) command can copy both file data and metadata--other than the .inodes file--to archive media. For example, if you create a SAM-QFS file system with a family-set name of samfs1, you can tell the archiver command to create an archive set also called samfs1. (See the archiver.cmd(4) man page for more information.) You can later retrieve damaged or destroyed file systems, files, and directories as long as the archive media onto which the archive copy was written has not been erased and as long as a recent metadata dump file is available.

The samfsdump(1M) command allows you to back up metadata separately from the file system data. The samfsdump command creates metadata dumps (including the .inodes file) either for a complete file system or of a portion of a file system. A cron(1M) job can be set up to automate the process.

If you dump metadata often enough using samfsdump, the metadata is always available to restore file data from the archives using samfsrestore(1M).

In summary, using the samfsdump method to dump metadata has the following advantages:

• The samfsdump command saves the relative path for each file.
• The samfsdump command is run on mounted file systems.
• The metadata dump file generated by the samfsdump command contains all information required for restoring a SAM-QFS file system. The metadata dump file contains the .inodes file, directory information, and symbolic links.
• The samfsdump and samfsrestore method is flexible. This process enables you to restore an entire file system, a directory hierarchy, or a single file. With samfsdump(1M) and samfsrestore(1M), you can split an existing file system into multiple file systems or you can join multiple file systems into a single file system.
• The samfsrestore command defragments the .inodes file, the file system name space, and file data.

During a file system restoration, files and directories are assigned new inode numbers based on directory location; only the required number of inodes are assigned. Inodes are assigned as the samfsrestore process restores the directory structure.

File data is defragmented because files that were written in a combination of small disk allocation units (DAUs) and large DAUs are staged back to the disk using appropriately sized DAUs.

• When the samfsrestore process is complete, all directories and symbolic links are online and files are ready to be accessed.

---

Creating samfsdump Dump Files

If you have multiple SAM-QFS file systems, make sure that you routinely dump the metadata for every file system. Look in /etc/vfstab for all file systems of type samfs.

Make sure to save the dump for each file system in a separate file.

The following procedures describe how to find all the samfs type file systems and to dump metadata using samfsdump(1M):

• To Find Sun StorEdge QFS File Systems
• To Create a Sun StorEdge SAM-FS Metadata Dump File Manually Using File System Manager
• To Create a Sun StorEdge SAM-FS Metadata Dump File Manually Using the Command Line
• To Create a Sun StorEdge SAM-FS Metadata Dump File Automatically From the File System Manager
• To Create a Sun StorEdge SAM-FS Metadata Dump File Automatically Using cron

Using samfsdump With the -u Option

The samfsdump(1M) command -u option causes unarchived file data to be interspersed with the metadata. Note the following about the use of the -u option:

• A samfsdump command run with the -u option on a version 3.5 or 4.x SAM-QFS file system cannot be restored to an earlier version (3.3.x) file system of the same type because versions 3.5 and 4.x have new data structures. Dumps from a 4.x version of either file system type can be restored on a 3.5 version and vice versa.
• A samfsdump dump taken using the -u option can be very large. The samfsdump command does not have any tape management or estimations such as those associated with ufsdump(1M). You need to weigh the amount of dump storage space available against the risks of having unarchived data when using the -u option (as you do when setting up any data protection procedures). For more information, see also the samfsdump and ufsdump man pages.

Look in the vfstab(4) file to find mount points for all samfs-type file systems.

CODE EXAMPLE 4-1 shows three file systems of type samfs with the file system names samfs1, samfs2, and samfs3. The mount points are /sam1, /sam2, and /sam3.

# vi /etc/vfstab

samfs1 -       /sam1 samfs   -       no high=80,low=70,partial=8

samfs2 -       /sam2 samfs   -       no high=80,low=50

samfs3 -       /sam3 samfs   -       no high=80,low=50

Taking a metadata snapshot through the File System Manager interface is the equivalent of using the samfsdump command from the command line. You can take a metadata snapshot from the File System Manager interface at any time.

To take a metadata snapshot:

1. From the Servers page, click the server on which the file system that you want to administer is located.

The File Systems Summary page is displayed.

2. Select the radio button next to the file system for which you want to schedule a metadata snapshot.

3. From the Operations menu, choose Take Metadata Snapshots.

The Take Metadata Snapshot pop-up window is displayed.

4. In the Fully Qualified Snapshot File field, type the path and the name of the snapshot file that you want to create.

5. Click Submit.

See the File System Manager online help file for complete information on creating metadata snapshots.

1. Log in as root.

2. Go to the mount point for the samfs type file system mount point or to the directory that you are dumping.

# cd /sam1

See To Find Sun StorEdge QFS File Systems if needed.

3. Enter the samfsdump(1M) command to create a metadata dump file.

CODE EXAMPLE 4-2 shows a SAM-QFS file system metadata dump file being created on February 14, 2004 in a dumps subdirectory in dump file system /dump_sam1/dumps. The output of the ls(1) command line shows the date is assigned in the yymmdd format as the dump file's name, 040214.

# samfsdump -f /dump_sam1/dumps/`date +\%y\%m\%d`

# ls /dump_sam1/dumps

040214

Scheduling a metadata snapshot through the File System Manager interface is the equivalent of creating a crontab(1) entry that automates the Sun StorEdge SAM-FS software samfsdump(1M) process.

To schedule a metadata snapshot:

1. From the Servers page, click the server on which the archiving file system that you want to administer is located.

The File Systems Summary page is displayed.

2. Select the radio button next to the archiving file system for which you want to schedule a metadata snapshot.

3. From the Operations menu, choose Schedule Metadata Snapshots.

The Schedule Metadata Snapshots page is displayed.

4. Specify values on the Schedule Metadata Snapshots page.

For complete instructions on using this page, see the File System Manager online help file.

5. Click Save.

1. Log in as root.

2. Enter the crontab(1M) command with the -e option to make an entry to dump the metadata for each file system.

The crontab entry in CODE EXAMPLE 4-3 runs at 10 minutes past 2 a.m. every day and does the following:

• In the dump file system's dumps directory (/dump_sam1/dumps), removes files older than three days.
• Dumps the metadata from/sam1.
• Assigns the date of the metadata dump as the file's name in yymmdd format.


CODE EXAMPLE 4-3 Crontab Entry
# crontab -e 10 2 * * * ( find /dump_sam1/dumps -type f -mtime +72 -print | xargs -l1 rm -f; cd /sam1 ; /opt/SUNWsamfs/sbin/samfsdump -f /dump_sam1/dumps/`date +\%y\%m\%d ` ) :wq

If the crontab entry in the previous screen example ran on March 20, 2005, the full pathname of the dump file would be: /dump_sam1/dumps/050320.

---

Disaster Recovery Commands and Tools

TABLE 4-6 summarizes the commands used most frequently in disaster recovery efforts.

For more information about these commands, see their man(1) pages. Other scripts and helpful sample files are located /opt/SUNWsamfs/examples or are available from Sun Microsystems.

TABLE 4-7 describes some disaster recovery utilities in the /opt/SUNWsamfs/examples directory and explains their purpose. You must modify all of the listed shell scripts, except for recover.sh(1M), to suit your configuration before using them. See the comments in the files.

Caution - Improper use of the restore.sh, recover.sh, or tarback.shscripts can damage user or system data. Please read the man pages for these scripts before attempting to use them. For additional help with using these scripts, contact Sun customer support.

---

The samexplorer Script

The /opt/SUNWsamfs/sbin/samexplorer script (called info.sh in software versions before 4U1) is not a backup utility, but it should be run whenever changes are made to the system's configuration.

The samexplorer(1M) script creates a file containing all the configuration information needed for reconstructing a SAM-QFS installation from scratch if you ever need to rebuild the system. You can use the crontab(1) command with the -e option to create a cron(1M) job to run the samexplorer script at desired intervals.

The samexplorer script writes the reconfiguration information to /tmp/SAMreport.

Make sure that the SAMreport file is moved from the /tmp directory after creation to a fixed disk that is separate from the configuration files and outside the SAM-QFS environment. For more information about managing the SAMreport file, see the samexplorer(1M) man page.

---

What to Back Up and How Often

TABLE 4-8 describes the files that should be backed up and how often the files should be backed up onto a location outside the file system environment.

Where "Regularly" is shown in the "Backup Frequency" column, each site's system administrator should decide the appropriate intervals based on that site's requirements. Except where specified, use whatever backup procedures you choose.

---

Additional Backup Considerations

The following is a list of questions to also consider when preparing your site's disaster recovery plan.

• What is the right number of samfsdump(1M) or qfsdump(1M) files to retain at your site?

TABLE 4-9 compares the types of dumps that are done in the various file system types.

Retain enough data and metadata to ensure that you can restore the file systems according to your site's needs. The appropriate number of dumps to save depends, in part, on how actively the system administrator monitors the dump output. If an administrator is monitoring the system daily to make sure the samfsdump(1M) or qfsdump(1M) dumps are succeeding (making sure enough tapes are available and investigating dump errors), then keeping a minimum number of dump files to cover vacations, long weekends, and other absences might be enough.

• If you are archiving data, are you actively recycling archive media? If so, make sure to schedule metadata copies to occur after recycling completes.

If your site is using the sam-recycler(1M) command to reclaim space on archive media, it is critical that you make metadata copies after sam-recycler has completed its work. If a metadata dump is created before the sam-recycler exits, the information in the metadump about archive copies becomes out of date as soon as sam-recyler runs. Also, some archive copies may be made inaccessible because the sam-recycler command may cause archive media to be relabeled.

Check root's crontab(1) entry to find out if and when the sam-recycler command is being run, and then, if necessary, schedule the creation of metadump files around the sam-recycler execution times. For more about recycling, see the Sun StorEdge SAM-FS Storage and Archive Management Guide.

• How much data should you store off site, and in what format?

Off-site data storage is an essential part of a disaster recovery plan. In the event of a disaster, the only safe data repository might be an offsite vault. Beyond the recommended two copies of all files and metadata that you should be keeping in house as a safeguard against media failure, consider making a third copy on removable media and storing it offsite.

Sun SAM-Remote offers you the additional alternative of making archive copies in remote locations on a LAN or WAN. Multiple Sun SAM-Remote servers can be configured as clients to one another in a reciprocal disaster recovery strategy.

• Is it sufficient to restore only the metadata to a predisaster state or do you need also to restore all files that were online when the disaster happened?
• The samfsrestore(1M) command can restore a SAM-QFS file or file system to the state reflected in the samfsdump(1M) file. After the samfsrestore(1M) command is run, the metadata is restored, but the file data remains offline.

If you need to restore all files that were online, you need to run the samfsrestore command with the -g option.

The log file generated by the samfsrestore command's -g option contains a list of all files that were on the disk when the samfsdump(1M) command was run. This log file can be used in conjunction with the restore.sh shell script to restore the files on disk to their predisaster state. The restore.sh script takes the log file as input and generates stage requests for files listed in the log. By default, the restore.sh script restores all files listed in the log file.

If your site has thousands of files that need to be staged, consider splitting the log file into manageable chunks and running the restore.sh script against each of those chunks separately to ensure that the staging process does not overwhelm the system. You can also use this approach to ensure that the most critical files are restored first. For more information, see the comments in /opt/SUNWsamfs/examples/restore.sh.

---

Using Archiver Logs

Archiver logging should be enabled in the archiver.cmd(4) file. Because archiver logs list all the files that have been archived and their locations on cartridges, archiver logs can be used to recover lost files that were archived since the last set of metadata dumps and backup copies were created.

Be aware of the following considerations:

• Processes writing to the archiver log continue to do so until they complete.
• If a log file is not found, the SAM-QFS file system creates a new log file when a process initiates a new write to the log.
• If a log file exists, data is appended to the existing file.
• The archiver log files grow over time, so they must be managed.

Set up and manage the archive logs by performing the following procedures:

• "To Set Up Archiver Logging"
• To Save Archiver Logs

Enable archive logging in the archiver.cmd file (in the /etc/opt/SUNWsamfs directory).

See the archiver.cmd(4) man page. The archiver log files are typically written to /var/adm/logfilename. The directory where you direct the logs to be written should reside on a disk outside the SAM-QFS environment.

Ensure that archiver log files are cycled regularly by creating a cron(1M) job that moves the current archiver log files to another location.

The screen example below shows how to create a dated copy of an archiver log named /var/adm/archlog every day at 3:15 a.m. The dated copy is stored in /var/archlogs.

---

How and Where to Keep Copies of Disaster Recovery Files and Metadata

Consider writing scripts to create tar(1) files that contain copies of all the relevant disaster recovery files and metadata described in this chapter and to store the copies outside the file system. Depending on your site's policies, put the files into one or more of the locations described in the following list:

• On another file system of any type.
• Directly on removable media files.

For information on removable media files, see the request(1) man page.

• If running the archiver(1M) on a SAM-QFS file system, store the files on a separate SAM-QFS file system that is being archived on a separate set of cartridges.

This approach ensures that the disaster recovery files and metadata are archived separately from file system to which they apply. You might also consider archiving multiple backup copies for additional redundancy.

Observe the following precautions:

• Keep a written (nonelectronic) listing of where the disaster recovery files are kept.

You can obtain lists of all directories containing removable media files by using the sls(1M) command. These listings can be emailed. For more information about obtaining file information, see the sls(1M) man page.

• Keep a written record of your hardware configuration.
• Do not assign the cartridges used to hold the removable media files to the archiver.

^{1 (Footnote) Protect this file only if you modify it.}

Sun StorEdge SAM-FS Troubleshooting Guide

819-2756-10

Copyright © 2005, Sun Microsystems, Inc. All Rights Reserved.