Archiving Operations

Operator actions are not required to archive files. The archiver writes files to a volume on the archive media. Archive media can contain one or more volumes. Each volume is identified by a unique identifier called a volume serial name (VSN).

By default, the archiver makes one copy of each archive set, but you can request up to four copies. An archive set and a copy number become synonyms for a collection of volumes. The copies of the archive set provide duplication of files on separate volumes.

To be a candidate for archiving or re-archiving, the data in a file must change. A file is not archived if it is accessed but not changed. For example, issuing a touch command on a file does not cause it to be archived or re-archived. If the mv command is used to rename a file, the file might move to a different archive set. In this case, the archiver software re-evaluates the archive copies to determine whether the file needs to be archived or re-archived.

The archive operation is affected by the following factors:

• Archive age The period of time that has passed since the file was last modified. The archive age can be defined for each archive copy.

  Use the touch command to change the default time references on their files to values in the past or future. Note that this practice can cause unexpected archiving results. To avoid problems, the archiver adjusts the references so that they are always at a point between the file creation time and the present time.

• Archive priority This value is computed from file property characteristics and from file property multipliers that are associated with the archive set, as follows:

  archive-priority = file-property-value x property-multiplier

  • Most file-property-value numbers are 1 (for true) or 0 (for false). For instance, the value of the property copy 1 is 1 if archive copy 1 is being made. The values of copy 2, copy 3, and copy 4 are therefore 0. Other properties, such as archive age and file size, can have values other than 0 or 1.

  • The property-multiplier value is determined from the -priority parameters for the archive set. Various aspects of a file, such as age or size, can be given values to determine the archive request's priority. The default value for all property multipliers is 0.0. For more information about the -priority parameter, see archiver.cmd(4) in Sun QFS and Sun Storage Archive Manager Reference Manual.

The archive-priority and the property-multiplier values are floating-point numbers.

The following sections describe the steps taken by the archiver from the initial file scan to the file copy process:

• Step 1: Identifying Files to Archive

• Step 2: Composing Archive Requests

• Step 3: Scheduling Archive Requests

• Step 4: Archiving the Files in an Archive Request

• Sample Default Output From the archiver -l Command

Step 1: Identifying Files to Archive

A separate sam-arfind process monitors each mounted file system to determine which files need archiving. The file system notifies this sam-arfind process whenever a file is changed in a manner that affects its archival state. Such changes are file modification, re-archiving, unarchiving, and renaming.

The sam-arfind process examines the file to determine the archive set to which the file belongs and what action to take.

The sam-arfind process uses the following file property descriptions to determine a file's archive set:

• The directory path portion of the file's name and, optionally, the complete file name using a regular expression

• The user name of the file's owner

• The group name of the file's owner

• A minimum file size

• A maximum file size

The sam-arfind process performs the following evaluation to determine the archive action:

• If the archive age of the file for one or more copies has been met or exceeded, the sam-arfind process adds the file to one or more archive requests for the archive set. An archive request is a collection of files that belong to the same archive set. The archive request resides in the /var/opt/SUNWsamfs/archiver/file_sys/ArchReq directory. The files in this directory are binary files. To display them, use the showqueue command. Separate archive requests are used for files that are not yet archived and for files that are being re-archived. This enables you to control scheduling independently for these two types of files.

• If the archive age of the file for one or more copies has not been met, the directory in which the file resides and the time at which the archive age is reached is added to a scan list. Directories are scanned as the scan list times are reached. Files that have reached their archive age are added to archive requests.

• If a file is offline, the sam-arfind process selects the volumes to be used as the source for the archive copy. If the file copy is being re-archived, the sam-arfind process selects the volume that contains the archive copy that is being rearchived.

• If a file is segmented, only those segments that have changed are selected for archiving. The index of a segmented file contains no user data, so it is treated as a member of the file system archive set and is archived separately.

The archive action is accomplished using the following methods:

• Continuous Archiving

• Scanned Archiving

Continuous Archiving

With the continuous archiving method, the archiver works with the file system to determine which files need to be archived.

Continuous archiving is the default archiving method (the archiver.cmd file parameter is examine=noscan) and operates with the following default start conditions:

• The archive starts every two hours.

• The archive waits until at least 90% of the archmax value of data is ready to archive.

• The archive waits until at least 500,000 files to archive.

  When any of the scheduling start conditions is reached, the sam-arfind process sends each archive request to the archiver daemon, sam-archiverd, to set the schedule for copying the file to archive media.

To control the schedule of the archive operation, specify the start conditions for each archive set by using the -startage, -startcount, and -startsize parameters. These conditions enable you to optimize archive timeliness in relation to archive work done. For example:

• If creating files that you want archived together takes an hour, set the -startage parameter to one hour (-startage 1h) to ensure that all files are created before the archive request occurs.

• If you want all archive requests to be no less than 150 gigabytes of data, specify a -startsize of 150 gigabytes (-startsize 150g) to direct the archiver to wait until 150 gigabytes of data are ready to be archived.

• If you know that 3000 files are available for archival, specify -startcount 3000 to ensure that the files are archived together.

For more information about archiving parameters, see Global Archiving Directives.

Scanned Archiving

With the scanning method, the archiver checks the file systems periodically and selects files for archiving. To enable the scanning method and to disable continuous scanning, specify examine=scan in the archiver.cmd file.

The sam-arfind process scans each file system periodically to determine which files need archiving and to place them in archive requests. The first scan is a directory scan, in which sam-arfind recursively descends through the directory tree. The process examines each file and sets the file status flag to archdone if the file does not need archiving. During successive scans, the sam-arfind process scans the .inodes file and examines only inodes without the archdone flag.

When the scan is complete, the sam-arfind process sends each archive request to the archiver daemon, sam-archiverd, to be scheduled for file copying to archive media. The sam-arfind process then sleeps for the duration specified by the interval= time directive. At the end of the interval, the sam-arfind process resumes scanning.

For information about controlling the setting of the archdone flag, see setarchdone Directive: Controlling the Setting of the archdone Flag.

Step 2: Composing Archive Requests

Regardless of the archiving method used, the sam-arfind process sends each archive request to the archiver daemon, sam-archiverd for composing. This section describes the composing process.

Composing is the process of selecting the files from the archive request to be archived at one time. Because of the capacity of the archive media or of the controls specified in the archiver command file, the files in an archive request might not all be archived simultaneously. When the archive copy operation is complete for an archive request, the archive request is recomposed if files remain to be archived.

The sam-archiverd daemon places the files in the archive request according to certain default and site-specific criteria. The default operation is to archive all the files in an archive request by their full path name so that files in the same directories are also together on the archive media. The site-specific criteria enable you to control the order in which files are archived and how they can be distributed on volumes. These criteria, called archive set parameters, are evaluated in the following order: -reserve, -sort, -rsort (reverse sort), and -drives. For more information, see archiver.cmd(4) in Sun QFS and Sun Storage Archive Manager Reference Manual.

When the -reserve owner parameter is specified, the sam-archiverd daemon orders the files in the archive request according to the file's directory path, user name, or group name. The files that belong to the owner are selected for archiving. The remaining files are archived later.

When the -sort or -rsort parameter is specified, the sam-archiverd daemon orders the files according to the specified sort method such as age, size, or directory location.

When an archive request contains both online and offline files, the online files are selected for archiving first.

In the absence of a specified sort method, the offline files are ordered by the volume on which the archive copies reside. This rule ensures that all files in each archive set on the same volume are staged simultaneously in the order in which they were stored on the media. When more than one archive copy of an offline file is being made, the offline file is not released until all required copies are made. All the files to be staged from the same volume as the first file are selected for archiving.

After being composed, the archive requests are entered in the sam-archiverd daemon's scheduling queue.

Step 3: Scheduling Archive Requests

The scheduler in the sam-archiverd daemon executes on demand when one of the following conditions exists:

• An archive request is entered in the scheduling queue.

• The archiving for an archive request has been completed.

• A change in media status is received from the catalog server.

• A message is received that changes the state of the archiver.

The archive requests in the scheduling queue are ordered by priority. Each time the scheduler runs, it examines all archive requests to determine whether they can be assigned to a sam-arcopy process to have their files copied to archive media.

The following must be true for archive requests to be scheduled:

• Drive usage must be available for making file copies.

• Volume usage must be available and have sufficient space to hold the files in the archive request.

Drive Usage

If the archive set has the -drives parameter specified, the sam-archiverd daemon divides the selected files in the archive request among the multiple drives. If the number of drives available is fewer than that specified by the -drives parameter, the smaller number of drives is used.

If the total size of files in the archive request is less than the -drivemin value, one drive is used. The -drivemin value is either the value specified by the -drivemin parameter or the archmax value. The archmax value is specified by the -archmax parameter or the value defined for the media. For more information about the -archmax parameter and the archmax= directive, see archiver.cmd(4) in Sun QFS and Sun Storage Archive Manager Reference Manual.

If the total size of files in the archive request is more than the -drivemin value, the number of drives used is determined by the total size of the files divided by the -drivemin value. If the number of drives available is fewer than that specified by the -drives parameter, the smaller number of drives is used.

Drives can take varying amounts of time to archive files. The -drivemax parameter specifies the maximum number of bytes to be written to a drive before that drive is rescheduled for more data. You can use the -drivemax parameter to obtain better drive utilization.

Volume Usage

For archiving to occur, at least one volume must have enough space to hold at least some of the files in the archive request. The volume that has most recently been used for the archive set is the one scheduled, if it has enough space. This volume must not be in use by the archiver.

If a volume usable for the archive set is busy, another is selected unless the -fillvsns parameter is specified. In this case, the archive request cannot be scheduled.

If an archive request is too big for one volume, the files that can fit on the volume are selected to be archived to the volume. If the archive request contains files that are too big to fit on one volume and volume overflow for the archive request is not selected, the files cannot be archived. An appropriate message for this condition is sent to the log.

You can specify volume overflow for the archive set by using the -ovflmin parameter, or for the media by using the ovflmin= directive. For more information about the -ovflmin parameter and the ovflmin= directive, see archiver.cmd(4) in Sun QFS and Sun Storage Archive Manager Reference Manual. The ovflmin specification determines the file size threshold above which additional volumes or media are assigned for archiving. An ovflmin value specified for the archive set takes precedence over an ovflmin value specified for the media.

If the size of the files is less than the value of ovflmin, the files cannot be archived. An appropriate message for this condition is sent to the log. If the size of the files is more than the value of ovflmin, additional volumes are assigned as required. Volumes are selected in order of decreasing size to minimize the number of volumes required. If no usable volumes can be found for the archive request, the archive request waits until volumes become available.

Certain properties, such as whether the file is online or offline, are used in conjunction with the archive priority to determine the scheduling priority for a particular archive request. For more information about customizing the priority multiplier, see the -priority parameters described on the archiver.cmd(4) man page.

For each archive request, the sam-archiverd daemon computes the scheduling priority by adding the archive priority to multipliers that are associated with various system resource properties. These properties are associated with the number of seconds for which the archive request has been queued, whether the first volume to be used in the archiving process is loaded into a drive, and so on.

Using the adjusted priorities, the sam-archiverd daemon assigns each ready archive request to be copied.

Step 4: Archiving the Files in an Archive Request

When an archive request is ready to be archived, the sam-archiverd daemon marks the archive file boundaries so that each archive file's size is less than the specified -archmax value. If a single file is larger than this value, it becomes the only file in an archive file.

For each archive request and each drive to be used, the sam-archiverd daemon assigns the archive request to a sam-arcopy process to copy the files to the archive media. The archive information is entered into the inode.

If archive logging is enabled, an archive log entry is created.

For each file that was staged, the disk space is released until all files in the list have been archived.

A variety of errors and file status changes can prevent a file from being successfully copied. Errors include read errors from the cache disk and write errors to the volumes. Status changes include modification since selection, a file that is open for writing, or a file that has been removed.

When the sam-arcopy process exits, the sam-archiverd daemon examines the archive request. If any files have not been archived, the archive request is recomposed.

You can also specify timeouts for archiving operations in the archiver.cmd file. The directive is as follows:

timeout = [operation | media] time

operation is one of the following:

• read – Read the file from the disk. The default timeout is 1 minute.

• request – Request the archive media. The default timeout is 15 minutes.

• stage – Stage the file to be archived. The default timeout is 0 minutes.

• write – Write to the archive media. The default timeout is 15 minutes.

The timeout value for the write operation can also be specified for individual media.

You can configure the -queue_time_limit time archive set parameter in the archiver.cmd file to notify the administrator when an archive request has been in the schedule queue longer than a certain amount of time. If the archive request remains in the queue at the end of this time, an email is sent to the system administrator.

Sample Default Output From the archiver -l Command

The following example shows sample output from the archiver -l command.

# archiver -l
Archive media:
default:mo
media:mo archmax:5000000
media:lt archmax:50000000
Archive devices:
device:mo20 drives_available:1 archive_drives:1
device:lt30 drives_available:1 archive_drives:1
Archive file selections:
Filesystem samfs1:
samfs1 Metadata
copy:1 arch_age:240
big path:. minsize:512000
copy:1 arch_age:240

all path:
copy:1 arch_age:30
Archive sets:
all
copy:1 media:mo
big
copy:1 media:lt
samfs1
copy:1 media:mo