File Retention Management

The share file retention policy provides a facility for data governance, legal holds, and compliance records retention. Both data governance and regulatory compliance can be used to help protect from cyber and ransomware attacks.

• Data Governance - Data governance locks datasets (snapshots, objects or files) for a period of time, thus protecting the data from deletion. You might need to protect certain datasets as part of internal business process requirements or to protect datasets as part of your cyber-protection strategy. Data governance allows for adjustments in the retention strategy from privileged users.

  File retention data governance is implemented by creating a new project and filesystem with the "privileged" file retention policy. Privileged mode allows you to create a default retention setting for all new files, and to change that setting in the future to a shorter or longer duration. Files inherit the retention setting in effect when they are created. Retention can also be adjusted manually to a longer duration by changing the unlock timestamp. Projects and filesystems cannot be deleted when they have locked files.

• Legal Holds - A legal hold preserves certain business data in response to potential or ongoing lawsuits. A legal hold does not have a defined retention period, and it remains in effect until removed. Once the legal hold is removed, all protected data is immediately eligible for deletion unless other retention rules still apply.

  File retention legal holds on files are implemented by manually increasing the retention period on individual files, or by setting a hold for individual files so that their expiry date extends indefinitely. Because a legal hold may be required for an indefinite period of time, it is recommended to periodically extend manual retention. Holds on files never expire; the hold must be explicitly turned off. These two methods allow file retention to expire after the need for the legal hold has passed.

• Regulatory Compliance - Your industry might require you to retain a certain class of data for a defined length of time. Your data retention regulations might also require that you lock the retention settings. Regulatory compliance only allows you to increase the retention time, if at all. Regulatory compliance is the most restrictive locking strategy, and it often does not allow anyone, even an administrator, to make changes affecting retention.

  File retention regulatory compliance is implemented by creating a new project and filesystem with the "mandatory (no override)" file retention policy. Mandatory mode does not allow you to decrease the file retention duration. However, retention can be adjusted manually to a longer duration by changing the unlock timestamp. Regulatory compliance uses the same mechanisms as data governance, but it is much more restrictive. The project and filesystem cannot be deleted when locked files exist, and the storage pool cannot be unconfigured when locked files exist within the pool. This mode also requires usage of an NTP server, and the root user is locked out of remote access.

File retention completes the trio of retention products for Oracle ZFS Storage Appliance: file retention, snapshot retention, and object storage retention.

When the file retention policy is enabled, files become retained when set to readonly. Each file has a retained-until-expiration timestamp. This expiration date is either explicitly set or calculated based on the retention policy set on the share. When automatic retention is enabled, a file that has not been modified for the grace period is automatically retained at the default period value. Automatically retained files can have a longer retained-until-expiration date by manually setting the value.

A retained file cannot be modified, even after its expiration; this includes its name and attributes. When the expired date has been reached, retained files can be deleted, but not modified. A retained file's expiration date can be lengthened, but never shortened.

File retention is set at share creation, and it can be set to off (default), privileged, or mandatory. After setting privileged or mandatory file retention, you define the retention periods: minimum, maximum, default, and optional grace period.

This section contains the following topics:

• Privileged File Retention

• Mandatory File Retention

• Automatic File Retention

• Mandatory with Automatic File Retention Guidelines

• Retention Period Settings

• Prerequisites

• Planning Guidelines for File Retention
• Creating a Filesystem or Project with File Retention

• Viewing the Retention Policy Type and Statistics

Privileged File Retention

The privileged policy provides an override mechanism to allow retained files to be deleted before the expiry date. This requires the share to be exported with root access for NFS, and the user on the NFS client must be root.

A file's retained-until-expiration date can be set either manually before the file is marked as readonly or calculated based on last modification. When automatic retention is enabled, a file that has not been modified for the grace period is automatically retained for the time indicated by the default period. Files that are automatically retained can have their retained-until-expiration date manually set longer.

Unlike mandatory file retention, privileged file retention does not protect the share, project nor storage pool in which the retained file is contained. Therefore, a share, project, or storage pool containing actively retained files can be destroyed (share and project) or unconfigured (storage pool) at any time.

Mandatory File Retention

In addition to privileged file retention restrictions, mandatory file retention has further limitations. The retentionMandatory user role authorization is required to create a file with the mandatory retention policy. No user, not even the root user, can modify or delete a retained file. Also, shares with retained but unexpired files cannot be deleted, and storage pools containing files with unexpired retention cannot be unconfigured. A share with mandatory retention can be destroyed after all retained files have expired, and a storage pool with only expired retained files can be unconfigured. Also, a mandatorily retained file protects its ancestors and clone descendants from destruction.

Caution:

Mandatory file retention affects the filesystem, project, and storage pool. Carefully plan mandatory usage so that storage resources, especially pools and their associated drives, are not consumed for longer than necessary or overfilled.

You can change the retention period forward into the future but, unlike privileged file retention, you cannot change the retention backward in time. Thus, for example, you cannot immediately delete a retained filesystem, project, or its storage pool, but must wait until all files' retention times have expired.

To further protect mandatory retention shares, the software cannot be rolled back to a previous version without mandatory file retention nor can a storage pool's disks be reused while retention is active. Furthermore, NTP cannot be disabled, and NTP always synchronizes the clock, regardless of the skew amount. You cannot use a striped storage pool profile with mandatory retention because it does not provide redundancy. Also, the appliance factory reset feature is disabled until all mandatory file retention expires.

Automatic File Retention

The automatic file retention feature can be used with both mandatory and privileged file retention. When the grace period is set to a non-zero value, the file is automatically retained for the default retention period when the file has not been modified for the grace period. Until the grace period expires, the file can be modified—including its name, attributes, and ACL—or deleted. If necessary, the grace period can be modified by a user with the retentionAuto authorization.

If a filesystem is written to, the share's, project's, and pool's expiration is set to now plus the grace period plus the default retention period to protect the share, project, and storage pool. For this reason, after automatic retention is in effect, the grace period can be decreased, but not increased or unset.

Mandatory with Automatic File Retention Guidelines

Extra caution should be observed when using both mandatory and automatic file retention together because filesystems might be preserved for longer than anticipated or consume more storage resources than originally planned. Remember that mandatory file retention affects the filesystem, its project, and its storage pool.

For mandatory with automatic file retention, the share, project, and storage pool are protected for the grace period plus the default period when automatic retention is first enabled, and then extended with each write to a file. The files in the share must have no writes for the grace period plus the default period before the share, project or storage pool can be deleted or unconfigured.

Retention Period Settings

The retention period settings restrict the time values for retaining a file, and they are used for both mandatory and privileged file retention. If a file is retained in a mandatory retention policy filesystem, the file's retention time also affects its share, project, and storage pool. The administrator must have the retentionPeriods role authorization.

If no value is entered for a period, it is assumed to be zero. When setting a value, also set the time measurement unit, such as seconds, minutes or years.

• Minimum file retention period - The minimum amount of time for file retention. The default value is 0 (zero), and the value must be less than 100 years.

• Maximum file retention period - The maximum amount of time for file retention, which must be less than 100 years. The default value is 5 years.

• Default file retention period - The default amount of time for which a file is retained if it is automatically retained, or retained manually without first changing the file's access time attribute. After the default period has been met, the file cannot be modified, but it can be deleted. The default value must be between the minimum and maximum retention periods, inclusive. The default value is 0 (zero).

• Automatic file retention grace period - The amount of time a file must remain unmodified before it is automatically retained. This is known as the automatic file retention grace period or automatic file retention. When the grace period expires, the file is automatically retained at the default retention period setting. The grace period is not constrained by either the minimum period nor the maximum period, and it can only be modified by a user with the retentionAuto authorization. However, after the grace period is enabled, it can only be shortened, never extended, for the share.

File Retention on Expiry Policy

Use the file retention on expiry policy to automatically delete files after expiration or to place a hold on all retained files. The default setting is off, which does not apply any behavior after a file's retention expires, and the file remains on the system.

When set to delete, retained files are automatically deleted after expiration. The default file retention period must be a non-zero value. Optionally, set the property retention.period.deletegrace to the amount of time to delay automatically deleting the files.

When set to hold, the file's expiry date is extended indefinitely, and the retention.policy.onexpiry property must be changed to off or delete to be able to delete the file. Therefore, the hold setting is especially beneficial for files with legal holds.

If the retention.policy.onexpiry property is set to hold on a filesystem or project with mandatory file retention, the filesystem and project cannot be deleted, and the storage pool cannot be unconfigured. When viewing a pool's statistics, the number of held filesystems is displayed, along with the number of filesystems with mandatory retention.

Caution:

Mandatory file retention affects the filesystem, project, and storage pool. Carefully plan mandatory usage so that storage resources, especially pools and their associated drives, are not consumed for longer than necessary or overfilled.

The hold setting does not affect filesystems within a replication package nor does it affect the storage pool containing such replication packages. However, if a replication package containing a held filesystem is cloned, the replication package with the cloned and held filesystem is also held. Additionally, held filesystems in replication packages are not included the storage pool's held filesystems statistic.

To use the file retention on expiry policy, apply deferred update File Retention on Expiry, which is available in software version OS8.8.63 or later. For information on applying deferred updates, see Deferred Updates in Oracle ZFS Storage Appliance Customer Service Manual, Release OS8.8.x.

Systems with software releases earlier than OS8.8.63 and that have not accepted the deferred update cannot import storage pools that use this feature; also, datasets cannot be received with property retention.policy.onexpiry set to anything except off.

User authorization retentionOnexpiry is required to set property retention.policy.onexpiry. See Assign Authorizations to User Roles.

Planning Guidelines for File Retention

When planning for file retention, observe the following guidelines:

• Mandatory file retention affects the filesystem, project, and storage pool. Carefully plan mandatory usage so that storage resources, especially pools and their associated drives, are not consumed for longer than necessary or overfilled. Note that to rename a storage pool, you must unconfigure it and then immediately import it with a new name. You cannot unconfigure a storage pool with mandatorily retained datasets.

• If you think there is a risk of someone unconfiguring a storage pool to destroy it at the same time that someone else is wrongly adding a mandatorily retained filesystem or project to the storage pool, set the maximum file retention period to a higher value.

• For mandatory file retention, the storage pool profile must provide redundancy. Therefore, the striped profile cannot be used with storage pools with mandatorily retained files.

• Directories cannot be renamed until all retained files within them have been deleted or moved to another directory. This preserves the name of the retained file, including its path.

File retention affects filesystem functionality in the following areas:

• Editing - Although a retained file cannot be modified, a user with role authorization retentionPeriods can edit a filesystem to change the retention periods only, not including the grace period. To edit the grace period, a user must have role authorization retentionAuto.
• Deleting - A filesystem with privileged retention can be deleted at any time, even if unexpired files exist. A filesystem with mandatory retention can be deleted after all of its files have expired. A file with the file retention on expiry policy set to hold can be deleted when the policy is set to other than hold.
• Moving - A filesystem with privileged or mandatory file retention can be moved to another project.
• Renaming - A file or project with privileged or mandatory file retention cannot be renamed, neither when file retention is active nor after the retention has expired.
• Snapshots:
  • Editing, Deleting, Moving, Renaming - The same principles apply as for retained files and projects not within a snapshot.
  • Cloning - A snapshot containing a filesystem with retention can be cloned. A file with mandatory retention protects its ancestors and clone descendants from destruction. The hold setting for the file retention on expiry policy applies to clones.
  • Rollback - Rollback can be performed on a filesystem with the privileged retention policy, even when unexpired retained files exist. Filesystems with the mandatory retention policy can never be rolled back, even when all retained files have expired.
• Cloud Backup - When a retained filesystem or project is used as the snapshot for a cloud backup, the same retention principles apply. Additionally, snapshots, themselves, can support snapshot retention. For more information, see Taking a Snapshot - BUI, CLI.

• Remote Replication - Snapshots of a retained filesystem or project have the same constraints as the file retention feature. Also, the parent and children of a snapshot containing a retained filesystem or project follow the same rules.

  When replicating to a target appliance, that appliance must support the file retention feature by accepting the File Retention deferred update, available with software version OS8.8.45 or later. When replicating to a different storage pool on the same appliance, the target pool must have a redundant profile for a file with mandatory retention. Therefore, the striped profile cannot be used with storage pools with mandatorily retained files. When replicating to an NFS server for offline replication, file retention is maintained.

  Reverse replication is not supported to a filesystem with mandatory retention. This action would require rolling back the filesystem prior to the replication reversal, which is not allowed.

  The hold setting for the file retention on expiry policy does not affect filesystems within a replication package nor does it affect the storage pool containing such replication packages.

• Appliance Factory Reset - The appliance factory reset feature is disabled if actively retained filesystems exist with mandatory file retention.

Creating a Filesystem or Project with File Retention

File retention is set at file creation, and a filesystem inherits this setting from its project if retention was set at the project level. Therefore, set retention when creating a new project or when creating a new filesystem within an unretained project. The available settings for the file retention policy are as follows:

• Disabled - No file retention policy is set. This is the default setting.
• Privileged override - Allows a user with the retentionPeriods role authorization to override the retention periods, except the grace period, for both mandatory and privileged file retention. Although a file cannot be retained for less than the minimum period, a user could change the other periods backwards, such that the file could be deleted earlier than originally set. The retention period can also be extended, but not beyond the maximum retention period setting.
• Mandatory (no override) - A filesystem with mandatory retention cannot be deleted before the retention period expires, and the retention period cannot be modified or overridden.

While creating file retention, set the retention period properties described in Retention Period Settings. In the BUI, these properties are located under the ACCESS tab. In the CLI, use the get command at the filesystem or project level to see the retention period properties. Set a value and a time measurement; no entry is the same as a zero-value entry.

Optionally, set the file retention on expiry policy at the project or filesystem level. This policy allows automatic deletion after file retention expiry or sets the file retention to an indefinite hold after expiry. In addition, the property retention.period.deletegrace can be set to the amount of time to delay automatically deleting the files. To remove a hold, set the policy to other than hold. The default setting is off and no action occurs after the file's retention expires. User authorization retentionOnexpiry is required to set property retention.policy.onexpiry. In the BUI, this property is located under the ACCESS tab. In the CLI, use the get command at the filesystem or project level to see the policy property. For information, see File Retention on Expiry Policy.

Optionally, set the file ACL/permission changes policy at the project or filesystem level. In the BUI, the property is located under the ACCESS tab. In the CLI, use the get command at the filesystem or project level to see the policy property. When set to on, property retention.policy.changeacl allows you to change ACL settings or file permissions, other than write, on a retained file. Thus, if the ACL or permissions were set incorrectly when a file was initially retained, this policy allows you to change those settings.

Manual file retention is accomplished in one of four ways:

• The file is retained after the minimum retention period if no maximum and/or default period was set.
• The file is retained after the maximum retention period if no minimum and/or default period was set.
• The file is retained after the default retention period if a minimum and/or maximum period was not set.
• For a file with automatic file retention set, the file can be manually retained if the minimum retention period is set to a lower value than the (automatic file retention) grace period, and the default period is set to a higher value than the grace period.

Viewing the Retention Policy Type and Statistics

To view the retention policy type and statistics, use the following procedures. Note that in the BUI and if a dataset cannot be destroyed, its trash icon is not highlighted.

• Filesystem - In the shares area of the software, navigate to the filesystem's project and then the filesystem. In the BUI, additionally select the ACCESS tab. In the CLI, issue the get command. The following file retention attributes are displayed:

  • Policy type

  • Minimum period

  • Maximum period

  • Default period

  • Grace retention period

  • Grace delete period

  • Number of filesystems retained

  • Retention status: expiration date, time, and if expired

  • On expiry policy

• Project - In the shares area of the software, navigate to the project. In the CLI, issue the get command. In the BUI, the policy type is displayed in the Static Properties column. Select the ACCESS tab to view the retention period. The following file retention expiration attributes are displayed for each filesystem:

  • Date

  • Time

  • If expired

  • Policy type: In the BUI, a solid locked icon indicates mandatory file retention, and a non-solid unlocked icon indicates privileged file retention.

• Storage Pool - In the configuration storage area of the software, navigate to the storage pool. In the CLI, issue the get command. The following file retention attributes are displayed:

  • Policy type is always mandatory; privileged file retention does not affect a storage pool

  • Retention status: expiration date, time, and if expired

  • Number of filesystems retained

  • Number of filesystems set to hold

Related Topics

• Creating a Filesystem - BUI, CLI

• Creating a Project - BUI, CLI

• Configuring a Storage Pool - BUI, CLI

• Project and Share Properties

• Editing Authorizations for a Role - BUI, CLI

• Deferred Updates in Oracle ZFS Storage Appliance Customer Service Manual, Release OS8.8.x