Go to main content

Managing ZFS File Systems in Oracle® Solaris 11.4

Exit Print View

Updated: August 2019
 
 

Saving, Sending, and Receiving ZFS Data

The zfs send command creates a stream representation of a snapshot that is written to standard output. By default, a full stream is generated. You can redirect the output to a file or to a different system. The zfs receive command creates a snapshot whose contents are specified in the stream that is provided on standard input. If a full stream is received, a new file system is created as well. You can also send ZFS snapshot data and receive ZFS snapshot data and file systems.

In this Oracle Solaris release, the zfs send command has been enhanced with –w compress option. This option enables a system to perform a raw data transfer. In this type of transfer, data blocks that are compressed are read as is on the source disk and written as is on the target. No decompression-recompression occurs during the operation.

. This system can still receive data transfers from a source that does not have the zfs send –w compress option, such as systems running previous Oracle Solaris releases. In this case, the default behavior applies, where the compressed data blocks are first decompressed before they are transferred to the target system. After the transfer is complete, the blocks are then recompressed on the receiving system. For more information, see Example 40, Sending ZFS Data Using Raw Transfer.

In addition, this release includes the ability to resume transfering ZFS data. In particular, the transfer of large amounts of ZFS data can be interrupted due to network outages or system failure. To prevent having to resend the whole thing again, the zfs send and zfs receive commands can be run with the –C option to resume sending the ZFS data. For more information, see Using Resumable Replication.

Note the following backup solutions for saving ZFS data:

  • Enterprise backup products – These products provide the following features:

    • Per-file restoration

    • Backup media verification

    • Media management

  • File system snapshots and rolling back snapshots – Create a copy of a file system and revert to a previous file system version, if necessary.

    For more information about creating and rolling back to a snapshot, see Overview of ZFS Snapshots.

  • Saving snapshots – Using the zfs send and zfs receive commands, you can save incremental changes between snapshots but you cannot restore files individually. You must restore the entire file system snapshot..

  • Remote replication – Copy a file system from one system to another system. This process is different from a traditional volume management product that might mirror devices across a WAN. No special configuration or hardware is required. Using the zfs send and zfs receive commands to replicate a ZFS file system enables you to re-create a file system on a storage pool on another system, and specify different levels of configuration for the newly created pool, such as RAID-Z, but with identical file system data.

  • Archive utilities – Save ZFS data with archive utilities such as tar, cpio, and pax or third-party backup products. Currently, both tar and cpio translate NFSv4-style ACLs correctly, but pax does not.

Saving ZFS Data With Other Backup Products

In addition to the zfs send and zfs receive commands, you can also use archive utilities such as the tar and cpio commands, to save ZFS files. These utilities save and restore ZFS file attributes and ACLs. Check the appropriate options for both the tar and cpio commands.

Types of ZFS Snapshot Streams

The zfs send command can be used to create a stream of one or more snapshots. Then, you can use the snapshot stream to re-create a ZFS file system or volume by using the zfs receive command.

The zfs send options used to create the snapshot stream determine the stream format type that is generated.

  • Full stream – Consists of all dataset content from the time that the dataset was created up to the specified snapshot.

    The default stream generated by the zfs send command is a full stream. It contains one file system or volume, up to and including the specified snapshot. The stream does not contain snapshots other than the snapshot specified in the command.

  • Incremental stream – Consists of the differences between one snapshot and another snapshot.

A stream package is a stream type that contains one or more full or incremental streams. The types of stream packages are:

  • Replication stream package – Consists of the specified dataset and its descendants. It includes all intermediate snapshots. If the origin of a cloned dataset is not a descendant of the snapshot specified on the command line, that origin dataset is not included in the stream package. To receive the stream, the origin dataset must exist in the destination storage pool.


    Note -  A self-contained replication stream does not have external dependencies. See the section on self-contained replication streams below.

    Assume that the following list of datasets and their origins were created in the order in which they appear.

    NAME                    ORIGIN
    pool/a                  -
    pool/a/1                -
    pool/a/1@clone          -
    pool/b                  -
    pool/b/1                pool/a/1@clone
    pool/b/1@clone2         -
    pool/b/2                pool/b/1@clone2
    pool/b@pre-send         -
    pool/b/1@pre-send       -
    pool/b/2@pre-send       -
    pool/b@send             -
    pool/b/1@send           -
    pool/b/2@send           -

    Suppose you have a replication stream package created with the following syntax:

    $ zfs send -R pool/b@send ....

    This package would consist of the following full and incremental streams:

    TYPE    SNAPSHOT                INCREMENTAL FROM
    full    pool/b@pre-send         -
    incr    pool/b@send             pool/b@pre-send
    incr    pool/b/1@clone2         pool/a/1@clone
    incr    pool/b/1@pre-send       pool/b/1@clone2
    incr    pool/b/1@send           pool/b/1@send
    incr    pool/b/2@pre-send       pool/b/1@clone2
    incr    pool/b/2@send           pool/b/2@pre-send

    In the output, the pool/a/1@clone snapshot is not included in the replication stream package. Therefore, this replication stream package can only be received in a pool that already has pool/a/1@clone snapshot.

  • Self-contained replication stream package - This type of package is not dependent on any datasets that are not included in the stream package. You create a replication stream package with syntax similar to the following example:

    $ zfs send -Rc pool/b@send ...

    This example package would consist of the following full and incremental streams:

    TYPE    SNAPSHOT             INCREMENTAL FROM 
    full    pool/b@pre-send      - 
    full    pool/b/1@clone2      - 
    incr    pool/b@send          pool/b@pre-send
    incr    pool/b/1@pre-send    pool/b/1@clone2 
    incr    pool/b/1@send        pool/b/1@send 
    incr    pool/b/2@pre-send    pool/b/1@clone2 
    incr    pool/b/2@send        pool/b/2@pre-send 
    

    Comparing with the non-self-contained replication stream, notice that this self-contained replication stream has an integrated full stream of the pool/b/1@clone2 snapshot. This snapshot is an integrated dataset that has clone origin bits merged into it as data; clone2 is no longer a full clone with a separate origin. This makes it possible to receive the pool/b/1 snapshot with no external dependencies.

  • Recursive stream package – Consists of the specified dataset and its descendants. Unlike replication stream packages, intermediate snapshots are not included unless they are the origin of a cloned dataset that is included in the stream. By default, if the origin of a dataset is not a descendant of the snapshot specified in the command, the behavior is similar to replication streams. Note that a self-contained recursive stream does not have external dependencies.


    Note -  A self-contained recursive stream does not have external dependencies. See the section on self-contained recursive streams below.

    You create a recursive stream package with syntax similar to the following example:

    $ zfs send -r pool/b@send ... 

    This example package would consist of the following full and incremental streams:

    TYPE    SNAPSHOT                INCREMENTAL FROM
    full    pool/b@send             -
    incr    pool/b/1@clone2         pool/a/1@clone
    incr    pool/b/1@send           pool/b/1@clone2
    incr    pool/b/2@send           pool/b/1@clone2

    In the output, the pool/a/1@clone snapshot is not included in the recursive stream package. Therefore, similar to the replication stream package,this recursive stream package can only be received in a pool that already has pool/a/1@clone snapshot. This behavior is similar to the replication stream package scenario described above.

  • Self-contained recursive stream package - This type of package is not dependent on any datasets that are not included in the stream package. You create a recursive stream package with syntax similar to the following example:

    $ zfs send -rc pool/b@send ...

    This example package would consist of the following full and incremental streams:

    TYPE    SNAPSHOT                INCREMENTAL FROM
    full    pool/b@send             -
    full    pool/b/1@clone2         -
    incr    pool/b/1@send           pool/b/1@clone2 
    incr    pool/b/2@send           pool/b/1@clone2 
    

Sending a ZFS Snapshot

You can use the zfs send command to send a copy of a snapshot stream and receive the snapshot stream in another pool on the same system or in another pool on a different system that is used to store backup data. For example, to send the snapshot stream on a different pool to the same system, use a command similar to the following example:

$ zfs send pool/dana@snap1 | zfs recv spool/ds01

Tip  -  You can use zfs recv as an alias for the zfs receive command.

If you are sending the snapshot stream to a different system, pipe the zfs send output through the ssh command. For example:

sys1$ zfs send pool/dana@snap1 | ssh sys2 zfs recv pool/dana

When you send a full stream, the destination file system must not exist.

If you need to store many copies, consider compressing a ZFS snapshot stream representation with the gzip command. For example:

$ zfs send pool/fs@snap | gzip > backupfile.gz
Example 39  Sending Incremental ZFS Data

You can send incremental data by using the zfs send –i option. For example:

sys1$ zfs send -i pool/dana@snap1 system1/dana@snap2 | ssh system2 zfs recv pool/dana

The first argument (snap1) is the earlier snapshot and the second argument (snap2) is the later snapshot. In this case, the pool/dana file system must already exist for the incremental receive to be successful.

You can specify the incremental snap1 source as the last component of the snapshot name. You would then have to specify only the name after the @ sign for snap1, which is assumed to be from the same file system as snap2. For example:

sys1$ zfs send -i snap1 pool/dana@snap2 | ssh system2 zfs recv pool/dana

This shortcut syntax is equivalent to the incremental syntax.

The following message is displayed if you attempt to generate an incremental stream from a different file system snapshot1:

cannot send 'pool/fs@name': not an earlier snapshot from the same fs

Accessing file information in the original received file system can cause the incremental snapshot receive operation to fail with a message similar to this one:

cannot receive incremental stream of pool/dana@snap2 into pool/dana:
most recent snapshot of pool/dana@snap2 does not match incremental source

Consider setting the atime property to off if you need to access file information in the original received file system and if you also need to receive incremental snapshots into the received file system.

If you need to store many copies, consider compressing a ZFS snapshot stream representation with the gzip command. For example:

$ zfs send pool/fs@snap | gzip > backupfile.gz
Example 40  Sending ZFS Data Using Raw Transfer

To send the stream in raw mode, use the –w compress option.

The following example shows that the raw transfer stream for a given snapshot is smaller, even though the file system created after receiving the stream is the same as the original. First, create a file system called pool/compressed-fs which you fill with data.

$ zfs create -o compression=gzip-6 pool/compressed-fs
$ cp /usr/dict/words /pool/compressed-fs/

Next, create a snapshot and check the compression ratio. For comparison purposes, create two streams to see the difference in sizes between a regular transfer and a raw transfer, Note that the rawstream file is smaller.

$ zfs snapshot pool/compressed-fs@snap
$ zfs get compressratio pool/compressed-fs@snap
NAME                        PROPERTY       VALUE  SOURCE
pool/compressed-fs@snap  compressratio  2.80x  -
$ zfs send pool/compressed-fs@snap > /tmp/stream
$ zfs send -w compress pool/compressed-fs@snap > /tmp/rawstream
$ ls -lh /tmp/*stream
-rw-r--r--   1 root     root        100K Dec 23 18:23 /tmp/rawstream
-rw-r--r--   1 root     root        304K Dec 23 18:23 /tmp/stream

Next, receive the raw transfer stream on its new location. Then to verify that the content is identical, compare the new file system to the original.

$ zfs receive pool/rawrecv </tmp/rawstream
$ diff -r /pool/compressed-fs/ /pool/rawrecv/
#
Example 41  Sending ZFS Data From a Oracle Solaris 11.4.0 Dataset

The ability to use per record checksums in the output data stream is enabled by default. To transfer data to older systems, you must disable this feature using the nocheck argument.

$ zfs send -s nocheck pool/dana@snap1 | zfs recv pool/ds01

Using Resumable Replication

If the transfer of data using zfs receive is interrupted, you can restart the process. For instance, if you started the transfer with this command.

system1$ zfs send pool/dana@snap1 | ssh system2 zfs recv pool/dana

If the transfer is interrupted, you will have an incomplete dataset. You can resume the transfer using this series of commands:

system1$ ssh system2 zfs receive -C pool/dana | zfs send -C pool/dana@snap1 | \
     ssh system2 zfs receive pool/dana

To see which datasets are incomplete use the zfs list –I command, see Listing Incomplete ZFS Datasets.

Receiving a ZFS Snapshot

Keep the following key points in mind when you receive a file system snapshot:

  • Both the snapshot and the file system are received.

  • The file system and all descendant file systems are unmounted.

  • The file systems are inaccessible while they are being received.

  • A file system with the same name as the source file system to be received must not exist on the target system. If the file system name exists on the target system, rename the file system.

For example:

$ zfs send system1/gozer@0830 > /bkups/gozer.083006
$ zfs receive system1/gozer2@today < /bkups/gozer.083006
$ zfs rename system1/gozer system1/gozer.old
$ zfs rename system1/gozer2 system1/gozer

If you make a change to the destination file system and you want to perform another incremental send of a snapshot, you must first roll back the receiving file system.

Consider the following example. First, make a change to the file system as follows:

sys2$ rm newsys/dana/file.1

Then, perform an incremental send of system1/dana@snap3. Note that either you must first roll back the receiving file system to receive the new incremental snapshot or eliminate the rollback step by using the –F option. For example:

sys1$ zfs send -i system1/dana@snap2 system1/dana@snap3 | ssh sys2 zfs recv -F newsys/dana

When you receive an incremental snapshot, the destination file system must already exist.

If you make changes to the file system and you do not roll back the receiving file system to receive the new incremental snapshot or you use the –F option, a message similar to the following example is displayed:

sys1$ zfs send -i system1/dana@snap4 system1/dana@snap5 | ssh sys2 zfs recv newsys/dana
cannot receive: destination has been modified since most recent snapshot

The following checks are performed before the –F option is successful:

  • If the most recent snapshot doesn't match the incremental source, neither the rollback nor the receive is completed, and an error message is returned.

  • If you accidentally provide the name of different file system that doesn't match the incremental source specified in the zfs receive command, neither the rollback nor the receive is completed, and the following error message is returned:

    cannot send 'pool/fs@name': not an earlier snapshot from the same fs

Applying Different Property Values to a ZFS Snapshot Stream

You can send a ZFS snapshot stream with a certain file system property value, and then either specify a different local property value when the snapshot stream is received or specify that the original property value be used when the snapshot stream is received to re-create the original file system. In addition, you can disable a file system property when the snapshot stream is received.

  • To revert a local property value to the received value, if any, use the zfs inherit –S command. If a property does not have a received value, the behavior of the –S option is the same as if you did not include the option. If the property does have a received value, the zfs inherit command masks the received value with the inherited value until issuing a zfs inherit –S command reverts it to the received value.

  • You can determine the columns displayed by the zfs get. Use the –o option to include the new non-default RECEIVED column. Use the –o all option to include all columns, including RECEIVED.

  • To include properties in the send stream without the –R option, use the –p option.

  • To use the last element of the sent snapshot name to determine the new snapshot name, use the –e option.

    The following example sends the poolA/bee/cee@1 snapshot to the poolD/eee file system and uses only the last element (cee@1) of the snapshot name to create the received file system and snapshot.

    $ zfs list -rt all poolA
    NAME              USED  AVAIL  REFER  MOUNTPOINT
    poolA             134K   134G    23K  /poolA
    poolA/bee          44K   134G    23K  /poolA/bee
    poolA/bee/cee      21K   134G    21K  /poolA/bee/cee
    poolA/bee/cee@1      0      -    21K  -
    $ zfs send -R poolA/bee/cee@1 | zfs receive -e poolD/eee
    $ zfs list -rt all poolD
    NAME              USED  AVAIL  REFER  MOUNTPOINT
    poolD             134K   134G    23K  /poolD
    poolD/eee          44K   134G    23K  /poolD/eee
    poolD/eee/cee      21K   134G    21K  /poolD/eee/cee
    poolD/eee/cee@1      0      -    21K  -

Retaining Original Property Values

In some cases, file system properties in a send stream might not apply to the receiving file system or local file system properties, such as the mountpoint property value, might interfere with a restore.

For example, suppose that the system1/data file system has the compression property disabled. A snapshot of the system1/data file system is sent with properties (–p option) to a backup pool and is received with the compression property enabled.

$ zfs get compression system1/data
NAME          PROPERTY     VALUE     SOURCE
system1/data  compression  off       default
$ zfs snapshot system1/data@snap1
$ zfs send -p system1/data@snap1 | zfs recv -o compression=on -d bpool
$ zfs get -o all compression bpool/data
NAME        PROPERTY     VALUE     RECEIVED  SOURCE
bpool/data  compression  on        off       local

In the example, the compression property is enabled when the snapshot is received into bpool. So, for bpool/data, the compression value is on.

If this snapshot stream is sent to a new pool, restorepool, for recovery purposes, you might want to keep all the original snapshot properties. In this case, you would use the zfs send –b option to restore the original snapshot properties. For example:

$ zfs send -b bpool/data@snap1 | zfs recv -d restorepool
$ zfs get -o all compression restorepool/data
NAME              PROPERTY     VALUE     RECEIVED  SOURCE
restorepool/data  compression  off       off       received 

In the example, the compression value is off, which represents the snapshot compression value from the original system1/data file system.

Disabling Original Property Values

If you have a local file system property value in a snapshot stream and you want to disable the property when it is received, use the zfs receive –x option. For example, the following command sends a recursive snapshot stream of home directory file systems with all file system properties reserved to a backup pool but without the quota property values:

$ zfs send -R system1/home@snap1 | zfs recv -x quota bpool/home
$ zfs get -r quota bpool/home
NAME                   PROPERTY  VALUE  SOURCE
bpool/home             quota     none   local
bpool/home@snap1       quota     -      -
bpool/home/lori        quota     none   default
bpool/home/lori@snap1  quota     -      -
bpool/home/mark        quota     none   default
bpool/home/mark@snap1  quota     -      -

If the recursive snapshot was not received with the –x option, the quota property would be set in the received file systems.

$ zfs send -R system1/home@snap1 | zfs recv bpool/home
$ zfs get -r quota bpool/home
NAME                   PROPERTY  VALUE  SOURCE
bpool/home             quota     none   received
bpool/home@snap1       quota     -      -
bpool/home/lori        quota     10G    received
bpool/home/lori@snap1  quota     -      -
bpool/home/mark        quota     10G    received
bpool/home/mark@snap1  quota     -      -

Sending and Receiving Complex ZFS Snapshot Streams

This section describes how to use the –I and –R options with the zfs send command to send and receive more complex snapshot streams.

    Keep the following points in mind when sending and receiving complex ZFS snapshot streams:

  • To send all incremental streams from one snapshot to a cumulative snapshot, use the –I option. You can also use this option to send an incremental stream from the original snapshot to create a clone. The original snapshot must already exist on the receiving side to accept the incremental stream.

  • To send a replication stream of all descendant file systems, use the –R option. When the replication stream is received, all properties, snapshots, descendant file systems, and clones are preserved.

  • Using the zfs send –r command or the zfs send –R command to send package streams without the –c option will omit the origin of clones in some circumstances. For more information, see Types of ZFS Snapshot Streams.

  • Use both options to send an incremental replication stream.

    • Changes to properties are preserved, as are snapshot and file system rename and destroy operations.

    • If the –F option is not specified when receiving the replication stream, dataset destroy operations are ignored. Thus, if necessary, you can undo the receive operation and restore the file system to its previous state.

    • When sending incremental streams, if –I is used, all snapshots between snapA and snapD are sent. If –i is used, only snapD (for all descendants) snapshots are sent.

  • To receive any of these types of zfs send streams, the receiving system must be running a software version capable of sending them. The stream version is incremented.

Example 42  Sending and Receiving Complex ZFS Snapshot Streams

You can combine A group of incremental snapshots into one snapshot by using the –I option. For example:

$ zfs send -I pool/fs@snapA pool/fs@snapD > /snaps/fs@all-I

You would then remove the incremental snapB, snapC, and snapD snapshots.

$ zfs destroy pool/fs@snapB
$ zfs destroy pool/fs@snapC
$ zfs destroy pool/fs@snapD

To receive the combined snapshot, you would use the following command.

$ zfs receive -d -F pool/fs < /snaps/fs@all-I
$ zfs list
NAME                      USED  AVAIL  REFER  MOUNTPOINT
pool                      428K  16.5G    20K  /pool
pool/fs                    71K  16.5G    21K  /pool/fs
pool/fs@snapA              16K      -  18.5K  -
pool/fs@snapB              17K      -    20K  -
pool/fs@snapC              17K      -  20.5K  -
pool/fs@snapD                0      -    21K  -

You can also use the –I command to combine a snapshot and a clone snapshot to create a combined dataset. For example:

$ zfs create pool/fs
$ zfs snapshot pool/fs@snap1
$ zfs clone pool/fs@snap1 pool/clone
$ zfs snapshot pool/clone@snapA
$ zfs send -I pool/fs@snap1 pool/clone@snapA > /snaps/fsclonesnap-I
$ zfs destroy pool/clone@snapA
$ zfs destroy pool/clone
$ zfs receive -F pool/clone < /snaps/fsclonesnap-I

To replicate a ZFS file system and all descendant file systems up to the named snapshot, use the –R option. When this stream is received, all properties, snapshots, descendant file systems, and clones are preserved.

The following example creates snapshots for user file systems. One replication stream is created for all user snapshots. Next, the original file systems and snapshots are destroyed and then recovered.

$ zfs snapshot -r users@today
$ zfs list
NAME                USED  AVAIL  REFER  MOUNTPOINT
users               187K  33.2G    22K  /users
users@today            0      -    22K  -
users/user1          18K  33.2G    18K  /users/user1
users/user1@today      0      -    18K  -
users/user2          18K  33.2G    18K  /users/user2
users/user2@today      0      -    18K  -
users/user3          18K  33.2G    18K  /users/user3
users/user3@today      0      -    18K  -
$ zfs send -R users@today > /snaps/users-R
$ zfs destroy -r users
$ zfs receive -F -d users < /snaps/users-R
$ zfs list
NAME                USED  AVAIL  REFER  MOUNTPOINT
users               196K  33.2G    22K  /users
users@today            0      -    22K  -
users/user1          18K  33.2G    18K  /users/user1
users/user1@today      0      -    18K  -
users/user2          18K  33.2G    18K  /users/user2
users/user2@today      0      -    18K  -
users/user3          18K  33.2G    18K  /users/user3
users/user3@today      0      -    18K  -

The following example uses the –R command to replicate the users file system and its descendants, and to send the replicated stream to another pool, users2.

$ zfs create users2 mirror c0t1d0 c1t1d0
$ zfs receive -F -d users2 < /snaps/users-R
$ zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
users                224K  33.2G    22K  /users
users@today             0      -    22K  -
users/user1           33K  33.2G    18K  /users/user1
users/user1@today     15K      -    18K  -
users/user2           18K  33.2G    18K  /users/user2
users/user2@today       0      -    18K  -
users/user3           18K  33.2G    18K  /users/user3
users/user3@today       0      -    18K  -
users2               188K  16.5G    22K  /users2
users2@today            0      -    22K  -
users2/user1          18K  16.5G    18K  /users2/user1
users2/user1@today      0      -    18K  -
users2/user2          18K  16.5G    18K  /users2/user2
users2/user2@today      0      -    18K  -
users2/user3          18K  16.5G    18K  /users2/user3
users2/user3@today      0      -    18K  -

Remote Replication of ZFS Data

You can use the zfs send and zfs recv commands to remotely copy a snapshot stream representation from one system to another system. For example:

$ zfs send system1/cindy@today | ssh newsys zfs recv sandbox/restfs@today

This command sends the system1/cindy@today snapshot data and receives it into the sandbox/restfs file system. The command also creates a restfs@today snapshot on the newsys system. In this example, the user has been configured to use ssh on the remote system.