Solaris ZFS Administration Guide

Sending and Receiving ZFS Data

The zfs send command creates a stream representation of a snapshot that is written to standard output. By default, a full stream is generated. You can redirect the output to a file or to a different system. The zfs receive command creates a snapshot whose contents are specified in the stream that is provided on standard input. If a full stream is received, a new file system is created as well. You can send ZFS snapshot data and receive ZFS snapshot data and file systems with these commands. See the examples in the next section.

The following backup solutions for saving ZFS data are available:

Enterprise backup products – If you need the following features then consider an enterprise backup solution:
- Per-file restoration
- Backup media verification
- media management
File system snapshots and rolling back snapshots – Use the zfs snapshot and zfs rollback commands if you want to easily create a copy of a file system and revert back to a previous file system version, if necessary. For example, if you want to restore a file or files from a previous version of a file system, you could use this solution.

For more information about creating and rolling back to a snapshot, see Overview of ZFS Snapshots.
Saving snapshots – Use the zfs send and zfs receive commands to send and receive a ZFS snapshot. You can save incremental changes between snapshots, but you cannot restore files individually. You must restore the entire file system snapshot. These commands do not provide a complete backup solution for saving your ZFS data.
Remote replication – Use the zfs send and zfs receive commands when you want to copy a file system from one system to another. This process is different from a traditional volume management product that might mirror devices across a WAN. No special configuration or hardware is required. The advantage of replicating a ZFS file system is that you can re-create a file system on a storage pool on another system, and specify different levels of configuration for the newly created pool, such as RAID-Z, but with identical file system data.
Archive utilities – Save ZFS data with archive utilities such as tar, cpio, and pax or third-party backup products. Currently, both tar and cpio translate NFSv4-style ACLs correctly, but pax does not translate NFSv4–style ACLs correctly.

Sending a ZFS Snapshot

You can use the zfs send command to send a copy of a snapshot and receive the snapshot in another pool on the same system or in another pool on a different system that is used to store backup data. For example, to send the snapshot on a different pool on the same system, use syntax similar to the following:

# zfs send tank/data@snap1 | zfs recv spool/ds01

You can use zfs recv as an alias for the zfs receive command.

If you are sending the snapshot stream to a different system, pipe the zfs send output through the ssh command. For example:

host1# zfs send tank/dana@snap1 | ssh host2 zfs recv newtank/dana

When sending a full stream, the destination file system must not exist.

You can send incremental data by using the zfs send -i option. For example:

host1# zfs send -i tank/dana@snap1 tank/dana@snap2 | ssh host2 zfs recv newtank/dana

Note that the first argument is the earlier snapshot (snap1) and the second argument (snap2) is the later snapshot. In this case, the newtank/dana file system must exist for the incremental receive to be successful.

The incremental snap1 source can be specified as the last component of the snapshot name. This shortcut means you only have to specify the name after the @ sign for snap1, which is assumed to be from the same file system as snap2. For example:

host1# zfs send -i snap1 tank/dana@snap2 > ssh host2 zfs recv newtank/dana

This syntax is equivalent to the above example of the incremental syntax.

The following message is displayed if you attempt to generate an incremental stream from a different file system snapshot1:

cannot send 'pool/fs@name': not an earlier snapshot from the same fs

If you need to store many copies, you might consider compressing a ZFS snapshot stream representation with the gzip command. For example:

# zfs send pool/fs@snap | gzip > backupfile.gz

Receiving a ZFS Snapshot

Keep the following key points in mind when you receive a file system snapshot:

The snapshot and the file system are received.
The file system and all descendent file systems are unmounted.
The file systems are inaccessible while they are being received.
The original file system to be received must not exist while it is being transferred.
If a conflicting file system name exists, zfs rename can be used to rename the file system.

For example:

# zfs send tank/gozer@0830 > /bkups/gozer.083006
# zfs receive tank/gozer2@today < /bkups/gozer.083006
# zfs rename tank/gozer tank/gozer.old
# zfs rename tank/gozer2 tank/gozer

If you make a change to the destination file system and you want to do another incremental send of a snapshot, you must first rollback the receiving file system.

For example, if you make a change to the file system as follows:

host2# rm newtank/dana/file.1

And you do an incremental send of tank/dana@snap3, you must first rollback the receiving file system to receive the new incremental snapshot. You can eliminate the rollback step by using the -F option. For example:

host1# zfs send -i tank/dana@snap2 tank/dana@snap3 | ssh host2 zfs recv -F newtank/dana

When you receive an incremental snapshot, the destination file system must already exist.

If you make changes to the file system and you do not rollback the receiving file system to receive the new incremental snapshot or you do not use the -F option, you will see the following message:

host1# zfs send -i tank/dana@snap4 tank/dana@snap5 | ssh host2 zfs recv newtank/dana
cannot receive: destination has been modified since most recent snapshot

The following checks are performed before the -F option is successful:

If the most recent snapshot doesn't match the incremental source, neither the rollback nor the receive is completed, and an error message is returned.
If you accidentally provide the name of different file system that doesn't match the incremental source to the zfs receive command, neither the rollback nor the receive is completed, and the following error message is returned.
cannot send 'pool/fs@name': not an earlier snapshot from the same fs

Sending and Receiving Complex ZFS Snapshot Streams

This section describes how to use the zfs send -I and -R options to send and receive more complex snapshot streams.

Keep the following points in mind when sending and receiving ZFS snapshot streams:

Use the zfs send -I option to send all incremental streams from one snapshot to a cumulative snapshot. Or, use this option to send an incremental stream from the original snapshot to create a clone. The original snapshot must already exist on the receiving side to accept the incremental stream.
Use the zfs send -R option to send a replication stream of all descendent file systems. When received, all properties, snapshots, descendent file systems, and clones are preserved.
Or use both options to send an incremental replication stream.
- Changes to properties and snapshot and file system renames and destroys are preserved.
- If zfs recv -F is not specified when receiving the replication stream, dataset destroys are ignored. The zfs recv -F syntax in this case also retains its rollback if necessary meaning.
- As with other (non zfs send -R) -i or -I cases, if -I is used, all snapshots between snapA and snapD are sent. If -i is used, only snapD (for all descendents) are sent.
To receive any of these new types of zfs send streams, the receiving system must be running a software version capable of sending them. The stream version is incremented.

However, you can access streams from older pool versions by using a newer software version. For example, you can send and receive streams created with the newer options to and from a version 3 pool. But, you must be running recent software to receive a stream sent with the newer options.

Example 7–1 Examples—Sending and Receiving Complex ZFS Snapshot Streams

A group of incremental snapshots can be combined into one snapshot by using the zfs send -I option. For example:

# zfs send -I pool/fs@snapA pool/fs@snapD > /snaps/fs@all-I

Remove snapshots B, C, and D.

# zfs destroy pool/fs@snapB
# zfs destroy pool/fs@snapC
# zfs destroy pool/fs@snapD

Receive the combined snapshot.

# zfs receive -d -F pool/fs < /snaps/fs@all-I
# zfs list
NAME                      USED  AVAIL  REFER  MOUNTPOINT
pool                      428K  16.5G    20K  /pool
pool/fs                    71K  16.5G    21K  /pool/fs
pool/fs@snapA              16K      -  18.5K  -
pool/fs@snapB              17K      -    20K  -
pool/fs@snapC              17K      -  20.5K  -
pool/fs@snapD                0      -    21K  -

You can also use the zfs send -I command to combine a snapshot and a clone snapshot to create a combined dataset. For example:

# zfs create pool/fs
# zfs snapshot pool/fs@snap1
# zfs clone pool/fs@snap1 pool/clone
# zfs snapshot pool/clone@snapA
# zfs send -I pool/fs@snap1 pool/clone@snapA > /snaps/fsclonesnap-I
# zfs destroy pool/clone@snapA
# zfs destroy pool/clone
# zfs receive -F pool/clone < /snaps/fsclonesnap-I

Use the zfs send -R command to replicate a ZFS file system and all descendent file systems, up to the named snapshot. When received, all properties, snapshots, descendent file systems, and clones are preserved.

In the following example, snapshots are created for user file systems. One replication stream is created for all user snapshots. Then, the original file systems and snapshots are destroyed and recovered.

# zfs snapshot -r users@today
# zfs list
NAME                USED  AVAIL  REFER  MOUNTPOINT
users               187K  33.2G    22K  /users
users@today            0      -    22K  -
users/user1          18K  33.2G    18K  /users/user1
users/user1@today      0      -    18K  -
users/user2          18K  33.2G    18K  /users/user2
users/user2@today      0      -    18K  -
users/user3          18K  33.2G    18K  /users/user3
users/user3@today      0      -    18K  -
# zfs send -R users@today > /snaps/users-R
# zfs destroy -r users
# zfs receive -F -d users < /snaps/users-R
# zfs list
NAME                USED  AVAIL  REFER  MOUNTPOINT
users               196K  33.2G    22K  /users
users@today            0      -    22K  -
users/user1          18K  33.2G    18K  /users/user1
users/user1@today      0      -    18K  -
users/user2          18K  33.2G    18K  /users/user2
users/user2@today      0      -    18K  -
users/user3          18K  33.2G    18K  /users/user3
users/user3@today      0      -    18K  -

You can use the zfs send -R command to replicate the users dataset and its descendents and send the replicated stream to another pool, users2.

# zfs create users2 mirror c0t1d0 c1t1d0
# zfs receive -F -d users2 < /snaps/users-R
# zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
users                224K  33.2G    22K  /users
users@today             0      -    22K  -
users/user1           33K  33.2G    18K  /users/user1
users/user1@today     15K      -    18K  -
users/user2           18K  33.2G    18K  /users/user2
users/user2@today       0      -    18K  -
users/user3           18K  33.2G    18K  /users/user3
users/user3@today       0      -    18K  -
users2               188K  16.5G    22K  /users2
users2@today            0      -    22K  -
users2/user1          18K  16.5G    18K  /users2/user1
users2/user1@today      0      -    18K  -
users2/user2          18K  16.5G    18K  /users2/user2
users2/user2@today      0      -    18K  -
users2/user3          18K  16.5G    18K  /users2/user3
users2/user3@today      0      -    18K  -

Remote Replication of ZFS Data

You can use the zfs send and zfs recv commands to remotely copy a snapshot stream representation from one system to another system. For example:

# zfs send tank/cindy@today | ssh newsys zfs recv sandbox/restfs@today

This command sends the tank/cindy@today snapshot data and receives it into the sandbox/restfs file system and also creates a restfs@today snapshot on the newsys system. In this example, the user has been configured to use ssh on the remote system.

Saving ZFS Data With Other Backup Products

In addition to the zfs send and zfs receive commands, you can also use archive utilities, such as the tar and cpio commands, to save ZFS files. All of these utilities save and restore ZFS file attributes and ACLs. Check the appropriate options for both the tar and cpio commands.

For up-to-date information about issues with ZFS and third-party backup products, please see the or the ZFS FAQ, here:

http://hub.opensolaris.org/bin/view/Community+Group+zfs/faq/#backupsoftware