Chapter 4 Managing Oracle Solaris ZFS Storage Pools
This chapter describes how to create and administer storage pools in Oracle Solaris ZFS.
The following sections are provided in this chapter:
Components of a ZFS Storage Pool
The following sections provide detailed information about the following storage pool components:
Using Disks in a ZFS Storage Pool
The most basic element of a storage pool is physical storage. Physical storage can be any block device of at least 128 MB in size. Typically, this device is a hard drive that is visible to the system in the /dev/dsk directory.
A storage device can be a whole disk (c1t0d0) or an individual slice (c0t0d0s7). The recommended mode of operation is to use an entire disk, in which case the disk does not require special formatting. ZFS formats the disk using an EFI label to contain a single, large slice. When used in this way, the partition table that is displayed by the format command appears similar to the following:
Current partition table (original): Total disk sectors available: 286722878 + 16384 (reserved sectors) Part Tag Flag First Sector Size Last Sector 0 usr wm 34 136.72GB 286722911 1 unassigned wm 0 0 0 2 unassigned wm 0 0 0 3 unassigned wm 0 0 0 4 unassigned wm 0 0 0 5 unassigned wm 0 0 0 6 unassigned wm 0 0 0 8 reserved wm 286722912 8.00MB 286739295 |
To use a whole disk, the disk must be named by using the /dev/dsk/cNtNdN naming convention. Some third-party drivers use a different naming convention or place disks in a location other than the /dev/dsk directory. To use these disks, you must manually label the disk and provide a slice to ZFS.
ZFS applies an EFI label when you create a storage pool with whole disks. For more information about EFI labels, see EFI Disk Label in System Administration Guide: Devices and File Systems.
A disk that is intended for a ZFS root pool must be created with an SMI label, not an EFI label. You can relabel a disk with an SMI label by using the format -e command.
Disks can be specified by using either the full path, such as /dev/dsk/c1t0d0, or a shorthand name that consists of the device name within the /dev/dsk directory, such as c1t0d0. For example, the following are valid disk names:
-
c1t0d0
-
/dev/dsk/c1t0d0
-
/dev/foo/disk
Using whole physical disks is the easiest way to create ZFS storage pools. ZFS configurations become progressively more complex, from management, reliability, and performance perspectives, when you build pools from disk slices, LUNs in hardware RAID arrays, or volumes presented by software-based volume managers. The following considerations might help you determine how to configure ZFS with other hardware or software storage solutions:
-
If you construct a ZFS configuration on top of LUNs from hardware RAID arrays, you need to understand the relationship between ZFS redundancy features and the redundancy features offered by the array. Certain configurations might provide adequate redundancy and performance, but other configurations might not.
-
You can construct logical devices for ZFS using volumes presented by software-based volume managers, such as Solaris Volume Manager (SVM) or Veritas Volume Manager (VxVM). However, these configurations are not recommended. Although ZFS functions properly on such devices, less-than-optimal performance might be the result.
For additional information about storage pool recommendations, see the ZFS best practices site:
http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide
Disks are identified both by their path and by their device ID, if available. On systems where device ID information is available, this identification method allows devices to be reconfigured without updating ZFS. Because device ID generation and management can vary by system, export the pool first before moving devices, such as moving a disk from one controller to another controller. A system event, such as a firmware update or other hardware change, might change the device IDs in your ZFS storage pool, which can cause the devices to become unavailable.
Using Slices in a ZFS Storage Pool
Disks can be labeled with a traditional Solaris VTOC (SMI) label when you create a storage pool with a disk slice.
For a bootable ZFS root pool, the disks in the pool must contain slices and the disks must be labeled with an SMI label. The simplest configuration would be to put the entire disk capacity in slice 0 and use that slice for the root pool.
On a SPARC based system, a 72-GB disk has 68 GB of usable space located in slice 0 as shown in the following format output:
# format . . . Specify disk (enter its number): 4 selecting c1t1d0 partition> p Current partition table (original): Total disk cylinders available: 14087 + 2 (reserved cylinders) Part Tag Flag Cylinders Size Blocks 0 root wm 0 - 14086 68.35GB (14087/0/0) 143349312 1 unassigned wm 0 0 (0/0/0) 0 2 backup wm 0 - 14086 68.35GB (14087/0/0) 143349312 3 unassigned wm 0 0 (0/0/0) 0 4 unassigned wm 0 0 (0/0/0) 0 5 unassigned wm 0 0 (0/0/0) 0 6 unassigned wm 0 0 (0/0/0) 0 7 unassigned wm 0 0 (0/0/0) 0 |
On an x86 based system, a 72-GB disk has 68 GB of usable disk space located in slice 0, as shown in the following format output. A small amount of boot information is contained in slice 8. Slice 8 requires no administration and cannot be changed.
# format . . . selecting c1t0d0 partition> p Current partition table (original): Total disk cylinders available: 49779 + 2 (reserved cylinders) Part Tag Flag Cylinders Size Blocks 0 root wm 1 - 49778 68.36GB (49778/0/0) 143360640 1 unassigned wu 0 0 (0/0/0) 0 2 backup wm 0 - 49778 68.36GB (49779/0/0) 143363520 3 unassigned wu 0 0 (0/0/0) 0 4 unassigned wu 0 0 (0/0/0) 0 5 unassigned wu 0 0 (0/0/0) 0 6 unassigned wu 0 0 (0/0/0) 0 7 unassigned wu 0 0 (0/0/0) 0 8 boot wu 0 - 0 1.41MB (1/0/0) 2880 9 unassigned wu 0 0 (0/0/0) 0 |
Using Files in a ZFS Storage Pool
ZFS also allows you to use UFS files as virtual devices in your storage pool. This feature is aimed primarily at testing and enabling simple experimentation, not for production use. The reason is that any use of files relies on the underlying file system for consistency. If you create a ZFS pool backed by files on a UFS file system, then you are implicitly relying on UFS to guarantee correctness and synchronous semantics.
However, files can be quite useful when you are first trying out ZFS or experimenting with more complicated configurations when insufficient physical devices are present. All files must be specified as complete paths and must be at least 64 MB in size.
Replication Features of a ZFS Storage Pool
ZFS provides data redundancy, as well as self-healing properties, in mirrored and RAID-Z configurations.
Mirrored Storage Pool Configuration
A mirrored storage pool configuration requires at least two disks, preferably on separate controllers. Many disks can be used in a mirrored configuration. In addition, you can create more than one mirror in each pool. Conceptually, a basic mirrored configuration would look similar to the following:
mirror c1t0d0 c2t0d0 |
Conceptually, a more complex mirrored configuration would look similar to the following:
mirror c1t0d0 c2t0d0 c3t0d0 mirror c4t0d0 c5t0d0 c6t0d0 |
For information about creating a mirrored storage pool, see Creating a Mirrored Storage Pool.
RAID-Z Storage Pool Configuration
In addition to a mirrored storage pool configuration, ZFS provides a RAID-Z configuration with either single-, double-, or triple-parity fault tolerance. Single-parity RAID-Z (raidz or raidz1) is similar to RAID-5. Double-parity RAID-Z (raidz2) is similar to RAID-6.
For more information about RAIDZ-3 (raidz3), see the following blog:
http://blogs.sun.com/ahl/entry/triple_parity_raid_z
All traditional RAID-5-like algorithms (RAID-4, RAID-6, RDP, and EVEN-ODD, for example) might experience a problem known as the “RAID-5 write hole.” If only part of a RAID-5 stripe is written, and power is lost before all blocks have been written to disk, the parity will remain unsynchronized with the data, and therefore forever useless, (unless a subsequent full-stripe write overwrites it). In RAID-Z, ZFS uses variable-width RAID stripes so that all writes are full-stripe writes. This design is only possible because ZFS integrates file system and device management in such a way that the file system's metadata has enough information about the underlying data redundancy model to handle variable-width RAID stripes. RAID-Z is the world's first software-only solution to the RAID-5 write hole.
A RAID-Z configuration with N disks of size X with P parity disks can hold approximately (N-P)*X bytes and can withstand P device(s) failing before data integrity is compromised. You need at least two disks for a single-parity RAID-Z configuration and at least three disks for a double-parity RAID-Z configuration. For example, if you have three disks in a single-parity RAID-Z configuration, parity data occupies disk space equal to one of the three disks. Otherwise, no special hardware is required to create a RAID-Z configuration.
Conceptually, a RAID-Z configuration with three disks would look similar to the following:
raidz c1t0d0 c2t0d0 c3t0d0 |
Conceptually, a more complex RAID-Z configuration would look similar to the following:
raidz c1t0d0 c2t0d0 c3t0d0 c4t0d0 c5t0d0 c6t0d0 c7t0d0 raidz c8t0d0 c9t0d0 c10t0d0 c11t0d0 c12t0d0 c13t0d0 c14t0d0 |
If you are creating a RAID-Z configuration with many disks, consider splitting the disks into multiple groupings. For example, a RAID-Z configuration with 14 disks is better split into two 7-disk groupings. RAID-Z configurations with single-digit groupings of disks should perform better.
For information about creating a RAID-Z storage pool, see Creating a RAID-Z Storage Pool.
For more information about choosing between a mirrored configuration or a RAID-Z configuration based on performance and disk space considerations, see the following blog entry:
http://blogs.sun.com/roch/entry/when_to_and_not_to
For additional information about RAID-Z storage pool recommendations, see the ZFS best practices site:
http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide
ZFS Hybrid Storage Pool
The ZFS hybrid storage pool, available in Oracle's Sun Storage 7000 product series, is a special storage pool that combines DRAM, SSDs, and HDDs, to improve performance and increase capacity, while reducing power consumption. With this product's management interface, you can select the ZFS redundancy configuration of the storage pool and easily manage other configuration options.
For more information about this product, see the Sun Storage Unified Storage System Administration Guide.
Self-Healing Data in a Redundant Configuration
ZFS provides self-healing data in a mirrored or RAID-Z configuration.
When a bad data block is detected, not only does ZFS fetch the correct data from another redundant copy, but it also repairs the bad data by replacing it with the good copy.
Dynamic Striping in a Storage Pool
ZFS dynamically stripes data across all top-level virtual devices. The decision about where to place data is done at write time, so no fixed-width stripes are created at allocation time.
When new virtual devices are added to a pool, ZFS gradually allocates data to the new device in order to maintain performance and disk space allocation policies. Each virtual device can also be a mirror or a RAID-Z device that contains other disk devices or files. This configuration gives you flexibility in controlling the fault characteristics of your pool. For example, you could create the following configurations out of four disks:
-
Four disks using dynamic striping
-
One four-way RAID-Z configuration
-
Two two-way mirrors using dynamic striping
Although ZFS supports combining different types of virtual devices within the same pool, avoid this practice. For example, you can create a pool with a two-way mirror and a three-way RAID-Z configuration. However, your fault tolerance is as good as your worst virtual device, RAID-Z in this case. A best practice is to use top-level virtual devices of the same type with the same redundancy level in each device.
Creating and Destroying ZFS Storage Pools
The following sections describe different scenarios for creating and destroying ZFS storage pools:
Creating and destroying pools is fast and easy. However, be cautious when performing these operations. Although checks are performed to prevent using devices known to be in use in a new pool, ZFS cannot always know when a device is already in use. Destroying a pool is easier than creating one. Use zpool destroy with caution. This simple command has significant consequences.
Creating a ZFS Storage Pool
To create a storage pool, use the zpool create command. This command takes a pool name and any number of virtual devices as arguments. The pool name must satisfy the naming requirements in ZFS Component Naming Requirements.
Creating a Basic Storage Pool
The following command creates a new pool named tank that consists of the disks c1t0d0 and c1t1d0:
# zpool create tank c1t0d0 c1t1d0 |
Device names representing the whole disks are found in the /dev/dsk directory and are labeled appropriately by ZFS to contain a single, large slice. Data is dynamically striped across both disks.
Creating a Mirrored Storage Pool
To create a mirrored pool, use the mirror keyword, followed by any number of storage devices that will comprise the mirror. Multiple mirrors can be specified by repeating the mirror keyword on the command line. The following command creates a pool with two, two-way mirrors:
# zpool create tank mirror c1d0 c2d0 mirror c3d0 c4d0 |
The second mirror keyword indicates that a new top-level virtual device is being specified. Data is dynamically striped across both mirrors, with data being redundant between each disk appropriately.
For more information about recommended mirrored configurations, see the following site:
http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide
Currently, the following operations are supported in a ZFS mirrored configuration:
-
Adding another set of disks for an additional top-level virtual device (vdev) to an existing mirrored configuration. For more information, see Adding Devices to a Storage Pool.
-
Attaching additional disks to an existing mirrored configuration. Or, attaching additional disks to a non-replicated configuration to create a mirrored configuration. For more information, see Attaching and Detaching Devices in a Storage Pool.
-
Replacing a disk or disks in an existing mirrored configuration as long as the replacement disks are greater than or equal to the size of the device to be replaced. For more information, see Replacing Devices in a Storage Pool.
-
Detaching a disk in a mirrored configuration as long as the remaining devices provide adequate redundancy for the configuration. For more information, see Attaching and Detaching Devices in a Storage Pool.
-
Splitting a mirrored configuration by detaching one of the disks to create a new, identical pool. For more information, see Creating a New Pool By Splitting a Mirrored ZFS Storage Pool.
You cannot outright remove a device that is not a log or a cache device from a mirrored storage pool. An RFE is filed for this feature.
Creating a ZFS Root Pool
You can install and boot from a ZFS root file system. Review the following root pool configuration information:
-
Disks used for the root pool must have a VTOC (SMI) label, and the pool must be created with disk slices.
-
The root pool must be created as a mirrored configuration or as a single-disk configuration. You cannot add additional disks to create multiple mirrored top-level virtual devices by using the zpool add command, but you can expand a mirrored virtual device by using the zpool attach command.
-
A RAID-Z or a striped configuration is not supported.
-
The root pool cannot have a separate log device.
-
If you attempt to use an unsupported configuration for a root pool, you see messages similar to the following:
ERROR: ZFS pool <pool-name> does not support boot environments
# zpool add -f rpool log c0t6d0s0 cannot add to 'rpool': root pool can not have multiple vdevs or separate logs
For more information about installing and booting a ZFS root file system, see Chapter 5, Installing and Booting an Oracle Solaris ZFS Root File System.
Creating a RAID-Z Storage Pool
Creating a single-parity RAID-Z pool is identical to creating a mirrored pool, except that the raidz or raidz1 keyword is used instead of mirror. The following example shows how to create a pool with a single RAID-Z device that consists of five disks:
# zpool create tank raidz c1t0d0 c2t0d0 c3t0d0 c4t0d0 /dev/dsk/c5t0d0 |
This example illustrates that disks can be specified by using their shorthand device names or their full device names. Both /dev/dsk/c5t0d0 and c5t0d0 refer to the same disk.
You can create a double-parity or triple-parity RAID-Z configuration by using the raidz2 or raidz3 keyword when creating the pool. For example:
# zpool create tank raidz2 c1t0d0 c2t0d0 c3t0d0 c4t0d0 c5t0d0
# zpool status -v tank
pool: tank
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c2t0d0 ONLINE 0 0 0
c3t0d0 ONLINE 0 0 0
c4t0d0 ONLINE 0 0 0
c5t0d0 ONLINE 0 0 0
errors: No known data errors
|
# zpool create tank raidz3 c0t0d0 c1t0d0 c2t0d0 c3t0d0 c4t0d0 c5t0d0 c6t0d0 c7t0d0
# zpool status -v tank
pool: tank
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz3-0 ONLINE 0 0 0
c0t0d0 ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c2t0d0 ONLINE 0 0 0
c3t0d0 ONLINE 0 0 0
c4t0d0 ONLINE 0 0 0
c5t0d0 ONLINE 0 0 0
c6t0d0 ONLINE 0 0 0
c7t0d0 ONLINE 0 0 0
errors: No known data errors
|
Currently, the following operations are supported in a ZFS RAID-Z configuration:
-
Adding another set of disks for an additional top-level virtual device to an existing RAID-Z configuration. For more information, see Adding Devices to a Storage Pool.
-
Replacing a disk or disks in an existing RAID-Z configuration as long as the replacement disks are greater than or equal to the size of the device to be replaced. For more information, see Replacing Devices in a Storage Pool.
Currently, the following operations are not supported in a RAID-Z configuration:
-
Attaching an additional disk to an existing RAID-Z configuration.
-
Detaching a disk from a RAID-Z configuration, except when you are detaching a disk that is replaced by a spare disk.
-
You cannot outright remove a device that is not a log or a cache device from a RAID-Z configuration. An RFE is filed for this feature.
For more information about a RAID-Z configuration, see RAID-Z Storage Pool Configuration.
Creating a ZFS Storage Pool With Log Devices
By default, the ZIL is allocated from blocks within the main pool. However, better performance might be possible by using separate intent log devices, such as NVRAM or a dedicated disk. For more information about ZFS log devices, see Setting Up Separate ZFS Log Devices.
You can set up a ZFS log device when the storage pool is created or after the pool is created.
The following example shows how to create a mirrored storage pool with mirrored log devices:
# zpool create datap mirror c1t1d0 c1t2d0 mirror c1t3d0 c1t4d0 log mirror c1t5d0 c1t8d0
# zpool status datap
pool: datap
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
datap ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
c1t3d0 ONLINE 0 0 0
c1t4d0 ONLINE 0 0 0
logs
mirror-2 ONLINE 0 0 0
c1t5d0 ONLINE 0 0 0
c1t8d0 ONLINE 0 0 0
errors: No known data errors
|
For information about recovering from a log device failure, see Example 11–2.
Creating a ZFS Storage Pool With Cache Devices
You can create a storage pool with cache devices to cache storage pool data. For example:
# zpool create tank mirror c2t0d0 c2t1d0 c2t3d0 cache c2t5d0 c2t8d0
# zpool status tank
pool: tank
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c2t0d0 ONLINE 0 0 0
c2t1d0 ONLINE 0 0 0
c2t3d0 ONLINE 0 0 0
cache
c2t5d0 ONLINE 0 0 0
c2t8d0 ONLINE 0 0 0
errors: No known data errors
|
Consider the following points when determining whether to create a ZFS storage pool with cache devices:
-
Using cache devices provides the greatest performance improvement for random-read workloads of mostly static content.
-
Capacity and reads can be monitored by using the zpool iostat command.
-
Single or multiple cache devices can be added when the pool is created. They can also be added and removed after the pool is created. For more information, see Example 4–4.
-
Cache devices cannot be mirrored or be part of a RAID-Z configuration.
-
If a read error is encountered on a cache device, that read I/O is reissued to the original storage pool device, which might be part of a mirrored or a RAID-Z configuration. The content of the cache devices is considered volatile, similar to other system caches.
Displaying Storage Pool Virtual Device Information
Each storage pool contains one or more virtual devices. A virtual device is an internal representation of the storage pool that describes the layout of physical storage and the storage pool's fault characteristics. As such, a virtual device represents the disk devices or files that are used to create the storage pool. A pool can have any number of virtual devices at the top of the configuration, known as a top-level vdev.
If the top-level virtual device contains two or more physical devices, the configuration provide data redundancy as mirror or RAID-Z virtual devices. These virtual devices consist of disks, disk slices, or files. A spare is a special virtual dev that tracks available hot spares for a pool.
The following example shows how to create a pool that consists of two top-level virtual devices, each a mirror of two disks:
# zpool create tank mirror c1d0 c2d0 mirror c3d0 c4d0 |
The following example shows how to create pool that consists of one top-level virtual device of four disks:
# zpool create mypool raidz2 c1d0 c2d0 c3d0 c4d0 |
You can add another top-level virtual device to this pool by using the zpool add command. For example:
# zpool add mypool raidz2 c2d1 c3d1 c4d1 c5d1 |
Disks, disk slices, or files that are used in nonredundant pools function as top-level virtual devices. Storage pools typically contain multiple top-level virtual devices. ZFS dynamically stripes data among all of the top-level virtual devices in a pool.
Virtual devices and the physical devices that are contained in a ZFS storage pool are displayed with the zpool status command. For example:
# zpool status tank
pool: tank
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0t1d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
c0t2d0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
c0t3d0 ONLINE 0 0 0
c1t3d0 ONLINE 0 0 0
errors: No known data errors
|
Handling ZFS Storage Pool Creation Errors
Pool creation errors can occur for many reasons. Some reasons are obvious, such as when a specified device doesn't exist, while other reasons are more subtle.
Detecting In-Use Devices
Before formatting a device, ZFS first determines if the disk is in-use by ZFS or some other part of the operating system. If the disk is in use, you might see errors such as the following:
# zpool create tank c1t0d0 c1t1d0 invalid vdev specification use '-f' to override the following errors: /dev/dsk/c1t0d0s0 is currently mounted on /. Please see umount(1M). /dev/dsk/c1t0d0s1 is currently mounted on swap. Please see swap(1M). /dev/dsk/c1t1d0s0 is part of active ZFS pool zeepool. Please see zpool(1M). |
Some errors can be overridden by using the -f option, but most errors cannot. The following conditions cannot be overridden by using the -f option, and you must manually correct them:
- Mounted file system
-
The disk or one of its slices contains a file system that is currently mounted. To correct this error, use the umount command.
- File system in /etc/vfstab
-
The disk contains a file system that is listed in the /etc/vfstab file, but the file system is not currently mounted. To correct this error, remove or comment out the line in the /etc/vfstab file.
- Dedicated dump device
-
The disk is in use as the dedicated dump device for the system. To correct this error, use the dumpadm command.
- Part of a ZFS pool
-
The disk or file is part of an active ZFS storage pool. To correct this error, use the zpool destroy command to destroy the other pool, if it is no longer needed. Or, use the zpool detach command to detach the disk from the other pool. You can only detach a disk from a mirrored storage pool.
The following in-use checks serve as helpful warnings and can be overridden by using the -f option to create the pool:
- Contains a file system
-
The disk contains a known file system, though it is not mounted and doesn't appear to be in use.
- Part of volume
-
The disk is part of a Solaris Volume Manager volume.
- Live upgrade
-
The disk is in use as an alternate boot environment for Oracle Solaris Live Upgrade.
- Part of exported ZFS pool
-
The disk is part of a storage pool that has been exported or manually removed from a system. In the latter case, the pool is reported as potentially active, as the disk might or might not be a network-attached drive in use by another system. Be cautious when overriding a potentially active pool.
The following example demonstrates how the -f option is used:
# zpool create tank c1t0d0 invalid vdev specification use '-f' to override the following errors: /dev/dsk/c1t0d0s0 contains a ufs filesystem. # zpool create -f tank c1t0d0 |
Ideally, correct the errors rather than use the -f option to override them.
Mismatched Replication Levels
Creating pools with virtual devices of different replication levels is not recommended. The zpool command tries to prevent you from accidentally creating a pool with mismatched levels of redundancy. If you try to create a pool with such a configuration, you see errors similar to the following:
# zpool create tank c1t0d0 mirror c2t0d0 c3t0d0 invalid vdev specification use '-f' to override the following errors: mismatched replication level: both disk and mirror vdevs are present # zpool create tank mirror c1t0d0 c2t0d0 mirror c3t0d0 c4t0d0 c5t0d0 invalid vdev specification use '-f' to override the following errors: mismatched replication level: 2-way mirror and 3-way mirror vdevs are present |
You can override these errors with the -f option, but you should avoid this practice. The command also warns you about creating a mirrored or RAID-Z pool using devices of different sizes. Although this configuration is allowed, mismatched levels of redundancy result in unused disk space on the larger device. The -f option is required to override the warning.
Doing a Dry Run of Storage Pool Creation
Attempts to create a pool can fail unexpectedly in different ways, and formatting disks is a potentially harmful action. For these reasons, the zpool create command has an additional option, -n, which simulates creating the pool without actually writing to the device. This dry run option performs the device in-use checking and replication-level validation, and reports any errors in the process. If no errors are found, you see output similar to the following:
# zpool create -n tank mirror c1t0d0 c1t1d0
would create 'tank' with the following layout:
tank
mirror
c1t0d0
c1t1d0
|
Some errors cannot be detected without actually creating the pool. The most common example is specifying the same device twice in the same configuration. This error cannot be reliably detected without actually writing the data, so the zpool create -n command can report success and yet fail to create the pool when the command is run without this option.
Default Mount Point for Storage Pools
When a pool is created, the default mount point for the top-level dataset is /pool-name. This directory must either not exist or be empty. If the directory does not exist, it is automatically created. If the directory is empty, the root dataset is mounted on top of the existing directory. To create a pool with a different default mount point, use the -m option of the zpool create command. For example:
# zpool create home c1t0d0 default mountpoint '/home' exists and is not empty use '-m' option to provide a different default # zpool create -m /export/zfs home c1t0d0 |
This command creates the new pool home and the home dataset with a mount point of /export/zfs.
For more information about mount points, see Managing ZFS Mount Points.
Destroying ZFS Storage Pools
Pools are destroyed by using the zpool destroy command. This command destroys the pool even if it contains mounted datasets.
# zpool destroy tank |
Caution – Be very careful when you destroy a pool. Ensure that you are destroying the right pool and you always have copies of your data. If you accidentally destroy the wrong pool, you can attempt to recover the pool. For more information, see Recovering Destroyed ZFS Storage Pools.
Destroying a Pool With Faulted Devices
The act of destroying a pool requires data to be written to disk to indicate that the pool is no longer valid. This state information prevents the devices from showing up as a potential pool when you perform an import. If one or more devices are unavailable, the pool can still be destroyed. However, the necessary state information won't be written to these unavailable devices.
These devices, when suitably repaired, are reported as potentially active when you create a new pool. They appear as valid devices when you search for pools to import. If a pool has enough faulted devices such that the pool itself is faulted (meaning that a top-level virtual device is faulted), then the command prints a warning and cannot complete without the -f option. This option is necessary because the pool cannot be opened, so whether data is stored there is unknown. For example:
# zpool destroy tank cannot destroy 'tank': pool is faulted use '-f' to force destruction anyway # zpool destroy -f tank |
For more information about pool and device health, see Determining the Health Status of ZFS Storage Pools.
For more information about importing pools, see Importing ZFS Storage Pools.
Managing Devices in ZFS Storage Pools
Most of the basic information regarding devices is covered in Components of a ZFS Storage Pool. After a pool has been created, you can perform several tasks to manage the physical devices within the pool.
Adding Devices to a Storage Pool
You can dynamically add disk space to a pool by adding a new top-level virtual device. This disk space is immediately available to all datasets in the pool. To add a new virtual device to a pool, use the zpool add command. For example:
# zpool add zeepool mirror c2t1d0 c2t2d0 |
The format for specifying the virtual devices is the same as for the zpool create command. Devices are checked to determine if they are in use, and the command cannot change the level of redundancy without the -f option. The command also supports the -n option so that you can perform a dry run. For example:
# zpool add -n zeepool mirror c3t1d0 c3t2d0
would update 'zeepool' to the following configuration:
zeepool
mirror
c1t0d0
c1t1d0
mirror
c2t1d0
c2t2d0
mirror
c3t1d0
c3t2d0
|
This command syntax would add mirrored devices c3t1d0 and c3t2d0 to the zeepool pool's existing configuration.
For more information about how virtual device validation is done, see Detecting In-Use Devices.
Example 4–1 Adding Disks to a Mirrored ZFS Configuration
In the following example, another mirror is added to an existing mirrored ZFS configuration on Oracle's Sun Fire x4500 system.
# zpool status tank
pool: tank
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0t1d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
c0t2d0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
errors: No known data errors
# zpool add tank mirror c0t3d0 c1t3d0
# zpool status tank
pool: tank
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0t1d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
c0t2d0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
c0t3d0 ONLINE 0 0 0
c1t3d0 ONLINE 0 0 0
errors: No known data errors
|
Example 4–2 Adding Disks to a RAID-Z Configuration
Additional disks can be added similarly to a RAID-Z configuration. The following example shows how to convert a storage pool with one RAID-Z device that contains three disks to a storage pool with two RAID-Z devices that contains three disks each.
# zpool status rzpool
pool: rzpool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
rzpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
c1t3d0 ONLINE 0 0 0
c1t4d0 ONLINE 0 0 0
errors: No known data errors
# zpool add rzpool raidz c2t2d0 c2t3d0 c2t4d0
# zpool status rzpool
pool: rzpool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
rzpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
c1t3d0 ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
c2t2d0 ONLINE 0 0 0
c2t3d0 ONLINE 0 0 0
c2t4d0 ONLINE 0 0 0
errors: No known data errors
|
Example 4–3 Adding and Removing a Mirrored Log Device
The following example shows how to add a mirrored log device to mirrored storage pool.For more information about using log devices in your storage pool, see Setting Up Separate ZFS Log Devices.
# zpool status newpool
pool: newpool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
newpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0t4d0 ONLINE 0 0 0
c0t5d0 ONLINE 0 0 0
errors: No known data errors
# zpool add newpool log mirror c0t6d0 c0t7d0
# zpool status newpool
pool: newpool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
newpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0t4d0 ONLINE 0 0 0
c0t5d0 ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
c0t6d0 ONLINE 0 0 0
c0t7d0 ONLINE 0 0 0
errors: No known data errors
|
You can attach a log device to an existing log device to create a mirrored log device. This operation is identical to attaching a device in a unmirrored storage pool.
Log devices can be removed by using the zpool remove command. The mirrored log device in the previous example can be removed by specifying the mirror-1 argument. For example:
# zpool remove newpool mirror-1
# zpool status newpool
pool: newpool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
newpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0t4d0 ONLINE 0 0 0
c0t5d0 ONLINE 0 0 0
errors: No known data errors
|
If your pool configuration only contains one log device, you would remove the log device by specifying the device name. For example:
# zpool status pool
pool: pool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
pool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
c0t8d0 ONLINE 0 0 0
c0t9d0 ONLINE 0 0 0
logs
c0t10d0 ONLINE 0 0 0
errors: No known data errors
# zpool remove pool c0t10d0
|
Example 4–4 Adding and Removing Cache Devices
You can add to your ZFS storage pool and remove them if they are no longer required..
Use the zpool add command to add cache devices. For example:
# zpool add tank cache c2t5d0 c2t8d0
# zpool status tank
pool: tank
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c2t0d0 ONLINE 0 0 0
c2t1d0 ONLINE 0 0 0
c2t3d0 ONLINE 0 0 0
cache
c2t5d0 ONLINE 0 0 0
c2t8d0 ONLINE 0 0 0
errors: No known data errors
|
Cache devices cannot be mirrored or be part of a RAID-Z configuration.
Use the zpool remove command to remove cache devices. For example:
# zpool remove tank c2t5d0 c2t8d0
# zpool status tank
pool: tank
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c2t0d0 ONLINE 0 0 0
c2t1d0 ONLINE 0 0 0
c2t3d0 ONLINE 0 0 0
errors: No known data errors
|
Currently, the zpool remove command only supports removing hot spares, log devices, and cache devices. Devices that are part of the main mirrored pool configuration can be removed by using the zpool detach command. Nonredundant and RAID-Z devices cannot be removed from a pool.
For more information about using cache devices in a ZFS storage pool, see Creating a ZFS Storage Pool With Cache Devices.
Attaching and Detaching Devices in a Storage Pool
In addition to the zpool add command, you can use the zpool attach command to add a new device to an existing mirrored or nonmirrored device.
If you are attaching a disk to create a mirrored root pool, see How to Create a Mirrored Root Pool (Post Installation).
If you are replacing a disk in a ZFS root pool, see How to Replace a Disk in the ZFS Root Pool.
Example 4–5 Converting a Two-Way Mirrored Storage Pool to a Three-way Mirrored Storage Pool
In this example, zeepool is an existing two-way mirror that is converted to a three-way mirror by attaching c2t1d0, the new device, to the existing device, c1t1d0.
# zpool status zeepool
pool: zeepool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
zeepool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0t1d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
errors: No known data errors
# zpool attach zeepool c1t1d0 c2t1d0
# zpool status zeepool
pool: zeepool
state: ONLINE
scrub: resilver completed after 0h0m with 0 errors on Fri Jan 8 12:59:20 2010
config:
NAME STATE READ WRITE CKSUM
zeepool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0t1d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
c2t1d0 ONLINE 0 0 0 592K resilvered
errors: No known data errors
|
If the existing device is part of a three-way mirror, attaching the new device creates a four-way mirror, and so on. Whatever the case, the new device begins to resilver immediately.
Example 4–6 Converting a Nonredundant ZFS Storage Pool to a Mirrored ZFS Storage Pool
In addition, you can convert a nonredundant storage pool to a redundant storage pool by using the zpool attach command. For example:
# zpool create tank c0t1d0
# zpool status tank
pool: tank
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
c0t1d0 ONLINE 0 0 0
errors: No known data errors
# zpool attach tank c0t1d0 c1t1d0
# zpool status tank
pool: tank
state: ONLINE
scrub: resilver completed after 0h0m with 0 errors on Fri Jan 8 14:28:23 2010
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0t1d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0 73.5K resilvered
errors: No known data errors
|
You can use the zpool detach command to detach a device from a mirrored storage pool. For example:
# zpool detach zeepool c2t1d0 |
However, this operation fails if no other valid replicas of the data exist. For example:
# zpool detach newpool c1t2d0 cannot detach c1t2d0: only applicable to mirror and replacing vdevs |
Creating a New Pool By Splitting a Mirrored ZFS Storage Pool
A mirrored ZFS storage pool can be quickly cloned as a backup pool by using the zpool split command.
Currently, this feature cannot be used to split a mirrored root pool.
You can use the zpool split command to detach disks from a mirrored ZFS storage pool to create a new pool with one of the detached disks. The new pool will have identical contents to the original mirrored ZFS storage pool.
By default, a zpool split operation on a mirrored pool detaches the last disk for the newly created pool. After the split operation, import the new pool. For example:
# zpool status tank
pool: tank
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
errors: No known data errors
# zpool split tank tank2
# zpool import tank2
# zpool status tank tank2
pool: tank
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
errors: No known data errors
pool: tank2
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank2 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
errors: No known data errors
|
You can identify which disk should be used for the newly created pool by specifying it with the zpool split command. For example:
# zpool split tank tank2 c1t0d0 |
Before the actual split operation occurs, data in memory is flushed to the mirrored disks. After the data is flushed, the disk is detached from the pool and given a new pool GUID. A new pool GUID is generated so that the pool can be imported on the same system on which it was split.
If the pool to be split has non-default dataset mount points and the new pool is created on the same system, then you will need to use the zpool split -R option to identify an alternate root directory for the new pool so that any existing mount points do not conflict. For example:
# zpool split -R /tank2 tank tank2 |
If you don't use the zpool split -R option and you can see that mount points conflict when you attempt to import the new pool, import the new pool with the -R option. If the new pool is created on a different system, then specifying an alternate root directory should not be necessary unless mount point conflicts occur.
Review the following considerations before using the zpool split feature:
-
This feature is not available for a RAIDZ configuration or a non-redundant pool of multiple disks.
-
Data and application operations should be quiesced before attempting a zpool split operation.
-
Having disks that honor, rather than ignore, the disk's flush write cache command is important.
-
A pool cannot be split if resilvering is in process.
-
Splitting a mirrored pool is optimal when composed of two to three disks, where the last disk in the original pool is used for the newly created pool. Then, you can use the zpool attach command to recreate your original mirrored storage pool or convert your newly created pool into a mirrored storage pool. No way currently exists to create a new mirrored pool from an existing mirrored pool by using this feature.
-
If the existing pool is a three-way mirror, then the new pool will contain one disk after the split operation. If the existing pool is a two-way mirror of two disks, then the outcome is two non-redundant pools of two disks. You will need to attach two additional disks to convert the non-redundant pools to mirrored pools.
-
A good way to keep your data redundant during a split operation is to split a mirrored storage pool that is composed of three disks so that the original pool is comprised of two mirrored disks after the split operation.
Example 4–7 Splitting a Mirrored ZFS Pool
In the following example, a mirrored storage pool called trinity, with three disks, c1t0d0, c1t2d0 and c1t3d0 is split. The two resulting pools are the mirrored pool trinity, with disks c1t0d0 and c1t2d0, and the new pool, neo, with disk c1t3d0. Each pool has identical content.
# zpool status trinity
pool: trinity
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
trinity ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
c1t3d0 ONLINE 0 0 0
errors: No known data errors
# zpool split trinity neo
# zpool import neo
# zpool status trinity neo
pool: neo
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
neo ONLINE 0 0 0
c1t3d0 ONLINE 0 0 0
errors: No known data errors
pool: trinity
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
trinity ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
errors: No known data errors
|
Onlining and Offlining Devices in a Storage Pool
ZFS allows individual devices to be taken offline or brought online. When hardware is unreliable or not functioning properly, ZFS continues to read data from or write data to the device, assuming the condition is only temporary. If the condition is not temporary, you can instruct ZFS to ignore the device by taking it offline. ZFS does not send any requests to an offline device.
Note –
Devices do not need to be taken offline in order to replace them.
You can use the zpool offline command when you need to temporarily disconnect storage. For example, if you need to physically disconnect an array from one set of Fibre Channel switches and connect the array to a different set, you can take offline the LUNs from the array that is used in the ZFS storage pools. After the array is reconnected and operational on the new set of switches, you can then bring the same LUNs online. Data that had been added to the storage pools while the LUNs were offline would resilver to the LUNs after they are brought back online.
This scenario is possible assuming that the systems in question can detect the storage after it is attached to the new switches, possibly through different controllers than before, and your pools are set up as RAID-Z or mirrored configurations.
Taking a Device Offline
You can take a device offline by using the zpool offline command. The device can be specified by path or by short name, if the device is a disk. For example:
# zpool offline tank c1t0d0 bringing device c1t0d0 offline |
Consider the following points when taking a device offline:
-
You cannot take a pool offline to the point where it becomes faulted. For example, you cannot take offline two devices in a raidz1 configuration, nor can you take offline a top-level virtual device.
# zpool offline tank c1t0d0 cannot offline c1t0d0: no valid replicas
-
By default, the OFFLINE state is persistent. The device remains offline when the system is rebooted.
To temporarily take a device offline, use the zpool offline -t option. For example:
# zpool offline -t tank c1t0d0 bringing device 'c1t0d0' offline
When the system is rebooted, this device is automatically returned to the ONLINE state.
-
When a device is taken offline, it is not detached from the storage pool. If you attempt to use the offline device in another pool, even after the original pool is destroyed, you see a message similar to the following:
device is part of exported or potentially active ZFS pool. Please see zpool(1M)
If you want to use the offline device in another storage pool after destroying the original storage pool, first bring the device online, then destroy the original storage pool.
Another way to use a device from another storage pool, while keeping the original storage pool, is to replace the existing device in the original storage pool with another comparable device. For information about replacing devices, see Replacing Devices in a Storage Pool.
Offline devices are in the OFFLINE state when you query pool status. For information about querying pool status, see Querying ZFS Storage Pool Status.
For more information on device health, see Determining the Health Status of ZFS Storage Pools.
Bringing a Device Online
After a device is taken offline, it can be brought online again by using the zpool online command. For example:
# zpool online tank c1t0d0 bringing device c1t0d0 online |
When a device is brought online, any data that has been written to the pool is resynchronized with the newly available device. Note that you cannot use bring a device online to replace a disk. If you take a device offline, replace the device, and try to bring it online, it remains in the faulted state.
If you attempt to bring online a faulted device, a message similar to the following is displayed:
# zpool online tank c1t0d0 warning: device 'c1t0d0' onlined, but remains in faulted state use 'zpool replace' to replace devices that are no longer present |
You might also see the faulted disk message displayed on the console or written to the /var/adm/messages file. For example:
SUNW-MSG-ID: ZFS-8000-D3, TYPE: Fault, VER: 1, SEVERITY: Major EVENT-TIME: Wed Jun 30 14:53:39 MDT 2010 PLATFORM: SUNW,Sun-Fire-880, CSN: -, HOSTNAME: neo SOURCE: zfs-diagnosis, REV: 1.0 EVENT-ID: 504a1188-b270-4ab0-af4e-8a77680576b8 DESC: A ZFS device failed. Refer to http://sun.com/msg/ZFS-8000-D3 for more information. AUTO-RESPONSE: No automated response will occur. IMPACT: Fault tolerance of the pool may be compromised. REC-ACTION: Run 'zpool status -x' and replace the bad device. |
For more information about replacing a faulted device, see Resolving a Missing Device.
You can use the zpool online -e command to expand the pool's size when a smaller disk is replaced by a larger disk. By default, pool size is not expanded to its full size unless the autoexpand pool property is enabled. You can expand the pool size automatically when a smaller disk is replaced by a larger disk by using the zpool online -ecommand even if the disk is already online or if the disk is currently offline. For example:
# zpool online -e tank c1t13d0 |
Clearing Storage Pool Device Errors
If a device is taken offline due to a failure that causes errors to be listed in the zpool status output, you can clear the error counts with the zpool clear command.
If specified with no arguments, this command clears all device errors within the pool. For example:
# zpool clear tank |
If one or more devices are specified, this command only clear errors associated with the specified devices. For example:
# zpool clear tank c1t0d0 |
For more information about clearing zpool errors, see Clearing Transient Errors.
Replacing Devices in a Storage Pool
You can replace a device in a storage pool by using the zpool replace command.
If you are physically replacing a device with another device in the same location in a redundant pool, then you might only need to identify the replaced device. ZFS recognizes that the device is a different disk in the same location on some hardware. For example, to replace a failed disk (c1t1d0) by removing the disk and replacing it in the same location, use the following syntax:
# zpool replace tank c1t1d0 |
If you are replacing a device in a storage pool with a disk in a different physical location, you will need to specify both devices. For example:
# zpool replace tank c1t1d0 c1t2d0 |
If you are replacing a disk in the ZFS root pool, see How to Replace a Disk in the ZFS Root Pool.
The following are the basic steps for replacing a disk:
-
Offline the disk, if necessary, with the zpool offline command.
-
Remove the disk to be replaced.
-
Insert the replacement disk.
-
Run the zpool replace command. For example:
# zpool replace tank c1t1d0
-
Bring the disk online with the zpool online command.
On some systems, such as the Sun Fire x4500, you must unconfigure a disk before you can take it offline. If you are replacing a disk in the same slot position on this system, then you can just run the zpool replace command as described in the first example in this section.
For an example of replacing a disk on a Sun Fire X4500 system, see Example 11–1.
Consider the following when replacing devices in a ZFS storage pool:
-
If you set the autoreplace pool property to on, then any new device found in the same physical location as a device that previously belonged to the pool is automatically formatted and replaced. You are not required to use the zpool replace command when this property is enabled. This feature might not be available on all hardware types.
-
The size of the replacement device must be equal to or larger than the smallest disk in a mirrored or RAID-Z configuration.
-
When a replacement device that is greater in size than the device it is replacing is added to a pool, is not automatically expanded to its full size. The autoexpand pool property value determines whether a pool is expanded to its full size when the disk is added to the pool. By default, the autoexpand property is disabled. You can enable this property to expand a pool's size before or after the larger disk is added to the pool.
In the following example, two 16-GB disks in a mirrored pool are replaced with two 72-GB disks. The autoexpand property is enabled after the disk replacements to expand the full disk sizes.
# zpool create pool mirror c1t16d0 c1t17d0 # zpool status pool: pool state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM pool ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t16d0 ONLINE 0 0 0 c1t17d0 ONLINE 0 0 0 zpool list pool NAME SIZE ALLOC FREE CAP HEALTH ALTROOT pool 16.8G 76.5K 16.7G 0% ONLINE - # zpool replace pool c1t16d0 c1t1d0 # zpool replace pool c1t17d0 c1t2d0 # zpool list pool NAME SIZE ALLOC FREE CAP HEALTH ALTROOT pool 16.8G 88.5K 16.7G 0% ONLINE - # zpool set autoexpand=on pool # zpool list pool NAME SIZE ALLOC FREE CAP HEALTH ALTROOT pool 68.2G 117K 68.2G 0% ONLINE - -
Replacing many disks in a large pool is time-consuming due to resilvering the data onto the new disks. In addition, you might consider running the zpool scrub command between disk replacements to ensure that the replacement devices are operational and that the data is written correctly.
-
If a failed disk has been replaced automatically with a hot spare, then you might need to detach the spare after the failed disk is replaced. For information about detaching a hot spare, see Activating and Deactivating Hot Spares in Your Storage Pool.
For more information about replacing devices, see Resolving a Missing Device and Replacing or Repairing a Damaged Device.
Designating Hot Spares in Your Storage Pool
The hot spares feature enables you to identify disks that could be used to replace a failed or faulted device in one or more storage pools. Designating a device as a hot spare means that the device is not an active device in the pool, but if an active device in the pool fails, the hot spare automatically replaces the failed device.
Devices can be designated as hot spares in the following ways:
-
When the pool is created with the zpool create command.
-
After the pool is created with the zpool add command.
The following example shows how to designate devices as hot spares when the pool is created:
# zpool create trinity mirror c1t1d0 c2t1d0 spare c1t2d0 c2t2d0
# zpool status trinity
pool: trinity
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
trinity ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
c2t1d0 ONLINE 0 0 0
spares
c1t2d0 AVAIL
c2t2d0 AVAIL
errors: No known data errors
|
The following example shows how to designate hot spares by adding them to a pool after the pool is created:
# zpool add neo spare c5t3d0 c6t3d0
# zpool status neo
pool: neo
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
neo ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c3t3d0 ONLINE 0 0 0
c4t3d0 ONLINE 0 0 0
spares
c5t3d0 AVAIL
c6t3d0 AVAIL
errors: No known data errors
|
Hot spares can be removed from a storage pool by using the zpool remove command. For example:
# zpool remove zeepool c2t3d0
# zpool status zeepool
pool: zeepool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
zeepool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
c2t1d0 ONLINE 0 0 0
spares
c1t3d0 AVAIL
errors: No known data errors
|
A hot spare cannot be removed if it is currently used by a storage pool.
Consider the following when using ZFS hot spares:
-
Currently, the zpool remove command can only be used to remove hot spares, cache devices, and log devices.
-
To add a disk as a hot spare, the hot spare must be equal to or larger than the size of the largest disk in the pool. Adding a smaller disk as a spare to a pool is allowed. However, when the smaller spare disk is activated, either automatically or with the zpool replace command, the operation fails with an error similar to the following:
cannot replace disk3 with disk4: device is too small
Activating and Deactivating Hot Spares in Your Storage Pool
Hot spares are activated in the following ways:
-
Manual replacement – You replace a failed device in a storage pool with a hot spare by using the zpool replace command.
-
Automatic replacement – When a fault is detected, an FMA agent examines the pool to determine if it has any available hot spares. If so, it replaces the faulted device with an available spare.
If a hot spare that is currently in use fails, the FMA agent detaches the spare and thereby cancels the replacement. The agent then attempts to replace the device with another hot spare, if one is available. This feature is currently limited by the fact that the ZFS diagnostic engine only generates faults when a device disappears from the system.
If you physically replace a failed device with an active spare, you can reactivate the original device by using the zpool detach command to detach the spare. If you set the autoreplace pool property to on, the spare is automatically detached and returned to the spare pool when the new device is inserted and the online operation completes.
You can manually replace a device with a hot spare by using the zpool replace command. See Example 4–8.
A faulted device is automatically replaced if a hot spare is available. For example:
# zpool status -x
pool: zeepool
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-2Q
scrub: resilver completed after 0h0m with 0 errors on Mon Jan 11 10:20:35 2010
config:
NAME STATE READ WRITE CKSUM
zeepool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
c1t2d0 ONLINE 0 0 0
spare-1 DEGRADED 0 0 0
c2t1d0 UNAVAIL 0 0 0 cannot open
c2t3d0 ONLINE 0 0 0 88.5K resilvered
spares
c2t3d0 INUSE currently in use
errors: No known data errors
|
Currently, you can deactivate a hot spare in the following ways:
-
By removing the hot spare from the storage pool.
-
By detaching a hot spare after a failed disk is physically replaced. See Example 4–9.
-
By temporarily or permanently swapping in the hot spare. See Example 4–10.
Example 4–8 Manually Replacing a Disk With a Hot Spare
In this example, the zpool replace command is used to replace disk c2t1d0 with the hot spare c2t3d0.
# zpool replace zeepool c2t1d0 c2t3d0
# zpool status zeepool
pool: zeepool
state: ONLINE
scrub: resilver completed after 0h0m with 0 errors on Wed Jan 20 10:00:50 2010
config:
NAME STATE READ WRITE CKSUM
zeepool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
spare-1 ONLINE 0 0 0
c2t1d0 ONLINE 0 0 0
c2t3d0 ONLINE 0 0 0 90K resilvered
spares
c2t3d0 INUSE currently in use
errors: No known data errors
|
Then, detach the disk c2t1d0.
# zpool detach zeepool c2t1d0
# zpool status zeepool
pool: zeepool
state: ONLINE
scrub: resilver completed after 0h0m with 0 errors on Wed Jan 20 10:00:50 2010
config:
NAME STATE READ WRITE CKSUM
zeepool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
c2t3d0 ONLINE 0 0 0 90K resilvered
errors: No known data errors
|
Example 4–9 Detaching a Hot Spare After the Failed Disk Is Replaced
In this example, the failed disk (c2t1d0) is physical replaced and ZFS is notified by using the zpool replace command.
# zpool replace zeepool c2t1d0
# zpool status zeepool
pool: zeepool
state: ONLINE
scrub: resilver completed after 0h0m with 0 errors on Wed Jan 20 10:08:44 2010
config:
NAME STATE READ WRITE CKSUM
zeepool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
spare-1 ONLINE 0 0 0
c2t3d0 ONLINE 0 0 0 90K resilvered
c2t1d0 ONLINE 0 0 0
spares
c2t3d0 INUSE currently in use
errors: No known data errors
|
Then, you can use the zpool detach command to return the hot spare back to the spare pool. For example:
# zpool detach zeepool c2t3d0
# zpool status zeepool
pool: zeepool
state: ONLINE
scrub: resilver completed with 0 errors on Wed Jan 20 10:08:44 2010
config:
NAME STATE READ WRITE CKSUM
zeepool ONLINE 0 0 0
mirror ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
c2t1d0 ONLINE 0 0 0
spares
c2t3d0 AVAIL
errors: No known data errors
|
Example 4–10 Detaching a Failed Disk and Using the Hot Spare
If you want to replace a failed disk by temporarily or permanently swap in the hot spare that is currently replacing it, then detach the original (failed) disk. If the failed disk is eventually replaced, then you can add it back to the storage pool as a spare. For example:
# zpool status zeepool
pool: zeepool
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-2Q
scrub: resilver in progress for 0h0m, 70.47% done, 0h0m to go
config:
NAME STATE READ WRITE CKSUM
zeepool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
c1t2d0 ONLINE 0 0 0
spare-1 DEGRADED 0 0 0
c2t1d0 UNAVAIL 0 0 0 cannot open
c2t3d0 ONLINE 0 0 0 70.5M resilvered
spares
c2t3d0 INUSE currently in use
errors: No known data errors
# zpool detach zeepool c2t1d0
# zpool status zeepool
pool: zeepool
state: ONLINE
scrub: resilver completed after 0h0m with 0 errors on Wed Jan 20 13:46:46 2010
config:
NAME STATE READ WRITE CKSUM
zeepool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
c2t3d0 ONLINE 0 0 0 70.5M resilvered
errors: No known data errors
(Original failed disk c2t1d0 is physically replaced)
# zpool add zeepool spare c2t1d0
# zpool status zeepool
pool: zeepool
state: ONLINE
scrub: resilver completed after 0h0m with 0 errors on Wed Jan 20 13:48:46 2010
config:
NAME STATE READ WRITE CKSUM
zeepool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
c2t3d0 ONLINE 0 0 0 70.5M resilvered
spares
c2t1d0 AVAIL
errors: No known data errors
|
Managing ZFS Storage Pool Properties
You can use the zpool get command to display pool property information. For example:
# zpool get all mpool NAME PROPERTY VALUE SOURCE pool size 68G - pool capacity 0% - pool altroot - default pool health ONLINE - pool guid 601891032394735745 default pool version 22 default pool bootfs - default pool delegation on default pool autoreplace off default pool cachefile - default pool failmode wait default pool listsnapshots on default pool autoexpand off default pool free 68.0G - pool allocated 76.5K - |
Storage pool properties can be set with the zpool set command. For example:
# zpool set autoreplace=on mpool # zpool get autoreplace mpool NAME PROPERTY VALUE SOURCE mpool autoreplace on default |
|
Property Name |
Type |
Default Value |
Description |
|---|---|---|---|
|
allocated |
String |
N/A |
Read-only value that identifies the amount of storage space within the pool that has been physically allocated. |
|
altroot |
String |
off |
Identifies an alternate root directory. If set, this directory is prepended to any mount points within the pool. This property can be used when you are examining an unknown pool, if the mount points cannot be trusted, or in an alternate boot environment, where the typical paths are not valid. |
|
autoreplace |
Boolean |
off |
Controls automatic device replacement. If set to off, device replacement must be initiated by using the zpool replace command. If set to on, any new device found in the same physical location as a device that previously belonged to the pool is automatically formatted and replaced. The property abbreviation is replace. |
|
bootfs |
Boolean |
N/A |
Identifies the default bootable dataset for the root pool. This property is typically set by the installation and upgrade programs. |
|
cachefile |
String |
N/A |
Controls where pool configuration information is cached. All pools in the cache are automatically imported when the system boots. However, installation and clustering environments might require this information to be cached in a different location so that pools are not automatically imported. You can set this property to cache pool configuration information in a different location. This information can be imported later by using the zpool import -c command. For most ZFS configurations, this property is not used. |
|
capacity |
Number |
N/A |
Read-only value that identifies the percentage of pool space used. The property abbreviation is cap. |
|
delegation |
Boolean |
on |
Controls whether a nonprivileged user can be granted access permissions that are defined for a dataset. For more information, see Chapter 9, Oracle Solaris ZFS Delegated Administration. |
|
failmode |
String |
wait |
Controls the system behavior if a catastrophic pool failure occurs. This condition is typically a result of a loss of connectivity to the underlying storage device or devices or a failure of all devices within the pool. The behavior of such an event is determined by one of the following values:
|
|
free |
String |
N/A |
Read-only value that identifies the number of blocks within the pool that are not allocated. |
|
guid |
String |
N/A |
Read-only property that identifies the unique identifier for the pool. |
|
health |
String |
N/A |
Read-only property that identifies the current health of the pool, as either ONLINE, DEGRADED, FAULTED, OFFLINE, REMOVED, or UNAVAIL. |
|
listsnapshots |
String |
on |
Controls whether snapshot information that is associated with this pool is displayed with the zfs list command. If this property is disabled, snapshot information can be displayed with the zfs list -t snapshot command. |
|
size |
Number |
N/A |
Read-only property that identifies the total size of the storage pool. |
|
version |
Number |
N/A |
Identifies the current on-disk version of the pool. The preferred method of updating pools is with the zpool upgrade command, although this property can be used when a specific version is needed for backwards compatibility. This property can be set to any number between 1 and the current version reported by the zpool upgrade -v command. |
Querying ZFS Storage Pool Status
The zpool list command provides several ways to request information regarding pool status. The information available generally falls into three categories: basic usage information, I/O statistics, and health status. All three types of storage pool information are covered in this section.
Displaying Information About ZFS Storage Pools
You can use the zpool list command to display basic information about pools.
Listing Information About All Storage Pools or a Specific Pool
With no arguments, the zpool listcommand displays the following information for all pools on the system:
# zpool list NAME SIZE ALLOC FREE CAP HEALTH ALTROOT tank 80.0G 22.3G 47.7G 28% ONLINE - dozer 1.2T 384G 816G 32% ONLINE - |
This command output displays the following information:
- NAME
-
The name of the pool.
- SIZE
-
The total size of the pool, equal to the sum of the sizes of all top-level virtual devices.
- ALLOC
-
The amount of physical space allocated to all datasets and internal metadata. Note that this amount differs from the amount of disk space as reported at the file system level.
For more information about determining available file system space, see ZFS Disk Space Accounting.
- FREE
-
The amount of unallocated space in the pool.
- CAP (CAPACITY)
-
The amount of disk space used, expressed as a percentage of the total disk space.
- HEALTH
-
The current health status of the pool.
For more information about pool health, see Determining the Health Status of ZFS Storage Pools.
- ALTROOT
-
The alternate root of the pool, if one exists.
For more information about alternate root pools, see Using ZFS Alternate Root Pools.
You can also gather statistics for a specific pool by specifying the pool name. For example:
# zpool list tank NAME SIZE ALLOC FREE CAP HEALTH ALTROOT tank 80.0G 22.3G 47.7G 28% ONLINE - |
Listing Specific Storage Pool Statistics
Specific statistics can be requested by using the -o option. This option provides custom reports or a quick way to list pertinent information. For example, to list only the name and size of each pool, you use the following syntax:
# zpool list -o name,size NAME SIZE tank 80.0G dozer 1.2T |
The column names correspond to the properties that are listed in Listing Information About All Storage Pools or a Specific Pool.
Scripting ZFS Storage Pool Output
The default output for the zpool list command is designed for readability and is not easy to use as part of a shell script. To aid programmatic uses of the command, the -H option can be used to suppress the column headings and separate fields by tabs, rather than by spaces. For example, to request a list of all pool names on the system, you would use the following syntax:
# zpool list -Ho name tank dozer |
Here is another example:
# zpool list -H -o name,size tank 80.0G dozer 1.2T |
Displaying ZFS Storage Pool Command History
ZFS automatically logs successful zfs and zpool commands that modify pool state information. This information can be displayed by using the zpool history command.
For example, the following syntax displays the command output for the root pool:
# zpool history History for 'rpool': 2010-05-11.10:18:54 zpool create -f -o failmode=continue -R /a -m legacy -o cachefile=/tmp/root/etc/zfs/zpool.cache rpool mirror c1t0d0s0 c1t1d0s0 2010-05-11.10:18:55 zfs set canmount=noauto rpool 2010-05-11.10:18:55 zfs set mountpoint=/rpool rpool 2010-05-11.10:18:56 zfs create -o mountpoint=legacy rpool/ROOT 2010-05-11.10:18:57 zfs create -b 8192 -V 2048m rpool/swap 2010-05-11.10:18:58 zfs create -b 131072 -V 1536m rpool/dump 2010-05-11.10:19:01 zfs create -o canmount=noauto rpool/ROOT/zfsBE 2010-05-11.10:19:02 zpool set bootfs=rpool/ROOT/zfsBE rpool 2010-05-11.10:19:02 zfs set mountpoint=/ rpool/ROOT/zfsBE 2010-05-11.10:19:03 zfs set canmount=on rpool 2010-05-11.10:19:04 zfs create -o mountpoint=/export rpool/export 2010-05-11.10:19:05 zfs create rpool/export/home 2010-05-11.11:11:10 zpool set bootfs=rpool rpool 2010-05-11.11:11:10 zpool set bootfs=rpool/ROOT/zfsBE rpool |
You can use similar output on your system to identify the actual ZFS commands that were executed to troubleshoot an error condition.
The features of the history log are as follows:
-
The log cannot be disabled.
-
The log is saved persistently on disk, which means that the log is saved across system reboots.
-
The log is implemented as a ring buffer. The minimum size is 128 KB. The maximum size is 32 MB.
-
For smaller pools, the maximum size is capped at 1 percent of the pool size, where the size is determined at pool creation time.
-
The log requires no administration, which means that tuning the size of the log or changing the location of the log is unnecessary.
To identify the command history of a specific storage pool, use syntax similar to the following:
# zpool history tank History for 'tank': 2010-05-13.14:13:15 zpool create tank mirror c1t2d0 c1t3d0 2010-05-13.14:21:19 zfs create tank/snaps 2010-05-14.08:10:29 zfs create tank/ws01 2010-05-14.08:10:54 zfs snapshot tank/ws01@now 2010-05-14.08:11:05 zfs clone tank/ws01@now tank/ws01bugfix |
Use the -l option to display a long format that includes the user name, the host name, and the zone in which the operation was performed. For example:
# zpool history -l tank History for 'tank': 2010-05-13.14:13:15 zpool create tank mirror c1t2d0 c1t3d0 [user root on neo] 2010-05-13.14:21:19 zfs create tank/snaps [user root on neo] 2010-05-14.08:10:29 zfs create tank/ws01 [user root on neo] 2010-05-14.08:10:54 zfs snapshot tank/ws01@now [user root on neo] 2010-05-14.08:11:05 zfs clone tank/ws01@now tank/ws01bugfix [user root on neo] |
Use the -i option to display internal event information that can be used for diagnostic purposes. For example:
# zpool history -i tank 2010-05-13.14:13:15 zpool create -f tank mirror c1t2d0 c1t23d0 2010-05-13.14:13:45 [internal pool create txg:6] pool spa 19; zfs spa 19; zpl 4;... 2010-05-13.14:21:19 zfs create tank/snaps 2010-05-13.14:22:02 [internal replay_inc_sync txg:20451] dataset = 41 2010-05-13.14:25:25 [internal snapshot txg:20480] dataset = 52 2010-05-13.14:25:25 [internal destroy_begin_sync txg:20481] dataset = 41 2010-05-13.14:25:26 [internal destroy txg:20488] dataset = 41 2010-05-13.14:25:26 [internal reservation set txg:20488] 0 dataset = 0 2010-05-14.08:10:29 zfs create tank/ws01 2010-05-14.08:10:54 [internal snapshot txg:53992] dataset = 42 2010-05-14.08:10:54 zfs snapshot tank/ws01@now 2010-05-14.08:11:04 [internal create txg:53994] dataset = 58 2010-05-14.08:11:05 zfs clone tank/ws01@now tank/ws01bugfix |
Viewing I/O Statistics for ZFS Storage Pools
To request I/O statistics for a pool or specific virtual devices, use the zpool iostat command. Similar to the iostat command, this command can display a static snapshot of all I/O activity, as well as updated statistics for every specified interval. The following statistics are reported:
- alloc capacity
-
The amount of data currently stored in the pool or device. This amount differs from the amount of disk space available to actual file systems by a small margin due to internal implementation details.
For more information about the differences between pool space and dataset space, see ZFS Disk Space Accounting.
- free capacity
-
The amount of disk space available in the pool or device. As with the used statistic, this amount differs from the amount of disk space available to datasets by a small margin.
- read operations
-
The number of read I/O operations sent to the pool or device, including metadata requests.
- write operations
-
The number of write I/O operations sent to the pool or device.
- read bandwidth
-
The bandwidth of all read operations (including metadata), expressed as units per second.
- write bandwidth
-
The bandwidth of all write operations, expressed as units per second.
Listing Pool-Wide I/O Statistics
With no options, the zpool iostat command displays the accumulated statistics since boot for all pools on the system. For example:
# zpool iostat
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
rpool 6.05G 61.9G 0 0 786 107
tank 31.3G 36.7G 4 1 296K 86.1K
---------- ----- ----- ----- ----- ----- -----
|
Because these statistics are cumulative since boot, bandwidth might appear low if the pool is relatively idle. You can request a more accurate view of current bandwidth usage by specifying an interval. For example:
# zpool iostat tank 2
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
tank 18.5G 49.5G 0 187 0 23.3M
tank 18.5G 49.5G 0 464 0 57.7M
tank 18.5G 49.5G 0 457 0 56.6M
tank 18.8G 49.2G 0 435 0 51.3M
|
In this example, the command displays usage statistics for the pool tank every two seconds until you type Control-C. Alternately, you can specify an additional count argument, which causes the command to terminate after the specified number of iterations. For example, zpool iostat 2 3 would print a summary every two seconds for three iterations, for a total of six seconds. If there is only a single pool, then the statistics are displayed on consecutive lines. If more than one pool exists, then an additional dashed line delineates each iteration to provide visual separation.
Listing Virtual Device I/O Statistics
In addition to pool-wide I/O statistics, the zpool iostat command can display I/O statistics for virtual devices. This command can be used to identify abnormally slow devices or to observe the distribution of I/O generated by ZFS. To request the complete virtual device layout as well as all I/O statistics, use the zpool iostat -v command. For example:
# zpool iostat -v
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
rpool 6.05G 61.9G 0 0 785 107
mirror 6.05G 61.9G 0 0 785 107
c1t0d0s0 - - 0 0 578 109
c1t1d0s0 - - 0 0 595 109
---------- ----- ----- ----- ----- ----- -----
tank 36.5G 31.5G 4 1 295K 146K
mirror 36.5G 31.5G 126 45 8.13M 4.01M
c1t2d0 - - 0 3 100K 386K
c1t3d0 - - 0 3 104K 386K
---------- ----- ----- ----- ----- ----- -----
|
Note two important points when viewing I/O statistics for virtual devices:
-
First, disk space usage statistics are only available for top-level virtual devices. The way in which disk space is allocated among mirror and RAID-Z virtual devices is particular to the implementation and not easily expressed as a single number.
-
Second, the numbers might not add up exactly as you would expect them to. In particular, operations across RAID-Z and mirrored devices will not be exactly equal. This difference is particularly noticeable immediately after a pool is created, as a significant amount of I/O is done directly to the disks as part of pool creation, which is not accounted for at the mirror level. Over time, these numbers gradually equalize. However, broken, unresponsive, or offline devices can affect this symmetry as well.
You can use the same set of options (interval and count) when examining virtual device statistics.
Determining the Health Status of ZFS Storage Pools
ZFS provides an integrated method of examining pool and device health. The health of a pool is determined from the state of all its devices. This state information is displayed by using the zpool status command. In addition, potential pool and device failures are reported by fmd, displayed on the system console, and logged in the /var/adm/messages file.
This section describes how to determine pool and device health. This chapter does not document how to repair or recover from unhealthy pools. For more information about troubleshooting and data recovery, see Chapter 11, Oracle Solaris ZFS Troubleshooting and Pool Recovery.
Each device can fall into one of the following states:
- ONLINE
-
The device or virtual device is in normal working order. Although some transient errors might still occur, the device is otherwise in working order.
- DEGRADED
-
The virtual device has experienced a failure but can still function. This state is most common when a mirror or RAID-Z device has lost one or more constituent devices. The fault tolerance of the pool might be compromised, as a subsequent fault in another device might be unrecoverable.
- FAULTED
-
The device or virtual device is completely inaccessible. This status typically indicates total failure of the device, such that ZFS is incapable of sending data to it or receiving data from it. If a top-level virtual device is in this state, then the pool is completely inaccessible.
- OFFLINE
-
The device has been explicitly taken offline by the administrator.
- UNAVAIL
-
The device or virtual device cannot be opened. In some cases, pools with UNAVAIL devices appear in DEGRADED mode. If a top-level virtual device is UNAVAIL, then nothing in the pool can be accessed.
- REMOVED
-
The device was physically removed while the system was running. Device removal detection is hardware-dependent and might not be supported on all platforms.
The health of a pool is determined from the health of all its top-level virtual devices. If all virtual devices are ONLINE, then the pool is also ONLINE. If any one of the virtual devices is DEGRADED or UNAVAIL, then the pool is also DEGRADED. If a top-level virtual device is FAULTED or OFFLINE, then the pool is also FAULTED. A pool in the FAULTED state is completely inaccessible. No data can be recovered until the necessary devices are attached or repaired. A pool in the DEGRADED state continues to run, but you might not achieve the same level of data redundancy or data throughput than if the pool were online.
Basic Storage Pool Health Status
You can quickly review pool health status by using the zpool status command as follows:
# zpool status -x all pools are healthy |
Specific pools can be examined by specifying a pool name in the command syntax. Any pool that is not in the ONLINE state should be investigated for potential problems, as described in the next section.
Detailed Health Status
You can request a more detailed health summary status by using the -v option. For example:
# zpool status -v tank
pool: tank
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-2Q
scrub: scrub completed after 0h0m with 0 errors on Wed Jan 20 15:13:59 2010
config:
NAME STATE READ WRITE CKSUM
tank DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
c1t0d0 ONLINE 0 0 0
c1t1d0 UNAVAIL 0 0 0 cannot open
errors: No known data errors
|
This output displays a complete description of why the pool is in its current state, including a readable description of the problem and a link to a knowledge article for more information. Each knowledge article provides up-to-date information about the best way to recover from your current problem. Using the detailed configuration information, you can determine which device is damaged and how to repair the pool.
In the preceding example, the faulted device should be replaced. After the device is replaced, use the zpool online command to bring the device online. For example:
# zpool online tank c1t0d0 Bringing device c1t0d0 online # zpool status -x all pools are healthy |
If the autoreplace property is on, you might not have to online the replaced device.
If a pool has an offline device, the command output identifies the problem pool. For example:
# zpool status -x
pool: tank
state: DEGRADED
status: One or more devices has been taken offline by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scrub: resilver completed after 0h0m with 0 errors on Wed Jan 20 15:15:09 2010
config:
NAME STATE READ WRITE CKSUM
tank DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
c1t0d0 ONLINE 0 0 0
c1t1d0 OFFLINE 0 0 0 48K resilvered
errors: No known data errors
|
The READ and WRITE columns provide a count of I/O errors that occurred on the device, while the CKSUM column provides a count of uncorrectable checksum errors that occurred on the device. Both error counts indicate a potential device failure, and some corrective action is needed. If non-zero errors are reported for a top-level virtual device, portions of your data might have become inaccessible.
The errors: field identifies any known data errors.
In the preceding example output, the offline device is not causing data errors.
For more information about diagnosing and repairing faulted pools and data, see Chapter 11, Oracle Solaris ZFS Troubleshooting and Pool Recovery.
Migrating ZFS Storage Pools
Occasionally, you might need to move a storage pool between systems. To do so, the storage devices must be disconnected from the original system and reconnected to the destination system. This task can be accomplished by physically recabling the devices, or by using multiported devices such as the devices on a SAN. ZFS enables you to export the pool from one machine and import it on the destination system, even if the system are of different architectural endianness. For information about replicating or migrating file systems between different storage pools, which might reside on different machines, see Sending and Receiving ZFS Data.
Preparing for ZFS Storage Pool Migration
Storage pools should be explicitly exported to indicate that they are ready to be migrated. This operation flushes any unwritten data to disk, writes data to the disk indicating that the export was done, and removes all information about the pool from the system.
If you do not explicitly export the pool, but instead remove the disks manually, you can still import the resulting pool on another system. However, you might lose the last few seconds of data transactions, and the pool will appear faulted on the original system because the devices are no longer present. By default, the destination system cannot import a pool that has not been explicitly exported. This condition is necessary to prevent you from accidentally importing an active pool that consists of network-attached storage that is still in use on another system.
Exporting a ZFS Storage Pool
To export a pool, use the zpool export command. For example:
# zpool export tank |
The command attempts to unmount any mounted file systems within the pool before continuing. If any of the file systems fail to unmount, you can forcefully unmount them by using the -f option. For example:
# zpool export tank cannot unmount '/export/home/eschrock': Device busy # zpool export -f tank |
After this command is executed, the pool tank is no longer visible on the system.
If devices are unavailable at the time of export, the devices cannot be identified as cleanly exported. If one of these devices is later attached to a system without any of the working devices, it appears as “potentially active.”
If ZFS volumes are in use in the pool, the pool cannot be exported, even with the -f option. To export a pool with a ZFS volume, first ensure that all consumers of the volume are no longer active.
For more information about ZFS volumes, see ZFS Volumes.
Determining Available Storage Pools to Import
After the pool has been removed from the system (either through an explicit export or by forcefully removing the devices), you can attach the devices to the target system. ZFS can handle some situations in which only some of the devices are available, but a successful pool migration depends on the overall health of the devices. In addition, the devices do not necessarily have to be attached under the same device name. ZFS detects any moved or renamed devices, and adjusts the configuration appropriately. To discover available pools, run the zpool import command with no options. For example:
# zpool import
pool: tank
id: 11809215114195894163
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:
tank ONLINE
mirror-0 ONLINE
c1t0d0 ONLINE
c1t1d0 ONLINE
|
In this example, the pool tank is available to be imported on the target system. Each pool is identified by a name as well as a unique numeric identifier. If multiple pools with the same name are available to import, you can use the numeric identifier to distinguish between them.
Similar to the zpool status command output, the zpool import output includes a link to a knowledge article with the most up-to-date information regarding repair procedures for the problem that is preventing a pool from being imported. In this case, the user can force the pool to be imported. However, importing a pool that is currently in use by another system over a storage network can result in data corruption and panics as both systems attempt to write to the same storage. If some devices in the pool are not available but sufficient redundant data exists to provide a usable pool, the pool appears in the DEGRADED state. For example:
# zpool import
pool: tank
id: 11809215114195894163
state: DEGRADED
status: One or more devices are missing from the system.
action: The pool can be imported despite missing or damaged devices. The
fault tolerance of the pool may be compromised if imported.
see: http://www.sun.com/msg/ZFS-8000-2Q
config:
NAME STATE READ WRITE CKSUM
tank DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
c1t0d0 UNAVAIL 0 0 0 cannot open
c1t3d0 ONLINE 0 0 0
|
In this example, the first disk is damaged or missing, though you can still import the pool because the mirrored data is still accessible. If too many faulted or missing devices are present, the pool cannot be imported. For example:
# zpool import
pool: dozer
id: 9784486589352144634
state: FAULTED
action: The pool cannot be imported. Attach the missing
devices and try again.
see: http://www.sun.com/msg/ZFS-8000-6X
config:
raidz1-0 FAULTED
c1t0d0 ONLINE
c1t1d0 FAULTED
c1t2d0 ONLINE
c1t3d0 FAULTED
|
In this example, two disks are missing from a RAID-Z virtual device, which means that sufficient redundant data is not available to reconstruct the pool. In some cases, not enough devices are present to determine the complete configuration. In this case, ZFS cannot determine what other devices were part of the pool, though ZFS does report as much information as possible about the situation. For example:
# zpool import
pool: dozer
id: 9784486589352144634
state: FAULTED
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
devices and try again.
see: http://www.sun.com/msg/ZFS-8000-6X
config:
dozer FAULTED missing device
raidz1-0 ONLINE
c1t0d0 ONLINE
c1t1d0 ONLINE
c1t2d0 ONLINE
c1t3d0 ONLINE
Additional devices are known to be part of this pool, though their
exact configuration cannot be determined.
|
Importing ZFS Storage Pools From Alternate Directories
By default, the zpool import command only searches devices within the /dev/dsk directory. If devices exist in another directory, or you are using pools backed by files, you must use the -d option to search alternate directories. For example:
# zpool create dozer mirror /file/a /file/b
# zpool export dozer
# zpool import -d /file
pool: dozer
id: 7318163511366751416
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:
dozer ONLINE
mirror-0 ONLINE
/file/a ONLINE
/file/b ONLINE
# zpool import -d /file dozer
|
If devices exist in multiple directories, you can specify multiple -d options.
Importing ZFS Storage Pools
After a pool has been identified for import, you can import it by specifying the name of the pool or its numeric identifier as an argument to the zpool import command. For example:
# zpool import tank |
If multiple available pools have the same name, you must specify which pool to import by using the numeric identifier. For example:
# zpool import
pool: dozer
id: 2704475622193776801
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:
dozer ONLINE
c1t9d0 ONLINE
pool: dozer
id: 6223921996155991199
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:
dozer ONLINE
c1t8d0 ONLINE
# zpool import dozer
cannot import 'dozer': more than one matching pool
import by numeric ID instead
# zpool import 6223921996155991199
|
If the pool name conflicts with an existing pool name, you can import the pool under a different name. For example:
# zpool import dozer zeepool |
This command imports the exported pool dozer using the new name zeepool.
If the pool was not cleanly exported, ZFS requires the -f flag to prevent users from accidentally importing a pool that is still in use on another system. For example:
# zpool import dozer cannot import 'dozer': pool may be in use on another system use '-f' to import anyway # zpool import -f dozer |
Note –
Do not attempt to import a pool that is active on one system to another system. ZFS is not a native cluster, distributed, or parallel file system and cannot provide concurrent access from multiple, different hosts.
Pools can also be imported under an alternate root by using the -R option. For more information on alternate root pools, see Using ZFS Alternate Root Pools.
Recovering Destroyed ZFS Storage Pools
You can use the zpool import -D command to recover a storage pool that has been destroyed. For example:
# zpool destroy tank
# zpool import -D
pool: tank
id: 5154272182900538157
state: ONLINE (DESTROYED)
action: The pool can be imported using its name or numeric identifier.
config:
tank ONLINE
mirror-0 ONLINE
c1t0d0 ONLINE
c1t1d0 ONLINE
|
In this zpool import output, you can identify the tank pool as the destroyed pool because of the following state information:
state: ONLINE (DESTROYED) |
To recover the destroyed pool, run the zpool import -D command again with the pool to be recovered. For example:
# zpool import -D tank
# zpool status tank
pool: tank
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE
mirror-0 ONLINE
c1t0d0 ONLINE
c1t1d0 ONLINE
errors: No known data errors
|
If one of the devices in the destroyed pool is faulted or unavailable, you might be able to recover the destroyed pool anyway by including the -f option. In this scenario, you would import the degraded pool and then attempt to fix the device failure. For example:
# zpool destroy dozer
# zpool import -D
pool: dozer
id: 13643595538644303788
state: DEGRADED (DESTROYED)
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-2Q
config:
NAME STATE READ WRITE CKSUM
dozer DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
c2t8d0 ONLINE 0 0 0
c2t9d0 ONLINE 0 0 0
c2t10d0 ONLINE 0 0 0
c2t11d0 UNAVAIL 0 35 1 cannot open
c2t12d0 ONLINE 0 0 0
errors: No known data errors
# zpool import -Df dozer
# zpool status -x
pool: dozer
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-2Q
scrub: scrub completed after 0h0m with 0 errors on Thu Jan 21 15:38:48 2010
config:
NAME STATE READ WRITE CKSUM
dozer DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
c2t8d0 ONLINE 0 0 0
c2t9d0 ONLINE 0 0 0
c2t10d0 ONLINE 0 0 0
c2t11d0 UNAVAIL 0 37 0 cannot open
c2t12d0 ONLINE 0 0 0
errors: No known data errors
# zpool online dozer c2t11d0
Bringing device c2t11d0 online
# zpool status -x
all pools are healthy
|
Upgrading ZFS Storage Pools
If you have ZFS storage pools from a previous Solaris release, such as the Solaris 10 10/09 release, you can upgrade your pools with the zpool upgrade command to take advantage of the pool features in the current release. In addition, the zpool status command has been modified to notify you when your pools are running older versions. For example:
# zpool status
pool: tank
state: ONLINE
status: The pool is formatted using an older on-disk format. The pool can
still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'. Once this is done, the
pool will no longer be accessible on older software versions.
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
errors: No known data errors
|
You can use the following syntax to identify additional information about a particular version and supported releases:
# zpool upgrade -v This system is currently running ZFS pool version 22. The following versions are supported: VER DESCRIPTION --- -------------------------------------------------------- 1 Initial ZFS version 2 Ditto blocks (replicated metadata) 3 Hot spares and double parity RAID-Z 4 zpool history 5 Compression using the gzip algorithm 6 bootfs pool property 7 Separate intent log devices 8 Delegated administration 9 refquota and refreservation properties 10 Cache devices 11 Improved scrub performance 12 Snapshot properties 13 snapused property 14 passthrough-x aclinherit 15 user/group space accounting 16 stmf property support 17 Triple-parity RAID-Z 18 Snapshot user holds 19 Log device removal 20 Compression using zle (zero-length encoding) 21 Reserved 22 Received properties For more information on a particular version, including supported releases, see the ZFS Administration Guide. |
Then, you can run the zpool upgrade command to upgrade all of your pools. For example:
# zpool upgrade -a |
Note –
If you upgrade your pool to a later ZFS version, the pool will not be accessible on a system that runs an older ZFS version.
- © 2010, Oracle Corporation and/or its affiliates