Replacing a Device in a ZFS Storage Pool
If device damage is permanent or future permanent damage is likely, the device must be replaced. Whether the device can be replaced depends on the configuration.
Determining if a Device Can Be Replaced
For a device to be replaced, the pool must be in the ONLINE state. The device must be part of a redundant configuration, or it must be healthy (in the ONLINE state). If the disk is part of a redundant configuration, sufficient replicas from which to retrieve good data must exist. If two disks in a four-way mirror are faulted, then either disk can be replaced because healthy replicas are available. However, if two disks in a four-way RAID-Z (raidz1) device are faulted, then neither disk can be replaced because not enough replicas from which to retrieve data exist. If the device is damaged but otherwise online, it can be replaced as long as the pool is not in the FAULTED state. However, any bad data on the device is copied to the new device unless there are sufficient replicas with good data.
In the following configuration, the disk c1t1d0 can be replaced, and any data in the pool is copied from the good replica, c1t0d0.
mirror DEGRADED
c1t0d0 ONLINE
c1t1d0 FAULTED
|
The disk c1t0d0 can also be replaced, though no self-healing of data can take place because no good replica is available.
In the following configuration, neither of the faulted disks can be replaced. The ONLINE disks cannot be replaced either, because the pool itself is faulted.
raidz FAULTED
c1t0d0 ONLINE
c2t0d0 FAULTED
c3t0d0 FAULTED
c4t0d0 ONLINE
|
In the following configuration, either top-level disk can be replaced, though any bad data present on the disk is copied to the new disk.
c1t0d0 ONLINE c1t1d0 ONLINE |
If either disk were faulted, then no replacement could be performed because the pool itself would be faulted.
Devices That Cannot be Replaced
If the loss of a device causes the pool to become faulted, or the device contains too many data errors in a non-redundant configuration, then the device cannot safely be replaced. Without sufficient redundancy, no good data with which to heal the damaged device exists. In this case, the only option is to destroy the pool and re-create the configuration, restoring your data in the process.
For more information about restoring an entire pool, see Repairing ZFS Storage Pool-Wide Damage.
Replacing a Device in a ZFS Storage Pool
Once you have determined that a device can be replaced, use the zpool replace command to replace the device. If you are replacing the damaged device with another different device, use the following command:
# zpool replace tank c1t1d0 c2t0d0 |
This command begins migrating data to the new device from the damaged device, or other devices in the pool if it is in a redundant configuration. When the command is finished, it detaches the damaged device from the configuration, at which point the device can be removed from the system. If you have already removed the device and replaced it with a new device in the same location, use the single device form of the command. For example:
# zpool replace tank c1t1d0 |
This command takes an unformatted disk, formats it appropriately, and then begins resilvering data from the rest of the configuration.
For more information about the zpool replace command, see Replacing Devices in a Storage Pool.
Example 11–1 Replacing a Device in a ZFS Storage Pool
The following example shows how to replace a device (c1t3d0) in the mirrored storage pool tank on a Sun Fire x4500 system. If you are going to replace the disk c1t3d0 with a new disk at the same location (c1t3d0), then unconfigure the disk before you attempt to replace it. The basic steps are as follows:
-
Offline the disk to be replaced first. You cannot unconfigure a disk that is currently being used.
-
Identify the disk (c1t3d0) to be unconfigured and unconfigure it. The pool will be degraded with the disk offlined in this mirrored configuration but the pool will continue to be available.
-
Physically replace the disk (c1t3d0). Make sure that the blue "Ready to Remove" LED is illuminated before you physically remove the faulted drive.
-
Reconfigure the disk (c1t3d0).
-
Bring the disk (c1t3d0) back online.
-
Run the zpool replace command to replace the disk (c1t3d0).
Note –If you had previously set the pool property autoreplace=on, then any new device, found in the same physical location as a device that previously belonged to the pool, is automatically formatted and replaced without using the zpool replace command. This feature might not be supported on all hardware.
-
If a failed disk is automatically replaced with a hot spare, you might need to detach the hot spare after the failed disk is replaced. For example, if c2t4d0 is still an active spare after the failed disk is replaced, then detach it.
# zpool detach tank c2t4d0
# zpool offline tank c1t3d0
# cfgadm | grep c1t3d0
sata1/3::dsk/c1t3d0 disk connected configured ok
# cfgadm -c unconfigure sata1/3
Unconfigure the device at: /devices/pci@0,0/pci1022,7458@2/pci11ab,11ab@1:3
This operation will suspend activity on the SATA device
Continue (yes/no)? yes
# cfgadm | grep sata1/3
sata1/3 disk connected unconfigured ok
<Replace the physical disk c1t3d0>
# cfgadm -c configure sata1/3
# cfgadm | grep sata1/3
sata1/3::dsk/c1t3d0 disk connected configured ok
# zpool online tank c1t3d0
# zpool replace tank c1t3d0
# zpool status
pool: tank
state: ONLINE
scrub: resilver completed after 0h0m with 0 errors on Tue Apr 22 14:44:46 2008
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror ONLINE 0 0 0
c0t1d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c0t2d0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c0t3d0 ONLINE 0 0 0
c1t3d0 ONLINE 0 0 0
errors: No known data errors
|
Note that the preceding zpool output might show both the new and old disks under a replacing heading. For example:
replacing DEGRADED 0 0 0 c1t3d0s0/o FAULTED 0 0 0 c1t3d0 ONLINE 0 0 0 |
This text means that the replacement process is in progress and the new disk is being resilvered.
If you are going to replace a disk (c1t3d0) with another disk (c4t3d0), then you only need to run the zpool replace command. For example:
# zpool replace tank c1t3d0 c4t3d0
# zpool status
pool: tank
state: DEGRADED
scrub: resilver completed after 0h0m with 0 errors on Tue Apr 22 14:54:50 2008
config:
NAME STATE READ WRITE CKSUM
tank DEGRADED 0 0 0
mirror ONLINE 0 0 0
c0t1d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c0t2d0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
mirror DEGRADED 0 0 0
c0t3d0 ONLINE 0 0 0
replacing DEGRADED 0 0 0
c1t3d0 OFFLINE 0 0 0
c4t3d0 ONLINE 0 0 0
errors: No known data errors
|
You might have to run the zpool status command several times until the disk replacement is complete.
# zpool status tank
pool: tank
state: ONLINE
scrub: resilver completed after 0h0m with 0 errors on Tue Apr 22 14:54:50 2008
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror ONLINE 0 0 0
c0t1d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c0t2d0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c0t3d0 ONLINE 0 0 0
c4t3d0 ONLINE 0 0 0
|
Example 11–2 Replacing a Failed Log Device
The following example shows to recover from a failed log device c0t5d0 in the storage pool, pool. The basic steps are as follows:
-
Review the zpool status -x output and FMA diagnostic message, described here:
-
Physically replace the failed log device.
-
Bring the log device online.
-
Clear the pool's error condition.
# zpool status -x
pool: pool
state: FAULTED
status: One or more of the intent logs could not be read.
Waiting for adminstrator intervention to fix the faulted pool.
action: Either restore the affected device(s) and run 'zpool online',
or ignore the intent log records by running 'zpool clear'.
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
pool FAULTED 0 0 0 bad intent log
mirror ONLINE 0 0 0
c0t1d0 ONLINE 0 0 0
c0t4d0 ONLINE 0 0 0
logs FAULTED 0 0 0 bad intent log
c0t5d0 UNAVAIL 0 0 0 cannot open
<Physically replace the failed log device>
# zpool online pool c0t5d0
# zpool clear pool
|
Viewing Resilvering Status
The process of replacing a drive can take an extended period of time, depending on the size of the drive and the amount of data in the pool. The process of moving data from one device to another device is known as resilvering, and can be monitored by using the zpool status command.
Traditional file systems resilver data at the block level. Because ZFS eliminates the artificial layering of the volume manager, it can perform resilvering in a much more powerful and controlled manner. The two main advantages of this feature are as follows:
-
ZFS only resilvers the minimum amount of necessary data. In the case of a short outage (as opposed to a complete device replacement), the entire disk can be resilvered in a matter of minutes or seconds, rather than resilvering the entire disk, or complicating matters with “dirty region” logging that some volume managers support. When an entire disk is replaced, the resilvering process takes time proportional to the amount of data used on disk. Replacing a 500-Gbyte disk can take seconds if only a few gigabytes of used space is in the pool.
-
Resilvering is interruptible and safe. If the system loses power or is rebooted, the resilvering process resumes exactly where it left off, without any need for manual intervention.
To view the resilvering process, use the zpool status command. For example:
# zpool status tank
pool: tank
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress for 0h2m, 16.43% done, 0h13m to go
config:
NAME STATE READ WRITE CKSUM
tank DEGRADED 0 0 0
mirror DEGRADED 0 0 0
replacing DEGRADED 0 0 0
c1t0d0 ONLINE 0 0 0
c2t0d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
|
In this example, the disk c1t0d0 is being replaced by c2t0d0. This event is observed in the status output by the presence of the replacing virtual device in the configuration. This device is not real, nor is it possible for you to create a pool by using this virtual device type. The purpose of this device is solely to display the resilvering progress, and to identify exactly which device is being replaced.
Note that any pool currently undergoing resilvering is placed in the ONLINE or DEGRADED state, because the pool cannot provide the desired level of redundancy until the resilvering process is complete. Resilvering proceeds as fast as possible, though the I/O is always scheduled with a lower priority than user-requested I/O, to minimize impact on the system. Once the resilvering is complete, the configuration reverts to the new, complete, configuration. For example:
# zpool status tank
pool: tank
state: ONLINE
scrub: resilver completed after 0h0m with 0 errors on Tue Sep 1 10:55:54 2009
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror ONLINE 0 0 0
c2t0d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
errors: No known data errors
|
The pool is once again ONLINE, and the original bad disk (c1t0d0) has been removed from the configuration.
- © 2010, Oracle Corporation and/or its affiliates