Review the following sections to determine whether pool problems or file system unavailability is related to a hardware problem, such as faulty system board, memory, device, HBA, or a misconfiguration.
For example, a failing or faulty disk on a busy ZFS pool can greatly degrade overall system performance.
If you start by diagnosing and identifying hardware problems first, which can be easier to detect and all your hardware checks out, you can then move on to diagnosing pool and file system problems as described in the rest of this chapter. If your hardware, pool, and file system configurations are healthy, consider diagnosing application problems, which are generally more complex to unravel and are not covered in this guide.
The Oracle Solaris Fault Manager tracks software, hardware and specific device problems by identifying error telemetry information that indicate a specific symptom in an error log and then reporting actual fault diagnosis when the error symptom results in an actual fault.
The following command identifies any software or hardware related fault.
# fmadm faulty
Use the above command routinely to identify failed services or devices.
Use the following command routinely to identify hardware or device related errors.
# fmdump -eV | more
Error messages in this log file that describe vdev.open_failed, checksum, or io_failure issues need your attention or they might evolve into actual faults that are displayed with the fmadm faulty command.
If the above indicates that a device is failing, then this is a good time to make sure you have a replacement device available.
You can also track additional device errors by using iostat command. Use the following syntax to identify a summary of error statistics.
# iostat -en ---- errors --- s/w h/w trn tot device 0 0 0 0 c0t5000C500335F95E3d0 0 0 0 0 c0t5000C500335FC3E7d0 0 0 0 0 c0t5000C500335BA8C3d0 0 12 0 12 c2t0d0 0 0 0 0 c0t5000C500335E106Bd0 0 0 0 0 c0t50015179594B6F11d0 0 0 0 0 c0t5000C500335DC60Fd0 0 0 0 0 c0t5000C500335F907Fd0 0 0 0 0 c0t5000C500335BD117d0
In the above output, errors are reported on an internal disk c2t0d0. Use the following syntax to display more detailed device errors.
Persistent SCSI transport errors that refer to retries or resets can be caused by down-rev firmware, a bad disk, a bad cable, or a faulty hardware connection. Some transient transport errors can be resolved by upgrading your HBA or device firmware. If transport errors persist after firmware is updated and all devices are deemed operational, then look for a bad cable or other faulty connection between hardware components.
In addition to persistently tracking errors within the pool, ZFS also displays syslog messages when events of interest occur. The following scenarios generate notification events:
Device state transition – If a device becomes FAULTED, ZFS logs a message indicating that the fault tolerance of the pool might be compromised. A similar message is sent if the device is later brought online, restoring the pool to health.
Data corruption – If any data corruption is detected, ZFS logs a message describing when and where the corruption was detected. This message is only logged the first time it is detected. Subsequent accesses do not generate a message.
Pool failures and device failures – If a pool failure or a device failure occurs, the fault manager daemon reports these errors through syslog messages as well as the fmdump command.
If ZFS detects a device error and automatically recovers from it, no notification occurs. Such errors do not constitute a failure in the pool redundancy or in data integrity. Moreover, such errors are typically the result of a driver problem accompanied by its own set of error messages.