Storage Pool Practices for Performance

• In general, keep pool capacity below 90% for best performance. The percentage where performance might be impacted depends greatly on workload:

  • If data is mostly added (write once, remove never), then it's very easy for ZFS to find new blocks. In this case, the percentage can be higher than normal; maybe up to 95%.

  • If data is made of large files or large blocks (such as 128K files or 1MB blocks) and the data is removed in bulk operations, the percentage can be higher than normal; maybe up to 95%.

  • If a large percentage (more than 50%) of the pool is made up of 8k chunks (DBfiles, iSCSI Luns, or many small files) and have constant rewrites, then the 90% rule should be followed strictly.

  • If all of the data is small blocks that have constant rewrites, then you should monitor your pool closely once the capacity gets over 80%. The sign to watch for is increased disk IOPS to achieve the same level of client IOPS.

• Mirrored pools are recommended over RAID-Z pools for random read/write workloads

• Separate log devices

  • Recommended to improve synchronous write performance

  • With a high synchronous write load, prevents fragmentation of writing many log blocks in the main pool

• Separate cache devices are recommended to improve read performance

• Scrub/resilver - A very large RAID-Z pool with lots of devices will have longer scrub and resilver times

• Pool performance is slow – Use the zpool status command to rule out any hardware problems that are causing pool performance problems. If no problems show up in the zpool status command, use the fmdump command to display hardware faults or use the fmdump -eV command to review any hardware errors that have not yet resulted in a reported fault.