1. Managing ZFS File Systems in Oracle Solaris 11.4
  2. Oracle Solaris ZFS Troubleshooting and Pool Recovery
  3. Resolving Data Problems in a ZFS Storage Pool
  4. Resolving ZFS Space Issues
  5. ZFS Storage Pool Space Reporting

ZFS Storage Pool Space Reporting

The SIZE value that is reported by the zpool list command is generally the amount of physical disk space in the pool, but varies depending on the pool's redundancy level. See the examples below. The zfs list command lists the usable space that is available to file systems, which is disk space minus ZFS pool redundancy metadata overhead, if any.

The following ZFS dataset configurations are tracked as allocated space by the zfs list command but they are not tracked as allocated space in the zpool list output:

  • ZFS file system quota

  • ZFS file system reservation

  • ZFS logical volume size

The following items describe how using different pool configurations, ZFS volumes and ZFS reservations can impact your consumed and available disk space. Depending upon your configuration, monitoring pool space should be tracked by using the steps listed below.

  • Non-redundant storage pool – When a pool is created with one 136GB disk, the zpool list command reports SIZE and initial FREE values as 136 GB. The initial AVAIL space reported by the zfs list command is 134 GB, due to a small amount of pool metadata overhead. For example: 

    $ zpool create system1 c0t6d0
$ zpool list system1
NAME      SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
system1   136G  95.5K   136G     0%  1.00x  ONLINE  -
$ zfs list system1
NAME      USED  AVAIL  REFER  MOUNTPOINT
system1    72K   134G    21K  /system1

  • Mirrored storage pool – When a pool is created with two 136GB disks, zpool list command reports SIZE as 136 GB and initial FREE value as 136 GB. This reporting is referred to as the deflated space value. The initial AVAIL space reported by the zfs list command is 134 GB, due to a small amount of pool metadata overhead. For example: 

    $ zpool create system1 mirror c0t6d0 c0t7d0
$ zpool list system1
NAME      SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
system1   136G  95.5K   136G     0%  1.00x  ONLINE  -
$ zfs list system1
NAME      USED  AVAIL  REFER  MOUNTPOINT
system1    72K   134G    21K  /system1

  • RAID-Z storage pool – When a raidz2 pool is created with three 136GB disks, the zpool list commands reports SIZE as 408 GB and initial FREE value as 408 GB. This reporting is referred to as the inflated disk space value, which includes redundancy overhead, such as parity information. The initial AVAIL space reported by the zfs list command is 133 GB, due to the pool redundancy overhead. The space discrepancy between the zpool list and the zfs list output for a RAID-Z pool is because zpool list reports the inflated pool space. 

    $ zpool create system1 raidz2 c0t6d0 c0t7d0 c0t8d0
$ zpool list system1
NAME      SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
system1   408G   286K   408G     0%  1.00x  ONLINE  -
$ zfs list system1
NAME      USED  AVAIL  REFER  MOUNTPOINT
system1  73.2K   133G  20.9K  /system1

  • NFS mounted file system space – Neither the zpool list or the zfs list account for NFS mounted file system space. However, local data files can be hidden under a mounted NFS file system. If you are missing file system space, ensure that you do not have data files hidden under an NFS file system.

  • Using ZFS Volumes – When a ZFS file system is created and pool space is consumed, you can view the file system space consumption by using the zpool list command. For example: 

    $ zpool create nova mirror c1t1d0 c2t1d0
$ zfs create nova/fs1
$ mkfile 10g /nova/fs1/file1_10g
$ zpool list nova
NAME  SIZE  ALLOC   FREE  CAP  DEDUP  HEALTH  ALTROOT
nova   68G  10.0G  58.0G  14%  1.00x  ONLINE  -
$ zfs list -r nova
NAME       USED  AVAIL  REFER  MOUNTPOINT
nova      10.0G  56.9G    32K  /nova
nova/fs1  10.0G  56.9G  10.0G  /nova/fs1

    If you create a 10GB ZFS volume, the space is not accounted for in the zpool list command. The space is accounted for in the zfs list command. If you are using ZFS volumes in your storage pools, monitor ZFS volume space consumption by using the zfs list command. For example: 

    $ zfs create -V 10g nova/vol1
$ zpool list nova
NAME  SIZE  ALLOC   FREE  CAP  DEDUP  HEALTH  ALTROOT
nova   68G  10.0G  58.0G  14%  1.00x  ONLINE  -
$ zfs list -r nova
NAME        USED  AVAIL  REFER  MOUNTPOINT
nova       20.3G  46.6G    32K  /nova
nova/fs1   10.0G  46.6G  10.0G  /nova/fs1
nova/vol1  10.3G  56.9G    16K  -

    Note in the above output that ZFS volume space is not tracked in the zpool list output so use the zfs list or the zfs list -o space command to identify space that is consumed by ZFS volumes.

    In addition, because ZFS volumes act like raw devices, some amount of space for metadata is automatically reserved through the refreservation property, which causes volumes to consume slightly more space then the amount specified when the volume was created. Do not remove the refreservation on ZFS volumes or you risk running out of volume space.

  • Using ZFS Reservations – If you create a file system with a reservation or add a reservation to an existing file system, reservations or refreservations are not tracked by the zpool list command.

    Identify space that is consumed by file system reservations by using the zfs list -r command to identify the increased USED space. For example: 

    $ zfs create -o reservation=10g nova/fs2
$ zpool list nova
NAME  SIZE  ALLOC   FREE  CAP  DEDUP  HEALTH  ALTROOT
nova   68G  10.0G  58.0G  14%  1.00x  ONLINE  -
$ zfs list -r nova
NAME        USED  AVAIL  REFER  MOUNTPOINT
nova       30.3G  36.6G    33K  /nova
nova/fs1   10.0G  36.6G  10.0G  /nova/fs1
nova/fs2     31K  46.6G    31K  /nova/fs2
nova/vol1  10.3G  46.9G    16K  -

    If you create a file system with a refreservation, it can be identified by using the zfs list -r command. For example: 

    $ zfs create -o refreservation=10g nova/fs3
$ zfs list -r nova
NAME        USED  AVAIL  REFER  MOUNTPOINT
nova       40.3G  26.6G    35K  /nova
nova/fs1   10.0G  26.6G  10.0G  /nova/fs1
nova/fs2     31K  36.6G    31K  /nova/fs2
nova/fs3     10G  36.6G    31K  /nova/fs3
nova/vol1  10.3G  36.9G    16K  -

    Use the following command to identify all existing reservations to account for total USED space.

    $ zfs get -r reserv,refreserv nova
NAME       PROPERTY        VALUE  SOURCE
nova       reservation     none   default
nova       refreservation  none   default
nova/fs1   reservation     none   default
nova/fs1   refreservation  none   default
nova/fs2   reservation     10G    local
nova/fs2   refreservation  none   default
nova/fs3   reservation     none   default
nova/fs3   refreservation  10G    local
nova/vol1  reservation     none   default
nova/vol1  refreservation  10.3G  local