Monitoring Migration Progress
Monitoring progress of a shadow migration is difficult given the context in which
the operation runs. A single filesystem can shadow all or part of a filesystem, or
multiple filesystems with nested mountpoints. As such, there is no way to request
statistics about the source and have any confidence in them being correct. In
addition, even with migration of a single filesystem, the methods used to calculate
the available size is not consistent across systems. For example, the remote
filesystem may use compression, or it may or not include metadata overhead. For
these reasons, it's impossible to display an accurate progress bar for any
particular migration.
The ZFSSA provides the following information that is guaranteed to be accurate:
-
Local size of the local filesystem so far
-
Logical size of the data copied so far
-
Time spent migrating data so far
These values are made available in the BUI and CLI through both the standard
filesystem properties as well as properties of the shadow migration node (or UI
panel). If you know the size of the remote filesystem, you can use this to estimate
progress. The size of the data copied consists only of plain file contents that
needed to be migrated from the source. Directories, metadata, and extended
attributes are not included in this calculation. While the size of the data migrated
so far includes only remotely migrated data, resuming background migration may
traverse parts of the filesystem that have already been migrated. This can cause it
to run fairly quickly while processing these initial directories, and slow down once
it reaches portions of the filesystem that have not yet been migrate.
While there is no accurate measurement of progress, the ZFSSA does attempt to make
an estimation of remaining data based on the assumption of a relatively uniform
directory tree. This estimate can range from fairly accurate to completely worthless
depending on the dataset, and is for information purposes only. For example, one
could have a relatively shallow filesystem tree but have large amounts of data in a
single directory that is visited last. In this scenario, the migration will appear
almost complete, and then rapidly drop to a very small percentage as this new tree
is discovered. Conversely, if that large directory was processed first, then the
estimate may assume that all other directories have a similarly large amount of
data, and when it finds them mostly empty the estimate quickly rises from a small
percentage to nearly complete. The best way to measure progress is to setup a test
migration, let it run to completion, and use that value to estimate progress for
filesystem of similar layout and size.