RSYNC is Slow When Copying Files

Symptom: rsync runs very slowly against a file system.

Cause: rsync is a serial operation, so it is slow when copying a large file system, especially if snapshots are included in the process.

Solution: Use one of the following alternatives:
  • GNU Parallel to run rsync in parallel. For example:
    time find /mnt/MyFileSystem -mindepth 1 -maxdepth 1 | parallel -P100 rsync --archive --perms --owner --group --xattrs --acls --recursive --delete --compress --exclude=.snapshot --ignore-errors --progress --log-file=$HOME/rsync/logs/test.log1 --quiet -e ssh {} root@10.0.3.6:/mnt/rsync_target

    For more information, see GNU Parallel - GNU Project.

  • File Storage Parallel Tools

    For more information and examples, see Using File Storage Parallel Tools.

  • The find command in combination with the xargs option. For example:
    find ${source_dir}  -mindepth 1 -maxdepth 1 | xargs -P 24 -I {} rsync --archive --perms --owner --group --xattrs --acls --recursive --delete --compress --log-file=<logfile_path> -quiet -e ssh {} <destination_user>@<destination_instance>:<destination_dir>

    See find(1)- Linux Man Page and xargs(1)- Linux Man Page for more information.

  • If you're using rsync to copy files from one region to another, see Copying Files from Region to Region Using RSYNC or FPSYNC is Slow.