Oracle Cloud Infrastructure Documentation

Storage Gateway Cloud Sync

Describes how to use the Storage Gateway Cloud Sync feature.

Use Cloud Sync to move on-premises datasets from a local NFS-mounted file system to Storage Gateway, where the data is then moved asynchronously to Oracle Cloud Infrastructure Object Storage. You can also use Cloud Sync to synchronize Storage Gateway file system changes back to the local file system.

Cloud Sync generates the following for each sync job:

  • Sync status (Created, Running, Completed, Failed, or Canceled).

  • Number of files and directories to be copied from the source to the target.

  • File and directory upload progress.

  • Number of files and directories uploaded to the target.

  • Time the job started, the time the job ended, and the run duration.

  • The number of files skipped. (Cloud Sync skips non-regular files, such as symlinks, in the source directory.)

  • A list of skipped files.

You can perform the following Cloud Sync tasks:

About Cloud Sync

Cloud Sync is an enhanced wrapper around the Linux rsync command and relies on rsync to detect new and changed files using size and modification time. Cloud Sync also relies on rsync to verify the files once the data transfer is complete using checksums of the files. Cloud Sync calls the Storage Gateway diagnostic commands to provide detailed data movement progress and status between your local server, Storage Gateway, and Oracle Cloud Infrastructure.

Understanding Failure Behavior

Because the file system is mounted over NFS locally, the NFS client running on the host handles any issues resulting from file system availability or connectivity.

Cloud Sync does the following:

  • Reports and logs any failures to list or copy specific files (for example, resulting from permission issues).
  • Monitors and reports on the pending and completed uploads to Object Storage.

Storage Gateway handles any connectivity and access issues to Object Storage and performs retry operations as needed.

Prerequisites for Cloud Sync

  • Create the Storage Gateway file system that serves as either the source or target destination for the sync operation. A file system on the Storage Gateway host maps to a bucket with an identical name in Oracle Cloud Infrastructure Object Storage. See Creating Your First Storage Gateway File System or Adding a Storage Gateway File System for details.
  • Obtain the proper credentials to mount the file system share from the local server.

  • The local server source must:

    • Have names services set up correctly so that UIDs and GIDs are preserved and are not remapped to nobody.
    • Be exported with root permissions to read and traverse the entire source tree.
    • The filesystem to be synced needs to be NFS mounted to a mount point under /cloudsync/mounts/
Important

If a Storage Gateway file system serves as the source, set the auto-refresh for new objects in the bucket to be synced using CloudSync. Use the following command to set the file system refresh at regular intervals:

ocisg set file_system_name dataset.refreshInterval=interval_in_minutes

Restart Storage Gateway, then run the following commands to apply the changes:

ocisg down
ocisg up

See Managing Storage Gateway File Systems for more information.