Note:

This tutorial requires access to Oracle Cloud. To sign up for a free account, see Get started with Oracle Cloud Infrastructure Free Tier.
It uses example values for Oracle Cloud Infrastructure credentials, tenancy, and compartments. When completing your lab, substitute these values with ones specific to your cloud environment.

Use Migration Tools to Move Data into OCI Cloud Storage Services

Introduction

This is tutorial 1 of a four tutorial series that shows you various ways to migrate data into Oracle Cloud Infrastructure (OCI) cloud storage services. The series is setup so you can review this first tutorial to get a broad understanding of the various tools and then proceed to the related tutorial(s) or documents relevant to your migration needs.

OCI provides customers with high-performance computing and low-cost cloud storage options. Through on-demand local, object, file, block, and archive storage, Oracle addresses key storage workload requirements and use cases.

OCI cloud storage services offer fast, secure, and durable cloud storage options for all your enterprise needs. Starting with the high performance options such as OCI File Storage with Lustre and OCI Block Volumes service; fully managed exabyte scale filesystems from OCI File Storage service with high performance mount targets; to highly durable and scalable OCI Object Storage. Our solutions can meet your demands, ranging from performance intensive applications such as AI/ML workloads, to exabyte-scale data lakes.

Many customers find the need to transfer data into OCI cloud storage service from on-prem, another provider, or between OCI cloud storage services. Based on origination, destination, and the direction of data transfer, the best method to accomplish the migration can vary. Once you have identified the basics of the data source and the destination in OCI you will then have to decide on a migration path and which tools you will need to use. Let our hands-on experience guide you towards the right migration tool and how to use it. This tutorial 1 in a series to introduce you to various tools and where they might best fit into the migration process.

Determine the amount of data that needs to be migrated, and the downtime available to cut-over to the new OCI storage platform. Batch migrations are a good choice to break down the migration into manageable increments. Batch migrations will enable you to schedule downtime for specific applications across different windows. Some customers have the flexibility to do a one-time migration over a scheduled maintenance window over 2-4 days. OCI FastConnect can be used to create a dedicated, private connection between OCI and your environment, with port speeds from 1G to 400G to speed up the data transfer process. OCI FastConnect can be integrated with partner solutions such as Megaport and ConsoleConnect to create a private connection to your data center or cloud-to-cloud interconnection to move data more directly from another cloud vendor into OCI cloud storage service. For more information, see FastConnect integration with Megaport Cloud Router.

Audience

DevOps engineers, developers, OCI cloud storage administrators and users, IT managers, OCI power users, and application administrators.

Objectives

Learn how to use various tools to copy and synchronize data into OCI cloud storage services:

Identify common migration tools.
Learn about the various tools for migrating filesystem data (local, NAS, cloud hosted) into OCI cloud storage services.
Learn which tool is best suited for various use cases.

Prerequisites

An understanding of your data migration set, overall size and what the files or objects look like (few large files, many small files, and so on).
Where in OCI cloud storage service (Object Storage, File Storage, or Block Volumes) the storage should land.
Your timelines and if you will be doing a bulk or incremental (batch) migration.

Migration Tools

Our customer experience and testing can help guide you to which migration tool will best fit your scenario. We encourage you to do a proof of concept and tests on sample data sets to verify the best migration method for your data set. We will explore common migration tools available to migrate data from on-premises and other cloud providers to OCI or within OCI cloud storage services. Test results are shared from sample datasets so you can extrapolate how the tools may perform on your data set.

Rclone: Rclone is an open source, command-line utility to migrate data to the cloud, or between cloud storage vendors. Rclone can be used to do one-time migration as well as periodical synchronization between source and destination storage. Rclone can migrate data to and from object storage, file storage, mounted drives, and between 70 supported storage types. OCI Object Storage is natively supported as a Rclone backend provider. Rclone processes can be scaled up and scaled out to increase the transfer performance using parameter options. Rclone has options like copy and sync to transfer data one-time and periodical respectively. For more information, see Install Rclone.
Resilio Active Anywhere: Resilio is a third party, agent-based data transfer application. It is graphic user interface (GUI) rich and gives great control on the transfer jobs as well as visualizing the performance metrics. The software is available in Oracle Cloud Marketplace and is licensed from Resilio for installation. Resilio can synchronize files in fixed timeframes in any direction in a one-to-many, many-to-one, or a many-to-many mesh enabling a global presence to a dataset.
Open Source Linux Sync Utilities: rsync and fpsync.
- Rsync: Common Unix based tool to do one-time copy or periodical sync between source and destination paths.
- Fpsync: Open source tool to do parallel sync. It uses rsync, tar or rclone underneath and runs a parallel wrapper on top. You can choose either of rsync, tar or rclone as the underlying sync tool.
Using File Storage Parallel Tools: OCI File Storage has delivered a parallel tools package that works optimally with the file system. It is available with Linux developer repository and can be installed directly from the yum repository. The package contains three tools that are parcp, parter and param which are parallel equivalent of standard Linux utilities cp, tar and rm respectively.
OCI Command Line Interface Object Storage Sync: OCI Object Storage Sync (os sync) is part of the OCI Command Line Interface (CLI) which synchronizes a filesystem directory with objects in a bucket. The command traverses sub-directories copying new and modified files or objects from the source to the destination and optionally deleting those that are not present in the source. It can run up to 1,00 parallel operations, depending on host machine resources.
OCI Object Storage Bulk Copy Python API: The following link is an example of bulk copy script using the python API for OCI. The API can be used to write scripts such as the example script which uses parallel threads to copy objects from one bucket into another. For more information on using the sample script, see Use Oracle Cloud Infrastructure Object Storage Python Utilities for Bulk Operations.
S3 Compatible tool (s5cmd): The s5cmd is an open source tool that can be used to migrate or synchronize local filesystems and NAS storage (on-prem and in OCI) into OCI Object Storage. It can also be used to migrate data bucket-to-bucket within the same region.

The Right Tool for the Job

There are many tools to choose from when doing a migration, and it can be overwhelming to research them all at once. Based on the origination, destination, direction of the migration, user experience, and user environment the best migration tool will vary.

The following table provides recommendations for common migration scenarios, including the migration of on-premises data, migration of data from another cloud vendor into OCI, copying OCI cloud storage data from one region to another, copying OCI cloud storage data within a region, and copying OCI File Storage data to OCI Object Storage.

Migrate Data From	Migrate Data To	Recommended Tool(s)	Notes	Documentation/ Tutorial Links
On-premises	OCI Object Storage	1. s5cmd (small/mixed files) 2. OCI Object Storage Sync (few large files) 3. Rclone (mixed)	Use the tool best for your data structures and you feel comfortable using. FastConnect should be reviewed to improve transfer time.	S5cmd and Object Storage Sync Tutorial, Rclone Tutorial
Another Cloud Vendor Object or Blob Storage	OCI Object Storage	Rclone	Rclone will be the most compatible supporting over 70 different cloud vendors.	Rclone Tutorial
OCI Object Storage	OCI Object Storage in another region	1. Object Replication 2. OCI Object Storage Bulk Copy Python API 3. Rclone	Using the native Object Replication is good for an exact replica of a new bucket that is currently empty, use OCI Object Storage Bulk Copy Python API or Rclone when you do not want to overwrite data in the target or to initialize copies of source which already has objects.	Object Storage Replication documentation, Use Oracle Cloud Infrastructure Object Storage Python Utilities for Bulk Operations, Rclone Tutorial
OCI Object Storage	OCI Object Storage in another tenancy (same or different region)	1. OCI Object Storage Bulk Copy Python API 2. Rclone	Pre-requisite: Use cross tenancy IAM policies to enable cross-tenancy copies. For more information, see Accessing Object Storage Resources Across Tenancies.	Use Oracle Cloud Infrastructure Object Storage Python Utilities for Bulk Operations, Rclone Tutorial
OCI File Storage	OCI Object Storage	1. s5cmd (small/mixed files) 2. Object Storage Sync (few large files) 3. Rclone (mixed)	Use the tool best for your data structures and you feel comfortable using.	S5cmd and Object Storage Sync Tutorial, Rclone Tutorial
On-premises	OCI File Storage	fpsync (Linux) and CIFS + fpsync (Windows)	Ensure that network connectivity is established between source and destination instances.	Fpsync documentation, fpsync tutorial
Another cloud vendor local disk or file storage	OCI File Storage	fpsync (Linux) and CIFS + fpsync (Windows)	Ensure that network connectivity is established between source and destination instances. FastConnect should be reviewed to improve transfer time.	Fpsync documentation, fpsync tutorial
OCI File Storage	OCI File Storage in another region	1. File System Replication 2. fpsync with instance-to-instance streaming	If you use replication, see replication’s Limitations and Considerations or if you use instance-to-instance streaming, ensure that network connectivity is established between source and destination instances.	Fpsync documentation, fpsync tutorial, File System Replication documentation
OCI File Storage	OCI File Storage within the same availability domain	1. File System Replication 2. Using File Storage Parallel Tools: parcp	If you use replication, see replication’s Limitations and Considerations or if you use parcp, ensure that both source and destination file systems are mounted in the instance.	File System Replication, Using File Storage Parallel Tools: parcp
On-premises, Another cloud vendor	OCI Object Storage or OCI File Storage	Resilio Active Anywhere Platform	Use Resilio Active Anywhere platform when you need multi-way synchronization of data, white glove service with support, and GUI interface. Resilio has been verified by the OCI cloud storage service product team and is available in the Oracle Cloud Marketplace.	For more information on using their platform reach out to the Resilio team.

Note: The migration tool series will not cover OCI Object Storage or OCI File System Replication, OCI Object Storage Bulk Copy Python API, and Resilio. See Related Links for more information.

Next Steps

Proceed to the related tutorial(s) relevant to your migration needs. To move data into OCI cloud storage services:

Using Rclone, see Tutorial 2: Move Data into OCI Cloud Storage Services using Rclone.
Using OCI Object Storage Sync and S5cmd, see Tutorial 3: Move Data into OCI Cloud Storage Services using OCI Object Storage Sync and S5cmd.
Using Fpsync and Rsync for file system data migrations, see Tutorial 4: Move Data into OCI Cloud Storage Services using Fpsync and Rsync for File System Data Migrations.

Acknowledgments

Authors - Vinoth Krishnamurthy (Principal Member of Technical Staff, OCI File Storage), Melinda Centeno (Senior Principal Product Manager, OCI Object Storage)
Contributors - Aboo Valappil (Consulting Member of Technical Staff, OCI File and Block Storage), Ashutosh Mate (Senior Principal Product Manager, OCI Object Storage)

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.

Title and Copyright Information

Use Migration Tools to Move Data into OCI Cloud Storage Services

G25415-01

January 2025