Note:

• This tutorial requires access to Oracle Cloud. To sign up for a free account, see Get started with Oracle Cloud Infrastructure Free Tier.
• It uses example values for Oracle Cloud Infrastructure credentials, tenancy, and compartments. When completing your lab, substitute these values with ones specific to your cloud environment.

Migrate Data to Oracle Cloud Infrastructure Object Storage Using Rclone

Introduction

Oracle Cloud Infrastructure (OCI) Object Storage service is an internet-scale, high-performance storage platform that offers reliable and cost-efficient data durability. The Object Storage service can store an unlimited amount of unstructured data of any content type, including analytic data and rich content, like images and videos.

Rclone is a versatile command line program that synchronizes files and directories to and from various cloud storage providers or between on-premises to cloud. Rclone is an effective tool for migrating data to OCI Object Storage. It is known for its simplicity, efficiency, and wide support for numerous cloud services, including OCI Object Storage. Rclone supports the following features and use cases.

• Backup (and encrypt) to cloud storage and restore (and decrypt) files from cloud storage.
• Multi-threaded downloads to local disk feature can be utilized.
• Mirror cloud data to other cloud services or locally.
• Migrate data to OCI, or between cloud storage vendors and OCI or from on-premises to OCI.
• Mount multiple, encrypted, cached or diverse cloud storage as a disk.
• Transfer operations can be restarted at any time.
• Copy operation of new or changed files to OCI Object Storage.
• Sync option (one way) to make a directory identical.
• Implies server-side transfers to minimize use of local bandwidth.
• Check for missing files between source and OCI Object Storage.
• Support move operation of files to OCI Object Storage deleting the source/local files after validation/verification.
• Preserves timestamps and verifies checksums.
• No need of local disk or local copy of data.
• Mount any local, cloud or virtual filesystem as a disk on Windows, MacOS, Linux and FreeBSD platforms.
• Support SFTP, HTTP, WebDAV, FTO, and DLNA protocols.
• Transfers over limited bandwidth, intermittent connections.

Rclone Architecture Overview

Rclone architecture works in two different use cases.

• Rclone as a centralized or gateway solution: You can deploy Rclone in an Oracle Linux 9 centralized location or gateway, mount Network File System (NFS) or Server Message Block (SMB) shares from different operating systems, sources, or Network attached storage (NAS) appliances, then utilize Rclone to copy the data from the local source to OCI Object Storage.

  

• Rclone deployed in the NFS or SMB Servers: You can deploy Rclone directly on the NFS or SMB servers to copy data from the local source directly to OCI Object Storage. This model does not require Rclone running on a server in a centralized location.

  

Audience

System administrators, application administrators, system and application owners, database owners, and developers who want to migrate data to OCI Object Storage.

Objective

• Effectively transfer data to OCI Object Storage using Rclone.

Prerequisites

• Oracle Linux 8 or 9 server or a Windows platform server.

  Note: Rclone supports several operating systems. For more information about currently supported operating systems, see Rclone install documentation.

• Access to an Oracle Cloud Infrastructure account and OCI Console.

• An OCI configuration file. For more information, see SDK and Configuration File.

• Permissions to manage OCI Object Storage resources. For more information, see Common Policies and the Policy Reference.

• An OCI Object Storage bucket: A bucket in OCI Object Storage where data will be transferred. For more information, see Creating an OCI Object Storage Bucket.

• Local Data: From a local server, NFS or SMB shares.

• Rclone software.

• libsmbclient.x86_64: On Linux, the latest version of the SMB client library is required, check the version and install it with the following commands.

  

Task 1: Install Rclone on Windows

Note: If using Linux, skip to Task 2.

1. Download and install Rclone latest version for Windows. For more information, see Rclone.

2. Extract the file and add the extracted file to Windows system PATH environment variable for easy command line access.

Task 2: Install Rclone on Linux

1. First method is to run a curl install script. Run the following curl command to download and install Rclone.

   curl https://rclone.org/install.sh | sudo bash
   

   Or

   Rclone install via precompiled binaries. Download the precompiled binary.

   curl -O https://downloads.rclone.org/rclone-current-linux-amd64.zip 
   or
   wget https://downloads.rclone.org/rclone-current-linux-amd64.zip
   

   Note: If you have used curl command to install the Rclone, then go to Task 3.

2. Uncompress the binary and change to the binary directory.

   unzip rclone-current-linux-amd64.zip
   cd rclone-*-linux-amd64
   

3. Copy the binary file to /usr/bin/ and set the permissions.

   cp rclone /usr/bin
   chown root:root /usr/bin/rclone
   chmod 755 /usr/bin/rclone
   

4. Install the Rclone manpage.

   mkdir -p /usr/local/share/man/man1
   cp rclone.1 /usr/local/share/man/man1/
   mandb
   

   

Task 3: Configure Rclone for OCI Object Storage

1. Open Oracle Linux CLI and run rclone config command. This command will start an interactive setup process.

   

2. Create a new remote name for OCI Object Storage, type n in the menu, and then enter a new name for your OCI remote configuration (for example: oci-object-01).

   

3. Choose the storage type, select the number corresponding to Oracle Cloud Infrastructure Object Storage 34.

   

4. Choose the authentication provider to be utilized by Rclone. For example, we will utilize option 2.

   

5. Enter the object storage namespace, object storage compartment ID, object storage region and the endpoint for object storage API.

   

   Note: Endpoint for object storage API is optional. Leave blank to use the default endpoint for the region.

6. Enter the full path for the OCI config file and the config_profile information.

   

7. The advanced config options are not required, select n to accept the default parameters.

   

8. A summary of the Rclone configuration will be displayed, if everything looks right, select y to save the configuration.

   

9. Perform a connection test to OCI Object Storage bucket before proceeding to migration and bulk data transfer activities with an rclone ls command.

   rclone ls OCIStorage:your-bucket-name
   

An Example Migration from a Local Linux directory

For this example, we are utilizing the /source directory as the local storage source in our Oracle Linux server, and an OCI Object Storage bucket named test-bucket as the target. Users on a Windows platform can follow along by creating a folder named source and creating files in the folder to copy to the OCI Object Storage bucket.

1. We start with the empty OCI Object Storage bucket named test-bucket.

   

   Note: Buckets can be created with the OCI Console or OCI CLI.

2. There are five text files in the /source directory.

   

   Note: You can create text files with the touch command or the editor of your preference such as vim. If you are using the Windows platform, you can create files using Notepad in a folder named source and continue using the rclone commands.

3. Use the rclone copy command to transfer data from the /source folder on the Oracle Linux server to the OCI Object Storage bucket test-bucket.

   rclone copy /path/to/local/data OCIStorage:your-bucket-name
   

   For example:

   rclone copy /source/ oci:test-bucket
   

4. From the OCI Console, go to the OCI Object Storage details page for the bucket named test-bucket. The bucket will now contain the text files migrated from the /source directory on the Oracle Linux server.

   

5. Delete all files in the OCI Object Storage test-bucket with the rclone delete command.

   rclone delete oci:test-bucket
   

6. The example command below includes advanced options such as multi-threaded streams, progress of the data transfer, buffer size, and so on. Explore Rclone advanced options to determine which options best suit your use case.

   rclone -vvv --progress --stats-one-line --max-stats-groups 10 --log-format date,time,UTC,longfile --fast-list --buffer-size 256Mi --oos-no-check-bucket --oos-upload-cutoff 10Mi --multi-thread-cutoff 16Mi --multi-thread-streams 3000 --transfers 3000 --checkers 64 --retries 2 --oos-chunk-size 10Mi --oos-upload-concurrency 10000 --oos-attempt-resume-upload --oos-leave-parts-on-error sync /root/source/ oci:test-bucket
   

   Note: Regularly update Rclone to utilize the latest features and improvements.

Related Links

• OCI Object Storage

• OCI Object Storage documentation

• Configure Rclone for OCI Object Storage

• Using Rclone to copy data in and out of OCI Object Storage

Acknowledgments

• Author - Anderson Souza (Senior Director, Product Management)

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.

---

Title and Copyright Information

Migrate Data to Oracle Cloud Infrastructure Object Storage Using Rclone

F91151-01

January 2024

Copyright © 2024, Oracle and/or its affiliates.