After your configuration is complete, you can start the transfer. Depending on the amount of data and the connection speed, this transfer can take a very long time (days or weeks).
To monitor progress, we recommend starting the transfer in verbose mode from GNU Screen, so you can run the process in the background and watch its progress as needed. GNU Screen is preinstalled with most Linux distributions.
Move the Data
Start the transfer in verbose mode from GNU Screen, so you can run the process in the background and watch its progress as needed.
- Start the GNU Screen:
- press Enter on the next screen to get a new shell.
- Start this screen in verbose mode. Replace
<destination_bucket>as needed (a bucket will be created automatically if one doesn't already exist).
$ rclone --verbose --cache-workers 64 --transfers 64 --retries 32 copy $SOURCE oci:<destination_bucket>
- While the transfer is ongoing, detach from GNU Screen by pressing Ctrl+a d
- To get back to GNU Screen, enter:
$ screen -r
- If you want to detach again, press Ctrl+a d.
Note:The optimal number of cache-workers and transfers depends on your specific use case, available bandwidth, resources available on the workstation or server, and throttling limits applied by your cloud provider. In general, the higher the number of transfers and cache-workers, the faster the transfer will complete. But beyond a certain number, you might start getting errors. We recommend that you start with 64 concurrent transfers and cache-workers and adjust the number as necessary,
When the transfer is complete, verify that no errors occurred. If you see errors, try decreasing the number of cache-workers and transfers and try again.
Verify the Transfer
Independent of how data is transferred to Object Storage (whether from a local file system or another cloud provider, or by some other means), you can use rclone to verify whether the data copied to Object Storage is identical to its source. You can verify the data quickly or more thoroughly.
$ rclone --verbose --cache-workers 64 --transfers 64 --retries 32 check $SOURCE oci:<destination_bucket>
This method detects differences in the source and destination buckets by verifying file sizes and MD5 hashes. However, MD5 hashes can’t be used for objects that are uploaded by using multipart uploads. As a result, large objects are verified only by comparing their file sizes, which provides only a cursory verification.
Although this method performs a more thorough comparison, it requires downloading copies from both object stores.
$ rclone --verbose --cache-workers 64 --transfers 64 --retries 32 check --download $SOURCE oci:<destination_bucket>
Note:When you perform a verification with downloads, depending on where your job is run, you might incur additional data transfer fees from your existing cloud provider, Oracle, or both.