- Move data to object storage in the cloud by using Rclone
- Configure Rclone for Object Storage
Configure Rclone for Object Storage
Note:
For the above referenced resources, see "Explore More", elsewhere in this playbook, for appropriate links.Set Up Authentication with an OCI User and API Key, Obtain the OCI Tenancy Namespace and Bucket Compartment
Generate an API signing pair, setup a config file, and obtain the tenancy namespace, compartment information, and the regional Object Storage API Endpoint.
- First generate an API signing key pair:
- View the user's details.
- If you're creating an API key for yourself, open the Profile menu (User menu icon) and click User Settings.
- If you're an administrator creating an API key for another user, in the Console, click Identity and then click Users. Locate the user in the list, and then click the user's name to view the details.
- On the left side of the page in the Resources section, click API Keys.
- Click Add API Key.
- Click Add API Key at the top left of the API Keys list. The Add API Key dialog appears.
- Click Download Private Key and save
the key to your
.oci
directory. In most cases, you do not need to download the public key.Note:
- If you do not have a
.oci
directory, create one in your home directory - If your browser downloads the private key to a
different directory, be sure to move it to your
.oci
directory
- If you do not have a
- Click Add.
The key is added to the OCI user's resources and the Configuration File Preview appears. Notice the profile is marked DEFAULT; you will need the profile name later when setting up the Rclone configuration. Copy and paste the configuration file snippet from the text box into a file into a file named config in your .oci directory. Update the the
key_file
path in the config file to match the location in step e, above. Save the path of file to setup the Rclone configuration.
- View the user's details.
- Obtain the Object Storage namespace string:
- Open the Profile menu and click Tenancy: <your_tenancy_name>. Your namespace string is listed under Object Storage Settings.
- Copy the namespace string, as you will need it when setting up Rclone.
- Determine the bucket compartment and region:
- In the OCI Console, navigate to Storage then Object Storage & Archive Storage then Buckets and find the name of the bucket where data will be migrated.
- Click the bucket name to view the details.
- Look in the upper left hand corner of the OCI Console page and note the region where your bucket is located and the region name. You will need this information when setting up Rclone.
- Click the name link next to Compartment to view the compartment details.
- On the Compartment details page, look under the Compartment Information tab and copy the compartment OCID. You will need this information when setting up Rclone,
Configure Rclone for OCI Object Storage
Next, you need to configure Rclone for OCI Object Storage. Use the following procedure.
- Navigate to your Linux system and run
rclone config
to start an interactive set-up process:$ rclone config No remotes found, make a new one? e) Edit existing remote n) New remote d) Delete remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config e/n/d/r/c/s/q>
- Select option
n) New remote
and, when prompted, enter a name for the new remote:Enter name for new remote. <enter your name here>
- The next prompt asks for the storage type and will present a list of options.
Select the number associated with OCI Object Storage (
oracleobjectstorage
):Option Storage. Type of storage to configure. Choose a number from below, or type in your own value. . . . 35 / OpenStack Swift (Rackspace Cloud Files, Blomp Cloud Storage, Memset Memstore, OVH) \ (swift) 36 / Oracle Cloud Infrastructure Object Storage \ (oracleobjectstorage) 37 / Pcloud \ (pcloud) <snip> Storage> 36
Note:
As of this writing, the option is number 36. This number changes with Rclone releases as they add additional storage choices. Be sure to verify that you are selecting the menu option for OCI Object Storage. - Select the authentication provider you want to use. For this example, option
2 OCI user and API Key
will be selected:Option provider. Choose your Auth Provider Choose a number from below, or type in your own string value. Press Enter for the default (env_auth). 1 / automatically pickup the credentials from runtime(env), first one to provide auth wins \ (env_auth) / use an OCI user and an API key for authentication. 2 | you’ll need to put in a config file your tenancy OCID, user OCID, region, the path, fingerprint to an API key. | https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdkconfig.htm \ (user_principal_auth) / use instance principals to authorize an instance to make API calls. 3 | each instance has its own identity, and authenticates using the certificates that are read from instance metadata. | https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/callingservicesfrominstances.htm \ (instance_principal_auth) / use workload identity to grant OCI Container Engine for Kubernetes workloads policy-driven access to OCI resources using OCI Identity and Access Management (IAM). 4 | https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contenggrantingworkloadaccesstoresources.htm \ (workload_identity_auth) 5 / use resource principals to make API calls \ (resource_principal_auth) 6 / no credentials needed, this is typically for reading public buckets \ (no_auth) provider> 2
- Enter the OCI Tenancy Object Storage namespace (which you created and recorded
when you performed step 2b in "Set Up Authentication with an OCI User and API
Key, Obtain the OCI Tenancy Namespace and Bucket Compartment"):
Option namespace. Object storage namespace Enter a value. namespace> namespace_name
- Enter the bucket compartment OCID (which you created and recorded when you
performed step 3e in "Set Up Authentication with an OCI User and API Key, Obtain
the OCI Tenancy Namespace and Bucket Compartment"):
Option compartment. Object storage compartment OCID Enter a value. compartment> ocid1.compartment.oc1xxx.xxxx.xxxx.xxxx
- Enter the bucket region (which you created and recorded when you performed step
3c in "Set Up Authentication with an OCI User and API Key, Obtain the OCI
Tenancy Namespace and Bucket Compartment"):
Option region. Object storage Region Enter a value. region> us-ashburn-1
- The next prompt is for the endpoint; leave this empty, which will select the
default region endpoint. Rclone will use the information collected to form the
full endpoint URL:
Option endpoint. Endpoint for Object storage API. Leave blank to use the default endpoint for the region. Enter a value. Press Enter to leave empty. endpoint>
- Enter the full path of the OCI config file location and config_profile
information (which you created and recorded when you performed step 1f in "Set
Up Authentication with an OCI User and API Key, Obtain the OCI Tenancy Namespace
and Bucket Compartment"), answer n for advanced
configuration:
Option config_file. Path to OCI config file Choose a number from below, or type in your own string value. Press Enter for the default (~/.oci/config). 1 / oci configuration file location \ (~/.oci/config) config_file> /home/username/.oci/config Option config_profile. Profile name inside the oci config file Choose a number from below, or type in your own string value. Press Enter for the default (Default). 1 / Use the default profile \ (Default) config_profile> DEFAULT Edit advanced config? y) Yes n) No (default) y/n> n
- If everything looks correct, Rclone will display a summary for the
configuration, select y to save the configuration:
Options: - type: oracleobjectstorage - provider: user_principal_auth - namespace: namespace - compartment: ocid1.compartment.oc1xxx.xxxx.xxxx.xxxx - region: us-ashburn-1 - config_file: /home/username/.oci/config - config_profile: DEFAULT Keep this "foo" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y Current remotes: Name Type ==== ==== remote_name oracleobjectstorage
- Exit the configuration interface:
e) Edit existing remote n) New remote d) Delete remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config e/n/d/r/c/s/q> q
- Test the Rclone connection by listing the buckets in the compartment:
$ rclone lsd remote_name: -1 2024-04-19 00:08:33 -1 bucket1