Configure Rclone for Object Storage

You can use Rclone with the native OCI Object Storage API to migrate data into OCI Object Storage Buckets. Have a target migration bucket in mind, or see the tutorial Put data into object storage to learn how to create an OCI Object Storage bucket.

The first steps include obtaining proper credentials and the API endpoint. This solution will use Rclone on a Linux host and authentication with an OCI user plus API key, see Rclone Oracle Object Storage documentation for additional authorization methods and information on how to download and install Rclone.

Note:

For the above referenced resources, see "Explore More", elsewhere in this playbook, for appropriate links.

Set Up Authentication with an OCI User and API Key, Obtain the OCI Tenancy Namespace and Bucket Compartment

Generate an API signing pair, setup a config file, and obtain the tenancy namespace, compartment information, and the regional Object Storage API Endpoint.

1. First generate an API signing key pair:
   1. View the user's details.
      • If you're creating an API key for yourself, open the Profile menu (User menu icon) and click User Settings.
      • If you're an administrator creating an API key for another user, in the Console, click Identity and then click Users. Locate the user in the list, and then click the user's name to view the details.
   2. On the left side of the page in the Resources section, click API Keys.
   3. Click Add API Key.
   4. Click Add API Key at the top left of the API Keys list. The Add API Key dialog appears.
   5. Click Download Private Key and save the key to your .oci directory. In most cases, you do not need to download the public key.

      Note:

      • If you do not have a .oci directory, create one in your home directory
      • If your browser downloads the private key to a different directory, be sure to move it to your .oci directory

      .
   6. Click Add.

      The key is added to the OCI user's resources and the Configuration File Preview appears. Notice the profile is marked DEFAULT; you will need the profile name later when setting up the Rclone configuration. Copy and paste the configuration file snippet from the text box into a file into a file named config in your .oci directory. Update the the key_file path in the config file to match the location in step e, above. Save the path of file to setup the Rclone configuration.

2. Obtain the Object Storage namespace string:
   1. Open the Profile menu and click Tenancy: <your_tenancy_name>. Your namespace string is listed under Object Storage Settings.
   2. Copy the namespace string, as you will need it when setting up Rclone.
3. Determine the bucket compartment and region:
   1. In the OCI Console, navigate to Storage then Object Storage & Archive Storage then Buckets and find the name of the bucket where data will be migrated.
   2. Click the bucket name to view the details.
   3. Look in the upper left hand corner of the OCI Console page and note the region where your bucket is located and the region name. You will need this information when setting up Rclone.
   4. Click the name link next to Compartment to view the compartment details.
   5. On the Compartment details page, look under the Compartment Information tab and copy the compartment OCID. You will need this information when setting up Rclone,

Configure Rclone for OCI Object Storage

Next, you need to configure Rclone for OCI Object Storage. Use the following procedure.

1. Navigate to your Linux system and run rclone config to start an interactive set-up process:

   $ rclone config
   No remotes found, make a new one?
   e) Edit existing remote
   n) New remote
   d) Delete remote
   r) Rename remote
   c) Copy remote
   s) Set configuration password
   q) Quit config
   e/n/d/r/c/s/q>

2. Select option n) New remote and, when prompted, enter a name for the new remote:

   Enter name for new remote.
   <enter your name here>

3. The next prompt asks for the storage type and will present a list of options. Select the number associated with OCI Object Storage (oracleobjectstorage):

   Option Storage.
   Type of storage to configure.
   Choose a number from below, or type in your own value.
   .
   .
   .
   35 / OpenStack Swift (Rackspace Cloud Files, Blomp Cloud Storage, Memset Memstore, OVH)
      \ (swift)
   36 / Oracle Cloud Infrastructure Object Storage
      \ (oracleobjectstorage)
   37 / Pcloud
      \ (pcloud)
   <snip>
   Storage> 36

   Note:

   As of this writing, the option is number 36. This number changes with Rclone releases as they add additional storage choices. Be sure to verify that you are selecting the menu option for OCI Object Storage.
4. Select the authentication provider you want to use. For this example, option 2 OCI user and API Key will be selected:

   Option provider.
   Choose your Auth Provider
   Choose a number from below, or type in your own string value.
   Press Enter for the default (env_auth).
    1 / automatically pickup the credentials from runtime(env), first one to provide auth wins
      \ (env_auth)
      / use an OCI user and an API key for authentication.
    2 | you’ll need to put in a config file your tenancy OCID, user OCID, region, the path, fingerprint to an API key.
      | https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdkconfig.htm
      \ (user_principal_auth)
      / use instance principals to authorize an instance to make API calls.
    3 | each instance has its own identity, and authenticates using the certificates that are read from instance metadata.
      | https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/callingservicesfrominstances.htm
      \ (instance_principal_auth)
      / use workload identity to grant OCI Container Engine for Kubernetes workloads policy-driven access to OCI resources using OCI Identity and Access Management (IAM).
    4 | https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contenggrantingworkloadaccesstoresources.htm
      \ (workload_identity_auth)
    5 / use resource principals to make API calls
      \ (resource_principal_auth)
    6 / no credentials needed, this is typically for reading public buckets
      \ (no_auth)
   provider> 2

5. Enter the OCI Tenancy Object Storage namespace (which you created and recorded when you performed step 2b in "Set Up Authentication with an OCI User and API Key, Obtain the OCI Tenancy Namespace and Bucket Compartment"):

   Option namespace.
   Object storage namespace
   Enter a value.
   namespace> namespace_name

6. Enter the bucket compartment OCID (which you created and recorded when you performed step 3e in "Set Up Authentication with an OCI User and API Key, Obtain the OCI Tenancy Namespace and Bucket Compartment"):

   Option compartment.
   Object storage compartment OCID
   Enter a value.
   compartment> ocid1.compartment.oc1xxx.xxxx.xxxx.xxxx

7. Enter the bucket region (which you created and recorded when you performed step 3c in "Set Up Authentication with an OCI User and API Key, Obtain the OCI Tenancy Namespace and Bucket Compartment"):

   Option region.
   Object storage Region
   Enter a value.
   region> us-ashburn-1

8. The next prompt is for the endpoint; leave this empty, which will select the default region endpoint. Rclone will use the information collected to form the full endpoint URL:

   Option endpoint.
   Endpoint for Object storage API.
   Leave blank to use the default endpoint for the region.
   Enter a value. Press Enter to leave empty.
   endpoint>

9. Enter the full path of the OCI config file location and config_profile information (which you created and recorded when you performed step 1f in "Set Up Authentication with an OCI User and API Key, Obtain the OCI Tenancy Namespace and Bucket Compartment"), answer n for advanced configuration:

   Option config_file.
   Path to OCI config file
   Choose a number from below, or type in your own string value.
   Press Enter for the default (~/.oci/config).
    1 / oci configuration file location
      \ (~/.oci/config)
   config_file> /home/username/.oci/config
    
   Option config_profile.
   Profile name inside the oci config file
   Choose a number from below, or type in your own string value.
   Press Enter for the default (Default).
    1 / Use the default profile
      \ (Default)
   config_profile> DEFAULT
    
   Edit advanced config?
   y) Yes
   n) No (default)
   y/n> n

10. If everything looks correct, Rclone will display a summary for the configuration, select y to save the configuration:

    
    Options:
    - type: oracleobjectstorage
    - provider: user_principal_auth
    - namespace: namespace
    - compartment: ocid1.compartment.oc1xxx.xxxx.xxxx.xxxx
    - region: us-ashburn-1
    - config_file: /home/username/.oci/config
    - config_profile: DEFAULT
    Keep this "foo" remote?
    y) Yes this is OK (default)
    e) Edit this remote
    d) Delete this remote
    y/e/d> y
     
    Current remotes:
     
    Name                 Type
    ====                 ====
    remote_name          oracleobjectstorage

11. Exit the configuration interface:

    e) Edit existing remote
    n) New remote
    d) Delete remote
    r) Rename remote
    c) Copy remote
    s) Set configuration password
    q) Quit config
    e/n/d/r/c/s/q> q

12. Test the Rclone connection by listing the buckets in the compartment:

    $ rclone lsd remote_name:
              -1 2024-04-19 00:08:33        -1 bucket1