Note:

Use Oracle Cloud Infrastructure Object Storage Service with OpenStack Swift API

Introduction

Oracle Cloud Infrastructure (OCI) Object Storage is a data storage architecture that manages and organizes digital information as objects, each comprising data, metadata, and a unique identifier. Unlike traditional file or block storage, OCI Object Storage does not rely on a hierarchical structure, allowing for efficient and scalable handling of vast amounts of unstructured data. Objects are stored in a flat address space, simplifying data retrieval and enabling seamless scalability. This solution is ideal for diverse data types, such as multimedia content and backups, making it a robust choice for cloud environments, archival storage, and distributed systems due to its flexibility, durability, and ease of access. Users interact with the object storage service using a REST interface over the HTTP protocol.

The OCI Object Storage service is a cloud-based storage solution where Oracle handles the management complexity, ensuring dependable and cost-effective data durability. The platform elasticity allows for a gradual start with the ability to scale seamlessly, all without compromising performance or service reliability.

OCI Object Storage provides API compatibility with the OpenStack Swift API and Amazon Simple Storage Service (Amazon S3) through Amazon S3 Compatibility API. This allows customers to continue using their existing tools (for example, SDK clients), minimizing the need to make changes to their applications. For more information, see Amazon S3 Compatibility API. This tutorial will focus on the OCI OpenStack Swift compatible API consumption.

Before we start to discuss about the OpenStack Swift API, we need to clarify some of the constructs used in OCI Object Storage.

The OpenStack Swift defines three constructs: Account, Container and Object.

We can now identify equivalent resources between OCI Object Storage and OpenStack Swift API.

Objectives

Prerequisites

Use OCI Swift API

The OCI Swift API regional endpoints use a consistent URL format of https://swiftobjectstorage.<region-identifier>.oraclecloud.com. For example, the native OCI Swift API endpoint in US East region (us-ashburn-1) is https://swiftobjectstorage.us-ashburn-1.oraclecloud.com.

Considering the tenant level isolation at the region level, the OCI Swift API endpoint becomes: https://swiftobjectstorage.<region-identifier>.oraclecloud.com/<tenancy-namespace>, where <tenancy-namespace> is the auto-generated Object Storage namespace string of the tenancy in which to create repositories. For more information, see Tenancy Information page.

OCI Swift API lacks the concept of compartment. By default, buckets created using the OCI Amazon S3 Compatibility API or the OCI Swift API are created in the root compartment of the Oracle Cloud Infrastructure tenancy. You can designate a different compartment for the OCI Amazon S3 Compatibility API or OCI Swift API to create buckets. For more information, see Editing Tenancy’s Amazon S3 Compatibility API and Swift API Compartment Designations.

OCI Swift API supports three approaches to authenticate requests.

Approach 1: Use Basic Auth Header

The username has the format <username>. For example, john.doe@acme.com. If your tenancy is federated with Oracle Identity Cloud Service, use the format oracleidentitycloudservice/<username>.

The password is the auth token generated for the user. For example, my-auth-token. For more information, see Getting an Auth Token.

Each request to the OCI Swift API should include the following header.

For example, if username is oracleidentitycloudservice/john.doe@acme.com and password is my-auth-token, the header will look like:

We will use some of the API calls described in the official documentation, see OpenStack Swift API.

Example:

  1. Initialize the following required environment variables.

    user='<username>'
password='<password>'
region='<oci-region>'
tenancy_namespace='<tenancy-namespace>'
bucket_name='tutorial-bucket'
object_name='sample.txt'

  2. List buckets in the account, only buckets from the selected OCI Swift API compartments will be listed.

    curl -sS -D /dev/stderr "https://swiftobjectstorage.${region}.oraclecloud.com/v1/${tenancy_namespace}" -u "${user}:${password}" | jq

  3. Create a bucket.

    curl -sS -i "https://swiftobjectstorage.${region}.oraclecloud.com/v1/${tenancy_namespace}/${bucket_name}" -X PUT -u "${user}:${password}"

  4. Upload a file to the bucket.

    echo 'sample file' > ${object_name}
curl -s -i "https://swiftobjectstorage.${region}.oraclecloud.com/v1/${tenancy_namespace}/${bucket_name}/${object_name}" -X PUT --data-binary "@${object_name}" -u "${user}:${password}"

  5. List the bucket content.

    curl -sS -D /dev/stderr "https://swiftobjectstorage.${region}.oraclecloud.com/v1/${tenancy_namespace}/${bucket_name}" -u "${user}:${password}" | jq

  6. Fetch a file from the bucket.

    curl -s -D /dev/stderr "https://swiftobjectstorage.${region}.oraclecloud.com/v1/${tenancy_namespace}/${bucket_name}/${object_name}" -u "${user}:${password}" -o downloaded_file.txt
cat downloaded_file.txt

  7. Delete the file from the bucket.

    curl -sS -i "https://swiftobjectstorage.${region}.oraclecloud.com/v1/${tenancy_namespace}/${bucket_name}/${object_name}" -X DELETE -u "${user}:${password}"

  8. Delete the bucket.

    curl -s -i "https://swiftobjectstorage.${region}.oraclecloud.com/v1/${tenancy_namespace}/${bucket_name}" -X DELETE -u "${user}:${password}"

Approach 2: Token Based Authentication using v1 Auth

You will have to call the OCI Swift API authorization endpoint for the region and include the X-Storage-User and X-Storage-Pass headers in the request. If the authentication request is successful, you will receive a response with X-Storage-Token header that can be used for request authorization, and X-Storage-Url header with the OCI Swift API endpoint for the account.

The X-Storage-User header has the format <tenancy-namespace>:<username>.

For example, if the tenancy-namespace is axaxnpcrorw5, and the username is the same as Approach 1, the value of the header is axaxnpcrorw5:oracleidentitycloudservice/john.doe@acme.com.

The X-Storage-Pass is the generated auth token.

Example:

  1. Initialize the following required environment variables.

    user='<username>'
password='<password>'
region='<oci-region>'
tenancy_namespace='<tenancy-namespace>'
bucket_name='tutorial-bucket'
object_name='sample.txt'

  2. Generate a token.

    curl -sS -D swift_headers.txt https://swiftobjectstorage.eu-frankfurt-1.oraclecloud.com/auth/v1.0 -H "X-Storage-User: ${tenancy_namespace}:${user}" -H "X-Storage-Pass: ${password}"
X_Auth_Token=$(cat swift_headers.txt | grep X-Auth-Token | cut -d":" -f 2)

  3. List buckets in the account, only buckets from the selected OCI Swift API compartments will be listed.

    curl -s -D /dev/stderr "https://swiftobjectstorage.${region}.oraclecloud.com/v1/${tenancy_namespace}" -H "X-Auth-Token: ${X_Auth_Token}" | jq

  4. Create a bucket.

    curl -s -i "https://swiftobjectstorage.${region}.oraclecloud.com/v1/${tenancy_namespace}/${bucket_name}" -X PUT -H "X-Auth-Token: ${X_Auth_Token}"

  5. Upload a file to the bucket.

    echo 'sample file' > ${object_name}
curl -s -i "https://swiftobjectstorage.${region}.oraclecloud.com/v1/${tenancy_namespace}/${bucket_name}/${object_name}" -X PUT --data-binary "@${object_name}" -H "X-Auth-Token: ${X_Auth_Token}"

  6. List the bucket content.

    curl -s -D /dev/stderr "https://swiftobjectstorage.${region}.oraclecloud.com/v1/${tenancy_namespace}/${bucket_name}" -H "X-Auth-Token: ${X_Auth_Token}" | jq

  7. Fetch a file from the bucket.

    curl -s -D /dev/stderr "https://swiftobjectstorage.${region}.oraclecloud.com/v1/${tenancy_namespace}/${bucket_name}/${object_name}" -H "X-Auth-Token: ${X_Auth_Token}" -o downloaded_file.txt
cat downloaded_file.txt

  8. Delete the file from the bucket.

    curl -s -i "https://swiftobjectstorage.${region}.oraclecloud.com/v1/${tenancy_namespace}/${bucket_name}/${object_name}" -X DELETE -H "X-Auth-Token: ${X_Auth_Token}"

  9. Delete the bucket.

    curl -s -i "https://swiftobjectstorage.${region}.oraclecloud.com/v1/${tenancy_namespace}/${bucket_name}" -X DELETE -H "X-Auth-Token: ${X_Auth_Token}"

Approach 3: Token Based Authentication using v2 Auth

To illustrate the OCI Swift API v2 authorization procedure, we will use the official OpenStack Swift API Python package python-swiftclient 4.5.0, and the required dependencies for the v2 auth python-keystoneclient 5.4.0.

Example:

  1. Run the following command to ensure you have access to a Python version above 3.6. For more information, see Download Python.

    python3 --version

  2. Install the required Python modules.

    python3 -m pip install python-swiftclient python-keystoneclient click

  3. You may use the following code to interact with the OCI Swift API v2. Make sure to replace the placeholders for <user>, <password>, <region> and <tenancy-namespace> in the following code before run.

    import click
import os

from swiftclient.client import Connection
from swiftclient.exceptions import ClientException

user='<username>'
password='<password>'
region='<oci-region>'
tenancy_namespace='<tenancy-namespace>'
bucket_name='tutorial-bucket'
object_name='sample.txt'

_authurl = f'https://swiftobjectstorage.{region}.oraclecloud.com/auth/v2.0/'
_auth_version = '2'
_user = f'{user}'
_key = f'{password}'
_tenant_name = f'{tenancy_namespace}'

conn = Connection(
    authurl=_authurl,
    user=_user,
    key=_key,
    tenant_name=_tenant_name,
    auth_version=_auth_version
)

@click.group(invoke_without_command=True)
@click.pass_context
def main(ctx):
    available_commands = {
        "a": ("List buckets in the account", get_account),
        "b": ("Create a new bucket", put_container),
        "c": ("Upload an object to the bucket", put_object),
        "d": ("List objects in the bucket", get_container),
        "e": ("Download object from the bucket", get_object),
        "f": ("Delete object from the bucket", delete_object),
        "g": ("Delete bucket from the account", delete_container)
    }
    while True:
        click.echo("Available actions")
        for index, command in available_commands.items():
            click.echo(f"{index} - {command[0]}")
        action = click.prompt("Please select an option from the list")
        if action in available_commands.keys():
            ctx.invoke(available_commands[action][1])
        print("###"*10)

@main.command()
def get_account(conn=conn):
    print("Listing buckets in the account.")
    try:
        resp_headers = conn.get_account()
        for entry in resp_headers[1]:
            print(entry["name"])
    except ClientException as e:
        print(f"Failed to get the buckets in the tenancy. Error: {e}")
        raise
    else:
        print(f"Successfuly listed bucket in the '{_tenant_name}' account.")

@main.command()
def put_container(conn=conn, bucket_name=bucket_name):
    print(f"Creating a new bucket named {bucket_name}.")
    try:
        resp_headers = conn.put_container(container=bucket_name)
    except ClientException as e:
        print(f"Failed to create the new bucket named {bucket_name} in the tenancy. Error: {e}")
        raise
    else:
        print(f"The '{bucket_name}' was successfuly created.")

@main.command()
def put_object(conn=conn, bucket_name=bucket_name, object_name=object_name):
    print(f"Uploading a new file, '{object_name}' to the new bucket '{bucket_name}'.")
    try:
        if not os.path.isfile(object_name):
            with open(object_name, "w") as f:
                f.write("sample file")
        with open(object_name) as f:
            resp_headers = conn.put_object(container=bucket_name, obj=object_name, contents=object_name)
    except ClientException as e:
        print(f"Failed to create a new object named '{object_name}' in the bucket '{bucket_name}'. Error: {e}")
        raise
    else:
        print(f"The '{object_name}' file was successfuly uploaded to '{bucket_name}' bucket.")

@main.command()
def get_container(conn=conn, bucket_name=bucket_name):
    print(f"List {bucket_name} bucket contents.")
    try:
        resp_headers = conn.get_container(container=bucket_name)
        for entry in resp_headers[1]:
            print(entry["name"])
    except ClientException as e:
        print(f"Failed to list objects in the bucket {bucket_name}. Error: {e}")
        raise
    else:
        print(f"The '{bucket_name}' content was successfuly listed.")

@main.command()
def get_object(conn=conn, bucket_name=bucket_name, object_name=object_name, dest_file="downloaded_file.txt"):
    print(f"Fetch {object_name} object from the bucket {bucket_name}.")
    try:
        resp_headers = conn.get_object(container=bucket_name, obj=object_name)
        with open(dest_file, "wb") as f:
            f.write(resp_headers[1])
    except ClientException as e:
        print(f"Failed to download {object_name} from {bucket_name} to local file named '{dest_file}'. Error: {e}")
        raise
    else:
        print(f"The '{object_name}' object was saved locally to '{dest_file}'.")

@main.command()
def delete_object(conn=conn, bucket_name=bucket_name, object_name=object_name):
    print(f"Deleting {object_name} object from the bucket {bucket_name}.")
    try:
        resp_headers = conn.delete_object(container=bucket_name, obj=object_name)
    except ClientException as e:
        print(f"Failed to delete {object_name} from {bucket_name}. Error: {e}")
        raise
    else:
        print(f"The '{object_name}' object was deleted.")

@main.command()
def delete_container(conn=conn, bucket_name=bucket_name):
    print(f"Deleting the bucket {bucket_name}.")
    try:
        resp_headers = conn.delete_container(container=bucket_name)
    except ClientException as e:
        print(f"Failed to delete {object_name} from {bucket_name}. Error: {e}")
        raise
    else:
        print(f"The '{bucket_name}' bucket was deleted.")


if __name__ == "__main__":
    main()

Acknowledgments

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.