Amazon S3 Compatibility API

Using the Amazon S3 Compatibility API, customers can continue to use their existing Amazon S3 tools (for example, SDK clients) and make minimal changes to their applications to work with Object Storage. The Amazon S3 Compatibility API and Object Storage datasets are congruent. If data is written to the Object Storage using the Amazon S3 Compatibility API, the data can be read back using the native Object Storage API and conversely.

Differences between the Object Storage API and the Amazon S3 Compatibility API

The Object Storage Service provided by Oracle Cloud Infrastructure and Amazon S3 use similar concepts and terminology. In both cases, data is stored as objects in buckets. The differences are in the implementation of features and tools for working with objects.

The following highlights the differences between the two storage technologies:

  • Compartments

    Amazon S3 doesn't use compartments. By default, buckets created using the Amazon S3 Compatibility API or the Swift API are created in the root compartment of the Oracle Cloud Infrastructure tenancy. Instead, you can designate a different compartment for the Amazon S3 Compatibility API or Swift API to create buckets in.

  • Global bucket namespace

    Object Storage doesn't use a global bucket namespace. In Object Storage, namespace serves as the top-level container for all buckets and objects. At account creation time, each Oracle Cloud Infrastructure tenant is assigned one unique system-generated and immutable Object Storage namespace name. The namespace spans all compartments within a region. You control bucket names, but those bucket names must be unique within a namespace. While the namespace is region-specific, the namespace name itself is the same in all regions. You can have a bucket named MyBucket in US West (Phoenix) and a bucket named MyBucket in Germany Central (Frankfurt).

  • Encryption

    The Oracle Cloud Infrastructure Object Storage service encrypts all data at rest by default. Encryption can't be turned on or off using the API.

  • Object Level Access Control Lists (ACLs)

    Oracle Cloud Infrastructure does not use ACLs for objects. Instead, an administrator needs to set up groups , compartments , and policies  that control which users can access which services, which resources, and the type of access. For example, the policies control who can create users and groups, create buckets, download objects, and manage Object Storage-related policies and rules.

For more information, see Overview of the Object Storage service.

Amazon S3 Compatibility API Prerequisites

To enable application access from Amazon S3 to Object Storage, you need to set up access to Oracle Cloud Infrastructure and modify your application.

Setting up access to Oracle Cloud Infrastructure

  1. Sign Up for Oracle Cloud Infrastructure and obtain a unique namespace.
  2. Any user of the Amazon S3 Compatibility API with Object Storage needs permission to work with the service. If you're not sure if you have permission, contact your administrator. For basic information about policies, see How Policies Work. For policies that enable use of Object Storage, see Common Policies and the Policy Reference.
  3. Use an existing or create a Customer Secret Key. A Customer Secret Key consists of an Access Key/Secret Key pair. See Working with Customer Secret Keys for details. To use or create the key pair:

Modifying your application

  1. Configure a new endpoint for the application that includes the namespace name and the region identifier. For example: mynamespace.compat.objectstorage.us-phoenix-1.oraclecloud.com.

  2. Set the target region as one of the Oracle Cloud Infrastructure regions.

    Important

    If your application does not support setting the region identifier to the correct Oracle Cloud Infrastructure identifier, you must either set the region to us-east-1 or leave it blank. Using this configuration, you can only use the Amazon S3 Compatibility API in your Oracle Cloud Infrastructure home region. If you can manually set the region, you can use the application against any Oracle Cloud Infrastructure region.
  3. Configure the application to use the Customer Secret key. The Customer Secret Key consists of an Access Key and Secret Key. Both of these keys must be supplied to the application.
  4. Use path-based access in your application. Virtual host-style access (accessing a bucket as bucketname.namespace.compat.objectstorage.region.oraclecloud.com) is not supported.

You can now use the Amazon S3 Compatibility API to access Object Storage in Oracle Cloud Infrastructure.

Amazon S3 Compatibility API Support

Amazon S3 Compatibility API support is provided at the bucket level and object level.

Bucket APIs

The following bucket APIs are supported:

Object APIs

The following object APIs are supported:

Multipart Upload APIs

The following multipart upload APIs are supported:

Tagging APIs

The following tagging APIs are supported:

SSE-C Support

Using optional API headers, you can provide your own 256-bit AES encryption key that is used to encrypt and decrypt objects uploaded to and downloaded from Object Storage.

If you want to use your own keys for server-side encryption, specify the following three request headers with the encryption key information:

Headers Description APIs Supported
x-amz-server-side-encryption-customer-algorithm Specifies "AES256" as the encryption algorithm.

GetObject

HeadObject

PutObject

InitiateMultipartUpload

UploadPart

x-amz-server-side-encryption-customer-key Specifies the base64-encoded 256-bit encryption key to use to encrypt or decrypt the data.
x-amz-server-side-encryption-customer-key-md5 Specifies the base64-encoded 128-bit MD5 digest of the encryption key. This value is used to check the integrity of the encryption key.

Object Storage has distinct APIs for copying objects and copying parts. Amazon S3 uses the presence of the following headers in PutObject and UploadPart to determine copy operations. To copy a source object that is encrypted with an SSE-C key, you must specify these three headers so that Object Storage can decrypt the object.

Headers Description APIs Supported
x-amz-copy-source-server-side-encryption-customer-algorithm Specifies "AES256" as the encryption algorithm to use to decrypt the source object.

PutObject

UploadPart

x-amz-copy-source-server-side-encryption-customer-key Specifies the base64-encoded 256-bit encryption key to use to decrypt the source object.
x-amz-copy-source-server-side-encryption-customer-key-md5 Specifies the base64-encoded 128-bit MD5 digest of the encryption key used to decrypt the source object.

Supported Amazon S3 Clients

You can configure various client applications to talk to Object Storage's Amazon S3-compatible endpoints. This topic provides some configuration examples for supported Amazon S3 Clients. Review the prerequisites in Amazon S3 Compatibility API Prerequisites.

AWS SDK for Java

The AWS SDK for Java repository, file download, and documentation links are available on GitHub: https://github.com/aws/aws-sdk-java.

Here is an example of configuring AWS SDK for Java to use Object Storage

                // Put the Access Key and Secret Key here
                
AWSCredentialsProvider credentials = new AWSStaticCredentialsProvider(new BasicAWSCredentials(
 "gQ4+YC530sBa8qZI6WcbUbtH8oar0exampleuniqueID",
 "7fa22331ebe62bf4605dc9a42aaeexampleuniqueID"))));

// Your namespace
String namespace = "namespace";

// The region to connect to
String region = "us-ashburn-1";

// Create an S3 client pointing at the region
String endpoint = String.format("%s.compat.objectstorage.%s.oraclecloud.com",namespace,region);
AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder.EndpointConfiguration(endpoint, region);
AmazonS3 client = AmazonS3Client.builder()
 .standard()
 .withCredentials(credentials)
 .withEndpointConfiguration(endpointConfiguration)
 .disableChunkedEncoding()
 .enablePathStyleAccess()
 .build();

AWS SDK for Javascript

The AWS SDK for Javascript repository, documentation links, and installation instructions are available on GitHub: https://github.com/aws/aws-sdk-js.

Here is an example of configuring AWS SDK for Javascript to use Object Storage
s3 = new AWS.S3({
  region: 'us-ashburn-1',
  endpoint: 'https://' + mynamespace + '.compat.objectstorage.us-ashburn-1.oraclecloud.com',
  accessKeyId: 'gQ4+YC530sBa8qZI6WcbUbtH8oar0exampleuniqueID',
  secretAccessKey: '7fa22331ebe62bf4605dc9a42aaeexampleuniqueID',
  s3ForcePathStyle: true,
  signatureVersion: 'v4',
});

AWS SDK for Python (Boto3)

The AWS SDK for Python (Boto3) repository, documentation links, and installation instructions are available on GitHub: https://github.com/boto/boto3.

Here is an example of configuring AWS SDK for Python to use Object Storage
import boto3
  
s3 = boto3.resource(
    's3',
    aws_access_key_id="gQ4+YC530sBa8qZI6WcbUbtH8oar0exampleuniqueID",
    aws_secret_access_key="7fa22331ebe62bf4605dc9a42aaeexampleuniqueID",
    region_name="us-phoenix-1", # Region name here that matches the endpoint
    endpoint_url="https://mynamespace.compat.objectstorage.us-phoenix-1.oraclecloud.com" # Include your namespace in the URL
)
  
# Print out the bucket names
for bucket in s3.buckets.all():
    print bucket.name

Mounting Object Storage buckets using s3fs

s3fs lets Linux and macOS mount Object Storage as a file system. The s3fs repository, documentation links, installation instructions, and examples are available on GitHub: https://github.com/s3fs-fuse/s3fs-fuse.

s3fs is not suitable for all applications. Understand the following limitations:

  • Object storage services have high latency compared to local file systems for time to first-byte and lack random write access. s3fs achieves the best throughput on workloads that only read large files.
  • You cannot partially update a file, so changing a single byte requires uploading the entire file.
  • Random writes or appends to files require rewriting the entire file.
  • s3fs does not support partial downloads, so even if you only want to read one byte of a file, you need to download the entire file.
  • s3fs does not support server-side file copies. Copied files must first be downloaded to the client and then uploaded to the new location.
  • Metadata operations, such as listing directories, have poor performance because of network latency.
  • s3fs does not support hard links or the atomic renames of files or directories.
  • s3fs provides no coordination between multiple clients mounting the same bucket.
To mount an Object Storage bucket as a file system
  1. Follow the installation instructions provided on GitHub: https://github.com/s3fs-fuse/s3fs-fuse.

    If you are unable to install using a pre-built package, follow the compilation instructions here: https://github.com/s3fs-fuse/s3fs-fuse/blob/master/COMPILATION.md.

  2. Review and perform the prerequisites in Amazon S3 Compatibility API Prerequisites. You need an Access Key/Secret Key pair and a proper IAM policy that lets you mount a bucket as a file system. For example:
    Allow group s3fsAdmins to manage object-family in compartment MyCompartment
  3. Enter your Access Key/Secret Key pair credentials in a ${HOME}/.passwd-s3fs credential file:
    cat ${HOME}/.passwd-s3fs
    <access_key>:<secret_key>

    For example:

    cat ${HOME}/.passwd-s3fs
    gQ4+YC530sBa8qZI6WcbUbtH8oar0exampleuniqueID:7fa22331ebe62bf4605dc9a42aaeexampleuniqueID

    Then, set owner-only permissions for the credential file:

    chmod 600 ${HOME}/.passwd-s3fs
  4. Create a mount point to mount an Object Storage bucket:
    mkdir /path/to/<local_directory_name>
    s3fs <bucket_name> <local_directory_name> -o passwd_file=${HOME}/.passwd-s3fs -o url=https://<namespace_name>.compat.objectstorage.<region_ID>.oraclecloud.com -o use_path_request_style [-o endpoint=<region_ID>]

    Where:

    • <bucket_name> is the name of the bucket that you want to mount.
    • <local_directory_name> is the name of the local directory where you want to mount the bucket.
    • <namespace_name> is the unique system-generated assigned to your tenancy at account creation time. You can use the CLI or the Console to obtain your namespace name. See Understanding Object Storage Namespaces for details.
    • <region_ID> is the region identifier where the bucket resides. See Regions and Availability Domains for details.
    • endpoint: If you want to mount a bucket that was created in your home region, you do not need to specify the endpoint parameter. If you want to mount a bucket that was created in a different region, you need to specify the endpoint parameter.
  5. If you want to automatically mount the bucket as a file system on system startup using s3fs, add following to the /etc/fstab file:

    <bucket_name> /path/to/<local_directory_name> fuse.s3fs use_path_request_style,passwd_file=/root/.s3fs-password,url=https://<namespace_name>.compat.objectstorage.<region_ID>.oraclecloud.com,endpoint=<region_ID>
  6. To verify the s3fs bucket mount, run the df -h command. The output shows the new mount point for the bucket. Navigate to the new mount point and run the ls command to list all objects in the bucket.

To troubleshoot mounting an Object Storage bucket
  • If you get authorization errors, review your IAM policies and ensure you have one that lets you mount a bucket as a file system. For example:
    Allow group s3fsAdmins to manage object-family in compartment MyCompartment
  • Ensure that you are using the correct namespace name in the URL in the s3fs command. To verify your namespace name, see Understanding Object Storage Namespaces.
  • Ensure that the named bucket that you are trying to mount exists and is in a compartment that you have access to. Use one of the following ways to verify the bucket name:
    • Log into the Console and find the named bucket is in the compartment that you have access to.
    • Use the CLI command oci os bucket list --namespace <object_storage_namespace> --compartment-id <target_compartment_id>.
  • If you are trying to mount a bucket that was created in a region other than your home region, you need to specify that other region in both the url and endpoint parameters.
  • If you mount a bucket as the root user, other users are not able to list or access objects in the bucket unless you add -o allow_other to the s3fs command or allow_other to the /etc/fstab mount options. You can also supply specific UID and GID parameters to specify user access details.
  • If you reviewed and verified the troubleshooting solutions and need to contact Support, run the mount command again in DEBUG mode to get more failure details. Add the following to the end of the command and save the output:
    -o dbglevel=info -f -o curldbg
To unmount an Object Storage bucket from a file system
Run the following command, specifying the mount point:
umount /path/to/<local_directory_name>