Upload a Large File

A file that's larger than 5 GB is considered a large object. A single object can hold up to 5 GB of data, but multiple objects can be linked together to hold more than 5 GB of contiguous data. You can create small objects as segments and upload them as one large object by using a manifest object.

Note:

A large object can have a maximum of 2048 segments. Each segment can be up to 5 GB. The maximum size of a file that you can upload to Oracle Cloud Infrastructure Object Storage Classic as a large object is 10 TB.

A user with the Service Administrator role or a role that is specified in the X-Container-Write ACL of the container can perform this task.

You can upload a large object by using the REST API.

1. Segment the large file locally into multiple sequential segment files, each smaller than 5 GB.

   On Linux, for example, you can use the following command:

   split -b segment_size file_name segment_name

2. List all the segment files.

   ls —al segment_name*

3. Create objects from each segment file. Upload all the objects in the same container.

   curl -v -X PUT \
           -H "X-Auth-Token:token" \
           -T segmentName \
           accountURL/containerName/objectName

   • token: The authentication token obtained from Oracle Cloud Infrastructure Object Storage Classic
   • segmentName: The full path and name of the segment file to be uploaded
   • containerName: The name of the container in which the object should be created
   • objectName: The name of the object to be created, which is same as the corresponding segment file name
4. Create a manifest file in JSON format, and ensure that the manifest file contains the following attributes for each segment object:
   • path: The container and object name in the format:

     containerName/segmentObjectName

   • etag: MD5 checksum of the segment object.

     You can find the value in the Etag header of the segment object.

   • size_bytes: Size of the segment object.

     You can find the value in the Content-Length header of the segment object.

   Sample Manifest File

   [
       {
           "path": "FirstContainer/segment_aa",
           "etag": "f1c9645dbc14efddc7d8a322685f26eb",
           "size_bytes": 10485760
       },
       {
           "path": "FirstContainer/segment_ab",
           "etag": "f1c9645dbc14efddc7d8a322685f26eb",
           "size_bytes": 10485760
       },
       {
           "path": "FirstContainer/segment_ac",
           "etag": "f1c9645dbc14efddc7d8a322685f26eb",
           "size_bytes": 10485760
       },
       {
           "path": "FirstContainer/segment_ad",
           "etag": "f1c9645dbc14efddc7d8a322685f26eb",
           "size_bytes": 10485760
       },
       
       ...    
       {
           "path": "FirstContainer/segment_aj",
           "etag": "f1c9645dbc14efddc7d8a322685f26eb",
           "size_bytes": 10485760
       }
   ]

5. Upload the manifest file that you just created. In the URI, include the ?multipart-manifest=put query parameter.

   curl -v -X PUT \
           -H "X-Auth-Token:token" \
           "accountURL/containerName/LargeFileName?multipart-manifest=put" \
           -T ./fileName.json

   • LargeFileName: The name of the large object
   • fileName.json: The name of the manifest file
   • ?multipart-manifest=put: The query parameter to upload the manifest file
6. Check the size of the large object.

   curl -v -X HEAD \
           -H "X-Auth-Token:token" \
           accountURL/containerName/LargeObjectName

   The size of the large object is the total size of all the segment objects.

Example:

The following example shows how to upload a large file, using an Oracle Cloud account with the following details:

• Account name: acme
• REST Endpoint URL: https://acme.storage.oraclecloud.com/v1/Storage-acme
• REST Endpoint (Permanent) URL: https://storage-7b16fede61e1417ab83eb52e06f0e365.storage.oraclecloud.com/v1/Storage-7b16fede61e1417ab83eb52e06f0e365

  Note:

  The REST Endpoint (Permanent) URL is displayed for the accounts created after November 2017.

The example uses the REST Endpoint URL for the sample Oracle Cloud account. To use the REST Endpoint (Permanent) URL, replace https://acme.storage.oraclecloud.com/v1/Storage-acme with https://storage-7b16fede61e1417ab83eb52e06f0e365.storage.oraclecloud.com/v1/Storage-7b16fede61e1417ab83eb52e06f0e365.

1. Segment the large file locally into multiple sequential segment files, each smaller than 5 GB:

   split -b 10m myLargeFile.zip segment_

2. List all the segment files:

   ls —al segment_*

   segment_aa
   segment_ab
   segment_ac
   segment_ad
   segment_ae
   segment_af
   segment_ag
   segment_ah
   segment_ai
   segment_aj

3. Create objects from each segment file (segment_aa, segment_ab...segment_aj), preserving the segment file names. Upload all the objects in the FirstContainer container. The following example shows one of the segment files:

   curl -v -X PUT \
            -H "X-Auth-Token: AUTH_tk5a58b7a8c34bb7b662523a59a5272650" \
            -T segment_aa 
            https://acme.storage.oraclecloud.com/v1/Storage-acme/FirstContainer/segment_aa

   The following example shows the output of this command:

   > User-Agent: curl/7.29.0
   > Host: acme.storage.oraclecloud.com
   > Accept: */*
   > X-Auth-Token: AUTH_tkc9305a46ebaa0585c4c7ae063c844f0b
   > Content-Length: 10485760
   > Expect: 100-continue
   >
   < HTTP/1.1 100 Continue
   * We are completely uploaded and fine
   < HTTP/1.1 201 Created
   < Date: Tue, 15 Dec 2015 10:18:26 GMT
   < Last-Modified: Tue, 15 Dec 2015 10:17:21 GMT
   < X-Trans-Id: tx85da332ec5ae4852b7d8c-00566fe8b0ga
   < Etag: f1c9645dbc14efddc7d8a322685f26eb
   < Connection: keep-alive
   < X-Last-Modified-Timestamp: 1450174640.10123
   < Content-Type: text/html;charset=UTF-8
   < Content-Length: 0
   <
   

4. Create a manifest file in JSON format.

   Example Manifest File

   [
       {
           "path": "FirstContainer/segment_aa",
           "etag": "f1c9645dbc14efddc7d8a322685f26eb",
           "size_bytes": 10485760
       },
       {
           "path": "FirstContainer/segment_ab",
           "etag": "f1c9645dbc14efddc7d8a322685f26eb",
           "size_bytes": 10485760
       },
       {
           "path": "FirstContainer/segment_ac",
           "etag": "f1c9645dbc14efddc7d8a322685f26eb",
           "size_bytes": 10485760
       },
       {
           "path": "FirstContainer/segment_ad",
           "etag": "f1c9645dbc14efddc7d8a322685f26eb",
           "size_bytes": 10485760
       },
       
       ...    
       {
           "path": "FirstContainer/segment_aj",
           "etag": "f1c9645dbc14efddc7d8a322685f26eb",
           "size_bytes": 10485760
       }
   ]

5. Upload the manifest file. Add the ?multipart-manifest=put query parameter to upload the manifest file.

   curl -v -X PUT \
           -H "X-Auth-Token: AUTH_tk5a58b7a8c34bb7b662523a59a5272650" 
           "https://acme.storage.oraclecloud.com/v1/Storage-acme/FirstContainer/myLargeFile.manifest?multipart-manifest=put" \
           -T ./manifest.json
   

6. Download the large object by sending a GET request. All the segment objects are concatenated and downloaded as one large object.

   curl -v -X GET \
           -H "X-Auth-Token: AUTH_tk5a58b7a8c34bb7b662523a59a5272650" 
           https://acme.storage.oraclecloud.com/v1/Storage-acme/FirstContainer/myLargeFile.manifest \
           -o ./myLargeFile
   

7. Download the manifest object by sending a GET request, and add the ?multipart-manifest=get query parameter.

   curl -v -X GET \
           -H "X-Auth-Token: AUTH_tk5a58b7a8c34bb7b662523a59a5272650" 
           "https://acme.storage.oraclecloud.com/v1/Storage-acme/FirstContainer/myLargeFile.manifest?multipart-manifest=get" \
           -o ./manifestFile

8. Run a HEAD request to view the size of the large object (myLargeFile) that you created:

   curl -v -X HEAD \
        -H "X-Auth-Token: AUTH_tkbaebb60dfa5b80d84e62b0d5d07031e5" 
   		 https://acme.storage.oraclecloud.com/v1/Storage-acme/FirstContainer/myLargeFile

   The following example shows the output of this command:

   > HEAD /v1/Storage-acme/FirstContainer/myLargeFile HTTP/1.1
   > User-Agent: curl/7.29.0
   > Host: acme.storage.oraclecloud.com
   > Accept: */*
   > X-Auth-Token: AUTH_tkc9305a46ebaa0585c4c7ae063c844f0b
   < Etag: "e6da53c20abee5c471fe8bf796abb1a4"
   < Accept-Ranges: bytes
   < Last-Modified: Tue, 15 Dec 2015 10:07:53 GMT
   < X-Timestamp: 1455012472.56679
   < X-Trans-Id: txcab964b91ba8474ca9193-0056b9bb6fga
   < Date: Tue, 15 Dec 2015 10:12:00 GMT
   < Connection: keep-alive
   < X-Last-Modified-Timestamp: 1455012472.56679
   < Content-Type: application/octet-stream;charset=UTF-8
   < Content-Length: 104857600
   curl: (18) transfer closed with 52428800 bytes remaining to read
   

   You can view the size of the large object in the Content-Length header. The size of the large object is the sum total of the sizes of the segment objects.