Upload a Large File
Note:
A large object can have a maximum of 2048 segments. Each segment can be up to 5 GB. The maximum size of a file that you can upload to Oracle Cloud Infrastructure Object Storage Classic as a large object is 10 TB.A user with the Service Administrator role or a role that is specified in the X-Container-Write
ACL of the container can perform this task.
You can upload a large object by using the REST API.
- Segment the large file locally into multiple sequential segment files, each smaller than 5 GB.
On Linux, for example, you can use the following command:
split -b segment_size file_name segment_name
- List all the segment files.
ls —al segment_name*
- Create objects from each segment file. Upload all the objects in the same container.
curl -v -X PUT \ -H "X-Auth-Token:token" \ -T segmentName \ accountURL/containerName/objectName
token
: The authentication token obtained from Oracle Cloud Infrastructure Object Storage ClassicsegmentName
: The full path and name of the segment file to be uploadedcontainerName
: The name of the container in which the object should be createdobjectName
: The name of the object to be created, which is same as the corresponding segment file name
- Create a manifest file in JSON format, and ensure that the manifest file contains the following attributes for each segment object:
path
: The container and object name in the format:containerName/segmentObjectName
etag
: MD5 checksum of the segment object.You can find the value in the
Etag
header of the segment object.size_bytes
: Size of the segment object.You can find the value in the
Content-Length
header of the segment object.
Sample Manifest File
[ { "path": "FirstContainer/segment_aa", "etag": "f1c9645dbc14efddc7d8a322685f26eb", "size_bytes": 10485760 }, { "path": "FirstContainer/segment_ab", "etag": "f1c9645dbc14efddc7d8a322685f26eb", "size_bytes": 10485760 }, { "path": "FirstContainer/segment_ac", "etag": "f1c9645dbc14efddc7d8a322685f26eb", "size_bytes": 10485760 }, { "path": "FirstContainer/segment_ad", "etag": "f1c9645dbc14efddc7d8a322685f26eb", "size_bytes": 10485760 }, ... { "path": "FirstContainer/segment_aj", "etag": "f1c9645dbc14efddc7d8a322685f26eb", "size_bytes": 10485760 } ]
- Upload the manifest file that you just created. In the URI, include the
?multipart-manifest=put
query parameter.curl -v -X PUT \ -H "X-Auth-Token:token" \ "accountURL/containerName/LargeFileName?multipart-manifest=put" \ -T ./fileName.json
LargeFileName
: The name of the large objectfileName.json
: The name of the manifest file?multipart-manifest=put
: The query parameter to upload the manifest file
- Check the size of the large object.
curl -v -X HEAD \ -H "X-Auth-Token:token" \ accountURL/containerName/LargeObjectName
The size of the large object is the total size of all the segment objects.
Example:
- Account name:
acme
- REST Endpoint URL:
https://acme.storage.oraclecloud.com/v1/Storage-acme
- REST Endpoint (Permanent) URL:
https://storage-7b16fede61e1417ab83eb52e06f0e365.storage.oraclecloud.com/v1/Storage-7b16fede61e1417ab83eb52e06f0e365
Note:
The REST Endpoint (Permanent) URL is displayed for the accounts created after November 2017.
The example uses the REST Endpoint URL for the sample Oracle Cloud account. To use the REST Endpoint (Permanent) URL, replace https://acme.storage.oraclecloud.com/v1/Storage-acme
with https://storage-7b16fede61e1417ab83eb52e06f0e365.storage.oraclecloud.com/v1/Storage-7b16fede61e1417ab83eb52e06f0e365
.
- Segment the large file locally into multiple sequential segment files, each smaller than 5 GB:
split -b 10m myLargeFile.zip segment_
- List all the segment files:
ls —al segment_*
segment_aa segment_ab segment_ac segment_ad segment_ae segment_af segment_ag segment_ah segment_ai segment_aj
- Create objects from each segment file (
segment_aa
,segment_ab
...segment_aj
), preserving the segment file names. Upload all the objects in theFirstContainer
container. The following example shows one of the segment files:curl -v -X PUT \ -H "X-Auth-Token: AUTH_tk5a58b7a8c34bb7b662523a59a5272650" \ -T segment_aa https://acme.storage.oraclecloud.com/v1/Storage-acme/FirstContainer/segment_aa
The following example shows the output of this command:
> User-Agent: curl/7.29.0 > Host: acme.storage.oraclecloud.com > Accept: */* > X-Auth-Token: AUTH_tkc9305a46ebaa0585c4c7ae063c844f0b > Content-Length: 10485760 > Expect: 100-continue > < HTTP/1.1 100 Continue * We are completely uploaded and fine < HTTP/1.1 201 Created < Date: Tue, 15 Dec 2015 10:18:26 GMT < Last-Modified: Tue, 15 Dec 2015 10:17:21 GMT < X-Trans-Id: tx85da332ec5ae4852b7d8c-00566fe8b0ga < Etag: f1c9645dbc14efddc7d8a322685f26eb < Connection: keep-alive < X-Last-Modified-Timestamp: 1450174640.10123 < Content-Type: text/html;charset=UTF-8 < Content-Length: 0 <
-
Create a manifest file in JSON format.
Example Manifest File
[ { "path": "FirstContainer/segment_aa", "etag": "f1c9645dbc14efddc7d8a322685f26eb", "size_bytes": 10485760 }, { "path": "FirstContainer/segment_ab", "etag": "f1c9645dbc14efddc7d8a322685f26eb", "size_bytes": 10485760 }, { "path": "FirstContainer/segment_ac", "etag": "f1c9645dbc14efddc7d8a322685f26eb", "size_bytes": 10485760 }, { "path": "FirstContainer/segment_ad", "etag": "f1c9645dbc14efddc7d8a322685f26eb", "size_bytes": 10485760 }, ... { "path": "FirstContainer/segment_aj", "etag": "f1c9645dbc14efddc7d8a322685f26eb", "size_bytes": 10485760 } ]
- Upload the manifest file. Add the
?multipart-manifest=put
query parameter to upload the manifest file.curl -v -X PUT \ -H "X-Auth-Token: AUTH_tk5a58b7a8c34bb7b662523a59a5272650" "https://acme.storage.oraclecloud.com/v1/Storage-acme/FirstContainer/myLargeFile.manifest?multipart-manifest=put" \ -T ./manifest.json
- Download the large object by sending a
GET
request. All the segment objects are concatenated and downloaded as one large object.curl -v -X GET \ -H "X-Auth-Token: AUTH_tk5a58b7a8c34bb7b662523a59a5272650" https://acme.storage.oraclecloud.com/v1/Storage-acme/FirstContainer/myLargeFile.manifest \ -o ./myLargeFile
-
Download the manifest object by sending a
GET
request, and add the?multipart-manifest=get
query parameter.curl -v -X GET \ -H "X-Auth-Token: AUTH_tk5a58b7a8c34bb7b662523a59a5272650" "https://acme.storage.oraclecloud.com/v1/Storage-acme/FirstContainer/myLargeFile.manifest?multipart-manifest=get" \ -o ./manifestFile
- Run a
HEAD
request to view the size of the large object (myLargeFile
) that you created:curl -v -X HEAD \ -H "X-Auth-Token: AUTH_tkbaebb60dfa5b80d84e62b0d5d07031e5" https://acme.storage.oraclecloud.com/v1/Storage-acme/FirstContainer/myLargeFile
The following example shows the output of this command:
> HEAD /v1/Storage-acme/FirstContainer/myLargeFile HTTP/1.1 > User-Agent: curl/7.29.0 > Host: acme.storage.oraclecloud.com > Accept: */* > X-Auth-Token: AUTH_tkc9305a46ebaa0585c4c7ae063c844f0b < Etag: "e6da53c20abee5c471fe8bf796abb1a4" < Accept-Ranges: bytes < Last-Modified: Tue, 15 Dec 2015 10:07:53 GMT < X-Timestamp: 1455012472.56679 < X-Trans-Id: txcab964b91ba8474ca9193-0056b9bb6fga < Date: Tue, 15 Dec 2015 10:12:00 GMT < Connection: keep-alive < X-Last-Modified-Timestamp: 1455012472.56679 < Content-Type: application/octet-stream;charset=UTF-8 < Content-Length: 104857600 curl: (18) transfer closed with 52428800 bytes remaining to read
You can view the size of the large object in the
Content-Length
header. The size of the large object is the sum total of the sizes of the segment objects.