Note:

This tutorial requires access to Oracle Cloud. To sign up for a free account, see Get started with Oracle Cloud Infrastructure Free Tier.
It uses example values for Oracle Cloud Infrastructure credentials, tenancy, and compartments. When completing your lab, substitute these values with ones specific to your cloud environment.

Use Oracle Cloud Infrastructure Object Storage Python Utilities for Bulk Operations

Introduction

Oracle Cloud Infrastructure (OCI) Object Storage enables customers to securely store any type of data in its native format. With built-in redundancy, OCI Object Storage is ideal for building modern applications that require scale and flexibility, as it can be used to consolidate multiple data sources for analytics, backup, or archive purposes.

OCI Object Storage is foundational to cloud workloads; it’s elasticity, scalability, reliability, and cost efficiency has made it the primary storage for unstructured data in the cloud. As a result of its popularity there is a vast ecosystem of tools to work with OCI Object Storage. This tutorial is the first in the series of “power tools” which aims to provide a pathway for users to become more familiar with the various tools available and how to utilize them in their workloads.

This first tutorial introduces the example scripts available in GitHub using the Python SDK with OCI Object Storage for bulk operations.

Audience

DevOps engineers, developers, OCI Object Storage administrators and users, IT managers, OCI power users, and application administrators.

Objective

Use the Python script examples to handle bulk operations in OCI Object Storage such as bulk restore, delete, copy, renaming, and to list object information.

Prerequisites

An OCI account.
A working config file in your home directory in a subdirectory called .oci. For more information, see Setting up the Configuration File.
Access to an OCI Object Storage bucket.
User permissions in OCI to use OCI Object Storage, have access to manage objects in at least 1 bucket. For more information, see Common Policies and Policy Reference.
A run environment: Access to OCI Cloud Shell, a Linux OCI instance.
Git package installed in run environment and familiarity with git commands.
Python 3 installed.
Familiarity with using Python and Python modules.

Task 1: Set up the Environment

Install the OCI Software Development Kit (SDK) package.
```
pip3 install --user oci
```

Clone the OCI SDK repo.

$ git clone https://github.com/oracle/oci-python-sdk
Cloning into 'oci-python-sdk'...
remote: Enumerating objects: 113452, done.
remote: Counting objects: 100% (23584/23584), done.
remote: Compressing objects: 100% (3022/3022), done.
remote: Total 113452 (delta 20706), reused 23195 (delta 20521), pack-reused 89868
Receiving objects: 100% (113452/113452), 77.07 MiB | 12.78 MiB/s, done.
Resolving deltas: 100% (104373/104373), done.

Note: These are example scripts using the OCI Python SDK, they are not official Oracle applications and are not maintained or supported by Oracle. Since they are examples, you may want to review, update, and customize them for your use case in your own software repository.

Change to the example directory. If installed in your home directory, run the following command.
```
cd $HOME/oci-python-sdk/examples/object_storage
```

Task 2: Use the Object Bulk Restore Script

The bulk restore script restores archived objects in the source bucket with parallel threads.

Run the bulk restore script without arguments to see the required arguments.

$ python object_storage_bulk_restore.py
Source bucket parameter is required !!!

usage: object_storage_bulk_restore.py [-h] [-t CONFIG_PROFILE] [-p PROXY]
                                      [-ip] [-dt] [-c CONFIG_FILE]
                                      [-sb SOURCE_BUCKET]
                                      [-sp SOURCE_PREFIX_INCLUDE]
                                      [-sr SOURCE_REGION]
                                      [-sn SOURCE_NAMESPACE]

optional arguments:
  -h, --help            show this help message and exit
  -t CONFIG_PROFILE     Config file section to use (tenancy profile)
  -p PROXY              Set Proxy (i.e. www-proxy-server.com:80)
  -ip                   Use Instance Principals for Authentication
  -dt                   Use Delegation Token for Authentication
  -c CONFIG_FILE        Config File (default=~/.oci/config)
  -sb SOURCE_BUCKET     Source Bucket Name
  -sp SOURCE_PREFIX_INCLUDE
                        Source Prefix Include
  -sr SOURCE_REGION     Source Region
  -sn SOURCE_NAMESPACE  Source Namespace (Default current connection)

Note: This tutorial will use config file for all scripts. The scripts will default to the region listed in the config file, if it is not specified as a script argument.

Run the script without using the prefix argument. This will restore all archive objects.

$ python object_storage_bulk_restore.py -c $HOME/.oci/config -sb MyBucket

Connecting to Object Storage Service...
Succeed.

##########################################################################################
#                          Running Object Storage Bulk Restore                           #
##########################################################################################
Written by Tim S and Adi Z, July 2020
Starts at             : 2024-05-30 16:44:18
Command Line          : -c /home/opc/.oci/config -sb MyBucket
Source Namespace      : namespace
Source Bucket         : MyBucket
Source Prefix Include :

##########################################################################################
#                                    Start Processing                                    #
##########################################################################################
16:44:18 - Creating 40 workers.
16:44:18 - Getting list of objects from source source_bucket (MyBucket). Restores will start immediately.
16:44:19 - Enqueued 9086 objects to be restored
16:44:19 - Waiting 60 seconds before checking status.
16:45:19 - Restoration of all objects has been requested.

##########################################################################################
#                                       Completed                                        #
##########################################################################################
Completed at :  2024-05-30 16:45:19

Note: Restored objects are not immediately available. Restoration takes at most an hour from the time an archive storage restore request is made, to the time the first byte of data is retrieved.

Run the script with the prefix argument.

$ python object_storage_bulk_restore.py -c $HOME/.oci/config -sb MyBucket -sp folder1

Connecting to Object Storage Service...
Succeed.

##########################################################################################
#                          Running Object Storage Bulk Restore                           #
##########################################################################################
Written by Tim S and Adi Z, July 2020
Starts at             : 2024-05-30 17:00:09
Command Line          : -c /home/opc/.oci/config -sb MyBucket -sp folder1
Source Namespace      : namespace
Source Bucket         : MyBucket
Source Prefix Include : folder1

##########################################################################################
#                                    Start Processing                                    #
##########################################################################################
17:00:09 - Creating 40 workers.
17:00:09 - Getting list of objects from source source_bucket (MyBucket). Restores will start immediately.
17:00:10 - Enqueued 1096 objects to be restored
17:00:10 - Waiting 60 seconds before checking status.
17:01:10 - Restoration of all objects has been requested.

##########################################################################################
#                                       Completed                                        #
##########################################################################################
Completed at :  2024-05-30 17:01:10

Note: All archived objects in the named prefix/folder and sub-prefixes/sub-folders will be restored.

Task 3: Use the Object Bulk Delete Script

The object bulk delete script deletes objects from a bucket with the option to filter.

Run the script without arguments to see the usage.

$ python object_storage_bulk_delete.py
Source bucket parameter is required !!!

usage: object_storage_bulk_delete.py [-h] [-t CONFIG_PROFILE] [-p PROXY] [-ip]
                                    [-dt] [-c CONFIG_FILE]
                                    [-sb SOURCE_BUCKET] [-sp SOURCE_PREFIX]
                                    [-se SOURCE_PREFIX_EXCLUDE]
                                    [-exclude_dirs] [-sn SOURCE_NAMESPACE]
                                    [-sr SOURCE_REGION]

optional arguments:
  -h, --help            show this help message and exit
  -t CONFIG_PROFILE     Config file section to use (tenancy profile)
  -p PROXY              Set Proxy (i.e. www-proxy-server.com:80)
  -ip                   Use Instance Principals for Authentication
  -dt                   Use Delegation Token for Authentication
  -c CONFIG_FILE        Config File (default=~/.oci/config)
  -sb SOURCE_BUCKET     Source Bucket Name
  -sp SOURCE_PREFIX     Source Prefix Include
  -se SOURCE_PREFIX_EXCLUDE
                        Source Prefix Exclude
  -exclude_dirs         Exclude Directories
  -sn SOURCE_NAMESPACE  Source Namespace (Default current connection)
  -sr SOURCE_REGION     Source Region

Note: Running the script without an include or exclude prefix will delete all items in the bucket.

Run the bulk delete script and set an include prefix, review the output and respond to the y/N prompt.

Note: All objects in the prefix/folder and sub-prefixes/sub-folders will be deleted, be careful when running the command. If no match is found, all items in the bucket will be deleted.

$ python object_storage_bulk_delete.py  -c $HOME/.oci/config -sb MyBucket -sp folder1

Connecting to Object Storage Service...
Succeed.

##########################################################################################
#                           Running Object Storage Bulk Delete                           #
##########################################################################################
Written by Adi Zohar, July 2020
Starts at             : 2024-05-30 18:51:32
Command Line          : -c /home/opc/.oci/config -sb MyBucket -sp folder1
Source Namespace      : namespace
Source Bucket         : MyBucket
Source Prefix Include : folder1
Source Prefix Exclude :
Source Region         : us-ashburn-1

Are you sure you want to continue deleting ? [y/N]: y

##########################################################################################
#                                    Start Processing                                    #
##########################################################################################
18:51:38 - Creating 40 workers.
18:51:38 - Getting list of objects from source source_bucket (MyBucket). delete will start immediately.
18:51:38 - Enqueued 700 objects to be deleted
18:51:38 - Waiting 60 seconds before checking status.
18:52:38 - deletion of all objects has been requested.

##########################################################################################
#                                       Completed                                        #
##########################################################################################
Completed at :  2024-05-30 18:52:38

Run the bulk delete script with the exclude prefix.

$ python object_storage_bulk_delete.py  -c $HOME/.oci/config -sb MyBucket -se folder1

Connecting to Object Storage Service...
Succeed.

##########################################################################################
#                           Running Object Storage Bulk Delete                           #
##########################################################################################
Written by Adi Zohar, July 2020
Starts at             : 2024-05-30 19:02:53
Command Line          : -c /home/opc/.oci/config -sb MyBucket -se folder1
Source Namespace      : namespace
Source Bucket         : MyBucket
Source Prefix Include :
Source Prefix Exclude : folder1
Source Region         : us-ashburn-1

Are you sure you want to continue deleting ? [y/N]: y

##########################################################################################
#                                    Start Processing                                    #
##########################################################################################
19:02:58 - Creating 40 workers.
19:02:58 - Getting list of objects from source source_bucket (MyBucket). delete will start immediately.
19:02:58 - Enqueued 60 objects to be deleted
19:02:58 - Waiting 60 seconds before checking status.
19:03:58 - deletion of all objects has been requested.

##########################################################################################
#                                       Completed                                        #
##########################################################################################
Completed at :  2024-05-30 19:03:58

Note: All objects in the bucket not in the exclude prefix/folder will be deleted. Be careful when running this script. If no match is found, all objects in the bucket will be deleted.

Task 4: Use the Object Bulk Rename Script

The bulk rename script can be used to rename objects or to move objects to a different folder/prefix. It is important to understand that the rename script simply changes the name metadata on the object, it is not a copy of the object, it is only a renaming.

Run the object bulk rename script without arguments to see the required parameters.

$ python object_storage_bulk_rename.py
Source bucket parameter is required !!!

usage: object_storage_bulk_rename.py [-h] [-t CONFIG_PROFILE] [-p PROXY] [-ip]
                                    [-dt] [-c CONFIG_FILE]
                                    [-sb SOURCE_BUCKET]
                                    [-sp SOURCE_PREFIX_INCLUDE]
                                    [-sr SOURCE_REGION]
                                    [-sn SOURCE_NAMESPACE]
                                    [-textrem TEXT_REMOVE]
                                    [-textadd TEXT_APPEND]

optional arguments:
  -h, --help            show this help message and exit
  -t CONFIG_PROFILE     Config file section to use (tenancy profile)
  -p PROXY              Set Proxy (i.e. www-proxy-server.com:80)
  -ip                   Use Instance Principals for Authentication
  -dt                   Use Delegation Token for Authentication
  -c CONFIG_FILE        Config File (default=~/.oci/config)
  -sb SOURCE_BUCKET     Source Bucket Name
  -sp SOURCE_PREFIX_INCLUDE
                        Source Prefix Include
  -sr SOURCE_REGION     Source Region
  -sn SOURCE_NAMESPACE  Source Namespace (Default current connection)
  -textrem TEXT_REMOVE  text remove prefix (can be used to remove folder)
  -textadd TEXT_APPEND  text append prefix (can be used to add folder)

Run an os object list to show current objects in bucket.

Existing Objects:

$ oci os object list --bucket-name MyBucket --query "data[].name"
[
  "file_1.txt",
  "file_10.txt",
  "file_2.txt",
  "file_3.txt",
  "file_4.txt",
  "file_5.txt",
  "file_6.txt",
  "file_7.txt",
  "file_8.txt",
  "file_9.txt",
  "hello_world"
]

Run object bulk rename script to change the name of the objects with file in the beginning and replace it with object.

$ python object_storage_bulk_rename.py -c $HOME/.oci/config -sb MyBucket -textrem file -textadd object

Connecting to Object Storage Service...
Succeed.

##########################################################################################
#                           Running Object Storage Bulk Rename                           #
##########################################################################################
Written by Adi Z, March 2021
Starts at             : 2024-05-31 23:09:13
Command Line          : -c /home/opc/.oci/config -sb MyBucket -textrem file -textadd object
Source Namespace      : namespace
Source Bucket         : MyBucket
Source Prefix Include : file
Text Remove Prefix    : file
Text Append Prefix    : object

##########################################################################################
#                                    Start Processing                                    #
##########################################################################################
23:09:13 - Creating 40 workers.
23:09:13 - Getting list of objects from source source_bucket (MyBucket). Rename will start immediately.
23:09:13 - Enqueued 10 objects to be Renamed
23:09:13 - Waiting 60 seconds before checking status.
23:10:13 - Rename of all objects has been requested.

##########################################################################################
#                                       Completed                                        #
##########################################################################################
Completed at :  2024-05-31 23:10:13

Run an os object list to show current objects in bucket and notice the name changes.

$ oci os object list --bucket-name MyBucket --query "data[].name"
[
  "hello_world",
  "object_1.txt",
  "object_10.txt",
  "object_2.txt",
  "object_3.txt",
  "object_4.txt",
  "object_5.txt",
  "object_6.txt",
  "object_7.txt",
  "object_8.txt",
  "object_9.txt"
]

Move all the objects with the string object in their name into a directory named dir1.

$ python object_storage_bulk_rename.py -c $HOME/.oci/config -sb MyBucket -sp object -textadd dir1/

Connecting to Object Storage Service...
Succeed.

##########################################################################################
#                           Running Object Storage Bulk Rename                           #
##########################################################################################
Written by Adi Z, March 2021
Starts at             : 2024-05-31 23:16:14
Command Line          : -c /home/opc/.oci/config -sb MyBucket -sp object -textadd dir1/
Source Namespace      : namespace
Source Bucket         : MyBucket
Source Prefix Include : object
Text Remove Prefix    :
Text Append Prefix    : dir1/

##########################################################################################
#                                    Start Processing                                    #
##########################################################################################
23:16:14 - Creating 40 workers.
23:16:14 - Getting list of objects from source source_bucket (MyBucket). Rename will start immediately.
23:16:15 - Enqueued 10 objects to be Renamed
23:16:15 - Waiting 60 seconds before checking status.
23:17:15 - Rename of all objects has been requested.

##########################################################################################
#                                       Completed                                        #
##########################################################################################
Completed at :  2024-05-31 23:17:15

Run the os object list to show current objects in bucket again and note the objects that have been moved into the folder/prefix named dir1.

$ oci os object list --bucket-name MyBucket --query "data[].name"
[
  "dir1/object_1.txt",
  "dir1/object_10.txt",
  "dir1/object_2.txt",
  "dir1/object_3.txt",
  "dir1/object_4.txt",
  "dir1/object_5.txt",
  "dir1/object_6.txt",
  "dir1/object_7.txt",
  "dir1/object_8.txt",
  "dir1/object_9.txt",
  "hello_world"
]

Task 5: Use the Object Bulk Copy Script

The bulk copy script copies objects between buckets, it can be used to copy objects between buckets in different regions or in different tenancies. This tutorial will demonstrate a copy between buckets in the same tenancy across regions, one bucket in us-ashburn-1 and another bucket in us-phoenix-1. If you like to use the script across tenancies, see Accessing Object Storage Resources Across Tenancies.

Note: There are more robust tools to bulk copy objects between buckets such as Rclone or Obect Storage Replication.

Run the object bulk copy script without arguments to understand the required parameters.

$ python object_storage_bulk_copy.py
usage: object_storage_bulk_copy.py [-h] [-t CONFIG_PROFILE] [-p PROXY] [-ip]
                                  [-dt] [-c CONFIG_FILE] [-sb SOURCE_BUCKET]
                                  [-sr SOURCE_REGION] [-sn SOURCE_NAMESPACE]
                                  [-sp SOURCE_PREFIX_INCLUDE]
                                  [-se SOURCE_PREFIX_EXCLUDE]
                                  [-db DESTINATION_BUCKET]
                                  [-dr DESTINATION_REGION]
                                  [-dn DESTINATION_NAMESPACE] [-ig]

optional arguments:
  -h, --help            show this help message and exit
  -t CONFIG_PROFILE     Config file section to use (tenancy profile)
  -p PROXY              Set Proxy (i.e. www-proxy-server.com:80)
  -ip                   Use Instance Principals for Authentication
  -dt                   Use Delegation Token for Authentication
  -c CONFIG_FILE        Config File (default=~/.oci/config)
  -sb SOURCE_BUCKET     Source Bucket Name
  -sr SOURCE_REGION     Source Region (Default current connection)
  -sn SOURCE_NAMESPACE  Source Namespace (Default current connection)
  -sp SOURCE_PREFIX_INCLUDE
                        Source Prefix Include
  -se SOURCE_PREFIX_EXCLUDE
                        Source Prefix Exclude
  -db DESTINATION_BUCKET
                        Destination Bucket Name
  -dr DESTINATION_REGION
                        Destination Region
  -dn DESTINATION_NAMESPACE
                        Destination Namespace (Default current connection)
  -ig                   Ignore Check if files exist at Destination

Run the object bulk object copy command to copy objects in a folder/prefix to a bucket in another region.

Note: In this example, the destination bucket is empty.

$ python object_storage_bulk_copy.py -c $HOME/.oci/config -sb MySourceBucket -db MyDestBucket -dr us-phoenix-1

##########################################################################################
#                              Connecting to Object Storage                              #
##########################################################################################

Connecting to Object Storage Service for source region - us-ashburn-1
Succeed.

Connecting to Object Storage Service for destination region - us-phoenix-1
Succeed.

##########################################################################################
#                            Running Object Storage Bulk Copy                            #
##########################################################################################
Written by Tim S and Adi Z, July 2020
Starts at        : 2024-05-31 19:38:52
Command Line     : -c /home/opc/.oci/config -sb MySourceBucket -db MyDestBucket
Source Namespace : namespace
Source Region    : us-ashburn-1
Source Bucket    : MySourceBucket
Source Prefix    :
Dest   Namespace : namespace
Dest   Region    : us-phoenix-1
Dest   Bucket    : MyDestBucket
State  File      : MySourceBucket.bulk.wrk

##########################################################################################
#                                    Start Processing                                    #
##########################################################################################
19:38:52 - Creating 50 copy request workers.
19:38:52 - Creating 50 status workers.
19:38:52 - Loading list of objects from destination bucket (MyDestBucket) to ignore exiting files.
19:38:52 -    Loaded 0 files.
19:38:52 - Getting list of objects from source bucket (MySourceBucket). Copies will start immediately.
19:38:52 -    Enqueued 5 objects to be copied

[...snip...]

##########################################################################################
#                          Finish queuing files, start checking                          #
##########################################################################################
19:38:52 - Waiting 60 seconds before checking status.
19:39:52 - Determining copy status
19:39:52 -    KNOWN: 1, REQUESTED: 0, COMPLETED: 4, FAILED: 0, CANCELED: 0

Task 6: Use the Object Storage List Folders Script

The object storage list folders script lists each folder/prefix in the bucket, the total size of all objects (kilobytes) in each folder, and a count of objects in each folder. There is an option to filter by prefix/folder.

Run the object list folder script without arguments to understand the command line options.

$ python  object_storage_list_folders.py
Source bucket parameter is required !!!

usage: object_storage_list_folders.py [-h] [-t CONFIG_PROFILE] [-p PROXY]
                                      [-ip] [-dt] [-c CONFIG_FILE]
                                      [-sb SOURCE_BUCKET] [-sp SOURCE_PREFIX]
                                      [-sr SOURCE_REGION]
                                      [-sn SOURCE_NAMESPACE] [-f FILE]

optional arguments:
  -h, --help            show this help message and exit
  -t CONFIG_PROFILE     Config file section to use (tenancy profile)
  -p PROXY              Set Proxy (i.e. www-proxy-server.com:80)
  -ip                   Use Instance Principals for Authentication
  -dt                   Use Delegation Token for Authentication
  -c CONFIG_FILE        Config File (default=~/.oci/config)
  -sb SOURCE_BUCKET     Source Bucket Name
  -sp SOURCE_PREFIX     Source Prefix Include
  -sr SOURCE_REGION     Source Region
  -sn SOURCE_NAMESPACE  Source Namespace (Default current connection)
  -f FILE               Output to file (as csv)

Run the object list folders script against an entire bucket.

$ python  object_storage_list_folders.py -c $HOME/.oci/config -sb MyBucket

Connecting to Object Storage Service...
Success.

##########################################################################################
#                               Running List/Count Objects                               #
##########################################################################################
Written By Adi Zohar, June 2020
Starts at           :2024-05-30 21:25:06
Command Line        : -c /home/opc/.oci/config -sb MyBucket
Source Namespace    : namespace
Source Bucket       : MyBucket
Source Prefix       :

##########################################################################################
#                                  Start Processing...                                   #
##########################################################################################
Folder Name                                        |  Size (KB) |   Count
--------------------------------------------------------------------------------
dir_1/                                             | 10240000.00 |      10
dir_10/                                            | 10240000.00 |      10
dir_2/                                             | 10240000.00 |      10
dir_3/                                             | 10240000.00 |      10
dir_4/                                             | 10240000.00 |      10
dir_5/                                             | 10240000.00 |      10
dir_6/                                             | 10240000.00 |      10
dir_7/                                             | 10240000.00 |      10
dir_8/                                             | 10240000.00 |      10
dir_9/                                             | 10240000.00 |      10

##########################################################################################
#                                       Completed                                        #
##########################################################################################
Completed at :  2024-05-30 21:25:06
Total Files  :                  103
Total Size   :      137,069,854,720

Run the object list folders script against a specific folder/ prefix.

$ python  object_storage_list_folders.py -c $HOME/.oci/config -sb  MyBucket -sp dir_1/

Connecting to Object Storage Service...
Success.

##########################################################################################
#                               Running List/Count Objects                               #
##########################################################################################
Written By Adi Zohar, June 2020
Starts at           :2024-05-30 21:27:41
Command Line        : -c /home/opc/.oci/config -sb MyBucket -sp dir_1/
Source Namespace    : namespace
Source Bucket       : MyBucket
Source Prefix       : dir_1/

##########################################################################################
#                                  Start Processing...                                   #
##########################################################################################
Folder Name                                        |  Size (KB) |   Count
--------------------------------------------------------------------------------
dir_1/                                             | 10240000.00 |      10

##########################################################################################
#                                       Completed                                        #
##########################################################################################
Completed at :  2024-05-30 21:27:42
Total Files  :                   10
Total Size   :       10,485,760,000

Task 7: Use the Object Storage List Objects Script

The object list script lists each object in the bucket, the size in bytes of each object, total size of all objects, and a count of all objects. There is an option to filter by prefix/folder.

Run the object list script against an entire bucket.

$ python object_storage_list_objects.py -c  $HOME/.oci/config -sb MyBucket

Connecting to Object Storage Service...
Success.

##########################################################################################
#                               Running List/Count Objects                               #
##########################################################################################
Written By Adi Zohar, June 2020
Starts at           :2024-05-31 18:25:16
Command Line        : -c /home/opc/.oci/config -sb MyBucket
Source Namespace    : namespace
Source Bucket       : MyBucket
Source Prefix       :
Source Pre-Exclude  :

##########################################################################################
#                                  Start Processing...                                   #
##########################################################################################
      10,737,418,240 | C 2024-01-30 23:58 | U 2024-01-30 23:58 | Standard          | 10GB-Testfile
      10,737,418,240 | C 2024-01-25 19:41 | U 2024-01-25 19:41 | Standard          | 10GB-file
      10,737,418,240 | C 2024-01-24 22:13 | U 2024-01-24 22:13 | Standard          | 10GB_file
      1,048,576,000 | C 2024-01-24 22:13 | U 2024-01-24 22:13 | Standard          | dir_1/file_1
      1,048,576,000 | C 2024-01-24 22:14 | U 2024-01-24 22:14 | Standard          | dir_1/file_10

[...snip...]

      1,048,576,000 | C 2024-01-24 23:10 | U 2024-01-24 23:10 | Standard          | dir_9/file_8
      1,048,576,000 | C 2024-01-24 23:10 | U 2024-01-24 23:10 | Standard          | dir_9/file_9

##########################################################################################
#                                       Completed                                        #
##########################################################################################
Completed at :  2024-05-31 18:25:16
Total Files  :                  103
Total Size   :      137,069,854,720

Run the list object script and filter by a folder/prefix.

$ python object_storage_list_objects.py -c  $HOME/.oci/config -sb MyBucket -sp dir_1/

Connecting to Object Storage Service...
Success.

##########################################################################################
#                               Running List/Count Objects                               #
##########################################################################################
Written By Adi Zohar, June 2020
Starts at           :2024-05-31 18:49:49
Command Line        : -c /home/opc/.oci/config -sb MyBucket -sp dir_1/
Source Namespace    : namespace
Source Bucket       : MyBucket
Source Prefix       : dir_1/
Source Pre-Exclude  :

##########################################################################################
#                                  Start Processing...                                   #
##########################################################################################
      1,048,576,000 | C 2024-01-24 22:13 | U 2024-01-24 22:13 | Standard          | dir_1/file_1
      1,048,576,000 | C 2024-01-24 22:14 | U 2024-01-24 22:14 | Standard          | dir_1/file_10
      1,048,576,000 | C 2024-01-24 22:15 | U 2024-01-24 22:15 | Standard          | dir_1/file_2
      1,048,576,000 | C 2024-01-24 22:16 | U 2024-01-24 22:16 | Standard          | dir_1/file_3
      1,048,576,000 | C 2024-01-24 22:17 | U 2024-01-24 22:17 | Standard          | dir_1/file_4
      1,048,576,000 | C 2024-01-24 22:17 | U 2024-01-24 22:17 | Standard          | dir_1/file_5
      1,048,576,000 | C 2024-01-24 22:17 | U 2024-01-24 22:17 | Standard          | dir_1/file_6
      1,048,576,000 | C 2024-01-24 22:18 | U 2024-01-24 22:18 | Standard          | dir_1/file_7
      1,048,576,000 | C 2024-01-24 22:19 | U 2024-01-24 22:19 | Standard          | dir_1/file_8
      1,048,576,000 | C 2024-01-24 22:20 | U 2024-01-24 22:20 | Standard          | dir_1/file_9

##########################################################################################
#                                       Completed                                        #
##########################################################################################
Completed at :  2024-05-31 18:49:49
Total Files  :                   10
Total Size   :       10,485,760,000

Acknowledgments

Author - Melinda Centeno (Senior Principal Product Manager)
Contributors - Adi Zohar (Master Principal Cloud Architect), Tim Sammut (Consulting Member of Technical Staff)

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.

Title and Copyright Information

Use Oracle Cloud Infrastructure Object Storage Python Utilities for Bulk Operations

G10195-01

June 2024

Oracle and/or its affiliates.

Use Oracle Cloud Infrastructure Object Storage Python Utilities for Bulk Operations

Introduction

Audience

Objective

Prerequisites

Task 1: Set up the Environment

Task 2: Use the Object Bulk Restore Script

Task 3: Use the Object Bulk Delete Script

Task 4: Use the Object Bulk Rename Script

Task 5: Use the Object Bulk Copy Script

Task 6: Use the Object Storage List Folders Script

Task 7: Use the Object Storage List Objects Script

Related Links

Acknowledgments

More Learning Resources