Note:

Delete objects in bulk/batch from Object Storage

Introduction

There are different ways to delete objects from Object Storage. You can perform single object deletion operations, via console, CLI or API/SDK. However, the problem becomes non-trivial when deleting an extremely large number of objects. This tutorial describes the following methods that can be used for bulk/batch deletion of objects:

Points to Note

Note the following points for deleting objects in bulk using the OCI CLI:

Note the following points for deleting objects in bulk using OLM:

Which method should I use?

The main difference between the two methods of bulk/batched object deletion is in the implementation. To reiterate, the OCI CLI bulk-delete command is implemented synchronously on the client side, whereas OLM is implemented asynchronously on the server side. As a general rule-of-thumb, use the following criteria to determine which method to use.

Use CLI bulk-delete if: Use OLM bulk-delete if:
The deletion can occur synchronously (because CLI is synchronous) The deletion must occur asynchronously (because OLM is asynchronous)
One-time deletion Recurring deletion that should happen forever
There are a small number of objects There are a large number of objects
The object deletion needs to happen immediately The object deletion can wait (have a short delay)

Objective

Prerequisites

Method 1: Delete objects in bulk/batch via OCI CLI

You can bulk-delete objects from the OCI CLI using any of the following methods.

Delete objects in bulk within a bucket

You can use the OCI CLI to do bulk deletion of objects within a bucket using the following command:

oci os object bulk-delete -bn <bucket_name>

Note: If you don’t want to run the deletion operation, but just want to see a plan of which objects will be deleted, you can use the --dry-run option.

oci os object bulk-delete -bn <bucket_name> --dry-run
{
  "delete-failures": {},
  "deleted-objects": [
    "1000_sales.csv",
    "rick_and_morty.json"
  ]
}

Delete objects in bulk by prefix

You can specify deletion of only objects that match a particular prefix, by adding the --prefix option. The following command deletes all objects that match the prefix <some_prefix>.

oci os object bulk-delete -bn <bucket_name> --prefix <some_prefix>

For example, if you want to delete objects that match the prefix “rick”, use the following command:

oci os object bulk-delete -bn test-bucket-1 --prefix rick
WARNING: This command will delete at least 1 objects. Are you sure you wish to continue? [y/N]: y
Deleted object rick_and_morty.json  [####################################]  100%
{
    "delete-failures": {},
    "deleted-objects": [
    "rick_and_morty.json"
    ]
}

Delete objects in bulk by pattern (Inclusion)

You can specify deletion of only objects that match a particular pattern, by adding the --include option. The following command deletes all objects that match the pattern <some_pattern>.

oci os object bulk-delete -bn <bucket_name> --include <some_pattern>

For example, if you want to delete only objects that are JSON file type, use the following command:

oci os object bulk-delete -bn test-bucket-1 --include "*.json"
WARNING: This command will delete at least 1 objects. Are you sure you wish to continue? [y/N]: y
Deleted object rick_and_morty.json  [####################################]  100%
{
    "delete-failures": {},
    "deleted-objects": [
    "rick_and_morty.json"
    ]
}

Delete objects in bulk by Inverse Pattern (Exclusion)

You can specify deletion of only objects that don’t match a particular pattern, by adding the --exclude option. The following command deletes all objects except those that match the pattern <some_pattern>.

oci os object bulk-delete -bn <bucket_name> --exclude <some_pattern>

For example, if you want to delete all objects except those that are JSON file type, use the following command:

oci os object bulk-delete -bn test-bucket-1 --exclude "*.json"
WARNING: This command will delete at least 1 objects. Are you sure you wish to continue? [y/N]: y
Deleted object 1000_sales.csv  [####################################]  100%  
{
    "delete-failures": {},
    "deleted-objects": [
    "1000_sales.csv"
    ]
}

Method 2: Delete objects in bulk/batch via OLM

Aside from the OCI CLI, you can also use OLM - Object Lifecycle Management - feature for bulk deletion of objects. The OLM feature allows you to create lifecycle rules which specify the action to take if a particular condition is met. For example, we can specify objects that are older than 30 days to be deleted.

Create a bulk deletion lifecycle rule on the OCI Console/CLI

Lifecycle rules can be created via the OCI Console. This can be done by clicking into a bucket and into the Lifecycle Policy Rules tab on the side. Then, a dialog box should pop up, allowing the user to configure their rule.

Create Lifecycle Rule Console Dialog Box

Lifecycle rules can also be created via the OCI CLI. The following command is an example of creating a Lifecycle rule using the OCI CLI. The rule deletes all objects that are older than 10 days.

oci os object-lifecycle-policy put --bucket-name test-bucket-1 --items '[
    {
        "action": "DELETE",
        "isEnabled": true,
        "name": "lifecycle-rule-delete-all",
        "target": "objects",
        "timeAmount": 10,
        "timeUnit": "DAYS"
    }
]'

Create a bulk deletion lifecycle rule by prefix

In the OCI Console, you can specify Object Name filters for objects which will be deleted, based on a particular condition. It can handle object name prefix matching. The screenshot below shows a lifecycle rule that will delete all objects with prefix test/.

Create Lifecycle Rule Console Prefix

Similar to the console, the OCI CLI can also specify which objects to delete via prefix matching.

oci os object-lifecycle-policy put --bucket-name <test_bucket> --items '[
    {
        ...
        "objectNameFilter": {
            "exclusionPatterns": [],
            "inclusionPatterns": [],
            "inclusionPrefixes": [
                <some_prefix>
            ]
        },
        ...
    }
]'

For example, the following OCI CLI command creates/updates a lifecycle rule that will delete all objects with prefix test/. Remember that the argument to the –-items option must be a valid JSON list.

oci os object-lifecycle-policy put --bucket-name test-bucket-1 --items '[
    {
        "action": "DELETE",
        "isEnabled": true,
        "name": "lifecycle-rule-delete-all",
        "objectNameFilter": {
            "exclusionPatterns": [],
            "inclusionPatterns": [],
            "inclusionPrefixes": [
                "test/"
            ]
        },
        "target": "objects",
        "timeAmount": 10,
        "timeUnit": "DAYS"
    }
]'

Create a bulk deletion lifecycle rule by Pattern Matching (Inclusions)

In the OCI Console, you can specify Object Name Filters to delete objects by pattern matching (inclusions). The screenshot below shows a lifecycle rule that will delete all .doc file types.

Create Lifecycle Rule Console Pattern Inclusion

You can also do deletion of only files that match a particular pattern from the OCI CLI.

oci os object-lifecycle-policy put --bucket-name <bucket_name> --items '[
    {
        ...
        "objectNameFilter": {
            "exclusionPatterns": [],
            "inclusionPatterns": [
                <some_pattern>
            ],
            "inclusionPrefixes": []
        },
        ...
    }
]'

For example, the following OCI CLI command creates/updates a Lifecycle rule that will delete all .doc file types.

oci os object-lifecycle-policy put --bucket-name test-bucket-1 --items '[
    {
        "action": "DELETE",
        "isEnabled": true,
        "name": "lifecycle-rule-delete-all",
        "objectNameFilter": {
            "exclusionPatterns": [],
            "inclusionPatterns": [
                "*.doc"
            ],
            "inclusionPrefixes": []
        },
        "target": "objects",
        "timeAmount": 10,
        "timeUnit": "DAYS"
    }
]'

Create a bulk deletion lifecycle rule by Inverse Pattern Matching (Exclusions)

In the OCI Console, you can specify Object Name Filters to delete objects by inverse pattern matching (exclusion) - that is, delete all objects, except those that match the pattern. The screenshot below shows a lifecycle rule that will delete everything except .txt file types.

Create Lifecycle Rule Console Pattern Exclusion

You can also delete all files except those that match a particular pattern from the OCI CLI.

oci os object-lifecycle-policy put --bucket-name <bucket_name> --items '[
    {
        ...
        "objectNameFilter": {
            "exclusionPatterns": [
                <some_pattern>
            ],
            "inclusionPatterns": [],
            "inclusionPrefixes": []
        },
        ...
    }
]'

For example, the following OCI CLI command creates/updates a lifecycle rule that will delete every object except .txt file types.

oci os object-lifecycle-policy put --bucket-name test-bucket-1 --items '[
    {
        "action": "DELETE",
        "isEnabled": true,
        "name": "lifecycle-rule-delete-all",
        "objectNameFilter": {
            "exclusionPatterns": [
                "*.txt"
            ],
            "inclusionPatterns": [],
            "inclusionPrefixes": []
        },
        "target": "objects",
        "timeAmount": 10,
        "timeUnit": "DAYS"
    }
]'

Acknowledgements

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.