Collect Logs from Your OCI Object Storage Bucket

You can collect log data continuously from Oracle Cloud Infrastructure (OCI) Object Storage. To enable the log collection, create ObjectCollectionRule resource using REST API or CLI. After the successful creation of this resource and having the required IAM policies, the log collection will be initiated.

You can use this method of collecting logs to ingest any type of log stored in an object storage bucket.

You can collect logs from Object Storage bucket in one of the following ways:

  • Live: For continuous collection of objects from the time of creation of the rule ObjectCollectionRule. This is the default method.

  • Historic: For onetime collection of objects for a specified time range.

  • Historic_Live: For collection of all historic logs in the bucket, and after that, continuous collection of all newly created objects containing logs.

Oracle Cloud Logging Analytics uses Events and Streaming services of OCI in conjunction with Object storage to collect and process objects. When you configure a bucket for log collection, Oracle Cloud Logging Analytics creates an Events rule to emit event notification for every new object upload to the bucket. The notifications will be delivered to a stream owned by Oracle Cloud Logging Analytics.

Note:

  • Per bucket, you can have only one ObjectCollectionRule of type Live or Historic_Live.

  • You can create up to 1000 unique object collection rules per tenancy in a region.

  • For proper functioning of log collection from object storage, ensure that the Event rules created by Oracle Cloud Logging Analytics are not tampered with.

Prerequisites: Before enabling log collection using this approach, ensure to:

  • Create a new log source or use an Oracle-defined log source that matches your log format. See Create a Source.
  • Create a log group or use an existing log group where you will store these logs to control the user access control to the logs and note the log group OCID. See Create Log Groups to Store Your Logs.
  • Optionally, if you want to map the logs you are uploading, then create entity or use an existing entity and note the entity OCID. See Map Your Host to an Entity.

To stop the collection of objects from the bucket, delete the ObjectCollectionRule rule. This will only delete the associated configuration with the bucket but will have no effect on the already collected log data or your objects in the bucket.

Permissions Required for Collecting Logs from Object Storage

For Logging Analytics policies documentation, see Details for Logging Analytics.

Permissions required for ObjectCollectionRule:

Task Permission Description

Creating ObjectCollectionRule

LOG_ANALYTICS_OBJECT_COLLECTION_RULE_CREATE

Allow rule creation

LOG_ANALYTICS_LOG_GROUP_UPLOAD_LOGS

Allow association of logs to the log group

LOG_ANALYTICS_ENTITY_UPLOAD_LOGS

Allow association of logs to the entity

LOG_ANALYTICS_SOURCE_READ

Allow using source to parse the logs

BUCKET_UPDATE

Allow updating the bucket to set the emit events flag

If this permission is not given, then the emit events flag must be set already.

Updating ObjectCollectionRule

LOG_ANALYTICS_OBJECT_COLLECTION_RULE_UPDATE

Allow rule updation

LOG_ANALYTICS_LOG_GROUP_UPLOAD_LOGS

Allow association of logs to the log group. Required only when updating the log group.

LOG_ANALYTICS_ENTITY_UPLOAD_LOGS

Allow association of logs to the entity. Required only when updating the entity.

LOG_ANALYTICS_SOURCE_READ

Allow using source to parse the logs. Only required when updating the source.

Deleting ObjectCollectionRule

LOG_ANALYTICS_OBJECT_COLLECTION_RULE_DELETE

Allow rule deletion

Note

When you delete and recreate a bucket, for the existing log collection rule to work, set the flag Emit Objects Events for the bucket after it is recreated.

For log collection to work, along with the above permissions for Creating ObjectCollectionRule, you must also give permission to Oracle Cloud Logging Analytics to read objects from the bucket in your tenancy. The following are some sample IAM policies to give the required permission:

  • allow service loganalytics to read buckets in compartment/tenancy
  • allow service loganalytics to read objects in compartment/tenancy
  • allow service loganalytics to manage cloudevents-rules in compartment/tenancy
  • allow service loganalytics to inspect compartments in tenancy
  • allow service loganalytics to use tag-namespaces in tenancy where all {target.tag-namespace.name = /oracle-tags/}

By default, Object Storage disables automatic emission of events at object level. You can either enable emission of events or have required permissions while creating ObjectCollectionRule. To enable event emission, see Managing Objects.

ObjectCollectionRule Operations

Using REST API or CLI, you can perform operations such as Create, Update, Delete, List, and Get on the ObjectCollectionRule resource.

For information about using the REST API and signing requests, see REST APIs and Security Credentials.

To communicate with OCI cloud services, create an API Signing Key and register it in your user account in OCI. To generate and register the key and to collect the Tenancy's OCID and user's OCID, see Security Credentials - API Signing Key.

For information about SDKs, see Software Development Kits and Command Line Interface.

Using CLI

For information about using the CLI, see Command Line Interface (CLI).

For a complete list of flags and options available for CLI commands, see Command Line Reference.

Run the following CLI commands to manage ObjectCollectionRule:

  • Create ObjectCollectionRule:

    oci log-analytics object-collection-rule create --from-json <json_file_name> --namespace-name <namespace_name>

    For Example:

    oci log-analytics object-collection-rule create --from-json file://create.json --namespace-name MyNamespace

    In the above example command, the referred sample create.json:

    {
        "name": "<Rule_Name>",
        "compartmentId": "<Compartment_OCID>",
        "osNamespace": "<Namespace>",
        "osBucketName": "<Bucket_Name>",
        "logGroupId": "<Log_Group_OCID>",
        "logSourceName": "<Log_Source>"
    }

    For logSourceName, run the CLI or REST API command to get the list of sources. In response to the command, you will get the list of sources with Name and Display Name of each source. In the above example command, you must use Name, and not Display Name.

    Sample response of the above example command:

    {
        "id": "ocid1.loganalyticsobjectcollectionrule.oc1..exampleuniqueID",
        "name": "My Rule",
        "compartmentId": "ocid.compartment.oc1..exampleuniqueID",
        "osNamespace": "MyNamespace",
        "osBucketName": "MyBucket1",
        "collectionType": "LIVE",
        "pollSince": "2020-09-08 14:06:28.028",
        "logGroupId": "ocid1.loganalyticsloggroup.oc1.. exampleuniqueID",
        "logSourceName": "MyLogSource",
        "lifecycleState": "ACTIVE",
        "timeCreated": "2020-09-08T14:06:28.028Z",
        "timeUpdated": "2020-09-08T14:06:28.028Z"
    }
  • Update ObjectCollectionRule: Following is a sample command to update a log source. Similarly, log group, entity, or any other resource can also be updated.

    oci log-analytics object-collection-rule update --namespace-name <Namespace> --object-collection-rule-id <object-collection-rule-OCID> --log-source-name <Log-Source>

    For Example:

    oci log-analytics object-collection-rule update --namespace-name “My Namespace” --object-collection-rule-id ocid1.loganalyticsobjectcollectionrule.oc1..exampleuniqueID --log-source-name MyLogSource
  • Delete ObjectCollectionRule :

    oci log-analytics object-collection-rule delete --namespace-name <Namespace> --object-collection-rule-id <object-collection-rule-OCID>

    For Example:

    oci log-analytics object-collection-rule delete --namespace-name “My Namespace” --object-collection-rule-id ocid1.loganalyticsobjectcollectionrule.oc1..exampleuniqueID
  • List ObjectCollectionRule:

    oci log-analytics object-collection-rule list --namespace-name <Namespace> --compartment-id <compartment-OCID>

    For Example:

    oci log-analytics object-collection-rule delete --namespace-name “My Namespace” --object-collection-rule-id ocid1.loganalyticsobjectcollectionrule.oc1..exampleuniqueID
  • Get ObjectCollectionRule:

    oci log-analytics object-collection-rule get --namespace-name <Namespace> --object-collection-rule-id <object-collection-rule-OCID>

    For Example:

    oci log-analytics object-collection-rule get --namespace-name “My Namespace” --object-collection-rule-id ocid1.loganalyticsobjectcollectionrule.oc1..exampleuniqueID

Cross-Tenancy Object Storage Log Collection

Set the following policies to configure the object collection rule for collecting logs from a bucket in a guest tenant.

Let Guest_Tenant be the guest tenant and Guest_Compartment the compartment in that guest tenant which has the object storage buckets from which the logs must be collected. Let Host_Tenant be the tenant which is subscribed to Oracle Cloud Logging Analytics.

For additional information about writing policies that let your tenancy access Object Storage resources in other tenancies, see Accessing Object Storage Resources Across Tenancies in Oracle Cloud Infrastructure Documentation.

Policies To Be Added in the Guest Tenant

Here is an example of policy statements which allow the IAM group Host_User_Group from the host tenant Host_Tenant and the Oracle Cloud Logging Analytics service to access the objects in the guest compartment Guest_Compartment in the guest tenancy:

define group <Host_User_Group> as <Host_User_Group_OCID>
define tenancy <Host_Tenant> as <Host_Tenant_OCID>
admit group <Host_User_Group> of tenancy <Host_Tenant> to use buckets in compartment <Guest_Compartment>
allow service loganalytics to read buckets in compartment <Guest_Compartment>
allow service loganalytics to read objects in compartment <Guest_Compartment>
allow service loganalytics to manage cloudevents-rules in compartment <Guest_Compartment>
allow service loganalytics to inspect compartments in tenancy
allow service loganalytics to use tag-namespaces in tenancy where all {target.tag-namespace.name = /oracle-tags /}

Policies To Be Added in the Host Tenant

Here is an example of policy statements which allow the host IAM group Host_User_Group to have USE access to the buckets in the guest tenancy Guest_Tenant:

define tenancy <Guest_Tenant> as <Guest_Tenant_OCID>
endorse group <Host_User_Group> to use buckets in tenancy <Guest_Tenant>
allow group <Host_User_Group> to use loganalytics-object-collection-rule in compartment <Rule_Compartment>
allow group <Host_User_Group> to {LOG_ANALYTICS_LOG_GROUP_UPLOAD_LOGS} in compartment <LogGroup_Compartment>
allow group <Host_User_Group> to {LOG_ANALYTICS_SOURCE_READ} in tenancy

Optionally, define this policy if ObjectCollectionRule has associated entities:

allow group <Host_User_Group> to {LOG_ANALYTICS_ENTITY_UPLOAD_LOGS} in compartment <Entity_Compartment>

In the above policies,

  • Rule_Compartment: The compartment in which ObjectCollectionRule must be created.

  • LogGroup_Compartment: The compartment of Oracle Cloud Logging Analytics log group in which the logs must be stored.

  • Entity_Compartment: The compartment of Oracle Cloud Logging Analytics entity.

After the required policies are created, you can create ObjectCollectionRule to collect the logs from guest tenancy Object Storage. Provide the namespace osNamespace and bucket name osBucketName of the guest tenant in the JSON file, as shown in the following example:

{
 "name": "<My_Rule>",
 "compartmentId": "<Compartment_OCID>",
 "osNamespace": "<Guest_Tenant_Namespace>", // Namespace of the guest tenant
"osBucketName": "<Guest_Tenant__Bucket1>", // Bucket in the guest tenant object store namespace
"logGroupId": "<Log_Group_OCID>",
 "logSourceName": "<My_Log_Source>"
 }

For details about creating ObjectCollectionRule, see ObjectCollectionRule Operations.