Update Your Object Collection Rule to Use Stream

Make the following changes to your existing setup to be able to use stream for object collection:

Topics:

Prerequisites

Retrieve the list of all LIVE and HISTORIC_LIVE object collection rules in your tenancy or compartment that have lifecycle state set to ACTIVE, and stream ID set to null. See List API Documentation.

You can use the following OCI CLI command to list object collection rules that meet this criteria:

oci log-analytics object-collection-rule list \
  --compartment-id <Compartment_OCID> \
  --namespace-name <Tenancy_Namespace_Name> \
| jq '.data.items[] | select((."stream-id" == null) and (."lifecycle-state" == "ACTIVE") and (."collection-type" == "LIVE" or ."collection-type" == "HISTORIC_LIVE"))'

Note the following details for each object collection rule:

Object Collection Rule OCID
Compartment OCID
Object Storage Namespace

If the object storage namespace matches the tenancy's namespace where the object collection rule exists, then follow the steps in Changes Needed for Same-Tenancy Object Collection. If the object storage namespace differs from the tenancy's namespace, then follow the steps in Changes Needed for Cross-Tenancy Object Collection.

Changes Needed for Same-Tenancy Object Collection

Create or Select a Stream:
- You can use an existing stream that was in use by a different object collection rule or create a new one. See Creating a Stream.
- While creating a stream, Oracle Log Analytics recommends the following:
  - Set the retention period of at least 48 hours.
  - Adjust the number of partitions in accordance with the throughput requirements. Each partition can handle up to 1,000 objects per second across all the buckets using the same stream. See Limits on Streaming Resources.
  - Optionally, consider having single stream per tenancy.
  - Use this stream exclusively for object collection to avoid issues during log processing.
- Note the stream OCID and the OCID of the compartment where the stream is located.
Change in Policy:

Add the following policies in addition to your existing object collection rule policies. See Allow Log Collection from Object Storage.
```
allow group <Group_Name> to {STREAM_CONSUME, STREAM_READ} in compartment <Stream_Compartment> 
allow DYNAMIC-GROUP <Dynamic_Group_Name> to {STREAM_CONSUME} in compartment <Stream_Compartment>
```
- Dynamic_Group_Name is the name of the dynamic group created in .
- Stream_Compartment refers to the compartment where the stream is located.
- Group_Name is the user group that is granted these permissions.
Update ObjectCollectionRule with stream ID:

Note

Updating the stream ID is an irreversible change. Once updated, it cannot be changed or removed. If an incorrect update occurs, delete the rule and create a new one.

Update the stream ID with the stream OCID and set streamCursorType to DEFAULT for all the identified object collection rules as below:
```
oci log-analytics object-collection-rule update --namespace-name <Tenancy_Namespace_Name> --object-collection-rule-id <Object_Collection_Rule_OCID> --stream-id <Stream_OCID> --stream-cursor-type DEFAULT
```
See Update API Documentation.

Changes Needed for Cross-Tenancy Object Collection

Let Guest_Tenant refer to the tenant from which logs are collected, and Bucket_Compartment be the compartment of Guest_Tenant where the object storage bucket resides. Let Host_Tenant refer to the tenant subscribed to Oracle Log Analytics, where the object collection currently exists.

Create or Select a Stream:
- You can use an existing stream that was in use by a different object collection rule or create a new one in Host_Tenant or Guest_Tenant. See Creating a Stream.
- While creating a stream, Oracle Log Analytics recommends the following:
  - Set the retention period of at least 48 hours.
  - Adjust the number of partitions in accordance with throughput requirements. Each partition can handle up to 1,000 objects per second across all buckets using the same stream. See Limits on Streaming Resources.
  - Optionally, consider having single stream per tenancy.
  - Use this stream exclusively for object collection to avoid issues during log processing.
- Note the stream OCID and the OCID of the compartment where the stream is located.

Change in Policy:

Stream created in Host_Tenant
If the stream is created in Host_Tenant, then add the following policies in addition to your existing object collection rule policies. For the complete policy requirements, see Allow Cross-Tenancy Log Collection from Object Storage.
- In the Host Tenant:
```
admit any-user of tenancy <Guest_Tenant> to {STREAM_PRODUCE} in tenancy where request.principal.type = 'eventrule' 
allow DYNAMIC-GROUP <Host_Dynamic_Group> to {STREAM_CONSUME} in compartment <Stream_Compartment> 
allow group <Host_User_Group> to {STREAM_CONSUME, STREAM_READ} in compartment <Stream_Compartment> 
```
- In the Guest Tenant:
```
endorse any-user to {STREAM_PRODUCE} in tenancy <Host_Tenant> where request.principal.type = 'eventrule'
```
- Dynamic_Group_Name is the name of the dynamic group created in Migrate Object Collection Workflow from Service Principal to Resource Principal.
- Stream_Compartmentis the compartment where the stream is located.
- Host_User_Group is the relevant user group in Host_Tenant.

Stream created in Guest_Tenant

If the stream is created in Guest_Tenant, then add the following policies in addition to your existing object collection rule policies. For the complete policy requirements, see Allow Cross-Tenancy Log Collection from Object Storage.

In the Host Tenant:

endorse group <Host_User_Group> to {STREAM_CONSUME, STREAM_READ} in compartment <Stream_Compartment> of tenancy <Guest_Tenant> 
endorse DYNAMIC-GROUP <Host_Dynamic_Group> to {STREAM_CONSUME} in compartment <Stream_Compartment> of tenancy <Guest_Tenant> 
endorse any-user to {STREAM_PRODUCE} in tenancy <Guest_Tenant> where request.principal.type = 'eventrule'

In the Guest Tenant:

admit group <Host_User_Group> of tenancy <Host_Tenant> to {STREAM_CONSUME, STREAM_READ} in compartment <Stream_Compartment> 
admit DYNAMIC-GROUP <Host_Dynamic_Group> of tenancy <Host_Tenant> to {STREAM_CONSUME} in compartment <Stream_Compartment> 
admit any-user of tenancy <Host_Tenant> to {STREAM_PRODUCE} in tenancy where request.principal.type = 'eventrule'

Dynamic_Group_Name is the name of the dynamic group created in Migrate Object Collection Workflow from Service Principal to Resource Principal.
Stream_Compartment is the compartment where the stream is located.
Host_User_Group is the relevant user group in Host_Tenant.

Update ObjectCollectionRule with stream ID:

Note

Updating the stream ID is an irreversible change. Once updated, it cannot be changed or removed. If an incorrect update occurs, delete the rule and create a new one.

Update the stream ID with the stream OCID and set streamCursorType to DEFAULT for all the identified object collection rules as below:
```
oci log-analytics object-collection-rule update --namespace-name <Tenancy_Namespace_Name> --object-collection-rule-id <Object_Collection_Rule_OCID> --stream-id <Stream_OCID> --stream-cursor-type DEFAULT 
```
See Update API Documentation.

Oracle Cloud Infrastructure Documentation

Update Your Object Collection Rule to Use Stream