Oracle Cloud Infrastructure Documentation

Ingest Custom Logs from OCI Streaming Service Using Service Connector

If you want to analyze and derive insights from your high-volume custom streaming data in OCI Streaming service by using Oracle Log Analytics, then complete the following steps to ingest the streaming logs.

Steps to collect custom logs from OCI Streaming using service connector:

  1. Provide permission to access and collect logs from OCI Streaming service. See Allow Collection of Logs from OCI Streaming Service.

  2. Identify or create the compartment and Log Analytics log group where the logs must be collected.

  3. Create a parser to parse your custom logs. Use the example log content from your streaming data to identify the fields. See Create a Parser.

    Note

    Log Analytics supports parsers of the formats Regex, XML, and JSON. Ensure the following:

    • The log data in a stream must be of a single format, Regex, XML, or JSON and not have multiple formats.

    • The parser identified for a stream must be of the same format as the log data in that stream.

  4. Create a log source that best suits your custom logs and include the parser the you created in the previous step. See Create a Source.

    Alternatively, you can select from the available Oracle-defined log sources. See Oracle-defined Sources.

  5. Create a new service connector to establish a connection between OCI Streaming and Oracle Log Analytics to route your custom logs. Specify Logging Analytics as the target and add new source corresponding to your Streaming Custom Logs.

    For detailed steps to create the service connector using the console, API, or CLI, see OCI Service Connector Hub Documentation.

After the connection is established with OCI Streaming, the custom logs will start flowing into Oracle Log Analytics.

To stop collecting your custom logs from OCI Streaming, follow steps similar to Stop Collecting Logs from Service Connector. However, the service connector referred in the steps must be the one you have created with OCI Streaming as the source and Oracle Log Analytics as the destination for the logs.

Allow Collection of Logs from OCI Streaming Service

Create policies to enable the users to collect the logs from OCI Streaming and ingest them into Oracle Log Analytics.

The following permissions are for uploading logs to Oracle Log Analytics from the service connector. You are prompted to add these policy statements when you create the service connector through OCI console. Alternatively, you can manually create the policy that includes the following policy statements: 

allow any-user to {LOG_ANALYTICS_LOG_GROUP_UPLOAD_LOGS} in compartment id <Log_Group_Compartment_OCID> where all {request.principal.type = 'serviceconnector', target.loganalytics-log-group.id = '<Log_Group_OCID>', request.principal.compartment.id = '<Service_Connector_Compartment_OCID>'}
allow group <userGroup> to MANAGE serviceconnectors in tenancy

In the above policy statements,

  • Log_Group_Compartment_OCID: The compartment OCID of the log group in Oracle Log Analytics where the logs must be stored.

  • Log_Group_OCID: The OCID of the log group in Oracle Log Analytics where the logs must be stored.

  • Service_Connector_Compartment_OCID: The compartment OCID of the service connector hub.

The following permissions are for collecting logs from OCI Streaming service. You are prompted to add these policy statements when you create the service connector through OCI console. Alternatively, you can manually create the policy that includes the following policy statements:

allow any-user to {STREAM_READ, STREAM_CONSUME} in compartment id <Source_Stream_Compartment_OCID> where all {request.principal.type='serviceconnector', target.stream.id='<Stream_OCID>', request.principal.compartment.id='<Service_Connector_Compartment_OCID>'}

In the above policy statements,

  • Source_Stream_Compartment_OCID: The compartment OCID of the stream data in OCI Streaming.

  • Stream_OCID: The OCID of the stream in OCI Streaming.

  • Service_Connector_Compartment_OCID: The compartment OCID of the service connector.