Note:
- This tutorial requires access to Oracle Cloud. To sign up for a free account, see Get started with Oracle Cloud Infrastructure Free Tier.
- It uses example values for Oracle Cloud Infrastructure credentials, tenancy, and compartments. When completing your lab, substitute these values with ones specific to your cloud environment.
Integrate Oracle Cloud Infrastructure Logging with Cribl using Oracle Cloud Infrastructure Streaming
Introduction
Oracle Cloud Infrastructure (OCI) Logging: OCI offers a native logging service that aggregates logs from different OCI services, including Compute, Networking, Object Storage, and Database. These logs are stored in a centralized location within OCI and can be accessed through the OCI Console or programmatically via APIs. While OCI Logging service provides essential capabilities, organizations often need to integrate OCI logs with external log management platforms like Cribl for advanced processing, analysis, and visualization.
Cribl is a next-generation log management platform that empowers organizations to take control of their log data. It offers features such as log routing, transformation, enrichment, and filtering, allowing users to optimize log streams according to their specific requirements. Cribl can easily send log data from various sources, such as OCI to destinations like Google Chronicle and Splunk. This integration enables organizations to gain deeper insights, enhance operational efficiency, and drive success in our data-driven world.
Objectives
- Integrate OCI Logging with Cribl using OCI Streaming. The integration of OCI Logging with Cribl involves setting up a streaming pipeline to ingest OCI logs into Cribl for further processing.
Prerequisites
-
Cribl user should be able to configure the Kafka pull. For more information, see Configuring Cribl Stream to Receive Data from Kafka Topics.
-
OCI user must have the required polices for managing OCI Connector Hub, Streaming and Logging services. Policy reference for all the services are here: Policy Reference.
Task 1: Create User, Group, Stream and Connector in OCI
-
Create a user and generate an auth token. To create auth token, see Getting an Auth Token.
Note: Note down the auth token to be provided in Cribl as the password.
-
Create a group and add the user to this group.
-
Create a policy.
Allow group ‘<domain_name>/<Cribl_User_Group>’ to use stream-pull in compartment <compartment_of_stream>, Replace the values of<domain_name>,<Cribl_User_Group>and<compartment_of_stream>accordingly.
-
To create a stream, go to Analytics & AI, Messaging, Streaming.
Note: Note down the stream name.
-
Once the stream is created. Copy the Bootstrap Servers, Username from Kafka connection settings under the respective stream pool. This will be used in Cribl during configuration.
-
To create a connector, go to Analytics & AI, Messaging, Connector Hub. Include the respective log group or logs. Remember to click the default create policy dialog box for logs to be pushed to streaming.
Task 2: Configure Cribl
-
In the Manage Sources section, select KafKa.
-
Configure source Kafka with the details from OCI Streaming. The Brokers are bootstrap servers, Topic is stream name.
-
Select Use TLS in TLS Setting while configuring the source.
-
Check the Authentication and add the required fields captured earlier from OCI stream. The password here is the auth token of the OCI user.
-
Go to Live Data for the logs consumed from OCI stream.
Related Links
Acknowledgments
- Author - Vishak Chittuvalapil (Senior Cloud Engineer)
More Learning Resources
Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.
For product documentation, visit Oracle Help Center.
Integrate Oracle Cloud Infrastructure Logging with Cribl using Oracle Cloud Infrastructure Streaming
F95354-01
March 2024