Note:

Collect logs from OCI Logging into Logstash using OCI Streaming as a Kafka server

Introduction

Oracle Cloud Infrastructure (OCI) Streaming lets users of Apache Kafka offload the setup, maintenance, and infrastructure management that hosting your own Zookeeper and Kafka cluster requires. Streaming is compatible with most Kafka APIs, allowing you to use applications written for Kafka to send messages to and receive messages from the Streaming service without having to rewrite your code.

In this tutorial, you can learn how to use OCI Streaming service as a Kafka server. This service will receive publish requests with logs from the OCI Logging service and subscribe requests from a Logstash server outside OCI.

Objectives

Prerequisites

Architecture

We will use the following OCI services in this tutorial: OCI Logging, OCI Service Connector Hub, and OCI Streaming, as shown in the following architecture:

Solution architecture

Task 1: Set up your Oracle Cloud Infrastructure

  1. Create a stream pool like a Kafka server.

    Create Stream Pool Window

  2. Create a stream. This will be considered a Kafka topic. You will need the stream’s name to specify the topic where to publish or subscribe to the OCI Streaming as a Kafka server.

    Create Stream Window

  3. Create a service connector. After you have the stream created you can create the service connector. You will add source logging and target streaming as shown in the following image:

    Create Service Connector Window

    • For the source configuration, leave the log group and logs values blank, so the connector will take all logs from the tenancy, as shown in the following image:

      Service Connector Source Window

    • For the target configuration, specify the stream name of the stream previously created.

      Service Connector Target Window

Task 2: Set up the Logstash Server

  1. Based on your preference, create a VM instance and configure it to have ssh access into it or use your local station with internet access.

  2. Download Logstash in the instance you want to have the Logstash server. In this tutorial, we have used v8.6.1.

  3. After the download is complete, unzip the package and go to the Logstash folder.

  4. Create two files: input.conf and jaas.conf as shown in the following code snippets.

    input.conf

    input {
  kafka {
    bootstrap_servers => "cell-1.streaming.<oci_region>.oci.oraclecloud.com:9092"
    topics => ["kafka_topic"]
    security_protocol => "SASL_SSL"
    sasl_mechanism => "PLAIN"
    jaas_path => "./jaas.conf"
    auto_offset_reset => "earliest"
    consumer_threads => 1
  }
}
output {
  stdout {
    codec => rubydebug
  }
}

    jaas.conf

    KafkaClient {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="<username_from_kafka_connection_settings>"
  password="<auth_token>";
};

  5. From the Stream Pool you have created previously, copy the configuration for jaas.conf file from SASL Connection Strings in Kafka Connections Settings.

  6. Run the following command to start Logstash with the Kafka client that will consume logs from OCI Streaming:

    ./bin/logstash -f input.conf

Acknowledgments

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.