Oracle Cloud Infrastructure Documentation

Ingesting Data into an OpenSearch Cluster with Logstash

Use Logstash to ingest log data into an OpenSearch cluster.

Logstash is a data processing pipeline that can ingest data from a variety of sources, process and transform it, and then send it to a destination such as an OpenSearch cluster.

Prerequisites

Complete the following tasks before proceeding with the steps described in this topic:

Install and Configure LogStash

  1. Download the Logstash OSS with OpenSearch Output Plugin from OpenSearch Ingest Tools (2.8) and install it on your local machine. This is currently only available for Linux and MacOS based systems.
  2. Extract the downloaded Logstash tarball and then navigate to Logstash directory.
  3. Create a pipeline configuration file, logstash.conf, in the config subdirectory of the Logstash directory.

    Following is a pipeline configuration file example that reads the log file from your local machine and ingests the log data to an OpenSearch cluster. Replace the applicable text with the details for your cluster. Specify the cluster's API endpoint for host, see Getting an OpenSearch Cluster's Details. For user and password, specify a user that has enough permissions to ingest data for the OpenSearch cluster. For example, you can use the primary user account for role-based access control that you specified when you created the cluster, see Role-Based Access Control in Search with OpenSearch.

    input {
  file{
    path => "/path/to/log/file/application.log"
    start_position => "beginning"
  }
}
 
filter{
  #Optional and can be empty
}
 
output {
  opensearch {
  hosts => "<cluster API endpoint>"
  user => "<cluster user>"
  password => "<cluster password>"
  index => "<index name>"
  ssl_certificate_verification => true
  }
}
  4. Start Logstash by running the following command: 
    bin/logstash -f config/logstash.conf
After Logstash starts, it continuously reads the file system for new logs using its file input plugin. Logstash then uses the OpenSearch output plugin to ingest new log data to an OpenSearch cluster.

You can verify that the log data was ingested into the OpenSearch cluster by connecting to the cluster's OpenSearch Dashboard to check for the index named what you specified for index in the pipleline configuration file. See Task 6: Connect to OpenSearch Dashboards and Quickstart guide for OpenSearch Dashboards.