Ingesting Data into an OpenSearch Cluster with Fluent Bit
Ingest log data into an OpenSearch cluster with Fluent Bit.
Fluent Bit is a lightweight logging and metrics processor and forwarder.
Complete the following tasks before proceeding with the steps described in this topic:
Install and Configure Fluent Bit
Run the applicable command for the VM instance's operating system to install the Fluent Bit agent. For example, run the following command to install the Fluent Bit agent on a Linux instance:
sudo rpm -i <fluent-bit-rpm file>
OUTPUTsection of the Fluent Bit configuration file,
/fluent-bit/fluent-bit.confto direct the output plugin to the OpenSearch cluster. For more information about this section, see Configuration File - Output.
The following is sample code for this section of the configuration file. Replace the applicable text with the details for your cluster. Specify the OpenSearch cluster's OCID for
host, see Getting an OpenSearch Cluster's Details. For
http_Passwd, specify a user that has sufficient permissions to ingest data for the OpenSearch cluster. For example, you can use the primary user account for role-based access control that you specified when you created the cluster, see Role-Based Access Control in Search with OpenSearch.
name - es host - <cluster_ID> port - 9200 tls - on tls.verify - on index - <index_name> http_User - <cluster_user> http_Passwd - <cluster_password> Suppress_Type_Name - On Replace_Dots - On Retry_Limit - 6 Buffer_Size - 400MB
INPUTsection of the Fluent Bit configuration file to specify the source for the log data and how it should be ingested. For more information about this section, see Configuration File - Input.
Name tail Tag file_log Path /etc/fluent-bit/project/file.log Buffer_Max_Size 600m Multiline On Parser_Firstline multiline Buffer_Chunk_Size 2m Skip_Long_Lines Off Skip_Empty_Lines Off Mem_Buf_Limit 700m DB /etc/fluent-bit/db/file.db DB.Sync normal Refresh_Interval 30 Rotate_Wait 20 Exit_On_Eof Off Ignore_Older 5m storage.type filesystem
Test Data Ingestion with Fluent Bit
This section walks through the steps to test log data ingestion with Fluent Bit based on the configuration you specified in Install and Configure Fluent Bit. Perform these steps while you are connected to the VM instance. The sample commands included are for Linux-based operating systems, for other operating systems, use the applicable commands for that operating system.
Run the following commands to install Java and Python:
sudo yum install python3
sudo yum install java
- Run the following command to create the directory for the log data:
Run the following command to start Fluent Bit:
sudo fluent-bit -c /etc/fluent-bit/fluent-bit.confb
Create the python script file
/etc/fluent-bit/projectand copy the following code into the file:
Import logging # Configure logging to write to a file logging.basicConfig(filename='file.log', level=logging.DEBUG) # Generate some log messages logging.debug('This is a debug message.') logging.info('This is an informational message.') logging.warning('This is a warning message.') logging.error('This is an error message.') logging.critical('This is a critical message.') # Example of logging additional details using formatting name = 'John' age = 30 logging.info('User %s, age %d, logged in.', name, age) # Example of logging an exception stack trace try: result = 10 / 0 except Exception as e: logging.exception('An error occurred: %s', str(e))
Use the following command to run the script created in the previous step:
Running this script creates a file
file.logand adds the specified logging messages to the file.
After you run the python script, you can verify that the log data was ingested into
the OpenSearch cluster by connecting to the cluster's OpenSearch Dashboard to check
for the index you specified in the
OUTPUT section of the Fluent Bit
configuration file. See Task 6: Connect to OpenSearch Dashboards
and Quickstart guide for OpenSearch