26 Centralized Log File Monitoring Using Elasticsearch and Kibana

If you are using Elasticsearch and Kibana, you can configure Filebeat to send the log files to the centralized Elasticearch/Kibana console. Configure Filebeat on each of the hosts you want to send data from.

The instructions in this section are applicable to hosts outside of the Kubernetes cluster. For example, web tier and database hosts.

Before completing these steps, ensure the following:

• You have access to a centralized Elasticsearch deployment.
• If this is a Kubernetes deployment of Elasticsearch and Kibana, you have configured external access through the NodePort Services.
• You have network access to the Kubernetes/Elasticsearch NodePort ports from the source host.
• If you are using an Elasticsearch self-signed certificate, ensure that the Kubernetes name attached to the certificate is resolvable on the origin hosts.
  For example:

  10.0.0.1 k8workers.example.com  elasticsearch-es-http.elkns.es.local

  If the Kubernetes name fails to resolve, you will encounter certificate exceptions.

For further information, see the official supplier documentation at https://www.elastic.co.

This chapter includes the following topics:

• Obtaining the Fingerprint of the Elasticsearch Certificate
  To configure the Filebeat module, you need to derive the fingerprint of the Elasticsearch certificate from the local copy you have already created.
• Obtaining and Installing Filebeat
  You should use the curl command to obtain and install Filebeat.
• Updating the Filebeat Configuration
  After you install Filebeat, you need to configure it so that it knows where Elasticsearch and Kibana instances are located.
• Sending OHS Logs to Elasticsearch
  Oracle HTTP Server is based on Apache. Therefore, you can use the built-in Filebeat Apache module to interpret log files for Oracle HTTP Servers.
• Sending the Database Audit Logs to Elasticsearch
  Elasticsearch has a predefined module for sending Oracle Database audit logs to the Elasticsearch server.
• Setting Up and Starting Filebeat
  When setting up Filebeat, ensure that the command succeeds without any error. If any errors are encountered, resolve them before continuing.

Obtaining the Fingerprint of the Elasticsearch Certificate

To configure the Filebeat module, you need to derive the fingerprint of the Elasticsearch certificate from the local copy you have already created. See Copying the Elasticsearch Certificate.

Obtain the fingerprint of the certificate by using the following command:

openssl x509 -noout -fingerprint -sha256 -inform pem -in ~/workdir/ELK/ca.crt  | sed 's/://g

The output appears as follows:

SHA256 Fingerprint=361A6E52F1936173795ABE36BB0F3A34185DD5A395BB9CECE0D8437EE16C2E44

Obtaining and Installing Filebeat

You should use the curl command to obtain and install Filebeat.

1. Use the following curl command:

   curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-<ELK_VER>-x86_64.rpm 
   sudo rpm -vi filebeat-<ELK_VER>-x86_64.rpm

   For example:

   curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.11.0-x86_64.rpm 
   sudo rpm -vi filebeat-8.11.0-x86_64.rpm

Updating the Filebeat Configuration

After you install Filebeat, you need to configure it so that it knows where Elasticsearch and Kibana instances are located.

1. To configure Filebeat, edit the /etc/filebeat/filebeat.yml file.
2. In the Kibana section of the file, add the following configuration information:

   setup.kibana:
     host: "https://elasticsearch-es-http.elkns.es.local:<ELK_KIBANA_K8>"
     ssl.verification_mode: "none"

   In the Elasticsearch section, add the following information

   output.elasticsearch:
     hosts: ["elasticsearch-es-http.elkns.es.local:<ELK_K8>"]
     protocol: "https"
     username: "elastic"
     password: "<ELK_PASSWORD>"
     ssl.ca_trusted_fingerprint: "361A6E52F1936173795ABE36BB0F3A34185DD5A395BB9CECE0D8437EE16C2E44"

   Note:

   The user specified here is the elastic user, used to ensure that you have access to create the Kibana dashboards. You can use a less permissive user such as logstash_internal after the setup is complete.

   For example:

   setup.kibana:
     host: "https://elasticsearch-es-http.elkns.es.local:31800”
     ssl.verification_mode: "none"

   output.elasticsearch:
     hosts: ["elasticsearch-es-http.elkns.es.local:31920"]
     protocol: "https"
     username: "elastic"
     password: "mypassword"
     ssl.ca_trusted_fingerprint: "361A6E52F1936173795ABE36BB0F3A33485DD5A395BB9CECE0D8437EE16C2E44"

3. Save the file.

Sending OHS Logs to Elasticsearch

Oracle HTTP Server is based on Apache. Therefore, you can use the built-in Filebeat Apache module to interpret log files for Oracle HTTP Servers.

• Enabling and Configuring the Apache Module
  

Enabling and Configuring the Apache Module

The Filebeat Apache module is used to send Oracle HTTP Server logs to the Elasticsearch server. To configure the Apache module:

1. Enable the module by using the following command:

   sudo filebeat modules enable apache

2. Edit the Apache configuration file to update the location of the Oracle HTTP Server log files. This file is located at /etc/filebeat/modules.d/apache.yml.
   For example, your configuration file may look as follows:

   # Module: apache
   # Docs: https://www.elastic.co/guide/en/beats/filebeat/8.3/filebeat-module-apache.html
   
   - module: apache
     # Access logs
     access:
       enabled: true
   
       # Set custom paths for the log files. If left empty,
       # Filebeat will choose the paths depending on your OS.
       var.paths:
        - '/u02/private/oracle/config/domains/ohsDomain/servers/ohs1/logs/access*'
   
     # Error logs
     error:
       enabled: true
   
       # Set custom paths for the log files. If left empty,
       # Filebeat will choose the paths depending on your OS.
       var.paths:
        - '/u02/private/oracle/config/domains/ohsDomain/servers/ohs1/logs/error*'

3. Save the file.

Sending the Database Audit Logs to Elasticsearch

Elasticsearch has a predefined module for sending Oracle Database audit logs to the Elasticsearch server.

• Enabling and Configuring the Oracle Module
  

Enabling and Configuring the Oracle Module

The Filebeat Oracle module is used to send Oracle Database audit logs to the Elasticsearch server. To configure the Oracle module:

1. Enable the module by using the following command:

   sudo filebeat modules enable oracle

2. Edit the Oracle configuration file to update the location of the Oracle Database audit files. This file is located at /etc/filebeat/modules.d/oracle.yml.
   For example, your configuration file may look as follows:

   # Module: oracle
   # Docs: https://www.elastic.co/guide/en/beats/filebeat/8.3/filebeat-module-oracle.html
   
   - module: oracle
     database_audit:
       enabled: true
   
       # Set which input to use between syslog or file (default).
       #var.input: file
   
       # Set paths for the log files when file input is used.
       # Should only be used together with file input
       var.paths: ["/u01/app/oracle/admin/iamdb1*/adump/*.aud"] 

3. Save the file.

Setting Up and Starting Filebeat

When setting up Filebeat, ensure that the command succeeds without any error. If any errors are encountered, resolve them before continuing.

1. Use the following command to set up Filebeat:

   sudo filebeat setup

   After verifying the configuration and ensuring that the Kibana dashboards are loaded, you can change the user and password to a lower privileged user, if required.

2. Start Filebeat by using the following command:

   sudo service filebeat start

   Your log files will now be shipped to Elasticsearch.