25 Centralized Monitoring Using Grafana and Prometheus

When using Prometheus and Grafana, you can configure non-Kubernetes products to send their data to the console. This data is in addition to data from the Kubernetes components.

This chapter includes the following topics:

• Oracle HTTP Server
  If you are using a centralized Prometheus and Grafana to monitor your infrastructure, you can send Oracle HTTP Server data to this application.
• Oracle Database
  You can send monitoring information from the Oracle Database to Prometheus. To send data, you should create a Database Exporter inside the Kubernetes cluster and connect it to your database.

Oracle HTTP Server

If you are using a centralized Prometheus and Grafana to monitor your infrastructure, you can send Oracle HTTP Server data to this application.

To send the Oracle HTTP Server data, perform the following steps:

• Enabling Oracle HTTP Server Status Monitoring
  
• Enabling Iptables/Firewalls
  
• Running the Apache Exporter
  
• Adding External Hosts to the Prometheus Operator
  

Enabling Oracle HTTP Server Status Monitoring

To enable Oracle HTTP Server status metrics:

1. Edit the httpd.conf file located in WEB_DOMAIN_HOME/config/fmwconfig/components/OHS/<component>/.
2. Uncomment out the section which looks as follows:

   <Location /server-status>
       SetHandler server-status
       Require host webhost1.example.com
       #  Require ip 127
   </Location>

   Require host and Require IP are optional. If you do not set one or more of these values, the statistics will be available from wherever requested. If you set Require host to be the name of the host running the OHS server, ensure that status information can be collected only when the request is sent from that host.

   In addition, you can set the directive ExtendedStatus to On. However, this setting may have a performance impact on your system.

3. Restart the Oracle HTTP Server using the following commands:

   cd WEB_DOMAIN_HOME/bin

   ./stopComponent.sh ohs1

   ./startComponent.sh ohs1

4. Validate that you can obtain the server statistics using the following command:

   wget http://webhost1.example.com:7777/server-status

   Also validate the command from one of the Kubernetes worker nodes.

Enabling Iptables/Firewalls

If the validation command does not work and you have a firewall enabled or Iptables running, you need to ensure that traffic can reach the web server from the Kubernetes worker hosts.

If you are using Iptables, run the following command:

sudo iptables -I INPUT -p tcp -m tcp --dport 9117 -j ACCEPT

Running the Apache Exporter

The earlier commands (see Enabling Oracle HTTP Server Status Monitoring and Enabling Iptables/Firewalls) enable you to expose the server information from the Oracle HTTP Server. After you expose the information, run the Apache Exporter for Prometheus. This step creates an agent on the webhost, which can be interrogated by Prometheus.

To create an agent, perform the following step:

• Downloading and Installing the Apache Exporter
  

Downloading and Installing the Apache Exporter

To install Apache Exporter:

1. Download the exporter to your webhost using the following command:

   https://api.github.com/repos/Lusitaniae/apache_exporter/releases/latest|grep browser_download_url|grep linux-amd64|cut -d '"' -f 4|wget -qi –

2. Extract the exporter and move it to a system directory such as /usr/local/bin. For example:

   tar xvf apache_exporter-*.linux-amd64.tar.gz
   sudo cp apache_exporter-*.linux-amd64/apache_exporter /usr/local/bin
   sudo chmod +x /usr/local/bin/apache_exporter

3. Run the Apache Exporter by using the following command:

   ./apache_exporter --insecure \
                     --scrape_uri=http://webhost1.example.com:7777/server-status?auto \
                     --telemetry.address=0.0.0.0:9100 \
                     --telemetry.endpoint=/metrics

   Where, scrape_uri is the URL you use to access the Oracle HTTP Server server-status page. The telemetry address includes the port which Prometheus will use to obtain the Oracle HTTP Server metrics

   Note:

   This command runs the exporter in the foreground of the current shell. This is sufficient for testing purposes. For production use, this should be converted to a service and run through init.d so that it starts whenever the host starts.

Adding External Hosts to the Prometheus Operator

After data is generated and made available to Prometheus, you have to configure Prometheus to periodically load the data. To load the data:

1. Edit the helm override file you used to deploy Prometheus and add the following lines in the Prometheus section:

   prometheus:
     service:
       nodePort: 30901
       type: NodePort
   
     prometheusSpec:
       additionalScrapeConfigs:
         - job_name: "External servers"
           static_configs:
             - targets: ["webhost1.example.com:9100","webhost2.example.com:9100"]

2. Upgrade the installation by using the following command:

   helm upgrade -n monitoring kube-prometheus prometheus-community/kube-prometheus-stack -f <WORKDIR>/override_prom.yaml

   You will see External Servers in the Prometheus Service Monitors page located at http://k8worker1.example.com:30901/service-discovery.

Oracle Database

You can send monitoring information from the Oracle Database to Prometheus. To send data, you should create a Database Exporter inside the Kubernetes cluster and connect it to your database.

Note:

If you are using a container database, you must connect to the service associated with the container database; not to individual PDBs.

• Creating an Oracle Database Exporter
  

Creating an Oracle Database Exporter

To create an Oracle Database exporter inside Kubernetes:

1. Create a file called dbexporter.yaml with the following contents:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: oracledb-exporter
     namespace: <PROMNS>
   spec:
     selector:
       matchLabels:
         app: oracledb-exporter
     strategy:
       type: Recreate
     replicas: 1
     template:
       metadata:
         labels:
           app: oracledb-exporter
         annotations:
           prometheus.io/scrape: "true"
           prometheus.io/port: "9161"
           prometheus.io/path: "/metrics"
       spec:
         containers:
         - name: oracledb-exporter
           ports:
           - containerPort: 9161
             name: scrape
             protocol: TCP
           image: iamseth/oracledb_exporter:alpine
           env:
           - name: DATA_SOURCE_NAME
             value: system/<SYSPWD>@//<DB_SCAN_ADDRESS>:1521/<DB_SERVICE_NAME>
   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: oracledb-exporter
     name: oracledb-exporter
     namespace: <PROMNS>
   
   spec:
     type: ClusterIP
     selector:
       app: oracledb-exporter
     ports:
     - name: scrape
       port: 9161
       protocol: TCP
       targetPort: 9161
   ---
   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   
   metadata:
     labels:
       release: kube-prometheus
     name: oracledb-exporter
     namespace: <PROMNS>
   
   spec:
     endpoints:
     - honorLabels: true
       interval: 15s
       path: /metrics
       port: scrape
       relabelings:
         - action: labelmap
           regex: __meta_kubernetes_service_label_(.+)
       scheme: http
       interval: 15s
       scrapeTimeout: 10s
   
     jobLabel: oracledb-exporter
   
     namespaceSelector:
       matchNames:
       - <PROMNS>
   
     selector:
       matchLabels:
         app: oracledb-exporter

2. Create the deployment using the following command:

   kubectl create -f ./dbexporter.yaml

3. Monitor the creation using the following command:

   kubectl get all -n <PROMNS>