Monitoring PDC with Prometheus and Grafana

57 Monitoring PDC with Prometheus and Grafana

Learn how to use external applications, such as Prometheus and Grafana, to monitor the system performance of Oracle Communications Pricing Design Center (PDC).

Topics in this document:

About Monitoring PDC

You set up the monitoring of PDC by using the following external applications:

WebLogic Monitoring Exporter: Use this Oracle web application to scrape runtime information from PDC and then expose the metric data in Prometheus format. It exposes different WebLogic Mbeans metrics, such as memory usage and sessions count, that are required for monitoring and maintaining the PDC application.
Prometheus: Use this open-source toolkit to scrape PDC metric data from WebLogic Monitoring Exporter and then store it in a time-series database.
Grafana: Use this open-source tool to view on a graphical dashboard all PDC metric data that is stored in Prometheus.

Setting Up Monitoring in PDC

To set up monitoring in PDC:

Install the following external software on your system:
- Prometheus. See "Installation" in the Prometheus documentation.
- Grafana. See "Install Grafana" in the Grafana documentation.
For the list of compatible software versions, see "Additional PDC Software Requirements" in BRM Compatibility Matrix.
Configure WebLogic Monitoring Exporter to scrape metric data from PDC. See "Configuring WebLogic Monitoring Exporter for PDC".
Configure Prometheus to scrape PDC metric data from WebLogic Monitoring Exporter. See "Configuring Prometheus for PDC".
Configure Grafana to display your PDC metric data in a graphic format. See "Creating Grafana Dashboards for PDC".

Configuring WebLogic Monitoring Exporter for PDC

To configure WebLogic Monitoring Exporter to scrape metric data from PDC:

Download the latest supported version of WebLogic Monitoring Exporter (getN.N.sh) from https://github.com/oracle/weblogic-monitoring-exporter/releases, where N.N is the version number.

For the list of supported versions, see "Additional PDC Software Requirements" in BRM Compatibility Matrix.
Edit the wls-exporter-config.yaml file to include the PDC metrics that you want scraped. For the list of supported metrics, see "WebLogic-Based Application Metrics".
Run the following command:
```
bash getN.N.sh wls-exporter-config.yaml
```
The wls-exporter.war is downloaded to your current directory.
Deploy the wls-exporter.war to the server in which PDC is deployed.

WebLogic Monitoring Exporter is deployed on your servers and begins scraping metric data for PDC.

WebLogic Monitoring Exporter exposes your PDC metrics in Prometheus format to the http://localhost:8080/wls-exporter/metrics endpoint on the WebLogic Server.

Example 57-1 wls-exporter-config.yaml Entries for Scraping PDC

The following shows sample wls-exporter-config.yaml entries for setting up WebLogic Monitoring Exporter to scrape metrics from PDC:

metricsNameSnakeCase: true
queries:
- key: name
  keyName: location
  prefix: wls_server_
  applicationRuntimes:
    key: name
    keyName: app
    componentRuntimes:
      prefix: wls_webapp_config_
      type: WebAppComponentRuntime
      key: name
      values: [deploymentState, contextRoot, sourceInfo, sessionsOpenedTotalCount, openSessionsCurrentCount, openSessionsHighCount]
      servlets:
        prefix: wls_servlet_
        key: servletName
- JVMRuntime:
    prefix: wls_jvm_
    key: name
- executeQueueRuntimes:
    prefix: wls_socketmuxer_
    key: name
    values: [pendingRequestCurrentCount]
- workManagerRuntimes:
    prefix: wls_workmanager_
    key: name
    values: [stuckThreadCount, pendingRequests, completedRequests]
- threadPoolRuntime:
    prefix: wls_threadpool_
    key: name
    values: [executeThreadTotalCount, queueLength, stuckThreadCount, hoggingThreadCount]
- JMSRuntime:
    key: name
    keyName: jmsruntime
    prefix: wls_jmsruntime_
    JMSServers:
      prefix: wls_jms_
      key: name
      keyName: jmsserver
      destinations:
        prefix: wls_jms_dest_
        key: name
        keyName: destination
 
- persistentStoreRuntimes:
    prefix: wls_persistentstore_
    key: name
- JDBCServiceRuntime:
    JDBCDataSourceRuntimeMBeans:
      prefix: wls_datasource_
      key: name
- JTARuntime:
    prefix: wls_jta_
    key: name

Configuring Prometheus for PDC

To configure a standalone version of Prometheus for PDC:

Configure Prometheus to scrape the PDC metrics exposed by WebLogic Monitoring Exporter.

To do so, add a scraping configuration to your prometheus.yaml file that is similar to the one shown below for each managed server and admin server in your cluster.

The following shows a sample admin server configuration for a PDC application that is deployed in the pdc-domain WebLogic domain:
```
scrape_configs:
  - job_name: 'wls-metric-pdc'
    scrape_interval: 1s
    metrics_path: /wls-exporter/metrics
    scheme: http
    static_configs:
    - targets: ['hostname:port] #hostname:port of the server where WebLogic Monitoring Exporter resides
      labels:       
        weblogic_domainName: pdc-domain #domain name
        weblogic_domainUID: pdc-domain  #domain UID
        weblogic_serverName: AdminServer #server name
    basic_auth:
      username: username
      password: password  
```
where username and password is your WebLogic Server user name and password.

Configure the alert rules in Prometheus.

To do so, add alert rules to your prometheus-rules.yaml file that are similar to the ones shown below:

groups:  
  -    
    name: alertmanager.rules    
    rules:     
      - alert: heapFreeSizeWarning       
        annotations:         
          message: WLS heap free size is less than 60%.       
        expr: wls_jvm_heap_free_percent{name="AdminServer"} < 60       
        for: 2m       
        labels:         
          domain_uid: pdc-domain         
          severity: critical         
          wls_domain_namespace: {{namespace}}     
      - alert: dataSourceReconnectWarning       
        annotations:         
          message: WLS PricingDB datasource reconnect failed.       
        expr: wls_datasource_failures_to_reconnect_count{name="PricingDB"} > 0       
        for: 2m       
        labels:         
          domain_uid: pdc-domain         
          severity: critical         
          wls_domain_namespace: {{namespace}}     
      - alert: WLSAdminServerNotRunningWarning       
        annotations:         
          message: WLS AdminServer stopped running.       
        expr: wls_server_state_val{location="AdminServer"} != 2       
        for: 2m       
        labels:         
          domain_uid: pdc-domain         
          severity: critical         
          wls_domain_namespace: {{namespace}}     
      - alert: PDCNotRunningWarning       
        annotations:         
          message: PDC stopped running.       
        expr: wls_webapp_config_deployment_state{location="AdminServer",app="PricingDesignCenter",name="AdminServer_/pdc"} != 2       
        for: 2m       
        labels:         
          domain_uid: pdc-domain         
          severity: critical         
          wls_domain_namespace: {{namespace}}            
      - alert: PricingDBRunningWarning       
        annotations:         
          message: PricingDB stopped running.       
        expr: wls_datasource_deployment_state{name="PricingDB"} != 2       
        for: 2m       
        labels:         
          domain_uid: pdc-domain         
          severity: critical         
          wls_domain_namespace: {{namespace}}

Configure Prometheus Alertmanager to raise alerts.

To do so, add alert configurations that are similar to the ones shown below to your prometheus.yaml file.

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
       - localhost:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  - "prometheus-rules.yaml"

Stop and restart Prometheus.

Creating Grafana Dashboards for PDC

You can create a dashboard in Grafana for displaying the metric data for PDC.

Alternatively, you can use the sample dashboards that are included with the PDC package. To use the sample dashboards, import the JSON file from the PDC_home/apps/Samples/monitoring directory into Grafana. For information about importing dashboards into Grafana, see "Export and Import" in the Grafana Dashboards documentation.

Note:

For the sample dashboard to work properly, the datasource name for the WebLogic Domain must be Prometheus.