Monitoring Billing Care and Billing Care REST API

About Monitoring Billing Care and Billing Care REST API

You set up the monitoring of Billing Care and the Billing Care REST API by using the following external applications:

WebLogic Monitoring Exporter: Use this Oracle web application to scrape runtime information from Billing Care and the Billing Care REST API, and then export the metric data into a format that can be processed by Prometheus. It exposes different WebLogic Mbeans metrics, such as memory usage and sessions count, that are required for monitoring and maintaining the application.
Prometheus: Use this open-source toolkit to aggregate and store the metric data from WebLogic Monitoring Exporter.
Grafana: Use this open-source tool to view on a graphical dashboard all Billing Care and Billing Care REST API metric data stored in Prometheus.

Setting Up Monitoring in Billing Care and Billing Care REST API

To set up monitoring in Billing Care and the Billing Care REST API:

Install the following external software on your system:
- Pometheus. See "Installation" in the Prometheus documentation.
- Grafana. See "Install Grafana" in the Grafana documentation.
For the list of compatible software versions, see "Billing Care Software Compatibility" in BRM Compatibility Matrix.
Configure WebLogic Monitoring Exporter to scrape metric data from Billing Care and the Billing Care REST API. See "Configuring WebLogic Monitoring Exporter for Billing Care and Billing Care REST API".
Configure Prometheus to scrape metric data from WebLogic Monitoring Exporter. See "Configuring Prometheus for Billing Care and Billing Care REST API".
Configure Grafana to display your Billing Care and Billing Care REST API metric data in a graphical format. See "Creating Grafana Dashboards for Billing Care and Billing Care REST API".

Configuring WebLogic Monitoring Exporter for Billing Care and Billing Care REST API

To configure WebLogic Monitoring Exporter to scrape metric data for Billing Care and the Billing Care REST API:

Download the latest supported version of WebLogic Monitoring Exporter (getN.N.sh) from https://github.com/oracle/weblogic-monitoring-exporter/releases, where N.N is the version number.

For the list of supported versions, see "Billing Care Software Compatibility" in BRM Compatibility Matrix.
Edit the wls-exporter-config.yaml file to include the metrics that you want scraped. For the list of supported metrics, see "WebLogic-Based Application Metrics".
Run the following command:
```
bash getN.N.sh wls-exporter-config.yaml
```
The wls-exporter.war is downloaded to your current directory.
Deploy the wls-exporter.war in the WebLogic Domain through the Admin Console, setting the target as all servers to be monitored.
Verify that metrics are getting exported by running the following cURL command:
```
curl -X GET -i -u WL_userName:WL_password http://localhost:8080/metrics
```
where WL_userName and WL_password is the user name and password for WebLogic Server.

WebLogic Monitoring Exporter is deployed on your servers and begins scraping metric data for Billing Care and the Billing Care REST API.

Example 43-1 Configuring WebLogic Monitoring Exporter to Scrape Metrics

The following shows sample wls-exporter-config.yaml entries for setting up WebLogic Monitoring Exporter to scrape metrics from Billing Care and Billing Care REST API:

metricsNameSnakeCase: true
queries:
- key: name
  keyName: location
  prefix: wls_server_
  applicationRuntimes:
    key: name
    keyName: app
    componentRuntimes:
      prefix: wls_webapp_config_
      type: WebAppComponentRuntime
      key: name
      values: [deploymentState, contextRoot, sourceInfo, sessionsOpenedTotalCount, openSessionsCurrentCount, openSessionsHighCount]
      servlets:
        prefix: wls_servlet_
        key: servletName
- JVMRuntime:
    prefix: wls_jvm_
    key: name
- executeQueueRuntimes:
    prefix: wls_socketmuxer_
    key: name
    values: [pendingRequestCurrentCount]
- workManagerRuntimes:
    prefix: wls_workmanager_
    key: name
    values: [stuckThreadCount, pendingRequests, completedRequests]
- threadPoolRuntime:
    prefix: wls_threadpool_
    key: name
    values: [executeThreadTotalCount, queueLength, stuckThreadCount, hoggingThreadCount]
- JTARuntime:
    prefix: wls_jta_
    key: name

Configuring Prometheus for Billing Care and Billing Care REST API

To configure a standalone version of Prometheus for Billing Care and the Billing Care REST API:

Configure Prometheus to scrape the required metrics exposed by WebLogic Monitoring Exporter.

To do so, add a scraping configuration to your prometheus.yaml file that is similar to the one shown below for each managed server and admin server in your cluster. In the following example, the Billing Care application is deployed in the billingcare-domain WebLogic domain and the server topology is:

The admin-server is on port 7001
A cluster named cluster-1 with two managed servers
managed-server1 is on port 7003
managed-server2 is on port 7005

The Billing Care REST API is deployed in the bcws-domain WebLogic domain and the server topology is:

The admin-server is on port 7011
A cluster named cluster-1 with two managed servers
managed-server1 is on port 7013
managed-server2 is on port 7015

scrape_configs:  
  - job_name: 'wls-metric-billingcare'    
    scrape_interval: 1s    
    metrics_path: /wls-exporter/metrics    
    scheme: http    
    static_configs:    
    - targets: ['localhost:7001']      
      labels:         
        weblogic_domainName: billingcare_domain        
        weblogic_domainUID: billingcare_domain        
        weblogic_serverName: admin-server    
    - targets: ['localhost:7003']      
      labels:        
        weblogic_clusterName: cluster-1        
        weblogic_domainName: billingcare_domain        
        weblogic_domainUID: billingcare_domain        
        weblogic_serverName: managed-server1    
    - targets: ['localhost:7005']      
      labels:        
        weblogic_clusterName: cluster-1        
        weblogic_domainName: billingcare_domain        
        weblogic_domainUID: billingcare_domain        
        weblogic_serverName: managed-server2    
    basic_auth:      
      username: <username>      
      password: <password>  
  - job_name: 'wls-metric-bcws'    
    scrape_interval: 1s    
    metrics_path: /wls-exporter/metrics    
    scheme: http    
    static_configs:    
    - targets: ['localhost:7011']      
      labels:        
        weblogic_domainName: bcws_domain        
        weblogic_domainUID: bcws_domain        
        weblogic_serverName: admin-server    
    - targets: ['localhost:7013']      
      labels:        
        weblogic_clusterName: cluster-1        
        weblogic_domainName: bcws_domain        
        weblogic_domainUID: bcws_domain        
        weblogic_serverName: managed-server1    
    - targets: ['localhost:7015']      
      labels:        
        weblogic_clusterName: cluster-1        
        weblogic_domainName: bcws_domain        
        weblogic_domainUID: bcws_domain       
        weblogic_serverName: managed-server2    
    basic_auth:      
     username: username      
     password: password

where username and password is your WebLogic Server user name and password.

Configure the alert rules in Prometheus.

To do so, add alert rules that are similar to the ones shown below to your prometheus-rules.yaml file. For examples of expressions that you can use for memory and CPU usage, see "Sample Expressions for CPU and Memory Usage".

groups:  
  - name: alertmanager.rules    
    rules:      
      - alert: ClusterWarning        
        annotations:          
          message: WLS cluster has less than 2 running server for more than 1 minutes.        
        expr: sum by(weblogic_domainUID, weblogic_clusterName, weblogic_serverName) (up{weblogic_domainUID="billingcare-domain"}) < 2        
        for: 1m        
        labels:          
          severity: critical     
      - alert: ClusterWarning        
        annotations:          
          message: WLS cluster has less than 2 running server for more than 1 minutes.        
        expr: sum by(weblogic_domainUID, weblogic_clusterName, weblogic_serverName) (up{weblogic_domainUID="bcws-domain"}) < 2        
        for: 1m        
        labels:          
          severity: critical

Configure Prometheus Alertmanger to raise alerts.

To do so, add an alert configuration to your prometheus.yaml file that is similar to the one shown below:

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
       - localhost:9093
# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  - "prometheus-rules.yaml"

Stop and restart Prometheus.

Creating Grafana Dashboards for Billing Care and Billing Care REST API

You can create a dashboard in Grafana for displaying your Billing Care and Billing Care REST API metric data.

Alternatively, you can use the sample dashboards that are included with the Billing Care and Billing Care SDK packages. To use the sample dashboard, import the following dashboard files into Grafana. For information about importing dashboards into Grafana, see "Export and Import" in the Grafana Dashboards documentation.

Billing Care: Domain_home/samples/monitoring/billingcare-dashboard.json
Billing Care REST API: SDK_home/BillingCareREST/samples/monitoring/billingcare-rest-api-dashboard.json

where Domain_home is the WebLogic Server domain home directory in which Billing Care is deployed, and SDK_home is the directory in which you installed the Billing Care SDK.

Note:

For the sample dashboard to work properly, the datasource name for the WebLogic Domain must be Prometheus.

Sample Expressions for CPU and Memory Usage

You can use the following sample expressions in your Prometheus alert rules that are based on current CPU and memory usage.

To raise an alert when the average CPU usage across all managed servers is greater than 70% for more that two minutes:

For the Billing Care REST API domain:

avg(avg_over_time(wls_jvm_process_cpu_load{weblogic_domainUID="bcws-domain",weblogic_serverName=~"managed-server.*"}[2m]))*100 > 70

For the Billing Care domain:

avg(avg_over_time(wls_jvm_process_cpu_load{weblogic_domainUID="billingcare-domain",weblogic_serverName=~"managed-server.*"}[2m]))*100 > 70

To raise an alert when the average memory usage across all managed servers is greater than 70% for more than two minutes:

For the Billing Care REST API domain:

100 - avg(avg_over_time(wls_jvm_heap_free_percent{weblogic_domainUID="bcws-domain",weblogic_serverName=~"managed-server.*"}[2m])) > 70

For the Billing Care domain:

100 - avg(avg_over_time(wls_jvm_heap_free_percent{weblogic_domainUID="billingcare-domain",weblogic_serverName=~"managed-server.*"}[2m])) > 70