Dealing with Performance and Throughput Problems
This part of the Operations Guide explains how to gather data that may help to diagnose performance or throughput problems and how to cooperate with Oracle in order to resolve these. This part of the guide assumes that the Oracle technology stack that is used to run Oracle Health Insurance Components applications is installed and configured in accordance with the instructions in the Installation Guide and that proper systems management is in place.
Configuration to Influence Performance
This section provides an overview of parameters and configuration items that influence performance of Oracle Health Insurance Components applications.
Memory Settings
The Installation Guide provides guidelines for memory settings for managed servers. Memory settings have a major impact on performance. If a managed server cannot free memory the responsiveness of the system is impacted badly and the managed server spends an increasing amount of time trying to free up memory (known as 'garbage collection'). The memory usage of a managed server can be monitored using a tool like Oracle Enterprise Manager.
Work Manager configuration
Oracle Health Insurance applications make use of Oracle Fusion Middleware Work Managers for managing the number of worker threads that process requests, like UI and web service requests, and tasks. A task is the equivalent of a unit of work. Oracle can help to determine the initial settings. These are usually based on available resources.
Oracle Health Insurance System Properties
The following table lists examples of system properties that influence performance of the system.
Category | System Property | Impact on Performance |
---|---|---|
User Interface |
ohi.ui.maxrowstoretrieve |
For ADF UI pages only. Memory usage and page load times are impacted by this value. This can be set on per page basis using the more specific "ohi.ui.maxrowstoretrieve.<function code>" variant. |
Web Services |
ohi.ws.client.connectiontimeout |
The time in milliseconds before the attempt to connect to an outbound service times out. If it takes a long time to establish the connection then the task may take a long time to complete. During this time, the thread sending the message is not available to perform other tasks. |
Web Services |
ohi.ws.client.readtimeout |
The time in milliseconds that the client waits for the server to respond to the request. If the service is slow to respond then the task may take a long time to complete. And again, during this time, the thread sending the message i not available to perform other tasks. |
Processing |
ohi.processing.filldepth ohi.processing.fillthreshhold |
The filldepth determines the number of tasks that are de-queued and submitted for processing to the Work Manager thread pool at any given time. The fillthreshhold determines when the system looks for more tasks to be submitted for processing. These go hand in hand and work in combination with the maximum number of threads that is set for the core work manager. If the fillthreshold value is too low the managed server may not have enough work to keep all its worker threads busy. Increasing the filldepth fetches more tasks or work items in one go but increases the memory footprint. The Installation Guide mentions default values that provide good starting points to start performance testing. |
Processing |
ohi.processing.groupsize |
The number of activities to be grouped (when applicable) into a collection of tasks that is to be put into the processing grid as one atomic unit. This complete collection is processed on one processing node. The parameter <0> refers to the specific activity type. If this value is too low, it might cause contention in the coherence and frequent commits to the database which impacts the performance. If this value is too high, it increases the memory footprint causing the JVM to run into memory related issues (for example: Frequent full GCs and OutOfMemoryError etc). The Installation Guide mentions default values that provide good starting points to start performance testing. |
These and other system properties are documented in more detail in the Installation Guide.
Deployment Architecture
By default (that is, if the load balancer configuration is set up that way) a managed server processes UI and Web Services requests as well as handle task processing. If that is the case, make sure to properly tune the associated work managers such that a burst of Web Services requests does not bring down the machine or adversely impacts task processing.
Additional processing capacity can be made available by adding managed servers, that is, scaling out in a vertical manner. It is also possible to set up specialized nodes, for example, a few nodes that handle UI requests, a few that handle task processing and a few that handle Web Services requests. This may be beneficial to better control and tune the machines for the various types of requests. How task processing can be disabled is documented elsewhere in this guide.
Performance Tests
Hardware, configuration, set up and usage of Oracle Health Insurance Components differs on per-customer basis. As a result, there is no simple recipe that leads to the optimal performance and throughput. Performance testing is an important part of the implementation project, to validate and tune environments and settings.
Gathering Performance Figures
Monitor Oracle Database Performance
The Oracle Database can be monitored via Enterprise Manager Grid control, and features a number of performance tuning and monitoring capabilities.
Oracle’s Automatic Workload Repository (AWR) is the source of information for database features like the Automatic Database Diagnostic Monitor (ADDM) and the SQL Tuning Advisor. Default snapshots to populate the AWR with relevant data are taken every hour and are retained for 7 days.
for RAC-clustered databases make sure that an AWR report is generated for each database node. Generate the AWR reports for the period in which the problem occurred and make sure to attach the AWR report for each active node in a RAC cluster to the Service Request (SR). |
Monitor a Managed Server using Oracle Enterprise Manager (OEM)
OEM can be used to gather performance figures for the Oracle Database and for the Oracle Fusion Middleware Application Server. It is also possible to define alarms on certain threshold values, for example, if the memory consumption exceeds a certain limit then a system administrator could be notified.
Third party tools may also be used to monitor the technology stack. In that case make sure that the monitoring tools themselves do not unnecessarily impact the performance. For example, use of specific (settings of) profiling tools could drastically impact the performance!
Using JVM Monitoring Tools to gather Performance Figures
Garbage Collection Statistics
The JVM provides the runtime environment in which both the WebLogic application server and the Oracle Health Insurance application runs. It is good practice to capture the JVM’s garbage collection statistics, for example by including the following in a startup script:
# GC logging TIMESTAMP=`date "+%Y-%m-%d_%H_%M_%S"` # LOG_BASE, APP_NAME and SERVER_NAME are used to put the GC logs in separate directories. Make sure they are set JAVA_OPTIONS="${JAVA_OPTIONS} -Xloggc:${LOG_BASE}/${APP_NAME}/gc/gc_${SERVER_NAME}_${TIMESTAMP}.log" JAVA_OPTIONS="${JAVA_OPTIONS} -XX:+PrintGCDetails -XX:+PrintTenuringDistribution -XX:+PrintGCTimeStamps"
Heap Dump
In case of memory issues, Oracle development requests you to take a heap dump of the managed server by using the jmap tool as follows:
jmap -dump:live,format=b,file=heap-<system_identifier>-<date>.dump [pid]
The live option triggers a full garbage collection cycle.
It is good practice to capture a heap dump in case of out of memory errors so that these are readily available for detailed analysis:
# LOG_BASE and APP_NAME are used to write the heap dump in a specific directory. Make sure they are set JAVA_OPTIONS="${JAVA_OPTIONS} -XX:+HeapDumpOnOutOfMemoryError" JAVA_OPTIONS="${JAVA_OPTIONS} -XX:HeapDumpPath=${LOG_BASE}/${APP_NAME}/heapdumps"
Running the Java Flight Recorder
Java Flight Recorder is a profiling and event collection framework built into the Oracle JDK. It allows Java administrators and developers to gather detailed low level information about how the Java Virtual Machine (JVM) and the Java application are behaving. Flight recordings can be interpreted with the Java Mission Control tool and help Oracle development to diagnose performance or memory related issues in Oracle Health Insurance Components applications.
To run the Flight Recorder set the following Java options:
# Java Mission Control: enable Flight Recorder JAVA_OPTIONS="${JAVA_OPTIONS} -XX:+UnlockCommercialFeatures -XX:+FlightRecorder”
Additional command line options can be specified, for example, to save the recording to disk and determine the max age of the recording (defaults to 15 minutes):
-XX:FlightRecorderOptions=defaultrecording=true,disk=true,repository=${LOG_BASE}/${APP_NAME}/{SERVER_NAME}/flightrecordings,maxage=1h
Additional details on the command line options are available in the product’s documentation.
Logging Performance Figures
ADF UI Request Handling Performance Logging
The following logger configuration enables the logging of timing information for ADF User Interface requests:
<logger name="com.oracle.healthinsurance.common.service.ui.userassistance.components" level="debug" />
For each ADF UI request, the elapsed time needed to process the request is logged like this:
DEBUG com.oracle.healthinsurance...SlaLogger - [UI] Request: userId=42;contextPath=/base; requestURL=http://localhost:7001/base/faces/SearchClaims;page=View Claim CL0025_reprocess_2_15;time(ms)=852
the entire message is logged on one line but was formatted differently in this guide for readability. |
REST Service Request Handling Performance Logging
The following logger configuration enables logging of timing information for REST Service requests:
<logger name="com.oracle.healthinsurance.http.jersey.filter.HttpInitFilter" level="debug" />
For each REST Service request, the elapsed time needed to process the request is logged as in the following example:
DEBUG com.oracle.healthinsurance...HttpInitFilter - [WS] Request: contextPath=/api; requestURL=http://localhost:7001/api/someRestService; time(ms)=58
the entire message is logged on one line but was formatted differently in this guide for readability. |
HTTP Connection Pooling
Oracle Health Insurance Components applications run REST interfaces over the HTTPS protocol. This section here applies for REST invocations that are triggered from an Oracle Health Insurance Components application. The Oracle Health Insurance Components application opens an HTTPS connection to a connected system to deliver the message. As opening an HTTP(S) connection is an expensive operation, a pool of connections is kept to reduce on connection setup time.
The size of this connection pool is controlled by two system properties. See the table below:
PropertyName | Description | Default Value |
---|---|---|
ohi.service.client.pool.maxsize |
The maximum number of connections in the pool |
40 |
ohi.service.client.pool.maxperroute |
The maximum number of connections in the pool for an individual route |
4 |
These system properties define the global behavior. It is not possible to set a value for an individual route. |
Several factors play a role when determining optimal values for those 2 parameters:
-
The "keep-alive" value returned by the other system. If the other system returns a keep-alive value of 10 seconds, the HTTP connection can be reused within 10 seconds.
-
The number of different destinations (routes) the Oracle Health Insurance Components application communicates with
-
How long the connection is used by a single worker thread.
-
The number of worker threads on the system that simultaneously may need a connection.
-
The number of worker threads on the system that simultaneously may need a connection for an individual route.
Due to all those factors, it is not possible to give hard rules for setting optimal values for those system properties, but only generic guidelines.
Examples
The next sections show some examples to illustrate the idea.
Single Route
-
The Oracle Health Insurance Components application communicates with a single route
-
16 worker threads do the processing. Each one sends a message every 5 seconds
-
each thread uses the connection for 1 second before returning to the pool.
-
the single route has returned kept-alive of 10 seconds
Optimal value for ohi.service.client.pool.maxsize: 16
Optimal value for ohi.service.client.pool.maxperroute: 16
Lower bound for optimal value for ohi.service.client.pool.maxsize: 4. Rationale: every thread needs the connection for 20% of the time. So 20% * 16 = 3.2 rounded to 4
Lower bound for ohi.service.client.pool.maxperroute: 4
Single Route, keep-alive is too short
-
The Oracle Health Insurance Components application communicates with a single route
-
8 worker threads do the processing. Each one sends a message every 120 seconds
-
each thread uses the connection for 1 second before returning to the pool.
-
the single route has returned kept-alive of 10 seconds No benefit of connection pooling. On average, a message is sent every 15 seconds (120/8). The lifetime of the connection is shorter, so the connection cannot be reused.
This example illustrates that connectivity optimization should work on both ends. Here connection pooling can give benefits if kept-alive is set to a higher value. |
Multiple Routes
-
The Oracle Health Insurance Components application communicates with two routes.
-
16 worker threads do the processing.
-
Route 1 has kept-alive of 60 seconds
-
Every thread sends a message every 8 seconds to route 1.
-
Route 2 has kept-alive of 30 seconds
-
Every thread sends a message every 2 seconds to route 2.
-
-
each thread uses the connection for 1 second before returning to the pool.
Assuming even spread of message delivery, to route 1 a thread uses a connection 12.5% of the time (1 second the connection is used / frequency of usage). As there are 16 threads, 2 connections in the pool are certainly needed.
To route 2 a thread uses a connection 50% of the time (1 second the connection is used / frequency of usage). As there are 16 threads, 8 connections in the pool are certainly needed.
Low bound for ohi.service.client.pool.maxsize: 2+8=10.
Low bound for ohi.service.client.pool.maxperroute: 8
High bound for ohi.service.client.pool.maxsize: 16+16=32.
High bound for ohi.service.client.pool.maxperroute: 16 (one for every thread)
The examples presented are for illustration and give understanding how those parameters work. They are not meant to force customers to exact those calculations. It is however important to analyze the connectivity pattern with each connected system, including analyzing their "keep-alive" settings. |
The examples illustrate that one can calculate with averages and accept that a connection is now always present in the pool. Or one can assume that all threads may need a connection at the same time and set the system properties higher. The latter approach is fine if memory usage is not an issue. The best approach can only be found by doing a performance test. |
Task Processing Performance
Task processing performance (over time) is a good measure or indicator for the performance of the system. The following queries, or variants of these, may provide a good starting point.
Average Processing Time in milliseconds per Task Type
The following query lists the average processing time in seconds per task type for tasks that were executed by the system on the current day:
select taty.name tasktype , round(avg(extract(day from(task.process_stop_datetime-task.process_start_datetime))*24*60*60 + extract(hour from(task.process_stop_datetime-task.process_start_datetime))*60*60 + extract(minute from(task.process_stop_datetime-task.process_start_datetime))*60 + extract(second from(task.process_stop_datetime-task.process_start_datetime))*1000), 0) avg_elapsed_time_ms from ohi_tasks task , ohi_task_types_tl taty where task.taty_id = taty.base_table_id and taty.language = 'en__OHI' and task.creation_date > trunc(sysdate) group by taty.name order by 2 desc nulls last;
Logging Performance Related Service Requests
The guidelines in this section should be followed for performance related Service Requests that are logged through Oracle Support in order to expedite problem resolution. If the problem is not clearly explained or the details to analyze the problem are not provided Support or Development Engineers ask for these first.
-
Always specify the exact name and version of the Oracle Health Insurance Components application.
-
Clear problem description, for example, pay attention to aspects like:
-
Is the problem specific to the UI, Web Services or Processing?
-
Were there any special circumstances, like a peak in Web Services traffic or in the number of objects that needed to be processed or were significant changes made to the configuration of the system?
-
List any recent changes to the environment, for example:
-
was there a substantial increase in the number of members or claims or policies or authorizations that the system must process?
-
does the system process more or new integration requests?
-
was the application recently upgraded?
-
-
Was it necessary to restart components?
-
Was the system responding slow or were out of memory exceptions encountered?
-
-
Specify details of the environment in which the problem occurs:
-
environment name;
-
purpose of the environment, for example: production, development, integration test, performance test;
-
database setup, for example: RAC or standalone
-
middleware domain setup:
-
number of nodes;
-
details about the purpose of these nodes (for example, nodes handling UI, Web Service and Task Processing or do you use specialized nodes)
-
-
-
Attach the following files to the Service Request:
File type | Description |
---|---|
AWR reports |
|
Application log files |
From the time period in which the problem occurred. Please attach entire files, not just snippets. If possible add timing information (see instructions elsewhere in this chapter to obtain these for UI or Web Services). |
Server log files |
Of the time period in which the problem occurred. Again, please attach entire files, not just snippets. |
Oracle Health Insurance properties file |
For the environment. |
Garbage collection statistics |
See instructions elsewhere in this chapter to automatically gather garbage collection data. |
Heap dump |
In case of out of memory issues. See instructions elsewhere in this chapter for automatically capturing a heap dump in case of out of memory exceptions. |
Java Flight Recording files |
See instructions elsewhere in this chapter for automatically creating these JFR files. |
User Interface Performance Checklist
Please provide answers to the following questions when encountering a UI specific performance issue:
-
What exact action was performed: for example: searched a claim, press view policies link, opened some dialog?
-
What is the time of the day? What other activities is the system processing around that time?
-
Can the problem be reproduced at will (at least 3 times) on the same machine?
-
Can the problem be reproduced at other environments at will?
In case the issue is related to an ADF User Interface page:
-
How many browser tabs were open?
-
Does the problem reproduce when all other tabs are closed?
-
What’s running on the local work station, for example, virus scanners, backup programs? Alternatively: what’s the CPU usage on the machine?
-
Does the problem reproduce after logging out from the system and starting from scratch?
-
Can the problem be reproduced at will on other work stations?
-
Does the problem reproduce if all other UI users log out and only one user logs in to reproduce?
-
What’s running on the local work station, for example, virus scanners, backup programs? Alternatively: what’s the CPU usage on the machine?
-
Does the problem reproduce after logging out from the system and starting from scratch?
-
Can the problem be reproduced at will on other work stations?
-
Does the problem reproduce if all other UI users log out and only one user logs in to reproduce?