28 Performance Considerations and Best Practices
Checking for performance problems requires observation of the effects that lead to the decision that a performance issue exists and access to configuration and performance information.
This chapter covers performance troubleshooting for OAAM.
28.1 General Performance Tuning and Troubleshooting
The troubleshooting process starts with the top-level troubleshooting flow:
-
Check rules and policy performance using the OAAM dashboard's performance section.
-
In the Dashboard list, click Performance.
-
In the Data list, click the Rules or Policies data type.
-
-
To check database performance, use Fusion Middleware Control to check the SQL query performance.
-
Examine diagnostics log, memory configuration check, network bottlenecks, CPU monitoring.
Review all the performance areas and concerns to isolate where the bottlenecks are occurring to ensure you are getting the best performance possible. Gathering data and investigating the I/O bottlenecks provides options for the DBA or the designer on how to fix the problem. If the system does not perform as expected, you can experience increased response times and frequent time-outs.
You can also monitor key metrics in Fusion Middleware Control.
-
Check connectivity between application server and database, jStack stack trace, or thread dump.
-
Check the following OAAM properties:
bharosa.db.query.performance.warning.print.stack=false bharosa.db.query.performance.warning.threshold.ms=500 (print WARN for every call that takes more than 0.5 secs, this can be adjusted for optimal value other than this). bharosa.db.query.performance.warning.print.stack=true
If the calls to the OAAM APIs begin to degrade over time, there are a few possible causes to check:
-
The database is becoming too large. Regular archiving and purging of old data from the database should prevent problems.
-
The hard drive on the serving machine may be too full. This could cause a variety of problems that would ultimately manifest as slow down or possibly a stopping of the application.
-
The binaries responsible for the proper working of the application server have become corrupt. Check for recent or unusual modification times within the application directory of the Web container.
-
The number of users or transactions per second has exceeded what the system was designed to handle*. Overcoming this will likely require additional hardware and/or more intelligent handling of incoming connections.
* Metrics obtained by conducting pre-deployment performance testing on your reference staging hardware/software environment should enable you to pinpoint the maximum concurrent sessions and sessions per second that should not be exceeded.
28.2 Performance Monitoring and Troubleshooting Tools
In order to monitor and improve performance for your environment you need to know what tools are available and how to use each of these tools.
This section covers the following tools to give you an introduction as to what they are used for an how you can use them to collect performance related data.
Use the dashboard for a better insight into the performance of your system. Through statistics, you can view how long rules, policies, checkpoints, and APIs are taking to process. What are the average processing times for rules, policy and checkpoint executions? You can identify the items that consume the most resources (For example, which rules are the most and least expensive?). You can view the data for today, the last 1 day, last 7 days, last 30 days, or last 90 days. You can view the number of logins, KBA or OTP challenges, and Transactions per minute.
The dashboard provides data about the system health. Use it to determine if there are any bottlenecks. The Session Details page is also a useful tool that provides an overview of what rules and policies were run as well as collected details such as execution time, the time stamp, run time, the number of rules that were run, as well additional data.
To launch the Dashboard, in the Navigation tree of OAAM, double-click Dashboard. The Dashboard will appear in the OAAM Administration Console's right side.
Create BI Publisher reports for rule logs to quickly obtain summary information and useful information that can assist you in determining where your performance bottlenecks may be.
Through reports, you can view detailed information about rules. Then, view the Session Details page for the execution time for rules. If the rule requires an unusual amount of time, perform further analysis. Also, check the execution of each runtime. How many seconds out of the box do we print queries? If query takes more than that threshold, print it.
If the CPU of the application seems high and database is not up to speed, you can use the JAVA profiling tools. These provide data on the methods, code, and classes that require a lot of time. Using these tools may answer questions about how many executions occur, and the total time taken for the executions.
Oracle Fusion Middleware Control
You can use Fusion Middleware Control to monitor Oracle Adaptive Access Manager performance and activity.
-
Select OAAM under Identity and Access to go to the home page.
In the home page, you can view a performance overview for Oracle Adaptive Access Manager.
-
Select Performance Summary from the Oracle Adaptive Access Manager menu in the upper left hand side of the home page to view performance metrics.
For information on monitoring status and performance with Fusion Middleware Control, see "Monitoring Oracle Fusion Middleware" in the Oracle Fusion Middleware Administrator's Guide.
Other Environment Dependant Tools
Use other environment dependant tools.
28.3 Policy and Rules - Performance Consideration
In most cases, creating of rules and policies should not be the focus when dealing with performance. However, as in any technology, there are tips and tricks that can be used to maximize performance when needed. Most of the considerations in this section are focused on the configuration.
Order Rules with Simple Conditions in the Beginning
Reordering rule conditions can improve the performance of rule evaluation in time, memory use, or both. Configure rules that are less complex, such as ones checking for a property or small variables, to run first in the order of evaluation. Configure the rules containing complex conditions like autolearning (a dynamic condition) to run after the simpler rules. The evaluation stops when a rule is evaluated to true. For example, if the first rule returns true, OAAM does not attempt to evaluate the second rule.
Consider Policy Structure and Behavior
Isolate the rules that require a great deal of time to process, then look at the policy to determine where performance issues might exists. For example, if it requires a great deal of time to execute all the rules associated to a policy, too many rules may be associated to a policy or more expensive rules are used. If there are too many rules, consider splitting them into different policies. If there are expensive rules consider optimizing the underlying SQL queries or optimizing/tuning database.
Running Conditions Multiple Times is Not Efficient/Can We Nest Them?
There policies where the outcome is to perform an action or give a score for a particular user or for when a feature is enabled. If a number of conditions are repeated that look for the particular user or an enabled feature in many rules, make the condition a decision point in one of your policies instead and continue from this point to a different flow. For example, if the condition is to check for a particular type of user, evaluating the same conditions many times evaluates the same users. If the condition is to check if a user is a mobile phone user and if he is, perform this action and give this score; and check if a user is a mobile phone user and if he is, perform that action and give another score, and so on. Instead of evaluating the condition in such a manner, considering nesting policies. Set up the evaluation to check if users are mobile phone customers and if they are, execute these different policies. In this way, you do not execute the conditions many times.
Order Rules So Not All Overrides are Evaluated
Are there overrides? Overrides (trigger combinations) are used to override the outcome of rules. In the Trigger Combination tab, each row in any trigger combination represents a rule that is presenting a policy. If there is a set of trigger combinations for a policy, each row corresponds to a rule. Each vertical column represents a combination of rules that are triggering or not. The trigger combinations evaluate sequentially, moving from column 1 to columns 2, 3, 4 and so on, and stop as soon as a rule return combination is matched.
Identify the rules that are more complex and arrange them in such a way that they are evaluated first. If the first column is true, the second column will not be looked at. Arrange rules so OAAM does not need to evaluate all the overrides.
Keep Only Required Rules, Patterns, and Logs
-
Keep only required rules. If you are not using a rule, you may want to disable or remove it. A rule adds queries for every session that runs. You do not want additional queries to run on the database if it is not necessary. Out of the box, device rules are shipped. If you are not planning to use devices in evaluations, you can disable all the rules for devices.
-
Keep only required patterns. If you are not using patterns in rules, disable or remove them. Each time you perform an operation and a session runs, autolearning can potentially run five to ten queries. If queries are run on the database, performance is affected.
-
Keep only required updates (rule logs for example). The OAAM database has rule logs as a feature. Rule logs will update tables in the database. If you do not want every session available for analysis for investigation, you may want to completely turn detailed rule logs off or you can enable them only for the time you need them (for example, for 5 seconds).
Determine the Resource Bundles Needed for the Deployment
Resource bundle are properties files that contains locale-specific data used in internationalizing the application.
Consider if you need all the resource bundles and whether or not you can remove some. If your deployment is not multilingual, or if do not need all the resources and locales, consider not using some of the resource bundles.
Use Lightweight Applications and Policies/Rule Configuration
Use lightweight applications and policies/rules. In custom development, consider the resources required when adding a policy, rule, and action.
28.4 Logging - Performance Considerations
Rule logging includes a large amount of data. Users who experience large numbers of logins per day will have many rows of data written in the logs.
You can configure rule logging such that detailed rule logs are created only if the execution time is more than a threshold. Then, the details are logged against the rules (runtime) with long execution time and hence the overhead of detailed logging is fair. If you want to set the minimum time required before detailed logging occurs, you need to set the following properties.
vcrypt.tracker.rulelog.detailed.minMillis=<time>
Oracle Adaptive Access Manager 11g components use the package java.util.logging as part of its logging infrastructure. This package is available in all Java environments. The OAAM loggers generate logging messages to report on errors and provide additional information about OAAM.
How much logging you need? The log level is used to define which messages should be written to the log. Reducing the amount of log output improves performance.
To access logging configuration:
-
Log in to Fusion Middleware Control as an administrator.
-
Under WebLogic Domain, click oaam_server_server1.
-
Expand the WebLogic Server tab, Logs, and select Log Configuration.
-
Click Log level and expand Root Logger > oracle > oracle.oaam.
-
Set log level to the level you need.
28.5 Database - Performance Considerations
The amount of database activity depends on several factors:
-
Whether the user, device, or browser is new or existing
-
If autolearning is used then the number of patterns
-
Number of policies and rules defined and the number of checkpoints
OAAM prints out every SQL if the property bharosa.db.query.performance.warning.threshold.ms is set to zero.
Set this property and try a typical login and then grep for the log messages with the string "ms execution for" or "SQLCall".
That will give you an estimate about the typical database activity of login.
Database Queries to Determine the Space Used
The following query can be used to determine the average size of row in tables:
.
select table_name,
avg_row_len
from user_tables
.
The following query can be used to determine the size of indexes of the tables:
.
select inds.table_name,
inds.index_name,
sum( inds.sizes ) as index_bytes_per_row
from (
select i.index_name,
i.table_name,
i.column_name,
decode(data_type, 'DATE' , 7,
'CHAR' , data_length,
'VARCHAR2', decode(
sign(data_length)-250, -1, .7*data_length+3, .7*data_length+1),
'NUMBER' ,
floor(nvl(data_precision,38)/2)+2 ) as sizes
from user_ind_columns i,
user_tab_columns t
where t.TABLE_NAME = i.table_name AND
t.COLUMN_NAME = i.COLUMN_NAME
order by i.table_name, i.column_name
) inds
group by inds.table_name, inds.index_name;
OAAM has indexes created out-of-the-box based on performance testing.
-
The out-of-the-box indexes should be sufficient for most deployments but database administrators may choose to add additional indexes if they feel they are required after performance testing. This is rare however unless transactional use cases are involved.
-
The database administrator may choose to adjust the database server parameters to tune I/O so that insets and updates are efficient
-
The database administrator should monitor the production environment until the database server is stable
Partitioning is an option that extends Oracle Database 11g Enterprise Edition out-of-the-box capabilities.
-
Deployments with more than 100K logins/transactions per day are recommended to utilize partitioning. When running the Oracle Fusion Middleware Repository Creation Utility (RCU) the partitioned scheme is used.
-
Databases 500 GB and over should use partitioning.
OAAM Database Indexes Contention - Oracle Real Application Cluster (RAC) Specific
If high index contention occurs, you may want to partition the following indexes:
-
PK_VT_TRX_LOGS
-
PK_VT_ENT_TRX_MAP
-
VT_TRX_LOGS_IDX3
-
VT_WF_MONTHS_IDX1
-
PK_VT_TRX_DATA
-
VT_WF_YEARS_IDX0
-
VT_WF_MONTHS_IDX0
-
VT_TRX_LOGS_IDX6
Also partition the VT_USER_PROFILE table.
Database I/O (input/output) performance problems may result if queries take a long time to run.
Query and audit activities tend to perform many sequential reads and table scans on the production index/tablespaces. To lessen the performance impact, you might consider maintaining a logical standby database using DataGuard where you can have an option to query, audit, and perform reporting using the logical standby database. The logical standby database would have all the data as production, except for the last one hour. The production database instance can just be used to perform its inserts, updates, and so on, and also for active monitoring and alerts.
28.6 Memory - Performance Considerations
Configure the application or content such that most of the executions are processing in memory.
The cache works by storing query results and making them available for later use. Use caching to return results in queries instead of pulling the information from the database each time. It is desirable that the information already exists in the cache. For example, cache user registration.
High Memory Usage Related to Caching
There are properties that you can use to control the maximum cache size. These properties are listed below:
Table 28-1 Cache Size Properties
| Property Name | Object Cached | Default Value |
|---|---|---|
|
vcrypt.fingerprint.cache.maxsize |
Fingerprints |
10000 |
|
bharosa.tracker.location.location.cache.maxsize |
IP Locations |
1000 |
|
bharosa.tracker.location.city.cache.maxsize |
Cities |
1000 |
|
bharosa.tracker.location.state.cache.maxsize |
States |
1000 |
|
bharosa.tracker.location.country.cache.maxsize |
Countries |
1000 |
A server restart is required after making changes.
When the appropriate property is set the cache will fill up to the size set and then the least recently used entries will be removed when a new entry is inserted.
-
Logging levels: Makes sure no unnecessary loggers are turned on.
-
Check that the JVM Settings are correct for specialized applications: Did you configure the right settings? Out of the box, most of these settings are optimal.
-
A slow database might add to disk I/O; thereby causing the application to run slowly.
28.7 Network - Performance Considerations
If the database is remote, queries to the database may require multiple hops. The more hops, the longer it takes for data to go from source to destination. Also, input can be affected if you are using Web services/SOAP and many sessions are being created.
Check if there are network issues.
Look for time-outs. Check that the network is set up correctly.
.Net/SOAP Web Service Integrated OAAM Deployment Seems Slow
Make sure the network communication between the .Net/SOAP application and OAAM Server is optimized. There should not be any extra hops which will add unwanted latency.
28.8 Hardware - Performance Considerations
-
Is there enough processing power? Check the processor and available memory. Are the disks fast enough?
-
Number of servers
-
Disk performance. Small system lose space because they are running a large number of log files so consider the number of processors configured for the database.
Insufficient CPU resources to satisfy demand may cause performance problems. For example, if there is not enough cache or the system is not up to requirements. Check the load and consumption on the CPU.