30 Oracle Entitlements Server Performance Tuning

This chapter provides guidelines for tuning and sizing Oracle Entitlements Server (OES). It contains these topics:

Section 30.1, "About Oracle Entitlements Server"
Section 30.2, "Performance Considerations for Oracle Entitlements Server"
Section 30.3, "Basic Tuning Considerations"

Note:

As with any enterprise class business application, there is no simple procedure for tuning that works for all systems. The tuning sections in this chapter provide (in some cases) sample configurations and outline the principles for tuning Oracle Identity Manager. Consider your own use case scenarios to determine which settings are appropriate.

30.1 About Oracle Entitlements Server

Oracle Entitlements Server (OES) is a standards-based, policy-driven security solution that provides real time fine-grained authorization in Application, Service-Oriented Architecture (SOA) and Database environments.

Oracle Entitlements Server allows an organization to protect its resources by defining and managing policies that control access to, and usage of, these resources. Access privileges are defined in a policy by specifying who can do what to which resource, when it can be done, and how. The policy can enforce controls on all types of resources including software components (URLs, Java Server Pages, Enterprise JavaBeans, methods, servlets and the like used to construct an application) and business objects (representations of user accounts, personal profiles and contracts such as bank accounts in a banking application, patient records in a health care application, or anything used to define a business relationship).

For more information on using Oracle Entitlements Server, see Oracle Fusion Middleware Administrator's Guide for Oracle Entitlements Server.

30.2 Performance Considerations for Oracle Entitlements Server

Effective Oracle Entitlement Server performance tuning starts with a good understanding of its usage and general performance issues. Before you begin tuning Oracle Entitlement Server performance, review this section as well as the recommendations discussed in Chapter 2, "Top Performance Areas".

Number of users	Understanding the overall user population size; group, membership, attribute counts, Grantees, Resource, permissions, entitlement & policies; data types, and configuration parameters of the LDAP and database is essential. See Chapter 3, "Performance Planning" for more information on using population data to improve performance.
Daily activity usage	It is important to know how many users are active during a 24-hour period and the expected traffic. Spikes in usage may require additional tuning to avoid performance issues. See Chapter 4, "Monitoring Oracle Fusion Middleware" for more information on collecting performance data.
Hardware resources and topology	Like any application deployed in demanding, real-time environments, proper server sizing and configuration is critical for acceptable Oracle Entitlements Server performance. Ensuring that your hardware is sufficient to prevent bottlenecks is a key factor in performance tuning Oracle Entitlements Server. See "Securing Sufficient Hardware Resources" for more information on optimizing hardware resources.
JVM and garbage collection	Optimal performance of Oracle Entitlements Server depends on correctly tuning JVM heap sizes and garbage collection. See "Configuring Garbage Collection" and "Specifying Heap Size Values" for more information.
Tuning the OES Caches	Understanding the OES caches (Decision cache and Permission cache) and tuning the same is important to achieve the optimal performance.

30.3 Basic Tuning Considerations

Depending on your OES usage and performance issues, you may consider tuning the following basic parameters. See Chapter 2, "Top Performance Areas" for additional tuning considerations.

This section includes the following topics:

Tuning the OES Policy Store
Tuning of OES Administration Server
Performance Tuning OES Security Modules

30.3.1 Tuning the OES Policy Store

OES policy store is the first component that needs to be tuned from the overall performance tuning. The tuning policy store involves tuning of the underlying database and the EclipseLink. The following sections provide information about the specific tuning aspects as a part of policy store tuning.

This section includes the following topics:

Oracle Database System Parameters Tuning
Table Spaces Tuning During User Schema Creation (RCU)
Tuning the OES Schema
EclipseLink Tuning

30.3.1.1 Oracle Database System Parameters Tuning

In addition to the standard database tuning parameters described in Section 2.6, "Tuning Database Parameters", the properties listed in Table 30-1 should also be tuned when using OES.

Table 30-1 Oracle Database System Parameters

Property Name	Value
`Processes`	1500
`sga_target`	3221225472 (3GB)
`audit_trail`	none
`open_cursors`	500
`pga_aggregate_target`	1610612736 (1.5GB)
`nls_sort`	BINARY
`filesystemio_options`	SETALL
`fast_start_mttr_target`	3600
`db_securefile`	ALWAYS
`session_cached_cursors`	500
`plsql_code_type`	NATIVE
`"_b_tree_bitmap_plans"`	False
`Memory_target`	0
`kernel.shmall`, `kernel.shmmax`, `kernel.sem` and `fs.file-max`	You must ensure that `kernel.shmall`, `kernel.shmmax`, `kernel.sem` and `fs.file-max` are set at optimal values (all kernel level parameter can be tuned and obtained from `/etc/sysctl.conf`.)

Note:

The above values are provided to show the common tuning configurations. There are exceptions and constraints. For example, SGA, PGA sizes are limited by the underlying operating system restrictions on the maximum available memory; on some platforms the processes can be calculated based on the connection pool settings. You must ensure that the tunings are validated by DBA's.

For more information about Oracle Database Tunings, see Section 2.7, "Reusing Database Connections".

30.3.1.2 Table Spaces Tuning During User Schema Creation (RCU)

The OES DB schema is created via RCU (Repository Creation Utility). To create a database schema using RCU, do the following:

Allocate large disk spaces for OES table spaces for optimal performances. Use 6GB files for IAS_OPSS and IAS_TEMP table spaces.
If you want to manage many large applications in the policy store, allocate huge table space size, for example, 40 GB for IAS_OPSS table space, 15 GB for IAS_TEMP table space.
To add applications to the policy store, mark the table space increment as 1GB, instead of 100MB, to avoid continuous extending of table space when it is full.

30.3.1.3 Tuning the OES Schema

The required OES database (DB) schema tuning should be complete if you have followed the steps mentioned in the above sections. Running the DB stats as the OES DB schema user is an on-going practice that must be monitored, because the policy data is stored in the DB.

Schema statistics should not be gathered for column JPS_DN.PARENTDN. Gathering histogram for column JPS_DN.PARENTDN impacts the query performance. To disable histogram, use the following SQL script:

EXECUTE dbms_stats.delete_column_stats(ownname=>'<SCHEMA_NAME>', tabname=>'JPS_DN', colname=>'PARENTDN', col_stat_type=>'HISTOGRAM');

Run the SQL script only once.

Run the command DBMS_STATS.gather_schema_stats('<SCHEMA_NAME>',DBMS_STATS.AUTO_SAMPLE_SIZE,no_invalidate=>FALSE)

You can automate the command by defining DB job to run the dB stats with required frequency. This collection must be done after a large policy data migration into the store.

30.3.1.4 EclipseLink Tuning

OES uses EclipseLink as JPA (Java Persistence API) implementation for Database operations. Table 30-2 defines the EclipseLink tuning parameters that can be set through jps-config.xml.

Table 30-2 EclipseLink Tuning

Property	Description	Default
`eclipselink.jdbc.connection_pool.default.max`	the maximum number of read connection in the internal connection pool	32
`eclipselink.jdbc.connection_pool.default.min`	the minimum number of read connection in the internal connection pool	32

The default property values can be overwritten by setting DB policy store service instance for OES admin, or PDP service in controlled-pull mode, as in the following:

      <propertySet name="props.db.1">
       <property name="jdbc.url"
       value="jdbc:oracle:thin:@slc04jpa.example.com:1521/orcl2"/>
       <property name="oracle.security.jps.farm.name" value="cn=my_domain"/>
       <property name="server.type" value="DB_ORACLE"/>
... ...

       <property name=" eclipselink.jdbc.connection_pool.default.min" value="16"/>
        <property name=" eclipselink.jdbc.connection_pool.default.min" value="64"/>
</propertySet>
... ...

<serviceInstance name="policystore.db" provider="policystore.provider">
       <property name="policystore.type" value="DB_ORACLE"/>
       <propertySetRef ref="props.db.1"/>
</serviceInstance>

For more information on EclipseLink properties, see http://www.eclipse.org/eclipselink/api/2.3/org/eclipse/persistence/config/PersistenceUnitProperties.html.

30.3.2 Tuning of OES Administration Server

This section provides information on how to tune the JVM parameters for optimal performance on large policy data for operations, like migration and distribution.

This section includes the following topics:

WLST Tuning
OES Admin Server Tuning

30.3.2.1 WLST Tuning

WebLogic Scripting Tool (WLST) is a command-line scripting interface of Oracle Middleware. It is used for OES for some management work, including migration of security store. For migrating large applications, you must tune the JVM parameters for the WLST tool for Java memory size, as in the following:

HotSpot:
"-Xms6144m -Xmx6144m -Xmn1900m -XX:PermSize=128m -XX:MaxPermSize=1024m"

JRockit:
"-Xms6144m -Xmx6144m -Xns1900m"

Note that the required heap size parameter (ms, mx, mn) could be based on the size of application you migrate. Table 30-3 provides a sample data:

Table 30-3 Data sample

Application	XML2DB	DB2XML	Number of Policies
Sample1KPolicyApp	650MB	1800 MB	1132
Sample5KPolicyApp	2.7 GB	6 GB	5351

30.3.2.2 OES Admin Server Tuning

OES Admin Server must be tuned for optimized performance for the policy distribution of large applications . The following are the tuning guidelines for OES Admin Server:

Use JDK version 1.6.0_31 or higher.
The graphic below provides information about the JDK memory size tuning. A large memory is required on Admin Server to generate application policy snapshot.

HotSpot:
"-Xms6144m -Xmx6144m -Xmn1900m -XX:PermSize=128m -XX:MaxPermSize=1024m"

JRockit:
"-Xms6144m -Xmx6144m -Xns1900m"

30.3.3 Performance Tuning OES Security Modules

When OES Security Module is configured as controlled pull mode, it has two main functionalities:

Acts as a PDP and/or PEP- it receives an authorization request, makes decision and enforces the decision.
Calls Policy Distribution component periodically to make configured policies and policy data available for evaluation.

When tuning the OES Security Module ensure that there is high throughput of authorization request processing and distribute the policy efficiently.

The size of the Policy data affects performance of security module. The number of functions, attributes, roles, resources, and policies have a great influence on authorization throughput. Distribution time and memory size is required for the Security Module.

For example:

Application Name:  myapp
oracle.security.jps.ldap.root.name:  cn=jpsroot
oracle.security.jps.farm.name:  cn=base_domain

Table 30-4 Security Module Tuning Attributes

Item	Number	Comment	Query SQL
Resource Type	1
Attribute definition	17
Function definition	269		select count `(*)` from `jps_dn` where parentdn like '`'cn=jpsroot,cn=jpscontext,cn=base_domain,cn=myapp,%cn=functions,'`;
Application Role	0	No application role is created. Weblogic Server's roles and principals are used.
Grantee	5351	Principals of policy are stored as grantee, one policy for one principal.	select count `(*)` from `jps_dn` where parentdn like `'cn=jpsroot,cn=jpscontext,cn=base_domain,cn=myapp,cn=jaas policy,cn=grantees,'`;
Permission	1772	Resource action pairs used in policies.	select count `(*)` from `jps_dn` where parentdn like `'cn=jpsroot,cn=jpscontext,cn=base_domain,cn=myapp,cn=jaas policy,cn=permissions,'`;
Resource	3173		select count `(*)` from `jps_dn` where parentdn like `'cn=jpsroot,cn=jpscontext,cn=base_domain,cn=myapp,%cn=resources,'`;
Entitlement	1806		select count `(*)` from `jps_dn` where parentdn like `'cn=jpsroot,cn=jpscontext,cn=base_domain,cn=myapp,%cn=permission sets,'`;
Policy	5351		select count `(*)` from `jps_dn` where parentdn like `'cn=jpsroot,cn=jpscontext,cn=base_domain,cn=myapp,%cn=policies,'`;
Rule	5351		select count `(*)` from `jps_dn` where parentdn like `'cn=jpsroot,cn=jpscontext,cn=base_domain,cn=myapp,%cn=rules,'`;
Attributes	1,484,356	Attributes of above entries	select count `(*)` from `jps_attrs` where `jps_dn_entryid` in (select entryid from `jps_dn` where parentdn like `'cn=jpsroot,cn=jpscontext,cn=base_domain,cn=myapp,%')`;
assignee attributes	1,220,877	Attributes for policy assignments	select count `(*)` from `ct_9_1` where `jps_dn_entryid` in (select entryid from `jps_dn` where parentdn like `'cn=jpsroot,cn=jpscontext,cn=base_domain,cn=myapp,cn=jaas policy,cn=permissions,')`;

For the above policy data sample runtime cache memory size can be computed as shown in Table 30-5

Table 30-5 Sample runtime cache memory size calculations

Cache Type	Total Heap Size	Formula
Resource Policy Cache	476.91M	0.4K * Assignee number
Permission Cache	143.78M	0.12K * Assignee number + 0.41K * Permission Number
Resource Cache	0.13M	44 * Resource number
Policy Cache	1.46MB	286 * Policy number
Resource Type Cache	Small one, ignore them
Function Cache	Small one, ignore them
Attribute Cache	Small one, ignore them

For a better performance of large data operations, a total of 6GB heap size is required for Security Module.

JVM Tunings for Security Module: (WLS/Tomcat/Java)

HotSpot JVM:

"-Xms6144m -Xmx6144m -Xmn1900m -XX:PermSize=128m -XX:MaxPermSize=1024m  -XX:+UseParallelGC "

-XX :+UseParallelGC: Use parallel garbage collection for scavenges. This collection algorithm is designed to maximize throughput while minimizing pauses and a suitable GC algorithm for Security Module.

JRockit:

"-Xms6144m -Xmx6144m -Xns1900m"

This section includes the following topics:

"Tuning OES Distribution Service"
"Tuning OES Cache"
"Network Considerations"
"Resource Intensive Operations"
"Enable Logging for Performance Measurement"

30.3.3.1 Tuning OES Distribution Service

PDP service configuration

oracle.security.jps.pd.client.PollingTimerInterval:

This property specifies PD periodic check interval. The default value is 600 seconds. Depending on how often policy data is changed, it can be increased or decreased as needed.

PDP service configurations for WSSM:

Table 30-6 lists the available parameters for WS SM

Table 30-6 Available Parameter for WSSM

Property	Description
`oracle.security.jps.pdp.sm.IdentityCacheEnabled`	Flag to set if using identity cache. If not set, identity cache is enabled by default
`oracle.security.jps.pdp.sm.IdentityMaxCacheSize`	Specify the maximum cache size. The default value is 20000. Tune this parameter according to the number of all subjects.
`oracle.security.jps.pdp.sm.IdentityCacheEvictionPercentage`	Specifies percentage of identities that must be evicted when cache has reached the maximum size. The default value is 20 percentage.
`oracle.security.jps.pdp.sm.IdentityCachedEntryTTL`	Specifies time-to-live in seconds for a identity record in the cache. The default value is 3600 seconds.
`oracle.security.jps.pdp.wssm.WSLoggingSoapHandlerEnabled`	Flag to set if enable `EnvelopLoggingSOAPHandler` of ws sm. The default value is false.

30.3.3.2 Tuning OES Cache

This section describes the decision cache attributes of the Security Module.

Set the following properties in the jps-config.xml of PDP:

<property name="oracle.security.jps.pdp.AuthorizationDecisionCacheEnabled" value="true"/>
<property name="oracle.security.jps.pdp.AuthorizationDecisionCacheEvictionCapacity" value="1500"/>
<property name="oracle.security.jps.pdp.AuthorizationDecisionCacheEvictionPercentage" value="10"/>
<property name="oracle.security.jps.pdp.AuthorizationPerUserDecisionCacheSize" value="1500"/>
<property name="oracle.security.jps.pdp.AuthorizationDecisionCacheTTL" value="300"/>

Table 30-7 Introduction to the Properties

Property Name	Description	Accepted Values	Optional/Mandatory	Comments
`oracle.security.jps.pdp.AuthorizationDecisionCacheEnable`	Specifies whether the the policy decision cache should be enabled.	"true/false" Default: true	Optional	Since DW PS1
`oracle.security.jps.pdp.AuthorizationDecisionCacheEvictionCapacity`	This is the number that is used to evict the decision cache if the decision cache size reaches this size.	Number Default:500	Optional	Since DW PS1
`oracle.security.jps.pdp.AuthorizationDecisionCacheEvictionPercentage`	The percentage of authorization decisions to drop when the decision cache has reached the maximum capacity.	Number Default:10	Optional	Since DW PS1
`oracle.security.jps.pdp.AuthorizationDecisionCacheTTL`	This is the TTL for the Decision Cache in seconds	Number Default:60	Optional	Since DW PS1
`oracle.security.jps.pdp.AuthorizationPerUserDecisionCacheSize`	This is the number that is used to evict the decision cache of each user (Subject) if the second level decision cache size reaches this size.	Number Default:1000	Optional	Since DW PS2

30.3.3.3 Network Considerations

The performance of the overall network is a major factor in the performance of the system. A reduction in network latency can improve network performance.

To control network latency, consider the following:

Keep database repositories close to the OES servers. Installing OES servers on a remote server may cause significant latency. Latency between the application tier and the database tier should be 5ms or less to maintain optimal performance.
Add an SSL accelerator or load balancer outside of the Oracle Entitlement Server system to improve the performance of your network.
Deploying a load balancer in front of the Web servers or application servers is a best practice for increasing availability and performance of Web-based applications, including Oracle Entitlement Server. However, load balancers are not recommended between the Oracle Entitlement Server components themselves.
Place the OES Servers closer to client applications than to the database.

30.3.3.4 Resource Intensive Operations

Optimization was made for process of PepRequestFactory.newQueryPepRequest().decide(), that does authorization check for a bunch of resources.

This feature is disabled by default and should be enabled for SM instances in controlled distribution mode.

To enable it, you should update jps-config.xml of SM instances. For a PDP service instance, add the following property.

<serviceInstance name="pdp.service" provider="pdp.service.provider">
         ... ...
         property name="oracle.security.jps.runtime.enableResourcePermissionCache" value="true"/>

30.3.3.5 Enable Logging for Performance Measurement

Logs are required for performance measurement for some key operations, including migration, snapshot generation and distribution.

There is log in INFO level for migration via WLST. You could see logs with timestamp print to WLST console directly, as shown in the following:

Apr 19, 2013 3:07:00 AM oracle.security.jps.internal.tools.utility.JpsUtilMigrationPolicyImpl
migrateAppPolicyData
INFO: Migration of Application Policies in progress.....

... ...
Apr 19, 2013 3:07:39 AM oracle.security.jps.internal.tools.utility.util.JpsMigrateUtil cloneResourceType
INFO: Migration of Resources started
Apr 19, 2013 3:08:12 AM oracle.security.jps.internal.tools.utility.util.JpsMigrateUtil cloneResourceType
INFO: Migration of Resources completed in 00:00:33

... ...
Apr 19, 2013 3:08:44 AM oracle.security.jps.internal.tools.utility.util.JpsMigrateUtil clonePermissionSet
INFO: Migration of Permission Sets completed in 00:00:31

... ...
Apr 19, 2013 3:11:35 AM oracle.security.jps.internal.tools.utility.destination.apibased.JpsDstPolicy clone
INFO: Completed Migrating 5,351 policies in [170,914] ms.

... ...
Apr 19, 2013 3:20:57 AM oracle.security.jps.internal.tools.utility.destination.apibased.JpsDstPolicy clone
INFO: Migration of Grants completed in 00:09:22

... ...
Apr 19, 2013 3:20:57 AM oracle.security.jps.internal.tools.utility.JpsUtilMigrationPolicyImpl
migrateAppPolicyData
INFO: Migration of Application Policies completed, Time taken for migration is 00:13:57

Enable Logging for Snapshot Generation Measurement

The following are the logging properties that need to be enabled to measure the timing for snapshot generation as a part of policy distribution. Create Java logging.properties file with log level setting as shown in the following:

handlers= java.util.logging.FileHandler
.level= INFO
############################################################
# Handler specific properties.
# Describes specific configuration info for Handlers.
############################################################# default file output is in user's home directory
java.util.logging.FileHandler.pattern =<Log_home>/performance.log
java.util.logging.FileHandler.limit = 10000000
java.util.logging.FileHandler.count = 200
java.util.logging.FileHandler.formatter =java.util.logging.SimpleFormatter
java.util.logging.FileHandler.level = FINE
java.util.logging.ConsoleHandler.level = INFO
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter
############################################################
# Facility specific properties.
############################################################
# Provides extra control for each logger.
############################################################
# For example, set the com.xyz.foo logger to only log SEVERE
# messages:
oracle.security.jps.internal.policystore.SnapshotWorker.level=FINEST

Enable log in <DOMAIN_HOME>/bin/startWeblogic.sh

${JAVA_HOME}/bin/java ${JAVA_VM} ${MEM_ARGS} -Djava.util.logging.config.file=./logging.properties -Dweblogic.Name=${SERVER_NAME} -Djava.security.policy=${WL_HOME}/server/lib/weblogic.policy ${JAVA_OPTIONS} ${PROXY_SETTINGS} ${SERVER_CLASS}

From the log file, you should see the following log message when you generate a snapshot of application policy:

Jan 31, 2013 1:07:49 AM oracle.security.jps.internal.policystore.SnapshotWorker run
FINE:SnapshotWorker begin.
.......... 
Jan 31, 2013 1:12:43 AM oracle.security.jps.internal.policystore.SnapshotWorker run
FINE: SnapshotWorker end.

Enable Logging for Distribution

The following are the logging properties that need to be enabled to measure the timing for snapshot generation as a part of policy distribution. Create a logging.properties file and add the following line:

oracle.security.jps.az.internal.runtime.pd.receiver.UpdatePolicySet.level=FINEST

You must update different script for different Security Module type.

Java SM

Edit the java SM start scripts to add log properties.

Take<OES_CLIENT_HOME>/oes_sm_instances/<SM_NAME>/run-j2se.sh as example.

${JAVA_HOME}/bin/java -Doracle.security.jps.config=${OES_INSTANCE_HOME}/config/jps-config.xml
${MEM_ARGS} -Djava.util.logging.config.file=./config/logging.properties
oracle.security.oes.tools.OESJ2SESampleApp PD

And enable oracle.security.jps.az.internal.runtime.pd.receiver.UpdatePolicySet.level=FINEST in the log property file.

WS SM

In <OESCLIENT>/oes_sm_instances/<wssm>/startWSServer.sh add following line:

JAVA_OPT="$JAVA_OPT -Djava.util.logging.config.file=./config/logging.properties".

Weblogic SM

Edit file <DOMAIN_HOME>/bin/startWeblogic.sh in the end:

${JAVA_HOME}/bin/java ${JAVA_VM} ${MEM_ARGS}
-Djava.util.logging.config.file=./config/logging.properties

Tomcat SM

Edit file <TOMCAT_HOME>/ bin/catalina.sh, and add the following line after the last JAVA_OPTS line:

JAVA_OPTS="$JAVA_OPTS -Djava.util.logging.config.file=./config/logging.properties".

Jan 31, 2013 10:11:19 PM oracle.security.jps.az.internal.runtime.pd.receiver.UpdatePolicySet commit
FINE: Starting ...

...........
Jan 31, 2013 10:20:06 PM oracle.security.jps.az.internal.runtime.pd.receiver.UpdatePolicySet commit
FINE: completed. Active Apps: [[]], Bounded Apps [[fidelity5000]
Jan 31, 2013 10:20:07 PM oracle.security.jps.az.internal.runtime.pd.receiver.UpdatePolicySet commit
FINE: afterCommit is finished
Jan 31, 2013 10:20:07 PM oracle.security.jps.az.internal.runtime.pd.receiver.UpdatePolicySet finishFirstDist