Oracle® Fusion Applications Performance and Tuning Guide 11g Release 5 (11.1.5) Part Number E16686-06 |
|
|
PDF · Mobi · ePub |
This chapter discusses how to find the information you need to examine so you can tune your system. It includes how to monitor and tune the database and Oracle Fusion Applications, and troubleshooting.
This chapter includes these sections:
Section 1.2, "Monitoring and Tuning Oracle Fusion Applications"
Section 1.3, "Tuning Platforms for Oracle Fusion Applications"
Every system of hardware and installed applications is different. Even though Oracle Fusion Applications are written and installed using industry-standard best practices, you can custom tailor your system to improve how it supports your environment.
But to tune your system, you need to locate and examine data. This chapter will explain what data you need to examine, and what tools you will use to gather the data.
In general, most of the settings that come default in Oracle Fusion Applications are already tuned.
These guidelines are provided to help ensure your Oracle Fusion Applications instance runs optimally. Note that all metrics listed are from Oracle Enterprise Manager Cloud Control.
Monitor the key host metrics, shown in Table 1-1, to ensure the underlying server hosts are healthy. Rather than constantly checking the metric values, you can set up alert thresholds in Cloud Control and receive notification when thresholds are exceeded. For more information, see the see "Creating Monitoring Templates" in the Oracle Fusion Applications Administrator's Guide.
Monitor the key component metrics, such as WebLogic server metrics, to ensure each component is healthy.
Monitor the number of incidents and logs to ensure the application is configured properly and not constantly wasting resources generating error messages. Review log levels to ensure they are not set too low. See "Troubleshooting Oracle Fusion Applications Using Incidents, Logs, QuickTrace, and Diagnostic Tests" in the Oracle Fusion Applications Administrator's Guide for more information
Monitor the database to ensure it is operating optimally. Follow the guidelines in Chapter 3, "Tuning the Database," to make sure that statistics are being collected.
Table 1-1 Key Host Metrics
Metric Category | Metric Name | Warning Threshold | Critical Threshold | Comments |
---|---|---|---|---|
Disk Activity |
Disk Device Busy |
>80% |
>95% |
|
Filesystems |
Filesystem Space Available |
<20% |
<5% |
|
Load |
CPU in I/O wait |
>60% |
>80% |
|
CPU Utilization |
>80% |
>95% |
||
Run Queue (5 min average) |
>2 |
>4 |
The run queue is normalized by the number of CPU cores. |
|
Swap Utilization |
>75% |
>90% |
||
Total Processes |
>15000 |
>25000 |
||
Logical Free Memory % |
<20 |
<10 |
||
CPU in System Mode |
>20% |
>40% |
||
Network Interfaces Summary |
All Network Interfaces Combined Utilization |
>80% |
>95% |
|
Switch/Swap Activity |
Total System Swaps |
>3 |
>5 |
Value is per second. |
Paging Activity |
Pages Paged-in (per second) |
|||
Pages Paged-out (per second) |
The combined value of Pages Paged-in and Pages Paged-out should be <=1000 |
Administrators will find it useful to study these suggestions on further analysis to undertake when a metric value exceeds threshold. The commands provided are for the Linux operating system.
When logical free memory/swap activity or paging activity is beyond threshold
This usually happens when memory is not sufficient to handle demands from all the running processes.
Linux: Check cat/proc/meminfo
and confirm total RAM is expected.
Windows: Open the Task Manager, click the Performance tab and check the Physical Memory section.
Check if there are unallocated huge pages. If there are and the WebLogic Server/Oracle instances are not expected to use them, reduce the huge page pool size.
Linux: Run top
and sort by resident memory (type OQ). Look for processes using the most resident memory and investigate those processes.
Windows: Open the Task Manager, click the Processes tab and click the Mem Usage column to sort the processes by memory usage.
When page activity is beyond threshold
Follow the steps in "When logical free memory/swap activity or paging activity is beyond threshold" to view and analyze memory usage.
When Network Interface Error Rates Is Beyond Threshold
The normal cause is misconfiguration between the host and the network switch. A bad network card or cabling also can cause this error. You can run /sbin/ifconfig
to identify which interface is having packet errors. Contact network administrator to ensure the host and the switch are using same data rate and duplex mode.
Otherwise, check if cabling or the network card is faulty and replace as appropriate.
When Packet Loss Rate Is Beyond Threshold
The normal cause of this error is network saturation of bad network hardware.
Run lsof -Pni | grep ESTAM
to determine which network paths are generating the problem.
Then run mtr <target host>
or ping <target host>
and look for packet lost on that segment.
20 packets transmitted, 20 received, 0% packet loss, time 18997ms rtt min/avg/max/mdev = 0.168/0.177/0.200/0.010 ms
The packet loss should be 0% and rtt should be less than .5 ms.
Ask the network monitoring staff to look for saturation or network packet loss from their side.
When Network Utilization Is Beyond Threshold
The normal cause is very heavy application load.
Run top
or lsof
to determine which processes are moving a lot of data.
Use tcpdump
to sample the network for usage patterns.
Use atop
, iftop
, ntop
or pkstat
to see which processes are moving data.
When CPU Usage or Run Queue Length Is Beyond Threshold
The normal cause is runaway demand, a poorly performing application, or poor capacity planning.
Linux: Run top
to identify which application/process is using time.
Windows: Open the Task Manager, click the Processes tab and click the CPU column to sort the processes based on CPU usage.
If top processes are WebLogic Server JVM processes, conduct a basic WebLogic Server health check. That is, review logs to see if there are configuration errors causing excessive exceptions, and review metrics to see if the load has increased. Use JVMD for a more detailed analysis.
If top processes are Oracle processes, use Enterprise Manager to look for high load SQL.
When System CPU Usage Is Beyond Threshold
High system CPU use could be due to kernel processes looking for pages to swap out during a memory shortage. Follow the steps listed in the "When logical free memory/swap activity or paging activity is beyond threshold" section to further diagnose the problem.
High system CPU use is also frequently related to various device failures. Run {{dmesg | less}}
and look for repeated messages about errors on some particular device, and also have hardware support personnel check the hardware console to see if there are any errors reported.
When Filesystem Usage Is Beyond Threshold
The normal cause is an application that is logging excessively or leaving behind temporary files.
Run lsof -d 1-99999 | grep REG | sort -nrk 7 | less
to see currently open files sorted by size from largest to smallest. Investigate the large files.
Run du -k /mount_point_running_out_of_space > /tmp/sizes
to get space used for directories under the mount point. This may take a long time. While it is running, run sort -nr /tmp/sizes
and find the directories using most space and investigate those first.
When Total Processes Is Beyond Threshold
The normal cause is runaway code or a stuck NFS filesystem.
Linux: Run ps aux
. If many processes are in status D, run df
to check for stuck mounts.
Windows: Run Task Manager, click the Processes tab, and check the list of running processes.
If there are hundreds or thousands of processes of a particular program, determine why.
Run ps o pid,nlwp,cmd | sort -nrk 2 | head
to look for processes with many threads.
When Disk Device Busy Is Beyond Threshold
Check for disk drive failure.
Linux: As root, check /var/log/messages*
and /var/log/mcelog
to see if there are any error messages indicating disk failure. For a RAID array, the disk controller needs to be checked. The commands will be specific to the controller manufacturer.
Windows: Run perfmon
and look at the Alert logs. Run chkdsk
to check for disk failure.
Look for processes that are using the disk. From a shell window, execute ps aux | grep ' D. '
several consecutive times to look for processes with "stat" D.
Poor performance is a major indicator of network connectivity problems.
Check for cumulative dropped packets drops for each host.
Linux
netstat -s | grep 'TCP data loss' 4007 segments retransmited 3302 TCP data loss events
Windows
netstat -s
Look for "TCP Statistics for IPv4" and for "Segments Retransmitted."
The counts for Linux and Windows should be 0 or growing very slowly over time.
Check for realtime dropped packets on specific network paths. The ping
command is the same for Windows and Linux systems.
ping -c 20 other_host 20 packets transmitted, 20 received, 0% packet loss, time 18997ms rtt min/avg/max/mdev = 0.168/0.177/0.200/0.010 ms
Packet loss should be 0%.
rtt should be less than .5 ms, except that it can be higher between the browser and load balancer.
In Windows, the command
ping -n 20 other_host
will show similar output, only with more information:
Ping statistics for 123.45.67.89: Packets: Sent = 20, Received = 20, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 96ms, Maximum = 121ms, Average = 104ms
Check for network interface errors.
Linux
/sbin/ifconfig eth0 | grep errors RX packets:842803463 errors:0 dropped:0 overruns:0 frame:0 TX packets:667946307 errors:0 dropped:0 overruns:0 carrier:0
Windows
ipconfig /all
will not show TX or RX packets information. However, Microsoft Network Monitor (free from the Microsoft Download site) will, when it is installed and configured properly.
These metrics provide an indication of whether the WebLogic Server is in a healthy state. Performance may degrade if any of the metrics is exceeding its threshold.
Table 1-2 describes the WebLogic Server metrics you should monitor in Cloud Control. See the "Creating Monitoring Templates" section in the Oracle Fusion Applications Administrator's Guide to create a monitoring template.
Table 1-2 WebLogic Server Metrics
Metric Category | Metric Name | Warning Threshold | Critical Threshold | Comments |
---|---|---|---|---|
Datasource Metrics |
Connections in Use |
>250 |
>400 |
|
Connection Requests that Waited (%) |
>10% |
>20% |
||
Connection Creation Time (ms) |
||||
JVM Garbage Collectors |
Garbage Collector - Percent Time spent (elapsed) |
>10% |
>20% |
|
JVM Metrics |
Heap Usage |
>90% |
>98% |
|
Response |
Status |
=Down |
This provides instance availability. |
|
Server Servlet/JSP Metrics |
Request Processing Time (ms) |
>10s |
>15s |
|
Server Work Manager Metrics |
Work Manager Stuck Threads |
>5 |
>10 |
|
JVM Threads |
Deadlocked Threads |
>2 |
>5 |
|
Module Metrics By Server |
Active Sessions |
When CPU Usage On Host Is Beyond Threshold and WebLogic Server Process Is Identified as Top CPU Consumer
Examine the % Time spent in the GC metric to see if JVM is doing excessive GC (>60 percent). If so, follow the process for diagnosing WebLogic Server heap pressure.
Look for incident creation rate and error logs and see if something is triggering a massive amount of logging/errors.
In JVMD, select the CPU state filter and look at top methods. Look for threads that are consistently in a CPU state.
When There Is a Spike in Active Web Sessions
Check access logs to see if there is a spike in the number of users.
Check if there are stuck threads, which could cause users to log in again.
Check session distribution across WebLogic Server managed servers and see if there is a problem with the load balancer.
Check session timeout in web.xml
, and see if it is too high or too low.
When There Are Stuck Threads On the System
Get the ECID from the stuck thread error in the WebLogic Server log.
From the Request Monitor, search for the ECID and get details from JVMD.
Alternatively, use JVMD to search for stuck threads and see the timing breakdown.
A stuck thread will also result in an incident with a JFR recording. Use JRMC to analyze the recording.
When There Are Deadlocks Detected On the System
In JVMD, inspect the threads that are in a blocked state.
Deadlock threads normally also will be reported as a stuck thread in the WebLogic Server log. Use the Request Monitor to search for the ECID and expand down into JVMD to show the blocking thread.
When Request Processing Time Is Beyond Threshold
Examine the % Time spent in GC metric to see if JVM is doing excessive garbage collection
Look for incident create rate and error logs and see if something is triggering a massive amount of logging/errors.
In JVMD, look at the thread states and see where most processing time is going.
Check the metric Garbage Collection - Invocation Time (ms) under the JVM Garbage Collectors metric category. Sometimes if you run many managed server instances on the same host, you may be able to reduce time spent in garbage collection by reducing the number of garbage collector threads in each JVM. The default is based on the number of CPUs and could be too high if there are multiple active JVMs running on the same machine. In those cases, if you are using JRockit, add the -XXgcThreads=4
option when starting the JVM. To add the option, edit the DOMAIN_HOME
/bin/fusionapps_start_params.properties
file, look for -Xgc:genpar
and add the -XXgcThreads=4
option after it (for example -Xgc:genpar -XXgcThreads=4
). The value 4 directs the JVM to use four threads to perform garbage collection. You can try different values from 4 to the number of CPU cores and observe if the % Time spent in GC metric improves. For other platforms, see Section 1.3, "Tuning Platforms for Oracle Fusion Applications."
When Percent Time Spent in GC Is Beyond Threshold
Check the session count. If there is a sudden surge of sessions due to user load, the JVM could be short on heap. Increase heap if possible, or add additional managed server instances.
Look at the stuck threads count. Stuck threads could increase the number of active session, as users could be launching new sessions hoping for a faster response.
Look at the incident creation rate and error logs and see if something is triggering a massive amount of logging/errors. The incident creation/logging operations could be causing a high amount of object creation and garbage collection stress.
Generate a heap dump using JVMD and analyze the top retainer of memory.
Use JRMC to connect and extract a JFR recording. Examine the Memory panel and allocation details to see what is doing a lot of allocations.
When Percent Connection Requests Waiting Is Beyond Threshold
Examine the number of sessions and request rate, and see if there is a spike in the load that would account for an increased demand for connections.
In JVMD, see where time is spent. For example, requests could be running longer due to slow SQLs (and retain the connection longer). In that case, identify and tune slow SQLs.
Consider increasing the initial capacity setting of the corresponding data source.
These metrics provide an indication of whether the Oracle HTTP Server is in a healthy state. Performance may degrade if any of the metrics is exceeding its threshold.
Table 1-3 describes the Oracle HTTP Server metrics you should monitor in Cloud Control. See the "Creating Monitoring Templates" section in the Oracle Fusion Applications Administrator's Guide to create a monitoring template. Also see Section 1.4, "Tuning Oracle HTTP Server," and "Monitoring the Oracle Fusion Applications Middle Tier" in the Oracle Fusion Applications Administrator's Guide.
Table 1-3 Oracle HTTP Server Metrics
Metric Category | Metric Name | Warning Threshold | Critical Threshold |
---|---|---|---|
OHS Server Metrics |
Busy Threads (%) |
>85% |
>95% |
Request Throughput (requests per second) |
TBD |
Yes |
|
OHS Response Code Metrics |
HTTP 4xx errors |
||
HTTP 5xx errors |
|||
OHS Virtual Host Metrics |
Request Processing Time for a Virtual Host |
>10s |
>15s |
When Busy Threads % Is Beyond Threshold
Check request throughput to see if load has increased. If the increased load is expected and CPU and memory resources on the OHS host has not exceeded threshold, consider increasing ServerLimit/MaxClients and ThreadsPerChild in httpd.conf
.
Check request process time on both OHS and underlying WebLogic Server to see if requests are taking longer. If WebLogic Server response time is increasing, check the key metrics for the WebLogic Server.
If possible, ensure the client browser cache is enabled to reduce number of requests submitted.
Check OHS Response Code Metrics. If there is a sudden increase of HTTP 4xx errors or HTTP 5xx errors, check the health of the underlying WebLogic Servers.
Check and increase the minimum and maximum spare threads for Oracle HTTP Server.
In the httpd.conf
file located in instance_home/config/ohs/<ohs_name>/httpd.conf
:
Increase MaxSpareThreads to 800.
Increase MinSpareThreads to 200.
When Request Processing Time for a Virtual Host Exceeds Threshold
Check the key host metrics to ensure the OHS host is healthy.
For each URL requested, OHS will first check DocumentRoot before passing the request to WebLogic Server. Check the utilization and health of the disk to which the DocumentRoot is pointing. If it is a NFS mount, check the health of the NFS mount point.
Check the key metrics for the underlying WebLogic Server(s) and see if they are healthy.
OHS accesses /tmp for each POST request, so check the performance of the /tmp filesystem.
These metrics provide an indication of whether the Oracle Business Intelligence Server is in a healthy state.
To start monitoring:
Log in to Oracle Enterprise Manager Fusion Applications Control. See the "Starting Fusion Applications Control" section in the Oracle Fusion Applications Administrator's Guide.
Note: Creating a monitoring template for Oracle Business Intelligence Server metric targets is not available in Cloud Control.
From the navigation pane, expand the farm, and then Business Intelligence.
Click coreapplication.
From the Business Intelligence Instance menu, choose Monitoring > Performance, as shown in Figure 1-1.
Use Fusion Applications Control to configure parameters related to Oracle Business Intelligence Suite Enterprise Edition.
Fusion Applications Control can monitor various BI components, including:
Weblogic Analytics Application
Oracle BI Presentation Services
Oracle BI Server
Oracle Weblogic Server (administration and managed servers)
Oracle Access Manager and Oracle Identity Manager are both WebLogic Server instances. See Section 1.2.3, "How to Analyze WebLogic Server Metrics" to monitor their health. See also Section 1.2.9, "How to Tune and Troubleshoot Oracle Identity Management."
Use Cloud Control to monitor the Oracle Internet Directory and Oracle Identity Manager databases. For information on creating monitoring templates in Cloud Control to obtain metrics, see the "Creating Monitoring Templates" section in the Oracle Fusion Applications Administrator's Guide.See Table 1-4 for Oracle Identity Manager metrics.
Table 1-4 Oracle Identity Manager Metrics
Metric Category | Metric Name | Warning Threshold | Critical Threshold |
---|---|---|---|
Oracle Identity Manager Cluster |
|||
Provisioning Requests |
Completed Provisioning Requests |
||
Failed Provisioning Requests |
|||
Reconciliations (Last 24 Hours) |
Jobs Completed |
||
Response |
Status |
||
Role Grant Requests |
Completed Role Grant Requests |
||
Completed Role Grant Requests Processing Time (per sec) |
|||
Failed Role Grant Requests |
|||
Pending Role Grant Requests |
|||
Self Service Requests |
Completed Self Service Requests |
||
Completed Self Service Requests Processing Time (per sec) |
|||
Failed Self Service Requests |
|||
Pending Self Service Requests |
|||
Oracle Identity Manager Server |
|||
Response |
Status |
||
Resource Utilization |
CPU Utilization (%) |
80 |
90 |
Memory Utilization (%) |
80 |
90 |
|
Adapters |
Average Adapter Execution Time (ms) |
||
Completed Adapter Executions |
|||
Maximum Adapter Execution Time (ms) |
|||
Minimum Adapter Execution Time (ms) |
See Table 1-5 for Oracle Internet Directory metrics.
Table 1-5 Oracle Internet Directory Metrics
Metric Category | Metric Name | Warning Threshold | Critical Threshold |
---|---|---|---|
LDAP Operation Response Time |
Bind Operation Response Time |
||
Request Processing Time (ms) |
|||
LDAP Server Resource Usage |
Total CPU Usage (%) |
80 |
90 |
Total Memory Usage (%) |
80 |
90 |
|
Response |
Status |
||
(Critical Events) System Resource Events (3113 Errors) |
Number of 3113 Error Occurrences |
||
(Critical Events) System Resource Events (3114 Errors) |
Number of 3114 Error Occurrences |
||
LDAP Failed Bind Operations Profile |
Failed Bind Operations |
||
LDAP Server Resource Usage |
Completed Bind Operations |
||
Completed Compare Operations |
|||
Completed Modify Operations |
|||
Completed Search Operations |
|||
Total Operations |
|||
User LDAP Operations Statistics |
Failed Base Search Operations |
||
Failed Bind Operations |
|||
Failed Compare Operations |
|||
Failed Delete Operations |
|||
Successful Base Search Operations |
To enable the collection of user LDAP Operation Statistics, edit the configuration using Fusion Applications Control:
From the navigation pane, expand the farm and then the Oracle Internet Directory target.
From the Administration menu, choose Server Properties.
Click User Statistics Collection to enable this feature.
Click Apply to save your changes.
From the Administration menu, choose Shared Properties.
Enter a valid DN (for example, cn=orcladmin
) to enable user statistics collection for that user.
The metrics shown in Table 1-6 provide an indication of whether the Enterprise Scheduler instance is performing well. See the "Creating Monitoring Templates" section in the Oracle Fusion Applications Administrator's Guide.
Table 1-6 Key Enterprise Scheduler Metrics
Metric Category | Metric Name | Warning Threshold | Critical Threshold | Comments |
---|---|---|---|---|
Completed Job Summary |
Average Elapsed Time (ms) |
You can define different thresholds for different job names. |
||
Long Running Job |
Elapsed Time (ms) |
|||
WorkAssignment Metrics aggregated across Group Members |
Average Wait Time for Requests in Ready State (seconds) |
When the Value of Average Elapsed Time for the Completed Jobs Is Higher Than Expected
Check the key host and WebLogic Server metrics and see if any component that could be involved in process batch jobs is in an unhealthy state.
Follow the steps listed in Section 2.1.4, "How to Troubleshoot Slow Batch Jobs" and analyze several jobs to see if there are any common causes.
When the Value of Elapsed Time Under the Long Running Job Category Is Higher Than Expected
Open the Enterprise Scheduler home page in Oracle Enterprise Manager Fusion Applications Control and examine the Top 10 Long Running Jobs.
Identify the job of interest, and follow the steps in Section 2.1.4, "How to Troubleshoot Slow Batch Jobs."
When Average Wait Time For Requests in Ready State (seconds) Is Higher Than Expected
Follow the steps in Section 2.1.4.1, "Troubleshooting Jobs That Are in Wait/Ready/Blocked State for a Long Time."
Monitoring SOA involves monitoring SOA Infrastructure, SOA Composite and SOA servers. See the "Creating Monitoring Templates" section in the Oracle Fusion Applications Administrator's Guide.
Use Table 1-7 to locate the key performance metrics for SOA Composite.
Table 1-7 SOA Composite Metrics
Metric Category | Metric Name |
---|---|
Mediator Case |
Invocation count throughput in last 5 minutes |
SOA Composite - Response Metrics |
Composite Status |
SOA Composite - Component Detail Metrics |
Component: Business Faults |
Component: Error Rate (%) |
|
SOA Composite - Services/References Detail Metrics |
Service/Reference: Average Incoming Messages Processing Time (ms) |
Service/Reference: Average Outbound Messages Processing Time (ms) |
|
SOA Composite: Error Rate (%) |
|
SOA Composite: Synchronous Response Time (ms) |
|
SOA Composite: Total Business |
Use Table 1-8 to locate the key performance metrics for SOA Infrastructure.
Follow the steps in this section to tune Oracle Identity Management specifically for Oracle Fusion Applications.
Most of these settings should be set by default if your environment is newly provisioned. If your environment is upgraded from a previous release, you will need to manually check and adjust the settings.
Optimize LDAP Search
Description: Optimize LDAP search by enabling search filters.
Solution:
Create an ldiff file named searchfilter_oid_tuning.ldif
with this content:
dn: cn=dsaconfig, cn=configsets, cn=oracle internet directory changetype: modify add: orclinmemfiltprocess;dn orclinmemfiltprocess;dn: cn=Roles,cn=fscm,cn=FusionDomain,cn=JPSContext,cn=FusionAppsPolicies orclinmemfiltprocess;dn: cn=Roles,cn=crm,cn=FusionDomain,cn=JPSContext,cn=FusionAppsPolicies orclinmemfiltprocess;dn: cn=Roles,cn=hcm,cn=FusionDomain,cn=JPSContext,cn=FusionAppsPolicies orclinmemfiltprocess;dn: cn=Permission Sets,cn=fscm,cn=FusionDomain,cn=JPSContext,cn=FusionAppsPolicies orclinmemfiltprocess;dn: cn=Permission Sets,cn=hcm,cn=FusionDomain,cn=JPSContext,cn=FusionAppsPolicies orclinmemfiltprocess;dn: cn=Permission Sets,cn=crm,cn=FusionDomain,cn=JPSContext,cn=FusionAppsPolicies orclinmemfiltprocess:dn; cn=Permissions,cn=JAAS Policy,cn=fscm,cn=FusionDomain,cn=JPSContext,cn=FusionAppsPolicies orclinmemfiltprocess;dn: cn=hcm,cn=FusionDomain,cn=JPSContext,cn=FusionAppsPolicies orclinmemfiltprocess;dn: cn=crm,cn=FusionDomain,cn=JPSContext,cn=FusionAppsPolicies orclinmemfiltprocess;dn: cn=fscm,cn=FusionDomain,cn=JPSContext,cn=FusionAppsPolicies
At the command prompt, run this command:
ldapmodify -p portNum -h hostname -D cn=orcladmin -f searchfilter_oid_tuning.ldif
Log Levels
Description: Oracle Identity Management stack WebLogic Server log levels are too fine-grained and need to be set to Severe.
Solution: In all WebLogic Servers in the Oracle Identity Management domain, change log levels to SEVERE. This is a two-part process.
Part 1: Manually edit the logging.xml
file, or by using the Oracle WebLogic Server Administration Console.
Edit the logging.xml
file that is in each server directory of the Oracle Identity Management Domain domain, such as OAM_Server1, OIM_Server1, and SOA, and set level='SEVERE'
for all log_handlers and loggers. The path to each logging.xml
file will resemble:
DOMAIN_HOME/config/fmwconfig/<servername>
Part 2: Edit the log levels in the Oracle WebLogic Server Administration Console:
Log in to the console (http://hostname:port/console
).
Click the Servers link.
Click the desired server.
Click the Logging tab.
Scroll down and click the Advanced link.
In the Message destination(s) section, change the log levels as shown here:
Log file : Severity level: warning Standard out :Severity level: error Domain log broadcaster :Severity level: error Memory buffer: Memory Buffer Severity level: error
Save the changes.
Repeat this for all WebLogic Servers in the Oracle Identity Management stack, such as OAM_Server1, OIM_Server1, and SOA.
Click Activate Changes.
Restart the server.
Tune Two OID Configuration Parameters
Description: Two Oracle Internet Directory configuration parameters, orclmaxcc
and orclserverprocs
, need to be appropriately tuned.
Solution: Change orclmaxcc
to 10 and tune the number of OID processes:
Name the sample script config_oid_tuning.ldif
. You will need to set cn=oid1
to your component name. In a multi-component environment, this needs to be changed accordingly. You will need to set orclserverprocs to the number of cores in the OID server that is used.
dn: cn=oid1,cn=osdldapd,cn=subconfigsubentry changetype: modify replace: orclmaxcc orclmaxcc: 10 orclserverprocs: <number of cores>
Apply the script by running this command at the command prompt:
ldapmodify -p portNum -h hostname -D cn=orcladmin -f config_oid_tuning.ldif
Enable Timing Logging
Description: Add parameters to enable timing logging for OID.
Solution:
Add this entry to the config.xml file in ./oid/user_projects/domains/oid_domain/config/
and the ./oim/user_projects/domains/oim_domain/config/
directories for each WebLogic Server in the Oracle Identity Management domain:
<web-server> <web-server-log> <file-name>logs/access.log.%yyyyMMdd%</file-name> <rotation-type>byTime</rotation-type> <number-of-files-limited>true</number-of-files-limited> <rotate-log-on-startup>true</rotate-log-on-startup> <buffer-size-kb>0</buffer-size-kb> <logging-enabled>true</logging-enabled> <elf-fields>date time time-taken bytes c-ip s-ip sc-status sc(X-ORACLE-DMS-ECID) cs-method cs-uri cs(User-Agent) cs(ECID-Context) cs(Proxy-Remote-User) cs(Proxy-Client-IP)</elf-fields> <log-file-format>extended</log-file-format> <log-time-in-gmt>false</log-time-in-gmt> <log-milli-seconds>true</log-milli-seconds> </web-server-log> </web-server>
To set the access log format, add this string to the httpd.conf
file in the /u01/ohsauth/ohsauth_inst/config/OHS/ohs1
path.
LogFormat "%h %l %u %t \"%r\" %>s %b %D %{X-ORACLE-DMS-ECID}o" common
Increase Policy Cache Timeout
Description: By default, entries in the security policy cache time out every 12 hours. When these entries time out, sporadic slowness may be experienced because they need to be repopulated. To avoid this, increase the timeout value.
Solution:
Open the DOMAIN_HOME
/config/fmwconfig/jps-config.xml
file.
Add the following entry to the <serviceInstance name="policystore.ldap" provider="policystore.provider"> section and select a timeout value (in milliseconds):
<property name="oracle.security.jps.policystore.refresh.purge.timeout" value="1296000000"/>
This example will set the timeout to 15 days. Adjust the property value as needed.
<serviceInstance name="pdp.service" provider="pdp.service.provider"> ... <property name="oracle.security.jps.policystore.refresh.purge.timeout" value="1296000000"/>
Adjust Connection Settings Between Managed Servers and LDAP Server
Description: Adjust the following LDAP settings to improve system stability and performance.
Solution:
From each Fusion Applications domain, login to the instance of WebLogic Server called the Administration Server.
Select Security realms and click myrealm.
Click Providers.
Click OIDAuthenticator.
Select the Provider Specific tab.
Check the Keep Alive Enabled option.
Set Connection Timeout to 60.
Set Parallel Connect Delay to 1.
Set Results Time Limit to 300000.
Select the Ignore Duplicate Membership option.
Adjust Connection Pool Sizes to Oracle Access Manager
Description: Increase the connection pool size to Oracle Access Manager to avoid contention when load is high.
Solution:
From each Fusion Applications domain, login to the Administration Server instance of WebLogic Server.
Select Security Realms and click myrealm.
Click Providers.
Click OAMIdentityAsserter.
Select the Provider Specific tab.
Set Minimum Access Server Connections In Pool to 10.
Set Maximum Access Server Connections In Pool to 100.
Set Connection Timeout to 60.
Set Parallel Connect Delay to 1.
Set Results Time Limit to 300000.
Select the Ignore Duplicate Membership option.
Adjust Configuration Settings for Oracle Access Manager Domain
Description: The following setting changes will ensure optimal performance for Oracle Access Manager under typical Oracle Fusion Applications workload.
Solution: Edit the settings shown in Table 1-9 in the OAM Domain Home
/config/fmwconfig/oam-config.xml
file and set the recommended values:
Table 1-9 Recommended Settings for Best Oracle Access Manager Performance
Location | Parameter | Value |
---|---|---|
|
|
|
|
|
|
|
|
|
|
|
Delete value |
|
|
Delete value |
|
|
|
If you are using the IBM AIX, Solaris SPARC, or Solaris x64 operating system, Oracle recommends that you incorporate these settings.
IBM AIX
Incorporate these settings for best performance when using IBM JVM 9:
-Xgcpolicy:gencon -Xcompressedrefs -XtlhPrefetch
Solaris SPARC
In all the domains, make these changes to the DOMAIN_HOME
/bin/fusionapps_start_params.properties
file:
Edit fusion.default.SunOS-sparc.memoryargs
and add these arguments:
-XX:ReservedCodeCacheSize=128m -XX:+UseParallelOldGC -XX:ParallelGCThreads=4
Restart all the servers of all domains.
In all the domains, make these changes to the DOMAIN_HOME
/bin/fusionapps_start_params.properties
file:
Add the fusion.AdminServer.SunOS-sparc.memoryargs
entry with these arguments:
-XX:PermSize=256m -XX:MaxPermSize=512m -XX:+UseParallelGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=@HEAP_DUMP_PATH@ -XX:+ParallelGCVerbose -XX:ReservedCodeCacheSize=128m -XX:+UseParallelOldGC -XX:ParallelGCThreads=2
Change @HEAP_DUMP_PATH@
to a valid path for the Heap Dump files to be written.
Restart the Admin servers of all the domains.
For the BIDomain, make these changes to the $BIDomain/bin/fusionapps_start_params_bi.properties
file:
Edit fusion.default.SunOS-sparc.memoryargs
and append these arguments:
-XX:ReservedCodeCacheSize=128m -XX:+UseParallelOldGC -XX:ParallelGCThreads=4
Restart all the servers of the BIDomain.
For the BIDomain, make these changes to the $BIDomain/bin/fusionapps_start_params_bi.properties
file:
Add the fusion.AdminServer.SunOS-sparc.memoryargs
entry with these arguments:
-XX:PermSize=256m -XX:MaxPermSize=512m -XX:+UseParallelGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=@HEAP_DUMP_PATH@ -XX:+ParallelGCVerbose -XX:ReservedCodeCacheSize=128m -XX:+UseParallelOldGC -XX:ParallelGCThreads=2
Change @HEAP_DUMP_PATH@
to a valid path for the Heap Dump files to be written.
Restart the Admin server of the BIDomain.
For the CommonDomain, make these changes to the $CommonDomain/bin/fusionapps_start_params.properties
file:
Edit fusion.HelpPortalCluster.SunOS-sparc.memoryargs
and add these arguments:
-XX:ReservedCodeCacheSize=128m -XX:+UseParallelOldGC -XX:ParallelGCThreads=4
Restart the HelpPortalCluster servers of the CommonDomain.
For the ProcurementDomain, make these changes to the $ProcurementDomain/bin/fusionapps_start_params.properties
file:
Edit fusion.ProcurementCluster.SunOS-sparc.memoryargs
to add the arguments:
-XX:ReservedCodeCacheSize=128m -XX:+UseParallelOldGC -XX:ParallelGCThreads=4
Restart the ProcurementCluster of the ProcurementDomain.
For the ProjectDomain, make these changes to the $ProjectDomain/bin/fusionapps_start_params.properties
file:
Edit fusion.ProjectsFinancialsCluster.SunOS-sparc.memoryargs
and add these arguments:
-XX:ReservedCodeCacheSize=128m -XX:+UseParallelOldGC -XX:ParallelGCThreads=4
Restart the ProjectFinancialCluster of the ProjectDomain.
Solaris x64
In all domains, make these changes to the DOMAIN_HOME
/bin/fusionapps_start_params.properties
file:
Edit fusion.default.SunOS-i386.memoryargs
to add these arguments:
-XX:ReservedCodeCacheSize=128m -XX:+UseParallelOldGC -XX:ParallelGCThreads=4
Restart all the servers of all the domains.
In all domains, make these changes to the DOMAIN_HOME
/bin/fusionapps_start_params.properties
file:
Add the fusion.AdminServer.SunOS-i386.memoryargs
entry with these arguments:
-XX:PermSize=256m -XX:MaxPermSize=512m -XX:+UseParallelGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=@HEAP_DUMP_PATH@ -XX:+ParallelGCVerbose -XX:ReservedCodeCacheSize=128m -XX:+UseParallelOldGC -XX:ParallelGCThreads=2
Change @HEAP_DUMP_PATH@
to a valid path for the Heap Dump files to be written.
Restart the Admin servers of all the domains.
For the BIDomain, make these changes to the $BIDomain/bin/fusionapps_start_params.properties
file:
Edit fusion.default.SunOS-i386.memoryargs
and append these arguments:
-XX:ReservedCodeCacheSize=128m -XX:+UseParallelOldGC -XX:ParallelGCThreads=4
Restart all the servers of the BIDomain.
For the BIDomain, make these changes to the $BIDomain/bin/fusionapps_start_params.properties
file:
Add the fusion.AdminServer.SunOS-i386.memoryargs
entry with these arguments:
-XX:PermSize=256m -XX:MaxPermSize=512m -XX:+UseParallelGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=@HEAP_DUMP_PATH@ -XX:+ParallelGCVerbose -XX:ReservedCodeCacheSize=128m -XX:+UseParallelOldGC -XX:ParallelGCThreads=2
Change @HEAP_DUMP_PATH@
to a valid path for the Heap Dump files to be written.
Restart the Admin server of the BIDomain.
For the CommonDomain, make these changes to the $CommonDomain/bin/fusionapps_start_params.properties
file:
Edit fusion.HelpPortalCluster.SunOS-i386.memoryargs
to add these arguments:
-XX:ReservedCodeCacheSize=128m -XX:+UseParallelOldGC -XX:ParallelGCThreads=4
Restart the HelpPortalCluster of the CommonDomain.
For the ProcurementDomain, make these changes to the ProcurementDomain/bin/fusionapps_start_params.properties
file:
Edit fusion.ProcurementCluster.SunOS-i386.memoryargs and add these arguments:
-XX:ReservedCodeCacheSize=128m -XX:+UseParallelOldGC -XX:ParallelGCThreads=4
Restart the ProcurementCluster server of the ProcurementDomain.
This section lists several Oracle HTTP Server configuration changes that may improve performance. These settings are the default if your environment is newly provisioned. If your environment in upgraded, you will have to manually apply these setting changes.
Follow these steps to tune the Oracle HTTP Server.
Avoid restarts of httpd-worker processes by increasing MaxSpareThreads and MinSpareThreads.
These restarts affect the recreation of connections and threads in Oracle HTTP Server processes during varying load patterns and could negatively affect performance. The recommendation is to increase the minimum and maximum spare threads for Oracle HTTP Server to 200 and 800 respectively.
Edit the httpd.conf
file located in instance_home/config/OHS/<ohs_name>/httpd.conf
so it resembles:
<IfModule mpm_worker_module> ... MinSpareThreads 200 MaxSpareThreads 800 ... </IfModule>
Increase the ThreadsPerChild setting from 50 to 250. Oracle HTTP Server processes maintain shared resources, such as connection pools to back-end servers, and memory for various purposes such as storing a cache of static files, and storing server information by pinging servers to obtain a dynamic server list from the cluster.
It would be efficient to have more threads created within a process so that they can effectively share the common resources. Setting more threads per process reduces the memory footprint and improves the efficiency of using the connection pool to the back-end servers. To create more threads within a process, change the details in the <instance home>/config/ohs/<ohs name>/httpd.conf
file, as shown here.
##<under worker mpm> <IfModule mpm_worker_module... StartServers 2 ThreadLimit 250 ThreadsPerChild 250 ...
Increase the Keep Alive timeout between Oracle HTTP Server and WebLogic Servers.
By default, a connection between an Oracle HTTP Server and a WebLogic Server is closed if it is idle for 20 seconds.
Since there is a cost in re-establishing these connections, it is beneficial to increase this timeout to 5 minutes. To do this, the mod_wl_ohs.conf
and config.xml
files for the target WebLogic Server domains need to be changed.
Edit the <instance home>/config/OHS/<ohs name>/mod_wl_ohs.conf
file so it resembles this example:
LoadModule weblogic_module ${ORACLE_HOME}/ohs/modules/mod_wl_ohs.so <IfModule weblogic_module> ... KeepAliveSecs 300 ... </IfModule>
Edit the WebLogic Server configuration.
Login to the WebLogic Server console of the target domain.
Select Environment > Servers.
Select the server you want to change.
Select the Protocols tab and then the HTTP subtab.
Change both the Duration and HTTPS Duration to 300.
Disable compression for .swf files.
Add this line to the <instance home>/config/OHS/<ohs name>/moduleconf/mod_deflate.conf
file.
SetEnvIfNoCase Request_URI \.(swf)$ no-gzip dont-vary
The .swf files are already compressed; there is no need for Oracle HTTP Server to compress them again.
Set the Expires header for Business Intelligence static resources. Adding the HTTP Expires header for static resources related to business intelligence improves performance by allowing the browser to locally cache those artifacts instead of making repeated requests for them.
Edit the instance home/config/OHS/
ohs name/moduleconf/FusionVirtualHost_bi.conf
file and add the entries noted in Example 1-1.
Example 1-1 Additions to FusionVirtualHost_bi.conf
#Internal virtual host for BI, replace biprove.domain.host and biprov.wls.managed.port with appropriate values <VirtualHost <OHS_INTERNAL_HOST>:<INTERNAL_PORT> > ServerName http://<INTERNAL_ENDPOINT_HOST>:<INTERNAL_ENDPOINT_PORT> RedirectMatch 301 ^/analytics$ /analytics/ # Add the following line RedirectMatch 301 ^/analytics/res$ /analytics/res/ <LocationMatch ^/analytics/> SetHandler weblogic-handler WeblogicCluster <biprov.domain.host>:<biprov.wls.managed.port> </LocationMatch> # Add the following section <LocationMatch ^/analytics/res/> SetHandler weblogic-handler WeblogicCluster <biprov.domain.host>:<biprov.wls.managed.port> SetOutputFilter DEFLATE SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary ExpiresActive on ExpiresDefault "access plus 1 weeks" Header set Cache-Control "Public" </LocationMatch> #External virtual host for BI <VirtualHost <OHS_EXTERNAL_HOST>:<EXTERNAL_PORT> > ServerName https://<EXTERNAL_ENDPOINT_HOST>:<EXTERNAL_ENDPOINT_PORT> RedirectMatch 301 ^/analytics$ /analytics/ # Add the followig line RedirectMatch 301 ^/analytics/res$ /analytics/res/ # BI EE <LocationMatch ^/analytics/> SetHandler weblogic-handler WeblogicCluster <biprov.domain.host>:<biprov.wls.managed.port> WLProxySSL ON WLProxySSLPassThrough ON RewriteEngine ON RewriteOptions inherit </LocationMatch> # Add the following section <LocationMatch ^/analytics/res/> SetHandler weblogic-handler WeblogicCluster <biprov.domain.host>:<biprov.wls.managed.port> WLProxySSL ON WLProxySSLPassThrough ON RewriteEngine ON RewriteOptions inherit SetOutputFilter DEFLATE SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary ExpiresActive on ExpiresDefault "access plus 1 weeks" Header set Cache-Control "Public" </LocationMatch>
Add the favicon.ico file to the online help system document root.
Most browsers will attempt to access the favicon.ico
(favorite icon) file from the root of a web site. The favorite icon is displayed next to the browser's address bar.
If the file does not exist, the browser will attempt to fetch it the next time the browser is started. If the file exists, it will be cached in the browser's cache.
Oracle Fusion Applications does not ship with a favicon.ico
file. If you inspect the online help system access log, you will see multiple requests for /favicon.ico
that result in the 404 File Not Found error response.
You can avoid many of these hits by placing a favicon.ico
file in the location specified by the DocumentRoot
location in the online help system configuration file. The icon file must be 16x16 pixels in Windows Icon (.ico) file format.
By default, the DocumentRoot is configured to be $
ORACLE_INSTANCE/config/
$COMPONENT_TYPE/$
COMPONENT_NAME/htdocs
. You can put a favicon.ico
file at that location. If you have multiple online help system instances, you should put the file in the DocumentRoot
location of each instance.