D Troubleshooting WebCenter Portal

This appendix contains the following troubleshooting information:

D.1 Troubleshooting Roadmap

Use this documentation roadmap to find troubleshooting information for Oracle WebCenter Portal. In Figure D-1, find the starting point that best describes your issue, then click the graphic or use the links in Table D-1 to jump to the section that you need.

Figure D-1 Troubleshooting WebCenter Portal Roadmap

Troubleshooting WebCenter Portal

Table D-1 Starting Points for Troubleshooting WebCenter Portal

What do you want to do? Link to Troubleshooting Section in the Guide

  • Troubleshoot WebCenter Portal configuration issues?

Section D.2, "Troubleshooting WebCenter Portal Configuration Issues"

  • Troubleshoot WebCenter Portal WLST issues?

Section D.3, "Troubleshooting WebCenter Portal WLST Command Issues"

  • Monitor WebCenter Portal logs and metrics?

Section 39, "Monitoring Oracle WebCenter Portal Performance"

Section 39.5, "Viewing and Configuring Log Information"

  • Troubleshoot WebCenter Portal performance issue?

Section D.4, "Troubleshooting WebCenter Portal Performance Issues"

D.2 Troubleshooting WebCenter Portal Configuration Issues

This section includes the following subsections:

D.2.1 How Do I Find Out Which WebCenter Portal Version Is Installed?

Always use Oracle's OPatch utility to obtain version information for WebCenter Portal products and components installed in your environment.

To run OPatch command with the lsinventory option:

  1. Set Opatch environment variables WC_ORACLE_HOME, MW_HOME, and PATH.

    See also, "OPatch Environment Variables" Oracle Fusion Middleware Patching Guide.

  2. Run the following OPatch command

    opatch lsinventory -details

    The output lists which components are installed and their version numbers, similar to that shown here:

    Oracle Upgrade Assistant for Webcenter             11.1.1.7.0
Oracle WebCenter Portal Suite                      11.1.1.7.0
Oracle WebCenter Portal Suite 11g                  11.1.1.7.0
Oracle WebCenter Portal: Activity Graph            11.1.1.7.0
Oracle WebCenter Portal: Analytics Collector       11.1.1.7.0
Oracle WebCenter Portal: Discussions Server        11.1.1.7.0
Oracle Webcenter Portal: Framework                 11.1.1.7.0
Oracle Webcenter Portal: Framework Core            11.1.1.7.0
Oracle WebCenter Portal: Pagelet Producers         11.1.1.7.0
Oracle WebCenter Portal: Personalization           11.1.1.7.0
Oracle Webcenter Portal: Portlet Server            11.1.1.7.0
Oracle WebCenter Portal: RCU                       11.1.1.7.0
Oracle Webcenter Portal: Spaces                    11.1.1.7.0
Oracle WebCenter Portal: Suite Components          11.1.1.7.0
Oracle WebCenter Portal: Wiki                      11.1.1.7.0
Oracle WebLogic Communications Service Client Library  11.1.1.7.0

    Note:

    Oracle WebCenter Portal Suite 11g is a child component of Oracle WebCenter Portal Suite and the versions are always the same.

See also, "About OPatch" in Oracle Fusion Middleware Patching Guide and "Lsinventory Command for Standalone OPatch" in Oracle Universal Installer and OPatch User's Guide for Windows and UNIX.

Note:

Always use OPatch to obtain the version number. Versions that display alongside WebCenter Portal product and component names in Fusion Middleware Control do not reflect the true version number of installed products. For example:

Incorrect version Incorrect version

D.2.2 WebCenter Portal Menu Does Not Display in Fusion Middleware Control

Problem

After logging into Fusion Middleware Control, you cannot find the WebCenter Portal option in the Application Deployment menu for your application.

Solution

Ensure the following:

  • Deployed application is a WebCenter Portal application, created using the WebCenter Portal – Framework Application template in JDeveloper.

    The WebCenter Portal option only displays for applications developed using the WebCenter Portal – Framework Application template in JDeveloper.

  • Deployed WebCenter Portal application is up and running.

  • Deployed WebCenter Portal application contains accurate information about the MDS repository and partition, and the MDS repository is accessible to the application. To verify this information, check the metadata-store-usages section in the adf-config.xml file. For information on MDS, see "Understanding the MDS Repository" in Oracle Fusion Middleware Administrator's Guide.

  • WebCenter Portal application is packaged with required artifacts to support configuration:

    • adf-jndi-config name space is configured in the application's adf-config.xml file. This is provisioned at design time. The following is an example (the text in bold) of the adf-jndi-config name space:

      <adf-config xmlns="http://xmlns.oracle.com/adf/config"
     xmlns:jndiC="http://xmlns.oracle.com/adf/jndi/config"
     xmlns:ns2="http://xmlns.oracle.com/mds/config"
     xmlns:ns3="http://xmlns.oracle.com/adf/mds/config">
  ...
  ... 
</adf-config>

    • Appropriate listeners exist in the web.xml file to register the MBeans. This is provisioned at design time. For example, see the text in bold in the following snippet of the web.xml file:

      <listener>
   <description>ADF Config MBeans</description>
   <display-name>ADF Config MBeans</display-name>
   <listener-class>oracle.adf.mbean.share.config.ADFConfigLifeCycleCallBack</listener-class>
</listener>
<listener>
    <description>ADF Connection MBeans</description>
    <display-name>ADF Connection MBeans</display-name>
    <listener-class>oracle.adf.mbean.share.connection.ADFConnectionLifeCycleCallBack</listener-class>
</listener>

  • ADFConfig and ADFConnection MBeans are registered for the WebCenter Portal application. You can verify whether these MBeans are registered through the System MBean Browser:

    1. In Fusion Middleware Control, open the System MBean Browser for your WebCenter Portal application. Do one of the following:

      For the Spaces application, select the menu option WebCenter Portal >System MBean Browser.

      For a Framework application, select the menu option Application Deployment > System MBean Browser.

    2. Locate ADFConnection MBeans for your application under Application Defined MBeans > oracle.adf.mbean.share.connection, as shown in Figure D-2.

    3. Similarly, locate ADFConfig MBeans for your application under Application Defined MBeans > oracle.adf.mbean.share.config, as shown in Figure D-2.

      Figure D-2 Application Defined MBeans

      Description of Figure D-2 follows
      Description of "Figure D-2 Application Defined MBeans"

    See also, Section 1.13.4, "System MBean Browser":

  • Review the latest configuration in adf-connections.xml and adf-config.xml to check the content is correct. Some typical problems include:

    • Configuration file is not compliant with its XML schema For example, there are duplicate configuration elements when the schema only allows for 1 occurrence.

    • XML namespace is missing for configuration referenced within the file.

    • XML element is not qualified with the XML namespace.

    See also, Appendix A "Exporting Configuration Files with MDS Customizations".

  • Check the application's diagnostic logs, analyze messages for the modules oracle.adf.mbean.share.connection and oracle.adf.mbean.share.config, and determine what must be done:

    • For the Spaces application, the log file is available in the DOMAIN_HOME/servers/ServerName/logs directory. The log file follows the naming convention of ServerName-diagnostic.log. See also, Section 39.5.1, "Spaces Application Logs".

    • For Framework applications, the log file is available in the DOMAIN_HOME/servers/ServerName/logs directory. The log file follows the naming convention ServerName-diagnostic.log. See also, Section 39.5.2, "Framework Application Logs".

D.2.3 Configuration Options Unavailable

Problem

When you try to configure a WebCenter Portal application through Fusion Middleware Control, the following message displays:

Configuration options currently unavailable. The application application_name might be down, did not start-up properly, or is incorrectly packaged.
Check the log files for further details.

For example, you try to change options available through the Application Settings screen or configure connections through the WebCenter Portal Service Configuration screen in Fusion Middleware Control.

Solution

Check the application's diagnostic logs:

  • For the Spaces application, the log file is available in the DOMAIN_HOME/servers/ServerName/logs directory. The log file follows the naming convention of ServerName-diagnostic.log. See also, Section 39.5.1, "Spaces Application Logs".

  • For Framework applications, the log file is available in the DOMAIN_HOME/servers/ServerName/logs directory. The log file follows the naming convention ServerName-diagnostic.log. See also, Section 39.5.2, "Framework Application Logs".

Analyze messages for the modules oracle.adf.mbean.share.connection and oracle.adf.mbean.share.config, and determine what must be done.

See also, Section D.2.2, "WebCenter Portal Menu Does Not Display in Fusion Middleware Control."

D.2.4 Configuration Issues with One or More WebCenter Portal Services

Do not attempt to configure services that your WebCenter Portal application does not support. WebCenter Portal configuration through Enterprise Manager and WLST fails if you try to configure a service, say discussions, that your application was not designed to use.

The Spaces application is designed to support all WebCenter Portal's services but applications that you build using WebCenter Portal: Framework only provide artifacts in the application's configuration for services that the developer specifically included during design time. For example, if the developer did not add or configure discussions for the application through JDeveloper, you cannot configure discussions postdeployment through Enterprise Manager and WLST.

If you are having issues configuring or connecting to one or more WebCenter Portal service, refer to the appropriate troubleshooting section:

D.2.5 Configuration for One Application Reflects in Another

Problem

You configured a WebCenter Portal application, but those configurations also show in another application.

For example, you created or edited a mail connection for an application named MyPortalApp1 and you discover that the connection changes are also seen in another application MyPortalApp2

Solution

This happens when multiple applications share the MDS partition in the same schema. To resolve this problem, deploy these applications again and ensure that each application either uses a different MDS schema or a different MDS partition. For information about creating a MDS repository or configuring an existing WebCenter Portal application to use a different MDS repository or partition, see section "Managing the Oracle Metadata Repository" in Oracle Fusion Middleware Administrator's Guide.

D.2.6 WebCenter Portal Logs Indicate Too Many Open Files

Problem

The Spaces application or your own portal application is inaccessible or displaying error messages and the diagnostic log files indicates that there is an issue with 'too many open files'.

Solution

Do the following:

  • Check the number of file handles configured on each of the back-end servers, primarily the database, and increase appropriately.

  • If the problem persists after increasing the file handles, check the value of fs.file-max in the /etc/sysctl.conf file and increase the value appropriately.

D.3 Troubleshooting WebCenter Portal WLST Command Issues

This section includes the following subsections:

See also, Section 1.13.3.1, "Running Oracle WebLogic Scripting Tool (WLST) Commands".

D.3.1 None of the WebCenter Portal WLST Commands Work

Problem

You are unable to run any WLST commands.

Solution

Ensure the following:

  • Always run WebCenter Portal WLST commands from your WebCenter Portal Oracle home directory (WC_ORACLE_HOME/common/bin).

    If you attempt to run WebCenter Portal WLST commands from the wrong directory you will see a NameError.

  • No files other than Python are stored in the WLST source directory: WC_ORACLE_HOME/common/bin/wlst. This directory must contain files with the .py extension only.

    The default set of files in this location contain legal Python files from Oracle. It is possible that a user copied some non-python script to this directory, for example, a backup file or a test python file with syntax errors.

  • webcenter-wlst.jar is located at WC_ORACLE_HOME/common/bin/wlst/lib.

See also, Section 1.13.3.1, "Running Oracle WebLogic Scripting Tool (WLST) Commands".

D.3.2 WLST Commands Do Not Work for a Particular WebCenter Portal Service

Problem

You are unable to run WLST commands for a particular service, and therefore, you cannot configure that service.

Solution

First, run generic non-WebCenter Portal commands, for example, listApplications() or displayMetricTableNames()to verify whether these commands work. If generic commands do not work, then apply the solution described in Section D.3.1, "None of the WebCenter Portal WLST Commands Work."

If generic commands work, then run test commands to check WebCenter Portal-specific commands for syntax errors. Run the appropriate WSLT check command (see Table D-2).

Table D-2 File Names and WLST Commands for WebCenter Portal Service

Service Name File Name WLST Command

Activity Graph

ActivityGraph.py

metadataAdminCheck()

Activity Stream

ActivityStream.py

asCheck()

Analytics

Analytics.py

OpenUsage.py

analyticsCheck()

openusageCheck()

Discussions and Announcements

Forum.py
JiveAdmin.py

fcpCheck()

Documents

Doclib.py

doclibCheck()

External Applications

ExtApp.py

extCheck()

Space Events

Community.py

ceCheck()

Instant Messaging and Presence

Imp.py

rtcCheck()

Mail

Mail.py

mailCheck()

Notifications

Notification.py

notificationCheck()

Personal Events

Personal.py

peCheck()

Producers

    

PDK-Java Producers

Pdk.py

pdkCheck()

WSRP Producers

Wsrp.py

wsrpCheck()

Pagelet Producers

Ensemble.py

ensembleCheck()

Producer Helper

Producer.py

producerHelperCheck()

RSS News Feed

RSS.py

rssCheck()

Search

Ses.py

sesCheck()

Worklist

Bpel.py

bpelCheck()

Export/Import - WebCenter Portal applications

Lifecycle.py

lifecycleCheck()

Export/Import - Spaces and Template

ExtImp.py

expimpCheck()

Synchronize Users

SynchronizeUser.py

userRenameCheck()

Rename Users

UserRename.py

userRenameCheck()

WebCenter Portal - General

    

Service Framework

WcServiceFwk.py

serviceFwkCheck()

General Settings

WebCenterGeneralSettings.py

generalSettingsCheck()

Spaces and SOA

WebCenterSpacesSOA.py

spaceCheck()

See also, Section 1.13.3.1, "Running Oracle WebLogic Scripting Tool (WLST) Commands".

For more information about WebCenter Portal's WLST commands, see Oracle Fusion Middleware WebLogic Scripting Tool Command Reference.

D.3.3 Connection Name Specified Already Exists

Problem

You are unable to create a connection with the name Connection_Name. The following message displays:

A connection with name Connection_Name already exists.

For example, you try to create an external application connection using the WLST command createExtAppConnection or connect to a mail server using createMailConnection.

Solution

Connection names must be unique (across all connection types) within the WebCenter Portal application. This error occurs when you try to create a connection with a name that is in use. Ensure that you use a unique name for your connection.

D.3.4 WLST Shell is Not Connected to the WebLogic Server

Problem

You must connect to the Administration Server for WebCenter Portal before running WLST commands. WebCenter Portal WLST commands do not work without a connection.

Solution

Run the following command to connect the WLST shell to the managed server:

connect(username, password , serverhost:serverport)

See also, Section D.3.1, "None of the WebCenter Portal WLST Commands Work" and Section 1.13.3.1, "Running Oracle WebLogic Scripting Tool (WLST) Commands."

D.3.5 More Than One Application with the Same Name Exists in the Domain

Problem

You attempt to perform an operation on a WebCenter Portal application, such as create a connection to a WebCenter Portal service or register a portlet producer, and the following message displays:

Another application named "YourApplicationName" exists. Specify the Server on which your application is deployed. Use: server="YourServerName".

This message displays if there are multiple applications with the same name in the domain. This usually happens in a cluster environment, where the same application is deployed to multiple managed servers.

For example, you tried to register a portlet producer for an application named "MyApp" using the following WLST command:

registerWSRPProducer(appName='myApp', name='MyWSRPSamples', url='http://myhost.com:9999/ portletapp/portlets/wsrp2?WSDL')

Solution

Specify on which managed server you want to run the WLST command, that is, include the server argument. For example:

registerWSRPProducer(appName='myApp', name='MyWSRPSamples', url='http://myhost.com:9999/portletapp/portlets/wsrp2?WSDL', server=WC_CustomPortal2)

See also, Section 1.13.3.1, "Running Oracle WebLogic Scripting Tool (WLST) Commands."

D.3.6 More Than One Application with the Same Name Exists on a Managed Server

Problem

You attempt to perform an operation on a WebCenter Portal application, such as create a connection to a WebCenter Portal service or register a portlet producer, and the following message displays:

Another application named "application_name" exists on the server managedServerName.

This message indicates that there are multiple applications with the same name on specified managed server. This usually happens when applications are assigned different versions.

For example, you tried to register a portlet producer for an application named "MyApp" using the following WLST command:

registerWSRPProducer(appName='myApp', name='MyWSRPSamples', url='http://myhost.com:9999/portletapp/portlets/wsrp2?WSDL')

Solution

Specify on which application version you want to run the WLST command, that is, include the server and applicationVersion arguments. For example:

registerWSRPProducer(appName='myApp', name='MyWSRPSamples', url='http://myhost.com:9999/portletapp/portlets/wsrp2?WSDL', server=WC_CustomPortal1, applicationVersion=2)

See also, Section 1.13.3.1, "Running Oracle WebLogic Scripting Tool (WLST) Commands."

D.3.7 Already in Domain Runtime Tree Message Displays

Problem

While running a WLST command, the following message displays:

Already in Domain Runtime Tree

Solution

None required. This is for information only.

D.4 Troubleshooting WebCenter Portal Performance Issues

Use the information in this section to help diagnose performance-related issues for WebCenter Portal.

This section contains the following sub sections:

D.4.1 About Performance Monitoring and Troubleshooting Tools

Various tools are available for monitoring and troubleshooting performance issues with your WebCenter Portal environment.

Table D-3 Performance Monitoring and Troubleshooting Tools

Tool Use to... See

Enterprise Manager

    

Fusion Middleware Control

Monitor WebCenter Portal metrics and log files in real-time mode for a single Oracle Fusion Middleware Farm.

Check service configuration, including MDS and partitions for WebCenter Portal deployments.

Starting Enterprise Manager Fusion Middleware Control

Grid Control

Monitor WebCenter Portal metrics in real time and from a historical perspective for trend analysis, as well as monitor the underlying host and operating system, databases, and more.

Oracle Enterprise Manager 11g Grid Control must be installed separately as it is not a part of the Oracle Fusion Middleware 11g installation. With Grid Control, you can centrally manage multiple Oracle Fusion Middleware Farms and WebLogic Domains.

Oracle Enterprise Manager 11g Grid Control

JConsole

Graphically monitor Java applications and Java virtual machines (JVM).

How to Use JConsole to Monitor JVM

JRockit Mission Control

Capture and present live data about memory, CPU usage, and other runtime metrics.

Troubleshooting Slow Requests Using JFR Recordings

Eclipse Memory Analyzer

Find memory leaks and reduce memory consumption.

Troubleshooting Memory Leaks and Heap Usage Problems

Threadlogic

Analyze thread dumps.

Generating Thread Dumps to Diagnose Extremely Slow Page Performance, High Thread Counts, and System Hangs

D.4.2 How to Troubleshoot Overall System Slowness

Use the actions listed inTable D-4, "Troubleshooting Overall System Slowness" to troubleshoot overall system slowness:

Table D-4 Troubleshooting Overall System Slowness

Action Description More Information

1

Verify key system resources.

Verifying System Resources (CPU and Memory)

2

Use top or vmstat to see if system slowest is caused by CPU, memory, or IO contention issues.

Monitoring System Resource Usage

3

Monitor the performance of your Java virtual machine (JVM).

Monitoring Java Virtual Machine (JVM) Usage

4

Verify OID and database connection pool settings.

Verifying Connection Pool Settings

5

Generate automatic workload repository (AWR) reports for Oracle databases to diagnose database-related issues.

Generating Automatic Workload Repository (AWR) Reports for the Database

6

Use tcpdump to investigate and diagnose network related problems.

Diagnosing Network Related Problems Using tcpdump

7

Use ping to measure network latency.

Measuring Network Latency Using ping

8

Collect thread dumps.

Generating Thread Dumps to Diagnose Extremely Slow Page Performance, High Thread Counts, and System Hangs

9

Look for errors in WC_Spaces-diagnostic.log and check the correct logging level is enabled.

Analyzing the Diagnostics Log

10

Use DMS Spy to monitor internal DMS metric data, such as activity in the Java Object Cache.

Using DMS Spy to Monitor Internal Performance Metric Tables

11

Look at the access.log for the WebLogic Server to check HTTP request/response cache settings.

Verify HTTP compression settings.

Verifying HTTP Request Caching

Verifying HTTP Compression

12

Use HTTP monitoring tools to analyze requests and response times.

Checking Browser Response Times

13

Warm up your system before taking performance measurements.

Warm up the System Before Re-Testing Performance

D.4.2.1 Verifying System Resources (CPU and Memory)

If you are experiencing performance issues its important to verify that you have sufficient system hardware resources, that is, adequate CPU and physical memory capacity for your WebCenter Portal installation.

Low system resources can cause many different problems. System resources are used by individual services. Regularly monitoring and recording system usage can help you determine whether you need to upgrade your system hardware, or if some services need to be moved to another machine.

To verify system CPU and memory:

  • On Linux, review CPU and memory information in the following files:

    • /proc/cpuinfo

      The cpuinfo file provides important CPU information including the model, CPU cores, CPU MHz, cache size, and flags which show what instruction sets are available on the processor. Systems with multiple processors or multiple cores have separate entries for each.

    • /proc/meminfo

      The important fields to look for in the meminfo file include MemTotal, MemFree, Cached, and SwapTotal.

  • On Windows, access CPU and memory information through the Task Manager (Performance > Resource Monitor).

D.4.2.2 Monitoring System Resource Usage

On Linux, you can use top and vmstat utilities to see if system slowness is caused by CPU, memory, or I/O contention issues. If you discover that your system resources are low, you can:

  • Move processes to other machines or shut down unused processes/programs to free up more physical memory and/or CPU cycles.

  • Upgrade your system hardware resources

For more information, see:

Note:

On Windows, use Task Manager to monitor system resources and shut down unused processes and programs. Refer to your Windows documentation for more information.

D.4.2.2.1 How to use top to monitor system resource usage on Linux

The top utility displays a continually updating report of system resource usage so you can identify the top memory and CPU consumers on your system.

The top portion of the report lists information such as the system time, uptime, CPU usage, physical and swap memory usage, and number of processes. Below that is a list of the processes sorted by CPU utilization.

Note:

Use Shift+M to sort by memory usage. Use Shift+P to sort by CPU usage.

# top
2:10:49  up 8 day,  3:47,  20 users,  load average: 0.34, 0.19, 0.10
75 processes: 20 sleeping, 2 running, 8 zombie, 0 stopped
CPU states:   5.1% user   1.1% system   0.0% nice   0.0% iowait  93.6% idle
Mem:   512216k av,  506176k used,    6540k free,    0k shrd,   21888k buff
Swap: 1044216k av,  161672k used,  882544k free     199388k cached

PID   USER  PRI NI  SIZE  RSS   SHARE STAT %CPU %MEM   TIME    CPU COMMAND
2330  admin  15  0  161M  70M   2132  S    4.9  14.0   1000m   0    oracle
2605  lin    15  0  8240  6340  3804  S    0.3  1.2    1:12    0    oracle
3413 harvey  15  0  6668  5324  3216  R    0.3  1.0    0:20    0    oracle

Troubleshooting Tips - top:

  • If free memory is < 100Mb and cached memory is < 1GB, system memory is running low.

  • If %wa (I/O WAIT) is always more than 10%, the system may be slow because it is blocked by physical I/O.

  • Ensure that the idle value is close to 100% and system/user CPU usage is close to 0% when there is no load on the system.

  • In the memory view of top, identify the % memory usage (%MEM) for each process (PID). If the memory is tight and swap space usage is high, consider:

    - moving process with high memory usage to another machine

    - increasing memory allocation to the virtual machine

    - adding physical memory.

D.4.2.2.2 How to use vmstat to monitor system resource usage on Linux

The vmstat utility shows running statistics on various parts of the system including system processes, memory, swap, I/O, CPU, and the Virtual Memory Manager. These statistics are generated using data from the last time the command was run to the present. The first time you run the command, data displays from the last reboot until the present.

When you run vmstat you can specify how often you want the statistics to refresh (in seconds) using the format: vmstat <refresh rate in seconds>

For example: vmstat 3

Frequent system resource usage updates, enable you to see trends in CPU/memory usage and how usage trend impact WebCenter Portal application performance. If the system is stable, the swap metrics (si and so) should register near zero.

# vmstat 3
   procs           memory              swap        io       system      cpu
r  b    swpd  free   buff    cache   si   so    bi    bo   in    cs us sy id wa
3  0    144 1058184 868324 12343056    0    0   220    83   33    14 10  1 88  1
0  0    144 1058184 868324 12343056    0    0     0   117  478   656  5  1 94  0
0  0    144 1058192 868324 12343056    0    0     0    69  696   860  9  1 90  0
2  0    144 1058264 868324 12343056    0    0     0     0  914  1127  5  2 93  0
0  0    144 1057864 868324 12343316    0    0     0   184  857  1021  7  1 92  0
2  0    144 1057592 868324 12343316    0    0     0   155 1596  1646  5  4 91  0
0  0    144 1057592 868324 12343316    0    0     0     0  737   839  5  2 94  0
0  0    144 1057592 868324 12343316    0    0     0    57  694   743  8  2 91  0
2  0    144 1057592 868324 12343316    0    0     0     0  358   437  2  0 98  0

Troubleshooting Tips - vmstat:

  • When the swap metrics (si and so) are frequently non-zero, system memory becomes tight, and the system may go into thrashing mode. Low system memory significantly affects performance, that is, any program running on the system becomes unusually slow. If you frequently experience low system memory, increase the memory on the machine.

  • If CPU idle time (id) is constantly less than 10%, the CPU is running at full or over capacity. Under such conditions, any more load on the system results in significant performance degradation.

D.4.2.3 Monitoring Java Virtual Machine (JVM) Usage

Your Java virtual machine (JVM) can significantly affect WebCenter Portal performance. Oracle recommends that you continually monitor your JVM to track:

  1. Memory usage

  2. CPU usage

  3. Thread activity

You can monitor all three metrics from WebCenter Portal's Recent WebLogic Server Metrics page in Fusion Middleware Control (Figure D-3). For details, see Section 39.1.8, "Understanding WebLogic Server Metrics" and Section 39.2, "Viewing Performance Information Using Fusion Middleware Control".

Figure D-3 Recent WebLogic Server Metrics Page - JVM Metrics

Recent WebLogic Server Metrics Page - JVM

  • Monitor memory trends - JVM frequently allocates and releases memory. To detect memory leaks and high memory usage, you must analyze the memory trend over a long period of time (an hour or several hours).

    When you see the bottom trend line on a memory graph increasing, it indicates a memory leak. The bottom points on a memory graph show how low memory usage can go after full garbage collection. If you take at least two memory dumps, one when the memory is healthy and another when the memory full garbage collection cannot recycle too much, you can compare the memory dumps and identify which components are consuming memory.

    See also, Section D.4.4.4, "Troubleshooting Memory Leaks and Heap Usage Problems".

  • Monitor CPU usage - Occasional spikes in CPU usage is normal but if CPU usage remains high (85-90%) over a long period of time, it normally indicates there is an issue with CPU which can impact performance.

    If CPU usage appears overloaded, comparing several thread dumps at fixed intervals (for example, every 5 seconds) can help reveal causes of high CPU usage.

    Alternatively, profiling tools, such as JRockit's Flight Recorder, can provide further insight into CPU usage.

  • Monitor thread activity and analyze thread dumps (stuck threads, blocked threads, deadlocks) - The number of threads should stabilize under stable load conditions. Sudden spikes or an increasing number of threads, generally indicates an issue in your system.

    When that is happening, you can take multiple thread dumps to understand the calling stack of stuck/blocked threads or deadlocks. You can use thread analysis tools to identify what is causing the extra threads and what are they doing, as well as detect contentious areas and slow performing areas. See also, Section D.4.4.2, "Troubleshooting Stuck Threads".

    See also, Section D.4.2.8, "Generating Thread Dumps to Diagnose Extremely Slow Page Performance, High Thread Counts, and System Hangs".

    Tip:

    Oracle Fusion Middleware's Diagnostic Framework collects and manages information about common problems, such as stuck threads and deadlocked threads, to help you diagnose and resolve such issues. Alternatively, you can send diagnostic dumps to Oracle Support Services. For more details, see "Diagnosing Problems" in Oracle Fusion Middleware Administrator's Guide.

There are various tools available to monitor JVM performance, including Fusion Middleware Control, JConsole, JVisualVM, and JRockit Mission Control. Use any of these tools to show the current state of the JVM, as well as all the active threads and their states. See also, "Monitoring and Profiling the JVM" in Oracle Fusion Middleware Performance and Tuning Guide.

D.4.2.3.1 How to Use JConsole to Monitor JVM

In the first instance, administrators can use WebCenter Portal's Recent WebLogic Server Metrics page in Fusion Middleware Control to monitor JVM (as shown in Figure D-3). If more aggressive real time metrics are required, another option is to use JConsole (shown in Figure D-4).

JConsole is available with all types of JVM, including HotSpot, JRockit, IBM JVM, and so on.

JConsole's executable is located at: JAVA_HOME/bin/jconsole.exe.

Figure D-4 JConsole

JConsole

D.4.2.4 Verifying Connection Pool Settings

This section describes how to verify connection pool settings for:

The information in this section might be useful if you are experiencing slow response times and your diagnostics log files contain connection/connection pool messages or a recent thread dump contains calling stacks that are waiting to get connections from connection pool.

D.4.2.4.1 WebCenter Portal Data Sources (JDBC Connection Pool Settings)

WebCenter Portal administrators can use Fusion Middleware Control to monitor JDBC connection metrics. Use the JDBC usage information on the WebLogic Server Metrics Page (Figure D-5) to assess whether JDBC configuration or the connection pool size needs to be adjusted.

See also, Section 39.2, "Viewing Performance Information Using Fusion Middleware Control".

Figure D-5 Recent WebLogic Server Metrics Page - JDBC Usage

Recent WebLogic Server Metrics Page - JDBC

The Recent JDBC Usage chart shows the number of JDBC connections currently open on the managed server. The JDBC session count that is displayed here, is the sum of the Current Active Connection Count metric for each JDBC data source.

If usage is high and the trend is rising, WebCenter Portal administrators can use WebLogic Server Administration Console to view and configure data source connection pool settings and monitor usage patterns for individual data sources in more detail. If you monitor data source usage over a period of time you can:

  • Determine the best connection pool size for a particular database connection.

  • Detect and resolve connection pool leakage, that is, if you notice that the number of data source connections do not decrease without load.

Tip:

Select "Customize the Table" when monitoring connection pools, to display additional metrics such as:

  • Connection Delay Time

  • Current Capacity High Count

  • Failures To Reconnect Count

  • Wait Seconds High Count

  • Waiting For Connection Current Count

  • Waiting For Connection Failure Total

For high concurrency systems, you may want to adjust the maximum number of connections in the pool (Maximum Capacity setting). Out-of-the-box maximum values for the various WebCenter Portal data sources are shown here:

WebCenter Portal
Data Sources		 Connection Pool
Default Maximum Capacity

WebCenterDS

50

ActivitiesDS

25

mds-SpacesDS

50

mds-owsmDS

15

mds-PageletProducerDS

50

mds-ServicesProducerDS

50

mds-wcpsDS

50

PersonalizationDS

25

Portlet-ServicesProducerDS

50

PortletDS

50

WC-ServicesProducerDS

25

Note:

Each connection uses memory in the WebLogic Server and consumes processes in the database so do not specify an unnecessarily large connection pool value.

  • To monitor a particular data source, log in to the WebLogic Server Administration Console, select Services>Data Sources, click the data source name, and then click the Monitoring tab.

  • To modify the connection pool for a particular data source, log in to the WebLogic Server Administration Console, select Services>Data Sources, click the name of the data source, and then click the Connection Pool tab.

See also, "Tuning Data Source Connection Pool Options" in Oracle Fusion Middleware Configuring and Managing JDBC Data Sources for Oracle WebLogic Server.

D.4.2.4.2 Identity Store (JNDI Connection Pool Settings)

Typically, JNDI connection pooling is always turned on. However, some additional configuration is required if the connection between WebCenter Portal and the identity store (this may be Oracle Internet Directory, Active Directory, and so on) uses SSL. By default, when you choose an SSL port, the JNDI connections are not pooled causing increased response time and decreased performance when logging in, looking up users, groups, or other identity store entities. For more information and instructions, see, "Tuning Identity Store Configuration" in Oracle Fusion Middleware Performance and Tuning Guide.

D.4.2.5 Generating Automatic Workload Repository (AWR) Reports for the Database

If database access is slow, for example, activity stream queries, MDS queries or JPA (Java Persistence API) queries are slow for various operations in your WebCenter Portal application, you can analyze Automatic Workload Repository (AWR) reports to diagnose the root cause of performance-related problems in your Oracle database.

Before generating the AWR report, first check the general health (CPU/Memory) of that machine hosting your database (see Section D.4.2.2, "Monitoring System Resource Usage"). If system resource limitations are not causing poor performance, examine the database performance information and statistics in the AWR report.

For more detail, see "Tuning Database Performance" in Oracle Fusion Middleware Performance and Tuning Guide, and "Generating Automatic Workload Repository Reports" in Oracle Fusion Middleware Performance and Tuning Guide.

D.4.2.6 Diagnosing Network Related Problems Using tcpdump

Use tcpdump, a network utility that listeners and captures network traffic to investigate and diagnose network related problems.

For example, if WebCenter Portal page performance metrics in Enterprise Manager indicate that server performance is operating normally while end users are reporting unstable/slow performance, you could have a problem with your network. You can use tcpdump or a similar network monitoring tool to trace network your traffic to see if any environmental issue (on the network) is causing an unexpected large latency.

Refer to tcpdump documentation for information on how to run this utility and analyze the network data.

D.4.2.7 Measuring Network Latency Using ping

Use ping to measure network latency.

WebCenter Portal installations depend on various backend components and services, such as Oracle HTTP Server, Oracle WebCenter Content Server, LDAP server, instant messaging and presence servers, mail servers, portlets servers, databases, and more. If all your required components are not on the same machine, Oracle recommends they are closely located to minimize network latency and, if possible, are on the same subnet.

Use ping from one machine to another to verify that network latency is less than 1ms.

D.4.2.8 Generating Thread Dumps to Diagnose Extremely Slow Page Performance, High Thread Counts, and System Hangs

Oracle recommends that you generate thread dumps to a dedicated thread dump file and then use a thread analysis tool such as ThreadLogic or a simple text editing tool, to understand and correlate the thread execution logic and progress.

To generate thread dumps for Sun JVM (HotSpot) to a file:

>jstack <pid> > threaddump.txt

To generate thread dumps for Oracle JRockit JVM to a file:

>jrcmd print_threads <pid> > threaddump.txt

Sometimes its useful to generate a series of thread dumps at fixed sampling intervals to confirm problems such as extremely slow method calls, stuck threads, deadlocks, and so on. For example, you could create a simple script to generate dumps every second or if you are diagnosing a slow request you can choose a suitable duration to cover the length of the slow request.

This example generates thread dumps every second:

#!/bin/bash
for i in {1..20}
do
 <java_home>/bin/jstack <pid> > thread_dump_$i.txt
 sleep 1
done

For more detail about the capabilities of ThreadLogic or any other thread analysis tool, refer the appropriate manufacturer's documentation.

Note:

When WebLogic servers detect a deadlock or stuck thread, a related thread dump is automatically generated in the server's output log file, located at:

DOMAIN_NAME\servers\SERVER_NAME\logs\SERVER_NAME.out

For example, the output log for WebCenter Portal's Spaces application is available at DOMAIM_NAME\servers\WC_Spaces\logs\WC_Spaces.out.

The output file is a simple text file that contains various server messages, including thread information.

D.4.2.9 Analyzing the Diagnostics Log

Look for errors, incidents, and warnings in the diagnostics log for the managed server that is hosting your WebCenter Portal application, that is, <managed_server_name>-diagnostic.log.

When your WebCenter Portal application is running with some error condition, it can have a big impact on performance. For example, if a connection to WebCenter Content Server becomes intermittent or not accessible, pages with content related components respond very slowing while attempting to connect and eventually may time out.

ERROR, INCIDENT, and WARNING messages due to timeouts, unavailable services, cache errors, and so on, are logged to a diagnostic log file which you can view from Fusion Middleware Control. For more information, see Section 39.5, "Viewing and Configuring Log Information".

Note:

When measuring performance, ensure that DEBUG/TRACE messages, that is, levels lower than CONFIG (700) are not being logged. When FINE, FINER, or FINEST messages display, the system is running in debugging mode and this means that most requests are significantly slower than normal. If you configure lower level logging temporarily to debug a problem, ensure that you change the log level before taking performance measurements.

D.4.2.10 Using DMS Spy to Monitor Internal Performance Metric Tables

The DMS Spy servlet provides access to internal DMS metric data from a Web browser. This servlet is useful if you want to monitor the system for long period of time and it can help you understand internal system behavior and performance.

You would not normally use DMS Spy to monitor WebCenter Portal-specific metrics since this information is more easily available through Fusion Middleware Control. However, if you interested in other underlying metrics, such as warning/error messages about the ADF application module (AM) pool display, you can investigate AM pool metrics here. Similarly, if messages relating to the Java Object Cache (JOC) display, you can turn on the JOC's DMS monitoring feature to observe activities in JOC.

To enable JOC DMS monitoring, edit the javacache.xml file (normally it is in <webcenter_domain>/config/fmwconfig/servers/WC_Spaces/javacache.xml)

  1. Open the javacache.xml file.

    Typically, the file is located at: <webcenter_domain>/config/fmwconfig/servers/<server_name>/javacache.xml

    For example, for the Spaces application at: <webcenter_domain>/config/fmwconfig/servers/WC_Spaces/javacache.xml

  2. Change the entry:

    <dms enabled="false"/>

    To: <dms enabled="true"/>

  3. Restart the managed server on which the WebCenter Portal application is deployed.

    For example, for the Spaces application, restart WC_Spaces.

Once enabled, you see two new entries in the left panel: java_cache_region and joc

java_cache_region shows how much cache memory is allocated to each area and the values displayed can help you determine cache size settings. For example, by default the MDS caching size is set to 100MB in adf-config.xml:

<max-size-kb>100000</max-size-kb>

Both Region Name: ADFApplication<N> and ADFApplication<N>/main_region are used by the MDS cache. If the sum of ADFApplication<N> and ADFApplication<N>/main_region memory is close to 100MB, and your JVM has enough unused heap memory, you could increase the MDS caching size (<max-size-kb>) so the MDS cache is not so full, and this will improve the efficiency of the MDS cache.

For more information, see "Viewing Performance Metrics Using the Spy Servlet" in Oracle Fusion Middleware Performance and Tuning Guide.

D.4.2.11 Verifying HTTP Request Caching

Most web pages include resources that do not change very often, such as images, CSS files, JavaScript files, and so on. As such resources take time to download over the network, the time taken to load a web page increases. HTTP caching allows such resources to be saved or cached, by a browser or proxy. Once a resource is cached, a browser or proxy can refer to the locally cached copy instead rather than downloading it again on subsequent visits to the web page. Caching reduces round-trip time by eliminating HTTP requests for the required resources. This not only helps significant reducing page load time for subsequent user visits, but also reduces the bandwidth and hosting costs for your site.

By default, static resources serviced by WebCenter Portal use the ADFCachingFilter to include the required response headers which allow browsers to cache static content. If your site is experiencing performance issues, you must confirm that your browser is caching static resources. If caching is not correctly working, static WebCenter Portal resources are repeatedly fetched from the server rather than being cached in the browser, and this can impact performance. A possible reason for caching issues with static portal resources could be dues to some loss introduced by a proxy or network component sitting inbetween WebCenter Portal and the browser.

Note:

For more information about ADFCachingFilter, refer to "ADF Faces Caching Filter" in Oracle Fusion Middleware Web User Interface Developer's Guide for Oracle Application Development Framework.

Use HTTP request monitoring tools, such as Firebug (Firefox) or httpWatch (Internet Explorer and Firefox), to monitor HTTP traffic between the browser and the server. With these tools, you can see if there is a problem with the cache, that is, whether static resources are fetched for each request, not cached for long enough, or not cached at all for static resources (such as JavaScript and CSS files). When using these monitoring tools, confirm that there are browser-cache related tags in the response, for example, Cache-Control header with a suitable max-age value or an Expires tag with a suitable cache expiry time.

You can also analyze HTTP caching by examining the access.log for the managed server on which your WebCenter Portal application is deployed. For example, look at the log to see whether a particular user/IP is repeatedly making requests for the same resource. If the log contains repeated requests, the cache header on requests might be wrong and you must investigate caching issue further. Start by accessing the static resource to eliminate various tiers. For example, use wget or curl to fetch the static resource directly using the WebLogic server port by issuing the same on the local machine. Next, access the same resource through any other entity front-ending the WebLogic server that is hosting the WebCenter Portal application, for example Oracle HTTP Serve or Apache. If needed, enable packet tracing to find the response from the WebCenter Portal tier to see whether issues are occurring on the WebCenter Portal side, or the problem is on the network.

Note:

The access log for the Spaces application is located at: ORACLE_HOME/user_projects/domains/wc_domain/servers/WC_Spaces/logs/access.log

While this section describes static resources serviced by WebCenter Portal, you must also consider similar issues for static resources in your environment. For example, if you have a rich UI where common pages reference static resources that you own, you must review how such content is cached. Consider using Apache Header directives to drive the caching of static resources, either based on file type (such as images, CSS), or based on URL path patterns. For example, the following configuration in Apache sends back a response header to the browser to cache all content under the URL path "/my_images/" for 30 days (2592000 is the number of seconds in 30 days):

<Location /my_images/>
  Header set Cache-Control "max-age=2592000, public"
</Location>

D.4.2.12 Verifying HTTP Compression

In addition to HTTP caching, it is important that you review HTTP compression characteristics for responses. HTTP Compression is a publicly defined way to compress content (mostly textual) that is transferred from web servers to browsers. Compression reduces the number of bytes that are transmitted which improves performance. HTTP Compression uses public domain compression algorithms to encode HTML, XML, JavaScript, CSS and other file formats at the server-side. This standards-based method for delivering compressed content is built into HTTP/1.1, and all modern web browsers support the HTTP/1.1 protocol, that is. they can decode compressed files automatically at the client-side. As a result, no additional software or user interaction is required on the client-side.

By default, static resources serviced by WebCenter Portal use the ADFCachingFilter () to compress static content like CSS and JavaScript. If your site is experiencing performance issues, you must confirm that your browser is seeing compressed responses. You can use one of the HTTP request monitoring tool described in Section D.4.2.11, "Verifying HTTP Request Caching" to scan responses for a Content-Encoding response header with the value gzip. A Content-Encoding response header is only sent if the client (that is, the browser) supports compressed content.

Note:

For more information about ADFCachingFilter (), refer to "ADF Faces Caching Filter" in Oracle Fusion Middleware Web User Interface Developer's Guide for Oracle Application Development Framework.

If the browser sends the request header Accept-Encoding: gzip, then you know the browser can process compressed content. If you do not see a compressed response, make sure that the client sends an Accept-Encoding header stating it can handle compressed content.

File format that typically require compression include: HTML, CSS, XML, and JavaScript. As most images, music, and videos are already compressed, you do not need to configure these file formats for compression.

Consider using Apache's mod_deflate or mod_gzip to drive the compression of resources, either based on MIME type (such as, text/css) or based on file extension (such as *.html). For example, the following configuration in Apache compresses resources listed in each AddOutputFilterByType:

<Location />
    SetOutputFilter DEFLATE
    AddOutputFilterByType DEFLATE text/plain
    AddOutputFilterByType DEFLATE text/xml
    AddOutputFilterByType DEFLATE text/html
    AddOutputFilterByType DEFLATE application/xhtml+xml
    AddOutputFilterByType DEFLATE text/css
    AddOutputFilterByType DEFLATE application/xml
    AddOutputFilterByType DEFLATE image/svg+xml
    AddOutputFilterByType DEFLATE application/rss+xml
    AddOutputFilterByType DEFLATE application/atom_xml
    AddOutputFilterByType DEFLATE application/x-javascript
    SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary
    SetEnvIfNoCase Request_URI \.(?:exe|t?gz|zip|bz2|sit|rar)$ no-gzip dont-vary
    SetEnvIfNoCase Request_URI \.(?:pdf|doc?x|ppt?x|xls?x)$ no-gzip dont-vary
    SetEnvIfNoCase Request_URI \.avi$ no-gzip dont-vary
    SetEnvIfNoCase Request_URI \.mov$ no-gzip dont-vary
    SetEnvIfNoCase Request_URI \.mp3$ no-gzip dont-vary
    SetEnvIfNoCase Request_URI \.mp4$ no-gzip dont-vary
</Location>

D.4.2.13 Checking Browser Response Times

Use HTTP request monitoring tools, such as Firebug (Firefox) or httpWatch (Internet Explorer and Firefox), to monitor HTTP traffic between the browser and the server. If you understand HTTP request flows and timings for common user actions it can help you diagnose and resolve WebCenter Portal application performance issues. For example, whenever a user logs in, several redirects can happen between the user's browser and servers. If requests to the login server are taking too long you will know to investigate the login server, rather than WebLogic Server. Similarly, if you see that requests to load the application's landing page is slow, you can focus on making the landing page faster.

D.4.2.14 Warm up the System Before Re-Testing Performance

If you restart WebCenter Portal managed servers for any reason, make sure that you warm up the system before re-testing performance to avoid Just-In-Time (JIT) compilation overhead.

To warm up your system, you can either manually perform the steps that you later want to measure for performance, or you can use a load test tool to run a few iterations of the steps. After restarting WebLogic managed servers the initial requests are often slower than usual, that is, not typical of the performance that end users would experience.

D.4.3 How to Identify Slow Pages

Use Fusion Middleware Control to determine the slowest pages in a WebCenter Portal application. If a poorly performing page is also very popular (the Invocation metric is high) then it makes sense for you to focus efforts to improve performance on those pages.

To find the slowest WebCenter Portal pages:

  1. Log in to Fusion Middleware Control and navigate to the home page for the Spaces or Framework application:

  2. Do one of the following:

    • For WebCenter Portal: Spaces - From the WebCenter Portal menu, select Monitoring > Recent Page Metrics.

    • For WebCenter Portal: Framework applications - From the Application Deployment menu, select WebCenter Portal > Recent Page Metrics.

    Page requests that respond slower than the pageResponseTime threshold display "red" in the chart at the top of the page.

  3. Click the Sort Descending arrow in the Time (ms) column to sort the page requests by response times.

    Page response times that exceed the threshold display "orange" in the table.

  4. Identify the slowest pages and make a note of the space in which the page displays.

  5. For more detailed metrics, including how frequently the slowest pages are requested, do one of the following:

    • For WebCenter Portal: Spaces - From the WebCenter Portal menu, select Monitoring > Overall Page Metrics.

      Note: Requests for pages in the Home space are excluded from the “Overall Page Metrics" page.

    • For WebCenter Portal: Framework applications - From the Application Deployment menu, select WebCenter Portal > Overall Page Metrics.

See also, Section 39.1.5, "Understanding Page Request Metrics" and Section 39.3, "Customizing Key Performance Metric Thresholds and Collection for WebCenter Portal".

D.4.4 How to Troubleshoot Slow Page Requests

Use the information in this section to diagnose issues relating to poor page performance:

D.4.4.1 Troubleshooting Live Requests

To troubleshoot slow page requests that are still running, extract and view a JRockit Flight Recorder (JFR) recording against the server on which the user session is running. See also, Section D.4.5, "How to Troubleshooting Requests using JRockit Flight Recordings".

Alternatively, take several evenly spaced thread dumps, for example, every 1 or 2 seconds as described in Section D.4.2.8, "Generating Thread Dumps to Diagnose Extremely Slow Page Performance, High Thread Counts, and System Hangs".

If you compare the thread dumps, you might see threads that spent a long time on certain method calls as the call stacks are the same in several consecutive thread dumps. For example, you might see a method call to a database, Oracle WebCenter Content Server, collaboration server, portlet producer, LDAP server, and so on, in which case you can investigate the associated backend server to diagnose the issue further.

D.4.4.2 Troubleshooting Stuck Threads

Stuck threads can occur for several reasons:

  • Server is nearly out of memory. If the server is close to out of memory, all requests slow down. To resolve out-of-memory issues, see Section D.4.4.4, "Troubleshooting Memory Leaks and Heap Usage Problems".

  • Deadlock threads. Take thread dumps and search for deadlock threads. This normally exposes an issue with the product code.

  • Extremely slow page requests. Take several evenly spaced thread dumps and find out which method is taking a long time to execute.

    If a request is taking longer than 10 minutes, the stuck thread is reported to Oracle WebLogic Server server_name.out in the following directories:

    (UNIX) DOMAIN_HOME/servers/server_name/logs
(Windows) DOMAIN_HOME\servers\server_name\logs

    For example:

    <Mar 4, 2012 7:44:08 AM PST> <Error> <WebLogicServer> <BEA-000337> 
<[STUCK] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "600" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@18986012[
GET /server_name/faces/PimDashboardUiShellPage?_afrLoop=1398820150000&_afrWindowMode=0&_adf.ctrl-state=a44e7uxcc_13 HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/x-ms-application, application/x-ms-xbap, application/vnd.ms-xpsdocument, application/xaml+xml,
application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*
Accept-Language: fr
UA-CPU: x86
...
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds
. Stack trace:
Thread-164 "[STUCK] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
    jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
    jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
    java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
    java.net.SocketInputStream.read(SocketInputStream.java:107)
...

Diagnosing a Stuck Thread

If the stack shows the thread is waiting for a response from another server, check the status of the other server and see it has performance problems before proceeding with the steps below.

To determine what the stuck thread was doing prior to becoming stuck, perform the following steps:

  1. Look at the next few log messages in server_name.out for a message indicating an incident has been created. For example:

    <Mar 4, 2012 7:44:10 AM PST> <Alert> <Diagnostics> <BEA-320016> 
<Creating diagnostic image in DOMAIN_HOME/servers
/server_name/adr/diag/ofm/MyDomain/
server_name_1/incident/incdir_394 with a lockout minute 
period of 1.>

    The above message may not always appear after each stuck thread reported. It is printed at most four times an hour. If the message does not appear, manually look for the incident directory by checking the readme file in the subdirectories under the following directories:

    (UNIX) DOMAIN_HOME/servers/server_name/adr/diag/ofm/domain_name/server_name/incident
(Windows) DOMAIN_HOME\servers\server_name\adr\diag\ofm\domain_name\server_name\incident

    The incident directory contains a WLDF diagnostic image which contains the JFR recording, and a file containing the thread dump.

    For more information about diagnosing incidents, see the "Diagnosing Problems" chapter in the Oracle Fusion Middleware Administrator's Guide.

  2. Review the thread dump to find the call stack of the thread. If the thread is blocked waiting for a lock, check what the thread holding the lock is doing.

  3. If the call stack shows that JDBC calls are taking a long time, generate an AWR report on the database to find the query and which table to look and tune.

    See Section D.4.2.5, "Generating Automatic Workload Repository (AWR) Reports for the Database".

  4. Review the JRockit flight recording file JRockitFlightRecorder.jfr for more details. You will also need the ECID of the request which is recorded in the readme.txt file of the incident directory, and also the Oracle WebLogic Server log.

    See also, Section D.4.5, "How to Troubleshooting Requests using JRockit Flight Recordings".

Since the ECID of the request that caused the stuck thread is recorded in the error message, you also can follow the steps for troubleshooting slow requests documented in Section D.4.4.5, "Troubleshooting Slow Requests for Content."

D.4.4.3 Troubleshooting Slow Requests Using JFR Recordings

See Section D.4.5, "How to Troubleshooting Requests using JRockit Flight Recordings".

D.4.4.4 Troubleshooting Memory Leaks and Heap Usage Problems

If WebCenter Portal application performance degrades over time, heap usage and garbage collection activity is increasing, and you see OutOfMemoryErrors, there could be memory leaks in the application causing the amount of free memory in the JVM to continuously decrease.

Figure D-6 shows a typical memory leak trend displayed in JConsole.

Figure D-6 Typical Memory Leak Trend in JConsole

Typical Memory Leak Trend in JConsole

To solve this problem:

  1. Determine the cause of OutOfMemoryErrors errors:

    • Review the server_name.out file for OutOfMemoryErrors errors.

      The server_name.out file is located at:

      (UNIX) DOMAIN_HOME/servers/server_name/logs
(Windows) DOMAIN_HOME\servers\server_name\logs

    • Take a memory dump when OutOfMemoryErrors errors occur.

      For example:

      On Sun HotSpot: jmap -dump:live,format=b,file=<path>/heap.hprof <pid>

      On JRockit: jrcmd <pid> hprofdump filename=<path>/heap.hprof

      You can configure JRockit to automatically generate a heap dump in HPROF binary format (.hprof file) each time an OutOfMemoryErrors occurs, by setting the JRockit JVM option, -XX:+HeapDumpOnOutOfMemoryError. For details, refer to the Oracle JRockit Command-Line Reference available at: http://docs.oracle.com/cd/E15289_01/doc.40/e15062/title.htm

  2. Restart the managed server.

    If the problem persists, proceed to Step 3.

  3. Open the heap.hprof file with a heap-dump analysis tool that can handle binary HPROF format, such as Eclipse Memory Analyzer.

  4. Determine which objects and classes are retaining the most memory.

  5. If necessary, take several heap dumps to determine which objects or classes are consuming and increasing the amount of memory.

    Take at least two memory dumps:

    • Take the first dump when the system is warmed up and stabilized.

    • Take the second dump, when the system is about to run out of memory, that is, full garbage collection gets less than 300MB from the maximum heap size.

    Instructions on how to take a heap dump using Sun HotSpot (jmap) or JRockit (jcrcmd) is described in step 1.

    See also "Running Diagnostic Commands" chapter in the Oracle JRockit JDK Tools Guide. Many heap dump analysis tools, such as Eclipse Memory Analyzer, enable you to compare two heap dumps to identify memory growth areas.

    Heap dumps provide information on why memory is retained (Retained Heap). Sometimes it is necessary to know how memory is allocated to further resolve the issue. For these cases, proceed to Step 6.

  6. Use the JRockit Memory Leak Detector tool that is part of JRockit Mission Control Client to understand how memory is allocated.

    For more information, see the JRockit Mission Control online help.

D.4.4.5 Troubleshooting Slow Requests for Content

If slow page performance is due to content/document-related components, for example, Documents service task flows, Content Presenter task flows, wikis or blogs, Oracle recommends that you review performance metrics for the backend Oracle WebCenter Content Server (System Audit Information page). For details, see "System Audit Information" in Oracle WebCenter Content System Administrator's Guide for Content Server.

Ensure that the systemdatabase tracing option is selected so you can see performance information for each query that is sent to the database. For details, see "Server-Wide Tracing" in Oracle WebCenter Content System Administrator's Guide for Content Server.

D.4.5 How to Troubleshooting Requests using JRockit Flight Recordings

JRockit Flight Recorder (JFR) files contain a record of various events that consume time. If requests are slow, you can analyze the JRockit Flight Recorder (JFR) file to find out why request are taking time.

To create a JFR file:

  1. Extract a JFR file from theOracle WebLogic Server server by running the following command:

    UNIX) JROCKIT_HOME/bin/jrcmd jrockit_pid dump_flightrecording recording=1 copy_to_file=path compress_copy=true

(Windows) JROCKIT_HOME\bin\jrcmd.exe jrockit_pid dump_flightrecording recording=1 copy_to_file=path compress_copy=true

    For more information about the jrcmd command-line tool, see "Running Diagnostic Commands" in the Oracle JRockit JDK Tools Guide.

  2. To view the file, start the JRockit Mission Control Client from the following directories:

    (UNIX) JAVA_HOME/bin/bin/jrmc

(Windows) JAVA_HOME\bin\jrmc.exe

  3. Select File > Open File to select the JFR file.

  4. Locate the slowest requests or investigate a specific request:

    To locate the slowest requests: To investigate a specific request:

    1. In the JRockitFlightRecorder.jfr page, click the Events icon.

    2. Click the Log tab at the bottom of the page.

    3. In the Event Type navigation pane on the left, locate Dynamic Monitoring System and then HttpRequest.

    4. Click HTTP request; de-select all the other event types.

    5. In the Log tab, in the Event Log section, click the Duration column to sort the duration in descending order.

      Each row corresponds to a HTTP Request and the duration column shows the response time for that request.

    6. Click the row in the table to view the attributes of the requests.

    7. In the Event Attributes sections, note the start time and the thread that serviced the request.

    1. Find the Execution Context Identifier (ECID) of that request.

      If the request is related to an incident triggered by a STUCK thread, the incident readme.txt file will contain the ECID.

      Alternatively, you can search the Oracle WebLogic Server HTTP access.log for requests from specific users. See the "Viewing and Searching Log Files" section in the Oracle Fusion Middleware Administrator's Guide.

    2. In the JRockit Mission Control Client, in the JRockitFlightRecorder.jfr page, select the WebLogic icon.

      Note: If the Weblogic icon is not available, select Help > Install Plugins to download the Oracle WebLogic Server plug-in.

    3. Click the ECIDs tab at the bottom of the page.

    4. In the ECIDs section, from Filter Column list, select ECID.

    5. Enter the ECID in the search box and select <Enter>.

    6. In the results table, highlight the row with the matching ECID and right-click to bring up the menu.

    7. Select Operative Set > Clear, and then Operative Set > Add matching ECID > ECID to add the ECID to the operative set.

      This enables users to view only events associated with the operative set.

    8. Click the Events icon.

    9. In the Event Type navigation pane on the left, locate Dynamic Monitoring System and then HttpRequest.

    10. Click HTTP request; de-select all the other event types. ** In the Event Log section, click Show Only Operative Set.

      Each row corresponds to the request with the matching ECID.

    11. Click the row in the table to view the attributes of the requests.

    12. Note the start time and the thread that serviced the request.

  5. Once you have identified the start time and the thread that serviced the request, navigate to the Logs tab, and drag the time selector at the top of the screen to include only the time window for the duration of the request.

  6. In the Event Log section, perform the following search:

    1. Deselect Show Only Operative Set.

    2. Enter the thread name in the search box.

    3. From the Filter Column list, select Thread.

    4. Select <Enter>.

  7. In the Event Type navigation pane on the left, click the events of interest. Typically, these events are located under nodes Dynamic Monitoring System, Java Application, and WebLogic > JDBC.

    The selected events appear in the table in the Event Log section.

  8. Click the Start Time column to sort the time when these events occur, or click the Duration column to view the events that took longest.

    The JDBC Statement Execute events corresponds to SQL execution. If there are slow SQL statements, the event details give the SQL text. These events do not have callstacks.

  9. To check the call stacks for slow SQL statements, view the Socket Read event that happens immediately after the JDBC Statement Execute event.

    This event corresponds to Oracle WebLogic Server waiting for the SQL results to return, and it has callstack in the event details.

  10. Review the call stacks for long Java Blocked and Java Wait events to see if you can identify what is causing slow performance.

    See also, "Analyzing Flight Recorder Data in JRockit Mission Control" in Oracle Fusion Middleware Configuring and Using the Diagnostics Framework for Oracle WebLogic Server.

  11. If you need more detail than the information captured in the default recording, and you can reproduce the slow requests, you can start an explicit recording.

    For details, see "Starting an Explicit Recording" in Oracle JRockit Flight Recorder Run Time Guide.

D.5 Using My Oracle Support for Additional Troubleshooting Information

You can use My Oracle Support (formerly MetaLink) to help resolve Oracle WebCenter Portal problems. My Oracle Support contains several useful troubleshooting resources, such as:

  • Knowledge base articles

  • Community forums and discussions

  • Patches and upgrades

  • Certification information

Note:

You can also use My Oracle Support to log a service request.

You can access My Oracle Support at https://support.oracle.com.