This appendix presents troubleshooting information for Oracle WebCenter Portal.
This appendix contains the following topics:
Section G.2, "Troubleshooting Oracle WebCenter Portal Configuration Issues"
Section G.3, "Troubleshooting Oracle WebCenter Portal WLST Command Issues"
Section G.4, "Troubleshooting Oracle WebCenter Portal Performance Issues"
Section G.5, "Using My Oracle Support for Additional Troubleshooting Information"
Section G.7, "Troubleshooting WebCenter Portal Import and Export"
Section G.8, "Troubleshooting Individual Portal and Portal Template Import and Export"
Use this documentation roadmap to find troubleshooting information for Oracle WebCenter Portal. In Figure G-1, find the starting point that best describes your issue, then click the graphic or use the links in Table G-1 to jump to the section that you need.
Figure G-1 Troubleshooting Oracle WebCenter Portal Roadmap
Table G-1 Starting Points for Troubleshooting WebCenter Portal
What do you want to do? | Link to Troubleshooting Section in the Guide |
---|---|
|
Section G.2, "Troubleshooting Oracle WebCenter Portal Configuration Issues" |
|
Section G.3, "Troubleshooting Oracle WebCenter Portal WLST Command Issues" |
|
Section 27, "Monitoring Oracle WebCenter Portal Performance" |
|
Section G.4, "Troubleshooting Oracle WebCenter Portal Performance Issues" |
This section includes the following subsections:
Section G.2.1, "How Do I Find Out Which Oracle WebCenter Portal Version Is Installed?"
Section G.2.2, "WebCenter Portal Menu Does Not Display in Fusion Middleware Control"
Section G.2.4, "Configuration Issues with One or More Tools or Services"
Section G.2.5, "Configuration for One Application Reflects in Another"
Always use Oracle's OPatch utility to obtain version information for Oracle WebCenter Portal products and components installed in your environment.
To run OPatch command with the lsinventory
option:
Set Opatch environment variables WCP_ORACLE_HOME
, MW_HOME
, and PATH
.
See the "Patching Oracle Fusion Middleware with Oracle OPatch" section in Oracle Fusion Middleware Patching Guide.
Run the following OPatch command
opatch lsinventory -details
The output lists which components are installed and their version numbers, similar to that shown here:
... Oracle Upgrade Assistant for Webcenter 11.1.1.8.0 Oracle WebCenter Portal Suite 11.1.1.8.0 Oracle WebCenter Portal Suite 11g 11.1.1.8.0 Oracle WebCenter Portal: Activity Graph 11.1.1.8.0 Oracle WebCenter Portal: Analytics Collector 11.1.1.8.0 Oracle WebCenter Portal: Discussions Server 11.1.1.8.0 Oracle Webcenter Portal: Framework 11.1.1.8.0 Oracle Webcenter Portal: Framework Core 11.1.1.8.0 Oracle WebCenter Portal: Pagelet Producers 11.1.1.8.0 Oracle WebCenter Portal: Personalization 11.1.1.8.0 Oracle Webcenter Portal: Portlet Server 11.1.1.8.0 Oracle WebCenter Portal: RCU 11.1.1.8.0 Oracle Webcenter Portal: Spaces 11.1.1.8.0 Oracle WebCenter Portal: Suite Components 11.1.1.8.0 Oracle WebCenter Portal: Wiki 11.1.1.8.0 ...
Note:
Oracle WebCenter Portal Suite 11g
is a child component of Oracle WebCenter Portal Suite
and the versions are always the same.
See the "Patching Oracle Fusion Middleware with Oracle OPatch" section in Oracle Fusion Middleware Patching Guide and the "Lsinventory Command for OUI-based Oracle Homes" section in Oracle Universal Installer and OPatch User's Guide for Windows and UNIX.
Note:
Always use OPatch to obtain the version number. Versions that display alongside Oracle WebCenter Portal product and component names in Fusion Middleware Control do not reflect the true version number of installed products, as shown in Figure G-2 and Figure G-3.
Figure G-2 Incorrect Version Number for WebCenter Portal in Fusion Middleware Control
Figure G-3 Incorrect Version Numbers for Other Applications in Fusion Middleware Control
After logging into Fusion Middleware Control, you cannot find the WebCenter Portal option in the Application Deployment menu for your application.
Ensure the following:
Deployed application is a Portal Framework application, created using the WebCenter Portal Framework Application template in JDeveloper.
The WebCenter Portal option only displays for applications developed using the WebCenter Portal Framework Application template in JDeveloper.
Deployed Portal Framework application is up and running.
Deployed Portal Framework application contains accurate information about the MDS repository and partition, and the MDS repository is accessible to the application. To verify this information, check the metadata-store-usages
section in the adf-config.xml
file. For information on MDS, see the "Understanding the MDS Repository" section in Oracle Fusion Middleware Administrator's Guide.
Portal Framework application is packaged with required artifacts to support configuration:
adf-jndi-config
namespace is configured in the application's adf-config.xml
file. This is provisioned at design time. The following is an example (the text in bold) of the adf-jndi-config
namespace:
<adf-config xmlns="http://xmlns.oracle.com/adf/config"
xmlns:jndiC="http://xmlns.oracle.com/adf/jndi/config"
xmlns:ns2="http://xmlns.oracle.com/mds/config"
xmlns:ns3="http://xmlns.oracle.com/adf/mds/config">
...
...
</adf-config>
Appropriate listeners exist in the web.xml
file to register the MBeans. This is provisioned at design time. For example, see the text in bold in the following snippet of the web.xml
file:
<listener> <description>ADF Config MBeans</description> <display-name>ADF Config MBeans</display-name> <listener-class>oracle.adf.mbean.share.config.ADFConfigLifeCycleCallBack</listener-class> </listener> <listener> <description>ADF Connection MBeans</description> <display-name>ADF Connection MBeans</display-name> <listener-class>oracle.adf.mbean.share.connection.ADFConnectionLifeCycleCallBack</listener-class> </listener>
ADFConfig
and ADFConnection
MBeans are registered for WebCenter Portal or your Portal Framework application. You can verify whether these MBeans are registered through the System MBean Browser:
In Fusion Middleware Control, open the System MBean Browser for your application. Do one of the following:
For WebCenter Portal, select the menu option WebCenter Portal >System MBean Browser.
For a Portal Framework application, select the menu option Application Deployment > System MBean Browser.
Locate ADFConnection
MBeans for your application under Application Defined MBeans > oracle.adf.mbean.share.connection, as shown in Figure G-4.
Similarly, locate ADFConfig
MBeans for your application under Application Defined MBeans > oracle.adf.mbean.share.config, as shown in Figure G-4.
See also, Section 1.13.4, "System MBean Browser."
Review the latest configuration in adf-connections.xml
and adf-config.xml
to check the content is correct. Some typical problems include:
Configuration file is not compliant with its XML schema For example, there are duplicate configuration elements when the schema only allows for 1 occurrence.
XML namespace is missing for configuration referenced within the file.
XML element is not qualified with the XML namespace.
See also, Appendix A "Exporting Configuration Files with MDS Customizations".
Check the application's diagnostic logs, analyze messages for the modules oracle.adf.mbean.share.connection
and oracle.adf.mbean.share.config
, and determine what must be done:
For WebCenter Portal, the log file is available in the DOMAIN_HOME
/servers/
ServerName
/logs
directory. The log file follows the naming convention of ServerName
-diagnostic.log
. See also, Section 28.2.1, "Viewing and Configuring WebCenter Portal Logs."
For Portal Framework applications, the log file is available in the DOMAIN_HOME
/servers/
ServerName
/logs
directory. The log file follows the naming convention ServerName
-diagnostic.log
. See also, Section 28.2.2, "Viewing and Configuring Portal Framework Application Logs."
When you try to configure WebCenter Portal or a Portal Framework application through Fusion Middleware Control, the following message displays:
Configuration options currently unavailable. The application application_name might be down, did not start-up properly, or is incorrectly packaged.
Check the log files for further details.
For example, you try to change options available through the Application Settings screen or configure connections through the WebCenter Portal Service Configuration screen in Fusion Middleware Control.
Check the application's diagnostic logs:
For WebCenter Portal, the log file is available in the DOMAIN_HOME
/servers/
ServerName
/logs
directory. The log file follows the naming convention of ServerName
-diagnostic.log
. See also, Section 28.2.1, "Viewing and Configuring WebCenter Portal Logs."
For Portal Framework applications, the log file is available in the DOMAIN_HOME
/servers/
ServerName
/logs
directory. The log file follows the naming convention ServerName
-diagnostic.log
. See also, Section 28.2.2, "Viewing and Configuring Portal Framework Application Logs."
Analyze messages for the modules oracle.adf.mbean.share.connection
and oracle.adf.mbean.share.config
, and determine what must be done.
See also, Section G.2.2, "WebCenter Portal Menu Does Not Display in Fusion Middleware Control."
Do not attempt to configure services that your application does not support. WebCenter Portal configuration through Enterprise Manager and WLST fails if you try to configure a service, say discussions, that your application was not designed to use.
The out-of-the-box application, WebCenter Portal, is designed to support all tools and services but Portal Framework applications that you build using JDeveloper only provide artifacts in the application's configuration for services that the developer specifically included during design time. For example, if the developer did not add or configure discussions for the application through JDeveloper, you cannot configure discussions postdeployment through Enterprise Manager and WLST.
If you are having issues configuring or connecting to one or more tool or service, refer to the appropriate troubleshooting section:
Section 9.13, "Troubleshooting Issues with Content Repositories"
Section 12.13, "Troubleshooting Issues with Announcements and Discussions"
Section 21.12.3, "Troubleshooting WebCenter Services Portlets"
Section 30.3, "Troubleshooting Security Configuration Issues"
Chapter G, "Troubleshooting WebCenter Portal Import and Export"
Chapter G, "Troubleshooting Individual Portal and Portal Template Import and Export"
You configured WebCenter Portal or a Portal Framework application, but those configurations also show in another application.
For example, you created or edited a mail connection for a Portal Framework application named MyPortalApp1 and you discover that the connection changes are also seen in another application MyPortalApp2
This happens when multiple applications share the MDS partition in the same schema. To resolve this problem, deploy these applications again and ensure that each application either uses a different MDS schema or a different MDS partition. For information about creating a MDS repository or configuring an existing application to use a different MDS repository or partition, see the "Managing the Metadata Repository" section in Oracle Fusion Middleware Administrator's Guide.
WebCenter Portal or a Portal Framework application is inaccessible or displaying error messages and the diagnostic log files indicates that there is an issue with 'too many open files
'.
Do the following:
Check the number of file handles configured on each of the back-end servers, primarily the database, and increase appropriately.
If the problem persists after increasing the file handles, check the value of fs.file-max
in the /etc/sysctl.conf
file and increase the value appropriately.
This section includes the following subsections:
Section G.3.1, "No Oracle WebCenter Portal WLST Commands Work"
Section G.3.2, "WLST Commands Do Not Work for a Particular Tool or Service"
Section G.3.4, "WLST Shell is Not Connected to the WebLogic Server"
Section G.3.5, "More Than One Application with the Same Name Exists in the Domain"
Section G.3.6, "More Than One Application with the Same Name Exists on a Managed Server"
Section G.3.7, "Already in Domain Runtime Tree Message Displays"
See also, Section 1.13.3.1, "Running Oracle WebLogic Scripting Tool (WLST) Commands."
You are unable to run any WLST commands.
Ensure the following:
Always run Oracle WebCenter Portal WLST commands from your WebCenter Portal Oracle home directory (WCP_ORACLE_HOME
/common/bin
).
If you attempt to run Oracle WebCenter Portal WLST commands from the wrong directory you will see a NameError
.
No files other than Python are stored in the WLST source directory: WCP_ORACLE_HOME
/common/bin/wlst
. This directory must contain files with the .py
extension only.
The default set of files in this location contain legal Python files from Oracle. It is possible that a user copied some non-python script to this directory, for example, a backup file or a test python file with syntax errors.
webcenter-wlst.jar
is located at WCP_ORACLE_HOME
/common/bin/wlst/lib
.
See also, Section 1.13.3.1, "Running Oracle WebLogic Scripting Tool (WLST) Commands."
You are unable to run WLST commands for a particular tool or service, and therefore, you cannot configure that tool/service.
First, run generic non-Oracle WebCenter Portal commands, for example, run listApplications
or displayMetricTableNames
to verify whether these commands work. If generic commands do not work, then apply the solution described in Section G.3.1, "No Oracle WebCenter Portal WLST Commands Work."
If generic commands work, then run test commands to check Oracle WebCenter Portal-specific commands for syntax errors. Run the appropriate WSLT check command (see Table G-2).
Table G-2 File Names and WLST Commands for Oracle WebCenter Portal Tools and Service
Service Name | File Name | WLST Command |
---|---|---|
Activity Graph |
|
|
Activity Stream |
|
|
Analytics |
|
|
Discussions and Announcements |
|
|
Documents |
|
|
External Applications |
|
|
Portal Events |
|
|
Instant Messaging and Presence |
|
|
|
|
|
Notifications |
|
|
Personal Events |
|
|
Producers |
||
PDK-Java Producers |
|
|
WSRP Producers |
|
|
Pagelet Producers |
|
|
Producer Helper |
|
|
RSS News Feed |
|
|
Search |
|
|
Worklist |
|
|
Export/Import - WebCenter Portal applications |
|
|
Export/Import - Portals and Template |
|
|
Synchronize Users |
|
|
Rename Users |
|
|
WebCenter Portal - General |
||
Service Framework |
|
|
General Settings |
|
|
WebCenter Portal and SOA |
|
|
See also, Section 1.13.3.1, "Running Oracle WebLogic Scripting Tool (WLST) Commands."
For more information about Oracle WebCenter Portal's WLST commands, see the "WebCenter Portal Custom WLST Commands" chapter in Oracle Fusion Middleware WebLogic Scripting Tool Command Reference.
You are unable to create a connection with the name Connection_Name
. The following message displays:
A connection with name Connection_Name already exists.
For example, you try to create an external application connection using the WLST command createExtAppConnection
or connect to a mail server using createMailConnection
.
Connection names must be unique (across all connection types) within WebCenter Portal or a Portal Framework application. This error occurs when you try to create a connection with a name that is in use. Ensure that you use a unique name for your connection.
You must connect to the Administration Server for Oracle WebCenter Portal before running WLST commands. Oracle WebCenter Portal WLST commands do not work without a connection.
Run the following command to connect the WLST shell to the managed server:
connect(username, password , serverhost:serverport)
See also, Section G.3.1, "No Oracle WebCenter Portal WLST Commands Work" and Section 1.13.3.1, "Running Oracle WebLogic Scripting Tool (WLST) Commands."
You attempt to perform an operation on WebCenter Portal or a Portal Framework application, such as create a connection for a service or register a portlet producer, and the following message displays:
Another application named "YourApplicationName" exists. Specify the Server on which your application is deployed. Use: server="YourServerName".
This message displays if there are multiple applications with the same name in the domain. This usually happens in a cluster environment, where the same application is deployed to multiple managed servers.
For example, you tried to register a portlet producer for an application named "MyApp" using the following WLST command:
registerWSRPProducer(appName='myApp', name='MyWSRPSamples', url='http://myhost.com:9999/ portletapp/portlets/wsrp2?WSDL')
Specify on which managed server you want to run the WLST command, that is, include the server
argument. For example:
registerWSRPProducer(appName='myApp', name='MyWSRPSamples', url='http://myhost.com:9999/portletapp/portlets/wsrp2?WSDL', server=WC_CustomPortal2)
See also, Section 1.13.3.1, "Running Oracle WebLogic Scripting Tool (WLST) Commands."
You attempt to perform an operation on WebCenter Portal or a Portal Framework application such as create a connection for a service or register a portlet producer, and the following message displays:
Another application named "application_name" exists on the server managedServerName.
This message indicates that there are multiple applications with the same name on specified managed server. This usually happens when applications are assigned different versions.
For example, you tried to register a portlet producer for an application named "MyApp" using the following WLST command:
registerWSRPProducer(appName='myApp', name='MyWSRPSamples', url='http://myhost.com:9999/portletapp/portlets/wsrp2?WSDL')
Specify on which application version you want to run the WLST command, that is, include the server
and applicationVersion
arguments. For example:
registerWSRPProducer(appName='myApp', name='MyWSRPSamples', url='http://myhost.com:9999/portletapp/portlets/wsrp2?WSDL', server=WC_CustomPortal1, applicationVersion=2)
See also, Section 1.13.3.1, "Running Oracle WebLogic Scripting Tool (WLST) Commands."
Use the information in this section to help diagnose performance-related issues for Oracle WebCenter Portal.
This section contains the following sub sections:
Various tools are available for monitoring and troubleshooting performance issues with your Oracle WebCenter Portal environment.
Table G-3 Performance Monitoring and Troubleshooting Tools
Tool | Use to... | See |
---|---|---|
Enterprise Manager |
||
Fusion Middleware Control |
Monitor WebCenter Portal metrics and log files in real-time mode for a single Oracle Fusion Middleware Farm. Check service configuration, including MDS and partitions for WebCenter Portal deployments. |
|
Grid Control |
Monitor WebCenter Portal metrics in real time and from a historical perspective for trend analysis, as well as monitor the underlying host and operating system, databases, and more. Oracle Enterprise Manager 11g Grid Control must be installed separately as it is not a part of the Oracle Fusion Middleware 11g installation. With Grid Control, you can centrally manage multiple Oracle Fusion Middleware Farms and WebLogic Domains. |
|
WebCenter Portal Page Performance Analyzer |
Analyze the performance of portal pages in WebCenter Portal. This tool dynamically measures and presents the performance of individual page components when you display pages in WebCenter Portal. |
|
JConsole |
Graphically monitor Java applications and Java virtual machines (JVM). |
|
JRockit Mission Control |
Capture and present live data about memory, CPU usage, and other runtime metrics. |
|
Eclipse Memory Analyzer |
Find memory leaks and reduce memory consumption. |
|
Threadlogic |
Analyze thread dumps. |
Use the actions listed in Table G-4 to troubleshoot overall system slowness:
Table G-4 Troubleshooting Overall System Slowness
Action | Description | More Information |
---|---|---|
1 |
Verify key system resources. |
|
2 |
Use |
|
3 |
Monitor the performance of your Java virtual machine (JVM). |
|
4 |
Verify OID and database connection pool settings. |
|
5 |
Generate automatic workload repository (AWR) reports for Oracle databases to diagnose database-related issues. |
Generating Automatic Workload Repository (AWR) Reports for the Database |
6 |
Use |
|
7 |
Use |
|
8 |
Collect thread dumps. |
|
9 |
Look for errors in |
|
10 |
Use DMS Spy to monitor internal DMS metric data, such as activity in the Java Object Cache. |
|
11 |
Look at the Verify HTTP compression settings. |
|
12 |
Use HTTP monitoring tools to analyze requests and response times. |
|
13 |
Warm up your system before taking performance measurements. |
If you are experiencing performance issues its important to verify that you have sufficient system hardware resources, that is, adequate CPU and physical memory capacity for your Oracle WebCenter Portal installation.
Low system resources can cause many different problems. System resources are used by individual services. Regularly monitoring and recording system usage can help you determine whether you need to upgrade your system hardware, or if some services need to be moved to another machine.
To verify system CPU and memory:
On Linux, review CPU and memory information in the following files:
/proc/cpuinfo
The cpuinfo
file provides important CPU information including the model, CPU cores, CPU MHz, cache size, and flags which show what instruction sets are available on the processor. Systems with multiple processors or multiple cores have separate entries for each.
/proc/meminfo
The important fields to look for in the meminfo file include MemTotal
, MemFree
, Cached
, and SwapTotal
.
On Windows, access CPU and memory information through the Task Manager (Performance > Resource Monitor).
On Linux, you can use top
and vmstat
utilities to see if system slowness is caused by CPU, memory, or I/O contention issues. If you discover that your system resources are low, you can:
Move processes to other machines or shut down unused processes/programs to free up more physical memory and/or CPU cycles.
Upgrade your system hardware resources
For more information, see:
Note:
On Windows, use Task Manager to monitor system resources and shut down unused processes and programs. Refer to your Windows documentation for more information.
The top
utility displays a continually updating report of system resource usage so you can identify the top memory and CPU consumers on your system.
The top portion of the report lists information such as the system time, uptime, CPU usage, physical and swap memory usage, and number of processes. Below that is a list of the processes sorted by CPU utilization.
Note:
Use Shift+M
to sort by memory usage. Use Shift+P
to sort by CPU usage.
# top 2:10:49 up 8 day, 3:47, 20 users, load average: 0.34, 0.19, 0.10 75 processes: 20 sleeping, 2 running, 8 zombie, 0 stopped CPU states: 5.1% user 1.1% system 0.0% nice 0.0% iowait 93.6% idle Mem: 512216k av, 506176k used, 6540k free, 0k shrd, 21888k buff Swap: 1044216k av, 161672k used, 882544k free 199388k cached PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND 2330 admin 15 0 161M 70M 2132 S 4.9 14.0 1000m 0 oracle 2605 lin 15 0 8240 6340 3804 S 0.3 1.2 1:12 0 oracle 3413 harvey 15 0 6668 5324 3216 R 0.3 1.0 0:20 0 oracle
Troubleshooting Tips - top:
If free
memory is < 100Mb and cached memory is < 1GB, system memory is running low.
If %wa
(I/O WAIT) is always more than 10%, the system may be slow because it is blocked by physical I/O.
Ensure that the idle
value is close to 100% and system
/user
CPU usage is close to 0% when there is no load on the system.
In the memory view of top
, identify the % memory usage (%MEM
) for each process (PID
). If the memory is tight and swap space usage is high, consider:
- moving process with high memory usage to another machine
- increasing memory allocation to the virtual machine
- adding physical memory.
The vmstat
utility shows running statistics on various parts of the system including system processes, memory, swap, I/O, CPU, and the Virtual Memory Manager. These statistics are generated using data from the last time the command was run to the present. The first time you run the command, data displays from the last reboot until the present.
When you run vmstat
you can specify how often you want the statistics to refresh (in seconds) using the format: vmstat <refresh rate in seconds>
For example: vmstat 3
Frequent system resource usage updates, enable you to see trends in CPU/memory usage and how usage trend impact WebCenter Portal or a Portal Framework application performance. If the system is stable, the swap
metrics (si
and so
) should register near zero.
# vmstat 3 procs memory swap io system cpu r b swpd free buff cache si so bi bo in cs us sy id wa 3 0 144 1058184 868324 12343056 0 0 220 83 33 14 10 1 88 1 0 0 144 1058184 868324 12343056 0 0 0 117 478 656 5 1 94 0 0 0 144 1058192 868324 12343056 0 0 0 69 696 860 9 1 90 0 2 0 144 1058264 868324 12343056 0 0 0 0 914 1127 5 2 93 0 0 0 144 1057864 868324 12343316 0 0 0 184 857 1021 7 1 92 0 2 0 144 1057592 868324 12343316 0 0 0 155 1596 1646 5 4 91 0 0 0 144 1057592 868324 12343316 0 0 0 0 737 839 5 2 94 0 0 0 144 1057592 868324 12343316 0 0 0 57 694 743 8 2 91 0 2 0 144 1057592 868324 12343316 0 0 0 0 358 437 2 0 98 0
Troubleshooting Tips - vmstat:
When the swap
metrics (si
and so
) are frequently non-zero, system memory becomes tight, and the system may go into thrashing mode. Low system memory significantly affects performance, that is, any program running on the system becomes unusually slow. If you frequently experience low system memory, increase the memory on the machine.
If CPU idle time (id
) is constantly less than 10%, the CPU is running at full or over capacity. Under such conditions, any more load on the system results in significant performance degradation.
Your Java virtual machine (JVM) can significantly affect Oracle WebCenter Portal performance. Oracle recommends that you continually monitor your JVM to track:
Memory usage
CPU usage
Thread activity
You can monitor all three metrics from your applications's Recent WebLogic Server Metrics page in Fusion Middleware Control (Figure G-5). For details, see Section 27.1.8, "Understanding WebLogic Server Metrics" and Section 27.2, "Viewing Performance Metrics Using Fusion Middleware Control."
Figure G-5 Recent WebLogic Server Metrics Page - JVM Metrics
Monitor memory trends - JVM frequently allocates and releases memory. To detect memory leaks and high memory usage, you must analyze the memory trend over a long period of time (an hour or several hours).
When you see the bottom trend line on a memory graph increasing, it indicates a memory leak. The bottom points on a memory graph show how low memory usage can go after full garbage collection. If you take at least two memory dumps, one when the memory is healthy and another when the memory full garbage collection cannot recycle too much, you can compare the memory dumps and identify which components are consuming memory.
See also, Section G.4.5.4, "Troubleshooting Memory Leaks and Heap Usage Problems."
Monitor CPU usage - Occasional spikes in CPU usage is normal but if CPU usage remains high (85-90%) over a long period of time, it normally indicates there is an issue with CPU which can impact performance.
If CPU usage appears overloaded, comparing several thread dumps at fixed intervals (for example, every 5 seconds) can help reveal causes of high CPU usage.
Alternatively, profiling tools, such as JRockit's Flight Recorder, can provide further insight into CPU usage.
Monitor thread activity and analyze thread dumps (stuck threads, blocked threads, deadlocks) - The number of threads should stabilize under stable load conditions. Sudden spikes or an increasing number of threads, generally indicates an issue in your system.
When that is happening, you can take multiple thread dumps to understand the calling stack of stuck/blocked threads or deadlocks. You can use thread analysis tools to identify what is causing the extra threads and what are they doing, as well as detect contentious areas and slow performing areas. See also, Section G.4.5.2, "Troubleshooting Stuck Threads."
Tip:
Oracle Fusion Middleware's Diagnostic Framework collects and manages information about common problems, such as stuck threads and deadlocked threads, to help you diagnose and resolve such issues. Alternatively, you can send diagnostic dumps to Oracle Support Services. For more details, see "Diagnosing Problems" in Oracle Fusion Middleware Administrator's Guide.
There are various tools available to monitor JVM performance, including Fusion Middleware Control, JConsole, JVisualVM, and JRockit Mission Control. Use any of these tools to show the current state of the JVM, as well as all the active threads and their states. See also, the "Tuning Java Virtual Machines (JVMs)" section in Oracle Fusion Middleware Performance and Tuning Guide
In the first instance, administrators can use the application's Recent WebLogic Server Metrics page in Fusion Middleware Control to monitor JVM (as shown in Figure G-5). If more aggressive real time metrics are required, another option is to use JConsole (shown in Figure G-6).
JConsole is available with all types of JVM, including HotSpot, JRockit, IBM JVM, and so on.
JConsole's executable is located at: JAVA_HOME/bin/jconsole.exe
.
This section describes how to verify connection pool settings for:
The information in this section might be useful if you are experiencing slow response times and your diagnostics log files contain connection/connection pool messages or a recent thread dump contains calling stacks that are waiting to get connections from connection pool.
Administrators can use Fusion Middleware Control to monitor JDBC connection metrics for WebCenter Portal or a Portal Framework applications. Use the JDBC usage information on the WebLogic Server Metrics Page (Figure G-7) to assess whether JDBC configuration or the connection pool size needs to be adjusted.
See also, Section 27.2, "Viewing Performance Metrics Using Fusion Middleware Control."
Figure G-7 Recent WebLogic Server Metrics Page - JDBC Usage
The Recent JDBC Usage chart shows the number of JDBC connections currently open on the managed server. The JDBC session count that is displayed here, is the sum of the Current Active Connection Count metric for each JDBC data source.
If usage is high and the trend is rising, administrators can use WebLogic Server Administration Console to view and configure data source connection pool settings and monitor usage patterns for individual data sources in more detail. If you monitor data source usage over a period of time you can:
Determine the best connection pool size for a particular database connection.
Detect and resolve connection pool leakage, that is, if you notice that the number of data source connections do not decrease without load.
Tip:
Select "Customize the Table" when monitoring connection pools, to display additional metrics such as:
Connection Delay Time
Current Capacity High Count
Failures To Reconnect Count
Wait Seconds High Count
Waiting For Connection Current Count
Waiting For Connection Failure Total
For high concurrency systems, you may want to adjust the maximum number of connections in the pool (Maximum Capacity setting). Out-of-the-box maximum values for the various WebCenter Portal data sources are shown here:
WebCenter Portal Data Sources |
Connection Pool Default Maximum Capacity |
---|---|
WebCenterDS |
50 |
ActivitiesDS |
25 |
mds-SpacesDS |
50 |
mds-owsmDS |
15 |
mds-PageletProducerDS |
50 |
mds-ServicesProducerDS |
50 |
mds-wcpsDS |
50 |
PersonalizationDS |
25 |
Portlet-ServicesProducerDS |
50 |
PortletDS |
50 |
WC-ServicesProducerDS |
25 |
Note:
Each connection uses memory in the WebLogic Server and consumes processes in the database so do not specify an unnecessarily large connection pool value.
To monitor a particular data source, log in to the WebLogic Server Administration Console, select Services>Data Sources, click the data source name, and then click the Monitoring tab.
To modify the connection pool for a particular data source, log in to the WebLogic Server Administration Console, select Services>Data Sources, click the name of the data source, and then click the Connection Pool tab.
See the "Tuning Data Source Connection Pools" section in Oracle Fusion Middleware Configuring and Managing JDBC Data Sources for Oracle WebLogic Server.
Typically, JNDI connection pooling is always turned on. However, some additional configuration is required if the connection between WebCenter Portal/Portal Framework application and the identity store (this may be Oracle Internet Directory, Active Directory, and so on) uses SSL. By default, when you choose an SSL port, the JNDI connections are not pooled causing increased response time and decreased performance when logging in, looking up users, groups, or other identity store entities. For more information and instructions, see the "Tuning Identity Store Configuration" section in Oracle Fusion Middleware Performance and Tuning Guide.
If database access is slow, for example, activity stream queries, MDS queries or JPA (Java Persistence API) queries are slow for various operations in your WebCenter Portal application, you can analyze Automatic Workload Repository (AWR) reports to diagnose the root cause of performance-related problems in your Oracle database.
Before generating the AWR report, first check the general health (CPU/Memory) of that machine hosting your database (see Section G.4.2.2, "Monitoring System Resource Usage"). If system resource limitations are not causing poor performance, examine the database performance information and statistics in the AWR report.
For more detail, see the "Tuning Database Parameters" section in Oracle Fusion Middleware Performance and Tuning Guide and the "Generating Automatic Workload Repository Reports" section in Oracle Fusion Middleware Performance and Tuning Guide.
Use tcpdump
, a network utility that listeners and captures network traffic to investigate and diagnose network related problems.
For example, if WebCenter Portal page performance metrics in Enterprise Manager indicate that server performance is operating normally while end users are reporting unstable/slow performance, you could have a problem with your network. You can use tcpdump
or a similar network monitoring tool to trace network your traffic to see if any environmental issue (on the network) is causing an unexpected large latency.
Refer to tcpdump
documentation for information on how to run this utility and analyze the network data.
Use ping
to measure network latency.
Oracle WebCenter Portal installations depend on various backend components and services, such as Oracle HTTP Server, Oracle WebCenter Content Server, LDAP server, instant messaging and presence servers, mail servers, portlets servers, databases, and more. If all your required components are not on the same machine, Oracle recommends they are closely located to minimize network latency and, if possible, are on the same subnet.
Use ping
from one machine to another to verify that network latency is less than 1ms.
Oracle recommends that you generate thread dumps to a dedicated thread dump file and then use a thread analysis tool such as ThreadLogic or a simple text editing tool, to understand and correlate the thread execution logic and progress.
To generate thread dumps for Sun JVM (HotSpot) to a file:
>jstack <pid> > threaddump.txt
To generate thread dumps for Oracle JRockit JVM to a file:
>jrcmd print_threads <pid> > threaddump.txt
Sometimes its useful to generate a series of thread dumps at fixed sampling intervals to confirm problems such as extremely slow method calls, stuck threads, deadlocks, and so on. For example, you could create a simple script to generate dumps every second or if you are diagnosing a slow request you can choose a suitable duration to cover the length of the slow request.
This example generates thread dumps every second:
#!/bin/bash
for i in {1..20}
do
JAVA_HOME/bin/jstack <pid> > thread_dump_$i.txt
sleep 1
done
For more detail about the capabilities of ThreadLogic or any other thread analysis tool, refer the appropriate manufacturer's documentation.
Note:
When WebLogic servers detect a deadlock or stuck thread, a related thread dump is automatically generated in the server's output log file, located at:
DOMAIN_NAME\servers\SERVER_NAME\logs\SERVER_NAME.out
For example, the output log for the out-of-the-box application WebCenter Portal is available at DOMAIM_NAME\servers\WC_Spaces\logs\WC_Spaces.out
.
The output file is a simple text file that contains various server messages, including thread information.
Look for errors, incidents, and warnings in the diagnostics log for the managed server that is hosting WebCenter Portal or a Portal Framework application, that is, <managed_server_name>
-diagnostic.log
.
When your application is running with some error condition, it can have a big impact on performance. For example, if a connection to WebCenter Content Server becomes intermittent or not accessible, pages with content related components respond very slowing while attempting to connect and eventually may time out.
ERROR, INCIDENT, and WARNING messages due to timeouts, unavailable services, cache errors, and so on, are logged to a diagnostic log file which you can view from Fusion Middleware Control. See also, Section 28.2.1, "Viewing and Configuring WebCenter Portal Logs."
Note:
When measuring performance, ensure that DEBUG/TRACE messages, that is, levels lower than CONFIG (700) are not being logged. When FINE, FINER, or FINEST messages display, the system is running in debugging mode and this means that most requests are significantly slower than normal. If you configure lower level logging temporarily to debug a problem, ensure that you change the log level before taking performance measurements.
The DMS Spy servlet provides access to internal DMS metric data from a web browser. This servlet is useful if you want to monitor the system for long period of time and it can help you understand internal system behavior and performance.
You would not normally use DMS Spy to monitor Oracle WebCenter Portal-specific metrics since this information is more easily available through Fusion Middleware Control. However, if you interested in other underlying metrics, such as warning/error messages about the ADF application module (AM) pool display, you can investigate AM pool metrics here. Similarly, if messages relating to the Java Object Cache (JOC) display, you can turn on the JOC's DMS monitoring feature to observe activities in JOC.
To enable JOC DMS monitoring, edit the javacache.xml
file (normally it is in <webcenter_portal_domain>/config/fmwconfig/servers/WC_Spaces/javacache.xml
)
Open the javacache.xml
file.
Typically, the file is located at: <webcenter_domain>/config/fmwconfig/servers/<server_name>/javacache.xml
For example, for WebCenter Portal, the file is located at: <webcenter_domain>/config/fmwconfig/servers/WC_Spaces/javacache.xml
Change the entry:
<dms enabled="false"/>
To: <dms enabled="true"/>
Restart the managed server on which WebCenter Portal or your Portal Framework applications deployed.
For example, for WebCenter Portal, restart WC_Spaces.
Once enabled, you see two new entries in the left panel: java_cache_region and joc
java_cache_region shows how much cache memory is allocated to each area and the values displayed can help you determine cache size settings. For example, by default the MDS caching size is set to 100MB in adf-config.xml
:
<max-size-kb>100000</max-size-kb>
Both Region Name: ADFApplication<N> and ADFApplication<N>/main_region are used by the MDS cache. If the sum of ADFApplication<N> and ADFApplication<N>/main_region memory is close to 100MB, and your JVM has enough unused heap memory, you could increase the MDS caching size (<max-size-kb>) so the MDS cache is not so full, and this will improve the efficiency of the MDS cache.
For more information, see the "Viewing Performance Metrics Using the Spy Servlet" section in Oracle Fusion Middleware Performance and Tuning Guide.
Most web pages include resources that do not change very often, such as images, CSS files, JavaScript files, and so on. As such resources take time to download over the network, the time taken to load a web page increases. HTTP caching allows such resources to be saved or cached, by a browser or proxy. Once a resource is cached, a browser or proxy can refer to the locally cached copy instead rather than downloading it again on subsequent visits to the web page. Caching reduces round-trip time by eliminating HTTP requests for the required resources. This not only helps significant reducing page load time for subsequent user visits, but also reduces the bandwidth and hosting costs for your site.
By default, static resources serviced by WebCenter Portal use the ADFCachingFilter
to include the required response headers which allow browsers to cache static content. If your site is experiencing performance issues, you must confirm that your browser is caching static resources. If caching is not correctly working, static WebCenter Portal resources are repeatedly fetched from the server rather than being cached in the browser, and this can impact performance. A possible reason for caching issues with static portal resources could be dues to some loss introduced by a proxy or network component sitting between WebCenter Portal and the browser.
Note:
For more information about ADFCachingFilter
, see Oracle Fusion Middleware Web User Interface Developer's Guide for Oracle Application Development Framework.
Use HTTP request monitoring tools, such as Firebug (Firefox) or httpWatch (Internet Explorer and Firefox), to monitor HTTP traffic between the browser and the server. With these tools, you can see if there is a problem with the cache, that is, whether static resources are fetched for each request, not cached for long enough, or not cached at all for static resources (such as JavaScript and CSS files). When using these monitoring tools, confirm that there are browser-cache related tags in the response, for example, Cache-Control
header with a suitable max-age
value or an Expires
tag with a suitable cache expiry time.
You can also analyze HTTP caching by examining the access.log
for the managed server on which WebCenter Portal or your Portal Framework application is deployed. For example, look at the log to see whether a particular user/IP is repeatedly making requests for the same resource. If the log contains repeated requests, the cache header on requests might be wrong and you must investigate caching issue further. Start by accessing the static resource to eliminate various tiers. For example, use wget
or curl
to fetch the static resource directly using the WebLogic server port by issuing the same on the local machine. Next, access the same resource through any other entity front-ending the WebLogic server that is hosting the application, for example Oracle HTTP Serve or Apache. If needed, enable packet tracing to find the response from the WebCenter Portal tier to see whether issues are occurring on the Oracle WebCenter Portal side, or the problem is on the network.
Note:
The access log for WebCenter Portal is located at: ORACLE_HOME/user_projects/domains/wc_domain/servers/WC_Spaces/logs/access.log
While this section describes static resources serviced by WebCenter Portal, you must also consider similar issues for static resources in your environment. For example, if you have a rich UI where common pages reference static resources that you own, you must review how such content is cached. Consider using Apache Header
directives to drive the caching of static resources, either based on file type (such as images, CSS), or based on URL path patterns. For example, the following configuration in Apache sends back a response header to the browser to cache all content under the URL path "/my_images/
" for 30 days (2592000 is the number of seconds in 30 days):
<Location /my_images/> Header set Cache-Control "max-age=2592000, public" </Location>
In addition to HTTP caching, it is important that you review HTTP compression characteristics for responses. HTTP Compression is a publicly defined way to compress content (mostly textual) that is transferred from web servers to browsers. Compression reduces the number of bytes that are transmitted which improves performance. HTTP Compression uses public domain compression algorithms to encode HTML, XML, JavaScript, CSS and other file formats at the server-side. This standards-based method for delivering compressed content is built into HTTP/1.1, and all modern web browsers support the HTTP/1.1 protocol, that is. they can decode compressed files automatically at the client-side. As a result, no additional software or user interaction is required on the client-side.
By default, static resources serviced by WebCenter Portal use the ADFCachingFilter ()
to compress static content like CSS and JavaScript. If your site is experiencing performance issues, you must confirm that your browser is seeing compressed responses. You can use one of the HTTP request monitoring tool described in Section G.4.2.11, "Verifying HTTP Request Caching" to scan responses for a Content-Encoding
response header with the value gzip
. A Content-Encoding
response header is only sent if the client (that is, the browser) supports compressed content.
Note:
For more information about ADFCachingFilter
, see in Oracle Fusion Middleware Web User Interface Developer's Guide for Oracle Application Development Framework.
If the browser sends the request header Accept-Encoding: gzip
, then you know the browser can process compressed content. If you do not see a compressed response, make sure that the client sends an Accept-Encoding
header stating it can handle compressed content.
File format that typically require compression include: HTML, CSS, XML, and JavaScript. As most images, music, and videos are already compressed, you do not need to configure these file formats for compression.
Consider using Apache's mod_deflate
or mod_gzip
to drive the compression of resources, either based on MIME type (such as, text/css) or based on file extension (such as *.html). For example, the following configuration in Apache compresses resources listed in each AddOutputFilterByType
:
<Location /> SetOutputFilter DEFLATE AddOutputFilterByType DEFLATE text/plain AddOutputFilterByType DEFLATE text/xml AddOutputFilterByType DEFLATE text/html AddOutputFilterByType DEFLATE application/xhtml+xml AddOutputFilterByType DEFLATE text/css AddOutputFilterByType DEFLATE application/xml AddOutputFilterByType DEFLATE image/svg+xml AddOutputFilterByType DEFLATE application/rss+xml AddOutputFilterByType DEFLATE application/atom_xml AddOutputFilterByType DEFLATE application/x-javascript SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary SetEnvIfNoCase Request_URI \.(?:exe|t?gz|zip|bz2|sit|rar)$ no-gzip dont-vary SetEnvIfNoCase Request_URI \.(?:pdf|doc?x|ppt?x|xls?x)$ no-gzip dont-vary SetEnvIfNoCase Request_URI \.avi$ no-gzip dont-vary SetEnvIfNoCase Request_URI \.mov$ no-gzip dont-vary SetEnvIfNoCase Request_URI \.mp3$ no-gzip dont-vary SetEnvIfNoCase Request_URI \.mp4$ no-gzip dont-vary </Location>
Use HTTP request monitoring tools, such as Firebug (Firefox) or httpWatch (Internet Explorer and Firefox), to monitor HTTP traffic between the browser and the server. If you understand HTTP request flows and timings for common user actions it can help you diagnose and resolve application performance issues. For example, whenever a user logs in, several redirects can happen between the user's browser and servers. If requests to the login server are taking too long you will know to investigate the login server, rather than WebLogic Server. Similarly, if you see that requests to load the application's landing page is slow, you can focus on making the landing page faster.
If you restart an Oracle WebCenter Portal managed server for any reason, make sure that you warm up the system before re-testing performance to avoid Just-In-Time (JIT) compilation overhead.
To warm up your system, you can either manually perform the steps that you later want to measure for performance, or you can use a load test tool to run a few iterations of the steps. After restarting WebLogic managed servers the initial requests are often slower than usual, that is, not typical of the performance that end users would experience.
Use Fusion Middleware Control to determine the slowest pages in a WebCenter Portal or a Portal Framework application. If a poorly performing page is also very popular (the Invocation metric is high) then it makes sense for you to focus efforts to improve performance on those pages.
To find the slowest pages:
Log in to Fusion Middleware Control and navigate to the home page for WebCenter Portal or Portal Framework application:
Do one of the following:
For WebCenter Portal - From the WebCenter Portal menu, select Monitoring > Recent Page Metrics.
For WebCenter Portal Framework applications - From the Application Deployment menu, select WebCenter Portal > Recent Page Metrics.
Page requests that respond slower than the pageResponseTime
threshold display "red" in the chart at the top of the page.
Click the Sort Descending arrow in the Time (ms) column to sort the page requests by response times.
Page response times that exceed the threshold display "orange" in the table.
Identify the slowest pages and make a note of the portal in which the page displays.
For more detailed metrics, including how frequently the slowest pages are requested, do one of the following:
For WebCenter Portal - From the WebCenter Portal menu, select Monitoring > Overall Page Metrics.
Note: Requests for pages in the Home portal are excluded from the "Overall Page Metrics" page.
For Portal Framework applications - From the Application Deployment menu, select WebCenter Portal > Overall Page Metrics.
See also, Section 27.1.5, "Understanding Page Request Metrics" and Section 27.3, "Customizing Key Performance Metric Thresholds and Collection."
Use WebCenter Portal's page performance analyzer to quickly see how long individual components take to display on a portal page, as well as the overall time taken to display a page. When enabled, this tool dynamically measures and presents the performance of individual page components whenever you display a portal page.
The portal page performance analyzer is useful to developers who are performing first level performance analysis, customers who build their own pages, and any user who customizes pages in WebCenter Portal.
This section includes the following subsections:
Displaying and Hiding Page Timing Information for Your Current Session
Using the Page Performance Analyzer to Troubleshoot Performance Issues
The portal page analyzer offers a simple way to diagnose slow pages and requires minimal set up or configuration. When this feature is on, the time spent on "high level" page components is calculated and displayed so you can see at a glance which components are slowing down your page. The overall time spent on the page also displays at the top left of the page (see Figure G-8).
Figure G-8 Portal Page Displays Timing Information
In WebCenter Portal, "high level" page components are wrapped in a ShowDetailFrame so they can be moved, hidden or shown on the page, and edited by Oracle Composer and it is the overall timing for each ShowDetailFrame that displays.
The overall page time is the sum of the individual page component timings, plus some additional processing time for page-level operations such as session replication, save and restore page state, page level security checks, and so on.
Performance timings display in various colors to help alert you to problem areas. Refer to the table below:
Color | Time to Display |
---|---|
Green |
< 100 ms |
Green/Yellow |
100 - 500 ms |
Yellow |
500 ms - 1 second |
Orange |
1 - 3 seconds |
Red |
> 3 seconds |
The portal page performance analyzer is disabled out-of-the-box. To make use of this feature, an administrator must specifically enable its use; while the impact on page performance to run this tool is minimal some additional page processing is required.
In a production environment, Oracle recommends that the analyzer is generally disabled to avoid the additional performance data collection and processing and then dynamically enabled when someone reports performance issues for a particular page.
If you do not want end users to see performance data in a production environment, this is another reason to disable the analyzer most of the time.
To enable or disable the portal page analyzer for a WebCenter Portal instance:
Use the WLST command exportMetadata
to export the base webcenter-config.xml
file from MDS.
For example:
exportMetadata(application='webcenter', server='WC_Spaces', toLocation='/tmp/mydata', docs='/oracle/webcenter/webcenterapp/metadata/webcenter-config.xml')
Open webcenter-config.xml
exported from MDS in a text editor and set the perfdebug-enabled
attribute to true
to enable or false
to disable this feature.
For example:
<webcenter:perfdebug-enabled>true</webcenter:perfdebug-enabled>
Save and close webcenter-config.xml
.
Import the updated webcenter-config.xml
file to MDS.
For example:
importMetadata(application='webcenter', server='WC_Spaces', fromLocation='/tmp/mydata', docs='/oracle/webcenter/webcenterapp/metadata/webcenter-config.xml')
There is no need to restart WebCenter Portal to effect this change.
Page performance information does not automatically display after you enable this feature. Anyone who wants to see timing information on portal pages must specifically request that the information displays. For details, see Section G.4.4.3, "Displaying and Hiding Page Timing Information for Your Current Session."
When an administrator enables the page performance analyzer in an WebCenter Portal instance, anyone with access to that WebCenter Portal instance can elect to display or hide page timing information, for their current user session, by appending the perfDebug
parameter to the page URL as follows:
To... | Add a perfDebug parameter to the page URL |
---|---|
Display timing information on portal pages |
|
Stop displaying page performance information |
|
To display timing information on portal pages:
Verify that your administrator enabled the page performance analyzer in your WebCenter Portal instance.
See also Section G.4.4.2, "Enabling and Disabling Portal Page Performance Analysis."
Log in to WebCenter Portal and navigate to the portal page that you want to investigate.
You do not need to log in if the page is a public page.
Add &perfDebug=on
to the end of the page URL (Figure G-9).
For example:
http://mycompany.com/webcenter/portal/MySalesPortal?_adf.ctrl-state=mji8314i6_4&_afrLoop=474789135539036&perfDebug=on
Click "Go" or press Enter to redisplay the page with timing information (as shown in Figure G-8).
All subsequent pages that you display show timing information as well.
Figure G-9 Appending the perfDebug Parameter to Page URLs
To stop displaying page timing information:
In your browser, add &perfDebug=on
to the end of any page URL (Figure G-9).
For example:
http://mycompany.com/webcenter/portal/MySalesPortal?_adf.ctrl-state=mji8314i6_4&_afrLoop=474789135539036&perfDebug=off
Click "Go" or press Enter to display the page again without timing information.
The steps in this section describe how to troubleshoot slow pages using WebCenter Portal tools:
If a user reports performance issues with a particular page, navigate to the slow pages and confirm that the slow performance consistently reproduces.
Alternatively, use Fusion Middleware Control to proactively identify the slowest pages in your application. See Section G.4.3, "How to Identify Slow Pages."
Append &perfDebug=on
to the page URL to display timing information for the page.
See also, Section G.4.4.3, "Displaying and Hiding Page Timing Information for Your Current Session."
Note:
If page timing information does not display, ask your administrator to enable the page performance analyzer. For details, see Section G.4.4.2, "Enabling and Disabling Portal Page Performance Analysis."
Identify the slowest page components, and troubleshoot the issue further:
For example, if the slow component contains:
Document, wiki, or content presenter, check the performance of the back-end Content Server and the database that Content Server is using.
Activity stream, use AWR reports to check database performance and to see if you can tune the database table used by activity stream.
Collaboration features, check the performance of the associated back-end server. For example, for announcements or discussions, monitor the performance of the discussions server.
Portlets, use Fusion Middleware Control to monitor portlet request timing information, errors, portlet producer performance, and so on
If necessary, add the slow page component to a separate "blank" page and then do further profiling.
For example, use JRockit flight recording to pinpoint the bottleneck.
The page performance analyzer works best with portal pages that you create with WebCenter Portal 11.1.1.8.0 and later. In previously releases, some page templates did not include the necessary tags to display component and total page timings and some task flows were not configured with activation=deferred
.
Use the information in this section to diagnose issues relating to poor page performance:
To troubleshoot slow page requests that are still running, extract and view a JRockit Flight Recorder (JFR) recording against the server on which the user session is running. See also, Section G.4.6, "How to Troubleshooting Requests using JRockit Flight Recordings."
Alternatively, take several evenly spaced thread dumps, for example, every 1 or 2 seconds as described in Section G.4.2.8, "Generating Thread Dumps to Diagnose Extremely Slow Page Performance, High Thread Counts, and System Hangs."
If you compare the thread dumps, you might see threads that spent a long time on certain method calls as the call stacks are the same in several consecutive thread dumps. For example, you might see a method call to a database, Oracle WebCenter Content Server, collaboration server, portlet producer, LDAP server, and so on, in which case you can investigate the associated backend server to diagnose the issue further.
Stuck threads can occur for several reasons:
Server is nearly out of memory. If the server is close to out of memory, all requests slow down. To resolve out-of-memory issues, see Section G.4.5.3, "Troubleshooting Slow Requests Using JFR Recordings."
Deadlock threads. Take thread dumps and search for deadlock threads. This normally exposes an issue with the product code.
Extremely slow page requests. Take several evenly spaced thread dumps and find out which method is taking a long time to execute.
If a request is taking longer than 10 minutes, the stuck thread is reported to Oracle WebLogic Server server_name
.out
in the following directories:
(UNIX) DOMAIN_HOME/servers/server_name/logs (Windows) DOMAIN_HOME\servers\server_name\logs
For example:
<Mar 4, 2012 7:44:08 AM PST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "600" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@18986012[ GET /server_name/faces/PimDashboardUiShellPage?_afrLoop=1398820150000&_afrWindowMod e=0&_adf.ctrl-state=a44e7uxcc_13 HTTP/1.1 Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/x-ms-application, application/x-ms-xbap, application/vnd.ms-xpsdocument, application/xaml+xml, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */* Accept-Language: fr UA-CPU: x86 ... ]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds . Stack trace: Thread-164 "[STUCK] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> { jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???) jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24) java.net.SocketInputStream.socketRead0(SocketInputStream.java:???) java.net.SocketInputStream.read(SocketInputStream.java:107) ...
If the stack shows the thread is waiting for a response from another server, check the status of the other server and see it has performance problems before proceeding with the steps below.
To determine what the stuck thread was doing prior to becoming stuck, perform the following steps:
Look at the next few log messages in server_name
.out
for a message indicating an incident has been created. For example:
<Mar 4, 2012 7:44:10 AM PST> <Alert> <Diagnostics> <BEA-320016>
<Creating diagnostic image in DOMAIN_HOME/servers
/server_name/adr/diag/ofm/MyDomain/
server_name_1/incident/incdir_394 with a lockout minute
period of 1.>
The above message may not always appear after each stuck thread reported. It is printed at most four times an hour. If the message does not appear, manually look for the incident
directory by checking the readme
file in the subdirectories under the following directories:
(UNIX) DOMAIN_HOME/servers/server_name/adr/diag/ofm/domain_name/server_name/incident (Windows) DOMAIN_HOME\servers\server_name\adr\diag\ofm\domain_name\server_name\incident
The incident directory contains a WLDF diagnostic image which contains the JFR recording, and a file containing the thread dump.
For more information about diagnosing incidents, see the "Diagnosing Problems" chapter in the Oracle Fusion Middleware Administrator's Guide.
Review the thread dump to find the call stack of the thread. If the thread is blocked waiting for a lock, check what the thread holding the lock is doing.
If the call stack shows that JDBC calls are taking a long time, generate an AWR report on the database to find the query and which table to look and tune.
See Section G.4.2.5, "Generating Automatic Workload Repository (AWR) Reports for the Database."
Review the JRockit flight recording file JRockitFlightRecorder.jfr
for more details. You will also need the ECID of the request which is recorded in the readme.txt
file of the incident directory, and also the Oracle WebLogic Server log.
See also, Section G.4.6, "How to Troubleshooting Requests using JRockit Flight Recordings."
The ECID of the request that caused the stuck thread is recorded in the error message.
See Section G.4.6, "How to Troubleshooting Requests using JRockit Flight Recordings."
If WebCenter Portal or Portal Framework application performance degrades over time, heap usage and garbage collection activity is increasing, and you see OutOfMemoryErrors
, there could be memory leaks in the application causing the amount of free memory in the JVM to continuously decrease.
Figure G-10 shows a typical memory leak trend displayed in JConsole.
Figure G-10 Typical Memory Leak Trend in JConsole
To solve this problem:
Determine the cause of OutOfMemoryErrors
errors:
Review the server_name.out
file for OutOfMemoryErrors
errors.
The server_name.out
file is located at:
(UNIX) DOMAIN_HOME/servers/server_name/logs (Windows) DOMAIN_HOME\servers\server_name\logs
Take a memory dump when OutOfMemoryErrors
errors occur.
For example:
On Sun HotSpot: jmap -dump:live,format=b,file=<path>/heap.hprof <pid>
On JRockit: jrcmd <pid> hprofdump filename=<path>/heap.hprof
You can configure JRockit to automatically generate a heap dump in HPROF binary format (.hprof
file) each time an OutOfMemoryErrors
occurs, by setting the JRockit JVM option, -XX:+HeapDumpOnOutOfMemoryError
. For details, refer to the Oracle JRockit Command-Line Reference.
Restart the managed server.
If the problem persists, proceed to Step 3.
Open the heap.hprof
file with a heap-dump analysis tool that can handle binary HPROF
format, such as Eclipse Memory Analyzer.
Determine which objects and classes are retaining the most memory.
If necessary, take several heap dumps to determine which objects or classes are consuming and increasing the amount of memory.
Take at least two memory dumps:
Take the first dump when the system is warmed up and stabilized.
Take the second dump, when the system is about to run out of memory, that is, full garbage collection gets less than 300MB from the maximum heap size.
Instructions on how to take a heap dump using Sun HotSpot (jmap
) or JRockit (jcrcmd
) is described in step 1.
See the "Running Diagnostic Commands" chapter in the Oracle JRockit JDK Tools Guide. Many heap dump analysis tools, such as Eclipse Memory Analyzer, enable you to compare two heap dumps to identify memory growth areas.
Heap dumps provide information on why memory is retained (Retained Heap). Sometimes it is necessary to know how memory is allocated to further resolve the issue. For these cases, proceed to Step 6.
Use the JRockit Memory Leak Detector tool that is part of JRockit Mission Control Client to understand how memory is allocated.
For more information, see the JRockit Mission Control online help.
If slow page performance is due to content/document-related components, for example, Documents service task flows, Content Presenter task flows, wikis or blogs, Oracle recommends that you review performance metrics for the backend Oracle WebCenter Content Server (System Audit Information page). For details, see the "Viewing System Audit Information" section in Oracle Fusion Middleware Administering Oracle WebCenter Content.
Ensure that the systemdatabase tracing option is selected so you can see performance information for each query that is sent to the database. For details, see the "Server-Wide Tracing" section in Oracle Fusion Middleware Administering Oracle WebCenter Content.
JRockit Flight Recorder (JFR) files contain a record of various events that consume time. If requests are slow, you can analyze the JRockit Flight Recorder (JFR) file to find out why request are taking time.
To create a JFR file:
Extract a JFR file from theOracle WebLogic Server server by running the following command:
UNIX) JROCKIT_HOME/bin/jrcmd jrockit_pid dump_flightrecording recording=1 copy_to_file=path compress_copy=true (Windows) JROCKIT_HOME\bin\jrcmd.exe jrockit_pid dump_flightrecording recording=1 copy_to_file=path compress_copy=true
For more information about the jrcmd
command-line tool, see the "Running Diagnostic Commands" section in Oracle JRockit JDK Tools Guide.
To view the file, start the JRockit Mission Control Client from the following directories:
(UNIX) JAVA_HOME/bin/bin/jrmc (Windows) JAVA_HOME\bin\jrmc.exe
Select File > Open File to select the JFR file.
Locate the slowest requests or investigate a specific request:
To locate the slowest requests: | To investigate a specific request: |
---|---|
|
|
Once you have identified the start time and the thread that serviced the request, navigate to the Logs tab, and drag the time selector at the top of the screen to include only the time window for the duration of the request.
In the Event Log section, perform the following search:
Deselect Show Only Operative Set.
Enter the thread name in the search box.
From the Filter Column list, select Thread.
Select <Enter>.
In the Event Type navigation pane on the left, click the events of interest. Typically, these events are located under nodes Dynamic Monitoring System, Java Application, and WebLogic > JDBC.
The selected events appear in the table in the Event Log section.
Click the Start Time column to sort the time when these events occur, or click the Duration column to view the events that took longest.
The JDBC Statement Execute events corresponds to SQL execution. If there are slow SQL statements, the event details give the SQL text. These events do not have callstacks.
To check the call stacks for slow SQL statements, view the Socket Read event that happens immediately after the JDBC Statement Execute event.
This event corresponds to Oracle WebLogic Server waiting for the SQL results to return, and it has callstack in the event details.
Review the call stacks for long Java Blocked and Java Wait events to see if you can identify what is causing slow performance.
See the "Analyzing Flight Recorder Data in JRockit Mission Control" section in Oracle Fusion Middleware Configuring and Using the Diagnostics Framework for Oracle WebLogic Server.
If you need more detail than the information captured in the default recording, and you can reproduce the slow requests, you can start an explicit recording.
See the "Starting an Explicit Recording" section in Oracle JRockit Flight Recorder Run Time Guide.
You can use My Oracle Support (formerly MetaLink) to help resolve Oracle WebCenter Portal problems. My Oracle Support contains several useful troubleshooting resources, such as:
Knowledge base articles
Community forums and discussions
Patches and upgrades
Certification information
Note:
You can also use My Oracle Support to log a service request.
You can access My Oracle Support at https://support.oracle.com
.
If you experience issues with WebCenter Portal workflows, review the following sections:
Section G.6.1, "Validating the WebCenter Portal Workflow Configuration"
Section G.6.2, "Troubleshooting Issues with WebCenter Portal Workflows"
The Oracle Fusion Middleware Installation Guide for Oracle WebCenter Portal describes how to install and configure WebCenter Portal workflows. For details, see the "Back-End Requirements for WebCenter Portal Workflows" section. You can validate the workflow configuration as follows:
Log in to WebCenter Portal.
Create a portal and then navigate to the Members tab (click the Administration link, then Security, then Members).
Invite a new member with any role (say User2
).
Log out, and then log in as User2
.
Navigate to a Worklist task flow.
Open the invite notification and click the Accept button.
Open the portal.
If the WebCenter Portal workflows are working properly, the newly created portal ia available to User2
. If the portal is not available or listed, there is some issue with the configuration.
If WebCenter Portal workflows are not working properly, follow these steps to help troubleshoot the issue:
Check that WebCenter Portal workflows are deployed on the Oracle SOA server:
Log in to Fusion Middleware Control.
Check that WebCenterWorklistDetailApp.ear
is deployed.
Verify that sca_CommunityWorkflows.jar
is deployed.
For details, see the "Oracle SOA Server - Extending the Domain" section in Oracle Fusion Middleware Installation Guide for Oracle WebCenter Portal.
Ensure the Web Service connection between the Oracle SOA server and the WebCenter Portal application is secure:
Check the alias in the keystore file on the Oracle SOA server.
For example, use the following command to list the content of the keystore file on the Oracle SOA server:
keytool -list -v -keystore bpel.jks -storepass <password>
There should be an entry with:
Alias name: webcenter_portals_ws
See the "Setting Up the SOA Domain" section in Oracle Fusion Middleware Installation Guide for Oracle WebCenter Portal.
Verify that the credential stores for both WebCenter Portal and Oracle SOA server are configured correctly.
Check that keystores exist at both ends of the connection, for example:
- webcenter.jks
(copied to WebCenter Portal server end)
- bpel.jks
(copied to Oracle SOA server end)
For example, the following commands generate webcenter.jks
and bpel.jks
:
keytool -genkeypair -keyalg RSA -dname "cn=webcenter,dc=us,dc=oracle,dc=com" -alias webcenter -keypass mypassword -keystore webcenter.jks -storepass mypassword -validity 360
keytool -exportcert -v -alias webcenter -keystore webcenter.jks -storepass mypassword -rfc -file webcenter.cer
keytool -importcert -alias webcenter_spaces_ws -file webcenter.cer -keystore bpel.jks -storepass mypassword
keytool -genkeypair -keyalg RSA -dname "cn=bpel,dc=us,dc=oracle,dc=com" -alias bpel -keypass mypassword -keystore bpel.jks -storepass mypassword -validity 360
keytool -exportcert -v -alias bpel -keystore bpel.jks -storepass mypassword -rfc -file bpel.cer
keytool -importcert -alias bpel -file bpel.cer -keystore webcenter.jks -storepass mypassword
See the "Creating the SOA Domain Keystore" section in Oracle Fusion Middleware Administering Oracle WebCenter Portal.
Configure role members for the BPMWorkflowAdmin
application role in Oracle SOA server (soa-infra
).
When associating the domain with an identity store that does not contain the user weblogic
, you must assign some other valid user to the application role BPMWorkflowAdmin
. Use WLST commands to do this from the SOA Oracle home, for example, to assign a user named "monty" that exists in LDAP:
cd $SOA_ORACLE_HOME/common/bin/ wlst.sh connect('<admin username>','<admin password>', 'mysoahost.example.com:7001') revokeAppRole(appStripe="soa-infra", appRoleName="BPMWorkflowAdmin", principalClass="oracle.security.jps.service.policystore.ApplicationRole", principalName="SOAAdmin") grantAppRole(appStripe="soa-infra", appRoleName="BPMWorkflowAdmin", principalClass="weblogic.security.principal.WLSUserImpl", principalName="monty")
See the "Overview of WLST Security Commands" section in Oracle Fusion Middleware WebLogic Scripting Tool Command Reference.
This section contains the following subsections:
The ResourceLimitException
error displays when you try to export all portals or the entire WebCenter Portal instance:
Weblogic.common.resourcepool.ResourceLimitException
Increase the maximum capacity in the JDBC connection pool. To reconfigure the connection pool, log in to the WLS Administration Console. From Services, select Data Sources, WebCenterDS, and then the Connection Pool tab.
A LockRefreshTask
warning displays similar to that below when you try to import or export an entire WebCenter Portal instance or a portal:
[WARNING] [][oracle.webcenter.lifecycle.operation.LockRefreshTask]
If you try the import or export operation again, an error similar to that shown here displays:
Starting WebCenter Portal application import...
WebCenter Portal application import started.
Error occurred while performing import
None
Check the WebCenter Portal log files for additional details.
Unable to contact MBeanServer for
oracle.webcenter.lifecycle:ApplicationName=webcenter,Location=WC_Spaces,name=Lifec
ycleManager,type=LifecycleManager,Application=webcenter,ApplicationVersion=11.1.1.
4.0
Error occurred while destroying MBean
The lock hasnt been released from the previous failed import.
Use the deleteMetadata
WLST command to delete unwanted locks in MDS that may be created and not destroyed due to an unexpected and unusual import or export operation failure. Depending on the operation that failed, run one of the following commands:
For WebCenter Portal application import failure, run:
deleteMetadata(application='webcenter', server='WC_Spaces', docs='/oracle/webcenter/lock/applicationImport/applicationImport.xml')
For WebCenter Portal application export failure, run:
deleteMetadata(application='webcenter', server='WC_Spaces', docs='/oracle/webcenter/lock/applicationExport/applicationExport.xml')
For portal import failure, run:
deleteMetadata(application='webcenter', server='WC_Spaces', docs='/oracle/webcenter/lock/scopeImport/**')
For portal export failure, run:
deleteMetadata(application='webcenter', server='WC_Spaces', docs='/oracle/webcenter/lock/gsExportImport/**')
When you first log in to WebCenter Portal after the import operation, the portals and portal templates that you migrated are not available as expected. This can sometimes occur if the portal or portal template cache fails to refresh properly.
Refresh the portal or portal template cache manually using the refreshGroupSpaceCache
and refreshSpaceTemplateCache
WLST commands.
To completely clear the cache (all portals):
refreshGroupSpaceCache(appName='webcenter', spaceNames='', syncMode=1,updateType='all', cleanCache=1)
To completely clear the cache (all portal templates):
refreshSpaceTemplateCache(appName='webcenter', spaceTemplateNames='',syncMode=1, updateType='all', cleanCache=1)
For detailed command syntax and examples, see the "refreshGroupSpaceCache" and "refreshSpaceTemplateCache" sections in Oracle Fusion Middleware WebLogic Scripting Tool Command Reference.
For information on how to run WLST commands, see Section 1.13.3.1, "Running Oracle WebLogic Scripting Tool (WLST) Commands."
You cannot migrate portals or portal templates between two different WebCenter Portal instances that share the same Content Server.
This section contains the following subsections:
Section G.8.1, "Portal Blocked After Unsuccessful Export or Import"
Section G.8.2, "Page or Portal Not Found Message After Import"
Section G.8.3, "Portal Import Archive Exceeds Maximum Upload File Size"
Section G.8.4, "Maximum Number of Portals Exceeded on Export"
Section G.8.6, "Exporting and Importing Portals with Tools and Services Configured"
Section G.8.10, "Exporting and Importing Portals in Multibyte Languages"
If an error occurs during a portal export/import operation, some portals may appear blocked. To unblock a portal, bring the portal back online temporarily, and then take the portal offline again to complete the export/import operation. Switching between the online and offline modes will unblock the portal. For more information, see Section 46.11, "Taking Any Portal Offline" and Section 46.12, "Bringing Any Portal Back Online." See also, the WLST command "setSpaceState
" in Oracle Fusion Middleware WebLogic Scripting Tool Command Reference.
When users first log in to WebCenter Portal after an import operation, they may see a "Page not found" or "Portal not found" message if the page or portal they last visited no longer exists. Last accessed page information is retained during import operations which is why these messages display sometimes.
There is a file size limitation uploading content to WebCenter Portal. If your export archive exceeds the maximum upload size, the import operation through WebCenter Portal Administration will fail.
Import the portal archive using WLST. For details, see Section 40.1.2.4.3, "Importing a Portal from an Archive Using WLST."
Alternatively, modify the maximum upload size in webcenter-config.xml
. The default maximum upload size is 2 GB. See Section 9.12, "Changing the Maximum File Upload Size."
The maximum number of portals that you can export must be less than or equal to 80% of the connection pool size specified for the MDS Data Source. If you try to export too many portals you might see a ResourceLimitException
error:
Weblogic.common.resourcepool.ResourceLimitException
Export fewer portals. Alternatively, modify the connection pool setting. For details, see the Oracle Fusion Middleware Performance and Tuning Guide.
Lists are not importing properly due to list definition differences in the source and target systems.
Consider exporting and importing list data. This ensures that list data is consistent with the list definitions being imported.
If you choose to import without data, the list data in the target system is migrated to be consistent with the imported list definitions. If a list column data type is changed, the column values are converted from the target data type to the imported data type, if possible, otherwise the value is deleted. If a list column is removed during import, the column values are deleted.
The following error message displays when you try to export a portal with tools and services configured, and try to import the same portal from an instance where some or all of those tools or services are not configured.
The following services are not configured: <list of tools and services>. Please configure these services and try again.
You can work around this problem by either adding the tools and services to the target, or removing the service-related info from the data.xml
file of the archive as described below.
To remove service-related info:
Extract the archive.
The archive contains two files: policy-store.xml
and transport.mar
.
Expand the transport.mar
into a directory.
The data.xml file is located in the oracle\webcenter\lifecycle\importexport
directory.
Remove the service tags for all the tools and services that are not present in the target as listed in the error message.
For the example error message above, we would need to remove the following:
<service id="oracle.webcenter.collab.forum" version="11.1.1.0"> <metadataUsages> <metadataUsage includeBaseDocuments="YES" includeSystemCustomizations="YES"> <paths> <include path="/oracle/webcenter/collab/forum/scopedMD/s516227ec_dde1_4991_9e18_28d487cb3bce/**"/> </paths> </metadataUsage> </metadataUsages> </service> <service id="oracle.webcenter.collab.rtc" version="11.1.1.0"/>
Repack the transport.mar
file by zipping the top-level directories Oracle
and pagedefs
into a file named transport.mar
.
Repack the export
archive by zipping the newly created transport.mar and the policy-store.xml
file into an archive.
Import the new archive.
When you navigate directly to the Tools and Services tab in portal administration after importing a portal, all the tools and services are disabled even though they were enabled in the source portal.
This can sometimes happen when the archive you import contains a portal that was exported from a release earlier than WebCenter Portal 11.1.1.8.
Select the Enable check box for tools and services, as required.
Alternatively, open the portal after you import instead of navigating to portal administration. When you access the portal for the first time, tools and services enable automatically.
When you import a portal from the Portals page, the imported portal does not automatically become a subportal of the current portal. The newly imported portal displays in the Portals switcher menu, Portals Browser task flow, or the Portals page, which display all the portals that are available to you.
You can import a portal as a subportal by selecting the parent portal on the Portals page before you import the archive.
You cannot export/import portals or portal templates between two different WebCenter Portal applications that share the same Content Server.
Similarly, you cannot use the Document Migration Utility to migrate portal documents between two different WebCenter Portal applications that share the same Content Server.
On Linux, individual portal export or import fails for one or more portals created in multibyte languages due to naming restrictions. Portal names are restricted to alphanumeric and portal characters ("a" through "z", "A" through "Z", "0" through "9", and the single-byte portal character, which WebCenter Portal replaces with "_"(underscore) ). If any other characters are used in the portal name, export or import fails.
Enforce the naming restriction on the server on which Oracle WebCenter Portal is deployed. To do this, set the environment variable LC_ALL
set to utf-8
.