G Troubleshooting Oracle WebCenter Portal

This appendix presents troubleshooting information for Oracle WebCenter Portal.

This appendix contains the following topics:

• Section G.1, "Troubleshooting Roadmap"

• Section G.2, "Troubleshooting Oracle WebCenter Portal Configuration Issues"

• Section G.3, "Troubleshooting Oracle WebCenter Portal WLST Command Issues"

• Section G.4, "Troubleshooting Oracle WebCenter Portal Performance Issues"

• Section G.5, "Using My Oracle Support for Additional Troubleshooting Information"

• Section G.6, "Troubleshooting WebCenter Portal Workflows"

• Section G.7, "Troubleshooting WebCenter Portal Import and Export"

• Section G.8, "Troubleshooting Individual Portal and Portal Template Import and Export"

G.1 Troubleshooting Roadmap

Use this documentation roadmap to find troubleshooting information for Oracle WebCenter Portal. In Figure G-1, find the starting point that best describes your issue, then click the graphic or use the links in Table G-1 to jump to the section that you need.

Figure G-1 Troubleshooting Oracle WebCenter Portal Roadmap

Description of ''Figure G-1 Troubleshooting Oracle WebCenter Portal Roadmap''

Table G-1 Starting Points for Troubleshooting WebCenter Portal

What do you want to do?	Link to Troubleshooting Section in the Guide
Troubleshoot Oracle WebCenter Portal configuration issues?	Section G.2, "Troubleshooting Oracle WebCenter Portal Configuration Issues"
Troubleshoot Oracle WebCenter Portal WLST issues?	Section G.3, "Troubleshooting Oracle WebCenter Portal WLST Command Issues"
Monitor Oracle WebCenter Portal logs and metrics?	Section 27, "Monitoring Oracle WebCenter Portal Performance" Section 28, "Managing Oracle WebCenter Portal Logs"
Troubleshoot Oracle WebCenter Portal performance issue?	Section G.4, "Troubleshooting Oracle WebCenter Portal Performance Issues"

G.2 Troubleshooting Oracle WebCenter Portal Configuration Issues

This section includes the following subsections:

• Section G.2.1, "How Do I Find Out Which Oracle WebCenter Portal Version Is Installed?"

• Section G.2.2, "WebCenter Portal Menu Does Not Display in Fusion Middleware Control"

• Section G.2.3, "Configuration Options Unavailable"

• Section G.2.4, "Configuration Issues with One or More Tools or Services"

• Section G.2.5, "Configuration for One Application Reflects in Another"

• Section G.2.6, "Logs Indicate Too Many Open Files"

G.2.1 How Do I Find Out Which Oracle WebCenter Portal Version Is Installed?

Always use Oracle's OPatch utility to obtain version information for Oracle WebCenter Portal products and components installed in your environment.

To run OPatch command with the lsinventory option:

1. Set Opatch environment variables WCP_ORACLE_HOME, MW_HOME, and PATH.

   See the "Patching Oracle Fusion Middleware with Oracle OPatch" section in Patching Guide.

2. Run the following OPatch command

   opatch lsinventory -details

   The output lists which components are installed and their version numbers, similar to that shown here:

   ...
   Oracle Upgrade Assistant for Webcenter             11.1.1.9.0
   Oracle WebCenter Portal Suite                      11.1.1.9.0
   Oracle WebCenter Portal Suite 11g                  11.1.1.9.0
   Oracle WebCenter Portal: Activity Graph            11.1.1.9.0
   Oracle WebCenter Portal: Analytics Collector       11.1.1.9.0
   Oracle WebCenter Portal: Discussions Server        11.1.1.9.0
   Oracle Webcenter Portal: Framework                 11.1.1.9.0
   Oracle Webcenter Portal: Framework Core            11.1.1.9.0
   Oracle WebCenter Portal: Pagelet Producers         11.1.1.9.0
   Oracle WebCenter Portal: Personalization           11.1.1.9.0
   Oracle Webcenter Portal: Portlet Server            11.1.1.9.0
   Oracle WebCenter Portal: RCU                       11.1.1.9.0
   Oracle Webcenter Portal: Spaces                    11.1.1.9.0
   Oracle WebCenter Portal: Suite Components          11.1.1.9.0
   Oracle WebCenter Portal: Wiki                      11.1.1.9.0
   ...
   

   Note:

   Oracle WebCenter Portal Suite 11g is a child component of Oracle WebCenter Portal Suite and the versions are always the same.

See the "Patching Oracle Fusion Middleware with Oracle OPatch" section in Patching Guide and the "Lsinventory Command for OUI-based Oracle Homes" section in Oracle Universal Installer and OPatch User's Guide for Windows and UNIX.

Note:

Always use OPatch to obtain the version number. Versions that display alongside Oracle WebCenter Portal product and component names in Fusion Middleware Control do not reflect the true version number of installed products, as shown in Figure G-2 and Figure G-3.

Figure G-2 Incorrect Version Number for WebCenter Portal in Fusion Middleware Control

Description of ''Figure G-2 Incorrect Version Number for WebCenter Portal in Fusion Middleware Control''

Figure G-3 Incorrect Version Numbers for Other Applications in Fusion Middleware Control

Description of ''Figure G-3 Incorrect Version Numbers for Other Applications in Fusion Middleware Control''

G.2.2 WebCenter Portal Menu Does Not Display in Fusion Middleware Control

Problem

After logging into Fusion Middleware Control, you cannot find the WebCenter Portal option in the Application Deployment menu for your application.

Solution

Ensure the following:

• Deployed application is a Portal Framework application, created using the WebCenter Portal Framework Application template in JDeveloper.

  The WebCenter Portal option only displays for applications developed using the WebCenter Portal Framework Application template in JDeveloper.

• Deployed Portal Framework application is up and running.

• Deployed Portal Framework application contains accurate information about the MDS repository and partition, and the MDS repository is accessible to the application. To verify this information, check the metadata-store-usages section in the adf-config.xml file. For information on MDS, see the "Understanding the MDS Repository" section in Administrator's Guide.

• Portal Framework application is packaged with required artifacts to support configuration:

  • adf-jndi-config namespace is configured in the application's adf-config.xml file. This is provisioned at design time. The following is an example (the text in bold) of the adf-jndi-config namespace:

    <adf-config xmlns="http://xmlns.oracle.com/adf/config"
         xmlns:jndiC="http://xmlns.oracle.com/adf/jndi/config"
         xmlns:ns2="http://xmlns.oracle.com/mds/config"
         xmlns:ns3="http://xmlns.oracle.com/adf/mds/config">
      ...
      ... 
    </adf-config>
    

  • Appropriate listeners exist in the web.xml file to register the MBeans. This is provisioned at design time. For example, see the text in bold in the following snippet of the web.xml file:

    <listener>
       <description>ADF Config MBeans</description>
       <display-name>ADF Config MBeans</display-name>
       <listener-class>oracle.adf.mbean.share.config.ADFConfigLifeCycleCallBack</listener-class>
    </listener>
    <listener>
        <description>ADF Connection MBeans</description>
        <display-name>ADF Connection MBeans</display-name>
        <listener-class>oracle.adf.mbean.share.connection.ADFConnectionLifeCycleCallBack</listener-class>
    </listener>
    

• ADFConfig and ADFConnection MBeans are registered for WebCenter Portal or your Portal Framework application. You can verify whether these MBeans are registered through the System MBean Browser:

  1. In Fusion Middleware Control, open the System MBean Browser for your application. Do one of the following:

     For WebCenter Portal, select the menu option WebCenter Portal >System MBean Browser.

     For a Portal Framework application, select the menu option Application Deployment > System MBean Browser.

  2. Locate ADFConnection MBeans for your application under Application Defined MBeans > oracle.adf.mbean.share.connection, as shown in Figure G-4.

  3. Similarly, locate ADFConfig MBeans for your application under Application Defined MBeans > oracle.adf.mbean.share.config, as shown in Figure G-4.

     Figure G-4 Application Defined MBeans

     
     Description of ''Figure G-4 Application Defined MBeans''
     
     

  See also, Section 1.13.4, "System MBean Browser."

• Review the latest configuration in adf-connections.xml and adf-config.xml to check the content is correct. Some typical problems include:

  • Configuration file is not compliant with its XML schema For example, there are duplicate configuration elements when the schema only allows for 1 occurrence.

  • XML namespace is missing for configuration referenced within the file.

  • XML element is not qualified with the XML namespace.

  See also, Appendix A "Exporting Configuration Files with MDS Customizations".

• Check the application's diagnostic logs, analyze messages for the modules oracle.adf.mbean.share.connection and oracle.adf.mbean.share.config, and determine what must be done:

  • For WebCenter Portal, the log file is available in the DOMAIN_HOME/servers/ServerName/logs directory. The log file follows the naming convention of ServerName-diagnostic.log. See also, Section 28.2.1, "Viewing and Configuring WebCenter Portal Logs."

  • For Portal Framework applications, the log file is available in the DOMAIN_HOME/servers/ServerName/logs directory. The log file follows the naming convention ServerName-diagnostic.log. See also, Section 28.2.2, "Viewing and Configuring Portal Framework Application Logs."

G.2.3 Configuration Options Unavailable

Problem

When you try to configure WebCenter Portal or a Portal Framework application through Fusion Middleware Control, the following message displays:

Configuration options currently unavailable. The application application_name might be down, did not start-up properly, or is incorrectly packaged.
Check the log files for further details.

For example, you try to change options available through the Application Settings screen or configure connections through the WebCenter Portal Service Configuration screen in Fusion Middleware Control.

Solution

Check the application's diagnostic logs:

• For WebCenter Portal, the log file is available in the DOMAIN_HOME/servers/ServerName/logs directory. The log file follows the naming convention of ServerName-diagnostic.log. See also, Section 28.2.1, "Viewing and Configuring WebCenter Portal Logs."

• For Portal Framework applications, the log file is available in the DOMAIN_HOME/servers/ServerName/logs directory. The log file follows the naming convention ServerName-diagnostic.log. See also, Section 28.2.2, "Viewing and Configuring Portal Framework Application Logs."

Analyze messages for the modules oracle.adf.mbean.share.connection and oracle.adf.mbean.share.config, and determine what must be done.

See also, Section G.2.2, "WebCenter Portal Menu Does Not Display in Fusion Middleware Control."

G.2.4 Configuration Issues with One or More Tools or Services

Do not attempt to configure services that your application does not support. WebCenter Portal configuration through Enterprise Manager and WLST fails if you try to configure a service, say discussions, that your application was not designed to use.

The out-of-the-box application, WebCenter Portal, is designed to support all tools and services but Portal Framework applications that you build using JDeveloper only provide artifacts in the application's configuration for services that the developer specifically included during design time. For example, if the developer did not add or configure discussions for the application through JDeveloper, you cannot configure discussions postdeployment through Enterprise Manager and WLST.

If you are having issues configuring or connecting to one or more tool or service, refer to the appropriate troubleshooting section:

• Section 9.13, "Troubleshooting Issues with Content Repositories"

• Section 10.8, "Troubleshooting Issues with Recommendations"

• Section 11.11, "Troubleshooting Issues with Analytics"

• Section 12.12, "Troubleshooting Issues with Announcements and Discussions"

• Section 13.9, "Troubleshooting Issues with Events"

• Section 15.11, "Troubleshooting Issues with Mail"

• Section 19.6, "Troubleshooting Issues with Notifications"

• Section 18.7, "Troubleshooting Issues with Oracle SES"

• Section 20.7, "Troubleshooting Issues with Worklists"

• Section 21.13, "Troubleshooting Portlet Producer Issues"

• Section 21.12.3, "Troubleshooting WebCenter Services Portlets"

• Section 30.3, "Troubleshooting Security Configuration Issues"

• Chapter 11489872, "Troubleshooting WebCenter Portal Import and Export"

• Chapter 11489872, "Troubleshooting Individual Portal and Portal Template Import and Export"

G.2.5 Configuration for One Application Reflects in Another

Problem

You configured WebCenter Portal or a Portal Framework application, but those configurations also show in another application.

For example, you created or edited a mail connection for a Portal Framework application named MyPortalApp1 and you discover that the connection changes are also seen in another application MyPortalApp2

Solution

This happens when multiple applications share the MDS partition in the same schema. To resolve this problem, deploy these applications again and ensure that each application either uses a different MDS schema or a different MDS partition. For information about creating a MDS repository or configuring an existing application to use a different MDS repository or partition, see the "Managing the Metadata Repository" section in Administrator's Guide.

G.2.6 Logs Indicate Too Many Open Files

Problem

WebCenter Portal or a Portal Framework application is inaccessible or displaying error messages and the diagnostic log files indicates that there is an issue with 'too many open files'.

Solution

Do the following:

• Check the number of file handles configured on each of the back-end servers, primarily the database, and increase appropriately.

• If the problem persists after increasing the file handles, check the value of fs.file-max in the /etc/sysctl.conf file and increase the value appropriately.

G.3 Troubleshooting Oracle WebCenter Portal WLST Command Issues

This section includes the following subsections:

• Section G.3.1, "No Oracle WebCenter Portal WLST Commands Work"

• Section G.3.2, "WLST Commands Do Not Work for a Particular Tool or Service"

• Section G.3.3, "Connection Name Specified Already Exists"

• Section G.3.4, "WLST Shell is Not Connected to the WebLogic Server"

• Section G.3.5, "More Than One Application with the Same Name Exists in the Domain"

• Section G.3.6, "More Than One Application with the Same Name Exists on a Managed Server"

• Section G.3.7, "Already in Domain Runtime Tree Message Displays"

See also, Section 1.13.3.1, "Running Oracle WebLogic Scripting Tool (WLST) Commands."

G.3.1 No Oracle WebCenter Portal WLST Commands Work

Problem

You are unable to run any WLST commands.

Solution

Ensure the following:

• Always run Oracle WebCenter Portal WLST commands from your WebCenter Portal Oracle home directory (WCP_ORACLE_HOME/common/bin).

  If you attempt to run Oracle WebCenter Portal WLST commands from the wrong directory you will see a NameError.

• No files other than Python are stored in the WLST source directory: WCP_ORACLE_HOME/common/bin/wlst. This directory must contain files with the .py extension only.

  The default set of files in this location contain legal Python files from Oracle. It is possible that a user copied some non-python script to this directory, for example, a backup file or a test python file with syntax errors.

• webcenter-wlst.jar is located at WCP_ORACLE_HOME/common/bin/wlst/lib.

See also, Section 1.13.3.1, "Running Oracle WebLogic Scripting Tool (WLST) Commands."

G.3.2 WLST Commands Do Not Work for a Particular Tool or Service

Problem

You are unable to run WLST commands for a particular tool or service, and therefore, you cannot configure that tool/service.

Solution

First, run generic non-Oracle WebCenter Portal commands, for example, run listApplications or displayMetricTableNames to verify whether these commands work. If generic commands do not work, then apply the solution described in Section G.3.1, "No Oracle WebCenter Portal WLST Commands Work."

If generic commands work, then run test commands to check Oracle WebCenter Portal-specific commands for syntax errors. Run the appropriate WSLT check command (see Table G-2).

Table G-2 File Names and WLST Commands for Oracle WebCenter Portal Tools and Service

Service Name	File Name	WLST Command
Activity Graph	ActivityGraph.py	metadataAdminCheck()
Activity Stream	ActivityStream.py	asCheck()
Analytics	Analytics.py OpenUsage.py	analyticsCheck() openusageCheck()
Discussions and Announcements	Forum.py JiveAdmin.py	fcpCheck()
Documents	Doclib.py	doclibCheck()
External Applications	ExtApp.py	extCheck()
Portal Events	Community.py	ceCheck()
Instant Messaging and Presence	Imp.py	rtcCheck()
Mail	Mail.py	mailCheck()
Notifications	Notification.py	notificationCheck()
Personal Events	Personal.py	peCheck()
Producers
PDK-Java Producers	Pdk.py	pdkCheck()
WSRP Producers	Wsrp.py	wsrpCheck()
Pagelet Producers	Ensemble.py	ensembleCheck()
Producer Helper	Producer.py	producerHelperCheck()
RSS News Feed	RSS.py	rssCheck()
Search	Ses.py	sesCheck()
Worklist	Bpel.py	bpelCheck()
Export/Import - WebCenter Portal applications	Lifecycle.py	lifecycleCheck()
Export/Import - Portals and Template	ExtImp.py	expimpCheck()
Synchronize Users	SynchronizeUser.py	userRenameCheck()
Rename Users	UserRename.py	userRenameCheck()
WebCenter Portal - General
Service Framework	WcServiceFwk.py	serviceFwkCheck()
General Settings	WebCenterGeneralSettings.py	generalSettingsCheck()
WebCenter Portal and SOA	WebCenterSpacesSOA.py	spaceCheck()

See also, Section 1.13.3.1, "Running Oracle WebLogic Scripting Tool (WLST) Commands."

For more information about Oracle WebCenter Portal's WLST commands, see the "WebCenter Portal Custom WLST Commands" chapter in WebLogic Scripting Tool Command Reference.

G.3.3 Connection Name Specified Already Exists

Problem

You are unable to create a connection with the name Connection_Name. The following message displays:

A connection with name Connection_Name already exists.

For example, you try to create an external application connection using the WLST command createExtAppConnection or connect to a mail server using createMailConnection.

Solution

Connection names must be unique (across all connection types) within WebCenter Portal or a Portal Framework application. This error occurs when you try to create a connection with a name that is in use. Ensure that you use a unique name for your connection.

G.3.4 WLST Shell is Not Connected to the WebLogic Server

Problem

You must connect to the Administration Server for Oracle WebCenter Portal before running WLST commands. Oracle WebCenter Portal WLST commands do not work without a connection.

Solution

Run the following command to connect the WLST shell to the managed server:

connect(username, password , serverhost:serverport)

See also, Section G.3.1, "No Oracle WebCenter Portal WLST Commands Work" and Section 1.13.3.1, "Running Oracle WebLogic Scripting Tool (WLST) Commands."

G.3.5 More Than One Application with the Same Name Exists in the Domain

Problem

You attempt to perform an operation on WebCenter Portal or a Portal Framework application, such as create a connection for a service or register a portlet producer, and the following message displays:

Another application named "YourApplicationName" exists. Specify the Server on which your application is deployed. Use: server="YourServerName".

This message displays if there are multiple applications with the same name in the domain. This usually happens in a cluster environment, where the same application is deployed to multiple managed servers.

For example, you tried to register a portlet producer for an application named "MyApp" using the following WLST command:

registerWSRPProducer(appName='myApp', name='MyWSRPSamples', url='http://myhost.com:9999/ portletapp/portlets/wsrp2?WSDL')

Solution

Specify on which managed server you want to run the WLST command, that is, include the server argument. For example:

registerWSRPProducer(appName='myApp', name='MyWSRPSamples', url='http://myhost.com:9999/portletapp/portlets/wsrp2?WSDL', server=WC_CustomPortal2)

See also, Section 1.13.3.1, "Running Oracle WebLogic Scripting Tool (WLST) Commands."

G.3.6 More Than One Application with the Same Name Exists on a Managed Server

Problem

You attempt to perform an operation on WebCenter Portal or a Portal Framework application such as create a connection for a service or register a portlet producer, and the following message displays:

Another application named "application_name" exists on the server managedServerName.

This message indicates that there are multiple applications with the same name on specified managed server. This usually happens when applications are assigned different versions.

For example, you tried to register a portlet producer for an application named "MyApp" using the following WLST command:

registerWSRPProducer(appName='myApp', name='MyWSRPSamples', url='http://myhost.com:9999/portletapp/portlets/wsrp2?WSDL')

Solution

Specify on which application version you want to run the WLST command, that is, include the server and applicationVersion arguments. For example:

registerWSRPProducer(appName='myApp', name='MyWSRPSamples', url='http://myhost.com:9999/portletapp/portlets/wsrp2?WSDL', server=WC_CustomPortal1, applicationVersion=2)

See also, Section 1.13.3.1, "Running Oracle WebLogic Scripting Tool (WLST) Commands."

G.3.7 Already in Domain Runtime Tree Message Displays

Problem

While running a WLST command, the following message displays:

Already in Domain Runtime Tree

Solution

None required. This is for information only.

G.4 Troubleshooting Oracle WebCenter Portal Performance Issues

Use the information in this section to help diagnose performance-related issues for Oracle WebCenter Portal.

This section contains the following sub sections:

• About Performance Monitoring and Troubleshooting Tools

• How to Troubleshoot Overall System Slowness

• How to Identify Slow Pages

• How to Identify Slow Page Components

• How to Troubleshoot Slow Page Requests

• How to Troubleshooting Requests using JRockit Flight Recordings

G.4.1 About Performance Monitoring and Troubleshooting Tools

Various tools are available for monitoring and troubleshooting performance issues with your Oracle WebCenter Portal environment.

Table G-3 Performance Monitoring and Troubleshooting Tools

Tool	Use to...	See
Enterprise Manager
Fusion Middleware Control	Monitor WebCenter Portal metrics and log files in real-time mode for a single Oracle Fusion Middleware Farm. Check service configuration, including MDS and partitions for WebCenter Portal deployments.	Starting Enterprise Manager Fusion Middleware Control
Grid Control	Monitor WebCenter Portal metrics in real time and from a historical perspective for trend analysis, as well as monitor the underlying host and operating system, databases, and more. Oracle Enterprise Manager 11g Grid Control must be installed separately as it is not a part of the Oracle Fusion Middleware 11g installation. With Grid Control, you can centrally manage multiple Oracle Fusion Middleware Farms and WebLogic Domains.	Oracle Enterprise Manager Grid Control
WebCenter Portal Page Performance Analyzer	Analyze the performance of portal pages in WebCenter Portal. This tool dynamically measures and presents the performance of individual page components when you display pages in WebCenter Portal.	How to Identify Slow Page Components
JConsole	Graphically monitor Java applications and Java virtual machines (JVM).	How to Use JConsole to Monitor JVM
JRockit Mission Control	Capture and present live data about memory, CPU usage, and other runtime metrics.	Troubleshooting Slow Requests Using JFR Recordings
Eclipse Memory Analyzer	Find memory leaks and reduce memory consumption.	Troubleshooting Memory Leaks and Heap Usage Problems
Threadlogic	Analyze thread dumps.	Generating Thread Dumps to Diagnose Extremely Slow Page Performance, High Thread Counts, and System Hangs

G.4.2 How to Troubleshoot Overall System Slowness

Use the actions listed in Table G-4 to troubleshoot overall system slowness:

Table G-4 Troubleshooting Overall System Slowness

Action	Description	More Information
1	Verify key system resources.	Verifying System Resources (CPU and Memory)
2	Use top or vmstat to see if system slowest is caused by CPU, memory, or IO contention issues.	Monitoring System Resource Usage
3	Monitor the performance of your Java virtual machine (JVM).	Monitoring Java Virtual Machine (JVM) Usage
4	Verify OID and database connection pool settings.	Verifying Connection Pool Settings
5	Generate automatic workload repository (AWR) reports for Oracle databases to diagnose database-related issues.	Generating Automatic Workload Repository (AWR) Reports for the Database
6	Use tcpdump to investigate and diagnose network related problems.	Diagnosing Network Related Problems Using tcpdump
7	Use ping to measure network latency.	Measuring Network Latency Using ping
8	Collect thread dumps.	Generating Thread Dumps to Diagnose Extremely Slow Page Performance, High Thread Counts, and System Hangs
9	Look for errors in WC_Spaces-diagnostic.log and check the correct logging level is enabled.	Analyzing the Diagnostics Log
10	Use DMS Spy to monitor internal DMS metric data, such as activity in the Java Object Cache.	Using DMS Spy to Monitor Internal Performance Metric Tables
11	Look at the access.log for the WebLogic Server to check HTTP request/response cache settings. Verify HTTP compression settings.	Verifying HTTP Request Caching Verifying HTTP Compression
12	Use HTTP monitoring tools to analyze requests and response times.	Checking Browser Response Times
13	Warm up your system before taking performance measurements.	Warm up the System Before Re-Testing Performance

G.4.2.1 Verifying System Resources (CPU and Memory)

If you are experiencing performance issues it is important to verify that you have sufficient system hardware resources, that is, adequate CPU and physical memory capacity for your Oracle WebCenter Portal installation.

Low system resources can cause many different problems. System resources are used by individual services. Regularly monitoring and recording system usage can help you determine whether you need to upgrade your system hardware, or if some services need to be moved to another machine.

To verify system CPU and memory:

• On Linux, review CPU and memory information in the following files:

  • /proc/cpuinfo

    The cpuinfo file provides important CPU information including the model, CPU cores, CPU MHz, cache size, and flags which show what instruction sets are available on the processor. Systems with multiple processors or multiple cores have separate entries for each.

  • /proc/meminfo

    The important fields to look for in the meminfo file include MemTotal, MemFree, Cached, and SwapTotal.

• On Windows, access CPU and memory information through the Task Manager (Performance > Resource Monitor).

G.4.2.2 Monitoring System Resource Usage

On Linux, you can use top and vmstat utilities to see if system slowness is caused by CPU, memory, or I/O contention issues. If you discover that your system resources are low, you can:

• Move processes to other machines or shut down unused processes/programs to free up more physical memory and/or CPU cycles.

• Upgrade your system hardware resources

For more information, see:

• How to use top to monitor system resource usage on Linux

• How to use vmstat to monitor system resource usage on Linux

Note:

On Windows, use Task Manager to monitor system resources and shut down unused processes and programs. Refer to your Windows documentation for more information.

G.4.2.2.1 How to use top to monitor system resource usage on Linux

The top utility displays a continually updating report of system resource usage so you can identify the top memory and CPU consumers on your system.

The top portion of the report lists information such as the system time, uptime, CPU usage, physical and swap memory usage, and number of processes. Below that is a list of the processes sorted by CPU utilization.

Note:

Use Shift+M to sort by memory usage. Use Shift+P to sort by CPU usage.

# top
2:10:49  up 8 day,  3:47,  20 users,  load average: 0.34, 0.19, 0.10
75 processes: 20 sleeping, 2 running, 8 zombie, 0 stopped
CPU states:   5.1% user   1.1% system   0.0% nice   0.0% iowait  93.6% idle
Mem:   512216k av,  506176k used,    6540k free,    0k shrd,   21888k buff
Swap: 1044216k av,  161672k used,  882544k free     199388k cached

PID   USER  PRI NI  SIZE  RSS   SHARE STAT %CPU %MEM   TIME    CPU COMMAND
2330  admin  15  0  161M  70M   2132  S    4.9  14.0   1000m   0    oracle
2605  lin    15  0  8240  6340  3804  S    0.3  1.2    1:12    0    oracle
3413 harvey  15  0  6668  5324  3216  R    0.3  1.0    0:20    0    oracle

Troubleshooting Tips - top:

• If free memory is < 100Mb and cached memory is < 1GB, system memory is running low.

• If %wa (I/O WAIT) is always more than 10%, the system may be slow because it is blocked by physical I/O.

• Ensure that the idle value is close to 100% and system/user CPU usage is close to 0% when there is no load on the system.

• In the memory view of top, identify the % memory usage (%MEM) for each process (PID). If the memory is tight and swap space usage is high, consider:

  - moving process with high memory usage to another machine

  - increasing memory allocation to the virtual machine

  - adding physical memory.

G.4.2.2.2 How to use vmstat to monitor system resource usage on Linux

The vmstat utility shows running statistics on various parts of the system including system processes, memory, swap, I/O, CPU, and the Virtual Memory Manager. These statistics are generated using data from the last time the command was run to the present. The first time you run the command, data displays from the last reboot until the present.

When you run vmstat you can specify how often you want the statistics to refresh (in seconds) using the format: vmstat <refresh rate in seconds>

For example: vmstat 3

Frequent system resource usage updates, enable you to see trends in CPU/memory usage and how usage trend impact WebCenter Portal or a Portal Framework application performance. If the system is stable, the swap metrics (si and so) should register near zero.

# vmstat 3
   procs           memory              swap        io       system      cpu
r  b    swpd  free   buff    cache   si   so    bi    bo   in    cs us sy id wa
3  0    144 1058184 868324 12343056    0    0   220    83   33    14 10  1 88  1
0  0    144 1058184 868324 12343056    0    0     0   117  478   656  5  1 94  0
0  0    144 1058192 868324 12343056    0    0     0    69  696   860  9  1 90  0
2  0    144 1058264 868324 12343056    0    0     0     0  914  1127  5  2 93  0
0  0    144 1057864 868324 12343316    0    0     0   184  857  1021  7  1 92  0
2  0    144 1057592 868324 12343316    0    0     0   155 1596  1646  5  4 91  0
0  0    144 1057592 868324 12343316    0    0     0     0  737   839  5  2 94  0
0  0    144 1057592 868324 12343316    0    0     0    57  694   743  8  2 91  0
2  0    144 1057592 868324 12343316    0    0     0     0  358   437  2  0 98  0

Troubleshooting Tips - vmstat:

• When the swap metrics (si and so) are frequently non-zero, system memory becomes tight, and the system may go into thrashing mode. Low system memory significantly affects performance, that is, any program running on the system becomes unusually slow. If you frequently experience low system memory, increase the memory on the machine.

• If CPU idle time (id) is constantly less than 10%, the CPU is running at full or over capacity. Under such conditions, any more load on the system results in significant performance degradation.

G.4.2.3 Monitoring Java Virtual Machine (JVM) Usage

Your Java virtual machine (JVM) can significantly affect Oracle WebCenter Portal performance. Oracle recommends that you continually monitor your JVM to track:

1. Memory usage

2. CPU usage

3. Thread activity

You can monitor all three metrics from your application's Recent WebLogic Server Metrics page in Fusion Middleware Control (Figure G-5). For details, see Section 27.1.8, "Understanding WebLogic Server Metrics" and Section 27.2, "Viewing Performance Metrics Using Fusion Middleware Control."

Figure G-5 Recent WebLogic Server Metrics Page - JVM Metrics

Description of ''Figure G-5 Recent WebLogic Server Metrics Page - JVM Metrics''

• Monitor memory trends - JVM frequently allocates and releases memory. To detect memory leaks and high memory usage, you must analyze the memory trend over a long period of time (an hour or several hours).

  When you see the bottom trend line on a memory graph increasing, it indicates a memory leak. The bottom points on a memory graph show how low memory usage can go after full garbage collection. If you take at least two memory dumps, one when the memory is healthy and another when the memory full garbage collection cannot recycle too much, you can compare the memory dumps and identify which components are consuming memory.

  See also, Section G.4.5.4, "Troubleshooting Memory Leaks and Heap Usage Problems."

• Monitor CPU usage - Occasional spikes in CPU usage is normal but if CPU usage remains high (85-90%) over a long period of time, it normally indicates there is an issue with CPU which can impact performance.

  If CPU usage appears overloaded, comparing several thread dumps at fixed intervals (for example, every 5 seconds) can help reveal causes of high CPU usage.

  Alternatively, profiling tools, such as JRockit's Flight Recorder, can provide further insight into CPU usage.

• Monitor thread activity and analyze thread dumps (stuck threads, blocked threads, deadlocks) - The number of threads should stabilize under stable load conditions. Sudden spikes or an increasing number of threads, generally indicates an issue in your system.

  When that is happening, you can take multiple thread dumps to understand the calling stack of stuck/blocked threads or deadlocks. You can use thread analysis tools to identify what is causing the extra threads and what are they doing, as well as detect contentious areas and slow performing areas. See also, Section G.4.5.2, "Troubleshooting Stuck Threads."

  See also, Section G.4.2.8, "Generating Thread Dumps to Diagnose Extremely Slow Page Performance, High Thread Counts, and System Hangs."

  Tip:

  Oracle Fusion Middleware's Diagnostic Framework collects and manages information about common problems, such as stuck threads and deadlocked threads, to help you diagnose and resolve such issues. Alternatively, you can send diagnostic dumps to Oracle Support Services. For more details, see "Diagnosing Problems" in Administrator's Guide.

There are various tools available to monitor JVM performance, including Fusion Middleware Control, JConsole, JVisualVM, and JRockit Mission Control. Use any of these tools to show the current state of the JVM, as well as all the active threads and their states. See also, the "Tuning Java Virtual Machines (JVMs)" section in Performance Tuning Guide

G.4.2.3.1 How to Use JConsole to Monitor JVM

In the first instance, administrators can use the application's Recent WebLogic Server Metrics page in Fusion Middleware Control to monitor JVM (as shown in Figure G-5). If more aggressive real time metrics are required, another option is to use JConsole (shown in Figure G-6).

JConsole is available with all types of JVM, including HotSpot, JRockit, IBM JVM, and so on.

JConsole's executable is located at: JAVA_HOME/bin/jconsole.exe.

Figure G-6 JConsole

Description of ''Figure G-6 JConsole''

G.4.2.4 Verifying Connection Pool Settings

This section describes how to verify connection pool settings for:

• WebCenter Portal Data Sources (JDBC Connection Pool Settings)

• Identity Store (JNDI Connection Pool Settings)

The information in this section might be useful if you are experiencing slow response times and your diagnostics log files contain connection/connection pool messages or a recent thread dump contains calling stacks that are waiting to get connections from connection pool.

G.4.2.4.1 WebCenter Portal Data Sources (JDBC Connection Pool Settings)

Administrators can use Fusion Middleware Control to monitor JDBC connection metrics for WebCenter Portal or a Portal Framework applications. Use the JDBC usage information on the WebLogic Server Metrics Page (Figure G-7) to assess whether JDBC configuration or the connection pool size needs to be adjusted.

See also, Section 27.2, "Viewing Performance Metrics Using Fusion Middleware Control."

Figure G-7 Recent WebLogic Server Metrics Page - JDBC Usage

Description of ''Figure G-7 Recent WebLogic Server Metrics Page - JDBC Usage''

The Recent JDBC Usage chart shows the number of JDBC connections currently open on the managed server. The JDBC session count that is displayed here, is the sum of the Current Active Connection Count metric for each JDBC data source.

If usage is high and the trend is rising, administrators can use WebLogic Server Administration Console to view and configure data source connection pool settings and monitor usage patterns for individual data sources in more detail. If you monitor data source usage over a period of time you can:

• Determine the best connection pool size for a particular database connection.

• Detect and resolve connection pool leakage, that is, if you notice that the number of data source connections do not decrease without load.

Tip:

Select "Customize the Table" when monitoring connection pools, to display additional metrics such as:

• Connection Delay Time

• Current Capacity High Count

• Failures To Reconnect Count

• Wait Seconds High Count

• Waiting For Connection Current Count

• Waiting For Connection Failure Total

For high concurrency systems, you may want to adjust the maximum number of connections in the pool (Maximum Capacity setting). Out-of-the-box maximum values for the various WebCenter Portal data sources are shown here:

WebCenter Portal Data Sources	Connection Pool Default Maximum Capacity
WebCenterDS	50
ActivitiesDS	25
mds-SpacesDS	50
mds-owsmDS	15
mds-PageletProducerDS	50
mds-ServicesProducerDS	50
mds-wcpsDS	50
PersonalizationDS	25
Portlet-ServicesProducerDS	50
PortletDS	50
WC-ServicesProducerDS	25

Note:

Each connection uses memory in the WebLogic Server and consumes processes in the database so do not specify an unnecessarily large connection pool value.

• To monitor a particular data source, log in to the WebLogic Server Administration Console, select Services>Data Sources, click the data source name, and then click the Monitoring tab.

• To modify the connection pool for a particular data source, log in to the WebLogic Server Administration Console, select Services>Data Sources, click the name of the data source, and then click the Connection Pool tab.

See the "Tuning Data Source Connection Pools" section in Configuring and Managing JDBC Data Sources for Oracle WebLogic Server.

G.4.2.4.2 Identity Store (JNDI Connection Pool Settings)

Typically, JNDI connection pooling is always turned on. However, some additional configuration is required if the connection between WebCenter Portal/Portal Framework application and the identity store (this may be Oracle Internet Directory, Active Directory, and so on) uses SSL. By default, when you choose an SSL port, the JNDI connections are not pooled causing increased response time and decreased performance when logging in, looking up users, groups, or other identity store entities. For more information and instructions, see the "Tuning Identity Store Configuration" section in Performance Tuning Guide.

G.4.2.5 Generating Automatic Workload Repository (AWR) Reports for the Database

If database access is slow, for example, activity stream queries, MDS queries or JPA (Java Persistence API) queries are slow for various operations in your WebCenter Portal application, you can analyze Automatic Workload Repository (AWR) reports to diagnose the root cause of performance-related problems in your Oracle database.

Before generating the AWR report, first check the general health (CPU/Memory) of that machine hosting your database (see Section G.4.2.2, "Monitoring System Resource Usage"). If system resource limitations are not causing poor performance, examine the database performance information and statistics in the AWR report.

For more detail, see the "Tuning Database Parameters" section in Performance Tuning Guide and the "Generating Automatic Workload Repository Reports" section in Oracle Database Performance Tuning Guide.

G.4.2.6 Diagnosing Network Related Problems Using tcpdump

Use tcpdump, a network utility that listeners and captures network traffic to investigate and diagnose network related problems.

For example, if WebCenter Portal page performance metrics in Enterprise Manager indicate that server performance is operating normally while end users are reporting unstable/slow performance, you could have a problem with your network. You can use tcpdump or a similar network monitoring tool to trace network your traffic to see if any environmental issue (on the network) is causing an unexpected large latency.

Refer to tcpdump documentation for information on how to run this utility and analyze the network data.

G.4.2.7 Measuring Network Latency Using ping

Use ping to measure network latency.

Oracle WebCenter Portal installations depend on various backend components and services, such as Oracle HTTP Server, Oracle WebCenter Content Server, LDAP server, instant messaging and presence servers, mail servers, portlets servers, databases, and more. If all your required components are not on the same machine, Oracle recommends they are closely located to minimize network latency and, if possible, are on the same subnet.

Use ping from one machine to another to verify that network latency is less than 1ms.

G.4.2.8 Generating Thread Dumps to Diagnose Extremely Slow Page Performance, High Thread Counts, and System Hangs

Oracle recommends that you generate thread dumps to a dedicated thread dump file and then use a thread analysis tool such as ThreadLogic or a simple text editing tool, to understand and correlate the thread execution logic and progress.

To generate thread dumps for Sun JVM (HotSpot) to a file:

>jstack <pid> > threaddump.txt

To generate thread dumps for Oracle JRockit JVM to a file:

>jrcmd print_threads <pid> > threaddump.txt

Sometimes it is useful to generate a series of thread dumps at fixed sampling intervals to confirm problems such as extremely slow method calls, stuck threads, deadlocks, and so on. For example, you could create a simple script to generate dumps every second or if you are diagnosing a slow request you can choose a suitable duration to cover the length of the slow request.

This example generates thread dumps every second:

#!/bin/bash
for i in {1..20}
do
 JAVA_HOME/bin/jstack <pid> > thread_dump_$i.txt
 sleep 1
done

For more detail about the capabilities of ThreadLogic or any other thread analysis tool, refer the appropriate manufacturer's documentation.

Note:

When WebLogic servers detect a deadlock or stuck thread, a related thread dump is automatically generated in the server's output log file, located at:

DOMAIN_NAME\servers\SERVER_NAME\logs\SERVER_NAME.out

For example, the output log for the out-of-the-box application WebCenter Portal is available at DOMAIM_NAME\servers\WC_Spaces\logs\WC_Spaces.out.

The output file is a simple text file that contains various server messages, including thread information.

G.4.2.9 Analyzing the Diagnostics Log

Look for errors, incidents, and warnings in the diagnostics log for the managed server that is hosting WebCenter Portal or a Portal Framework application, that is, <managed_server_name>-diagnostic.log.

When your application is running with some error condition, it can have a big impact on performance. For example, if a connection to WebCenter Content Server becomes intermittent or not accessible, pages with content related components respond very slowing while attempting to connect and eventually may time out.

ERROR, INCIDENT, and WARNING messages due to timeouts, unavailable services, cache errors, and so on, are logged to a diagnostic log file which you can view from Fusion Middleware Control. See also, Section 28.2.1, "Viewing and Configuring WebCenter Portal Logs."

Note:

When measuring performance, ensure that DEBUG/TRACE messages, that is, levels lower than CONFIG (700) are not being logged. When FINE, FINER, or FINEST messages display, the system is running in debugging mode and this means that most requests are significantly slower than normal. If you configure lower level logging temporarily to debug a problem, ensure that you change the log level before taking performance measurements.

G.4.2.10 Using DMS Spy to Monitor Internal Performance Metric Tables

The DMS Spy servlet provides access to internal DMS metric data from a web browser. This servlet is useful if you want to monitor the system for long period of time and it can help you understand internal system behavior and performance.

You would not normally use DMS Spy to monitor Oracle WebCenter Portal-specific metrics since this information is more easily available through Fusion Middleware Control. However, if you interested in other underlying metrics, such as warning/error messages about the ADF application module (AM) pool display, you can investigate AM pool metrics here. Similarly, if messages relating to the Java Object Cache (JOC) display, you can turn on the JOC's DMS monitoring feature to observe activities in JOC.

To enable JOC DMS monitoring, edit the javacache.xml file (normally it is in <webcenter_portal_domain>/config/fmwconfig/servers/WC_Spaces/javacache.xml)

1. Open the javacache.xml file.

   Typically, the file is located at: <webcenter_domain>/config/fmwconfig/servers/<server_name>/javacache.xml

   For example, for WebCenter Portal, the file is located at: <webcenter_domain>/config/fmwconfig/servers/WC_Spaces/javacache.xml

2. Change the entry:

   <dms enabled="false"/>

   To: <dms enabled="true"/>

3. Restart the managed server on which WebCenter Portal or your Portal Framework applications deployed.

   For example, for WebCenter Portal, restart WC_Spaces.

Once enabled, you see two new entries in the left panel: java_cache_region and joc

java_cache_region shows how much cache memory is allocated to each area and the values displayed can help you determine cache size settings. For example, by default the MDS caching size is set to 100MB in adf-config.xml:

<max-size-kb>100000</max-size-kb>

Both Region Name: ADFApplication<N> and ADFApplication<N>/main_region are used by the MDS cache. If the sum of ADFApplication<N> and ADFApplication<N>/main_region memory is close to 100MB, and your JVM has enough unused heap memory, you could increase the MDS caching size (<max-size-kb>) so the MDS cache is not so full, and this will improve the efficiency of the MDS cache.

For more information, see the "Viewing Performance Metrics Using the Spy Servlet" section in Performance Tuning Guide.

G.4.2.11 Verifying HTTP Request Caching

Most web pages include resources that do not change very often, such as images, CSS files, JavaScript files, and so on. As such resources take time to download over the network, the time taken to load a web page increases. HTTP caching allows such resources to be saved or cached, by a browser or proxy. Once a resource is cached, a browser or proxy can refer to the locally cached copy instead rather than downloading it again on subsequent visits to the web page. Caching reduces round-trip time by eliminating HTTP requests for the required resources. This not only helps significant reducing page load time for subsequent user visits, but also reduces the bandwidth and hosting costs for your site.

By default, static resources serviced by WebCenter Portal use the ADFCachingFilter to include the required response headers which allow browsers to cache static content. If your site is experiencing performance issues, you must confirm that your browser is caching static resources. If caching is not correctly working, static WebCenter Portal resources are repeatedly fetched from the server rather than being cached in the browser, and this can impact performance. A possible reason for caching issues with static portal resources could be dues to some loss introduced by a proxy or network component sitting between WebCenter Portal and the browser.

Note:

For more information about ADFCachingFilter, see Web User Interface Developer's Guide for Oracle Application Development Framework.

Use HTTP request monitoring tools, such as Firebug (Firefox) or httpWatch (Internet Explorer and Firefox), to monitor HTTP traffic between the browser and the server. With these tools, you can see if there is a problem with the cache, that is, whether static resources are fetched for each request, not cached for long enough, or not cached at all for static resources (such as JavaScript and CSS files). When using these monitoring tools, confirm that there are browser-cache related tags in the response, for example, Cache-Control header with a suitable max-age value or an Expires tag with a suitable cache expiry time.

You can also analyze HTTP caching by examining the access.log for the managed server on which WebCenter Portal or your Portal Framework application is deployed. For example, look at the log to see whether a particular user/IP is repeatedly making requests for the same resource. If the log contains repeated requests, the cache header on requests might be wrong and you must investigate caching issue further. Start by accessing the static resource to eliminate various tiers. For example, use wget or curl to fetch the static resource directly using the WebLogic server port by issuing the same on the local machine. Next, access the same resource through any other entity front-ending the WebLogic server that is hosting the application, for example Oracle HTTP Serve or Apache. If needed, enable packet tracing to find the response from the WebCenter Portal tier to see whether issues are occurring on the Oracle WebCenter Portal side, or the problem is on the network.

Note:

The access log for WebCenter Portal is located at: ORACLE_HOME/user_projects/domains/wc_domain/servers/WC_Spaces/logs/access.log

While this section describes static resources serviced by WebCenter Portal, you must also consider similar issues for static resources in your environment. For example, if you have a rich UI where common pages reference static resources that you own, you must review how such content is cached. Consider using Apache Header directives to drive the caching of static resources, either based on file type (such as images, CSS), or based on URL path patterns. For example, the following configuration in Apache sends back a response header to the browser to cache all content under the URL path "/my_images/" for 30 days (2592000 is the number of seconds in 30 days):

<Location /my_images/>
  Header set Cache-Control "max-age=2592000, public"
</Location>

G.4.2.12 Verifying HTTP Compression

In addition to HTTP caching, it is important that you review HTTP compression characteristics for responses. HTTP Compression is a publicly defined way to compress content (mostly textual) that is transferred from web servers to browsers. Compression reduces the number of bytes that are transmitted which improves performance. HTTP Compression uses public domain compression algorithms to encode HTML, XML, JavaScript, CSS and other file formats at the server-side. This standards-based method for delivering compressed content is built into HTTP/1.1, and all modern web browsers support the HTTP/1.1 protocol, that is. they can decode compressed files automatically at the client-side. As a result, no additional software or user interaction is required on the client-side.

By default, static resources serviced by WebCenter Portal use the ADFCachingFilter () to compress static content like CSS and JavaScript. If your site is experiencing performance issues, you must confirm that your browser is seeing compressed responses. You can use one of the HTTP request monitoring tool described in Section G.4.2.11, "Verifying HTTP Request Caching" to scan responses for a Content-Encoding response header with the value gzip. A Content-Encoding response header is only sent if the client (that is, the browser) supports compressed content.

Note:

For more information about ADFCachingFilter, see in Web User Interface Developer's Guide for Oracle Application Development Framework.

If the browser sends the request header Accept-Encoding: gzip, then you know the browser can process compressed content. If you do not see a compressed response, make sure that the client sends an Accept-Encoding header stating it can handle compressed content.

File format that typically require compression include: HTML, CSS, XML, and JavaScript. As most images, music, and videos are already compressed, you do not need to configure these file formats for compression.

Consider using Apache's mod_deflate or mod_gzip to drive the compression of resources, either based on MIME type (such as, text/css) or based on file extension (such as *.html). For example, the following configuration in Apache compresses resources listed in each AddOutputFilterByType:

<Location />
    SetOutputFilter DEFLATE
    AddOutputFilterByType DEFLATE text/plain
    AddOutputFilterByType DEFLATE text/xml
    AddOutputFilterByType DEFLATE text/html
    AddOutputFilterByType DEFLATE application/xhtml+xml
    AddOutputFilterByType DEFLATE text/css
    AddOutputFilterByType DEFLATE application/xml
    AddOutputFilterByType DEFLATE image/svg+xml
    AddOutputFilterByType DEFLATE application/rss+xml
    AddOutputFilterByType DEFLATE application/atom_xml
    AddOutputFilterByType DEFLATE application/x-javascript
    SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary
    SetEnvIfNoCase Request_URI \.(?:exe|t?gz|zip|bz2|sit|rar)$ no-gzip dont-vary
    SetEnvIfNoCase Request_URI \.(?:pdf|doc?x|ppt?x|xls?x)$ no-gzip dont-vary
    SetEnvIfNoCase Request_URI \.avi$ no-gzip dont-vary
    SetEnvIfNoCase Request_URI \.mov$ no-gzip dont-vary
    SetEnvIfNoCase Request_URI \.mp3$ no-gzip dont-vary
    SetEnvIfNoCase Request_URI \.mp4$ no-gzip dont-vary
</Location>

G.4.2.13 Checking Browser Response Times

Use HTTP request monitoring tools, such as Firebug (Firefox) or httpWatch (Internet Explorer and Firefox), to monitor HTTP traffic between the browser and the server. If you understand HTTP request flows and timings for common user actions it can help you diagnose and resolve application performance issues. For example, whenever a user logs in, several redirects can happen between the user's browser and servers. If requests to the login server are taking too long you will know to investigate the login server, rather than WebLogic Server. Similarly, if you see that requests to load the application's landing page is slow, you can focus on making the landing page faster.

G.4.2.14 Warm up the System Before Re-Testing Performance

If you restart an Oracle WebCenter Portal managed server for any reason, make sure that you warm up the system before re-testing performance to avoid Just-In-Time (JIT) compilation overhead.

To warm up your system, you can either manually perform the steps that you later want to measure for performance, or you can use a load test tool to run a few iterations of the steps. After restarting WebLogic managed servers the initial requests are often slower than usual, that is, not typical of the performance that end users would experience.

G.4.3 How to Identify Slow Pages

Use Fusion Middleware Control to determine the slowest pages in a WebCenter Portal or a Portal Framework application. If a poorly performing page is also very popular (the Invocation metric is high) then it makes sense for you to focus efforts to improve performance on those pages.

To find the slowest pages:

1. Log in to Fusion Middleware Control and navigate to the home page for WebCenter Portal or Portal Framework application:

   • Section 6.2, "Navigating to the Home Page for WebCenter Portal"

   • Section 6.3, "Navigating to the Home Page for Portal Framework Applications"

2. Do one of the following:

   • For WebCenter Portal - From the WebCenter Portal menu, select Monitoring > Recent Page Metrics.

   • For WebCenter Portal Framework applications - From the Application Deployment menu, select WebCenter Portal > Recent Page Metrics.

   Page requests that respond slower than the pageResponseTime threshold display "red" in the chart at the top of the page.

3. Click the Sort Descending arrow in the Time (ms) column to sort the page requests by response times.

   Page response times that exceed the threshold display "orange" in the table.

4. Identify the slowest pages and make a note of the portal in which the page displays.

5. For more detailed metrics, including how frequently the slowest pages are requested, do one of the following:

   • For WebCenter Portal - From the WebCenter Portal menu, select Monitoring > Overall Page Metrics.

     Note: Requests for pages in the Home portal are excluded from the "Overall Page Metrics" page.

   • For Portal Framework applications - From the Application Deployment menu, select WebCenter Portal > Overall Page Metrics.

See also, Section 27.1.5, "Understanding Page Request Metrics" and Section 27.3, "Customizing Key Performance Metric Thresholds and Collection."

G.4.4 How to Identify Slow Page Components

Use WebCenter Portal's page performance analyzer to quickly see how long individual components take to display on a portal page, as well as the overall time taken to display a page. When enabled, this tool dynamically measures and presents the performance of individual page components whenever you display a portal page.

The portal page performance analyzer is useful to developers who are performing first level performance analysis, customers who build their own pages, and any user who customizes pages in WebCenter Portal.

This section includes the following subsections:

• About the Portal Page Performance Analyzer

• Enabling and Disabling Portal Page Performance Analysis

• Displaying and Hiding Page Timing Information for Your Current Session

• Using the Page Performance Analyzer to Troubleshoot Performance Issues

• Limitations

G.4.4.1 About the Portal Page Performance Analyzer

The portal page analyzer offers a simple way to diagnose slow pages and requires minimal set up or configuration. When this feature is on, the time spent on "high level" page components is calculated and displayed so you can see at a glance which components are slowing down your page. The overall time spent on the page also displays at the top left of the page (see Figure G-8).

Figure G-8 Portal Page Displays Timing Information

Description of ''Figure G-8 Portal Page Displays Timing Information''

About Page Component Timings

In WebCenter Portal, "high level" page components are wrapped in a ShowDetailFrame so they can be moved, hidden or shown on the page, and edited by Oracle Composer and it is the overall timing for each ShowDetailFrame that displays.

About Overall Page Time

The overall page time is the sum of the individual page component timings, plus some additional processing time for page-level operations such as session replication, save and restore page state, page level security checks, and so on.

Color Coding

Performance timings display in various colors to help alert you to problem areas. Refer to the table below:

Color	Time to Display
Green	< 100 ms
Green/Yellow	100 - 500 ms
Yellow	500 ms - 1 second
Orange	1 - 3 seconds
Red	> 3 seconds

G.4.4.2 Enabling and Disabling Portal Page Performance Analysis

The portal page performance analyzer is disabled out-of-the-box. To make use of this feature, an administrator must specifically enable its use; while the impact on page performance to run this tool is minimal some additional page processing is required.

In a production environment, Oracle recommends that the analyzer is generally disabled to avoid the additional performance data collection and processing and then dynamically enabled when someone reports performance issues for a particular page.

If you do not want end users to see performance data in a production environment, this is another reason to disable the analyzer most of the time.

To enable or disable the portal page analyzer for a WebCenter Portal instance:

1. Use the WLST command exportMetadata to export the base webcenter-config.xml file from MDS.

   For example:

   exportMetadata(application='webcenter', server='WC_Spaces', toLocation='/tmp/mydata', docs='/oracle/webcenter/webcenterapp/metadata/webcenter-config.xml')
   

2. Open webcenter-config.xml exported from MDS in a text editor and set the perfdebug-enabled attribute to true to enable or false to disable this feature.

   For example:

   <webcenter:perfdebug-enabled>true</webcenter:perfdebug-enabled>
   

3. Save and close webcenter-config.xml.

4. Import the updated webcenter-config.xml file to MDS.

   For example:

   importMetadata(application='webcenter', server='WC_Spaces', fromLocation='/tmp/mydata', docs='/oracle/webcenter/webcenterapp/metadata/webcenter-config.xml')
   

There is no need to restart WebCenter Portal to effect this change.

Page performance information does not automatically display after you enable this feature. Anyone who wants to see timing information on portal pages must specifically request that the information displays. For details, see Section G.4.4.3, "Displaying and Hiding Page Timing Information for Your Current Session."

G.4.4.3 Displaying and Hiding Page Timing Information for Your Current Session

When an administrator enables the page performance analyzer in an WebCenter Portal instance, anyone with access to that WebCenter Portal instance can elect to display or hide page timing information, for their current user session, by appending the perfDebug parameter to the page URL as follows:

To...	Add a perfDebug parameter to the page URL
Display timing information on portal pages	&perfDebug=on
Stop displaying page performance information	&perfDebug=off

To display timing information on portal pages:

1. Verify that your administrator enabled the page performance analyzer in your WebCenter Portal instance.

   See also Section G.4.4.2, "Enabling and Disabling Portal Page Performance Analysis."

2. Log in to WebCenter Portal and navigate to the portal page that you want to investigate.

   You do not need to log in if the page is a public page.

3. Add &perfDebug=on to the end of the page URL (Figure G-9).

   For example:

   http://mycompany.com/webcenter/portal/MySalesPortal?_adf.ctrl-state=mji8314i6_4&_afrLoop=474789135539036&perfDebug=on
   

4. Click "Go" or press Enter to redisplay the page with timing information (as shown in Figure G-8).

   All subsequent pages that you display show timing information as well.

Figure G-9 Appending the perfDebug Parameter to Page URLs

Description of ''Figure G-9 Appending the perfDebug Parameter to Page URLs''

To stop displaying page timing information:

1. In your browser, add &perfDebug=on to the end of any page URL (Figure G-9).

   For example:

   http://mycompany.com/webcenter/portal/MySalesPortal?_adf.ctrl-state=mji8314i6_4&_afrLoop=474789135539036&perfDebug=off
   

2. Click "Go" or press Enter to display the page again without timing information.

G.4.4.4 Using the Page Performance Analyzer to Troubleshoot Performance Issues

The steps in this section describe how to troubleshoot slow pages using WebCenter Portal tools:

1. If a user reports performance issues with a particular page, navigate to the slow pages and confirm that the slow performance consistently reproduces.

   Alternatively, use Fusion Middleware Control to proactively identify the slowest pages in your application. See Section G.4.3, "How to Identify Slow Pages."

2. Append &perfDebug=on to the page URL to display timing information for the page.

   See also, Section G.4.4.3, "Displaying and Hiding Page Timing Information for Your Current Session."

   Note:

   If page timing information does not display, ask your administrator to enable the page performance analyzer. For details, see Section G.4.4.2, "Enabling and Disabling Portal Page Performance Analysis."

3. Identify the slowest page components, and troubleshoot the issue further:

   For example, if the slow component contains:

   • Document, wiki, or content presenter, check the performance of the back-end Content Server and the database that Content Server is using.

   • Activity stream, use AWR reports to check database performance and to see if you can tune the database table used by activity stream.

   • Collaboration features, check the performance of the associated back-end server. For example, for announcements or discussions, monitor the performance of the discussions server.

   • Portlets, use Fusion Middleware Control to monitor portlet request timing information, errors, portlet producer performance, and so on

4. If necessary, add the slow page component to a separate "blank" page and then do further profiling.

   For example, use JRockit flight recording to pinpoint the bottleneck.

G.4.4.5 Limitations

The page performance analyzer works best with portal pages that you create with WebCenter Portal 11.1.1.8.0 and later. In previous releases, some page templates did not include the necessary tags to display component and total page timings and some task flows were not configured with activation=deferred.

G.4.5 How to Troubleshoot Slow Page Requests

Use the information in this section to diagnose issues relating to poor page performance:

• Troubleshooting Live Requests

• Troubleshooting Stuck Threads

• Troubleshooting Slow Requests Using JFR Recordings

• Troubleshooting Memory Leaks and Heap Usage Problems

• Troubleshooting Slow Requests for Content

G.4.5.1 Troubleshooting Live Requests

To troubleshoot slow page requests that are still running, extract and view a JRockit Flight Recorder (JFR) recording against the server on which the user session is running. See also, Section G.4.6, "How to Troubleshooting Requests using JRockit Flight Recordings."

Alternatively, take several evenly spaced thread dumps, for example, every 1 or 2 seconds as described in Section G.4.2.8, "Generating Thread Dumps to Diagnose Extremely Slow Page Performance, High Thread Counts, and System Hangs."

If you compare the thread dumps, you might see threads that spent a long time on certain method calls as the call stacks are the same in several consecutive thread dumps. For example, you might see a method call to a database, Oracle WebCenter Content Server, collaboration server, portlet producer, LDAP server, and so on, in which case you can investigate the associated backend server to diagnose the issue further.

G.4.5.2 Troubleshooting Stuck Threads

Stuck threads can occur for several reasons:

• Server is nearly out of memory. If the server is close to out of memory, all requests slow down. To resolve out-of-memory issues, see Section G.4.5.3, "Troubleshooting Slow Requests Using JFR Recordings."

• Deadlock threads. Take thread dumps and search for deadlock threads. This normally exposes an issue with the product code.

• Extremely slow page requests. Take several evenly spaced thread dumps and find out which method is taking a long time to execute.

  If a request is taking longer than 10 minutes, the stuck thread is reported to Oracle WebLogic Server server_name.out in the following directories:

  (UNIX) DOMAIN_HOME/servers/server_name/logs
  (Windows) DOMAIN_HOME\servers\server_name\logs
  

  For example:

  <Mar 4, 2012 7:44:08 AM PST> <Error> <WebLogicServer> <BEA-000337> 
  <[STUCK] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'
   has been busy for "600" seconds working on the request
   "weblogic.servlet.internal.ServletRequestImpl@18986012[
  
  GET
   /server_name/faces/PimDashboardUiShellPage?_afrLoop=1398820150000&_afrWindowMod
  e=0&_adf.ctrl-state=a44e7uxcc_13 HTTP/1.1
  Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg,
   application/x-shockwave-flash, application/x-ms-application,
   application/x-ms-xbap, application/vnd.ms-xpsdocument, application/xaml+xml,
   application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword,
   */*
  Accept-Language: fr
  UA-CPU: x86
  ...
  ]", which is more than the configured time (StuckThreadMaxTime) of "600"
   seconds
  . Stack trace:
  Thread-164 "[STUCK] ExecuteThread: '19' for queue: 'weblogic.kernel.Default
   (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
      jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
      jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
      java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
      java.net.SocketInputStream.read(SocketInputStream.java:107)
  ...
  

Diagnosing a Stuck Thread

If the stack shows the thread is waiting for a response from another server, check the status of the other server and see it has performance problems before proceeding with the steps below.

To determine what the stuck thread was doing prior to becoming stuck, perform the following steps:

1. Look at the next few log messages in server_name.out for a message indicating an incident has been created. For example:

   <Mar 4, 2012 7:44:10 AM PST> <Alert> <Diagnostics> <BEA-320016> 
    <Creating diagnostic image in DOMAIN_HOME/servers
    /server_name/adr/diag/ofm/MyDomain/
    server_name_1/incident/incdir_394 with a lockout minute 
    period of 1.>
   

   The above message may not always appear after each stuck thread reported. It is printed at most four times an hour. If the message does not appear, manually look for the incident directory by checking the readme file in the subdirectories under the following directories:

   (UNIX) DOMAIN_HOME/servers/server_name/adr/diag/ofm/domain_name/server_name/incident
   (Windows) DOMAIN_HOME\servers\server_name\adr\diag\ofm\domain_name\server_name\incident
   

   The incident directory contains a WLDF diagnostic image which contains the JFR recording, and a file containing the thread dump.

   For more information about diagnosing incidents, see the "Diagnosing Problems" chapter in the Administrator's Guide.

2. Review the thread dump to find the call stack of the thread. If the thread is blocked waiting for a lock, check what the thread holding the lock is doing.

3. If the call stack shows that JDBC calls are taking a long time, generate an AWR report on the database to find the query and which table to look and tune.

   See Section G.4.2.5, "Generating Automatic Workload Repository (AWR) Reports for the Database."

4. Review the JRockit flight recording file JRockitFlightRecorder.jfr for more details. You will also need the ECID of the request which is recorded in the readme.txt file of the incident directory, and also the Oracle WebLogic Server log.

   See also, Section G.4.6, "How to Troubleshooting Requests using JRockit Flight Recordings."

The ECID of the request that caused the stuck thread is recorded in the error message.

G.4.5.3 Troubleshooting Slow Requests Using JFR Recordings

See Section G.4.6, "How to Troubleshooting Requests using JRockit Flight Recordings."

G.4.5.4 Troubleshooting Memory Leaks and Heap Usage Problems

If WebCenter Portal or Portal Framework application performance degrades over time, heap usage and garbage collection activity is increasing, and you see OutOfMemoryErrors, there could be memory leaks in the application causing the amount of free memory in the JVM to continuously decrease.

Figure G-10 shows a typical memory leak trend displayed in JConsole.

Figure G-10 Typical Memory Leak Trend in JConsole

Description of ''Figure G-10 Typical Memory Leak Trend in JConsole''

To solve this problem:

1. Determine the cause of OutOfMemoryErrors errors:

   • Review the server_name.out file for OutOfMemoryErrors errors.

     The server_name.out file is located at:

     (UNIX) DOMAIN_HOME/servers/server_name/logs
     (Windows) DOMAIN_HOME\servers\server_name\logs
     

   • Take a memory dump when OutOfMemoryErrors errors occur.

     For example:

     On Sun HotSpot: jmap -dump:live,format=b,file=<path>/heap.hprof <pid>

     On JRockit: jrcmd <pid> hprofdump filename=<path>/heap.hprof

     You can configure JRockit to automatically generate a heap dump in HPROF binary format (.hprof file) each time an OutOfMemoryErrors occurs, by setting the JRockit JVM option, -XX:+HeapDumpOnOutOfMemoryError. For details, refer to the Oracle JRockit Command-Line Reference.

2. Restart the managed server.

   If the problem persists, proceed to Step 3.

3. Open the heap.hprof file with a heap-dump analysis tool that can handle binary HPROF format, such as Eclipse Memory Analyzer.

4. Determine which objects and classes are retaining the most memory.

5. If necessary, take several heap dumps to determine which objects or classes are consuming and increasing the amount of memory.

   Take at least two memory dumps:

   • Take the first dump when the system is warmed up and stabilized.

   • Take the second dump, when the system is about to run out of memory, that is, full garbage collection gets less than 300MB from the maximum heap size.

   Instructions on how to take a heap dump using Sun HotSpot (jmap) or JRockit (jcrcmd) is described in step 1.

   See the "Running Diagnostic Commands" chapter in the Oracle JRockit JDK Tools Guide. Many heap dump analysis tools, such as Eclipse Memory Analyzer, enable you to compare two heap dumps to identify memory growth areas.

   Heap dumps provide information on why memory is retained (Retained Heap). Sometimes it is necessary to know how memory is allocated to further resolve the issue. For these cases, proceed to Step 6.

6. Use the JRockit Memory Leak Detector tool that is part of JRockit Mission Control Client to understand how memory is allocated.

   For more information, see the JRockit Mission Control online help.

G.4.5.5 Troubleshooting Slow Requests for Content

If slow page performance is due to content/document-related components, for example, Documents service task flows, Content Presenter task flows, wikis or blogs, Oracle recommends that you review performance metrics for the backend Oracle WebCenter Content Server (System Audit Information page). For details, see the "Viewing System Audit Information" section in Administering Oracle WebCenter Content.

Ensure that the systemdatabase tracing option is selected so you can see performance information for each query that is sent to the database. For details, see the "Server-Wide Tracing" section in Administering Oracle WebCenter Content.

G.4.6 How to Troubleshooting Requests using JRockit Flight Recordings

JRockit Flight Recorder (JFR) files contain a record of various events that consume time. If requests are slow, you can analyze the JRockit Flight Recorder (JFR) file to find out why request are taking time.

To create a JFR file:

1. Extract a JFR file from Oracle WebLogic Server by running the following command:

   UNIX) JROCKIT_HOME/bin/jrcmd jrockit_pid dump_flightrecording recording=1 copy_to_file=path compress_copy=true
   
   (Windows) JROCKIT_HOME\bin\jrcmd.exe jrockit_pid dump_flightrecording recording=1 copy_to_file=path compress_copy=true
   

   For more information about the jrcmd command-line tool, see the "Running Diagnostic Commands" section in Oracle JRockit JDK Tools Guide.

2. To view the file, start the JRockit Mission Control Client from the following directories:

   (UNIX) JAVA_HOME/bin/bin/jrmc
   
   (Windows) JAVA_HOME\bin\jrmc.exe
   

3. Select File > Open File to select the JFR file.

4. Locate the slowest requests or investigate a specific request:

   To locate the slowest requests:

   To investigate a specific request:

   In the JRockitFlightRecorder.jfr page, click the Events icon. Click the Log tab at the bottom of the page. In the Event Type navigation pane on the left, locate Dynamic Monitoring System and then HttpRequest. Click HTTP request; deselect all the other event types. In the Log tab, in the Event Log section, click the Duration column to sort the duration in descending order. Each row corresponds to a HTTP Request and the duration column shows the response time for that request. Click the row in the table to view the attributes of the requests. In the Event Attributes sections, note the start time and the thread that serviced the request.

   Find the Execution Context Identifier (ECID) of that request. If the request is related to an incident triggered by a STUCK thread, the incident readme.txt file will contain the ECID. Alternatively, you can search the Oracle WebLogic Server HTTP access.log for requests from specific users. See the "Viewing and Searching Log Files" section in the Administrator's Guide. In the JRockit Mission Control Client, in the JRockitFlightRecorder.jfr page, select the WebLogic icon. Note: If the Weblogic icon is not available, select Help > Install Plugins to download the Oracle WebLogic Server plug-in. Click the ECIDs tab at the bottom of the page. In the ECIDs section, from Filter Column list, select ECID. Enter the ECID in the search box and select <Enter>. In the results table, highlight the row with the matching ECID and right-click to bring up the menu. Select Operative Set > Clear, and then Operative Set > Add matching ECID > ECID to add the ECID to the operative set. This enables users to view only events associated with the operative set. Click the Events icon. In the Event Type navigation pane on the left, locate Dynamic Monitoring System and then HttpRequest. Click HTTP request; deselect all the other event types. ** In the Event Log section, click Show Only Operative Set. Each row corresponds to the request with the matching ECID. Click the row in the table to view the attributes of the requests. Note the start time and the thread that serviced the request.

   
   

5. Once you have identified the start time and the thread that serviced the request, navigate to the Logs tab, and drag the time selector at the top of the screen to include only the time window for the duration of the request.

6. In the Event Log section, perform the following search:

   1. Deselect Show Only Operative Set.

   2. Enter the thread name in the search box.

   3. From the Filter Column list, select Thread.

   4. Select <Enter>.

7. In the Event Type navigation pane on the left, click the events of interest. Typically, these events are located under nodes Dynamic Monitoring System, Java Application, and WebLogic > JDBC.

   The selected events appear in the table in the Event Log section.

8. Click the Start Time column to sort the time when these events occur, or click the Duration column to view the events that took longest.

   The JDBC Statement Execute events corresponds to SQL execution. If there are slow SQL statements, the event details give the SQL text. These events do not have callstacks.

9. To check the call stacks for slow SQL statements, view the Socket Read event that happens immediately after the JDBC Statement Execute event.

   This event corresponds to Oracle WebLogic Server waiting for the SQL results to return, and it has callstack in the event details.

10. Review the call stacks for long Java Blocked and Java Wait events to see if you can identify what is causing slow performance.

    See the "Analyzing Flight Recorder Data in JRockit Mission Control" section in Configuring and Using the Diagnostics Framework for Oracle WebLogic Server.

11. If you need more detail than the information captured in the default recording, and you can reproduce the slow requests, you can start an explicit recording.

    See the "Starting an Explicit Recording" section in Oracle JRockit Flight Recorder Run Time Guide.

G.5 Using My Oracle Support for Additional Troubleshooting Information

You can use My Oracle Support (formerly MetaLink) to help resolve Oracle WebCenter Portal problems. My Oracle Support contains several useful troubleshooting resources, such as:

• Knowledge base articles

• Community forums and discussions

• Patches and upgrades

• Certification information

Note:

You can also use My Oracle Support to log a service request.

You can access My Oracle Support at https://support.oracle.com.

G.6 Troubleshooting WebCenter Portal Workflows

If you experience issues with WebCenter Portal workflows, review the following sections:

• Section G.6.1, "Validating the WebCenter Portal Workflow Configuration"

• Section G.6.2, "Troubleshooting Issues with WebCenter Portal Workflows"

G.6.1 Validating the WebCenter Portal Workflow Configuration

The Installation Guide for Oracle WebCenter Portal describes how to install and configure WebCenter Portal workflows. For details, see the "Back-End Requirements for WebCenter Portal Workflows" section. You can validate the workflow configuration as follows:

1. Log in to WebCenter Portal.

2. Create a portal and then navigate to the Members tab (click the Administration link, then Security, then Members).

3. Invite a new member with any role (say User2).

4. Log out, and then log in as User2.

5. Navigate to a Worklist task flow.

6. Open the invite notification and click the Accept button.

7. Open the portal.

If the WebCenter Portal workflows are working properly, the newly created portal is available to User2. If the portal is not available or listed, there is some issue with the configuration.

G.6.2 Troubleshooting Issues with WebCenter Portal Workflows

If WebCenter Portal workflows are not working properly, follow these steps to help troubleshoot the issue:

1. Check that WebCenter Portal workflows are deployed on the Oracle SOA server:

   1. Log in to Fusion Middleware Control.

   2. Check that WebCenterWorklistDetailApp.ear is deployed.

   3. Verify that sca_CommunityWorkflows.jar is deployed.

   For details, see the "Oracle SOA Server - Extending the Domain" section in Installation Guide for Oracle WebCenter Portal.

2. Ensure the Web Service connection between the Oracle SOA server and the WebCenter Portal application is secure:

   1. Check the alias in the keystore file on the Oracle SOA server.

      For example, use the following command to list the content of the keystore file on the Oracle SOA server:

      keytool -list -v -keystore bpel.jks -storepass <password>

      There should be an entry with:

      Alias name: webcenter_portals_ws
      
      

      See the "Setting Up the SOA Domain" section in Installation Guide for Oracle WebCenter Portal.

   2. Verify that the credential stores for both WebCenter Portal and Oracle SOA server are configured correctly.

   3. Check that keystores exist at both ends of the connection, for example:

      - webcenter.jks (copied to WebCenter Portal server end)

      - bpel.jks (copied to Oracle SOA server end)

      For example, the following commands generate webcenter.jks and bpel.jks:

      keytool -genkeypair -keyalg RSA -dname "cn=webcenter,dc=us,dc=oracle,dc=com" -alias webcenter -keypass mypassword -keystore webcenter.jks -storepass mypassword -validity 360
      keytool -exportcert -v -alias webcenter -keystore webcenter.jks -storepass mypassword -rfc -file webcenter.cer
      keytool -importcert -alias webcenter_spaces_ws  -file webcenter.cer -keystore bpel.jks -storepass mypassword
      keytool -genkeypair -keyalg RSA -dname "cn=bpel,dc=us,dc=oracle,dc=com" -alias bpel -keypass mypassword -keystore bpel.jks -storepass mypassword -validity 360
      keytool -exportcert -v -alias bpel -keystore bpel.jks -storepass mypassword -rfc -file bpel.cer
      keytool -importcert -alias bpel -file bpel.cer -keystore webcenter.jks -storepass mypassword
      

      See the "Creating the SOA Domain Keystore" section in Administering Oracle WebCenter Portal.

   4. Configure role members for the BPMWorkflowAdmin application role in Oracle SOA server (soa-infra).

      When associating the domain with an identity store that does not contain the user weblogic, you must assign some other valid user to the application role BPMWorkflowAdmin. Use WLST commands to do this from the SOA Oracle home, for example, to assign a user named "monty" that exists in LDAP:

      cd $SOA_ORACLE_HOME/common/bin/
      wlst.sh
       
      connect('<admin username>','<admin password>', 'mysoahost.example.com:7001')
      revokeAppRole(appStripe="soa-infra", appRoleName="BPMWorkflowAdmin", principalClass="oracle.security.jps.service.policystore.ApplicationRole", principalName="SOAAdmin")
      grantAppRole(appStripe="soa-infra", appRoleName="BPMWorkflowAdmin", principalClass="weblogic.security.principal.WLSUserImpl", principalName="monty")
      
      

      See the "Overview of WLST Security Commands" section in WebLogic Scripting Tool Command Reference.

G.7 Troubleshooting WebCenter Portal Import and Export

This section contains the following subsections:

• Section G.7.1, "ResourceLimitException Issue"

• Section G.7.2, "LockRefreshTask Issue"

• Section G.7.3, "Portals and Portal Templates Not Available After Import"

• Section G.7.4, "Unable to Migrate Portals or Documents If the Source and Target Applications Share the Same Content Server"

G.7.1 ResourceLimitException Issue

Problem

The ResourceLimitException error displays when you try to export all portals or the entire WebCenter Portal instance:

Weblogic.common.resourcepool.ResourceLimitException

Solution

Increase the maximum capacity in the JDBC connection pool. To reconfigure the connection pool, log in to the WLS Administration Console. From Services, select Data Sources, WebCenterDS, and then the Connection Pool tab.

G.7.2 LockRefreshTask Issue

Problem

A LockRefreshTask warning displays similar to that below when you try to import or export an entire WebCenter Portal instance or a portal:

[WARNING] [][oracle.webcenter.lifecycle.operation.LockRefreshTask]

If you try the import or export operation again, an error similar to that shown here displays:

Starting WebCenter Portal application import...
WebCenter Portal application import started.

Error occurred while performing import
None
Check the WebCenter Portal log files for additional details.
Unable to contact MBeanServer for
oracle.webcenter.lifecycle:ApplicationName=webcenter,Location=WC_Spaces,name=Lifec
ycleManager,type=LifecycleManager,Application=webcenter,ApplicationVersion=11.1.1.
4.0
Error occurred while destroying MBean
The lock hasnt been released from the previous failed import.

Solution

Use the deleteMetadata WLST command to delete unwanted locks in MDS that may be created and not destroyed due to an unexpected and unusual import or export operation failure. Depending on the operation that failed, run one of the following commands:

For WebCenter Portal application import failure, run:

deleteMetadata(application='webcenter', server='WC_Spaces', docs='/oracle/webcenter/lock/applicationImport/applicationImport.xml')

For WebCenter Portal application export failure, run:

deleteMetadata(application='webcenter', server='WC_Spaces', docs='/oracle/webcenter/lock/applicationExport/applicationExport.xml')

For portal import failure, run:

deleteMetadata(application='webcenter', server='WC_Spaces', docs='/oracle/webcenter/lock/scopeImport/**')

For portal export failure, run:

deleteMetadata(application='webcenter', server='WC_Spaces', docs='/oracle/webcenter/lock/gsExportImport/**') 

G.7.3 Portals and Portal Templates Not Available After Import

Problem

When you first log in to WebCenter Portal after the import operation, the portals and portal templates that you migrated are not available as expected. This can sometimes occur if the portal or portal template cache fails to refresh properly.

Solution

Refresh the portal or portal template cache manually using the refreshGroupSpaceCache and refreshSpaceTemplateCache WLST commands.

To completely clear the cache (all portals):

refreshGroupSpaceCache(appName='webcenter', spaceNames='', syncMode=1,updateType='all', cleanCache=1)

To completely clear the cache (all portal templates):

refreshSpaceTemplateCache(appName='webcenter', spaceTemplateNames='',syncMode=1, updateType='all', cleanCache=1) 

For detailed command syntax and examples, see the "refreshGroupSpaceCache" and "refreshSpaceTemplateCache" sections in WebLogic Scripting Tool Command Reference.

For information on how to run WLST commands, see Section 1.13.3.1, "Running Oracle WebLogic Scripting Tool (WLST) Commands."

G.7.4 Unable to Migrate Portals or Documents If the Source and Target Applications Share the Same Content Server

You cannot migrate portals or portal templates between two different WebCenter Portal instances that share the same Content Server.

G.8 Troubleshooting Individual Portal and Portal Template Import and Export

This section contains the following subsections:

• Section G.8.1, "Portal Blocked After Unsuccessful Export or Import"

• Section G.8.2, "Page or Portal Not Found Message After Import"

• Section G.8.3, "Portal Import Archive Exceeds Maximum Upload File Size"

• Section G.8.4, "Maximum Number of Portals Exceeded on Export"

• Section G.8.5, "Lists Not Imported Properly"

• Section G.8.6, "Exporting and Importing Portals with Tools and Services Configured"

• Section G.8.7, "Tools and Services Disabled After Import"

• Section G.8.8, "Importing from the Subportals Page"

• Section G.8.9, "Unable to Import a Portal If the Source and Target Applications Share the Same Content Server"

• Section G.8.10, "Exporting and Importing Portals in Multibyte Languages"

G.8.1 Portal Blocked After Unsuccessful Export or Import

If an error occurs during a portal export/import operation, some portals may appear blocked. To unblock a portal, bring the portal back online temporarily, and then take the portal offline again to complete the export/import operation. Switching between the online and offline modes will unblock the portal. For more information, see Section 46.11, "Taking Any Portal Offline" and Section 46.12, "Bringing Any Portal Back Online." See also, the WLST command "setSpaceState" in WebLogic Scripting Tool Command Reference.

G.8.2 Page or Portal Not Found Message After Import

When users first log in to WebCenter Portal after an import operation, they may see a "Page not found" or "Portal not found" message if the page or portal they last visited no longer exists. Last accessed page information is retained during import operations which is why these messages display sometimes.

G.8.3 Portal Import Archive Exceeds Maximum Upload File Size

Problem

There is a file size limitation uploading content to WebCenter Portal. If your export archive exceeds the maximum upload size, the import operation through WebCenter Portal Administration will fail.

Solution

Import the portal archive using WLST. For details, see Section 40.1.2.4.3, "Importing a Portal from an Archive Using WLST."

Alternatively, modify the maximum upload size in webcenter-config.xml. The default maximum upload size is 2 GB. See Section 9.12, "Changing the Maximum File Upload Size."

G.8.4 Maximum Number of Portals Exceeded on Export

Problem

The maximum number of portals that you can export must be less than or equal to 80% of the connection pool size specified for the MDS Data Source. If you try to export too many portals you might see a ResourceLimitException error:

Weblogic.common.resourcepool.ResourceLimitException

Solution

Export fewer portals. Alternatively, modify the connection pool setting. For details, see the Performance Tuning Guide.

G.8.5 Lists Not Imported Properly

Problem

Lists are not importing properly due to list definition differences in the source and target systems.

Solution

Consider exporting and importing list data. This ensures that list data is consistent with the list definitions being imported.

If you choose to import without data, the list data in the target system is migrated to be consistent with the imported list definitions. If a list column data type is changed, the column values are converted from the target data type to the imported data type, if possible, otherwise the value is deleted. If a list column is removed during import, the column values are deleted.

G.8.6 Exporting and Importing Portals with Tools and Services Configured

Problem

The following error message displays when you try to export a portal with tools and services configured, and try to import the same portal from an instance where some or all of those tools or services are not configured.

The following services are not configured: <list of tools and services>. Please configure these services and try again.

Solution

You can work around this problem by either adding the tools and services to the target, or removing the service-related info from the data.xml file of the archive as described below.

To remove service-related info:

1. Extract the archive.

   The archive contains two files: policy-store.xml and transport.mar.

2. Expand the transport.mar into a directory.

   The data.xml file is located in the oracle\webcenter\lifecycle\importexport directory.

3. Remove the service tags for all the tools and services that are not present in the target as listed in the error message.

   For the example error message above, we would need to remove the following:

         <service id="oracle.webcenter.collab.forum" version="11.1.1.0">
            <metadataUsages>
               <metadataUsage includeBaseDocuments="YES" includeSystemCustomizations="YES">
                  <paths>
                     <include path="/oracle/webcenter/collab/forum/scopedMD/s516227ec_dde1_4991_9e18_28d487cb3bce/**"/>
                  </paths>
               </metadataUsage>
            </metadataUsages>
         </service>
    
      <service id="oracle.webcenter.collab.rtc" version="11.1.1.0"/>
   

4. Repack the transport.mar file by zipping the top-level directories Oracle and pagedefs into a file named transport.mar.

5. Repack the export archive by zipping the newly created transport.mar and the policy-store.xml file into an archive.

6. Import the new archive.

G.8.7 Tools and Services Disabled After Import

Problem

When you navigate directly to the Tools and Services tab in portal administration after importing a portal, all the tools and services are disabled even though they were enabled in the source portal.

This can sometimes happen when the archive you import contains a portal that was exported from a release earlier than WebCenter Portal 11.1.1.8.

Solution

Select the Enable check box for tools and services, as required.

Alternatively, open the portal after you import instead of navigating to portal administration. When you access the portal for the first time, tools and services enable automatically.

G.8.8 Importing from the Subportals Page

Problem

When you import a portal from the Portals page, the imported portal does not automatically become a subportal of the current portal. The newly imported portal displays in the Portals switcher menu, Portals Browser task flow, or the Portals page, which display all the portals that are available to you.

Solution

You can import a portal as a subportal by selecting the parent portal on the Portals page before you import the archive.

G.8.9 Unable to Import a Portal If the Source and Target Applications Share the Same Content Server

You cannot export/import portals or portal templates between two different WebCenter Portal applications that share the same Content Server.

Similarly, you cannot use the Document Migration Utility to migrate portal documents between two different WebCenter Portal applications that share the same Content Server.

G.8.10 Exporting and Importing Portals in Multibyte Languages

Problem

On Linux, individual portal export or import fails for one or more portals created in multibyte languages due to naming restrictions. Portal names are restricted to alphanumeric and portal characters ("a" through "z", "A" through "Z", "0" through "9", and the single-byte portal character, which WebCenter Portal replaces with "_"(underscore) ). If any other characters are used in the portal name, export or import fails.

Solution

Enforce the naming restriction on the server on which Oracle WebCenter Portal is deployed. To do this, set the environment variable LC_ALL set to utf-8.