A Troubleshooting Oracle Web Cache

This appendix describes common problems that you might encounter when using Oracle Web Cache and explains how to solve them. It contains the following topics:

A.1 Problems and Solutions

This section describes common problems and solutions. It contains the following topics:

A.1.1 No Response from Application Web Server Error

Problem

If an 11g Release 1 (11.1.1) Oracle Web Cache is reverse proxying 10g components, such as Oracle Portal, Oracle Forms Services, or Oracle Business Intelligence Discoverer, and SSL is enabled, the following browser error may return:

No response from Application Web Server

In addition, messages similar to the following appear in the event log:

[2009-02-11T03:44:55-08:00] [webcache] [ERROR:32] [WXE-11904] [security]
[ecid: ] SSL handshake fails NZE-29024
[2009-02-11T03:44:55-08:00] [webcache] [WARNING:1] [WXE-11905] [security]
[ecid: ] SSL additional information: The certificate sent by the other side could not be validated. This may occur if the certificate has expired, has been revoked, or is invalid for another reason.
[2009-02-11T03:44:55-08:00] [webcache] [WARNING:1] [WXE-11905] [security]
[ecid: ] SSL additional information: Remote IP [140.87.11.159]:9002
[2009-02-11T03:44:55-08:00] [webcache] [WARNING:1] [WXE-11905] [security]
[ecid: ] SSL additional information: Local IP 152.68.64.152:2832
[2009-02-11T03:44:55-08:00] [webcache] [WARNING:1] [WXE-11906] [security]
[ecid: ] SSL details: SSL error during handshake (details: internal=The
certificate sent by the other side could not be validated. This may occur if the certificate has expired, has been revoked, or is invalid for another reason. system=Success)

This error indicates that the wallet you selected for Oracle Web Cache contains a certificate that does not match the wallet used by the 10g components.

Solution

To resolve this problem, modify the wallet you specify for Oracle Web Cache to use the wallet you are using for the 10g components. See Section 5.4.3.

A.1.2 Load Issues on Oracle Web Cache Computer

On UNIX operating systems, the top and uptime utilities report a higher than expected average load when the Oracle Web Cache computer is idle.

Problem

This effect occurs because Oracle Web Cache performs light maintenance work, even when it is idle. One operation Oracle Web Cache performs is garbage collection. During idle mode, the following effect occurs:

  • The uptime load—the average kernel scheduler queue length—is going to be longer. Oracle Web Cache increases the average queue length (uptime output) by approximately one.

  • The CPU load is still low because the work Oracle Web Cache performs is minimal.

A.1.3 Performance Degradation and Memory

Because Oracle Web Cache is an in-memory cache, it is best to deploy Oracle Web Cache on a dedicated computer to minimize paging. Unless the computer is dedicated to run Oracle Web Cache, ensure the maximum cache size does not exceed 20 percent of the total memory.

This section describes workarounds to invalidation timeouts:

Problem 1: Paging

If the time taken to cache or invalidate objects increases, the computer may be paging. Paging can severely degrade performance.

Solution 1

To configure Oracle Web Cache to work efficiently on a computer with paging, either deploy Oracle Web Cache on a dedicated computer or reduce the maximum cache size and maximum cached object size. See Section 2.11.5 to configure these settings.

Problem 2: Oracle Web Cache Using Memory than the Maximum Cache Size

If Oracle Web Cache uses more memory than the maximum cache size, the increase may be caused by numerous simultaneous requests for objects that are larger than the maximum cached object size. In this situation, because the objects are not cached, Oracle Web Cache uses more memory processing the requests and forwarding them to the origin server than it would to cache the objects.

Solution 2

Review access logs to determine if many simultaneous requests for large objects have been made and adjust the size of the maximum cached object size so that those objects are cached. In addition, check that a caching rule or response header specifies that the objects are to be cached.

To modify the maximum cache size or the maximum cached object size, set new limits. See Section 2.11.5 to configure these settings.

A.1.4 Invalidation Timeouts in a Cache Cluster

Invalidation has a default timeout of 300 seconds for the propagation of invalidation requests in the cache hierarchy or cache cluster deployments. See Section 7.10.2 for an overview of invalidation propagation.

Problem

When the timeout is exceeded in a cache cluster, a message similar to the following is displayed in the response to the invalidation request:

Can't connect to the web cache's invalidation listening port.

Solution

To resolve this error:

  1. On cache cluster members, use a text editor to open the webcache.xml file.

  2. Locate the CALYPSONETINFO element:

    <CALYPSONETINFO...INV_PEER_TIMEOUT="300"
                    INV_GLOBAL_TIMEOUT="300".../>
    
  3. Modify the value of INV_PEER_TIMEOUT attribute.

    In a cache cluster, it is likely that cache cluster members run in a LAN environment. Therefore, decreasing the value of INV_PEER_TIMEOUT typically improves efficiency.

  4. Save webcache.xml.

  5. Restart Oracle Web Cache with the following command:

    opmnctl restartproc ias-component=component_name
    

    This executable is found in the following directory:

    (UNIX) ORACLE_INSTANCE/bin
    (Windows) ORACLE_INSTANCE\bin
    
  6. Synchronize configuration changes to all cache cluster members and restart the other cache cluster members. See Section 3.6.5 to perform this task in Fusion Middleware Control and Section 3.7.4 to perform this task in Oracle Web Cache Manager.

A.1.5 Capacity Issues on Origin Server

Problem

If an application Web server has reached capacity, the following error message appears when accessing pages of a Web site:

The application Web server is busy. Possible reach capacity.

In addition, messages similar to the following appear in the event log:

[2009-04-16T11:56:23-04:00] [webcache] [WARNING:1] [WXE-14021] [backend] [ecid: 453611135182,0] Capacity, as set by Web Cache, is exceeded for all origin servers that are available for processing this request.
[2009-04-16T11:56:23-04:00] [webcache] [ERROR:32] [WXE-11365] [frontend] [ecid: 453611135182,0] Server busy response is returned.

These errors indicate that the application Web server has exceeded the maximum number of concurrent connections.

Solution

To resolve this problem, you can either:

  • Increase capacity. See Section 2.11.2.

  • Evaluate the caching rules to determine if additional content can be cached. See Section A.3.

A.1.6 Browsers Not Receiving Complete Responses

The client browser is not receiving the complete response.

Problem

If the actual length of a page is less than value of the Content-Length response-header field set by the origin server and sent to a client browser by Oracle Web Cache, the browser expects more data to arrive and the connection does eventually time out. If the actual length of the page is greater than the Content-Length, the browser does not receive the complete page. This problem does not occur for cache hits because Oracle Web Cache correctly calculates the content length itself for pages stored in the cache.

Messages similar to the following appear in the event log:

[2009-04-16T13:23:32-04:00] [webcache] [WARNING:1] [WXE-11093] [backend] [ecid: 268932876536,0] Content-Length value 7755 sent byorigin server does not match number of bytes received, that is 454 bytes for localhost:7785/cgi-bin/test-cgi.

Solution

For cache misses, there are two workarounds for the improper content-length problem:

  • Fix your application to ensure that the value of the Content-Length response-header field is correct.

  • Configure the browser or client emulator to send HTTP/1.0 requests without the Connection: keep-alive request-header field.

A.1.7 Browser Presenting a Page Not Displayed Error

Client browsers return an error saying that a page cannot be displayed.

Problem

Microsoft Internet Explorer has known issues with trying to reuse SSL connections after they have timed out. Due to this limitation, users connecting to a Web site using Internet Explorer 5.5 or later release with the following Oracle Web Cache configuration conditions, may receive an error saying that the page cannot be displayed:

  • Oracle Web Cache has at least one listener port to set to accept HTTPS client requests

  • The network connection keep-alive timeout is set a value more than 0.

Please see the following related sections:

  • Section 5.4 for further information about configuring Oracle Web Cache to accept HTTPS requests

  • Section 2.11.5 for further information about configuring the keep-alive timeout

  • Section 5.4.4.1 for further information about configuring the Oracle HTTP Server to maintain keep-alive connections from Oracle Web Cache for Internet Explorer browsers

When Oracle Web Cache is configured with these settings, Internet Explorer may send HTTPS requests after Oracle Web Cache has already tried to close the connection. Then, the browser returns an error saying that the page cannot be displayed.

Solution

To correct his problem, you can upgrade all clients to use the correct Microsoft patches. For information about the Internet Explorer problem, its workarounds, and links to updates to Internet Explorer, see the following:

In configurations with public Web sites, this option may not feasible. For these configurations, the Web site administrator can either enable or disable keep-alive timeouts on HTTPS connections from Internet Explorer in the webcache.xml file. By default, Oracle Web Cache disables keep-alive for HTTPS connections from Internet Explorer.

To re-enable keep-alive connection for HTTPS requests from Internet Explorer, perform the following steps. In a cache cluster, you must perform this procedure for each cluster member:

  1. Use a text editor to open the webcache.xml file.

  2. Locate the CALYPSONETINFO element:

    <CALYPSONETINFO...KEEPALIVE4MSIE_SSL="NO".../>
    
  3. Modify the value of KEEPALIVE4MSIE_SSL attribute to YES.

  4. Save webcache.xml.

  5. Restart Oracle Web Cache with the following command:

    opmnctl restartproc ias-component=component_name
    

    This executable is found in the following directory:

    (UNIX) ORACLE_INSTANCE/bin
    (Windows) ORACLE_INSTANCE\bin
    

A.1.8 ESI Errors with IBM Websphere Application Server

Due to an incompatibility with the ESI capability token between IBM Websphere Application server and Oracle Web Cache, you may see errors.

Problem

If Oracle Web Cache is deployed to cache content for an IBM Websphere Application server, the following errors may result:

  • Error page with text "Some ESI Features in this page are not supported"

  • Images do not render

Solution

To resolve this issue

  1. In the WebSphere Application Server administrative console, navigate to Servers > Application Servers.

  2. In the Application servers page, select the server1 application server.

    server1 is the server name when IBM Websphere Application Server is installed with default options. If you specified a different name, select that name instead.

  3. In the Server Infrastructure section of the Configuration tab, select Java and Process Management, and then Process Definition.

  4. In the Additional Properties section, select Java Virtual Machine, and then Custom Properties.

  5. Click New to create an entry.

  6. In the Name field, enter com.ibm.servlet.file.esi.control.

  7. In the Value field, enter the following string:

    max-age=300, cacheid="URL",content="ESI/1.0".
    
  8. Click Apply to save the settings.

  9. Restart IBM Websphere Application Server.

A.1.9 XML Parsing Errors of webcache.xml Appears in Event Viewer

XML parsing errors related to not being able to read the webcache.xml file are displayed to the Event Viewer rather than to the screen on Windows.

A.2 Common Configuration Mistakes

Common configuration mistakes include:

  • Not mapping sites correctly to origin servers.

    When sites are not mapped, Oracle Web Cache directs requests to the default Oracle HTTP Server. Browsers may receive an HTTP 500 error code.

    Other site configuration errors include:

    • Not specifying all the site aliases

    • Misuse of the wildcard character *

    • Creating multiple site-to-server mappings for a site with multiple origin servers

    See Section 2.11.3 for further information about configuring sites.

  • Ping URL

    When configuring the ping URL, how you enter the URL depends on the origin server. For an application Web Server, enter either a relative or fully qualified URL that includes the domain name, or site name, representing the virtual host of the application Web server. For a proxy server, enter a fully qualified URL that includes the domain name, or site name, representing the virtual host of the origin server behind the proxy server. Ensure the URL is cached.

    See Section 2.11.2 for further information about configuring origin servers.

  • Running Oracle Web Cache with root privilege

    On UNIX, you must configure Oracle Web Cache to run with root privilege in the following cases:

    • Privileged port numbers less than 1024 are being used for Oracle Web Cache listening ports.

    • There are more than 1,024 file descriptors being used for connections to Oracle Web Cache.

    • The current opmnctl user does not match the configured process identity user.

    See Section 5.9 for further information about configuring origin servers.

The TRACE verbosity event-logging level can help you validate Oracle Web Cache configuration settings, such as:

  • Site resolution

  • Site-to-server mappings route to the correct origin servers

  • Compression

  • Session binding

  • Caching rules

  • ESI processing

See Section 9.3 to configure TRACE verbosity in the event logs.

A.3 Diagnosing Cache Content Results

To diagnose if caching rules are set up to serve wrong or outdated content:

  1. Determine the contents of the cache by:

    • Listing the most popular requests, either cached or not cached requests, along with the caching rules associated with cached objects. See Section 8.6.

    • Listing the contents of the cache. See Section 8.6.

    • Previewing invalidation without invalidating actual content. See Section 7.7.2 to use the Preview list of objects that match invalidation criteria option.

  2. Enable event logging with a logging level of TRACE. Then, resubmit the request.

    Trace level logging shows whether an object is cached and which caching rule it matches.

    See Section 9.3 for further information about enabling event logging.

  3. Compare the contents of the cache to the caching rules to determine discrepancies or syntax errors.

    Adjust caching rules by adding or removing rules, adjusting expression type syntax, or changing the precedence of rules.

    See Section 6.6 for further information about configuring caching rules.

  4. Enable access logging. Then, send an explicit request for the object.

    By analyzing the access log determine, you can determine if Oracle Web Cache is serving the object from its cache or is forwarding the request to the origin server.

    See Section 9.4 for further information about enabling access logging.

A.4 Diagnosing Common Edge Side Includes (ESI) Syntax Errors

The majority of ESI errors are the result of syntax errors in either the template or fragment pages. By analyzing the ESI output in the event log, you can easily diagnose most ESI syntax errors. To avoid unnecessary reporting in the event log, use a verbosity level of WARNING, as described in Section 9.3. It is also useful to display the diagnostic information and event log information in the HTML response body, as described in Section 8.3.

The following topics describe using the event log and HTML response body to diagnose template and fragment syntax errors:

A.4.1 Template Syntax Error Example

Consider a template named exlusion.html that contains syntax for a nonexistent ESI tag named <esi:exclude>:

<html><body> 
Simple inclusion test.
<esi:exclude src="/cgi-bin/esi-headers.sh?/esi/fragment1.html"/> 
</body></html> 

The response returned to the browser follows:

<HTML><HEAD><TITLE>Unsupported ESI feature</TITLE></HEAD> 
<BODY>Some ESI features on this page are not supported. 
</BODY></HTML> 

The following shows an event log excerpt that indicates a problem with the <esi:exlude> keyword:

[24/Jul/2005:16:02:12 -0500] [detail] [ecid: 25732665668,0] [client: 127.0.0.1] [host: www.company.com:80] [url: /cgi-bin/esi-headers.sh?/esi/exclusion.html] 
[24/Jul/2005:16:02:12 -0500] [error 12086] [ecid: 25732665668,0] ESI syntax error. Unrecognized keyword exclude is at line 2. 
[24/Jul/2005:16:02:12 -0500] [warning 11064] [ecid: 25732665668,0] ESI object
 www.company.com:80/cgi-bin/esi-headers.sh?/esi/exclusion.html parsing error 

A.4.2 Fragment Syntax Error Example

Consider a template named inclusion_exclusion.html that contains the following syntax for including fragment exlusion1.html. Notice that HTML does not contain any ESI exception handling tags or attributes.

<html><body> 
Simple inclusion test.
<esi:include src="/cgi-bin/esi-headers.sh?/esi/frag_exclusion.html"/> 
</body></html> 

Fragment frag_exclusion.html contains syntax for a nonexistent ESI tag named <esi:exclude>:

<esi:exclude src="/cgi-bin/esi-headers.sh?/esi/fragment1.html"/> 

The response returned to the browser follows:

<HTML><HEAD><TITLE>ESI Processing Exception</TITLE></HEAD> 
<BODY>The page caused an ESI processing exception. 
</BODY></HTML> 

The following shows an event log excerpt that indicates a problem with the <esi:exlude> keyword. As a result of this error and the fact that the ESI in the template does not specify any alternative fragment to serve, the browser is served an ESI exception.

[24/Jul/2005:16:10:40 -0500] [detail] [ecid: 25733186204,0] [client: 127.0.0.1] [host: www.company.com:80] [url: /cgi-bin/esi-headers.sh?/esi/inclusion_exclusion.html] 
[24/Jul/2005:16:10:40 -0500] [error 12086] [ecid: 25733186204,0] ESI syntax error. Unrecognized keyword exclude is at line 2. 
[24/Jul/2005:16:10:40 -0500] [warning 11064] [ecid: 25733186204,0] ESI object www.company.com:80/cgi-bin/esi-headers.sh?/esi/frag_exclusion.html parsing error 
[24/Jul/2005:16:10:40 -0500] [warning 12009] [ecid: 25733186204,0] Incorrect ESI fragment exception in ESI template www.company.com:80/cgi-bin/esi-headers.sh?/esi/inclusion_exclusion.html, fragment www.company.com:80/cgi-bin/esi-headers.sh?/esi/frag_exclusion.html 
[24/Jul/2005:16:10:40 -0500] [error 12012] [ecid: 25733186204,0] No exception handler is defined in template www.company.com:80/cgi-bin/esi-headers.sh?/esi/inclusion_exclusion.html:. 
[24/Jul/2005:16:10:40 -0500] [error 11368] [ecid: 25733186204,0] ESI exception error response is returned. 

A.4.3 Fragment Syntax Error with Exception Handling Example

Consider the same inclusion_exclusion.html template that contains the following syntax for including fragment frag_exclusion.html or alternative fragment fragment1.html. When the exlusion1.html fragment specified cannot be fetched, the fragment1.html fragment specified with the alt attribute is served in its place.

<html><body> 
Simple inclusion test.
<esi:include src="/cgi-bin/esi-headers.sh?/esi/frag_exclusion.html" alt="/esi/fragment1.html"/> 
</body></html> 

Fragment frag_exclusion.html contains syntax for a nonexistent ESI tag named <esi:exclude>:

Simple inclusion succeeded.
<esi:exclude src="/cgi-bin/esi-headers.sh?/esi/fragment1.html"/> 

Therefore, fragment fragment1.html is used instead of frag_exclusion.html as the fragment:

Simple inclusion succeeded.

The response returned to the browser follows:

<html><body> 
Simple inclusion test. Simple inclusion succeeded. 
</body></html> 

The following shows an event log excerpt that indicates a problem with the <esi:exlude> keyword. Because of the exception handling, the browser is served the alternative fragment instead of an ESI exception.

[24/Jul/2005:16:14:41 -0500] [detail] [ecid: 25733432444,0] [client: 127.0.0.1]
 [host: www.company.com:80] [url: /cgi-bin/esi-headers.sh?/esi/inclusion_exclusion.html] 
[24/Jul/2005:16:14:41 -0500] [error 12086] [ecid: 25733432444,0] ESI syntax error. Unrecognized keyword exclude is at line 2.
[24/Jul/2005:16:14:41 -0500] [warning 11064] [ecid: 25733432444,0] ESI object www.company.com:80/cgi-bin/esi-headers.sh?/esi/frag_exclusion.html parsing error 
[24/Jul/2005:16:14:41 -0500] [warning 12009] [ecid: 25733432444,0] Incorrect ESI
 fragment exception in ESI template www.company.com:80/cgi-bin/esi-headers.sh?/
esi/inclusion_exclusion.html, fragment www.company.com:80/cgi-bin/esi-headers.sh?/esi/frag_exclusion.html 

In addition to analyzing the event log for the sequence of events, you can also view the diagnostic and event log results in the HTML response. The following shows the HTML response when the string +wcdebug is appended is the URL. The template diagnostic information, TU;max-age=30+60;age=0, means the following:

  • T means this page is composed an ESI template.

  • U means this request resulted in an update of stale object.

  • max-age=30+60 means that the object is to expire in 30 seconds from population and to be removed from the cache 60 seconds from the expiration. This provides a total of 90 seconds from population.

  • age=0 in age means that 0 seconds have passed since population of the cache, meaning there are 30 seconds to expiration and 60 seconds to removal.

The fragment diagnostic information, FM;max-age=30+0;age=0, means the following:

  • F means this page is an ESI fragment.

  • U means this request resulted in a cache miss.

  • max-age=30+0 means that the object is to expire in 30 seconds from population and to be removed from the cache 0 seconds from the expiration. This provides a total of 30 seconds from population.

  • age=0 in age means that 0 seconds have passed since population of the cache, meaning there are 30 seconds to expiration and removal.

Web Cache Debug Info: Web Cache Debug Info: TU;max-age=30+60;age=0 
Simple inclusion test: Web Cache Debug Info: Web Cache Debug Info: TU;max-age=30+60;age=0 
Web Cache Debug Info: FM;max-age=30+0;age=0 

[EVENTLOG] 
[24/Jul/2005:16:17:23 -0500] [detail] [ecid: 25733598670,0] [client: 127.0.0.1] [host: www.company.com:80] [url: /cgi-bin/esi-headers.sh?/esi/inclusion_exclusion.html] 
[24/Jul/2005:16:17:23 -0500] [error 12086] [ecid: 25733598670,0] ESI syntax error. Unrecognized keyword exclude is at line 2. 
[24/Jul/2005:16:17:23 -0500] [warning 11064] [ecid: 25733598670,0] ESI object www.company.com:80/cgi-bin/esi-headers.sh?/esi/frag_exclusion.html parsing error 
[24/Jul/2005:16:17:23 -0500] [warning 12009] [ecid: 25733598670,0] Incorrect ESI
 fragment exception in ESI template www.company.com:80/cgi-bin/esi-headers.sh?/
esi/inclusion_exclusion.html, fragment www.company.com:80/cgi-bin/esi-headers.sh?/esi/frag_exclusion.html 
Simple inclusion succeeded.

A.5 Impact of HTTP Traffic Changes

When Oracle Web Cache is added to an existing application Web server environment, HTTP traffic changes effect the following aspects of the application:

  • Protocol/Hostname/Port Mapping

    To ensure traffic is directed through Oracle Web Cache, configure all absolute URLs to use the protocol, host name, and port number of Oracle Web Cache. Also, ensure the Port directive in the Oracle HTTP Server httpd.conf file specifies the Oracle Web Cache listening port.

  • SSL Processing

    Add certificate management to Oracle Web Cache, if the connection between the client and Oracle Web Cache is HTTPS, but the connection between Oracle Web Cache and the origin server is HTTP.

  • Page Delivery Timing

    For compressed pages or pages that requires processing, Oracle Web Cache waits for an entire page from the origin server before it sends it to the browser.

  • HTTP Protocol

    Oracle Web Cache transparently performs the following:

    • Oracle Web Cache upgrades and downgrades the protocol version between the origin server and client.

    • For cacheable objects, Oracle Web Cache sends content to clients with the Content-Length response header instead of chunked encoding for the initial request.

    • For cache hits, Oracle Web Cache overwrites the Content-Length response-header field whenever it is different from what the origin server sent. This feature ensures browsers receive full page content.

A.6 Need More Help?

You can find more solutions at the following locations: