C Troubleshooting Oracle HTTP Server

This appendix describes common problems that you might encounter when using Oracle HTTP Server (OHS), and explains how to solve them. It includes the following topics:

C.1 Oracle HTTP Server Unable to Start Due to Port Conflict

You can get the following error if Oracle HTTP Server cannot start due to port conflict:

[VirtualHost: main] (98)Address already in use: make_sock: could not bind to address [::]:7777

Solution

Determine what process is already using that port, and then either change the IP:port address of Oracle HTTP Server or the port of the conflicting process.

C.2 System Overloaded by Number of httpd Processes

When too many httpd processes run on a system, the response time degrades because there are insufficient resources for normal processing.

Solution

Lower the value of MaxClients to a value the machine can accommodate.

C.3 Permission Denied When Starting Oracle HTTP Server On a Port Below 1024

You will get the following error if you try to start Oracle HTTP Server on a port below 1024:

[VirtualHost: main] (13)Permission denied: make_sock: could not bind to address [::]:443

Oracle HTTP Server will not start on ports below 1024 because root privileges are needed to bind these ports.

Solution

Follow the steps in Section 4.3.2.4, "Starting Oracle HTTP Server Instances on a Privileged Port (UNIX Only)" to start Oracle HTTP Server on a Privileged Port.

C.4 Exception Thrown when Unsetting PerSetEnv and Removing Variable

If you configure mod_perl by using the Fusion Middleware Control mod_perl configuration page and try to remove a previously configured PerSetEnv variable from the Environment field, this error is thrown:

Failed to invoke operation save on MBean
oracle.as.management.mbeans.register:type=component,name=ohs1,instance=webtier
 _inst7971,Location=AdminServer
Apply failed, modify required parameters and save again. Validation of
configuration trying to apply failed 

Solution

To correct this situation:

  1. Close the pop-up error and click Revert.

  2. Remove the PerSetEnv by doing one of the following:

    • Go to the Advanced Configuration page of Fusion Middleware Control and modify the mod_perl.conf file directly.

      OR

    • Go to the DOMAIN_HOME/config/fmwconfig/components/OHS/componentName/moduleconf/mod_perl.conf and edit the configuration file directly to remove the PerSetEnv value.

  3. Restart OHS.

C.5 Using Log Files to Locate Errors

You can use the following log files to help locate errors:

C.5.1 Rewrite Log

This log file is necessary for debugging when mod_rewrite is used. The log file produces a detailed analysis of how the rewriting engine transforms requests. The RewriteLogLevel directive controls the level of detail.

C.5.2 Script Log

This log file enables you to record the input to and output from the CGI scripts. This should only be used in testing, and not for production servers.

See Also:

ScriptLog in the Apache HTTP Server documentation at:

http://httpd.apache.org/docs/current/mod/mod_cgi.html#scriptlog

C.5.3 Error Log

This log file records overall server problems. Refer to Chapter 8, "Managing Oracle HTTP Server Logs" for details on configuring and viewing error logs.

C.6 Recovering an OHS Instance on a Remote Host

If you need to recover an Oracle HTTP Server instance that is installed on a remote host (that is, a host with just managed servers but no Administration Server), you must use tar and untar; pack.sh and unpack.sh do not work in this scenario.

C.7 Oracle HTTP Server Performance Issues

The following are performance issues, along with their solutions, that you might encounter when running Oracle HTTP Server:

C.7.1 Special Runtime Files Reside on a Network File System

Oracle HTTP Server uses locks for its internal processing, which in turn use lock files. These files are created dynamically when the lock is created and are accessed every time the lock is taken or released. If these files reside on a slower file system (for example, network file system), then there could be severe performance degradation. To counter this issue:

  • On Linux

    1. In httpd.conf, change AcceptMutex fcntl to AcceptMutex sysvsem (two places).

    2. In httpd.conf, comment-out the LockFile directive (three places).

  • On Solaris

    1. In httpd.conf, change AcceptMutex fcntl to AcceptMutex pthread (two places).

    2. In httpd.conf, comment-out the LockFile directive (three places).

  • Other UNIX Platforms

    In httpd.conf, change the LockFile directive to point to a local filesystem (three places).

C.7.2 UNIX Sockets on a Network File System

mod_cgid and mod_fastcgi are not enabled by default. If enabled, these modules use UNIX sockets internally. If UNIX sockets reside on a slower file system (for example, network file system), a severe performance degradation could be observed. You can set the following directives to avoid the issue:

  • If mod_cgid is enabled, use the ScriptSock directive to place mod_cgid's UNIX socket on a local filesystem.

  • If mod_fastcgi is enabled, use the FastCgiIpcDir directive to place mod_fastcgi's UNIX sockets on a local filesystem.

    Note:

    Although it is possible to use mod_fastcgi with both worker and prefork Multi-Processing Modules (MPMs), it should only be used with the prefork MPM. For more information, see Section 6.3, "Configuring Oracle HTTP Server Performance Directives".

C.7.3 DocumentRoot on a Slow File System

If you are using mod_wl_ohs to route the requests to back-end WLS server/cluster, and the DocumentRoot is on a slower file system (for example, network file system), then every request that is routed to the backend server can experience performance issues. This can be overcome by setting WLSRequest to ON instead of SetHandler weblogic-handler.

C.8 Out of DMS Shared Memory

In some extreme configurations, you might see the following message in the OHS error log:

dms_fail_shm_expansion: out of DMS shared memory in pid XXX, disabling DMS; increase DMSProcSharedMem directive from YYY

This is because of an incorrect calculation of required shared memory for OHS DMS. This can be resolved by setting DMSProcSharedMem to a larger value than the default of 4096. Continue setting DMSProcSharedMem 50% higher until the problem is resolved. The minimum value for DMSProcSharedMem is 256 and the maximum value is 65536.

In a configuration with a very large number of virtual hosts (hundreds or thousands), if the above workaround does not work, you can instead, set the environment variable OHS_DMS_BLOCKSIZE to a large enough value that Oracle HTTP Server starts without error. The value of this variable is in kilobytes and a value of 524288 is a good starting point. If the error persists, continue to increase the value by 50% until Oracle HTTP Server starts without error.

C.9 Missing Libraries Might Cause HTTPD to Exit Without Notice

On the Windows platform, Oracle HTTP Server requires Microsoft Visual C++ run-time libraries to be installed on the system to function. If they are not installed, Oracle HTTP Server might exit with no output or generate an error dialog box, depending on registry settings.

Workaround

If Oracle HTTP Server fails to start and no error messages are logged by Oracle HTTP Server in the server error log or in logs, follow this procedure to diagnose the problem:

  1. Verify that the following run-time library is installed:

    Microsoft Visual C++ 2008 Feature Pack Redistributable Package (x64), at

    http://www.microsoft.com/en-us/download/details.aspx?id=16771

    Important:

    After installing this package, install any service packs or other fixes to this package which are recommended by Microsoft.

    If this package was not installed previously, try to start Oracle HTTP Server again after installing it.

  2. Attempt to start Oracle HTTP Server from a command prompt.

    1. Create the DOMAIN_HOME/servers/<ohs-instancename>/logs directory if it has not yet been created by Oracle HTTP Server.

    2. Try to start Oracle HTTP Server using Fusion Middleware Control or WLST.

    3. Copy the file DOMAIN_HOME/servers/<ohs-instancename>/logs/lastinvocation.log to a new file startohs.cmd.

    4. Open a new command prompt and run startohs.cmd in that new command prompt.

    5. If it generates error messages after the script runs, use those messages to diagnose the startup failure.

    6. If it does not generate error messages after the script runs, proceed to the next step.

      Note:

      Oracle HTTP Server does not function properly when started from a command prompt in this manner. This mechanism for starting Oracle HTTP Server is for diagnostic purposes only.
  3. Collect Windows error information when attempting to start Oracle HTTP Server

    Error reporting for Oracle HTTP Server library errors might be inhibited on your system. Refer to the following Microsoft article for information on verifying or correcting error reporting:

    "HOWTO: How To Change Hard Error Popup Handling in Windows NT", at

    http://support.microsoft.com/kb/128642

    If ErrorMode is set to 1, library errors might not be reported for Oracle HTTP Server. Change the mode to 0 to enable hard error pop-ups or change the mode to 2 to enable reporting to the Windows event log.

    Important:

    If this is changed to 0, set it back to the previous value immediately after collecting diagnostic information as it can result in operational problems if errors occur when the server is unattended.

    The recommended setting is 2.

    1. Change ErrorMode to the desired value.

    2. Try to start Oracle HTTP Server using Fusion Middleware Control or WLST or the startohs.cmd script which was created in the previous step.

    3. Check for diagnostic information in a hard error popup or in the Windows event log, depending on the setting of ErrorMode.

      To check the Windows event log, open Windows Event Viewer and view the logs for Event Viewer (Local) / Windows Logs / Application

      Find the event log entries written at the time of the attempt to start Oracle HTTP Server.

      Errors locating run-time libraries might be reported as SideBySide errors with event ids 33 or 35, with text describing the Oracle library attempting to load the library and the specific library being loaded. Example:

      level=Error
      Source=SideBySide
      Event Id = 33
      Activation context generation failed for "C:\work\Oracle\MD1212\ORACLE~1\bin\oraiau.dll".
      Dependent Assembly Microsoft.VC90.CRT,processorArchitecture="amd64",
      publicKeyToken="1fc8b3b9a1e18e3b",type="win32",version="9.0.21022.8" could not be found.
      Please use sxstrace.exe for detailed diagnosis.
      

      The indicated library must be installed. This example shows VC90, which is the Microsoft Visual C++ 2008 library described in step 1 above.

    4. Restore ErrorMode to the previous setting.

C.10 Using AES Encrypted Wallet with SSLFIPS

If an AES encrypted wallet is not used with SSLFIPS ON (FIPS mode on), Oracle HTTP Server may fail to start. When this issue occurs, the following message in the Oracle HTTP Server log (<OHS_instance>.log):

A full restart is needed when wallet file changed on-the-fly.

To resolve this problem, encrypt the wallet with AES, or create a new wallet that is encrypted with AES. Enabling SSLFIPS mode in Oracle HTTP Server requires a wallet created with AES encrypted (compat_v12) headers. To create a new wallet or to convert an existing wallet, see these sections in Administering Oracle Fusion Middleware

"Orapki"

"Creating an Oracle Wallet with AES Encryption"

"Converting an Existing Wallet to Use AES Encryption"