15 Diagnosing and Troubleshooting Problems

This chapter describes the methods and information sources you can use for diagnosing and solving problems that you might encounter while using Oracle Traffic Director.

This chapter contains the following sections:

15.1 Roadmap for Troubleshooting Oracle Traffic Director

This section provides the sequence of tasks you can perform to diagnose and solve problems with Oracle Traffic Director.

  1. Verify whether the system configuration is correct.

    For information about the supported platforms and operating systems, see the Oracle Fusion Middleware Supported System Configurations at:

    http://www.oracle.com/technetwork/middleware/ias/downloads/fusion-certification-100350.html

  2. Look for a solution to the problem in Section 15.2, "Solutions to Common Errors."

  3. Check whether the information in Section 15.3, "Frequently Asked Questions" helps you understand or solve the problem.

  4. Try to diagnose the problem.

    1. Review the messages logged in the server log. Look for messages of type WARNING, ERROR, and INCIDENT_ERROR.

      For messages of type WARNING and ERROR, try to solve the problem by following the directions, if any, in the error message.

      An INCIDENT_ERROR message indicates a serious problem caused by unknown reasons. You should contact Oracle for support.

    2. Increase the verbosity of the server log, and try to reproduce the problem.

      Oracle Traffic Director supports several log levels for the server log, as described in Table 11-1, "Server Log Levels". The default log level is NOTIFICATION:1. The least verbose log level is INCIDENT_ERROR, at which only serious error messages are logged. At the TRACE:1, TRACE:16, or TRACE:32 levels, the logs are increasingly verbose, but provide more detailed information, which can be useful for diagnosing problems.

      Increase the log verbosity and then try to reproduce the problem. When the problem occurs again, review the messages logs for pointers to the cause of the problem.

      For information about changing the server log level, see Section 11.3, "Configuring Log Preferences."

  5. Contact Oracle for support, as described in Section 15.4, "Contacting Oracle for Support."

15.1.1 Troubleshooting High Availability Configuration Issues

This section provides information about the tasks you can perform to diagnose and solve problems with an Oracle Traffic Director high availability configuration.

  • The Oracle Traffic Director configuration must be deployed on two nodes. For more information, see Section 13.2, "Creating and Managing Failover Groups."

  • The router ID for each failover group has to be unique.

  • Make sure that KeepAlived is installed. In most cases KeepAlived software is installed by default on both the Exalogic compute nodes (or VMs) where Oracle Traffic Director instances are running. To check if KeepAlived is installed, run the following command:

    rpm -qa | grep keepalived
    

    If KeepAlived is correctly installed, an output similar to the following is displayed:

    keepalived-1.2.2-1.el5
    

    Note that if KeepAlived is not installed, the RPM can be found in the software repository. For more information about KeepAlived, see Section 13.2.1, "How Failover Works."

  • For KeepAlived specific information, check the logs in the /var/log/messages.

  • Make sure to provide the correct VIP address and the appropriate subnet mask (netmask) bit-size for successfully completing the high availability configuration. In addition, ensure that you provide the netmask bits and not the actual netmask value. For more information, see Section 13.2.3, "Creating Failover Groups."

15.2 Solutions to Common Errors

This section provides solutions to the following problems:

15.2.1 Startup failure: could not bind to port

This error occurs when one or more HTTP listeners in the configuration are assigned to a TCP port number that is already in use by another process.

[ERROR:32] startup failure: could not bind to port port (Address already in use)
[ERROR:32] [OTD-10380] http-listener-1: http://host:port: Error creating socket (Address already in use)
[ERROR:32] [OTD-10376] 1 listen sockets could not be created
[ERROR:32] server initialization failed

You can find out the process that is listening on a given port by running the following command:

> netstat -npl | grep :port | grep LISTEN

If the configured HTTP listener port is being used by another process, then either free the port or change it as described in Section 9.3, "Modifying a Listener."

15.2.2 Unable to start server with HTTP listener port 80

This error occurs if you configure an HTTP listener port up to 1024 (say 80) and attempt to start the Oracle Traffic Director instance as a non-root user.

The following messages are written to the server log:

[ERROR:32] [OTD-10376] 1 listen sockets could not be created
[ERROR:32] [OTD-10380] http-listener-1: http://soa.example.com:80:
 Error creating socket (No access rights)

Port numbers up to 1024 are assigned by the Internet Assigned Numbers Authority (IANA) to various services. These port numbers are accessible only by the root user.

To solve this problem, you can do one of the following:

  • Configure the Oracle Traffic Director listener with a port number higher than 1024 (say, 8080), and create an IP packet-filtering rule to internally redirect requests received at port 80 to the configured Oracle Traffic Director port, as shown in the following examples:

    # /sbin/iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8080
    # /sbin/iptables -t nat -A PREROUTING -p udp -m udp --dport 80 -j REDIRECT --to-ports 8080
    

    Make sure that the iptables service is started by default when the server restarts by running the chkconfig command, as shown in the following example:

    # chkconfig --level 35 iptables on
    
  • If xinetd is installed in the system, create a file (named otd, for example) in the /etc/xinetd.d/ directory with the following entry:

    service otd
    {
    type = UNLISTED
    disable = no
    socket_type = stream
    protocol = tcp
    user = root
    wait = no
    port = 80
    redirect = 127.0.0.1 8080
    }
    

    This entry redirects all incoming TCP traffic received on port 80 to port 8080 on the local machine.

    For more information, see the Linux xinetd documentation.

15.2.3 Oracle Traffic Director consumes excessive memory at startup

When you start an Oracle Traffic Director instance, the values for certain parameters—maximum number of keep-alive connections, size of the connection queue, and maximum number of connections to origin servers—are assigned automatically based on the system's file descriptor limit.

If the file descriptor limit is very high, the auto-assigned values for undefined parameters can be needlessly high, causing Oracle Traffic Director to consume an excessive amount of memory. To avoid this problem, explicitly configure the maximum number of keep-alive connections (Section 14.5.3, "Tuning Keep-Alive Settings"), the size of the connection queue (Section 14.3.4, "Tuning the Thread Pool and Connection Queue Settings"), and the maximum number of connections to individual origin servers (Section 6.3, "Modifying an Origin Server").

15.2.4 Operating system error: Too many open files in system

This operating system error occurs in Linux when the number of allocated file descriptors reaches the limit for the system.

The following message is written to the server log:

[ERROR:16] [OTD-10546] Insufficient file descriptors for optimum configuration.

To avoid this error, increase the file descriptor limit on Linux from the default of 1024 to a reasonable number. For more information, see Section 14.2, "Tuning the File Descriptor Limit."

15.2.5 Unable to stop instance after changing the temporary directory

This error occurs when, after changing the temporary directory for a configuration, you deploy the change without stopping the instances, and then attempt to stop the instances later. The temporary directory is the directory (on the administration node) in which the process ID and socket information for the instances of the configuration are stored.

When this error occurs, the following message is written to the server log:

OTD-63585 An error occurred while stopping the server. For details, see the server log.

To Avoid This Error

If you change the temporary directory for a configuration, you should first stop all the instances of the configuration, deploy the changes, and then start the instances.

To Solve This Problem

Kill the Oracle Traffic Director instance.

  1. Find out the current temporary directory for the configuration by doing one of the following:

    • Run the otd_getConfigurationProperties WLST command, as shown in the following example:

      props = {}
      props['configuration'] = 'foo'
      otd_getConfigurationProperties(props)
      
      temp-path=/tmp/net-test-a46e5844
      ...
      
    • Log in to Fusion Middleware Control, select the required configuration, and select Advanced Settings. On the resulting page, look for the Temporary Directory field.

    Note the path to the temporary directory.

  2. Find out the process ID of the running instance by running the following command:

    cat temp_dir/pid
    

    temp_dir is the full path to the temporary directory that you noted in step 1.

    Note the process ID that this command returns.

  3. Kill the process, by running the following command:

    kill pid
    

    pid is the process ID that you noted in step 2.

15.2.6 Unable to restart the administration server

In Linux systems, the cron script tmpwatch, located at /etc/cron.daily/tmpwatch, is set to execute everyday by default. This script removes all files that are older than 240 hours (10 days) from all /tmp directories in the administration server. Hence, if the administration server is not restarted for more than 10 days, the default pid file is removed. This in turn prevents the administration server from being restarted after 10 days.

To Avoid This Problem

  • Change temp-path location: In the file, <otd-home>/admin-server/config/server.xml, change the temp-path value to a location where the server user has exclusive rights. For example, change it to, <temp-path>/var/tmp/https-test-1234</temp-path>. In addition, make sure that the new temp-path is not being monitored by the tmpwatch script.

  • Change the cron script: Remove the value 240 /tmp from the cron script for tmpwatch. Use the -X/--exclude-pattern option to exclude a directory from being monitored by tmpwatch. For more information about using this option, see the man-page for tmpwatch.

15.2.7 Oracle Traffic Director does not maintain session stickiness

Oracle Traffic Director can maintain session stickiness as follows:

Cookie Based Session Persistence

This is a common scenario where clients accept cookies from web or application servers. In this scenario, Oracle Traffic Director, while load balancing HTTP traffic, ensures session persistence using its own cookie. This ensures that sticky requests, requests containing HTTP Session cookie, are routed to the same back-end application server where this session cookie originated.

Oracle Traffic Director 11. 1.1.5 needs to be explicitly configured to honor session persistence when a back-end application server uses HTTP Session cookie other than the default JSESSIONID. On the other hand, Oracle Traffic Director 11. 1.1.6 honors session persistence on receiving any cookie from the origin server.

Note:

Oracle Traffic Director needs additional patches within WebLogic 10.3.x to maintain URI based session stickiness.

URI Based Session Persistence

This is not a very common scenario. In this case, cookies are disabled on clients and back-end web or application servers maintain session persistence by appending HTTP session information to the URI.

In this scenario, Oracle Traffic Director can honor session persistence if the back-end application server appends Oracle Traffic Director's JRoute cookie to the URI. Origin servers like WebLogic Server 10.3.6.2 and higher, 12.1 and higher, and GlassFish 2.0 and higher have the ability to append this JRoute cookie to the URI. Hence, Oracle Traffic Director is able to maintain URI based session persistence only with these origin servers.

15.3 Frequently Asked Questions

This section contains the following subsections:

15.3.1 What is a "configuration"?

A configuration, in Oracle Traffic Director terminology, is a collection of configurable elements (metadata) that determine the run-time behavior of an Oracle Traffic Director instance.

For more information, see Section 1.4, "Oracle Traffic Director Terminology."

15.3.2 How do I access Fusion Middleware Control?

See Section 1.7.2, "Displaying Fusion Middleware Control."

15.3.3 Why do I see a certificate warning when I access Fusion Middleware Control for the first time?

The browser displays a warning because the administration server has a self-signed certificate. To proceed, you should choose to trust the certificate.

15.3.4 Can I manually edit configuration files?

The files in the configuration store are updated automatically when you edit a configuration by using either Fusion Middleware Control or the WLST. Unless otherwise instructed in the Oracle Traffic Director documentation, DO NOT edit the files in the configuration store manually.

For the configuration changes to take effect, you should activate the configuration to the instances as described in Section 3.3, "Activate Configuration Changes."

15.3.5 In Fusion Middleware Control, what is the difference between saving a configuration and deploying it?

When you save a configuration, the changes you made are saved in the configuration store on the administration server. For the changes to take effect in the instances of the configuration, you must activate the configuration as described in Section 3.3, "Activate Configuration Changes."

15.3.6 Why is the "Deployment Pending" message displayed in Fusion Middleware Control?

The Deployment Pending message is displayed in Fusion Middleware Control when you change a configuration and save it on the administration server. It indicates that the changes are yet to be copied over to the instances of the configuration.

If you have finished making the required configuration changes, you can deploy the changes to all of the instances by clicking Deploy Changes in Fusion Middleware Control or by running the activate WLST command, as described in Section 3.3, "Activate Configuration Changes."

15.3.7 Why is the "Instance Configuration Deployed" message is displayed in Fusion Middleware Control?

The Instance Configuration Deployed message is displayed in Fusion Middleware Control when you manually edit the configuration files of an instance. It indicates that the configuration files of one or more instances are different from the corresponding configuration files stored in the configuration store on the administration server.

15.3.8 Why does Fusion Middleware Control session end abruptly?

If an Fusion Middleware Control session remains inactive for 30 minutes, it ends automatically. You should log in again.

15.3.9 How do I access the WLST?

See Section 1.7.1, "Accessing WebLogic Scripting Tool."

15.3.10 Why is a certificate warning message displayed when I tried to access the WLST for the first time?

The WLST connects to the SSL port of the administration server. The administration server has a self-signed certificate. The message that you see when you connect to the administration server for the first time is a prompt to choose whether you trust the certificate. Make sure that you are connecting to the correct server and port, and enter y to trust the certificate. For subsequent invocations of the CLI, the warning message is not displayed.

15.3.11 How do I find out the short names for the options of a WLST command?

See help for the command, by running the command with the --help option.

15.3.12 Why am I unable to select TCP as the health-check protocol when dynamic discovery is enabled?

When dynamic discovery is enabled, Oracle Traffic Director needs to send, at a specified interval, an HTTP request containing specific headers to determine whether the origin servers specified in the pool are Oracle WebLogic Server instances and whether the servers belong to a cluster. The response to a TCP health-check request would not provide the necessary information to determine the presence of Oracle WebLogic Server instances. So when dynamic discovery is enabled, the health-check protocol must be set to HTTP.

15.3.13 After I changed the origin servers in a pool to Oracle WebLogic Servers, they are not discovered automatically, though dynamic discovery is enabled. Why?

If dynamic discovery is enabled, when the Oracle Traffic Director instance starts, it determines whether or not the configured origin server is an Oracle WebLogic Server instance.

So if you initially configured, say, an Oracle GlassFish Server instance as the origin server, then at startup, Oracle Traffic Director determines that the origin server is not an Oracle WebLogic Server instance. Subsequently, if you replace the origin server with an Oracle WebLogic Server instance, then for Oracle Traffic Director to determine afresh that the origin server is now an Oracle WebLogic Server instance, you must either restart the Oracle Traffic Director instances or reconfigure them.

If you want to change the origin servers from Oracle WebLogic Server instances to other servers, or vice versa, without restarting the instances, do the following:

  1. Create a new origin-server pool with the required origin servers, and delete the old pool. For more information, see Chapter 5, "Managing Origin-Server Pools."

  2. Update the appropriate routes to point to the new pool, as described in Section 7.4, "Configuring Routes."

  3. Reconfigure the Oracle Traffic Director instances by using the softRestart WLST command, as described in Section 4.4, "Updating Oracle Traffic Director Instances Without Restarting."

15.3.14 How do I view the request and response headers sent and received by Oracle Traffic Director?

You can enable logging of the request and response headers in the server log by modifying the appropriate route, using either Fusion Middleware Control or the WLST.

  • Using Fusion Middleware Control

    1. Log in to Fusion Middleware Control, as described in Section 1.7.2, "Displaying Fusion Middleware Control."

    2. Click the WebLogic Domain button at the upper left corner of the page.

    3. Select Administration > OTD Configurations.

      A list of the available configurations is displayed.

    4. Select the configuration for which you want to configure routes.

    5. Click the Traffic Director Configuration In the Common Tasks pane.

    6. Select Administration > Virtual Servers.

      The Virtual Servers page is displayed.

    7. In the navigation pane, expand Virtual Servers, expand the name of the virtual server for which you want to edit routes, and select Routes.

      The Routes page is displayed. It lists the routes that are currently defined for the selected virtual server.

    8. Click the Name of the route that you want to configure.

      The Route Settings page is displayed.

    9. Go to the Advanced Settings section of the Route Settings page, and scroll down to the Client Configuration for Connections with Origin Servers subsection.

    10. Select the Log Headers check box.

    11. Click OK.

  • Using WLST

    Run the otd_setRouteProperties command, as shown in the following example:

    props = {}
    props['configuration'] = 'foo'
    props['virtual-server'] = 'bar'
    props['route'] = 'route-1'
    props['log-headers'] = 'true'
    otd_setRouteProperties(props)
    

    This command enables logging of the headers that Oracle Traffic Director sends to, and receives from, the origin servers associated with the route named route-1 in the virtual server bar of the configuration foo.

    For more information about the WLST commands mentioned in this section, see WebLogic Scripting Tool Command Reference for Oracle Traffic Director.

The headers are logged in the server log as shown in the following example:

[2011-11-11T03:45:00.000-08:00] [net-test] [NOTIFICATION] [OTD-11008] []
 [pid: 8184] for host 10.177.243.152 trying to OPTIONS / while trying to GET
 /favicon.ico, service-http reports: request headers sent to origin server(soa.example.com:1900) :[[
OPTIONS / HTTP/1.1
Proxy-agent: Oracle-Traffic-Director/12.2.1.0
Surrogate-capability: otd="Surrogate/1.0"
Host: dadvma0178:1900
Proxy-ping: true
X-weblogic-force-jvmid: unset
Via: 1.1 net-test
Connection: keep-aliv   e
]]
[2011-11-11T03:45:00.000-08:00] [net-test] [NOTIFICATION] [OTD-11009] []
 [pid: 8184] for host 10.177.243.152 trying to OPTIONS / while trying to GET
 /favicon.ico, service-http reports: response headers received from origin server(soa.example.com:1900) :[[
HTTP/1.1 200 OK
date: Fri, 11 Nov 2011 11:45:00 GMT
server: Apache/2.2.17 (Unix)
allow: GET,HEAD,POST,OPTIONS,TRACE
content-length: 0
keep-alive: timeout=5, max=100
connection: Keep-Alive
content-type: text/html]

15.3.15 How do I enable SSL/TLS for an Oracle Traffic Director instance?

See Section 10.1, "Configuring SSL/TLS Between Oracle Traffic Director and Clients."

15.3.16 How do I find out which SSL/TLS cipher suites are supported and enabled?

See Section 10.1.4, "Configuring SSL/TLS Ciphers for a Listener."

15.3.17 How do I view a list of installed certificates?

See Section 10.3.4, "Viewing a List of Certificates."

15.3.18 How do I issue test requests to an SSL/TLS-enabled Oracle Traffic Director instance?

Run the following command:

$ openssl s_client -host hostname -port portnumber -quiet
  • If you omit the -quiet option, information about the SSL/TLS connection—such as the server DN, certificate name, and the negotiated cipher suite—is displayed.

  • For testing with a specific cipher, specify the -cipher option.

After the SSL/TLS connection is established, enter an HTTP request, as shown in the following example.

GET /

For more information, see the s_client man page.

15.3.19 How do I analyze SSL/TLS connections?

Several tools are available to observe request and response data over SSL/TLS connections. One such tool is ssltap, which serves as a simple proxy between the client and the Oracle Traffic Director and displays information about the connections that it forwards.

Run the following command:

$ ssltap -l -s otd_host:otd_port

For example, to observe the communication between clients and the SSL/TLS-enabled Oracle Traffic Director listener soa.example.com:1905, run the following command:

$ ssltap -l -s soa.example.com:8080

The following messages are displayed:

Looking up "localhost"...
Proxy socket ready and listening

By default, ssltap listens on port 1924. Connect to https://localhost:1924 by using your browser.

You will see an output similar to the following:

Connection #1 [Tue Oct 25 04:29:46 2011]
Connected to localhost:8080
--> [
(177 bytes of 172)
SSLRecord { [Tue Oct 25 04:29:46 2011]
   type    = 22 (handshake)
   version = { 3,1 }
   length  = 172 (0xac)
   handshake {
      type = 1 (client_hello)
      length = 168 (0x0000a8)
         ClientHelloV3 {
            client_version = {3, 1}
            random = {...}
            session ID = {
                length = 0
                contents = {...}
            }
            cipher_suites[29] = {
                (0x00ff) TLS_EMPTY_RENEGOTIATION_INFO_SCSV
                (0xc00a) TLS/ECDHE-ECDSA/AES256-CBC/SHA
                (0xc014) TLS/ECDHE-RSA/AES256-CBC/SHA
                (0x0039) TLS/DHE-RSA/AES256-CBC/SHA
                (0x0038) TLS/DHE-DSS/AES256-CBC/SHA
                (0xc00f) TLS/ECDH-RSA/AES256-CBC/SHA
                (0xc005) TLS/ECDH-ECDSA/AES256-CBC/SHA
                (0x0035) TLS/RSA/AES256-CBC/SHA
                (0xc007) TLS/ECDHE-ECDSA/RC4-128/SHA
                (0xc009) TLS/ECDHE-ECDSA/AES128-CBC/SHA
                (0xc011) TLS/ECDHE-RSA/RC4-128/SHA
                (0xc013) TLS/ECDHE-RSA/AES128-CBC/SHA
                (0x0033) TLS/DHE-RSA/AES128-CBC/SHA
                (0x0032) TLS/DHE-DSS/AES128-CBC/SHA
                (0xc00c) TLS/ECDH-RSA/RC4-128/SHA
                (0xc00e) TLS/ECDH-RSA/AES128-CBC/SHA
                (0xc002) TLS/ECDH-ECDSA/RC4-128/SHA
                (0xc004) TLS/ECDH-ECDSA/AES128-CBC/SHA
                (0x0004) SSL3/RSA/RC4-128/MD5
                (0x0005) SSL3/RSA/RC4-128/SHA
                (0x002f) TLS/RSA/AES128-CBC/SHA
                (0xc008) TLS/ECDHE-ECDSA/3DES-EDE-CBC/SHA
                (0xc012) TLS/ECDHE-RSA/3DES-EDE-CBC/SHA
                (0x0016) SSL3/DHE-RSA/3DES192EDE-CBC/SHA
                (0x0013) SSL3/DHE-DSS/DES192EDE3CBC/SHA
                (0xc00d) TLS/ECDH-RSA/3DES-EDE-CBC/SHA
                (0xc003) TLS/ECDH-ECDSA/3DES-EDE-CBC/SHA
                (0xfeff) SSL3/RSA-FIPS/3DESEDE-CBC/SHA
                (0x000a) SSL3/RSA/3DES192EDE-CBC/SHA
            }
            compression[1] = {
                (00) NULL
            }
            extensions[55] = {
              extension type server_name, length [29] = {
   0: 00 1b 00 00  18 64 61 64  76 6d 61 30  31 37 38 2e  | .....soa.
  10: 75 73 2e 6f  72 61 63 6c  65 2e 63 6f  6d           | example.com
              }
              extension type elliptic_curves, length [8] = {
   0: 00 06 00 17  00 18 00 19                            | ........
              }
              extension type ec_point_formats, length [2] = {
   0: 01 00                                               | ..
              }
              extension type session_ticket, length [0]
            }
         }
   }
}

This is the SSL/TLS client hello sent from the browser to the Oracle Traffic Director instance. Note the list of cipher suites sent by the browser. These are the cipher suites that the browser is configured to handle, sorted in order of preference. The server selects one of the cipher suites for the handshake. If the server is not configured any of the cipher suites indicated by the client, the connection fails. In the above example, the session ID is empty, indicating that the browser does not have any cached SSL/TLS session with the specified server.

The Oracle Traffic Director instance's response would be similar to the following output:

<-- [
(823 bytes of 818)
SSLRecord { [Tue Oct 25 04:29:46 2011]
   type    = 22 (handshake)
   version = { 3,1 }
   length  = 818 (0x332)
   handshake {
      type = 2 (server_hello)
      length = 77 (0x00004d)
         ServerHello {
            server_version = {3, 1}
            random = {...}
            session ID = {
                length = 32
                contents = {...}
            }
            cipher_suite = (0x0035) TLS/RSA/AES256-CBC/SHA
            compression method = (00) NULL
            extensions[5] = {
              extension type renegotiation_info, length [1] = {
   0: 00                                                  | .
              }
            }
         }
      type = 11 (certificate)
      length = 729 (0x0002d9)
         CertificateChain {
            chainlength = 726 (0x02d6)
            Certificate {
               size = 723 (0x02d3)
               data = { saved in file 'cert.001' }
            }
         }
      type = 14 (server_hello_done)
      length = 0 (0x000000)
   }
}
]
--> [

The server selected the cipher suite, TLS/RSA/AES256-CBC/SHA and a session ID, which the client will include in subsequent requests.

The server also sent its certificate chain for the browser to verify. ssltap saved the certificates in the file cert.001. You can examine the certificates with any tool that can parse X.509 certificates. For example, run the following command:

$ openssl x509 -in cert.001 -text -inform DER

Note:

ssltap is a single threaded proxy server. So if you issue multiple requests through it, the requests will get serialized. If you need to analyze a specific problem with your application that only occurs on concurrent requests through SSL/TLS, try running multiple ssltap instances.

15.3.20 How do I view details of SSL/TLS communication between Oracle Traffic Director instances and Oracle WebLogic Server origin servers?

Configure SSL debugging for the Oracle WebLogic Server instance by adding the -Dssl.debug=true system property in the start script of the serve. For more information, see "SSL Debugging" in the Oracle Fusion Middleware Securing Oracle WebLogic Server.

Increase the verbosity of the Oracle Traffic Director server log by setting the log level to TRACE:32, as described in Section 11.3, "Configuring Log Preferences."

15.3.21 Why are certain SSL/TLS-enabled origin servers marked offline after health checks, even though the servers are up?

This error can occur for the following origin servers:

  • SSL/TLS-enabled origin servers that are configured in the origin-server pool by using IP addresses instead of host names.

  • Dynamically discovered, SSL/TLS-enabled Oracle WebLogic Server origin servers. Oracle Traffic Director refers to them using their IP addresses rather than the host names.

While Oracle Traffic Director refers to such origin servers by using their IP addresses, the certificates of the origin servers contain the servers' host names. So, in response to health-check requests, when the origin servers present certificates, Oracle Traffic Director attempts, unsuccessfully, to validate them. The SSL/TLS handshake fails. As a result, the health checks show such origin servers to be offline. Note that server-certificate validation is enabled by default.

If you set the server-log level to TRACE:32, you can view the message logged for this failure, as shown in the following example:

[2011-11-21T09:50:54-08:00] [net-soa] [TRACE:1] [OTD-10969] [] [pid: 22466]
 trying to OPTIONS /, service-http reports: error sending request
 (SSL_ERROR_BAD_CERT_DOMAIN: Requested domain name does not match the server's certificate.)

To solve this problem, disable validation of the origin-server certificates for the origin server, by running the otd_setOriginServerPoolSslProperties WLST command, as shown in the following example:

props = {}
props['configuration'] = 'foo'
props['origin-server-pool'] = 'origin-server-pool-1'
props['validate-server-cert'] = 'false'
otd_setOriginServerPoolSslProperties(props)

For more information about the WLST commands mentioned in this section, see WebLogic Scripting Tool Command Reference for Oracle Traffic Director.

15.3.22 Does Oracle Traffic Director rewrite the source IP address of clients before forwarding requests to the origin servers?

The default behavior of Oracle Traffic Director is to rewrite the source IP address. However, Oracle Traffic Director does send the client IP address in an additional request header Proxy-client-ip. You can set up Oracle Traffic Director to block or forward Proxy-client-ip and other request headers by configuring the appropriate route as described in Section 7.4, "Configuring Routes".

Note that Oracle Traffic Director cannot maintain case sensitivity of the HTTP request headers while forwarding them to origin servers.

15.3.23 Why does Oracle Traffic Director return a 405 status code?

If an HTTP request does not meet the conditions specified in any of the defined routes and there is no default (=unconditional) route in the configuration, then Oracle Traffic Director returns the 405 status code. This error indicates that Oracle Traffic Director did not find any valid route for the request. This situation can occur only if the default route, which is used when the request does not meet the conditions specified in any of the other routes, is deleted manually in the obj.conf configuration file. To solve this issue the administrator must create a valid route.

Note:

The default (=unconditional) route cannot be deleted through Fusion Middleware Control and the WLST, and should not be deleted manually.

15.4 Contacting Oracle for Support

If you have a service agreement with Oracle, you can contact Oracle Support (http://support.oracle.com) for help with Oracle Traffic Director problems.

Before Contacting Oracle Support

Before contacting Oracle Support, do the following:

  • Try all the appropriate diagnostics and troubleshooting guidelines described in this document Oracle Traffic Director Administrator's Guide).

  • Check whether the problem you are facing, or a similar problem, has been discussed in the OTN Discussion Forums at http://forums.oracle.com/.

    If the information available on the forum is not sufficient to help you solve the problem, post a question on the forum. Other Oracle Traffic Director users on the forum might respond to your question.

  • To the extent possible, document the sequence of actions you performed just before the problem occurred.

  • Where possible, try to restore the original state of the system, and reproduce the problem using the documented steps. This helps to determine whether the problem is reproducible or an intermittent issue.

  • If the issue can be reproduced, try to narrow down the steps for reproducing the problem. Problems that can be reproduced by small test cases are typically easier to diagnose when compared with large test cases.

    Narrowing down the steps for reproducing problems enables Oracle Support to provide solutions for potential problems faster.

Information You Should Provide to Oracle Support

When you contact Oracle for support, provide the following information.

  • The release number of Oracle Traffic Director.

  • A brief description of the problem, including the actions you performed just before the problem occurred.

  • If you need support with using the administration interfaces, the name of the command-line subcommand or the title of the Fusion Middleware Control screen for which you require help.

  • Zip file containing the configuration files for the configuration in which you encountered the error.

    INSTANCE_HOME/admin-server/config-store/config_name/current.zip
    
  • Zip file containing the configuration files for the last error-free configuration.

    INSTANCE_HOME/admin-server/config-store/config_name/backup/date_time.zip
    
  • The latest server and access log files.

    Note:

    When you send files to Oracle Support, remember to provide the MD5 checksum value for each file, so that Oracle Support personnel can verify the integrity of the files before using them for troubleshooting the problem.