15 Diagnosing and Troubleshooting Problems

This error occurs when one or more HTTP listeners in the configuration are assigned to a TCP port number that is already in use by another process.

[ERROR:32] startup failure: could not bind to port port (Address already in use)
[ERROR:32] [OTD-10380] http-listener-1: http://host:port: Error creating socket (Address already in use)
[ERROR:32] [OTD-10376] 1 listen sockets could not be created
[ERROR:32] server initialization failed

You can find out the process that is listening on a given port by running the following command:

> netstat -npl | grep :port | grep LISTEN

If the configured HTTP listener port is being used by another process, then either free the port or change it as described in Modifying a Listener.

This error occurs if you configure an HTTP listener port up to 1024 (say 80) and attempt to start the Oracle Traffic Director instance as a non-root user.

The following messages are written to the server log:

[ERROR:32] [OTD-10376] 1 listen sockets could not be created
[ERROR:32] [OTD-10380] http-listener-1: http://soa.example.com:80:
 Error creating socket (No access rights)

Port numbers up to 1024 are assigned by the Internet Assigned Numbers Authority (IANA) to various services. These port numbers are accessible only by the root user.

To solve this problem, you can do one of the following:

• Configure the Oracle Traffic Director listener with a port number higher than 1024 (say, 8080), and create an IP packet-filtering rule to internally redirect requests received at port 80 to the configured Oracle Traffic Director port, as shown in the following examples:

  # /sbin/iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8080
  # /sbin/iptables -t nat -A PREROUTING -p udp -m udp --dport 80 -j REDIRECT --to-ports 8080
  

  Make sure that the iptables service is started by default when the server restarts by running the chkconfig command, as shown in the following example:

  # chkconfig --level 35 iptables on
  

• If xinetd is installed in the system, create a file (named otd, for example) in the /etc/xinetd.d/ directory with the following entry:

  service otd
  {
  type = UNLISTED
  disable = no
  socket_type = stream
  protocol = tcp
  user = root
  wait = no
  port = 80
  redirect = 127.0.0.1 8080
  }
  

  This entry redirects all incoming TCP traffic received on port 80 to port 8080 on the local machine.

  For more information, see the Linux xinetd documentation.

This error occurs when, for an SSL-enabled configuration, you set or change the PKCS#11 token pin, and then deploy the updated configuration while the instances are running.

The following messages are written to the server log:

[ERROR:32] [OTD-10094] NSS PKCS #11 initialization failed 
(SEC_ERROR_BAD_PASSWORD: Security password entered is incorrect.)
[ERROR:32] [OTD-10492] New configuration not installed
[ERROR:32] [OTD-10520] The new configuration is incompatible with the existing
 configuration (Enabling PKCS #11 or SSL requires a server restart)

To solve this problem, start the instance by using the start-instance CLI command or by clicking the Start/Restart Instances button in the administration console. At the resulting prompt, enter the pin for each token that is protected with a pin.

To avoid this error, after you set or change the PKCS#11 token pin for an SSL-enabled configuration, first stop the running instances, deploy the changes, and then start the instances.

This error usually occurs when the configured SNMP subagent port is being used by another process.

The following message is written to the administration server log.

OTD-63410 The SNMP subagent failed to start.

Check whether the configured port for the SNMP subagent on the node is already used by another process, by using the following command.

> netstat -npl --udp | grep :port

To solve this problem, either free the port or change it in the INSTANCE_HOME/admin-server/config/snmpagt.conf file, as described in Configuring the SNMP Subagent.

This error occurs when you run the tadm command in the following situations:

• The value specified for the --port option is not correct.

• The --port option was not specified, and the administration server is running on a port other than the default SSL port 8989.

Run the command again with the correct value for the --port option.

When you start an Oracle Traffic Director instance, the values for certain parameters—maximum number of keep-alive connections, size of the connection queue, and maximum number of connections to origin servers—are assigned automatically based on the system's file descriptor limit.

If the file descriptor limit is very high, the auto-assigned values for undefined parameters can be needlessly high, causing Oracle Traffic Director to consume an excessive amount of memory. To avoid this problem, explicitly configure the maximum number of keep-alive connections (Tuning Keep-Alive Settings), the size of the connection queue (Tuning the Thread Pool and Connection Queue Settings), and the maximum number of connections to individual origin servers (Modifying an Origin Server).

This operating system error occurs in Linux when the number of allocated file descriptors reaches the limit for the system.

The following message is written to the server log:

[ERROR:16] [OTD-10546] Insufficient file descriptors for optimum configuration.

To avoid this error, increase the file descriptor limit on Linux from the default of 1024 to a reasonable number. For more information, see Tuning the File Descriptor Limit.

This error occurs when, after changing the temporary directory for a configuration, you deploy the change without stopping the instances, and then attempt to stop the instances later. The temporary directory is the directory (on the administration node) in which the process ID and socket information for the instances of the configuration are stored.

When this error occurs, the following message is written to the server log:

OTD-63585 An error occurred while stopping the server. For details, see the server log.

To Avoid This Error

If you change the temporary directory for a configuration, you should first stop all the instances of the configuration, deploy the changes, and then start the instances.

To Solve This Problem

Kill the Oracle Traffic Director instance.

1. Find out the current temporary directory for the configuration by doing one of the following:

   • Run the get-config-prop CLI command, as shown in the following example:

     tadm> get-config-prop --config=soa temp-path
     /tmp/net-test-a46e5844
     

   • Log in to the administration console, select the required configuration, and select Advanced Settings. On the resulting page, look for the Temporary Directory field.

   Note the path to the temporary directory.

2. Find out the process ID of the running instance by running the following command:

   cat temp_dir/pid
   

   temp_dir is the full path to the temporary directory that you noted in step 1.

   Note the process ID that this command returns.

3. Kill the process, by running the following command:

   kill pid
   

   pid is the process ID that you noted in step 2.

In Linux systems, the cron script tmpwatch, located at /etc/cron.daily/tmpwatch, is set to execute everyday by default. This script removes all files that are older than 240 hours (10 days) from all /tmp directories in the administration server. Hence, if the administration server is not restarted for more than 10 days, the default pid file is removed. This in turn prevents the administration server from being restarted after 10 days.

To Avoid This Problem

• Change temp-path location: In the file, <otd-home>/admin-server/config/server.xml, change the temp-path value to a location where the server user has exclusive rights. For example, change it to, <temp-path>/var/tmp/https-test-1234</temp-path>. In addition, make sure that the new temp-path is not being monitored by the tmpwatch script.

• Change the cron script: Remove the value 240 /tmp from the cron script for tmpwatch. Use the -X/--exclude-pattern option to exclude a directory from being monitored by tmpwatch. For more information about using this option, see the man-page for tmpwatch.

Oracle Traffic Director can maintain session stickiness as follows:

Cookie Based Session Persistence

This is a common scenario where clients accept cookies from web or application servers. In this scenario, Oracle Traffic Director, while load balancing HTTP traffic, ensures session persistence using its own cookie. This ensures that sticky requests, requests containing HTTP Session cookie, are routed to the same back-end application server where this session cookie originated.

Oracle Traffic Director 11. 1.1.5 needs to be explicitly configured to honor session persistence when a back-end application server uses HTTP Session cookie other than the default JSESSIONID. On the other hand, Oracle Traffic Director 11. 1.1.6 honors session persistence on receiving any cookie from the origin server.

URI Based Session Persistence

This is not a very common scenario. In this case, cookies are disabled on clients and back-end web or application servers maintain session persistence by appending HTTP session information to the URI.

In this scenario, Oracle Traffic Director can honor session persistence if the back-end application server appends Oracle Traffic Director's JRoute cookie to the URI. Origin servers like WebLogic Server 10.3.6.2 and higher, 12.1 and higher, and GlassFish 2.0 and higher have the ability to append this JRoute cookie to the URI. Hence, Oracle Traffic Director is able to maintain URI based session persistence only with these origin servers.

Run the reset-admin-password CLI command as described in the "Changing the Administrator User Name and Password Using the CLI" section.

A configuration, in Oracle Traffic Director terminology, is a collection of configurable elements (metadata) that determine the run-time behavior of an Oracle Traffic Director instance.

For more information, see Oracle Traffic Director Terminology.

See Accessing the Administration Console.

The browser displays a warning because the administration server has a self-signed certificate. To proceed, you should choose to trust the certificate.

The files in the configuration store are updated automatically when you edit a configuration by using either the administration console or the CLI. Unless otherwise instructed in the Oracle Traffic Director documentation, DO NOT edit the files in the configuration store manually.

For the configuration changes to take effect, you should deploy the configuration to the instances as described in Deploying a Configuration.

If you inadvertently edit a configuration file of an instance, and if you want to retain the change and replicate it in all the instances of the configuration, you can pull the configuration from the instance to the administration server, as described Synchronizing Configurations Between the Administration Server and Nodes.

When you save a configuration, the changes you made are saved in the configuration store on the administration server. For the changes to take effect in the instances of the configuration, you must deploy the configuration as described in Deploying a Configuration.

The Deployment Pending message is displayed in the administration console when you change a configuration and save it on the administration server. It indicates that the changes are yet to be copied over to the instances of the configuration.

If you have finished making the required configuration changes, you can deploy the changes to all of the instances by clicking Deploy Changes in the administration console or by running the deploy-config CLI command, as described in Deploying a Configuration.

The Instance Configuration Deployed message is displayed in the administration console when you manually edit the configuration files of an instance. It indicates that the configuration files of one or more instances are different from the corresponding configuration files stored in the configuration store on the administration server. For information about what you should do in this situation, see Synchronizing Configurations Between the Administration Server and Nodes.

If an administration console session remains inactive for 30 minutes, it ends automatically. You should log in again.

See Accessing the Command-Line Interface.

For the specified CLI subcommand to be executed, it must precede all the options, including the common options like --user and --host. Otherwise, the CLI assumes that you attempting to invoke the shell mode.

The CLI connects to the SSL port of the administration server. The administration server has a self-signed certificate. The message that you see when you connect to the administration server for the first time is a prompt to choose whether you trust the certificate. Make sure that you are connecting to the correct server and port, and enter y to trust the certificate. For subsequent invocations of the CLI, the warning message is not displayed.

See help for the command, by running the command with the --help option.

Yes. Store the password in a file in the format, tadm_password=password. When you run a CLI command, specify the full path to the password file by using the --password-file option.

When dynamic discovery is enabled, Oracle Traffic Director needs to send, at a specified interval, an HTTP request containing specific headers to determine whether the origin servers specified in the pool are Oracle WebLogic Server instances and whether the servers belong to a cluster. The response to a TCP health-check request would not provide the necessary information to determine the presence of Oracle WebLogic Server instances. So when dynamic discovery is enabled, the health-check protocol must be set to HTTP.

If dynamic discovery is enabled, when the Oracle Traffic Director instance starts, it determines whether or not the configured origin server is an Oracle WebLogic Server instance.

So if you initially configured, say, an Oracle GlassFish Server instance as the origin server, then at startup, Oracle Traffic Director determines that the origin server is not an Oracle WebLogic Server instance. Subsequently, if you replace the origin server with an Oracle WebLogic Server instance, then for Oracle Traffic Director to determine afresh that the origin server is now an Oracle WebLogic Server instance, you must either restart the Oracle Traffic Director instances or reconfigure them.

If you want to change the origin servers from Oracle WebLogic Server instances to other servers, or vice versa, without restarting the instances, do the following:

1. Create a new origin-server pool with the required origin servers, and delete the old pool. For more information, see Managing Origin-Server Pools.

2. Update the appropriate routes to point to the new pool, as described in Configuring Routes.

3. Reconfigure the Oracle Traffic Director instances by using the reconfig-instance CLI command, as described in Updating Oracle Traffic Director Instances Without Restarting.

You can enable logging of the request and response headers in the server log by modifying the appropriate route, using either the administration console or the CLI.

• Using the administration console

  1. Log in to the administration console, as described in Accessing the Administration Console.

  2. Click the Configurations button that is situated at the upper left corner of the page.

     A list of the available configurations is displayed.

  3. Select the configuration for which you want to configure routes.

  4. In the navigation pane, expand Virtual Servers, expand the name of the virtual server for which you want to configure routes, and select Routes.

     The Routes page is displayed. It lists the routes that are currently defined for the selected virtual server.

  5. Click the Name of the route that you want to configure.

     The Route Settings page is displayed.

  6. Go to the Advanced Settings section of the Route Settings page, and scroll down to the Client Configuration for Connections with Origin Servers subsection.

  7. Select the Log Headers check box.

  8. Click Save.

  9. Click Deploy Changes.

• Using the CLI

  Run the set-route-prop command, as shown in the following example:

  tadm> set-route-prop --config=soa --vs=soa.example.com --route=default-route log-headers=true
  

  This command enables logging of the headers that Oracle Traffic Director sends to, and receives from, the origin servers associated with the route named default-route in the virtual server soa.example.com of the configuration soa.

  For the updated configuration to take effect, deploy it to the Oracle Traffic Director instances by using the deploy-config command.

  For more information about the CLI commands mentioned in this section, see the Oracle Traffic Director Command-Line Reference or run the commands with the --help option.

The headers are logged in the server log as shown in the following example:

[2011-11-11T03:45:00.000-08:00] [net-test] [NOTIFICATION] [OTD-11008] []
 [pid: 8184] for host 10.177.243.152 trying to OPTIONS / while trying to GET
 /favicon.ico, service-http reports: request headers sent to origin server(soa.example.com:1900) :[[
OPTIONS / HTTP/1.1
Proxy-agent: Oracle-Traffic-Director/11.1.1.7
Surrogate-capability: otd="Surrogate/1.0"
Host: dadvma0178:1900
Proxy-ping: true
X-weblogic-force-jvmid: unset
Via: 1.1 net-test
Connection: keep-aliv   e
]]
[2011-11-11T03:45:00.000-08:00] [net-test] [NOTIFICATION] [OTD-11009] []
 [pid: 8184] for host 10.177.243.152 trying to OPTIONS / while trying to GET
 /favicon.ico, service-http reports: response headers received from origin server(soa.example.com:1900) :[[
HTTP/1.1 200 OK
date: Fri, 11 Nov 2011 11:45:00 GMT
server: Apache/2.2.17 (Unix)
allow: GET,HEAD,POST,OPTIONS,TRACE
content-length: 0
keep-alive: timeout=5, max=100
connection: Keep-Alive
content-type: text/html]

See Configuring SSL/TLS Between Oracle Traffic Director and Clients.

See Configuring SSL/TLS Ciphers for a Listener.

See Viewing a List of Certificates.

Run the following command:

$ openssl s_client -host hostname -port portnumber -quiet

• If you omit the -quiet option, information about the SSL/TLS connection—such as the server DN, certificate name, and the negotiated cipher suite—is displayed.

• For testing with a specific cipher, specify the -cipher option.

After the SSL/TLS connection is established, enter an HTTP request, as shown in the following example.

GET /

For more information, see the s_client man page.

Several tools are available to observe request and response data over SSL/TLS connections. One such tool is ssltap, which serves as a simple proxy between the client and the Oracle Traffic Director and displays information about the connections that it forwards.

Run the following command:

$ ssltap -l -s otd_host:otd_port

For example, to observe the communication between clients and the SSL/TLS-enabled Oracle Traffic Director listener soa.example.com:1905, run the following command:

$ ssltap -l -s soa.example.com:8080

The following messages are displayed:

Looking up "localhost"...
Proxy socket ready and listening

By default, ssltap listens on port 1924. Connect to https://localhost:1924 by using your browser.

You will see an output similar to the following:

Connection #1 [Tue Oct 25 04:29:46 2011]
Connected to localhost:8080
--> [
(177 bytes of 172)
SSLRecord { [Tue Oct 25 04:29:46 2011]
   type    = 22 (handshake)
   version = { 3,1 }
   length  = 172 (0xac)
   handshake {
      type = 1 (client_hello)
      length = 168 (0x0000a8)
         ClientHelloV3 {
            client_version = {3, 1}
            random = {...}
            session ID = {
                length = 0
                contents = {...}
            }
            cipher_suites[29] = {
                (0x00ff) TLS_EMPTY_RENEGOTIATION_INFO_SCSV
                (0xc00a) TLS/ECDHE-ECDSA/AES256-CBC/SHA
                (0xc014) TLS/ECDHE-RSA/AES256-CBC/SHA
                (0x0039) TLS/DHE-RSA/AES256-CBC/SHA
                (0x0038) TLS/DHE-DSS/AES256-CBC/SHA
                (0xc00f) TLS/ECDH-RSA/AES256-CBC/SHA
                (0xc005) TLS/ECDH-ECDSA/AES256-CBC/SHA
                (0x0035) TLS/RSA/AES256-CBC/SHA
                (0xc007) TLS/ECDHE-ECDSA/RC4-128/SHA
                (0xc009) TLS/ECDHE-ECDSA/AES128-CBC/SHA
                (0xc011) TLS/ECDHE-RSA/RC4-128/SHA
                (0xc013) TLS/ECDHE-RSA/AES128-CBC/SHA
                (0x0033) TLS/DHE-RSA/AES128-CBC/SHA
                (0x0032) TLS/DHE-DSS/AES128-CBC/SHA
                (0xc00c) TLS/ECDH-RSA/RC4-128/SHA
                (0xc00e) TLS/ECDH-RSA/AES128-CBC/SHA
                (0xc002) TLS/ECDH-ECDSA/RC4-128/SHA
                (0xc004) TLS/ECDH-ECDSA/AES128-CBC/SHA
                (0x0004) SSL3/RSA/RC4-128/MD5
                (0x0005) SSL3/RSA/RC4-128/SHA
                (0x002f) TLS/RSA/AES128-CBC/SHA
                (0xc008) TLS/ECDHE-ECDSA/3DES-EDE-CBC/SHA
                (0xc012) TLS/ECDHE-RSA/3DES-EDE-CBC/SHA
                (0x0016) SSL3/DHE-RSA/3DES192EDE-CBC/SHA
                (0x0013) SSL3/DHE-DSS/DES192EDE3CBC/SHA
                (0xc00d) TLS/ECDH-RSA/3DES-EDE-CBC/SHA
                (0xc003) TLS/ECDH-ECDSA/3DES-EDE-CBC/SHA
                (0xfeff) SSL3/RSA-FIPS/3DESEDE-CBC/SHA
                (0x000a) SSL3/RSA/3DES192EDE-CBC/SHA
            }
            compression[1] = {
                (00) NULL
            }
            extensions[55] = {
              extension type server_name, length [29] = {
   0: 00 1b 00 00  18 64 61 64  76 6d 61 30  31 37 38 2e  | .....soa.
  10: 75 73 2e 6f  72 61 63 6c  65 2e 63 6f  6d           | example.com
              }
              extension type elliptic_curves, length [8] = {
   0: 00 06 00 17  00 18 00 19                            | ........
              }
              extension type ec_point_formats, length [2] = {
   0: 01 00                                               | ..
              }
              extension type session_ticket, length [0]
            }
         }
   }
}

This is the SSL/TLS client hello sent from the browser to the Oracle Traffic Director instance. Note the list of cipher suites sent by the browser. These are the cipher suites that the browser is configured to handle, sorted in order of preference. The server selects one of the cipher suites for the handshake. If the server is not configured any of the cipher suites indicated by the client, the connection fails. In the above example, the session ID is empty, indicating that the browser does not have any cached SSL/TLS session with the specified server.

The Oracle Traffic Director instance's response would be similar to the following output:

<-- [
(823 bytes of 818)
SSLRecord { [Tue Oct 25 04:29:46 2011]
   type    = 22 (handshake)
   version = { 3,1 }
   length  = 818 (0x332)
   handshake {
      type = 2 (server_hello)
      length = 77 (0x00004d)
         ServerHello {
            server_version = {3, 1}
            random = {...}
            session ID = {
                length = 32
                contents = {...}
            }
            cipher_suite = (0x0035) TLS/RSA/AES256-CBC/SHA
            compression method = (00) NULL
            extensions[5] = {
              extension type renegotiation_info, length [1] = {
   0: 00                                                  | .
              }
            }
         }
      type = 11 (certificate)
      length = 729 (0x0002d9)
         CertificateChain {
            chainlength = 726 (0x02d6)
            Certificate {
               size = 723 (0x02d3)
               data = { saved in file 'cert.001' }
            }
         }
      type = 14 (server_hello_done)
      length = 0 (0x000000)
   }
}
]
--> [

The server selected the cipher suite, TLS/RSA/AES256-CBC/SHA and a session ID, which the client will include in subsequent requests.

The server also sent its certificate chain for the browser to verify. ssltap saved the certificates in the file cert.001. You can examine the certificates with any tool that can parse X.509 certificates. For example, run the following command:

$ openssl x509 -in cert.001 -text -inform DER

Configure SSL debugging for the Oracle WebLogic Server instance by adding the -Dssl.debug=true system property in the start script of the serve. For more information, see "SSL Debugging" in the Oracle Fusion Middleware Securing Oracle WebLogic Server.

Increase the verbosity of the Oracle Traffic Director server log by setting the log level to TRACE:32, as described in Configuring Log Preferences.

This error can occur for the following origin servers:

• SSL/TLS-enabled origin servers that are configured in the origin-server pool by using IP addresses instead of host names.

• Dynamically discovered, SSL/TLS-enabled Oracle WebLogic Server origin servers. Oracle Traffic Director refers to them using their IP addresses rather than the host names.

While Oracle Traffic Director refers to such origin servers by using their IP addresses, the certificates of the origin servers contain the servers' host names. So, in response to health-check requests, when the origin servers present certificates, Oracle Traffic Director attempts, unsuccessfully, to validate them. The SSL/TLS handshake fails. As a result, the health checks show such origin servers to be offline. Note that server-certificate validation is enabled by default.

If you set the server-log level to TRACE:32, you can view the message logged for this failure, as shown in the following example:

[2011-11-21T09:50:54-08:00] [net-soa] [TRACE:1] [OTD-10969] [] [pid: 22466]
 trying to OPTIONS /, service-http reports: error sending request
 (SSL_ERROR_BAD_CERT_DOMAIN: Requested domain name does not match the server's certificate.)

To solve this problem, disable validation of the origin-server certificates for the required virtual-server routes, by running the set-route-prop CLI command, as shown in the following example:

tadm> set-route-prop --config=soa  --vs=vs1 --route=route1 validate-server-cert=false

For the updated route to take effect, deploy the configuration by running the deploy-config command, as shown in the following example:

tadm> deploy-config soa

For more information about the CLI commands mentioned in this section, see the Oracle Traffic Director Command-Line Reference or run the command with the --help option.

The default behavior of Oracle Traffic Director is to rewrite the source IP address. However, Oracle Traffic Director does send the client IP address in an additional request header Proxy-client-ip. You can set up Oracle Traffic Director to block or forward Proxy-client-ip and other request headers by configuring the appropriate route as described in Configuring Routes.

Note that Oracle Traffic Director cannot maintain case sensitivity of the HTTP request headers while forwarding them to origin servers.

If an HTTP request does not meet the conditions specified in any of the defined routes and there is no default (=unconditional) route in the configuration, then Oracle Traffic Director returns the 405 status code. This error indicates that Oracle Traffic Director did not find any valid route for the request. This situation can occur only if the default route, which is used when the request does not meet the conditions specified in any of the other routes, is deleted manually in the obj.conf configuration file. To solve this issue the administrator must create a valid route.