16.2 Solutions to Common Errors

This section provides solutions to the following problems:

16.2.1 Startup failure: could not bind to port

This error occurs when one or more HTTP listeners in the configuration are assigned to a TCP port number that is already in use by another process.

[ERROR:32] startup failure: could not bind to port port (Address already in use)
[ERROR:32] [OTD-10380] http-listener-1: http://host:port: Error creating socket (Address already in use)
[ERROR:32] [OTD-10376] 1 listen sockets could not be created
[ERROR:32] server initialization failed

You can find out the process that is listening on a given port by running the following command:

> netstat -npl | grep :port | grep LISTEN

If the configured HTTP listener port is being used by another process, then either free the port or change it as described in Section 10.3, "Modifying a Listener."

16.2.2 Unable to start server with HTTP listener port 80

This error occurs if you configure an HTTP listener port up to 1024 (say 80) and attempt to start the Oracle Traffic Director instance as a non-root user.

The following messages are written to the server log:

[ERROR:32] [OTD-10376] 1 listen sockets could not be created
[ERROR:32] [OTD-10380] http-listener-1: http://soa.example.com:80:
 Error creating socket (No access rights)

Port numbers up to 1024 are assigned by the Internet Assigned Numbers Authority (IANA) to various services. These port numbers are accessible only by the root user.

To solve this problem, you can do one of the following:

  • Configure the Oracle Traffic Director listener with a port number higher than 1024 (say, 8080), and create an IP packet-filtering rule to internally redirect requests received at port 80 to the configured Oracle Traffic Director port, as shown in the following examples:

    # /sbin/iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8080
    # /sbin/iptables -t nat -A PREROUTING -p udp -m udp --dport 80 -j REDIRECT --to-ports 8080
    

    Make sure that the iptables service is started by default when the server restarts by running the chkconfig command, as shown in the following example:

    # chkconfig --level 35 iptables on
    
  • If xinetd is installed in the system, create a file (named otd, for example) in the /etc/xinetd.d/ directory with the following entry:

    service otd
    {
    type = UNLISTED
    disable = no
    socket_type = stream
    protocol = tcp
    user = root
    wait = no
    port = 80
    redirect = 127.0.0.1 8080
    }
    

    This entry redirects all incoming TCP traffic received on port 80 to port 8080 on the local machine.

    For more information, see the Linux xinetd documentation.

16.2.3 Unable to restart SSL/TLS-enabled server after changing the PKCS#11 token pin

This error occurs when, for an SSL-enabled configuration, you set or change the PKCS#11 token pin, and then deploy the updated configuration while the instances are running.

The following messages are written to the server log:

[ERROR:32] [OTD-10094] NSS PKCS #11 initialization failed 
(SEC_ERROR_BAD_PASSWORD: Security password entered is incorrect.)
[ERROR:32] [OTD-10492] New configuration not installed
[ERROR:32] [OTD-10520] The new configuration is incompatible with the existing
 configuration (Enabling PKCS #11 or SSL requires a server restart)

To solve this problem, start the instance by using the start-instance CLI command or by clicking the Start/Restart Instances button in the administration console. At the resulting prompt, enter the pin for each token that is protected with a pin.

To avoid this error, after you set or change the PKCS#11 token pin for an SSL-enabled configuration, first stop the running instances, deploy the changes, and then start the instances.

16.2.4 Unable to start the SNMP subagent

This error usually occurs when the configured SNMP subagent port is being used by another process.

The following message is written to the administration server log.

OTD-63410 The SNMP subagent failed to start.

Check whether the configured port for the SNMP subagent on the node is already used by another process, by using the following command.

> netstat -npl --udp | grep :port

To solve this problem, either free the port or change it in the INSTANCE_HOME/admin-server/config/snmpagt.conf file, as described in Section 13.6.2, "Configuring the SNMP Subagent."

16.2.5 Unable to communicate with the administration server: connection refused

This error occurs when you run the tadm command in the following situations:

  • The value specified for the --port option is not correct.

  • The --port option was not specified, and the administration server is running on a port other than the default SSL port 8989.

Run the command again with the correct value for the --port option.

16.2.6 Oracle Traffic Director consumes excessive memory at startup

When you start an Oracle Traffic Director instance, the values for certain parameters—maximum number of keep-alive connections, size of the connection queue, and maximum number of connections to origin servers—are assigned automatically based on the system's file descriptor limit.

If the file descriptor limit is very high, the auto-assigned values for undefined parameters can be needlessly high, causing Oracle Traffic Director to consume an excessive amount of memory. To avoid this problem, explicitly configure the maximum number of keep-alive connections (Section 15.2.3.3), the size of the connection queue (Section 15.2.1.4), and the maximum number of connections to individual origin servers (Section 7.3).

16.2.7 Operating system error: Too many open files in system

This operating system error occurs in Linux when the number of allocated file descriptors reaches the limit for the system.

The following message is written to the server log:

[ERROR:16] [OTD-10546] Insufficient file descriptors for optimum configuration.

To avoid this error, increase the file descriptor limit on Linux from the default of 1024 to a reasonable number. For more information, see Section 15.3, "Tuning the File Descriptor Limit."

16.2.8 Unable to stop instance after changing the temporary directory

This error occurs when, after changing the temporary directory for a configuration, you deploy the change without stopping the instances, and then attempt to stop the instances later. The temporary directory is the directory (on the administration node) in which the process ID and socket information for the instances of the configuration are stored.

When this error occurs, the following message is written to the server log:

OTD-63585 An error occurred while stopping the server. For details, see the server log.

To Avoid This Error

If you change the temporary directory for a configuration, you should first stop all the instances of the configuration, deploy the changes, and then start the instances.

To Solve This Problem

Kill the Oracle Traffic Director instance.

  1. Find out the current temporary directory for the configuration by doing one of the following:

    • Run the get-config-prop CLI command, as shown in the following example:

      tadm> get-config-prop --config=soa temp-path
      /tmp/net-test-a46e5844
      
    • Log in to the administration console, select the required configuration, and select Advanced Settings. On the resulting page, look for the Temporary Directory field.

    Note the path to the temporary directory.

  2. Find out the process ID of the running instance by running the following command:

    cat temp_dir/pid
    

    temp_dir is the full path to the temporary directory that you noted in step 1.

    Note the process ID that this command returns.

  3. Kill the process, by running the following command:

    kill pid
    

    pid is the process ID that you noted in step 2.

16.2.9 Unable to restart the administration server

In Linux systems, the cron script tmpwatch, located at /etc/cron.daily/tmpwatch, is set to execute everyday by default. This script removes all files that are older than 240 hours (10 days) from all /tmp directories in the administration server. Hence, if the administration server is not restarted for more than 10 days, the default pid file is removed. This in turn prevents the administration server from being restarted after 10 days.

To Avoid This Problem

  • Change temp-path location: In the file, <otd-home>/admin-server/config/server.xml, change the temp-path value to a location where the server user has exclusive rights. For example, change it to, <temp-path>/var/tmp/https-test-1234</temp-path>. In addition, make sure that the new temp-path is not being monitored by the tmpwatch script.

  • Change the cron script: Remove the value 240 /tmp from the cron script for tmpwatch. Use the -X/--exclude-pattern option to exclude a directory from being monitored by tmpwatch. For more information about using this option, see the man-page for tmpwatch.

16.2.10 Oracle Traffic Director does not maintain session stickiness

Oracle Traffic Director can maintain session stickiness as follows:

Cookie Based Session Persistence

This is a common scenario where clients accept cookies from web or application servers. In this scenario, Oracle Traffic Director, while load balancing HTTP traffic, ensures session persistence using its own cookie. This ensures that sticky requests, requests containing HTTP Session cookie, are routed to the same back-end application server where this session cookie originated.

Oracle Traffic Director 11. 1.1.5 needs to be explicitly configured to honor session persistence when a back-end application server uses HTTP Session cookie other than the default JSESSIONID. On the other hand, Oracle Traffic Director 11. 1.1.6 honors session persistence on receiving any cookie from the origin server.

Note:

Oracle Traffic Director needs additional patches within WebLogic 10.3.x to maintain URI based session stickiness.

URI Based Session Persistence

This is not a very common scenario. In this case, cookies are disabled on clients and back-end web or application servers maintain session persistence by appending HTTP session information to the URI.

In this scenario, Oracle Traffic Director can honor session persistence if the back-end application server appends Oracle Traffic Director's JRoute cookie to the URI. Origin servers like WebLogic Server 10.3.6.2 and higher, 12.1 and higher, and GlassFish 2.0 and higher have the ability to append this JRoute cookie to the URI. Hence, Oracle Traffic Director is able to maintain URI based session persistence only with these origin servers.