Troubleshooting Load Balancing

Learn how to troubleshoot an HTTP 502 bad gateway error for a Load Balancer resource.

In addition to monitoring and management, load balancing logging helps you to identify, isolate, and troubleshoot issues with your load balancer infrastructure. The following procedure illustrates how to troubleshoot a 502 Bad Gateway error encountered when deploying a new web application, example.com. The example uses an Oracle Cloud Infrastructure public load balancer as the front end in a development environment. The task fails with a 502 Bad Gateway error on the browser. Troubleshoot the issue using load balancer access and error logs, as follows:

1. Confirm the error using the curl utility, as follows:

   curl -v http://example.com

   > GET / HTTP/1.1
   > Host: 192.0.2.99
   > User-Agent: curl/7.54.0
   > Accept: */*
   >
   < HTTP/1.1 502 Bad Gateway
   < Content-Type: text/html
   < Content-Length: 161
   < Connection: keep-alive

2. Search the load balancer access and error logs for "lbStatusCode" and "backendStatusCode."

   • If the results include backendStatusCode: 502, then:

     Possible causes:

     • Issue is an improperly configured backend.

     • Backend is likely another reverse proxy or LB.

     Possible resolutions:

     • Examine upstream proxy logs to determine why the 502 error is being returned.

     • Resolve any issues on the ultimate backend that is causing the upstream proxy to return a 502 error.

   • If the results include backendStatusCode: 504, then:

     Possible causes:

     • When a 504 error occurs from the backend, it typically indicates that the backend is another proxy or load balancer service instance. The error typically occurs when a proxy is unable to connect to an upstream server in a specified amount of time.

     • Examine the logs of the upstream system to determine what is causing the upstream proxy from connecting to the backend.

     Possible resolutions:

     • Increase the amount of time for the connection timeout.

     • Determine why the backend is taking longer to respond than usual using a utility, such as tcpdump, and built-in application tools.

   • If the results include backendStatusCode: 500, then:

     Possible causes:

     • When a 500 error occurs from the backend, it typically indicates a server-side error, commonly known as an "Internal Server Error." Backend applications typically cause this error.

     • Inability to connect to upstream resources, such as databases, APIs, and services.

     Possible resolutions:

     Resolve application-level issue that is causing the error.

   • If the results include backendStatusCode: with no error code, then:

     • Typically, when no backend status code accompanies lbStatusCode: 502, no backend is available to send the connections.

     • You might also notice a No healthy backends available in associated backendSet message in the load balancer error Logs.

     • Ensure that the backends are healthy. If the backends are healthy, then confirm that the health check is properly configured.

Learn how to debug a timeout error associated with a backend server used by a Load Balancer resource.

When the backend server exceeds the response time when responding to a request, a 504 error occurs indicating that the backend server is either down or not responding to the request forwarded by the load balancer. The client application receives the following response code: HTTP/1.1 504 Gateway Timeout.

Errors can occur for the following reasons:

• The load balancer failed to establish a connection to the backend server before the connection timeout expired.

• The load balancer established a connection to the backend server but the backend did not respond before the idle timeout period elapsed.

• The security lists or network security groups for the subnet or the VNIC did not allow traffic from the backends to the load balancer.

• The backend server or application server failed.

Follow these steps to troubleshoot the backend server timeout errors:

1. Use the curl utility to directly test the backend server from a host in the same network.

   curl -i http://backend_ip_address

   If this test takes longer than one second to respond, an application-level issue is causing latency. Oracle recommends that you check any upstream dependencies that might cause latency, including:

   • Network attached storage such as iSCSI or NFS

   • Database latency

   • An off-premise API

   • An application tier

2. Check the application by accessing it directly from the backend server. Check its access logs to determine if the application can be accessed and is functioning properly.

3. If the load balancer and the backend server are in different subnets, then check whether the security lists contain rules to allow traffic. If no rules exist, then traffic is not allowed.

4. Enter the following commands to determine whether firewall rules exist on the backend servers that block traffic:

   iptables -L lists all firewall rules enforced by iptables

   sudo firewall-cmd --list-all lists all firewall rules enforced by firewalld

5. Enable logging on the load balancer to determine whether the load balancer or the backend server is causing the latency.

Learn how to test the TCP and HTTP backend servers used by a Load Balancer resource.

This topic describes how to troubleshoot a load balancer connection. The topology used in this procedure has a public load balancer in a public subnet and the backends are in the same subnet.

Oracle recommends that you use the Oracle Cloud Infrastructure Logging service to troubleshoot issues. (See Details for Load Balancer Logs.)

In addition to using Oracle Cloud Infrastructure logging, however, you can use other utilities listed in this section to troubleshoot the traffic that is processed by the load balancer and sent to a backend. To perform these tests, Oracle recommends that you create an instance in the same network as your load balancer and allow the traffic in the same network security groups and security lists. Use the following tools to troubleshoot:

• ping

  Before using the more advanced utilities listed here, Oracle recommends that you perform a basic ping test. For this test to succeed, you must allow ICMP traffic between the test instance and the backend.

  $ ping backend_ip_address

  The response should look similar to:

  PING 192.0.2.2 (192.0.2.2) 56(84) bytes of data.
  64 bytes from 192.0.2.2: icmp_seq=1 ttl=64 time=0.028 ms
  64 bytes from 192.0.2.2: icmp_seq=2 ttl=64 time=0.044 ms

  If you receive a message that contains "64 bytes from...", then the ping succeeded.

  Receiving a message that contains "Destination Host Unreachable" indicates that the system does not exist.

  Receiving no message indicates that the system exists but the ICMP protocol is not allowed. Check all firewalls, security lists, and network security groups to ensure ICMP is allowed.

• curl

  Use the curl utility to send HTTP requests to a specific host, port, or URL.

  • The following example shows using curl to connect to a backend that is sending a 403 Forbidden error:

    $ curl -I http://backend_ip_address/health
    HTTP/1.1 403 Forbidden
    Date: Tue, 17 Mar 2021 17:47:10 GMT
    Content-Type: text/html; charset=UTF-8
    Content-Length: 3539
    Connection: keep-alive
    Last-Modified: Tue, 10 Mar 2021 20:33:28 GMT
    ETag: "dd3-5b3c6975e7600"
    Accept-Ranges: bytes

    In the preceding example, the health check fails, returning a 403 error, indicating that the backend does not have local file permissions configured properly for the health check page.

  • The following example shows using curl to connect to a backend that is sending a 404 Not Found error:

    $ curl -I http://backend_ip_address/health
    HTTP/1.1 404 Not Found
    Date: Tue, 17 Mar 2021 17:47:10 GMT
    Content-Type: text/html; charset=UTF-8
    Content-Length: 3539
    Connection: keep-alive
    Last-Modified: Tue, 10 Mar 2021 20:33:28 GMT
    ETag: "dd3-5b3c6975e7600"
    Accept-Ranges: bytes

    In the preceding example, the health check fails, returning a 404 error, indicating that the health check page does not exist in the expected location.

  • The following example shows a backend that exists and either a network security group, the security lists, or a local firewall is blocking the traffic:

    $ curl -I backend_ip_address
    curl: (7) Failed connect to backend_ip_address:port; Connection refused

  • The following example shows a backend that does not exist:

    $ curl -I backend_ip_address
    curl: (7) Failed connect to backend_ip_address:port; No route to host

• Netcat

  Netcat is a networking utility for reading from and writing to network connections using TCP or UDP.

  • The following example shows using the netcat utility at the TCP level to ensure that the destination backend server can receive a connection:

    $ nc -vz backend_ip_address port
    Ncat: Connected to backend_ip_address:port.

    In the preceding example, port is open for connections.

  • $ nc -vn backend_ip_address port
    Ncat: Connection timed out.

    In the preceding example, port is closed.

• Tcpdump

  Use the tcpdump utility to capture all traffic to a backend to ensure which traffic is coming from a load balancer and what is being returned to the load balancer.

  sudo tcpdump -i any -A port port src load_balancer_ip_address
  11:25:54.799014 IP 192.0.2.224.39224 > 192.0.2.224.80: Flags [P.], seq 1458768667:1458770008, ack 2440130792, win 704, options [nop,nop,TS val 461552632 ecr 208900561], length 1341: HTTP: POST /health HTTP/1.1

• OpenSSL

  When troubleshooting SSL issues between the load balancer instance and the backend servers, Oracle recommends using the openssl utility. This utility opens an SSL connection to a specific host name and port, and prints the SSL certificate and other parameters.

  Other options for troubleshooting issues are:

  • -showcerts

    This option prints all certificates in the certificate chain presented by the backend server. Use this option to identify issues, such as a missing intermediate certificate authority certificate.

  • -cipher cipher_name

    This option forces the client and server use a specific cipher suite and helps to rule out whether the backend is allowing specific ciphers.

• Netstat

  Use the netstat -natp command to ensure that the application running on the backend server is up and running. For TCP or HTTP traffic, the backend application, IP address, and port must all be in listen mode. If the application port on the backend server is not in listen mode, then the TCP port of the application is not up.

  To resolve this issue, ensure that the application is up and running by either restarting the application or the backend server.

Learn about common load balancer errors associated with the Load Balancing service.

Common load balancer errors include, series 500 and series 400 errors, health check errors, client errors, and SSL errors. The subsequent topics in this section describe these common errors and detail troubleshooting procedures for each, where applicable.