Setting Up Load Balancing by Using Keepalived

Installing and Configuring Keepalived

Before you can set up load balancing by using Keepalived, you must install and configure the feature.

Install the keepalived package on each server:
```
sudo dnf install keepalived
```
Edit /etc/keepalived/keepalived.conf to configure Keepalived on each server. See Keepalived Configuration Directives.
Enable IP forwarding in /etc/sysctl.conf:
```
net.ipv4.ip_forward = 1
```
Verify that the IP forwarding has been applied:
```
sudo sysctl -p
```
```
net.ipv4.ip_forward = 1
```

Add firewall rules to accept VRRP communication by using the multicast IP address 224.0.0.18 and the VRRP protocol (112) on each network interface that Keepalived controls, for example:

sudo firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 \
  --in-interface enp0s8 --destination 224.0.0.18 --protocol vrrp -j ACCEPT

sudo firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 \
  --out-interface enp0s8 --destination 224.0.0.18 --protocol vrrp -j ACCEPT

sudo firewall-cmd --reload

Enable and start the keepalived service on each server:
```
sudo systemctl enable --now keepalived
```
If you change the Keepalived configuration, reload the keepalived service:
```
sudo systemctl reload keepalived
```

Keepalived Configuration Directives

The /etc/keepalived/keepalived.conf configuration file is divided into the following sections:

global_defs: Defines global settings such as the email addresses for sending notification messages, the IP address of an SMTP server, the timeout value for SMTP connections in seconds, a string that identifies the host machine, the VRRP IPv4 and IPv6 multicast addresses, and whether SNMP traps are enabled.
static_ipaddress
static_routes: Define static IP addresses and routes, which VRRP can't change. These sections aren't required if the addresses and routes are already defined on the servers and these servers already have network connectivity.
vrrp_sync_group: Defines a VRRP synchronization group of VRRP instances that fail over together.
vrrp_instance: Defines a moveable virtual IP address for a member of a VRRP synchronization group's internal or external network interface, which follows other group members during a state transition. Each VRRP instance must have a unique value of virtual_router_id, which identifies which interfaces on the primary and backup servers can be assigned a specified virtual IP address. You can also specify scripts that are run on state transitions to BACKUP, MASTER, and FAULT, and whether to trigger SMTP alerts for state transitions.
vrrp_script: Defines a tracking script that Keepalived can run at regular intervals to perform monitoring actions from a vrrp_instance or vrrp_sync_group section.
virtual_server_group: Defines a virtual server group, through a real server can be configured to be a member of several virtual server groups.
virtual_server: Defines a virtual server for load balancing, which is composed of several real servers.

For more information about setting up load balancing with Keepalived, see Setting Up Load Balancing by Using Keepalived

Setting Up Load Balancing in NAT Mode

The following example shows how you would use Keepalived in NAT mode to implement a basic failover and load balancing configuration on two servers. One server acts as the primary, the other acts as a backup, with the primary server having a higher priority than the backup server. Both servers use VRRP to monitor the current routing state. For more information about VRRP, see Using Keepalived With VRRP.

Each of the servers has two network interfaces, where one interface is connected to the an external network (192.168.1.0/24). The other interface is connected to an internal network (10.0.0.0/24), on which two web servers are accessible.

The following figure shows that the Keepalived primary server has the following network addresses: 192.168.1.10, 192.168.1.1 (virtual), 10.0.0.10, and 10.0.0.100 (virtual).

The Keepalived backup server has he following network addresses: 192.168.1.11 and 10.0.0.11.

For IP addresses, websrv1 has 10.0.0.71 and websrv2 has 10.0.0.72.

Figure 3-1 Keepalived Configuration for Load Balancing in NAT Mode

The figure shows that the Keepalived primary server has network addresses 192.168.1.10, 192.168.1.1 (virtual), 10.0.0.10, and 10.0.0.100 (virtual). The Keepalived backup server has the network addresses 192.168.1.11 and 10.0.0.11. For IP addresses, websrv1 has 10.0.0.71 and websrv2 has 10.0.0.72.

The following is an example of the configuration in the /etc/keepalived/keepalived.conf file on the primary server:

global_defs {
   notification_email {
     root@example.com
   }
   notification_email_from srv1@example.com
   smtp_server localhost
   smtp_connect_timeout 30
}

vrrp_sync_group VRRP1 {
#   Group the external and internal VRRP instances so they fail over together
    group {
        external
        internal
        }
}

vrrp_instance external {
    state MASTER
    interface enp0s8
    virtual_router_id 91
    priority 200
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1215
    }
#   Define the virtual IP address for the external network interface
    virtual_ipaddress {
        192.168.1.1/24
    }
}

vrrp_instance internal {
    state MASTER
    interface enp0s9
    virtual_router_id 92
    priority 200
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1215
    }
#   Define the virtual IP address for the internal network interface
    virtual_ipaddress {
        10.0.0.100/24
    }
}

# Define a virtual HTTP server on the virtual IP address 192.168.1.1
virtual_server 192.168.1.1 80 {
    delay_loop 10
    protocol TCP
#   Use round-robin scheduling in this example
    lb_algo rr
#   Use NAT to hide the back-end servers
    lb_kind NAT
#   Persistence of client sessions times out after 2 hours
    persistence_timeout 7200
    real_server 10.0.0.71 80 {
        weight 1
        TCP_CHECK {
          connect_timeout 5
          connect_port 80
        }
    }

    real_server 10.0.0.72 80 {
       weight 1
       TCP_CHECK {
          connect_timeout 5
          connect_port 80
        }
    }
}

The previous configuration includes both a vrrp_sync_group section so that the network interfaces are assigned together on failover, and a virtual_server section to define the real backend servers that Keepalived uses for load balancing. The value of lb_kind is set to use NAT, which means the Keepalived server handles both inbound and outbound network traffic from and to the client on behalf of the backend servers.

The configuration of the backup server is the same, except for the values of notification_email_from, state, priority, and possibly interface, if the system hardware configuration is different:

global_defs {
   notification_email {
     root@example.com
   }
   notification_email_from srv2@example.com
   smtp_server localhost
   smtp_connect_timeout 30
}

vrrp_sync_group VRRP1 {
#   Group the external and internal VRRP instances so they fail over together
    group {
        external
        internal
        }
}

vrrp_instance external {
    state BACKUP
    interface enp0s8
    virtual_router_id 91
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1215
    }
#   Define the virtual IP address for the external network interface
    virtual_ipaddress {
        192.168.1.1/24
    }
}

vrrp_instance internal {
    state BACKUP
    interface enp0s9
    virtual_router_id 92
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1215
    }
#   Define the virtual IP address for the internal network interface
    virtual_ipaddress {
        10.0.0.100/24
    }
}

# Define a virtual HTTP server on the virtual IP address 192.168.1.1
virtual_server 192.168.1.1 80 {
    delay_loop 10
    protocol TCP
#   Use round-robin scheduling in this example
    lb_algo rr
#   Use NAT to hide the back-end servers
    lb_kind NAT
#   Persistence of client sessions times out after 2 hours
    persistence_timeout 7200
 
    real_server 10.0.0.71 80 {
        weight 1
        TCP_CHECK {
          connect_timeout 5
          connect_port 80
        }
    }

    real_server 10.0.0.72 80 {
        weight 1
        TCP_CHECK {
          connect_timeout 5
          connect_port 80
        }
    }
}

The following configuration changes are required:

Configure the firewall rules on each Keepalived server (primary and backup) that you are configuring as a load balancer. See Configuring Firewall Rules for Keepalived NAT-Mode Load Balancing.
Configure a default route for the virtual IP address of the load balancer's internal network interface on each backend server that you intend to use with the Keepalived load balancer. See Configuring Backend Server Routing for Keepalived NAT-Mode Load Balancing.

See Installing and Configuring Keepalived for more information.

Configuring Firewall Rules for Keepalived NAT-Mode Load Balancing

If you configure Keepalived to use NAT mode for load balancing with the servers on the internal network, the Keepalived server handles all inbound and outbound network traffic and hides the existing backend servers by rewriting the source IP address of the real backend server in outgoing packets with the virtual IP address of the external network interface.

The following example shows how to move interface enp0s9 to the internal zone, while interface enp0s8 remains in the public zone.

To configure a Keepalived server to use NAT mode for load balancing:

Check the state of any active firewall zones on the system:

sudo firewall-cmd --get-active-zones

public
  interfaces: enp0s8 enp0s9

Configure the firewall so that the interfaces on the external network side are in a zone that's different from the interfaces on the internal network side.

sudo firewall-cmd --zone=public --remove-interface=enp0s9

sudo firewall-cmd --zone=internal --add-interface=enp0s9

sudo firewall-cmd --permanent --zone=public --remove-interface=enp0s9

sudo firewall-cmd --permanent --zone=internal --add-interface=enp0s9

Confirm that the changes have been applied:

sudo firewall-cmd --get-active-zones

internal
  interfaces: enp0s9
public
  interfaces: enp0s8

Configure NAT mode (masquerading) on the external network interface, for example:
```
sudo firewall-cmd --zone=public --add-masquerade
```
```
sudo firewall-cmd --permanent --zone=public --add-masquerade
```
Optionally, you can query each NAT mode to ensure that both of them have been set correctly. The query for public zone would return a yes response, and the query for internal zone would return a no response.
```
sudo firewall-cmd --zone=public --query-masquerade
```
```
sudo firewall-cmd --zone=internal --query-masquerade
```

If not already enabled for the firewall, configure forwarding rules between the external and internal network interfaces, for example:

sudo firewall-cmd --direct --permanent --add-rule ipv4 filter FORWARD 0 \
  -i enp0s8 -o enp0s9 -m state --state RELATED,ESTABLISHED -j ACCEPT

sudo firewall-cmd --direct --permanent --add-rule ipv4 filter FORWARD 0 \
  -i enp0s9 -o enp0s8 -j ACCEPT

sudo firewall-cmd --direct --permanent --add-rule ipv4 filter FORWARD 0 \
  -j REJECT --reject-with icmp-host-prohibited

sudo firewall-cmd --reload

Enable access to the services or ports that you want Keepalived to handle.

Configuring Backend Server Routing for Keepalived NAT-Mode Load Balancing

On each backend real servers that you intend to use with the Keepalived load balancer, ensure that the routing table contains a default route for the virtual IP address of the load balancer's internal network interface.

For example, if the virtual IP address is 10.0.0.100, use the ip command to examine the routing table:

sudo ip route show

10.0.0.0/24 dev enp0s8  proto kernel  scope link  src 10.0.0.71

You can also use the ip command to add the default route, and then confirm the changes:

sudo ip route add default via 10.0.0.100 dev enp0s8
sudo ip route show

default via 10.0.0.100 dev enp0s8 
10.0.0.0/24 dev enp0s8  proto kernel  scope link  src 10.0.0.71

To make the default route for enp0s8 persist across sysem reboots, create the /etc/sysconfig/network-scripts/route-enp0s8 file:

sudo echo "default via 10.0.0.100 dev enp0s8" |sudo tee /etc/sysconfig/network-scripts/route-enp0s8

Enhancing Load Balancing by Using Keepalived With HAProxy

You can use Keepalived to provide failover services for backup routers, while at the same time also using HAProxy for load balancing and to achieve high availability across distributed servers. The advantage of this approach is that the packet and application layers are separated, which means that the health checks that are performed by Keepalived for the load-balancing servers aren't impacted by the inbound HTTP or TCP traffic that HAproxy is managing. Also, failover routing, which is achieved by using VRRP, can be activated automatically without waiting for a client response to time out. To learn more about the usefulness of VRRP, see Using Keepalived With VRRP.

The practicality of using this method is that if the public-facing HAProxy load balancer goes offline, Keepalived automatically detects this event and dynamically switches to another HAProxy server. If the Keepalived primary router goes offline, the VRRP settings that you configured ensure that traffic is automatically handled by the Keepalived backup router.

The role of HAProxy in the setup is to provide inbound load balancing and session persistence to the backend servers: Keepalived is solely responsible for monitoring the status of HAProxy and providing an alternative routing mechanism. Using both tools in combination provides a highly available and resilient load-balancing solution.

The instructions in the following example are similar to those in Setting Up Load Balancing in NAT Mode. However, here HAProxy is installed on both the Keepalived primary server and the Keepalived backup server.

The external virtual IP address is 192.168.1.1 on the 192.168.1.0/24 external network. This IP address is dynamically assigned through NAT between the Keepalived primary server whose IP address is 192.168.1.10, and the Keepalived backup server, whose external IP address is 192.168.1.11.

The internal network is hosted on the 10.0.0.0/24 subnet. For IP addresses, websvr1 has 10.0.0.71 while websvr2 has 10.0.0.72.

Figure 3-2 Keepalived and HAProxy Configuration for High Availability Load Balancing

The figure shows that the Keepalived primary server running HAProxy has network addresses 192.168.1.10, 10.0.0.10, and 10.0.0.100 (virtual). The Keepalived backup server also running HAProxy has the network addresses 192.168.1.11 and 10.0.0.11, and it will become the new primary router if the current primary router becomes unavailable. For IP addresses, websvr1 has 10.0.0.71 while websvr2 has 10.0.0.72.

The following example shows the configuration in the /etc/keepalived/keepalived.conf file on the primary server

global_defs {
 
  notification_email {
    root@example.com
  }
 
  notification_email_from srv1@example.com
  smtp_server localhost
  smtp_connect_timeout 30
}
 
vrrp_sync_group vg1 {
  group {
    external
    internal
  }
}
 
vrrp_script chk_haproxy {
  script "killall -0 haproxy" # check the haproxy process
  interval 2 # every 2 seconds
  weight 2 # add 2 points if OK
}
 
vrrp_instance external {
  state MASTER
  interface enp0s8
  virtual_router_id 91
  priority 200
  advert_int 1
  authentication {
    auth_type PASS
    auth_pass 1215
  }
  virtual_ipaddress {
    192.168.1.1/24
  }
  track_script {
    chk_haproxy
  }
}
 
vrrp_instance internal {
  state MASTER
  interface enp0s9
  virtual_router_id 92
  priority 200
  advert_int 1
  authentication {
    auth_type PASS
    auth_pass 1215
  }
  virtual_ipaddress {
    10.0.0.100/24
  }
}

In the previous example, the configuration for the backup Keepalived server is identical, but the state value must be set to BACKUP. You don't need to set up a virtual_server because in this scenario, Keepalived is only used to route traffic, not to perform load balancing.

For more information about configuring Keepalived and setting the appropriate firewall rules, see Setting Up Load Balancing by Using Keepalived.

The HAProxy settings are configured in the /etc/haproxy/haproxy.cfg file. The settings are identical for both HAProxy installations because Keepalived dynamically routes from one configuration to the other automatically, as needed:

global
  daemon
  log 127.0.0.1 local0 debug
  maxconn 4000
  nbproc 1
 
defaults
  mode          http
  retries       3
  timeout connect 5s
  timeout client 25s
  timeout server 25s
  timeout queue 10s

listen http-incoming
  mode http
  bind internal-server-ip:80
  option http-server-close
  option forwardfor
  default_backend app

backend app
  balance roundrobin
  option httpchk HEAD / HTTP/1.1\r\nHost:\ localhost
  option httpclose
  option forwardfor
  server websrv1 192.168.1.71:80 weight 1 maxconn 512 check
  server websrv2 192.168.1.72:80 weight 1 maxconn 512 check

In the previous example the option http-server-close and option httpclose options are used to stop idle connections. This configuration shows the round-robin, load-balancing strategy. If no option is specified, then HAProxy defaults to using the option http-keep-alive option, which keeps any new connections open until every request and response journey that is associated with them is processed.

For more information about configuring HAProxy and setting the appropriate firewall rules, see Setting Up Load Balancing by Using HAProxy.