Provisioning OCI Load Balancers for Kubernetes Services of Type LoadBalancer

oci.oraclecloud.com/load-balancer-type: "lb"

Note that lb is the default value of the oci.oraclecloud.com/load-balancer-type annotation. If you do not explicitly include the annotation in the service definition, the default value of the annotation is used.

For example, consider the following configuration file, nginx_lb.yaml. It defines a deployment (kind: Deployment) for the nginx app, followed by a definition of a service of type of LoadBalancer (type: LoadBalancer) that balances http traffic on port 80 for the nginx app.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
  labels:
    app: nginx
  annotations:
    oci.oraclecloud.com/load-balancer-type: "lb"
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: nginx

The first part of the configuration file defines an Nginx deployment, requesting that it be hosted on 3 pods running the nginx:latest image, and accept traffic to the containers on port 80.

The second part of the configuration file defines the Nginx service, which uses type LoadBalancer to balance Nginx traffic on port 80 amongst the available pods.

To create the deployment and service defined in nginx_lb.yaml while connected to your Kubernetes cluster, enter the command:

kubectl apply -f nginx_lb.yaml

This command outputs the following upon successful creation of the deployment and the load balancer:

deployment "my-nginx" created
service "my-nginx-svc" created

The load balancer may take a few minutes to go from a pending state to being fully operational. You can view the current state of your cluster by entering:

kubectl get all

The output from the above command shows the current state:

NAME                                  READY     STATUS    RESTARTS   AGE
po/my-nginx-431080787-0m4m8           1/1       Running   0          3m
po/my-nginx-431080787-hqqcr           1/1       Running   0          3m
po/my-nginx-431080787-n8125           1/1       Running   0          3m

NAME               CLUSTER-IP      EXTERNAL-IP      PORT(S)        AGE
svc/kubernetes     203.0.113.1     <NONE>           443/TCP        3d
svc/my-nginx-svc   203.0.113.7     192.0.2.22       80:30269/TCP   3m

NAME                      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/my-nginx           3         3         3            3           3m

NAME                            DESIRED   CURRENT   READY     AGE
rs/my-nginx-431080787           3         3         3         3m

The output shows that the my-nginx deployment is running on 3 pods (the po/my-nginx entries), that the load balancer is running (svc/my-nginx-svc) and has an external IP (192.0.2.22) that clients can use to connect to the app that's deployed on the pods.

When you provision an Oracle Cloud Infrastructure (OCI) load balancer for a Kubernetes service of type LoadBalancer, you can configure it to terminate SSL/TLS traffic. This configuration, often referred to as frontend SSL, ensures that encrypted traffic from the internet is decrypted at the load balancer before being forwarded to worker nodes.

Implementing SSL termination at the load balancer level offloads the intensive decryption process from application pods, simplifying certificate management and improving backend performance.

Note that you can implement full point-to-point SSL encryption between clients and application pods running on worker nodes. To do so, configure SSL termination at the load balancer (as described in this section) and also implement backend SSL between the load balancer and worker nodes (see Implementing SSL/TLS between the Load Balancer and Worker Nodes (Backend SSL)).

oci.oraclecloud.com/oci-load-balancer-ssl-config: '{"CipherSuiteName":"<cipher-suite-name>", "Protocols":["<tls-version>"]}'

oci.oraclecloud.com/oci-load-balancer-ssl-config: '{"CipherSuiteName":"oci-default-http2-tls-12-13-ssl-cipher-suite-v1", "Protocols":["TLSv1.2", "TLSv1.3"]}'

When Kubernetes Engine provisions a load balancer for a Kubernetes service of type LoadBalancer, you can specify that you want to implement SSL between the load balancer and the backend servers (worker nodes) in the backend set. This configuration is known as backend SSL. To implement backend SSL, you associate an SSL certificate with the load balancer's backend set.

Note that you can implement full point-to-point SSL encryption between clients and application pods running on worker nodes. To do so, associate an SSL certificate with the load balancer's backend set (as described in this section), and also create a load balancer with SSL termination (see Terminating SSL/TLS at the Load Balancer (Frontend SSL)).

While frontend SSL termination secures the connection between the client and the load balancer, backend SSL secures the connection between the load balancer and worker nodes. This configuration ensures that traffic remains encrypted even as it travels through the internal Virtual Cloud Network (VCN), providing an additional layer of defense-in-depth.

When you implement backend SSL, the load balancer decrypts incoming traffic using a frontend certificate, performs any necessary Layer 7 inspections or routing, and then re-encrypts the traffic before sending it to the destination pods. This is a common requirement for high-security environments, where cleartext traffic is restricted at every hop.

oci.oraclecloud.com/oci-load-balancer-backendset-ssl-config: '{"CipherSuiteName":"<cipher-suite-name>", "Protocols":["<tls-version>"]}'

oci.oraclecloud.com/oci-load-balancer-backendset-ssl-config: '{"CipherSuiteName":"oci-default-http2-tls-12-13-ssl-cipher-suite-v1", "Protocols":["TLSv1.2", "TLSv1.3"]}'

The shape of an Oracle Cloud Infrastructure load balancer specifies its maximum total bandwidth (that is, ingress plus egress). By default, load balancers are created with a shape of 100Mbps. Other shapes are available, including 400Mbps and 8000Mbps.

To specify an alternative shape for a load balancer, add the following annotation in the metadata section of the manifest file:

service.beta.kubernetes.io/oci-load-balancer-shape: <value>

where value is the bandwidth of the shape (for example, 100Mbps, 400Mbps, 8000Mbps).

For example:

apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
  labels:
    app: nginx
  annotations:
    oci.oraclecloud.com/load-balancer-type: "lb"
    service.beta.kubernetes.io/oci-load-balancer-shape: 400Mbps
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: nginx

Sufficient load balancer quota must be available in the region for the shape you specify. Enter the following kubectl command to confirm that load balancer creation did not fail due to lack of quota:

kubectl describe service <service-name>

Note that Oracle recommends you implement Kubernetes services of type LoadBalancer as cost-efficient flexible load balancers rather than as fixed-shape (dynamic) load balancers (see Specifying Flexible Load Balancer Shapes).

The shape of an Oracle Cloud Infrastructure load balancer specifies its maximum total bandwidth (that is, ingress plus egress). As described in Specifying Alternative Load Balancer Shapes, you can specify different load balancer shapes.

In addition, you can also specify a flexible shape for an Oracle Cloud Infrastructure load balancer, by defining a minimum and a maximum bandwidth for the load balancer.

To specify a flexible shape for a load balancer, add the following annotations in the metadata section of the manifest file:

service.beta.kubernetes.io/oci-load-balancer-shape: "flexible"
service.beta.kubernetes.io/oci-load-balancer-shape-flex-min: "<min-value>"
service.beta.kubernetes.io/oci-load-balancer-shape-flex-max: "<max-value>"

where:

• "<min-value>" is the minimum bandwidth for the load balancer, in Mbps (for example, "10")
• "<max-value>" is the maximum bandwidth for the load balancer, in Mbps (for example, "100")

Note that you do not include a unit of measurement when specifying bandwidth values for flexible load balancer shapes (unlike for pre-defined shapes). For example, specify the minimum bandwidth as 10 rather than as 10Mbps.

For example:

apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
  labels:
    app: nginx
  annotations:
    oci.oraclecloud.com/load-balancer-type: "lb"
    service.beta.kubernetes.io/oci-load-balancer-shape: "flexible"
    service.beta.kubernetes.io/oci-load-balancer-shape-flex-min: "10"
    service.beta.kubernetes.io/oci-load-balancer-shape-flex-max: "100"
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: nginx

When provisioning an Oracle Cloud Infrastructure load balancer for a Kubernetes service of type LoadBalancer, you can specify the maximum idle time (in seconds) allowed between two successive receive or two successive send operations between the client and backend servers.

To explicitly specify a maximum idle time, add the following annotation in the metadata section of the manifest file:

service.beta.kubernetes.io/oci-load-balancer-connection-idle-timeout: <value>

where value is the number of seconds.

For example:

apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
  labels:
    app: nginx
  annotations:
    oci.oraclecloud.com/load-balancer-type: "lb"
    service.beta.kubernetes.io/oci-load-balancer-connection-idle-timeout: 100
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: nginx

Note that if you don't explicitly specify a maximum idle time, a default value is used. The default value depends on the type of listener:

• for TCP listeners, the default maximum idle time is 300 seconds
• for HTTP listeners, the default maximum idle time is 60 seconds

When provisioning an Oracle Cloud Infrastructure load balancer for a Kubernetes service of type LoadBalancer, you can specify the maximum number of simultaneous connections that the load balancer is allowed to make to backend servers (worker nodes) in the backend set.

To explicitly specify the maximum number of simultaneous connections that are allowed between the load balancer and each individual backend server in the backend set, add the following annotation in the metadata section of the manifest file:

oci-load-balancer.oraclecloud.com/backendset-backend-max-connections: "<value>"

where <value> is a valid value for the number of allowed connections, as follows:

• A number between "256" and "65535" (inclusive). If you specify a number outside of this range (apart from 0), an error is returned.
• "0". If you specify 0 as the maximum number of allowed connections, an infinite number of connections between the load balancer and each individual backend server in the backend set are allowed.

For example:

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  annotations:
    oci-load-balancer.oraclecloud.com/backendset-backend-max-connections: "256"
spec:
  selector:
    app: nginx
  type: LoadBalancer
  ports:
    - name: http
      port: 80
      targetPort: 80

Note the following:

• The number you specify for the annotation is not only used as the value of the backend set's backendMaxConnections property, but is also inherited by each individual backend server as the value of its maxConnections property. As a result, the number you specify is used as the number of connections that are allowed between the load balancer and each individual backend server in the backend set, and not as the overall total number of connections that are allowed between the load balancer and the backend set.
• If you do not include the annotation in the manifest (or subsequently remove the annotation), an infinite number of connections between the load balancer and each backend server are allowed.
• For related information about specifying the maximum number of concurrent connections to the load balancer's listener that are allowed from the same originating IP address, see Specifying Max Listener Connection Rules.

When Kubernetes Engine provisions an Oracle Cloud Infrastructure load balancer for a Kubernetes service of type LoadBalancer, you can specify the maximum number of concurrent connections to the load balancer's listener that are allowed from the same originating IP address. You use an annotation to define a default maximum listener connections value that applies to all IP addresses. The annotation defines a maximum listener connection rule for the load balancer. If you don't set this default maximum value, there's no limit to the number of connections that an IP address can make to the listener.

For more information about maximum listener connection rules, see Max Listener Connection Rules in the OCI Load Balancer documentation.

To specify the maximum number of concurrent listener connections that are allowed from the same originating IP address, add the oci.oraclecloud.com/oci-load-balancer-rule-sets annotation in the metadata section of the manifest file. Note that you specify the value of the annotation as formatted JSON, as follows:

oci.oraclecloud.com/oci-load-balancer-rule-sets: |
  {
    "<rule-set-name>": {
      "items": [
        {
          "action": "IP_BASED_MAX_CONNECTIONS",
          "maxConnectionsPerIp": <value>
        }
      ]
    }
  }

where:

• <rule-set-name> is your choice of name for the rule set (between 1 and 32 characters in length). For example, IpLimitRuleSet .
• <value> is the maximum number of simultaneous listener connections that are allowed from the same originating IP address.

For example:

apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
  labels:
    app: nginx
  annotations:
    oci.oraclecloud.com/load-balancer-type: "lb"
    oci.oraclecloud.com/oci-load-balancer-rule-sets: |
      {
        "IpLimitRuleSet": {
          "items": [
            {
              "action": "IP_BASED_MAX_CONNECTIONS",
              "maxConnectionsPerIp": 20
            }
          ]
        }
      }
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: nginx

This configuration creates a maximum listener connection rule set named IpLimitRuleSet that restricts each originating IP address to a maximum of 20 concurrent connections.

To subsequently change the maximum number of concurrent connections, update the oci.oraclecloud.com/oci-load-balancer-rule-sets annotation and re-apply the manifest.

When Kubernetes Engine provisions a load balancer for a Kubernetes service of type LoadBalancer, you can specify the maximum size of the HTTP header that is accepted by the load balancer's listeners. You specify the maximum size of the HTTP header by using an annotation to specify the size (in KB) of the buffer used to read the header. The allowed values for the buffer size are 8, 16, 32 and 64. The annotation defines an HTTP header rule for the load balancer. The HTTP header rule applies to all of the load balancer's listeners, on all ports.

For more information about HTTP header rules, see HTTP Header Rules in the OCI Load Balancer documentation.

To specify the maximum size of the HTTP header accepted by the load balancer's listeners, add the oci.oraclecloud.com/oci-load-balancer-rule-sets annotation in the metadata section of the manifest file. Note that you specify the value of the annotation as formatted JSON, as follows:

oci.oraclecloud.com/oci-load-balancer-rule-sets:  |
      {
        "<rule-set-name>": {
          "items": [
            {
              "action": "HTTP_HEADER",
              "httpLargeHeaderSizeInKB": <value>
            }
          ]
        }
      }

where:

• <rule-set-name> is your choice of name for the rule set (between 1 and 32 characters in length). For example, header-size .
• <value> is the maximum size of the HTTP header in KB, and is one of 8, 16, 32, or 64 .

For example:

apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
  labels:
    app: nginx
  annotations:
    oci.oraclecloud.com/load-balancer-type: "lb"
    oci.oraclecloud.com/oci-load-balancer-rule-sets: |
      {
        "header-size": {
          "items": [
            {
              "action": "HTTP_HEADER",
              "httpLargeHeaderSizeInKB": 16
            }
          ]
        }
      }
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: nginx

To subsequently change the maximum size of the HTTP header, update the oci.oraclecloud.com/oci-load-balancer-rule-sets annotation and re-apply the manifest.

When Kubernetes Engine provisions a load balancer for a Kubernetes service of type LoadBalancer, you can define the type of traffic accepted by the listener by specifying the protocol on which the listener accepts connection requests.

Note that if you don't explicitly specify a protocol, "TCP" is used as the default value.

To explicitly specify the load balancer listener protocol when Kubernetes Engine provisions a load balancer for a Kubernetes service of type LoadBalancer, add the following annotation in the metadata section of the manifest file:

service.beta.kubernetes.io/oci-load-balancer-backend-protocol: <value>

where <value> is the protocol that defines the type of traffic accepted by the listener. For example, "HTTP". Valid protocols include "HTTP", "TCP", and "GRPC".

For example:

apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
  labels:
    app: nginx
  annotations:
    oci.oraclecloud.com/load-balancer-type: "lb"
    service.beta.kubernetes.io/oci-load-balancer-backend-protocol: "HTTP"
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: nginx

Note that if you specify GRPC as the protocol, you must configure both of the following:

• Frontend SSL, using the service.beta.kubernetes.io/oci-load-balancer-ssl-ports, oci.oraclecloud.com/oci-load-balancer-ssl-listener-config, and service.beta.kubernetes.io/oci-load-balancer-tls-secret annotations. For more information, see Terminating SSL/TLS at the Load Balancer (Frontend SSL).
• Backend SSL, using the oci.oraclecloud.com/oci-load-balancer-backendset-ssl-config and service.beta.kubernetes.io/oci-load-balancer-tls-backendset-secret annotations. For more information, see Implementing SSL/TLS between the Load Balancer and Worker Nodes (Backend SSL).

For example:

apiVersion: v1
kind: Service
metadata:
  name: hello-grpc-service
  annotations:
    oci.oraclecloud.com/load-balancer-type: "lb"
    service.beta.kubernetes.io/oci-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/oci-load-balancer-tls-secret: ssl-certificate-secret
    service.beta.kubernetes.io/oci-load-balancer-backend-protocol: "GRPC"
    oci.oraclecloud.com/oci-load-balancer-listener-ssl-config: '{"CipherSuiteName":"oci-default-http2-ssl-cipher-suite-v1", "Protocols":["TLSv1.2"]}'
    oci.oraclecloud.com/oci-load-balancer-backendset-ssl-config: '{"CipherSuiteName":"oci-default-http2-ssl-cipher-suite-v1", "Protocols":["TLSv1.2"]}'
    service.beta.kubernetes.io/oci-load-balancer-tls-backendset-secret: ca-ser-secret
spec:
  type: LoadBalancer
  selector:
    app: hello-grpc
  ports:
    - port: 443
      name: grpc
      targetPort: 50051

You can use the security list management feature to configure how security list rules are managed for an Oracle Cloud Infrastructure load balancer that Kubernetes Engine provisions for a Kubernetes service of type LoadBalancer. This feature is useful if you are new to Kubernetes, or for basic deployments.

To specify how the Kubernetes security list management feature manages security lists when Kubernetes Engine provisions a load balancer for a Kubernetes service of type LoadBalancer, add the following annotation in the metadata section of the manifest file:

service.beta.kubernetes.io/oci-load-balancer-security-list-management-mode: <value>

where <value> is one of:

• "None": (recommended) No security list management is enabled. You have to set up a security rule that allows inbound traffic to the appropriate ports for node port ranges, the kube-proxy health port, and the health check port ranges. Additionally, you have to set up security rules to allow inbound traffic to load balancers (see Security Rules for Load Balancers and Network Load Balancers).
• "All": (default) All required security list rules for load balancer services are managed.
• "Frontend": Only security list rules for ingress to load balancer services are managed. You have to set up a security rule that allows inbound traffic to the appropriate ports for node port ranges, the kube-proxy health port, and the health check port ranges.

Oracle recommends that you explicitly set service.beta.kubernetes.io/oci-load-balancer-security-list-management-mode to None.

In clusters with managed nodes, if you don't explicitly specify a management mode or you specify an invalid value, all security list rules are managed (equivalent to "All"). Be aware that in this case, Kubernetes Engine creates a security rule that allows inbound traffic from 0.0.0.0/0 (or from the source port ranges specified in the manifest file) to listener ports. In clusters with virtual nodes, security list management is never enabled and you always have to manually configure security rules (equivalent to "None").

Note that there are limits to the number of ingress and egress rules that are allowed in a security list (see Security List Limits). If the number of ingress or egress rules exceeds the limit, and <value> is set to "All" or "Frontend", creating or updating the load balancer fails.

For example:

apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
  labels:
    app: nginx
  annotations:
    oci.oraclecloud.com/load-balancer-type: "lb"
    service.beta.kubernetes.io/oci-load-balancer-security-list-management-mode: "Frontend"
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: nginx