Network Resource Configuration for Cluster Creation and Deployment

Before you can use Container Engine for Kubernetes to create and deploy clusters in the regions in a tenancy:

  • Within the tenancy, there must already be a compartment to contain the necessary network resources (such as a VCN, subnets, internet gateway, route table, security lists). If such a compartment does not exist already, you will have to create it. Note that the network resources can reside in the root compartment. However, if you expect multiple teams to create clusters, best practice is to create a separate compartment for each team.
  • Within the compartment, network resources (such as a VCN, subnets, internet gateway, route table, security lists) must be appropriately configured in each region in which you want to create and deploy clusters. When creating a new cluster, you can have Container Engine for Kubernetes automatically create and configure new network resources in the 'Quick Create' workflow. Alternatively, you can explicitly specify the existing network resources to use in the 'Custom Create' workflow. If you specify existing network resources, you or somebody else must have already configured those resources appropriately, as described in this topic.

This topic describes the necessary configuration for each network resource. To see details of a typical configuration, see Example Network Resource Configurations.

For an introductory tutorial, see Creating a Cluster with Oracle Cloud Infrastructure Container Engine for Kubernetes. A number of related Developer Tutorials are also available.

VCN Configuration

The VCN in which you want to create and deploy clusters must be configured as follows:

  • The VCN must have a CIDR block defined that is large enough for the number of subnets you specify for the clusters you create. For example, to create a highly available cluster in a region with three availability domains will typically require two regional subnets (recommended) or five AD-specific subnets to support the necessary number of worker nodes and load balancers. However, you can create clusters with fewer subnets. A /16 CIDR block would be large enough for almost all use cases (10.0.0.0/16 for example). The CIDR block you specify for the VCN must not overlap with the CIDR block you specify for pods and for the Kubernetes services (see CIDR Blocks and Container Engine for Kubernetes).
  • The VCN must have an appropriate number of subnets defined for worker nodes, load balancers and the Kubernetes API endpoint. Best practice is to use regional subnets to make failover across availability domains simpler to implement. See Subnet Configuration.
  • The VCN must have security lists defined for each subnet. See Security List Configuration.
  • Oracle recommends DNS Resolution is selected for the VCN.
  • If you are using public subnets, the VCN must have an internet gateway. See Internet Gateway Configuration.
  • If you are using private subnets, the VCN must have a NAT gateway and a service gateway. See NAT Gateway Configuration and Service Gateway Configuration.
  • If the VCN has a NAT gateway, service gateway, or internet gateway, it must have a route table with appropriate rules defined. See Route Table Configuration.

See VCNs and Subnets and Example Network Resource Configurations.

Internet Gateway Configuration

If you intend to use public subnets (for worker nodes, load balancers, or the Kubernetes API endpoint) and the subnets require access to/from the internet, the VCN must have an internet gateway. The internet gateway must be specified as the target for the destination CIDR block 0.0.0.0/0 as a route rule in a route table.

See VCNs and Subnets and Example Network Resource Configurations.

NAT Gateway Configuration

If you intend to use private subnets (for worker nodes or the Kubernetes API endpoint) and the subnets require access to the internet, the VCN must have a NAT gateway. For example, if you expect deployed applications to download software or to access third party services.

The NAT gateway must be specified as the target for the destination CIDR block 0.0.0.0/0 as a route rule in a route table.

See NAT Gateway and Example Network Resource Configurations.

Service Gateway Configuration

If you intend to use private subnets for worker nodes or the Kubernetes API endpoint, the VCN must have a service gateway.

Create the service gateway in the same VCN and compartment as the worker nodes subnet and the Kubernetes API endpoint subnet, and select the All <region> Services in Oracle Services Network option.

The service gateway must be specified as the target for All <region> Services in Oracle Services Network as a route rule in a route table.

See Access to Oracle Services: Service Gateway and Example Network Resource Configurations.

Route Table Configuration

Route Table for Worker Nodes Subnets

If you intend to deploy worker nodes in a public subnet, the subnet route table must have a route rule that specifies the internet gateway as the target for the destination CIDR block 0.0.0.0/0.

If you intend to deploy worker nodes in a private subnet, the subnet route table must have::

  • a route rule that specifies the service gateway as the target for All <region> Services in Oracle Services Network
  • a route rule that specifies the NAT gateway as the target for the destination CIDR block 0.0.0.0/0

Route Table for Kubernetes API Endpoint Subnets

If you intend to deploy the Kubernetes API endpoint in a public subnet, the subnet route table must have a route rule that specifies the internet gateway as the target for the destination CIDR block 0.0.0.0/0.

If you intend to deploy the Kubernetes API endpoint in a private subnet, the subnet route table must have:

  • a route rule that specifies the service gateway as the target for All <region> Services in Oracle Services Network
  • a route rule that specifies the NAT gateway as the target for the destination CIDR block 0.0.0.0/0

Route Table for Load Balancer Subnets

If you intend to deploy load balancers in public subnets, the subnet route table must have a route rule that specifies the internet gateway as the target for the destination CIDR block 0.0.0.0/0.

If you intend to deploy load balancers in private subnets, the subnet route table can be empty.

Notes about Route Table Configuration

DHCP Options Configuration

The VCN in which you want to create and deploy clusters must have DHCP Options configured. The default value for DNS Type of Internet and VCN Resolver is acceptable.

See DHCP Options and Example Network Resource Configurations.

Security List Configuration

The VCN in which you want to create and deploy clusters must have security lists defined for worker node subnets, for the Kubernetes API endpoint subnet, and for load balancer subnets (if specified). The worker node subnets, Kubernetes API endpoint subnet, and load balancer subnets have different security rule requirements, as described in this topic. You can add additional rules to meet your specific needs.

See Security Lists and Example Network Resource Configurations.

Security List for Kubernetes API Endpoint Subnet

The security list for the Kubernetes API endpoint subnet must have the following required ingress rules:

State Source Protocol/Dest. Port Description
Stateful Worker Nodes CIDR TCP/6443 Kubernetes worker to Kubernetes API endpoint communication.
Stateful Worker Nodes CIDR TCP/12250 Kubernetes worker to control plane communication.
Stateful Worker Nodes CIDR ICMP 3,4 Path Discovery.

The security list for the Kubernetes API endpoint subnet can have the following optional ingress rules:

State Source Protocol/Dest. Port Description
Stateful 0.0.0.0/0 or specific subnets TCP/6443 Client access to Kubernetes API endpoint

The security list for the Kubernetes API endpoint subnet must have the following egress rules:

State Destination Protocol/Dest. Port Description
Stateful All <region> Services in Oracle Services Network TCP/443 Allow Kubernetes control plane to communicate with OKE.
Stateful Worker Nodes CIDR TCP/ALL All traffic to worker nodes.
Stateful Worker Nodes CIDR ICMP 3,4 Path Discovery.

Security List for Worker Node Subnets

Worker nodes are created with public or private IP addresses, according to whether you specify public or private subnets when defining the node pools in a cluster. Container Engine for Kubernetes must be able to access worker nodes.

Security lists for worker node subnets must have the following required ingress rules:

State Source Protocol/Dest. Port Description
Stateful Worker Nodes CIDR ALL/ALL Allow pods on one worker node to communicate with pods on other worker nodes.
Stateful Kubernetes API Endpoint CIDR TCP/ALL Allow Kubernetes control plane to communicate with worker nodes.
Stateful 0.0.0.0/0 ICMP 3,4 Path Discovery.

Security lists for worker node subnets can have the following optional ingress rules:

State Source Protocol/Dest. Port Description
Stateful 0.0.0.0/0 or subnet CIDR TCP/22 (optional) Allow inbound SSH traffic to worker nodes.

Security lists for worker node subnets must have the following required egress rules:

State Destination Protocol/Dest. Port Description
Stateful Worker Nodes CIDR ALL/ALL Allow pods on one worker node to communicate with pods on other worker nodes.
Stateful 0.0.0.0/0 ICMP 3,4 Path Discovery.
Stateful All <region> Services in Oracle Services Network TCP/ALL Allow nodes to communicate with OKE.
Stateful Kubernetes API Endpoint CIDR TCP/6443 Kubernetes worker to Kubernetes API endpoint communication.
Stateful Kubernetes API Endpoint CIDR TCP/12250 Kubernetes worker to control plane communication.

Security lists for worker node subnets can have the following optional egress rules:

State Destination Protocol/Dest. Port Description
Stateful 0.0.0.0/0 TCP/ALL (optional) Allow worker nodes to communicate with internet.

Security List for Load Balancers

Security lists for load balancers do not require any security rules. The Kubernetes Cloud Controller Manager sets security rules automatically when Kubernetes services are deployed.

You can use the security list management feature in Kubernetes to manage security list rules yourself. See Specifying Load Balancer Security List Management Options.

Subnet Configuration

The characteristics of the cluster you want to create will determine the number of subnets to configure. Best practice is to use regional subnets to make failover across availability domains simpler to implement.

The VCN in which you want to create and deploy clusters must have at least two (optionally, three) different subnets:

  • a Kubernetes API endpoint subnet
  • a worker nodes subnet
  • (optionally) one or two load balancer subnets

Although you can choose to combine the subnets, and also to combine security rules, this approach is not recommended because it makes security harder to manage.

The subnet CIDR blocks must not overlap with CIDR blocks you specify for pods running in the cluster (see CIDR Blocks and Container Engine for Kubernetes).

The subnets must be configured with the following properties:

See VCNs and Subnets and Example Network Resource Configurations.

Kubernetes API Endpoint Subnet Configuration

The Kubernetes API endpoint subnet hosts an endpoint that provides access to the Kubernetes API on the cluster control plane.

The Kubernetes API endpoint subnet must be a regional subnet, and can be a private or a public subnet:

  • If you specify a public subnet for the Kubernetes API endpoint, you can optionally assign a public IP address to the Kubernetes API endpoint subnet. The public IP address enables third party cloud services (such as SaaS CI/CD services) to access the Kubernetes API endpoint.
  • If you specify a private subnet for the Kubernetes API endpoint, traffic can access the Kubernetes API endpoint subnet from another subnet in your VCN, from another VCN, or from your on-premise network (peered with FastConnect). You can also set up a bastion host on a public subnet to reach the Kubernetes API endpoint.

Worker Node Subnet Configuration

A worker node subnet hosts the worker nodes in a node pool.

The worker node subnet can be either a single regional subnet or multiple AD-specific subnets (one in each of the availability domains).

The worker node subnet can be either a public subnet or a private subnet. However, we recommend the worker node subnet is a private subnet for maximum security.

Load Balancer Subnet Configuration

Load balancer subnet(s) connect Oracle Cloud Infrastructure load balancers created by Kubernetes services of type LoadBalancer.

The load balancer subnets can be single regional subnets or AD-specific subnets (one in each of the availability domains). However, we recommend regional subnets.

The load balancer subnets can be either public or private subnets, depending on how applications deployed on the cluster will be accessed. If applications will be accessed internally from within your VCN, use private subnets for the load balancer subnets. If applications will be accessed from the internet, use public subnets for the load balancer subnets.