Network Resource Configuration for Cluster Creation and Deployment

Before you can use Container Engine for Kubernetes to create and deploy clusters in the regions in a tenancy:

  • Within the tenancy, there must already be a compartment to contain the necessary network resources (such as a VCN, subnets, internet gateway, route table, security lists and/or network security groups). If such a compartment does not exist already, you will have to create it. Note that the network resources can reside in the root compartment. However, if you expect multiple teams to create clusters, best practice is to create a separate compartment for each team.
  • Within the compartment, network resources (such as a VCN, subnets, internet gateway, route table, security lists and/or network security groups) must be appropriately configured in each region in which you want to create and deploy clusters. When creating a new cluster, you can have Container Engine for Kubernetes automatically create and configure new network resources in the 'Quick Create' workflow. Alternatively, you can explicitly specify the existing network resources to use in the 'Custom Create' workflow. If you specify existing network resources, you or somebody else must have already configured those resources appropriately, as described in this topic.

This topic describes the necessary configuration for each network resource. To see details of a typical configuration, see Example Network Resource Configurations.

For an introductory tutorial, see Creating a Cluster with Oracle Cloud Infrastructure Container Engine for Kubernetes. A number of related Developer Tutorials are also available.

VCN Configuration

The VCN in which you want to create and deploy clusters must be configured as follows:

  • The VCN must have a CIDR block defined that is large enough for the number of subnets you specify for the clusters you create. For example, to create a highly available cluster in a region with three availability domains will typically require two regional subnets (recommended) or five AD-specific subnets to support the necessary number of worker nodes and load balancers. However, you can create clusters with fewer subnets. A /16 CIDR block would be large enough for almost all use cases (10.0.0.0/16 for example). The CIDR block you specify for the VCN must not overlap with the CIDR block you specify for pods and for the Kubernetes services (see CIDR Blocks and Container Engine for Kubernetes).
  • The VCN must have an appropriate number of subnets defined for worker nodes, load balancers and the Kubernetes API endpoint. Best practice is to use regional subnets to make failover across availability domains simpler to implement. See Subnet Configuration.
  • The VCN must have security rules defined (in either or both security lists for each subnet and/or network security groups). See Security Rule Configuration in Security Lists and/or Network Security Groups.
  • Oracle recommends DNS Resolution is selected for the VCN.
  • If you are using public subnets, the VCN must have an internet gateway. See Internet Gateway Configuration.
  • If you are using private subnets, the VCN must have a NAT gateway and a service gateway. See NAT Gateway Configuration and Service Gateway Configuration.
  • If the VCN has a NAT gateway, service gateway, or internet gateway, it must have a route table with appropriate rules defined. See Route Table Configuration.

See VCNs and Subnets and Example Network Resource Configurations.

Internet Gateway Configuration

If you intend to use public subnets (for worker nodes, load balancers, or the Kubernetes API endpoint) and the subnets require access to/from the internet, the VCN must have an internet gateway. The internet gateway must be specified as the target for the destination CIDR block 0.0.0.0/0 as a route rule in a route table.

See VCNs and Subnets and Example Network Resource Configurations.

NAT Gateway Configuration

If you intend to use private subnets (for worker nodes or the Kubernetes API endpoint) and the subnets require access to the internet, the VCN must have a NAT gateway. For example, if you expect deployed applications to download software or to access third party services.

The NAT gateway must be specified as the target for the destination CIDR block 0.0.0.0/0 as a route rule in a route table.

See NAT Gateway and Example Network Resource Configurations.

Service Gateway Configuration

If you intend to use private subnets for worker nodes or the Kubernetes API endpoint, the VCN must have a service gateway.

Create the service gateway in the same VCN and compartment as the worker nodes subnet and the Kubernetes API endpoint subnet, and select the All <region> Services in Oracle Services Network option.

The service gateway must be specified as the target for All <region> Services in Oracle Services Network as a route rule in a route table.

See Access to Oracle Services: Service Gateway and Example Network Resource Configurations.

Route Table Configuration

Route Table for Worker Nodes Subnets

If you intend to deploy worker nodes in a public subnet, the subnet route table must have a route rule that specifies the internet gateway as the target for the destination CIDR block 0.0.0.0/0.

If you intend to deploy worker nodes in a private subnet, the subnet route table must have::

  • a route rule that specifies the service gateway as the target for All <region> Services in Oracle Services Network
  • a route rule that specifies the NAT gateway as the target for the destination CIDR block 0.0.0.0/0

Route Table for Kubernetes API Endpoint Subnets

If you intend to deploy the Kubernetes API endpoint in a public subnet, the subnet route table must have a route rule that specifies the internet gateway as the target for the destination CIDR block 0.0.0.0/0.

If you intend to deploy the Kubernetes API endpoint in a private subnet, the subnet route table must have:

  • a route rule that specifies the service gateway as the target for All <region> Services in Oracle Services Network
  • a route rule that specifies the NAT gateway as the target for the destination CIDR block 0.0.0.0/0

Route Table for Load Balancer Subnets

If you intend to deploy load balancers in public subnets, the subnet route table must have a route rule that specifies the internet gateway as the target for the destination CIDR block 0.0.0.0/0.

If you intend to deploy load balancers in private subnets, the subnet route table can be empty.

Notes about Route Table Configuration

DHCP Options Configuration

The VCN in which you want to create and deploy clusters must have DHCP Options configured. The default value for DNS Type of Internet and VCN Resolver is acceptable.

See DHCP Options and Example Network Resource Configurations.

Security Rule Configuration in Security Lists and/or Network Security Groups

The VCN in which you want to create and deploy clusters must have security rules defined. You can define the security rules in either or both the following ways:

  • In Security lists that are already associated with the subnets that you select for the worker nodes, for the Kubernetes API endpoint, and for load balancers (if specified) when you create a cluster.
  • In Network security groups that you select for node pools, for the Kubernetes API endpoint, and for load balancers (if specified) when you create a cluster.

The worker nodes, Kubernetes API endpoint, and load balancer have different security rule requirements, as described in this topic. You can add additional rules to meet your specific needs.

If you specify security rules in both security lists and network security groups, all the security rules are applied (that is, the union of the security rules).

For more information, see:

Security Rules for the Kubernetes API Endpoint

The following ingress rules must be defined for the Kubernetes API endpoint, in a security list and/or in a network security group:

State Source Protocol/Dest. Port Description
Stateful Worker Nodes CIDR TCP/6443 Kubernetes worker to Kubernetes API endpoint communication.
Stateful Worker Nodes CIDR TCP/12250 Kubernetes worker to control plane communication.
Stateful Worker Nodes CIDR ICMP 3,4 Path Discovery.

The following optional ingress rules can be defined for the Kubernetes API endpoint, in a security list and/or in a network security group:

State Source Protocol/Dest. Port Description
Stateful 0.0.0.0/0 or specific subnets TCP/6443 Client access to Kubernetes API endpoint

The following egress rules must be defined for the Kubernetes API endpoint, in a security list and/or in a network security group:

State Destination Protocol/Dest. Port Description
Stateful All <region> Services in Oracle Services Network TCP/443 Allow Kubernetes control plane to communicate with OKE.
Stateful Worker Nodes CIDR TCP/ALL All traffic to worker nodes.
Stateful Worker Nodes CIDR ICMP 3,4 Path Discovery.

Security Rules for Worker Nodes

Worker nodes are created with public or private IP addresses, according to whether you specify public or private subnets when defining the node pools in a cluster. Container Engine for Kubernetes must be able to access worker nodes.

The following ingress rules must be defined for worker nodes, in a security list and/or in a network security group:

State Source Protocol/Dest. Port Description
Stateful Worker Nodes CIDR ALL/ALL Allow pods on one worker node to communicate with pods on other worker nodes.
Stateful Kubernetes API Endpoint CIDR TCP/ALL Allow Kubernetes control plane to communicate with worker nodes.
Stateful 0.0.0.0/0 ICMP 3,4 Path Discovery.

The following optional ingress rules can be defined for worker nodes, in a security list and/or in a network security group:

State Source Protocol/Dest. Port Description
Stateful 0.0.0.0/0 or subnet CIDR TCP/22 (optional) Allow inbound SSH traffic to worker nodes.

The following egress rules must be defined for worker nodes, in a security list and/or in a network security group:

State Destination Protocol/Dest. Port Description
Stateful Worker Nodes CIDR ALL/ALL Allow pods on one worker node to communicate with pods on other worker nodes.
Stateful 0.0.0.0/0 ICMP 3,4 Path Discovery.
Stateful All <region> Services in Oracle Services Network TCP/ALL Allow nodes to communicate with OKE.
Stateful Kubernetes API Endpoint CIDR TCP/6443 Kubernetes worker to Kubernetes API endpoint communication.
Stateful Kubernetes API Endpoint CIDR TCP/12250 Kubernetes worker to control plane communication.

The following optional egress rules can be defined for worker nodes, in a security list and/or in a network security group:

State Destination Protocol/Dest. Port Description
Stateful 0.0.0.0/0 TCP/ALL (optional) Allow worker nodes to communicate with internet.

Security Rules for Load Balancers

Load balancers do not require pre-defined security rules.

The Kubernetes Cloud Controller Manager sets security rules automatically when Kubernetes services are deployed.

When deploying a service of type LoadBalancer, you can optionally:

Subnet Configuration

The characteristics of the cluster you want to create will determine the number of subnets to configure. Best practice is to use regional subnets to make failover across availability domains simpler to implement.

The VCN in which you want to create and deploy clusters must have at least two (optionally, three) different subnets:

  • a Kubernetes API endpoint subnet
  • a worker nodes subnet
  • (optionally) one or two load balancer subnets

Although you can choose to combine the subnets, and also to combine security rules, this approach is not recommended because it makes security harder to manage.

The subnet CIDR blocks must not overlap with CIDR blocks you specify for pods running in the cluster (see CIDR Blocks and Container Engine for Kubernetes).

The subnets must be configured with the following properties:

See VCNs and Subnets and Example Network Resource Configurations.

Kubernetes API Endpoint Subnet Configuration

The Kubernetes API endpoint subnet hosts an endpoint that provides access to the Kubernetes API on the cluster control plane.

The Kubernetes API endpoint subnet must be a regional subnet, and can be a private or a public subnet:

  • If you specify a public subnet for the Kubernetes API endpoint, you can optionally assign a public IP address to the Kubernetes API endpoint subnet. The public IP address enables third party cloud services (such as SaaS CI/CD services) to access the Kubernetes API endpoint.
  • If you specify a private subnet for the Kubernetes API endpoint, traffic can access the Kubernetes API endpoint subnet from another subnet in your VCN, from another VCN, or from your on-premise network (peered with FastConnect). You can also set up a bastion host on a public subnet to reach the Kubernetes API endpoint.

Worker Node Subnet Configuration

A worker node subnet hosts the worker nodes in a node pool.

The worker node subnet can be either a single regional subnet or multiple AD-specific subnets (one in each of the availability domains).

The worker node subnet can be either a public subnet or a private subnet. However, we recommend the worker node subnet is a private subnet for maximum security.

Load Balancer Subnet Configuration

Load balancer subnet(s) connect Oracle Cloud Infrastructure load balancers created by Kubernetes services of type LoadBalancer.

The load balancer subnets can be single regional subnets or AD-specific subnets (one in each of the availability domains). However, we recommend regional subnets.

The load balancer subnets can be either public or private subnets, depending on how applications deployed on the cluster will be accessed. If applications will be accessed internally from within your VCN, use private subnets for the load balancer subnets. If applications will be accessed from the internet, use public subnets for the load balancer subnets.